Repository navigation
Implement Sentencepiece Unigram tokenizer #7186
Description
Is your feature request related to a problem? Please describe.
I want to use a multilingual model from Huggingface ( https://huggingface.co/intfloat/multilingual-e5-large ) and the tokenizer is a sentencepiece unigram tokenizer, so I am unable to port it to C#/ONNX
Describe the solution you'd like
Support for the unigram sentencepiece tokenizer in the Microsoft.ML.Tokenizers package.
Describe alternatives you've considered
Blingfire, but seems not maintained anymore and unclear if it would return exactly the same token-id's.
Thank you for your time and effort (the library in general is great!)
Activity
ericstj commentedon Aug 5, 2024
@tarekgh do any of our existing tokenizers support this, or is this new work?
tarekgh commentedon Aug 5, 2024
This is a new model that needs to implement.