Sparse Embeddings (BM25) and Persian Language Support

[alibaghernejad]

Hello,
I would like to ask about the limitations of sparse embeddings, specifically BM25 when it comes to language support, and in particular Persian (Farsi).

supported_languages = [
    "arabic",
    "danish",
    "dutch",
    "english",
    "finnish",
    "french",
    "german",
    "greek",
    "hungarian",
    "italian",
    "norwegian",
    "portuguese",
    "romanian",
    "russian",
    "spanish",
    "swedish",
    "tamil",
    "turkish",
]

Are the limitations mainly related to stop word handling, tokenization, or other language-specific preprocessing steps?

If the limitation is primarily related to stop words or preprocessing, is there a recommended way to extend or customize BM25 for Persian?

I am eager to contribute and would be happy to provide the necessary data (e.g., stop word lists, etc.) to enable better Persian support.

https://github.com/qdrant/fastembed/blob/main/fastembed/sparse/bm25.py
https://huggingface.co/Qdrant/bm25/tree/main

Thanks for your guidance!

[Debankur04]

Hi @alibaghernejad I am planning to check the issue and fix it if possible. It would be great if you can assign this to me. Thanks

[Debankur04]

Hi! I’ve implemented a solution for this issue in PR #684: PR #684

It adds Persian BM25 support with lightweight preprocessing, Persian stopword handling, and accompanying tests.

Would appreciate it if you could take a look when you get a chance. Feedback is very welcome!