Repository navigation
fasttext quantization documentation is confusing/misleading #108
Description
https://docs.rspamd.com/modules/neural/ says
# Download and extract a full model
wget https://dl.fbaipublicfiles.com/fasttext/vectors-crawl/cc.en.300.bin.gz
gunzip cc.en.300.bin.gz
# Quantize to reduce size and dimensions
./fasttext quantize -input cc.en.300 -output cc.en.300.q
but:
$ fasttext quantize -input cc.en.300 -output cc.en.300.q
terminate called after throwing an instance of 'std::invalid_argument'
what(): cc.en.300.q.bin cannot be opened for loading!
Aborted (core dumped)
This is with
ii fasttext 0.9.2+ds-7build1 arm64 Efficient learning of word representations and sentence classification library
My assumption is that the CLI changed since the docs were written. But then this version of fasttext is 6 years old. So maybe the docs are just totally wrong?
https://www.facebook.com/groups/1174547215919768/posts/1668216449886173/ lead me to try:
$ fasttext quantize -input cc.en.300 -output cc.en.300
terminate called after throwing an instance of 'std::invalid_argument'
what(): For now we only support quantization of supervised models
Aborted (core dumped)
... research this leads to Gemini saying stuff like "Because cc.en.300.bin is a set of raw word vectors (unsupervised), it lacks the classification headers required by the quantize utility." and at this point I have to throw my hands up and admit I have no earthly idea what any of this means!
Activity