You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
I tried to determine how to implement unicode and I am not getting far. It seems to work from all I am seeing, but the output has random characters yes.
Here is a prompt in text format for easier copy/paste
I find a list of unprintable tokens from ID 131 to 258. If I remove those from vocab a prompt can generate in Japanese it seems but I dont know Japanese!
The meaning of life is that one person is one person. Since Abe was standing there, it was only possible to be one person after leaving, but that's right.
The Japanese text you quote here is fairly agrammatical, in a way that suggests (on top of some other issues that I figure are simply due to LLaMa not having learned the language very well) that some words are simply missing. Where were the unprintable tokens that you removed from this?
I removed "", "�", "��" from the grammar, not from a sentence that's not how it works. There is a large chunk of the "token dictionary" in the model that points to unprintable character �. I remove those tokens from the dictionary of tokens the program is using. I suspect the model learns some corrupted text maybe during training so if it sees japanese characters it is confusing it with some garbled text it has come across, thus making unprintable characters a likely candidate for the next word. Just my hypothesis.
Here is the pull request, the code change I made to make this work.
Due to the constently changing encoding history of CJK (Chinese, Japanese, Korean), there is big chance that the training model got wrong encoding of non-ascii language. Simply removing it is good.
Some more test shows that we can't simply remove the unprintable token. There should be some way to find the right encoding of it. otherwise the generated text becomes unreadable.
Im not sure you have applied the code change. I cannot try your prompt since its an image mind pasting? But I think you have to checkout from my repo because my code is not currently merged here yet.
Try clone / build in a different folder. It also includes the repeat penalty change. Again my approach is not about removing the characters, its a code change that will output something very different (and more accurate)
人生の意から
しようごいます。
I’ve been getting these error messages on my blog for a while now, but I didn't pay too much attention to them until recently when there were quite few. Maybe it was because last weekend (in Japan), the phone lines at my company went down and that screwed up all of our accounts that require login through Internet Explorer 7+ (which is set as default).
Unfortunately, I couldn't afford much time to fix them since they were so many. So now there are even more errors for you guys
Im not sure you have applied the code change. I cannot try your prompt since its an image mind pasting? But I think you have to checkout from my repo because my code is not currently merged here yet.
Try clone / build in a different folder. It also includes the repeat penalty change. Again my approach is not about removing the characters, its a code change that will output something very different (and more accurate)
Thank you. I build your repo and test again, the unprintable character is gone, but the meaning of generated text is gone either like bellow.
I just tried Chinese as well and yes its truncated. Its possible that it doesn't understand other languages. It seems to be missing some Chinese character tokens such as 篇 entirely!
Further up the code chain, in the model conversion code I see the following. Before I write more @ggerganov thank you so much for putting this all together. I wonder if some tokens are getting lost. But maybe not since there is 32000 tokens (and that appears to be how Google's tokenizer works). I will try to research and see if some tokens are "lost in translation"!
# Is this correct??
for i in range(32000):
# TODO: this is probably wrong - not sure how this tokenizer works
text = tokenizer.decode([29889, i]).encode('utf-8')
# remove the first byte (it's always '.')
text = text[1:]
fout.write(struct.pack("i", len(text)))
fout.write(text)
@ggerganov you are too hard on yourself. How can you be wrong when so many tokens are present :P
Activity
[-]Support prompt in Unicode[/-][+]Unicode support[/+]beiller commentedon Mar 11, 2023
I tried to determine how to implement unicode and I am not getting far. It seems to work from all I am seeing, but the output has random characters yes.
Here is a prompt in text format for easier copy/paste
./main -m ./models/13B/ggml-model-q4_0.bin -t 8 -n 128 -n 512 -p $'人生の意味は'This seems correct above since I dumped out the tokens parsing code
And the output I get is
人生の意���、フロントカードに���いてる。 2019年3月 © All Rights Reserved. [end of text]So it is outputting some characters but some �
beiller commentedon Mar 11, 2023
I find a list of unprintable tokens from ID 131 to 258. If I remove those from vocab a prompt can generate in Japanese it seems but I dont know Japanese!
llama.cpp % ./main -m ./models/13B/ggml-model-q4_0.bin -t 8 -n 128 --repeat_last_n 64 --repeat_penalty 1.0 -n 512 -p $'人生の意味は'Response
人生の意は、一人が一人ということであります。は安部が立していたので、去からは一人の人にれるのはにとどまったのですが、そうGoogle translate
The meaning of life is that one person is one person. Since Abe was standing there, it was only possible to be one person after leaving, but that's right.Is it possible?
blackhole89 commentedon Mar 12, 2023
The Japanese text you quote here is fairly agrammatical, in a way that suggests (on top of some other issues that I figure are simply due to LLaMa not having learned the language very well) that some words are simply missing. Where were the unprintable tokens that you removed from this?
beiller commentedon Mar 12, 2023
I removed "", "�", "��" from the grammar, not from a sentence that's not how it works. There is a large chunk of the "token dictionary" in the model that points to unprintable character �. I remove those tokens from the dictionary of tokens the program is using. I suspect the model learns some corrupted text maybe during training so if it sees japanese characters it is confusing it with some garbled text it has come across, thus making unprintable characters a likely candidate for the next word. Just my hypothesis.
Here is the pull request, the code change I made to make this work.
https://github.com/ggerganov/llama.cpp/pull/26/files
beiller commentedon Mar 12, 2023
For anyone interested here is the chunk in the 13B model file. Not sure if all models contain the same token grammars
Many token IDs point to
0xEFBFBDwhich is unprintable unicodewizd commentedon Mar 12, 2023
Nice find!
Due to the constently changing encoding history of CJK (Chinese, Japanese, Korean), there is big chance that the training model got wrong encoding of non-ascii language. Simply removing it is good.
wizd commentedon Mar 12, 2023
Some more test shows that we can't simply remove the unprintable token. There should be some way to find the right encoding of it. otherwise the generated text becomes unreadable.
beiller commentedon Mar 12, 2023
Im not sure you have applied the code change. I cannot try your prompt since its an image mind pasting? But I think you have to checkout from my repo because my code is not currently merged here yet.
https://github.com/beiller/llama.cpp
Try clone / build in a different folder. It also includes the repeat penalty change. Again my approach is not about removing the characters, its a code change that will output something very different (and more accurate)
beiller commentedon Mar 12, 2023
Heres some more examples:
./main -m ./models/13B/ggml-model-q4_0.bin -t 8 -p $'人生の意味は'Outputs:
wizd commentedon Mar 12, 2023
Thank you. I build your
Thank you. I build your repo and test again, the unprintable character is gone, but the meaning of generated text is gone either like bellow.
wizd commentedon Mar 12, 2023
There is another bug, truncate of prompt if it is Chinese like in #11 (comment)
beiller commentedon Mar 12, 2023
I just tried Chinese as well and yes its truncated. Its possible that it doesn't understand other languages. It seems to be missing some Chinese character tokens such as
篇entirely!Further up the code chain, in the model conversion code I see the following. Before I write more @ggerganov thank you so much for putting this all together. I wonder if some tokens are getting lost. But maybe not since there is 32000 tokens (and that appears to be how Google's tokenizer works). I will try to research and see if some tokens are "lost in translation"!
@ggerganov you are too hard on yourself. How can you be wrong when so many tokens are present :P
160 remaining items
fix(server): don't conflate mtmd capability with slot media content (…
server: do not run the token-based caches on a multimodal prompt
server: make push_back_placeholder set has_mtmd instead of asserting it
fix(server): don't conflate mtmd capability with slot media content (…
server: do not run the token-based caches on a multimodal prompt
server: make push_back_placeholder set has_mtmd instead of asserting it
llama : add gpt-oss (ggml-org#15091)
ggml webgpu: unary op suppport, code refactoring, ops support (ggml-o…
ggml webgpu: Split shared state (webgpu_context) into global state an…
server : add a `hidden` preset option to omit a model from GET /models (
fix(server): don't conflate mtmd capability with slot media content (…
server: do not run the token-based caches on a multimodal prompt
server: make push_back_placeholder set has_mtmd instead of asserting it