Skip to content

llama : restore prefix space in llama tokenizer - #4081

Merged
cebtenzzre merged 2 commits into
masterfrom
ceb/restore-prefix-space
Nov 15, 2023
Merged

llama : restore prefix space in llama tokenizer#4081
cebtenzzre merged 2 commits into
masterfrom
ceb/restore-prefix-space

do not add space prefix if the first token is special

5e89942
Select commit
Loading
Failed to load commit list.

Workflow runs completed with no jobs