Eval bug: the vector of jina-embeddings-v2-base-zh retrieval result is incorrect.

Name and Version

./build/bin/llama-cli --version
version: 7449 (6853bee)
built with GNU 11.4.0 for Linux x86_64

Operating systems

Linux

GGML backends

CPU

Hardware

AMD EPYC 9K84 96-Core Processor

Models

jina-embeddings-v2-base-zh

Problem description & steps to reproduce

1.Convert HF model to GGUF model：
/app/jczhao/miniconda3/envs/llamacpp/bin/python convert_hf_to_gguf.py
/app/jczhao/jina-embeddings-v2-base-zh
--outfile /app/jczhao/jina-embeddings-v2-base-zh-f32.gguf
--outtype f32
--metadata "general.architecture=encoder"
--metadata "encoder.context_length=8192"
--metadata "general.name=jina-embeddings-v2-base-zh"
--verbose
2.Llama server deployment command：
sudo nohup ./build/bin/llama-server -m /app/jczhao/jina-embeddings-v2-base-zh-f32.gguf -t 24 -c 8192 -b 512 --host 0.0.0.0 --port 7868 --embeddings --cont-batching --no-mmap > llama-server-7.log 2>&1 &
Compared to the official HF model and ONNX model,the vector of jina-embeddings-v2-base-zh gguf model retrieval result is incorrect!
Please help me solve the issue,thanks!

First Bad Commit

No response

Relevant log output

Logs

@zhaojc001

Hi @zhaojc001, Could you provide the cosine similarity score between the embeddings from llama.cpp and the Python implementation?

embedding_similarity_result.json

@o7si

Hello @o7si Please take a detailed look and fix the issue as soon as possible. let me know if you need any further assistance.

This could be a tokenizer issue. Let me take some time to look into it.

This is the code of jina v2 onnx server,to help fix and solve the issue.

@CISC

I've done some investigation on this issue and believe the problem might be related to the tokenizer implementation.

The jina-embeddings-v2-base-zh model uses a simple Whitespace pre-tokenizer in its tokenizer.json:

"pre_tokenizer": {
  "type": "Whitespace"
},

This means the vocabulary stores tokens as raw Unicode (e.g., "你好" → token 49226).

However, llama.cpp's BPE tokenizer always applies GPT-2 byte encoding via unicode_byte_encoding_process() in src/unicode.cpp:

llama.cpp/src/unicode.cpp

Lines 223 to 239 in 9ac2693

    
           static std::vector<std::string> unicode_byte_encoding_process(const std::vector<std::string> & bpe_words) { 
        
               std::vector<std::string> bpe_encoded_words; 
        
               for (const auto & word : bpe_words) { 
        
                   std::string text_utf; 
        
                   auto utf_word =  unicode_cpts_from_utf8(word); 
        
                   for (size_t i = 0; i < utf_word.size(); ++i) { 
        
                       text_utf += unicode_cpt_to_utf8(utf_word[i]); 
        
                   } 
        
                   std::string encoded_token; 
        
                   for (char & c : text_utf) { 
        
                       encoded_token += unicode_byte_to_utf8(c); 
        
                   } 
        
                   bpe_encoded_words.emplace_back(encoded_token); 
        
               } 
        
               return bpe_encoded_words; 
        
           }

This works correctly for models with ByteLevel pre-tokenizer (like Qwen, LLaMA), where the vocabulary stores byte-encoded tokens (e.g., "ä½łå¥½" → token 108386).

But for Jina, after byte encoding, "你好" becomes "ä½łå¥½", which doesn't exist in the vocabulary, causing fallback to single-byte tokens.

I'm considering adding a GGUF metadata field (set during convert_hf_to_gguf.py) to control ByteLevel encoding, but there might be a cleaner approach.

Hi @CISC, suggestions for a more elegant approach?

Expected output (HuggingFace tokenizer)

$ git clone https://huggingface.co/jinaai/jina-embeddings-v2-base-zh 
$ cat << 'EOF' > test_jina_tokenizer.py
from transformers import AutoTokenizer

tokenizer = AutoTokenizer.from_pretrained("jina-embeddings-v2-base-zh")
ids = tokenizer.encode("你好")
for id in ids:
    print(id, tokenizer.decode(id))
EOF
$ python test_jina_tokenizer.py
0 <s>
49226 你好
2 </s>

Actual output (llama.cpp tokenizer)

$ git clone https://huggingface.co/jinaai/jina-embeddings-v2-base-zh 
$ python convert_hf_to_gguf.py jina-embeddings-v2-base-zh --outtype f32
$ ./build/bin/llama-tokenize -m jina-embeddings-v2-base-zh/Jina-Bert-Implementation-160M-F32.gguf -p "你好"
ggml_metal_device_init: tensor API disabled for pre-M5 and pre-A19 devices
ggml_metal_library_init: using embedded metal library
ggml_metal_library_init: loaded in 0.032 sec
ggml_metal_rsets_init: creating a residency set collection (keep_alive = 180 s)
ggml_metal_device_init: GPU name:   Apple M4
ggml_metal_device_init: GPU family: MTLGPUFamilyApple9  (1009)
ggml_metal_device_init: GPU family: MTLGPUFamilyCommon3 (3003)
ggml_metal_device_init: GPU family: MTLGPUFamilyMetal3  (5001)
ggml_metal_device_init: simdgroup reduction   = true
ggml_metal_device_init: simdgroup matrix mul. = true
ggml_metal_device_init: has unified memory    = true
ggml_metal_device_init: has bfloat            = true
ggml_metal_device_init: has tensor            = false
ggml_metal_device_init: use residency sets    = true
ggml_metal_device_init: use shared buffers    = true
ggml_metal_device_init: recommendedMaxWorkingSetSize  = 11453.25 MB
llama_model_load_from_file_impl: using device Metal (Apple M4) (unknown id) - 10922 MiB free
llama_model_loader: loaded meta data with 33 key-value pairs and 184 tensors from jina-embeddings-v2-base-zh/Jina-Bert-Implementation-160M-F32.gguf (version GGUF V3 (latest))
llama_model_loader: Dumping metadata keys/values. Note: KV overrides do not apply in this output.
llama_model_loader: - kv   0:                       general.architecture str              = jina-bert-v2
llama_model_loader: - kv   1:                               general.type str              = model
llama_model_loader: - kv   2:                               general.name str              = Jina Bert Implementation
llama_model_loader: - kv   3:                       general.organization str              = Jinaai
llama_model_loader: - kv   4:                         general.size_label str              = 160M
llama_model_loader: - kv   5:                            general.license str              = apache-2.0
llama_model_loader: - kv   6:                               general.tags arr[str,6]       = ["sentence-transformers", "feature-ex...
llama_model_loader: - kv   7:                          general.languages arr[str,2]       = ["en", "zh"]
llama_model_loader: - kv   8:                   jina-bert-v2.block_count u32              = 12
llama_model_loader: - kv   9:                jina-bert-v2.context_length u32              = 8192
llama_model_loader: - kv  10:              jina-bert-v2.embedding_length u32              = 768
llama_model_loader: - kv  11:           jina-bert-v2.feed_forward_length u32              = 3072
llama_model_loader: - kv  12:          jina-bert-v2.attention.head_count u32              = 12
llama_model_loader: - kv  13:  jina-bert-v2.attention.layer_norm_epsilon f32              = 0.000000
llama_model_loader: - kv  14:                          general.file_type u32              = 0
llama_model_loader: - kv  15:              jina-bert-v2.attention.causal bool             = false
llama_model_loader: - kv  16:                  jina-bert-v2.pooling_type u32              = 1
llama_model_loader: - kv  17:               general.quantization_version u32              = 2
llama_model_loader: - kv  18:                       tokenizer.ggml.model str              = gpt2
llama_model_loader: - kv  19:                         tokenizer.ggml.pre str              = jina-v1-en
llama_model_loader: - kv  20:                      tokenizer.ggml.tokens arr[str,61056]   = ["<s>", "<pad>", "</s>", "<unk>", "<m...
llama_model_loader: - kv  21:                  tokenizer.ggml.token_type arr[i32,61056]   = [3, 3, 3, 3, 3, 1, 1, 1, 1, 1, 1, 1, ...
llama_model_loader: - kv  22:                      tokenizer.ggml.merges arr[str,39382]   = ["t h", "i n", "a n", "e r", "th e", ...
llama_model_loader: - kv  23:                tokenizer.ggml.bos_token_id u32              = 0
llama_model_loader: - kv  24:                tokenizer.ggml.eos_token_id u32              = 2
llama_model_loader: - kv  25:            tokenizer.ggml.unknown_token_id u32              = 3
llama_model_loader: - kv  26:          tokenizer.ggml.seperator_token_id u32              = 2
llama_model_loader: - kv  27:            tokenizer.ggml.padding_token_id u32              = 1
llama_model_loader: - kv  28:               tokenizer.ggml.mask_token_id u32              = 4
llama_model_loader: - kv  29:               tokenizer.ggml.add_bos_token bool             = true
llama_model_loader: - kv  30:               tokenizer.ggml.add_eos_token bool             = true
llama_model_loader: - kv  31:               tokenizer.ggml.add_sep_token bool             = true
llama_model_loader: - kv  32:            tokenizer.ggml.token_type_count u32              = 2
llama_model_loader: - type  f32:  184 tensors
print_info: file format = GGUF V3 (latest)
print_info: file type   = all F32
print_info: file size   = 611.20 MiB (32.00 BPW) 
load: empty token at index 5
init_tokenizer: initializing tokenizer for type 2
load: model vocab missing newline token, using special_pad_id instead
load: 0 unused tokens
load: control token:      0 '<s>' is not marked as EOG
load: control token:      4 '<mask>' is not marked as EOG
load: control token:      1 '<pad>' is not marked as EOG
load: control token:      3 '<unk>' is not marked as EOG
load: control token:      2 '</s>' is not marked as EOG
load: special_eos_id is not in special_eog_ids - the tokenizer config may be incorrect
load: printing all EOG tokens:
load:   - 2 ('</s>')
load: special tokens cache size = 5
load: token to piece cache size = 1.5060 MB
print_info: arch             = jina-bert-v2
print_info: vocab_only       = 1
print_info: no_alloc         = 0
print_info: model type       = ?B
print_info: model params     = 160.22 M
print_info: general.name     = Jina Bert Implementation
print_info: vocab type       = BPE
print_info: n_vocab          = 61056
print_info: n_merges         = 39382
print_info: BOS token        = 0 '<s>'
print_info: EOS token        = 2 '</s>'
print_info: UNK token        = 3 '<unk>'
print_info: SEP token        = 2 '</s>'
print_info: PAD token        = 1 '<pad>'
print_info: MASK token       = 4 '<mask>'
print_info: EOG token        = 2 '</s>'
print_info: max token length = 45
llama_model_load: vocab only - skipping tensors
llama_context: constructing llama_context
llama_context: n_seq_max     = 1
llama_context: n_ctx         = 512
llama_context: n_ctx_seq     = 512
llama_context: n_batch       = 512
llama_context: n_ubatch      = 512
llama_context: causal_attn   = 1
llama_context: flash_attn    = auto
llama_context: kv_unified    = false
llama_context: freq_base     = 0.0
llama_context: freq_scale    = 1
llama_context: n_ctx_seq (512) > n_ctx_train (0) -- possible training context overflow
     0 -> '<s>'
   168 -> '�'
   159 -> '�'
   224 -> '�'
   169 -> '�'
   135 -> '�'
   159 -> '�'
     2 -> '</s>'

I'm considering adding a GGUF metadata field (set during convert_hf_to_gguf.py) to control ByteLevel encoding, but there might be a cleaner approach.

Hi @CISC, suggestions for a more elegant approach?

This is technically a very basic, but unsupported tokenizer, perhaps it should just be treated as one (ie, set tokenizer.ggml.model to whitespace)?

@zhaojc001

Hi @zhaojc001, I've submitted a PR (#18756) for this. The implementation approach might still have some issues, but my current results align with the Python tokenizer. Give it a try if you'd like!

@o7si

@o7si Thank you very much.

This issue was closed because it has been inactive for 14 days since being marked as stale.

Navigation Menu

Repository navigation

Eval bug: the vector of jina-embeddings-v2-base-zh retrieval result is incorrect. #18452

Name and Version

Operating systems

GGML backends

Hardware

Models

Problem description & steps to reproduce

First Bad Commit

Relevant log output

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

	static std::vector<std::string> unicode_byte_encoding_process(const std::vector<std::string> & bpe_words) {
	std::vector<std::string> bpe_encoded_words;
	for (const auto & word : bpe_words) {
	std::string text_utf;
	auto utf_word = unicode_cpts_from_utf8(word);
	for (size_t i = 0; i < utf_word.size(); ++i) {
	text_utf += unicode_cpt_to_utf8(utf_word[i]);
	}

	std::string encoded_token;
	for (char & c : text_utf) {
	encoded_token += unicode_byte_to_utf8(c);
	}
	bpe_encoded_words.emplace_back(encoded_token);
	}
	return bpe_encoded_words;
	}

Eval bug: the vector of jina-embeddings-v2-base-zh retrieval result is incorrect. #18452

Description

Name and Version

Operating systems

GGML backends

Hardware

Models

Problem description & steps to reproduce

First Bad Commit

Relevant log output

Activity

o7si commented on Dec 31, 2025

zhaojc001 commented on Jan 7, 2026

zhaojc001 commented on Jan 7, 2026

o7si commented on Jan 8, 2026

zhaojc001 commented on Jan 9, 2026

o7si commented on Jan 10, 2026

o7si commented on Jan 10, 2026

CISC commented on Jan 10, 2026

o7si commented on Jan 11, 2026

zhaojc001 commented on Jan 13, 2026

github-actions commented on Feb 28, 2026

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions