[Security] Reachable Rust panic (denial of service) when tokenizing with an untrusted encoder missing single-byte tokens #568

Activity

1. geo-chen commented on Jul 13, 2026

   

   geo-chen

   on Jul 13, 2026

   Author

   More actions

   Summary

   Loading an untrusted tiktoken encoder that omits some single-byte tokens and then tokenizing text causes a Rust panic. The BPE core indexes the ranks map directly, so when the input contains a byte the encoder does not cover, the lookup panics. The panic crosses the Python FFI boundary as a PanicException and aborts the in-progress encode call; in any embedding built with panic equals abort it terminates the process. Loading a user-supplied encoder or .tiktoken file and tokenizing user text are deliberate, common actions in LLM apps.

   Details

   The BPE core indexes the ranks map with the panicking index operator instead of a checked lookup. Three reachable sites in src/lib.rs:

   • line 202, byte_pair_encode, the single-piece branch: return vec![ranks[piece]];
   • line 207, byte_pair_encode, the under-100-byte path: .map(|part| ranks[&piece[part[0].0..part[1].0]])
   • line 133, _byte_pair_merge_large, the final loop: result.push(ranks[&piece[i..state[i].end]]);

   The constructor CoreBPE::new_internal (src/lib.rs:618) only asserts encoder.len() == decoder.len(); it never checks that all 256 single-byte tokens are present. The Python side is equally permissive: tiktoken/load.py:159 load_tiktoken_bpe parses any base64 rank file with no completeness check, and tiktoken/core.py:16 Encoding.__init__ performs no byte-coverage validation. The developers know the invariant: the pure-Python reference enforces it at tiktoken/_educational.py:124 ("vocab_size must be at least 256, so we can encode all bytes"), but the production Rust path does not.

   Sink: encode_ordinary / encode (src/py.rs:30/35) call byte_pair_encode, which panics on the first input byte not covered by the loaded vocab. No advisory covers this; the v0.13.0 fix addressed a different issue (super-linear merge on huge pieces), and the v0.9.0 change only improved the parse-error message.

   PoC

   pip install tiktoken
   python poc_tiktoken_panic.py
   

   poc_tiktoken_panic.py:

   import os, base64, tiktoken
   from tiktoken.load import load_tiktoken_bpe
   
   CL100K_PAT = (r"""'(?i:[sdmt]|ll|ve|re)|[^\r\n\p{L}\p{N}]?+\p{L}++|\p{N}{1,3}+"""
                 r"""| ?[^\s\p{L}\p{N}]++[\r\n]*+|\s++$|\s*[\r\n]|\s+(?!\S)|\s""")
   
   # Attacker-controlled vocab file in the standard .tiktoken format (base64 token + rank).
   # Includes only bytes 'a','b' and the merge "ab"; all other single bytes are absent.
   path = os.path.expanduser("~/evil.tiktoken")
   with open(path, "wb") as f:
       for rank, tok in enumerate((b"a", b"b", b"ab")):
           f.write(base64.b64encode(tok) + b" " + str(rank).encode() + b"\n")
   
   ranks = load_tiktoken_bpe(path)                                   # accepted, no completeness check
   enc = tiktoken.Encoding(name="evil", pat_str=CL100K_PAT, mergeable_ranks=ranks, special_tokens={})
   print("loaded; encode('ab') ->", enc.encode_ordinary("ab"))
   enc.encode_ordinary("c")   # -> pyo3_runtime.PanicException: no entry found for key

   Observed (tiktoken 0.13.0): pyo3_runtime.PanicException: no entry found for key at src/lib.rs:202:26, with the same panic confirmed at the under-100 and large paths. The benign encode_ordinary("ab") succeeds, so the encoder is otherwise functional; only an uncovered input byte triggers the crash.

   Impact

   Denial of service: a service that loads a user-supplied tokenizer or .tiktoken file and tokenizes text aborts the in-progress encode call when the input contains an uncovered byte (trivially attacker-controlled). The panic is catchable in CPython unwinding builds but still aborts the call and prints a panic to stderr; in encode_batch it surfaces as a worker exception, and in any embedding built with panic equals abort it hard-aborts the process.