Repository navigation
Unofficial bindings / ports in other languages #97
Description
The following projects are not maintained by OpenAI. I cannot vouch that any of them are correct or safe to use. Use at your own risk.
Note that if a tokeniser fails to exactly match tiktoken's behaviour, you may get worse results when sampling from models, with no warning.
Javascript
- https://github.com/dqbd/tiktoken
- https://github.com/ceifa/tiktoken-node
- https://github.com/niieani/gpt-tokenizer
- The gpt-3-encoder package will work for most GPT-3 models. However, it will often appear to work for Codex or GPT-3.5 while actually being out of distribution, and will not at all work for GPT-4 or embeddings models.
Rust
Java
- https://github.com/eisber/tiktoken
- https://github.com/knuddelsgmbh/jtokkit
Ruby
C#
Go
- https://github.com/tiktoken-go/tokenizer
- https://github.com/pkoukk/tiktoken-go
PHP
Kotlin
Thanks to everyone for building useful things!
I'm happy to link to other projects in this comment.
Reacted by Baptiste Lombard, Zhou Fang, oneice, jiacheo, Roger Zurawicki, Dan Vanderboom and Mouaad Aallam
Activity
[-]Unofficial bindings in other languages[/-][+]Unofficial bindings / ports in other languages[/+]bluescreen10 commentedon Apr 6, 2023
👋,
I built a port for go that you can find in the link below
https://github.com/tiktoken-go/tokenizer
fang2hou commentedon Apr 9, 2023
I am currently using a another port in Go.
https://github.com/pkoukk/tiktoken-go
rex-remind101 commentedon May 13, 2023
Hello @hauntsaninja , I was looking at https://github.com/openai/tiktoken/blob/main/src/lib.rs and it appears to be written in Rust. Could this be open sourced into a crate of its own?
hauntsaninja commentedon May 13, 2023
See the FAQ #98
danielcompton commentedon May 15, 2023
@hauntsaninja would it be possible to publish the full test suite publicly? That would make it easier to tell whether a given implementation matches (or is close to) the official implementation.
niieani commentedon Jun 1, 2023
Here's a pure JavaScript / TypeScript port of tiktoken: https://github.com/niieani/gpt-tokenizer
Playground online: https://gpt-tokenizer.dev
ShylockWu commentedon Sep 4, 2023
niieani commentedon Sep 5, 2023
@ShylockWu they're not incorrect. You've set gpt-tokenizer to tokenize using GPT-3.5/GPT-4 encoding, whereas the official openAI token calculator uses the older GPT-3. If you switch the playground to use the older model, you'll get the same result.
danny50610 commentedon Sep 11, 2023
👋
I ported a version of PHP, link here
https://github.com/danny50610/bpe-tokeniser
aallam commentedon Oct 11, 2023
I have built and published a port for Kotlin: https://github.com/aallam/ktoken :)
Gabriella439 commentedon Aug 31, 2024
Pure Haskell implementation of tiktoken: https://hackage.haskell.org/package/tiktoken
riyas686 commentedon Sep 17, 2026