You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Reacted by John Iburg, Daniel Faust, ericatdropzone, Vittorio "LapsuS" Banfi, Taiga Nitta, aleivano, Djehouty, Sam Yamashita, Denis Palnitsky, kashankrm and 12 more
Based on what i know it is the same tokenizer as gpt 4o. they just added some extra stuff to it.
I did a small test like this:
fromopenaiimportOpenAIclient=OpenAI()
defgenerate_response(prompt: str, model: str="o3-mini") ->str:
print(f"Generating response for {model} model...")
response=client.chat.completions.create(
model=model,
messages=[
{"role": "user", "content": prompt}
],
)
# print(response.choices[0].message.content)print("input tokens:", response.usage.prompt_tokens)
print("reasoning tokens:", response.usage.completion_tokens_details.reasoning_tokens)
print("output tokens:", response.usage.completion_tokens-response.usage.completion_tokens_details.reasoning_tokens)
print("\n ---------------- \n")
returnresponse.choices[0].message.contentmessages= [
"Complete this sentence: The quick brown fox jumps",
"kajsh ekr jn as kemnralsekjr la sekjrl skejlsmcelkamc .skenrs",
"code, make sure you have installed the OpenAI Python"
]
formessageinmessages:
print("testing for message:", message)
generate_response(message)
generate_response(message, model="o1")
generate_response(message, model="o1-mini")
generate_response(message, model="o1-preview")
generate_response(message, model="gpt-4o")
here is the output:
testing for message: Complete this sentence: The quick brown fox jumps
Generating response for o3-mini model...
input tokens: 15
reasoning tokens: 192
output tokens: 20
----------------
Generating response for o1 model...
input tokens: 15
reasoning tokens: 256
output tokens: 21
----------------
Generating response for o1-mini model...
input tokens: 17
reasoning tokens: 128
output tokens: 22
----------------
Generating response for o1-preview model...
input tokens: 17
reasoning tokens: 128
output tokens: 29
----------------
Generating response for gpt-4o model...
input tokens: 16
reasoning tokens: 0
output tokens: 6
----------------
testing for message: kajsh ekr jn as kemnralsekjr la sekjrl skejlsmcelkamc .skenrs
Generating response for o3-mini model...
input tokens: 34
reasoning tokens: 128
output tokens: 42
----------------
Generating response for o1 model...
input tokens: 34
reasoning tokens: 384
output tokens: 56
----------------
Generating response for o1-mini model...
input tokens: 36
reasoning tokens: 192
output tokens: 41
----------------
Generating response for o1-preview model...
input tokens: 36
reasoning tokens: 320
output tokens: 39
----------------
Generating response for gpt-4o model...
input tokens: 35
reasoning tokens: 0
output tokens: 29
----------------
testing for message: code, make sure you have installed the OpenAI Python
Generating response for o3-mini model...
input tokens: 17
reasoning tokens: 320
output tokens: 326
----------------
Generating response for o1 model...
input tokens: 17
reasoning tokens: 320
output tokens: 434
----------------
Generating response for o1-mini model...
input tokens: 19
reasoning tokens: 448
output tokens: 622
----------------
Generating response for o1-preview model...
input tokens: 19
reasoning tokens: 896
output tokens: 403
----------------
Generating response for gpt-4o model...
input tokens: 18
reasoning tokens: 0
output tokens: 428
----------------
notice that o3-mini and o1 have same input tokens, o1-preview and o1-mini have same and then we have gpt-4o. And all of them are offset by few tokens. I also did a similar test with a system prompt and there the difference was about 9 tokens, but always predictable.
Reacted by Wei Xu, Gabriel Romero, Terry Tao and aleivano
This is a pretty famous PIP library for tons of people why don't you just go through in the code and explicitly define parameters for every open a I model and the second there's news that open a I released a new model just find out the pricing and update your library.... If you're really busy maybe you could research on GPT how to advertise or promote your other interests along the way to make that possible... Using your famous Github page
But in response to the other guy 03 mini and o1-mini don't have the same tokens it seems weird To have to sit here and do math... when it's easy to implement into the library but 03 mini and o1-mini have the same tokens input and output.... I mean there's this complex library I don't know all the parameters of it but it seems like there should be somewhere where you could explicitly define that 03 mini has $1.10 for input tokens and $4.40 for output tokens and just make a dictionary based on that and then Design your code around that dictionary so whenever there's like a new model or they change the prices then you just gotta update the price in the dictionary and then the code would just use that...
This is a pretty famous PIP library for tons of people why don't you just go through in the code and explicitly define parameters for every open a I model and the second there's news that open a I released a new model just find out the pricing and update your library.... If you're really busy maybe you could research on GPT how to advertise or promote your other interests along the way to make that possible... Using your famous Github page
Is that supposed to be in response to me? If so, I have no idea what this word salad is supposed to convey.
OKI just took 5 minutes to do the math on the function and it appears when I give it the oh one mini model it outputs the same price as the 03 mini model so I'm just going to make all my code whenever he needs to calculate 03 many tokens use the OH-1 mini model parameter
On Thu, Feb 13, 2025 at 7:06 AM Roberto Anić Banić ***@***.***> wrote:
This is a pretty famous PIP library for tons of people why don't you just
go through in the code and explicitly define parameters for every open a I
model and the second there's news that open a I released a new model just
find out the pricing and update your library.... If you're really busy
maybe you could research on GPT how to advertise or promote your other
interests along the way to make that possible... Using your famous Github
page
Is that supposed to be in response to me? If so, I have no idea what this
word salad is supposed to convey.
—
Reply to this email directly, view it on GitHub
<#374 (comment)>,
or unsubscribe
<https://github.com/notifications/unsubscribe-auth/AWFQGIOA7JTRP62PIWOLJRT2PSYIFAVCNFSM6AAAAABWIKKQRCVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDMNJWHA4TAMJZGM>
.
You are receiving this because you commented.Message ID:
***@***.***>
[image: Nicba1010]*Nicba1010* left a comment (openai/tiktoken#374)
<#374 (comment)>
This is a pretty famous PIP library for tons of people why don't you just
go through in the code and explicitly define parameters for every open a I
model and the second there's news that open a I released a new model just
find out the pricing and update your library.... If you're really busy
maybe you could research on GPT how to advertise or promote your other
interests along the way to make that possible... Using your famous Github
page
Is that supposed to be in response to me? If so, I have no idea what this
word salad is supposed to convey.
—
Reply to this email directly, view it on GitHub
<#374 (comment)>,
or unsubscribe
<https://github.com/notifications/unsubscribe-auth/AWFQGIOA7JTRP62PIWOLJRT2PSYIFAVCNFSM6AAAAABWIKKQRCVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDMNJWHA4TAMJZGM>
.
You are receiving this because you commented.Message ID:
***@***.***>
Activity
tarekgh commentedon Jan 31, 2025
Is it known which vocab file the tokenizer is going to use with this model?
kashankrm commentedon Feb 3, 2025
Based on what i know it is the same tokenizer as gpt 4o. they just added some extra stuff to it.
I did a small test like this:
here is the output:
notice that o3-mini and o1 have same input tokens, o1-preview and o1-mini have same and then we have gpt-4o. And all of them are offset by few tokens. I also did a similar test with a system prompt and there the difference was about 9 tokens, but always predictable.
blender-tv-studio commentedon Feb 13, 2025
This is a pretty famous PIP library for tons of people why don't you just go through in the code and explicitly define parameters for every open a I model and the second there's news that open a I released a new model just find out the pricing and update your library.... If you're really busy maybe you could research on GPT how to advertise or promote your other interests along the way to make that possible... Using your famous Github page
blender-tv-studio commentedon Feb 13, 2025
But in response to the other guy 03 mini and o1-mini don't have the same tokens it seems weird To have to sit here and do math... when it's easy to implement into the library but 03 mini and o1-mini have the same tokens input and output.... I mean there's this complex library I don't know all the parameters of it but it seems like there should be somewhere where you could explicitly define that 03 mini has $1.10 for input tokens and $4.40 for output tokens and just make a dictionary based on that and then Design your code around that dictionary so whenever there's like a new model or they change the prices then you just gotta update the price in the dictionary and then the code would just use that...
Nicba1010 commentedon Feb 13, 2025
Is that supposed to be in response to me? If so, I have no idea what this word salad is supposed to convey.
blender-tv-studio commentedon Feb 13, 2025
OKI just took 5 minutes to do the math on the function and it appears when I give it the oh one mini model it outputs the same price as the 03 mini model so I'm just going to make all my code whenever he needs to calculate 03 many tokens use the OH-1 mini model parameter
blender-tv-studio commentedon Feb 13, 2025
hauntsaninja commentedon Feb 14, 2025
This is resolved in tiktoken 0.9
blender-tv-studio commentedon Feb 14, 2025
tarekgh commentedon Feb 14, 2025
adding the link of mapping the model's name to the encoding file just for clarity
tiktoken/tiktoken/model.py
Line 8 in e35ab09
blender-tv-studio commentedon Feb 14, 2025