Skip to content

o3 and o3-mini support for tokenizer #7375

Closed
#7394
Closed
o3 and o3-mini support for tokenizer#7375
#7394
Assignees
tarekgh
Labels
enhancementNew feature or requestin-prA pull request is open to resolve this issue
Milestone
ML.NET 5.0

Description

@Peter-Juhasz
Peter-Juhasz
opened on Feb 2, 2025

Is your feature request related to a problem? Please describe.
Related to openai/tiktoken#374

Describe the solution you'd like
TiktokenTokenizer.CreateForModel support for new models

Describe alternatives you've considered
Configure encoding and tokenizer manually.

Additional context

Activity

  1. Peter-Juhasz
    added
    enhancementNew feature or request
    on Feb 2, 2025
  2. dotnet-policy-service
    added
    untriagedNew issue has not been triaged
    on Feb 2, 2025
  3. tarekgh
    added this to the ML.NET 5.0 milestone on Feb 3, 2025
  4. tarekgh
    added
    Tokenizers
    and removed
    untriagedNew issue has not been triaged
    on Feb 3, 2025
  5. tarekgh
    self-assigned this
    on Feb 3, 2025
  6. dotnet-policy-service
    added
    in-prA pull request is open to resolve this issue
    on Feb 21, 2025
  7. tarekgh
    closed this as completedin #7394on Feb 21, 2025
  8. github-actions
    locked and limited conversation to collaborators on Mar 23, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

enhancementNew feature or requestin-prA pull request is open to resolve this issue

Type

No type

Projects

No projects

    Milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions