Repository navigation

[Tokenizers] Port CodeGenTokenizer & byte-level BPE algorithm #6992

Closed

Assignees

Labels

Milestone

opened

Port Codegen Tokenizer to enable Phi-2 models

Reference:
https://huggingface.co/docs/transformers/main/en/model_doc/codegen
https://github.com/huggingface/transformers/blob/0549000c5bf6c7249f411917f2a6f0b6d0f06da1/src/transformers/models/codegen/tokenization_codegen.py#L98

Paper:
https://arxiv.org/abs/2203.13474
https://arxiv.org/pdf/2203.13474.pdf

ericstj
added
enhancementNew feature or request
on Feb 8, 2024
ghost added
untriagedNew issue has not been triaged
on Feb 8, 2024
ericstj
added this to the ML.NET 4.0 milestone on Feb 8, 2024
ghost removed
untriagedNew issue has not been triaged
on Feb 8, 2024
ericstj
added
P1
on Feb 8, 2024
ericstj
assigned
tarekgh
on Feb 13, 2024
tarekgh commented on Jun 7, 2024
tarekgh
on Jun 7, 2024
Member
This is already done!
tarekgh
closed this as completed on Jun 7, 2024
github-actions
locked and limited conversation to collaborators on Jul 7, 2024

to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Assignees

tarekgh

Labels

Type

No type

Projects

No projects

Milestone

ML.NET 4.0

Relationships

None yet

Development

No branches or pull requests