Skip to content
Navigation Menu
Sign in
Appearance settings
Platform
AI CODE CREATION
GitHub Copilot
Write better code with AI
GitHub Copilot app
Direct agents from issue to merge
MCP Registry
Integrate external tools
DEVELOPER WORKFLOWS
Actions
Automate any workflow
Codespaces
Instant dev environments
Issues
Plan and track work
Code Review
Manage code changes
Code Quality
Enforce quality at merge
APPLICATION SECURITY
GitHub Advanced Security
Find and fix vulnerabilities
Code security
Secure your code as you build
Secret protection
Stop leaks before they start
EXPLORE
Why GitHub
Documentation
Blog
Changelog
Marketplace
View all features
Solutions
BY COMPANY SIZE
Enterprises
Small and medium teams
Startups
Nonprofits
BY USE CASE
App Modernization
DevSecOps
DevOps
CI/CD
View all use cases
BY INDUSTRY
Healthcare
Financial services
Manufacturing
Government
View all industries
View all solutions
Resources
EXPLORE BY TOPIC
AI
Software Development
DevOps
Security
View all topics
EXPLORE BY TYPE
Customer stories
Events & webinars
Ebooks & reports
Business insights
GitHub Skills
SUPPORT & SERVICES
Documentation
Customer support
Community forum
Trust center
Partners
View all resources
Open Source
COMMUNITY
GitHub Sponsors
Fund open source developers
PROGRAMS
Security Lab
Maintainer Community
GitHub Stars
Archive Program
REPOSITORIES
Topics
Trending
Collections
Enterprise
ENTERPRISE SOLUTIONS
Enterprise platform
AI-powered developer platform
AVAILABLE ADD-ONS
GitHub Advanced Security
Enterprise-grade security features
Copilot for Business
Enterprise-grade AI features
Premium Support
Enterprise-grade 24/7 support
Pricing
Search
/
Sign in
Sign up
Appearance settings
You signed in with another tab or window.
Reload
to refresh your session.
You signed out in another tab or window.
Reload
to refresh your session.
You switched accounts on another tab or window.
Reload
to refresh your session.
Dismiss alert
{{ message }}
huggingface
/
tokenizers
Public
Notifications
You must be signed in to change notification settings
Fork
1.2k
Star
11.1k
Code
Issues
116
Pull requests
89
Actions
Projects
Security and quality
0
Insights
Additional navigation options
Code
Issues
Pull requests
Actions
Projects
Security and quality
Insights
Actions: huggingface/tokenizers
Actions
All workflows
Workflows
Node Release
Node Release
Python Release
Python Release
Rust Release
Rust Release
atomnorm
atomnorm
atomsplit
atomsplit
Backfill documentation
Backfill documentation
bitcannon
bitcannon
Build documentation
Build documentation
Build PR Documentation
Build PR Documentation
C
C
Show more workflows...
Management
Caches
Deployments
All workflows
All workflows
Actions
Loading...
Loading
Sorry, something went wrong.
Uh oh!
There was an error while loading.
Please reload this page
.
will be ignored since log searching is not yet available
Showing runs from all workflows
will be ignored since log searching is not yet available
2,500+ workflow runs
2,500+ workflow runs
Workflow
Filter by Workflow
Sorry, something went wrong.
Filter
Loading
Sorry, something went wrong.
No matching workflows.
Event
Filter by Event
Sorry, something went wrong.
Filter
Loading
Sorry, something went wrong.
No matching events.
Status
Filter by Status
Sorry, something went wrong.
Filter
Loading
Sorry, something went wrong.
No matching statuses.
Branch
Filter by Branch
Sorry, something went wrong.
Filter
Loading
Sorry, something went wrong.
No matching branches.
Actor
Filter by Actor
Sorry, something went wrong.
Filter
Loading
Sorry, something went wrong.
No matching users.
Close stale issues and PRs
Close stale issues and PRs
#1029:
Scheduled
23s
main
main
23s
View workflow file
bpe: do not slice merge tokens at a byte offset that may not be a boundary
Build PR Documentation
#3302:
Pull request
#2398
synchronize by
apollo-2006
Action required
apollo-2006:fix-bpe-merge-prefix-slicing
apollo-2006:fix-bpe-merge-prefix-slicing
Action required
View #2398
View workflow file
bpe: do not slice merge tokens at a byte offset that may not be a boundary
Python
#5078:
Pull request
#2398
synchronize by
apollo-2006
Action required
apollo-2006:fix-bpe-merge-prefix-slicing
apollo-2006:fix-bpe-merge-prefix-slicing
Action required
View #2398
View workflow file
bpe: do not slice merge tokens at a byte offset that may not be a boundary
Node
#4916:
Pull request
#2398
synchronize by
apollo-2006
Action required
apollo-2006:fix-bpe-merge-prefix-slicing
apollo-2006:fix-bpe-merge-prefix-slicing
Action required
View #2398
View workflow file
bpe: do not slice merge tokens at a byte offset that may not be a boundary
Rust
#5933:
Pull request
#2398
synchronize by
apollo-2006
Action required
apollo-2006:fix-bpe-merge-prefix-slicing
apollo-2006:fix-bpe-merge-prefix-slicing
Action required
View #2398
View workflow file
Upload PR Documentation
Upload PR Documentation
#2539:
completed by
bhaskargurram-ai
9s
9s
View workflow file
Upload PR Documentation
Upload PR Documentation
#2538:
completed by
SBrandeis
28s
28s
View workflow file
fix(byte-level): Support sparse byte atoms
Build PR Documentation
#3301:
Pull request
#2466
synchronize by
SBrandeis
1m 22s
fix/sparse-atoms
fix/sparse-atoms
1m 22s
View #2466
View workflow file
fix(byte-level): Support sparse byte atoms
Rust
#5932:
Pull request
#2466
synchronize by
SBrandeis
1m 42s
fix/sparse-atoms
fix/sparse-atoms
1m 42s
View #2466
View workflow file
fix(byte-level): Support sparse byte atoms
Python
#5077:
Pull request
#2466
synchronize by
SBrandeis
5m 6s
fix/sparse-atoms
fix/sparse-atoms
5m 6s
View #2466
View workflow file
fix(byte-level): Support sparse byte atoms
Node
#4915:
Pull request
#2466
synchronize by
SBrandeis
1m 16s
fix/sparse-atoms
fix/sparse-atoms
1m 16s
View #2466
View workflow file
bump hf revision
Secret Leaks
#2793:
Commit
cde465c
pushed by
SBrandeis
22s
fix/sparse-atoms
fix/sparse-atoms
22s
View #2466
View workflow file
Upload PR Documentation
Upload PR Documentation
#2537:
completed by
SBrandeis
20s
20s
View workflow file
fix(BucketVocabStore): Explicit duplicate handling
Build PR Documentation
#3300:
Pull request
#2467
opened by
SBrandeis
1m 36s
fix/vocab-store-duplicates
fix/vocab-store-duplicates
1m 36s
View #2467
View workflow file
fix(BucketVocabStore): Explicit duplicate handling
Rust
#5931:
Pull request
#2467
opened by
SBrandeis
1m 44s
fix/vocab-store-duplicates
fix/vocab-store-duplicates
1m 44s
View #2467
View workflow file
fix(BucketVocabStore): Explicit duplicate handling
Python
#5076:
Pull request
#2467
opened by
SBrandeis
4m 42s
fix/vocab-store-duplicates
fix/vocab-store-duplicates
4m 42s
View #2467
View workflow file
fix(BucketVocabStore): Explicit duplicate handling
Node
#4914:
Pull request
#2467
opened by
SBrandeis
1m 28s
fix/vocab-store-duplicates
fix/vocab-store-duplicates
1m 28s
View #2467
View workflow file
BucketVocabStore::build is now fallible
Secret Leaks
#2792:
Commit
b764bd0
pushed by
SBrandeis
26s
fix/vocab-store-duplicates
fix/vocab-store-duplicates
26s
View #2467
View workflow file
Upload PR Documentation
Upload PR Documentation
#2536:
completed by
SBrandeis
16s
16s
View workflow file
fix(byte-level): Support sparse byte atoms
Build PR Documentation
#3299:
Pull request
#2466
synchronize by
SBrandeis
1m 47s
fix/sparse-atoms
fix/sparse-atoms
1m 47s
View #2466
View workflow file
fix(byte-level): Support sparse byte atoms
Rust
#5930:
Pull request
#2466
synchronize by
SBrandeis
2m 8s
fix/sparse-atoms
fix/sparse-atoms
2m 8s
View #2466
View workflow file
fix(byte-level): Support sparse byte atoms
Python
#5075:
Pull request
#2466
synchronize by
SBrandeis
5m 16s
fix/sparse-atoms
fix/sparse-atoms
5m 16s
View #2466
View workflow file
fix(byte-level): Support sparse byte atoms
Node
#4913:
Pull request
#2466
synchronize by
SBrandeis
1m 43s
fix/sparse-atoms
fix/sparse-atoms
1m 43s
View #2466
View workflow file
bump hf revision
Secret Leaks
#2791:
Commit
3c5fa4e
pushed by
SBrandeis
34s
fix/sparse-atoms
fix/sparse-atoms
34s
View workflow file
Upload PR Documentation
Upload PR Documentation
#2535:
completed by
SBrandeis
22s
22s
View workflow file
Previous
1
2
3
4
5
…
99
100
101
Next
You can’t perform that action at this time.