Skip to content

Filter low-frequency suffixes from dense ZeroTrie - #8733

Merged
sffc merged 2 commits into
unicode-org:mainfrom
wozhenbang2004:patch-2
Nov 19, 2026
Merged

sffc merged 2 commits into
unicode-org:mainfrom
wozhenbang2004:patch-2

Conversation

@wozhenbang2004

Copy link
Copy Markdown
Contributor

Fixes #7302. Follow-up to #7439.

Keep nonempty suffixes in dense columns only when they appear in at least two rows and 2% of all rows. Preserve dense storage for encodable empty suffixes, whose sparse key would collide with the row offset. The existing basic fixture shrinks from 106 to 82 bytes.

Tested with cargo test -p zerotrie --all-features, including the 100/101-row threshold boundary, rare empty suffixes, and encoding round trips.

Changelog

zerotrie: Exclude low-frequency suffixes from dense trie columns.

@wozhenbang2004
wozhenbang2004 requested a review from a team as a code owner November 18, 2026 17:03
@Manishearth
Manishearth requested a review from sffc November 18, 2026 18:31
@Manishearth

Copy link
Copy Markdown
Member

I can review this but for zerotrie code it is preferable if Shane does.

@sffc sffc left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nice work!

@sffc

sffc commented Nov 19, 2026

Copy link
Copy Markdown
Member

CI semver is complaining about something in normalizer, but this PR does not touch normalizer.

@sffc
sffc merged commit 3f9c5d4 into unicode-org:main Nov 19, 2026
33 of 34 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Filter low-frequency suffixes from Dense ZeroTrie

3 participants