Fix issue 406

[ooples]

This commit implements a comprehensive tokenization framework for AiDotNet, replacing the naive whitespace tokenization with state-of-the-art subword tokenization algorithms required by modern NLP systems.

Core Tokenizers Implemented:

• BPE (Byte-Pair Encoding) for GPT models
• WordPiece for BERT-family models
• SentencePiece (Unigram) for multilingual models

Key Features:

• Vocabulary training from corpus
• Special tokens management ([CLS], [SEP], [PAD], [UNK], [MASK], etc.)
• Encoding/decoding with padding and truncation
• Attention mask generation
• HuggingFace pretrained tokenizer compatibility
• Load/save tokenizers in HuggingFace format
• Batch encoding/decoding support

Code Tokenization:

• Language-aware tokenization (C#, Python, Java, JavaScript, TypeScript)
• Identifier splitting (camelCase, snake_case, PascalCase)
• Keyword recognition
• CodeBERT-compatible tokenizer for program synthesis
• Combined code + natural language encoding

Implementation Details:

• 16 new source files in src/Tokenization/
• Complete interfaces (ITokenizer, IVocabulary)
• Abstract base class (TokenizerBase) for common functionality
• Three algorithm implementations with training support
• HuggingFace compatibility layer
• Code-specific tokenization support
• Comprehensive test suite (4 test files)
• Full documentation (README.md)

This resolves issue #406 and unblocks:

• Issue [Phase 3] Implement Program Synthesis and Code Generation #404: Program Synthesis (CodeBERT tokenizer ready)
• Issues Automatic Speech Recognition (ASR) – Comprehensive Model Implementation (~95 models) #269-273: Multimodal systems
• All BERT/GPT/T5 model implementations

Files created: 20 total

• 14 implementation files
• 2 HuggingFace compatibility files
• 4 test files

User Story / Context

• Reference: [US-XXX] (if applicable)
• Base branch: merge-dev2-to-master

Summary

• What changed and why (scoped strictly to the user story / PR intent)

Verification

• Builds succeed (scoped to changed projects)
• Unit tests pass locally
• Code coverage >= 90% for touched code
• Codecov upload succeeded (if token configured)
• TFM verification (net46, net6.0, net8.0) passes (if packaging)
• No unresolved Copilot comments on HEAD

Copilot Review Loop (Outcome-Based)

Record counts before/after your last push:

• Comments on HEAD BEFORE: [N]
• Comments on HEAD AFTER (60s): [M]
• Final HEAD SHA: [sha]

Files Modified

• List files changed (must align with scope)

Notes

• Any follow-ups, caveats, or migration details

[coderabbitai]

Note

Other AI code review bot(s) detected

CodeRabbit has detected other AI code review bot(s) in this pull request and will avoid duplicating their findings in the review comments. This may lead to a less comprehensive review.

Summary by CodeRabbit

Release Notes

• New Features

  • Added comprehensive tokenization framework supporting multiple algorithms (BPE, WordPiece, SentencePiece, Unigram, Character).
  • Integrated tokenization into model builder for automatic text encoding/decoding during inference.
  • Added HuggingFace tokenizer compatibility with model loading and caching.
  • Introduced specialized tokenizers for code, phoneme, and MIDI processing.
  • Enabled batch tokenization with configurable padding, truncation, and encoding options.

• Documentation

  • Added tokenization framework guide and examples.

_{✏️ Tip: You can customize this high-level summary in your review settings.}

Walkthrough

Implements a comprehensive modern tokenization framework for AiDotNet, including multiple tokenization algorithms (BPE, WordPiece, SentencePiece, Unigram, Character), HuggingFace hub integration, code-aware tokenization, specialized tokenizers (MIDI, Phoneme), vocabulary management, and integration with the model builder pipeline. Adds ~30 production files and extensive test/benchmark coverage.

Changes

Cohort / File(s)	Summary
Core Tokenization Interfaces & Base Classes src/Tokenization/Interfaces/ITokenizer.cs, src/Tokenization/Interfaces/IVocabulary.cs, src/Tokenization/Core/TokenizerBase.cs	New tokenizer interface contract (encode/decode/tokenize), vocabulary management interface, and abstract base class implementing encoding workflow with special tokens, truncation, padding, attention mask, and position ID generation.
Tokenization Algorithm Implementations src/Tokenization/Algorithms/{BpeTokenizer,WordPieceTokenizer,SentencePieceTokenizer,UnigramTokenizer,CharacterTokenizer}.cs	Five concrete tokenizer implementations: BPE with merge-rule training, WordPiece with greedy longest-match, SentencePiece with Viterbi segmentation, Unigram with probabilistic Viterbi, and Character with optional lowercasing and whitespace handling.
Vocabulary Management src/Tokenization/Vocabulary/Vocabulary.cs	Bidirectional token↔ID mapping with unknown token handling, bulk addition, and iteration support.
Configuration & Result Models src/Tokenization/Models/{EncodingOptions,SpecialTokens,TokenizationResult}.cs, src/Tokenization/Configuration/{TokenizationConfig,PretrainedTokenizerModel}.cs	Configuration classes for encoding behavior (padding, truncation, attention masks), special token definitions (BERT/GPT/T5 presets), tokenization output structure, and strongly-typed enum for pretrained model references with HuggingFace Hub ID mapping.
HuggingFace Integration src/Tokenization/HuggingFace/{AutoTokenizer,HuggingFaceTokenizerLoader,TokenizerConfig}.cs	AutoTokenizer convenience methods (FromPretrained, caching, listing), HuggingFaceTokenizerLoader for loading tokenizers from Hub or local directories with support for multiple formats (tokenizer.json, vocab+merges), and config model for JSON deserialization.
Code Tokenization src/Tokenization/CodeTokenization/{CodeTokenizer,CodeBertTokenizer}.cs	Language-aware code tokenizer with identifier splitting, keyword preservation, and comment/string handling; CodeBERT-style encoder combining code and natural language with token type IDs for segment identification.
Specialized Tokenizers src/Tokenization/Specialized/{MidiTokenizer,PhonemeTokenizer}.cs	MIDI note-to-token converter with REMI strategy and bar/pitch/velocity/duration tokens; phoneme tokenizer with G2P rules, ARPAbet presets, and phoneme set selection.
Model Builder Integration src/Interfaces/IPredictionModelBuilder.cs, src/PredictionModelBuilder.cs, src/Models/Results/PredictionModelResult.cs	New interface methods and implementations for tokenizer configuration (manual, from pretrained models, from HuggingFace Hub with async support); PredictionModelResult gains tokenizer attachment and public Tokenize/TokenizeBatch/Detokenize/HasTokenizer APIs.
Documentation TOKENIZATION_IMPLEMENTATION_SUMMARY.md, src/Tokenization/README.md	High-level summary of architecture and components; comprehensive README with usage examples, directory structure, and compatibility notes.
Comprehensive Test Suites tests/AiDotNet.Tests/UnitTests/Tokenization/{VocabularyTests,BpeTokenizerTests,WordPieceTokenizerTests,CharacterTokenizerTests,CodeTokenizerTests,HuggingFaceLoaderTests,SpecializedTokenizerTests,UnigramTokenizerTests,AutoTokenizerTests}.cs, tests/AiDotNet.Tests/Tokenization/{BpeTokenizerTests,CodeTokenizerTests,WordPieceTokenizerTests}.cs	Unit tests covering training, tokenization, encoding/decoding, special tokens, padding/truncation, batch operations, HuggingFace loading, code/MIDI/phoneme tokenization.
Performance Benchmarks tests/AiDotNet.Tests/Benchmarks/TokenizerBenchmarks.cs	Throughput and memory benchmarks for BPE, WordPiece, SentencePiece, Unigram, and Character tokenizers across tokenization, encoding, decoding, and batch operations.

Sequence Diagram(s)

sequenceDiagram
    participant App as Application
    participant Builder as PredictionModelBuilder
    participant Tokenizer as ITokenizer
    participant Vocab as IVocabulary
    participant Result as PredictionModelResult

    App->>Builder: ConfigureTokenizer(tokenizer) or ConfigureTokenizerFromPretrained(model)
    Builder->>Builder: Store tokenizer & config internally
    App->>Builder: BuildAsync()
    Builder->>Result: Create PredictionModelResult
    Builder->>Result: AttachTokenizer(tokenizer, config)
    Result->>Result: Store tokenizer & config
    
    rect rgb(200, 220, 255)
    Note over App,Result: Later: Use tokenization
    App->>Result: Tokenize(text)
    Result->>Tokenizer: Encode(text, options)
    Tokenizer->>Vocab: ConvertTokensToIds(tokens)
    Vocab-->>Tokenizer: token IDs
    Tokenizer-->>Result: TokenizationResult {tokens, IDs, masks}
    Result-->>App: TokenizationResult
    end

Loading

Estimated code review effort

🎯 4 (Complex) | ⏱️ ~75 minutes

Areas requiring extra attention:

• Algorithm correctness: BPE merge logic, Viterbi segmentation in SentencePiece/Unigram, greedy longest-match in WordPiece—verify mathematical soundness and edge case handling (empty input, single characters, unknown tokens)
• HuggingFace format compatibility: Parsing of tokenizer.json, vocab/merges files, special token extraction from multiple file formats; ensure round-trip consistency between load and save
• Integration with builder/result lifecycle: Verify tokenizer is correctly attached across all build paths (meta-learning, regular, RL), preserved during deserialization, and accessible from PredictionModelResult
• Identifier splitting edge cases: CodeTokenizer camelCase/snake_case splitting regex patterns may miss or misclassify identifiers; test with varied naming conventions
• Batch processing logic: Ensure padding/truncation/attention mask generation is consistent and correct across batch sizes and variable-length inputs
• MIDI/Phoneme tokenizer domain logic: Verify quantization, Viterbi scoring, and phoneme mapping correctness for specialized domains

Possibly related issues

• [CRITICAL Infrastructure] Implement Modern Tokenization Framework #406: This PR directly implements the full tokenization framework infrastructure (all algorithms, HuggingFace compatibility, vocabulary, interfaces, configuration) that was requested in the issue.

Possibly related PRs

• Feature: Complete RAG Framework Concrete Implementations (#303) #304: Both PRs modify PredictionModelBuilder and PredictionModelResult to wire additional pipeline components (this PR adds tokenizers, #304 adds RAG systems) through the model lifecycle using the same attach/configuration pattern.

Poem

🐰 Tokens dance through BPE's gentle merge,
WordPiece cuts and builds with verge,
HuggingFace arrives on model's call,
CodeBERT speaks and codes enthrall,
From vocab's heart to hub's embrace—
The tokenization framework finds its place! ✨

Pre-merge checks and finishing touches

❌ Failed checks (1 warning, 1 inconclusive)

Check name	Status	Explanation	Resolution
Docstring Coverage	⚠️ Warning	Docstring coverage is 37.79% which is insufficient. The required threshold is 80.00%.	You can run @coderabbitai generate docstrings to improve docstring coverage.
Title check	❓ Inconclusive	The title 'Fix issue 406' is vague and does not clearly summarize the main change. While it references an issue number, it provides no meaningful information about the tokenization framework implementation that constitutes the core change.	Revise the title to be more descriptive, e.g., 'Implement comprehensive tokenization framework with BPE, WordPiece, and SentencePiece' or 'Add subword tokenization support (BPE, WordPiece, SentencePiece, HuggingFace)' to clearly convey the primary change.

✅ Passed checks (1 passed)

Check name	Status	Explanation
Description check	✅ Passed	The description is comprehensive and directly related to the changeset. It clearly explains the tokenization framework implementation, core features, code tokenization support, and provides detailed context about what was implemented and why.

✨ Finishing touches

• 📝 Generate docstrings

🧪 Generate unit tests (beta)

• Create PR with unit tests
• Post copyable unit tests in a comment
• Commit unit tests in branch claude/fix-issue-406-011CUvqPDDa4qHeWevRhmZPR

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands and usage tips.}

[Copilot]

Pull Request Overview

This PR implements a comprehensive modern tokenization framework for AiDotNet, adding support for state-of-the-art subword tokenization algorithms (BPE, WordPiece, SentencePiece), HuggingFace compatibility, and specialized code tokenization capabilities. This addresses Issue #406 and unblocks multiple downstream features.

• BPE, WordPiece, and SentencePiece tokenizers with training from corpus
• HuggingFace pretrained tokenizer loading and saving
• Language-aware code tokenization with identifier splitting for C#, Python, Java, JavaScript

Reviewed Changes

Copilot reviewed 20 out of 20 changed files in this pull request and generated 18 comments.

Show a summary per file

File	Description
src/Tokenization/Interfaces/ITokenizer.cs	Defines tokenizer interface with encode/decode operations
src/Tokenization/Interfaces/IVocabulary.cs	Defines vocabulary management interface
src/Tokenization/Models/TokenizationResult.cs	Result model containing tokens, IDs, and attention masks
src/Tokenization/Models/EncodingOptions.cs	Configuration model for encoding with padding and truncation options
src/Tokenization/Models/SpecialTokens.cs	Special token management with factory methods for BERT/GPT/T5 styles
src/Tokenization/Core/TokenizerBase.cs	Abstract base class implementing common tokenization functionality
src/Tokenization/Vocabulary/Vocabulary.cs	Token-to-ID mapping implementation with unknown token handling
src/Tokenization/Algorithms/BpeTokenizer.cs	Byte-Pair Encoding implementation with merge-based tokenization
src/Tokenization/Algorithms/WordPieceTokenizer.cs	WordPiece algorithm with greedy longest-match-first approach
src/Tokenization/Algorithms/SentencePieceTokenizer.cs	Unigram language model with Viterbi segmentation
src/Tokenization/HuggingFace/TokenizerConfig.cs	HuggingFace configuration format model
src/Tokenization/HuggingFace/HuggingFaceTokenizerLoader.cs	Loads and saves pretrained tokenizers in HuggingFace format
src/Tokenization/CodeTokenization/CodeTokenizer.cs	Language-aware tokenizer with identifier splitting and keyword recognition
src/Tokenization/CodeTokenization/CodeBertTokenizer.cs	CodeBERT-compatible tokenizer for code and natural language
tests/AiDotNet.Tests/Tokenization/VocabularyTests.cs	Tests for vocabulary operations
tests/AiDotNet.Tests/Tokenization/BpeTokenizerTests.cs	Tests for BPE tokenizer functionality
tests/AiDotNet.Tests/Tokenization/WordPieceTokenizerTests.cs	Tests for WordPiece tokenizer
tests/AiDotNet.Tests/Tokenization/CodeTokenizerTests.cs	Tests for code tokenization features
src/Tokenization/README.md	Comprehensive documentation with usage examples
TOKENIZATION_IMPLEMENTATION_SUMMARY.md	Implementation summary and architecture overview

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[coderabbitai]

Actionable comments posted: 19

♻️ Duplicate comments (4)

src/Tokenization/Algorithms/WordPieceTokenizer.cs (1)

137-148: Consider LINQ pattern for cleaner code.

The iteration can be simplified using LINQ's SelectMany for a more functional style, as noted in a previous review.

-    var outputTokens = new List<string>();
-
-    // Basic whitespace tokenization
-    var words = text.Split(new[] { ' ', '\t', '\n', '\r' }, StringSplitOptions.RemoveEmptyEntries);
-
-    foreach (var word in words.Select(w => w.ToLowerInvariant()))
-    {
-        var wordTokens = TokenizeWord(word);
-        outputTokens.AddRange(wordTokens);
-    }
-
-    return outputTokens;
+    // Basic whitespace tokenization
+    var words = text.Split(new[] { ' ', '\t', '\n', '\r' }, StringSplitOptions.RemoveEmptyEntries);
+
+    return words
+        .Select(w => w.ToLowerInvariant())
+        .SelectMany(TokenizeWord)
+        .ToList();

src/Tokenization/CodeTokenization/CodeBertTokenizer.cs (1)

82-84: Past review feedback addressed.

The attention mask initialization now uses the efficient Enumerable.Repeat(1, tokenIds.Count).ToList() pattern.

src/Tokenization/Algorithms/BpeTokenizer.cs (1)

84-95: Consider using LINQ for cleaner initialization.

The foreach loop can be refactored to use .ToDictionary() for the splits initialization, which is more idiomatic.

-            var splits = new Dictionary<string, List<string>>();
-            foreach (var word in wordFreqs.Keys)
-            {
-                var charStrings = word.Select(c => c.ToString()).ToList();
-                splits[word] = charStrings;
-
-                // Add characters to vocabulary
-                foreach (var charStr in charStrings)
-                {
-                    vocabulary.AddToken(charStr);
-                }
-            }
+            var splits = wordFreqs.Keys.ToDictionary(
+                word => word,
+                word => word.Select(c => c.ToString()).ToList()
+            );
+
+            // Add characters to vocabulary
+            foreach (var word in wordFreqs.Keys)
+            {
+                foreach (var charStr in splits[word])
+                {
+                    vocabulary.AddToken(charStr);
+                }
+            }

src/Tokenization/Core/TokenizerBase.cs (1)

191-200: TruncateSequence implementation is correct.

The ternary operator cleanly handles left vs right truncation.

🧹 Nitpick comments (33)

src/Tokenization/README.md (1)

184-207: Specify a language for the architecture code block to satisfy markdownlint (MD040).

The fenced block showing the Tokenization/ directory tree has no language, which triggers MD040. Consider annotating it as plain text:

-```
+```text
 Tokenization/
 ├── Interfaces/
 ...
 └── CodeTokenization/
-```
+```

tests/AiDotNet.Tests/UnitTests/Tokenization/CodeTokenizerTests.cs (1)

67-93: Align identifier‑splitting tests with their intent (or relax the comments).

Tokenize_CamelCaseIdentifier_SplitsIdentifier and Tokenize_SnakeCaseIdentifier_SplitsIdentifier currently only assert that tokens is non‑empty, while the comments say they verify splitting. As written, they don’t prove that identifiers were actually split.

Consider either:

• Strengthening the assertions (e.g., checking tokens.Count > 1 or for expected pieces where it’s stable), or
• Relaxing the comments/test names to describe these as smoke tests if stronger checks would be too brittle with the underlying subword tokenizer.

tests/AiDotNet.Tests/UnitTests/Tokenization/UnigramTokenizerTests.cs (1)

148-160: Clarify or tighten the “LongWord_BreaksIntoSubwords” assertion.

The test name and comment say long unknown words “should be broken into subwords”, but the assertion only checks tokens.Count >= 1, which would pass even with a single <unk> or whole‑word token.

Consider either:

• Asserting something like Assert.True(tokens.Count > 1);, or
• Renaming/rewriting the comment if single‑token behavior is actually acceptable here.

tests/AiDotNet.Tests/UnitTests/Tokenization/AutoTokenizerTests.cs (1)

217-222: Optional: mirror whitespace validation in async FromPretrained tests.

The sync FromPretrained_WithEmptyPath_ThrowsArgumentException test covers both "" and " ", while the async variant only checks "". For consistency, you could add a whitespace‑only case to the async test as well.

TOKENIZATION_IMPLEMENTATION_SUMMARY.md (3)

16-25: Add a language to the directory-tree fenced block (MD040).

The src/Tokenization/ tree is in a bare fenced block. To satisfy markdownlint and improve rendering, annotate it as plain text:

-```
+```text
 src/Tokenization/
 ├── Interfaces/
 ...
 └── CodeTokenization/
-```
+```

---

221-221: Avoid using bold as a pseudo‑heading (MD036).

The “Total: 19 files + this summary = 20 files” line is formatted as bold text, which markdownlint flags as heading‑like emphasis. Consider turning it into an actual heading:

-**Total: 19 files + this summary = 20 files**
+### Total: 19 files + this summary = 20 files

or simply a normal sentence without bold.

---

194-221: Update the “Files Created” list to match the current implementation.

The enumerated “Core/HuggingFace/Code Tokenization/Tests” file list appears to be slightly out of date compared to the rest of this PR (e.g., additional algorithms like UnigramTokenizer, extra unit test files under tests/AiDotNet.Tests/UnitTests/Tokenization, AutoTokenizer tests).

Refreshing this section to reflect the final set of components and their actual paths will keep this summary aligned with the codebase and prevent confusion for future contributors.

tests/AiDotNet.Tests/UnitTests/Tokenization/WordPieceTokenizerTests.cs (1)

70-88: Make the continuation‑prefix test assert on ## tokens.

Tokenize_ContinuationPrefix_UsesHashHash currently only checks that tokens is non‑null; the continuationTokens list is computed but never asserted, so the test doesn’t actually verify WordPiece’s ## convention.

You could strengthen it along these lines:

     var hasSubwords = tokens.Count > 1;
     if (hasSubwords)
     {
-        // Some tokens should have ## prefix (continuation)
-        var continuationTokens = tokens.Skip(1).Where(t => t.StartsWith("##")).ToList();
-        // May or may not have continuation depending on vocabulary
+        // Some tokens should have ## prefix (continuation)
+        var continuationTokens = tokens.Skip(1).Where(t => t.StartsWith("##")).ToList();
+        Assert.NotEmpty(continuationTokens);
+        Assert.All(continuationTokens, t => Assert.StartsWith("##", t));
     }

so the test actually enforces the expected continuation marker when subwords are produced.

tests/AiDotNet.Tests/Tokenization/BpeTokenizerTests.cs (2)

45-53: Verify test expectation aligns with BPE behavior.

The test constructs merges that should combine characters into "hello" (lines 39-42), but the assertion on line 52 may be fragile. If the BPE algorithm doesn't fully merge due to the specific merge ordering or if the vocabulary lookup fails, "hello" might not appear as a single token.

Consider adding an assertion that checks the actual token count or verifies the merge was applied, rather than just checking for the presence of "hello" in the token list.

     // Act
     var tokens = tokenizer.Tokenize("hello");

     // Assert
     Assert.NotEmpty(tokens);
-    Assert.Contains("hello", tokens);
+    // Verify merging occurred - should have fewer tokens than characters
+    Assert.True(tokens.Count < 5, "BPE merges should reduce token count");
+    Assert.Contains("hello", tokens); // Verify final merged token

---

90-98: Potential round-trip preservation issue.

Line 97 expects exact equality after encode/decode, but BPE tokenizers may not preserve exact whitespace or may introduce artifacts. This test could be flaky if the tokenizer normalizes whitespace differently during encoding vs decoding.

Consider a more lenient assertion or explicitly document the whitespace preservation guarantee:

     // Act
     var decoded = tokenizer.Decode(encoded.TokenIds, skipSpecialTokens: true);

     // Assert
-    Assert.Equal(text, decoded);
+    // BPE should preserve content, but whitespace handling may vary
+    Assert.Equal(text.Trim(), decoded.Trim());

src/Tokenization/Configuration/PretrainedTokenizerModel.cs (2)

177-178: Duplicate model ID mapping.

Both PretrainedTokenizerModel.CodeBertBase and PretrainedTokenizerModel.MicrosoftCodeBert map to the same HuggingFace model ID "microsoft/codebert-base". While this works, it creates ambiguity about which enum value to use.

Consider one of these approaches:

1. Remove duplicate - Keep only MicrosoftCodeBert or CodeBertBase (recommend MicrosoftCodeBert for clarity)
2. Document the alias - Add XML comments explaining these are aliases for the same model
3. Mark one as obsolete - If one is preferred, mark the other [Obsolete("Use MicrosoftCodeBert instead")]

Example:

     PretrainedTokenizerModel.CodeBertBase => "microsoft/codebert-base",
-    PretrainedTokenizerModel.MicrosoftCodeBert => "microsoft/codebert-base",
+    // MicrosoftCodeBert is handled by CodeBertBase mapping (same model)

---

157-181: Silent fallback may mask errors.

The switch expression uses a default case (line 180) that returns "bert-base-uncased" for any unmapped enum values. If new enum members are added without updating ToModelId, they'll silently fall back to BERT instead of failing fast.

Consider throwing an exception for unmapped values to catch configuration errors early:

-    _ => "bert-base-uncased" // Fallback to default
+    _ => throw new ArgumentOutOfRangeException(nameof(model), model, 
+        $"No HuggingFace model ID mapping defined for {model}. Please update ToModelId extension method.")

If you want to keep the fallback, at least log a warning or add a comment explaining why silent fallback is acceptable.

src/Interfaces/IPredictionModelBuilder.cs (1)

891-891: Clarify behavior when all parameters are null.

Both parameters of ConfigureTokenizer are optional (nullable with default null). It's unclear what calling builder.ConfigureTokenizer() with no arguments does. Does it:

1. Clear any previously configured tokenizer?
2. Use some default tokenizer?
3. Effectively do nothing?

Consider one of these approaches:

1. Require at least one parameter - Make tokenizer non-nullable OR add validation
2. Document the behavior - Update XML comments to explain what happens when both are null
3. Add validation - Throw if both are null (if that's not a valid configuration)

Example:

/// <remarks>
/// ...
/// If both tokenizer and config are null, this method has no effect and any
/// previously configured tokenizer remains active.
/// ...
/// </remarks>

src/Tokenization/Algorithms/CharacterTokenizer.cs (1)

95-95: Redundant namespace qualification.

Line 95 uses new Vocabulary.Vocabulary(...) with the namespace qualifier, but the using statements should already resolve Vocabulary. This is redundant and reduces readability.

-    var vocabulary = new Vocabulary.Vocabulary(specialTokens.UnkToken);
+    var vocabulary = new Vocabulary(specialTokens.UnkToken);

src/Models/Results/PredictionModelResult.cs (1)

1928-2003: Well-designed tokenization integration.

The tokenization support is well-integrated into PredictionModelResult:

Strengths:

• Internal attachment method (AttachTokenizer) maintains encapsulation
• Public methods (Tokenize, TokenizeBatch, Detokenize) provide clean API
• Proper null checking with clear error messages
• HasTokenizer property enables safe consumer checks
• Consistent error handling across all public methods

Optional improvement:

Consider adding a guard clause to AttachTokenizer to prevent attaching null config when tokenizer is non-null:

 internal void AttachTokenizer(
     ITokenizer? tokenizer,
     TokenizationConfig? config = null)
 {
+    // If attaching a tokenizer, ensure we have a config (use default if needed)
+    if (tokenizer != null && config == null)
+        config = new TokenizationConfig();
+    
     Tokenizer = tokenizer;
     TokenizationConfig = config;
 }

This ensures TokenizationConfig is always available when a tokenizer is present, avoiding potential null reference issues in Tokenize at line 1966.

src/Tokenization/Algorithms/WordPieceTokenizer.cs (1)

90-124: Simplified training approach may miss edge cases.

The training algorithm uses frequency-based subword selection rather than the likelihood-based approach used in the original WordPiece paper. This works for basic use cases but may produce suboptimal vocabularies for complex corpora.

Additionally, single characters added in step 3 (lines 82-88) might be overwritten or excluded if vocabSize is reached before all necessary subwords are added, potentially causing tokenization failures.

Consider ensuring critical single-character tokens remain in the vocabulary:

 foreach (var subword in sortedSubwords)
 {
     if (vocabulary.Size >= vocabSize)
         break;
 
-    vocabulary.AddToken(subword);
+    // Skip if already added (e.g., single characters)
+    if (!vocabulary.ContainsToken(subword))
+        vocabulary.AddToken(subword);
 }

src/Tokenization/CodeTokenization/CodeTokenizer.cs (1)

163-168: Minor redundancy in token processing.

The .Select(token => token.Trim()) after filtering out whitespace-only tokens is slightly redundant since whitespace tokens are already filtered. However, it's harmless and adds defensive trimming.

-    tokens.AddRange(matches.Cast<Match>()
-        .Select(m => m.Value)
-        .Where(token => !string.IsNullOrWhiteSpace(token))
-        .Select(token => token.Trim()));
+    tokens.AddRange(matches.Cast<Match>()
+        .Select(m => m.Value.Trim())
+        .Where(token => !string.IsNullOrEmpty(token)));

tests/AiDotNet.Tests/Tokenization/VocabularyTests.cs (1)

108-137: Consider adding a test to verify vocabulary is usable after Clear().

The Clear_RemovesAllTokens test only verifies size becomes 0. Consider adding assertions to verify the vocabulary can be reused after clearing (e.g., adding new tokens works correctly and UNK token handling is consistent). Based on the relevant code snippet, Clear() resets _nextId to 0 but doesn't re-add the UNK token, which could be an edge case worth testing.

[Fact]
public void Clear_AllowsReuseAfterClearing()
{
    // Arrange
    var vocab = new Vocabulary("[UNK]");
    vocab.AddTokens(new[] { "hello", "world" });
    vocab.Clear();

    // Act
    var id = vocab.AddToken("test");

    // Assert
    Assert.Equal(1, vocab.Size);
    Assert.Equal(0, id); // First token after clear gets ID 0
}

src/Tokenization/Algorithms/SentencePieceTokenizer.cs (1)

195-211: Insert(0, ...) in a loop is O(n²) — consider reversing at the end.

Using List.Insert(0, ...) repeatedly shifts all existing elements on each insertion. For long texts, this can degrade performance.

 var tokens = new List<string>();
 int pos = n;

 while (pos > 0)
 {
     if (backtrack[pos] == -1)
     {
-        tokens.Insert(0, SpecialTokens.UnkToken);
-        break;
+        tokens.Add(SpecialTokens.UnkToken);
+        pos--;
+        continue;
     }

     var start = backtrack[pos];
     var piece = text.Substring(start, pos - start);
-    tokens.Insert(0, piece);
+    tokens.Add(piece);
     pos = start;
 }

+tokens.Reverse();
 return tokens;

src/Tokenization/HuggingFace/TokenizerConfig.cs (1)

59-63: Consider making DoLowerCase nullable to distinguish "not specified" from "false".

Currently DoLowerCase defaults to false, which makes it impossible to distinguish between a config file that explicitly sets do_lower_case: false versus one that omits the field entirely. If downstream logic needs to apply different defaults based on tokenizer type, nullable would help.

-[JsonProperty("do_lower_case")]
-public bool DoLowerCase { get; set; }
+[JsonProperty("do_lower_case")]
+public bool? DoLowerCase { get; set; }

src/Tokenization/Models/SpecialTokens.cs (1)

10-48: Consider using empty string defaults for model-specific tokens.

Defaulting all tokens to non-empty values means any instance of SpecialTokens created via new SpecialTokens() will include all seven tokens regardless of whether the model actually uses them. This could lead to incorrect vocabulary initialization or unexpected behavior when tokens like [BOS] are added to BERT-style models.

An alternative approach is to default model-specific tokens (BosToken, EosToken) to empty strings and only set them in factories that need them.

-public string BosToken { get; set; } = "[BOS]";
-public string EosToken { get; set; } = "[EOS]";
+public string BosToken { get; set; } = "";
+public string EosToken { get; set; } = "";

src/Tokenization/Specialized/PhonemeTokenizer.cs (1)

108-127: Dictionary allocation on every GetDefaultPhoneme call.

The phoneme mapping dictionaries are recreated for each unmatched character. For text with many unmatched characters, this creates unnecessary allocations.

Consider making the mappings static readonly:

+    private static readonly Dictionary<char, string> ArpabetMappings = new()
+    {
+        {'a', "AE"}, {'b', "B"}, {'c', "K"}, {'d', "D"}, {'e', "EH"}, {'f', "F"},
+        {'g', "G"}, {'h', "HH"}, {'i', "IH"}, {'j', "JH"}, {'k', "K"}, {'l', "L"},
+        {'m', "M"}, {'n', "N"}, {'o', "AA"}, {'p', "P"}, {'r', "R"}, {'s', "S"},
+        {'t', "T"}, {'u', "AH"}, {'v', "V"}, {'w', "W"}, {'y', "Y"}, {'z', "Z"}
+    };
+
+    private static readonly Dictionary<char, string> XsampaMappings = new()
+    {
+        {'a', "ae"}, {'b', "b"}, {'c', "k"}, {'d', "d"}, {'e', "E"}, {'f', "f"},
+        {'g', "g"}, {'h', "h"}, {'i', "I"}, {'j', "dZ"}, {'k', "k"}, {'l', "l"},
+        {'m', "m"}, {'n', "n"}, {'o', "A"}, {'p', "p"}, {'r', "r"}, {'s', "s"},
+        {'t', "t"}, {'u', "V"}, {'v', "v"}, {'w', "w"}, {'y', "j"}, {'z', "z"}
+    };
+
     private string GetDefaultPhoneme(char c)
     {
-        var mappings = _phonemeSet == PhonemeSet.ARPAbet
-            ? new Dictionary<char, string>
-            {
-                // ... dictionary contents ...
-            }
-            : new Dictionary<char, string>
-            {
-                // ... dictionary contents ...
-            };
+        var mappings = _phonemeSet == PhonemeSet.ARPAbet ? ArpabetMappings : XsampaMappings;
 
         return mappings.TryGetValue(char.ToLowerInvariant(c), out string? phoneme) ? phoneme ?? string.Empty : string.Empty;
     }

src/Tokenization/Algorithms/BpeTokenizer.cs (1)

101-150: Training loop is correct but has O(n) pair frequency counting per iteration.

The BPE training algorithm correctly finds the most frequent pair and merges it. However, for large vocabularies/corpora, rebuilding pairFreqs from scratch each iteration can be slow. For production use with large datasets, consider incremental frequency updates.

src/Tokenization/HuggingFace/AutoTokenizer.cs (2)

66-79: Async method returns synchronously for local directories.

When modelNameOrPath is a local directory, the async method returns synchronously via HuggingFaceTokenizerLoader.LoadFromDirectory. Consider wrapping in Task.FromResult or using an async-aware file check to maintain async consistency, though this is a minor concern.

             // Check if it's a local path
             if (Directory.Exists(modelNameOrPath))
             {
-                return HuggingFaceTokenizerLoader.LoadFromDirectory(modelNameOrPath);
+                return await Task.FromResult(HuggingFaceTokenizerLoader.LoadFromDirectory(modelNameOrPath));
             }

---

128-150: ClearCache has redundant null check after IsNullOrWhiteSpace.

Line 141 checks modelName is not null after line 135 already handled the null/whitespace case. The else-if is unreachable for null values.

             if (string.IsNullOrWhiteSpace(modelName))
             {
                 // Clear all cached tokenizers
                 Directory.Delete(cacheDir, true);
                 Directory.CreateDirectory(cacheDir);
             }
-            else if (modelName is not null)
+            else
             {
                 // Clear specific model cache
                 var modelCacheDir = Path.Combine(cacheDir, modelName.Replace("/", "--"));
                 if (Directory.Exists(modelCacheDir))
                 {
                     Directory.Delete(modelCacheDir, true);
                 }
             }

src/Tokenization/Configuration/TokenizationConfig.cs (1)

14-18: Potential confusion between DefaultEncodingOptions and ToEncodingOptions.

DefaultEncodingOptions is initialized as a new EncodingOptions(), but ToEncodingOptions() creates a separate EncodingOptions from the config's properties. This could lead to confusion - if a user sets DefaultEncodingOptions.MaxLength, it won't be reflected in ToEncodingOptions() output.

Consider clarifying the purpose or removing DefaultEncodingOptions if ToEncodingOptions is the primary conversion mechanism.

tests/AiDotNet.Tests/Benchmarks/TokenizerBenchmarks.cs (1)

207-228: Duplicate GenerateText/GenerateLargeText/GenerateSmallText helper methods.

The same text generation logic is duplicated across all three benchmark classes with minor variations. Consider extracting to a shared helper class or base class to reduce duplication.

// Consider creating a shared TestDataGenerator class:
internal static class TestDataGenerator
{
    private static readonly string[] Words = { 
        "the", "quick", "brown", "fox", "jumps", "over", "lazy", "dog",
        "machine", "learning", "artificial", "intelligence", "natural",
        "language", "processing", "deep", "neural", "network", "transformer",
        "attention", "embedding", "tokenizer", "vocabulary", "sequence",
        "model", "training", "inference", "batch", "parallel", "efficient"
    };

    public static string GenerateText(int wordCount, int? seed = 42)
    {
        var random = seed.HasValue ? new Random(seed.Value) : new Random();
        var sb = new StringBuilder(wordCount * 8);
        for (int i = 0; i < wordCount; i++)
        {
            if (i > 0) sb.Append(' ');
            sb.Append(Words[random.Next(Words.Length)]);
        }
        return sb.ToString();
    }
}

Also applies to: 279-318, 389-407

src/Tokenization/Core/TokenizerBase.cs (1)

117-120: EncodeBatch could benefit from parallel processing for large batches.

The current implementation processes texts sequentially. Consider adding parallel processing support using Parallel.ForEach or PLINQ for batches above a certain threshold, which aligns with the EnableParallelBatchProcessing and ParallelBatchThreshold properties in TokenizationConfig.

tests/AiDotNet.Tests/UnitTests/Tokenization/BpeTokenizerTests.cs (1)

169-186: Consider testing left-side padding as well.

The padding test only validates right-side padding. Adding a test for left-side padding (PaddingSide = "left") would improve coverage, especially since the base class supports both padding sides.

src/Tokenization/Specialized/MidiTokenizer.cs (3)

118-121: Unused variable currentTick after assignment.

The variable currentTick is assigned on line 120 but never used after the loop completes. This is dead code that should be removed for clarity.

                 tokens.Add($"Duration_{QuantizeDuration(note.Duration)}");
-
-                currentTick = note.StartTick + note.Duration;
             }

---

22-22: Unused tokenization strategies CPWord and SimpleNote.

The enum defines three strategies but only REMI is implemented. Consider either documenting these as future work or removing them until implemented to avoid confusion.

---

105-112: Hardcoded 4/4 time signature assumption.

The bar calculation note.StartTick / (_ticksPerBeat * 4) assumes 4 beats per bar (4/4 time signature). This is a reasonable default but should be documented or made configurable for other time signatures.

src/Tokenization/HuggingFace/HuggingFaceTokenizerLoader.cs (1)

264-280: Silent failure on download hides potential issues.

While silently ignoring HttpRequestException is reasonable for optional files, consider logging at debug/trace level to aid troubleshooting. Also note that File.WriteAllBytes could throw IOException which would propagate unexpectedly.

         private static async Task DownloadFileAsync(string modelName, string fileName, string localPath)
         {
             var url = $"{HuggingFaceHubUrl}/{modelName}/resolve/main/{fileName}";
             try
             {
                 var response = await _httpClient.GetAsync(url);
                 if (response.IsSuccessStatusCode)
                 {
                     var content = await response.Content.ReadAsByteArrayAsync();
                     File.WriteAllBytes(localPath, content);
                 }
             }
-            catch (HttpRequestException)
+            catch (Exception ex) when (ex is HttpRequestException or IOException)
             {
-                // Silently ignore - not all tokenizers have all files
+                // Optional files may not exist or may fail to write - ignore
             }
         }

📜 Review details

Configuration used: CodeRabbit UI

Review profile: CHILL

Plan: Pro

📥 Commits

Reviewing files that changed from the base of the PR and between 64247d5 and 7f105fb.

📒 Files selected for processing (40)

• TOKENIZATION_IMPLEMENTATION_SUMMARY.md (1 hunks)
• src/Interfaces/IPredictionModelBuilder.cs (2 hunks)
• src/Models/Results/PredictionModelResult.cs (3 hunks)
• src/Optimizers/CMAESOptimizer.cs (1 hunks)
• src/PredictionModelBuilder.cs (7 hunks)
• src/Tokenization/Algorithms/BpeTokenizer.cs (1 hunks)
• src/Tokenization/Algorithms/CharacterTokenizer.cs (1 hunks)
• src/Tokenization/Algorithms/SentencePieceTokenizer.cs (1 hunks)
• src/Tokenization/Algorithms/UnigramTokenizer.cs (1 hunks)
• src/Tokenization/Algorithms/WordPieceTokenizer.cs (1 hunks)
• src/Tokenization/CodeTokenization/CodeBertTokenizer.cs (1 hunks)
• src/Tokenization/CodeTokenization/CodeTokenizer.cs (1 hunks)
• src/Tokenization/Configuration/PretrainedTokenizerModel.cs (1 hunks)
• src/Tokenization/Configuration/TokenizationConfig.cs (1 hunks)
• src/Tokenization/Core/TokenizerBase.cs (1 hunks)
• src/Tokenization/HuggingFace/AutoTokenizer.cs (1 hunks)
• src/Tokenization/HuggingFace/HuggingFaceTokenizerLoader.cs (1 hunks)
• src/Tokenization/HuggingFace/TokenizerConfig.cs (1 hunks)
• src/Tokenization/Interfaces/ITokenizer.cs (1 hunks)
• src/Tokenization/Interfaces/IVocabulary.cs (1 hunks)
• src/Tokenization/Models/EncodingOptions.cs (1 hunks)
• src/Tokenization/Models/SpecialTokens.cs (1 hunks)
• src/Tokenization/Models/TokenizationResult.cs (1 hunks)
• src/Tokenization/README.md (1 hunks)
• src/Tokenization/Specialized/MidiTokenizer.cs (1 hunks)
• src/Tokenization/Specialized/PhonemeTokenizer.cs (1 hunks)
• src/Tokenization/Vocabulary/Vocabulary.cs (1 hunks)
• tests/AiDotNet.Tests/Benchmarks/TokenizerBenchmarks.cs (1 hunks)
• tests/AiDotNet.Tests/Tokenization/BpeTokenizerTests.cs (1 hunks)
• tests/AiDotNet.Tests/Tokenization/CodeTokenizerTests.cs (1 hunks)
• tests/AiDotNet.Tests/Tokenization/VocabularyTests.cs (1 hunks)
• tests/AiDotNet.Tests/Tokenization/WordPieceTokenizerTests.cs (1 hunks)
• tests/AiDotNet.Tests/UnitTests/Tokenization/AutoTokenizerTests.cs (1 hunks)
• tests/AiDotNet.Tests/UnitTests/Tokenization/BpeTokenizerTests.cs (1 hunks)
• tests/AiDotNet.Tests/UnitTests/Tokenization/CharacterTokenizerTests.cs (1 hunks)
• tests/AiDotNet.Tests/UnitTests/Tokenization/CodeTokenizerTests.cs (1 hunks)
• tests/AiDotNet.Tests/UnitTests/Tokenization/HuggingFaceLoaderTests.cs (1 hunks)
• tests/AiDotNet.Tests/UnitTests/Tokenization/SpecializedTokenizerTests.cs (1 hunks)
• tests/AiDotNet.Tests/UnitTests/Tokenization/UnigramTokenizerTests.cs (1 hunks)
• tests/AiDotNet.Tests/UnitTests/Tokenization/WordPieceTokenizerTests.cs (1 hunks)

🧰 Additional context used

🧬 Code graph analysis (20)

src/Tokenization/HuggingFace/TokenizerConfig.cs (1)

src/Tokenization/Models/SpecialTokens.cs (1)

• List (53-68)

src/Tokenization/Algorithms/WordPieceTokenizer.cs (4)

src/Tokenization/Core/TokenizerBase.cs (10)

• TokenizerBase (12-206)
• TokenizerBase (32-36)
• List (117-120)
• List (144-147)
• List (152-152)
• List (157-160)
• List (165-168)
• List (173-186)
• List (191-200)
• CleanupTokens (205-205)

src/Tokenization/Models/SpecialTokens.cs (6)

• SpecialTokens (8-107)
• SpecialTokens (73-80)
• SpecialTokens (85-91)
• SpecialTokens (96-101)
• SpecialTokens (106-106)
• List (53-68)

src/Tokenization/Vocabulary/Vocabulary.cs (6)

• IEnumerable (135-138)
• Vocabulary (11-149)
• Vocabulary (37-45)
• Vocabulary (52-58)
• AddToken (65-77)
• ContainsToken (116-119)

src/Tokenization/Interfaces/IVocabulary.cs (3)

• IEnumerable (60-60)
• AddToken (20-20)
• ContainsToken (47-47)

tests/AiDotNet.Tests/UnitTests/Tokenization/WordPieceTokenizerTests.cs (1)

tests/AiDotNet.Tests/UnitTests/Tokenization/BpeTokenizerTests.cs (2)

• Fact (34-43)
• Fact (45-57)

src/Tokenization/Models/SpecialTokens.cs (3)

src/Tokenization/HuggingFace/HuggingFaceTokenizerLoader.cs (1)

• SpecialTokens (312-340)

src/Tokenization/Algorithms/CharacterTokenizer.cs (1)

• List (36-59)

src/Tokenization/Core/TokenizerBase.cs (7)

• List (117-120)
• List (144-147)
• List (152-152)
• List (157-160)
• List (165-168)
• List (173-186)
• List (191-200)

tests/AiDotNet.Tests/UnitTests/Tokenization/SpecializedTokenizerTests.cs (1)

src/Tokenization/Specialized/MidiTokenizer.cs (1)

• MidiNote (27-33)

src/Tokenization/Interfaces/IVocabulary.cs (1)

src/Tokenization/Vocabulary/Vocabulary.cs (8)

• AddToken (65-77)
• AddTokens (83-89)
• IEnumerable (135-138)
• GetTokenId (96-99)
• GetToken (106-109)
• ContainsToken (116-119)
• ContainsId (126-129)
• Clear (143-148)

src/Tokenization/Vocabulary/Vocabulary.cs (1)

src/Tokenization/Interfaces/IVocabulary.cs (8)

• AddToken (20-20)
• AddTokens (26-26)
• IEnumerable (60-60)
• GetTokenId (33-33)
• GetToken (40-40)
• ContainsToken (47-47)
• ContainsId (54-54)
• Clear (75-75)

src/Tokenization/Specialized/PhonemeTokenizer.cs (2)

src/Tokenization/Core/TokenizerBase.cs (10)

• TokenizerBase (12-206)
• TokenizerBase (32-36)
• List (117-120)
• List (144-147)
• List (152-152)
• List (157-160)
• List (165-168)
• List (173-186)
• List (191-200)
• CleanupTokens (205-205)

src/Tokenization/Models/SpecialTokens.cs (6)

• SpecialTokens (8-107)
• SpecialTokens (73-80)
• SpecialTokens (85-91)
• SpecialTokens (96-101)
• SpecialTokens (106-106)
• List (53-68)

src/Tokenization/Algorithms/BpeTokenizer.cs (5)

src/Tokenization/Core/TokenizerBase.cs (10)

• TokenizerBase (12-206)
• TokenizerBase (32-36)
• List (117-120)
• List (144-147)
• List (152-152)
• List (157-160)
• List (165-168)
• List (173-186)
• List (191-200)
• CleanupTokens (205-205)

src/Tokenization/Models/SpecialTokens.cs (6)

• SpecialTokens (8-107)
• SpecialTokens (73-80)
• SpecialTokens (85-91)
• SpecialTokens (96-101)
• SpecialTokens (106-106)
• List (53-68)

src/Tokenization/Vocabulary/Vocabulary.cs (5)

• IEnumerable (135-138)
• Vocabulary (11-149)
• Vocabulary (37-45)
• Vocabulary (52-58)
• AddToken (65-77)

src/Tokenization/Interfaces/IVocabulary.cs (2)

• IEnumerable (60-60)
• AddToken (20-20)

src/Tokenization/Interfaces/ITokenizer.cs (1)

• List (35-35)

src/Tokenization/Models/TokenizationResult.cs (2)

src/Tokenization/Core/TokenizerBase.cs (8)

• TokenizationResult (41-112)
• List (117-120)
• List (144-147)
• List (152-152)
• List (157-160)
• List (165-168)
• List (173-186)
• List (191-200)

src/Tokenization/Interfaces/ITokenizer.cs (1)

• TokenizationResult (27-27)

src/Tokenization/HuggingFace/AutoTokenizer.cs (1)

src/Tokenization/HuggingFace/HuggingFaceTokenizerLoader.cs (1)

• HuggingFaceTokenizerLoader (18-415)

tests/AiDotNet.Tests/UnitTests/Tokenization/CodeTokenizerTests.cs (1)

tests/AiDotNet.Tests/UnitTests/Tokenization/BpeTokenizerTests.cs (2)

• Fact (34-43)
• Fact (45-57)

src/Tokenization/Core/TokenizerBase.cs (6)

src/Tokenization/Vocabulary/Vocabulary.cs (5)

• Vocabulary (11-149)
• Vocabulary (37-45)
• Vocabulary (52-58)
• GetTokenId (96-99)
• GetToken (106-109)

src/Tokenization/Models/SpecialTokens.cs (6)

• SpecialTokens (8-107)
• SpecialTokens (73-80)
• SpecialTokens (85-91)
• SpecialTokens (96-101)
• SpecialTokens (106-106)
• List (53-68)

src/Tokenization/Models/TokenizationResult.cs (3)

• TokenizationResult (10-73)
• TokenizationResult (60-62)
• TokenizationResult (67-72)

src/Tokenization/Interfaces/ITokenizer.cs (2)

• TokenizationResult (27-27)
• Decode (43-43)

src/Tokenization/Models/EncodingOptions.cs (2)

• EncodingOptions (6-69)
• EncodingOptions (66-68)

src/Tokenization/Interfaces/IVocabulary.cs (2)

• GetTokenId (33-33)
• GetToken (40-40)

tests/AiDotNet.Tests/UnitTests/Tokenization/HuggingFaceLoaderTests.cs (2)

src/Tokenization/HuggingFace/HuggingFaceTokenizerLoader.cs (2)

• HuggingFaceTokenizerLoader (18-415)
• SaveToDirectory (196-223)

src/Tokenization/HuggingFace/TokenizerConfig.cs (1)

• TokenizerConfig (9-76)

src/Tokenization/HuggingFace/HuggingFaceTokenizerLoader.cs (6)

src/Tokenization/HuggingFace/TokenizerConfig.cs (1)

• TokenizerConfig (9-76)

src/Tokenization/Models/SpecialTokens.cs (6)

• SpecialTokens (8-107)
• SpecialTokens (73-80)
• SpecialTokens (85-91)
• SpecialTokens (96-101)
• SpecialTokens (106-106)
• List (53-68)

src/Tokenization/Algorithms/WordPieceTokenizer.cs (5)

• List (132-149)
• List (154-195)
• WordPieceTokenizer (15-222)
• WordPieceTokenizer (27-36)
• WordPieceTokenizer (46-127)

src/Tokenization/Algorithms/SentencePieceTokenizer.cs (5)

• List (136-148)
• List (153-214)
• SentencePieceTokenizer (15-231)
• SentencePieceTokenizer (28-37)
• SentencePieceTokenizer (47-131)

src/Tokenization/Algorithms/BpeTokenizer.cs (4)

• List (158-184)
• BpeTokenizer (16-248)
• BpeTokenizer (29-42)
• BpeTokenizer (52-153)

src/Tokenization/Vocabulary/Vocabulary.cs (3)

• Vocabulary (11-149)
• Vocabulary (37-45)
• Vocabulary (52-58)

src/Tokenization/Algorithms/UnigramTokenizer.cs (2)

src/Tokenization/Core/TokenizerBase.cs (3)

• TokenizerBase (12-206)
• TokenizerBase (32-36)
• CleanupTokens (205-205)

src/Tokenization/Models/SpecialTokens.cs (5)

• SpecialTokens (8-107)
• SpecialTokens (73-80)
• SpecialTokens (85-91)
• SpecialTokens (96-101)
• SpecialTokens (106-106)

src/Tokenization/Specialized/MidiTokenizer.cs (2)

src/Tokenization/Core/TokenizerBase.cs (10)

• TokenizerBase (12-206)
• TokenizerBase (32-36)
• List (117-120)
• List (144-147)
• List (152-152)
• List (157-160)
• List (165-168)
• List (173-186)
• List (191-200)
• CleanupTokens (205-205)

src/Tokenization/Models/SpecialTokens.cs (6)

• SpecialTokens (8-107)
• SpecialTokens (73-80)
• SpecialTokens (85-91)
• SpecialTokens (96-101)
• SpecialTokens (106-106)
• List (53-68)

src/PredictionModelBuilder.cs (3)

src/Tokenization/Configuration/TokenizationConfig.cs (4)

• TokenizationConfig (13-150)
• TokenizationConfig (105-116)
• TokenizationConfig (121-133)
• TokenizationConfig (138-149)

src/Tokenization/HuggingFace/AutoTokenizer.cs (1)

• AutoTokenizer (23-176)

src/Tokenization/Configuration/PretrainedTokenizerModel.cs (1)

• ToModelId (155-182)

src/Models/Results/PredictionModelResult.cs (1)

src/Tokenization/Configuration/TokenizationConfig.cs (4)

• TokenizationConfig (13-150)
• TokenizationConfig (105-116)
• TokenizationConfig (121-133)
• TokenizationConfig (138-149)

tests/AiDotNet.Tests/Tokenization/VocabularyTests.cs (1)

src/Tokenization/Vocabulary/Vocabulary.cs (9)

• Vocabulary (11-149)
• Vocabulary (37-45)
• Vocabulary (52-58)
• ContainsToken (116-119)
• AddToken (65-77)
• GetTokenId (96-99)
• GetToken (106-109)
• AddTokens (83-89)
• Clear (143-148)

🪛 markdownlint-cli2 (0.18.1)

src/Tokenization/README.md

184-184: Fenced code blocks should have a language specified

(MD040, fenced-code-language)

TOKENIZATION_IMPLEMENTATION_SUMMARY.md

16-16: Fenced code blocks should have a language specified

(MD040, fenced-code-language)

---

221-221: Emphasis used instead of a heading

(MD036, no-emphasis-as-heading)

⏰ Context from checks skipped due to timeout of 90000ms. You can increase the timeout in your CodeRabbit configuration to a maximum of 15 minutes (900000ms). (3)

• GitHub Check: Analyze (csharp)
• GitHub Check: Build All Frameworks
• GitHub Check: Codacy Security Scan

🔇 Additional comments (63)

src/Tokenization/Interfaces/IVocabulary.cs (1)

1-76: IVocabulary contract and documentation look consistent and well‑designed.

The interface cleanly captures the needed vocabulary operations, with clear XML docs and sensible exposure of read‑only maps for advanced scenarios.

tests/AiDotNet.Tests/UnitTests/Tokenization/CodeTokenizerTests.cs (2)

1-195: Good breadth of coverage for CodeTokenizer behavior.

These tests cover language selection, keyword handling, encode/decode, empty input, strings, and numbers, giving solid baseline coverage for CodeTokenizer semantics.

---

197-337: CodeBertTokenizer tests exercise key behaviors effectively.

The suite validates construction from a vocabulary, code‑only vs code+NL encoding, special tokens, padding/truncation, attention masks, decode, and the exposed underlying tokenizer, which should catch most integration regressions.

tests/AiDotNet.Tests/UnitTests/Tokenization/UnigramTokenizerTests.cs (1)

1-161: UnigramTokenizer tests provide solid coverage of key behaviors.

Training, Viterbi segmentation, space markers, empty input, encode/decode, position IDs, small‑vocab training, and round‑trip checks are all covered and look consistent with SentencePiece‑style expectations.

src/Tokenization/Models/EncodingOptions.cs (1)

1-69: EncodingOptions surface and defaults look appropriate.

The option set aligns well with typical tokenizer configurations (HF‑style), and defaults (e.g., special tokens on, padding/truncation off by default) are sensible for general use.

tests/AiDotNet.Tests/UnitTests/Tokenization/AutoTokenizerTests.cs (1)

1-255: AutoTokenizer tests give strong confidence in local and cache behaviors.

The suite thoroughly exercises local loading, default cache directory, cache presence checks, listing, clearing (single and all models), name normalization (org--model ↔ org/model), and async loading, which should make regressions in HuggingFace‑style integration very visible.

tests/AiDotNet.Tests/UnitTests/Tokenization/WordPieceTokenizerTests.cs (1)

1-209: WordPieceTokenizer tests robustly exercise core behaviors.

The suite covers training, simple and unknown tokenization, special tokens, attention masks (with and without padding), decode behavior, round‑trip content, special‑token configuration, token type IDs, and batch encoding, giving good confidence in the WordPiece implementation.

tests/AiDotNet.Tests/Tokenization/WordPieceTokenizerTests.cs (1)

1-148: LGTM!

The WordPiece tokenizer tests are well-structured and comprehensive. They properly cover:

• Vocabulary training
• Subword tokenization with ## prefix handling
• Unknown token fallback
• Special token insertion/removal
• Batch encoding
• Round-trip encode/decode

The test expectations align with WordPiece tokenization behavior.

src/Tokenization/Algorithms/CharacterTokenizer.cs (1)

46-50: Index out of range exception when first character is whitespace.

When _includeWhitespace is false and the first character in processedText is whitespace, line 48 will throw an IndexOutOfRangeException because characters.Count is 0.

Add a bounds check before accessing the last element:

     if (!_includeWhitespace && char.IsWhiteSpace(c))
     {
-        if (characters.Count == 0 || characters[characters.Count - 1] != " ")
+        if (characters.Count > 0 && characters[characters.Count - 1] != " ")
             characters.Add(" ");
+        else if (characters.Count == 0)
+            characters.Add(" ");
     }

Or more concisely:

     if (!_includeWhitespace && char.IsWhiteSpace(c))
     {
-        if (characters.Count == 0 || characters[characters.Count - 1] != " ")
+        if (characters.Count == 0 || characters[^1] != " ")
             characters.Add(" ");
     }

Likely an incorrect or invalid review comment.

src/Tokenization/Algorithms/WordPieceTokenizer.cs (3)

27-36: LGTM!

The constructor properly initializes the tokenizer with sensible defaults for BERT-style WordPiece tokenization. The null-coalescing for specialTokens with SpecialTokens.Bert() is appropriate.

---

154-195: LGTM!

The greedy longest-match-first algorithm is correctly implemented. The behavior of returning [UNK] for the entire word when any subword fails is consistent with standard WordPiece semantics.

---

200-221: LGTM!

The CleanupTokens method correctly reconstructs text by handling the continuation prefix and spacing between words.

tests/AiDotNet.Tests/UnitTests/Tokenization/HuggingFaceLoaderTests.cs (3)

24-37: LGTM!

The Dispose pattern is appropriate for test cleanup. Swallowing exceptions during directory deletion is acceptable to prevent test failures due to file locking or other transient cleanup issues.

---

125-139: LGTM!

The roundtrip test is valuable for verifying that vocabulary preservation works correctly through the save/load cycle. This provides good integration coverage for the HuggingFace compatibility layer.

---

263-308: LGTM!

The CreateTokenizerJsonFile helper correctly handles the different vocabulary formats for BPE, WordPiece, and Unigram models. The Unigram format with [token, score] pairs matches the expected HuggingFace format.

src/Tokenization/CodeTokenization/CodeTokenizer.cs (3)

41-52: LGTM!

The constructor properly validates the baseTokenizer parameter and delegates vocabulary/special tokens to the base class.

---

103-138: LGTM!

The tokenization logic correctly handles keywords (preserving them as-is), identifier splitting, and delegation to the base tokenizer. This approach is well-suited for code understanding tasks.

---

182-207: LGTM!

The identifier splitting logic correctly handles both snake_case and camelCase/PascalCase patterns. The regex ([A-Z]?[a-z]+|[A-Z]+(?=[A-Z][a-z]|\b)) properly handles edge cases like "XMLParser" → ["XML", "Parser"].

tests/AiDotNet.Tests/Tokenization/CodeTokenizerTests.cs (3)

12-48: LGTM!

The tests for camelCase and snake_case identifier splitting are well-structured. Using Assert.Contains allows flexibility for any additional tokens the tokenizer might produce.

---

50-90: LGTM!

The keyword recognition tests verify that language-specific keywords are correctly identified and preserved in the tokenization output.

---

92-132: LGTM!

The CodeBertTokenizer tests verify the dual-segment encoding (code + natural language) with proper token type IDs and special token handling. The assertions on TokenTypeIds containing both 0 and 1 confirm segment differentiation is working.

src/Tokenization/Models/TokenizationResult.cs (1)

47-55: LGTM!

The Length and TotalLength computed properties provide useful abstractions. The Length property correctly represents the non-padded token count when AttentionMask is properly populated.

tests/AiDotNet.Tests/Tokenization/VocabularyTests.cs (3)

1-18: LGTM!

The test correctly validates that the constructor initializes the vocabulary with the UNK token and sets the size to 1.

---

20-48: LGTM!

Good coverage of basic AddToken behavior and idempotent duplicate handling.

---

50-106: LGTM!

Tests for GetTokenId, GetToken, and AddTokens are well-structured and cover expected behaviors including unknown token fallback and invalid ID handling.

src/Tokenization/Algorithms/SentencePieceTokenizer.cs (3)

28-37: LGTM!

Constructor properly validates required parameters and delegates to base class.

---

47-131: Training implementation looks reasonable.

The training flow covers character coverage selection, seed vocabulary initialization, and subword candidate scoring. The approach follows standard SentencePiece training concepts.

One minor observation: the subword generation at lines 104-112 has O(corpus_size × text_length²) complexity which could be slow for large corpora, but this is acceptable for training which is typically done offline.

---

219-230: LGTM!

CleanupTokens correctly reverses the whitespace symbol substitution and trims the result.

src/Tokenization/Algorithms/UnigramTokenizer.cs (4)

21-30: LGTM!

Constructor properly validates tokenScores and stores configuration.

---

44-83: Good implementation with proper O(n) reconstruction.

The Viterbi implementation correctly uses Add followed by Reverse() for O(n) backward pass reconstruction, unlike the SentencePiece implementation. The fallback at lines 67-68 gracefully handles positions with no valid tokens by advancing one character with a penalty score, and line 77 ensures unknown tokens are properly substituted.

---

93-138: Training implementation is sound.

The training flow correctly builds substring frequency counts, selects top candidates, and converts to log probabilities. The vocabulary is properly initialized with special tokens before adding scored substrings.

---

85-88: LGTM!

CleanupTokens correctly reverses the space marker substitution and trims leading whitespace.

src/Tokenization/HuggingFace/TokenizerConfig.cs (1)

9-76: LGTM — Clean DTO for HuggingFace config deserialization.

The class correctly maps HuggingFace tokenizer_config.json fields with appropriate JsonProperty attributes. Property types are well-chosen for deserialization.

src/Tokenization/Models/SpecialTokens.cs (1)

53-68: LGTM!

GetAllSpecialTokens() correctly filters out empty tokens before aggregating, which works well with the factory pattern.

src/Tokenization/Vocabulary/Vocabulary.cs (1)

65-77: LGTM!

The AddToken method correctly validates input, handles duplicates by returning the existing ID, and maintains both dictionaries in sync.

src/Tokenization/CodeTokenization/CodeBertTokenizer.cs (1)

30-38: LGTM!

Constructor correctly composes WordPieceTokenizer with CodeTokenizer, using BERT-style special tokens by default.

tests/AiDotNet.Tests/UnitTests/Tokenization/CharacterTokenizerTests.cs (4)

14-23: LGTM!

Basic factory method test verifies the tokenizer is created with a non-empty vocabulary.

---

25-85: LGTM!

Tokenization tests cover key scenarios: character splitting, case handling, whitespace preservation, and empty input handling.

---

102-131: LGTM!

The encode/decode and roundtrip tests properly verify that tokenization is reversible for ASCII text.

---

163-202: LGTM!

Edge case tests for non-ASCII handling, position IDs, and vocabulary size bounds are well-designed.

src/Tokenization/Specialized/PhonemeTokenizer.cs (3)

27-36: LGTM!

Constructor properly validates the g2pRules parameter and stores the phoneme set.

---

41-66: LGTM!

The tokenization logic correctly handles word segmentation and phoneme conversion. The <space> token management properly handles inter-word boundaries.

---

137-156: LGTM!

The ARPAbet factory provides a convenient way to create a phoneme tokenizer with sensible defaults. The duplicate UNK token addition is harmless as AddTokens handles duplicates gracefully.

src/PredictionModelBuilder.cs (4)

22-24: LGTM!

Global usings for tokenization namespaces are appropriately added to support the new tokenization integration.

---

92-94: LGTM!

Tokenization fields follow the established pattern for optional builder configuration.

---

638-642: LGTM!

Tokenizer attachment is consistently applied across all build paths (meta-learning, regular training, RL, and deserialization), ensuring the tokenizer is available on the result regardless of which build method is used.

Also applies to: 1000-1004, 1194-1198, 1313-1317

---

1927-1938: LGTM!

The string-based overload provides good flexibility for custom models while defaulting to a sensible choice when no model is specified. The documentation clearly explains the behavior.

src/Tokenization/Algorithms/BpeTokenizer.cs (3)

1-42: LGTM - Constructor and initialization are well-structured.

The constructor properly validates the merges parameter and provides sensible defaults (GPT-2 pattern for pre-tokenization, GPT special tokens). The regex is compiled for performance.

---

158-184: Tokenize method is correct with effective caching.

The pre-tokenization and caching strategy is sound. The cache stores space-joined tokens which are then split back - this works correctly.

---

189-236: BpeEncode correctly implements the greedy BPE merge algorithm.

The implementation finds the lowest-rank merge pair and applies it iteratively until no more merges are possible. Logic is correct.

src/Tokenization/HuggingFace/AutoTokenizer.cs (2)

39-52: LGTM - FromPretrained correctly distinguishes local paths from Hub models.

The method properly validates input and uses Directory.Exists to determine the loading path.

---

157-175: ListCachedModels implementation is correct.

The method properly handles non-existent cache directories and correctly converts filesystem-safe names back to model names.

src/Tokenization/Configuration/TokenizationConfig.cs (2)

102-149: Factory methods are well-designed with sensible defaults.

ForBert, ForGpt, and ForCode provide appropriate presets for their respective model families. The configurations align with standard practices (e.g., BERT uses token type IDs, GPT truncates from left).

---

66-82: EnableParallelBatchProcessing and ParallelBatchThreshold properties are unused.

EnableParallelBatchProcessing and ParallelBatchThreshold are defined but have no references in the codebase beyond their definition. Remove these unused properties or implement their consumption in tokenization logic.

Note: EnableCaching is actively used throughout the codebase (JitCompiler, AutoTuningPass, DeploymentRuntime, etc.) and should not be removed.

Likely an incorrect or invalid review comment.

tests/AiDotNet.Tests/Benchmarks/TokenizerBenchmarks.cs (2)

32-66: Good benchmark setup with reproducible data generation.

The setup properly trains tokenizers with diverse corpus data and uses a fixed seed for reproducibility. The benchmark data spans multiple text lengths (short/medium/long) for comprehensive coverage.

---

193-203: WordPieceThroughput benchmark is declared but the description says "tokens/sec" without measuring time.

The throughput benchmarks return total token count, but BenchmarkDotNet measures execution time separately. The description "tokens/sec" is accurate since BenchmarkDotNet will compute throughput. This is fine.

src/Tokenization/Core/TokenizerBase.cs (1)

12-36: Well-structured base class with proper null validation.

The constructor correctly validates required dependencies. Properties are appropriately scoped with protected setters.

tests/AiDotNet.Tests/UnitTests/Tokenization/BpeTokenizerTests.cs (2)

17-32: Test fixture design is appropriate.

The shared trained tokenizer in the constructor is a reasonable approach for these tests. The training corpus is diverse enough to exercise the BPE algorithm, and a vocabulary size of 500 provides sufficient coverage.

---

188-205: Truncation test correctly validates max length constraint.

The test properly verifies that the result respects the maximum length. Consider adding a complementary test that verifies truncation with special tokens enabled to ensure [CLS] and [SEP] are preserved correctly when truncation occurs.

src/Tokenization/HuggingFace/HuggingFaceTokenizerLoader.cs (4)

20-21: Static HttpClient is correctly used.

Using a static HttpClient instance is the recommended practice to avoid socket exhaustion. Good implementation.

---

67-86: Tokenizer type detection is reasonable but fragile.

The detection logic relies on string matching in TokenizerClass. This works for common cases but may fail for custom or newer tokenizer classes. The fallback to WordPiece is a safe default.

---

312-340: Special token extraction uses heuristic matching.

The substring-based detection (e.g., lower.Contains("unk")) works for most HuggingFace tokenizers but could misidentify tokens in edge cases. This is an acceptable trade-off for compatibility.

---

388-414: Unigram loader correctly extracts token scores.

The implementation properly parses the vocab array format [[token, score], ...] used by unigram models and preserves the scores for proper Viterbi segmentation. Good implementation.

[coderabbitai]

Actionable comments posted: 4

🧹 Nitpick comments (3)

src/Tokenization/CodeTokenization/CodeTokenizer.cs (1)

156-220: Consider caching regexes and revisiting identifier splitting edge cases

PreTokenizeCode and SplitIdentifier create new Regex instances on every call; if this tokenizer is used heavily (e.g., for large corpora), promoting these to static readonly Regex (optionally with RegexOptions.Compiled) would reduce allocations and improve throughput.

Also, the camelCase/PascalCase pattern intentionally ignores digits inside identifiers (e.g., Http2Request → ["Http", "Request"]), which may or may not be what you want for code models. If preserving numeric suffixes matters, you could extend the pattern or add a post-pass to keep trailing digits as their own token.

These are behavioral/perf tweaks only; current logic is functionally sound.

src/PredictionModelBuilder.cs (1)

92-95: Tokenizer wiring is consistent; consider making ConfigureTokenizer config-only friendly

The new _tokenizer/_tokenizationConfig fields are attached in all the right places (both BuildAsync overloads, RL path, meta-learning path, and DeserializeModel), so any configured tokenizer flows through to PredictionModelResult reliably. The sync/async ConfigureTokenizerFromPretrained* overloads also provide a good balance between convenience and non-blocking HuggingFace I/O.

One ergonomic edge case: calling

builder.ConfigureTokenizer(config: new TokenizationConfig { /* ... */ });

after a previous ConfigureTokenizerFromPretrained* will silently clear _tokenizer because the optional tokenizer parameter defaults to null and is always assigned. If you want to allow “update config, keep existing tokenizer” semantics, you could gate the assignment:

public IPredictionModelBuilder<T, TInput, TOutput> ConfigureTokenizer(
-    ITokenizer? tokenizer = null,
-    TokenizationConfig? config = null)
+    ITokenizer? tokenizer = null,
+    TokenizationConfig? config = null)
{
-    _tokenizer = tokenizer;
-    _tokenizationConfig = config ?? new TokenizationConfig();
+    if (tokenizer != null)
+        _tokenizer = tokenizer;
+
+    if (config != null)
+        _tokenizationConfig = config;
+    else if (_tokenizationConfig == null)
+        _tokenizationConfig = new TokenizationConfig();
     return this;
}

That keeps today’s behavior when you explicitly pass a tokenizer, while making it safer to call the method just to adjust configuration.

Also applies to: 638-642, 1000-1004, 1194-1198, 1313-1317, 1825-1994

src/Tokenization/Core/TokenizerBase.cs (1)

191-199: Make TruncateSequence side selection more robust (enum or validated string)

TruncateSequence relies on side == "left"; any other value silently falls back to right-side truncation. Consider using an enum (or centralized constants with validation) so invalid values are caught early instead of changing behavior implicitly.

📜 Review details

Configuration used: CodeRabbit UI

Review profile: CHILL

Plan: Pro

📥 Commits

Reviewing files that changed from the base of the PR and between 7f105fb and 53220de.

📒 Files selected for processing (14)

• src/Interfaces/IPredictionModelBuilder.cs (2 hunks)
• src/PredictionModelBuilder.cs (7 hunks)
• src/Tokenization/Algorithms/SentencePieceTokenizer.cs (1 hunks)
• src/Tokenization/CodeTokenization/CodeBertTokenizer.cs (1 hunks)
• src/Tokenization/CodeTokenization/CodeTokenizer.cs (1 hunks)
• src/Tokenization/Core/TokenizerBase.cs (1 hunks)
• src/Tokenization/HuggingFace/HuggingFaceTokenizerLoader.cs (1 hunks)
• src/Tokenization/Models/SpecialTokens.cs (1 hunks)
• src/Tokenization/Models/TokenizationResult.cs (1 hunks)
• src/Tokenization/Specialized/MidiTokenizer.cs (1 hunks)
• src/Tokenization/Vocabulary/Vocabulary.cs (1 hunks)
• tests/AiDotNet.Tests/Tokenization/BpeTokenizerTests.cs (1 hunks)
• tests/AiDotNet.Tests/UnitTests/Tokenization/CharacterTokenizerTests.cs (1 hunks)
• tests/AiDotNet.Tests/UnitTests/Tokenization/SpecializedTokenizerTests.cs (1 hunks)

🚧 Files skipped from review as they are similar to previous changes (5)

• tests/AiDotNet.Tests/UnitTests/Tokenization/SpecializedTokenizerTests.cs
• src/Tokenization/CodeTokenization/CodeBertTokenizer.cs
• src/Tokenization/Models/SpecialTokens.cs
• tests/AiDotNet.Tests/Tokenization/BpeTokenizerTests.cs
• tests/AiDotNet.Tests/UnitTests/Tokenization/CharacterTokenizerTests.cs

🧰 Additional context used

🧬 Code graph analysis (6)

src/Tokenization/Models/TokenizationResult.cs (2)

src/Tokenization/Core/TokenizerBase.cs (8)

• TokenizationResult (41-112)
• List (117-120)
• List (144-147)
• List (152-152)
• List (157-160)
• List (165-168)
• List (173-186)
• List (191-200)

src/Tokenization/Interfaces/ITokenizer.cs (1)

• TokenizationResult (27-27)

src/Tokenization/Vocabulary/Vocabulary.cs (1)

src/Tokenization/Interfaces/IVocabulary.cs (8)

• AddToken (20-20)
• AddTokens (26-26)
• IEnumerable (60-60)
• GetTokenId (33-33)
• GetToken (40-40)
• ContainsToken (47-47)
• ContainsId (54-54)
• Clear (75-75)

src/Tokenization/CodeTokenization/CodeTokenizer.cs (4)

src/Tokenization/Core/TokenizerBase.cs (11)

• TokenizerBase (12-206)
• TokenizerBase (32-36)
• List (117-120)
• List (144-147)
• List (152-152)
• List (157-160)
• List (165-168)
• List (173-186)
• List (191-200)
• CleanupTokens (205-205)
• Decode (125-139)

src/Tokenization/Vocabulary/Vocabulary.cs (3)

• Vocabulary (11-168)
• Vocabulary (38-47)
• Vocabulary (54-72)

src/Tokenization/Models/SpecialTokens.cs (6)

• SpecialTokens (8-116)
• SpecialTokens (73-82)
• SpecialTokens (87-96)
• SpecialTokens (101-110)
• SpecialTokens (115-115)
• List (53-68)

src/Tokenization/Interfaces/ITokenizer.cs (5)

• List (35-35)
• List (51-51)
• List (58-58)
• List (65-65)
• Decode (43-43)

src/Tokenization/HuggingFace/HuggingFaceTokenizerLoader.cs (6)

src/Tokenization/Models/SpecialTokens.cs (6)

• SpecialTokens (8-116)
• SpecialTokens (73-82)
• SpecialTokens (87-96)
• SpecialTokens (101-110)
• SpecialTokens (115-115)
• List (53-68)

src/Tokenization/Algorithms/SentencePieceTokenizer.cs (5)

• List (136-148)
• List (153-216)
• SentencePieceTokenizer (15-233)
• SentencePieceTokenizer (28-37)
• SentencePieceTokenizer (47-131)

src/Tokenization/CodeTokenization/CodeTokenizer.cs (3)

• List (116-151)
• List (156-182)
• List (195-220)

src/Tokenization/Algorithms/WordPieceTokenizer.cs (5)

• List (132-149)
• List (154-195)
• WordPieceTokenizer (15-222)
• WordPieceTokenizer (27-36)
• WordPieceTokenizer (46-127)

src/Tokenization/Algorithms/BpeTokenizer.cs (3)

• BpeTokenizer (16-248)
• BpeTokenizer (29-42)
• BpeTokenizer (52-153)

src/Tokenization/Vocabulary/Vocabulary.cs (3)

• Vocabulary (11-168)
• Vocabulary (38-47)
• Vocabulary (54-72)

src/Tokenization/Core/TokenizerBase.cs (5)

src/Tokenization/Vocabulary/Vocabulary.cs (5)

• Vocabulary (11-168)
• Vocabulary (38-47)
• Vocabulary (54-72)
• GetTokenId (110-113)
• GetToken (120-123)

src/Tokenization/Models/SpecialTokens.cs (6)

• SpecialTokens (8-116)
• SpecialTokens (73-82)
• SpecialTokens (87-96)
• SpecialTokens (101-110)
• SpecialTokens (115-115)
• List (53-68)

src/Tokenization/Models/TokenizationResult.cs (3)

• TokenizationResult (10-82)
• TokenizationResult (60-62)
• TokenizationResult (68-81)

src/Tokenization/Interfaces/ITokenizer.cs (2)

• TokenizationResult (27-27)
• Decode (43-43)

src/Tokenization/Interfaces/IVocabulary.cs (2)

• GetTokenId (33-33)
• GetToken (40-40)

src/Tokenization/Specialized/MidiTokenizer.cs (2)

src/Tokenization/Core/TokenizerBase.cs (10)

• TokenizerBase (12-206)
• TokenizerBase (32-36)
• List (117-120)
• List (144-147)
• List (152-152)
• List (157-160)
• List (165-168)
• List (173-186)
• List (191-200)
• CleanupTokens (205-205)

src/Tokenization/Models/SpecialTokens.cs (6)

• SpecialTokens (8-116)
• SpecialTokens (73-82)
• SpecialTokens (87-96)
• SpecialTokens (101-110)
• SpecialTokens (115-115)
• List (53-68)

⏰ Context from checks skipped due to timeout of 90000ms. You can increase the timeout in your CodeRabbit configuration to a maximum of 15 minutes (900000ms). (3)

• GitHub Check: Analyze (csharp)
• GitHub Check: Codacy Security Scan
• GitHub Check: Build All Frameworks

🔇 Additional comments (4)

src/Tokenization/Models/TokenizationResult.cs (1)

12-81: Constructor validation and shape invariants look solid

The non-null checks plus the explicit tokens.Count == tokenIds.Count guard give you a clear, consistent contract between Tokens, TokenIds, and AttentionMask. The DTO-style mutable lists are reasonable here given how the type is used across the tokenization stack.

src/Tokenization/Vocabulary/Vocabulary.cs (1)

38-72: UNK handling and next-id initialization are consistent

The updated constructor logic (_nextId from Max()+1 with empty-guard and explicit insertion of unkToken when missing) plus the Clear() re-registration of the UNK token give you a stable unknown-id contract across all code paths.

src/Interfaces/IPredictionModelBuilder.cs (1)

865-986: Tokenization builder API surface is coherent and matches implementation

The new ConfigureTokenizer* overloads (sync + async, enum + string) fit the existing fluent builder pattern, their XML docs reflect the actual defaults (BertBaseUncased / "bert-base-uncased"), and the async variants give a non-blocking path for HuggingFace downloads.

src/Tokenization/CodeTokenization/CodeTokenizer.cs (1)

57-108: Language keyword handling, including TypeScript, looks comprehensive

The per-language keyword sets are sensible, and adding a dedicated ProgrammingLanguage.TypeScript case (instead of falling through to empty) fixes the earlier gap for TS code-awareness without impacting other languages.

[github-advanced-security]

CodeQL found more than 20 potential problems in the proposed changes. Check the Files changed tab for more details.