Repository navigation
Conversation
jadenfix
marked this pull request as ready for review
October 13, 2026 02:40
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Enabling
Bigram.train_bigram(..., smoothing=True)raised AttributeError because it called a nonexistent method. Perplexity also assigned zero to uncached bigrams instead of applying the configured interpolation, defeating smoothing on unseen transitions.Call the existing interpolation method and evaluate uncached transitions through MLE with the stored weights. Handle absent contexts/words using the existing zero/floor policy, validate mixture weights and the required unigram, and clear old probability entries on retraining. Correct README composition/add-k claims and the broken training example; add an executable toy interpolation example.
Validation at main
01c7a0ff921614949f62f55337b574dda5e541ba:python -m unittest -vpasses all 13 tests. New checks use explicit probabilities (.8, .4, .1), an unseen-bigram sentence with hand-calculated perplexity 5, normalized distributions for observed nonterminal contexts, retraining/invalid-weight cases, and the shared trigram interpolation path. Existing unsmoothed/boundary/unknown-word tests andpython demo.py --text 'the agent tests the system'pass. Before the repair, the initial expanded suite produced 10 AttributeError failures across interpolation cases. Both Python 3.12/3.13 CI jobs and GitGuardian pass at head102973cda642bb7beb6e6dc0e3b4c237a98456ef: https://github.com/jadenfix/NLP/actions/runs/38337854923 (PR) and run38337835745 (push). Both README model examples were executed successfully against temporary toy corpus files. All four published blobs exactly match the tested local bytes.The historical benchmark corpus is absent and its results are not reproduced here. Add-k/unigram smoothing is not implemented; the original 1e-10 floor remains, including unknown words without trained UNK mass. This is a focused extracted-module repair, not production-model or accuracy certification. No media captured. Coordinator 100e79 remains sole merger; no frozen InftyThink or CSV branches changed.