Skip to content

Make bigram interpolation work for training and unseen transitions - #2

Open
jadenfix wants to merge 1 commit into
mainfrom
codex/bigram-interpolation-20261013
Open

jadenfix wants to merge 1 commit into
mainfrom
codex/bigram-interpolation-20261013

Conversation

@jadenfix

@jadenfix jadenfix commented Oct 13, 2026 •

Copy link
Copy Markdown
Owner

Enabling Bigram.train_bigram(..., smoothing=True) raised AttributeError because it called a nonexistent method. Perplexity also assigned zero to uncached bigrams instead of applying the configured interpolation, defeating smoothing on unseen transitions.

Call the existing interpolation method and evaluate uncached transitions through MLE with the stored weights. Handle absent contexts/words using the existing zero/floor policy, validate mixture weights and the required unigram, and clear old probability entries on retraining. Correct README composition/add-k claims and the broken training example; add an executable toy interpolation example.

Validation at main 01c7a0ff921614949f62f55337b574dda5e541ba: python -m unittest -v passes all 13 tests. New checks use explicit probabilities (.8, .4, .1), an unseen-bigram sentence with hand-calculated perplexity 5, normalized distributions for observed nonterminal contexts, retraining/invalid-weight cases, and the shared trigram interpolation path. Existing unsmoothed/boundary/unknown-word tests and python demo.py --text 'the agent tests the system' pass. Before the repair, the initial expanded suite produced 10 AttributeError failures across interpolation cases. Both Python 3.12/3.13 CI jobs and GitGuardian pass at head 102973cda642bb7beb6e6dc0e3b4c237a98456ef: https://github.com/jadenfix/NLP/actions/runs/38337854923 (PR) and run38337835745 (push). Both README model examples were executed successfully against temporary toy corpus files. All four published blobs exactly match the tested local bytes.

The historical benchmark corpus is absent and its results are not reproduced here. Add-k/unigram smoothing is not implemented; the original 1e-10 floor remains, including unknown words without trained UNK mass. This is a focused extracted-module repair, not production-model or accuracy certification. No media captured. Coordinator 100e79 remains sole merger; no frozen InftyThink or CSV branches changed.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant