Repository navigation
fix(compat): reconcile manifest ownership and fix trace perplexity - #1474
Merged
Merged
Conversation
The b10621 compatibility manifest accumulated three kinds of drift during wave 1 of epic #1431, and the teacher-forced trace comparison reported the reciprocal of the metric it named. Nine entries were filed under an issue that cannot fix them, because their shard's owner set left no alternative. `GET /health` and `GET /v1/health` sat under #1441 while their recorded divergence names #1440; `GET /models` and `GET /v1/models` likewise name #1438; `POST /completion` and `POST /completions` name #1466; `POST /embedding` and `POST /embeddings` name #1452; and `--alias` sat under #1434 while naming #1438. Each moves to the issue that owns closing its gap, and `pin.json` gains the corresponding owners so the ownership check accepts them: `routes` gains #1438, #1440 and #1466, `model-source` gains #1438. No shard becomes editable by two chains that run concurrently, because #1434 is closed and every other owner of those two shards belongs to chain A. Twenty-four entries were owned by #1450 and #1453, which wave 1 merged and closed while those entries were still deferred. The gate requires a deferred entry's issue to be open, so closing them turned the `llama-compat manifest` job red for every open pull request in the repository. The remainder now belongs to #1472 (the thirteen context and batching entries) and #1473 (the eleven prompt-cache and slot-scheduling entries), which carry each entry's divergence text verbatim, and `runtime-and-context` gains both as owners. #1450 stays an owner because six supported entries still reference it. The two issue numbers were also baked into operator-facing startup diagnostics and asserted by their tests, so closing the issues would have pointed operators at closed issues. `--cache-reuse`'s refusal, the YaRN refusal and the RoPE override refusal now name #1473 and #1472, and the assertions follow. `scripts/compare_logit_traces.py` computed `exp(-mean(nll))` and printed it as perplexity. Column four of a trace is the true NLL, positive, because `examples/logit_trace.rs` writes `-log_softmax(target)`, so that expression is the geometric mean probability, the reciprocal of perplexity. A candidate that got worse therefore showed a negative percentage under a header that read like an improvement. Eight sub-issues of this epic still owe teacher-forced logit traces, so the sign matters. The output now carries "higher is worse". Also syncs the shard ownership table in the operator documentation, including the `chat-templates` row that gained #1470 during wave 1 without the table following.
This was referenced Aug 27, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Reconciles the b10621 compatibility manifest after wave 1 of epic #1431, and corrects a sign error in the teacher-forced trace comparison.
Ownership drift: nine entries filed under an issue that cannot fix them
Each of these carried a
divergencenaming a different issue than its ownissuefield, because its shard's owner set left no alternative. They move to the issue that owns closing the gap:GET /healthandGET /v1/healthto #1440,GET /modelsandGET /v1/modelsto #1438,POST /completionandPOST /completionsto #1466,POST /embeddingandPOST /embeddingsto #1452, and--aliasto #1438.pin.jsongains the matching owners so the ownership check accepts them.This does not make any shard editable by two concurrently running chains.
routesgains #1438, #1440 and #1466, all of which belong to chain A alongside its existing #1441, #1442 and #1452.model-sourcegains #1438 next to #1434, and #1434 is closed, so chain B will not edit that shard again.The reopened issues: twenty-four entries move to tracked follow-ups
Wave 1 merged and closed #1450 and #1453 while twenty-four of their entries were still
deferred. The gate requires a deferred entry's issue to be open, so that turned thellama-compat manifestjob red for every open pull request in the repository, twice. #1472 now owns the thirteen context and batching entries and #1473 the eleven prompt-cache and slot-scheduling entries, each reproducing the entry's recorded divergence verbatim;runtime-and-contextgains both as owners. #1450 remains an owner because sixsupportedentries still reference it.The issue numbers were shipped to operators
--cache-reuse's startup refusal, the YaRN refusal and the RoPE override refusal all ended withTracked by #1450.orTracked by #1453., asserted by their unit tests. Closing those issues would have sent operators to closed issues, so the diagnostics and the assertions now name #1472 and #1473.The trace comparison reported the reciprocal of perplexity
scripts/compare_logit_traces.pycomputedexp(-mean(nll))and printed it under aperplexityheader. Column four of a trace is the true NLL and is positive, becauseexamples/logit_trace.rswrites-log_softmax(target), so that expression is the geometric mean probability, which is the reciprocal of perplexity. A candidate that got worse therefore reported a negative percentage under a header that read like an improvement. Eight sub-issues of this epic still owe teacher-forced logit traces, so the sign is load-bearing. Verified against a synthetic pair with NLL 0.1 and 1.0: the fixed script reports 1.1052 against 2.7183 at +145.960%, matchingexp(0.1)andexp(1.0)analytically. The line now also says "higher is worse".Verification
python3 scripts/ci/check_llama_compat_manifest.pypasses: 249 help entries, 323 spellings, 136 env vars, 53 routes, 74 native fields, states{supported: 45, aliased: 12, not_applicable: 61, deferred: 258}, 20 distinct deferred issues. A generated cross-check confirms all 17 rows of the shard ownership table indocs/llama-server-compat.mdnow matchpin.json; that check is what caught thechat-templatesrow still omitting #1470, which wave 1 added topin.jsonwithout updating the table.cargo test --lib cli::137 passed,--test llama_compat_manifest4 passed including the archive-gated conformance test,cargo fmt --all -- --checkclean.Part of #1431.