Skip to content

fix(compat): reconcile manifest ownership and fix trace perplexity - #1474

Merged
inureyes merged 1 commit into
mainfrom
fix/epic-1431-consolidation
Aug 27, 2026
Merged

inureyes merged 1 commit into
mainfrom
fix/epic-1431-consolidation

Conversation

@inureyes

Copy link
Copy Markdown
Member

Reconciles the b10621 compatibility manifest after wave 1 of epic #1431, and corrects a sign error in the teacher-forced trace comparison.

Ownership drift: nine entries filed under an issue that cannot fix them

Each of these carried a divergence naming a different issue than its own issue field, because its shard's owner set left no alternative. They move to the issue that owns closing the gap: GET /health and GET /v1/health to #1440, GET /models and GET /v1/models to #1438, POST /completion and POST /completions to #1466, POST /embedding and POST /embeddings to #1452, and --alias to #1438. pin.json gains the matching owners so the ownership check accepts them.

This does not make any shard editable by two concurrently running chains. routes gains #1438, #1440 and #1466, all of which belong to chain A alongside its existing #1441, #1442 and #1452. model-source gains #1438 next to #1434, and #1434 is closed, so chain B will not edit that shard again.

The reopened issues: twenty-four entries move to tracked follow-ups

Wave 1 merged and closed #1450 and #1453 while twenty-four of their entries were still deferred. The gate requires a deferred entry's issue to be open, so that turned the llama-compat manifest job red for every open pull request in the repository, twice. #1472 now owns the thirteen context and batching entries and #1473 the eleven prompt-cache and slot-scheduling entries, each reproducing the entry's recorded divergence verbatim; runtime-and-context gains both as owners. #1450 remains an owner because six supported entries still reference it.

The issue numbers were shipped to operators

--cache-reuse's startup refusal, the YaRN refusal and the RoPE override refusal all ended with Tracked by #1450. or Tracked by #1453., asserted by their unit tests. Closing those issues would have sent operators to closed issues, so the diagnostics and the assertions now name #1472 and #1473.

The trace comparison reported the reciprocal of perplexity

scripts/compare_logit_traces.py computed exp(-mean(nll)) and printed it under a perplexity header. Column four of a trace is the true NLL and is positive, because examples/logit_trace.rs writes -log_softmax(target), so that expression is the geometric mean probability, which is the reciprocal of perplexity. A candidate that got worse therefore reported a negative percentage under a header that read like an improvement. Eight sub-issues of this epic still owe teacher-forced logit traces, so the sign is load-bearing. Verified against a synthetic pair with NLL 0.1 and 1.0: the fixed script reports 1.1052 against 2.7183 at +145.960%, matching exp(0.1) and exp(1.0) analytically. The line now also says "higher is worse".

Verification

python3 scripts/ci/check_llama_compat_manifest.py passes: 249 help entries, 323 spellings, 136 env vars, 53 routes, 74 native fields, states {supported: 45, aliased: 12, not_applicable: 61, deferred: 258}, 20 distinct deferred issues. A generated cross-check confirms all 17 rows of the shard ownership table in docs/llama-server-compat.md now match pin.json; that check is what caught the chat-templates row still omitting #1470, which wave 1 added to pin.json without updating the table. cargo test --lib cli:: 137 passed, --test llama_compat_manifest 4 passed including the archive-gated conformance test, cargo fmt --all -- --check clean.

Part of #1431.

The b10621 compatibility manifest accumulated three kinds of drift during wave 1 of epic #1431, and the teacher-forced trace comparison reported the reciprocal of the metric it named.

Nine entries were filed under an issue that cannot fix them, because their shard's owner set left no alternative. `GET /health` and `GET /v1/health` sat under #1441 while their recorded divergence names #1440; `GET /models` and `GET /v1/models` likewise name #1438; `POST /completion` and `POST /completions` name #1466; `POST /embedding` and `POST /embeddings` name #1452; and `--alias` sat under #1434 while naming #1438. Each moves to the issue that owns closing its gap, and `pin.json` gains the corresponding owners so the ownership check accepts them: `routes` gains #1438, #1440 and #1466, `model-source` gains #1438. No shard becomes editable by two chains that run concurrently, because #1434 is closed and every other owner of those two shards belongs to chain A.

Twenty-four entries were owned by #1450 and #1453, which wave 1 merged and closed while those entries were still deferred. The gate requires a deferred entry's issue to be open, so closing them turned the `llama-compat manifest` job red for every open pull request in the repository. The remainder now belongs to #1472 (the thirteen context and batching entries) and #1473 (the eleven prompt-cache and slot-scheduling entries), which carry each entry's divergence text verbatim, and `runtime-and-context` gains both as owners. #1450 stays an owner because six supported entries still reference it.

The two issue numbers were also baked into operator-facing startup diagnostics and asserted by their tests, so closing the issues would have pointed operators at closed issues. `--cache-reuse`'s refusal, the YaRN refusal and the RoPE override refusal now name #1473 and #1472, and the assertions follow.

`scripts/compare_logit_traces.py` computed `exp(-mean(nll))` and printed it as perplexity. Column four of a trace is the true NLL, positive, because `examples/logit_trace.rs` writes `-log_softmax(target)`, so that expression is the geometric mean probability, the reciprocal of perplexity. A candidate that got worse therefore showed a negative percentage under a header that read like an improvement. Eight sub-issues of this epic still owe teacher-forced logit traces, so the sign matters. The output now carries "higher is worse".

Also syncs the shard ownership table in the operator documentation, including the `chat-templates` row that gained #1470 during wave 1 without the table following.
@inureyes inureyes added type:bug Bug fixes, error corrections, or issue resolutions priority:high High priority area:docs User and developer documentation area:architecture Architecture and code structure changes status:review Under review status:done Completed and removed status:review Under review labels Aug 27, 2026
@inureyes
inureyes merged commit f4aec89 into main Aug 27, 2026
12 checks passed
@inureyes
inureyes deleted the fix/epic-1431-consolidation branch August 27, 2026 14:49
@inureyes inureyes self-assigned this Aug 31, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area:architecture Architecture and code structure changes area:docs User and developer documentation priority:high High priority status:done Completed type:bug Bug fixes, error corrections, or issue resolutions

Projects

None yet

Development

Successfully merging this pull request may close these issues.

fix(inference): align context, batching, RoPE, and runtime override semantics

1 participant