The shipped oracle fixture does not match a container built by a default conversion

[isenbek]

engine → engine matches the PyTorch oracle fails against a Kimi-Linear container converted with current defaults. The engine and the fixture agree on the answer and disagree on precision, which does not look like an engine error — but I cannot rule one out from the fixture alone, which is why this is an issue rather than a PR.

Measurement

tests/fixtures/oracle_kimilinear_16tok.bin (655 360 bytes = 163 840 floats = vocab), against test_forward on the same 16 ids tests/run.sh uses:

lengths:      engine 163840   oracle 163840
argmax agree: True
correlation:  0.964251
max |diff|:   3.2840          mean |diff|: 0.5484      (check threshold: 1e-3)

engine top-5 ids [318, 5243, 387, 14509, 10484]
oracle top-5 ids [318, 5243, 14509, 387, 10484]

Same argmax, same top-5 set with two middle ranks swapped, correlation 0.96. That is the signature of the same computation over differently-quantized weights, not of a wrong one. A genuine engine bug would not track the reference this closely across 163 840 logits.

Why I think it is the fixture

The fixture has not changed since 0d9c995 (git diff --stat ce96e38..HEAD -- tests/fixtures/ is empty; md5 e4b19db9c9abb503fcee46bc97cfe747). tools/kimi_ref.py computes its logits from a WASTE container, so a fixture is only comparable to a container quantized the same way. My container is a plain tools/convert.py --src ... --out ... with no flags, giving a Q4G/Q8G/F32 trunk.

The README's 3.6e-06 agreement is only achievable when the oracle and the engine read the same weights. Nothing records which conversion settings this fixture corresponds to, so a contributor converting from the released shards today gets a failure with no way to tell whether the engine or the fixture is at fault.

What would settle it

Regenerating the oracle from the container under test with tools/kimi_ref.py and re-diffing. That is a pure-PyTorch 48 B forward pass on CPU, and I can run it and report the number if that is useful — it is the only version of this check that means anything on a locally-converted container.

Possible resolutions, all yours to choose

• record the conversion settings the fixture was generated under, and have tests/run.sh skip with that reason when the container's trunk formats differ;
• ship the fixture for the default conversion instead, and regenerate it when the default moves;
• make the check generate its own oracle when uv and the source weights are present, and skip otherwise.

Reproduced on

v0.6.1 (c4d45c5), Linux x86_64 / AVX2. Identical numbers on 0.6.0, so it is not a flake. The same container passes dequantized weights match the source — the round-trip against the original 91.5 GB of safetensors — and C tokenizer matches Python tiktoken.

🤖 Generated with Claude Code

[isenbek]

Ran the regeneration. It settles the question: the engine is correct, and the fixture describes a different container.

Method

tools/kimi_ref.py against the same container the engine reads, on the 16 ids tests/run.sh uses. --dump writes logits[-1] straight after prefill, which is what test_forward CONTAINER IDS out.bin 0 produces, and --tokens 0 skips the generation loop that follows it:

uv run --no-project --with torch --with fla-core --with einops \
  python tools/kimi_ref.py --container ~/models/kimi-linear.waste \
    --prompt-ids 1008,10484,318,15383,387,11,316,276,10484,318,19509,387,31082,13,646,10484 \
    --tokens 0 --dump oracle_ours.bin

loaded in 3.6s: 27 layers, 20 KDA / 7 MLA
prefill 16 tok in 193.8s; logits (16, 163840), argmax 318, max 17.986

Result

comparison	max |diff|	mean |diff|	correlation	argmax
engine v0.6.1 vs fresh oracle	4.58e-05	7.37e-06	1.000000	318 == 318
engine v0.6.1 vs shipped fixture	3.28e+00	5.48e-01	0.964251	318 == 318
fresh oracle vs shipped fixture	3.28e+00	5.48e-01	0.964251	318 == 318

Threshold is 1e-3. The engine agrees with PyTorch on the same weights to 4.58e-05, two orders of magnitude inside it.

The third row is the one that closes it. The PyTorch reference itself diverges from the shipped fixture by exactly the amount the engine does — 3.28 / 0.548 / 0.964251, identical to the digit. kimi_ref.py shares no code with src/model.c, so a discrepancy that reproduces identically in both cannot originate in either; it comes from the weights they both read. The fixture was generated from a container quantized differently from what tools/convert.py produces today.

So nothing in the engine needs fixing for this. What is left is the fixture's provenance, as in the issue body: regenerate it for the default conversion, or record the settings it corresponds to and skip when the container's trunk formats differ.

Two incidental notes from the run:

• the C engine did that same 16-token prefill in 34.5 s against the reference's 193.8 s — ~5.6x, while matching it to 7e-06;
• kimi_ref.py runs fine on Linux under uv with fla-core and einops. The docstring frames the pure-PyTorch path as a way around Triton being unavailable on macOS, and it is equally usable as a portable oracle here.

Environment: v0.6.1 (c4d45c5), Linux x86_64 / AVX2, container from tools/convert.py with no flags (trunk Q4G/Q8G/F32).

🤖 Generated with Claude Code

[marcobambini]

Your follow-up settles it, and it reproduces here independently — different OS, different arch, different container.

macOS arm64 / NEON, against a container converted with today's defaults on this machine:

comparison	max |diff|	mean |diff|	correlation	argmax
engine vs fresh oracle	4.77e-05	7.52e-06	1.000000	318 == 318
engine vs shipped fixture	2.425	0.461	0.979884	318 == 318
fresh oracle vs shipped fixture	2.425	0.461	0.979884	318 == 318

Same structure as yours, different magnitude because the containers differ. The third row matching the second to the digit is the part that closes it, exactly as you argued.

Where the fixture came from

File timestamps answer it. The fixture was written 2026-07-27 18:32, from the container at ~/models/kimi-linear.waste built the same day at 16:38 — whose trunk is Q8G/F32, i.e. a --trunk8 conversion. --trunk-bits has defaulted to 4 for the bulk since before 1fd5136, so the fixture never corresponded to a default conversion, and nothing anywhere recorded that. Exactly the gap you described.

It is now recorded, in tests/fixtures/oracle_kimilinear_16tok.json beside the fixture, and the fixture has been regenerated against a default conversion.

But regenerating it would not have fixed your case

This is the part worth having measured, because it changes the answer. We converted one expert layer with --device cpu and compared it against the same layer converted on mps:

• the trunk is bit-identical — quantization is deterministic arithmetic;
• the expert banks are not — train_codebooks seeds its generator, but k-means on a different device converges to different books;
• splicing a single cpu-trained layer (1 of 26) into an otherwise mps container moved the final logits by 1.24 max / 0.19 mean, against a 1e-3 threshold.

So a regenerated fixture would have given you the right trunk width and still failed, on the codebooks, with a smaller diff and the same ambiguity. No shipped fixture can be portable across conversions. Yours was not a special case; the whole approach has a ceiling.

So we took your third option as well

tests/run.sh now generates its own oracle from the container under test, and keeps the fixture only as the fallback for hosts without uv — where it checks the recorded trunk against the container's and skips, with the reason, rather than producing a diff that reads as an engine bug.

The number that made this the obvious choice is one you nearly had: kimi_ref.py reads the container, not the source shards, so generating costs 16.9 s here, not the 193.8 s of a cold start on 92 GB of safetensors. The check now passes on both containers on this machine — the --trunk8 one and the default one — because each is measured against its own oracle.

Two notes back on yours:

• the 5.6x you measured is a fair figure and matches what we see; the same 16-token prefill is 2.7 s in the engine against 16.9 s for the reference here;
• you are right that kimi_ref.py's pure-PyTorch path is a portable oracle and not just a macOS workaround. The docstring undersold it, and run.sh now depends on that being true on Linux too.

Thanks for both reports — and for not guessing at the fix. The choice in #6 did change engine behaviour, and this one changed what the suite claims to cover.

[marcobambini]

Landed in aa57bff, with the reasoning written up as §33 of docs/LEARNED.md in 3be917d.

Your provenance question has an answer, and it is the one you suspected. File timestamps: the fixture was written 2026-07-27 18:32, from a container built the same day at 16:38 whose trunk is Q8G/F32 — a --trunk8 conversion. The bulk of a trunk has been 4-bit by default since before the Q3G sweep, so the fixture never corresponded to a default conversion and nothing recorded it.

What shipped

The fixture is regenerated against a default conversion, and tests/fixtures/oracle_kimilinear_16tok.json now records what it came from.

But that is the smaller half, because regenerating it would not have fixed your case, which is worth stating since you would have hit it again. One expert layer converted with --device cpu against the same layer on mps:

• the trunk is bit-identical — quantization is deterministic arithmetic;
• the expert bank is not. train_codebooks seeds its generator, but k-means on a different device converges to different books;
• splicing that one layer of 26 into an otherwise-mps container moved the final logits by 1.24 max / 0.19 mean, against the 1e-3 threshold.

So you would have got the right trunk width and failed on the codebooks instead, with a smaller diff and the same ambiguity. No shipped fixture can be portable across conversions.

Hence your third option as well: tests/run.sh now generates its own oracle from the container under test, and keeps the fixture only as the fallback for hosts without uv — where it compares the recorded trunk against the container's and skips, with the reason, rather than producing a diff that reads as an engine bug.

The number that made that the obvious choice is one you nearly had. kimi_ref.py reads the container, not the source shards, so generating costs 16.9 s here — not the 193.8 s of a cold start over 92 GB of safetensors. All four paths verified:

container	uv	result
default (Q4G/Q8G/F32)	yes	PASS, oracle built from the container
--trunk8 (Q8G/F32)	yes	PASS, oracle built from the container
default	no	PASS, shipped fixture, trunk matches
--trunk8	no	SKIP, naming the mismatch

Reproduced independently

macOS arm64 / NEON, a different container from yours, and the same three-row shape:

comparison	max |diff|	mean	correlation
engine vs a fresh oracle	4.77e-05	7.52e-06	1.000000
engine vs the old fixture	2.425	0.461	0.979884
fresh oracle vs the old fixture	2.425	0.461	0.979884

Different magnitude from your 3.28 / 0.548 / 0.964251, because the containers differ. Same structure, and the third row matching the second to the digit is what closes it — exactly as you argued.

One guard came out of this that the issue did not ask for but the fix would otherwise have armed: after regeneration the fixture's trunk is Q4G/Q8G/F32, which is also K3's, so a run pointed at K3 would have compared its logits against a Kimi-Linear fixture with the format check seeing nothing wrong. That now skips on the architecture.

Thanks for the report, and for insisting on the measurement rather than the plausible explanation.