[Engine] Diagnostic: fixed-config CUDA split A/A matches early state but varies per-verify timing and later path #4069

Summary

On a four-stage native CUDA Qwen3.8-Flash-Next setup, two controlled A/A requests produced identical output and identical valid state at three early checkpoints, but showed large differences in individual verify-call durations and a later difference in the recorded speculative execution path.

This is a bounded diagnostic observation, not a confirmed correctness bug, a root-cause claim, or a performance result. Only two generation requests were executed. We stopped before the planned 8192-chunk and MTP-off comparisons.

Related: #3976. This report uses the ordinary chunk-major path; it does not test or validate the layer-major branch proposed there.

Tested configuration and scope

• Test date: 2026-10-25.
• Baseline source: 5c8dd322528cc84f68f461ad54a506b9da8fa8d4, with local diagnostic instrumentation in native generation/state export and a service configuration override. This is not a reproduction on current upstream main or v0.2.4.2.
• Diagnostic executable SHA-256: bb472f0af8145c18c1b324f4056e8319ac8dedd8f364b018c2c135f963f2a1d9. Both arms used this same executable.
• Model: Qwen3.8-Flash-Next, GSQ-RCO-IQ3_S weights; Q2_0 MTP sidecar enabled in both arms. Model/profile/reserve settings were held fixed.
• Four CUDA stages, layer split 12,24,36; configured parallel slots: 16, with one diagnostic request at a time.
• --prefill 2048 --pcie-frac 0.55 --spec 4 --kv int8 --expert-cache auto; observed PCIe fraction was 0.55 on all four stages. Requested --spec 4 is not a claim that every effective verify window has T=4; the recorded verify maximum was 8.
• Same 130,658-token six-needle retrieval input, seed 20261018, and max_new=32768.
• Each arm started a fresh candidate engine; conversation-cache reuse was zero. OS page cache was not cleared, and "cold" here means conversation-cache-cold.
• Effective configuration and the exported initial logical expert-residency maps matched. These maps are logical host-side records, not independent physical GPU-memory readbacks.

The raw prompt, weights and full state exports are not attached. This is therefore a diagnostic summary, not a self-contained public reproducer.

Early state and output checks

There were 93 / 94 / 93 comparison items at the respective checkpoints, covering all four stage sessions. Unused paged tails were excluded. Existing acceptance thresholds were kept at NRMSE <= 2e-4 and logits KL <= 1e-5; no tolerance was relaxed.

The actual boundary matters: before first verify, p=130657 and T=1. After verify, valid main K/V extends to 130658 while the recurrent checkpoint remains at 130657; after commit, both reach 130658. Draft K/V at that commit checkpoint precedes the subsequent MTP write. We did not compare invalid/speculative rows as if they were committed state.

Both requests returned exactly the same correct six-needle JSON, 231 output tokens, and a natural stop. These checks establish equality at the exported early checkpoints; they do not establish equality of all later hidden states.

Timing and execution-path observations

Timings below are monotonic host-side intervals around the instrumented native calls, not individual CUDA-kernel timings.

The first 26 windows matched in the recorded execution fields (p, T, T_mtp, merged, from_sfx, chain_n). Thus the second-window difference of 591.667 ms compares the same recorded position, T and draft source; it does not prove identical kernel launches or scheduling.

The first recorded path difference occurred at zero-based window 26, p=130769:

• A1: T=6, T_mtp=6, from_sfx=true, merged=false, chain_n=0.
• A2: T=4, T_mtp=4, from_sfx=false, merged=false, chain_n=0.

Windows after that point should not be paired mechanically by index. A2 also had a 395.988 ms T=2 verify interval at its own zero-based window 45; we do not treat it as a matched A1 event.

Stop decision and measurement limitations

The aggregate timing gate would pass: aggregate intervals were close despite the per-window changes. On observing the per-window timing/path differences, we stopped the campaign under the predeclared "unstable A/A => stop attribution" rule. The request ledger contained only A1 and A2; no 8192 arm or MTP-off arm was sent. Subsequent work was offline analysis of those two requests.

• Neither request had OOM, host swap or a resource-guard failure. Minimum sampled free GPU memory was 1604 / 1600 MiB; peak sampled temperatures were 59 / 63 C; host available memory remained above 100 GiB.
• Per-GPU sampling intervals had a median of about 100.4 ms and maximum of about 111.6 ms. All four GPUs were observed at PCIe Gen4 x16. Across 25 samples spanning the slow A1 T=4 interval, sampled SM clocks were 2520 MHz. This does not exclude shorter events, scheduling, synchronization or initialization effects.
• CPU/expert counters cover only the primary drive, not all stages; zero counters cannot establish absence of CPU/cache work elsewhere.
• First-kernel/CUDA-graph capture counters were unavailable and recorded as unknown. We do not have a complete kernel-launch trace.
• State export, CUDA synchronization and per-window tracing perturb execution. Subtracting export time does not produce an uninstrumented benchmark.
• Two runs are not a statistical characterization. Path variability might be expected under some scheduling conditions; this report does not establish that it is erroneous.
• No conclusion is established about MTP causality, cross-chunk numerical equivalence, a specific kernel, layer-major speedup, or suitability for deployment.

Question for maintainers

For this native CUDA split path, is the observed later suffix/MTP path variation expected with the same request seed and matching early state? Which existing counter or trace point would best distinguish verify-call synchronization/initialization cost from actual model work before attempting any cross-chunk attribution?

We are keeping this separate from the cache-copy work in #4002 and linking #3976 only for validation-method context. No fix is claimed.

Only allowlisted configuration fields, hashes and aggregate diagnostic measurements are included here. No internal addresses, hostnames, filesystem paths, credentials, raw request text, model files or raw state dumps are published.