[dsenchankau]

Issue: extends #1337 (shipped in 0.1.41); diagnostics for #2188 (still open)

Problem and result

--calibrate restarts the engine for every setting it measures. Since 0.1.41, a candidate whose engine won't start just skips that candidate and keeps what was measured so far (#1337). But an engine that starts fine and then dies during its own measurement still fails the whole calibration.

I hit this on a UD-Q4_K_XL run: --adapt-swaps 80 measured fine, then the --adapt-swaps 160 engine died at the adaptive tier's copy gate (the same error as #2188). About twenty minutes of --pcie-frac and --spec-min-p results were lost.

After this change a dead engine drops only its own candidate. The earlier settings survive and the winner is picked from what actually measured; in my run 80 swaps beat the default by 10% and was still written even though 160 died. The engine's timeout line for that gate now also prints how far the copies got, which is what's missing in #2188.

Related discussion

#1337 (closed; this extends its rule from failed starts to mid-measurement deaths) and #2188 (open; the engine part here is diagnostics only, it fixes nothing). I don't think this needs a scope agreement first: a calibration where nothing dies behaves exactly as before, and the new C++ code only runs after a wait has already timed out.

Approach

Steps 4, 5 and 6 (worker counts, expert tier, the AMD shared-expert fork) already had the failed-start rule through restart() and failed_starts. I routed a mid-measurement death through the same path and folded the three nearly identical blocks into one restart-and-measure helper. Step 6 had no mid-measurement protection at all and failed the whole run the same way. The first start (steps 1-3) still fails the run; nothing has been measured yet, so there is nothing to keep.

On the engine side, when the adaptive gate's wait times out the message now carries the gate's reached counter, its atomic, and the copy stream's pinned value, and says that the #267 release drains the window's own streams and never the tier's copy stream. With STRATA_VERIFY_TRACE=1 it also dumps the window's trace. Nothing changed for the case where the wait succeeds.

Validation

Revision eb07e55. Ubuntu 24.04, i7-13700F, RTX 4080 SUPER (sm_89), CUDA 13.4 (nvcc V13.4.92, cuda_13.4.r13.4/compiler.38855100_0). I re-ran everything on the upstream main commit this branch is rebased on, as a baseline, and the failing lists below are the same on both.

Passed:

cmake -G Ninja -S . -B build-tests -DCMAKE_BUILD_TYPE=Release \
  -DSTRATA_ENABLE_CUDA=ON -DSTRATA_BUILD_TESTS=ON \
  -DSTRATA_Q6K_EXPERTS=ON -DSTRATA_MMQ_KQUANTS=ON -DCMAKE_CUDA_ARCHITECTURES=89
cmake --build build-tests -j "$(nproc)"
ctest --test-dir build-tests --output-on-failure

Build is clean. ctest: "96% tests passed, 6 tests failed out of 149". expert_multi_test skips itself on this CPU (no AVX-512). The six failures (ple_parity, ggml_tq_overlay, native_hadamard_test, tq_cpu_parity, tq_cpu_parity_avx2, isa_guard_check) are the same six on the baseline, same command, same build dir. Nothing new fails on this branch.

PYTHONPATH=. .venv/bin/python tools/test_calibrate.py

33 tests, OK. The baseline has 30; the three new ones cover the cases above: a tier candidate dies and the faster candidate still wins, a worker count dies and the rest still measures, a fork candidate dies and the run still returns.

for t in tools/test_*.py serve/test_*.py bench/test_*.py; do
  PYTHONPATH=. .venv/bin/python "$t" >/dev/null 2>&1 || echo "FAILED: $t"
done

Six failures, the same six on the baseline: test_iq_pack, test_nvfp4_local_pack, test_setup_download, test_setup_engine_hash, test_setup_golden, test_gfx906_qsa_query_swizzle_guard. All environmental, none caused by this change.

Failed: the ctest and python failures listed above. Every one of them reproduces the same way on the baseline, so none comes from this change.

Not run:

• HIP compile of src/core/verify.cpp, I don't have an AMD card. The new block is ordinary C++ in the file CUDA and HIP share, and it compiles on its own under -Wall -Wextra -Wformat=2, but that's a static check, not a HIP build. Could someone with a HIP setup compile this one file?
• Actually seeing the new timeout line. It needs the real 0.1.43.1, Windows: STRATA_ADAPT_OVERLAP=1 stops the first request with "the adaptive tier's copies for layer 45 never landed" (RTX 4070 SUPER); STRATA_ADAPT_SIGNAL=host works #2188 stop (GPU + adaptive tier + STRATA_ADAPT_OVERLAP=1), and the STRATA_TEST_VERIFY_STALL hook stalls the window's flag waits, not the gate, so it can't fake it either. The 0.1.43.1, Windows: STRATA_ADAPT_OVERLAP=1 stops the first request with "the adaptive tier's copies for layer 45 never landed" (RTX 4070 SUPER); STRATA_ADAPT_SIGNAL=host works #2188 reporters: a rerun on this branch with STRATA_VERIFY_TRACE=1 would give the evidence the thread needs.
• SYCL. The sycl tree has no adaptive gate at all (no adapt_gate_wait there), so this file is not built for SYCL.

Limits

The python tests exercise calibrate.py against a fake engine; nothing crashes a real engine end to end in the tests. And this doesn't fix #2188, it only prints the values its diagnosis needs.

[dsenchankau]

Supersedes #2935 as GitHub refuses to reopen it after the rebase (Validation Failed)

[Niko1221]

Thank you. Fixed in 0.2.7: An engine that dies mid-measurement drops only its candidate. Your name is on the commit.

Release: https://github.com/Niko1221/Strata/releases/tag/v0.2.7

[dsenchankau]

@Niko1221 that's actually a false statement. My name is nowhere to be found.

• This PR is closed, not merged. And because of that I'm not on the contributors list.
• Your commit 8486336 which merges 7 community-originated PRs lists no names.
• The 0.2.7 release message lists basically no names and doesn't even mention this PR. And likely some or all of the other 6 community PRs from your merge commit.