[dsenchankau]

Issue: extends #1337 (shipped in 0.1.41); diagnostics for #2188 (still open)

Problem and result

--calibrate restarts the engine for every setting it measures. Since 0.1.41, a candidate whose engine won't start just skips that candidate and keeps what was measured so far (#1337). But an engine that starts fine and then dies during its own measurement still fails the whole calibration.

I hit this on a UD-Q4_K_XL run: --adapt-swaps 80 measured fine, then the --adapt-swaps 160 engine died at the adaptive tier's copy gate (the same error as #2188). About twenty minutes of --pcie-frac and --spec-min-p results were lost.

After this change a dead engine drops only its own candidate. The earlier settings survive and the winner is picked from what actually measured; in my run 80 swaps beat the default by 10% and was still written even though 160 died. The engine's timeout line for that gate now also prints how far the copies got, which is what's missing in #2188.

Related discussion

#1337 (closed; this extends its rule from failed starts to mid-measurement deaths) and #2188 (open; the engine part here is diagnostics only, it fixes nothing). I don't think this needs a scope agreement first: a calibration where nothing dies behaves exactly as before, and the new C++ code only runs after a wait has already timed out.

Approach

Steps 4, 5 and 6 (worker counts, expert tier, the AMD shared-expert fork) already had the failed-start rule through restart() and failed_starts. I routed a mid-measurement death through the same path and folded the three nearly identical blocks into one restart-and-measure helper. Step 6 had no mid-measurement protection at all and failed the whole run the same way. The first start (steps 1-3) still fails the run; nothing has been measured yet, so there is nothing to keep.

On the engine side, when the adaptive gate's wait times out the message now carries the gate's reached counter, its atomic, and the copy stream's pinned value, and says that the #267 release drains the window's own streams and never the tier's copy stream. With STRATA_VERIFY_TRACE=1 it also dumps the window's trace. Nothing changed for the case where the wait succeeds.

Validation

Revision eb07e55. Ubuntu 24.04, i7-13700F, RTX 4080 SUPER (sm_89), CUDA 13.4 (nvcc V13.4.92, cuda_13.4.r13.4/compiler.38855100_0). I re-ran everything on the upstream main commit this branch is rebased on, as a baseline, and the failing lists below are the same on both.

Passed:

cmake -G Ninja -S . -B build-tests -DCMAKE_BUILD_TYPE=Release \
  -DSTRATA_ENABLE_CUDA=ON -DSTRATA_BUILD_TESTS=ON \
  -DSTRATA_Q6K_EXPERTS=ON -DSTRATA_MMQ_KQUANTS=ON -DCMAKE_CUDA_ARCHITECTURES=89
cmake --build build-tests -j "$(nproc)"
ctest --test-dir build-tests --output-on-failure

Build is clean. ctest: "96% tests passed, 6 tests failed out of 149". expert_multi_test skips itself on this CPU (no AVX-512). The six failures (ple_parity, ggml_tq_overlay, native_hadamard_test, tq_cpu_parity, tq_cpu_parity_avx2, isa_guard_check) are the same six on the baseline, same command, same build dir. Nothing new fails on this branch.

PYTHONPATH=. .venv/bin/python tools/test_calibrate.py

33 tests, OK. The baseline has 30; the three new ones cover the cases above: a tier candidate dies and the faster candidate still wins, a worker count dies and the rest still measures, a fork candidate dies and the run still returns.

for t in tools/test_*.py serve/test_*.py bench/test_*.py; do
  PYTHONPATH=. .venv/bin/python "$t" >/dev/null 2>&1 || echo "FAILED: $t"
done

Six failures, the same six on the baseline: test_iq_pack, test_nvfp4_local_pack, test_setup_download, test_setup_engine_hash, test_setup_golden, test_gfx906_qsa_query_swizzle_guard. All environmental, none caused by this change.

Failed: the ctest and python failures listed above. Every one of them reproduces the same way on the baseline, so none comes from this change.

Not run:

• HIP compile of src/core/verify.cpp, I don't have an AMD card. The new block is ordinary C++ in the file CUDA and HIP share, and it compiles on its own under -Wall -Wextra -Wformat=2, but that's a static check, not a HIP build. Could someone with a HIP setup compile this one file?
• Actually seeing the new timeout line. It needs the real 0.1.43.1, Windows: STRATA_ADAPT_OVERLAP=1 stops the first request with "the adaptive tier's copies for layer 45 never landed" (RTX 4070 SUPER); STRATA_ADAPT_SIGNAL=host works #2188 stop (GPU + adaptive tier + STRATA_ADAPT_OVERLAP=1), and the STRATA_TEST_VERIFY_STALL hook stalls the window's flag waits, not the gate, so it can't fake it either. The 0.1.43.1, Windows: STRATA_ADAPT_OVERLAP=1 stops the first request with "the adaptive tier's copies for layer 45 never landed" (RTX 4070 SUPER); STRATA_ADAPT_SIGNAL=host works #2188 reporters: a rerun on this branch with STRATA_VERIFY_TRACE=1 would give the evidence the thread needs.
• SYCL. The sycl tree has no adaptive gate at all (no adapt_gate_wait there), so this file is not built for SYCL.

Limits

The python tests exercise calibrate.py against a fake engine; nothing crashes a real engine end to end in the tests. And this doesn't fix #2188, it only prints the values its diagnosis needs.

[Niko1221]

Thanks for the pull request.

We're resetting the tracker for 0.2.0-beta. There are more than 1,300 open issues and pull requests, too many for us to review properly, so we're closing everything that isn't being worked on right now. Closed here does not mean rejected.

To reopen it takes about two minutes:

1. Rebase on main if it no longer applies, and re-run your checks on 0.2.0-beta.
2. Start the title with its area: [Engine], [GUI], [TUI], [Benchmark], [Docs] or [Other].
3. Fill in the pull request template: the problem and result, and the validation (what you ran, on which revision, what you saw).
4. Comment /reopen. A check runs and either reopens the pull request or lists what is still missing.

The rules are in CONTRIBUTING.md.

[dsenchankau]

/reopen

[github-actions]

Not reopened yet. Still missing:

• Fill in "Validation": the exact commands, the revision and environment tested, what you saw. Keep Passed, Failed and Not run apart.

Edit the pull request and comment /reopen again. Details: CONTRIBUTING.md

[dsenchankau]

/reopen

[github-actions]

The check passed, but GitHub would not reopen it (Validation Failed: ). If the branch was deleted, restore it or open a new pull request and link this one.