CI integrity: a shard whose runner dies is dropped from the ledger, and the gate reports "improved"

[ooples]

Summary

On a baseline-establishing run (no baseline to compare against), a shard whose runner dies before uploading its artifact is omitted from the ledger entirely rather than recorded as failed or incomplete. The run then self-reports fully green, and the resulting ledger — which later pull requests are measured against — is silently missing that shard.

This means the gate can report Regression status: **improved** on a run that contained a dead shard.

Evidence

Run 33938630584 on master @ 2a8ca6b2d, the most recent run that actually executed test shards.

What the CI jobs say:

• Tests (net10.0) - Integration D → conclusion failure
• Its log ends at 2026-09-05T06:34:24 with every remaining step reporting ##[debug]Skip evaluate condition on runner shutdown — including Record shard outcome, Upload coverage + test results and Upload crash evidence. The runner died ~2.7 minutes into the test step (started 06:31:38). No artifacts were uploaded.

What the ledger says (test-outcome-ledger-2a8ca6b2..., summary.md):

Current master ledger: 115 shards, 115 passed, 0 failed, 0 incomplete.
The TRX files report 0 failing results representing 0 distinct failing tests.
This push establishes the TRX baseline artifact used by later pull requests.

shards.csv contains 115 rows, every one Passed. The 15 Integration shards it lists are:

A-B, C - ComputerVision Detection, C - ComputerVision Segmentation Contracts, C - ComputerVision Segmentation Models, C - Core, E-G, H-L, M, N-O, P-Q, R, S, T-Z, CPU Parallelism Probe, Serial Perf

There is no Integration D row at all.

What the aggregate analysis says (ci-test-analysis.md):

Metric	Reported
Unique failing test cases	0
Failing shards	0
Shards without a result artifact	0
Executed / discovered	67755 / 67792

Regression status: **improved**.

So a run containing a dead shard produced a clean bill of health, and reported zero shards missing an artifact while one was in fact missing.

Root cause

.github/scripts/test-regression-analysis.ps1 does handle dropped artifacts, and the intent is documented clearly at lines 564–568:

# An absent artifact is an incomplete shard, even when that shard was already red on master.
# Without this synthetic entry, dropping a red artifact also drops its failures and can make a
# killed run look like an improvement.

But that detection is baseline-relative only. The loop that synthesizes status = 'Missing' entries iterates the baseline:

foreach ($baselineShard in $baseline.shards) {
    $baselineKey = [string] $baselineShard.key
    if ($currentShardMap.ContainsKey($baselineKey)) { continue }
    ...
}

When there is no baseline, the script takes the mode = 'inventory' path at line ~507 (if (-not $baseline) { ... mode = 'inventory' ... }). In inventory mode there is no expected-shard list at all, so shard count is simply "however many artifacts showed up". A shard that never uploaded is indistinguishable from a shard that never existed.

The guard immediately above it shows the same failure mode was already anticipated for a different input:

# actions/download-artifact treats a pattern with no matches as a
# successful step, so accepting an empty directory here would classify every
# current failure as new against an invented all-green baseline.

The gap is the same class of problem, one level up: absence is being read as success.

Why this matters for v1

1. Green cannot currently be trusted. Any master run may contain a shard that died and was dropped, and it will still report improved.
2. It is self-perpetuating. The inventory ledger from this run becomes the baseline later PRs are compared against — and that baseline has no Integration D. A future PR that also loses Integration D would be diffed against a baseline that never contained it, so the missing-shard detection would not fire for it either.
3. It masks exactly the failure mode the project keeps hitting (host death / OOM-kill), which is the hardest class to notice from the summary.

Suggested fix

Give inventory mode an expected-shard list so absence is detectable without a baseline:

• Derive the expected shard keys from the matrix definition in .github/workflows/sonarcloud.yml (the same source that generates the jobs) and emit it as a manifest artifact alongside the ledger.
• In Read-TestLedger / inventory mode, diff the uploaded shard set against that expected set and synthesize the same status = 'Missing' / policyStatus = 'Missing' entries the baseline path already produces.
• Fail the gate when Missing > 0 and no approved shard-change manifest covers it — mirroring the existing Read-ApprovedShardChanges escape hatch, so intentional matrix edits still pass.

A cheap independent backstop, worth having regardless: assert in the aggregate step that shards counted == test-shard jobs in the run, and fail loudly when they disagree.

Repro

1. Open run 33938630584.
2. Note Tests (net10.0) - Integration D concluded failure.
3. Download test-outcome-ledger-2a8ca6b2da3e48b0af9e988dfe3b4a3344b17df2 and grep shards.csv for Integration D — no match; 115 rows, all Passed.
4. Download ci-test-analysis-2a8ca6b2... and read Failing shards: 0, Shards without a result artifact: 0, Regression status: improved.

---

Filed from a full re-audit of all 45 open issues against master on 2026-09-05. Triaged v1 Tier 1 (release-blocking) — until this is fixed, no green run can be used as release evidence.

[ooples]

Integration D reproduced on a clean rerun — and it is deterministic, not a flake

I re-ran the failed Tests (net10.0) - Integration D job on run 33938630584 to get a non-cancelled signal. It failed again, identically.

	Attempt 1	Attempt 2 (rerun)
Job id	101235459783	101304741104
Test run starts	06:31:38	12:22:26
Last test event logged	Data.Benchmarks.TinyStoriesDataLoaderTests (06:32:07)	Data.Benchmarks.SvhnDataLoaderTests (12:23:22)
Runner shutdown	06:34:24	12:25:42
Artifacts uploaded	none	none
Ledger row	absent	absent

Both attempts: ##[error]The runner has received a shutdown signal, roughly two minutes after the last emitted test event, with every remaining step reporting Skip evaluate condition on runner shutdown — including Record shard outcome and Upload coverage + test results.

Not a resource-exhaustion red herring on disk: the pre-test snapshot shows /dev/root 145G 41G 105G 28% /.

Localization: in both independent runs the last thing logged is an AiDotNet.Tests.IntegrationTests.Data.Benchmarks.*DataLoaderTests case, then silence until the host dies. That is a reproducible pointer at the dataset-loader tests in this shard rather than a random infrastructure flake.

Relevance to this issue: this is precisely the scenario the ledger cannot represent. Two consecutive, deterministic failures of a real shard produced:

Current master ledger: 115 shards, 115 passed, 0 failed, 0 incomplete.
Failing shards: 0   |   Shards without a result artifact: 0
Regression status: improved

A shard that dies twice in a row is invisible to the gate. The workflow already anticipated this failure mode — .github/workflows/sonarcloud.yml:1123-1130 documents that these shards "die with The runner has received a shutdown signal 2-6 min into test execution. The dotnet test step exits before producing TRX output so we have no idea what was running at OOM time" — and added a pre-test resource snapshot for forensics. The missing piece is that the ledger still treats the resulting absence as success.

Note also that Integration D is not in the $heavyShards list that gets serialized parallelism (sonarcloud.yml:1140+), so it runs at default 4-way collection parallelism. If the cause is memory, that is a candidate lever.