ci: the integration job grew from ~3m to ~9m with the distributed tests #701

Area: CI / tests — the Integration tests job

On a PR, CI went from 3–4 minutes to about 9 when the distributed-backends work landed. The whole difference is the Integration tests job, which is now the critical path. Every other job is unchanged.

Measured

GitHub Actions, Integration tests job:

Over August and most of September, successful PR runs of ci.yml averaged 3.7–4.2 minutes (438 runs). Nothing else drifted.

Local: go test -tags integration -race -count=1 -json ./tests/integration/ on main @ 6fa9723 took 343s. The top-level tests add up to 313s, so the package runs almost serially. Only 5 of its files call t.Parallel(). The slowest tests:

The tests from the distributed work add up to about 215s, which accounts for the growth.

Why they are slow (inferred from the tests' code)

Most of that time is spent waiting on real clocks, not doing work:

• Lease and pin timers. TestShardClaims_BlockedOwnerKeepsItsShard blocks an insert for 13s to outlast the 10s pinned TTL. TestShardClaims_TableOrderAcrossHandoverAndStop makes the old owner's inserts take 6s each. The crash test waits out a pin TTL. TestRoles_SeparateProcesses waits for real membership leases (15s unrenewed) across its handover steps, in sequence.
• Serial execution. None of these tests calls t.Parallel(), although each starts its own NATS server, and most start their own processes and fake ClickHouse.
• Serial gotestsum calls. The test-integration Makefile target runs internal/mq's integration tests as a second call, after tests/integration finishes.

Fix directions, roughly cheapest first

1. Run the pieces concurrently. Split Integration tests into matrix jobs: tests/integration on its own; internal/mq integration plus internal/cache plus natsspike. Or shard tests/integration by -run pattern. .github/workflows/README.md has a deferred sharding design with a working implementation (see "Deferred optimizations"). Coverage merges already happen in the Coverage job from artifacts. This alone should bring the job to about the slowest shard.
2. t.Parallel() on the self-contained slow tests: TestShardClaims_*, TestRoles_SeparateProcesses, TestNATSBackend_EndToEnd, TestSharedCache_*. Check each first for shared state: fixed ports, the shared ClickHouse container's tables (use per-test table names), and TestMain-built binaries. The -race CPU cost on a 4-core runner caps how much this buys, so measure it.
3. Shorten the clocks under test. Where a test waits on a production default (pinned TTL 10s, lease 15s, ack_wait 60s), pass the smallest value the code accepts through the config the test already builds, and scale the sleeps to match. The pinned-TTL floor of 10s is a check at boot; a test-only override would need care so it cannot reach production config.
4. Decide whether internal/mq/natsspike belongs in CI at all. It is an experiment package: 18s per run.
5. Local make ci pays the same cost, and it is the pre-push gate, so this slows every push too, not just the PR check.

Done when

• A PR's CI is back to about 4–5 minutes, with the same tests and coverage gates. Compare gh run list -w ci.yml before and after.
• Coverage totals are unchanged (cov gate).
• Nothing is skipped: related to bug(mq): internal/mq integration tests are selected by name pattern, so a new prefix is silently skipped #692, where internal/mq's integration tests are chosen by name pattern. If the jobs are restructured, fix that selection at the same time.

Related: #624, #613, #692.