Skip to content

perf(devloop): parallel tests, test-time budgets, 1s probes and a dev-loop bench - #74

Merged
jayleaton merged 10 commits into
mainfrom
perf/devloop
Oct 22, 2026
Merged

jayleaton merged 10 commits into
mainfrom
perf/devloop

Conversation

@jayleaton

@jayleaton jayleaton commented Oct 22, 2026 •

Copy link
Copy Markdown
Owner

Summary

This keeps the Go test and lint loop fast as the template grows:

  • Every test runs in parallel, and lint enforces it.
  • Tests never sleep, and lint enforces that too.
  • make test and CI fail a package or test that gets slow, and a probe proves that check trips.
  • The coverage probes no longer recompile three packages under -race.
  • make bench-devloop prints the timing tables, including an A/B against main on the same runner.
 go test -race -json ./... | go run ./scripts/testbudget      # make test + CI Test step
+  plain go test output, then "slowest tests", then FAIL if a package > 60s or a leaf test/subtest > 10s
 .golangci.yml
+  paralleltest: every test calls t.Parallel() (exceptions need //nolint:paralleltest // <why>; none were needed)
+  tparallel; forbidigo bans time.Sleep in _test.go (synctest fake-clock sleeps go through one helper)
 tests
+  auth: migrate a template once and clone it (shared DB + newDatabase) instead of migrating each DB
+  UI crawls: one handler per state+method, routes as parallel subtests sharing it; body-only checks run once per distinct body
+  shared globals removed (crawlMethod, templ buffer size); per-case mocks, fakes and servers
 scripts/check-test-probes.sh
-  go test -race -coverprofile on 3 packages with planted files      (10-16s in CI)
+  plant the uncovered statements in the test run's real profile     (1s), plus probes for testbudget
 make bench-devloop   (+ "Bench dev loop" workflow: run it by hand, or push bench/<name>)
+  build time, size and cold start; generators; lint (also per linter); probes; tests per package with and without -race;
+  A/B of the test run against a base ref on the same machine (BENCH_BASE)

AGENTS.md gains a "Fast tests" section: parallel tests, no sleeps, cloned databases, cheap crypto, table tests, checking immutable inputs once, the budgets, and measuring with the A/B. README lists make bench-devloop. ci.yml is unchanged. Inside the check action, the Test step now pipes through testbudget, and the probes step moves after Test and reads the coverage profile (PROFILE).

Evidence

All numbers come from GitHub Actions ubuntu-latest runners (2 vCPU) with Go 1.26.8.

Tests: A/B on one runner. Base main 7fcc091 vs HEAD 056ce8d, go test -race -coverprofile ./... as CI runs it, warm cache, 3 alternating rounds (run):

rounds median
Before (main) 30.5s 30.0s 31.2s 30.5s
After 28.0s 29.7s 29.1s 29.1s (−5%)
test binary, median, -race before after
apps/admin/ui 13.09s 9.85s (−25%)
apps/web/ui 12.96s 10.47s (−19%)
packages/auth 7.89s 7.41s
everything else ≤ 2.0s ≤ 2.2s
slowest single test 11.09s (apps/web/ui TestCrawlExposesNothing) ~2.7s (one crawl route)

The suite total moves less than the UI packages because two packages already share the 2 vCPUs. Parallelism inside packages mostly removes the long tail. It also keeps the suite from growing linearly with routes and tests, which the budgets now enforce.

An earlier A/B on the pre-overhaul main (8b120a3) measured 50.0s → 43.2s, −14% (run).

CI probes step: 10s and 11s on main vs 1s on this branch, across steady-state workflow_dispatch runs of both.

Budgets proven on CI. The final PR run on e3e3c32 (run) is green and lists the slowest tests. Two earlier runs of this PR failed the first budgets:

  • Per-state crawl subtests took 6–11s, which led to the regrouping above.
  • A CPU-bound Better Auth scrypt test took 5.28s while the cold build cache compiled on the same 2 vCPUs, which is why the budgets sit at about 2× today's slowest. make lint-tests / CI prove that a slow package and a slow test both fail.

Dev loop (bench on current HEAD, run; this PR doesn't change these numbers):

cold warm
build server / admin 28.6s / 27.1s 0.1s / 0.1s (one-line edit: 1.8s / 2.3s)
make generate (sqlc / mockgen / templ) ~117s (99s / 19s / 33s) ~3s (0.2s / 2.2s / 0.5s)
golangci-lint ./... 58–77s 0.6–1.2s
tests -race -cover / plain 90.0s / 45.7s 28.8s / 6.8s

Lint per linter, each run alone on an empty lint cache (times include loading packages):

  • staticcheck ~12s, govet ~5s, tparallel ~4.7s.
  • The SSA-based linters (bodyclose, nilerr, noctx, rowserrcheck, sqlclosecheck) take ~4.3s each, sharing one SSA build when run together.
  • paralleltest 1.1s; the rest ≤ 1.7s.

A warm lint takes about 1s, so a separate fast local mode doesn't pay off, and every linter stays. sqlc's cold build (~100s, cgo) is a one-time cost per toolchain/go.sum and is cached after that.

Runtime sanity: server 15.7 MB, admin 16.1 MB. Cold start to /healthz takes 134–245 ms. RSS after start is 80–84 MB, the same before and after. Most of it is auth's boot-time argon2 dummy hash (64 MiB, production cost) from the original auth port (#2), not from today's changes.

Local checks: gofmt; go vet ./...; golangci-lint (0 issues on the root and on deploy/); check-test-boundaries; check-test-probes (13/13); actionlint; plain tests of the touched packages. I also checked by mutation that a leak planted in app.css is still reported in every crawl state.

Follow-up (not in this PR): DB-backed tests run through go test -exec wait-db. -exec turns off Go's test-result cache, so unchanged DB packages always re-run on main pushes. Dropping it needs a readiness wait in packages/database's own tests, because pool_internal_test.go connects without retrying.

Merge Danger

Door: two-way. The change touches tests, lint config, scripts, the check action and a new manual/bench/** workflow. No production code changes.

Blast Radius: contributors

What changes for people writing tests:

  • New tests must call t.Parallel() and must not sleep; lint enforces both.
  • Tests must stay within 60s per package and 10s per leaf test on CI; make test and CI fail otherwise. The constants live in scripts/testbudget.

🤖 Generated with Claude Code

@jayleaton
jayleaton marked this pull request as ready for review October 22, 2026 13:16
@jayleaton
jayleaton force-pushed the perf/devloop branch 2 times, most recently from 8e0fe96 to 3a5648b Compare October 22, 2026 13:30
jayleaton and others added 10 commits October 22, 2026 20:47
… and per-package test timings

Adds scripts/bench-devloop.sh (cold + warm), the Bench dev loop workflow
(manual, or push a bench/* branch) and scripts/testbudget, which renders
go test -json as plain go test output, lists the slowest tests and enforces
per-package and per-test time budgets.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
… body once

paralleltest + tparallel make t.Parallel() the rule (no exceptions were
needed) and forbidigo bans time.Sleep in tests (synctest fake-clock sleeps go
through one documented helper). Fixes that made it safe: the UI crawls no
longer share a package-level crawlMethod, admin render tests stop mutating
templ's global buffer size, TestDopplerErrors and the web services table get
per-case fakes and controllers, and packages/auth migrates a template once and
clones it (shared database and newDatabase) instead of migrating per database.
The crawls split into parallel subtests and memoize the body-only checks:
static assets come back byte-identical in every state, and re-scanning them
was most of the crawl's time under -race.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
… the test profile

make test and CI pipe go test -json through scripts/testbudget: plain go test
output, the slowest tests, and a failure when a package runs over 30s or a
leaf test over 5s (make lint-tests proves both trip). The coverage probes
plant their uncovered statements in the real profile instead of recompiling
three packages under -race (CI runs them after the tests). AGENTS.md
"Fast tests" documents the rules; make bench-devloop the measurements.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The cold run uses a throwaway cache, so the warm run was compiling every
-race variant; an untimed run now primes the cache first.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…CI does

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
… too

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
… on one machine

Per-route crawl subtests built a handler and mock set each: twice the CPU of
the serial crawl, which 2 vCPUs cannot pay back. One parallel subtest per
state (web: and method) keeps the setup at main's level and still uses both
CPUs. Separate CI runs of identical code varied by up to 50%, so the bench
gains BENCH_BASE: base and HEAD test runs alternate on the same machine and
their medians are compared (the workflow compares with main).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
… state and method

Per-state crawl subtests ran 6-11s each on CI's single job and tripped the
5s test budget. Routes are subtests again, now sharing their state's
handler: at GOMAXPROCS=2 the crawls use main's CPU or less and finish
29-38% sooner.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…e, 10s test)

5s flaked on a CPU-bound scrypt test (5.28s) while a cold build cache
compiled on the same 2 vCPUs.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@jayleaton
jayleaton merged commit 48e6c80 into main Oct 22, 2026
3 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant