Repository navigation
perf(devloop): parallel tests, test-time budgets, 1s probes and a dev-loop bench - #74
Merged
Merged
Conversation
jayleaton
marked this pull request as ready for review
October 22, 2026 13:16
jayleaton
force-pushed
the
perf/devloop
branch
2 times, most recently
from
October 22, 2026 13:30
8e0fe96 to
3a5648b
Compare
… and per-package test timings Adds scripts/bench-devloop.sh (cold + warm), the Bench dev loop workflow (manual, or push a bench/* branch) and scripts/testbudget, which renders go test -json as plain go test output, lists the slowest tests and enforces per-package and per-test time budgets. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
… body once paralleltest + tparallel make t.Parallel() the rule (no exceptions were needed) and forbidigo bans time.Sleep in tests (synctest fake-clock sleeps go through one documented helper). Fixes that made it safe: the UI crawls no longer share a package-level crawlMethod, admin render tests stop mutating templ's global buffer size, TestDopplerErrors and the web services table get per-case fakes and controllers, and packages/auth migrates a template once and clones it (shared database and newDatabase) instead of migrating per database. The crawls split into parallel subtests and memoize the body-only checks: static assets come back byte-identical in every state, and re-scanning them was most of the crawl's time under -race. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
… the test profile make test and CI pipe go test -json through scripts/testbudget: plain go test output, the slowest tests, and a failure when a package runs over 30s or a leaf test over 5s (make lint-tests proves both trip). The coverage probes plant their uncovered statements in the real profile instead of recompiling three packages under -race (CI runs them after the tests). AGENTS.md "Fast tests" documents the rules; make bench-devloop the measurements. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The cold run uses a throwaway cache, so the warm run was compiling every -race variant; an untimed run now primes the cache first. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…CI does Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
… too Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
… on one machine Per-route crawl subtests built a handler and mock set each: twice the CPU of the serial crawl, which 2 vCPUs cannot pay back. One parallel subtest per state (web: and method) keeps the setup at main's level and still uses both CPUs. Separate CI runs of identical code varied by up to 50%, so the bench gains BENCH_BASE: base and HEAD test runs alternate on the same machine and their medians are compared (the workflow compares with main). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
… state and method Per-state crawl subtests ran 6-11s each on CI's single job and tripped the 5s test budget. Routes are subtests again, now sharing their state's handler: at GOMAXPROCS=2 the crawls use main's CPU or less and finish 29-38% sooner. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…e, 10s test) 5s flaked on a CPU-bound scrypt test (5.28s) while a cold build cache compiled on the same 2 vCPUs. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
jayleaton
force-pushed
the
perf/devloop
branch
from
October 22, 2026 13:47
3a5648b to
e3e3c32
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
This keeps the Go test and lint loop fast as the template grows:
make testand CI fail a package or test that gets slow, and a probe proves that check trips.-race.make bench-devloopprints the timing tables, including an A/B againstmainon the same runner.AGENTS.md gains a "Fast tests" section: parallel tests, no sleeps, cloned databases, cheap crypto, table tests, checking immutable inputs once, the budgets, and measuring with the A/B. README lists
make bench-devloop.ci.ymlis unchanged. Inside thecheckaction, the Test step now pipes through testbudget, and the probes step moves after Test and reads the coverage profile (PROFILE).Evidence
All numbers come from GitHub Actions
ubuntu-latestrunners (2 vCPU) with Go 1.26.8.Tests: A/B on one runner. Base
main7fcc091 vs HEAD 056ce8d,go test -race -coverprofile ./...as CI runs it, warm cache, 3 alternating rounds (run):apps/web/ui TestCrawlExposesNothing)The suite total moves less than the UI packages because two packages already share the 2 vCPUs. Parallelism inside packages mostly removes the long tail. It also keeps the suite from growing linearly with routes and tests, which the budgets now enforce.
An earlier A/B on the pre-overhaul main (8b120a3) measured 50.0s → 43.2s, −14% (run).
CI probes step: 10s and 11s on
mainvs 1s on this branch, across steady-stateworkflow_dispatchruns of both.Budgets proven on CI. The final PR run on e3e3c32 (run) is green and lists the slowest tests. Two earlier runs of this PR failed the first budgets:
make lint-tests/ CI prove that a slow package and a slow test both fail.Dev loop (bench on current HEAD, run; this PR doesn't change these numbers):
make generate(sqlc / mockgen / templ)Lint per linter, each run alone on an empty lint cache (times include loading packages):
A warm lint takes about 1s, so a separate fast local mode doesn't pay off, and every linter stays. sqlc's cold build (~100s, cgo) is a one-time cost per toolchain/go.sum and is cached after that.
Runtime sanity: server 15.7 MB, admin 16.1 MB. Cold start to
/healthztakes 134–245 ms. RSS after start is 80–84 MB, the same before and after. Most of it is auth's boot-time argon2 dummy hash (64 MiB, production cost) from the original auth port (#2), not from today's changes.Local checks: gofmt;
go vet ./...; golangci-lint (0 issues on the root and ondeploy/); check-test-boundaries; check-test-probes (13/13); actionlint; plain tests of the touched packages. I also checked by mutation that a leak planted inapp.cssis still reported in every crawl state.Follow-up (not in this PR): DB-backed tests run through
go test -exec wait-db.-execturns off Go's test-result cache, so unchanged DB packages always re-run on main pushes. Dropping it needs a readiness wait inpackages/database's own tests, becausepool_internal_test.goconnects without retrying.Merge Danger
Door: two-way. The change touches tests, lint config, scripts, the
checkaction and a new manual/bench/**workflow. No production code changes.Blast Radius: contributors
What changes for people writing tests:
t.Parallel()and must not sleep; lint enforces both.make testand CI fail otherwise. The constants live inscripts/testbudget.🤖 Generated with Claude Code