feat(observability): instrument hot path + add bench harness

[fclairamb]

Summary

• Add the perf-debugging signals SolidPing was missing: DB pool stats, per-stage timing inside executeJob, per-route HTTP latency, claim-jobs outcomes, busy-retry counters.
• Build cmd/loadgen + make bench-checks so optimization work has a reproducible baseline (writes markdown reports under bench-results/).
• Wire up the pre-existing solidping_check_executions_total / solidping_check_duration_seconds counters — they were defined but never observed in production.
• No optimization changes. A follow-up spec drives actual perf fixes once we have numbers.

Driven by specs/todos/2026-05-16-01-perf-instrumentation-and-loadgen.md.

Signal it surfaces

A 20-second SQLite mini-run with 30 checks at period=5s, 5ms target latency, on an idle laptop:

Metric	Value
Achieved checks/min	369
DB pool wait_count delta	749
DB pool wait_duration delta	359ms
save_result p99	25.4ms (p50: 3.3ms)
execute p99	10.0ms

That wait_count = 749 / 20s is the SQLite single-writer pool ceiling becoming visible for the first time — exactly the kind of signal the follow-up optimization spec will target.

Metric families added

Family	Notes
solidping_db_pool_*{backend}	Custom Collector wrapping sql.DB.Stats(); no idle cost
solidping_db_query_duration_seconds{operation,backend,status}	Emitted from the Bun sloghook
solidping_db_busy_retries_total{backend}	SQLITE_BUSY + PG 40001 serialization-failure classification
solidping_check_stage_duration_seconds{stage}	fetch / execute / save_result / process_incident / release_lease
solidping_http_request_duration_seconds{method,route,status} + _requests_total	Keyed by bunrouter pattern — low cardinality
solidping_claim_jobs_result_total{outcome}	jobs / empty / lock_conflict / error

Telemetry surfaces (all independently toggleable, all already existed)

• `SP_PROMETHEUS_ENABLED` (default `true`) — gates `/metrics` mount
• `SP_PROFILER_ENABLED` (default `false`) — pprof on `SP_PROFILER_LISTEN`
• `SP_OTEL_ENABLED` (default `false`) — OTLP traces/metrics/logs export

Documented in CLAUDE.md alongside the rate-limit knobs.

Bench harness

```bash
make bench-checks-sqlite # SQLite
make bench-checks-postgres # embedded PostgreSQL
make bench-checks # both, sequentially
make bench-checks BENCH_CHECKS=500 BENCH_DURATION=5m # turn the knobs
```

Each run writes `bench-results/-.md` (gitignored) with achieved checks/min, p50/p95/p99 per stage, DB pool snapshot, query timings, and claim-jobs outcomes.

Test plan

• `make build` (with placeholder embed dirs) — green
• `go test -short ./...` — 1823 passed, 0 failed across 123 packages
• `golangci-lint run ./...` — no issues
• Smoke: `/metrics` lists all new families with non-zero values; `SP_PROMETHEUS_ENABLED=false` returns 404; pprof off by default
• End-to-end loadgen run against SQLite: 369 checks/min, DB-pool wait signal visible

🤖 Generated with Claude Code