Repository navigation
Conversation
…ole check budget A single lost DNS UDP packet used to fail a check outright: the stdlib resolver waits resolv.conf's per-exchange timeout (default 5s) — as long as the scheduler's cost-aware timeout floor — so fast checks died with "lookup <host>: i/o timeout" at exactly 5s (~0.007% of runs on the k8xp deployment, every region). Add checkerdef.LookupIPAddr: up to 3 attempts of 1.5s each, retrying only transient errors (timeouts, temporary server failures), returning immediately on NXDOMAIN/cancellation. A re-issued lookup uses a fresh source port (new conntrack flow, often a different DNS endpoint), which beats waiting out a packet that is never coming. Wired into the nine checkers where resolution is a precondition (ssl, imap, smtp, pop3, udp, sip, tcp, ssh, icmp). checkdns/checkdnsbl are untouched — there the lookups are the measurement itself. Per-checker error messages are unchanged. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
fclairamb
added a commit
that referenced
this pull request
Aug 4, 2026
…ole check budget (#194) ## Problem Checks sporadically fail with `failed to resolve hostname: lookup <host>: i/o timeout` at exactly ~5001ms (e.g. the hourly SSL check on `ts-seminor.abyla.fr`, org stonaltech, k8xp dev). Over 48h: 39 such errors out of 561k results, across all regions and many unrelated hostnames. Root cause chain: 1. Fast checks get the **cost-aware timeout floor of 5s** (`CostTimeoutFloorMs: 5000`) as their whole budget. 2. The stdlib resolver's per-exchange DNS timeout is **also 5s** (resolv.conf default), so **one lost DNS UDP packet stalls resolution for the entire check budget** — no retry ever fires. 3. On the k8xp cluster the DNS path crosses intercontinental WireGuard hops as single UDP packets (companion infra fix: node-local CoreDNS in the k8xp repo), but any deployment with occasional DNS packet loss reproduces this. ## Fix New `checkerdef.LookupIPAddr`: up to 3 attempts × 1.5s, retrying only transient errors (timeout / temporary), immediate return on NXDOMAIN or cancellation. Re-issued lookups leave from a fresh source port (new conntrack flow, often a different DNS server), so a retry is genuinely a second chance rather than more waiting. Worst case 4.5s still fits the 5s floor. Wired into the nine checkers where resolution is a precondition: ssl, imap, smtp, pop3, udp, sip, tcp, ssh, icmp. `checkdns` / `checkdnsbl` untouched — their lookups are the measurement. Per-checker error messages unchanged. ## Testing - New unit tests for the retry loop (injectable lookup): first-try success, retry-then-success, no retry on NXDOMAIN/non-DNS errors, max-attempts cap, parent-context stop, IP-literal passthrough, transient-error classification. - `go test` green on all touched packages; `golangci-lint` clean. _Supersedes #192 (closed unmerged)._ 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
Checks sporadically fail with
failed to resolve hostname: lookup <host>: i/o timeoutat exactly ~5001ms (e.g. the hourly SSL check onts-seminor.abyla.fr, org stonaltech, k8xp dev). Over 48h: 39 such errors out of 561k results, across all regions and many unrelated hostnames.Root cause chain:
CostTimeoutFloorMs: 5000) as their whole budget.Fix
New
checkerdef.LookupIPAddr: up to 3 attempts × 1.5s, retrying only transient errors (timeout / temporary), immediate return on NXDOMAIN or cancellation. Re-issued lookups leave from a fresh source port (new conntrack flow, often a different DNS server), so a retry is genuinely a second chance rather than more waiting. Worst case 4.5s still fits the 5s floor.Wired into the nine checkers where resolution is a precondition: ssl, imap, smtp, pop3, udp, sip, tcp, ssh, icmp.
checkdns/checkdnsbluntouched — their lookups are the measurement. Per-checker error messages unchanged.Testing
go testgreen on all touched packages;golangci-lintclean.🤖 Generated with Claude Code