Skip to content

fix(checkers): retry transient DNS failures instead of burning the whole check budget - #192

Closed
fclairamb wants to merge 1 commit into
mainfrom
fix/dns-resolve-retry
Closed

fclairamb wants to merge 1 commit into
mainfrom
fix/dns-resolve-retry

Conversation

@fclairamb

Copy link
Copy Markdown
Owner

Problem

Checks sporadically fail with failed to resolve hostname: lookup <host>: i/o timeout at exactly ~5001ms (e.g. the hourly SSL check on ts-seminor.abyla.fr, org stonaltech, k8xp dev). Over 48h: 39 such errors out of 561k results, across all regions and many unrelated hostnames.

Root cause chain:

  1. Fast checks get the cost-aware timeout floor of 5s (CostTimeoutFloorMs: 5000) as their whole budget.
  2. The stdlib resolver's per-exchange DNS timeout is also 5s (resolv.conf default), so one lost DNS UDP packet stalls resolution for the entire check budget — no retry ever fires.
  3. On the k8xp cluster the DNS path crosses intercontinental WireGuard hops as single UDP packets (companion infra fix: node-local CoreDNS in the k8xp repo), but any deployment with occasional DNS packet loss reproduces this.

Fix

New checkerdef.LookupIPAddr: up to 3 attempts × 1.5s, retrying only transient errors (timeout / temporary), immediate return on NXDOMAIN or cancellation. Re-issued lookups leave from a fresh source port (new conntrack flow, often a different DNS server), so a retry is genuinely a second chance rather than more waiting. Worst case 4.5s still fits the 5s floor.

Wired into the nine checkers where resolution is a precondition: ssl, imap, smtp, pop3, udp, sip, tcp, ssh, icmp. checkdns / checkdnsbl untouched — their lookups are the measurement. Per-checker error messages unchanged.

Testing

  • New unit tests for the retry loop (injectable lookup): first-try success, retry-then-success, no retry on NXDOMAIN/non-DNS errors, max-attempts cap, parent-context stop, IP-literal passthrough, transient-error classification.
  • go test green on all touched packages; golangci-lint clean.

🤖 Generated with Claude Code

…ole check budget

A single lost DNS UDP packet used to fail a check outright: the stdlib
resolver waits resolv.conf's per-exchange timeout (default 5s) — as long
as the scheduler's cost-aware timeout floor — so fast checks died with
"lookup <host>: i/o timeout" at exactly 5s (~0.007% of runs on the k8xp
deployment, every region).

Add checkerdef.LookupIPAddr: up to 3 attempts of 1.5s each, retrying only
transient errors (timeouts, temporary server failures), returning
immediately on NXDOMAIN/cancellation. A re-issued lookup uses a fresh
source port (new conntrack flow, often a different DNS endpoint), which
beats waiting out a packet that is never coming.

Wired into the nine checkers where resolution is a precondition (ssl,
imap, smtp, pop3, udp, sip, tcp, ssh, icmp). checkdns/checkdnsbl are
untouched — there the lookups are the measurement itself. Per-checker
error messages are unchanged.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@fclairamb fclairamb closed this Aug 4, 2026
fclairamb added a commit that referenced this pull request Aug 4, 2026
…ole check budget (#194)

## Problem

Checks sporadically fail with `failed to resolve hostname: lookup
<host>: i/o timeout` at exactly ~5001ms (e.g. the hourly SSL check on
`ts-seminor.abyla.fr`, org stonaltech, k8xp dev). Over 48h: 39 such
errors out of 561k results, across all regions and many unrelated
hostnames.

Root cause chain:
1. Fast checks get the **cost-aware timeout floor of 5s**
(`CostTimeoutFloorMs: 5000`) as their whole budget.
2. The stdlib resolver's per-exchange DNS timeout is **also 5s**
(resolv.conf default), so **one lost DNS UDP packet stalls resolution
for the entire check budget** — no retry ever fires.
3. On the k8xp cluster the DNS path crosses intercontinental WireGuard
hops as single UDP packets (companion infra fix: node-local CoreDNS in
the k8xp repo), but any deployment with occasional DNS packet loss
reproduces this.

## Fix

New `checkerdef.LookupIPAddr`: up to 3 attempts × 1.5s, retrying only
transient errors (timeout / temporary), immediate return on NXDOMAIN or
cancellation. Re-issued lookups leave from a fresh source port (new
conntrack flow, often a different DNS server), so a retry is genuinely a
second chance rather than more waiting. Worst case 4.5s still fits the
5s floor.

Wired into the nine checkers where resolution is a precondition: ssl,
imap, smtp, pop3, udp, sip, tcp, ssh, icmp. `checkdns` / `checkdnsbl`
untouched — their lookups are the measurement. Per-checker error
messages unchanged.

## Testing

- New unit tests for the retry loop (injectable lookup): first-try
success, retry-then-success, no retry on NXDOMAIN/non-DNS errors,
max-attempts cap, parent-context stop, IP-literal passthrough,
transient-error classification.
- `go test` green on all touched packages; `golangci-lint` clean.

_Supersedes #192 (closed unmerged)._

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant