Repository navigation
fix(checkers): retry transient DNS failures instead of burning the whole check budget - #194
Merged
Merged
Conversation
…ole check budget A single lost DNS UDP packet used to fail a check outright: the stdlib resolver waits resolv.conf's per-exchange timeout (default 5s) — as long as the scheduler's cost-aware timeout floor — so fast checks died with "lookup <host>: i/o timeout" at exactly 5s (~0.007% of runs on the k8xp deployment, every region). Add checkerdef.LookupIPAddr: up to 3 attempts of 1.5s each, retrying only transient errors (timeouts, temporary server failures), returning immediately on NXDOMAIN/cancellation. A re-issued lookup uses a fresh source port (new conntrack flow, often a different DNS endpoint), which beats waiting out a packet that is never coming. Wired into the nine checkers where resolution is a precondition (ssl, imap, smtp, pop3, udp, sip, tcp, ssh, icmp). checkdns/checkdnsbl are untouched — there the lookups are the measurement itself. Per-checker error messages are unchanged. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
fclairamb
enabled auto-merge (squash)
August 4, 2026 22:10
Merged
fclairamb
added a commit
that referenced
this pull request
Aug 5, 2026
🤖 I have created a release *beep* *boop* --- ## [0.7.2](v0.7.1...v0.7.2) (2026-08-05) ### Bug Fixes * **checkers:** retry transient DNS failures instead of burning the whole check budget ([#194](#194)) ([deda12e](deda12e)) --- This PR was generated with [Release Please](https://github.com/googleapis/release-please). See [documentation](https://github.com/googleapis/release-please#release-please). --------- Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
This was referenced Sep 8, 2026
Closed
Closed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
Checks sporadically fail with
failed to resolve hostname: lookup <host>: i/o timeoutat exactly ~5001ms (e.g. the hourly SSL check onts-seminor.abyla.fr, org stonaltech, k8xp dev). Over 48h: 39 such errors out of 561k results, across all regions and many unrelated hostnames.Root cause chain:
CostTimeoutFloorMs: 5000) as their whole budget.Fix
New
checkerdef.LookupIPAddr: up to 3 attempts × 1.5s, retrying only transient errors (timeout / temporary), immediate return on NXDOMAIN or cancellation. Re-issued lookups leave from a fresh source port (new conntrack flow, often a different DNS server), so a retry is genuinely a second chance rather than more waiting. Worst case 4.5s still fits the 5s floor.Wired into the nine checkers where resolution is a precondition: ssl, imap, smtp, pop3, udp, sip, tcp, ssh, icmp.
checkdns/checkdnsbluntouched — their lookups are the measurement. Per-checker error messages unchanged.Testing
go testgreen on all touched packages;golangci-lintclean.Supersedes #192 (closed unmerged).
🤖 Generated with Claude Code