Repository navigation
Don't retry task heartbeats according to [workers] execution_api_retries - #73244
Merged
Merged
Conversation
nickstenning
requested review from
amoghrajesh,
ashb and
kaxil
as code owners
September 16, 2026 14:04
nickstenning
force-pushed
the
dont-retry-heartbeat-requests
branch
from
September 16, 2026 14:05
21c8262 to
ffaa52b
Compare
ashb
reviewed
Sep 16, 2026
kaxil
reviewed
Sep 17, 2026
…tries` Heartbeats had two stacked retry mechanisms: the Task SDK API client retried every request, including heartbeats, and the supervisor separately counted consecutive heartbeat failures against `[workers] max_failed_heartbeats`. The client retries also had the unwanted side-effect of blocking the supervisor's monitor loop while it backed off, so during an API server outage the supervisor would be delayed forwarding task logs and/or noticing that the task subprocess had exited. Heartbeat retries are now governed solely by `max_failed_heartbeats`. I've adjusted the default value of that tunable from 3 to 30. This gives a failure tolerance that (depending on the failure mode) is roughly equivalent to the default value of `[scheduler] task_instance_heartbeat_timeout`, and in any event slightly *more* tolerant of an unreachable API server than the previous configuration. Specifically, if every heartbeat failure is instant (e.g. a DNS lookup failure or an explicitly rejected connection), then the new defaults will result in task termination after 150s. If every heartbeat failure takes a full 5s timeout, this extends to 300s, matching the scheduler's tolerance for a missing task.
nickstenning
force-pushed
the
dont-retry-heartbeat-requests
branch
from
September 24, 2026 09:13
ffaa52b to
5d548fb
Compare
ashb
approved these changes
Sep 24, 2026
kaxil
approved these changes
Sep 24, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Heartbeats had two stacked retry mechanisms: the Task SDK API client retried every request, including heartbeats, and the supervisor separately counted consecutive heartbeat failures against
[workers] max_failed_heartbeats. The client retries also had the unwanted side-effect of blocking the supervisor's monitor loop while it backed off, so during an API server outage the supervisor would be delayed forwarding task logs and/or noticing that the task subprocess had exited.Heartbeat retries are now governed solely by
max_failed_heartbeats. I've adjusted the default value of that tunable from 3 to 30. This gives a failure tolerance that (depending on the failure mode) is roughly equivalent to the default value of[scheduler] task_instance_heartbeat_timeout, and in any event slightly more tolerant of an unreachable API server than the previous configuration.Specifically, if every heartbeat failure is instant (e.g. a DNS lookup failure or an explicitly rejected connection), then the new defaults will result in task termination after 150s. If every heartbeat failure takes a full 5s timeout, this extends to 300s, matching the scheduler's tolerance for a missing task.