Repository navigation
Improve LocalExecutor bookkeeping: correctly add tasks to the running list - #74142
Merged
Merged
Conversation
ashb
added this pull request to stack #73919
October 3, 2026 09:12
ashb
requested review from
dheerajturaga,
hussein-awala and
pierrejeambrun
as code owners
October 3, 2026 09:12
ashb
force-pushed
the
local-executor-bookkeeping
branch
from
October 3, 2026 09:13
59eb578 to
67a2a7c
Compare
Member
Author
|
Addressing the issue raised by Shahar in #73916 (comment) |
ashb
force-pushed
the
local-executor-bookkeeping
branch
from
October 3, 2026 11:40
de6cbb5 to
2b2b21b
Compare
…g` list Since #73916 landed, BaseExecutor now keeps a UUID-to-coordinates map for each attempt and drops an entry once the key is no longer queued, running or in the event buffer. The issue was that LocalExecutor never added dispatched work to `running`, so the entry was dropped while the task was still executing. The scheduler then received the final event with no coordinates ("Received executor event with state success for task instance <uuid> (coordinates=None)"). This PR fixes that, and addresses a few gotchas that could, in edge cases around callbacks or post-task finalization, lead to a forever-dead/locked worker slot. Holding the key in `running` until the worker finishes needs LocalExecutor to know when that is, and workers share one activity queue so the parent cannot tell which worker took which workload. Results now carry the worker's pid and START tells the parent which pid owns the key. That lets it: - release the slot only on a terminal result from the owning worker, ignoring results from unknown pids or pids that don't own the key - fail the workload when its worker dies, instead of leaving it in `running` until the scheduler's heartbeat timeout A resumed task reuses its key, so a per-key dispatch count stops the old run's terminal result from releasing the new run's slot.
ashb
force-pushed
the
local-executor-bookkeeping
branch
from
October 3, 2026 18:33
1c1b063 to
302c914
Compare
eladkal
approved these changes
Oct 5, 2026
1 of 2 tasks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Since #73916 landed, BaseExecutor now keeps a UUID-to-coordinates map for each
attempt and drops an entry once the key is no longer queued, running or in the
event buffer. The issue was that LocalExecutor never added dispatched work to
running, so the entry was dropped while the task was still executing. Thescheduler then received the final event with no coordinates ("Received executor
event with state success for task instance (coordinates=None)").
This PR fixes that, and addresses a few gotchas that could, in edge cases
around callbacks or post-task finalization, lead to a forever-dead/locked
worker slot.
Holding the key in
runninguntil the worker finishes needs LocalExecutorto know when that is, and workers share one activity queue so the parent
cannot tell which worker took which workload. Results now carry the
worker's pid and START tells the parent which pid owns the key. That lets
us:
ignoring results from unknown pids or pids that don't own the key
runninguntil the scheduler's heartbeat timeoutA resumed task reuses its key, so a per-key dispatch count stops the old
run's terminal result from releasing the new run's slot.
{pr_number}.significant.rst, in airflow-core/newsfragments. You can add this file in a follow-up commit after the PR is created so you know the PR number.Stack created with GitHub Stacks CLI • Give Feedback 💬