Drain result_queue in LocalExecutor to prevent process join deadlock by SakshamKapoor2911 · Pull Request #67881 · apache/airflow

SakshamKapoor2911 · 2026-06-02T00:24:30Z

Description

This PR fixes a deadlock in the LocalExecutor during executor shutdown (end()).

Currently, the executor calls proc.join() on active worker processes. However, if a worker has written enough results to the result_queue to fill the OS-level pipe buffer (typically 64KB), the worker process blocks indefinitely on its put() call.
Because the parent scheduler process is blocked on the unbounded proc.join() and not reading from result_queue, a classic multiprocessing deadlock occurs. The scheduler hangs indefinitely, stopping heartbeats and preventing systemd/Kubernetes from restarting the process (as it never exits).

Changes

Draining during Join: Updated the shutdown loop in LocalExecutor.end() to continuously drain the result_queue using _read_results() while waiting for worker processes to join with a small timeout.
Robust Queue Reading: Wrapped the queue drainage loop in _read_results() with a try/except (OSError, EOFError) block to gracefully handle cases where pipes are already broken or closed during task exit.
Forced Terminate: Implemented the LocalExecutor.terminate() method to forcefully kill any remaining workers when a hard stop is requested.

Context

This contribution was created as part of the required capstone of the CodePath AI301 course, where students learn how to make responsible and effective use of generative AI tools for open-source software contributions.

Was generative AI tooling used to co-author this PR?

Yes (Antigravity IDE powered by Gemini)

Generated-by: Antigravity IDE (Gemini) following the guidelines

Note

✅ Ready for review · @SakshamKapoor2911 → @potiuk · 2026-06-12 13:30 UTC

Thanks @SakshamKapoor2911 — all checks are green and this PR is marked ready for maintainer review. The ball is with the maintainers now; a maintainer will take the next look.

_{Automated triage — may be imperfect.}

boring-cyborg · 2026-06-02T00:24:34Z

Congratulations on your first Pull Request and welcome to the Apache Airflow community! If you have any issues or are unsure about any anything please check our Contributors' Guide
Here are some useful points:

Pay attention to the quality of your code (ruff, mypy and type annotations). Our prek-hooks will help you with that.
In case of a new feature add useful documentation (in docstrings or in docs/ directory). Adding a new operator? Check this short guide Consider adding an example Dag that shows how users should use it.
Consider using Breeze environment for testing locally, it's a heavy docker but it ships with a working Airflow and a lot of integrations.
Be patient and persistent. It might take some time to get a review or get the final approval from Committers.
Please follow ASF Code of Conduct for all communication including (but not limited to) comments on Pull Requests, Mailing list and Slack.
Be sure to read the Airflow Coding style.
Always keep your Pull Requests rebased, otherwise your build might fail due to changes not related to your commits.
Apache Airflow is a community-driven project and together we are making it better 🚀.
In case of doubts contact the developers at:
Mailing List: dev@airflow.apache.org
Slack: https://s.apache.org/airflow-slack

Copilot

Pull request overview

Note

Copilot was unable to run its full agentic suite in this review.

Fixes a potential deadlock in LocalExecutor shutdown where workers blocked on writing to a full result_queue would never exit, hanging the scheduler. Also makes terminate() actually terminate worker processes.

Changes:

Drain result_queue while waiting for workers to join in end(), with KeyboardInterrupt/SystemExit fallback that force-terminates workers.
Harden _read_results against OSError/EOFError from a closed/broken queue.
Implement terminate() to forcefully terminate all worker processes (previously a no-op).

SakshamKapoor2911 · 2026-06-04T01:49:38Z

@potiuk Fixed the Ruff pre-commit hook failures on this PR by addressing the following two lint issues:

Ruff SIM105: Replaced the try-except-pass block when closing worker processes with contextlib.suppress(ValueError).
Ruff D401: Rewrote the docstring of the terminate method to begin with an imperative verb ("Terminate all worker processes..." instead of "Forcefully terminate...").

All local checks now pass successfully and should pass the CI tests.

o-nikolas

Thanks for the updates, as long as the CI goes green it looks good to me

SakshamKapoor2911 · 2026-06-30T21:30:56Z

@o-nikolas could you please approve the pending GitHub Actions workflows for the latest push?

I fixed the single CI failure by removing the Conventional Commits-style fix: prefix from the PR title and rewriting the branch commit subjects. This was metadata-only; the code tree did not change. The latest head is 28fa007d66.

On the previous run, the only failed job was CI image checks / Static checks; all other executed jobs passed, with the remaining jobs skipped by CI selection.

o-nikolas · 2026-07-01T00:57:41Z

@o-nikolas could you please approve the pending GitHub Actions workflows for the latest push?

I fixed the single CI failure by removing the Conventional Commits-style fix: prefix from the PR title and rewriting the branch commit subjects. This was metadata-only; the code tree did not change. The latest head is 28fa007d66.

On the previous run, the only failed job was CI image checks / Static checks; all other executed jobs passed, with the remaining jobs skipped by CI selection.

Done!

SakshamKapoor2911 · 2026-07-01T03:24:19Z

@o-nikolas CI is green now, including the previously failing CI image checks / Static checks. Thanks again for approving the workflows. Would you mind merging this when you get a chance?

boring-cyborg · 2026-07-02T17:47:39Z

Awesome work, congrats on your first merged pull request! You are invited to check our Issue Tracker for additional contributions.

…alExecutor Call _read_results() before each activity_queue.put() in _process_workloads so workers blocked on a full result_queue pipe can resume reading from activity_queue, preventing the circular deadlock described in apache#70526. This is the dispatch-side equivalent of the fix applied to end() in apache#67881.

pierrejeambrun · 2026-08-17T10:18:12Z

@o-nikolas should this be included and backported in 3.3.2? Also #70545 depends on that too.

o-nikolas · 2026-08-17T23:33:07Z

@o-nikolas should this be included and backported in 3.3.2? Also #70545 depends on that too.

@pierrejeambrun I don't feel too strongly about it personally. But since the call has been made for #70545 then we should pull this one in as well I reckon.

CC @vatsrahul1001

pierrejeambrun · 2026-08-20T09:37:42Z

Actually after second consideration I've moved that to 3.4.0 so it has time to sink in before 3.4.0 (and spend some time under testing before being released). But i'm ok if we want this in 3.3.2.

o-nikolas · 2026-08-25T00:18:00Z

Actually after second consideration I've moved that to 3.4.0 so it has time to sink in before 3.4.0 (and spend some time under testing before being released). But i'm ok if we want this in 3.3.2.

Yeah, I think that's a good call @pierrejeambrun!

…e dispatch deadlock Fixes the runtime/dispatch-path deadlock reported in apache#70526: during _process_workloads the scheduler could block in activity_queue.put() while workers were blocked writing results into a full result_queue pipe, a circular wait. Draining the result queue before each put breaks the cycle. The end()/join path is already handled by apache#67881; this covers normal dispatch. Adds a call-order unit test and a bounded real-queue regression test.

…e dispatch deadlock (#70545) Fixes the runtime/dispatch-path deadlock reported in #70526: during _process_workloads the scheduler could block in activity_queue.put() while workers were blocked writing results into a full result_queue pipe, a circular wait. Draining the result queue before each put breaks the cycle. The end()/join path is already handled by #67881; this covers normal dispatch. Adds a call-order unit test and a bounded real-queue regression test.

…event runtime dispatch deadlock (#70545) (#74949) Fixes the runtime/dispatch-path deadlock reported in #70526: during _process_workloads the scheduler could block in activity_queue.put() while workers were blocked writing results into a full result_queue pipe, a circular wait. Draining the result queue before each put breaks the cycle. The end()/join path is already handled by #67881; this covers normal dispatch. Adds a call-order unit test and a bounded real-queue regression test. (cherry picked from commit b00cfeb) Co-authored-by: Saksham Kapoor <sakshamkapoor2911@gmail.com>

…e dispatch deadlock (apache#70545) Fixes the runtime/dispatch-path deadlock reported in apache#70526: during _process_workloads the scheduler could block in activity_queue.put() while workers were blocked writing results into a full result_queue pipe, a circular wait. Draining the result queue before each put breaks the cycle. The end()/join path is already handled by apache#67881; this covers normal dispatch. Adds a call-order unit test and a bounded real-queue regression test.

Copilot AI review requested due to automatic review settings June 2, 2026 00:24

SakshamKapoor2911 requested review from XD-DENG, ashb, dheerajturaga, hussein-awala, o-nikolas and pierrejeambrun as code owners June 2, 2026 00:24

boring-cyborg Bot added the area:Executors-core LocalExecutor & SequentialExecutor label Jun 2, 2026

Copilot AI reviewed Jun 2, 2026

View reviewed changes

potiuk added the ready for maintainer review Set by triage when all criteria pass, or by the fork CI reconciler when fork CI is green. label Jun 12, 2026

o-nikolas reviewed Jun 22, 2026

View reviewed changes

Comment thread airflow-core/src/airflow/executors/local_executor.py Outdated

Comment thread airflow-core/src/airflow/executors/local_executor.py Outdated

Comment thread airflow-core/src/airflow/executors/local_executor.py

SakshamKapoor2911 added 4 commits June 27, 2026 09:52

Drain result_queue in LocalExecutor to prevent process join deadlock

f52f438

Resolve SimpleQueue API mismatch and close queues in finally block

a628f80

Fix ruff linting violations in local_executor.py

2aa1345

Address LocalExecutor shutdown review comments

28fa007

SakshamKapoor2911 force-pushed the fix-local-executor-deadlock branch from 33cd755 to 5684360 Compare June 27, 2026 14:36

o-nikolas approved these changes Jun 30, 2026

View reviewed changes

SakshamKapoor2911 changed the title ~~fix: drain result_queue in LocalExecutor to prevent process join dead…~~ Drain result_queue in LocalExecutor to prevent process join deadlock Jun 30, 2026

SakshamKapoor2911 force-pushed the fix-local-executor-deadlock branch from 5684360 to 28fa007 Compare June 30, 2026 21:14

o-nikolas merged commit 670785d into apache:main Jul 2, 2026
77 checks passed

FrankYang0529 mentioned this pull request Jul 7, 2026

Fix flaky test timeout test_end_drains_result_queue_to_avoid_join_deadlock on Python 3.14 #69525

Closed

1 task done

darren-cheung-sa mentioned this pull request Jul 27, 2026

Scheduler deadlocks in pipe write during task dispatch (LocalExecutor) — runtime variant of #67881 #70526

Closed

1 of 2 tasks

SakshamKapoor2911 mentioned this pull request Jul 27, 2026

Should we replace spot-fixes with a background drain thread in LocalExecutor? #70543

Closed

SakshamKapoor2911 mentioned this pull request Jul 27, 2026

Drain result_queue before dispatch to prevent runtime deadlock in LocalExecutor #70545

Merged

1 task done

SakshamKapoor2911 mentioned this pull request Jul 28, 2026

Add aggregate reassembly byte budget for DTLS handshake state openssl/openssl#32095

Open

pierrejeambrun modified the milestones: Airflow 3.4.0, Airflow 3.3.2 Aug 20, 2026

pierrejeambrun added the backport-to-v3-3-test label Aug 20, 2026

pierrejeambrun modified the milestones: Airflow 3.3.2, Airflow 3.4.0 Aug 20, 2026

pierrejeambrun removed the backport-to-v3-3-test label Aug 20, 2026

github-actions Bot mentioned this pull request Oct 18, 2026

[v3-4-test] Drain result_queue before dispatch in LocalExecutor to prevent runtime dispatch deadlock (#70545) #74949

Merged

vatsrahul1001 mentioned this pull request Oct 28, 2026

Status of testing of Apache Airflow 3.4.0rc1 #75865

Open

Conversation

SakshamKapoor2911 commented Jun 2, 2026 • edited by potiuk Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

Changes

Context

Was generative AI tooling used to co-author this PR?

Uh oh!

boring-cyborg Bot commented Jun 2, 2026

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

SakshamKapoor2911 commented Jun 4, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

o-nikolas left a comment

Choose a reason for hiding this comment

Uh oh!

SakshamKapoor2911 commented Jun 30, 2026

Uh oh!

o-nikolas commented Jul 1, 2026

Uh oh!

SakshamKapoor2911 commented Jul 1, 2026

Uh oh!

Uh oh!

boring-cyborg Bot commented Jul 2, 2026

Uh oh!

pierrejeambrun commented Aug 17, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

o-nikolas commented Aug 17, 2026

Uh oh!

pierrejeambrun commented Aug 20, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

o-nikolas commented Aug 25, 2026

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

SakshamKapoor2911 commented Jun 2, 2026 •

edited by potiuk

Loading

SakshamKapoor2911 commented Jun 4, 2026 •

edited

Loading

pierrejeambrun commented Aug 17, 2026 •

edited

Loading

pierrejeambrun commented Aug 20, 2026 •

edited

Loading