Repository navigation
Scheduler deadlocks in pipe write during task dispatch (LocalExecutor) — runtime variant of #67881 #70526
Description
Under which category would you file this issue?
Airflow Core
Apache Airflow version
3.3.0
What happened and how to reproduce it?
Issue Description
The scheduler permanently deadlocks during normal task dispatch (not shutdown), blocked in a pipe write() to the LocalExecutor's activity queue. All dispatch halts; the SchedulerJob heartbeat stops while the systemd service stays "active". ~10 minutes later, every running task's supervisor kills its task ("Server indicated the task shouldn't be running anymore"), burning retry attempts.
This appears to be the runtime/dispatch-path variant of the shutdown deadlock fixed in #67881 (merged 2026-07-02, not yet released — 3.3.0 still ships the pre-fix end() and empty terminate()).
Occurred 4 times in ~14 hours on one production host once dispatch volume rose (~150–250 concurrent tasks). Identical signature each time.
Evidence captured mid-deadlock:
/proc/<scheduler-pid>/stack→ blocked inpipe_write/proc/<scheduler-pid>/syscall→write(fd=9); fd 9 →pipe:[<inode>]- Scanning
/proc/*/fd: the same pipe inode held (read fd 7 + write fd 9) by the scheduler and every LocalExecutor worker process - Host healthy: 100+ GB free RAM, idle CPU, no OOM, clean kernel log
- One occurrence happened with zero api-server errors/timeouts, ruling out API-side causes
Steps to reproduce:
Race condition — probabilistic and load-correlated, so not minimally deterministic, but reliably triggered in our environment within minutes-to-hours:
- LocalExecutor,
parallelism = 256, defaultmax_tis_per_query(512), PostgreSQL backend - Generate dispatch bursts of 150–250 mostly lightweight, long-running tasks (ours poll remote Spark jobs), with tasks completing (returning results) concurrently with new dispatch
- Scheduler eventually blocks forever in
pipe_write; heartbeat stops
Notably there is no clean load threshold: the same host survived a 30-minute soak at 256 running tasks, then deadlocked at ~40 running during a later dispatch wave — consistent with a timing race on the shared queues, not resource exhaustion.
Suggested synthetic repro: a DAG with ~300 mapped time.sleep(300) tasks, cleared repeatedly to force dispatch bursts while earlier tasks are completing.
Mitigations tried (none sufficient):
| Change | Result |
|---|---|
[api] workers 4 → 12 |
fixed unrelated api saturation; deadlock recurred with 0 timeouts |
default_pool 300 → 150 |
reduced frequency; recurred |
[scheduler] max_tis_per_query 512 → 32 |
reduced frequency; recurred |
| Pausing the high-volume DAGs | stable (current workaround, at large throughput cost) |
What you think should happen instead?
The scheduler should never block indefinitely on executor queue writes — a full result/activity queue should trigger draining (or a bounded/non-blocking put with retry), so a slow consumer can't freeze the entire scheduling loop.
Suspected mechanism — same as #67881 describes, but hit from the dispatch side:
- Workers fill
result_queue's OS pipe buffer (~64KB) and block onput() - Blocked workers stop draining
activity_queue - Scheduler blocks in
_process_workloads()→self.activity_queue.put(workload)(airflow/executors/local_executor.py) - Being single-threaded at that point, the scheduler never reaches
_read_results(), soresult_queuenever drains → circular wait, permanent
#67881 adds result-queue draining during end()/join, but _process_workloads() still calls a blocking put() with no draining fallback, so the runtime deadlock remains possible on main as well.
Operating System
Amazon Linux 2023
Deployment
Virtualenv installation
Apache Airflow Provider(s)
No response
Versions of Apache Airflow Providers
n/a — core issue (apache-airflow-task-sdk 1.3.0 for reference)
Official Helm Chart version
Not Applicable
Kubernetes Version
No response
Helm Chart configuration
No response
Docker Image customizations
No response
Anything else?
Standalone deployment: systemd units for scheduler (LocalExecutor), api-server, and dag-processor on a single EC2 host; PostgreSQL metadata DB.
Happy to provide full /proc captures, journald excerpts of the supervisor mass-terminations, and timing data from all 4 occurrences on request.
Mainly using Claude to diagnose & type this report out.
- Looked myself for existing GitHub issues, and the one I linked was the closest thing I could find to ours.
- Looked myself at our Airflow server, and found running tasks randomly killed (with the
[2026-07-27 13:41:09] INFO - spark app is running...check in again in 120
[2026-07-27 13:43:15] INFO - spark app is running...check in again in 120
[2026-07-27 13:45:20] INFO - spark app is running...check in again in 120
[2026-07-27 13:47:24] INFO - spark app is running...check in again in 120
[2026-07-27 13:49:29] INFO - spark app is running...check in again in 120
[2026-07-27 13:50:40] ERROR - Server indicated the task shouldn't be running anymore. Terminating process detail={"reason":"not_found","message":"Task Instance not found, it may have been moved to the Task Instance History table"`)
message). This is what Claude diagnoses as an effect of a scheduler deadlock
Are you willing to submit PR?
- Yes I am willing to submit a PR!
Code of Conduct
- I agree to follow this project's Code of Conduct
Activity
boring-cyborg commentedon Jul 27, 2026
Thanks for opening your first issue here! Be sure to follow the issue template! If you are willing to raise PR to address this issue please do so, no need to wait for approval.
darren-cheung-sa commentedon Jul 27, 2026
@SakshamKapoor2911 wondering if you could confirm this issue was similar to yours / shed some light on it / suggest solution
I don't think your fix was included in 3.3.0, was it?
SakshamKapoor2911 commentedon Jul 27, 2026
Great diagnosis — this is indeed the dispatch-side gap that #67881 (shutdown
end()) did not cover. The same drain-before-write pattern applies: calling_read_results()before eachactivity_queue.put()in_process_workloadsensures workers blocked on a full result pipe get unblocked and can resume consuming from the activity queue, preventing the circular wait.Verified locally with existing tests (17 pass, 0 regressions) plus two new ordering tests. Reference branch with fix + tests:
https://github.com/SakshamKapoor2911/airflow/tree/fix-local-executor-dispatch-deadlock
Feel free to cherry-pick, adapt, or ignore — I will not open a competing PR since you have already put in the diagnostic work. Happy to review/support yours when ready.
Drafted-by: OpenCode (DeepSeek v4 Pro); reviewed by me before posting
darren-cheung-sa commentedon Jul 27, 2026
Thanks @SakshamKapoor2911 , I will take a look at your changes and try to adapt them for our use case. If successful, I'll open a PR and tag you for review, much appreciated
Could you explain how you proceeded after that / how I should proceed after that? I assume your fix isn't in the published Airflow 3.3.0 version. Are you somehow running your tasks on a different version of Airflow (e.g. the one with your fix), and if so, could you tell me how I could do that?
SakshamKapoor2911 commentedon Jul 27, 2026
Correct — neither fix is in the published 3.3.0. #67881 was merged to
mainafter the 3.3.0 cut and will ship in the next release.Both can be hot-patched directly into your 3.3.0 installation by editing a single file:
<venv>/lib/python3.x/site-packages/airflow/executors/local_executor.py.Shutdown deadlock (#67881) — apply the merged diff, which drains
result_queueduring worker join inend()and implementsterminate()to force-kill workers (currently a no-op).Dispatch deadlock (#70526) — add one line in
_process_workloads(), inside thefor workload in workload_list:loop, beforeself.activity_queue.put(workload):This drains the result queue before each dispatch, unblocking workers so they can resume consuming the activity queue — breaking the circular wait during normal runtime.
I didn't deploy either in production myself — both were verified against unit tests. My branch with the dispatch fix + tests is here. Would you like me to open a PR? I think you should be able to drive it — you've already done the hard diagnostic work here — but happy to co-author or take it if you'd rather focus on getting your deployment stable first.
darren-cheung-sa commentedon Jul 27, 2026
@SakshamKapoor2911 It would be great if you could take on the PR. I'm new/unfamiliar to this issue and to contributing to this repo (and as you said, also trying to make fixes to our pipelines). After that, I'll try to pull your fixes into my deployment (also new to that, but your steps seem easy enough)
Thanks for all the work & communication!
SakshamKapoor2911 commentedon Jul 27, 2026
I opened #70543 for discussion. It asks whether a continuous background drain thread would be a better long-term fix than the spot-drain approach both our PRs use.
The spot-fix from #70526 (drain before each put) is correct and works. The question in #70543 is whether we should prevent the underlying condition altogether so future contributors do not need to keep adding these defensive drains. No strong opinion either way, I wanted to surface it for maintainer guidance.
Would welcome your thoughts there since you have diagnosed the same root cause from the production side.
Drain result_queue before dispatch to prevent runtime deadlock in Loc…
Add tests for _process_workloads result-queue draining
SakshamKapoor2911 commentedon Jul 27, 2026
PR is open at #70545. The fix is the one-line change described above — you can hot-patch your 3.3.0 install right away while CI runs on the PR.
darren-cheung-sa commentedon Jul 27, 2026
I will take a look at your design in 70543 once I have a chance. And I will apply the 70545 fix locally if my other temp-fixes fail (obviously ideally I get my workflow working with official Airflow versions, but it might not be possible)
Thank you
Drain result_queue before dispatch to prevent runtime deadlock in Loc…
Add tests for _process_workloads result-queue draining
Drain result_queue before dispatch in LocalExecutor to prevent runtim…
Drain result_queue before dispatch in LocalExecutor to prevent runtim…
Drain result_queue before dispatch in LocalExecutor to prevent runtim…
[v3-4-test] Drain result_queue before dispatch in LocalExecutor to pr…
[v3-4-test] Drain result_queue before dispatch in LocalExecutor to pr…
Drain result_queue before dispatch in LocalExecutor to prevent runtim…