Tasks are immediately restarted after unknown "Server indicated the task shouldn't be running anymore", resulting in HTTP 409 #59378

Activity

1. rjh-yext

   added

   kind:bugThis is a clearly a bugThis is a clearly a bug

   area:core

   needs-triagelabel for new issues that we didn't triage yetlabel for new issues that we didn't triage yet

   on Dec 12, 2025
2. boring-cyborg commented on Dec 12, 2025

   

   boring-cyborgBot

   on Dec 12, 2025 – with boring-cyborg

   More actions

   Thanks for opening your first issue here! Be sure to follow the issue template! If you are willing to raise PR to address this issue please do so, no need to wait for approval.
3. danielford831 commented on Dec 17, 2025

   

   danielford831

   on Dec 17, 2025

   Last edited by danielford831

   More actions

   Also seeing this on 3.1.4 (which I'll bump to 3.1.5 today, but I notice @rjh-yext is on 3.1.3 so hopefully unrelated)

   2 schedulers on EKS. also useful to note there's no mention of an attempt 1 in the audit log and the UI, but the log for attempt 1 is populated in EFS, attempt 2 log is empty
4. davidmarshmallow commented on Dec 17, 2025

   

   davidmarshmallow

   on Dec 17, 2025

   More actions

   Also seeing this on 3.1.4
5. vinitpayal commented on Dec 17, 2025

   

   vinitpayal

   on Dec 17, 2025

   Contributor

   More actions

   Same for us in Airflow 3.1.4.

   In UI randomly seeing task_number starting from 11, 16 etc and because of which task retries we are setting aren't working.

   Screenshot for reference(only one retry with try number 11)

   

   Same is the case in metadata database as well

   

   👍React with 👍1Reacted by Ryan Pope
6. guillaumecollombet commented on Dec 18, 2025

   

   guillaumecollombet

   on Dec 18, 2025

   More actions

   Same thing in 3.1.5
7. abhijeets25012-tech commented on Dec 26, 2025

   

   abhijeets25012-tech

   on Dec 26, 2025

   Contributor

   More actions

   Scheduler A Scheduler B
   | |
   RUNNING |
   | |
   Task starts |
   | |
   |--------------> SCHEDULED
   | |
   Heartbeat (409) |
   | |
   Task killed & retried |
   I’m new to open source and this is my first PR, so feedback is very welcome.

   While investigating an issue with multiple schedulers, I observed that a
   TaskInstance can regress from RUNNING back to SCHEDULED due to a race condition.
   This leads to heartbeat conflicts (409), premature task termination, and
   repeated task retries.

   This PR adds a small guard in TaskInstance.set_state() to prevent the invalid
   RUNNING → SCHEDULED transition while allowing all valid state transitions.
   This avoids killing already-running tasks and preserves correct execution.

   Tested locally with a multi-scheduler setup.

   👀React with 👀1Reacted by David Blain
8. dabla commented on Jan 14, 2026

   

   dabla

   on Jan 14, 2026

   Contributor

   More actions

   Scheduler A Scheduler B | | RUNNING | | | Task starts | | | |--------------> SCHEDULED | | Heartbeat (409) | | | Task killed & retried | I’m new to open source and this is my first PR, so feedback is very welcome.

   While investigating an issue with multiple schedulers, I observed that a TaskInstance can regress from RUNNING back to SCHEDULED due to a race condition. This leads to heartbeat conflicts (409), premature task termination, and repeated task retries.

   This PR adds a small guard in TaskInstance.set_state() to prevent the invalid RUNNING → SCHEDULED transition while allowing all valid state transitions. This avoids killing already-running tasks and preserves correct execution.

   Tested locally with a multi-scheduler setup.

   @abhijeets25012-tech I can't find the PR you're mentioning?
9. dabla commented on Jan 14, 2026

   

   dabla

   on Jan 14, 2026

   Last edited by dabla

   Contributor

   More actions

   Issue still there with Airflow 3.1.6, running with 3 schedulers:

   [2026-01-14 06:25:42] INFO - DAG bundles loaded: dags-folder
   [2026-01-14 06:25:42] INFO - Filling up the DagBag from /usr/local/airflow/dags/jb_sccm/dag.py
   [2026-01-14 06:25:42] WARNING - Skipping masking for a secret as it's too short (<5 chars)
   [2026-01-14 06:25:42] INFO - Establishing WinRM connection to host: obfuscated-server-name
   [2026-01-14 06:25:43] INFO - Running command: 'python C:\python_scripts\airflow_bods_trigger\bods_job_trigger.py -j JB_SCCM'...
   [2026-01-14 06:28:29] ERROR - Server indicated the task shouldn't be running anymore. Terminating process detail={"detail":{"reason":"not_found","message":"Task Instance not found"}}
   [2026-01-14 06:28:34] ERROR - Task killed!
   
10. dabla commented on Jan 14, 2026

    

    dabla

    on Jan 14, 2026

    Last edited by dabla

    Contributor

    More actions

    In the meantime found another related issue #5761.
11. bohdan-pd commented on Jan 26, 2026

    

    bohdan-pd

    on Jan 26, 2026

    More actions

    Same thing in airflow 3.0.6

    👍React with 👍2Reacted by Christian Ichebi and Ann von Brincken
12. shahar1

    removed

    needs-triagelabel for new issues that we didn't triage yetlabel for new issues that we didn't triage yet

    on Jan 31, 2026
13. vin-c-ent commented on Feb 11, 2026

    

    vin-c-ent

    on Feb 11, 2026

    Last edited by vin-c-ent

    Contributor

    More actions

    same here, airflow 3.1.3 with 4 schedulers
14. 8 remaining items

    Load more
15. eladkal

    added

    affected_version:3.1Issues Reported for 3.1Issues Reported for 3.1

    on Mar 12, 2026
16. sam-dumont

    added 4 commits that reference this issue on Mar 13, 2026

    Fix ti_skip_downstream overwriting RUNNING tasks to SKIPPED

    ...

    This issue will close once commit f7a4f14 is merged into the 'main' branch.

    f7a4f14

    Fix ti_skip_downstream overwriting RUNNING tasks to SKIPPED

    ...

    This issue will close once commit 31f4050 is merged into the 'main' branch.

    31f4050

    Fix ti_skip_downstream overwriting RUNNING tasks to SKIPPED

    ...

    This issue will close once commit 08ece10 is merged into the 'main' branch.

    08ece10

    Fix ti_skip_downstream overwriting RUNNING tasks to SKIPPED

    ...

    This issue will close once commit 9dd13fb is merged into the 'main' branch.

    9dd13fb
17. sam-dumont commented on Mar 13, 2026

    

    sam-dumont

    on Mar 13, 2026

    Contributor

    More actions

    We hit this on our prod cluster (830 DAGs, ~25k tasks/day, 2 schedulers, KubernetesExecutor, ~500 peak concurrent workers on EKS) and spent 2+ weeks instrumenting it. Sharing what we found in case it helps.

    We traced it to at least 2 independent race conditions, both unguarded UPDATEs on task instance state that can overwrite a RUNNING task :

    1. schedule_tis() in dagrun.py : a competing scheduler overwrites RUNNING → SCHEDULED. Worker heartbeats, gets 409 with current_state: scheduled, task killed. PR Fix duplicate task execution when running multiple schedulers (HA) #60330 by @ephraimbuddy addresses this.

    2. ti_skip_downstream() in the Execution API : BranchOperator marks unchosen tasks as SKIPPED, but the UPDATE has no state guard, so it can overwrite tasks already RUNNING. Worker heartbeats, gets 409 with current_state: skipped, task killed. We opened PR Fix ti_skip_downstream overwriting RUNNING tasks to SKIPPED #63266 for this one.

    Both seem to exist in every Airflow 3.x release with 2+ schedulers. The frequency likely scales with concurrency : more schedulers and more tasks = wider race window.

    What worked for us :

    We patched both code paths at Docker build time (patching the installed .py files so every process gets the fix). Our apply_patches.py script does this (auto-detects already-patched files, self-disables when upstream fix lands). Got us from 374 errors/day to 0.

    One thing we learned the hard way : monkey patches in airflow_local_settings.py worked for schedule_tis() (runs in the scheduler) but NOT for ti_skip_downstream(). That one runs on the API server (FastAPI/uvicorn), which has a different startup path and never imports airflow_local_settings.py. We had zero 409s for 18h, then skipped 409s exploded to 131/day when load increased. Build-time patching was the only approach that covered all processes.

    Diagnosing which vector you're hitting :

    Parsing the current_state field from the 409 response body was the key to telling them apart :

    current_state	What's happening	Relevant PR
    scheduled	schedule_tis() race : RUNNING → SCHEDULED	PR #60330
    failed	Cascade from the above	PR #60330
    skipped	ti_skip_downstream() race : RUNNING → SKIPPED (branching DAGs)	PR #63266
    not_found	K8s executor duplicate pod / stale UUID after scheduler restart	No fix yet (#57618)

    Our timeline (2+ weeks of continuous monitoring) :

    Date	409s	What changed
    Feb 27-Mar 5	14-169	baseline, no fixes
    Mar 6	22	schedule_tis patch deployed : scheduled+failed dropped to 0
    Mar 7-9	3-4	only skipped remaining
    Mar 10	0	both patches active
    Mar 13	0	build-time patches on PROD : 0 race 409s, 0 orphaned tasks

    Note if you're on 3.1.7 specifically : on top of the race conditions above, 3.1.7 has a missing task_reschedule index + DepContext mutation leak (#59604) that slows the scheduler, grows memory (ours peaked at 5.92/6 GiB), and widens the race window for both bugs. This created a feedback loop on our cluster where 409s escalated exponentially (5 → 26 → 133 → 374/day). Upgrading to 3.1.8 broke the loop (#60931, #62089), but the race conditions still need the patches on top.

    Related PRs :

    What	PR	Status
    schedule_tis() state guard	#60330 (@ephraimbuddy)	Open, validated on our prod
    ti_skip_downstream() state guard	#63266 (ours)	Open
    Missing index + DepContext leak (3.1.7 only)	#60931, #62089	Fixed in 3.1.8

    (AI-assisted investigation with Claude Code. All monitoring data from our prod cluster.)

    🚀React with 🚀4Reacted by David Blain, Sam Wheating, prayaagmahajan and Bohdan Bondar
18. dabla commented on Mar 13, 2026

    

    dabla

    on Mar 13, 2026

    Last edited by dabla

    Contributor

    More actions

    We hit this on our prod cluster (830 DAGs, ~25k tasks/day, 2 schedulers, KubernetesExecutor, ~500 peak concurrent workers on EKS) and spent 2+ weeks instrumenting it. Sharing what we found in case it helps.

    We traced it to at least 2 independent race conditions, both unguarded UPDATEs on task instance state that can overwrite a RUNNING task :

    1. schedule_tis() in dagrun.py : a competing scheduler overwrites RUNNING → SCHEDULED. Worker heartbeats, gets 409 with current_state: scheduled, task killed. PR Fix HA scheduler try_number double increment #60330 by @ephraimbuddy addresses this.
    2. ti_skip_downstream() in the Execution API : BranchOperator marks unchosen tasks as SKIPPED, but the UPDATE has no state guard, so it can overwrite tasks already RUNNING. Worker heartbeats, gets 409 with current_state: skipped, task killed. We opened PR Fix ti_skip_downstream overwriting RUNNING tasks to SKIPPED #63266 for this one.

    Both seem to exist in every Airflow 3.x release with 2+ schedulers. The frequency likely scales with concurrency : more schedulers and more tasks = wider race window.

    What worked for us :

    We patched both code paths at Docker build time (patching the installed .py files so every process gets the fix). Our apply_patches.py script does this (auto-detects already-patched files, self-disables when upstream fix lands). Got us from 374 errors/day to 0.

    One thing we learned the hard way : monkey patches in airflow_local_settings.py worked for schedule_tis() (runs in the scheduler) but NOT for ti_skip_downstream(). That one runs on the API server (FastAPI/uvicorn), which has a different startup path and never imports airflow_local_settings.py. We had zero 409s for 18h, then skipped 409s exploded to 131/day when load increased. Build-time patching was the only approach that covered all processes.

    Diagnosing which vector you're hitting :

    Parsing the current_state field from the 409 response body was the key to telling them apart :

    current_state What's happening Relevant PR
    scheduled schedule_tis() race : RUNNING → SCHEDULED PR #60330
    failed Cascade from the above PR #60330
    skipped ti_skip_downstream() race : RUNNING → SKIPPED (branching DAGs) PR #63266
    not_found K8s executor duplicate pod / stale UUID after scheduler restart No fix yet (#57618)
    Our timeline (2+ weeks of continuous monitoring) :

    Date 409s What changed
    Feb 27-Mar 5 14-169 baseline, no fixes
    Mar 6 22 schedule_tis patch deployed : scheduled+failed dropped to 0
    Mar 7-9 3-4 only skipped remaining
    Mar 10 0 both patches active
    Mar 13 0 build-time patches on PROD : 0 race 409s, 0 orphaned tasks
    Note if you're on 3.1.7 specifically : on top of the race conditions above, 3.1.7 has a missing task_reschedule index + DepContext mutation leak (#59604) that slows the scheduler, grows memory (ours peaked at 5.92/6 GiB), and widens the race window for both bugs. This created a feedback loop on our cluster where 409s escalated exponentially (5 → 26 → 133 → 374/day). Upgrading to 3.1.8 broke the loop (#60931, #62089), but the race conditions still need the patches on top.

    Related PRs :

    What PR Status
    schedule_tis() state guard #60330 (@ephraimbuddy) Open, validated on our prod
    ti_skip_downstream() state guard #63266 (ours) Open
    Missing index + DepContext leak (3.1.7 only) #60931, #62089 Fixed in 3.1.8
    (AI-assisted investigation with Claude Code. All monitoring data from our prod cluster.)

    Nice investigation and explanation, we don’t have those problems as often anymore. Will try do do the same at our side and see if it also fixes the issue.

    We are running with 3 concurrent schedulers but since we now migrated from YugabyteDB to Postgres again that maybe also helped already. As YugabyteDB isn’t officially supported on Airflow we had to do some post alterations to make it run correctly on YugabyteDB as the Airflow migrations scripts aren’t optimized for it.

    We are running on 3.1.8 so that may also help improve it as you stated in your comment.
19. bohdan-pd commented on Mar 16, 2026

    

    bohdan-pd

    on Mar 16, 2026

    More actions

    @sam-dumont Thank you! I tested your patch on MWAA and still got this error. Maybe there is another bug as well:
    TI is no longer in the running state and task should terminate
20. sam-dumont commented on Mar 16, 2026

    

    sam-dumont

    on Mar 16, 2026

    Contributor

    More actions

    @sam-dumont Thank you! I tested your patch on MWAA and still got this error. Maybe there is another bug as well:
    TI is no longer in the running state and task should terminate

    Oh too bad 😔 we still have a few 409 but they are legitimate timeouts (explicit timeouts at the task level). I haven't use MWAA in a while so I'm not sure how easy it is to patch, do you have any control over the docker images?

    If you have control over the airflow_local_settings.py you can use the monkeypatch in this gist : https://gist.github.com/sam-dumont/c362da9c42dd5930266c855c4b8e42fa

    The first one only, the second does not work as it's running on the api server.

    From what I remember MWAA was quite limited in customization and this is why we took the 'custom' road, but we tested 3 years ago so it probably changed since them.

    Hope it'll help... 🤞
21. kaxil

    added a commit that references this issue on Mar 19, 2026

    Fix ti_skip_downstream overwriting RUNNING tasks to SKIPPED

    ...

    4d57ba4
22. kaxil

    closed this as completedin 6d1794aon Mar 19, 2026
23. aws-airflow-bot

    added a commit that references this issue on Mar 19, 2026

    Fix downstream tasks being incorrectly skipped in HA scheduler mode (a…

    ...

    This issue will close once commit 6d1794a is merged into the 'main' branch.

    6d1794a
24. fat-catTW

    added a commit that references this issue on Mar 22, 2026

    Fix downstream tasks being incorrectly skipped in HA scheduler mode (a…

    ...

    This issue will close once commit f4193a1 is merged into the 'main' branch.

    f4193a1
25. Suraj-kumar00

    added a commit that references this issue on Apr 7, 2026

    Fix downstream tasks being incorrectly skipped in HA scheduler mode (a…

    ...

    This issue will close once commit 6be4738 is merged into the 'main' branch.

    6be4738
26. abhijeets25012-tech

    added a commit that references this issue on Apr 9, 2026

    Fix downstream tasks being incorrectly skipped in HA scheduler mode (a…

    ...

    This issue will close once commit bd3f270 is merged into the 'main' branch.

    bd3f270