Repository navigation
Add ability to disable a running DAG only after after it's in a finished state #22006
Description
Activity
Good idea. Would you like to work on maybe ? this is the best way to make sure it happens :)
I wonder if this should be added as a configuration for pausing/unpausing DAGs? Having a separate button for this could be confusing.
The use case as described is "Don't schedule new runs but let the current running dags to finish" - This sounds more like a scheduler feature rather than UI.
As a user, I'd like to have functionality in the UI that allows me to either "disable and stop the DAG immediately", or "disable DAG once finished". How to implement such a feature in the UI, such that it's not confusing is of course up for discussion - button, switch, etc.
The underlying functionality to perform the "drain" will likely be need to be developed in the scheduler. So, this feature I think, falls under both scheduler, and UI development areas.Also, @potiuk - I can try to pick this task up in a couple of weeks. Let's leave it unassigned for now, and I can let you know once I am ready to pick it up.
Reacted by Jarek Potiuk, andrew-washington and UrielSSure!
I think this involves some changes to the DB schema as probably it means that
is_pausedis no longer just bool of True / False but actually it has several states.I think this involves some changes to the DB schema as probably it means that
is_pausedis no longer just bool of True / False but actually it has several states.This is why I am partial to it being a config for the scheduler because then we can't represent
is_pausedas an on/off switch.33 remaining items
Thanks for reminding me that I am free to open a PR in an open source project, I am aware. I'd sincerely love to l, but whether I have the skill and time is another problem. The people who opened and upvoted this issue are proof that there is a need for the same feature I am advocating for (gracefully finish all running instances of a DAG). I am not asking for anything more or different.
Sure. If someone will take it and provide a PR, that will be accepted, it will be done. I am not telling anything different than that (and that IMHO dag versioning will solve the problem). And yes, there is a possibility I do not understand how DAG versioning will not solve it.
But again - anyone is free to implement it. I understand you migh not have the skills, the second best thing is to find somoene who will implement it. And you are also free to seek that person. Or just hope someone will take it and implement it.
But again my personal assesment here (which is unchanged) has nothing to do with it. Maybe someone will propose PR and implement it regardless. It's a possibility my explanation on how DAG versioning solve it will not be enough convincing for them as well.
@potiuk, @jscheffl, @eladkal Revisiting this since a lot has changed since the initial discussion. I see both variants of pause: "freeze immediately" and "allow existing dag runs to finish" valid and very useful. I would like to see a drain existing runs option myself aswell.
I like @eladkal proposal of a
pause_mode: "suspend" (default) | "drain"and "drain eventually is switched to paused when the drain is completed"@jscheffl On the scheduler-performance concern,
dag.is_pausedhas no index today, so the existing gating already assumes a predicate on the already-joined dag row is cheap. A mode check on that same row is the same cost class, and only ever set transiently.If everyone is in alignment, I can take a jab at this
If everyone is in alignment, I can take a jab at this
This might need smallish AIP but I think you can start and we will figure this as we go.
If everyone is in alignment, I can take a jab at this
This might need smallish AIP but I think you can start and we will figure this as we go.
let me work on a draft
Reacted by andrew-washington and Howard Martinez- added 8 commits that reference this issue
on Sep 14, 2026 - added a commit that references this issue
on Sep 26, 2026
Description
I'd like the ability to "drain" a DAG from the UI. This would mean:
Use case/motivation
Currently, if you disable a DAG, it stops the DAG midway - the running tasks are completed, and queued tasks remain on the queue. I would like a way to disable the DAG, but if there is an active Dag Run it should continue running until it reaches a finished state.
For us, this would be especially helpful when upgrading our Airflow images/environments. To my understanding, Airflow doesn't have a failure-free zero-downtime upgrade (please point me to documentation if its possible). When upgrading, we don't want to fail/kill running DAGs, so we let them reach a finished state. Once none of the DAGs are in a running state, we perform the upgrades. Precisely timing the DAG disable operation such that a subsequent run doesn't start, is not a practical/viable solution.
For upgrades in the past, we have changed all the DAG schedules to None, which allows a running DAG to complete but doesn't start a subsequent run. Once upgrades are done, we update the DAG schedules back to their original value.
We would like to move away from this solution, and have some built-in functionality to deal with such a case.
Related issues
No response
Are you willing to submit a PR?
Code of Conduct