Add ability to automatically set DAG to off after X times it failed sequentially

[eladkal]

Body

This idea may raise some edge cases but the core concept of it is: If a DAG fails more than X times sequentially then it probably needs some manual action thus there is no point in keeping as active. When running large deployment of Airflow I discovered that many of the DAGs were some experiments/ad hoc/tests that were leftovers and no one noticed or needed.

The value of this feature is that it helps to reduce load from the scheduler and workers. This can be very helpful for large Airflow deployments with many DAG authors that share the same instance.

This should be optional functionality with default as -1 (which means feature is off - for backward compatibility)

Committer

• I acknowledge that I am a maintainer/committer of the Apache Airflow project.

[pateash]

Hi @eladkal,
I can check this out if there are no takers.

[eladkal]

@pateash assigned

[pateash]

@eladkal My initial thoughts on this, we have few options

1. We can have config variable/env variables, similar to

[core]
dags_are_paused_at_creation 

2. Adding an argument in Dag Object to do this at every dag level, which would require an entry in DagModel and table.
   similar to
   is_paused_upon_creation in DAG object

3. Adding both similar to above is_paused_upon_creation
   Let me know your thoughts.

[eladkal]

@pateash i prefer to have it as parameter on dag level as it allows flexibility to set the needed behavior per dag

[pateash]

Makes sense, this will require

• An additional database columns
• by default this value will be null/None, and user will have an ability to set value from Dag Params

[eladkal]

cool

you can also add default setting in core (as you mentioned) but that is in addition. Should be quite simple to do once you have the functionality working.

[jscheffl]

I am wondering a bit, do we really need a feature in the core for this? So maybe this is convenience but as a user you could add a on_failure_callback and add some custom logic there as well?

[eladkal]

I am wondering a bit, do we really need a feature in the core for this? So maybe this is convenience but as a user you could add a on_failure_callback and add some custom logic there as well?

This capability has a big advantage of saving $$ as it simply stop executing code that constantly fail. To my prespective its important capability that worth implementing