Skip to content

Adding ability to automatically set DAG to off after X times it failed sequentially - #36935

Merged
eladkal merged 42 commits into
apache:mainfrom
pateash:36239-automatically-set-DAG-off
Mar 15, 2024
Merged

Adding ability to automatically set DAG to off after X times it failed sequentially#36935
eladkal merged 42 commits into
apache:mainfrom
pateash:36239-automatically-set-DAG-off

Conversation

@pateash

@pateash pateash commented Jan 21, 2024 •
edited
Loading

Copy link
Copy Markdown
Contributor

closes: #36239

Description

This PR adds ability to automatically turn off DAG after X number of consecutive failures.

There are 2 ways to configure this

  1. in DAG() using max_consecutive_failed_dag_runs parameter this will override any config (as below)
  2. Using core config inside airflow.cfg or variable AIRFLOW__CORE_MAX_CONSECUTIVE_FAILED_DAG_RUNS_PER_DAG
    environment variable.

@eladkal eladkal changed the title 36239 automatically set dag off Add ability to automatically set DAG to off after X times it failed sequentially Jan 21, 2024
@eladkal eladkal added this to the Airflow 2.9.0 milestone Jan 21, 2024
@eladkal eladkal added the type:new-feature Changelog: New Features label Jan 21, 2024
dirrao
dirrao requested changes Jan 24, 2024
View reviewed changes

@dirrao dirrao left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Great work. Can you add the test cases for this feature?

Comment thread airflow/models/dagrun.py Outdated
BasPH
BasPH reviewed Jan 24, 2024
View reviewed changes

@BasPH BasPH left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think this would be a helpful feature. Left a few comments around naming conventions

Comment thread airflow/config_templates/config.yml Outdated
Comment thread airflow/config_templates/config.yml Outdated
Comment thread airflow/models/dag.py Outdated
Comment thread airflow/models/dagrun.py Outdated
Comment thread airflow/models/dagrun.py Outdated
Comment thread airflow/models/dagrun.py Outdated
Comment thread airflow/models/dagrun.py Outdated
Comment thread airflow/models/dagrun.py Outdated
Comment thread airflow/models/dagrun.py Outdated
Comment thread airflow/models/dagrun.py Outdated
@pateash pateash changed the title Add ability to automatically set DAG to off after X times it failed sequentially Adding ability to automatically set DAG to off after X times it failed sequentially Jan 27, 2024
@pateash
pateash force-pushed the 36239-automatically-set-DAG-off branch 2 times, most recently from d9d29c2 to 3ff75bf Compare February 1, 2024 10:42
@pateash
pateash force-pushed the 36239-automatically-set-DAG-off branch 2 times, most recently from 8e54af0 to 6df778f Compare February 5, 2024 06:15
@pateash
pateash force-pushed the 36239-automatically-set-DAG-off branch from 394d028 to 6d7e91d Compare February 12, 2024 18:48
@pateash
pateash marked this pull request as ready for review February 13, 2024 06:02
@pateash
pateash force-pushed the 36239-automatically-set-DAG-off branch from e836d92 to 081176f Compare February 13, 2024 06:03
@pateash
pateash requested review from BasPH and dirrao February 13, 2024 06:03
@pateash

pateash commented Feb 13, 2024 •
edited
Loading

Copy link
Copy Markdown
Contributor Author

TODO:

  • adding test for core config
  • adding test for environment variable

@pateash
pateash force-pushed the 36239-automatically-set-DAG-off branch 2 times, most recently from 80170e4 to 81ed11f Compare February 13, 2024 14:35
@pateash
pateash requested a review from potiuk as a code owner February 13, 2024 18:34
@pateash
pateash force-pushed the 36239-automatically-set-DAG-off branch 2 times, most recently from dfaa53a to f34cd9e Compare February 14, 2024 07:10
@eladkal
eladkal force-pushed the 36239-automatically-set-DAG-off branch from a4af994 to bc56ee4 Compare March 15, 2024 10:51
@eladkal
eladkal merged commit 777a216 into apache:main Mar 15, 2024
@eladkal

eladkal commented Mar 15, 2024

Copy link
Copy Markdown
Contributor

🎉 🎉 🎉

@pateash
pateash deleted the 36239-automatically-set-DAG-off branch March 15, 2024 12:58
@pateash

pateash commented Mar 15, 2024

Copy link
Copy Markdown
Contributor Author

🎉 🎉 🎉

Thanks a lot @eladkal

@o-nikolas

o-nikolas commented Mar 15, 2024

Copy link
Copy Markdown
Contributor

Looks like the diagrams weren't updated along with this migration? When you run precommit locally now, diagram changes are made for code that is unrelated.

@o-nikolas o-nikolas mentioned this pull request Mar 15, 2024
Merged
This was referenced Mar 17, 2024
Merged
Merged
@bbovenzi bbovenzi mentioned this pull request Mar 20, 2024
Closed
1 of 2 tasks
@sean-rose sean-rose mentioned this pull request Oct 31, 2024
Merged
@kaxil kaxil mentioned this pull request Apr 13, 2026
Closed
1 of 2 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@eladkal eladkal eladkal approved these changes
@ephraimbuddy ephraimbuddy Awaiting requested review from ephraimbuddy
@pierrejeambrun pierrejeambrun Awaiting requested review from pierrejeambrun
@kaxil kaxil Awaiting requested review from kaxil
@ashb ashb Awaiting requested review from ashb
@bolkedebruin bolkedebruin Awaiting requested review from bolkedebruin
@XD-DENG XD-DENG Awaiting requested review from XD-DENG
@BasPH BasPH Awaiting requested review from BasPH
@dirrao dirrao Awaiting requested review from dirrao
@potiuk potiuk Awaiting requested review from potiuk
@amoghrajesh amoghrajesh Awaiting requested review from amoghrajesh
@uranusjr uranusjr Awaiting requested review from uranusjr

Assignees

No one assigned

Labels

area:API Airflow's REST/HTTP API area:CLI area:serialization type:new-feature Changelog: New Features

Projects

None yet

Milestone

Airflow 2.9.0

Development

Successfully merging this pull request may close these issues.

Add ability to automatically set DAG to off after X times it failed sequentially

8 participants

@pateash @potiuk @eladkal @o-nikolas @uranusjr @BasPH @amoghrajesh @dirrao