Repository navigation
Running tasks marked as skipped on DagRun timeout #30264
Description
Apache Airflow version
2.5.2
What happened
Users are experiencing the following:
- A DAG begins to run
- Task(s) go into running state, as expected
- The DagRun times out, marking any currently running task as SKIPPED
- Because tasks are not mark as failed the
on_failure_callbacknever gets revoked
Here are some example logs:
[2023-03-22, 16:30:02 PDT] {local_task_job.py:266} WARNING - DagRun timed out after 4:00:02.394287.
[2023-03-22, 16:30:07 PDT] {local_task_job.py:266} WARNING - DagRun timed out after 4:00:07.447373.
[2023-03-22, 16:30:07 PDT] {local_task_job.py:272} WARNING - State of this instance has been externally set to skipped. Terminating instance.
[2023-03-22, 16:30:07 PDT] {process_utils.py:129} INFO - Sending Signals.SIGTERM to group 8515. PIDs of all processes in the group: [8515]
What you think should happen instead
Once a DagRun times out, tasks that are currently in RUNNING should be marked as FAILED and downstream tasks should be marked as UPSTREAM_FAILED
How to reproduce
The following DAG will cause this intermittently
import time
import logging
from airflow.decorators import dag, task
from airflow.utils.dates import datetime, timedelta
@task
def task_1():
import random
pulses = random.randint(5, 10)
for i in range(pulses):
logging.info(f"pulsing: pulse...{i}")
time.sleep(4)
@task
def task_2():
import random
pulses = random.randint(10, 20)
for i in range(pulses):
logging.info(f"pulsing: pulse...{i}")
time.sleep(5)
@task
def downstream_finished_task():
logging.info("task finished")
time.sleep(20)
@dag(dag_id="dagrun_interval_test",
schedule_interval="*/5 * * * *",
start_date=datetime(2023, 3, 23),
dagrun_timeout=timedelta(seconds=30),
catchup=False)
def my_dag():
return [task_1(), task_2()] >> downstream_finished_task()
dag = my_dag()- Running tasks marked skipped
- Downstream left with
no status
See screenshot
Operating System
MacOS
Versions of Apache Airflow Providers
N/A
Deployment
Astronomer
Deployment details
Airflow Version 2.5.2
Anything else
Every time a DagRun times out
Are you willing to submit PR?
- Yes I am willing to submit a PR!
Code of Conduct
- I agree to follow this project's Code of Conduct
Activity
eladkal commentedon Mar 23, 2023
The dag timeouted so the dag status is marked as failed.
Tasks did not finish so why should they be set to fail?
airflow/airflow/jobs/scheduler_job.py
Lines 1302 to 1315 in 3239720
erdos2n commentedon Mar 24, 2023
@eladkal Maybe they should not be set to fail, but they should also not be set to skipped. The task was not skipped, technically speaking.
An issue that has come up is that a user wants an alert for a specific task failure, so they don't want to set the
on_failure_callbackon the dag level. That specific task gets marked skipped on adagrun_timeoutand theon_failure_callbackisn't triggered.I believe it's worth discussing either marking these tasks that are stopped mid-run as FAILED or introducing a new state into the task instance.
I'm curious if
SHUTDOWNmakes more sense in this instance. It seems to fit what is occurring more than skipped.https://github.com/apache/airflow/blob/main/airflow/utils/state.py#L42
Thoughts?
wolfier commentedon Mar 24, 2023
Instead of setting the state to SKIPPED, I propose calling handle_failure such that the callbacks are executed.
erdos2n commentedon Mar 24, 2023
Hey Alan, did more digging.
handle_failureis only called on failures, which means that my initial proposal ofSHUTDOWNwould not work, anyways. But if the remaining tasks are marked asupstream_failedthen they would trigger thehandle_failurecallback.So in short, adding the
handle_failurecallback would work, or marking downstream tasks asupstream_failedwould work. Thoughts?eladkal commentedon Mar 24, 2023
Just to clarify if the goal is to change the current behavior from skipped to fail this is a breaking change and can not happen before Airflow 3.
Before discussing how to get it done I suggest first to discuss if this should be done. I'm not convinced setting tasks to failure when dag timeout is the desired behavior.
erdos2n commentedon Mar 24, 2023
The goal is to change the behavior, but not necessarily from skipped to fail, just something to trigger the
handle_failuremethod so callbacks can exhibit (more) expected behavior from users.hussein-awala commentedon Mar 25, 2023
Since the task didn't fail, I don't see the need to run the failure callback in every stopped task, the dag failure callback is enough to handle this case, where we can check if the run failed due to timeout, and select skipped tasks in metadata to do what we need to do. WDYT?
erdos2n commentedon Mar 25, 2023
Well this user wants a callback if this specific task fails, so not on the dag level. Could be that we need a on skipped callback. Thoughts?
eladkal commentedon Apr 12, 2023
I'm OK with adding
on_skipped_callback(regardless of what we discuss here, this is probably something we should add)eladkal commentedon Apr 27, 2023
Should we scope this issue to adding
on_skipped_callback?@erdos2n is that a suitable solution for your use case?
pankajkoti commentedon Jul 21, 2023
@erdos2n would you have an update on the last question from Elad?
34 remaining items
raphaelauv commentedon Dec 12, 2024
No stale
CyberDuck79 commentedon Dec 18, 2024
Same behavior for us on MWAA; tasks are killed by SIGTERM. Could it launch any callback? The skipped callback was not configured in our case.
We discovered this behavior accidentally due to a bad configuration, but it raised 3 questions:
For the DAG-level callbacks, it's not the first time we have encountered an issue with our alerting system because tasks were killed (but it was due to the MWAA Airflow internal architecture). So, it could be a good feature, even if I think it is out of the scope of this issue.
CyberDuck79 commentedon Dec 18, 2024
Well, after experimentation, it seems there is confusion about callback configurations.
There is DAG level callbacks: #38897
But we were also confused by the configurations:
default_args(our old way of configuring callbacks) -> it configures the callbacks at the task level for all tasks.on_failure_callback, for example) -> callback triggered at dag level by dag state (so in case of dag run timeout, failed state -> failure callback)The second configuration would allow the callback to be triggered even if the task process is killed.
For the question of the skipped status:
Maybe the skipped status is not very clear because the task could have already started, but this information is included in the skipped callback so the DAG run timeout case can be handled. It seems that in Airflow, the meaning of the skipped state includes unfinished tasks.
An unfinished state could be more clear, but is it worse to add a new state to handle?
In regard to these discoveries, I think the issue should be closed because there is nothing that cannot be handled, and a conversation about a possible new "unfinished" state for those who find it interesting should be initiated.
I hope my comments could help to clarify the confusions that I read in this conversation.
morooshka commentedon Jan 16, 2025
Also faced this. I see there one confusion - we fail the task on execution timeout but not fail on DAG timeout. To my mind the behaviour should be similar - fail the task
P.S. In which case one should need to skip the task if the whole DAG times out?
AlvaroCavalcante commentedon Feb 11, 2025
I was facing this issue, but I was able to call the on_failure_callback when I set this callback in the DAG definition, as stated here: https://stackoverflow.com/questions/69312630/airflow-triggering-the-on-failure-callback-when-the-dagrun-timeout-is-exceed
github-actions commentedon Feb 11, 2026
This issue has been automatically marked as stale because it has been open for 365 days without any activity. There has been several Airflow releases since last activity on this issue. Kindly asking to recheck the report against latest Airflow version and let us know if the issue is reproducible. The issue will be closed in next 30 days if no further activity occurs from the issue author.
morooshka commentedon Feb 11, 2026
@AlvaroCavalcante I would prefer to have the default
on_failure_callbackcalled here without additional efforts. Wouldn't it be more clear here?