Task is taking more than 5 minutes to fail

Apache Airflow version

main (development)

If "Other Airflow 2 version" selected, which one?

No response

What happened?

When there are 2 assets with same name and there is task which has these 2 assets as outlet then task is taking more than 5 minutes to fail.

What you think should happen instead?

No response

How to reproduce

Run below dag on main latest and notice 'task1' is taking more than 5 minutes to fail

from airflow.providers.standard.operators.empty import EmptyOperator
from airflow.providers.standard.operators.python import PythonOperator
from airflow.sdk import Asset
from airflow.sdk import DAG
from airflow.utils.trigger_rule import TriggerRule
from pendulum import today

from dags.plugins.airflow_dag_introspection import assert_the_task_states

left = Asset(name='asset_name')
right = Asset(name='asset_name', uri="asset_uri")

with DAG(
    dag_id="asset_with_duplicate_name",
    start_date=today('UTC').add(days=-2),
    schedule=None,
    max_active_runs=1,
    tags=["asset", "AIP-74"],
) as dag:
    task1 = EmptyOperator(task_id='task1', outlets=[left, right])
    status = PythonOperator(
        task_id="assert_task_status",
        python_callable=assert_the_task_states,
        op_args=[{"task1": "failed"}],
        trigger_rule=TriggerRule.ALL_DONE,
    )
    task1 >> status

Operating System

Linux

Versions of Apache Airflow Providers

No response

Deployment

Other

Deployment details

No response

Anything else?

No response

Are you willing to submit PR?

Yes I am willing to submit a PR!

Code of Conduct

I agree to follow this project's Code of Conduct

@Lee-W

@Lee-W this is happening with a Dag which is trying to use 2 assets with the same name.
I am expecting task to fail fast.

It fails immediately on my end. Is there another setup I can try?

another potential bug found, let's see whether that fixes this issue

Due to the latest task sdk change, this issue occurs in all tasks instead of just the empty operator. The checking logic is totally removed somehow. New root cause found. A total rework needed

We should fix this in TaskRunner by handling the exceptions https://github.com/apache/airflow/blob/main/task-sdk/src/airflow/sdk/execution_time/task_runner.py#L939

After a few more tests, I discovered that my statement was not accurate. The API server as a whole doesn't die. Only the API endpoint itself dies without being correctly handled. It's still the root cause of why the task is hanging. But what we need to do is not handle

airflow/task-sdk/src/airflow/sdk/execution_time/task_runner.py

Line 939 in 807ba69

finally:

but

airflow/airflow-core/src/airflow/api_fastapi/execution_api/routes/task_instances.py

Line 318 in 22d7ba8

def ti_update_state(

.

The task fails to update it state correctly and hangs there till it fails

Navigation Menu

Repository navigation

Task is taking more than 5 minutes to fail #50654

Apache Airflow version

If "Other Airflow 2 version" selected, which one?

What happened?

What you think should happen instead?

How to reproduce

Operating System

Versions of Apache Airflow Providers

Deployment

Deployment details

Anything else?

Are you willing to submit PR?

Code of Conduct

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Task is taking more than 5 minutes to fail #50654

Description

Apache Airflow version

If "Other Airflow 2 version" selected, which one?

What happened?

What you think should happen instead?

How to reproduce

Operating System

Versions of Apache Airflow Providers

Deployment

Deployment details

Anything else?

Are you willing to submit PR?

Code of Conduct

Activity

atuls4441 commented on May 16, 2025

Lee-W commented on May 19, 2025

Lee-W commented on May 19, 2025

Lee-W commented on May 23, 2025

phanikumv commented on Jun 2, 2025

Lee-W commented on Jun 2, 2025

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions