Skip to content

Statistics not updated after failure #52254

Closed
Closed
Statistics not updated after failure#52254
Labels
affected_version:3.0Issues Reported for 3.0area:corearea:metricskind:bugThis is a clearly a bug
Milestone
Airflow 3.1.8

Description

@zlokomatic
zlokomatic
opened on Jun 25, 2025

Apache Airflow version

3.0.2

If "Other Airflow 2 version" selected, which one?

No response

What happened?

After a task failed the values for ti_failures are not increased.

As we are using statsd/prometheus to monitor airflow_ti_failures for task failures we are not alarmed about this.

What you think should happen instead?

A failing task should increase ti_failures

How to reproduce

Environment

    AIRFLOW__METRICS__STATSD_ON: 'true'
    AIRFLOW__METRICS__STATSD_HOST: 'statsd_exporter'
    AIRFLOW__METRICS__STATSD_PORT: '9125'
    AIRFLOW__METRICS__STATSD_DATADOG_ENABLED: 'true'

Activate statsd server

Dag:

from airflow import DAG
from airflow.operators.bash import BashOperator
from datetime import datetime, timedelta

with DAG(
    dag_id="failure_handling_dag",
    start_date=datetime(2025, 4, 1),
    schedule="@daily",
    catchup=False,
) as dag:

    fail_task = BashOperator(
        task_id="fail_task",
        bash_command="sleep 5 && exit 1",
    )

Operating System

Ubuntu 22.04.5 LTS

Versions of Apache Airflow Providers

Default with apache-airflow 3.0.2 docker image

Deployment

Docker-Compose

Deployment details

No response

Anything else?

No response

Are you willing to submit PR?

  • Yes I am willing to submit a PR!

Code of Conduct

Activity

  1. zlokomatic
    added
    kind:bugThis is a clearly a bug
    area:core
    needs-triagelabel for new issues that we didn't triage yet
    on Jun 25, 2025

  2. boring-cyborg commented on Jun 25, 2025

    @boring-cyborg
    boring-cyborgbot
    on Jun 25, 2025 – with boring-cyborg

    Thanks for opening your first issue here! Be sure to follow the issue template! If you are willing to raise PR to address this issue please do so, no need to wait for approval.

  3. dosubot
    added
    area:metrics
    on Jun 25, 2025

  4. kempspo commented on Jul 6, 2025

    @kempspo
    kempspo
    on Jul 6, 2025

    Can confirm also seeing the same thing, I think #52336 also has the same issue but for a different metric.

  5. potiuk
    added
    affected_version:3.0Issues Reported for 3.0
    and removed
    needs-triagelabel for new issues that we didn't triage yet
    on Jul 18, 2025

  6. aaron-y-chen commented on Jul 28, 2025

    @aaron-y-chen
    aaron-y-chen
    on Jul 28, 2025
    Contributor

    Yes, I believe this is the same issue. I tested on Airflow 2.11.0 and both ti_successes and ti_failures metrics are emitted, but they no longer appear in 3.0.0 or later.

  7. potiuk
    added this to the Airflow 3.0.4 milestone on Jul 29, 2025

  8. larryyu1285 commented on Jul 31, 2025

    @larryyu1285
    larryyu1285
    on Jul 31, 2025

    I use version 3.0.3, both ti_failures and ti_successes are not emitted.

  9. larryyu1285 commented on Aug 4, 2025

    @larryyu1285
    larryyu1285
    on Aug 4, 2025

    The metrics below are from my statsd-exporter logs after I triggered a DAG that later failed. As the logs show, only task_instance_created was emitted.

    I use airflow version 3.0.3

    2025-08-04 11:45:45 time=2025-08-04T03:45:45.059Z level=DEBUG source=listener.go:96 msg="Incoming line" proto=udp line=airflow.serde.load_serializers:8.303375|ms
2025-08-04 11:45:47 time=2025-08-04T03:45:47.928Z level=DEBUG source=listener.go:96 msg="Incoming line" proto=udp line=airflow.serde.load_serializers:2.348084|ms
2025-08-04 11:46:03 time=2025-08-04T03:46:03.859Z level=DEBUG source=listener.go:96 msg="Incoming line" proto=udp line=airflow.serde.load_serializers:2.921875|ms
2025-08-04 11:46:03 time=2025-08-04T03:46:03.879Z level=DEBUG source=listener.go:96 msg="Incoming line" proto=udp line=airflow.serde.load_serializers:3.580958|ms
2025-08-04 11:46:48 time=2025-08-04T03:46:48.204Z level=DEBUG source=listener.go:96 msg="Incoming line" proto=udp line=airflow.serde.load_serializers:6.642375|ms
2025-08-04 11:46:48 time=2025-08-04T03:46:48.209Z level=DEBUG source=listener.go:96 msg="Incoming line" proto=udp line=airflow.serde.load_serializers:7.632917|ms
2025-08-04 11:46:52 time=2025-08-04T03:46:52.237Z level=DEBUG source=listener.go:96 msg="Incoming line" proto=udp line=airflow.serde.load_serializers:39.680208|ms
2025-08-04 11:47:04 time=2025-08-04T03:47:04.355Z level=DEBUG source=listener.go:96 msg="Incoming line" proto=udp line=airflow.serde.load_serializers:4.704292|ms
2025-08-04 11:47:05 time=2025-08-04T03:47:05.901Z level=DEBUG source=listener.go:96 msg="Incoming line" proto=udp line=airflow.task_instance_created_BashOperator:1|c
2025-08-04 11:47:05 time=2025-08-04T03:47:05.903Z level=DEBUG source=listener.go:96 msg="Incoming line" proto=udp line=airflow.task_instance_created:1|c
2025-08-04 11:47:05 time=2025-08-04T03:47:05.903Z level=DEBUG source=listener.go:96 msg="Incoming line" proto=udp line=airflow.task_instance_created_HttpOperator:1|c
2025-08-04 11:47:05 time=2025-08-04T03:47:05.903Z level=DEBUG source=listener.go:96 msg="Incoming line" proto=udp line=airflow.task_instance_created:1|c
2025-08-04 11:47:08 time=2025-08-04T03:47:08.776Z level=DEBUG source=listener.go:96 msg="Incoming line" proto=udp line=airflow.serde.load_serializers:195.017208|ms
2025-08-04 11:47:14 time=2025-08-04T03:47:14.907Z level=DEBUG source=listener.go:96 msg="Incoming line" proto=udp line=airflow.serde.load_serializers:2.884000|ms

    Also, I noticed that sometimes even task_instance_created metrics were not emitted.

  10. kempspo commented on Aug 4, 2025

    @kempspo
    kempspo
    on Aug 4, 2025

    I was looking through this, I think it's related to this PR, particularly the task_instance.py changes and similar issue to this one too.

    Almost all the stats that used to get emitted in task_instance.py were removed or not migrated properly.

    I'm happy to try and open a PR for this, but probably need some guidance.

  11. ashb
    modified the milestones: Airflow 3.0.4, Airflow 3.0.5 on Aug 4, 2025

  12. 12 remaining items

  13. kaxil
    modified the milestones: Airflow 3.1.2, Airflow 3.1.3 on Oct 31, 2025
  14. ephraimbuddy
    modified the milestones: Airflow 3.1.3, Airflow 3.1.4 on Nov 10, 2025

  15. antonlin1 commented on Dec 4, 2025

    @antonlin1
    antonlin1
    on Dec 4, 2025
    Contributor

    specifically for the missing task metrics I put up this PR. Could I have review on that one please?

    The missing failure metrics are not being addressed in this PR which is handling the awaited #45511.

  16. ephraimbuddy
    modified the milestones: Airflow 3.1.4, Airflow 3.1.5 on Dec 8, 2025
  17. potiuk
    modified the milestones: Airflow 3.1.5, Airflow 3.1.6 on Dec 14, 2025
  18. ephraimbuddy
    modified the milestones: Airflow 3.1.6, Airflow 3.1.7 on Jan 6, 2026
  19. ephraimbuddy
    modified the milestones: Airflow 3.1.7, Airflow 3.1.8 on Jan 28, 2026

  20. amoghrajesh commented on Feb 12, 2026

    @amoghrajesh
    amoghrajesh
    on Feb 12, 2026
    Contributor

    ti_failures were handled by: #59731 and ti_successes by #55322, closing this issue with that intent, let us know if you think it still has something missing

  21. amoghrajesh
    closed this as completedon Feb 12, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    affected_version:3.0Issues Reported for 3.0area:corearea:metricskind:bugThis is a clearly a bug

    Type

    No type

    Projects

    No projects

      Milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions