Skip to content

Better error handling for running ti heartbeats after task is cleared #53140

Description

@amoghrajesh

Apache Airflow version

main (development)

If "Other Airflow 2 version" selected, which one?

No response

What happened?

When a running TI is cleared, it is immediately removed from the ti table and moved to TIH. This causes subsequent heartbeats from the running TI to receive a 404 (Not Found) from the API server.

As a result, the client (worker/task) logs an error like: "TaskInstance not found", while it could be running.

What you think should happen instead?

Instead of a 404, the API should handle this more gracefully by checking TIH when a TI is not found in TI.

How to reproduce

  1. Start a task
  2. Clear it while running
  3. Observe 404 errors from heartbeat API calls and the "TaskInstance not found" message.

Operating System

MacOS

Versions of Apache Airflow Providers

No response

Deployment

Official Apache Airflow Helm Chart

Deployment details

No response

Anything else?

No response

Are you willing to submit PR?

  • Yes I am willing to submit a PR!

Code of Conduct

Activity

  1. self-assigned this
    on Jul 10, 2025
  2. added this to the Airflow 3.0.4 milestone on Jul 10, 2025
  3. amoghrajesh commented on Jul 30, 2025

    @amoghrajesh
    ContributorAuthor

    I will pick this issue up today/tomorrow.

  4. moved this from Next 2 Weeks to Todo in AIP-72 - Task Execution Interface and SDKon Aug 8, 2025
  5. removed this from the Airflow 3.0.6 milestone on Aug 22, 2025
  6. 11 remaining items

  7. added a commit that references this issue on Feb 8, 2026
    9a654e4
  8. added a commit that references this issue on Mar 11, 2026
    dbdf55f
  9. added a commit that references this issue on Apr 3, 2026
    48a0fc2
  10. added a commit that references this issue on Apr 3, 2026
    ce1270b
  11. added a commit that references this issue on Apr 8, 2026
    1d4a552
  12. added 5 commits that reference this issue on Apr 8, 2026
    64fe17d
    c31b22f
    e061fd2
    a4ee379
    1b96819
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Type

    No type

    Projects

    No projects

      Milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions