[amoghrajesh]

---

Was generative AI tooling used to co-author this PR?

• Yes (please specify the tool below)

Summary

Clearing a task instance now discards its task_state_store entries, so the next attempt starts over instead of resuming from a checkpoint or reconnecting to an external job recorded by the attempt that was cleared.

Clearing a task instance now discards its task_state_store entries, so the next attempt starts
over. Retries are unaffected and still resume.

Why

A retry and a clear were treated identically. Both kept the task's state, both resumed. But they mean different things.

A retry happens because infrastructure failed. Nothing about the work changed, so carrying on from the checkpoint is exactly right, and that is what crash recovery is for. A clear happens because a human intervened, and the usual reason a human clears a task is that something did change: the code, the upstream data, a connection, a config value.

So the behaviour was tuned for the case where nothing changed, and then applied to the case where
something had.

What that cost in practice:

Fixing a bug and clearing left the bug's output in place. A task gets through files 1 to 6, hits bad data on the 7th, you fix the transform and clear. It resumed at 7. Files 1 to 6 still held output from the code you had just fixed, now silently mixed with the corrected work. The task went green, and nothing anywhere said otherwise.

Clearing a succeeded durable task did nothing at all. The operator read back the stored job id, saw the external job had already finished, and returned the stored result in a couple of seconds. This needs no unusual configuration, and there is no reading of "clear" under which doing nothing is what the user asked for.

The two failure modes are asymmetric, which is what decides the default rather than just moving the
problem elsewhere. Discarding when you wanted to resume costs repeated work you can watch happen.
Resuming when you wanted a fresh start produces wrong output you cannot see.

What changed

• keep_task_state on ClearTaskInstancesBody, defaulting to false
• The discard runs in post_clear_task_instances, inside the existing if not dry_run: block and
  after clear_task_instances succeeds, so a preview discards nothing and a failed clear cannot
  take the task state with it
• _clear_task_state_store_on_success refactored into a shared discard_task_state_store helper:
  two callers, two gates, one implementation
• A "Keep task state and resume" checkbox in both clear dialogs, unchecked by default, next to the
  existing "Prevent rerun if task is running"
• Concept docs rewritten. The section previously asserted the opposite, under the heading "Clearing
  a task is treated the same as a retry"

When to tick the box

Two situations, both documented.

Nothing changed and you only want the task to carry on. Retries normally cover this, so you reach it when retries are exhausted.

An external job is still running. Most operators cancel theirs in on_kill, so clearing a running task leaves nothing to reconnect to. But clearing a failed task never runs on_kill, so a job that outlived its worker is still going, and discarding the stored id submits a second one. Same for operators configured to leave the job alive, such as KubernetesPodOperator with on_kill_action="keep_pod".

Compatibility

This changes behaviour introduced in 3.3 and released in 3.3.1. Anyone relying on clear-to-resume needs keep_task_state=true.

A single default rather than per-operator behaviour is acceptable precisely because the user keeps an override: if Airflow decided silently per operator, a wrong guess would be unrecoverable, whereas a wrong default is one checkbox.

Tests

Trying to run a dag like this:

from __future__ import annotations

import logging
from datetime import datetime

from airflow.sdk import DAG, NEVER_EXPIRE, Variable, task

log = logging.getLogger("airflow.task")

ROWS = ["100", "200", "N/A", "400", "500"]


with DAG(
    dag_id="clear_after_fix",
    schedule=None,
    start_date=datetime(2026, 1, 1),
    catchup=False,
    tags=["task-state-store"],
    doc_md=__doc__,
):

    @task(retries=0)
    def transform_amounts(**context):
        store = context["task_state_store"]
        handle_missing = Variable.get("handle_missing", default="false") == "true"

        done = set(store.get("done", default=[]))
        if done:
            log.info("Resuming, %d of %d rows already done", len(done), len(ROWS))
        else:
            log.info("Starting fresh, %d rows to transform", len(ROWS))

        total = 0
        for row in ROWS:
            if row in done:
                log.info("  %-5s skipped, already done", row)
                continue

            if row == "N/A" and not handle_missing:
                raise ValueError(f"cannot transform {row!r}: fix the transform and clear this task")

            total += 0 if row == "N/A" else int(row)
            done.add(row)
            store.set("done", sorted(done), retention=NEVER_EXPIRE)
            log.info("  %-5s transformed", row)

        log.info("Done. %d rows, total %d", len(done), total)
        return total

    transform_amounts()

This dag showcases some data transformation and mimics a case where bad data came in as N/A controlled by a variable value.

So showing it:

First run:

Task state store contains this:

Set the variable rightly now:

airflow variables set handle_missing true

Clear with the default:

Next run:

State store:

Now if I cleared by overriding the checkbox for another try (ie: keeping task state):

Observe the logs:

More testing

Tried same test as above from clear from TI page as bulk

Clearing with airflowctl:

amoghrajesh.desai  …/airflow   task-state-clear-on-clear $!?    orbstack  ♥ 12:28  airflowctl tasks clear --help
Usage: airflowctl tasks clear [-h] [--api-token API_TOKEN] [--dag-run-id DAG_RUN_ID] [--dry-run | --no-dry-run] [--end-date END_DATE] [-e ENV]
                              [--include-downstream | --no-include-downstream] [--include-future | --no-include-future] [--include-past | --no-include-past]
                              [--include-upstream | --no-include-upstream] [--keep-task-state | --no-keep-task-state] [--note NOTE] [--only-failed | --no-only-failed]
                              [--only-running | --no-only-running] [--prevent-running-task | --no-prevent-running-task] [--reset-dag-runs | --no-reset-dag-runs]
                              [--run-on-latest-version | --no-run-on-latest-version] [--start-date START_DATE] [--task-ids TASK_IDS] [--output (table, json, yaml, plain)]
                              dag_id

Clear task instances of a Dag by its ID

Positional Arguments:
  dag_id                dag_id for clear operation in TasksOperations

Options:
  -h, --help            show this help message and exit
  --api-token API_TOKEN
                        The token to use for authentication
  --dag-run-id DAG_RUN_ID
                        dag_run_id for clear_task_instances operation
  --dry-run, --no-dry-run
                        dry_run for clear_task_instances operation
  --end-date END_DATE   end_date for clear_task_instances operation
  -e, --env ENV         The environment to run the command in
  --include-downstream, --no-include-downstream
                        include_downstream for clear_task_instances operation
  --include-future, --no-include-future
                        include_future for clear_task_instances operation
  --include-past, --no-include-past
                        include_past for clear_task_instances operation
  --include-upstream, --no-include-upstream
                        include_upstream for clear_task_instances operation
  --keep-task-state, --no-keep-task-state
                        keep_task_state for clear_task_instances operation
  --note NOTE           note for clear_task_instances operation
  --only-failed, --no-only-failed
                        only_failed for clear_task_instances operation
  --only-running, --no-only-running
                        only_running for clear_task_instances operation
  --prevent-running-task, --no-prevent-running-task
                        prevent_running_task for clear_task_instances operation
  --reset-dag-runs, --no-reset-dag-runs
                        reset_dag_runs for clear_task_instances operation
  --run-on-latest-version, --no-run-on-latest-version
                        run_on_latest_version for clear_task_instances operation
  --start-date START_DATE
                        start_date for clear_task_instances operation
  --task-ids TASK_IDS   task_ids for clear_task_instances operation
  --output, -o (table, json, yaml, plain)
                        Output format. Allowed values: json, yaml, plain, table (default: json)

Ran this:

[Breeze:3.11.15] root@3c493b69b8c3:/opt/airflow$ airflowctl dags tasks clear_after_fix --run-id manual__2026-09-09T07:30:58.621788+00:00
Please enter password for encrypted keyring:
Dag: clear_after_fix
Dag runs to clear: 1
  - manual__2026-09-09T07:30:58.621788+00:00 (logical date: 2026-09-09T07:30:57+00:00, partition
date: -)
Clear task instances for these Dag runs? [y/N] y
Cleared 1 task instance(s) across 1 Dag run(s)

---

• Read the Pull Request Guidelines for more information. Note: commit author/co-author name and email in commits become permanently public when merged.
• For fundamental code changes, an Airflow Improvement Proposal (AIP) is needed.
• When adding dependency, check compliance with the ASF 3rd Party License Policy.
• For significant user-facing changes create newsfragment: {pr_number}.significant.rst, in airflow-core/newsfragments. You can add this file in a follow-up commit after the PR is created so you know the PR number.

[bbovenzi]

UI code lgtm

[amoghrajesh]

@kaxil would you mind taking another look at this?

[amoghrajesh]

@kaxil can you take a look at this, trying to make this for 3.4

[github-actions]

Backport failed to create: airflow-ctl/v0-1-test. View the failure log Run details

Note: As of Merging PRs targeted for Airflow 3.X
the committer who merges the PR is responsible for backporting the PRs that are bug fixes (generally speaking) to the maintenance branches.

In matter of doubt please ask in #release-management Slack channel.

Status	Branch	Result
❌	airflow-ctl/v0-1-test

You can attempt to backport this manually by running:

cherry_picker 5e785ac airflow-ctl/v0-1-test

This should apply the commit to the airflow-ctl/v0-1-test branch and leave the commit in conflict state marking
the files that need manual conflict resolution.

After you have resolved the conflicts, you can continue the backport process by running:

cherry_picker --continue

If you don't have cherry-picker installed, see the installation guide.