Skip to content

Surface the retry policy decision on task instances page - #73030

Merged
amoghrajesh merged 15 commits into
apache:mainfrom
astronomer:retry-policy-ui-improvements
Sep 29, 2026
Merged

amoghrajesh merged 15 commits into
apache:mainfrom
astronomer:retry-policy-ui-improvements

Conversation

@amoghrajesh

@amoghrajesh amoghrajesh commented Sep 12, 2026 •

Copy link
Copy Markdown
Contributor

Was generative AI tooling used to co-author this PR?
  • Yes (please specify the tool below)

What

A retry policy can already classify why a task failed (auth error, rate limit, and so on), and #73027 makes sure that reason is actually saved to the database on the failure path, not just the retry path. But that reason still isn't visible anywhere - not through the API, not in the UI. This PR closes that gap: it exposes retry_reason through the public REST API and renders it on the Task Instance page, so a Dag author looking at a failed task can finally see a plain-English reason instead of just "FAILED" with no explanation. Stacked stacked on #73027.

Current behaviour

retry_reason is a real column on both task_instance and task_instance_history (written by #73027), but neither the public REST API's TaskInstanceResponse/TaskInstanceHistoryResponse models nor the Task Instance UI expose it. The data sits in the database with no way to see it.

Proposed change

Backend (core API):

  • Added retry_reason: str | None = None to both TaskInstanceResponse and TaskInstanceHistoryResponse (airflow-core/src/airflow/api_fastapi/core_api/datamodels/). Both models needed the field - the Task Instance page's per try data comes from the try-details endpoint, which returns TaskInstanceHistoryResponse backed by either the live row or an archived history row.
  • No route changes needed: both GET .../taskInstances/{task_id} and GET .../tries/{try_number} already return the ORM object directly, so FastAPI/Pydantic auto-maps the existing retry_reason column.
  • Regenerated the OpenAPI spec, the UI's generated TS client, and airflow-ctl's generated datamodels via their respective hooks (never hand-edited).

Frontend (Details.tsx, the Task Instance details page):

  • A "Reason for state" row directly under the State row in the details table, showing the reason for whichever try is currently selected in the Task Tries strip.
  • A colored alert banner above the Task Tries strip, summarizing the latest try's reason at a glance: red for a stopped/failed outcome, orange for an exhausted-retries outcome. Reuses the repo's existing Alert system-component (the same one WarningAlert/ErrorAlert use), so it's visually consistent with how errors are already surfaced elsewhere in the UI.
  • One new English translation key (taskInstance.retryReason: "Reason for state"); other locales fall back to English until translated separately, since the i18n validity check doesn't enforce cross-locale key parity.

Testing

Ran the same example_llm_retry_policy dag.

Task that fails:

image image

Task in retry state:
image

And when it moves to failed:

image
  • Read the Pull Request Guidelines for more information. Note: commit author/co-author name and email in commits become permanently public when merged.
  • For fundamental code changes, an Airflow Improvement Proposal (AIP) is needed.
  • When adding dependency, check compliance with the ASF 3rd Party License Policy.
  • For significant user-facing changes create newsfragment: {pr_number}.significant.rst, in airflow-core/newsfragments. You can add this file in a follow-up commit after the PR is created so you know the PR number.

Comment thread airflow-core/src/airflow/ui/src/pages/TaskInstance/Details.tsx Outdated
Comment thread airflow-core/src/airflow/ui/src/pages/TaskInstance/Details.tsx Outdated
Comment thread airflow-core/src/airflow/ui/src/pages/TaskInstance/Details.tsx Outdated
Comment thread airflow-core/src/airflow/api_fastapi/core_api/datamodels/task_instances.py Outdated
Comment thread task-sdk/src/airflow/sdk/execution_time/task_runner.py Outdated
Comment thread task-sdk/tests/task_sdk/execution_time/test_task_runner.py Outdated
Comment thread airflow-core/src/airflow/api_fastapi/execution_api/routes/task_instances.py Outdated
@amoghrajesh
amoghrajesh requested a review from kaxil September 21, 2026 08:55
Comment thread task-sdk/src/airflow/sdk/execution_time/schema/schema.json
Comment thread airflow-core/src/airflow/ui/src/pages/TaskInstance/Details.tsx Outdated
Comment thread task-sdk/src/airflow/sdk/execution_time/task_runner.py Outdated
Comment thread task-sdk/tests/task_sdk/execution_time/schema/test_migrator.py Outdated
Comment thread providers/common/ai/docs/retry_policies.rst Outdated
Comment thread airflow-core/src/airflow/ui/src/pages/TaskInstance/Details.test.tsx Outdated
Comment thread task-sdk/tests/task_sdk/execution_time/test_task_runner.py Outdated
Comment thread airflow-core/src/airflow/api_fastapi/core_api/datamodels/task_instances.py Outdated
Comment thread airflow-core/src/airflow/ui/src/pages/TaskInstance/Details.tsx Outdated
Comment thread airflow-core/src/airflow/ui/src/pages/TaskInstance/Details.test.tsx Outdated
@amoghrajesh
amoghrajesh requested a review from kaxil September 24, 2026 11:58
@amoghrajesh

Copy link
Copy Markdown
Contributor Author

@kaxil @bbovenzi I think I have handled all your comments, wdyt now?

Comment thread airflow-core/src/airflow/ui/src/pages/TaskInstance/Details.tsx Outdated
Comment thread airflow-core/src/airflow/api_fastapi/core_api/datamodels/task_instances.py Outdated
Comment thread airflow-core/src/airflow/api_fastapi/core_api/datamodels/task_instances.py Outdated
Comment thread airflow-core/tests/unit/api_fastapi/core_api/routes/public/test_task_instances.py Outdated
Comment thread airflow-core/src/airflow/ui/src/pages/TaskInstance/Details.test.tsx Outdated
Comment thread providers/common/ai/docs/retry_policies.rst Outdated
@amoghrajesh

Copy link
Copy Markdown
Contributor Author

@bbovenzi @kaxil handled your comments in the last commit: d9e1026

I have also updated the PR description with the latest screenshots.

@bbovenzi bbovenzi left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM UI-wise. I may play with the placement of the state reason more once I use this more.

@kaxil

kaxil commented Sep 25, 2026

Copy link
Copy Markdown
Member

Approving. The masking now happens in the worker, where the secret is registered, and the items from the earlier rounds are all in. Nothing below needs to hold the merge.

  • schedule_tis bumps try_number without clearing retry_reason (dagrun.py:2302-2308), and the only reset is in ti_run. When the next try fails before it starts, which the stuck-in-queued limit (scheduler_job_runner.py:3357) and an executor-reported failure on a queued TI (:1772) both do, the row keeps the previous try's reason and the Header banner shows "Stopped on try 3 of 4" over try 2's text. Could the staleness follow-up cover schedule_tis as well as clear_task_instances? Adding retry_reason=None to its UPDATE is enough, since the retry branch archives the earlier try with its reason before that runs.
  • The comment on the state_reason validator went stale with this push, because _evaluate_retry_policy now redacts every reason. What the validator still guards is a reason written by an older task-sdk, which is worth saying so nobody later removes it as redundant. Same in task_instance_history.py.
  • from dataclasses import replace in _evaluate_retry_policy can move to the top of the file; dataclasses is stdlib with no cycle.
  • The as Record<string, ...> cast in stateReason.ts lets a misspelt state compile and then never display. Partial<Record<TaskInstanceState, ...>> would turn that into a compile error.

@amoghrajesh amoghrajesh added this to the Airflow 3.4.0 milestone Sep 28, 2026
@amoghrajesh

Copy link
Copy Markdown
Contributor Author

Thanks, folks. Did the last 3.

  1. Validator comment now says what it actually guards: the worker redacts; this only catches rows from an older task-sdk. Same note on the history copy.
  2. Moved import to top.
  3. stateReason.ts is now Partial<Record<TaskInstanceState, ...>> with the cast gone. A misspelt state is a compile error - checked it: error TS2561: ... 'up_for_retryy' does not exist.
    That last one turned up something worth knowing: the root tsconfig.json is {"files": [], "references": [...]}, so tsc -p tsconfig.json checks nothing at all. Even a blatant type error passes. The real check is tsc -p tsconfig.app.json, which is what pnpm lint runs. I'd been using the wrong one. Re-ran the right one across everything here: clean.

On schedule_tis: good catch, and worse than the cases we already knew about, since the counts and the reason come from different attempts rather than just being stale. Taking it into the clearing follow up alongside clear_task_instances.

@amoghrajesh

Copy link
Copy Markdown
Contributor Author

Yea the failing checks aren't related. Merging.

@amoghrajesh
amoghrajesh merged commit 8b3d4d3 into apache:main Sep 29, 2026
123 of 124 checks passed
@amoghrajesh
amoghrajesh deleted the retry-policy-ui-improvements branch September 29, 2026 08:14
@github-actions

Copy link
Copy Markdown
Contributor

Backport failed to create: airflow-ctl/v0-1-test. View the failure log Run details

Note: As of Merging PRs targeted for Airflow 3.X
the committer who merges the PR is responsible for backporting the PRs that are bug fixes (generally speaking) to the maintenance branches.

In matter of doubt please ask in #release-management Slack channel.

Status Branch Result
❌ airflow-ctl/v0-1-test Commit Link

You can attempt to backport this manually by running:

cherry_picker 8b3d4d3 airflow-ctl/v0-1-test

This should apply the commit to the airflow-ctl/v0-1-test branch and leave the commit in conflict state marking
the files that need manual conflict resolution.

After you have resolved the conflicts, you can continue the backport process by running:

cherry_picker --continue

If you don't have cherry-picker installed, see the installation guide.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants