Add TaskGroup-level retries to retry a whole group as a unit

[yuseok89]

related: #21867

Implements the minimal v1 scope for TaskGroup retries I proposed on the dev list, a first step toward #21867. A TaskGroup can now be retried as a unit, the way SubDag once could. The richer options from the issue and thread are left for follow-up PRs.

What's in this v1

@task_group(retries=2)        # or: with TaskGroup("g", retries=2): ...
def collect_and_publish():
    publish(collect())

• When any task in the group fails (after its own task-level retries), the whole group is cleared and re-run, up to retries times; once exhausted the run fails as usual. Default retries=0 keeps today's behavior.
• Tasks outside the group are untouched (no upstream_failed spill-over) and running siblings are restarted. Nested groups retry innermost-first with independent counters.

Group retries run at the top of DagRun.update_state, before scheduling decisions, so a failure is cleared before it can propagate downstream. Covered by SDK, serialization, and scheduler integration tests.

Out of v1 (follow-ups)
retry_condition (any/last/custom), retry_strategy (all / only-failed), group-level retry_delay, mapped-group retries.

Demo

The example DAG has a collect_and_publish group with retries=2, and publish is intentionally made to fail on its first two attempts and succeed on the third, so the group is retried twice before it goes green. In the run below:

• collect and publish both run 3 times. A group retry re-runs every task in the group, not just the one that failed.
• start and finish, which sit outside the group, each run once.

Screen.Recording.2026-06-06.at.11.47.22.PM.mov

---

Was generative AI tooling used to co-author this PR?

• Yes (please specify the tool below)
  • Opus 4.8

---

• Read the Pull Request Guidelines for more information. Note: commit author/co-author name and email in commits become permanently public when merged.
• For fundamental code changes, an Airflow Improvement Proposal (AIP) is needed.
• When adding dependency, check compliance with the ASF 3rd Party License Policy.
• For significant user-facing changes create newsfragment: {pr_number}.significant.rst, in airflow-core/newsfragments. You can add this file in a follow-up commit after the PR is created so you know the PR number.

---

Important

🛠️ Maintainer triage note for @yuseok89 · by @potiuk · 2026-07-18 16:36 UTC

I've removed the ready for maintainer review label — the next step here is yours:

• ❌ Merge conflicts with main. See docs.

It'll return to the maintainer queue automatically once resolved — no need to re-add the label by hand. See the Pull Request quality criteria for details.

The ball is in your court — you've been assigned. Rebase onto main, push, then mark it Ready for review.

_{Automated triage — may be imperfect; a maintainer takes the next look.}

[potiuk]

@yuseok89 — I've removed the ready for maintainer review label because the next step here is yours: this branch now has merge conflicts with main. It'll go back into the maintainer queue automatically once you rebase onto the latest main and push — no need to re-add the label by hand. There's no rush.

_{Automated triage note drafted by an AI-assisted tool — may get things wrong; a real Apache Airflow maintainer takes the next look once it's green. (why automated)}

---

Drafted-by: Claude Code (Opus 4.8); reviewed by @potiuk before posting

[potiuk]

Hello @yuseok89 - thank you for your contributions to Apache Airflow!

The Airflow community has introduced a limit of 5 open pull requests at a time for contributors without write access to the repository. You currently have 11 open pull requests, so - as a one-time step of introducing the limit - we closed the ones where maintainers have not engaged yet:

• Add deferred task resume flow to task lifecycle diagram #65323 - Add deferred task resume flow to task lifecycle diagram
• Add TaskGroup-level retries to retry a whole group as a unit #68139 - Add TaskGroup-level retries to retry a whole group as a unit
• Dedent task docstrings used as doc_md #71108 - Dedent task docstrings used as doc_md
• Fix task duration increasing forever after Dag run timeout skips it #71549 - Fix task duration increasing forever after Dag run timeout skips it
• Ignore task start_date and end_date on manual runs #71758 - Ignore task start_date and end_date on manual runs
• Make HttpSensor defer when response_check is provided #71922 - Make HttpSensor defer when response_check is provided
• Flag ui-field-behaviour drift between hooks and provider.yaml in static checks #72089 - Flag ui-field-behaviour drift between hooks and provider.yaml in static checks
• Skip GCS folder-marker keys in GCSToSFTPOperator #72638 - Skip GCS folder-marker keys in GCSToSFTPOperator
• i18n(ko): Add missing translations (Sep25) #73677 - i18n(ko): Add missing translations (Sep25)

These pull requests stay open because maintainers are already engaged in them - they count towards your limit:

• Show recent task states on the Dags list page #69619 - Show task state counts for the latest Dag run on the Dags list page
• UI: Add a time-window scope to the Dags list run state filter #70470 - UI: Add a time-window scope to the Dags list run state filter

This is not a judgement of you or of your changes. We never told contributors before that opening many pull requests at once was a problem, so there is nothing to feel bad about - and nothing is lost: your branches, commits and the review history stay where they are.

What we ask you to do is to make your first prioritization decision: choose which of the pull requests above matter most to you, and reopen them (up to 5 open at a time, including the ones still open) with the "Reopen pull request" button or gh pr reopen <PR_NUMBER> --repo apache/airflow. Reopen the ones you are ready to follow through - keep them rebased, respond to review comments and fix failing checks.

While your pull requests are waiting for review, the most valuable thing you can do is help in other ways - reviewing other contributors' pull requests, helping with issues, and taking part in the discussions on the devlist and Slack.

Why we introduced the limit, what it means for you and how to reopen or restore a pull request is explained in https://github.com/apache/airflow/blob/main/contributing-docs/32_open_pull_request_limit.rst.

---

Drafted-by: Claude Code (Opus 5); reviewed by @potiuk before posting