Make the scheduler's HITL and orphan sweeps retry transient DB errors

[rjgoyln]

Summary

check_awaiting_input_timeouts and adopt_or_reset_orphaned_tasks both run inside run_with_db_retries, so a deadlock should cost a scheduler tick rather than the scheduler. Neither loop survives one.

A failed flush deactivates the session: Session._flush rolls its subtransaction back on any exception, so a plain StaleDataError poisons it as surely as a DBAPI error. The retry's first statement then raises PendingRollbackError, which the predicate does not match, so it escapes the loop — and the awaiting-input sweep sits on a timer without non_fatal=True, taking the scheduler loop with it.

retry_db_transaction rolls back for exactly this reason; neither hand-rolled loop picked it up.

Change

• Roll back before re-raising in check_awaiting_input_timeouts, as retry_db_transaction does.
• Widen the orphaned-task sweep's guard to (DBAPIError, StaleDataError).

Both guards now name exactly the tuple run_with_db_retries retries. That widens no retries — tenacity already retried every DBAPIError in the orphaned-task sweep; those retries just died on the next statement.

The sweep body is unchanged apart from the indent under the new try:; check_trigger_timeouts is the third loop of this shape, and #72595 covers it.

Tests

Both tests drive a session that deactivates on a failed flush the way SQLAlchemy does; all four cases fail on main with PendingRollbackError.

---

Was generative AI tooling used to co-author this PR?

• Yes — Claude Code (Opus 5)

Generated-by: Claude Code (Opus 5) following the guidelines

[potiuk]

Hello @rjgoyln - thank you for your contributions to Apache Airflow!

The Airflow community has introduced a limit of 5 open pull requests at a time for contributors without write access to the repository. You currently have 24 open pull requests, so - as a one-time step of introducing the limit - we closed the ones where maintainers have not engaged yet:

• Support OpenTelemetry auto-instrumentation and vendor distros #70758 - Support OpenTelemetry auto-instrumentation and vendor distros
• Honor OTEL_PROPAGATORS when propagating trace context across Airflow #70840 - Honor OTEL_PROPAGATORS when propagating trace context across Airflow
• Fix API client errors losing detail behind a reverse proxy #70936 - Fix API client errors losing detail behind a reverse proxy
• Fix airflowctl config lint filters matching partial config names #70940 - Fix airflowctl config lint filters matching partial config names
• Stop failing reschedule-mode sensors on a stale executor success #71194 - Stop failing reschedule-mode sensors on a stale executor success
• Drop the cncf.kubernetes provider dependency from airflow-core tests #71868 - Drop the cncf.kubernetes provider dependency from airflow-core tests
• Drop the git provider dependency from airflow-core tests #72041 - Drop the git provider dependency from airflow-core tests
• Fail deferred sensors on non-timeout errors when soft_fail is set #72053 - Fail deferred sensors on non-timeout errors when soft_fail is set
• Drop the celery provider dependency from airflow-core tests #72249 - Drop the celery provider dependency from airflow-core tests
• Honor config_file in deferrable Kubernetes tasks #72301 - Honor config_file in deferrable Kubernetes tasks
• Fix Dataproc job not cancelled when killing a deferred task #72591 - Fix Dataproc job not cancelled when killing a deferred task
• Stop the scheduler deadlocking with the triggerer on deferred-task timeouts #72595 - Stop the scheduler deadlocking with the triggerer on deferred-task timeouts
• Make the scheduler's HITL and orphan sweeps retry transient DB errors #72596 - Make the scheduler's HITL and orphan sweeps retry transient DB errors
• Keep the triggerer running when a trigger constructor raises #72758 - Keep the triggerer running when a trigger constructor raises
• Stop failing scheduled tasks when a serialized Dag is briefly missing #72814 - Stop failing scheduled tasks when a serialized Dag is briefly missing
• Stop fetching the run conf once per task instance in the scheduler #73022 - Stop fetching the run conf once per task instance in the scheduler
• Catch format-string arguments passed to exception constructors #73076 - Catch format-string arguments passed to exception constructors
• Fail deferrable KPO when a GC'd pod takes its XCom with it #73131 - Fail deferrable KPO when a GC'd pod takes its XCom with it
• Block new EM102 f-string exception messages in prek #73166 - Block new EM102 f-string exception messages in prek
• Fail the local yml mounts prek hook when synchronization fails #73170 - Fail the local yml mounts prek hook when synchronization fails
• Fix provider.yaml doc URL check with several suspended providers #73204 - Fix provider.yaml doc URL check with several suspended providers

These pull requests stay open because maintainers are already engaged in them - they count towards your limit:

• Batch upstream map-length lookups when resolving mapped kwargs #70620 - Batch upstream map-length lookups when resolving mapped kwargs
• Re-anchor custom queued deadline references when a Dag run is cleared #71850 - Re-anchor custom queued deadline references when a Dag run is cleared
• Make the default task failure email say what failed #73221 - Make the default task failure email say what failed

This is not a judgement of you or of your changes. We never told contributors before that opening many pull requests at once was a problem, so there is nothing to feel bad about - and nothing is lost: your branches, commits and the review history stay where they are.

What we ask you to do is to make your first prioritization decision: choose which of the pull requests above matter most to you, and reopen them (up to 5 open at a time, including the ones still open) with the "Reopen pull request" button or gh pr reopen <PR_NUMBER> --repo apache/airflow. Reopen the ones you are ready to follow through - keep them rebased, respond to review comments and fix failing checks.

While your pull requests are waiting for review, the most valuable thing you can do is help in other ways - reviewing other contributors' pull requests, helping with issues, and taking part in the discussions on the devlist and Slack.

Why we introduced the limit, what it means for you and how to reopen or restore a pull request is explained in https://github.com/apache/airflow/blob/main/contributing-docs/32_open_pull_request_limit.rst.

---

Drafted-by: Claude Code (Opus 5); reviewed by @potiuk before posting