Fail deferred sensors on non-timeout errors when soft_fail is set

[rjgoyln]

Summary

While poking, a sensor with soft_fail=True skips only on a timeout or AirflowFailException and fails on anything else, so a misconfigured sensor is visible as a failure. In deferrable mode it skipped on every error it saw when resuming, a crashed trigger included, leaving a sensor that never worked indistinguishable from one that legitimately gave up. soft_fail now covers the same exceptions in both paths; never_fail is unchanged.

Narrowing the handler also picks up AirflowTaskTimeout, which derives from BaseException and so escaped the previous except AirflowException entirely — a deferred sensor that hit its execution_timeout failed even with soft_fail set.

Behavior change

A deferrable sensor with soft_fail=True that raises a plain AirflowException from execute_complete now fails instead of skipping. Sensors that raise AirflowSkipException themselves, such as ExternalTaskSensor and TimeDeltaSensor, are unaffected.

closes: #71255

---

Was generative AI tooling used to co-author this PR?

• Yes — Claude Code (Opus 5)

Generated-by: Claude Code (Opus 5) following the guidelines

[raghavan-arvind]

+1, would love to see this merged

[potiuk]

Hello @rjgoyln - thank you for your contributions to Apache Airflow!

The Airflow community has introduced a limit of 5 open pull requests at a time for contributors without write access to the repository. You currently have 24 open pull requests, so - as a one-time step of introducing the limit - we closed the ones where maintainers have not engaged yet:

• Support OpenTelemetry auto-instrumentation and vendor distros #70758 - Support OpenTelemetry auto-instrumentation and vendor distros
• Honor OTEL_PROPAGATORS when propagating trace context across Airflow #70840 - Honor OTEL_PROPAGATORS when propagating trace context across Airflow
• Fix API client errors losing detail behind a reverse proxy #70936 - Fix API client errors losing detail behind a reverse proxy
• Fix airflowctl config lint filters matching partial config names #70940 - Fix airflowctl config lint filters matching partial config names
• Stop failing reschedule-mode sensors on a stale executor success #71194 - Stop failing reschedule-mode sensors on a stale executor success
• Drop the cncf.kubernetes provider dependency from airflow-core tests #71868 - Drop the cncf.kubernetes provider dependency from airflow-core tests
• Drop the git provider dependency from airflow-core tests #72041 - Drop the git provider dependency from airflow-core tests
• Fail deferred sensors on non-timeout errors when soft_fail is set #72053 - Fail deferred sensors on non-timeout errors when soft_fail is set
• Drop the celery provider dependency from airflow-core tests #72249 - Drop the celery provider dependency from airflow-core tests
• Honor config_file in deferrable Kubernetes tasks #72301 - Honor config_file in deferrable Kubernetes tasks
• Fix Dataproc job not cancelled when killing a deferred task #72591 - Fix Dataproc job not cancelled when killing a deferred task
• Stop the scheduler deadlocking with the triggerer on deferred-task timeouts #72595 - Stop the scheduler deadlocking with the triggerer on deferred-task timeouts
• Make the scheduler's HITL and orphan sweeps retry transient DB errors #72596 - Make the scheduler's HITL and orphan sweeps retry transient DB errors
• Keep the triggerer running when a trigger constructor raises #72758 - Keep the triggerer running when a trigger constructor raises
• Stop failing scheduled tasks when a serialized Dag is briefly missing #72814 - Stop failing scheduled tasks when a serialized Dag is briefly missing
• Stop fetching the run conf once per task instance in the scheduler #73022 - Stop fetching the run conf once per task instance in the scheduler
• Catch format-string arguments passed to exception constructors #73076 - Catch format-string arguments passed to exception constructors
• Fail deferrable KPO when a GC'd pod takes its XCom with it #73131 - Fail deferrable KPO when a GC'd pod takes its XCom with it
• Block new EM102 f-string exception messages in prek #73166 - Block new EM102 f-string exception messages in prek
• Fail the local yml mounts prek hook when synchronization fails #73170 - Fail the local yml mounts prek hook when synchronization fails
• Fix provider.yaml doc URL check with several suspended providers #73204 - Fix provider.yaml doc URL check with several suspended providers

These pull requests stay open because maintainers are already engaged in them - they count towards your limit:

• Batch upstream map-length lookups when resolving mapped kwargs #70620 - Batch upstream map-length lookups when resolving mapped kwargs
• Re-anchor custom queued deadline references when a Dag run is cleared #71850 - Re-anchor custom queued deadline references when a Dag run is cleared
• Make the default task failure email say what failed #73221 - Make the default task failure email say what failed

This is not a judgement of you or of your changes. We never told contributors before that opening many pull requests at once was a problem, so there is nothing to feel bad about - and nothing is lost: your branches, commits and the review history stay where they are.

What we ask you to do is to make your first prioritization decision: choose which of the pull requests above matter most to you, and reopen them (up to 5 open at a time, including the ones still open) with the "Reopen pull request" button or gh pr reopen <PR_NUMBER> --repo apache/airflow. Reopen the ones you are ready to follow through - keep them rebased, respond to review comments and fix failing checks.

While your pull requests are waiting for review, the most valuable thing you can do is help in other ways - reviewing other contributors' pull requests, helping with issues, and taking part in the discussions on the devlist and Slack.

Why we introduced the limit, what it means for you and how to reopen or restore a pull request is explained in https://github.com/apache/airflow/blob/main/contributing-docs/32_open_pull_request_limit.rst.

---

Drafted-by: Claude Code (Opus 5); reviewed by @potiuk before posting