Make HttpSensor defer when response_check is provided

[yuseok89]

closes: #40209

HttpSensor(deferrable=True) with a response_check silently fell back to the synchronous poke loop, occupying a worker slot for the whole wait. It now defers like any other deferrable sensor: the triggerer waits for the endpoint to respond without error, and the check runs on the worker. If the check returns False, the task defers again while preserving the original sensor timeout and poke_interval pacing.

The callable stays on the worker. The trigger serializes and returns the HTTP response only when a response_check is configured; otherwise, it preserves the existing lightweight success event. This follows the same general pattern that S3KeySensor uses for check_fn.

Verified with a live Dag

An HttpSensor(deferrable=True, poke_interval=20) polling this Airflow's own api-server, with a response_check that only passes two minutes after the run was triggered.

Before the change the task never left running, since a response_check forced it onto the synchronous path. After the change it enters deferred after the initial worker-side poke, wakes every 20 seconds to evaluate the check on the worker, and succeeds once the check passes:

---

Was generative AI tooling used to co-author this PR?

• Yes (please specify the tool below)
  • Opus 4.8

---

• Read the Pull Request Guidelines for more information. Note: commit author/co-author name and email in commits become permanently public when merged.
• For fundamental code changes, an Airflow Improvement Proposal (AIP) is needed.
• When adding dependency, check compliance with the ASF 3rd Party License Policy.
• For significant user-facing changes create newsfragment: {pr_number}.significant.rst, in airflow-core/newsfragments. You can add this file in a follow-up commit after the PR is created so you know the PR number.

[jroachgolf84]

Seems like the self.response_check behvaior is being dropped? Is that an issue?

[yuseok89]

As I understand it, this condition causes the behavior reported in #40209 because setting response_check forces the sensor onto the synchronous path. The check itself is still evaluated by self.poke(context) before deferral and by execute_complete() after the trigger returns a response. A false result defers the task again.

[jroachgolf84]

Is this a pattern that is being used elsewhere (initial_delay, that is)?

[yuseok89]

I could not find another trigger using the same initial_delay parameter. I added it because a failed response_check creates a new trigger, which would otherwise send its first request immediately instead of respecting poke_interval.
Since this introduces a new trigger parameter, I would be happy to continue discussing in this PR whether keeping it local to HttpSensorTrigger is appropriate or whether a different design would be preferable.

[potiuk]

Hello @yuseok89 - thank you for your contributions to Apache Airflow!

The Airflow community has introduced a limit of 5 open pull requests at a time for contributors without write access to the repository. You currently have 11 open pull requests, so - as a one-time step of introducing the limit - we closed the ones where maintainers have not engaged yet:

• Add deferred task resume flow to task lifecycle diagram #65323 - Add deferred task resume flow to task lifecycle diagram
• Add TaskGroup-level retries to retry a whole group as a unit #68139 - Add TaskGroup-level retries to retry a whole group as a unit
• Dedent task docstrings used as doc_md #71108 - Dedent task docstrings used as doc_md
• Fix task duration increasing forever after Dag run timeout skips it #71549 - Fix task duration increasing forever after Dag run timeout skips it
• Ignore task start_date and end_date on manual runs #71758 - Ignore task start_date and end_date on manual runs
• Make HttpSensor defer when response_check is provided #71922 - Make HttpSensor defer when response_check is provided
• Flag ui-field-behaviour drift between hooks and provider.yaml in static checks #72089 - Flag ui-field-behaviour drift between hooks and provider.yaml in static checks
• Skip GCS folder-marker keys in GCSToSFTPOperator #72638 - Skip GCS folder-marker keys in GCSToSFTPOperator
• i18n(ko): Add missing translations (Sep25) #73677 - i18n(ko): Add missing translations (Sep25)

These pull requests stay open because maintainers are already engaged in them - they count towards your limit:

• Show recent task states on the Dags list page #69619 - Show task state counts for the latest Dag run on the Dags list page
• UI: Add a time-window scope to the Dags list run state filter #70470 - UI: Add a time-window scope to the Dags list run state filter

This is not a judgement of you or of your changes. We never told contributors before that opening many pull requests at once was a problem, so there is nothing to feel bad about - and nothing is lost: your branches, commits and the review history stay where they are.

What we ask you to do is to make your first prioritization decision: choose which of the pull requests above matter most to you, and reopen them (up to 5 open at a time, including the ones still open) with the "Reopen pull request" button or gh pr reopen <PR_NUMBER> --repo apache/airflow. Reopen the ones you are ready to follow through - keep them rebased, respond to review comments and fix failing checks.

While your pull requests are waiting for review, the most valuable thing you can do is help in other ways - reviewing other contributors' pull requests, helping with issues, and taking part in the discussions on the devlist and Slack.

Why we introduced the limit, what it means for you and how to reopen or restore a pull request is explained in https://github.com/apache/airflow/blob/main/contributing-docs/32_open_pull_request_limit.rst.

---

Drafted-by: Claude Code (Opus 5); reviewed by @potiuk before posting