Add trigger `queue` support for `AsyncCallback` and `BaseEventTrigger` by zach-overflow · Pull Request #71346 · apache/airflow

zach-overflow · 2026-08-09T15:03:51Z

What?

Adds trigger queue support for the BaseEventTrigger and the AsyncCallback categories of triggers.
Allows users to control the triggerer host assignment for all 3 categories of triggers (task-initiated, event-based, and callback-initiated).

Why?

The trigger queues feature was first introduced in Airflow 3.2.0, but only for task-based triggers. This was because at the time, the BaseEventTrigger and AsyncCallback patterns were in the middle of development.
Trigger queues provide triggerer host assignment controls distinct from, and complementary to, the multi-team triggerer assignment. Since the multi-team assignment supports all three types of triggers, trigger queues should too.

Was generative AI tooling used to co-author this PR?

Yes (please specify the tool below)

Generated-by: Claude Code following the guidelines
Scope of AI use:
- I used Claude for tracing and modifying the trigger record creation pathways for event-based triggers / AsyncCallback triggers after I added the initial queue assignment to the base classes.
- Documentation was hand-written
- tests were 50% AI written, followed by manual cleanup.

uranusjr

With nits

zach-overflow · 2026-08-11T13:36:09Z

@uranusjr Thanks for the review! I believe I've addressed everything, but please let me know if you have any other feedback.

uranusjr

What do we do with the argument collision? As this stands, the queue cannot actually be set by the user, can it?

zach-overflow · 2026-08-13T23:02:46Z

What do we do with the argument collision? As this stands, the queue cannot actually be set by the user, can it?

@uranusjr Fair point, I think I was a bit overzealous when I saw that the MessageQueueTrigger already had the queue parameter marked as deprecated. After some adjustments, I think I found a decent middle ground until whenever that deprecated parameter is fully removed: For the MessageQueueTrigger the queue parameter's meaning remains the same as before, while it's set to a private instance attribute instead (_queue) so we there's no collision with the BaseTrigger's queue.

Users can then assign specifically the MessageQueueTrigger class to a "trigger queue" with the trigger_queue parameter. A little ugly, but I'm ok with it. I'm also ok with fully replacing the meaning of the queue value in that class so everything is consistent. No strong feelings either way on my end.

zach-overflow · 2026-08-18T18:19:45Z

cc @uranusjr is this good to merge? Should I get any other reviewers eyes on this? Please let me know if there's anyone in particular I might want to tag here if so, and I'll be happy to do so 👍

hussein-awala

LGTM once the newsfragment is removed

…t__()

MessageQueueTrigger's deprecated `queue` constructor param (a broker URI) was stored under self.queue, silently colliding with the new triggerer-routing `queue` attribute added to BaseTrigger/BaseEventTrigger. Any Dag still using that deprecated call style would have its asset-watcher Trigger row's queue column set to the broker URI, and since every triggerer started without --queues filters on queue IS NULL, the trigger would never be picked up again, silently.

PR #41 renamed MessageQueueTrigger's deprecated broker-URI storage from self.queue to self.queue_uri to avoid colliding with the new BaseEventTrigger.queue triggerer-routing attribute, but missed this provider-local test asserting on the old attribute name.

…gger.queue Provider distributions must keep passing tests against previously released Airflow versions. BaseEventTrigger.queue is a new attribute added by this branch, so any Airflow release before it ships doesn't have the attribute at all, and BaseTrigger has no class-level default either in those older releases. Assertions on trigger.queue therefore raised AttributeError under the 3.0.6/3.1.8/3.3.0 compat test jobs. Guard with getattr(trigger, "queue", None), matching the same pattern already used for this attribute in airflow.serialization.encoders.

The `queue` constructor keyword is already claimed by the deprecated broker queue URI, so there was no way for a user to route a MessageQueueTrigger to a specific triggerer queue via BaseEventTrigger.queue. Add a `triggerer_queue` parameter that is forwarded to BaseEventTrigger.__init__ instead.

BaseEventTrigger.queue is unreleased (targets 3.4.0), so the Compat test matrix against older published airflow-core wheels fails with AttributeError when asserting on it directly.

…mpat tests" This reverts commit 1993585.

…ion, and fix trigger-queue terminology The previous fix relied on BaseEventTrigger.__init__ to set the queue attribute, which older published airflow-core releases silently drop, breaking the Compat test matrix. Storing the value under our own name and exposing it through a `queue` property/setter works regardless of the installed core version, so no version-gated tests are needed. Also corrects "triggerer queue" to "trigger queue" throughout, which is the term used elsewhere for this concept.

apache#71346) * Add trigger queue support for `BaseEventTrigger` * Enable trigger queue support for async callbacks * Add newsfragment * Address classattr bug * Address feedback * remove unintentional uv.lock metadata addition * Adjust queue value to accomodate subclasses not calling super().__init__() * Fix MessageQueueTrigger queue collision with trigger queue assignment MessageQueueTrigger's deprecated `queue` constructor param (a broker URI) was stored under self.queue, silently colliding with the new triggerer-routing `queue` attribute added to BaseTrigger/BaseEventTrigger. Any Dag still using that deprecated call style would have its asset-watcher Trigger row's queue column set to the broker URI, and since every triggerer started without --queues filters on queue IS NULL, the trigger would never be picked up again, silently. * Fix IBM MQ test asserting on renamed MessageQueueTrigger attribute PR apache#41 renamed MessageQueueTrigger's deprecated broker-URI storage from self.queue to self.queue_uri to avoid colliding with the new BaseEventTrigger.queue triggerer-routing attribute, but missed this provider-local test asserting on the old attribute name. * Fix compat-test failures from asserting on unreleased MessageQueueTrigger.queue Provider distributions must keep passing tests against previously released Airflow versions. BaseEventTrigger.queue is a new attribute added by this branch, so any Airflow release before it ships doesn't have the attribute at all, and BaseTrigger has no class-level default either in those older releases. Assertions on trigger.queue therefore raised AttributeError under the 3.0.6/3.1.8/3.3.0 compat test jobs. Guard with getattr(trigger, "queue", None), matching the same pattern already used for this attribute in airflow.serialization.encoders. * Let MessageQueueTrigger set the triggerer queue via a distinct parameter The `queue` constructor keyword is already claimed by the deprecated broker queue URI, so there was no way for a user to route a MessageQueueTrigger to a specific triggerer queue via BaseEventTrigger.queue. Add a `triggerer_queue` parameter that is forwarded to BaseEventTrigger.__init__ instead. * Skip triggerer-queue assertions on pre-3.4 Airflow-core in compat tests BaseEventTrigger.queue is unreleased (targets 3.4.0), so the Compat test matrix against older published airflow-core wheels fails with AttributeError when asserting on it directly. * Revert "Skip triggerer-queue assertions on pre-3.4 Airflow-core in compat tests" This reverts commit 1993585. * Let MessageQueueTrigger's trigger queue work on any Airflow-core version, and fix trigger-queue terminology The previous fix relied on BaseEventTrigger.__init__ to set the queue attribute, which older published airflow-core releases silently drop, breaking the Compat test matrix. Storing the value under our own name and exposing it through a `queue` property/setter works regardless of the installed core version, so no version-gated tests are needed. Also corrects "triggerer queue" to "trigger queue" throughout, which is the term used elsewhere for this concept. * Undo `uv.lock` extras shift from local `uv run` call. * remove news fragment --------- Co-authored-by: Xu Han <xu.han@datadoghq.com>

boring-cyborg Bot added area:dag-processor area:deadline-alerts AIP-86 (former AIP-57) area:task-sdk area:Triggerer kind:documentation labels Aug 9, 2026

zach-overflow commented Aug 9, 2026

View reviewed changes

Comment thread airflow-core/docs/authoring-and-scheduling/deferring.rst

Comment thread airflow-core/docs/howto/deadline-alerts.rst

zach-overflow force-pushed the zo/global-trigger-q-support branch from 1ca1494 to bb21aa1 Compare August 9, 2026 15:27

zach-overflow marked this pull request as ready for review August 9, 2026 15:27

zach-overflow requested review from XD-DENG, amoghrajesh, ashb, bolkedebruin, ephraimbuddy, jedcunningham and kaxil as code owners August 9, 2026 15:27

zach-overflow changed the title ~~feat: Add trigger queue support for AsyncCallback and BaseEventTrigger~~ Add trigger queue support for AsyncCallback and BaseEventTrigger Aug 9, 2026

uranusjr reviewed Aug 9, 2026

View reviewed changes

Comment thread task-sdk/src/airflow/sdk/definitions/callback.py

uranusjr reviewed Aug 9, 2026

View reviewed changes

Comment thread airflow-core/src/airflow/models/trigger.py Outdated

uranusjr approved these changes Aug 9, 2026

View reviewed changes

zach-overflow force-pushed the zo/global-trigger-q-support branch from d7f6479 to f297fc0 Compare August 10, 2026 13:07

hanxdatadog mentioned this pull request Aug 10, 2026

Fix MessageQueueTrigger queue collision with trigger queue assignment DataDog/airflow#41

Merged

1 task done

zach-overflow requested a review from vincbeck as a code owner August 10, 2026 18:25

zach-overflow force-pushed the zo/global-trigger-q-support branch from b169e5a to 24bde08 Compare August 10, 2026 19:22

zach-overflow requested a review from dabla as a code owner August 10, 2026 19:22

zach-overflow force-pushed the zo/global-trigger-q-support branch from 549f90a to a52b817 Compare August 11, 2026 12:07

zach-overflow force-pushed the zo/global-trigger-q-support branch from a52b817 to 3a3c2a7 Compare August 12, 2026 12:39

uranusjr approved these changes Aug 12, 2026

View reviewed changes

hussein-awala approved these changes Aug 19, 2026

View reviewed changes

Comment thread airflow-core/newsfragments/71346.feature.rst Outdated

hussein-awala added this to the Airflow 3.4.0 milestone Aug 19, 2026

hussein-awala added the type:new-feature Changelog: New Features label Aug 19, 2026

zach-overflow and others added 16 commits August 19, 2026 14:08

Add trigger queue support for BaseEventTrigger

f0e9a72

Enable trigger queue support for async callbacks

d617b45

Add newsfragment

2595e0a

Address classattr bug

f8c6fc0

Address feedback

f6b8844

remove unintentional uv.lock metadata addition

5f72575

Adjust queue value to accomodate subclasses not calling super().__ini…

7e49985

…t__()

Skip triggerer-queue assertions on pre-3.4 Airflow-core in compat tests

4a6ff20

BaseEventTrigger.queue is unreleased (targets 3.4.0), so the Compat test matrix against older published airflow-core wheels fails with AttributeError when asserting on it directly.

Revert "Skip triggerer-queue assertions on pre-3.4 Airflow-core in co…

72dfb2e

…mpat tests" This reverts commit 1993585.

Undo uv.lock extras shift from local uv run call.

f8a98e3

remove news fragment

122a949

zach-overflow force-pushed the zo/global-trigger-q-support branch from f8e6b94 to 122a949 Compare August 19, 2026 18:41

hussein-awala merged commit e606e1b into apache:main Aug 20, 2026
108 checks passed

zach-overflow deleted the zo/global-trigger-q-support branch August 26, 2026 17:06

vincbeck mentioned this pull request Sep 10, 2026

Status of testing Providers that were prepared on September 09, 2026 #72902

Closed

dabla mentioned this pull request Sep 25, 2026

Mark the IBM MQ and IBM Db2 providers as ready for release #73708

Merged

1 task done

Navigation Menu

Repository navigation

Add trigger `queue` support for `AsyncCallback` and `BaseEventTrigger` - #71346

Add trigger `queue` support for `AsyncCallback` and `BaseEventTrigger`#71346
hussein-awala merged 16 commits into
apache:mainfrom
DataDog:zo/global-trigger-q-support

zach-overflow commented Aug 9, 2026 •

edited

Loading

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

uranusjr left a comment

Uh oh!

zach-overflow commented Aug 11, 2026

Uh oh!

uranusjr left a comment

Uh oh!

zach-overflow commented Aug 13, 2026

Uh oh!

zach-overflow commented Aug 18, 2026

Uh oh!

hussein-awala left a comment

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

Conversation

zach-overflow commented Aug 9, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What?

Why?

Was generative AI tooling used to co-author this PR?

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

uranusjr left a comment

Choose a reason for hiding this comment

Uh oh!

zach-overflow commented Aug 11, 2026

Uh oh!

uranusjr left a comment

Choose a reason for hiding this comment

Uh oh!

zach-overflow commented Aug 13, 2026

Uh oh!

zach-overflow commented Aug 18, 2026

Uh oh!

hussein-awala left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

zach-overflow commented Aug 9, 2026 •

edited

Loading