Repository navigation
Fix Dataproc triggers swallowing task cancellation - #74192
Merged
potiuk merged 1 commit intoOct 5, 2026
Merged
Conversation
The except asyncio.CancelledError handlers in DataprocSubmitTrigger.run(), DataprocSubmitJobDirectTrigger.run() and DataprocClusterTrigger.run() performed cleanup but never re-raised, so triggerer migration (sibling steals trigger rows, original cancels its coroutines) looked like a crash: Trigger exited without sending an event, failing all dependent deferred tasks. Add the missing bare raise, mirroring the BigQuery fix in apache#63730, so cancellation propagates and cleanup_finished_triggers takes the transparent-migration path. Update the cancellation tests to require CancelledError instead of also accepting StopAsyncIteration.
|
Congratulations on your first Pull Request and welcome to the Apache Airflow community! If you have any issues or are unsure about any anything please check our Contributors' Guide
|
potiuk
approved these changes
Oct 5, 2026
potiuk
left a comment
Member
There was a problem hiding this comment.
Approving. Re-raising CancelledError after cleanup lets the triggerer tell a reassigned trigger from a user cancel, while still emitting the final DELETING event first on older Airflow. The tightened tests now really cancel mid-await. Thanks!
Drafted-by: Claude Code (Opus 5.5); reviewed by @potiuk before posting
|
Awesome work, congrats on your first merged pull request! You are invited to check our Issue Tracker for additional contributions. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
The triggerer tells migration, user action and shutdown apart only by
asyncio.CancelledErrorpropagating out oftrigger.run(). Theexcept asyncio.CancelledErrorhandlers inDataprocSubmitTrigger.run(),DataprocSubmitJobDirectTrigger.run()andDataprocClusterTrigger.run()performed cleanup (cancel the job/cluster whensafe_to_cancel()) but never re-raised, so a transient triggerer stall followed by trigger migration producedTrigger exited without sending an event. Dependent tasks will be failed.for every hosted deferred task instead of a transparent failover.This is the same bug shape fixed for
BigQueryInsertJobTriggerin #63730: add the missing bareraiseafter cleanup socleanup_finished_triggers()takes the expected cancelled path.DataprocBatchTrigger,DataprocDeleteClusterTriggerandDataprocOperationTriggerare unaffected (noCancelledErrorhandler, andexcept Exceptiondoes not catchCancelledError).The cancellation tests now require
CancelledErrorinstead of also acceptingStopAsyncIteration, and the cluster not-safe test actually injects cancellation (it previously completed before cancelling).closes: #74087
Was generative AI tooling used to co-author this PR?
Generated-by: Muse Spark following the guidelines