Skip to content

Fix Dataproc triggers swallowing task cancellation - #74192

Merged
potiuk merged 1 commit into
apache:mainfrom
atharvaajmera:fix-dataproc-trigger-cancelled-error-74087
Oct 5, 2026
Merged

potiuk merged 1 commit into
apache:mainfrom
atharvaajmera:fix-dataproc-trigger-cancelled-error-74087

Conversation

@atharvaajmera

Copy link
Copy Markdown
Contributor

Summary

The triggerer tells migration, user action and shutdown apart only by asyncio.CancelledError propagating out of trigger.run(). The except asyncio.CancelledError handlers in DataprocSubmitTrigger.run(), DataprocSubmitJobDirectTrigger.run() and DataprocClusterTrigger.run() performed cleanup (cancel the job/cluster when safe_to_cancel()) but never re-raised, so a transient triggerer stall followed by trigger migration produced Trigger exited without sending an event. Dependent tasks will be failed. for every hosted deferred task instead of a transparent failover.

This is the same bug shape fixed for BigQueryInsertJobTrigger in #63730: add the missing bare raise after cleanup so cleanup_finished_triggers() takes the expected cancelled path. DataprocBatchTrigger, DataprocDeleteClusterTrigger and DataprocOperationTrigger are unaffected (no CancelledError handler, and except Exception does not catch CancelledError).

The cancellation tests now require CancelledError instead of also accepting StopAsyncIteration, and the cluster not-safe test actually injects cancellation (it previously completed before cancelling).

closes: #74087


Was generative AI tooling used to co-author this PR?
  • Yes — Muse Spark

Generated-by: Muse Spark following the guidelines

The except asyncio.CancelledError handlers in DataprocSubmitTrigger.run(), DataprocSubmitJobDirectTrigger.run() and DataprocClusterTrigger.run() performed cleanup but never re-raised, so triggerer migration (sibling steals trigger rows, original cancels its coroutines) looked like a crash: Trigger exited without sending an event, failing all dependent deferred tasks. Add the missing bare raise, mirroring the BigQuery fix in apache#63730, so cancellation propagates and cleanup_finished_triggers takes the transparent-migration path. Update the cancellation tests to require CancelledError instead of also accepting StopAsyncIteration.
@atharvaajmera
atharvaajmera requested a review from shahar1 as a code owner October 4, 2026 15:58
@boring-cyborg boring-cyborg Bot added area:providers provider:google Google (including GCP) related issues labels Oct 4, 2026
@boring-cyborg

boring-cyborg Bot commented Oct 4, 2026

Copy link
Copy Markdown

Congratulations on your first Pull Request and welcome to the Apache Airflow community! If you have any issues or are unsure about any anything please check our Contributors' Guide
Here are some useful points:

  • Pay attention to the quality of your code (ruff, mypy and type annotations). Our prek-hooks will help you with that.
  • In case of a new feature add useful documentation (in docstrings or in docs/ directory). Adding a new operator? Check this short guide Consider adding an example Dag that shows how users should use it.
  • Consider using Breeze environment for testing locally, it's a heavy docker but it ships with a working Airflow and a lot of integrations.
  • Be patient and persistent. It might take some time to get a review or get the final approval from Committers.
  • Please follow ASF Code of Conduct for all communication including (but not limited to) comments on Pull Requests, Mailing list and Slack.
  • Be sure to read the Airflow Coding style.
  • Always keep your Pull Requests rebased, otherwise your build might fail due to changes not related to your commits.
    Apache Airflow is a community-driven project and together we are making it better 🚀.
    In case of doubts contact the developers at:
    Mailing List: dev@airflow.apache.org
    Slack: https://s.apache.org/airflow-slack

@potiuk potiuk left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Approving. Re-raising CancelledError after cleanup lets the triggerer tell a reassigned trigger from a user cancel, while still emitting the final DELETING event first on older Airflow. The tightened tests now really cancel mid-await. Thanks!


Drafted-by: Claude Code (Opus 5.5); reviewed by @potiuk before posting

@potiuk
potiuk merged commit d339bb2 into apache:main Oct 5, 2026
87 checks passed
@boring-cyborg

boring-cyborg Bot commented Oct 5, 2026

Copy link
Copy Markdown

Awesome work, congrats on your first merged pull request! You are invited to check our Issue Tracker for additional contributions.

@atharvaajmera
atharvaajmera deleted the fix-dataproc-trigger-cancelled-error-74087 branch October 5, 2026 14:34
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area:providers provider:google Google (including GCP) related issues

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Deferrable Dataproc triggers swallow CancelledError during triggerer migration, mass-failing deferred tasks (same as #63730 for BigQuery)

2 participants