Repository navigation
Validate GCP storage transfer job body after template rendering - #70529
Merged
Merged
Conversation
mitre88
requested review from
amoghrajesh,
ashb,
bugraoz93,
gopidesupavan,
jason810496,
jscheffl,
potiuk and
shahar1
as code owners
July 27, 2026 15:31
boring-cyborg
Bot
added
area:dev-tools
area:providers
backport-to-v3-3-test
provider:google
potiuk
added
the
ready for maintainer review
1 task done
potiuk
approved these changes
Jul 30, 2026
potiuk
left a comment
potiuk
left a comment
Member
There was a problem hiding this comment.
Thanks — correct move. body is in template_fields, so _validate_inputs() in __init__ was running TransferJobValidator against the un-rendered value; a templated body could never pass validation and a bad one broke Dag parsing instead of failing the task. Deferring both the deepcopy and the validation to execute fixes that, and removing the exemptions entry closes it out.
Good that the test asserts create_transfer_job.assert_not_called() — that shows it bails before touching the API.
One coverage gap inline.
Drafted-by: Claude Code (Opus 5); reviewed by @potiuk before posting
75 of 78 tasks
Contributor
|
This pull request has been automatically marked as stale because it has not had recent activity. It will be closed in 5 days if no further activity occurs. Thank you for your contributions. |
github-actions
Bot
added
the
stale
shahar1
removed
the
stale
The operator's body is a template field, but it was deep-copied and validated in __init__, so a fully templated body skipped validation entirely — the checks ran against the Jinja expression instead of the rendered dict, silently bypassing the AWS-credential restriction. Part of the burn-down tracked in apache#70296.
1 task
Copying the body into a local keeps the AWS access key and secret that TransferJobPreprocessor injects off the live task object, and leaves the operator's own body pristine so a retry validates the value the user passed rather than the credential-laden one from the previous attempt. The regression test now drives real Jinja rendering through a Dag with render_template_as_native_obj, so it proves validation sees the rendered mapping instead of standing in for rendering with a direct assignment. The Dag-level flag also keeps the test working on the Airflow versions in the provider compatibility matrix, where the operator-level override does not exist yet.
shahar1
force-pushed
the
fix-storage-transfer-create-job
branch
from
September 23, 2026 06:56
eb56680 to
1c52845
Compare
shahar1
approved these changes
Sep 23, 2026
shahar1
changed the title
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
5 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Part of the template-field validation burn-down tracked in #70296.
CloudDataTransferServiceCreateJobOperatorlistsbodyintemplate_fieldsbut deep-copies and validates it in__init__. With a fully templatedbody(a Jinja expression or XComArg), the validator ran against the un-rendered expression, soTransferJobValidator's checks — the AWS-credential restriction and the single-data-source rule — were silently bypassed. The deep copy and validation now run at the start ofexecute(), against the rendered value, right beforeTransferJobPreprocessormutates the body.Added a test constructing the operator with a templated
bodythat renders to a body embedding AWS credentials — with the previous implementation the credential check never fires and the job is created; now it raises before calling the hook. The class is removed from the exemption list and thevalidate-operators-initcheck passes locally.Per the discussion in #70505 this is a genuine value read (validation of the rendered dict), not an argument-provision check.
Was generative AI tooling used to co-author this PR?
Generated-by: Claude Code (Fable 5) following the guidelines