Repository navigation
Return files destination uris in all ToGCS operators #11323
Description
Activity
Thanks for opening your first issue here! Be sure to follow the issue template!
This issue spawn from #10991 (comment)
It probably can be useful but it's also a breaking change and so far I didn't see users complaining about it.
Could you please clarify the TODO part on this issue?
What operators need adjustments? Is it localized to GCP or also other providers?Yeah I don’t see how this could break user code in practice. Should be doable.
Yeah. Nice feature to add. Would you like to implement it @FHoffmannCode ? Should I assign it to you ?
Yeah I don’t see how this could break user code in practice.
The suggestion was to change the returned value to make operators consistent. If changed then the value pushed to xcom is not as users are expecting.
It depends on what are correctly being returned; I’m assuming most (if not all) are currently returning None, which won’t have backward compatibility issues (except if someone is depending on an XCom is not pushed, but why would anyone do that). Are there instances of an operator currently pushing something else?
Yeah. Same here. I think it's no harm to return "more".
This issue has been automatically marked as stale because it has been open for 30 days with no response from the author. It will be closed in next 7 days if no further activity occurs from the issue author.
- addedstaleStale PRs per the .github/workflows/stale.yml policy fileStale PRs per the .github/workflows/stale.yml policy file
on Mar 9, 2022 - added and removedstaleStale PRs per the .github/workflows/stale.yml policy fileStale PRs per the .github/workflows/stale.yml policy file
on Mar 9, 2022 48 remaining items
@yuseok89 We are unassigning you from this issue as part of our updated assignment policy.
This is not meant to discourage your contribution — quite the opposite! You are still very welcome to work on this issue and submit a PR for it. Simply comment that you are working on it and open a PR when ready.
We found that formal assignments were not working well, as they often prevented others from contributing when the assignee was not actively working on the issue.
@nailo2c We are unassigning you from this issue as part of our updated assignment policy.
This is not meant to discourage your contribution — quite the opposite! You are still very welcome to work on this issue and submit a PR for it. Simply comment that you are working on it and open a PR when ready.
We found that formal assignments were not working well, as they often prevented others from contributing when the assignee was not actively working on the issue.
@Abhishekmishra2808 We are unassigning you from this issue as part of our updated assignment policy.
This is not meant to discourage your contribution — quite the opposite! You are still very welcome to work on this issue and submit a PR for it. Simply comment that you are working on it and open a PR when ready.
We found that formal assignments were not working well, as they often prevented others from contributing when the assignee was not actively working on the issue.
@Prab-27 We are unassigning you from this issue as part of our updated assignment policy.
This is not meant to discourage your contribution — quite the opposite! You are still very welcome to work on this issue and submit a PR for it. Simply comment that you are working on it and open a PR when ready.
We found that formal assignments were not working well, as they often prevented others from contributing when the assignee was not actively working on the issue.
@Srabasti We are unassigning you from this issue as part of our updated assignment policy.
This is not meant to discourage your contribution — quite the opposite! You are still very welcome to work on this issue and submit a PR for it. Simply comment that you are working on it and open a PR when ready.
We found that formal assignments were not working well, as they often prevented others from contributing when the assignee was not actively working on the issue.
Hi @shahar1, if there are any PRs that need system tests, please let me know, I think I could help.
Reacted by Shahar EpsteinClosing this issue, as there's only a single operator left which will be treated separately.
Thanks again everyone!Reacted by Yuseok Jo and Aaron ChenThis issue was open for a long time, and it's great to see it finally closed. Thanks to everyone who contributed, and special thanks to @shahar1 for driving this to completion.
Reacted by Shahar Epstein and Pratiksha BadhekaThanks for organizing this issue, great job @shahar1!
Reacted by Shahar Epstein and Pratiksha Badheka
Description
Currently some storage operators return files destination uris list for example
GoogleSheetsToGCSOperator. It would be a good idea to make all of such operators to return this list.Use case / motivation
Edit (by @shahar1 on Jan. 31, 2026)
For operators that currently return a single entity, add the flag
unwrap_singlewhich controls whether the returned value is eitherlist[obj]orobj(usuallyobjisstrbut could also be adict). Until we apply consistency for all operators (unwrap_single=False), which will be a breaking change - set for now the default to return the single entity (unwrap_single=True).Tracking (added by @shahar1 on Jan. 24, 2026)
All operators should return
list[str]of destination URIs (gs://bucket/object).To Do (Generated by AI, please double check if current situation is as described)
ADLSToGCSOperator (
transfers/adls_to_gcs.py)list[str](ADLS source paths) → Need: destination URIsAzureFileShareToGCSOperator (
transfers/azure_fileshare_to_gcs.py)list[str](source filenames) → Need: destination URIsAzureBlobStorageToGCSOperator (
transfers/azure_blob_to_gcs.py)str→ Need:list[str](wrap in list)GCSToGCSOperator (
transfers/gcs_to_gcs.py)None→ Need: destination URIsLocalFilesystemToGCSOperator (
transfers/local_to_gcs.py)None→ Need: destination URIsGoogleDriveToGCSOperator (
transfers/gdrive_to_gcs.py)GoogleDriverToGCSOperatorandSheetsToGCSOperator#61347HttpToGCSOperator (
transfers/http_to_gcs.py)None→ Need: destination URIsFacebookAdsReportToGcsOperator (
transfers/facebook_ads_to_gcs.py)None→ Need: destination URIsGoogleCalendarToGCSOperator (
transfers/calendar_to_gcs.py)gs://URIsGoogleSheetsToGCSOperator (
transfers/sheets_to_gcs.py)list[str](object names only) → Need: fullgs://URIsGoogleDriverToGCSOperatorandSheetsToGCSOperator#61347SalesforceToGcsOperator (
transfers/salesforce_to_gcs.py)str→ Need:list[str](wrap in list)GoogleAdsToGcsOperator (
ads/transfers/ads_to_gcs.py)None→ Need: destination URIsGoogleDisplayVideo360SDFtoGCSOperator (
marketing_platform/operators/display_video.py)gs://URIsS3ToGcsOperator (
cloud/transfers/s3_to_gcs.py)