Skip to content

Create GCS folder markers as directories in GCSToSFTPOperator - #73880

Merged
shahar1 merged 1 commit into
apache:mainfrom
yuseok89:skip-gcs-folder-markers-in-gcs-to-sftp-v2
Oct 3, 2026
Merged

shahar1 merged 1 commit into
apache:mainfrom
yuseok89:skip-gcs-folder-markers-in-gcs-to-sftp-v2

Conversation

@yuseok89

@yuseok89 yuseok89 commented Sep 29, 2026 •

Copy link
Copy Markdown
Contributor

related: #72638

Picks up #72638, closed under the open pull request limit. A GCS folder marker (folder/) was stored as an empty file at <dest>/folder, because os.path.normpath strips the slash, and the folder's children then failed to transfer.

Review feedback on #72638 and how it is addressed

  • @aaron-y-chen: move_object left skipped markers in the bucket, and a later run with only folder/ failed. A marker is now created as a directory and then deleted when moving. A later run is a no-op, since SFTPHook.create_directory returns for an existing directory.
  • @olegkachur-e: name/ always means a folder; prefer a switch over corner-case handling. Any name ending in / is created as a directory, never a file, and the earlier overlap and existing-directory checks are removed. The switch is the existing create_intermediate_dirs rather than a new parameter. With False, markers are skipped and left in the bucket.
  • @olegkachur-e: bug fix or breaking change? Added a warning to the Google provider changelog, including that an empty file left by an earlier version must be removed. The classification is left to the release manager.

Tests: test_gcs_to_sftp.py passes; with the operator reverted to main, only the 4 new tests fail.


Was generative AI tooling used to co-author this PR?
  • Yes (please specify the tool below)
    • Claude Code (Opus 5.5)

  • Read the Pull Request Guidelines for more information. Note: commit author/co-author name and email in commits become permanently public when merged.
  • For fundamental code changes, an Airflow Improvement Proposal (AIP) is needed.
  • When adding dependency, check compliance with the ASF 3rd Party License Policy.
  • For significant user-facing changes create newsfragment: {pr_number}.significant.rst, in airflow-core/newsfragments. You can add this file in a follow-up commit after the PR is created so you know the PR number.

@boring-cyborg boring-cyborg Bot added area:providers kind:documentation provider:google Google (including GCP) related issues labels Sep 29, 2026
@yuseok89
yuseok89 marked this pull request as ready for review September 29, 2026 11:27
@yuseok89
yuseok89 requested a review from shahar1 as a code owner September 29, 2026 11:27

@olegkachur-e olegkachur-e left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM, great work!👍

@aaron-y-chen aaron-y-chen left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks, LGTM!

@shahar1
shahar1 merged commit b9c5295 into apache:main Oct 3, 2026
162 of 163 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area:providers kind:documentation provider:google Google (including GCP) related issues

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants