Repository navigation
Create GCS folder markers as directories in GCSToSFTPOperator - #73880
Merged
shahar1 merged 1 commit intoOct 3, 2026
Merged
Conversation
yuseok89
marked this pull request as ready for review
September 29, 2026 11:27
shahar1
approved these changes
Oct 3, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
related: #72638
Picks up #72638, closed under the open pull request limit. A GCS folder marker (
folder/) was stored as an empty file at<dest>/folder, becauseos.path.normpathstrips the slash, and the folder's children then failed to transfer.Review feedback on #72638 and how it is addressed
move_objectleft skipped markers in the bucket, and a later run with onlyfolder/failed. A marker is now created as a directory and then deleted when moving. A later run is a no-op, sinceSFTPHook.create_directoryreturns for an existing directory.name/always means a folder; prefer a switch over corner-case handling. Any name ending in/is created as a directory, never a file, and the earlier overlap and existing-directory checks are removed. The switch is the existingcreate_intermediate_dirsrather than a new parameter. WithFalse, markers are skipped and left in the bucket.Tests:
test_gcs_to_sftp.pypasses; with the operator reverted tomain, only the 4 new tests fail.Was generative AI tooling used to co-author this PR?
{pr_number}.significant.rst, in airflow-core/newsfragments. You can add this file in a follow-up commit after the PR is created so you know the PR number.