Skip to content

Fix prefix handling in Amazon transfer operators - #73269

Merged
vincbeck merged 3 commits into
apache:mainfrom
yuseok89:fix-s3-transfer-prefix-handling
Sep 23, 2026
Merged

vincbeck merged 3 commits into
apache:mainfrom
yuseok89:fix-s3-transfer-prefix-handling

Conversation

@yuseok89

@yuseok89 yuseok89 commented Sep 17, 2026 •

Copy link
Copy Markdown
Contributor

This PR aligns filename prefix handling with the documented behavior of the Amazon FTP and SFTP transfer operators.

Previously, string prefixes were matched anywhere in a filename. This could select unintended files. Renaming also replaced every occurrence of the prefix, including occurrences later in the filename.

Documented behavior. Each operator's docstring already describes prefix semantics, not substring semantics:

:param s3_filenames: ... or a string prefix that all filenames must match.
:param ftp_filenames: ... prefix that replaces the s3 prefix.

The old implementation didn't match this. This PR makes the code match the docs, not the other way around.

The updated behavior:

  • Selects only filenames that start with the configured source prefix.
  • Replaces only the leading source prefix when a destination prefix is configured.
  • Warns with the names of entries the previous substring rule selected but a prefix does not, naming up to ten.
  • Matches and builds the destination key on the basename in FTP-to-S3, because some servers qualify nlst entries with the listed directory.
  • Matches S3 keys on their file name, so nested keys under s3_key keep their directory at the destination.
  • Applies consistently to FTP-to-S3, SFTP-to-S3, S3-to-FTP, and S3-to-SFTP transfers.
  • Adds regression tests covering both file selection and prefix replacement.

Example, with ftp_filenames="pre_" / s3_filenames="new_":

Filename Should match? Before After
pre_one.txt yes new_one.txt new_one.txt
xpre_two.txt no selected (bug) excluded
pre_again_pre_.txt yes new_again_new_.txt (corrupted mid-filename) new_again_pre_.txt

Users relying on the undocumented substring-matching behavior may observe different file selection after this change.


Was generative AI tooling used to co-author this PR?
  • Yes (please specify the tool below)
    • gpt-5.6

  • Read the Pull Request Guidelines for more information. Note: commit author/co-author name and email in commits become permanently public when merged.
  • For fundamental code changes, an Airflow Improvement Proposal (AIP) is needed.
  • When adding dependency, check compliance with the ASF 3rd Party License Policy.
  • For significant user-facing changes create newsfragment: {pr_number}.significant.rst, in airflow-core/newsfragments. You can add this file in a follow-up commit after the PR is created so you know the PR number.

@boring-cyborg boring-cyborg Bot added area:providers provider:amazon AWS/Amazon - related issues labels Sep 17, 2026
@yuseok89
yuseok89 marked this pull request as ready for review September 17, 2026 14:18
@yuseok89
yuseok89 requested a review from o-nikolas as a code owner September 17, 2026 14:18
@vincbeck

Copy link
Copy Markdown
Contributor

@ramitkataria @seanghaeli

Comment thread providers/amazon/src/airflow/providers/amazon/aws/transfers/ftp_to_s3.py Outdated
@seanghaeli

Copy link
Copy Markdown
Contributor

Looks good, pending merge conflicts

@yuseok89
yuseok89 force-pushed the fix-s3-transfer-prefix-handling branch from 858ff23 to 4b67f7a Compare September 22, 2026 00:27

@seanghaeli seanghaeli left a comment •

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Happy to merge this @o-nikolas @vincbeck @ferruzzi

edit: pending Ramit's below comments are addressed

Comment thread providers/amazon/src/airflow/providers/amazon/aws/transfers/s3_to_ftp.py Outdated
@vincbeck
vincbeck merged commit 3816831 into apache:main Sep 23, 2026
83 checks passed
@vincbeck

Copy link
Copy Markdown
Contributor

Note for the next release manager, this a bug fix. It is a breaking change but considered as bug fix because the doc and code was not matching

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area:providers provider:amazon AWS/Amazon - related issues

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants