Support for passing KMS key when copying S3 objects across AWS accounts using S3CopyObjectOperator

[RajasGujarathi]

I'm using the S3CopyObjectOperator in Apache Airflow to copy objects between two S3 buckets that reside in different AWS accounts. The destination bucket is encrypted using a KMS key.

However, I couldn't find a way to pass the KMS key to the operator for the destination bucket. This results in a permissions or encryption error during the copy operation.

🔍 Use Case
Source bucket: Account A
Destination bucket: Account B (KMS-encrypted)
Operator: S3CopyObjectOperator

❓ Question
How can I pass the KMS key ID for the destination bucket in this scenario?

Originally posted by @RajasGujarathi in #55658

[jroachgolf84]

@RajasGujarathi, are you looking to pass the ARN for a KMS key to this Operator?

[jroachgolf84]

It seems like this can be done using boto3; I don't see any reason why we couldn't get this functionality into the S3Hook, and therefore, into the S3CopyObjectOperator.

[jroachgolf84]

Which of these parameters would you need to pass in?

• ServerSideEncryption
• SSECustomerAlgorithm
• SSECustomerKey
• SSEKMSKeyId
• SSEKMSEncryptionContext

[RajasGujarathi]

@RajasGujarathi, are you looking to pass the ARN for a KMS key to this Operator?

Yes, I am looking to pass the KMS Key ARN of the destination bucket.

[RajasGujarathi]

It seems like this can be done using boto3; I don't see any reason why we couldn't get this functionality into the S3Hook, and therefore, into the S3CopyObjectOperator.

Yes, even this was my understanding. That it should be possible.

[RajasGujarathi]

Which of these parameters would you need to pass in?

• ServerSideEncryption
• SSECustomerAlgorithm
• SSECustomerKey
• SSEKMSKeyId
• SSEKMSEncryptionContext

In my current scenario, I am looking at "SSEKMSKeyId"

[dominikhei]

An easy way would be to add these two params: ServerSideEncryption="aws:kms" and SSEKMSKeyId="arn:aws:kms:xxx" as kwargs to the Operator and then pass them to the hook, which is currently not done. We would simply need to pass them forward and use them in the boto3 call.

I will draft a PR for this.

[dominikhei]

@RajasGujarathi if the destination bucket has default SSE-KMS encryption, you do not need to pass a KMS key in the copy operation, as the default one is used. Could it simply be that the role used has not access to this key? or do you want to use a different, non-default KMS key