Repository navigation
GCSSynchronizeBucketsOperator fails on 30-second timeout on large files #27488
Description
Apache Airflow Provider(s)
Versions of Apache Airflow Providers
apache-airflow-providers-google==6.7.0
Apache Airflow version
v2.2.3+composer
Operating System
whatever Linux version Cloud Composer uses
Deployment
Composer
Deployment details
No response
What happened
Using GCSSynchronizeBucketsOperator to clone a folder between two buckets. There are several thousand objects, and they range in size from 50MB to 10GB.
At some point the operation fails with this error: `google.api_core.exceptions.GoogleAPICallError: 413 POST https://storage.googleapis.com/storage/v1/b//o/zoominfo%2FZI_COMP_DESCRIPTION_20211001.csv.gz/copyTo/b//o/zoominfo2%2FZI_COMP_DESCRIPTION_20211001.csv.gz?prettyPrint=false: Copy spanning locations and/or storage classes could not complete within 30 seconds. Please use the Rewrite method in the JSON API (https://cloud.google.com/storage/docs/json_api/v1/objects/rewrite) instead.
What you think should happen instead
GCSSynchronizeBucketsOperator should use the Rewrite method so that it can deal with copies that do not complete within 30 seconds.
How to reproduce
Use GCSSynchronizeBucketsOperator to copy a 10GB file.
Anything else
No response
Are you willing to submit PR?
- Yes I am willing to submit a PR!
Code of Conduct
- I agree to follow this project's Code of Conduct
Activity
boring-cyborg commentedon Nov 3, 2022
Thanks for opening your first issue here! Be sure to follow the issue template!
eladkal commentedon Nov 5, 2022
feel free to submit PR.
It seems like the operator is using the copy function though the rewrite function is available on the hook. It looks simple enough to make this change.
For your PR please provide detail explanation (in the PR) about why this change is better and clarify what is expected to happen to users who relay on current operator behavior.
jmelot commentedon Mar 20, 2023
Hi @eladkal, I ran into what seemed like a similar issue the other day. If it's still useful, I'll try to submit a PR for this within the next week or so.
eladkal commentedon Mar 20, 2023
Sure. Assigned to you
kevgeo commentedon Feb 1, 2024
Hi @jmelot, are you still working on this issue? I was able to reproduce to it and can provide a fix.
jmelot commentedon Feb 10, 2024
@kevgeo I never was able to get a chance to work on this, glad you have a fix!