Repository navigation
Status of testing Providers that were prepared on June 07, 2024 #40115
Description
Body
I have a kind request for all the contributors to the latest provider packages release.
Could you please help us to test the RC versions of the providers?
The guidelines on how to test providers can be found in
Verify providers by contributors
Let us know in the comment, whether the issue is addressed.
Those are providers that require testing as there were some substantial changes introduced:
Provider amazon: 8.24.0rc2
- ECS Overrides for AWS Batch submit_job (#39903): @yehoshuadimarsky
- feat: implement amazon s3 to dynamodb transfer operator (#39654): @dondaum
Linked issues: - Adding Glue Data Quality Rule Recommendation Run (#40014): @gopidesupavan
- Allow user-specified object attributes to be used in check_fn for S3KeySensor (#39950): @ellisms
Linked issues: - Adding Amazon Glue Data Quality Service (#39923): @gopidesupavan
- Deduplicate model name in SageMakerTransformOperator (#39956): @mateuslatrova
Linked issues: - fix: DynamoDBToS3Operator - remove process_func from templated_fields (#39948): @raphaelauv
- Fix aws assume role session creation when deferrable (#40051): @gopidesupavan
Linked issues:- Linked Issue #39970: @arollet-decathlon
- Resolving ECS fargate deprecated warnings (#39834): @dirrao
- Resolving EMR notebook deprecated warnings (#39829): @dirrao
- Bump boto min versions (#40052): @ferruzzi
- Implement per-provider tests with lowest-direct dependency resolution (#39946): @potiuk
- Resolve aws emr deprecations in tests (#40020): @dirrao
Provider celery: 3.7.2rc1
- Fixing exception types to include TypeError, which is what is raised in (#40012): @leakec
Linked issues: - catch sentry flush if exception happens in _execute_in_fork finally block (#40060): @Lee-W
- add PID and return code to _execute_in_fork logging (#40058): @Lee-W
Provider cncf.kubernetes: 8.3.1rc1
- Fixes K8s trigger failing running pods with timeout (#40019): @Usiel
- Refresh properties on KubernetesPodOperator on token expiration also when logging (#39789): @paolo-moriello
- Fix Pod Number Exception in the sync mode if reattach_on_restart parameter is False (#39329): @e-galan
- Avoid resetting adopted task instances when retrying for kubernetes executor (#39406): @tanvn
- Include fatal reason for pod pending events (#39924): @wolfier
All users involved in the PRs:
@potiuk @mateuslatrova @ferruzzi @gopidesupavan @dirrao @leakec @tanvn @paolo-moriello @wolfier @raphaelauv @ellisms @Usiel @yehoshuadimarsky @e-galan @dondaum @Lee-W
Committer
- I acknowledge that I am a maintainer/committer of the Apache Airflow project.
Activity
potiuk commentedon Jun 7, 2024
My changes were just in tests but I tested that indeed the new celery provider fixes the problem with celery executor reported in #40011 (Impacting also previous versions of airflow). WDYT @eladkal, maybe we should yank the 3.7.1 of celery provider already - to avoid people to install it accidentally ? It's pretty bad when you install 3.7.1 for all previous Airlfow version that use celery provider.
eladkal commentedon Jun 7, 2024
Agreed
potiuk commentedon Jun 7, 2024
Yenked 3.7.1
dondaum commentedon Jun 7, 2024
Tested #39654, works as expected.
fdemiane commentedon Jun 7, 2024
No issues from my side
tanvn commentedon Jun 7, 2024
Tested #39406 by installing cncf.kubernetes: 8.3.1rc1 upon my Airflow 2.8.4, confirmed it is working as expected.
arollet22 commentedon Jun 7, 2024
Tested #40051 it fixed #39970 ! Huge thanks @gopidesupavan
yehoshuadimarsky commentedon Jun 7, 2024
Tested Amazon provider, works for me
leakec commentedon Jun 7, 2024
Tested
celery: 3.7.2rc1and verified it implements #40012 and that it fixes #40011.gopidesupavan commentedon Jun 7, 2024
Tested both #40014 and #39923 working fine.
gopidesupavan commentedon Jun 7, 2024
Thanks @arollet-decathlon for testing #40051
Updating setup.cfg with RC provider packages on apache/airflow#40115
Updating setup.cfg with RC provider packages on apache/airflow#40115
dirrao commentedon Jun 8, 2024
Tested #39834, #39829 and #40020. They looks good to me.
Usiel commentedon Jun 9, 2024
Tested #40019 with Airflow 2.9.1, all good 👍
eladkal commentedon Jun 9, 2024
Thank you everyone. Providers are released.
I invite everyone to help improve providers for the next release, a list of open issues can be found here.