Status of testing Providers that were prepared on March 04, 2024

@vincbeck

Body

I have a kind request for all the contributors to the latest provider packages release.
Could you please help us to test the RC versions of the providers?

The guidelines on how to test providers can be found in

Verify providers by contributors

Let us know in the comment, whether the issue is addressed.

Those are providers that require testing as there were some substantial changes introduced:

Provider amazon: 8.19.0rc1

Provider apache.beam: 5.6.2rc1

Add Python 3.12 exclusions in providers/pyproject.toml (#37404): @potiuk

Provider apache.druid: 3.9.0rc1

Provider apache.hdfs: 4.3.3rc1

Turn Pydantic into an optional dependency (#37320): @potiuk

Provider apache.hive: 7.0.1rc1

Provider apache.livy: 3.7.3rc1

Fix rendering LivyOperator.spark_params (#37361): @Taragolis

Provider apache.pinot: 4.3.1rc1

Bump lower-limit for pinotdb (#37690): @potiuk

Provider celery: 3.6.1rc1

Provider cncf.kubernetes: 8.0.1rc1

Provider common.sql: 1.11.1rc1

Provider dbt.cloud: 3.7.0rc1

add reuse_existing_run for allowing DbtCloudRunJobOperator to reuse existing run (#37474): @Lee-W

Provider docker: 3.9.2rc1

Fix construct docker.TLSConfig for docker>=7 (#37481): @Taragolis

Provider exasol: 4.4.3rc1

Limit pandas to <2.2 (#37748): @Taragolis

Provider google: 10.16.0rc1

Provider hashicorp: 3.6.4rc1

Fix unexpected keyword argument 'secret_path' in secrets.kv.v2 (#37626): @tungbq

Provider http: 4.10.0rc1

Provider microsoft.azure: 9.0.1rc1

fix: Pass proxies config when using ClientSecretCredential in AzureDataLakeStorageV2Hook (#37103): @dabla

Provider microsoft.psrp: 2.6.0rc2

Add PSRP connection type (#34766): @arjunanan6

Provider mysql: 5.5.4rc1

Provider openlineage: 1.6.0rc1

Provider opensearch: 1.1.2rc1

Fix "Exception statement has no effect" (#37722): @Taragolis

Provider postgres: 5.10.2rc1

Implement AIP-60 Dataset URI formats (#37005): @uranusjr

Provider presto: 5.4.2rc1

Limit pandas to <2.2 (#37748): @Taragolis

Provider salesforce: 5.6.3rc1

Limit pandas to <2.2 (#37748): @Taragolis

Provider smtp: 1.6.1rc1

Provider telegram: 4.4.0rc2

Add telegram connection with documnetation (#37515): @romsharon98

Provider trino: 5.6.3rc1

Provider weaviate: 1.3.3rc1

Limit pandas to <2.2 (#37748): @Taragolis

All users involved in the PRs:
@vincbeck @VladaZakharova @pankajastro @dabla @m1racoli @RNHTTR @danielbe11 @Taragolis @pavansharma36 @kacpermuda @moiseenkov @dirrao @kevgeo @okirialbert @eejbyfeldt @mattiabertorello @MaksYermak @hter

Committer

I acknowledge that I am a maintainer/committer of the Apache Airflow project.

Updating setup.cfg with RC provider packages on apache/airflow#37890

hi,
#37282 works as expected

All the PR's for the Provider openlineage: 1.6.0rc1 works as expected.

@m1racoli

I've check my changes, most of them pretty simple.

Replace usage of datetime.utcnow and datetime.utcfromtimestamp in providers #37138 in google Provider changes should be fixed by fix: use offset-naive datetime in _CredentialsToken #37539, cc: @m1racoli
Fix rendering LivyOperator.spark_params #37361: I've checked that it rendered correctly, it would be nice if @LudvigLundberg have a time to look on changes as original reporter of the issue

@m1racoli

I've check my changes, most of them pretty simple.

Replace usage of datetime.utcnow and datetime.utcfromtimestamp in providers #37138 in google Provider changes should be fixed by fix: use offset-naive datetime in _CredentialsToken #37539, cc: @m1racoli

Fix rendering LivyOperator.spark_params #37361: I've checked that it rendered correctly, it would be nice if @LudvigLundberg have a time to look on changes as original reporter of the issue

Can confirm that the changes in #37361 have solved the issue I was experiencing! Big thanks for fixing it!

#37711 looks good.

Huge thank you to everyone that helped resolve this mini fire last week (Jarek, Andrey, Elad 🙇 )

#34766 works as expected. :)

I can confirm #37486. It works as expected.

Although I had to manually require apache-airflow-providers-cncf-kubernetes>=8.0.0 for my DAG to parse. Before that I had version 7.14.0 installed and getting

Broken DAG: [/usr/local/airflow/dags/test_gke.py] Traceback (most recent call last):
  File "/usr/local/airflow/dags/test_gke.py", line 3, in <module>
    from airflow.providers.google.cloud.operators.kubernetes_engine import GKEStartPodOperator, OnFinishAction
  File "/usr/local/lib/python3.11/site-packages/airflow/providers/google/cloud/operators/kubernetes_engine.py", line 35, in <module>
    from airflow.providers.cncf.kubernetes.operators.job import KubernetesJobOperator
ModuleNotFoundError: No module named 'airflow.providers.cncf.kubernetes.operators.job'

I assume this is because of #36847. Are we missing a requirement for apache-airflow-providers-cncf-kubernetes>=8.0.0 in the google provider's dependencies?

@m1racoli

I assume this is because of #36847. Are we missing a requirement for apache-airflow-providers-cncf-kubernetes>=8.0.0 in the google provider's dependencies?

@m1racoli , yes you're right. Thank you!
@eladkal , can we please exclude google from this release for now, merge the fix #37899 and release google afterwards with RC2?

All my changes are there :)

Verified #36794

@moiseenkov

@moiseenkov I don't think this qualify for RC2. It's not a regression and there is a simple fix (bump k8s provider if not using latest). Also the RC1 contains many features and fixes that not worth delaying.
We can always improve the mechanism and have better error message for next release.

@moiseenkov

@moiseenkov I don't think this qualify for RC2. It's not a regression and there is a simple fix (bump k8s provider if not using latest). Also the RC1 contains many features and fixes that not worth delaying.
We can always improve the mechanism and have better error message for next release.

I totally agree. We will adjust error messages in the google provider package for the next release and will follow this convention from now. Thank you!

#37634 looks good. thanks!

Thank you everyone. Providers are released.

I invite everyone to help improve providers for the next release, a list of open issues can be found here.

Maybe a bit late but I tested #37103 and I can confirm it works :)

Never too late for good news :)

Status of testing Providers that were prepared on March 04, 2024 #37890

Description

Body

Provider amazon: 8.19.0rc1

Provider apache.beam: 5.6.2rc1

Provider apache.druid: 3.9.0rc1

Provider apache.hdfs: 4.3.3rc1

Provider apache.hive: 7.0.1rc1

Provider apache.livy: 3.7.3rc1

Provider apache.pinot: 4.3.1rc1

Provider celery: 3.6.1rc1

Provider cncf.kubernetes: 8.0.1rc1

Provider common.sql: 1.11.1rc1

Provider dbt.cloud: 3.7.0rc1

Provider docker: 3.9.2rc1

Provider exasol: 4.4.3rc1

Provider google: 10.16.0rc1

Provider hashicorp: 3.6.4rc1

Provider http: 4.10.0rc1

Provider microsoft.azure: 9.0.1rc1

Provider microsoft.psrp: 2.6.0rc2

Provider mysql: 5.5.4rc1

Provider openlineage: 1.6.0rc1

Provider opensearch: 1.1.2rc1

Provider postgres: 5.10.2rc1

Provider presto: 5.4.2rc1

Provider salesforce: 5.6.3rc1

Provider smtp: 1.6.1rc1

Provider telegram: 4.4.0rc2

Provider trino: 5.6.3rc1

Provider weaviate: 1.3.3rc1

Committer

Activity

moiseenkov commented on Mar 5, 2024

kacpermuda commented on Mar 5, 2024

Taragolis commented on Mar 5, 2024

LudvigLundberg commented on Mar 5, 2024

vchiapaikeo commented on Mar 5, 2024

arjunanan6 commented on Mar 5, 2024

m1racoli commented on Mar 5, 2024

moiseenkov commented on Mar 5, 2024

potiuk commented on Mar 6, 2024

pavansharma36 commented on Mar 6, 2024

eladkal commented on Mar 6, 2024

moiseenkov commented on Mar 6, 2024

pankajastro commented on Mar 6, 2024

eladkal commented on Mar 8, 2024

dabla commented on Mar 8, 2024

potiuk commented on Mar 8, 2024

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions