Repository navigation
Status of testing Providers that were prepared on February 12, 2024 #37358
Description
Body
I have a kind request for all the contributors to the latest provider packages release.
Could you please help us to test the RC versions of the providers?
The guidelines on how to test providers can be found in
Verify providers by contributors
Let us know in the comment, whether the issue is addressed.
Those are providers that require testing as there were some substantial changes introduced:
Provider amazon: 8.18.0rc1
- ECS Executor - Add backoff on failed task retry (#37109): @ferruzzi
- SqlToS3Operator: feat/ add max_rows_per_file parameter (#37055): @selimchergui
- Adding Amazon Neptune Hook and Operators (#37000): @ellisms
- Fix
log_queryto format SQL statement correctly inAthenaOperator(#36962): @ferruzzi - check sagemaker training job status before deferring SageMakerTrainingOperator (#36685): @Lee-W
- Merge all ECS executor configs following recursive python dict update (#37137): @o-nikolas
- Update default value for
BatchSensor(#37234): @vincbeck - Remove info log from amazon provider-> S3 Hook -> download_file (#37211): @rawwar
- feature: S3ToRedshiftOperator templating aws_conn_id (#37195): @raphaelauv
- Standardize deprecations in providers [part1] (#36876): @kacpermuda
Provider apache.beam: 5.6.1rc1
Provider apache.drill: 2.6.1rc1
Provider apache.druid: 3.8.1rc1
Provider apache.hive: 7.0.0rc1
- Simplify hive client connection (#37043): @romsharon98
- Fix pyhive hive_pure_sasl extra name (#37323): @hussein-awala
Provider apache.livy: 3.7.2rc1
Provider apprise: 1.2.2rc1
Provider celery: 3.6.0rc1
- Add
task_acks_lateconfiguration to Celery Executor (#37066): @ninsbl - prevent celery command autoscale misconfig (#36576): @Bowrna
Provider cncf.kubernetes: 8.0.0rc1
- The task is stuck in a queued state forever in case of pod launch errors (#36882): @dirrao
- Modify KPO to log container log periodically (#37279): @pankajastro
- Fix occasional attr-undefined for the python_kubernetes_script (#37318): @potiuk
- Fix hanging KPO on deferrable task with do_xcom_push (#37300): @vchiapaikeo
- Fix rendering
SparkKubernetesOperator.template_body(#37271): @Taragolis - Fix assignment of template field in
__init__inKubernetesPodOperator(#37010): @romsharon98 - Standardize deprecations in providers [part1] (#36876): @kacpermuda
- Kubernetes version bump (#37040): @dirrao
Provider common.io: 1.3.0rc1
Provider common.sql: 1.11.0rc2
- Enhancement: Performance enhancement for insert_rows method DbApiHook with fast executemany + SAP Hana support (#37246): @dabla
- Standardize deprecations in providers [part1] (#36876): @kacpermuda
Provider databricks: 6.2.0rc1
- Update DatabricksSqlOperator to work with namedtuple (#37025): @SuperBo
- Bump aiohttp min version to avoid CVE-2024-23829 and CVE-2024-23334 (#37110): @hussein-awala
- Standardize deprecations in providers [part1] (#36876): @kacpermuda
- Get rid of pytest-httpx as dependency (#37334): @potiuk
Provider dbt.cloud: 3.6.1rc1
- Bump aiohttp min version to avoid CVE-2024-23829 and CVE-2024-23334 (#37110): @hussein-awala
- Standardize deprecations in providers [part1] (#36876): @kacpermuda
Provider elasticsearch: 5.3.3rc1
- Avoid
pendulum.from_timestampusage (#37160): @Taragolis - Standardize deprecations in providers [part1] (#36876): @kacpermuda
Provider google: 10.15.0rc1
- add service_file support to GKEPodAsyncHook (#37081): @Lee-W
- Update GCP Dataproc ClusterGenerator to support GPU params (#37036): @dmedora
- Create DataprocStartClusterOperator and DataprocStopClusterOperator (#36996): @molcay
- Implement deferrable mode for CreateHyperparameterTuningJobOperator (#36594): @moiseenkov
- Enable '_enable_tcp_keepalive' functionality for GKEPodHook (#36999): @MaksYermak
- fix how GKEPodAsyncHook.service_file_as_context is used (#37306): @Lee-W
- Fix metadata override for ComputeEngineSSHHook (#37192): @molcay
- Fix assignment of template field in
__init__incustom_job(#36789): @romsharon98 - Fix location requirement in DataflowTemplatedJobStartOperator (#37069): @shardbread
- Fix assignment of template field in
__init__inCloudDataTransferServiceCreateJobOperator(#36909): @romsharon98 - Fixed the hardcoded default namespace value for GCP Data Fusion links. (#35379): @synsh
- Do not ignore the internal_ip_only if set to false in Dataproc cluster config (#37014): @ahzaz
- Revert protection against back-compatibilty issue with google-core-api (#37111): @potiuk
- Standardize deprecations in providers [part1] (#36876): @kacpermuda
Provider hashicorp: 3.6.3rc1
- feat(hashicorp): Add
VAULT_TOKENsupport (#37337): @V0lantis - Standardize deprecations in providers [part1] (#36876): @kacpermuda
Provider http: 4.9.1rc1
- Bump aiohttp min version to avoid CVE-2024-23829 and CVE-2024-23334 (#37110): @hussein-awala
- Standardize deprecations in providers [part1] (#36876): @kacpermuda
- Get rid of pytest-httpx as dependency (#37334): @potiuk
Provider influxdb: 2.4.1rc1
Provider jdbc: 4.2.2rc1
Provider microsoft.mssql: 3.6.1rc1
Provider mongo: 4.0.0rc1
- Adding certificate validation for ssl in mongo hook (#37214): @amoghrajesh
- Change
MongoHook.conn_name_attrtomongo_conn_id(#37104): @Taragolis - Change docs type in insert_many to list of dicts (#37230): @MezentsevIlya
Provider mysql: 5.5.3rc1
Provider oracle: 3.9.2rc1
Provider pagerduty: 3.6.2rc1
Provider postgres: 5.10.1rc1
Provider qdrant: 1.0.0rc1
Provider sftp: 4.9.0rc1
Provider slack: 8.6.1rc1
- Remove the remaining references to use the token bypassing the Connection in the Slack provider. (#37112): @Taragolis
- Standardize deprecations in providers [part1] (#36876): @kacpermuda
Provider snowflake: 5.3.1rc1
Provider sqlite: 3.7.1rc1
Provider ssh: 3.10.1rc1
Provider tableau: 4.4.2rc1
Provider trino: 5.6.2rc1
Provider vertica: 3.7.1rc1
Provider weaviate: 1.3.2rc1
Provider yandex: 3.9.0rc1
- Add secrets-backends section into the Yandex provider yaml definition (#37065): @Taragolis
- fix: using endpoint from connection if not specified (#37076): @ImpressionableRaccoon
All users involved in the PRs:
@ellisms @moiseenkov @synsh @anush008 @infohash @shardbread @raphaelauv @hussein-awala @molcay @pankajastro @selimchergui @dabla @rawwar @SuperBo @ninsbl @V0lantis @kacpermuda @ahzaz @dirrao
Committer
- I acknowledge that I am a maintainer/committer of the Apache Airflow project.
Activity
moiseenkov commentedon Feb 12, 2024
#36594 works as expected
moiseenkov commentedon Feb 12, 2024
@potiuk , @eladkal , hi,
Could we please include #37282 into the release?
eladkal commentedon Feb 12, 2024
If PR missed cut off time there is no option to add it. It will have to wait for next release / RC2 if we will have one.
Updating setup.cfg with RC provider packages on apache/airflow#37358
Updating setup.cfg with RC provider packages on apache/airflow#37358
pankajkoti commentedon Feb 12, 2024
Observing an issue with common.sql: 1.11.0rc2
Posted a question on the PR
potiuk commentedon Feb 12, 2024
Nice catch @pankajkoti
pankajkoti commentedon Feb 12, 2024
Actually, it was the (Astronomer) Providers RC automation test suite that reported this failure :)
Thank you @potiuk for providing the fix PR #37359 so quick. Appreciate it 🙏🏽
infohash commentedon Feb 12, 2024
vchiapaikeo commentedon Feb 12, 2024
#37300 looks good. Also, @Lee-W , thank you for that fix to GKE providers.
However, I have deferrable tasks that should be completing that are not completing due to #37279. Will comment on the PR there. cc @pankajastro @pankajkoti
Link to comment: #37279 (comment)
pankajastro commentedon Feb 12, 2024
Thanks for testing it. please share the DAG and other details if possible.
ninsbl commentedon Feb 12, 2024
Hmm... I am struggeling to build a recent version of Airflow with
celery 3.6.0rc1. The issue seems unrelated to the changes in the provider. I tried to build a container with 2.9.0 but there FAB provider seems to be missing / not installable...potiuk commentedon Feb 12, 2024
Yes. FAB provider and Airflow 2.9.0 are not yet usable to install. We might want to change that and release FAB provider in the next wave
12 remaining items
anush008 commentedon Feb 13, 2024
Tested DAGs. Works.
eladkal commentedon Feb 13, 2024
Providers
amazon,cncf.kubernetesandcommon.sqlwill be excluded from RC1.SuperBo commentedon Feb 13, 2024
Databricks providers, confirm #37025 works when xcom is disabled. However it raises error with xcom enabled when serializing named tuple to json.
Taragolis commentedon Feb 13, 2024
Serializing named tuple should be add into the core first, e.g. it might be added by #37168
Taragolis commentedon Feb 13, 2024
Checked all my changes, looking good
ahzaz commentedon Feb 13, 2024
#37014 works as expected.
pankajkoti commentedon Feb 14, 2024
#37117 works as expected
utkarsharma2 commentedon Feb 14, 2024
raphaelauv commentedon Feb 14, 2024
Fix hanging KPO on deferrable task with do_xcom_push (#37300): @vchiapaikeowork 👍
pankajastro commentedon Feb 14, 2024
Hey @vchiapaikeo would it be possible for you to test #37363 with your use case?
eladkal commentedon Feb 14, 2024
Since we are having RC2 for amazon provider there is time to fix it if you can raise a PR? @o-nikolas
ninsbl commentedon Feb 15, 2024
#37066 Seems to work fine. The environment variable is acknowledged in the airflow config and long running tasks have been finnishing successfull with Celery 3.6.0rc1.
Lee-W commentedon Feb 15, 2024
I tested #36685 a few more times today. The issue can no longer be reproduced.
eladkal commentedon Feb 15, 2024
Thank you everyone. Providers are released.
Providers:
amazon,common.sql,cncf.kuberneteswere excluded from this release.I invite everyone to help improve providers for the next release, a list of open issues can be found here.