Repository navigation
Status of testing Providers that were prepared on November 24, 2023 #35845
Description
Body
I have a kind request for all the contributors to the latest provider packages release.
Could you please help us to test the RC versions of the providers?
The guidelines on how to test providers can be found in
Verify providers by contributors
Let us know in the comment, whether the issue is addressed.
Those are providers that require testing as there were some substantial changes introduced:
Provider amazon: 8.12.0rc1
- Add EC2HibernateInstanceOperator and EC2RebootInstanceOperator (#35790): @dashton90
- Add OpenLineage support to S3FileTransformOperator (#35819): @kacpermuda
- Add OpenLineage support to S3Operators - Copy, Delete and Create Object (#35796): @kacpermuda
- Added Feature: retry strategy parameter to Amazon AWS Batch Operator (#35789): @evgenyslab
- Added name field to template_fields in EmrServerlessStartJobOperator (#35648): @sseung00921
- openlineage, aws: Add OpenLineage support for AthenaOperator. (#35090): @JDarDagran
- Implement login and logout in AWS auth manager (#35488): @vincbeck
- Fix recent change to Batch operator's retry_strategy (#35808): @ferruzzi
- Fix and reapply templates for provider documentation (#35686): @potiuk
- Make EksPodOperator exec config not rely on log level (#35771): @syedahsn
- Update emr.py fix serverless operator without runtime override (#35787): @tnyz
- Updated docstring:
check_key_asyncis now in line with description of_check_key_async(#35799): @bolleyboll - Check attr on parent not self re TaskContextLogger set_context (#35780): @dstandish
- Allow a wider range of watchtower versions (#35713): @o-nikolas
- Extend task context logging support for remote logging using AWS S3 (#32950): @pankajkoti
- Log failure reason for containers if a task fails for ECS Executor (#35496): @syedahsn
Provider apache.impala: 1.2.1rc1
Provider atlassian.jira: 2.3.0rc1
Provider cncf.kubernetes: 7.10.0rc1
- Add annotations to template_fields in KubernetesPodOperator (#35641): @shohamy7
- feat: K8S resource operator - CRD (#35600): @raphaelauv
- Revert Remove PodLoggingStatus object #35422 (#35822): @dstandish
- Fix K8S executor override config using pod_override_object (#35185): @hussein-awala
- Fix and reapply templates for provider documentation (#35686): @potiuk
- Remove inconsequential code bits in KPO logging (#35416): @dstandish
- KubernetesExecutor observability Improvements (#35579): @dirrao
Provider common.io: 1.1.0rc1
Provider common.sql: 1.8.1rc1
- Make pyodbc.Row and databricks.Row JSON-serializable via new
make_serializablemethod (#32319): @Joffreybvn - Remove backcompat inheritance for DbApiHook (#35754): @Taragolis
- Use reproducible builds for provider packages (#35693): @potiuk
- Fix and reapply templates for provider documentation (#35686): @potiuk
- Work around typing issue in examples and providers (#35494): @uranusjr
- Pre-upgrade
ruff==0.0.292changes in providers (#35053): @Taragolis
Provider databricks: 5.0.1rc1
- Make pyodbc.Row and databricks.Row JSON-serializable via new
make_serializablemethod (#32319): @Joffreybvn
Provider dbt.cloud: 3.4.1rc1
Provider docker: 3.8.2rc1
Provider elasticsearch: 5.2.0rc1
- Add task context logging feature to allow forwarding messages to task logs (#32646): @pankajkoti
- Extend task context logging support for remote logging using Elasticsearch (#32977): @pankajkoti
- Update es read query to not use body (#34792): @dstandish
- Check attr on parent not self re TaskContextLogger set_context (#35780): @dstandish
- Remove backcompat inheritance for DbApiHook (#35754): @Taragolis
Provider google: 10.12.0rc1
- Add PubSub Topic params for schema_settings and message_retention_duration (#35767): @dmedora
- Add OpenLineage support to GCSToBigQueryOperator (#35778): @kacpermuda
- Add dedicated OpenLineage support for bigquery_to_gcs transfer (#35660): @kacpermuda
- Add support for driver pool, instance flexibility policy, and min_num_instances for Dataproc (#34172): @ahzaz
- Add "NON_PREEMPTIBLE" as a valid value for Dataproc workers (#35669): @tskir
- Add ability to pass impersonation_chain to BigQuery triggers (#35629): @VladaZakharova
- Add a filter for local files in GoogleDisplayVideo360CreateQueryOperator (#35635): @hussein-awala
- Extend task context logging support for remote logging using GCS (#32970): @pankajkoti
- Fix and reapply templates for provider documentation (#35686): @potiuk
- Fix the logic of checking dataflow job state (#34785): @champon1020
- Remove usage of deprecated method from BigQueryToBigQueryOperator (#35605): @MaksYermak
- Check attr on parent not self re TaskContextLogger set_context (#35780): @dstandish
- Remove backcompat with Airflow 2.3/2.4 in providers (#35727): @Taragolis
- Restore delegate_to param in GoogleDiscoveryApiHook (#35728): @austinletson
- Remove usage of deprecated methods from BigQueryCursor (#35606): @MaksYermak
Provider microsoft.azure: 8.3.0rc1
- Update provider.yaml: To view Azure Synapse Pipeline Connection type in Airflow UI (#35709): @ambika-garg
- Add task context logging feature to allow forwarding messages to task logs (#32646): @pankajkoti
- Add operator to invoke Azure-Synapse pipeline (#35091): @ambika-garg
- Extend task context logging support for remote logging using WASB (Azure Blob Storage) (#32972): @pankajkoti
- Check attr on parent not self re TaskContextLogger set_context (#35780): @dstandish
- Remove backcompat with Airflow 2.3/2.4 in providers (#35727): @Taragolis
Provider odbc: 4.2.0rc1
- Make pyodbc.Row and databricks.Row JSON-serializable via new
make_serializablemethod (#32319): @Joffreybvn
Provider openai: 1.0.1rc1
Provider opsgenie: 5.3.0rc1
Provider papermill: 3.5.0rc1
Provider redis: 3.4.1rc1
Provider snowflake: 5.1.2rc1
Provider trino: 5.4.1rc1
All users involved in the PRs:
@ferruzzi @champon1020 @tskir @Eric-Mendes @pubudu91 @dashton90 @o-nikolas @raphaelauv @bolleyboll @ccorleis @MaksYermak @kacpermuda @uranusjr @akshaychitneni @dmedora @Joffreybvn @evgenyslab
@ambika-garg @pankajkoti @potiuk @hussein-awala @ahzaz @tnyz @syedahsn @VladaZakharova @shohamy7 @austinletson @sseung00921 @dstandish @JDarDagran @bolkedebruin @vincbeck @ylnsnv @dirrao @Taragolis
Committer
- I acknowledge that I am a maintainer/committer of the Apache Airflow project.
Activity
raphaelauv commentedon Nov 24, 2023
#35600 work ✅
Updating setup.cfg with RC provider packages on apache/airflow#35845
Updating setup.cfg with RC provider packages on apache/airflow#35845
sseung00921 commentedon Nov 25, 2023
#35648 work ✅
bolleyboll commentedon Nov 25, 2023
#35799 work ✅
dashton90 commentedon Nov 25, 2023
#35790 works
It has caused a problem with the example DAG (addressed in #35839) but the operator changes themselves work
eladkal commentedon Nov 26, 2023
Thanks. Since functionality work as expected this will not block rc1 release.
hussein-awala commentedon Nov 26, 2023
I tested #35185 and #35635, they work as expected.
pankajkoti commentedon Nov 26, 2023
Tested my set of PRs, all work fine, thank you for the release efforts!
bolkedebruin commentedon Nov 26, 2023
I would like to see a re-discussion of #32319 before including it.
potiuk commentedon Nov 26, 2023
I think this is misunderstanding what #32319 wants to achieve. Commented there.
potiuk commentedon Nov 26, 2023
Conflirmed my template + reproducible builds worl (works great! PR shortly to a better PMC verification step is coming soon :) )
dirrao commentedon Nov 27, 2023
Tested my PR #35579 and it is working as expected.
Lee-W commentedon Nov 27, 2023
We tested it with our example DAGs with
apache-airflow-providers-amazon,apache-airflow-providers-cncf-kubernetes,apache-airflow-providers-databricks,apache-airflow-providers-dbt-cloud,apache-airflow-providers-google,apache-airflow-providers-cncf-kubernetes,apache-airflow-providers-microsoft-azure,apache-airflow-providers-snowflakeand everything works as expected. Thanks!Updating setup.cfg with RC provider packages on apache/airflow#35845
Updating setup.cfg with RC provider packages on apache/airflow#35845
Updating setup.cfg with RC provider packages on apache/airflow#35845
ferruzzi commentedon Nov 27, 2023
#35808 appears to be working as expected.
Joffreybvn commentedon Nov 27, 2023
#32319 works as expected !
akshaychitneni commentedon Nov 28, 2023
#34840 works as expected.
champon1020 commentedon Nov 28, 2023
#34785 works as expected
ambika-garg commentedon Nov 28, 2023
eladkal commentedon Nov 29, 2023
Thank you everyone.
Providers are released
I invite everyone to help improve providers for the next release, a list of open issues can be found here.