Repository navigation
Status of testing Providers that were prepared on August 03, 2024 #41237
Description
Body
I have a kind request for all the contributors to the latest provider packages release.
Could you please help us to test the RC versions of the providers?
The guidelines on how to test providers can be found in
Verify providers by contributors
Let us know in the comment, whether the issue is addressed.
Those are providers that require testing as there were some substantial changes introduced:
Provider amazon: 8.27.0rc2
- Add RedriveExecution support to StepFunctionStartExecutionOperator (#40976): @gopidesupavan
- AIP-62: add hook lineage collection support for S3Hook (#40819): @mobuchowski
- Introduce Amazon Kinesis Analytics V2 (Managed Service for Apache Flink application) (#40765): @gopidesupavan
- Make EMR Container Trigger max attempts retries match the Operator (#41008): @o-nikolas
Linked issues: - Fix
RdsStopDbOperatoroperator in deferrable mode (#41059): @vincbeck - Fix
RedshiftCreateClusterOperatorto always specifyPubliclyAccessible(#40872): @vincbeck - Fix AWS Redshift operators and sensors (#41191): @vincbeck
- Fix
EmrServerlessStartJobOperator(#41103): @vincbeck - Update
example_redshiftandexample_redshift_s3_transfersto useRedshiftDataHookinstead ofRedshiftSQLHook(#40970): @vincbeck - openlineage: migrate OpenLineage provider to V2 facets. (#39530): @JDarDagran
- [AIP-62] Translate AIP-60 URI to OpenLineage (#40173): @kacpermuda
Linked issues: - Move AWS Managed Service for Apache Flink sensor states to Hook (#40896): @gopidesupavan
- Replace usages of task context logger with the log table (#40867): @dstandish
- Deprecate
SageMakerTrainingPrintLogTrigger(#41158): @vincbeck
Provider apache.beam: 5.7.2rc2
Provider apache.drill: 2.7.3rc1
Provider apache.druid: 3.10.2rc1
Provider apache.impala: 1.4.2rc1
Provider apache.pinot: 4.4.2rc1
Provider apprise: 1.3.2rc1
Provider celery: 3.7.3rc1
Provider cncf.kubernetes: 8.3.4rc1
- Pass content of kube/config file to triggerer as a dictionary (#41178): @VladaZakharova
- Fix confusing log message in kubernetes executor (#41035): @RNHTTR
- Fix ApiException handling when adopting completed pods (#41109): @jedcunningham
Provider common.compat: 1.1.0rc1
- AIP-62: add method to common.compat to not force hooks to try/except every time (#40812): @mobuchowski
- openlineage: migrate OpenLineage provider to V2 facets. (#39530): @JDarDagran
- AIP-62: add hook lineage collection support for S3Hook (#40819): @mobuchowski
Provider common.io: 1.4.0rc1
- [AIP-62] Translate AIP-60 URI to OpenLineage (#40173): @kacpermuda
Linked issues: - AIP-62: add file dataset type support into common.io provider (#40817): @mobuchowski
- openlineage: migrate OpenLineage provider to V2 facets. (#39530): @JDarDagran
- AIP-62: add hook lineage collection support for S3Hook (#40819): @mobuchowski
Provider common.sql: 1.15.0rc1
- Create SQLAlchemy engine from connection in DB Hook and added autocommit param to insert_rows method (#40669): @dabla
Provider databricks: 6.8.0rc1
- Add DatabricksWorkflowPlugin (#40724): @pankajkoti
Linked issues: - DatabricksPlugin - Fix dag view redirect URL by using url_for redirect (#41040): @pankajkoti
- Fix named parameters templating in Databricks operators (#40864): @boraberke
Linked issues: - [Databricks Provider] Revert PRs #40864 and #40471 (#41050): @pankajkoti
Linked issues:
Provider docker: 3.12.3rc1
Provider elasticsearch: 5.4.2rc1
Provider exasol: 4.5.3rc1
Provider ftp: 3.10.1rc1
Provider google: 10.21.1rc2
- Dynamic import MySqlHook/PostgresHook in CloudSQLDatabaseHook (#41009): @jhongy1994
Linked issues: - Make google_analytics_admin system tests self sufficient (#40951): @moiseenkov
- Fix Custom Training Job operators to accept results without a model in def mode (#40685): @e-galan
Linked issues: - Fix behavior for reattach_state parameter in BigQueryInsertJobOperator (#40664): @VladaZakharova
- Fix cloudsql-query system tests (#41092): @moiseenkov
- Refactor dataproc system tests (#40720): @VladaZakharova
- openlineage: migrate OpenLineage provider to V2 facets. (#39530): @JDarDagran
- Resolve google cloud sql deprecation in tests (#40834): @dirrao
- Update default value for job_name in DataflowConfiguration (#40645): @e-galan
Linked issues:
Provider jdbc: 4.4.0rc1
- Create SQLAlchemy engine from connection in DB Hook and added autocommit param to insert_rows method (#40669): @dabla
- Clean up remaining getattr connection DbApiHook (#40665): @dabla
Provider microsoft.azure: 10.3.0rc1
- Added priority to Azure Container Instances (#40616): @liamalsopp
Linked issues: - Bump minimum version for azure containerinstance. (#40767): @potiuk
Linked issues:
Provider microsoft.mssql: 3.8.0rc1
- Added support for replace option when using insert_rows with MsSqlHook (#40836): @dabla
- Clean up remaining getattr connection DbApiHook (#40665): @dabla
Provider mysql: 5.6.3rc1
Provider odbc: 4.6.3rc1
Provider openlineage: 1.10.0rc1
- openlineage: add AirflowRun on COMPLETE/FAIL events (#40996): @mobuchowski
- openlineage: extend custom_run_facets to on_failed and on_complete (#40953): @kacpermuda
Linked issues: - openlineage: migrate OpenLineage provider to V2 facets. (#39530): @JDarDagran
- openlineage: Add AirflowDagRunFacet to dag runEvents (#40854): @dolfinus
Linked issues: - [AIP-62] Translate AIP-60 URI to OpenLineage (#40173): @kacpermuda
Linked issues: - Ability to add custom facet in OpenLineage events (#38982): @anandhimurali
- AIP-62: add method to common.compat to not force hooks to try/except every time (#40812): @mobuchowski
- openlineage: use airflow provided getters in conf (#40790): @kacpermuda
Linked issues: - openlineage: add config to include 'full' task info based on conf setting (#40589): @mobuchowski
- Add log_url to OpenLineage AirflowRunFacet (#40797): @dolfinus
- openlineage: add deferrable information to task info in airflowRunFacet (#40682): @kacpermuda
Linked issues: - openlineage: adjust DefaultExtractor's on_failure detection for airflow 2.10 fix (#41094): @kacpermuda
Linked issues: - openlineage: make value of slots in attrs.define consistent (#40992): @kacpermuda
Linked issues: - openlineage: Set
slotsto True for facets used in dag level (#40972): @JDarDagran - openlineage: fix / add some task attributes in AirflowRunFacet (#40725): @kacpermuda
Linked issues: - openlineage: replace dt.now with airflow.utils.timezone.utcnow (#40887): @kacpermuda
- openlineage: remove deprecated parentRun facet key (#40681): @kacpermuda
Provider pgvector: 1.2.2rc1
- updating pgvector to !=0.3.0 to avoid recent issue in latest releases(0.3.0) (#40683): @vatsrahul1001
Provider postgres: 5.11.3rc1
Provider qdrant: 1.1.2rc1
Provider sftp: 4.10.3rc1
Provider slack: 8.8.0rc1
- Add the ability to toggle the unfurl behavior for slack notifier (#40694): @armand-sauzay
- Add newly added unfurl args to the docstring of SlackNotifier (#40709): @armand-sauzay
Provider snowflake: 5.6.1rc1
- openlineage: migrate OpenLineage provider to V2 facets. (#39530): @JDarDagran
- Clean up remaining getattr connection DbApiHook (#40665): @dabla
Provider sqlite: 3.8.2rc1
Provider ssh: 3.12.0rc1
- Add on kill to ssh (#40377): @MRLab12
Linked issues:
Provider teradata: 2.5.0rc1
- Implemented Query Band Support for the Teradata provider (#40716): @sc250072
- Clean up remaining getattr connection DbApiHook (#40665): @dabla
Provider ydb: 1.2.0rc1
- ydb provider: add bulk upsert support (#40631): @uzhastik
- Clean up remaining getattr connection DbApiHook (#40665): @dabla
All users involved in the PRs:
@MRLab12 @uzhastik @dstandish @kacpermuda @jedcunningham @mr1holmes @gopidesupavan @moiseenkov @anush008 @dirrao @o-nikolas @vatsrahul1001 @dabla @romsharon98 @e-galan @JDarDagran @ephraimbuddy
Committer
- I acknowledge that I am a maintainer/committer of the Apache Airflow project.
Activity
Updating setup.cfg with RC provider packages on apache/airflow#41237
jhongy1994 commentedon Aug 4, 2024
I checked #41009 (Linked issue #40175) and working as expected :)
moiseenkov commentedon Aug 5, 2024
#40951, #41092 work as expected
anush008 commentedon Aug 5, 2024
Tested and verified working.
VladaZakharova commentedon Aug 5, 2024
#40664 and #40720 work as expected
potiuk commentedon Aug 5, 2024
Checked my changes - both look good !
pankajkoti commentedon Aug 5, 2024
Tested my changes in the Databricks provider, all look good!
dolfinus commentedon Aug 5, 2024
#40854 and #40797 are fine
sc250072 commentedon Aug 5, 2024
#40716 Tested and working fine. All system tests are getting success.
dabla commentedon Aug 5, 2024
Tested following PR with odbc/jdbc/mssql and looking good:
kacpermuda commentedon Aug 5, 2024
I've tested all OpenLineage provider changes and my PRs with some example DAGs, all seems to work fine.
gopidesupavan commentedon Aug 5, 2024
Have tested #40976 , #40765 and #40896. All looks good. thank you.
JDarDagran commentedon Aug 6, 2024
Tested all my changes - all look fine.
uzhastik commentedon Aug 6, 2024
ydb provider works as expected, checked #40631
thank you!
mr1holmes commentedon Aug 6, 2024
Tested #40705
Works as expected.
eladkal commentedon Aug 6, 2024
Thank you everyone. Providers are released.
I invite everyone to help improve providers for the next release, a list of open issues can be found here.