Repository navigation
Resolve AirflowProviderDeprecationWarning in providers system tests #39485
Description
Body
Time to time we deprecate some attributes/arguments and even entire Operators, however maintainer might forget to change it as result system tests which use as DAG/Operators example might suggest to use deprecated stuff, this should not be a case after the #39468 PR
The purpose of this task is to fixing existing usage of deprecated stuff in Airflow Providers system tests
This one is a special case for #38642
How to check that deprecations resolved after the changes
For example you make a changes in tests/system/providers/google/cloud/gcs/example_gcs_to_gcs.py and want to check that it resolved
-
Remove reference to specific system test from that list
https://github.com/apache/airflow/blob/694d3038fca7c6127feebc370255707aceaddb01/tests/always/test_example_dags.py#L50-L80 -
Run test case
tests/always/test_example_dags.py::test_should_be_importable, you might specify relative path as pytest parametrise by define it inside of[]brackets
pytest 'tests/always/test_example_dags.py::test_should_be_importable[tests/system/providers/google/cloud/gcs/example_gcs_to_gcs.py]'Warning
System tests also uses as an example in providers documentation, so if you change system test please make sure that you also remove / change information about deprecated Operators / attributes / arguments in to the documentation.
Provider amazon
airflow/providers/amazon/aws/operators/ecs.py:498:
airflow.exceptions.AirflowProviderDeprecationWarning:Call to deprecated function (or staticmethod) region. (`region` is deprecated and will be removed in the future. Please use `region_name` instead.)
tests/system/providers/amazon/aws/example_eks_with_nodegroups.py:124:
airflow.exceptions.AirflowProviderDeprecationWarning:You have not set parameter `on_finish_action` in class EksPodOperator. Currently the default for this parameter is `keep_pod` but in a future release the default will be changed to `delete_pod`. To ensure pods are not deleted in the future you will need to set `on_finish_action=keep_pod` explicitly.
tests/system/providers/amazon/aws/example_emr.py:143:
airflow.exceptions.AirflowProviderDeprecationWarning:The parameter waiter_check_interval_seconds has been deprecated to standardize naming conventions. Please use waiter_delay instead. In the future this will default to None and defer to the waiter's default value.
tests/system/providers/amazon/aws/example_emr_notebook_execution.py:54:
airflow.exceptions.AirflowProviderDeprecationWarning:The parameter waiter_check_interval_seconds has been deprecated to standardize naming conventions. Please use waiter_delay instead. In the future this will default to None and defer to the waiter's default value.
tests/system/providers/amazon/aws/example_emr_notebook_execution.py:54:
airflow.exceptions.AirflowProviderDeprecationWarning:The parameter waiter_countdown has been deprecated to standardize naming conventions. Please use waiter_max_attempts instead. In the future this will default to None and defer to the waiter's default value.
tests/system/providers/amazon/aws/example_emr_notebook_execution.py:75:
airflow.exceptions.AirflowProviderDeprecationWarning:The parameter waiter_check_interval_seconds has been deprecated to standardize naming conventions. Please use waiter_delay instead. In the future this will default to None and defer to the waiter's default value.
tests/system/providers/amazon/aws/example_emr_notebook_execution.py:75:
airflow.exceptions.AirflowProviderDeprecationWarning:The parameter waiter_countdown has been deprecated to standardize naming conventions. Please use waiter_max_attempts instead. In the future this will default to None and defer to the waiter's default value.
tests/system/providers/amazon/aws/example_emr_notebook_execution.py:87:
airflow.exceptions.AirflowProviderDeprecationWarning:The parameter waiter_check_interval_seconds has been deprecated to standardize naming conventions. Please use waiter_delay instead. In the future this will default to None and defer to the waiter's default value.
tests/system/providers/amazon/aws/example_emr_notebook_execution.py:87:
airflow.exceptions.AirflowProviderDeprecationWarning:The parameter waiter_countdown has been deprecated to standardize naming conventions. Please use waiter_max_attempts instead. In the future this will default to None and defer to the waiter's default value.
Provider apache.drill
tests/system/providers/apache/drill/example_drill_dag.py:41:
airflow.exceptions.AirflowProviderDeprecationWarning:Call to deprecated class DrillOperator. (Please use `airflow.providers.common.sql.operators.sql.SQLExecuteQueryOperator`.)
Provider dbt.cloud
tests/system/providers/dbt/cloud/example_dbt_cloud.py:82:
airflow.exceptions.AirflowProviderDeprecationWarning:Call to deprecated class DbtCloudJobRunAsyncSensor. (Class `DbtCloudJobRunAsyncSensor` is deprecated and will be removed in a future release. Please use `DbtCloudJobRunSensor` and set `deferrable` attribute to `True` instead)
Provider docker
airflow/providers/docker/operators/docker_swarm.py:118:
airflow.exceptions.AirflowProviderDeprecationWarning:bool value for `auto_remove` is deprecated and will be removed in the future. Please use 'never', 'success', or 'force' instead
Provider google
Important
Please mention @VladaZakharova in PR if you would like to to make changes in Google Provider System Tests. See: #39485 (comment)
tests/system/providers/google/cloud/azure/example_azure_fileshare_to_gcs.py:56:
airflow.exceptions.AirflowProviderDeprecationWarning:Use 'directory_path' instead of 'directory_name'.
tests/system/providers/google/cloud/bigquery/example_bigquery_operations.py:66:
airflow.exceptions.AirflowProviderDeprecationWarning:Passing table parameters via keywords arguments will be deprecated. Please provide table definition using `table_resource` parameter.
tests/system/providers/google/cloud/bigquery/example_bigquery_sensors.py:101:
airflow.exceptions.AirflowProviderDeprecationWarning:Call to deprecated class BigQueryTableExistenceAsyncSensor. (Class `BigQueryTableExistenceAsyncSensor` is deprecated and will be removed in a future release. Please use `BigQueryTableExistenceSensor` and set `deferrable` attribute to `True` instead)
tests/system/providers/google/cloud/bigquery/example_bigquery_sensors.py:141:
airflow.exceptions.AirflowProviderDeprecationWarning:Call to deprecated class BigQueryTableExistencePartitionAsyncSensor. (Class `BigQueryTableExistencePartitionAsyncSensor` is deprecated and will be removed in a future release. Please use `BigQueryTablePartitionExistenceSensor` and set `deferrable` attribute to `True` instead)
airflow/providers/google/cloud/operators/kubernetes_engine.py:342:
airflow.exceptions.AirflowProviderDeprecationWarning:The body field 'initial_node_count' is deprecated. Use 'node_pool.initial_node_count' instead.
tests/system/providers/google/cloud/gcs/example_gcs_sensor.py:144:
airflow.exceptions.AirflowProviderDeprecationWarning:Call to deprecated class GCSObjectExistenceAsyncSensor. (Class `GCSObjectExistenceAsyncSensor` is deprecated and will be removed in a future release. Please use `GCSObjectExistenceSensor` and set `deferrable` attribute to `True` instead)
tests/system/providers/google/cloud/gcs/example_gcs_to_gcs.py:169:
airflow.exceptions.AirflowProviderDeprecationWarning:Usage of wildcard (*) in 'source_object' is deprecated, utilize 'match_glob' instead
tests/system/providers/google/cloud/gcs/example_gcs_to_gcs.py:189:
airflow.exceptions.AirflowProviderDeprecationWarning:Usage of 'delimiter' is deprecated, please use 'match_glob' instead
airflow/providers/google/cloud/operators/kubernetes_engine.py:342:
airflow.exceptions.AirflowProviderDeprecationWarning:The body field 'initial_node_count' is deprecated. Use 'node_pool.initial_node_count' instead.
airflow/providers/google/cloud/operators/kubernetes_engine.py:342:
airflow.exceptions.AirflowProviderDeprecationWarning:The body field 'initial_node_count' is deprecated. Use 'node_pool.initial_node_count' instead.
airflow/providers/google/cloud/operators/kubernetes_engine.py:342:
airflow.exceptions.AirflowProviderDeprecationWarning:The body field 'initial_node_count' is deprecated. Use 'node_pool.initial_node_count' instead.
airflow/providers/google/cloud/operators/kubernetes_engine.py:342:
airflow.exceptions.AirflowProviderDeprecationWarning:The body field 'initial_node_count' is deprecated. Use 'node_pool.initial_node_count' instead.
airflow/providers/google/cloud/operators/kubernetes_engine.py:342:
airflow.exceptions.AirflowProviderDeprecationWarning:The body field 'initial_node_count' is deprecated. Use 'node_pool.initial_node_count' instead.
tests/system/providers/google/cloud/life_sciences/example_life_sciences.py:105:
airflow.exceptions.AirflowProviderDeprecationWarning:Call to deprecated class LifeSciencesRunPipelineOperator. (Consider using Google Cloud Batch Operators instead.The Life Sciences API (beta) will be discontinued on July 8, 2025 in favor of Google Cloud Batch.)
tests/system/providers/google/cloud/life_sciences/example_life_sciences.py:113:
airflow.exceptions.AirflowProviderDeprecationWarning:Call to deprecated class LifeSciencesRunPipelineOperator. (Consider using Google Cloud Batch Operators instead.The Life Sciences API (beta) will be discontinued on July 8, 2025 in favor of Google Cloud Batch.)
tests/system/providers/google/marketing_platform/example_analytics.py:55:
airflow.exceptions.AirflowProviderDeprecationWarning:Call to deprecated class GoogleAnalyticsListAccountsOperator. (The `GoogleAnalyticsListAccountsOperator` class is deprecated, please use `GoogleAnalyticsAdminListAccountsOperator` instead.)
tests/system/providers/google/marketing_platform/example_analytics.py:59:
airflow.exceptions.AirflowProviderDeprecationWarning:Call to deprecated class GoogleAnalyticsGetAdsLinkOperator. (The `GoogleAnalyticsGetAdsLinkOperator` class is deprecated, please use `GoogleAnalyticsAdminGetGoogleAdsLinkOperator` instead.)
tests/system/providers/google/marketing_platform/example_analytics.py:68:
airflow.exceptions.AirflowProviderDeprecationWarning:Call to deprecated class GoogleAnalyticsRetrieveAdsLinksListOperator. (The `GoogleAnalyticsRetrieveAdsLinksListOperator` class is deprecated, please use `GoogleAnalyticsAdminListGoogleAdsLinksOperator` instead.)
tests/system/providers/google/marketing_platform/example_analytics.py:73:
airflow.exceptions.AirflowProviderDeprecationWarning:Call to deprecated class GoogleAnalyticsDataImportUploadOperator. (The `GoogleAnalyticsDataImportUploadOperator` class is deprecated, please use `GoogleAnalyticsAdminCreateDataStreamOperator` instead.)
tests/system/providers/google/marketing_platform/example_analytics.py:82:
airflow.exceptions.AirflowProviderDeprecationWarning:Call to deprecated class GoogleAnalyticsDeletePreviousDataUploadsOperator. (The `GoogleAnalyticsDeletePreviousDataUploadsOperator` class is deprecated, please use `GoogleAnalyticsAdminDeleteDataStreamOperator` instead.)
Provider jdbc
tests/system/providers/jdbc/example_jdbc_queries.py:43:
airflow.exceptions.AirflowProviderDeprecationWarning:Call to deprecated class JdbcOperator. (Please use `airflow.providers.common.sql.operators.sql.SQLExecuteQueryOperator`.)
tests/system/providers/jdbc/example_jdbc_queries.py:52:
airflow.exceptions.AirflowProviderDeprecationWarning:Call to deprecated class JdbcOperator. (Please use `airflow.providers.common.sql.operators.sql.SQLExecuteQueryOperator`.)
Provider microsoft.mssql
tests/system/providers/microsoft/mssql/example_mssql.py:115:
airflow.exceptions.AirflowProviderDeprecationWarning:Call to deprecated class MsSqlOperator. (Please use `airflow.providers.common.sql.operators.sql.SQLExecuteQueryOperator`.Also, you can provide `hook_params={'schema': <database>}`.)
tests/system/providers/microsoft/mssql/example_mssql.py:123:
airflow.exceptions.AirflowProviderDeprecationWarning:Call to deprecated class MsSqlOperator. (Please use `airflow.providers.common.sql.operators.sql.SQLExecuteQueryOperator`.Also, you can provide `hook_params={'schema': <database>}`.)
tests/system/providers/microsoft/mssql/example_mssql.py:131:
airflow.exceptions.AirflowProviderDeprecationWarning:Call to deprecated class MsSqlOperator. (Please use `airflow.providers.common.sql.operators.sql.SQLExecuteQueryOperator`.Also, you can provide `hook_params={'schema': <database>}`.)
tests/system/providers/microsoft/mssql/example_mssql.py:53:
airflow.exceptions.AirflowProviderDeprecationWarning:Call to deprecated class MsSqlOperator. (Please use `airflow.providers.common.sql.operators.sql.SQLExecuteQueryOperator`.Also, you can provide `hook_params={'schema': <database>}`.)
tests/system/providers/microsoft/mssql/example_mssql.py:89:
airflow.exceptions.AirflowProviderDeprecationWarning:Call to deprecated class MsSqlOperator. (Please use `airflow.providers.common.sql.operators.sql.SQLExecuteQueryOperator`.Also, you can provide `hook_params={'schema': <database>}`.)
tests/system/providers/microsoft/mssql/example_mssql.py:98:
airflow.exceptions.AirflowProviderDeprecationWarning:Call to deprecated class MsSqlOperator. (Please use `airflow.providers.common.sql.operators.sql.SQLExecuteQueryOperator`.Also, you can provide `hook_params={'schema': <database>}`.)
Provider mysql
tests/system/providers/mysql/example_mysql.py:42:
airflow.exceptions.AirflowProviderDeprecationWarning:Call to deprecated class MySqlOperator. (Please use `airflow.providers.common.sql.operators.sql.SQLExecuteQueryOperator`.Also, you can provide `hook_params={'schema': <database>}`.)
tests/system/providers/mysql/example_mysql.py:50:
airflow.exceptions.AirflowProviderDeprecationWarning:Call to deprecated class MySqlOperator. (Please use `airflow.providers.common.sql.operators.sql.SQLExecuteQueryOperator`.Also, you can provide `hook_params={'schema': <database>}`.)
Provider postgres
tests/system/providers/postgres/example_postgres.py:41:
airflow.exceptions.AirflowProviderDeprecationWarning:Call to deprecated class PostgresOperator. (Please use `airflow.providers.common.sql.operators.sql.SQLExecuteQueryOperator`.Also, you can provide `hook_params={'schema': <database>}`.)
tests/system/providers/postgres/example_postgres.py:54:
airflow.exceptions.AirflowProviderDeprecationWarning:Call to deprecated class PostgresOperator. (Please use `airflow.providers.common.sql.operators.sql.SQLExecuteQueryOperator`.Also, you can provide `hook_params={'schema': <database>}`.)
tests/system/providers/postgres/example_postgres.py:69:
airflow.exceptions.AirflowProviderDeprecationWarning:Call to deprecated class PostgresOperator. (Please use `airflow.providers.common.sql.operators.sql.SQLExecuteQueryOperator`.Also, you can provide `hook_params={'schema': <database>}`.)
tests/system/providers/postgres/example_postgres.py:72:
airflow.exceptions.AirflowProviderDeprecationWarning:Call to deprecated class PostgresOperator. (Please use `airflow.providers.common.sql.operators.sql.SQLExecuteQueryOperator`.Also, you can provide `hook_params={'schema': <database>}`.)
Provider snowflake
tests/system/providers/snowflake/example_snowflake.py:53:
airflow.exceptions.AirflowProviderDeprecationWarning:Call to deprecated class SnowflakeOperator. (This class is deprecated. Please use `airflow.providers.common.sql.operators.sql.SQLExecuteQueryOperator`. Also, you can provide `hook_params={'warehouse': <warehouse>, 'database': <database>, 'role': <role>, 'schema': <schema>, 'authenticator': <authenticator>,'session_parameters': <session_parameters>}`.)
tests/system/providers/snowflake/example_snowflake.py:55:
airflow.exceptions.AirflowProviderDeprecationWarning:Call to deprecated class SnowflakeOperator. (This class is deprecated. Please use `airflow.providers.common.sql.operators.sql.SQLExecuteQueryOperator`. Also, you can provide `hook_params={'warehouse': <warehouse>, 'database': <database>, 'role': <role>, 'schema': <schema>, 'authenticator': <authenticator>,'session_parameters': <session_parameters>}`.)
tests/system/providers/snowflake/example_snowflake.py:61:
airflow.exceptions.AirflowProviderDeprecationWarning:Call to deprecated class SnowflakeOperator. (This class is deprecated. Please use `airflow.providers.common.sql.operators.sql.SQLExecuteQueryOperator`. Also, you can provide `hook_params={'warehouse': <warehouse>, 'database': <database>, 'role': <role>, 'schema': <schema>, 'authenticator': <authenticator>,'session_parameters': <session_parameters>}`.)
tests/system/providers/snowflake/example_snowflake.py:63:
airflow.exceptions.AirflowProviderDeprecationWarning:Call to deprecated class SnowflakeOperator. (This class is deprecated. Please use `airflow.providers.common.sql.operators.sql.SQLExecuteQueryOperator`. Also, you can provide `hook_params={'warehouse': <warehouse>, 'database': <database>, 'role': <role>, 'schema': <schema>, 'authenticator': <authenticator>,'session_parameters': <session_parameters>}`.)
tests/system/providers/snowflake/example_snowflake.py:69:
airflow.exceptions.AirflowProviderDeprecationWarning:Call to deprecated class SnowflakeOperator. (This class is deprecated. Please use `airflow.providers.common.sql.operators.sql.SQLExecuteQueryOperator`. Also, you can provide `hook_params={'warehouse': <warehouse>, 'database': <database>, 'role': <role>, 'schema': <schema>, 'authenticator': <authenticator>,'session_parameters': <session_parameters>}`.)
Provider sqlite
tests/system/providers/sqlite/example_sqlite.py:48:
airflow.exceptions.AirflowProviderDeprecationWarning:Call to deprecated class SqliteOperator. (Please use `airflow.providers.common.sql.operators.sql.SQLExecuteQueryOperator`.)
tests/system/providers/sqlite/example_sqlite.py:80:
airflow.exceptions.AirflowProviderDeprecationWarning:Call to deprecated class SqliteOperator. (Please use `airflow.providers.common.sql.operators.sql.SQLExecuteQueryOperator`.)
Provider trino
tests/system/providers/trino/example_trino.py:43:
airflow.exceptions.AirflowProviderDeprecationWarning:Call to deprecated class TrinoOperator. (Please use `airflow.providers.common.sql.operators.sql.SQLExecuteQueryOperator`.)
tests/system/providers/trino/example_trino.py:48:
airflow.exceptions.AirflowProviderDeprecationWarning:Call to deprecated class TrinoOperator. (Please use `airflow.providers.common.sql.operators.sql.SQLExecuteQueryOperator`.)
tests/system/providers/trino/example_trino.py:57:
airflow.exceptions.AirflowProviderDeprecationWarning:Call to deprecated class TrinoOperator. (Please use `airflow.providers.common.sql.operators.sql.SQLExecuteQueryOperator`.)
tests/system/providers/trino/example_trino.py:63:
airflow.exceptions.AirflowProviderDeprecationWarning:Call to deprecated class TrinoOperator. (Please use `airflow.providers.common.sql.operators.sql.SQLExecuteQueryOperator`.)
tests/system/providers/trino/example_trino.py:72:
airflow.exceptions.AirflowProviderDeprecationWarning:Call to deprecated class TrinoOperator. (Please use `airflow.providers.common.sql.operators.sql.SQLExecuteQueryOperator`.)
tests/system/providers/trino/example_trino.py:78:
airflow.exceptions.AirflowProviderDeprecationWarning:Call to deprecated class TrinoOperator. (Please use `airflow.providers.common.sql.operators.sql.SQLExecuteQueryOperator`.)
Provider weaviate
tests/system/providers/weaviate/example_weaviate_cohere.py:81:
airflow.exceptions.AirflowProviderDeprecationWarning:Passing 'input_json' to WeaviateIngestOperator is deprecated and you should use 'input_data' instead
tests/system/providers/weaviate/example_weaviate_openai.py:79:
airflow.exceptions.AirflowProviderDeprecationWarning:Passing 'input_json' to WeaviateIngestOperator is deprecated and you should use 'input_data' instead
tests/system/providers/weaviate/example_weaviate_operator.py:120:
airflow.exceptions.AirflowProviderDeprecationWarning:Passing 'input_json' to WeaviateIngestOperator is deprecated and you should use 'input_data' instead
tests/system/providers/weaviate/example_weaviate_operator.py:130:
airflow.exceptions.AirflowProviderDeprecationWarning:Passing 'input_json' to WeaviateIngestOperator is deprecated and you should use 'input_data' instead
tests/system/providers/weaviate/example_weaviate_operator.py:230:
airflow.exceptions.AirflowProviderDeprecationWarning:Passing 'input_json' to WeaviateIngestOperator is deprecated and you should use 'input_data' instead
tests/system/providers/weaviate/example_weaviate_operator.py:240:
airflow.exceptions.AirflowProviderDeprecationWarning:Passing 'input_json' to WeaviateIngestOperator is deprecated and you should use 'input_data' instead
Committer
- I acknowledge that I am a maintainer/committer of the Apache Airflow project.
Activity
19 remaining items
Bowrna commentedon May 21, 2024
oops @dirrao I am sorry I raised a PR without noticing it :(
Bowrna commentedon May 21, 2024
Raising PR for Snowflake, Drill and SQLite too!
Edit: Raising for JDBC, MSSQL too
Bowrna commentedon May 21, 2024
The open ones in this task are
tests/system/providers/google/cloud/life_sciences/example_life_sciences.py
tests/system/providers/google/marketing_platform/example_analytics.py
tests/system/providers/google/cloud/bigquery/example_bigquery_operations.py
tests/system/providers/amazon/aws/example_ecs_fargate.py
tests/system/providers/amazon/aws/example_emr_notebook_execution.py
tests/system/providers/amazon/aws/example_emr.py
dirrao commentedon May 21, 2024
No issues. I will drop mine.
VladaZakharova commentedon May 22, 2024
Hi @Taragolis ! Thank you for opening this issue to improve system tests for Google provider :)
From our team's side we can offer checking all the PR and also running them in Google env to check if they work good with new changes.
I would like to ask you only update the description of this issue and ask people who will create PRs to mention me in there, so we will be able to see all new changes. Thank you!
Resolve DbtCloudJobRunAsyncSensor deprecation in system tests (apache…
Resolve deprecation warnings in Azure FileShare-to-GCS tests (apache#…
kunaljubce commentedon Mar 6, 2025
@Taragolis Validated that
AirflowProviderDeprecationWarningis not present in any of the remaining provider system tests:providers/mysql/tests/system/mysql/example_mysql.pyproviders/microsoft/mssql/tests/system/microsoft/mssql/example_mssql.pyproviders/jdbc/tests/system/jdbc/example_jdbc_queries.pyproviders/google/tests/system/google/marketing_platform/example_analytics_admin.pyproviders/google/tests/system/google/cloud/life_sciences/example_life_sciences.pyproviders/google/tests/system/google/cloud/kubernetes_engine/example_kubernetes_engine_resource.pyproviders/google/tests/system/google/cloud/kubernetes_engine/example_kubernetes_engine_kueue.pyproviders/google/tests/system/google/cloud/kubernetes_engine/example_kubernetes_engine_job.pyproviders/google/tests/system/google/cloud/kubernetes_engine/example_kubernetes_engine_async.pyproviders/google/tests/system/google/cloud/kubernetes_engine/example_kubernetes_engine.pyproviders/google/tests/system/google/cloud/gcs/example_gcs_to_gcs.pyproviders/google/tests/system/google/cloud/gcs/example_gcs_sensor.pyproviders/google/tests/system/google/cloud/dataproc/example_dataproc_gke.pyproviders/google/tests/system/google/cloud/bigquery/example_bigquery_sensors.pyproviders/google/tests/system/google/cloud/bigquery/example_bigquery_operations.pyproviders/apache/drill/tests/system/apache/drill/example_drill_dag.pyproviders/amazon/tests/system/amazon/aws/example_emr_notebook_execution.pySeems we're good to close this issue?
potiuk commentedon Mar 6, 2025
let's