Skip to content

Migrate Google example DAGs to new design #22447

Closed
Closed
Migrate Google example DAGs to new design#22447
Labels
AIP-47AIP-47 New Design of System Testskind:metaHigh-level information important to the community

Description

@potiuk
potiuk
opened on Mar 22, 2022

There is a new design of system tests that was introduced by the AIP-47.

All current example dags need to be migrated and converted into system tests, so they can be run in the CI process automatically before releases.

This is an aggregated issue for all example DAGs related to Google provider. It is created to track progress of their migration.

List of paths to example DAGs:

  • airflow/providers/google/leveldb/example_dags/example_leveldb.py
  • airflow/providers/google/firebase/example_dags/example_firestore.py
  • airflow/providers/google/ads/example_dags/example_ads.py
  • airflow/providers/google/suite/example_dags/example_sql_to_sheets.py
  • airflow/providers/google/suite/example_dags/example_gcs_to_sheets.py
  • airflow/providers/google/suite/example_dags/example_gcs_to_gdrive.py
  • airflow/providers/google/suite/example_dags/example_sheets.py
  • airflow/providers/google/marketing_platform/example_dags/example_analytics.py
  • airflow/providers/google/marketing_platform/example_dags/example_display_video.py
  • airflow/providers/google/marketing_platform/example_dags/example_campaign_manager.py
  • airflow/providers/google/marketing_platform/example_dags/example_search_ads.py
  • airflow/providers/google/cloud/example_dags/example_bigquery_queries.py
  • airflow/providers/google/cloud/example_dags/example_gcs_to_sftp.py
  • airflow/providers/google/cloud/example_dags/example_oracle_to_gcs.py
  • airflow/providers/google/cloud/example_dags/example_compute_igm.py
  • airflow/providers/google/cloud/example_dags/example_automl_vision_classification.py
  • airflow/providers/google/cloud/example_dags/example_datacatalog.py
  • airflow/providers/google/cloud/example_dags/example_bigquery_to_mssql.py
  • airflow/providers/google/cloud/example_dags/example_mssql_to_gcs.py
  • airflow/providers/google/cloud/example_dags/example_mysql_to_gcs.py
  • airflow/providers/google/cloud/example_dags/example_bigquery_to_bigquery.py
  • airflow/providers/google/cloud/example_dags/example_natural_language.py
  • airflow/providers/google/cloud/example_dags/example_life_sciences.py
  • airflow/providers/google/cloud/example_dags/example_automl_nl_text_extraction.py
  • airflow/providers/google/cloud/example_dags/example_automl_video_intelligence_classification.py
  • airflow/providers/google/cloud/example_dags/example_trino_to_gcs.py
  • airflow/providers/google/cloud/example_dags/example_bigquery_transfer.py
  • airflow/providers/google/cloud/example_dags/example_cloud_storage_transfer_service_gcp.py
  • airflow/providers/google/cloud/example_dags/example_translate.py
  • airflow/providers/google/cloud/example_dags/example_local_to_gcs.py
  • airflow/providers/google/cloud/example_dags/example_compute_ssh.py
  • airflow/providers/google/cloud/example_dags/example_dataflow_sql.py
  • airflow/providers/google/cloud/example_dags/example_text_to_speech.py
  • airflow/providers/google/cloud/example_dags/example_presto_to_gcs.py
  • airflow/providers/google/cloud/example_dags/example_looker.py
  • airflow/providers/google/cloud/example_dags/example_sftp_to_gcs.py
  • airflow/providers/google/cloud/example_dags/example_facebook_ads_to_gcs.py
  • airflow/providers/google/cloud/example_dags/example_gcs_to_bigquery.py
  • airflow/providers/google/cloud/example_dags/example_postgres_to_gcs.py
  • airflow/providers/google/cloud/example_dags/example_spanner.py
  • airflow/providers/google/cloud/example_dags/example_gdrive_to_gcs.py
  • airflow/providers/google/cloud/example_dags/example_dlp.py
  • airflow/providers/google/cloud/example_dags/example_bigquery_dts.py
  • airflow/providers/google/cloud/example_dags/example_datastore.py
  • airflow/providers/google/cloud/example_dags/example_bigquery_operations.py
  • airflow/providers/google/cloud/example_dags/example_automl_nl_text_classification.py
  • airflow/providers/google/cloud/example_dags/example_bigquery_to_gcs.py
  • airflow/providers/google/cloud/example_dags/example_automl_vision_object_detection.py
  • airflow/providers/google/cloud/example_dags/example_bigtable.py
  • airflow/providers/google/cloud/example_dags/example_gcs_to_local.py
  • airflow/providers/google/cloud/example_dags/example_vertex_ai.py
  • airflow/providers/google/cloud/example_dags/example_automl_video_intelligence_tracking.py
  • airflow/providers/google/cloud/example_dags/example_gcs_timespan_file_transform.py
  • airflow/providers/google/cloud/example_dags/example_dataproc_metastore.py
  • airflow/providers/google/cloud/example_dags/example_salesforce_to_gcs.py
  • airflow/providers/google/cloud/example_dags/example_functions.py
  • airflow/providers/google/cloud/example_dags/example_compute.py
  • airflow/providers/google/cloud/example_dags/example_sheets_to_gcs.py
  • airflow/providers/google/cloud/example_dags/example_dataflow.py
  • airflow/providers/google/cloud/example_dags/example_datafusion.py
  • airflow/providers/google/cloud/example_dags/example_cloud_sql_query.py
  • airflow/providers/google/cloud/example_dags/example_translate_speech.py
  • airflow/providers/google/cloud/example_dags/example_automl_tables.py
  • airflow/providers/google/cloud/example_dags/example_cloud_memorystore.py
  • airflow/providers/google/cloud/example_dags/example_bigquery_sensors.py
  • airflow/providers/google/cloud/example_dags/example_gcs.py
  • airflow/providers/google/cloud/example_dags/example_mlengine.py
  • airflow/providers/google/cloud/example_dags/example_dataprep.py
  • airflow/providers/google/cloud/example_dags/example_cloud_storage_transfer_service_aws.py
  • airflow/providers/google/cloud/example_dags/example_pubsub.py
  • airflow/providers/google/cloud/example_dags/example_gcs_to_gcs.py
  • airflow/providers/google/cloud/example_dags/example_s3_to_gcs.py
  • airflow/providers/google/cloud/example_dags/example_gdrive_to_local.py
  • airflow/providers/google/cloud/example_dags/example_azure_fileshare_to_gcs.py
  • airflow/providers/google/cloud/example_dags/example_workflows.py
  • airflow/providers/google/cloud/example_dags/example_kubernetes_engine.py
  • airflow/providers/google/cloud/example_dags/example_dataproc.py
  • airflow/providers/google/cloud/example_dags/example_speech_to_text.py
  • airflow/providers/google/cloud/example_dags/example_cloud_composer.py
  • airflow/providers/google/cloud/example_dags/example_automl_translation.py
  • airflow/providers/google/cloud/example_dags/example_stackdriver.py
  • airflow/providers/google/cloud/example_dags/example_tasks.py
  • airflow/providers/google/cloud/example_dags/example_calendar_to_gcs.py
  • airflow/providers/google/cloud/example_dags/example_video_intelligence.py
  • airflow/providers/google/cloud/example_dags/example_cloud_build.py
  • airflow/providers/google/cloud/example_dags/example_dataflow_flex_template.py
  • airflow/providers/google/cloud/example_dags/example_automl_nl_text_sentiment.py
  • airflow/providers/google/cloud/example_dags/example_cloud_sql.py
  • airflow/providers/google/cloud/example_dags/example_vision.py

In progress:

  • airflow/providers/google/cloud/example_dags/example_bigquery_to_bigquery.py
  • airflow/providers/google/cloud/example_dags/example_dataflow.py
  • airflow/providers/google/cloud/example_dags/example_datafusion.py
  • airflow/providers/google/cloud/example_dags/example_dataproc.py
  • airflow/providers/google/cloud/example_dags/example_dataproc_metastore.py
  • airflow/providers/google/cloud/example_dags/example_datastore.py
  • airflow/providers/google/cloud/example_dags/example_gcs.py
  • airflow/providers/google/cloud/example_dags/example_gcs_timespan_file_transform.py
  • airflow/providers/google/cloud/example_dags/example_gcs_to_gcs.py
  • airflow/providers/google/cloud/example_dags/example_gcs_to_local.py
  • airflow/providers/google/cloud/example_dags/example_local_to_gcs.py

Activity

  1. potiuk
    added
    AIP-47AIP-47 New Design of System Tests
    kind:metaHigh-level information important to the community
    on Mar 22, 2022

  2. mnojek commented on Mar 24, 2022

    @mnojek
    mnojek
    on Mar 24, 2022
    Contributor

    Hey, please update the checklist.

    These example DAGs are being removed by PR #22311:

    • airflow/providers/google/cloud/example_dags/example_bigquery_operations.py
    • airflow/providers/google/cloud/example_dags/example_bigquery_queries.py
    • airflow/providers/google/cloud/example_dags/example_bigquery_sensors.py

    And these are WIP (Work In Progress) (not sure how we are going to mark them, but it would be good to notify others so that we will avoid work duplication):

    • airflow/providers/google/cloud/example_dags/example_bigquery_to_bigquery.py
    • airflow/providers/google/cloud/example_dags/example_dataflow.py
    • airflow/providers/google/cloud/example_dags/example_datafusion.py
    • airflow/providers/google/cloud/example_dags/example_dataproc.py
    • airflow/providers/google/cloud/example_dags/example_dataproc_metastore.py
    • airflow/providers/google/cloud/example_dags/example_datastore.py
    • airflow/providers/google/cloud/example_dags/example_gcs.py
    • airflow/providers/google/cloud/example_dags/example_gcs_timespan_file_transform.py
    • airflow/providers/google/cloud/example_dags/example_gcs_to_gcs.py
    • airflow/providers/google/cloud/example_dags/example_gcs_to_local.py
    • airflow/providers/google/cloud/example_dags/example_local_to_gcs.py

  3. potiuk commented on Mar 28, 2022

    @potiuk
    potiuk
    on Mar 28, 2022
    MemberAuthor

    Updated

  4. potiuk commented on Apr 28, 2022

    @potiuk
    potiuk
    on Apr 28, 2022
    MemberAuthor

    BTW. I will automate refreshing the list shortly :) @bhirsz :)

  5. chethanuk commented on Jun 2, 2022

    @chethanuk
    chethanuk
    on Jun 2, 2022
    Contributor

    Hi @potiuk Can we update the checklist if and when possible [So Can know which needs to be done, which are WIP]. Thanks :)

  6. potiuk commented on Jun 3, 2022

    @potiuk
    potiuk
    on Jun 3, 2022
    MemberAuthor

    Sure. I merge all the green ones and re-run the script to update the checklists

  7. chethanuk
    added 7 commits that reference this issue on Jun 5, 2022

    AIP-47 - Migrate google leveldb DAGs to new design #apache#22447

    efe78db

    AIP-47 - Migrate google marketing DAGs to new design apache#22447

    945b20c
  8. potiuk
    added a commit that references this issue on Jun 6, 2022

    Migrate Google leveldb system tests to new design AIP-47 (#24255)

    203fe71

  9. 35 remaining items

  10. potiuk
    added 11 commits that reference this issue on Jul 12, 2022

    Migrate Google ads example to new design AIP-47 (#24941)

    a038b52

    Migrate Google sheets example to new design AIP-47 (#24975)

    a13c51e

    Migrate Google analytics example to new design AIP-47 (#25006)

    9b7e4a7

    Migrate Google campaign manager example to new design AIP-47 (#25069)

    4473b27

    Migrate Google example gcs_to_sftp to new design AIP-47 (#25107)

    c0ce8a8

    Migrate Google example automl_vision to new design AIP-47 (#25152)

    3a80b36

    Migrate Google example compute_igm to new design AIP-47 (#25132)

    9644451

    Migrate Google example bigquery_to_mssql to new design AIP-47 (#25174)

    66b3ca1

    Delete redundant system test bigquery_to_bigquery (#25261)

    b288bf4

    Migrate Google example natural_language to new design AIP-47 (#25262)

    10c9a36

    Migrate Google example life_sciences to new design AIP-47 (#25264)

    28db8c1
  11. potiuk
    added 2 commits that reference this issue on Aug 2, 2022

    Migrate Google example automl_nl_text_extraction to new design AIP-47 (…

    d290002

    Migrate Google example trino_to_gcs to new design AIP-47 (#25420)

    c8cacad

  12. eladkal commented on Jun 2, 2024

    @eladkal
    eladkal
    on Jun 2, 2024
    Contributor

    Do we still need this tracking issue?

  13. potiuk
    closed this as completedon Jun 2, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    AIP-47AIP-47 New Design of System Testskind:metaHigh-level information important to the community

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions