Status of testing Providers that were prepared on June 22, 2024 #40382

Activity

1. eladkal

   added

   kind:metaHigh-level information important to the communityHigh-level information important to the community

   testing statusStatus of testing releasesStatus of testing releases

   on Jun 22, 2024
2. uzhastik commented on Jun 22, 2024

   

   uzhastik

   on Jun 22, 2024

   Contributor

   More actions

   Hello. Could you please exclude from this release ydb provider? It’s too young for that ;) it has known bug related to auth and I’d like to add new feature there later. Thank you in advance.
3. eladkal commented on Jun 22, 2024

   

   eladkal

   on Jun 22, 2024

   Last edited by eladkal

   ContributorAuthor

   More actions

   Hello. Could you please exclude from this release ydb provider? It’s too young for that ;) it has known bug related to auth and I’d like to add new feature there later. Thank you in advance.

   In your PR you marked it as ready to be released
   

   airflow/airflow/providers/ydb/provider.yaml

   Line 24 in 6e5ae26

   state: ready

   Missing features are not a blocker for release. We can always add new features later on.
4. uzhastik commented on Jun 22, 2024

   

   uzhastik

   on Jun 22, 2024

   Contributor

   More actions

   Hello. Could you please exclude from this release ydb provider? It’s too young for that ;) it has known bug related to auth and I’d like to add new feature there later. Thank you in advance.

   In your PR you marked it as ready to be released

   airflow/airflow/providers/ydb/provider.yaml

   Line 24 in 6e5ae26

   state: ready

   Missing features are not a blocker for release. We can always add new features later on.

   To be honest yaml was just copied and bug was found two days ago;) auth info from file does not work, but there is workaround to provide auth info inplace.
5. potiuk commented on Jun 22, 2024

   

   potiuk

   on Jun 22, 2024

   Member

   More actions

   Checked that all changes are there. The common.compat does not have any code yet, so we could skip releasing it, but there is no harm in doing so - having a 1.0.0 version released in PyPI is generally a good idea.
6. gopidesupavan commented on Jun 22, 2024

   

   gopidesupavan

   on Jun 22, 2024

   Member

   More actions

   Tested #40287, working fine. Thanks

   👍React with 👍1Reacted by Elad Kalif
7. rahul-madaan commented on Jun 22, 2024

   

   rahul-madaan

   on Jun 22, 2024

   Contributor

   More actions

   tested #40290, working fine 👍🏻

   👍React with 👍1Reacted by Elad Kalif
8. mehdigati commented on Jun 22, 2024

   

   mehdigati

   on Jun 22, 2024

   Contributor

   More actions

   tested #39955, working fine, thanks !

   👍React with 👍1Reacted by Elad Kalif
9. shahar1 commented on Jun 22, 2024

   

   shahar1

   on Jun 22, 2024

   Contributor

   More actions

   #40142, #40237, #39831 are ok

   👍React with 👍1Reacted by Elad Kalif
10. airflow-oss-bot

    added a commit that references this issue on Jun 23, 2024

    Updating setup.cfg with RC provider packages on apache/airflow#40382

    74594a1
11. jasonspeck commented on Jun 23, 2024

    

    jasonspeck

    on Jun 23, 2024

    Contributor

    More actions

    Confirmed #40206 works as expected

    👍React with 👍1Reacted by Elad Kalif
12. airflow-oss-bot

    added a commit that references this issue on Jun 23, 2024

    Updating setup.cfg with RC provider packages on apache/airflow#40382

    47e699c
13. kylase commented on Jun 23, 2024

    

    kylase

    on Jun 23, 2024

    Contributor

    More actions

    I have tested #39991 and it works as expected.

    [2024-06-23, 04:47:48 UTC] {beam.py:119} WARNING - {"message":"org.apache.beam.runners.dataflow.DataflowRunner - Dataflow SDK version: 2.56.0","severity":"INFO"}
    [2024-06-23, 04:47:49 UTC] {beam.py:119} WARNING - {"message":"org.apache.beam.runners.dataflow.DataflowRunner - To access the Dataflow monitoring console, please navigate to https://console.cloud.google.com/dataflow/jobs/<redacted>","severity":"INFO"}
    [2024-06-23, 04:47:49 UTC] {beam.py:119} WARNING - {"message":"org.apache.beam.runners.dataflow.DataflowRunner - Submitted job: <redacted>","severity":"INFO"}
    [2024-06-23, 04:47:49 UTC] {beam.py:119} WARNING - {"message":"org.apache.beam.runners.dataflow.DataflowRunner - To cancel the job using the \u0027gcloud\u0027 tool, run:\n\u003e gcloud dataflow jobs --project\u003d<redacted> cancel --region\u003<redacted> <redacted>","severity":"INFO"}
    [2024-06-23, 04:47:49 UTC] {beam.py:172} INFO - Process exited with return code: 0
    [2024-06-23, 04:47:49 UTC] {dataflow.py:461} INFO - Start waiting for done.
    [2024-06-23, 04:47:49 UTC] {dataflow.py:403} INFO - Google Cloud DataFlow job <redacted> is state: JOB_STATE_PENDING
    [2024-06-23, 04:47:49 UTC] {dataflow.py:464} INFO - Waiting for done. Sleep 10 s
    [2024-06-23, 04:47:59 UTC] {dataflow.py:403} INFO - Google Cloud DataFlow job <redacted> is state: JOB_STATE_PENDING
    [2024-06-23, 04:47:59 UTC] {dataflow.py:464} INFO - Waiting for done. Sleep 10 s
    [2024-06-23, 04:48:09 UTC] {dataflow.py:403} INFO - Google Cloud DataFlow job <redacted> is state: JOB_STATE_RUNNING
    [2024-06-23, 04:48:09 UTC] {taskinstance.py:1401} INFO - Marking task as SUCCESS. dag_id=<redacted>, task_id=start_streaming, map_index=0, execution_date=20240623T044637, start_date=20240623T044717, end_date=20240623T044809
    [2024-06-23, 04:48:09 UTC] {local_task_job_runner.py:228} INFO - Task exited with return code 0
    [2024-06-23, 04:48:09 UTC] {taskinstance.py:2781} INFO - 0 downstream tasks scheduled from follow-on schedule check
    

    👍React with 👍1Reacted by Elad Kalif
14. dirrao commented on Jun 23, 2024

    

    dirrao

    on Jun 23, 2024

    Contributor

    More actions

    Changes #40253 works as expected.

    👍React with 👍1Reacted by Elad Kalif
15. 13 remaining items

    Load more

16. pankajastro commented on Jun 24, 2024

    

    pankajastro

    on Jun 24, 2024

    Member

    More actions

    #39348 looks good. thank you!

    👍React with 👍1Reacted by Elad Kalif
17. kacpermuda commented on Jun 24, 2024

    

    kacpermuda

    on Jun 24, 2024

    Collaborator

    More actions

    @eladkal please exclude openlineage provider from this wave, and if possible, let's go for rc2.

    #40353 is causing the scheduler crash, I prepared a revert commit in #40402 that should be included in rc2. I also described the bug and provided some logs in #40403, but i guess for now just reverting this will allow us to move forward with the release.

    👍React with 👍3Reacted by Elad Kalif, Tatiana Al-Chueyr and Maciej Obuchowski
18. sc250072 commented on Jun 24, 2024

    

    sc250072

    on Jun 24, 2024

    Contributor

    More actions

    Can you please remove teradata #40378 PR from this 2.3.0 release as we wanted to release compute cluster functionality along with this. We are in process of creating new PR for compute cluster and we would like to release 2.3.0 with these two features as per our roadmap.

    It's of course @eladkal (Release Manager's) decision - but no @satish-chinthanippu , this is not how our provider's release process work. We release every changed provider from main, and we only exclude a provider (not individual PRs) from release if a serious bug has been found. Manipulating and manually modifying stuff during the release is not easy, takes time and effort, might break various parts of the process (like documentation generation, or package preparation, publishing, signing and verification) and introduces serious overhead for ther release process that we don't want.

    We have more than 90 providers and we cannot afford individual treatment and such "custom" approach.

    If there is a bug / regression in the existing functionality that is a blocker - we might remove whole provider from the set of the providers being voted - but that's about all flexibility, and unless there is a bug in Teradata provider, it will be released as-is.

    We do not look at other's roadmaps - this is a bit of price to pay for making the provider a "community managed" one and we've been very clear about the process when we accepted Teradata - one of the reasons why Teradata could choose to release their own provider, was that this could help them to manage their own release roadmap and schedule. Once this is a community provider, we do expect Teradata to keep it updated, with dashboard and fix bugs (in their own interest) but this also means that anyone can contribute changes (and Airflow committers make decisions what goes in and out) but also that we release it together with other providers with the same cadence.

    I hope that explains how it works :). This is not a complaint or being nasty (we appreciate all the work done, system test dashboards, and all the new features you guys, add). It's just the way how we have to manage 90+ providers in release has to have some limitations and structure, and also the governance requirements of the ASF is very clear that once the code is submitted to our repo it has to follow the rules of ASF where only Airflow committers can decide on the code modifications. So I wanted to make it clear.

    Thank you @potiuk for your detailed information. Understood the process and considerations regarding the release. Actually, @eladkal suggested to raise individual PRs for each related functionality implemented in Airflow Teradata Provider to make the review process simpler. So, in line with this suggestion, we thought of raising individual PRs for the two features we have planned for this release.
    We're committed to aligning with the community standards and appreciate the governance framework outlined by ASF.
    Given this understanding, we'll proceed with the new PR for the compute cluster functionality alongside Teradata's provider updates as per the standard release cadence. We'll ensure that our contributions meet the necessary criteria and are compatible with the overall release process.
    Please let us know if there are any specific guidelines or additional steps we should follow as we prepare these updates with multiple PRs for a single release.
19. sc250072 commented on Jun 24, 2024

    

    sc250072

    on Jun 24, 2024

    Contributor

    More actions

    #40378 working as expected.
20. pankajkoti commented on Jun 24, 2024

    

    pankajkoti

    on Jun 24, 2024

    Member

    More actions

    Tested #40013, #39771 and #39941. All works fine. Thank you for the release efforts!

    👍React with 👍1Reacted by Elad Kalif
21. ahidalgob commented on Jun 24, 2024

    

    ahidalgob

    on Jun 24, 2024

    Contributor

    More actions

    #40062 working as expected

    👍React with 👍1Reacted by Elad Kalif
22. e-galan commented on Jun 24, 2024

    

    e-galan

    on Jun 24, 2024

    Contributor

    More actions

    #40278 works as expected

    👍React with 👍1Reacted by Elad Kalif
23. boraberke commented on Jun 24, 2024

    

    boraberke

    on Jun 24, 2024

    Contributor

    More actions

    Tested all of my work (#38868, #39154, #40048, #40086, #40162) and they all work as expected!

    Thanks! 🥂

    👍React with 👍1Reacted by Elad Kalif
24. eladkal commented on Jun 24, 2024

    

    eladkal

    on Jun 24, 2024

    ContributorAuthor

    More actions

    Actually, @eladkal suggested to raise individual PRs for each related functionality implemented in Airflow Teradata Provider to make the review process simpler.

    Which we stand for.
    I do not understand the concern you raised. Merged PR = ready to release.
    What is the problem with releasing it as is?
25. potiuk commented on Jun 24, 2024

    

    potiuk

    on Jun 24, 2024

    Last edited by potiuk

    Member

    More actions

    Which we stand for.
    I do not understand the concern you raised. Merged PR = ready to release.
    What is the problem with releasing it as is?

    Yep. This is all good and I stand for it too. Raising PR <> merging PR. If you wish PR to wait because there is a need to release it together with another - related - PR, the PR can be kept in draft or with an unresolved conversation explaining that it shoud not yet been merged, and other PR might be added on top (based on the first PR) and both could be rebased until both are ready to be released. I believe there was a lack of understanding that "merged" = "ready to release" for providers, so I hope it's now clear.
26. sc250072 commented on Jun 24, 2024

    

    sc250072

    on Jun 24, 2024

    Last edited by sc250072

    Contributor

    More actions

    @potiuk and @eladkal understood. These steps clarifies on the process to follow to release related features with multiple PRs in a single release. Thank you for providing detailed information.
27. sc250072 commented on Jun 24, 2024

    

    sc250072

    on Jun 24, 2024

    Contributor

    More actions

    Actually, @eladkal suggested to raise individual PRs for each related functionality implemented in Airflow Teradata Provider to make the review process simpler.

    Which we stand for. I do not understand the concern you raised. Merged PR = ready to release. What is the problem with releasing it as is?

    Please release it. #40378 Tested and working as expected.
28. dabla commented on Jun 24, 2024

    

    dabla

    on Jun 24, 2024

    Contributor

    More actions

    40301
    
    40300
    
    40297 are all working as expected

    👍React with 👍1Reacted by Elad Kalif
29. eladkal commented on Jun 27, 2024

    

    eladkal

    on Jun 27, 2024

    ContributorAuthor

    More actions

    Thank you everyone. Providers are released.
    openlineage provider is excluded from this wave

    I invite everyone to help improve providers for the next release, a list of open issues can be found here.

    🎉React with 🎉2Reacted by Kacper Muda and Tatiana Al-Chueyr
30. eladkal

    closed this as completedon Jun 27, 2024