Repository navigation
Status of testing of Apache Airflow 2.7.1rc2 #34065
Description
Activity
hi @ephraimbuddy , wondering why #33039 is part of Airflow RC? I think it went in the Sqoop provider, no?
hi @ephraimbuddy , wondering why #33039 is part of Airflow RC? I think it went in the Sqoop provider, no?
It's because of conflict resolution, trying to cherry-pick cleanly. We had many refactoring PRs that addressed issues across core and providers. If you look at the description on the sync PR: #33826 I mentioned about provider changes getting cherry-picked to core.
You do not need to test this in this caseReacted by Pankaj KotiI encountered an issue when upgrading to 2.7.1rc1, linked to the pydantic upgrade. Do I open a seperate issue for this or can this be discussed here?
Great expectations does not yet support pydantic > 2.0:
fivetran/great_expectations#8604Because of this, we are unable to upgrade our airflow version
Reacted by Alex GalstCheck on fresh Airflow 2.7.1.rc1
- Add limit 1 if required first value from query result #33672 - Less intensive queries to DB. Shorter execution time, less traffic between backend and Airflow
- Use
dialect.namein custom SA types #33503 - Seems working with Postgres/MySQL as expected - Sort data before groupby in TIS duration calculation #33535 - Work as expected in the UI
I encountered an issue when upgrading to 2.7.1rc1, linked to the pydantic upgrade. Do I open a seperate issue for this or can this be discussed here? Great expectations does not yet support pydantic > 2.0: great-expectations/great_expectations#8604
Because of this, we are unable to upgrade our airflow version
Ops. Looks like we don't currently have great expectation in our codebase but would like to hear what others think cc @potiuk
- I encountered an issue when upgrading to 2.7.1rc1, linked to the pydanticupgrade. Do I open a seperate issue for this or can this be discussed here? Great expectations does not yet support pydantic > 2.0: fivetran/great_expectations#8604 <fivetran/great_expectations#8604> Airflow doesn't have an official integration/support for great expectations, so if great expectations doesn't support pydantic v2 which we use in 2.7.1, IMHO this should not be considered a breaking change.…On Mon, Sep 4, 2023 at 5:28 PM Andrey Anshin ***@***.***> wrote: Check on fresh Airflow 2.7.1.rc1 - #33672 <#33672> - Less intensive queries to DB. Shorter execution time, less traffic between backend and Airflow - #33503 <#33503> - Seems working with Postgres/MySQL as expected - #33535 <#33535> - Work as expected in the UI — Reply to this email directly, view it on GitHub <#34065 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AFCS775S3YTSFWWQA7LUH6LXYXXQLANCNFSM6AAAAAA4J6KWGM> . You are receiving this because you were mentioned.Message ID: ***@***.***>Reacted by Alejandro Vidal Castillo
I encountered an issue when upgrading to 2.7.1rc1, linked to the pydantic
upgrade. Do I open a seperate issue for this or can this be discussed here? Great expectations does not yet support pydantic > 2.0: great-expectations/great_expectations#8604 <great-expectations/great_expectations#8604> Airflow doesn't have an official integration/support for great expectations, so if great expectations doesn't support pydantic v2 which we use in 2.7.1, IMHO this should not be considered a breaking change.
…
On Mon, Sep 4, 2023 at 5:28 PM Andrey Anshin @.> wrote: Check on fresh Airflow 2.7.1.rc1 - #33672 <#33672> - Less intensive queries to DB. Shorter execution time, less traffic between backend and Airflow - #33503 <#33503> - Seems working with Postgres/MySQL as expected - #33535 <#33535> - Work as expected in the UI — Reply to this email directly, view it on GitHub <#34065 (comment)>, or unsubscribe https://github.com/notifications/unsubscribe-auth/AFCS775S3YTSFWWQA7LUH6LXYXXQLANCNFSM6AAAAAA4J6KWGM . You are receiving this because you were mentioned.Message ID: @.>Not a breaking change but great expectation is widely used, I think we should consider it
Not a breaking change but great expectation is widely used, I think we should consider it
My only concern is that pydantic v2 was added to fix a bug or just an improvement? if not maybe we should move it to 2.8.0
Not a breaking change but great expectation is widely used, I think we should consider it
My only concern is that pydantic v2 was added to fix a bug or just an improvement? if not maybe we should move it to 2.8.0
Yeah, I think it's for improvement
Not a breaking change but great expectation is widely used, I think we should consider it
My only concern is that pydantic v2 was added to fix a bug or just an improvement? if not maybe we should move it to 2.8.0
Yeah, I think it's for improvement
Specifically for AIP-44 so yes ww move it down to v1 and we will release for 2.8.0 (when I hope AIP-44 will be ready to ship). By then Pydantic 2 should be much more widespread.
Reacted by Ephraim Anierobi and Pankaj Kotihi,
Tested the below PRs are included and they all work as expected.- Remove the replace to <br> tag in Provider's view #33326 - The BR tag no longer appears in the providers view. Works as expected
- Use a trimmed version of README.md for PyPI #33637 - The generated/PYPI_README.md file is present in the RC and is pointed to in the setup.cfg for long_description for the package.
- Remove print introduced in PR #32261 #34008 : The endpoint no longer prints - Works as expected
As noted in comment #34065 (comment),
Ignored cherry-picked PR for conflict resolution #33039Thank you for including all of these 🙏🏽
Reacted by Ephraim AnierobiHi,
I verified that this change works as expected in UI in 2.7.1rc1- Fix cleaning zombie RESTARTING tasks #33706 - a task that was in RESTARTING state always goes into RUNNING state after scheduler start, the scheduler logs indicate that it was adopted
Reacted by Jarek Potiuk and Ephraim Anierobi- changed the title
[-]Status of testing of Apache Airflow 2.7.1rc1[/-][+]Status of testing of Apache Airflow 2.7.1rc2[/+]on Sep 4, 2023 Confirmed all my changes are in.! Looks good.
Reacted by Ephraim Anierobi+1. Checked my changes from RC1 (#34065 (comment)) are included here too and all work fine.
Reacted by Jarek Potiuk and Ephraim AnierobiI have pulled the 2.7.1rc2 commit and verified the following fixes in Breeze:
#33735 is a docs-only fix and should be fine.
I'm trying to verify #33632 but running into issues triggering the permissions sync (this issue might predate this release?) I will keep investigating and confirm.
Update: I have found some issues preventing #33632 from working as expected. I will work on a fix and provide updates in #34114, but in the meantime we could also remove this commit from the release and save it for 2.7.2? I will leave it up to you.
Reacted by Jarek Potiuk, Ephraim Anierobi and Schumgood for me
Reacted by Jarek Potiuk and Ephraim AnierobiChecked #33141 and can confirm it is working.
Found a minor "glitch" of string formatting as we forked code of Flask-AppBuilder (manager.py) into the Airflow repo and the Flask Appbuilder has changed the string formatting upstream, the constants we source in manager.py:499 for example are now printed with
%sinstead of the content.
I assume not critical, will open a follow-up bug ticket (if not already open)Reacted by Ephraim AnierobiAirflow 2.7.1 is now released. Thank you all for testing this release
Found a minor "glitch" of string formatting as we forked code of Flask-AppBuilder (manager.py) into the Airflow repo and the Flask Appbuilder has changed the string formatting upstream, the constants we source in manager.py:499 for example are now printed with
%sinstead of the content. I assume not critical, will open a follow-up bug ticket (if not already open)@jens-scheffler-bosch could you check if #34139 fix the bug you are talking about?
Found a minor "glitch" of string formatting as we forked code of Flask-AppBuilder (manager.py) into the Airflow repo and the Flask Appbuilder has changed the string formatting upstream, the constants we source in manager.py:499 for example are now printed with
%sinstead of the content. I assume not critical, will open a follow-up bug ticket (if not already open)@jens-scheffler-bosch could you check if #34139 fix the bug you are talking about?
Perfect. Yes. that was what I was referring to. Did not find the issue by search... so fixed already, GREAT!
Body
We are kindly requesting that contributors to Apache Airflow RC 2.7.1rc1 help test the RC.
Please let us know by commenting if the issue is addressed in the latest RC.
expand_kwargsmethod (#32272): @Talvarenga @hussein-awalaLinked issues:
Linked issues:
Linked issues:
orm_moderename (#33220): @ephraimbuddytag in Provider's view (#33326): @eladkal @pankajkoti
Linked issues:
processor_subdir(#33357): @pierrejeambrunLinked issues:
CronTriggerTimetable(#32921)Linked issues:
Linked issues:
conf.setcase insensitive (#33452): @vandonr-amzformatDurationmethod (#33486): @hussein-awaladialect.namein custom SA types (#33503): @agomez-etsy @TaragolisLinked issues:
Linked issues:
execution_dateparameter is malformed (#33478)Linked issues:
execution_dateparameter is malformed (#33478)len()in utils (#33567): @eumiroLinked issues:
Linked issues:
Linked issues:
Linked issues:
Linked issues:
get_url_for_loginin security manager (#33660): @vincbeck @cesar-vermeulenLinked issues:
Linked issues:
formatin Airflow core (#33753): @hussein-awalaLinked issues:
Linked issues:
Linked issues:
Linked issues:
Thanks to all who contributed to the release (probably not a complete list!):
@agomez-etsy @Lee-W @jedcunningham @potiuk @cesar-vermeulen @pankajkoti @abhishekbhakat @dstandish @vandonr-amz @SamWheating @joeknize-bc @ephraimbuddy @ohaibbq @alexbegg @Michalosu @BohdanSemonov @pierrejeambrun @ferruzzi @eumiro @mobuchowski @Taragolis @Talvarenga @aipatr @LipuFei @Bisk1 @uranusjr @hussein-awala @vincbeck @s0neq @klexadoc @boushphong @eladkal @jens-scheffler-bosch @Owen-CH-Leung @nathadfield @ivan-afonichkin
Committer