Status of testing Apache Airflow 3.0.4rc2 and TaskSDK 1.0.4rc1

[ashb]

We are kindly requesting that contributors to Apache Airflow RC 3.0.4rc1 help test the RC.

Please let us know by commenting if the issue is addressed in the latest RC.

• use of debugpy to run airflow components via breeze (#51763): @rawwar
• [Backport v3-0-test] Add back dag parsing pre-import optimization (#50371) (#52698): @Lzzz666 @kaxil
  Linked issues:
  • Add back dag parsing pre-import optimization (#50371)
• [v3-0-test] Dev: Fix KeyError in release issue content generation (#52751) (#52756): @kaxil
• [v3-0-test] Advanced auto classification for provider documentation (#52902) (#52925): @amoghrajesh
• [v3-0-test] Workaround the need to use latest version on Lucas pre-commit's (#52931) (#52942): @potiuk
• [v3-0-test] Make the update_installer_and_pre_commit script more resilient to re matches (#52960) (#52961): @amoghrajesh
• [v3-0-test] Remove upper-binding for "python-requires" (#52980) (#52984): @potiuk
• [v3-0-test] Remove remnants of ~= used in requires-python. (#52985) (#52987): @potiuk
  Linked issues:
  • Remove upper-binding for "python-requires" (#52980)
• [v3-0-test] Unify connection not found exceptions between AF2 and AF3 (#52968) (#53093): @Lzzz666 @github-actions[bot] @potiuk @amoghrajesh
• [v3-0-test] docs: update public interface doc to reflect airflow.sdk and AIP-72 (#52197) (#53117): @nrnavaneet
• Run Task failure callbacks on DAG Processor when task is externally killed (#53058) (#53143): @kaxil
  Linked issues:
  • Run Task failure callbacks on DAG Processor when task is externally killed (#53058)
• [v3-0-test] Resolve OOM When Reading Large Logs in Webserver (#53167): @jason810496
  Linked issues:
  • Resolve OOM When Reading Large Logs in Webserver (#49470)
• [v3-0-test] Move --no-binary flag to uv command line from pyproject.toml (#53179) (#53185): @potiuk
• [v3-0-test] Install python from source again (#53150) (#53202): @aritra24 @potiuk
• [v3-0-test] Fix package typo in build images (#53209) (#53210): @gopidesupavan
• [v3-0-test] Bump uv to 0.7.20 (#53207) (#53220): @potiuk @gopidesupavan
• [v3-0-test] Pass GITHUB_TOKEN to update checks (#53219) (#53221): @potiuk
• [v3-0-test] Bring back quick image build check and increase timeout minutes (#53227) (#53230): @potiuk @gopidesupavan
  Linked issues:
  • Run check-that-image-builds-quickly job only in canary (#53212)
• [v3-0-test] bump remaining uv versions (#53228) (#53232): @bugraoz93 @potiuk
• [v3-0-test] fix(ci): bump python versions (#53229) (#53233): @bugraoz93
• [v3-0-test] Update mypy script with --warn-unused-ignores (#53240) (#53245): @gopidesupavan
• [v3-0-test] Fix mypy cmd (#53246) (#53247): @gopidesupavan
• [v3-0-test] Fix broken link in contributors doc on providers (#53344) (#53347): @zach-overflow
• [v3-0-test] fix: Make task_success_overtime configurable (#53342) (#53351): @kacpermuda
• [v3-0-test] Remove gunicorn daemonize for api-server (#52860) (#53372): @jason810496
  Linked issues:
  • Respect apps flags for api_server_command (#52929)
  • Remove gunicorn daemonize for api-server (#52860)
• [v3-0-test] Update docs for Airflow CLI help command (#51042) (#53383): @rapsealk
• [v3-0-test] Update doc string for BaseOperator (#53403) (#53404): @eladkal
• [v3-0-test] Fixing uv upgrade failures on main (#53406) (#53451): @potiuk @amoghrajesh
• [v3-0-test] Make GITHUB_TOKEN variable mandatory when updating installers (#53452) (#53457): @potiuk
• [v3-0-test] DagBag in file parsing used to collect dags in dag processing uses default value for safe_mode, which resolves to value in configuration (#52694) (#53507): @BBQing
• [v3-0-test] Handle invalid execution API urls gracefully in supervisor (#53082) (#53518): @amoghrajesh
• [v3-0-test] Remove warning for BaseOperator.executor because its false (#53496) (#53519): @amoghrajesh
• [v3-0-test] Add changelog for regenerated image for Airflow 3.0.3 (#5… (#53522): @potiuk
• [v3-0-test] Add sql extras to pandas in the providers where sqlalchem… (#53539): @potiuk
• [v3-0-test] Upgdate to uv 0.8.0 (#53552) (#53564): @potiuk
• [v3-0-test] Include fixed navbar for anchor position calculation. (#5… (#53581): @simi @potiuk
• [v3-0-test] fix: accordian child contents to be visible when overflown (#53595) (#53602): @naseemshah
• [v3-0-test] fix(systemd): Update systemd unit files and documentation for Airflow 3.0 (#52294) (#53609): @nrnavaneet
• [v3-0-test] Skip empty dag run conf rows and set statement timeout (#50788) (#53619): @dstandish
• [v3-0-test] Add deprecation notice for using Connection from models (#53594) (#53621): @amoghrajesh
• [v3-0-test] Ignore cursor rules from the git tree (#53658) (#53659): @amoghrajesh
• Fix incorrect reference to SimpleHttpOperator in 3.0.0 release notes (#53661): @eladkal
• [v3-0-test] Fix a UI bug where "All" does not show all dags when hide_paused_dags_by_default = True (#53656) (#53672): @zachliu
• [v3-0-test] Add GITHUB_TOKEN in test helm release job (#53674) (#53677): @potiuk
• [v3-0-test] Add possibility to apply a commit with patch to docs when publishing (apache#53668) (#53678): @potiuk
• [v3-0-test] Improve validation of breeze workflow-run command (#53685) (#53688): @potiuk
• [v3-0-test] Fix typo in AIRFLOW_COMMIT(S) diagnostics in workflow_run (#53686) (#53689): @potiuk
• [v3-0-test] CHG: cover v1, v2 (#53636) (#53709): @kyungjunleeme
• [v3-0-test] Bump gitpython from 3.1.44 to 3.1.45 in /airflow-core (#5… (#53731): @dependabot[bot] @potiuk
• [v3-0-test] Bump gitpython from 3.1.44 to 3.1.45 (#53724) (#53732): @dependabot[bot] @potiuk
• [v3-0-test] Unconditionally disable start_from_trigger (#53744) (#53750): @uranusjr
• [v3-0-test] Allow Remote logging providers to load connections from the API Server (#53719) (#53761): @ashb
• [v3-0-test] Remove extra slash from endpoint URL (#53755) (#53764): @lpillmann
• [v3-0-test] Respect apps flags for api_server_command (#52929) (#53775): @jason810496
  Linked issues:
  • Respect apps flags for api_server_command (#52929)
• [v3-0-test] Make branch tests more resilient to provider dependency c… (#53777): @potiuk
• [v3-0-test] Better type handling to account for types-docutils bump (… (#53778): @potiuk @amoghrajesh
• [v3-0-test] sphinx: avoid repeated isinstance checks (#53367) (#53783): @Dev-iL @gopidesupavan
• [v3-0-test] Add documentation for self-signed certificate (#53788) (#53792): @danieldean
• [v3-0-test] Add cert used for apiserver to client (fix for self-signe… (#53793): @danieldean @potiuk
• [v3-0-test] Add user facing docs for running a separate Task Execution API-server (#53789) (#53794): @jason810496
• [v3-0-test] Remove unnecessary group_by clause in event logs query (#53733) (#53807): @xiaohuanlin @gvnn-gov
  Linked issues:
  • column "log.dttm" must appear in the GROUP BY clause or be used in an aggregate function (#53695)
• [v3-0-test] Make BaseOperator on_kill functionality work with TaskSDK (#53832): @gopidesupavan
• [v3-0-test] Serialize nan and infinities to string (#53835) (#53844): @uranusjr
• [v3-0-test] Fix scheduler heartbeat timeout failures with DetachedInstanceError (#53838) (#53858): @kaxil
• [v3-0-test] Revert "Fix editing connection with sensitive extra field (#52403)" (#53889): @pierrejeambrun
• docs: Update DAG author as Dag author (#53857) (#53950): @Lee-W
• [v3-0-test] Removed bold formatting for Public Interface for Airflow 3.0+ (#53955) (#53964): @sunank200
• Handle unmodified sensitived fields when updating connections (#53943) (#53973): @pierrejeambrun
• [v3-0-test] Add type annotation for dev system tests (#53979) (#53996): @amoghrajesh
• [v3-0-test] Fix nth-child warning (#53982) (#54000): @pierrejeambrun
• Revert "[v3-0-test] Remove gunicorn daemonize for api-server (#52860) (#53372)" (#54002): @jason810496 @jedcunningham
  Linked issues:
  • Remove gunicorn daemonize for api-server (#52860)
• [v3-0-test] Fix version for parsing_pre_import_modules config option move (#53995) (#54011): @jedcunningham
• [v3-0-test] Fix AIRFLOW_API_APPS constant in api_server_command (#54007) (#54012): @jason810496
• [v3-0-test] fix(diagrams): update labels from "Webserver(s)" to "API Server(s)" i… (#54020): @potiuk @aayushbisen
• [v3-0-test] build: update uv to 0.8.4 and pip to 25.2 (#53954) (#54021): @Lee-W @potiuk
• [v3-0-test] Unmount dialog & add onOpen (#53307) (#54025): @pierrejeambrun @Jasperora
• [v3-0-test] Flexible form use ReactMarkdown instead of default Markdo… (#54040): @pierrejeambrun @jscheffl
• [v3-0-test] Fix _extract_template_fields mypy error (#53855) (#54047): @choo121600
• [v3-0-test] Make log redaction safer in edge case when redaction has an error (#54046) (#54048): @ashb
• [v3-0-test] Only forward debugger ports when debug components are used (#54051) (#54053): @rawwar @potiuk
  Linked issues:
  • use of debugpy to run airflow components via breeze (#51763)
• [v3-0-test] Simplify installing airflow from GitHub repo in breeze (#54070) (#54076): @potiuk
• Limit paramiko to < 4.0.0 till we remove DSS support (#54078): @amoghrajesh

Thanks to all who contributed to the release (probably not a complete list!):
@Lzzz666 @jscheffl @gvnn-gov @aayushbisen @zach-overflow @bugraoz93 @jason810496 @sunank200 @jedcunningham @lpillmann @zachliu @dstandish @potiuk @kyungjunleeme @amoghrajesh @xiaohuanlin @Lee-W @gopidesupavan @Jasperora @nrnavaneet @BBQing @aritra24 @rawwar @rapsealk @dependabot[bot] @kacpermuda @uranusjr @naseemshah @ashb @pierrejeambrun @danieldean @simi @choo121600 @kaxil @eladkal @Dev-iL

[potiuk]

Tested and checked all my changes - they look good. I found one issue with #54020 -> there was another cherry-pick missing - #54129 -> which caused generated diagrams to miss icons (see here: https://airflow.staged.apache.org/docs/apache-airflow/stable/core-concepts/overview.html) . I am cherry-picking the missing change to v3-0-test (#54129). No need to fix anything - we will just have to add the commit (alongside #54100) when building documentation.

[amoghrajesh]

CI / Dev tool fixes:

• [v3-0-test] Make the update_installer_and_pre_commit script more resilient to re matches (#52960) #52961
• [v3-0-test] Advanced auto classification for provider documentation (#52902) #52925
• [v3-0-test] Fixing uv upgrade failures on main (#53406) #53451
• [v3-0-test] Ignore cursor rules from the git tree (#53658) #53659
• [v3-0-test] Better type handling to account for types-docutils bump (… #53778
• [v3-0-test] Add type annotation for dev system tests (#53979) #53996
• Limit paramiko to < 4.0.0 till we remove DSS support #54078

Actual fixes:

• [v3-0-test] Unify connection not found exceptions between AF2 and AF3 (#52968) #53093
• [v3-0-test] Handle invalid execution API urls gracefully in supervisor (#53082) #53518
• [v3-0-test] Remove warning for BaseOperator.executor because its false (#53496) #53519
• [v3-0-test] Add deprecation notice for using Connection from models (#53594) #53621

All of it works as expected.

[pierrejeambrun]

Tested a bunch of my changes

[bugraoz93]

My changes look good. Thanks, Ash!

[danieldean]

#53793 works perfectly. Thanks!

[BBQing]

#53507 Tested and it works

[naseemshah]

My change #53602 works well!

[Lee-W]

Tested a few dags with airflow==3.0.4rc1 and task-sdk==1.0.4rc1. Works fine.

One issue I encountered was that after running uv add apache-airflow==3.0.4rc1 --prerelease=allow, it does not use task-sdk==1.0.4rc1. Additionally, airflow 3.0.4rc1 does not work with task-sdk 1.0.3. Not sure whether we need to pin any deps

[choo121600]

My change only updates pre-commit scripts, so no issues with this rc 😉

[aritra24]

My changes were with the build system so shouldn't effect the rc

[Lzzz666]

Tested with 3.0.4rc1:

• My change #50371 works well.
• The backport #52698 does not affect the RC functionality.

Looks good from my side.

[ashb]

Tested a few dags with airflow==3.0.4rc1 and task-sdk==1.0.4rc1. Works fine.

One issue I encountered was that after running uv add apache-airflow==3.0.4rc1 --prerelease=allow, it does not use task-sdk==1.0.4rc1. Additionally, airflow 3.0.4rc1 does not work with task-sdk 1.0.3. Not sure whether we need to pin any deps

Fixed in apache-airflow rc2