Status of testing of Apache Airflow 2.2.5rc3

@subkanthi

Body

We have a kind request for all the contributors to the latest Apache Airflow RC 2.2.5rc3.

Could you please help us to test the RC versions of Airflow?

Please let us know in the comment if the issue is addressed in the latest RC.

Set X-Frame-Options header to DENY unless X_FRAME_ENABLED is set to true (#19491): @subkanthi @planoe-cloudera
Linked issues:
- x_frame_enabled logic broken in Airflow 2 (#17255)
Fix: Do not render undefined graph edges (#19684): @bbovenzi
Fix Tasks getting stuck in scheduled state (#19747): @vapiravfif @tanelk
Linked issues:
- Tasks get stuck in "scheduled" state and starved when dags with huge amount of tasks is scheduled (#19622)
Rename to_delete to to_cancel in TriggerRunner (#20658): @dstandish
Fix duplicate trigger creation race condition (#20699): @MatrixManAtYrService @dstandish
Linked issues:
- TriggerEvent fires, and then defers a second time (doesn't fire a second time though). (#18392)
- Track completed triggers until deleted from database (#19546)
Truncate stack trace to DAG user code for exceptions raised during execution (#20731): @malthe
fix: Update custom connection field processing (#20883): @mike-mcdonald
Linked issues:
- Cannot edit custom fields on provider connections (#20839)
Ensure clear_task_instances sets valid run state (#21116): @uranusjr @uplsh580
Linked issues:
- Running airflow dags backfill --reset-dagruns <dag_id> -s <execution_start_dt> -e <execution_end_dt> results in error when run twice. (#21058)
- Fix 'airflow dags backfill --reset-dagruns' errors when run twice (#21062)
Update ExternalTaskSensorLink to handle templated external_dag_id (#21192): @csp98 @josh-fell
Linked issues:
- Webserver "External DAG" button on ExternalTaskSensor not working when dag_id is templated (#21183)
A trigger might use a connection; make sure we mask passwords (#21207): @malthe
Log traceback in trigger excs (#21213): @malthe
Fix trigger dag redirect from task instance log view (#21239): @dstandish
Fix race condition between triggerer and scheduler (#21316): @malthe
Reduce DB load incurred by Stale DAG deactivation (#21399): @SamWheating
Fix max_active_runs=1 not scheduling runs when min_file_process_interval is high (#21413): @ephraimbuddy
Fix Resources eq check (#21442): @pingzh
Filter out default configs when overrides exist. (#21539): @xyu @ashb
Linked issues:
- PR #18772 breaks sql_alchemy_conn_cmd config (#20092)
- Don't bake ENV and _cmd into tmp config for non-sudo (#18772)
Fix logging JDBC SQL error when task fails (#21540): @stijn-meersman @hubert-pietron @jyotsa09
Linked issues:
- No SQL error shown when using the JDBC operator (#18482)
- JDBC operator not logging errors (#16295)
Dispose not used connection pool (#21565): @pingzh
FIX: extends typing-extensions to be installed with python 3.8+ #21566 (#21567): @frankcash
Linked issues:
- typing_extensions package isn't installed with apache-airflow-providers-amazon causing an issue for SqlToS3Operator (#21566)
Disable default_pool delete on web ui (#21658): @chenglongyan
Log exception in local executor (#21667): @uranusjr
Correctly handle multiple '=' in LocalFileSystem secrets. (#21694): @kadai0308 @joeyberkovitz
Linked issues:
- LocalFileSystem Secret Backend Bug (#21625)
Fix stray order_by(TaskInstance.execution_date) (#21705): @uranusjr @TidesMind
Linked issues:
- Airflow >= 2.2.0 execution date change is failing TaskInstance get_task_instances method and possibly others (#21656)
Fix filesystem sensor for directories (#21729): @mixilchenko
Fix graph autorefresh on page load (#21736): @bbovenzi
Fix triggerer --capacity parameter (#21753): @thesuperzapper @msumit
Linked issues:
- triggerer --capacity parameter does not work (#21752)
Fix assignment of unassigned triggers (#21770): @jkramer-ginkgo
Fix incorrect data provided to tries & landing times charts (#21928): @millin
Fixed dask executor and tests (#22027): @subkanthi @potiuk
Linked issues:
- Removes limitations from Dask dependencies (#22017)
Add the new Airflow Trove Classifier to setup.cfg (#22241): @potiuk
Fix broken links to celery documentation (#22364): @davidhagens
Add back celery intersphinx mapping (#22370): @kaxil
Replace timedelta.max with year long timdelta in test_manager (#22527): @potiuk
Check and disallow a relative path for sqlite (#22530): @yenchenLiu

Thanks to all who contributed to the release (probably not a complete list!):
@frankcash @yenchenLiu @bbovenzi @SamWheating @ephraimbuddy @csp98 @joeyberkovitz @planoe-cloudera @hubert-pietron @dstandish @stijn-meersman @jyotsa09 @kadai0308 @chenglongyan @pingzh @TidesMind @josh-fell @potiuk @ashb @mike-mcdonald @vapiravfif @thesuperzapper @msumit @xyu @uplsh580 @tanelk @subkanthi @uranusjr @jkramer-ginkgo @millin @malthe @mixilchenko @davidhagens @MatrixManAtYrService @kaxil

Committer

I acknowledge that I am a maintainer/committer of the Apache Airflow project.

@subkanthi

All good.

Checked #22241 - > classifier is there

#21905 and #22527 were mostly to make sure the tests pass (they did :) )

#22027 - > I checked that dependencies are ok (below). Tests passed too. It would be great @subkanthi if you could check it (I have no Dask account ) :)

http https://pypi.org/pypi/apache-airflow/2.2.5rc1/json | jq ".info" | grep "extra == 'dask'":

    "cloudpickle (>=1.4.1) ; extra == 'dask'",
    "dask (>=2.9.0) ; extra == 'dask'",
    "distributed (>=2.11.1) ; extra == 'dask'",

#21729 looks good for 2.2.5 release

#18797 look good

with python 3.6

[2022-03-27, 18:45:42 UTC] {kubernetes_pod.py:525} INFO - Creating pod airflow-test-pod.2e925545978340a197ae86ee955e660a with labels: {'dag_id': 'kubernetes_dag', 'task_id': 'task-one', 'run_id': 'scheduled__2022-03-18T0000000000-90832bb28', 'try_number': '1'}
[2022-03-27, 18:45:42 UTC] {pod_manager.py:163} WARNING - Pod not yet started: airflow-test-pod.2e925545978340a197ae86ee955e660a
[2022-03-27, 18:45:43 UTC] {pod_manager.py:163} WARNING - Pod not yet started: airflow-test-pod.2e925545978340a197ae86ee955e660a
[2022-03-27, 18:45:44 UTC] {pod_manager.py:163} WARNING - Pod not yet started: airflow-test-pod.2e925545978340a197ae86ee955e660a
[2022-03-27, 18:45:45 UTC] {pod_manager.py:163} WARNING - Pod not yet started: airflow-test-pod.2e925545978340a197ae86ee955e660a
[2022-03-27, 18:45:47 UTC] {pod_manager.py:203} INFO - toto
[2022-03-27, 18:46:03 UTC] {pod_manager.py:203} INFO - finish
[2022-03-27, 18:46:04 UTC] {kubernetes_pod.py:418} INFO - Deleting pod: airflow-test-pod.2e925545978340a197ae86ee955e660a
[2022-03-27, 18:46:04 UTC] {taskinstance.py:1288} INFO - Marking task as SUCCESS. dag_id=kubernetes_dag, task_id=task-one, execution_date=20220318T000000, start_date=20220327T184542, end_date=20220327T184604
[2022-03-27, 18:46:04 UTC] {local_task_job.py:154} INFO - Task exited with return code 0
[2022-03-27, 18:46:04 UTC] {local_task_job.py:264} INFO - 0 downstream tasks scheduled from follow-on schedule check

with python 3.9

[2022-03-27, 18:29:26 UTC] {kubernetes_pod.py:525} INFO - Creating pod airflow-test-pod.87f39cb63ab949c1afc6b92b43e04370 with labels: {'dag_id': 'kubernetes_dag', 'task_id': 'task-one', 'run_id': 'scheduled__2022-03-26T0000000000-09f4abc4d', 'try_number': '1'}
[2022-03-27, 18:29:27 UTC] {pod_manager.py:159} WARNING - Pod not yet started: airflow-test-pod.87f39cb63ab949c1afc6b92b43e04370
[2022-03-27, 18:29:28 UTC] {pod_manager.py:159} WARNING - Pod not yet started: airflow-test-pod.87f39cb63ab949c1afc6b92b43e04370
[2022-03-27, 18:29:29 UTC] {pod_manager.py:159} WARNING - Pod not yet started: airflow-test-pod.87f39cb63ab949c1afc6b92b43e04370
[2022-03-27, 18:29:30 UTC] {pod_manager.py:159} WARNING - Pod not yet started: airflow-test-pod.87f39cb63ab949c1afc6b92b43e04370
[2022-03-27, 18:29:31 UTC] {pod_manager.py:159} WARNING - Pod not yet started: airflow-test-pod.87f39cb63ab949c1afc6b92b43e04370
[2022-03-27, 18:29:32 UTC] {pod_manager.py:159} WARNING - Pod not yet started: airflow-test-pod.87f39cb63ab949c1afc6b92b43e04370
[2022-03-27, 18:29:33 UTC] {pod_manager.py:199} INFO - toto
[2022-03-27, 18:29:48 UTC] {pod_manager.py:199} INFO - finish
[2022-03-27, 18:29:49 UTC] {pod_manager.py:214} WARNING - Pod airflow-test-pod.87f39cb63ab949c1afc6b92b43e04370 log read interrupted but container base still running
[2022-03-27, 18:29:50 UTC] {pod_manager.py:199} INFO - finish
[2022-03-27, 18:29:50 UTC] {kubernetes_pod.py:418} INFO - Deleting pod: airflow-test-pod.87f39cb63ab949c1afc6b92b43e04370
[2022-03-27, 18:29:50 UTC] {taskinstance.py:1278} INFO - Marking task as SUCCESS. dag_id=kubernetes_dag, task_id=task-one, execution_date=20220326T000000, start_date=20220327T182926, end_date=20220327T182950
[2022-03-27, 18:29:50 UTC] {local_task_job.py:154} INFO - Task exited with return code 0
[2022-03-27, 18:29:50 UTC] {local_task_job.py:264} INFO - 0 downstream tasks scheduled from follow-on schedule check

#21694 looks good - tested with 2.2.5rc1 docker image.

conn.env:

TEST_CONN=mysql://username:password@host:1234/schema?param1=val1&param2=val2

airflow connections get TEST_CONN:

id   | conn_id   | conn_type | description | host | schema | login    | password | port | is_encrypted | is_extra_encrypted | extra_dejson                         | get_uri                                                           
=====+===========+===========+=============+======+========+==========+==========+======+==============+====================+======================================+===================================================================
None | TEST_CONN | mysql     | None        | host | schema | username | password | 1234 | False        | False              | {'param1': 'val1', 'param2': 'val2'} | mysql://username:password@host:1234/schema?param1=val1&param2=val2

Just brought up a clean docker image with the RC to validate #21567

$ python --version && airflow version && pip list | grep typing_extension
Python 3.8.13
2.2.5rc1
typing_extensions                      4.1.1

Looks good to me

@subkanthi

All good.

Checked #22241 - > classifier is there

#21905 and #22527 were mostly to make sure the tests pass (they did :) )

#22027 - > I checked that dependencies are ok (below). Tests passed too. It would be great @subkanthi if you could check it (I have no Dask account ) :)

http https://pypi.org/pypi/apache-airflow/2.2.5rc1/json | jq ".info" | grep "extra == 'dask'":
    "cloudpickle (>=1.4.1) ; extra == 'dask'",
    "dask (>=2.9.0) ; extra == 'dask'",
    "distributed (>=2.11.1) ; extra == 'dask'",

Tested #22027 on the tag 2.2.5rc1. Thanks

Tested #20883 on 2.2.5rc2. Tests passed, using the connection form to test saving also worked as expected.

#21770 <#21770> confirmed with heavy load

…

Validated #21567

$ python --version && airflow version && pip list | grep typing_extension
Python 3.8.13
2.2.5rc2
typing_extensions                      4.1.1

All good with RC3 from my point of view - including updated dependencies.

@ephraimbuddy

Verified #21192. Thanks @ephraimbuddy for organizing this release!

Closing since 2.2.5 was released.

Navigation Menu

Repository navigation

Status of testing of Apache Airflow 2.2.5rc3 #22549

Body

Committer

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Status of testing of Apache Airflow 2.2.5rc3 #22549

Description

Body

Committer

Activity

potiuk commented on Mar 27, 2022

mixilchenko commented on Mar 27, 2022

raphaelauv commented on Mar 27, 2022

joeyberkovitz commented on Mar 28, 2022

frankcash commented on Mar 28, 2022

subkanthi commented on Mar 28, 2022

mike-mcdonald commented on Mar 29, 2022

jkramer-ginkgo commented on Mar 29, 2022

frankcash commented on Mar 29, 2022

potiuk commented on Mar 31, 2022

josh-fell commented on Mar 31, 2022

eladkal commented on Apr 5, 2022

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions