After upgrade from Airflow 2.2.4, grid disappears for some DAGs

Apache Airflow version

2.3.0 (latest released)

What happened

After the upgrade from 2.2.4 to 2.3.0, some DAGs grid data seems missing and it renders the UI blank

What you think should happen instead

When I click the grid for a specific execution date, I expect to be able to click the tasks and view the log, render jinja templating, and clear status

How to reproduce

Run an upgrade from 2.2.4 to 2.3.0 with a huge database (we have ~750 DAGs with a minimum of 10 tasks each).
In addition, we heavily rely on XCom.

Operating System

Ubuntu 20.04.3 LTS

Versions of Apache Airflow Providers

apache-airflow apache_airflow-2.3.0-py3-none-any.whl
apache-airflow-providers-amazon apache_airflow_providers_amazon-3.3.0-py3-none-any.whl
apache-airflow-providers-ftp apache_airflow_providers_ftp-2.1.2-py3-none-any.whl
apache-airflow-providers-http apache_airflow_providers_http-2.1.2-py3-none-any.whl
apache-airflow-providers-imap apache_airflow_providers_imap-2.2.3-py3-none-any.whl
apache-airflow-providers-mongo apache_airflow_providers_mongo-2.3.3-py3-none-any.whl
apache-airflow-providers-mysql apache_airflow_providers_mysql-2.2.3-py3-none-any.whl
apache-airflow-providers-pagerduty apache_airflow_providers_pagerduty-2.1.3-py3-none-any.whl
apache-airflow-providers-postgres apache_airflow_providers_postgres-4.1.0-py3-none-any.whl
apache-airflow-providers-sendgrid apache_airflow_providers_sendgrid-2.0.4-py3-none-any.whl
apache-airflow-providers-slack apache_airflow_providers_slack-4.2.3-py3-none-any.whl
apache-airflow-providers-sqlite apache_airflow_providers_sqlite-2.1.3-py3-none-any.whl
apache-airflow-providers-ssh apache_airflow_providers_ssh-2.4.3-py3-none-any.whl
apache-airflow-providers-vertica apache_airflow_providers_vertica-2.1.3-py3-none-any.whl

Deployment

Virtualenv installation

Deployment details

Python 3.8.10

Anything else

For the affected DAGs, all the time

Are you willing to submit PR?

Yes I am willing to submit a PR!

Code of Conduct

I agree to follow this project's Code of Conduct

Thanks for opening your first issue here! Be sure to follow the issue template!

Video that shows the problem

Getting the same thing - It appears to be an issue with the getTask API (there's several javascript errors in console). Most notably the /tasks endpoint for a given dag.

Python version: 3.9.12
Airflow version: 2.3.0+astro.3

Traceback (most recent call last):
  File "/usr/local/lib/python3.9/site-packages/flask/app.py", line 2447, in wsgi_app
    response = self.full_dispatch_request()
  File "/usr/local/lib/python3.9/site-packages/flask/app.py", line 1952, in full_dispatch_request
    rv = self.handle_user_exception(e)
  File "/usr/local/lib/python3.9/site-packages/flask/app.py", line 1821, in handle_user_exception
    reraise(exc_type, exc_value, tb)
  File "/usr/local/lib/python3.9/site-packages/flask/_compat.py", line 39, in reraise
    raise value
  File "/usr/local/lib/python3.9/site-packages/flask/app.py", line 1950, in full_dispatch_request
    rv = self.dispatch_request()
  File "/usr/local/lib/python3.9/site-packages/flask/app.py", line 1936, in dispatch_request
    return self.view_functions[rule.endpoint](**req.view_args)
  File "/usr/local/lib/python3.9/site-packages/connexion/decorators/decorator.py", line 68, in wrapper
    response = function(request)
  File "/usr/local/lib/python3.9/site-packages/connexion/decorators/uri_parsing.py", line 149, in wrapper
    response = function(request)
  File "/usr/local/lib/python3.9/site-packages/connexion/decorators/validation.py", line 399, in wrapper
    return function(request)
  File "/usr/local/lib/python3.9/site-packages/connexion/decorators/response.py", line 112, in wrapper
    response = function(request)
  File "/usr/local/lib/python3.9/site-packages/connexion/decorators/parameter.py", line 116, in wrapper
    return function(**kwargs)
  File "/usr/local/lib/python3.9/site-packages/airflow/api_connexion/security.py", line 49, in decorated
    return func(*args, **kwargs)
  File "/usr/local/lib/python3.9/site-packages/airflow/api_connexion/endpoints/task_endpoint.py", line 67, in get_tasks
    return task_collection_schema.dump(task_collection)
  File "/usr/local/lib/python3.9/site-packages/marshmallow/schema.py", line 552, in dump
    result = self._serialize(processed_obj, many=many)
  File "/usr/local/lib/python3.9/site-packages/marshmallow/schema.py", line 520, in _serialize
    value = field_obj.serialize(attr_name, obj, accessor=self.get_attribute)
  File "/usr/local/lib/python3.9/site-packages/marshmallow/fields.py", line 338, in serialize
    return self._serialize(value, attr, obj, **kwargs)
  File "/usr/local/lib/python3.9/site-packages/marshmallow/fields.py", line 765, in _serialize
    return [self.inner._serialize(each, attr, obj, **kwargs) for each in value]
  File "/usr/local/lib/python3.9/site-packages/marshmallow/fields.py", line 765, in <listcomp>
    return [self.inner._serialize(each, attr, obj, **kwargs) for each in value]
  File "/usr/local/lib/python3.9/site-packages/marshmallow/fields.py", line 634, in _serialize
    return schema.dump(nested_obj, many=many)
  File "/usr/local/lib/python3.9/site-packages/marshmallow/schema.py", line 552, in dump
    result = self._serialize(processed_obj, many=many)
  File "/usr/local/lib/python3.9/site-packages/marshmallow/schema.py", line 520, in _serialize
    value = field_obj.serialize(attr_name, obj, accessor=self.get_attribute)
  File "/usr/local/lib/python3.9/site-packages/marshmallow/fields.py", line 338, in serialize
    return self._serialize(value, attr, obj, **kwargs)
  File "/usr/local/lib/python3.9/site-packages/marshmallow/fields.py", line 1870, in _serialize
    return self._serialize_method(obj)
  File "/usr/local/lib/python3.9/site-packages/airflow/api_connexion/schemas/task_schema.py", line 70, in get_params
    return {k: v.dump() for k, v in params.items()}
  File "/usr/local/lib/python3.9/site-packages/airflow/api_connexion/schemas/task_schema.py", line 70, in <dictcomp>
    return {k: v.dump() for k, v in params.items()}
AttributeError: 'dict' object has no attribute 'dump'

@norm

Hmm thats a new one. @norm would you mind taking a look at get tasks?

Just wanted to update, that this appears to only be happening for DAGs that have TaskGroups. When viewing my "non-taskgroup" DAGs, the grid-view works as expected. When trying to select individual tasks on a DAG with task groups (and therefore have expand/collapsible grid), clicking on an individual task the entire grid view disappears.

Can someone give us a DAG and re-production steps please?

Also, what are the other js console errors aside from the get task endpoint? Even if that endpoint throws an error, the UI shouldn't crash

I get first error message twice when then the second when the Grid view loads:

# First
VM35:1          
GET https://****************/api/v1/dags/some_dag/tasks 500
    (anonymous)                 @   VM35:1
    (anonymous)                 @   tree.ac7202f….js:2
    e.exports                   @   tree.ac7202f….js:2
    e.exports                   @   tree.ac7202f….js:2
    c.request                   @   tree.ac7202f….js:2
    r.forEach.c.<computed>      @   tree.ac7202f….js:2
    (anonymous)                 @   tree.ac7202f….js:2
    (anonymous)                 @   tree.ac7202f….js:2
    fetchFn                     @   tree.ac7202f….js:2
    s                           @   tree.ac7202f….js:2
    c                           @   tree.ac7202f….js:2
    t.fetch                     @   tree.ac7202f….js:2
    n.executeFetch              @   tree.ac7202f….js:2
    n.onSubscribe               @   tree.ac7202f….js:2
    t.subscribe                 @   tree.ac7202f….js:2
    (anonymous)                 @   tree.ac7202f….js:2
    Dl                          @   tree.ac7202f….js:2
    t.unstable_runWithPriority  @   tree.ac7202f….js:2
    Go                          @   tree.ac7202f….js:2
    Bl                          @   tree.ac7202f….js:2
    bl                          @   tree.ac7202f….js:2
    (anonymous)                 @   tree.ac7202f….js:2
    t.unstable_runWithPriority  @   tree.ac7202f….js:2
    Go                          @   tree.ac7202f….js:2
    Yo                          @   tree.ac7202f….js:2
    Xo                          @   tree.ac7202f….js:2
    xl                          @   tree.ac7202f….js:2
    is                          @   tree.ac7202f….js:2
    t.render                    @   tree.ac7202f….js:2
    (anonymous)                 @   tree.ac7202f….js:2
    n                           @   tree.ac7202f….js:2
    (anonymous)                 @   tree.ac7202f….js:2
    n                           @   tree.ac7202f….js:2
    (anonymous)                 @   tree.ac7202f….js:2
    (anonymous)                 @   tree.ac7202f….js:2
    (anonymous)                 @   tree.ac7202f….js:2
    (anonymous)                 @   tree.ac7202f….js:2

# Second
Error: Request failed with status code 500
    at e.exports (tree.ac7202f96db7173e9baf.js:2:141127)
    at e.exports (tree.ac7202f96db7173e9baf.js:2:297309)
    at XMLHttpRequest.E (tree.ac7202f96db7173e9baf.js:2:139518)

And finally, the following error twice when I click one of the tasks:

tree.ac7202f96db7173e9baf.js:2 
TypeError: Cannot read properties of undefined (reading 'tasks')
    at B_ (tree.ac7202f96db7173e9baf.js:2:739128)
    at ui (tree.ac7202f96db7173e9baf.js:2:228656)
    at Qu (tree.ac7202f96db7173e9baf.js:2:280737)
    at Pl (tree.ac7202f96db7173e9baf.js:2:268010)
    at jl (tree.ac7202f96db7173e9baf.js:2:267938)
    at Cl (tree.ac7202f96db7173e9baf.js:2:267801)
    at bl (tree.ac7202f96db7173e9baf.js:2:264788)
    at tree.ac7202f96db7173e9baf.js:2:214575
    at t.unstable_runWithPriority (tree.ac7202f96db7173e9baf.js:2:291432)
    at Go (tree.ac7202f96db7173e9baf.js:2:214352)
cu                          @ tree.ac7202f96db7173e9baf.js:2
n.callback                  @ tree.ac7202f96db7173e9baf.js:2
ma                          @ tree.ac7202f96db7173e9baf.js:2
gu                          @ tree.ac7202f96db7173e9baf.js:2
Il                          @ tree.ac7202f96db7173e9baf.js:2
t.unstable_runWithPriority  @ tree.ac7202f96db7173e9baf.js:2
Go                          @ tree.ac7202f96db7173e9baf.js:2
Tl                          @ tree.ac7202f96db7173e9baf.js:2
bl                          @ tree.ac7202f96db7173e9baf.js:2
(anonymous)                 @ tree.ac7202f96db7173e9baf.js:2
t.unstable_runWithPriority  @ tree.ac7202f96db7173e9baf.js:2
Go                          @ tree.ac7202f96db7173e9baf.js:2
Yo                          @ tree.ac7202f96db7173e9baf.js:2
Xo                          @ tree.ac7202f96db7173e9baf.js:2
Me                          @ tree.ac7202f96db7173e9baf.js:2
Yt                          @ tree.ac7202f96db7173e9baf.js:2

Great thanks -- the second error is likely a side effect of the first.

What error appears in the webserver logs when you do the first action?

@rotemseekingalpha

The web server error is what I posted above... ends with:

AttributeError: 'dict' object has no attribute 'dump'

I too am getting the same JS errors that @rotemseekingalpha is showing as well. My theory about taskgroups however, appears wrong because it's also happening on "non-taskgroup" DAGs. In my case they're "previously parsed" dags, and were there before the upgrade to 2.3.0 - so I assume there's something in the migration for these dags or that happened to them in the past that 2.3.0 just doesn't like.

We have the same problem in some DAGs (others working as expected) after upgrade from 1.0.15 to 2.5.3).
No errors both in browser console and webserver logs

I believe I found the root cause of the problem.

A new feature was introduced in 2.3.0 - "details drawer" in Grid View (#22123). Under the hood in calls /confirm endpoint here. The problem is that this endpoint requires
CAN_EDIT RESOURCE_DAG and CAN_EDIT RESOURCE_TASK_INSTANCE permissions, which are obviously not granted to the Viewer role by default.

As a result the /confirm XHR call is redirected to the login page and returns its HTML content, which breaks the logic behind the details drawer and we get a blank page.

On top of that #30373 introduced more improvements to the details drawer and now there's also a POST being sent to /clear which requires even more permissions.

EDIT: It's not actually the details drawer itself that is problematic, but its "Mark task as" modals.

@bbovenzi

cc: @bbovenzi @pierrejeambrun ^^

Navigation Menu

Repository navigation

After upgrade from Airflow 2.2.4, grid disappears for some DAGs #23588

Apache Airflow version

What happened

What you think should happen instead

How to reproduce

Operating System

Versions of Apache Airflow Providers

Deployment

Deployment details

Anything else

Are you willing to submit PR?

Code of Conduct

38 remaining items

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

After upgrade from Airflow 2.2.4, grid disappears for some DAGs #23588

Description

Apache Airflow version

What happened

What you think should happen instead

How to reproduce

Operating System

Versions of Apache Airflow Providers

Deployment

Deployment details

Anything else

Are you willing to submit PR?

Code of Conduct

Activity

boring-cyborg commented on May 9, 2022

rotemseekingalpha commented on May 9, 2022

jpipas commented on May 9, 2022

bbovenzi commented on May 9, 2022

jpipas commented on May 14, 2022

ashb commented on May 17, 2022

bbovenzi commented on May 18, 2022

rotemseekingalpha commented on May 19, 2022

ashb commented on May 19, 2022

jpipas commented on May 19, 2022

38 remaining items

radiophysicist commented on Jun 23, 2023

mjkonarski-b commented on Jun 26, 2023

potiuk commented on Jun 26, 2023

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions