Task in 3.0 gets stuck when fetching larger XCom that can be passed without issue in 2.10 + "Direct database access not allowed error upon clearing said task"

Apache Airflow version

3.0.0

If "Other Airflow 2 version" selected, which one?

current main

What happened?

I encountered this passing an embedding vector (list of lists) as XCom using a dag that works in 2.10. The below dag reproduces the issue:

from airflow.sdk import dag
from airflow.decorators import task 

@dag() 
def lazy_list_xcom():

    @task 
    def create_long_list():
        big_list = [[i for i in range(100)] for _ in range(1000)]
        return big_list
    
    @task
    def print_list(x):
        print(x)


    print_list(create_long_list())


lazy_list_xcom()

When running this dag the create_long_list task finishes successfully and the print_list task gets stuck in running state (without any error that I can see in the components).

Once I clear the print_list task I get the following error in the logs:

Log message source details: sources=["/root/airflow/logs/dag_id=lazy_list_xcom/run_id=manual__2025-04-06T18:45:31.764375+00:00/task_id=print_list/attempt=1.log"]
[[2](http://localhost:28080/dags/lazy_list_xcom/runs/manual__2025-04-06T18:45:31.764375+00:00/tasks/print_list?try_number=1#2)025-04-06, 18:45:33] INFO - DAG bundles loaded: dags-folder: source="airflow.dag_processing.bundles.manager.DagBundlesManager"
[2025-04-06, 18:45:[3](http://localhost:28080/dags/lazy_list_xcom/runs/manual__2025-04-06T18:45:31.764375+00:00/tasks/print_list?try_number=1#3)3] INFO - Filling up the DagBag from /files/dags/lazy_list_xcom.py: source="airflow.models.dagbag.DagBag"
[2025-04-06, 18:[4](http://localhost:28080/dags/lazy_list_xcom/runs/manual__2025-04-06T18:45:31.764375+00:00/tasks/print_list?try_number=1#4)6:53] ERROR - Exception rendering Jinja template for task 'print_list', field 'op_args'. Template: (XComArg(<Task(_PythonDecoratedOperator): create_long_list>),): source="airflow.sdk.definitions._internal.abstractoperator"
RuntimeError: Direct database access via the ORM is not allowed in Airflow 3.0
File "/opt/airflow/task-sdk/src/airflow/sdk/definitions/_internal/abstractoperator.py", line 379 in _do_render_template_fields

File "/opt/airflow/task-sdk/src/airflow/sdk/definitions/_internal/templater.py", line 193 in render_template

File "/opt/airflow/task-sdk/src/airflow/sdk/definitions/_internal/templater.py", line 193 in <genexpr>

File "/opt/airflow/task-sdk/src/airflow/sdk/definitions/_internal/templater.py", line 189 in render_template

File "/opt/airflow/task-sdk/src/airflow/sdk/definitions/xcom_arg.py", line 345 in resolve

File "/opt/airflow/task-sdk/src/airflow/sdk/execution_time/task_runner.py", line 348 in xcom_pull

File "/opt/airflow/task-sdk/src/airflow/sdk/bases/xcom.py", line 256 in get_one

File "/opt/airflow/task-sdk/src/airflow/sdk/execution_time/task_runner.py", line 564 in get_message

File "/opt/airflow/airflow-core/src/airflow/jobs/scheduler_job_runner.py", line 227 in _exit_gracefully

File "/opt/airflow/airflow-core/src/airflow/utils/session.py", line 101 in wrapper

File "/usr/local/lib/python3.9/contextlib.py", line 119 in __enter__

File "/opt/airflow/airflow-core/src/airflow/utils/session.py", line 41 in create_session

File "/opt/airflow/task-sdk/src/airflow/sdk/execution_time/supervisor.py", line 218 in __init__

The movie:

2025-04-06_20-52-26.mp4

What you think should happen instead?

The task should run successfully and print the list of lists.

How to reproduce

Run the dag above. See the second task get stuck. Then clear the task to see the database access error.

Operating System

MacOs

Versions of Apache Airflow Providers

latest main

Deployment

Other

Deployment details

breeze

Anything else?

I think this might be an issue that occurs dependent on lazy xcom access... small XComs work fine.

Are you willing to submit PR?

Yes I am willing to submit a PR!

Code of Conduct

I agree to follow this project's Code of Conduct

I can reproduce this issue. Also, there's a UI problem — when we try to view this large XCom value in the UI, all table columns appear blank except for the 'value' column. I will raise a separate issue for this

@kaxil

cc: @kaxil @amoghrajesh @ashb

We are now using a client-server model and basically sending a lot of data over the network for this.

To give an estimate, your data: big_list = [[i for i in range(100)] for _ in range(1000)] will be roughly 380KB. It shoudln't usually be an issue to do this, but I think this is a valid enough case to use custom backends?

I can see this in the task though:

2025-04-07 07:05:51 [debug    ] Received message from task runner [supervisor] msg=GetXCom(key='return_value', dag_id='lazy_list_xcom', run_id='manual__2025-04-07T07:05:49.119778+00:00', task_id='create_long_list', map_index=None, include_prior_dates=False, type='GetXCom')
[2025-04-07T07:05:51.279+0000] {_client.py:1026} INFO - HTTP Request: GET http://localhost:8080/execution/xcoms/lazy_list_xcom/manual__2025-04-07T07:05:49.119778+00:00/create_long_list/return_value "HTTP/1.1 200 OK"

I tried with the API directly, its not an issue. It responds fine:

Let me investigate it further

@cmarteepants

We are now using a client-server model and basically sending a lot of data over the network for this.

To give an estimate, your data: big_list = [[i for i in range(100)] for _ in range(1000)] will be roughly 380KB. It shoudln't usually be an issue to do this, but I think this is a valid enough case to use custom backends?

It is definitely a situation where we would (and will in this case, the pipeline where I noticed the issue is for a tutorial) recommend a custom xcom backend once the pipeline is moved to production.

But many users will pass XCom of this size without a custom backend in their existing dags (for example with a larger metadb and frequent cleanup this would work fine in production even if not best practice). So it would be quite a breaking change we'd have to call out and that is tricky to assess across many pipelines since users likely wont know the size of all their XCom they pass, if they'd be over the threshold or not. It would be tedious to assess if they need to make a change and to require custom xcom backends for local development...

So if it is possible I'd strongly favor not limiting the XCom size that can be passed (beyond the db-level limits) to have parity with Airflow 2.10.

If it is not possible I'd need the size cutoff as soon as possible to put into upgrading checklists as a breaking change. And I'd vote to have an error message show in the task logs that explains that the issue was an XCom was too large and to please use a custom XCom backend.

cc: @cmarteepants just fyi there might be another thing to call out in release notes/ upgrade guides

@TJaniF

Thanks for the explanation @TJaniF!

I am not saying / enforcing any threshold here. That was just an idea i was throwing around.

I intend to fix this issue and i am working on it actively

Thanks for the explanation @TJaniF!

I am not saying / enforcing any threshold here. That was just an idea i was throwing around.

I intend to fix this issue and i am working on it actively

That sounds great! And thanks for taking this on, I was quite confused at first, not having touched the metadb at all in my dag, how I managed to get that specific error 😅

And I agree that custom xcom backends should almost be a default best practice, especially now that there is an easier option to set one up with the common IO provider! I'll make sure to call that out more in our guides and webinars going forward :)

@TJaniF

Fixed by #48949

@TJaniF please validate and let me know if it works.

Fixed by #48949

@TJaniF please validate and let me know if it works.

Can confirm it works great! Thank you so much! :)

Navigation Menu

Repository navigation

Task in 3.0 gets stuck when fetching larger XCom that can be passed without issue in 2.10 + "Direct database access not allowed error upon clearing said task" #48852

Apache Airflow version

If "Other Airflow 2 version" selected, which one?

What happened?

What you think should happen instead?

How to reproduce

Operating System

Versions of Apache Airflow Providers

Deployment

Deployment details

Anything else?

Are you willing to submit PR?

Code of Conduct

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Task in 3.0 gets stuck when fetching larger XCom that can be passed without issue in 2.10 + "Direct database access not allowed error upon clearing said task" #48852

Description

Apache Airflow version

If "Other Airflow 2 version" selected, which one?

What happened?

What you think should happen instead?

How to reproduce

Operating System

Versions of Apache Airflow Providers

Deployment

Deployment details

Anything else?

Are you willing to submit PR?

Code of Conduct

Activity

vatsrahul1001 commented on Apr 7, 2025

vatsrahul1001 commented on Apr 7, 2025

amoghrajesh commented on Apr 7, 2025

amoghrajesh commented on Apr 7, 2025

TJaniF commented on Apr 7, 2025

amoghrajesh commented on Apr 7, 2025

TJaniF commented on Apr 7, 2025

amoghrajesh commented on Apr 10, 2025

TJaniF commented on Apr 10, 2025

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions