Couldn't able to access Airflow variables inside @task.virtualenv operator

Apache Airflow version

3.0.1

If "Other Airflow 2 version" selected, which one?

No response

What happened?

I am trying to acess airflow variable inside @task.virtualenv operator but it's failing with below error

What you think should happen instead?

No response

How to reproduce

Create task using @task.virtualenv
Try to access airflow variables inside it

Operating System

mac

Versions of Apache Airflow Providers

No response

Deployment

Other

Deployment details

Airflow Standalone

Anything else?

No response

Are you willing to submit PR?

Yes I am willing to submit a PR!

Code of Conduct

I agree to follow this project's Code of Conduct

Thanks for opening your first issue here! Be sure to follow the issue template! If you are willing to raise PR to address this issue please do so, no need to wait for approval.

@amoghrajesh

Yep. We likely need to make task-sdk works across venvs.

CC: @amoghrajesh @ashb @kaxil

Allow virtualenv code to access connections/variables and send logs (a…

Just upgraded to v3.1.3.

I'm trying to get connection inside @task.virtualenv. The connection is defined as an environment variable. I can successfully read it with os.environ["AIRFLOW_CONN_VAULT_DEFAULT"].strip(), but not with Connection.get("vault_default"):

[2025-11-15 09:56:20] INFO - Executing cmd: uv venv --allow-existing --seed --python python --system-site-packages /tmp/venvys70w7pu
[2025-11-15 09:56:20] INFO - Output:
[2025-11-15 09:56:21] INFO - Using CPython 3.12.12 interpreter at: /usr/python/bin/python
[2025-11-15 09:56:21] INFO - Creating virtual environment with seed packages at: /tmp/venvys70w7pu
[2025-11-15 09:56:21] INFO -  + pip==25.3
[2025-11-15 09:56:21] INFO - Executing cmd: uv pip install --python /tmp/venvys70w7pu/bin/python -r /tmp/venvys70w7pu/requirements.txt
[2025-11-15 09:56:21] INFO - Output:
[2025-11-15 09:56:22] INFO - Using Python 3.12.12 environment at: /tmp/venvys70w7pu
[2025-11-15 09:56:22] INFO - Resolved 9 packages in 292ms
[2025-11-15 09:56:22] INFO - Downloading psycopg-binary (4.2MiB)
[2025-11-15 09:56:22] INFO -  Downloading psycopg-binary
[2025-11-15 09:56:22] INFO - Prepared 9 packages in 287ms
[2025-11-15 09:56:22] INFO - Installed 9 packages in 24ms
[2025-11-15 09:56:22] INFO -  + clickhouse-driver==0.2.9
[2025-11-15 09:56:22] INFO -  + psycopg==3.2.12
[2025-11-15 09:56:22] INFO -  + psycopg-binary==3.2.12
[2025-11-15 09:56:22] INFO -  + python-dotenv==1.2.1
[2025-11-15 09:56:22] INFO -  + pytz==2025.2
[2025-11-15 09:56:22] INFO -  + stamina==25.1.0
[2025-11-15 09:56:22] INFO -  + tenacity==9.1.2
[2025-11-15 09:56:22] INFO -  + typing-extensions==4.15.0
[2025-11-15 09:56:22] INFO -  + tzlocal==5.3.1
[2025-11-15 09:56:22] INFO - Use 'pickle' as serializer.
[2025-11-15 09:56:22] INFO - Executing cmd: /tmp/venvys70w7pu/bin/python /tmp/venv-call1zlxzyye/script.py /tmp/venv-call1zlxzyye/script.in /tmp/venv-call1zlxzyye/script.out /tmp/venv-call1zlxzyye/string_args.txt /tmp/venv-call1zlxzyye/termination.log /tmp/venv-call1zlxzyye/airflow_context.json
[2025-11-15 09:56:22] INFO - Output:
[2025-11-15 09:56:26] INFO - Traceback (most recent call last):
[2025-11-15 09:56:26] INFO -   File "/tmp/venv-call1zlxzyye/script.py", line 63, in <module>
[2025-11-15 09:56:26] INFO -     res = callable_virtualenv(*arg_dict["args"], **arg_dict["kwargs"])
[2025-11-15 09:56:26] INFO -           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[2025-11-15 09:56:26] INFO -   File "/tmp/venv-call1zlxzyye/script.py", line 31, in callable_virtualenv
[2025-11-15 09:56:26] INFO -     conn = Connection.get("vault_default")
[2025-11-15 09:56:26] INFO -            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[2025-11-15 09:56:26] INFO -   File "/home/airflow/.local/lib/python3.12/site-packages/airflow/sdk/definitions/connection.py", line 225, in get
[2025-11-15 09:56:26] INFO -     return _get_connection(conn_id)
[2025-11-15 09:56:26] INFO -            ^^^^^^^^^^^^^^^^^^^^^^^^
[2025-11-15 09:56:26] INFO -   File "/home/airflow/.local/lib/python3.12/site-packages/airflow/sdk/execution_time/context.py", line 174, in _get_connection
[2025-11-15 09:56:26] INFO -     raise AirflowNotFoundException(f"The conn_id `{conn_id}` isn't defined")
[2025-11-15 09:56:26] INFO - airflow.exceptions.AirflowNotFoundException: The conn_id `vault_default` isn't defined

As a workaround i created my own VaultHook to avoid calling self.connection = self.get_connection(vault_conn_id) and replaced it with

conn_var = f"AIRFLOW_CONN_{self.default_conn_name.upper()}"
self.connection = Connection.from_json(
    conn_id=self.default_conn_name, value=os.environ[conn_var].strip()
)

@spirkaa

@spirkaa This didn't make it in to 3.1.3, the change only got included back in core airflow (non providers) for 3.1.4 which is not yet released.

😉

[v3-1-test] Add support for "reconnecting" Supervisor Comms and logs …

Allow virtualenv code to access connections/variables and send logs (a…

Fixes apache#57515 - Fix test_async_write_logs_should_execute_success…

[v3-1-test] Add support for "reconnecting" Supervisor Comms and logs …

Allow virtualenv code to access connections/variables and send logs (a…

@spirkaa

This bug still seems to exist in a different form, no crash but indefinitely hangs.
Exactly the same conditions as @spirkaa , Airflow 3.1.7.

Inside @task.virtualenv:

@task.virtualenv(
    requirements="requirements/tasks/requirements.txt",
    system_site_packages=False,
    venv_cache_path="/tmp/",
    retries=1,
)
def import_data():                                                                                                                                                                                            

    from airflow.sdk import Connection

    pg_conn = Connection.get(conn_id="pg")  # hangs forever

Add support for "reconnecting" Supervisor Comms and logs in task proc…

Allow virtualenv code to access connections/variables and send logs (a…

Navigation Menu

Repository navigation

Couldn't able to access Airflow variables inside @task.virtualenv operator #51422

Apache Airflow version

If "Other Airflow 2 version" selected, which one?

What happened?

What you think should happen instead?

How to reproduce

Operating System

Versions of Apache Airflow Providers

Deployment

Deployment details

Anything else?

Are you willing to submit PR?

Code of Conduct

34 remaining items

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Couldn't able to access Airflow variables inside @task.virtualenv operator #51422

Description

Apache Airflow version

If "Other Airflow 2 version" selected, which one?

What happened?

What you think should happen instead?

How to reproduce

Operating System

Versions of Apache Airflow Providers

Deployment

Deployment details

Anything else?

Are you willing to submit PR?

Code of Conduct

Activity

boring-cyborg commented on Jun 5, 2025

potiuk commented on Jun 5, 2025

34 remaining items

spirkaa commented on Nov 15, 2025

ashb commented on Nov 16, 2025

vperron commented on Mar 16, 2026

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions