Skip to content

Stop loading the task runner on every Airflow process startup - #74496

Merged
kaxil merged 2 commits into
apache:mainfrom
astronomer:skip-task-runner-import-when-masking
Oct 11, 2026
Merged

kaxil merged 2 commits into
apache:mainfrom
astronomer:skip-task-runner-import-when-masking

Conversation

@kaxil

@kaxil kaxil commented Oct 9, 2026

Copy link
Copy Markdown
Member

mask_secret() imported airflow.sdk.execution_time.task_runner only to check whether SUPERVISOR_COMMS is set. settings.initialize() calls conf.mask_secrets() at startup, which calls mask_secret() for each sensitive config value, so every process that imports airflow loaded the whole task runtime: the operator base classes, the API client, the coordinators, and the SDK configuration with a second full provider discovery. That covers every CLI command and every scheduler, API server, triggerer and dag-processor start. Outside a task process none of it is used.

SUPERVISOR_COMMS is now read from sys.modules. That loses nothing: every place that sets it (the task runner's main(), the dag-processor, the triggerer, the callback supervisor, in-process dag.test) assigns it on the imported airflow.sdk.execution_time.task_runner module, so the module is always in sys.modules when comms exist. The module itself only declares the name as an annotation, so importing it never produced a value in a process where nothing had set one.

Measured in the CI image (Python 3.11, all providers installed, bytecode cached), median of 9 runs, alternating with main:

main This PR
python -c "import airflow" 1.36s 0.80s
airflow version 1.37s 1.01s
airflow --help 1.41s 1.03s
airflow config get-value core executor 1.35s 1.04s
Shell completion of airflow dags list --o 1.24s 0.91s

The new subprocess test fails on main, because task_runner ends up in sys.modules after a single mask_secret() call. The two forwarding tests keep the task-execution path covered for both mask_secret and amask_secret.


  • Read the Pull Request Guidelines for more information. Note: commit author/co-author name and email in commits become permanently public when merged.
  • For fundamental code changes, an Airflow Improvement Proposal (AIP) is needed.
  • When adding dependency, check compliance with the ASF 3rd Party License Policy.
  • For significant user-facing changes create newsfragment: {pr_number}.significant.rst, in airflow-core/newsfragments. You can add this file in a follow-up commit after the PR is created so you know the PR number.

mask_secret() imported airflow.sdk.execution_time.task_runner only to check
whether SUPERVISOR_COMMS is set. settings.initialize() masks sensitive config
values at startup, so every CLI command and every scheduler, API server and
dag-processor start paid for importing the whole task runtime, including a
second provider discovery for the SDK configuration. Every process that sets
SUPERVISOR_COMMS has already imported task_runner, so it is now read from
sys.modules instead.
@kaxil
kaxil force-pushed the skip-task-runner-import-when-masking branch from 7caa2aa to cf21563 Compare October 9, 2026 14:29
…artup

Masking a config value at startup no longer imports the task runner, and
non-DB tests had quietly relied on that import:

- configure_orm() never succeeds against the bad connection string non-DB
  tests use, so the hook that imports every model before SQLAlchemy
  configures mappers was never installed. The task runner import was what
  registered DagBundleModel, so Team.dag_bundles stopped resolving and the
  first ORM object built in a worker failed every later test. Install the
  same hook in the pytest plugin for non-DB runs.
- test_execute_deferrable patched builtins.open for the whole process. The
  task runner is now first imported inside execute(), so the metrics
  registry read the mocked file instead of its YAML. Patch open only in
  the module that reads the kube config.
@kaxil
kaxil merged commit 72ac4fb into apache:main Oct 11, 2026
158 checks passed
@kaxil
kaxil deleted the skip-task-runner-import-when-masking branch October 11, 2026 23:18
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants