Repository navigation
Stop loading the task runner on every Airflow process startup - #74496
Merged
kaxil merged 2 commits intoOct 11, 2026
Merged
Conversation
ashb
approved these changes
Oct 9, 2026
mask_secret() imported airflow.sdk.execution_time.task_runner only to check whether SUPERVISOR_COMMS is set. settings.initialize() masks sensitive config values at startup, so every CLI command and every scheduler, API server and dag-processor start paid for importing the whole task runtime, including a second provider discovery for the SDK configuration. Every process that sets SUPERVISOR_COMMS has already imported task_runner, so it is now read from sys.modules instead.
kaxil
force-pushed
the
skip-task-runner-import-when-masking
branch
from
October 9, 2026 14:29
7caa2aa to
cf21563
Compare
…artup Masking a config value at startup no longer imports the task runner, and non-DB tests had quietly relied on that import: - configure_orm() never succeeds against the bad connection string non-DB tests use, so the hook that imports every model before SQLAlchemy configures mappers was never installed. The task runner import was what registered DagBundleModel, so Team.dag_bundles stopped resolving and the first ORM object built in a worker failed every later test. Install the same hook in the pytest plugin for non-DB runs. - test_execute_deferrable patched builtins.open for the whole process. The task runner is now first imported inside execute(), so the metrics registry read the mocked file instead of its YAML. Patch open only in the module that reads the kube config.
kaxil
requested review from
hussein-awala,
jedcunningham and
jscheffl
as code owners
October 10, 2026 11:45
This was referenced Oct 13, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
mask_secret()importedairflow.sdk.execution_time.task_runneronly to check whetherSUPERVISOR_COMMSis set.settings.initialize()callsconf.mask_secrets()at startup, which callsmask_secret()for each sensitive config value, so every process that importsairflowloaded the whole task runtime: the operator base classes, the API client, the coordinators, and the SDK configuration with a second full provider discovery. That covers every CLI command and every scheduler, API server, triggerer and dag-processor start. Outside a task process none of it is used.SUPERVISOR_COMMSis now read fromsys.modules. That loses nothing: every place that sets it (the task runner'smain(), the dag-processor, the triggerer, the callback supervisor, in-processdag.test) assigns it on the importedairflow.sdk.execution_time.task_runnermodule, so the module is always insys.moduleswhen comms exist. The module itself only declares the name as an annotation, so importing it never produced a value in a process where nothing had set one.Measured in the CI image (Python 3.11, all providers installed, bytecode cached), median of 9 runs, alternating with
main:mainpython -c "import airflow"airflow versionairflow --helpairflow config get-value core executorairflow dags list --oThe new subprocess test fails on
main, becausetask_runnerends up insys.modulesafter a singlemask_secret()call. The two forwarding tests keep the task-execution path covered for bothmask_secretandamask_secret.{pr_number}.significant.rst, in airflow-core/newsfragments. You can add this file in a follow-up commit after the PR is created so you know the PR number.