Skip to content

Flag execute_tasks_new_python_interpreter as Edge-only in docs and lint - #70901

Closed
ignacioparicio wants to merge 2 commits into
apache:mainfrom
ignacioparicio:flag-execute-tasks-new-python-interpreter-ineffective-on-celery
Closed

ignacioparicio wants to merge 2 commits into
apache:mainfrom
ignacioparicio:flag-execute-tasks-new-python-interpreter-ineffective-on-celery

Conversation

@ignacioparicio

@ignacioparicio ignacioparicio commented Aug 1, 2026 •

Copy link
Copy Markdown
Contributor

[core] execute_tasks_new_python_interpreter is still documented as a live option, but since Airflow 3.0 CeleryExecutor and LocalExecutor ignore it. The only thing left that reads it is the Edge worker.

That happened in #46265 (I think as an unintended side effect), as the code that reads the variable is behind a if not AIRFLOW_V_3_0_PLUS: gate. What is left is somewhat inconsistent.

Why it's inconsistent
  • CeleryExecutor and LocalExecutor silently ignore execute_tasks_new_python_interpreter. (Note: The Edge worker does honour it on Airflow 3, added in Allow using fresh interpreter besides fork() in Edge Worker #65943 ).
  • config.yml documents the option under [core] with no deprecation marker.
  • plugins.rst still tells users to set it to pick up plugin changes without restarting the worker.
  • airflow config lint, the designated 2 to 3 upgrade check, does not mention it, so an upgrade gives no signal at all.

What this PR does:

  • config.yml: adds a note to the option saying that since Airflow 3.0 only the Edge worker reads it.
  • plugins.rst: removes the advice to set it (that workaround no longer works on Celery or Local).
  • airflow config lint: adds an entry so the 2-to-3 upgrade check flags the option when it is set, with a pointer to what changed.

No behaviour changes. The goal is just that an Airflow 2->3 upgrade gives some signal instead of none.

If that's OK I leave restoring the functionality of execute_tasks_new_python_interpreter to a separate discussion (#70280 looks like the right place).

Optional context: how we ran into it

We hit this upgrading a CeleryExecutor deployment from 2.11 to 3.3. We had the option set to True, so every task ran in a fresh interpreter. On 3.3 tasks are forked from the Celery prefork worker instead, and any task that starts a multiprocessing child now inherits the worker's daemon flag and dies with daemonic processes are not allowed to have children. Nothing in the upgrade told us the option had stopped working; we found it by reading the provider source.


Was generative AI tooling used to co-author this PR?
  • Yes — Claude Code (Opus 5)

Used to map the blast radius: which code paths, docs, and CLI checks still reference the option, and which components still read it.

@boring-cyborg

boring-cyborg Bot commented Aug 1, 2026

Copy link
Copy Markdown

Congratulations on your first Pull Request and welcome to the Apache Airflow community! If you have any issues or are unsure about any anything please check our Contributors' Guide
Here are some useful points:

  • Pay attention to the quality of your code (ruff, mypy and type annotations). Our prek-hooks will help you with that.
  • In case of a new feature add useful documentation (in docstrings or in docs/ directory). Adding a new operator? Check this short guide Consider adding an example Dag that shows how users should use it.
  • Consider using Breeze environment for testing locally, it's a heavy docker but it ships with a working Airflow and a lot of integrations.
  • Be patient and persistent. It might take some time to get a review or get the final approval from Committers.
  • Please follow ASF Code of Conduct for all communication including (but not limited to) comments on Pull Requests, Mailing list and Slack.
  • Be sure to read the Airflow Coding style.
  • Always keep your Pull Requests rebased, otherwise your build might fail due to changes not related to your commits.
    Apache Airflow is a community-driven project and together we are making it better 🚀.
    In case of doubts contact the developers at:
    Mailing List: dev@airflow.apache.org
    Slack: https://s.apache.org/airflow-slack

@ignacioparicio ignacioparicio changed the title Warn when execute_tasks_new_python_interpreter has no effect Flag execute_tasks_new_python_interpreter as Edge-only in docs and lint Aug 1, 2026
@github-actions

Copy link
Copy Markdown
Contributor

This pull request has been automatically marked as stale because it has not had recent activity. It will be closed in 5 days if no further activity occurs. Thank you for your contributions.

@github-actions github-actions Bot added the stale Stale PRs per the .github/workflows/stale.yml policy file label Sep 19, 2026
@ignacioparicio

Copy link
Copy Markdown
Contributor Author

I still think this docs-only change would save others some headaches, especially since this was an Airflow 3 regression

@ignacioparicio

Copy link
Copy Markdown
Contributor Author

on second look looks like this is superseded by #72164, which restores execute_tasks_new_python_interpreter. Closing this.

@ignacioparicio
ignacioparicio deleted the flag-execute-tasks-new-python-interpreter-ineffective-on-celery branch September 19, 2026 10:34
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area:CLI area:ConfigTemplates area:plugins kind:documentation stale Stale PRs per the .github/workflows/stale.yml policy file

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant