Repository navigation
dags folder not present in PYTHONPATH #53617
Description
Apache Airflow version
3.0.3
If "Other Airflow 2 version" selected, which one?
No response
What happened?
The DAGs folder is expected in PYTHONPATH, and we should be able to import modules in the folder, as specified in https://airflow.apache.org/docs/apache-airflow/stable/administration-and-deployment/modules_management.html
However after Airflow 3.0.2 the folder is no longer exist in PYTHONPATH and we cannot import modules in dag files.
What you think should happen instead?
No response
How to reproduce
- run
airflow info, the dags folder is not be presented
Paths info
airflow_home | /app
system_path | /opt/oracle/instantclient_19_5:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
python_path | /usr/local/bin:/usr/lib64/python39.zip:/usr/lib64/python3.9:/usr/lib64/python3.9/lib-dynload:/usr/local/lib64/python3.9/site-packages:/usr/local/lib/python3.9/site-packages:/usr/lib64/python3.9/site-packages:/usr/lib/python3.9/site-packages:/app/config:/app/plugins
airflow_on_path | True
- Import a module in a dag file, the module should be in the dags folder. It will fail.
Operating System
rhel8
Versions of Apache Airflow Providers
No response
Deployment
Other 3rd-party Helm chart
Deployment details
No response
Anything else?
Related commit: b5db146
Are you willing to submit PR?
- Yes I am willing to submit a PR!
Code of Conduct
- I agree to follow this project's Code of Conduct
Activity
potiuk commentedon Jul 22, 2025
I believe this is a documentation issue. Since we hav dag bundles, which are really only evaluated by dag file processor, there is no reason why the old
dag foldershould be added to PYTHONPATH for all kinds ofairflowcommands.It might matter though for some commands (like dag test) that are supposed to be run in the environment where the dags are present (note that for example scheduler and api_server are not even supposed to have dag folder present). But even that is not supposed to change
airflow infooutput IMHO. And in your "task" context, the right folder from which Dag is imported should indeed be already in the PYTHONPATH (otherwise many things would stop working@jedcunningham @ephraimbuddy -> Am I right with this assesment?
potiuk commentedon Jul 22, 2025
Also if that will be confirmed - are you willing to help and update our docs @widewing ? I think - since you are apparently user of our documentation and unlike many users - read it, you'd be probably one of the best people to update the docs if it is clarified what the behaviour is and in the way that will be easy to understand by others like you who read it.
And it's easy - just click "suggest a change on this page" and you will get PR where you will be able to update the docs.
Can we count on your help there?
widewing commentedon Jul 22, 2025
Hi @potiuk I'm in the process of migrating Airflow 2 to Airflow 3 and I'm still learning the changes and new concepts. This issue was found indeed when it failed to test dags. I don't think I'm the best person to update the doc since I am not familiar with the new system yet and don't have much knowledge of expected behaviors.
jedcunningham commentedon Jul 22, 2025
You explained it well. I'll just add that there is no single "folder" to add any more - you can have many dag bundles at the same time! And some may not even be a meaningful folder for users (e.g. with git, the repo is manged by airflow itself).
That said, some cli commands are known to not add the bundle root to sys.path. @simi has a wip change to address it, or at least provide the ability to address it.
simi commentedon Jul 22, 2025
mentioned change is at simi@e9166cb, I'll open PR with some tests added soon 🙏
potiuk commentedon Jul 22, 2025
Cool. Just nice thing to mention that likely (as mentioned above) some CLI commands probably should not even try (api-server, scheduler) :). I think eventually - when we continue splitting our codebase, maybe some day we extract the dagbag (parsing) part to a separate shared library that will be used only by specialized distributions (triggerer, worker, dag processor) - and you will have only those commands available in
airflowcli that you have coming from installed distributions (but that's likely airflow 3.4 or 3.5 if at all)suhail-zemoso commentedon Jul 28, 2025
✅ Workarounds to Resolve the Issue
We’ve identified a few workarounds for the missing PYTHONPATH entry for the dags/ folder in Airflow 3.x:
Option 1 (✅ Recommended)
Move your custom module to a separate directory:
Preferably place it inside the plugins/ folder (which is already included in sys.path),
OR
Create a new directory (e.g., libs/) and manually add it to the PYTHONPATH (e.g., via environment variable or modifying sys.path).
Option 2 (Migration Support)
If you're migrating from Airflow 2.x and have already placed custom modules inside the dags/ folder:
You can continue using them only if you explicitly add the dags/ directory to the PYTHONPATH.
Option 3 (Dev Only)
⚠️ Note: This behavior is specific to Breeze and should not be relied on in production.
If you're using the Breeze development environment, /files/dags is implicitly added to sys.path at both parse time and runtime.
📌 Rationale
Keeping custom modules outside the dags/ folder promotes separation of concerns between DAG definitions and business logic.
It improves modularity, reusability, and testability of your Python code.
Avoids relying on environment-specific behavior — especially important because /dags being added to sys.path works in Breeze but fails silently in other production-like deployments.
@potiuk Can we add these as documentation?
simi commentedon Jul 28, 2025
@suhail-zemoso And how to modify
sys.path?This is IMHO regression since until 3.0.1 dags folder was in
sys.pathand by no announcement it was removed. It should be added back.Plugins seems to have different purpose. https://airflow.apache.org/docs/apache-airflow/stable/administration-and-deployment/plugins.html#what-for
suhail-zemoso commentedon Jul 28, 2025
Option 1: Modify sys.path in code
You can dynamically add the path at runtime within your DAG Python file
import sys
import os
Add the absolute path of libs/ to sys.path
libs_path = os.path.join(os.path.dirname(file), "..", "libs")
sys.path.insert(0, os.path.abspath(libs_path))
Now you can import your custom module
from my_module import my_function
Option 2: Set PYTHONPATH Environment Variable
permanently add a folder to sys.path by setting the PYTHONPATH environment variable:
export PYTHONPATH=/path/to/your/libs:$PYTHONPATH
Agreed — plugins_folder is meant for extending Airflow (e.g., custom operators, hooks), not for business logic. For shared code or utilities, it's better to create a separate folder like libs/ and add it to the PYTHONPATH for clean and maintainable DAGs.
ralichkov commentedon Jul 31, 2025
This came about due to #50385
So the solution is DAG bundles? Does that make most of this page obsolete?
https://airflow.apache.org/docs/apache-airflow/stable/administration-and-deployment/modules_management.html
jedcunningham commentedon Jul 31, 2025
There is no singular "dags folder" any longer, there is one or more bundles, which may or may not include the AF2 style dags folder. The bundle root is added on the fly during execution and parsing of dags, but some CLI commands were missed and have not been moved over to this new world.
Until 3.0.2, we were incorrectly adding the AF2 style dags folder onto sys.path, even if you weren't using it. You can easily flip this and say "the dag bundle root wasn't on sys.path in 3.0.0-3.0.1, but was fixed in 3.0.2". The current behavior is right.
That page is nearly all still applicable, the 2 places where the dags folder is mentioned needs to change to talk about the bundle root instead.