Repository navigation
breeze bug --mount-sources all #53330
Description
Apache Airflow version
3.0.2
If "Other Airflow 2 version" selected, which one?
No response
What happened?
While working on #53241, I realized that I still don’t fully understand how the breeze command works. Since I had already built the image, I assumed that I needed to use breeze shell to run tests inside that image.
So I ran breeze ci-image build --python 3.10 to build the CI image with Python 3.10.
Then I ran breeze shell --python 3.10 --verbose to enter the shell and confirm, using the --verbose flag, that I was indeed working inside the CI image.
Inside the shell, I ran export SHOW_UNUSED_MYPY_WARNINGS=true (note: this is not related to the issue), and then executed ./scripts/ci/pre_commit/mypy_folder.py providers/apache/pinot
After reviewing the official documentation again, I noticed the --mount-sources all option, which seemed relevant to the issue. So I re-ran the shell using breeze shell --python 3.10 --mount-sources all, and this time, I was able to confirm that Dockerfile.ci was properly included in the environment. After going through this sequence, everything worked as expected.
First of all, is it possible that I ran the test incorrectly?
Also, could what I observed here be considered a bug?
What you think should happen instead?
No response
How to reproduce
breeze ci-image build --python 3.10
breeze shell --python 3.10 --mount-sources allOperating System
m2 max (arm64)
Versions of Apache Airflow Providers
No response
Deployment
Official Apache Airflow Helm Chart
Deployment details
No response
Anything else?
No response
Are you willing to submit PR?
- Yes I am willing to submit a PR!
Code of Conduct
- I agree to follow this project's Code of Conduct
Activity
potiuk commentedon Jul 14, 2025
The
./scripts/ci/pre_commit/mypy_folder.py providers/apache/pinotshould be run outside of breeze image. All the pre-commits are supposed to be run outside of the image, really. Most of them will also run inside, but this is not guaranteed. Specifically whatrun_mypy.shscript does - it actuallly starts the breeze image - using different docker compose project, so while the script is run in the host environment, it will start breeze image and run the mypy command inside it.There is a good reason why we are only mounting selected files and directories to inside the image - namely performance on MacOS. Part of our software chain is installing UI dependencies and compliling assets and it downloads (among others) node_modules and places them inside the sources, thera are also a number of other build artifacts that are generated in airflow sources - and mounting it all to the image is causing quite a lof of problems, for example when the folders are scanned - because MacOS filesystem used to mount files to (essentially running in a Linux VM) breeze container causes a lot of delays and slowdowns of a number of operations, also they are bad for reproducibility - because some of the generated folders and fiels (excluded by .gitignore and they are also ignored by .dockerignore - so not part of the CI image) might change the result of tests and starting airflow for example.
That's why we only mount selected sources by default -> and Dockerfile.ci is not among those - https://github.com/apache/airflow/blob/main/scripts/ci/docker-compose/local.yml -> this is a default set of files and folders that are mounted.
--mount-sources all uses https://github.com/apache/airflow/blob/main/scripts/ci/docker-compose/local-all-sources.yml which essentially mounts all "root" folder to inside the image.
Generally spaeaking, if you want to easily see what should be executed "in container" vs. "in host" - all the scripts that are supposed to be executed in container are in the https://github.com/apache/airflow/tree/main/scripts/in_container folder.
potiuk commentedon Jul 14, 2025
Closing because it's not an issue :)