Repository navigation
Enable Airflow DAG processor horizontally scalable #32966
Description
Description
Right now, the airflow DAG processor running in standalone or non standalone mode. Running multiple instances of the DAG processor does duplicate work. It's good idea to run the DAG processor in distributed mode to share the parsing workload.
Use case/motivation
Running the Airflow at a large scale requires running all the components of the airflow in distributed mode. Right now, the webserver and scheduler can able to scale horizontally. Right now, the DAG processor is vertically scalable but not horizontally scalable. This feature request is to make the DAG processor horizontally scalable.
Related issues
No response
Are you willing to submit a PR?
- Yes I am willing to submit a PR!
Code of Conduct
- I agree to follow this project's Code of Conduct
Reacted by DevilShubert
Activity
potiuk commentedon Jul 31, 2023
Yep. It's an intresting feature. It would be great if someone takes it and implement it. Nothing against such feature, it just needs someone to do it.
pegasas commentedon Jul 31, 2023
Hi, @potiuk , @dirrao
I am also following and interested in some related work which is similar to this item.
I am willing to submit a PR of this feature.
Count me in if you don't mind.
potiuk commentedon Jul 31, 2023
Sure @pegasas !
raphaelauv commentedon Jul 31, 2023
You can manually horizontally scale the DAG processor by using the
subdiroption https://airflow.apache.org/docs/apache-airflow/stable/cli-and-env-variables-ref.html#dag-processor.I agree that an automatic distribution of the files to parse (pondered by the number of dags present in each file) to every up and running dag processes would be nice
dirrao commentedon Aug 1, 2023
Yes. We can do that for the time being. However, we have to work on multiple external things (manual segregation of DAGs, HA, etc.). It would be nice to handle this out of the box.
pegasas commentedon Aug 1, 2023
Hi, @dirrao , @raphaelauv , @potiuk
I have one performance concern, maybe it is naive:
Our architecture only allows one scheduler to access meta database,
Will horizontal DAG processors be worsening DB lock contention?
Especially currently we use "select ... for update"?
Indeed, DAG Processing is a CPU-Bound job and Python parallelism is limited by GIL.
Is there any other evidence showing that multiple singletons are our bottleneck of Airflow?
Correct me if I am wrong.
potiuk commentedon Aug 1, 2023
No. Dag FileProcessor generally does not manipulate Dag Runs and Task instance - and those are the ones that locks are mainly used for. Creating DagRuns and queuing tasks is all happening in schduler.
We are also heavily using multiprocessing (Especially for Dag FileProcessor) - basically each file is processed in a new forked python process, so gil and singletons have nothing to do with it.
BTW. Horizontal scaling of DAG file processor would really be needed if your Python Parsing gets a lot of CPU cycles. And mostly when you have big number of smal Python files.
stephen-bracken commentedon Jan 20, 2026
Hi all, is this still a relevant issue? With the introduction of dag bundles / AIP-67 it makes sense to me to have a deployment pattern where we have one dag processor per dag bundle, which would help to isolate teams from noisy neighbour issues caused by parsing dags in other teams. I am interested in the feature regardless because I have had issues with teams creating very heavy (dynamic) dags in the past, so being able to scale horizontally would be a good addition here.