Repository navigation
A high value of min_file_process_interval & max_active_runs=1 causes stuck dags #21083
Description
Apache Airflow version
2.2.2
What happened
When the value of AIRFLOW__SCHEDULER__MIN_FILE_PROCESS_INTERVAL is set to 86400, all dags whose MAX_ACTIVE_RUNS is set to 1 stop executing & remains stuck forever. If the MAX_ACTIVE_RUNS is set to 2 or above, or AIRFLOW__SCHEDULER__MIN_FILE_PROCESS_INTERVAL is set to a lower value (bw 30-300), dags work just fine.
What you expected to happen
These 2 settings should be exclusive to each other and there should be no direct impact of one another.
How to reproduce
set AIRFLOW__SCHEDULER__MIN_FILE_PROCESS_INTERVAL to 86400
set MAX_ACTIVE_RUNS to 1 on any dag & observe its execution dates.
Operating System
Debian GNU/Linux 11 (bullseye)
Versions of Apache Airflow Providers
apache-airflow-providers-amazon==1!2.3.0
apache-airflow-providers-cncf-kubernetes==1!2.1.0
apache-airflow-providers-elasticsearch==1!2.1.0
apache-airflow-providers-ftp==1!2.0.1
apache-airflow-providers-google==1!6.1.0
apache-airflow-providers-http==1!2.0.1
apache-airflow-providers-imap==1!2.0.1
apache-airflow-providers-microsoft-azure==1!3.3.0
apache-airflow-providers-mysql==1!2.1.1
apache-airflow-providers-postgres==1!2.3.0
apache-airflow-providers-redis==1!2.0.1
apache-airflow-providers-slack==1!4.1.0
apache-airflow-providers-sqlite==1!2.0.1
apache-airflow-providers-ssh==1!2.3.0
Deployment
Astronomer
Deployment details
Deployed on AKS via Astronomer Helm chart.
Anything else
No response
Are you willing to submit PR?
- Yes I am willing to submit a PR!
Code of Conduct
- I agree to follow this project's Code of Conduct
Activity