Repository navigation
Airflow web UI is slow #8907
Description
Apache Airflow version:
1.10.10
Kubernetes version (if you are using kubernetes) (use kubectl version):
1.13.12
Environment:
- Cloud provider or hardware configuration: Azure
- OS (e.g. from /etc/os-release):
- Kernel (e.g.
uname -a): - Install tools:
- Others:
What happened:
Every HTTP requests of the UI takes at least 5s, even static content.
/admin/metrics/, /health endpoints and 404 page have the same problems.
Here a graphs showing CPU usage of all ariflow components:
Each container has a 1s limit (left Y axis) so none of them is currently CPU bound.
What you expected to happen:
How to reproduce it:
Anything else we need to know:
Activity
boring-cyborg commentedon May 19, 2020
Thanks for opening your first issue here! Be sure to follow the issue template!
sylr commentedon May 19, 2020
The graphs shows the queries going through a reverse proxy but I've also tried directly with a
kubectl port-forwardand the problem persists.ashb commentedon May 20, 2020
@sylr A flat 5s latency sounds odd, and isn't something Airflow itself is doing, so there's something environmental going on here.
Things to check: if you run it locally, do you see this?
Is there anything of note in the webserver logs?
sylr commentedon May 21, 2020
I don't see anything suspicious is the logs.
But know that I think of it, 5s latency looks very much like a DNS timeout.
Any chance each request might trigger a DNS resolve ?
sylr commentedon May 21, 2020
After tcpdump'ing the calico interface of the pod I see DNS storms happening regularly:
Details
sylr commentedon May 21, 2020
Turning off the pgbouncer makes the endpoint
/static/bootstrap-toggle.min.jsunresponsive.ashb commentedon Jun 5, 2020
Which DNS pod (kube-dns or core-dns?) are you using in your kube cluster?
sylr commentedon Jun 5, 2020
coredns
vishal-swarankar-sdl commentedon Aug 4, 2020
I am also hitting the exact same issue? do we have an update on this issue? @ashb @sylr
sylr commentedon Aug 4, 2020
Nope, I'm at lost here.
natemoseman commentedon Aug 26, 2020
I had a similar issue with airflow running on kubernetes cluster that I was, fortunately, able to solve.
Regular HTTP connections were taking a minimal of 5 seconds to complete. Even using curl to fetch static content, like a CSS file, was taking 5 seconds. When looking at the logs of the airflow web process it didn't show anything. Although following them 'realtime' showed that the GET was showing up in the logs with the same 5 second delays as the curl command was taking.
by-passing everything and running curl directly from the container, and thus by-passing all the kubernetes networking stuff, was still having the delays.
It took me a few days to realize what was going on with my setup.
As it turned out it was due to the 'type: LoadBalancer' service I was using to expose the airflow webserver to outside the cluster. The loadbalancer was a external network load balancer that connected to the service via NodePort on each virtual machine in the node. For whatever reason this meant that there was a large number of connections just kept open to the webserver at any time.
In a 20 node cluster this meant 20 connections.
So when I killed the LoadBalancer service and started using nginx-ingress instead then the problem instantly resolved itself. No more delays. Admin web UI went back to normal.
I am not exactly sure what was going on here. But I suspect that having a large number of connections always open was causing gunicorn process to delay routing new connections to the pool of webserver worker processes. I was only using 4 processes at the time.
So if you are seeing these strange 5 second delays then use netstat or similar tool to count the number of "ESTABLISHED" connections to the webserver process. If you have a lot of connections and you are using service 'type: LoadBalancer' then try switching to using a ingress controller. Also increasing the number of worker processes to exceed the number of established connections will probably work too.
Hope that helps.
vishal-swarankar-sdl commentedon Aug 27, 2020
I had the flexibility of updating the k8s cluster to 1.16.5, and that solved this issue for me.
sylr commentedon Aug 27, 2020
@natemoseman Thank you for your insights, I will take a look as soon as I get back to work.
danielnazareth89 commentedon Aug 28, 2020
I fixed this by changing the default worker gunicorn from sync to an asynchronous class namely gevent. Please see the below AWS thread for the weird behaviour between gunicorn and ELB's.
https://forums.aws.amazon.com/thread.jspa?messageID=419138
So, simply set AIRFLOW__WEBSERVER__WORKER_CLASS: "gevent" in your config, should be better
10 remaining items
github-actions commentedon Jul 2, 2021
This issue has been closed because it has not received response from the issue author.
rafraph commentedon Aug 3, 2021
I think in my case it also was DNS issue since when I use the IP instead of the URL the UI become very fast
ashb commentedon Aug 4, 2021
It's always DNS.
Even when you think it's not DNS and you check for DNS problems, it's always DNS problems.
jcborras commentedon Aug 3, 2022
FWIW:
I encountered a similar issue lately which could be traced to slow HTTP responses when requesting resources
/dag_statsand/task_stats. The culprit is the database query which under certain circumstances (namely large number of rows and large number of active DAGs) it shows a) long response time and b) low susceptibility to optimization.This situation has a bad fix though: DB query ops are done through the ORM which in turn makes a poor job writing SQL code under the circumstances above.
nshetty15 commentedon Aug 3, 2022
This solution helped me. - #8907 (comment)
Moved from Load balancer service to Ingress. Airflow UI runs perfectly fine without any delays.
lovishchaudhary-dd commentedon May 24, 2024
Setting worked class to
geventresolved the issue but Airflow Web UI is unable to fetch the Dag/Task logs from S3 and the webserver logs show theRecursionError: maximum recursion depth exceededwhich is a known issuedimon222 commentedon May 24, 2024
@lovishchaudhary-dd
https://airflow.apache.org/docs/apache-airflow/stable/configurations-ref.html#worker-class
Set the extra patch env variable for monkeypatching before launching airflow.
aswathinj-pai commentedon Oct 28, 2024
@dimon222 I'm still encountering the issue even after setting
AIRFLOW__WEBSERVER__WORKER_CLASSto'gevent'and_AIRFLOW_PATCH_GEVENTto1. The logs continue to show aRecursionError: maximum recursion depth exceeded.dimon222 commentedon Oct 28, 2024
@aswathinj-pai what version of airflow? is it happening in webserver?
aswathinj-pai commentedon Nov 4, 2024
@dimon222 The airflow version is 2.4.3. The Log issue is happening for all the dags in the airflow.
potiuk commentedon Nov 4, 2024
Look at the release notes. I believe there was a similiar issue fixed some 1.5 years ago in Airflow 2.5 or 2.6. it is so long ago that only release notes can tell you for sure - but I would strongly recommend you to upgrade to latest t version of airflow (2.10.2) as you are missing on literally thousands of bug fixes applied since - not mentioning a number of security issues that were fixed since your version.
dimon222 commentedon Nov 4, 2024
this flag for monkeypatching was introduced in 2.5.1
#28283
aswathinj-pai commentedon Nov 5, 2024
@dimon222 @potiuk . Thank you for the input. I'll pass this along to my team.
EugeneBad commentedon Jan 3, 2025
We had a similar experience, but the cause turned out to be the database backend. Airflow makes a database call(s) for most web requests including the admin UI; to fetch static assets. If the database is already slow, then this impact will be cascaded upstream.
In our case, we had a long-running DAG with a locking SQL query being run from multiple airflow pods. This left many open connections, and a high DBLoad.
AWS RDS' "Performance Insights" makes it easy to assess the database's health, and diagnose such issues.
Removing this query restored the UI's performance. Hope this helps someone.