Repository navigation
No logs after task completion #45376
Description
Official Helm Chart version
1.15.0 (latest released)
Apache Airflow version
2.10.4
Kubernetes Version
1.28
Helm Chart configuration
No response
Docker Image customizations
No response
What happened
We use CeleryKubernetes Executor. Not in all cases, but sometimes after the task is completed, the output with logs disappears. In order for the output with logs to appear in the UI, you have to refresh the page.
What you think should happen instead
No response
How to reproduce
We launch the pipeline, select any task and wait for its completion. After the task is completed, the output disappears. We refresh the page, the output with logs is in place. Sometimes refreshing the page does not immediately display the logs, but you need to wait about a minute.
Anything else
No response
Are you willing to submit PR?
- Yes I am willing to submit a PR!
Code of Conduct
- I agree to follow this project's Code of Conduct
Activity
boring-cyborg commentedon Jan 3, 2025
Thanks for opening your first issue here! Be sure to follow the issue template! If you are willing to raise PR to address this issue please do so, no need to wait for approval.
javier-salazar-zefr commentedon Jan 3, 2025
We experienced a similar issue with version
2.10.3. In our case what was happening was that it would fail to fetch remote logs. From looking at the requests it seemed it only tried to get attempt=1 which did not exist in the bucket then after refreshing it would get attempt=2.log and that would successfully show the logs in the UI, this behavior was consistent with every task on the DAG.After trying different things we ended up downgrading to 2.9.2 and the issue went away. In version 2.9.2 it seems that it gets all attempts on the first time loading the screen so it showed 1 and 2, if we looked at 1 we could see the same error we saw on that first load on 2.10.3 but clicking on the
2tab showed the full logs from the bucket.Unsure if this would be the same issue but they sound similar.
potiuk commentedon Jan 6, 2025
That sounds like related to num_try change @dstandish (#39336)
asemelianov commentedon Jan 7, 2025
@javier-salazar-zefr @potiuk thanks!
dstandish commentedon Jan 7, 2025
This issue (i.e. the one reported by @asemelianov ) does not seem try number related. The reported issue is after task completion, the logs are temporarily absent. This is not terribly surprising because while the task is running, airlfow will read logs from the worker. Then, when task done, the logs are uploaded to blob storage (most commonly) and then the webserver will have to read from there instead. There were other recent meddlings with this area of code, i.e. dealing with scenarios when the logs are not found on worker, and that also could change this behavior.
I'm not quite clear on the scenario from @javier-salazar-zefr but it may warrant a separate issue with more detailed explanation.
Jorricks commentedon Jan 14, 2025
I think this might be unrelated to the issue posted here, but decided to reply here as #39336 was mentioned as a potential issue here. I think the try_number is actually correct except in a very specific case;
We just did the Airflow upgrade of 2.3.4 to 2.10.4. Usually, we are very strict on turning of all components during any sort of maintenance, this time we were a bit more Yolo (as it was our beta environment). The only preparation was setting some pools to 0, leading to quite some tasks being queued in the pools.
We found out that the tasks that were queued before the upgrade were uploading logs to
try_number=0. This checks out with the change regarding thetry_number; #39336. This is because previously with Airflow 2.3.4 we'd increment the version in the workers, now with 2.10.4 we increment it in the scheduler. For the tasks that were queued with 2.3.4, but executed on celery workers of 2.10.4, they never got an incremented try_number.We were actually very impressed with the upgrade as we left some 2.3.4 workers on (on k8s in terminating state) that were still finishing some tasks, and even during the DB upgrade & after the DB upgrade, the jobs they were executing finished fine and were marked in the DB as a success.
TLDR; we only had issues with the ones that are scheduled with <2.10 and executed with celery workers >=2.10, as we could not fetch their logs.
eladkal commentedon Jan 13, 2026
I'm closing this one as reported on Airflow 2.10.
If anyone experience similar problem on latest Airflow version please open a new issue and we will look into it.