Add ability to generate temporary downloadable link for task logs(stored on cloud storage) on UI #44753

Activity

1. rawwar

   added

   kind:featureFeature RequestsFeature Requests

   needs-triagelabel for new issues that we didn't triage yetlabel for new issues that we didn't triage yet

   on Dec 7, 2024
2. rawwar

   self-assigned this

   on Dec 7, 2024
3. dosubot

   added

   area:logging

   area:UIRelated to UI/UX. For Frontend Developers.Related to UI/UX. For Frontend Developers.

   on Dec 7, 2024
4. rawwar

   changed the title [-]Add ability to generate downloadable link for task logs on UI[/-] [+]Add ability to generate downloadable link for task logs(stored on cloud storage) on UI[/+] on Dec 7, 2024
5. rawwar

   changed the title [-]Add ability to generate downloadable link for task logs(stored on cloud storage) on UI[/-] [+]Add ability to generate temporary downloadable link for task logs(stored on cloud storage) on UI[/+] on Dec 7, 2024
6. vatsrahul1001

   removed

   needs-triagelabel for new issues that we didn't triage yetlabel for new issues that we didn't triage yet

   on Dec 7, 2024
7. jscheffl commented on Dec 7, 2024

   

   jscheffl

   on Dec 7, 2024

   Contributor

   More actions

   I have also espcially seen that the webserver gets OOM even if you download because the FileTaskHandler tries to sort+merge different log sources. One important thing is not only the download but also the FileTaskHandler must 1:1 stream the logs from the backend, else it will go OOM as well.

   This included when I took a look last time... why actually log sorting and merging is implemented in general I doubt the usefulness because I also had situations where log sorting by timestamp gave "strange" results. In my view logs should not be modified/sorted/merged when served from webserver. This would also remove the OOM problem in general... whereas I have seen situations as well where the browser/client "dies" in rendering. So a limit in logs provided to browser might be a important feature as well (but please not like Github is doing it)

   👍React with 👍2Reacted by Kalyan R and Jarek Potiuk
8. potiuk commented on Dec 7, 2024

   

   potiuk

   on Dec 7, 2024

   Last edited by potiuk

   Member

   More actions

   Mostly agree with @jscheffl -> but I still think merging logs might be useful in some cases, though the "naive" version it is done now should be either limited to certain log size that should be able to fit in memory or fixed to support arbitrary log size. Loading whole log to memory is generally bad idea (but OK if we can confirm they will fit in memory).

   There are some algorithms that could be used to do it "well" even when the logs are huge, but they require much more sophisticated behaviour and local file storage and likely are not suitable to run in an "API" call. So I am not sure if it at all worth doing it (airflow is NOT a sophisticated logging solution) - but if we have this "download full log" (and even there we could download zipped logs from several sources without merging them) - that could be a useful counterpart for merging for big files.

   There are two options how to do it, I think:

   1. if the files are huge (and we could set arbitrary value here), only show the original task log (streaming it) and add a link to "download" the .zip file where you see the missing logs as well
   2. if the files are huge just download the "max" part of the files - perform merge and add a note that the logs are incomplete and that you should download the whole .zip content of the several logs

   Generally, yes, I think we should implement it (and prevent the OOM from happening).

   👍React with 👍1Reacted by Kalyan R
9. jason810496 commented on Dec 8, 2024

   

   jason810496

   on Dec 8, 2024

   Last edited by jason810496

   Member

   More actions

   Hi, after tracing down related GitHub issues and attempting to reproduce the OOM issue, I have some questions and ideas about related improvements.

   Related Issues and Context

   • Fix grid logs for large logs #29390
     • Limits UI rendering to 10,000 lines, but the webserver still reads the full logs.
     • The browser tab will get stuck also.
   • Add pagination to get_log in the rest API #29405
     • description added for pagination in get_log api  #30729
     • Only adds a description of log_pos and end_of_log metadata using URLSafeSerializer, the get_log API remain unchanged.
   • Add pagination to the task log #33625
     • Pagination on UI side, haven't been resolved yet.
     • Suggests exposing log_pos and end_of_log as query parameters for better UI pagination.
       As noted by @bbovenzi, this remains unresolved.

   Ideas for Related Improvements

   I’d be happy to take on these improvements if there are no objections!

   1. Expose log_pos and end_of_log as Query Parameters:

      • Implement in Flask and backport to v2.10.x.
      • Implement in FastAPI. ( new UI will have to implement the pagination logic as well )

   2. Pagination for Legacy UI:

      • Implement pagination for logs in the legacy UI after addressing the above.
      • Add pagination to the task log #33625

   3. Refactor FileTaskHandler:

      • Address the root cause of OOM directly at the handler level.

   Though I couldn't reliably reproduce OOM on the webserver, the browser tab crashing due to large logs remains an issue.

   Question

   How can I reliably reproduce the OOM issue when fetching large task logs?
   I’ve tried observing memory usage with Memray but found it challenging to reproduce.

   • Steps Taken:
     • Used the DAG mentioned in PR #29390.
     • Triggered runs multiple times and managed to reproduce OOM only once.
     • Used breeze start-airflow --python 3.9 --backend sqlite as the environment.

   Any tips on how to consistently reproduce the issue? Or is there a specific environment setup recommended for this?
10. rawwar commented on Dec 8, 2024

    

    rawwar

    on Dec 8, 2024

    ContributorAuthor

    More actions

    @jason810496 , I think you need to put resource limits on each airflow component. On managed services, the webserver is usually limited, On Astro its about 1 vcpu and 2GiB memory.

    ❤️React with ❤️1Reacted by Jason(Zhe-You) Liu
11. jason810496 commented on Dec 8, 2024

    

    jason810496

    on Dec 8, 2024

    Member

    More actions

    @jason810496 , I think you need to put resource limits on each airflow component. On managed services, the webserver is usually limited, On Astro its about 1 vcpu and 2GiB memory.

    Thanks, I will try it with docker compose !
12. rawwar

    removed their assignment

    on Jan 27, 2025
13. AryanK1511 commented on Jul 1, 2025

    

    AryanK1511

    on Jul 1, 2025 · Hidden as outdated

    show commentMore actions
14. rawwar

    self-assigned this

    on Aug 13, 2025
15. rawwar commented on Oct 28, 2025

    

    rawwar

    on Oct 28, 2025

    ContributorAuthor

    More actions

    Not sure if we need this now. Since AF3 can just stream logs. closing this.
16. rawwar

    closed this as not plannedon Oct 28, 2025