Repository navigation
Container Log Groomer should be configurable with a log retention time <1d - OTEL Collector FileLogReceiver #61814
Description
Description
Hi, thank you for the amazing ecosystem ❤️
We are using remote_logging and want to somehow use delete_local_logs, but with a delay (we are exporting the logs using OpenTelemetry Collector FileLog Receiver)
So in our current solution, we have set logging.delete_local_logs = "False" and the environment variable AIRFLOW__LOG_RETENTION_DAYS=1. The env is getting used in https://github.com/apache/airflow/blob/main/scripts/docker/clean-logs.sh
Now, even with only 1d retention, there are too many logs in our system, and the OTEL FileLog Receiver is unfortunately working very poorly with many files, I guess the receiver uses polling in some way.
The https://github.com/apache/airflow/blob/main/scripts/docker/clean-logs.sh script is using the command
find -type f -mtime +${RETENTION} where RETENTION is set to the AIRFLOW__LOG_RETENTION_DAYS env.
Why not just use (or additionally offer) AIRFLOW__LOG_RETENTION_MINUTES with find -type f -mmin +${RETENTION_MINUTES}? 🙂
Use case/motivation
More granular configuration option for log retention and thus better offering for largely scaled systems
Related issues
No response
Are you willing to submit a PR?
- Yes I am willing to submit a PR!
Code of Conduct
- I agree to follow this project's Code of Conduct
Activity
boring-cyborg commentedon Feb 12, 2026
Thanks for opening your first issue here! Be sure to follow the issue template! If you are willing to raise PR to address this issue please do so, no need to wait for approval.
shahar1 commentedon Feb 12, 2026
Thanks for suggesting, feel free to contribute a fix!
pugarte7 commentedon Feb 12, 2026
Hi @shahar1! I think this could be a very good first issue for me! Could you assign it to me? I think the approach given by @n-badtke-cg could be a good idea.
shreeharshshinde commentedon Feb 13, 2026
I’ve worked on this and opened a PR implementing minute-level log retention while keeping the existing behavior unchanged. Please feel free to share any feedback or suggestions.
pugarte7 commentedon Feb 13, 2026
Hi! I have created a PR to address this issue.
I think it's has better flexibility and safety than previous proposals:
AIRFLOW__LOG_RETENTION_MINUTESto 0, the script automatically falls back to the original daily retention logic. Users can toggle between minutes and days without breaking the cleanup cycle.I have verified that there is not documentation talking about AIRFLOW__LOG_RETENTION_DAYS, but I'm happy to add a section for this new variable to ensure it's discoverable for Docker users.
Also, if the maintainers agree with this logic, I am more than happy to extend this to the Helm Chart templates and update the relevant tests to keep everything in sync.
n-badtke-cg commentedon Feb 13, 2026
Ouh, we now have two PRs open for this issue 😅
n-badtke-cg commentedon Feb 13, 2026
@Wastelander777 look into #61853 , there are already some refinements
But thank you @Wastelander777 anyway <3 also already a thanks to @shreeharshshinde
n-badtke-cg commentedon Feb 13, 2026
Okay, three PRs now?
n-badtke-cg commentedon Feb 16, 2026
The first PR #61853 got closed without a good reason imo. @potiuk please elaborate
potiuk commentedon Feb 17, 2026
Yeah. Accidental. You need to make sure to fix the issues there (are conflicts and static checks are failing) .
potiuk commentedon Feb 17, 2026
I reopened it.
pugarte7 commentedon Mar 6, 2026
Hi @n-badtke-cg! I have updated the PR with the latest updates, and fixing the errors that were found on @shreeharshshinde 's branch. I have also added him as a co-author on the PR. Thank you!
n-badtke-cg commentedon Mar 6, 2026
@Wastelander777 I've reviewed #61855
n-badtke-cg commentedon Mar 12, 2026
PR #61855 was merged but accidentally included a bug unfortunately.
Currently, there is no follow up PR yet.Update: I created PR #63421
Update 2: Issues fixed with merge of PR #63421