Repository navigation
Add support of max log length for OTEL task log event #43868
Description
Apache Airflow version
2.10.3
If "Other Airflow 2 version" selected, which one?
No response
What happened?
When the OTEL_TASK_LOG_EVENT is enabled, scheduler job performs extraction of the task log as span event which would get added to the task span (
airflow/airflow/jobs/scheduler_job_runner.py
Line 805 in cf3f9c7
What you think should happen instead?
OTEL_TASK_LOG_EVENT should support MAX length of the log entry, such that it should have a safe default value of the max length (e.g. 64k characters), which will effectively trim the task log once that max limit is reached, preventing any operational risk of scheduler if the log message is too large to be practical.
How to reproduce
Generate a huge log (hundreds of megabytes?) and observe scheduler to hang (or causing delay) processing the log as span event.
Operating System
Linux (amd64)
Versions of Apache Airflow Providers
N/A
Deployment
Official Apache Airflow Helm Chart
Deployment details
No response
Anything else?
No response
Are you willing to submit PR?
- Yes I am willing to submit a PR!
Code of Conduct
- I agree to follow this project's Code of Conduct
Activity
potiuk commentedon Nov 11, 2024
+1
dstandish commentedon Nov 12, 2024
-1
If you are talking about uploading task execution logs from the scheduler loop, this seems like still very much not a good idea. Even if just a snippet of logs, you don't want to be retrieving connections, reading from s3, etc. It's not the place to do that.
potiuk commentedon Nov 12, 2024
Who is talking about doing it from scheduler loop?
dstandish commentedon Nov 12, 2024
that's where the code in question is right now and that's the context for this is it not?
dstandish commentedon Nov 12, 2024
look in the top of this issue @potiuk
dstandish commentedon Nov 12, 2024
so it sounds like this issue is just about reducing the size of the log message in the area where it's currently added.
but it cannot be added ther.
potiuk commentedon Nov 12, 2024
Oh, I have not seen that - this should be definitely done from task/worker..
dstandish commentedon Nov 12, 2024
😌 (relieved face)
ashb commentedon Nov 12, 2024
No, remove this feature entirely. Logs do not belong in Otel traces/spans. Use OTel logging and associate it with a span if that's what you want to do, but this code cannot exist in the scheduler.