Description
Currently, the log outputs for TaskInstance state changes within TaskInstance.py do not include the run_id.
i.e https://github.com/apache/airflow/blob/2.9.0/airflow/models/taskinstance.py#L1200
This omission makes it difficult to directly trace and verify the status of specific Task Instances without relying on inference from the execution_date. Including run_id in these logs would greatly enhance the traceability and debugging process, particularly in environments where multiple runs might be managed concurrently.
I propose changing the log outputs to include run_id in TaskInstance._log_state and TaskInstance._run_raw_task#TaskDeferred log entry. The run_id is one of the primary keys for TaskInstances and I think its inclusion in the logs should not introduce any issues.
Current Log Outputs
- Pausing task as DEFERRED: dag_id=Test1, task_id=task1, execution_date=..., start_date=...
- Marking task as SUCCESS: dag_id=Test1, task_id=task1, execution_date=..., start_date=..., end_date=...
Proposed Log Outouts
- Pausing task as DEFERRED: dag_id=Test1, task_id=task1, run_id=RUN_ID, execution_date=..., start_date=...
- Marking task as SUCCESS: dag_id=Test1, task_id=task1, run_id=RUN_ID, execution_date=..., start_date=..., end_date=...
Use case/motivation
Debugging and troubleshooting specific DAGRuns on Workers can be a little bit cumbersome without the ability to search logs by run_id.
Currently, we need infer execution_date from run_id to search worker log to find the final status of a TaskInstance on a Worker. This leads to unnecessary queries and steps. Including run_id in the logs would simplify log searches in distributed environments(like Airflow on Kubernetes) and likely assist other users facing similar challenges.
Related issues
No response
Are you willing to submit a PR?
Code of Conduct
Description
Currently, the log outputs for TaskInstance state changes within
TaskInstance.pydo not include therun_id.i.e https://github.com/apache/airflow/blob/2.9.0/airflow/models/taskinstance.py#L1200
This omission makes it difficult to directly trace and verify the status of specific Task Instances without relying on inference from the
execution_date. Includingrun_idin these logs would greatly enhance the traceability and debugging process, particularly in environments where multiple runs might be managed concurrently.I propose changing the log outputs to include
run_idinTaskInstance._log_stateandTaskInstance._run_raw_task#TaskDeferredlog entry. Therun_idis one of the primary keys for TaskInstances and I think its inclusion in the logs should not introduce any issues.Current Log Outputs
Proposed Log Outouts
Use case/motivation
Debugging and troubleshooting specific DAGRuns on Workers can be a little bit cumbersome without the ability to search logs by
run_id.Currently, we need infer
execution_datefromrun_idto search worker log to find the final status of a TaskInstance on a Worker. This leads to unnecessary queries and steps. Includingrun_idin the logs would simplify log searches in distributed environments(like Airflow on Kubernetes) and likely assist other users facing similar challenges.Related issues
No response
Are you willing to submit a PR?
Code of Conduct