Repository navigation
AWS EMR Operators: Add Log URI in task logs to speed up debugging #29939
Description
Activity
Thanks for opening your first issue here! Be sure to follow the issue template!
Isn't that done already via https://airflow.apache.org/docs/apache-airflow/stable/howto/define_extra_link.html ? There is EMR link already in the latest provider #28180 - maybe you are not using the latest provider simply? Or maybe it needs to be used in other operators?
Can you please @akumar1-godaddy
a) close the issue if it is already solved in the latest provider or
b) simply add PR to add them (following the one I copied)Thanks @potiuk for the inputs and reference links.
I checked the EMR operators, as of now the EMR Operators only have links to the EMR Cluster Summary(EMR console) and the EMR Sensors have links to the S3 logs folder containing logs for nodes, steps and container.
I think adding the log uri to the operators would be helpful wherever applicable e.g. for failed steps it would be useful to have loguri forEmrAddStepsOperator.Will work on the PR.
Reacted by Shubham Mehta, Jarek Potiuk and Vincent+1 to suggestion from @akumar1-godaddy; having additional extra link should only improve the UX.
Description
Airflow is widely used to launch, interact and submit jobs on AWS EMR Clusters. Existing EMR operators do not provide links to the EMR logs (Job Flow/Step logs), as a result in case of failures the users need to switch to EMR Console or go to AWS S3 console to locate the logs for EMR Jobs and Steps using the job_flow_id available in the EMR Operators and in Xcom.
It will be really convenient and help with debugging if the EMR log links are present in Operator Task logs, it will obviate the need to switch to AWS S3 or AWS EMR consoles from Airflow and lookup the logs using job_flow_ids. It will be a nice improvement for the developer experience.
LogUri for Cluster is available in DescribeCluster
LogFile path for Steps in case of failure is available in ListSteps
Use case/motivation
Ability to go to EMR logs directly from Airflow EMR Task logs.
Related issues
N/A
Are you willing to submit a PR?
Code of Conduct