Repository navigation
Add extra links for EMR Serverless #31620
Description
Activity
Thanks! I don't worry too much about the overhead of generating the link each time the dag popup loads, but I 1000% agree with the poor user experience, we need to have an answer for that.
I like the idea of a new class of extra links. Instead of a link it would be a button that says something like "generate link to " and then upon clicking that some client side code runs to generate the link (the user sees a spinner while that's happening but it should be fast) and then they are presented with the link once it's ready.
Thanks @phanikumv - I do have an early branch ready, but I still need to figure out how to handle the one-time URL generated by the
GetDashboardForJobRun API.My current thought is to add new endpoint (something like
/extra_link_ephemeral) that generates the link and redirects the user, but this is a bit more of a core change to Airflow.I am yes, hope to push something up early next week.
- Don't think I will try to figure out a new approach for ephemeral links. Some changes were made on EMR side to make the user experience better
- Also considering add more links (like to S3 and/or CloudWatch, but then the links started going off the pop up dialog. :/
Description
Add extra links to the EMR Serverless operator for S3 logs as well as the
GetDashboardForJobRunAPI that returns a one-time link to the UI.Use case/motivation
Allow for easier debugging of EMR Serverless jobs. Today, folks have to navigate through the AWS console to find the corresponding application and job run ID in order to gain access to the Spark UI or stdout/stderr logs. It'd be great to have direct links for both of those things.
One potential challenge is that the URL returned by the
GetDashboardForJobRunAPI is both short-lived (expires after 1 hour) and one-time (expires on first use). I'm concerned about both poor user experience of clicking an expired or already used URL and the API impact of generating a URL every time the DAG popup is triggered even if it's not used. This may require some additional functionality in Airflow to implement.Related issues
#29939
Are you willing to submit a PR?
Code of Conduct