Repository navigation
Sagemaker Transform Job fails if there are job with Same name #21941
Description
Description
Sagemaker Transform Job fails if there are job with Same name exist. Let say I create a job name as 'transform-2021-01-01T00-30-00' . So if I clear the airflow task run id for this so that the operator re-triggers then the Sagemaker Job creation fails because job with same name exists. So can we add 'action_if_job_exists flag where Behaviour if the job name already exists. Possible options are "increment" (default) and "fail".'
Use case/motivation
Now in production environment failures are inevitable and with Sagemaker Jobs we have to ensure there is unique name for each run of the Job. So like the Sagemaker Processing Job operator or training operator we have an option to increment a job name by appending the count like if I run same job twice the job name will be 'transform-2021-01-01T00-30-00-1' where 1 is appended at end with the help of 'action_if_job_exists (str) -- Behaviour if the job name already exists. Possible options are "increment" (default) and "fail".'
I have faced this issue personally on one of the task I am working on and think will save time and cost instead of running entire workflow again to get unique job names if there are other dependent task in the job by just clearing failed task id post fixing the failure in Sagemaker code , docker image input etc and that will just continue from where it failed
Related issues
No response
Are you willing to submit a PR?
- Yes I am willing to submit a PR!
Code of Conduct
- I agree to follow this project's Code of Conduct
Activity
boring-cyborg commentedon Mar 2, 2022
Thanks for opening your first issue here! Be sure to follow the issue template!
potiuk commentedon Mar 2, 2022
Assigned uou @hsrocks
schirag1993 commentedon Mar 8, 2022
Hey @potiuk and @hsrocks,
Would it make sense to add a random suffix to the end of the Transform job name vs. incrementing a number to it as a suffix?
Since the suffix would be irrelevant to the actual run of the job and serves only to avoid name collisions, a random suffix would make the code a lot simpler and serve the same purpose.
Thoughts?
potiuk commentedon Mar 8, 2022
Random is fine.
schirag1993 commentedon Mar 8, 2022
Awesome. Had a word with @hsrocks, I'll be working on this one!
celeriev commentedon Jun 30, 2022
Hello ! I think this feature could indeed be very useful ! We are facing the issue currently with my team. I can help if needed @hsrocks
celeriev commentedon Jul 11, 2022
Hi everyone, what would be the update status for this issue ? @hsrocks @potiuk @schirag1993, I'd be happy to work on it if needed. This would be a useful fix for next release !
potiuk commentedon Jul 11, 2022
I added you - not sure if @schirag1993 had done much on it - but if you still want @schirag1993 - chime in.
schirag1993 commentedon Jul 12, 2022
Hey y'all, I hadn't started work. Please feel free to get on it!