Skip to content

[AIRFLOW-6882] Implement SageMakerHook list_training_jobs method + convenience method for iterating list_* requests - #7504

Closed
BasPH wants to merge 2 commits into
apache:masterfrom
BasPH:airflow-6882
Closed

BasPH wants to merge 2 commits into
apache:masterfrom
BasPH:airflow-6882

Conversation

@BasPH

@BasPH BasPH commented Feb 22, 2020 •

Copy link
Copy Markdown
Contributor

The SageMakerHook currently does not support any list_* operations (there are a few... https://boto3.amazonaws.com/v1/documentation/api/latest/reference/services/sagemaker.html). This PR adds a generic method for iterating over list_* operation results and one implementation for list_training_jobs.

By default, boto3 caps the number of results to 10. This is configurable up to 100. I've implemented _list_request as a convenience function for making multiple requests to AWS in case more results are available. SageMakerHook.list_training_jobs() makes use of this convenience function.

Not sure if tests should be added because all existing SageMakerHook tests simply test the number of calls to the underlying client's methods, which doesn't really validate correct behaviour IMO?

Example usage:

from airflow.providers.amazon.aws.hooks.sagemaker import SageMakerHook

hook = SageMakerHook()
hook.list_training_jobs(max_results=150)

Say there are 2000 training jobs, then _list_request() will select 150 training jobs in 2 batches (first 100, next 50).


Issue link: AIRFLOW-6882

Make sure to mark the boxes below before creating PR: [x]

  • Description above provides context of the change
  • Commit message/PR title starts with [AIRFLOW-NNNN]. AIRFLOW-NNNN = JIRA ID*
  • Unit tests coverage for changes (not needed for documentation changes)
  • Commits follow "How to write a good git commit message"
  • Relevant documentation is updated including usage instructions.
  • I will engage committers as explained in Contribution Workflow Example.

* For document-only changes commit message can start with [AIRFLOW-XXXX].


In case of fundamental code change, Airflow Improvement Proposal (AIP) is needed.
In case of a new dependency, check compliance with the ASF 3rd Party License Policy.
In case of backwards incompatible changes please leave a note in UPDATING.md.
Read the Pull Request Guidelines for more information.

@boring-cyborg boring-cyborg Bot added the provider:amazon AWS/Amazon - related issues label Feb 22, 2020
@BasPH

BasPH commented Mar 1, 2020

Copy link
Copy Markdown
Contributor Author

Closing in favour of #7598

@BasPH BasPH closed this Mar 1, 2020
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

provider:amazon AWS/Amazon - related issues

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant