Repository navigation
Use Job API 2.1 version for databricks provider hooks and #19447
Description
Description
According with Airflow Databricks hook documentation, this hook use the latest endpoint version of the databricks job API.
Databricks hook.
This hook enable the submitting and running of jobs to the Databricks platform. Internally the
operators talk to the ``api/2.0/jobs/runs/submit``
`endpoint <https://docs.databricks.com/api/latest/jobs.html#runs-submit>`_.
However, the provided link, point to new 2.1 API version instead of the 2.0 version endpoints used in the job api.
Use case/motivation
Since there are a new versions of the Jobs API (2.1) the Job API endpoints should be updated to version 2.1:
namely:
RUN_NOW_ENDPOINT = ('POST', 'api/2.1/jobs/run-now')
SUBMIT_RUN_ENDPOINT = ('POST', 'api/2.1/jobs/runs/submit')
GET_RUN_ENDPOINT = ('GET', 'api/2.1/jobs/runs/get')
CANCEL_RUN_ENDPOINT = ('POST', 'api/2.1/jobs/runs/cancel')
Related issues
No response
Are you willing to submit a PR?
- Yes I am willing to submit a PR!
Code of Conduct
- I agree to follow this project's Code of Conduct
Activity
josh-fell commentedon Nov 7, 2021
@jtvmatos Does #19412 cover this feature request?
jtvmatos commentedon Nov 8, 2021
Hi @josh-fell,
I believe your PR #19412 resolves this issue.
Can you need any help?
Tks.
josh-fell commentedon Nov 8, 2021
Not my PR but wanted to make you aware of it since you were willing to submit a PR here. Didn't want you to do waste your time with a PR if you didn't need to 😄