Databricks: add SQL endpoint operators

[dinowernli]

Description

Operators to manipulate Databricks SQL endpoints.

Use case/motivation

Support conveniently managing the lifecycle of Databricks SQL endpoints. This likely entails building one or more operators which talk to the endpoints API and support:

• Creating endpoints
• Deleting endpoints
• Starting endpoints
• Stopping endpoints

These operations map cleanly to individual API calls.

Related issues

No response

Are you willing to submit a PR?

• Yes I am willing to submit a PR!

Code of Conduct

• I agree to follow this project's Code of Conduct

[besenthil]

@dinowernli

Would the function _handle_databricks_operator_execution be applicable for API endpoints which are not ASYNC like the creation of sql endpoint?

[dinowernli]

Right now, the logic in that function is specific to the kind of operators defined in the same file, i.e., to operators which create runs. I think something similar would be useful for other operators, but it would likely have to operate on the status of a query execution or similar (rather than the state of a job run).

[besenthil]

Thanks @dinowernli I will take a look at it.

[besenthil]

@dinowernli

Is there a test stub that I can use for testing the API call?

[alexott]

@besenthil we don't have stubs for testing, so you need to have access to a workspace with Dataricks SQL, plus corresponding permissions to create SQL Endpoints.

From implementation perspective, you need to use _do_api_call function - first POST definition of SQL Endpoint, and then check for it's status. Please note that request will fail if you submit an endpoint with the same name, so you will need to check it (I have a code for that in PR for Databricks SQL Operator)

[besenthil]

Thanks @alexott

[alexott]

@besenthil is implementation in progress, or you need an additional help with it?

[besenthil]

Hi Alex, Yes it is in progress.

…

On Mon, 28 Feb, 2022, 3:29 am Alex Ott, ***@***.***> wrote: @besenthil <https://github.com/besenthil> is implementation in progress, or you need an additional help with it? — Reply to this email directly, view it on GitHub <#21377 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAHSRRYDFG7INIMAJBE24VDU5KNFRANCNFSM5NW7VFOA> . Triage notifications on the go with GitHub Mobile for iOS <https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675> or Android <https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub>. You are receiving this because you were mentioned.Message ID: ***@***.***>

[adam133]

Found this as part of debugging another issue... Databricks renamed the sql/endpoints API to sql/warehouses: https://docs.databricks.com/api/workspace/warehouses/list
The "list endpoints" endpoint no longer exists that's referenced here:

airflow/airflow/providers/databricks/hooks/databricks_sql.py

Line 30 in 9eae662

LIST_SQL_ENDPOINTS_ENDPOINT = ("GET", "api/2.0/sql/endpoints")

[wojkam73159]

Hi, is this issue is still open?