Databricks runs fail to execute on 0.235.4 with thrift error

[davem-bis]

Summary

When running with SQLMesh 0.235.4 and Databricks SQL Connector, existing environments fail to run in Databricks with the error:

ERROR:databricks.sql.thrift_backend:ThriftBackend.attempt_request: Exception: encoding without a string argument.

Downgrading to 0.235.3 resolves this issue.

Root Cause

The underlying model SQL I can see being executed successfully against a Databricks Warehouse, the failure occurs when acquiring the built tables' column types.

I believe that this has been caused by PRs #5783 and #5816, as when I remove the DatabricksEngineAdapter.columns method added by those PRs to sqlmesh/core/engine_adapter/databricks.py this error does not occur.

Potentially the fix would be to change line 434 from:

self.cursor.execute(query)

to:

self.cursor.execute(query.sql(dialect="databricks"))

Environment Tested

Environments:

1. Type: Databricks Cluster 17.3 LTS
   Connection mode: Databricks SQL Connector
   Python: 3.12.3
2. Type: Windows 11
   Connection mode: Databricks SQL Connector
   Python: 3.12.3

The Databricks SQL Connector and thrift versions do not make a difference in success, it's purely whether version 0.235.3 or 0.235.4 is running.