Repository navigation
DatabricksSQLOperator failed to save to csv #37007
Description
Apache Airflow version
2.8.1
If "Other Airflow 2 version" selected, which one?
No response
What happened?
When using DatabricksSQLOperator with csv as output format, Operation failed.
Log
[2024-01-25, 08:48:30 UTC] {client.py:258} INFO - Closing session 01eebb5e-8163-1e9c-a85e-6d23051ebfea
[2024-01-25, 08:48:30 UTC] {taskinstance.py:2698} ERROR - Task failed with exception
Traceback (most recent call last):
File "/home/airflow/.local/lib/python3.11/site-packages/airflow/models/taskinstance.py", line 433, in _execute_task
result = execute_callable(context=context, **execute_callable_kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/airflow/.local/lib/python3.11/site-packages/airflow/providers/common/sql/operators/sql.py", line 296, in execute
return self._process_output([output], hook.descriptions)[-1]
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/airflow/.local/lib/python3.11/site-packages/airflow/providers/databricks/operators/databricks_sql.py", line 149, in _process_output
writer.writerow(row.asDict())
^^^^^^^^^^
AttributeError: 'Row' object has no attribute 'asDict'
[2024-01-25, 08:48:30 UTC] {taskinstance.py:1138} INFO - Marking task as FAILED. dag_id=example_dag, task_id=select_account_info, execution_date=20240125T072737, start_date=20240125T084825, end_date=20240125T084830
[2024-01-25, 08:48:30 UTC] {standard_task_runner.py:107} ERROR - Failed to execute job 61 for task select_account_info ('Row' object has no attribute 'asDict'; 391)
[2024-01-25, 08:48:30 UTC] {local_task_job_runner.py:234} INFO - Task exited with return code 1
[2024-01-25, 08:48:30 UTC] {taskinstance.py:3280} INFO - 0 downstream tasks scheduled from follow-on schedule check
What you think should happen instead?
It shouldn't raise error.
How to reproduce
Create following operator
select_account_info = DatabricksSqlOperator(
task_id="select_account_info",
sql="SELECT a, b, c FROM tabl",
output_format="csv",
output_path="/var/tmp/output.csv",
)Operating System
MacOS
Versions of Apache Airflow Providers
apache-airflow-providers-databricks==6.0.0
Deployment
Docker-Compose
Deployment details
No response
Anything else?
No response
Are you willing to submit PR?
- Yes I am willing to submit a PR!
Code of Conduct
- I agree to follow this project's Code of Conduct
Activity
boring-cyborg commentedon Jan 25, 2024
Thanks for opening your first issue here! Be sure to follow the issue template! If you are willing to raise PR to address this issue please do so, no need to wait for approval.
Joffreybvn commentedon Jan 25, 2024
The Databricks hooks now returns a namedtuple instead of a Databricks.Row object. The whole
_process_outputmethod needs to be updated to userow._asdict(). Are you going to make a PR ?SuperBo commentedon Jan 25, 2024
Sure thing @Joffreybvn, are there anything I need to aware before making changes?
Joffreybvn commentedon Jan 25, 2024
You can setup Breeze, do the changes, add some tests (this case is not covered yet). And then run the tests and the static checks via breeze (those are the most common reasons for your CI-CD to fail). Take a look at CONTRIBUTING.rst. There is also a Slack #development channel if you need help :)
nathadfield commentedon Jan 25, 2024
@SuperBo Shall I assign you this issue then if you gonna attempt a PR?
SuperBo commentedon Jan 25, 2024
@nathadfield , yeah, please go ahead. I will start working on this tomorrow.