apache-airflow-providers-jdbc fails with jaydebeapi.Error

Apache Airflow Provider(s)

jdbc

Versions of Apache Airflow Providers

I am using apache-airflow-providers-jdbc==3.0.0 for Airflow 2.3.3 as per constraint file

Apache Airflow version

2.3.3 (latest released)

Operating System

K8 on Linux

Deployment

Official Apache Airflow Helm Chart

Deployment details

No response

What happened

I am using JdbcOperator to execute one ALTER sql statement but it returns the following error:
File "/usr/local/airflow/.local/lib/python3.10/site-packages/airflow/providers/jdbc/operators/jdbc.py", line 76, in execute
return hook.run(self.sql, self.autocommit, parameters=self.parameters, handler=fetch_all_handler)
File "/usr/local/airflow/.local/lib/python3.10/site-packages/airflow/hooks/dbapi.py", line 213, in run
result = handler(cur)
File "/usr/local/airflow/.local/lib/python3.10/site-packages/airflow/providers/jdbc/operators/jdbc.py", line 30, in fetch_all_handler
return cursor.fetchall()
File "/usr/local/airflow/.local/lib/python3.10/site-packages/jaydebeapi/init.py", line 593, in fetchall
row = self.fetchone()
File "/usr/local/airflow/.local/lib/python3.10/site-packages/jaydebeapi/init.py", line 558, in fetchone
raise Error()
jaydebeapi.Error

What you think should happen instead

The introduction of handler=fetch_all_handler in File "/usr/local/airflow/.local/lib/python3.10/site-packages/airflow/providers/jdbc/operators/jdbc.py", line 76, in execute
return hook.run(self.sql, self.autocommit, parameters=self.parameters, handler=fetch_all_handler) is breaking the script. With the previous version which did not have fetch_all_handler in jdbc.py, it was running perfectly.

How to reproduce

Try submitting ALTER statement in airflow jdbcOperator.

Anything else

No response

Are you willing to submit PR?

Yes I am willing to submit a PR!

Code of Conduct

I agree to follow this project's Code of Conduct

Thanks for opening your first issue here! Be sure to follow the issue template!

@kazanzhy

@kazanzhy -> maybe you can help? (In the meantime @akashgangulyhf -> you can always downgrade the provider to previous version - this is the nice thing about providers that you can downgrade them individuall if you see error even if "blessed" version is different.

@potiuk

Hi @potiuk , Thanks for picking this up. I tried doing so but we have a dependency check with the constraint file and Airflow 2.3.3 doesn't allow <3.0.0 for jdbc operator

@akashgangulyhf

Don't use constraints for that @akashgangulyhf. See https://airflow.apache.org/docs/apache-airflow/stable/installation/installing-from-pypi.html#installing-upgrading-downgrading-providers-separately-from-airflow-core

For installing the providers you should not use any constraint files (the constraints are for installing Airflow with providers, not to install providers separately).

This should be a separate pip install <provider>==x.y.z command if you want to downgrade/upgrade provider separately - without constraints.

I see, let me see if I can go with that approach. Thanks.

@potiuk

@potiuk @akashgangulyhf I just want to point out that this isn't related to the jaydebeapi driver.

Currently any sql providers using the fetch_all_handler will fail since https://github.com/apache/airflow/pull/23971/files#diff-6e1b2f961cb951d05d66d2d814ef5f6d8f8bf8f43c40fb5d40e27a031fed8dd7R35 changed the fetch_all_handler implementation, making it non dbapi2 compliant but only compatible with sqlalchemy (we do not use sqlalchemy in DbApiHook.run).

PR #25430 should fix it

Navigation Menu

Repository navigation

apache-airflow-providers-jdbc fails with jaydebeapi.Error #25388