Repository navigation
apache-airflow-providers-common-sql==1.3.0 breaks BigQuery operators #27838
Description
Activity
Thanks for opening your first issue here! Be sure to follow the issue template!
Apologies - just noticed there's a separate template for bug reports related to providers. If someone could update the labels, that would be grand :)
One option to fix this would be to backport this commit and create a hotfix version 8.4.1 of the Google provider package.
Did you tried
apache-airflow-providers-google==8.5.0?@Taragolis that would be the ideal fix... as I can see that the issue is fixed on that release.
Cloud Composer 2.0.32 doesn't seem to like that though.
apache-airflow-providers-google==8.5.0pulls in a new version ofgoogle-cloud-compute, which itself wants to pull in new version of other dependencies that aren't compatible with the other GCP SDK packages...Error message below:
+ python3 -m pip check google-cloud-compute 1.6.1 has requirement google-api-core[grpc]<3.0.0dev,>=2.10.2, but you have google-api-core 2.8.1. google-cloud-compute 1.6.1 has requirement proto-plus<2.0.0dev,>=1.22.0, but you have proto-plus 1.19.6. google-cloud-compute 1.6.1 has requirement protobuf!=3.20.0,!=3.20.1,!=4.21.0,!=4.21.1,!=4.21.2,!=4.21.3,!=4.21.4,!=4.21.5,<5.0.0dev,>=3.19.5, but you have protobuf 3.20.0. The command '/bin/sh -c bash installer.sh $COMPOSER_PYTHON_VERSION fail' returned a non-zero code: 1Here's the full log from the Cloud Build job that Composer triggers when explicitly installing
apache-airflow-providers-google==8.5.0.A minor version upgrade of apache-airflow-providers-common-sql (1.2.0 to 1.3.0) should not break other providers (e.g. apache-airflow-providers-google==8.4.0)
The issue is not with sql provider. From the sql provider perspective only feature was added thus it's just a feature release.
If this becomes a breaking change in other package (other provider) than this package needs to have a major release.In Airflow we support only latest releases so if there is no issue with latest release then there is no issue. If this causes problems due to Compose limitations then I think it's best to contact Composer support.
That said, if one wants to fix previous version - it is possible. The procedure is documented on https://github.com/apache/airflow#release-process-for-providers and if someone is willing to backport a fix we can release
8.4.1If we made a mistake and we should have release google provider version 9.0.0 rather than 8.5.0 we can yank the wrong version and release a new one (but at least from the description at the moment I'm not sure this is the case?)
@eladkal that's a fair assessment and I don't think you need to yank version 8.5.0 given that it's compatible with the sql provider.
I guess that the real underlying problem is that it's currently not straight forward to get version 8.5.0 of the Google provider installed on the latest release of Cloud Composer as per #27838 (comment)
That is a question for Composer support then :)
I would like also to point that hopefully someone from Google will take ownership over #15933 the dependencies of this provider are too coupled. It should be several independent packages (Like for example the Apache provider)The issue is with how past bigquery providers mis-used protected method of the common-sql provider, indeed. Unfortunately, the only way to fix it, is to bring the method back in 1.3.1 and yank common.sql 1.3.0, because this mis-use made common.sql provider 1.3.0 backwards-incompatible.
I am afraid it is on us rather than on Composer team, because the problem is with the google provider versions we release and maintain. We cannot break older released google provider versions by installing new version of common.sql provider which is not backwrds-compatible (and in this case unfortunately OUR google provider made use of something that OUR common.sql provider thought was an internal detail. In any case it is OUR problem to solve. I will release common.sql
It is what it is, unfortunately, and one more learning that in case of common code like that we need to be extra careful and in the future we should use
__(double underscore) methods for internal commnds rather than_because the_methods are not sufficiently protected agains accidental mis-use.Cool - I'll poke some people on the Google side to see what we can do :)
I think not much. It will have to wait for 1.3.1 release of common.sql provider.
That said, I appreciate where @eladkal is coming from. I'll poke some folks on the Google side to see if they can engage with #15933.
Just be aware tht this is something that at very least take months once seriously started. The experience with separating common.sql provider had shown something that I knew is difficult. I chose common.sql as the case of seeing where separating common code and reusing it across several providers leads to. This i think 4th or 5th issue with common.sql provider that we discovered as result of this experiment - the fact is that making common code into a separate packages leads to exactly this kind of problems - the code is closely coupled (implicitly), yet we want to make sure that the code should evolve, And it is extremely difficult to make sure that we will prevent and handle all such problems.
And splitting the Google provider in similar fashion like common sql will lead to many, many, many more problems/couplings like that.
There is far more common code in google provider between multiple entitiies, and there will be many more such implicit dependencies that we (or Google) will miss. This is quite a difficult tasks (And now I know it not by intuition but also by seing what happened with common.sql case). If anything, the common.sql experience have reinforced my believe splitting google provider might simply never happen because no-one will be brave enough to take on the task.
27 remaining items
- added a commit that references this issue
on Jan 30, 2023
Apache Airflow version
Other Airflow 2 version (please specify below)
What happened
Airflow version: 2.3.4 (Cloud Composer 2.0.32)
Issue:
apache-airflow-providers-common-sql==1.3.0breaks all BigQuery operators provided by theapache-airflow-providers-google==8.4.0package. The error is as follows:Why this issue is tricky: other providers such as
apache-airflow-providers-microsoft-mssql==3.3.0andapache-airflow-providers-oracle==3.5.0have a dependency onapache-airflow-providers-common-sql>=1.3.0and will therefore install it when adding to the Composer environmentCurrent mitigation: Downgrade provider packages such that
apache-airflow-providers-common-sql==1.2.0is installed insteadWhat you think should happen instead
A minor version upgrade of
apache-airflow-providers-common-sql(1.2.0 to 1.3.0) should not break other providers (e.g. apache-airflow-providers-google==8.4.0)How to reproduce
composer-2.0.32-airflow-2.3.4apache-airflow-providers-common-sql==1.3.0via Pypi package install featureOperating System
Ubuntu 18.04.6 LTS
Versions of Apache Airflow Providers
Deployment
Composer
Deployment details
No response
Anything else
No response
Are you willing to submit PR?
Code of Conduct