Repository navigation
BigQueryInsertJobOperator doesn't include BigQuery region/location in deferrable mode #35833
Description
Activity
Thanks for opening your first issue here! Be sure to follow the issue template! If you are willing to raise PR to address this issue please do so, no need to wait for approval.
- changed the title
[-]BigQuery Operators don't include BigQuery region/location in deferrable mode[/-][+]BigQuery BigQueryInsertJobOperator doesn't include BigQuery region/location in deferrable mode[/+]on Nov 24, 2023 - changed the title
[-]BigQuery BigQueryInsertJobOperator doesn't include BigQuery region/location in deferrable mode[/-][+]BigQueryInsertJobOperator doesn't include BigQuery region/location in deferrable mode[/+]on Nov 24, 2023 feel free to open PR to address the issue
I would like to add another note.
Some other BigQuery operators (For exampleGCSToBigQueryOperator) might have the same problem as well. Because if the operator usesBigQueryInsertJobTrigger, it won't pass thelocationparam.@kadai0308
Is it ok for me to take this issue. It will be my first commit ever to airflow but I would like to challenge it!@kadai0308 Is it ok for me to take this issue. It will be my first commit ever to airflow but I would like to challenge it!
Sure, please go ahead.
Reacted by Le Duc ManhI believe this issue is the same as #29307
We can't fix it until talkiq/gcloud-aio#584 is resolved
Now that #36849 has been merged, there is no longer a requirement on the talkiq library (and the outstanding PR) to enable impersonation in deferrable mode. This should mean that this issue can now be addressed.
Now that #36849 has been merged, there is no longer a requirement on the talkiq library (and the outstanding PR) to enable impersonation in deferrable mode. This should mean that this issue can now be addressed.
It is really great that aforementioned PR was merged and the impersonation is working well now. However, the described problem is still valid for regions other than
USorEU, because the used async client simply doesn't support location when making API call (in contrast sync client supports it). This PR is intended to fix the problem, so after it is merged andgcloud-io-bigqueryupgraded, we could fix the problem on our side by simply passing location from the operator to the trigger and than to the async hook.Please correct me if I'm wrong.
Ah, yeah, you're right. My misunderstanding.
That PR was merged in today and now the 7.1.0 release of gcloud-aio accepts a location param. Seems like with your recent PR @moiseenkov that we may not need to utilize this new param.
Apache Airflow version
Other Airflow 2 version (please specify below)
What happened
I tested
BigQueryInsertJobOperatorwith deferrable mode enabled.I used Google Cloud Composer 2.4.0 with airflow version 2.4.3
The BigQuery's region is
asia-northeast1region. The operator ran fine before but I gotJob not founderror after enabling deferrable mode. At first I thought it was because I didn't specifylocationparameter inBigQueryInsertJobOperator. However, even after changing it, I still got theJob not founderror.After taking a quick look at the source code, I guess maybe the cause is BigQuery Trigger doesn't include
locationin it's parameters.The code
airflow/airflow/providers/google/cloud/triggers/bigquery.py
Line 90 in 0e157b3
BigQuery get job API doc
What you think should happen instead
No response
How to reproduce
Execute
BigQueryInsertJobOperatorusingasia-northeast1BigQuery regionOperating System
It's not OS. I used Google Cloud Composer 2.4.0 with airflow version 2.4.3
Versions of Apache Airflow Providers
No response
Deployment
Google Cloud Composer
Deployment details
No response
Anything else
No response
Are you willing to submit PR?
Code of Conduct