Repository navigation
Impersonation logic missing in BigQuery Async operators #34727
Description
Activity
@shahar1 I am bumping up against this issue currently and was wondering if you had been able to make any progress?
it looks like #35629 made an attempt to get BigQuery triggers working with impersonation. however i think one place was missed, specifically this invocation of BigQueryInsertJobTrigger. should be a simple fix to pass in
self.impersonation_chainthere as well@melugoyal I think it is more complicated than that. As mentioned above, if you follow the trigger through it leads to
GoogleBaseHook.provide_gcp_credential_file_as_context()where there isn't any provision for dealing with the impersonation chain.Reacted by Mehul Goyal and Shahar Epstein@melugoyal I think it is more complicated than that. As mentioned above, if you follow the trigger through it leads to
GoogleBaseHook.provide_gcp_credential_file_as_context()where there isn't any provision for dealing with the impersonation chain.Apologies for the delayed response - I haven't managed to look up into this until now.
As you stated, the problem is indeed more complicated, asGoogleBaseHook.provide_gcp_credential_file_as_context()is not provisioned with the impersonation chain.
It seems that the reason that it hasn't been supported until now, is that thegcloud-aiolibrary (which in turn creates the asyncJobinstance) does not support impersonated credentials. See: #29535 and talkiq/gcloud-aio#421.The good news are that there's a fresh PR for implementing it: talkiq/gcloud-aio#665
@eladkal FYI
Apache Airflow version
main (development)
What happened
BigQueryAsyncOperators uses writing credentials to a file using
GoogleBaseHook.provide_gcp_credential_file_as_context()which has logic forkey_pathandkeyfile_dictbut logic forimpersonation_chainmethod is missing.When using Impersonation chain method, the Operator goes into the deferred state and the tasks fail with
403 Access Deniederror.What you think should happen instead
When the operator goes into deferred state, the triggerer should try generating credentials using an impersonated service account instead of the default service account.the
How to reproduce
We have set up an impersonation chain for authentication to BigQuery. Here's how it works:
We assign a Service Account to the Kubernetes namespace.
This namespace-level Service Account impersonates our team's Service Account, which has the necessary roles to access BigQuery.
When the operator runs, the worker inserts a job into BigQuery using the team's Service Account. After that, it defers itself and starts executing get_job to check the job's status.
However, during this process, we encountered a "403 Access Denied" error. After some debugging, we discovered that the Triggerer is checking the job status using the namespace-level Service Account, rather than the team's Service Account. To confirm this, we granted the necessary role to the namespace-level Service Account for checking job status, and after that, the task succeeded.
To setup impersonation_chain, we can refer to this documentation.
Operating System
Debian GNU/Linux 11 (bullseye)
Versions of Apache Airflow Providers
Deployment
Astronomer
Deployment details
Deploy Airflow on Kubernetes so that you can annotate gcloud caller service account to the Airflow worker service account and impersonate a privileged service account that has bigquery permissions to generate short-term credentials.
Anything else
This problem occurs everytime when you use bigquery operators in async mode.
Are you willing to submit PR?
Code of Conduct