Repository navigation
Incorrect implementation of GCP auth type in VaultBackend? #17500
Description
Activity
Thanks for opening your first issue here! Be sure to follow the issue template!
Unfortunately I ran into this very issue trying to configure Vault as secrets backend using GCP Auth. The "credentials" returned by the
/auth/gcp/configcall cannot be serialized into JSON , triggering the following:[2021-11-24 07:45:00,858] {taskinstance.py:1463} ERROR - Task failed with exception Traceback (most recent call last): File "/home/airflow/.local/lib/python3.8/site-packages/airflow/models/taskinstance.py", line 1165, in _run_raw_task self._prepare_and_execute_task_with_callbacks(context, task) File "/home/airflow/.local/lib/python3.8/site-packages/airflow/models/taskinstance.py", line 1283, in _prepare_and_execute_task_with_callbacks result = self._execute_task(context, task_copy) File "/home/airflow/.local/lib/python3.8/site-packages/airflow/models/taskinstance.py", line 1313, in _execute_task result = task_copy.execute(context=context) File "/home/airflow/.local/lib/python3.8/site-packages/airflow/operators/python.py", line 150, in execute return_value = self.execute_callable() File "/home/airflow/.local/lib/python3.8/site-packages/airflow/operators/python.py", line 161, in execute_callable return self.python_callable(*self.op_args, **self.op_kwargs) File "/opt/airflow/dags/fa43208e955dbded1f92788fdabc5945cc1cabfb/vault_test.py", line 8, in get_secrets conn = BaseHook.get_connection(kwargs['my_conn_id']) File "/home/airflow/.local/lib/python3.8/site-packages/airflow/hooks/base.py", line 68, in get_connection conn = Connection.get_connection_from_secrets(conn_id) File "/home/airflow/.local/lib/python3.8/site-packages/airflow/models/connection.py", line 376, in get_connection_from_secrets conn = secrets_backend.get_connection(conn_id=conn_id) File "/home/airflow/.local/lib/python3.8/site-packages/airflow/providers/hashicorp/secrets/vault.py", line 226, in get_connection response = self.get_response(conn_id) File "/home/airflow/.local/lib/python3.8/site-packages/airflow/providers/hashicorp/secrets/vault.py", line 193, in get_response return self.vault_client.get_secret(secret_path=secret_path) File "/home/airflow/.local/lib/python3.8/site-packages/airflow/providers/hashicorp/_internal_client/vault_client.py", line 375, in get_secret response = self.client.secrets.kv.v2.read_secret_version( File "/home/airflow/.local/lib/python3.8/site-packages/airflow/providers/hashicorp/_internal_client/vault_client.py", line 205, in client if not self._client.is_authenticated(): File "/usr/local/lib/python3.8/functools.py", line 967, in __get__ val = self.func(instance) File "/home/airflow/.local/lib/python3.8/site-packages/airflow/providers/hashicorp/_internal_client/vault_client.py", line 228, in _client self._auth_gcp(_client) File "/home/airflow/.local/lib/python3.8/site-packages/airflow/providers/hashicorp/_internal_client/vault_client.py", line 307, in _auth_gcp _client.auth.gcp.configure(credentials=credentials) File "/home/airflow/.local/lib/python3.8/site-packages/hvac/api/auth_methods/gcp.py", line 59, in configure return self._adapter.post( File "/home/airflow/.local/lib/python3.8/site-packages/hvac/adapters.py", line 121, in post return self.request("post", url, **kwargs) File "/home/airflow/.local/lib/python3.8/site-packages/hvac/adapters.py", line 356, in request response = super(JSONAdapter, self).request(*args, **kwargs) File "/home/airflow/.local/lib/python3.8/site-packages/hvac/adapters.py", line 305, in request response = self.session.request( File "/home/airflow/.local/lib/python3.8/site-packages/requests/sessions.py", line 528, in request prep = self.prepare_request(req) File "/home/airflow/.local/lib/python3.8/site-packages/requests/sessions.py", line 456, in prepare_request p.prepare( File "/home/airflow/.local/lib/python3.8/site-packages/requests/models.py", line 319, in prepare self.prepare_body(data, files, json) File "/home/airflow/.local/lib/python3.8/site-packages/requests/models.py", line 471, in prepare_body body = complexjson.dumps(json, allow_nan=False) File "/usr/local/lib/python3.8/json/__init__.py", line 234, in dumps return cls( File "/usr/local/lib/python3.8/json/encoder.py", line 199, in encode chunks = self.iterencode(o, _one_shot=True) File "/usr/local/lib/python3.8/json/encoder.py", line 257, in iterencode return _iterencode(o, 0) File "/usr/local/lib/python3.8/json/encoder.py", line 179, in default raise TypeError(f'Object of type {o.__class__.__name__} ' TypeError: Object of type Credentials is not JSON serializableThis issue has been automatically marked as stale because it has been open for 365 days without any activity. There has been several Airflow releases since last activity on this issue. Kindly asking to recheck the report against latest Airflow version and let us know if the issue is reproducible. The issue will be closed in next 30 days if no further activity occurs from the issue author.
This issue has been closed because it has not received response from the issue author.
Please reopen.
Why? Can you confirm it is still open in the latest version of Google Provider @fpopic ? Can you then open a new issue with new stack trace showing the problem ? Otherwise (if there is no new data on latest versions ) there is no way anyone is able to take any action on it since it's open for 2 years.
But if you still have the issue, really the best way from your side to get attention is to provide more fresh information - ideally in a new issue.
Please assign https://github.com/messam88
@potiuk @mik-laj can you reopen the issue, please? I wrote update here #35855 (comment). Not sure how to proceed with Closed Issue and Closed PR by bot.
@potiuk I can confirm that this issue still exists.
Instead of login (
auth/{gcp}/login), airflow calls configure (auth/{gcp}/config).
References:- https://github.com/apache/airflow/blob/main/providers/hashicorp/src/airflow/providers/hashicorp/_internal_client/vault_client.py#L306
- https://github.com/apache/airflow/blob/main/providers/hashicorp/src/airflow/providers/hashicorp/_internal_client/vault_client.py#L308
It should be changed to
_client.auth.gcp.login(...).@fpopic thanks for opening #35855, this is a good start.
Moreover, for the same reason,
azureandradiusauthn won't work as well.Reacted by Filip Popić@aaabramov -> I suggest to open a new issue about it. Ass I already commented above - tt makes very little sense to comment on an issue opened 3 years ago. Quite likely it's related- but different issue and many of the things that were valid 3 years ago are not valid any more. And there is no point in tagging me again - I might or might not be involved in any way in diagnosing and solving the issue. Possibly people from google will get interested when you open a new issue (and link to that one marking it as imilar) and they migh help with it. Or other people might get interest and implement it.
By commenting on a closed issue you are limiting your visibility and a chance that someone will look at it. Highly recommend to open a new issue.
Reacted by Andrii Abramov@pbetkier can you create a clone issue and I will make a new PR?
@fpopic -> if you have a fix, you can create PR regardless from having an issue. We do not need issues in Airflow. We need PRs solving them and we are perfectly fine to have PRs just solving issues describing issues the are solving. We do not need double-accounting for that.
I can't reopen it either. You need to recreate it by pushing a new branch and creating new PR.
Reacted by Filip Popić
Short
I'm pretty sure the authentication flow for GCP in the
airflow.providers.hashicorp.secrets.vault.VaultBackendis incorrect and not usable. The Vault API used/auth/gcp/confighas a different purpose,/auth/gcp/loginshould be used instead.I've seen the
/auth/<type>/configAPI being used in a couple other auth types, e.g. RADIUS and Azure, so the problem is probably broader than just GCP.It's a bold statement as it means nobody has really used
VaultBackendwith these auth types. I may be wrong, but I would be surprised.Details
I've analysed Airflow's
VaultBackendcode while researching how to integrate Vault with GCP for a different project. Here's the discrepancy I found between the current implementation and the Vault's design.Current implementation
GCP auth is implemented in vault_client.py#L277. The implementation delegates to official Vault client's method that calls
/auth/gcp/configwhile providing the Airflow's GCP credentials to the call.This is however an administrative API to configure Vault's capability to verify JWT tokens via GCP API. The credentials expected are credentials for Vault to use when communicating with GCP IAM API, not application's credentials. This is explained in /auth/gcp/config API docs. The API doesn't respond with any data, so from the point of view of
VaultBackendthis call is basically just a no-op./auth/gcp/config call requires
X-Vault-Tokenheader with a token that has proper permissions. I'm guessing either this auth method was not tested at all or some administrative (root?) Vault token was passed (by mistake?) during manual tests? I haven't seen any integration tests for the auth methods, I understand they would be difficult/costly.Expected implementation
The proper API to use is /auth/gcp/login. It requires a signed JWT token issued by IAM's signJwt based on application's GCP credentials. It returns a Vault token to be passed in subsequent requests.
This is described in the Vault docs on GCP auth and on Google blog along with an implementation example Authenticating to HashiCorp Vault using Google Cloud IAM.
Am I missing something? What do you think about this?