Repository navigation
Null key reference in bigquery cursor query results #53127
Description
Apache Airflow Provider(s)
Versions of Apache Airflow Providers
Issue faced on two versions:
apache-airflow-providers-google==12.0.0
apache-airflow-providers-google==15.1.0
Apache Airflow version
2.10.5
Operating System
Debian Linux 12
Deployment
Docker-Compose
Deployment details
Docker version 27.5.1
No tools used other than airflow.
What happened
Code reference that causes the issue:
airflow/providers/google/src/airflow/providers/google/cloud/hooks/bigquery.py
Lines 1663 to 1666 in e142ab9
airflow/providers/google/src/airflow/providers/google/cloud/hooks/bigquery.py
Lines 2061 to 2062 in e142ab9
While executing an external table creation job, I faced an issue that the above code runs into an error:
File "/home/airflow/.local/lib/python3.12/site-packages/airflow/providers/google/cloud/hooks/bigquery.py", line 2077, in _format_schema_for_description
for field in schema["fields"]:
~~~~~~^^^^^^^^^^
KeyError: 'fields'
After debugging I noticed that this kind of jobs return an empty dictionary for the key "schema"
This causes the second code reference in line 2061-2062 to try to loop over an empty dictionary and then it fails as there are no key "fields".
What you think should happen instead
What should happen is that the following script should check if the value for key "schema" exists and not check if the key it self exists
airflow/providers/google/src/airflow/providers/google/cloud/hooks/bigquery.py
Lines 1663 to 1666 in e142ab9
If we make the logic to check the value instead of the key then it will make sure that it won't call function '_format_schema_for_description()' without a proper schema values.
To fix this we can modify the script at line 1663-1666 to the following script:
if query_results["schema"]:
self.description = _format_schema_for_description(query_results["schema"])
else:
self.description = []
How to reproduce
To reproduce the problem you need the following:
1- Airflow and airflow-providers installed < versions dont matter much as this script exists in multiple versions.
2- Create a sql script that create an external table < This will assure no schema will be returned.
3- execute the sql script using the cursor from the bigqueryhook
hook = BigQueryHook(gcp_conn_id=gcp_conn_id, use_legacy_sql=False) conn = hook.get_conn() cursor = conn.cursor() cursor.execute(operation='SELECT 1 AS NUMBER')
Anything else
This problem is part of the script and will appear whenever anyone tries to execute a non resulting query.
Something like creating and dropping tables, data transformation, and other system queries.
Are you willing to submit PR?
- Yes I am willing to submit a PR!
Code of Conduct
- I agree to follow this project's Code of Conduct
Activity
boring-cyborg commentedon Jul 10, 2025
Thanks for opening your first issue here! Be sure to follow the issue template! If you are willing to raise PR to address this issue please do so, no need to wait for approval.
jroachgolf84 commentedon Jul 10, 2025
I'm not quite sure if this code snippet below will do the trick. Even if
query_results["schema"]exists, that does not guarantee that the"fields"key will exist. The change needs to be made in here:airflow/providers/google/src/airflow/providers/google/cloud/hooks/bigquery.py
Lines 2061 to 2062 in e142ab9
FaresDev8 commentedon Jul 13, 2025
@jroachgolf84 you are correct, even if we check for schema values that will not resolve the issue.
Will look into it.
prohackerzzz commentedon Jul 13, 2025
Hi! I'd like to work on this issue. Please let me know if that's okay.
prohackerzzz commentedon Jul 13, 2025
what if we just use this it will not cause any exception if there are no fields
schema = query_results.get("schema", {})
fields = schema.get("fields", [])
self.description = _format_schema_for_description(fields) if fields else []
FaresDev8 commentedon Jul 13, 2025
Hey @prohackerzzz, as @jroachgolf84 mentioned the solution need to be in the (_format_schema_for_description) function it self here:
airflow/providers/google/src/airflow/providers/google/cloud/hooks/bigquery.py
Lines 2061 to 2062 in e142ab9
The solution can be as simple as adding an if statement to check if the "fields" key exists and since this function only loops on "fields" then this check would do.
if "fields" in schema:Fix for Null key reference in bigquery cursor query results apache#53127