Under which category would you file this issue?
Providers
Apache Airflow version
main (unreleased)
What happened and how to reproduce it?
BigQueryHook.create_table()'s table_resource parameter is typed as
dict[str, Any] | Table | TableReference | TableListItem, and the docstring explicitly documents
passing a bare reference ("If table is a reference, an empty table is created with the specified
ID"). But the implementation only actually works when table_resource is a plain dict:
_table_resource: dict[str, Any] = {}
if isinstance(table_resource, Table):
_table_resource = Table.from_api_repr(table_resource) # type: ignore
if schema_fields:
_table_resource["schema"] = {"fields": schema_fields}
table_resource_final = {**table_resource, **_table_resource} # type: ignore
- Passing a real
Table instance: Table.from_api_repr(table_resource) is backwards —
from_api_repr expects an API-JSON dict, not a Table object. Its first line does
"tableReference" not in resource, and since Table has no __contains__/__iter__, this
raises TypeError: argument of type 'Table' is not iterable.
- Passing a
TableReference or TableListItem: neither is a Table subclass, so the isinstance
branch is skipped, and {**table_resource, **_table_resource} requires table_resource to be a
mapping. Neither class implements the mapping protocol, so this raises
TypeError: 'TableReference' object is not a mapping /
'TableListItem' object is not a mapping.
To reproduce:
from airflow.providers.google.cloud.hooks.bigquery import BigQueryHook
hook = BigQueryHook()
table = hook.get_table(project_id="my-project", dataset_id="my_dataset", table_id="my_table")
hook.create_table(
dataset_id="my_dataset",
table_id="my_new_table",
table_resource=table, # a Table instance -> TypeError
)
This is reachable from a public path, not just internal to the hook —
BigQueryCreateTableOperator has the identical type hint on its own table_resource param and
forwards it straight into bq_hook.create_table() unmodified. A Dag author who follows the
documented type hint and passes a Table/TableReference/TableListItem — e.g. one obtained from
BigQueryHook.get_table() or list_tables(), which is the natural thing to do — hits a crash. All
existing unit tests for create_table() only ever pass a plain dict, so this path is untested.
Root cause: Table, TableReference, and TableListItem all expose .to_api_repr() to convert
themselves into a dict, but the code calls Table.from_api_repr() (dict into object) and does
raw **-unpacking on the object — both wrong direction / wrong assumption about the object's
protocol.
What you think should happen instead?
create_table() should accept any of the types its own signature and docstring already promise
(dict, Table, TableReference, TableListItem) without raising, by converting each to a plain
dict via .to_api_repr() (wrapping a bare TableReference under "tableReference", since unlike
Table/TableListItem its to_api_repr() is flat) before merging in schema_fields and building
the final Table.
Operating System
No response
Deployment
Other
Deployment details
No response
Anything else?
No response
Are you willing to submit PR?
Code of Conduct
Under which category would you file this issue?
Providers
Apache Airflow version
main (unreleased)
What happened and how to reproduce it?
BigQueryHook.create_table()'stable_resourceparameter is typed asdict[str, Any] | Table | TableReference | TableListItem, and the docstring explicitly documentspassing a bare reference ("If
tableis a reference, an empty table is created with the specifiedID"). But the implementation only actually works when
table_resourceis a plaindict:Tableinstance:Table.from_api_repr(table_resource)is backwards —from_api_reprexpects an API-JSON dict, not aTableobject. Its first line does"tableReference" not in resource, and sinceTablehas no__contains__/__iter__, thisraises
TypeError: argument of type 'Table' is not iterable.TableReferenceorTableListItem: neither is aTablesubclass, so theisinstancebranch is skipped, and
{**table_resource, **_table_resource}requirestable_resourceto be amapping. Neither class implements the mapping protocol, so this raises
TypeError: 'TableReference' object is not a mapping/'TableListItem' object is not a mapping.To reproduce:
This is reachable from a public path, not just internal to the hook —
BigQueryCreateTableOperatorhas the identical type hint on its owntable_resourceparam andforwards it straight into
bq_hook.create_table()unmodified. A Dag author who follows thedocumented type hint and passes a
Table/TableReference/TableListItem— e.g. one obtained fromBigQueryHook.get_table()orlist_tables(), which is the natural thing to do — hits a crash. Allexisting unit tests for
create_table()only ever pass a plaindict, so this path is untested.Root cause:
Table,TableReference, andTableListItemall expose.to_api_repr()to convertthemselves into a dict, but the code calls
Table.from_api_repr()(dict into object) and doesraw
**-unpacking on the object — both wrong direction / wrong assumption about the object'sprotocol.
What you think should happen instead?
create_table()should accept any of the types its own signature and docstring already promise(
dict,Table,TableReference,TableListItem) without raising, by converting each to a plaindict via
.to_api_repr()(wrapping a bareTableReferenceunder"tableReference", since unlikeTable/TableListItemitsto_api_repr()is flat) before merging inschema_fieldsand buildingthe final
Table.Operating System
No response
Deployment
Other
Deployment details
No response
Anything else?
No response
Are you willing to submit PR?
Code of Conduct