Apache Airflow Provider(s)
postgres
Versions of Apache Airflow Providers
apache-airflow-providers-postgres==6.1.3
Apache Airflow version
2.11.0
Operating System
Debian GNU/Linux 12 (bookworm)
Deployment
Official Apache Airflow Helm Chart
Deployment details
No response
What happened
The changes introduced by #49120 made PostgresHook ignore custom adapters registered via psycopg2.extensions.register_adapter for the dict and list types.
Code that relied on custom adapters for dicts and lists is now broken because PostgresHook._serialize_cell is changing their type from dict or list to Json, making psycopg2 ignore the custom adapters registered for dicts and lists.
Automatically using Json to adapt list is a specially bad case because it prevents users from adapting python's list[int] or list[str] to postgres' int[] or text[].
The default Json adapter is also often times insufficient, because it doesn't support uuid.UUIDs, decimal.Decimals and datetime.datetimes, for example.
What you think should happen instead
I think PostgresHook should not introduce a special case when serialing dict and list values and it should respect the custom adapters registered using the register_adapter mechanism provided by psycopg2.
#49120 should be reverted and code that relied on it should use the code below to replicate the desired behaviour.
import psycopg2
from psycopg2.extras import Json
psycopg2.extensions.register_adapter(list, Json)
psycopg2.extensions.register_adapter(dict, Json)
How to reproduce
Using apache-airflow-providers-postgres==6.1.3, register a custom adapter for dict or list using psycopg2.extensions.register_adapter and try to run a query with PostgresHook.insert_rows which adapts a dict or list. The adapter will never be called.
Anything else
No response
Are you willing to submit PR?
Code of Conduct
Apache Airflow Provider(s)
postgres
Versions of Apache Airflow Providers
apache-airflow-providers-postgres==6.1.3
Apache Airflow version
2.11.0
Operating System
Debian GNU/Linux 12 (bookworm)
Deployment
Official Apache Airflow Helm Chart
Deployment details
No response
What happened
The changes introduced by #49120 made PostgresHook ignore custom adapters registered via
psycopg2.extensions.register_adapterfor thedictandlisttypes.Code that relied on custom adapters for
dicts andlists is now broken becausePostgresHook._serialize_cellis changing their type fromdictorlisttoJson, making psycopg2 ignore the custom adapters registered fordicts andlists.Automatically using
Jsonto adaptlistis a specially bad case because it prevents users from adapting python'slist[int]orlist[str]to postgres'int[]ortext[].The default
Jsonadapter is also often times insufficient, because it doesn't supportuuid.UUIDs,decimal.Decimals anddatetime.datetimes, for example.What you think should happen instead
I think PostgresHook should not introduce a special case when serialing
dictandlistvalues and it should respect the custom adapters registered using theregister_adaptermechanism provided by psycopg2.#49120 should be reverted and code that relied on it should use the code below to replicate the desired behaviour.
How to reproduce
Using
apache-airflow-providers-postgres==6.1.3, register a custom adapter fordictorlistusingpsycopg2.extensions.register_adapterand try to run a query withPostgresHook.insert_rowswhich adapts adictorlist. The adapter will never be called.Anything else
No response
Are you willing to submit PR?
Code of Conduct