RecursionError with new refactored serialization #28741

Apache Airflow version

main (development)

What happened

>>> from airflow.utils.json import XComEncoder
>>> XComEncoder().encode(set())
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/opt/airflow/airflow/utils/json.py", line 91, in encode
    return super().encode(o)
  File "/usr/local/lib/python3.7/json/encoder.py", line 199, in encode
    chunks = self.iterencode(o, _one_shot=True)
  File "/usr/local/lib/python3.7/json/encoder.py", line 257, in iterencode
    return _iterencode(o, 0)
  File "/opt/airflow/airflow/utils/json.py", line 82, in default
    return serialize(o)
  File "/opt/airflow/airflow/serialization/serde.py", line 97, in serialize
    if isinstance(o, _primitives):
RecursionError: maximum recursion depth exceeded in __instancecheck__

What you think should happen instead

The encoder should correctly encode a value, or raise TypeError if it is not encodable.

How to reproduce

Simply use XComEncoder to encode anything that’s not encodable by the built-in JSONEncoder (i.e. something that’d go through the custom default hook).

Operating System

Any

Versions of Apache Airflow Providers

Irrelevant

Deployment

Official Apache Airflow Helm Chart

Deployment details

No response

Anything else

The problem here is because serialize does not guarantee to return a JSON-serializable object. So when default receives an set, it returns a set, and the encoder would try to serialize that set again, calling default, and eventually the stack explodes.

The simplest solution would be to add an additional flag to serialize so it can return a JSON-compatible type if requested.

Are you willing to submit PR?

• Yes I am willing to submit a PR!

Code of Conduct

• I agree to follow this project's Code of Conduct