Repository navigation
Try to serialize a set to XCom, get unhelpful error message. #28048
Copy link
Copy link
Closed
Labels
Description
Apache Airflow version
main (development)
What happened
Here's a dag, and an exception from its task logs, as generated from 2.5.0 (as of e2455d8)
from airflow.models import DAG
from airflow.operators.python import PythonOperator
from datetime import datetime, timedelta
def cant_serialize():
return {1: {"one", "two"}, 2: {"check", "one", "two"}}
with DAG(
dag_id="some_sets",
start_date=datetime(2022, 1, 1),
schedule_interval=None,
) as dag:
PythonOperator(
task_id="some_sets",
python_callable=cant_serialize,
)
# Traceback (most recent call last):
# File "/Users/matt/src/airflow/airflow/utils/session.py", line 72, in wrapper
# return func(*args, **kwargs)
# File "/Users/matt/src/airflow/airflow/models/taskinstance.py", line 2294, in xcom_push
# XCom.set(
# File "/Users/matt/src/airflow/airflow/utils/session.py", line 72, in wrapper
# return func(*args, **kwargs)
# File "/Users/matt/src/airflow/airflow/models/xcom.py", line 234, in set
# value = cls.serialize_value(
# File "/Users/matt/src/airflow/airflow/models/xcom.py", line 627, in serialize_value
# return json.dumps(value, cls=XComEncoder).encode("UTF-8")
# File "/Users/matt/.pyenv/versions/3.10.6/lib/python3.10/json/__init__.py", line 238, in dumps
# **kw).encode(obj)
# File "/Users/matt/src/airflow/airflow/utils/json.py", line 176, in encode
# return super().encode(o)
# File "/Users/matt/.pyenv/versions/3.10.6/lib/python3.10/json/encoder.py", line 199, in encode
# chunks = self.iterencode(o, _one_shot=True)
# File "/Users/matt/.pyenv/versions/3.10.6/lib/python3.10/json/encoder.py", line 257, in iterencode
# return _iterencode(o, 0)
# File "/Users/matt/src/airflow/airflow/utils/json.py", line 153, in default
# CLASSNAME: o.__module__ + "." + o.__class__.__qualname__,
# AttributeError: 'set' object has no attribute '__module__'. Did you mean: '__reduce__'?2.4.3's exception is TypeError: Object of type set is not JSON serializable
What you think should happen instead
The message here should be as good or better than it was in 2.4.3
How to reproduce
Run the dag above
Operating System
macOS / venv
Versions of Apache Airflow Providers
n/a
Deployment
Virtualenv installation
Deployment details
No response
Anything else
No response
Are you willing to submit PR?
- Yes I am willing to submit a PR!
Code of Conduct
- I agree to follow this project's Code of Conduct
Reactions are currently unavailable
Activity
tirkarthi commentedon Dec 2, 2022
XCom seems to have custom encoder and decoder with e9d618f that could be causing this whereas before this commit the
AirflowJSONEncoderwas used. cc: @bolkedebruinuranusjr commentedon Jan 5, 2023
This was fixed as a part of #28067. But the fix introduced a new bug; I’ll open a new issue for it.