Skip to content

Try to serialize a set to XCom, get unhelpful error message. #28048

Closed

Description

@MatrixManAtYrService

Apache Airflow version

main (development)

What happened

Here's a dag, and an exception from its task logs, as generated from 2.5.0 (as of e2455d8)

from airflow.models import DAG
from airflow.operators.python import PythonOperator
from datetime import datetime, timedelta

def cant_serialize():
    return {1: {"one", "two"}, 2: {"check", "one", "two"}}

with DAG(
    dag_id="some_sets",
    start_date=datetime(2022, 1, 1),
    schedule_interval=None,
) as dag:

    PythonOperator(
        task_id="some_sets",
        python_callable=cant_serialize,
    )
# Traceback (most recent call last):
#   File "/Users/matt/src/airflow/airflow/utils/session.py", line 72, in wrapper
#     return func(*args, **kwargs)
#   File "/Users/matt/src/airflow/airflow/models/taskinstance.py", line 2294, in xcom_push
#     XCom.set(
#   File "/Users/matt/src/airflow/airflow/utils/session.py", line 72, in wrapper
#     return func(*args, **kwargs)
#   File "/Users/matt/src/airflow/airflow/models/xcom.py", line 234, in set
#     value = cls.serialize_value(
#   File "/Users/matt/src/airflow/airflow/models/xcom.py", line 627, in serialize_value
#     return json.dumps(value, cls=XComEncoder).encode("UTF-8")
#   File "/Users/matt/.pyenv/versions/3.10.6/lib/python3.10/json/__init__.py", line 238, in dumps
#     **kw).encode(obj)
#   File "/Users/matt/src/airflow/airflow/utils/json.py", line 176, in encode
#     return super().encode(o)
#   File "/Users/matt/.pyenv/versions/3.10.6/lib/python3.10/json/encoder.py", line 199, in encode
#     chunks = self.iterencode(o, _one_shot=True)
#   File "/Users/matt/.pyenv/versions/3.10.6/lib/python3.10/json/encoder.py", line 257, in iterencode
#     return _iterencode(o, 0)
#   File "/Users/matt/src/airflow/airflow/utils/json.py", line 153, in default
#     CLASSNAME: o.__module__ + "." + o.__class__.__qualname__,
# AttributeError: 'set' object has no attribute '__module__'. Did you mean: '__reduce__'?

2.4.3's exception is TypeError: Object of type set is not JSON serializable

What you think should happen instead

The message here should be as good or better than it was in 2.4.3

How to reproduce

Run the dag above

Operating System

macOS / venv

Versions of Apache Airflow Providers

n/a

Deployment

Virtualenv installation

Deployment details

No response

Anything else

No response

Are you willing to submit PR?

  • Yes I am willing to submit a PR!

Code of Conduct

Activity

  1. tirkarthi commented on Dec 2, 2022

    @tirkarthi
    Contributor

    XCom seems to have custom encoder and decoder with e9d618f that could be causing this whereas before this commit the AirflowJSONEncoder was used. cc: @bolkedebruin

  2. uranusjr commented on Jan 5, 2023

    @uranusjr
    Member

    This was fixed as a part of #28067. But the fix introduced a new bug; I’ll open a new issue for it.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions