Skip to content

Use BaseXCom serialize_value when objectstorage_threshold is less than given input - #49173

Merged
vatsrahul1001 merged 7 commits into
apache:mainfrom
gopidesupavan:fix-xcom-object-storage-deser
Apr 14, 2025
Merged

vatsrahul1001 merged 7 commits into
apache:mainfrom
gopidesupavan:fix-xcom-object-storage-deser

Conversation

@gopidesupavan

Copy link
Copy Markdown
Member

closes: #49159

image


^ Add meaningful description above
Read the Pull Request Guidelines for more information.
In case of fundamental code changes, an Airflow Improvement Proposal (AIP) is needed.
In case of a new dependency, check compliance with the ASF 3rd Party License Policy.
In case of backwards incompatible changes please leave a note in a newsfragment file, named {pr_number}.significant.rst or {issue_number}.significant.rst, in airflow-core/newsfragments.

Comment thread providers/common/io/src/airflow/providers/common/io/xcom/backend.py Outdated
@gopidesupavan

Copy link
Copy Markdown
Member Author

Post update: @vatsrahul1001

image

@vatsrahul1001 vatsrahul1001 changed the title Fix XComObjectStorageBackend deserialize_value to use json loads Use BaseXCom serialize_value when objectstorage_threshold is less than given input Apr 14, 2025
@vatsrahul1001
vatsrahul1001 merged commit efdca94 into apache:main Apr 14, 2025

@amoghrajesh amoghrajesh left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good, one comment

Comment thread providers/common/io/src/airflow/providers/common/io/xcom/backend.py
# likely, we need to deserialize it using json.loads
data = json.loads(base_xcom_deser_result, cls=XComDecoder)
except (TypeError, ValueError):
pass

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Should we re raise this one?

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think this is fine, this raises error happen only when base_xcom_deser_result is object store path, likely this is string that means we should continue to next below to read from ObjectStore with path. WDYT?

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yep seems ok

@gopidesupavan
gopidesupavan deleted the fix-xcom-object-storage-deser branch April 14, 2025 07:38
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Example_xcom failing when XComObjectStorageBackend enabled

3 participants