Repository navigation
BytesCodec's endian does not roundtrip #3416
Description
Zarr version
3.1.1.dev52+gc019a5f26.d20250830 (current main, b8dbf56)
Numcodecs version
0.16.1
Python Version
3.12.3
Operating System
ubuntu 24.04
Installation
git
Description
If endian=None (and thus it is omitted from the json), BytesCodec.from_dict replaces it with endian=<system default> (by unpacking a empty dict into the kwargs of BytesCodec.__init__ we get the default for endian).
This means that roundtripping BytesCodec with to_dict and from_dict will change the endian. This has pretty limited impact, as for all dtypes where omitting the endianness is allowed changing that setting does not affect anything (or maybe somewhere else in the stack there's code that corrects this?). It did pop up when trying to write tests for zarr-developers/VirtualiZarr#787, though.
(I hope I'm not confusing to_dict and from_dict with "to / from JSON")
Fixing this could be as simple as calling parsed_configuration.setdefault("endian", None) in BytesCodec.from_dict.
Steps to reproduce
# /// script
# requires-python = ">=3.11"
# dependencies = [
# "zarr@git+https://github.com/zarr-developers/zarr-python.git@main",
# ]
# ///
import zarr
from zarr.codecs import BytesCodec
codec = BytesCodec(endian=None)
roundtripped = BytesCodec.from_dict(codec.to_dict())
assert codec.endian == roundtripped.endian, "did not roundtrip the endianness correctly"Additional output
No response
Activity
d-v-b commentedon Aug 30, 2025
seems like a bug. If endian is unset in the dict, it should be
Noneon the codec classkeewis commentedon Aug 30, 2025
I'll look into fixing that, then. Do you have tests for
BytesCodec.from_dict(orBytesCodecin general)? If so I couldn't find them.d-v-b commentedon Aug 30, 2025
tbh I think the spec is deficient here -- The spec disallows having
"endian"unset for multi-byte dtypes, but the spec allows settingendian: "little" | "big"for single-byte dtypes (this setting won't do anything though). IMO the simplest thing then would be to require the"endian"key in all cases. Too late to make this change, but we can use this concept in Zarr 4d-v-b commentedon Aug 30, 2025
the tests are here: https://github.com/zarr-developers/zarr-python/blob/main/tests/test_codecs/test_endian.py
This function needs to be renamed to
test_bytes.py