Repository navigation
String arrays written with zarr python 3 zarr spec 2 cannot be read in zarr python 2 #3132
Description
Zarr version
v2.18.7, v3.0.8
Numcodecs version
v0.15.1
Python Version
3.12.10
Operating System
Mac
Installation
Pixi from conda
Description
Current behavior:
- Write an array of strings using zarr python 3 and zarr format 2
- The same array cannot be read using zarr python 2
Traceback (most recent call last):
File "/Users/schwartzm10/Code/geff/sandbox/zarr_string.py", line 33, in <module>
read(path)
File "/Users/schwartzm10/Code/geff/sandbox/zarr_string.py", line 20, in read
ids = z['strings'][:]
~~~~~~~~~~~~^^^
File "/Users/schwartzm10/Code/geff/.pixi/envs/zarr2/lib/python3.12/site-packages/zarr/core.py", line 799, in __getitem__
result = self.get_basic_selection(pure_selection, fields=fields)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/Users/schwartzm10/Code/geff/.pixi/envs/zarr2/lib/python3.12/site-packages/zarr/core.py", line 925, in get_basic_selection
return self._get_basic_selection_nd(selection=selection, out=out, fields=fields)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/Users/schwartzm10/Code/geff/.pixi/envs/zarr2/lib/python3.12/site-packages/zarr/core.py", line 967, in _get_basic_selection_nd
return self._get_selection(indexer=indexer, out=out, fields=fields)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/Users/schwartzm10/Code/geff/.pixi/envs/zarr2/lib/python3.12/site-packages/zarr/core.py", line 1342, in _get_selection
self._chunk_getitems(
File "/Users/schwartzm10/Code/geff/.pixi/envs/zarr2/lib/python3.12/site-packages/zarr/core.py", line 2187, in _chunk_getitems
self._process_chunk(
File "/Users/schwartzm10/Code/geff/.pixi/envs/zarr2/lib/python3.12/site-packages/zarr/core.py", line 2100, in _process_chunk
chunk = self._decode_chunk(cdata)
^^^^^^^^^^^^^^^^^^^^^^^^^
File "/Users/schwartzm10/Code/geff/.pixi/envs/zarr2/lib/python3.12/site-packages/zarr/core.py", line 2370, in _decode_chunk
chunk = chunk.view(self._dtype)
^^^^^^^^^^^^^^^^^^^^^^^
File "/Users/schwartzm10/Code/geff/.pixi/envs/zarr2/lib/python3.12/site-packages/numpy/_core/_internal.py", line 565, in _view_is_safe
raise TypeError("Cannot change data-type for array of references.")
TypeError: Cannot change data-type for array of references.
Steps to reproduce
write.py
# /// script
# requires-python = ">=3.12,<3.13"
# dependencies = [
# "zarr>3,<4",
# ]
# ///
import zarr
import numpy as np
if __name__ == "__main__":
ids = np.array([f"1_{i}" for i in range(1000)])
print(zarr.__version__)
z = zarr.open(f'test.zarr', mode='a', zarr_format=2)
z['strings'] = idsread.py
# /// script
# requires-python = ">=3.12,<3.13"
# dependencies = [
# "zarr>=2,<3",
# ]
# ///
import zarr
if __name__ == "__main__":
z = zarr.open('test.zarr')
ids = z['strings'][:]
print(ids)Additional output
No response
Activity
d-v-b commentedon Jun 13, 2025
@msschwartz21 thanks for the report, this should have been fixed by #3100, can you run your scripts against the main branch to confirm?
d-v-b commentedon Jun 13, 2025
for context, zarr-python 2 is complaining because it has decoded a chunk into a numpy object dtype array, and the reason it did that is because zarr-python 3 was erroneously inserting a
vlen-utf8codec into the filters for arrays with dtypeU. Because the numpyUdtype is fixed-size, there is no reason to encode it withvlen-utf8, so removing that behavior brings back zarr-python 2 compatibility.msschwartz21 commentedon Jun 13, 2025
@d-v-b you're right! Installing from main fixed the bug. I'll keep an eye out for the next release.
d-v-b commentedon Jul 19, 2025
this should now be fixed in the latest release of zarr python (3.1), let me know if you run into any problems with it