Read performance regression via Store backed by FSMap (fsspec, GCS) #1296

Activity

1. ravwojdyla

   added

   bugPotential issues with the zarr-python libraryPotential issues with the zarr-python library

   on Dec 15, 2022
2. rabernat commented on Dec 15, 2022

   

   rabernat

   on Dec 15, 2022

   Contributor

   More actions

   @ravwojdyla I am very interested in optimizing the performance of this code path. My first recommendation would be to try your test instead with

   store = zarr.FSStore(...)
   np.asarray(zarr.open(store))

   rather than fsspec.get_mapper. Does this make any difference?
3. ravwojdyla commented on Dec 15, 2022

   

   ravwojdyla

   on Dec 15, 2022

   ContributorAuthor

   More actions

   @rabernat thanks for a quick response. Yes that will bypass the issue the same way reading via np.asarray(zarr.open(<path>)) would also avoid this issue.
4. rabernat commented on Dec 15, 2022

   

   rabernat

   on Dec 15, 2022

   Contributor

   More actions

   Ok, thanks for confirming that the problem goes away if you pass an FSStore rather than an fsspec.FSMap.

   We could resolve the regression by automatically promoting an FSMap to an FSStore. We discussed this a bit in #911 (comment). I believe that the only blocker to that idea was resolved in fsspec/filesystem_spec#939.

   🎉React with 🎉1Reacted by Rafal Wojdyla
5. ravwojdyla commented on Dec 15, 2022

   

   ravwojdyla

   on Dec 15, 2022

   ContributorAuthor

   More actions

   @rabernat that sounds great! +1
6. rabernat commented on Dec 15, 2022

   

   rabernat

   on Dec 15, 2022

   Contributor

   More actions

   Now this the time when I try to nerd-swipe you into making the PR yourself! 🤓

   Basically, we just need to add a block in this function (and also in the equivalent V3 version):

   zarr-python/zarr/storage.py

   Lines 132 to 156 in e7c0eb4

   	def _normalize_store_arg_v2(store: Any, storage_options=None, mode="r") -> BaseStore:
   	# default to v2 store for backward compatibility
   	zarr_version = getattr(store, '_store_version', 2)
   	if zarr_version != 2:
   	raise ValueError("store must be a version 2 store")
   	if store is None:
   	store = KVStore(dict())
   	return store
   	if isinstance(store, os.PathLike):
   	store = os.fspath(store)
   	if isinstance(store, str):
   	if "://" in store or "::" in store:
   	return FSStore(store, mode=mode, **(storage_options or {}))
   	elif storage_options:
   	raise ValueError("storage_options passed with non-fsspec path")
   	if store.endswith('.zip'):
   	return ZipStore(store, mode=mode)
   	elif store.endswith('.n5'):
   	from zarr.n5 import N5Store
   	return N5Store(store)
   	else:
   	return DirectoryStore(store)
   	else:
   	store = Store._ensure_store(store)
   	return store

   which checks isinstance(store, fsspec.FSMap) and then uses the FSMap attributes to initialize an FSStore.

   Is this something you think you could take on? We would be happy to help guide you through your first PR to Zarr. We are always looking for new contributors, and it seems like you understand this stuff pretty well already.
7. ravwojdyla commented on Dec 15, 2022

   

   ravwojdyla

   on Dec 15, 2022

   ContributorAuthor

   More actions

   @rabernat well played :P Sure, will give it a try. Thank for the context ^
8. rabernat commented on Feb 2, 2023

   

   rabernat

   on Feb 2, 2023

   Contributor

   More actions

   This was closed by #1304.
9. rabernat

   closed this as completedon Feb 2, 2023