zarr v3 scale-offset codec, and should the codec API be defined upstream

now that a spec-compliant cast_value codec implementation has been published, and we have a spec for the scale-offset codec, we can in principle deprecate the numcodecs.fixedscaleoffset codec for zarr v3 data and replace it with the zarr v3 native codecs.

But there's a complication: the cast_value codec is an external dependency, because we decided it was too big to fit in #3774. There's no external scale_offset package, but I'm happy to make one as it's not a lot of code)

How do we want to move forward here? IMO the current numcodecs.fixedscaleoffset is not fit for zarr v3 use so we need to deprecate it. If we deprecate it, we should direct users an alternative that functions correctly. Here are two options:

use the numcodecs.fixescaleoffset deprecation warning message / docs to recommend that users use the cast_value and scale_offset codecs instead, with links to the packages that implement these codecs.
add cast_value and scale_offset codecs as optional dependencies for zarr-python. IMO this would introduce an urgent need to replace our codec ABC base class with protocols, because the cast_value codec depends on zarr-python to implement the codec abc, and we really don't want another circular dependency nightmare like we had with numcodecs.

thinking this through, it seems like the current codec API makes it impossible to a) avoid circular dependencies and b) allow zarr-python to declare stand-alone codec libraries as dependencies. The only recourse is to define all codecs in zarr-python itself, which isn't workable long term. IMO fixing this API should be considered a high priority by folks interested in seeing a thriving ecosystem of codecs that are not all defined directly in zarr-python.

I agree that a protocol based approach is a better design for codecs because it provides independent extensibility.

Can you provide a bit more details on the urgency? Who is trying to use cast_value and scale_offset and what is their timeline for producing data using those codecs? How much flexibility is there in that timeline?

we need the scale-offset codec for work described here. Zarr-python currently ships the numcodecs.fixedscaleoffset codec, but nobody should use it, because it has no spec and will not safely generalize to other implementations.

but the urgency is only in the specific case: if we deprecated the numcodecs.fixedscaleoffset codec, and included the replacement codecs as optional zarr-python dependencies, we would have a circular dependency situation, because any external codec implementation must depend on zarr-python. But we can avoid this by moving the codec abc outside of zarr-python, and making zarr-python depend on it.

are the codec abc classes purely to define the interface, or are there also default implementations? If it's the latter, you won't really be able to avoid circular dependencies unless you move these default implementations elsewhere (though imports from within codec methods may work around that) or somehow find a way to avoid the default implementations.

I guess that means we'd either way end up with a separate codecs package.

I don't think we have any load-bearing default implementations, and even if we did, defining them in zarr_codec_shapes.v1.abc would solve this problem I think.

I updated the title, the subject of this issue now includes the question of whether the codec API should be defined in a package upstream of zarr-python. I'm leaning toward "yes", and I think the name zarr_codec_shapes seems fine (because this package will contain both ABCs and protocols)

I don't think we have any load-bearing default implementations

FWIW there's the default implementations for batching, not sure how much that is used.

and there's a big hitch: our codec API depends on our data type API. So if we try to push codecs abcs / protocols upstream, we have to push data type abcs / protocols even farther upstream. To be clear I am 100% up for this, and I think it will be healthy, but we are looking at a lot of LOC moved around

This might be a naïve/uninformed idea, but isn't it possible to define the encoding logic/code in an third-party package, and then import that and use it in a small Codec wrapper class inside zarr-python?

@dstansby

@dstansby that's actually a viable short-term solution. I would basically take this code and we put it in zarr-python. It's won't exactly be a small codec wrapper, but I can make that work.

One unfortunate about this approach: in cast-value.py I use the entrypoints system to register the cast_value codec implementation with zarr-python, which means zarr-python automatically adds that codec implementation to the registry at import time. If we define another cast_value codec implementation in zarr-python, presumably that one would take priority over the codec defined in cast_value.py, which sort of defeats the intention of the entrypoints system. But IMO that's part of a broader problem with how we set up the codec registry.

@TomAugspurger

@TomAugspurger (asking because you have the most relevant experience with this issue) what do you think about defining the codec API upstream of zarr python, in a separate package? I think we could keep that package in this gh repo by making zarr-python a monorepo. Then, zarr-python would import the codec and data type base classes instead of defining them, and we could make zarr-python depend on externally defined codecs without circular dependency issues.

Can you help me understand the packaging issues caused by having the interfaces defined here in zarr-python instead of some zarr-interfaces library? I'm not sure I see it.

We have some library providing a codec, and it does that by implementing some interface (currently an ABC, could also be a Protocol, doesn't really matter).

Now the bit I don't understand is where this would ever involve zarr-python's packaging. I would never expect zarr-python to import, in its source code, a codec implemented by a third party library. Instead, I would expect those to only ever be loaded at runtime.

And if we only ever load it at runtime, what's the packaging side? You could in theory have an extra optional-dependency pip install zarr[cast-value-codec] or whatever you call it, but IIUC that's only a convenience to users; an alternative to pip install zarr zarr-cast-value-codec. Since zarr-python wouldn't literally import it, zarr-python itself wouldn't benefit from specifying that dependency.

Does that sound right, or am I missing something?

I would never expect zarr-python to import, in its source code, a codec implemented by a third party library.

I don't think it's a bad idea to import codecs. This is exactly what we used to do in zarr-python 2.x: the library we imported them from was numcodecs. We also briefly did this in zarr-python 3.x, when we defined v3-compatible codecs in numcodecs. But because this required numcodecs to depend on zarr-python (for the v3 codec API), we had a circular dependency that was a massive headache to unwind.

And if we only ever load it at runtime, what's the packaging side? You could in theory have an extra optional-dependency pip install zarr[cast-value-codec] or whatever you call it, but IIUC that's only a convenience to users; an alternative to pip install zarr zarr-cast-value-codec. Since zarr-python wouldn't literally import it, zarr-python itself wouldn't benefit from specifying that dependency.

scale_offset + cast_value replace the current numcodecs.fixedscaleoffset codec that zarr-python currently defines. There's a package inzarr-developers that implements cast_value, but not one for scale_offset (yet). One option would be to deprecate numcodecs.fixedscaleoffset, and in that message instruct people to install zarr alongside the external packages that implement the scale_offset and cast_value codecs to get that functionality. IMO this is not a great user experience.

If the zarr-python developers are maintaining these external packages exclusively for use with zarr-python, and we are committed to ensuring compatibility with zarr-python (which we can do, as the zarr-python devs), then why wouldn't we give users the option to install bundle these codecs with zarr-python? And that is currently not tenable, because of something very simple: we define the codec ABCs in zarr-python, when they could be defined in an upstream package.

What code would we have in zarr-python that imports an externally defined codec (the scale-offset code, say)? I was imagining it'd all be dynamic:

See that you have a codec with some name
Load the codec from the codec registry (which is where things like entrypoints come it)
Call the loaded codec's decode method

But if that's incorrect I've got a pretty fundamental misunderstanding and my opinion isn't worth much.

Setting that aside, the headaches of packaging something like zarr-scale-offset separately from zarr-python doesn't seem too worth it to me if a.) users aren't using it independently of zarr-python, and b.) zarr-python always includes it by a required or optional dependency. To me, it'd make sense to include that in zarr-python itself if we're the ones maintaining it anyway. But again, I might not have the full context here.

What code would we have in zarr-python that imports an externally defined codec (the scale-offset code, say)? I was imagining it'd all be dynamic:

See that you have a codec with some name
Load the codec from the codec registry (which is where things like entrypoints come it)
Call the loaded codec's decode method

But if that's incorrect I've got a pretty fundamental misunderstanding and my opinion isn't worth much.

The dynamic thing works, but we probably don't want to structure zarr-python with the expectation that literally every codec is dynamic. get_codec('gzip') -> SomeCodec is much vaguer than from x import GzIpCodec. In zarr-python 2.x, we directly imported the core codecs
. In 3.x, import codec implementations from numcodecs and write v3-compatible wrappers around them.

To me, it'd make sense to include that in zarr-python itself if we're the ones maintaining it anyway. But again, I might not have the full context here.

the codec in question is implemented in rust, so including it in zarr-python means adding the rust toolchain. Historically we put this kind of stuff in an external package (numcodecs). I proposed adding a pure-python implementation in this PR, but then I got the feedback that it was too much for zarr-python. So i'm not really sure what to do here.

edit: I didn't add the cast-value-rs implementation to numcodecs because it has nothing in common with the other codecs there besides the same consumer (zarr-python). and also, I think numcodecs should be split into separate packages.

So i'm not really sure what to do here.

You could use the existing registry approach. I'm not convinced that the argument "get_codec('gzip') -> SomeCodec is much vaguer than from x import GzIpCodec" merits the complexity of #3875. There's no circular dependency with pip install zarr cast-value

so your recommendation is this:

scale_offset + cast_value replace the current numcodecs.fixedscaleoffset codec that zarr-python currently defines. There's a package inzarr-developers that implements cast_value, but not one for scale_offset (yet). One option would be to deprecate numcodecs.fixedscaleoffset, and in that message instruct people to install zarr alongside the external packages that implement the scale_offset and cast_value codecs to get that functionality. IMO this is not a great user experience.

We can have cast-value be an optional dependency in zarr-python via the registry approach without circular dependencies. That's simple with entrypoints.

pip install zarr[cast-value] and everything working seems like a fine user experience to me.

pip install zarr[cast-value] and everything working seems like a fine user experience to me.

this requires:

zarr-python to optionally depend on cast-value (we declare cast-value as an optional dep)
cast-value to depend on zarr-python (for the codec API)

can you explain how this does not introduce a circular dependency?

I should clarify that there's no problematic circular dependency, because a circular dependency is only a problem when it creates a cycle in the import graph or an unsolvable cycle in the resolver graph which would not be the case here.

We can break down dependency cycles into three categories:

Import-time cycles. module A imports module B. module B imports module A at the top level. This is a real problem and happened with numcodecs.
Package metadata cycles. package A lists B as a dependency, B lists A as a dependency. Modern resolvers handle this fine. We do this in VirtualiZarr with virtual-tiff as an optional dependency that satisfies a protocol with no issues.
Runtime discovery via entrypoints. No dependency relationship at all at import time. Package B registers a class, package A discovers it lazily through the registry.

The zarr + cast-value case is (3), with an optional dependency convenience in the metadata (2). It is not (1), which is when there'd be a problem.

I would argue that a circular dependency is always a problem, because it means you have to change 2 things at once instead of changing 1 thing, and then the other thing. Your examples are all different examples of pathologies this can cause, but there are more that you didn't list. here's an example:

zarr-python has codec API v1
cast-value depends on zarr-python for a codec that implements codec API v1
zarr-python declares cast-value as a dep
zarr-python starts codec API v2 (some breaking change to the codec API). This cannot be released without co-releasing a new version of cast-value at the same time, and pinning the minimum version of cast-value to that new release. But the new version of cast value can't be released until the new codec API is available. otherwise the cast-value codec targeting the v1 API would be pulled in and potentially error or worse.

As long as the codec api is defined inside zarr-python, we are extremely limited in how we can change it while also pulling in external packages that use our API. entrypoints only makes the problem more subtle -- instead of an import-time error, you push the error somewhere else. but the fundamental problem is the circular dependency.

because it means you have to change 2 things at once instead of changing 1 thing, and then the other thing

I think that argues for either

Gradually changing APIs, as opposed to outright breaks (not always easy!). Or,
Bundling everything togethers, so there's only one unit to change.

But given that we do have a codec interface that we want people building on, I think that gradual changes might be our best option. Though I'm not sure what all changes you have in mind and that might stray a bit far from this issue. If we're creative enough we mind be able to find a solution.

@LDeakin

But given that we do have a codec interface that we want people building on, I think that gradual changes might be our best option. Though I'm not sure what all changes you have in mind and that might stray a bit far from this issue. If we're creative enough we mind be able to find a solution.

honestly I think the codec API needs a full rewrite, and codec behavior should probably be expressed via a family of protocols. Based on my recent experience working on improving chunk encoding performance, here are a few changes we need:

encode / decode should be sync by default, encode_async and decode_async should be separate methods. Async is costly overhead for any CPU-bound codecs.
we should have encode_into / decode_into methods that take an output array / buffer. This lets the codec pipeline manage memory allocation.
encode / decode should work on a single chunk, not an iterator of chunks. batching should be handled by a separate method, if we even want codecs to know how to batch.
the codec pipeline is clunky and does way too much. it needs to be decomposed into logically separate routines. this process should take feedback from @LDeakin and @ilan-gold based on their experience wrapping zarrs with the zarrs codec pipeline.

We might disagree on the details of some of these changes, but doing even a subset of this gradually would be a lot of work, and require a fair amount of coordination, in a corner of the busy zarr-python codebase. I think it's also harder to motivate people with gradual changes, as opposed a concerted renovation effort.

As opposed to gradually morphing the codec API in-place, I think this alternative has some advantages worth considering:

we upstream the codec API and every other necessary API into a new package(s)
version 0.1.0 is what we have today.
zarr-python depends on 0.1.0 exactly. nothing changes inside zarr-python besides imports and pyproject.toml
we develop 0.2.0 with a bunch of changes, and we test against zarr-python. this is straightforward with a monorepo.
external codec developers can also use the 0.2.0 codec API before zarr-python supports it, to test their codecs.
when 0.2.0 is ready and useful, we release a new version of zarr-python that can use it.

@LDeakin it would be useful to hear what your development cadence is like in the zarrs monorepo. I've never used a monorepo before so the list above is speculative / optimistic :)

@LDeakin

@LDeakin it would be useful to hear what your development cadence is like in the zarrs monorepo

zarrs was originally monolithic and that was great for initial development. I split it into several crates for various reasons and that has since reduced the overall development effort in some areas quite a bit. I wouldn't call zarrs a monorepo exactly. It is a Rust workspace with several core crates, but there are something like 15 crates split over many repos in the org. Maintenance across this ecosystem is pretty easy now due to the relative stability of some of the core API crates. For example, I almost never touch the store crates (e.g. zarrs_object_store, zarrs_opendal, zarrs_icechunk, zarrs_http, zarrs_zip, ...). I just update them when their public dependency (object_store, opendal, icechunk, etc.) gets a major bump.

Overall, it was definitely worth it (for the linked reasons), but it brings complexities around crate versioning/publishing/PR review and it was a substantial initial effort. I gradually split out zarrs over a number of major releases over ~6 months.

and there's a big hitch: our codec API depends on our data type API

Indeed. Data types and codecs are highly coupled and it took some time for me to realise an extensible API for it. zarrs_codec depends on zarrs_data_type, and the data types themselves define how they are encoded/decoded by certain array-to-array and array-to-bytes codecs (see codec data type traits).

Navigation Menu

Repository navigation

zarr v3 scale-offset codec, and should the codec API be defined upstream #3867

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

zarr v3 scale-offset codec, and should the codec API be defined upstream #3867

Description

Activity

maxrjones commented on Apr 3, 2026

d-v-b commented on Apr 3, 2026

keewis commented on Apr 3, 2026

d-v-b commented on Apr 3, 2026

d-v-b commented on Apr 3, 2026

keewis commented on Apr 3, 2026

d-v-b commented on Apr 3, 2026

dstansby commented on Apr 3, 2026

d-v-b commented on Apr 4, 2026

d-v-b commented on Apr 6, 2026

TomAugspurger commented on Apr 11, 2026

d-v-b commented on Apr 11, 2026

TomAugspurger commented on Apr 12, 2026

d-v-b commented on Apr 12, 2026

maxrjones commented on Apr 14, 2026

d-v-b commented on Apr 14, 2026

maxrjones commented on Apr 14, 2026

d-v-b commented on Apr 14, 2026

maxrjones commented on Apr 14, 2026

d-v-b commented on Apr 14, 2026

TomAugspurger commented on Apr 18, 2026

d-v-b commented on Apr 19, 2026

LDeakin commented on Apr 19, 2026

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions