Feature: ReadYourWritesConsistency

Upstream officially marked the RV as comparable: kubernetes/enhancements#5505

This allows us to extend the client to block reads until all relevant writes made it back into the cache.

Tasks

v0.25
- Design: ✨ Read your own write client design #3473
- Initial implementation: ✨ Add a read-your-own-write client #3472
v0.25.1
- 🐛 client: fix subresource create RV parse error under read-your-writes consistency #3591
Follow-ups: (more to be added once we merged the design)
- Future goals from the design:
  - Allow more fine-granular configuration about what to wait on as this warrants its own discussion
  - Optimize blocking of namespaced list to only consider objects in the namespace
- Consider adding an additional request-level timeout per default in the consistency client: ✨ Read your own write client design #3473 (comment)
- Ensure Create with GenerateName also works as expected: ✨ Read your own write client design #3473 (comment)
- Make it work with: multiNamespaceCache/multiNamespaceInformer with >1 informer
  - Provide a way to check the resourceVersion of a specific informer in multiNamespaceInformer #3607
- Consider evaluating shouldByPassCache and only wait if the client would use the cache
- Consider adding a method to check if the cache is up-to-date (revisit ✨ Read your own write client design #3473 (comment))
- Consider fail writes that use optimistic locking clientside (if the cache already has a higher RV)
- Optionally use Applyconfigurations: Generate GetResourceVersion kubernetes/kubernetes#141688 (once merged)
- Allow sharing the consistency between two clients that for example differ in user-agent

In general makes sense to me.

I think we should definitely have an option to turn it off though for better performance.

Blocking in mutating operations until the RV in the cache is >= response RV

I think this one absolutely makes sense

Blocking all reading operations in the cache until it it got to >= response RV

I"m not so sure about this one, I assume this would essentially block all reads for a kind if there is a single write for that kind.
I think this could slow down controllers considerably. Especially if there are a lot of reconcilers that are using the same objects.

Delete will be tricky due to #3059

Not sure I follow. Can't we wait until the object is gone from the cache?

(just have to be careful for the case where another object with the same name is created right after deletion)

EDIT: Ah, do you mean the case where a finalizer is set or not? I would hope we can handle both cases

=> wait until object is gone or
=> wait until resourceVersion is high enough

Yeah, because of finalizers. The problem is that since we don't deserialize the object back if it isn't gone from storage, we don't know the post-delete RV, so 2) in your list doesn't have an obvious solution

Got it. If we can't come up with a good user-facing API for Delete we could overwrite Delete in the cached client and handle the response differently there?

@alvaroaleman

Hey everyone, I look at KEP-5505 in a little different perspective, would love to hear your opinion on this. My background is in etcd and API server cache performance/correctness, so I'm newer to the intricacies of controller-runtime.

For me, KEP-5505 is a huge deal. It addresses the complexity of eventual consistency caused by watch. A tradeoff Kubernetes made for scale that put a burden on controller authors. It unlocks the biggest advantage of etcd: the totally ordered revision for every operation. This is the same "magic" that powered the recent API server cache snapshotting improvements (https://kubernetes.io/blog/2025/09/09/kubernetes-v1-34-snapshottable-api-server-cache/,), and now it can be used to build more reliable controllers.

I've seen production incidents where this complexity leads to failure: controllers falling minutes behind, resulting in a DaemonSet creating duplicate pods or a scheduler placing pods on nodes without capacity. It's an unreasonable burden to expect every developer to perfectly handle unbounded staleness, especially when it's only discovered during an incident.

Because of this, I strongly agree with @alvaroaleman's idea of Blocking all reading operations in the cache until it got to >= response RV even make it a default guard rail. As @sbueringer mentioned, it should absolutely be disable-able for performance-critical scenarios, but the default should be safety over performance.

To address the performance concerns, perhaps the API could be more flexible than a simple on/off switch. What if a controller could specify an acceptable staleness? We could also expose informer delay as a metric (like in etcd-io/etcd#20260), allowing controllers to automatically abort reconciliation if staleness exceeds a critical threshold.

I'm not sure what the best user-facing API is or where it should live (informer, controller-runtime, etc.). That's why I'm hoping to combine my perspective with your controller expertise here. Thanks!

I think the following could also be an interesting idea:

Provide a guarantee that a read call will see at least all previous writes done by the same controller/binary/client
If the cache catches up quickly enough (configurable "timeout"), serve from cache
Otherwise fallback to live call

(I don't remember exactly, but I think this would be similar to how one/some of the apiserver caches are working?)

Provide a guarantee that a read call will see at least all previous writes done by the same controller/binary/client

Right, that has been available in API as Not older than request, but not in informer (https://kubernetes.io/docs/reference/using-api/api-concepts/#semantics-for-get-and-list).

If the cache catches up quickly enough (configurable "timeout"), serve from cache

This is what we do for consistent read in watch cache, where we wait for cache to catch up to etcd. Default timeout is 3 seconds. https://kubernetes.io/blog/2024/08/15/consistent-read-from-cache-beta/. Consistent read only differs in that instead of waiting for RV from previous write, it asks etcd what is the latest RV. Either of should be good.

Otherwise fallback to live call

I would refrain from live calls for LIST if possible. They don't have a bounded cost, even with limit as object might differ a lot in size causing high decoding cost. The only exception would be Gets on single object. API server never serves Gets from cache, only from etcd as a way to reduce lock contention. Might be something we consider to avoid issues like kubernetes/kubernetes#130767.

I've been exploring a similar approach and would like to share my prototype implementation for feedback.

Approach: ConsistentClient Wrapper

Instead of modifying the existing Client interface directly, I implemented a ConsistentClient wrapper that provides additional methods with cache consistency guarantees:

type ConsistentClient interface {
    Client
    CreateConsistent(ctx context.Context, obj Object, opts ...CreateOption) error
    UpdateConsistent(ctx context.Context, obj Object, opts ...UpdateOption) error
    PatchConsistent(ctx context.Context, obj Object, opts ...PatchOption) error
    DeleteConsistent(ctx context.Context, obj Object, opts ...DeleteOption) error
    StatusConsistent() ConsistentSubResourceWriter
}

Usage:

consistentClient := client.NewConsistentClient(mgr.GetClient(), mgr.GetCache())
err := consistentClient.CreateConsistent(ctx, obj)
// After this returns, cache is guaranteed to be synced

Implementation Details

The implementation polls the cache until synchronization is confirmed. The key challenge is correctly identifying when the cache is "up-to-date", especially in edge cases.

For Create/Update/Patch (`waitForCacheSync`)

poll:
  ├── NotFound → keep waiting
  │
  ├── UID differs:
  │     ├── cache.CreationTimestamp > target → success (cache has newer object)
  │     └── cache.CreationTimestamp <= target → keep waiting (stale data)
  │
  └── UID matches && RV >= target → success

// waitForCacheSync polls the cache until the object with matching UID has
// a ResourceVersion greater than or equal to the target ResourceVersion.
//
// Note: If the object is deleted after the write operation and no new object
// is created, this will keep polling until the context is cancelled.
// Callers should use an appropriate context timeout.
func (c *consistentClient) waitForCacheSync(ctx context.Context, obj Object) error {
	targetRV := obj.GetResourceVersion()
	targetUID := obj.GetUID()
	targetCreationTimestamp := obj.GetCreationTimestamp()
	key := ObjectKeyFromObject(obj)

	return wait.PollUntilContextCancel(ctx, c.pollInterval, true, func(ctx context.Context) (bool, error) {
		current := obj.DeepCopyObject().(Object)
		if err := c.cacheReader.Get(ctx, key, current); err != nil {
			if apierrors.IsNotFound(err) {
				// Object not yet in cache (or was deleted), keep polling.
				// We cannot distinguish between "not yet synced" and "deleted",
				// so we rely on context timeout for the deleted case.
				return false, nil
			}
			return false, err
		}

		// Check UID to ensure we're looking at the same object.
		if current.GetUID() != targetUID {
			// UID differs - cache has a different object with the same name.
			// Check CreationTimestamp to determine if we should keep waiting:
			// - If cache object is newer (created after ours), cache is up-to-date
			//   with the latest state (our object was deleted and replaced).
			// - If cache object is older, it's stale data not yet removed - keep waiting.
			if current.GetCreationTimestamp().After(targetCreationTimestamp.Time) {
				return true, nil
			}
			return false, nil
		}

		// Compare ResourceVersion to check if cache is synced
		cmp, err := resourceversion.CompareResourceVersion(current.GetResourceVersion(), targetRV)
		if err != nil {
			// If comparison fails, continue polling
			return false, nil
		}
		return cmp >= 0, nil
	})
}

For Delete (`waitForCacheDelete`)

poll:
  ├── NotFound → success
  │
  ├── UID differs:
  │     ├── cache.CreationTimestamp > target → success (object replaced)
  │     └── cache.CreationTimestamp <= target → keep waiting (stale data)
  │
  └── UID matches → keep waiting

// waitForCacheDelete polls the cache until the object with the specified UID
// is no longer present.
//
// Success conditions:
// - Object not found in cache (NotFound error)
// - Object exists but has different UID and newer CreationTimestamp (original was deleted, new one created)
func (c *consistentClient) waitForCacheDelete(ctx context.Context, obj Object, uid types.UID, creationTimestamp metav1.Time) error {
	key := ObjectKeyFromObject(obj)

	return wait.PollUntilContextCancel(ctx, c.pollInterval, true, func(ctx context.Context) (bool, error) {
		current := obj.DeepCopyObject().(Object)
		err := c.cacheReader.Get(ctx, key, current)
		if apierrors.IsNotFound(err) {
			// Object is gone from cache - success
			return true, nil
		}
		if err != nil {
			return false, err
		}

		// If UID is different, check CreationTimestamp to determine cache state
		if current.GetUID() != uid {
			// If cache object is newer, original was deleted and replaced - success
			// If cache object is older, it's stale data - keep waiting
			if current.GetCreationTimestamp().After(creationTimestamp.Time) {
				return true, nil
			}
			return false, nil
		}

		// Same object still exists in cache, keep polling
		return false, nil
	})
}

Why UID + CreationTimestamp?

Comparing only ResourceVersion is insufficient for these edge cases:

Object deleted and recreated with same name: The new object will have a different UID and potentially lower RV. Without UID check, we might incorrectly wait forever or match the wrong object.
Cache contains stale object: When UID differs, we need CreationTimestamp to distinguish between:
- Cache has an older object (stale, not yet removed) → keep waiting
- Cache has a newer object (our object was deleted and replaced) → cache is up-to-date

Important: Context Timeout Behavior

In certain edge cases, the "keep waiting" state will continue polling until context cancellation/timeout:

Create/Update/Patch: If the object is deleted immediately after write and no new object is created, cache returns NotFound indefinitely. We cannot distinguish between "not yet synced" and "actually deleted".
Delete: If the cache has a stale object with the same UID that never gets cleared (unlikely but theoretically possible), polling continues.

This means callers MUST use an appropriate context timeout:

ctx, cancel := context.WithTimeout(ctx, 5*time.Second)
defer cancel()
err := consistentClient.CreateConsistent(ctx, obj)
if errors.Is(err, context.DeadlineExceeded) {
    // Handle timeout - object may have been deleted after creation
}

This is a deliberate design choice - we prefer explicit timeout handling over potentially incorrect "success" responses. However, I'd like to discuss if there are better approaches for these edge cases.

Trade-offs

Pros:

Non-invasive: Doesn't change existing Client interface
Opt-in: Users choose when they need consistency guarantees
Uses official k8s.io/apimachinery/pkg/util/resourceversion.CompareResourceVersion for RV comparison

Cons:

Polling-based (vs event-driven) - though with small intervals (10ms default), this is negligible
Requires passing both Client and Cache reader to constructor
Relies on context timeout for certain edge cases (see above)

Questions for Discussion

Would this approach be acceptable for inclusion in controller-runtime, or would you prefer modifying the core Client interface?
Should we consider an "overlay cache" approach (as described in Java Operator SDK blog) for immediate consistency without polling?
Any concerns about the UID + CreationTimestamp logic for edge case handling?

Block on write vs block on read?

My current implementation blocks on the mutating operation (CUD). As you mentioned in the issue, an alternative is to block subsequent read operations. Trade-offs:

Approach	Pros	Cons
Block on CUD	Simpler mental model; write returns = cache synced	Increases write latency; caller may not need immediate read
Block on Get	Write returns faster; only pays cost when needed	More complex state tracking; need to track "pending" RVs per object

Which approach do you think fits better with controller-runtime's design philosophy?

Happy to share the full implementation or submit a PR if there's interest!

@halfcrazy

@halfcrazy if we add something to controller-runtime itself, I want it to be push- and not poll-based so the perf impact is minimal

Hi All, just putting here our 2 cents, in Java Operator SDK we solve this with an additional cache on top of Informer's cache - (calling this it read-cache-after-write consistency ) - see details here:
operator-framework/java-operator-sdk#2944
So basically a push based approach. We have a poll based utility also, but that will be deprecated after this is released.

Just one note, since would be interested also on your opinion: as far we can see this is not just about cache, so what we do in the current implementation is that we also filter out the event for our change (or up to our change). Thus if we did the change on a resource, and we are now sure that the most recent resource is in the cache, we don't trigger the reconiliation for our own change (thus for the event for that change), only if someone else is changes the resource (not our controller).
I think this is desirable in the general case.
What do you think?

(Of course, users have the option always to reschedule reconciliation explicitly)

Note also that handling delete is an issue properly is an issue with the current Kubernetes API delete response:
operator-framework/java-operator-sdk#3117

thank you!

@alvaroaleman

@alvaroaleman the way we are doing it upstream is by opting in controllers that we've seen issues with with an alpha gate. E.g daemonset controller. Here is what's being done for the daemonset controller as an example of the approach we are taking. The underlying rv store querying should likely be merged in the next couple weeks, at which point any client-go user can use the same approach.

kubernetes/kubernetes#134937

kubernetes/kubernetes#134937

Yeah, I am watching that one :) I don't actually think we need the client-go changes from that for controller-runtime, just the Bookmark() extension on the handler should be sufficient. The difference is that in the DS controller change, the check happens in the beginning of the reconciliation whereas in controller-runtime we want to block all read operations that started after a write operation until a) the write is observed in the cache b) the write errored c) timeout. This is possible because we have a client that reads from cache and writes to the api. Upstream doesn't have such a client, so unless that is added, the approach isn't possible there.

Proposing a design for this in #3473

Hi All, just putting here, how we did this is Java Operator SDK, although it is bit opinionated (with the event filtering), might help:
https://javaoperatorsdk.io/blog/2026/03/13/welcome-read-cache-after-write-consistency/

Also in case let us know if this makes sense. thank you!

Hey folks. I tried EnableReadYourWritesConsistency (v0.25.2) against Karpenter's access pattern: every controller shares one client, so NodeClaim writes are in flight while other loops List NodeClaims. With the blocking design that's too slow for us to turn on. A Java Operator SDK–style overlay gives the same consistency at close to today's latency.

Setup (quick test, not a rigorous benchmark). EKS 1.37 with Karpenter's KWOK provider and 2,000 NodeClaims. The benchmark client is in-region, with about 10ms write round trips. Client-side rate limiting was turned off on both the benchmark and Karpenter. Each client patches NodeClaims from W goroutines and Lists them from another, all through the same client.

NodeClaim writes in flight	today (cache client)	overlay (prototype)	ReadYourWritesConsistency
0	8.1ms	10.0ms	7.8ms
4	8.5ms	10.1ms	20.5ms
16	8.2ms	9.6ms	37.2ms
64	9.1ms	11.9ms	136ms
256	10.8ms	11.6ms	2.03s

In Karpenter's own client metrics during unthrottled scale-ups of 1,000 nodes, we see 2.6–4.8 NodeClaim writes in flight on average and 35–87 at peak. That makes read-your-writes 2–4× slower than today at the mean and up to about 15× at peaks. The overlay costs roughly 1–3ms.

With Karpenter's default client limits (200 QPS / 300 burst) it's worse. The write barrier is taken before the request goes out, so a List also waits behind writes still queued in the client-side rate limiter. At 512 writes in flight, a List takes 2.57s, which is almost exactly 512 / 200 QPS. Under defaults, our scale-ups showed 13–53 writes in flight on average and 270–544 at peak.

We could use DisableReadYourWritesConsistency per call. But the calls we'd opt out of are our hot loops, like our cluster-state sync check, which lists NodeClaims and Nodes on every provisioning and disruption pass. That's exactly where we want the consistency.

The overlay we prototyped:

Writes: a successful write's response goes through the cache's transform and into an overlay keyed by object.
Reads: Get and List merge in the overlay copy when its resourceVersion is newer than the informer's, compared with resourceversion.CompareResourceVersion.
Eviction: push-based. An event handler on the informer drops an entry once the store reaches that resourceVersion.
Event handlers: they never see overlay objects, because the informer's store isn't touched. KEP-5504 removes the objection in Update cache to implement client.Writer and update cache with updated objects returned from API server writes #2209 that versions couldn't be ordered.
Ordering: reads have to snapshot the overlay before reading the cache. Otherwise an eviction landing between the two reads returns the stale copy. Our first version got this wrong.
Gaps: one kind only, and no Delete, field selectors or indexes yet. Delete needs a UID tombstone, plus deserializing the delete response when finalizers keep the object (the same gap as Object doesn't get updated after Client.Delete(object) #3059).

Would you be open to an overlay mode alongside the blocking one? Prototype available in a draft PR, be warned, I've not reviewed it, its probably slop <>

Would you be open to an overlay mode alongside the blocking one?

I don't really want us to maintain two approaches and the drawback of write-through is that you can observe collections that never existed like that on the server. The part that we could optimize is what exactly to wait for in list requests rather than always waiting on everything.

Completely understand not wanting to maintain both approaches!

I don't know what the other controller-runtime users need, but if it matters, that drawback is a non-issue for our usecase. We are deprecating the one use we have for 'aggregate across all of object X, where X has multiple writers and the aggregate needs to be accurate'.

I don't know what the other controller-runtime users need, but if it matters, that drawback is a non-issue for our usecas

Yes, I am aware that this isn't always needed. For something we want to enable by default however, IMHO the correct tradeoff is correctness > performance. In situations like yours were that isn't the correct tradeoff, just don't use this and build something downstream.

@alvaroaleman

@alvaroaleman could you please elaborate on this:

write-through is that you can observe collections that never existed like that on the server

thank you!

write-through is that you can observe collections that never existed like that on the server

Your cache has RV 1 with one object. RV 2 deletes that object. You do a request that creates a different object, get RV3 back and persist it locally in your write-through cache. Listing against the cache now shows two objects, despite the server never having had more than one object.

Not sure if I follow but if the resources are cached based on resource ID (name,ns), there can be no more versions of the same resource in the cache.

(Assuming RV is resource version of the same resource )

RV is a global clock in etcd and the RV on an object is the version at which the object was written (technically per GVK since the KAS allows to have an etcd per GVK). The above refers to this global clock, not any specific object. In the example above, the object that got created at RV1 gets deleted at RV2 and the object created at RV3 is a different one/has a different name or namespace.

Now considering the following logic without overlay cache, resource A1 and A2 is created (RV1, RV2), the A1 gets deleted with RV3. Let's assume lastSyncResourceVersion of the Informer is still RV2 (we are waiting for the delete event), if at this point the user lists resources from informer cache, still sees A1, A2 while in the API server is already just A1. Basically just eventual consistency.

So since we do level based logic in the controllers, the user's logic needs to be the equivalent to handle this scenario as the scenario with overlay cache (OC) in the case you mentioned. In this sense read-after-write is correct, but we realize that this is a bit relaxed consistency model (in generic sense). However from users perspective AFAIK this is correct / good enough since controller logic does not have to change.

Just for the sake of completeness I have to mention that there is one special case where we had to do some tweaking in Informer event handler API:

Consider this scenario (with OC):

We create a resource A1 and cache it
However the informer websocket is down, and will have to do a re-list (resource version too old)
Some deletes the A1 before the re-list
We never receive even a delete event about the A1 (or any event). So A1 would stay in the overlay cache.

For this reason we had to add onList callback to check for those resources and remove them from overlay (but it is trivial to detect those) . But again this is equivalent with a scenario that can happen with eventual consistency anyway.

Navigation Menu

Repository navigation

Feature: ReadYourWritesConsistency #3320

Tasks

Approach: ConsistentClient Wrapper

Implementation Details

For Create/Update/Patch (`waitForCacheSync`)

For Delete (`waitForCacheDelete`)

Why UID + CreationTimestamp?

Important: Context Timeout Behavior

Trade-offs

Questions for Discussion

2 remaining items

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Feature: ReadYourWritesConsistency #3320

Description

Tasks

Activity

sbueringer commented on Sep 16, 2025

alvaroaleman commented on Sep 24, 2025

sbueringer commented on Sep 24, 2025

alvaroaleman commented on Sep 24, 2025

sbueringer commented on Sep 24, 2025

serathius commented on Sep 25, 2025

sbueringer commented on Sep 25, 2025

serathius commented on Sep 25, 2025

halfcrazy commented on Jan 16, 2026

Approach: ConsistentClient Wrapper

Implementation Details

For Create/Update/Patch (waitForCacheSync)

For Delete (waitForCacheDelete)

Why UID + CreationTimestamp?

Important: Context Timeout Behavior

Trade-offs

Questions for Discussion

alvaroaleman commented on Jan 16, 2026

csviri commented on Jan 28, 2026

2 remaining items

michaelasp commented on Jan 28, 2026

alvaroaleman commented on Jan 28, 2026

alvaroaleman commented on Mar 8, 2026

csviri commented on Mar 16, 2026

DerekFrank commented on Oct 23, 2026

alvaroaleman commented on Oct 23, 2026

DerekFrank commented on Oct 26, 2026

alvaroaleman commented on Oct 26, 2026

csviri commented on Oct 26, 2026

alvaroaleman commented on Oct 26, 2026

csviri commented on Oct 26, 2026

alvaroaleman commented on Oct 26, 2026

csviri commented on Oct 27, 2026

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions

For Create/Update/Patch (`waitForCacheSync`)

For Delete (`waitForCacheDelete`)