bug: ClusterCachedResource replication stops after a leader-lease loss; status stays Ready #4451

Describe the bug

On kcp v0.33.1, one loss of the kcp-controllers lease stops ClusterCachedResource replication.
The process keeps running. Writes never reach the cache server. Status stays Ready with
ReplicationStarted=True. Only a process restart recovers it.

After re-election, every reconcile fails:

handler {…} was not added to shared informer because it has stopped already

Each failed retry leaks one event handler. A single write then logs queuing object N times.
Nothing is replicated.

A second trigger is fully silent: delete and recreate the replicated CRD. Replication stops with no
log line and no error.

We first hit this in our deployment on v0.32.5 (CachedResource): three stalls in two days. The logs
show both signatures (evidence/10-stall-2.txt, evidence/20-stall-3.txt). main has the same
code as v0.33.1.

10-stall-2.txt

20-stall-3.txt

Steps To Reproduce

Standalone v0.33.1 shard, embedded etcd and cache server:

kcp start --feature-gates=CacheAPIs=true --enable-leader-election=true

1. In a workspace, create a cluster-scoped CRD widgets.widgets.example.com, one Widget, and a
   ClusterCachedResource for widgets.example.com/v1alpha1/widgets.
2. Annotate the Widget. The cache copy updates at
   /services/cache/shards/root/clusters/<cluster>/apis/widgets.example.com/v1alpha1/widgets/<name>.
   The log shows one queuing object from local and one from global.
3. Take the lease away. A missed 5 s renew does the same.

   kubectl --server …/clusters/system:admin -n kube-system patch lease kcp-controllers --type=merge \
     -p '{"spec":{"holderIdentity":"intruder","renewTime":"<now>","leaseDurationSeconds":60}}'

   The shard logs stopped leading, then started leading about 60 s later.
4. Every reconcile now fails with stopped already. Retries back off and never succeed. They were
   still failing after 22 min.
5. Annotate the Widget again. The cache copy does not update. The write logs N queuing object
   lines, all from="local". N equals the failed retries so far (13–14 in the first minute).
   Status stays Ready.

Reproduced 2/2. A restart recovers it. Details are in evidence/40-repro.txt.

Silent variant: delete the CRD and wait for removing dynamic informer for gvr. Recreate the CRD
and Widget, then write. Nothing replicates, and nothing is logged.

40-repro.txt

Expected Behaviour

• A lease loss does not break replication.
• Global informers belong to a running factory, not to one reconcile's context.
• A controller whose informer stops is re-created.
• Handlers are removed on every exit path.
• Ready and ReplicationStarted are set after sync, and cleared when replication stops.

Additional Context

The paths below are v0.33.1, under pkg/; evidence/30-source.txt has more detail.
R = reconciler/cache/clustercachedresources.

Trigger

• server/server.go:223 builds CachePartialMetadataDDSIF, and server/controllers.go:2015
  passes it as the Global factory.
• Nothing starts that factory. Only PartialMetadataDDSIF gets StartWorker (server.go:682).
• So the Global informer's only Run is go replicated.Global.Run(ctx.Done())
  (R/clustercachedresources_reconcile_replication.go:131).
• That ctx is leaderElectionCtx (server.go:870), and RenewDeadline is 5 s (server.go:859).
• On lease loss the informer stops but stays in the factory's map. After re-election,
  ForResource keeps returning it.
• Every ClusterCachedResource on the shard breaks at once.
• Local informers usually survive. Their factory already started them, so the reconcile's Run
  does nothing.

Defects

1. Handlers leak (R/replication/replication_controller.go:101-131).
   • NewController adds the Local handler, then the Global one. Only Start removes them
     (:153-158).
   • When the Global add fails, the Local handler stays registered. It fills a queue that has no
     workers. That is one leak per retry.
   • The WaitForCacheSync failure path (R/clustercachedresources_reconcile_replication.go:142-145)
     also skips Start, so it leaks both handlers.
2. Registered controllers are never re-checked (R/clustercachedresources_reconcile_replication.go:70-74).
   • A controller is built only when the registry has none.
   • When the GVR leaves discovery, the local factory drops its informer
     (informer/informer.go:691-699).
   • The registered controller keeps the dead informer. This is the silent variant.
3. Status is set before sync (R/clustercachedresources_reconcile_replication.go:133-136).
   • WaitForCacheSync (:142) has no timeout.
   • Nothing clears the status when replication stops.

Other versions

• main (373d921): byte-identical to v0.33.1 in these files.
• v0.32.5 (our deployment): same bug, reproduced 2/2. Its handler,
  {0x36c2ee0 {0x36c3080 0x36c2fe0 0x36c2f60}}, matches stall 2 exactly.

No existing issue found. #3812 (flake: TestCachedResources) never loses the lease. It may share
defect 1 or 2.

30-source.txt