Aggregated apiserver rejects all writes with "not yet ready to handle request" on k8s 1.36+ when SA lacks RBAC for MutatingAdmissionPolicy #1841

What steps did you take:

1. Install a kapp-controller build that vendors k8s.io/apiserver v0.36.0+ on a Kubernetes 1.36+ cluster (reproduced on kind 1.36 and EKS 1.36).
2. Ensure the deployed ClusterRole for the kapp-controller ServiceAccount does NOT include list + watch on mutatingadmissionpolicies and mutatingadmissionpolicybindings in the
   admissionregistration.k8s.io group. This state arises naturally when: (a) an operator upgrades kapp-controller in-place without re-applying the shipped ClusterRole, (b) a distribution ships its own ClusterRole
   derived from an older release, or (c) an operator has deliberately narrowed RBAC.
3. Create a PackageRepository targeting any OCI bundle:
   kubectl apply -f - <<EOF
   apiVersion: packaging.carvel.dev/v1alpha1
   kind: PackageRepository
   metadata:
   name: repro
   namespace: default
   spec:
   fetch:
   imgpkgBundle:
   image:
   EOF
4. Observe the reconciler state:
   kubectl describe packagerepository/repro

What happened:

The PackageRepository fails to reconcile with:

Reconcile failed: kapp: Error: create packagemetadata/<name> (data.packaging.carvel.dev/v1alpha1) namespace: default:
    Creating resource packagemetadata/<name> (data.packaging.carvel.dev/v1alpha1) namespace: default:
      API server says: packagemetadatas.data.packaging.carvel.dev "<name>" is forbidden:
      not yet ready to handle request (reason: Forbidden)

Every mutating request into data.packaging.carvel.dev/v1alpha1 fails the same way. Reads (get, list, watch) continue to work. The kapp-controller pod reports Ready 2/2. The aggregated APIService reports
Status.Available=True. Pod logs contain a high-frequency stream of Failed to watch *v1.MutatingAdmissionPolicy: ... is forbidden: User "system:serviceaccount:kapp-controller:kapp-controller-sa" cannot list resource "mutatingadmissionpolicies".

Root cause:

k8s.io/apiserver v0.36 added the MutatingAdmissionPolicy admission plugin to RecommendedPluginOrder by default, and the plugin's feature gate went GA-on in Kubernetes 1.36. On 1.36+ clusters the plugin loads automatically; its SharedInformer tries to list+watch MutatingAdmissionPolicy and MutatingAdmissionPolicyBinding; if RBAC blocks that, HasSynced() never returns true and every mutating
admission decision returns admission.NewForbidden(a, fmt.Errorf("not yet ready to handle request")) from vendor/k8s.io/apiserver/pkg/admission/plugin/policy/generic/plugin.go. pkg/apiserver/apiserver.go
already guards each admission-policy plugin behind a discovery-based serverResourceExists check, but discovery only proves the resource exists in the cluster — not that the ServiceAccount can watch it. On 1.36+
the resource always exists, so the guard passes and the plugin loads even when RBAC is insufficient.

The same failure mode can affect ValidatingAdmissionPolicy on k8s 1.30+ when its RBAC is out of sync.

What did you expect:
Loading an admission-policy plugin whose backing SharedInformer cannot start should not permanently break the aggregated apiserver. The apiserver should either:

1. Not load the plugin when the ServiceAccount lacks the RBAC required to run its informers, and emit a clear, actionable warning naming the specific resource and verb that would need to be granted; OR
2. Surface the failure loudly at startup (fail-fast or prominent error-level log) rather than passing readiness checks and silently rejecting every write.

At a minimum, APIService.Status.Available=True should not be reported when the admission chain is permanently in the "not yet ready" state.

Anything else you would like to add:

Suggested fix: extend the existing plugin guard in pkg/apiserver/apiserver.go to also probe the ServiceAccount's permissions via SelfSubjectAccessReview before loading the plugin. Concretely, replace the two serverResourceExists(...) calls in the plugin-order block with a new helper canUseAdmissionPolicyPlugin(discoveryClient, coreClient, policyResource, bindingResource, logger) that returns true only when:

• Both resources are present under admissionregistration.k8s.io/v1 in discovery, AND
• A SelfSubjectAccessReview for list and for watch on each of the two resources comes back with Allowed=true.

When the check fails due to missing RBAC, log a single warning naming the specific (resource, verb) pair the SA is missing, so operators can grep and fix. The SelfSubjectAccessReview probe uses
subjectaccessreviews/create, which is already granted in the shipped ClusterRole — no new RBAC required for the probe itself.

The proposed fix keeps the aggregated apiserver functional across every supported k8s version regardless of the operator's RBAC state, and turns a silent 403-forever loop into a single actionable log line at startup.

Environment:

• kapp-controller version: any build vendoring k8s.io/apiserver v0.36.0+ (including develop at HEAD). Releases vendoring older k8s.io/apiserver do not exhibit this bug because the MutatingAdmissionPolicy
  plugin does not exist in those library versions.
• Kubernetes version:
  Client Version: v1.36
  Server Version: v1.36

Vote on this request

This is an invitation to the community to vote on issues, to help us prioritize our backlog. Use the "smiley face" up to the right of this comment to vote.

👍 "I would like to see this addressed as soon as possible"
👎 "There are other more important things to focus on right now"

We are also happy to receive and review Pull Requests if you want to help working on this issue.