Repository navigation
HealthChecks failing with "no matches for kind..." whereas CRD is installed. #1680
Description
Using kustomize controller v1.8.5, we have a kustomization running this healthcheck (among others):
healthChecks:
- apiVersion: infrastructure.cluster.x-k8s.io/v1beta1
kind: OpenStackCluster
name: mgmt-2620952448-rke2-capo
namespace: sylva-system
that fails to reconcile with following error (first occurence at 2026-06-22T20:09:10.247Z):
health check failed after 30.034806535s: timeout waiting for: [OpenStackCluster/sylva-system/mgmt-2620952448-rke2-capo status: 'Unknown': no matches for kind "OpenStackCluster" in group "infrastructure.cluster.x-k8s.io"]
whereas the crd is already installed by another kustomization, and has a single version:
kubectl get customresourcedefinitions.apiextensions.k8s.io openstackclusters.infrastructure.cluster.x-k8s.io -o yaml | yq -r '.metadata.creationTimestamp, .spec.versions[].name'
2026-06-22T20:06:23Z
v1beta1
and the custom resource already exists in cluster:
❯ kubectl get openstackclusters.infrastructure.cluster.x-k8s.io mgmt-2620952448-rke2-capo -o yaml | yq .metadata.creationTimestamp
2026-06-22T20:06:47Z
This looks similar to that issue fluxcd/flux2#531, but this issue is fixed in v1.8.5, and the CRD has a single version installed...
Side note: we don't have the DisableStatusPollerCache feature enabled in our kustomize controller deployments, maybe it could help fixing that issue, as it looks like a cache miss. But this shouldn't have occurred anyway...
Reacted by Martin Hansen
Reacted by Thomas Morin
Activity
zioc commentedon Jul 6, 2026
After digging a bit further, it turns out that this issue comes from a caching issue in controller-runtime RESTMapper (see upstream issue)
This issue is fairly critical, since the kustomize controller may endlessly fail to perform healthChecks on a groupKind that has been installed in a cluster after the initial discovery of the group, as it has been observed in sylva project.
As a short-term workaround, we've configured the failing Kustomizations with a serviceAccount. In that case the kustomize controller creates a new RESTMapper during each reconciliation, and avoids the cache invalidation issue.
This is issue could also be fixed in kstatus library by adding the group version to RESTMapping requests when it is available in kustomization healthCheck's NamespacedObjectKindReference: When version is provided to the restmapper, it is not affected by the cache invalidation issue described above, and it is much more efficient as it uses serverResourcesForGroupVersion under the hood that fetches a single group version in the API instead of ServerGroupsAndResources that requests all api groups and versions.
stefanprodan commentedon Jul 6, 2026
I think no one hit this issue because most people deploy Flux with multi-tenancy enabled. Running each reconciliation under a dedicated SA should not be just "a short-term workaround", it's how Flux should run in production system.
zioc commentedon Jul 7, 2026
Sure, but for now it also implies that statusPoller cache won't be shared across reconciliations, which can also lead to some some performance issues as described in #1436.
stefanprodan commentedon Jul 7, 2026
A shared cache comes with downsides too, as many reconciliation run in parallel and they all block on the cache write mutex. The tradeoff is goroutine contention vs increased memory usage. Anyway if this gets fixed upstream, we'll update controller-runtime to fix this bug for users who don't run Flux in multi-tenancy mode.
matheuscscp commentedon Jul 7, 2026
A shared cache implies RBAC cannot gate tenants.
zioc commentedon Jul 7, 2026
Thanks for your feedbacks. In sylva project we have a fairly specific use of flux where all flux resource are managed by a single entity, so we were no urged to use multi-tenancy (but still plan to do so, it's a work in progress).
I was not aware of the shared cache write mutex thought, it makes a point indeed.