You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Prototype a KLT implementation that uses Pod Scheduling Readiness instead of the Scheduling Plugin.
Technical Details
KLT currently uses a custom plugin for the scheduler to forbid Pods from being bound to a node. With the new addition of the Readiness Gates, we could eliminate the extra binary and use the new API. This has some consequences for our mutating webhook: it should add a gate when a manifest is applied (on update it won't). When the pre-checks are completed successfully, the gates shall be removed.
The new controller runtime v 0.15.0 will introduce multiple breaking changes that will require code refactoring/changes in all our operators:
the server struct for the webhooks is now an interface, so we need to change all reference of it since we use it to override configs when setting up certificates
few ways to setup the controller manager are deprecated: to set up the port we should use WebhookServer.Port instead and for ClientDisableCacheFor weshould use Client.Cache.DisableCacheFor.
multiple controller unit tests still fail in the poc after the new library version, so they need refactoring. Maybe I missed even other breaking changes 😿
After these changes and the import of controller runtime and k8s 1.27.1 libraries, we can substitute the scheduler with gates, in the poc this is done by adding an extra webhook reacting to the creation of new pods:
extract common functions from the mutating webhook
introduce a gating webhook that simply adds a keptn gated at creation
remove the gate in the webhook instance controller after all pre check passed
modify operator/config/default/webhooknamespaces_patch.yaml and operator/config/webhook/manifests.yaml to include the new webhook
add the extra webhook in the list passed to builder.run in operator/main.go
It is left do decide also what to do with the scheduler. Shall we do a breaking change and delete it or add some logic to enable/disable gates or scheduler for at least one version?
Activity
thisthat commentedon Apr 13, 2023
Old PoC that used a workaround with extra labels: #632
thisthat commentedon May 4, 2023
PR kubernetes-sigs/controller-runtime#2189 has been merged and unlocks this research
RealAnna commentedon May 22, 2023
The new controller runtime v 0.15.0 will introduce multiple breaking changes that will require code refactoring/changes in all our operators:
After these changes and the import of controller runtime and k8s 1.27.1 libraries, we can substitute the scheduler with gates, in the poc this is done by adding an extra webhook reacting to the creation of new pods:
It is left do decide also what to do with the scheduler. Shall we do a breaking change and delete it or add some logic to enable/disable gates or scheduler for at least one version?