[Defilan]

What

A key removed from an InferenceService's spec.podLabels or spec.podAnnotations is now removed from the pods. Before, changing a value worked but removing a key never did.

Why

Fixes #2287

The update path merges the pod template with mergePreservingExternal, so keys other tools add survive (#456). That helper can't tell a key the user removed from a key another tool added, so it kept both. ModelRouter's new spec.proxy.podLabels had the same leak, and #2285 fixed it there.

How

• Record: the Deployment records the user keys it applied, in the inference.llmkube.dev/applied-pod-labels and inference.llmkube.dev/applied-pod-annotations annotations. These are the annotations feat(router): podLabels and podAnnotations for router-proxy pods #2285 writes on router-proxy Deployments. They live on the Deployment's own metadata, not the template, so writing them doesn't roll the pods.
• Prune: before the merge, a recorded key the spec no longer sets is dropped. Keys other tools add are never recorded, so restartedAt, sidecar annotations and GitOps labels still survive.
• Both computations: the prune runs before the desired-template hash as well as inside the update retry. A removal therefore changes the hash and goes through the RolloutPolicy gate, waitForIdle drain included. With the prune only in the update, the pods would roll with the hash unchanged, skipping the gate. A test catches that case; see the bite checks.
• Operator keys:
  • Labels: the operator's labels win, so a user label on an operator key (app) is never recorded and never pruned.
  • Annotations: the user's values win, so every user annotation is recorded.
  • Removing an override of an operator annotation, such as prometheus.io/scrape: "false" while the operator emits scrape annotations, hands the key back to the operator. It isn't deleted.
• MultiNode: not affected. Member pods are built straight from the template, and a group-hash change recreates the group (SpecChanged).
• Shared code: the record and prune helpers move from router_deployment_builder.go to merge.go, and both reconcilers use them. The annotation names don't change, so records written by 0.10.13 routers stay valid.
• Docs: the PodAnnotations field doc said the operator sets no annotations on inference pods. It sets karpenter.sh/do-not-disrupt and the scrape hints, and the doc now says so. Both field docs now say a removed key leaves the pods.

Upgrade note

• Only removals after the upgrade are pruned. A Deployment written by an older operator has no record, so keys removed before the upgrade stay until removed by hand, for example with kubectl annotate or kubectl label on the Deployment's template.
• Removals now take effect. A user who removed karpenter.sh/do-not-disrupt from podAnnotations will now see it leave the pods, unless the operator's own disruption protection sets it.

Consumers

• Deployment metadata: gains the two record annotations, only while user keys are set.
• Pod template: loses keys the user removed, and the pods roll once, through RolloutPolicy.
• ModelRouter: same behavior as before, now through the shared helpers. The feat(router): podLabels and podAnnotations for router-proxy pods #2285 tests pass unchanged apart from the constant names.
• CRD: description text only, +156 bytes on the InferenceService CRD.

Testing

make gate passes: regen, drift, fmt, vet, lint-all and the full suite.

New tests in internal/controller/inferenceservice_podmeta_test.go drive reconcileDeployment against the fake client:

• TestReconcileDeploymentPrunesRemovedPodMetadata follows one service through these steps:

  1. Create it with two labels and an annotation.
  2. Have another tool add a label and restartedAt.
  3. Restamp the hash with an unchanged spec.
  4. Remove one label and the annotation without changing any value.
  5. Remove everything.

  It asserts that removed keys leave the template, the other tool's keys survive, the hash changes on a pure removal, the record tracks the spec, and the selector never changes.

• TestReconcileDeploymentNeverPrunesOperatorKeys: a user app label isn't recorded, and removing a prometheus.io/scrape: "false" override restores the operator's true.

Bite checks, with each change reverted on its own:

• No prune before the hash:

  inferenceservice_podmeta_test.go:139: desired-template hash unchanged after a removal; the change would skip the RolloutPolicy gate
  

• No prune in the update:

  inferenceservice_podmeta_test.go:129: label tier removed from the spec is still on the template: map[...]
  inferenceservice_podmeta_test.go:150: label team still on the template after podLabels was removed: map[...]
  

• Record not written at create:

  inferenceservice_podmeta_test.go:129: label tier removed from the spec is still on the template: map[...]
  inferenceservice_podmeta_test.go:132: annotation removed from the spec is still on the template: map[...]
  

• Record not synced on update:

  inferenceservice_podmeta_test.go:142: applied labels record = "team,tier", want team
  

• Operator keys recorded:

  inferenceservice_podmeta_test.go:173: applied labels record = "app", want none: app is the operator's
  

All five were run against the final test; map[...] stands in for the full label maps.

Checklist

• Tests added/updated
• make test passes locally
• make lint passes locally
• Commit messages follow conventional commits
• All commits are signed off (git commit -s) per DCO
• AI assistance (if any) is disclosed above, per CONTRIBUTING.md
• Documentation updated (if user-facing change)

AI assistance: this change was AI-assisted. I reviewed the diffs and supervised the build.

[codecov]

Codecov Report

❌ Patch coverage is 97.87234% with 1 line in your changes missing coverage. Please review.

Files with missing lines	Patch %	Lines
internal/controller/merge.go	96.77%	1 Missing ⚠️

📢 Thoughts on this report? Let us know!