Repository navigation
feat(janitor): implement ExternalRemediationRequest reconciler - #1392
Merged
natherz97 merged 27 commits intoJun 30, 2026
Merged
Conversation
Contributor
|
🌿 Fern Docs Preview: https://nvidia-preview-pull-request-1392.docs.buildwithfern.com/nvsentinel |
Contributor
|
Note Reviews pausedIt looks like this branch is under active development. To avoid overwhelming you with review comments due to an influx of new commits, CodeRabbit has automatically paused this review. You can configure this behavior by changing the Use the following commands to manage reviews:
Use the checkboxes below for quick actions:
📝 WalkthroughWalkthroughIntroduces the ChangesExternal Remediation Reconciliation Flow
Sequence Diagram(s)sequenceDiagram
participant Test
participant Webhook
participant ExtRRController
participant Node as Kubernetes Node
Test->>Webhook: CreateExtRRCR (spec.healthEvent.nodeName)
Webhook-->>Test: Accepted
ExtRRController->>ExtRRController: reconcileInitialize (finalizer + Unknown conditions)
ExtRRController->>Node: Patch release taint + managed=false label
ExtRRController->>ExtRRController: NVSentinelOwnershipReleased=True
Test->>ExtRRController: SetExtRRComplete(True)
ExtRRController->>Node: Remove release taint + delete managed label
ExtRRController->>ExtRRController: recordCloseMetrics + markCompletionTime
Test->>Node: assertNodeHasNoReleaseTaint
Estimated code review effort🎯 5 (Critical) | ⏱️ ~120 minutes Possibly related PRs
Suggested reviewers
Poem
🚥 Pre-merge checks | ✅ 4 | ❌ 1❌ Failed checks (1 warning)
✅ Passed checks (4 passed)
✨ Finishing Touches🧪 Generate unit tests (beta)
Comment |
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
Contributor
There was a problem hiding this comment.
Actionable comments posted: 8
🧹 Nitpick comments (1)
janitor/pkg/controller/externalremediationrequest_controller.go (1)
117-141: 💤 Low valueConsider extracting "unknown" as a package constant.
The string literal
"unknown"appears multiple times in this file (lines 123, 127, 130). Extract it as a package-level constant to improve maintainability and reduce the risk of typos.🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@janitor/pkg/controller/externalremediationrequest_controller.go` around lines 117 - 141, Extract the repeated literal "unknown" into a package-level constant (e.g., const unknownLabel = "unknown") and replace all occurrences inside recommendedActionLabel and errNodeLabel (and any other uses in this file) with that constant; update the functions recommendedActionLabel(extrrObj *nvsentinelv1.ExternalRemediationRequest) and errNodeLabel(extrrObj *nvsentinelv1.ExternalRemediationRequest) to return unknownLabel instead of the raw string to centralize the value and avoid typos.Source: Linters/SAST tools
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@janitor/pkg/condition/condition.go`:
- Around line 26-31: The import block in condition.go is misformatted; run gofmt
-s -w or goimports -w to reorder and format the imports so they conform to
gofmt/goimports rules (group standard library, external, then internal packages)
and remove/adjust the blank line between metav1, timestamppb and protos; check
the imports that reference metav1, timestamppb and protos remain unchanged after
formatting.
In `@janitor/pkg/controller/externalremediationrequest_controller.go`:
- Around line 262-302: In setInitialConditions, add a blank line immediately
above the final return statement so there is whitespace separating the call to
r.patchStatusConditions(...) and the subsequent "return patched, err" line; this
improves readability around the end of the function where conditions are patched
and returned (references: setInitialConditions, r.patchStatusConditions,
patched, err).
- Around line 202-222: The function needsInitialization in
ExternalRemediationRequestReconciler has a line that exceeds the 120-character
limit; locate the long conditional check (the meta.FindStatusCondition call that
checks ConditionNVSentinelOwnershipReleased on the conds slice inside
needsInitialization) and break it into two shorter lines (e.g., assign
meta.FindStatusCondition(conds, ConditionNVSentinelOwnershipReleased) to a local
variable or wrap the if condition across lines) so no source line exceeds 120
characters while preserving the same logic and returning true when the condition
is nil.
- Around line 167-200: In Reconcile (method
ExternalRemediationRequestReconciler.Reconcile) insert a blank line immediately
above the defer span.End() to separate the
tracing.StartSpanWithLinkFromTraceContext(...) call/assignment from the defer;
locate the tracing.StartSpanWithLinkFromTraceContext invocation that returns
ctx, span and the subsequent defer span.End() and add one empty line for
improved readability and to satisfy the static analysis style rule.
- Around line 360-441: reconcileApply is over the cyclomatic complexity
threshold; extract three helpers to simplify control flow: (1) move node
retrieval and missing-node requeue logic out of reconcileApply into a helper
like getNodeForApply(ctx, extrrObj) that returns (*corev1.Node, ctrl.Result,
error) to encapsulate the IsNotFound requeue behavior; (2) extract the taint
ownership/drift checks into checkTaintOwnership(node *corev1.Node, extrrObj
*nvsentinelv1.ExternalRemediationRequest) which returns an enum/struct or
(status, message) indicating "owned-by-us", "label-missing", "owned-by-other" so
reconcileApply can early-return to transitionToReleaseSuccess or
transitionToReleaseFailure without nested branches; and (3) pull the Patch error
handling into handlePatchError(err, nodeName, extrrObj) which returns
(ctrl.Result, error) and encapsulates the apierrors.IsForbidden branch and the
generic fmt.Errorf branch; update reconcileApply to call these helpers to reduce
branching and meet the complexity threshold.
- Around line 586-659: reconcileCleanup is too complex—extract two helpers: (1)
a node fetch helper on ExternalRemediationRequestReconciler (e.g.
fetchNodeForCleanup(ctx, nodeName) returning (*corev1.Node, bool, error)) that
wraps r.Get and implements the IsNotFound short-circuit used today, and (2) a
taint decision helper (e.g. shouldRemoveReleaseTaint(taints []corev1.Taint,
extrrName string) returning (newTaints []corev1.Taint, removed bool, ownerDrift
bool)) that uses findTaintByKey, ReleaseTaintKey and removeTaintByKey to decide
whether to remove the taint (match on extrrObj.Name) or leave it and signal
drift; then simplify reconcileCleanup to call these two helpers, adjust changed
logic and keep the existing Patch and logging paths intact.
- Around line 808-827: The reconciler currently uses the deprecated
mgr.GetEventRecorderFor and client-go record.EventRecorder; update
ExternalRemediationRequestReconciler.SetupWithManager to call
mgr.GetEventRecorder(...) and change the reconciler's Recorder field type and
import from k8s.io/client-go/tools/record.EventRecorder to
k8s.io/client-go/tools/events.EventRecorder (update imports and any uses of
Recorder accordingly), update tests to replace record.NewFakeRecorder usages
with the appropriate events-based fake/constructor, and add RBAC markers for
events.k8s.io if controller-runtime requires them.
In `@janitor/pkg/metrics/err_metrics.go`:
- Around line 48-58: The comment above the ExtRRTotal declaration has a line
exceeding the 120-char limit; edit the block comment describing phases (the long
sentence that starts with "phase=created ..." and continues through
"operator_deleted (kubectl delete err)") to wrap or split into multiple shorter
lines (or shorten wording) so no single comment line exceeds 120 characters
while preserving the same phase/result descriptions and labels for ExtRRTotal.
---
Nitpick comments:
In `@janitor/pkg/controller/externalremediationrequest_controller.go`:
- Around line 117-141: Extract the repeated literal "unknown" into a
package-level constant (e.g., const unknownLabel = "unknown") and replace all
occurrences inside recommendedActionLabel and errNodeLabel (and any other uses
in this file) with that constant; update the functions
recommendedActionLabel(extrrObj *nvsentinelv1.ExternalRemediationRequest) and
errNodeLabel(extrrObj *nvsentinelv1.ExternalRemediationRequest) to return
unknownLabel instead of the raw string to centralize the value and avoid typos.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: CHILL
Plan: Enterprise
Run ID: f37c4f65-f22b-4fb2-a71c-b6ce4b3db6d8
📥 Commits
Reviewing files that changed from the base of the PR and between 29e14d9 and 14bf562c725eef0859cb510c121a4b6a6840a7fd.
📒 Files selected for processing (8)
janitor/main.gojanitor/pkg/condition/condition.gojanitor/pkg/condition/condition_test.gojanitor/pkg/controller/externalremediationrequest_controller.gojanitor/pkg/controller/externalremediationrequest_controller_test.gojanitor/pkg/controller/suite_test.gojanitor/pkg/metrics/err_metrics.gojanitor/pkg/metrics/metrics.go
jtschelling
force-pushed
the
feature/jsc-103-err-reconciler
branch
from
June 11, 2026 20:20
0ea43b0 to
94158b1
Compare
Contributor
There was a problem hiding this comment.
🧹 Nitpick comments (1)
janitor/pkg/controller/externalremediationrequest_controller_test.go (1)
1064-1084: 💤 Low valueConsider removing or documenting unused helper.
snapshotConditionsis defined but not called anywhere in this test file. If it's intended for debugging or future use, consider adding a comment indicating that purpose, or remove it to avoid dead code.🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@janitor/pkg/controller/externalremediationrequest_controller_test.go` around lines 1064 - 1084, The helper function snapshotConditions (func snapshotConditions(extrrObj *nvsentinelv1.ExternalRemediationRequest) string) is currently unused; either delete it to remove dead code or keep it but add a short comment explaining its purpose (e.g., "helper for debugging/test snapshots of ExternalRemediationRequest conditions") and/or call it from a relevant test so it is exercised; update or add a linter suppression only if you intentionally keep it unused.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Nitpick comments:
In `@janitor/pkg/controller/externalremediationrequest_controller_test.go`:
- Around line 1064-1084: The helper function snapshotConditions (func
snapshotConditions(extrrObj *nvsentinelv1.ExternalRemediationRequest) string) is
currently unused; either delete it to remove dead code or keep it but add a
short comment explaining its purpose (e.g., "helper for debugging/test snapshots
of ExternalRemediationRequest conditions") and/or call it from a relevant test
so it is exercised; update or add a linter suppression only if you intentionally
keep it unused.
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: CHILL
Plan: Enterprise
Run ID: dfb20a53-f3ff-40ea-8f2b-71953cbd0142
📥 Commits
Reviewing files that changed from the base of the PR and between 0ea43b0f9f578268dac680f6c4263efd4baeefbe and 94158b1.
📒 Files selected for processing (8)
janitor/main.gojanitor/pkg/condition/condition.gojanitor/pkg/condition/condition_test.gojanitor/pkg/controller/externalremediationrequest_controller.gojanitor/pkg/controller/externalremediationrequest_controller_test.gojanitor/pkg/controller/suite_test.gojanitor/pkg/metrics/err_metrics.gojanitor/pkg/metrics/metrics.go
🚧 Files skipped from review as they are similar to previous changes (6)
- janitor/main.go
- janitor/pkg/controller/suite_test.go
- janitor/pkg/metrics/err_metrics.go
- janitor/pkg/condition/condition.go
- janitor/pkg/condition/condition_test.go
- janitor/pkg/controller/externalremediationrequest_controller.go
Contributor
Merging this branch will decrease overall coverage
Coverage by fileChanged files (no unit tests)
Please note that the "Total", "Covered", and "Missed" counts above refer to code statements instead of lines of code. The value in brackets refers to the test coverage of that file in the old version of the code. |
Contributor
Author
|
@coderabbitai review |
Contributor
✅ Action performedReview finished.
|
Contributor
Merging this branch changes the coverage (2 decrease, 1 increase)
Coverage by fileChanged files (no unit tests)
Please note that the "Total", "Covered", and "Missed" counts above refer to code statements instead of lines of code. The value in brackets refers to the test coverage of that file in the old version of the code. Changed unit test files
|
Contributor
There was a problem hiding this comment.
Actionable comments posted: 1
🧹 Nitpick comments (1)
janitor/pkg/controller/externalremediationrequest_controller.go (1)
15-15: 💤 Low valueAdd package-level godoc for the controller package.
Per coding guidelines, all Go packages should include package-level godoc. Consider adding a comment above the
package controllerline documenting the package's purpose.📝 Suggested addition
+// Package controller implements Kubernetes reconcilers for NVSentinel custom +// resources including RebootNode, TerminateNode, GPUReset, and +// ExternalRemediationRequest controllers. package controller🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@janitor/pkg/controller/externalremediationrequest_controller.go` at line 15, Add a package-level godoc comment immediately above the "package controller" declaration describing the purpose and responsibilities of the controller package (e.g., that it implements reconciliation logic for ExternalRemediationRequest resources and related helpers), mention the main abstractions it contains (controllers, reconcilers, and helpers for ExternalRemediationRequest), and keep it concise and formatted as a proper Go comment block to satisfy godoc requirements.Source: Coding guidelines
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@janitor/pkg/controller/externalremediationrequest_controller.go`:
- Around line 148-152: Rename the local variable currently named err (type
nvsentinelv1.ExternalRemediationRequest) to extrrObj in the Reconcile method of
ExternalRemediationRequestReconciler to avoid shadowing the conventional error
name; update the Get call and all subsequent uses (currently lines referencing
err) to use extrrObj, and rename the temporary error variable e (from the Get
call) to the conventional err so the function reads: declare extrrObj :=
nvsentinelv1.ExternalRemediationRequest (or var extrrObj
nvsentinelv1.ExternalRemediationRequest), call r.Get(..., &extrrObj) and handle
the returned error as err := r.Get(...) / if err != nil { return ctrl.Result{},
client.IgnoreNotFound(err) } ensuring all references to the ExtRR object are
updated to extrrObj.
---
Nitpick comments:
In `@janitor/pkg/controller/externalremediationrequest_controller.go`:
- Line 15: Add a package-level godoc comment immediately above the "package
controller" declaration describing the purpose and responsibilities of the
controller package (e.g., that it implements reconciliation logic for
ExternalRemediationRequest resources and related helpers), mention the main
abstractions it contains (controllers, reconcilers, and helpers for
ExternalRemediationRequest), and keep it concise and formatted as a proper Go
comment block to satisfy godoc requirements.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: CHILL
Plan: Enterprise
Run ID: 67c3c8d4-d01d-440f-ab5c-f7af09f94d3c
📥 Commits
Reviewing files that changed from the base of the PR and between 0ea43b0f9f578268dac680f6c4263efd4baeefbe and 859e5bc.
📒 Files selected for processing (8)
janitor/main.gojanitor/pkg/condition/condition.gojanitor/pkg/condition/condition_test.gojanitor/pkg/controller/externalremediationrequest_controller.gojanitor/pkg/controller/externalremediationrequest_controller_test.gojanitor/pkg/controller/suite_test.gojanitor/pkg/metrics/err_metrics.gojanitor/pkg/metrics/metrics.go
🚧 Files skipped from review as they are similar to previous changes (7)
- janitor/pkg/controller/suite_test.go
- janitor/pkg/metrics/metrics.go
- janitor/main.go
- janitor/pkg/condition/condition.go
- janitor/pkg/condition/condition_test.go
- janitor/pkg/metrics/err_metrics.go
- janitor/pkg/controller/externalremediationrequest_controller_test.go
Contributor
Merging this branch will decrease overall coverage
Coverage by fileChanged files (no unit tests)
Please note that the "Total", "Covered", and "Missed" counts above refer to code statements instead of lines of code. The value in brackets refers to the test coverage of that file in the old version of the code. |
Contributor
Merging this branch will decrease overall coverage
Coverage by fileChanged files (no unit tests)
Please note that the "Total", "Covered", and "Missed" counts above refer to code statements instead of lines of code. The value in brackets refers to the test coverage of that file in the old version of the code. Changed unit test files
|
Adds the controller-runtime reconciler that drives the ADR-040 ExtRR lifecycle. Owns the six-branch state machine: init (finalizer + initial Unknown conditions), deletion-cleanup, apply (release taint + managed=false in a single strategic-merge PATCH), True-cleanup, asymmetric False no-op, and the released steady-state. Includes: * Proto Condition ↔ metav1.Condition adapter (pkg/condition) so the reconciler uses standard meta.SetStatusCondition helpers while the proto stays authoritative on the apiserver. * Field indexer on spec.healthEvent.nodeName so the Node watch maps to ExtRRs in O(1) instead of scanning every ExtRR per Node event. * Predicate that drops Node events outside the release-state surface (managed label + release taint), so kubelet heartbeats don't re-enqueue every ExtRR. * Belt-and-suspenders nil-spec guard in Reconcile — the validating webhook enforces the contract, but a webhook outage shouldn't crashloop the controller. * Prometheus metrics in the nvsentinel_external_remediation_ namespace (total / open / age_seconds) with phase + result labels per ADR-040. * Ginkgo envtest suite covering init, apply, drift, RBAC denial, branch 2 + 4 cleanup paths, asymmetric False, and observability assertions. * Focused unit tests for the field indexer, node-watch predicate, and mapper using a fake client with the index pre-registered. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
…quest Adds a typed validator that enforces the contract the reconciler depends on: spec, spec.healthEvent, and spec.healthEvent.nodeName must all be populated, and nodeName is immutable after creation (the release taint carries the ExtRR name so flipping nodeName would orphan a node). The proto-backed CRD wrapper has pointer Spec/Status fields (forced by the proto's embedded sync.Mutex), so the apiserver's OpenAPI schema accepts spec: null and similar incomplete objects. The webhook closes that gap with failurePolicy=Fail so the reconciler can rely on a well-formed Spec. ExtRR intentionally has no Config.Enabled gate: per ADR-040 it is an unconditional capability — disabling it would strand any external system mid-remediation with no way to release affected nodes. Includes: * extrrValidator in pkg/webhook/v1alpha1 modelled on the existing RebootNode / TerminateNode / GPUReset validators. * kubebuilder marker emitting the standard webhook path. * Helm chart entry that wires the same cert/service infra as the sibling webhooks. * Ginkgo unit tests covering create rejection paths and update immutability. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Adds five end-to-end tests against a chart-deployed janitor + webhook that prove the wiring works in a real cluster (cert provisioning, RBAC, manifest correctness, reconciler boot): * TestExtRRWebhookRejectsInvalidSpec — exercises the four webhook rejection paths (nil spec, nil healthEvent, empty nodeName, immutable nodeName on update). Proves the kubebuilder marker, helm chart entry, and cert wiring are all aligned. * TestExtRRLifecycleHappyPath — full ADR-040 happy path: apply → release taint + managed=false → Complete=True → cleanup → garbage collection. * TestExtRRAsymmetricFalse — Complete=False is a no-op; node remains released. Complete=True retry from False then closes the ExtRR. * TestExtRROperatorDeleteEscape — kubectl delete extrr while stuck at Complete=False drives the finalizer-based cleanup. * TestExtRRForeignTaintDrift — drift-safety: a fresh ExtRR for a node already tainted by another ExtRR transitions to NVSentinelOwnershipReleased=False, leaving the foreign taint alone. Each test's Teardown does a belt-and-suspenders scrub of the test node's taint+label via ScrubExtRRStateFromNode, so a mid-test failure doesn't leak state into subsequent tests sharing the same Node. Helpers added in tests/helpers/kube.go: CreateExtRRCR, CreateMalformedExtRR, SetExtRRComplete, WaitForExtRRCondition, WaitForExtRRGone, ScrubExtRRStateFromNode, plus the ExternalRemediationRequestGVK constant. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
jtschelling
force-pushed
the
feature/jsc-103-err-reconciler
branch
from
June 19, 2026 20:37
74f8f7b to
e3ae6d4
Compare
…collection The CI failure (TestExtRRLifecycleHappyPath/Complete=True_triggers_cleanup; ExtRR_garbage-collected) timed out at 600s. Confirmed by the janitor controller logs from the failing run: 21:04:24.020 applied release taint + managed=false 21:04:29.025 removed release taint and managed label 21:14:29.042 removed cleanup finalizer (Teardown's DeleteAllCRs, 10m later) The reconciler scrubbed the Node within 5 seconds of Complete=True but left the ExtRR alive with its finalizer attached — which is exactly the ADR-040 design (ExtRR is the historical record; the operator decides when to delete it). The envtest spec "removes the release taint and managed=false label; ExtRR stays with finalizer" asserts this same contract. The e2e test contradicted the design by calling WaitForExtRRGone after Complete=True — that wait can only succeed via operator delete, so it always timed out and starved the remaining lifecycle tests (AsymmetricFalse, OperatorDeleteEscape, ForeignTaintDrift) which never ran. Replaces WaitForExtRRGone with a new WaitForNodeReleaseStateCleared helper that polls the Node directly, plus an assertion that the ExtRR itself remains with the cleanup finalizer attached. Same fix applied to TestExtRRAsymmetricFalse's True-retry-after-False step. TestExtRROperatorDeleteEscape's WaitForExtRRGone is unchanged — that path explicitly deletes the ExtRR, so GC is the correct expectation. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Contributor
Author
|
@coderabbitai review |
Contributor
✅ Action performedReview finished.
|
Contributor
There was a problem hiding this comment.
Actionable comments posted: 2
🧹 Nitpick comments (3)
janitor/pkg/metrics/extrr_metrics.go (1)
99-104: ⚡ Quick winAdd a doc comment for exported
ObserveExtRRAge.
ObserveExtRRAgeis exported and should have a leading comment for Go lint/doc consistency.Suggested patch
+// ObserveExtRRAge records the creation-to-close age of an ExtRR by +// recommended action and close result. func (m *ActionMetrics) ObserveExtRRAge(recommendedAction, result string, ageSeconds float64) { ExtRRAgeSeconds.With(prometheus.Labels{ "recommended_action": recommendedAction, "result": result, }).Observe(ageSeconds) }As per coding guidelines, "Include function comments for exported Go functions."
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@janitor/pkg/metrics/extrr_metrics.go` around lines 99 - 104, Add a leading doc comment to the exported method ObserveExtRRAge in the ActionMetrics type. The comment should start with the method name followed by a description of what it does. Follow Go's convention where the comment begins with the receiver and method name, then explains the purpose of observing the ExtRR age metric with the provided recommended action, result, and age in seconds.Source: Coding guidelines
janitor/pkg/controller/externalremediationrequest_setup_test.go (1)
170-174: 🏗️ Heavy liftUse envtest instead of a fake client for this controller mapping test.
mapNodeToExtRRsis controller behavior, and fake-client semantics can diverge from apiserver behavior (especially around indexed list behavior over time). Running this with envtest keeps this path aligned with real runtime behavior.As per coding guidelines, "Use envtest for testing Kubernetes controllers, not fake clients, in Go test code."
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@janitor/pkg/controller/externalremediationrequest_setup_test.go` around lines 170 - 174, Replace the fake client created with fake.NewClientBuilder() in the externalremediationrequest_setup_test.go test with envtest infrastructure. The mapNodeToExtRRs controller behavior needs to be tested against actual apiserver semantics rather than fake client semantics, particularly for indexed list operations. Set up an envtest environment that creates a real API server for this test to ensure the controller mapping logic is validated against runtime-accurate behavior.Source: Coding guidelines
janitor/pkg/controller/externalremediationrequest_controller_test.go (1)
919-962: ⚡ Quick winRefactor
TestExtRRReconciler_NeedsInitializationinto a table-driven test.This block repeats the same setup/assert pattern across scenarios; converting to a single table improves maintainability and makes adding new init cases safer.
As per coding guidelines, "Write table-driven tests when testing multiple scenarios in Go."
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@janitor/pkg/controller/externalremediationrequest_controller_test.go` around lines 919 - 962, Refactor the TestExtRRReconciler_NeedsInitialization test from multiple t.Run blocks into a single table-driven test. Create a slice of test case structs containing the test name, setup function (to configure the extrrObj with Finalizers and Status.Conditions), and the expected result. Then iterate through this table once, executing the same test logic for each case with r.needsInitialization. This eliminates the repeated pattern of creating newTestExtRR objects, setting Finalizers and Status fields, and calling assert.True or assert.False, making it easier to add new initialization scenarios in the future.Source: Coding guidelines
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@janitor/pkg/webhook/v1alpha1/janitor_webhook.go`:
- Around line 502-503: The type `extrrValidator` is declared more than once in
the same scope within the janitor_webhook.go file, causing a compile-time
redeclaration error. Search the entire file for all instances of the
`extrrValidator` type declaration and remove the duplicate definition, keeping
only one single declaration of the `extrrValidator` struct that embeds
`*JanitorCustomValidator`.
In `@tests/external_remediation_test.go`:
- Around line 96-100: The cleanup function in the test block after creating the
ExtRR with CreateExtRRCR only calls client.Resources().Delete(ctx, extrr)
without waiting for the finalizer-driven async deletion to complete. This allows
node taints and labels to leak into subsequent tests if deletion hasn't
finished. Modify the t.Cleanup function to wait deterministically for the ExtRR
to be fully deleted by polling or using a wait mechanism that confirms the CR no
longer exists and any associated node taints or labels have been removed before
cleanup returns.
---
Nitpick comments:
In `@janitor/pkg/controller/externalremediationrequest_controller_test.go`:
- Around line 919-962: Refactor the TestExtRRReconciler_NeedsInitialization test
from multiple t.Run blocks into a single table-driven test. Create a slice of
test case structs containing the test name, setup function (to configure the
extrrObj with Finalizers and Status.Conditions), and the expected result. Then
iterate through this table once, executing the same test logic for each case
with r.needsInitialization. This eliminates the repeated pattern of creating
newTestExtRR objects, setting Finalizers and Status fields, and calling
assert.True or assert.False, making it easier to add new initialization
scenarios in the future.
In `@janitor/pkg/controller/externalremediationrequest_setup_test.go`:
- Around line 170-174: Replace the fake client created with
fake.NewClientBuilder() in the externalremediationrequest_setup_test.go test
with envtest infrastructure. The mapNodeToExtRRs controller behavior needs to be
tested against actual apiserver semantics rather than fake client semantics,
particularly for indexed list operations. Set up an envtest environment that
creates a real API server for this test to ensure the controller mapping logic
is validated against runtime-accurate behavior.
In `@janitor/pkg/metrics/extrr_metrics.go`:
- Around line 99-104: Add a leading doc comment to the exported method
ObserveExtRRAge in the ActionMetrics type. The comment should start with the
method name followed by a description of what it does. Follow Go's convention
where the comment begins with the receiver and method name, then explains the
purpose of observing the ExtRR age metric with the provided recommended action,
result, and age in seconds.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: CHILL
Plan: Enterprise
Run ID: f560c2cc-4116-4a84-b996-9547fa0418a4
📒 Files selected for processing (14)
distros/kubernetes/nvsentinel/charts/janitor/templates/webhook.yamljanitor/main.gojanitor/pkg/condition/condition.gojanitor/pkg/condition/condition_test.gojanitor/pkg/controller/externalremediationrequest_controller.gojanitor/pkg/controller/externalremediationrequest_controller_test.gojanitor/pkg/controller/externalremediationrequest_setup_test.gojanitor/pkg/controller/suite_test.gojanitor/pkg/metrics/extrr_metrics.gojanitor/pkg/metrics/metrics.gojanitor/pkg/webhook/v1alpha1/janitor_webhook.gojanitor/pkg/webhook/v1alpha1/janitor_webhook_test.gotests/external_remediation_test.gotests/helpers/kube.go
✅ Files skipped from review due to trivial changes (1)
- janitor/main.go
🚧 Files skipped from review as they are similar to previous changes (4)
- janitor/pkg/metrics/metrics.go
- janitor/pkg/condition/condition.go
- janitor/pkg/condition/condition_test.go
- janitor/pkg/controller/externalremediationrequest_controller.go
Removes WHAT-restating doc blocks, comments that paraphrase well-named identifiers, and verbose multi-paragraph rationales. Keeps load-bearing WHY notes (drift-safety, asymmetric False semantics, the SSA tradeoff on status patches, the no-Config.Enabled gate on the ExtRR webhook, slice-aliasing invariants on the taint helpers, etc.). Net: ~245 fewer comment lines (392 removed, 147 added in tighter form) across 9 files. No functional changes; envtest + webhook unit suites pass. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Contributor
Merging this branch changes the coverage (3 decrease, 1 increase)
Coverage by fileChanged files (no unit tests)
Please note that the "Total", "Covered", and "Missed" counts above refer to code statements instead of lines of code. The value in brackets refers to the test coverage of that file in the old version of the code. Changed unit test files
|
Locally reproduced the CI failure with a focused envtest harness (creating, snapshotting rv, then doing a stale-rv Update gets the exact "the object has been modified" 409 error, just like CI). The apiserver treats the client-supplied resourceVersion as a precondition on Update; when the reconciler bumps the rv between the test's Get and Update, the request short-circuits with 409 BEFORE admission webhooks run, so the test never sees the webhook's "nodeName cannot be changed" rejection. The sibling lifecycle tests don't hit this because their Setup waits for NVSentinelOwnershipReleased=True (which parks the dispatcher in the no-op branch). Mirror that gate here — once Released=True there are no more reconciler writes to race against, so the fresh Get→Update has a stable rv and admission gets to run. Also tightens the Teardown to mirror the other lifecycle tests: drive finalizer-based cleanup via DeleteAllCRs, plus a Node scrub as belt-and-suspenders if the finalizer path didn't complete. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
natherz97
reviewed
Jun 23, 2026
natherz97
reviewed
Jun 23, 2026
The close-bucket metrics (closed + external_response + ExtRROpen-1 +
age histogram) were gated on whether reconcileCleanup actually mutated
the Node. That made the "Maestro terminates the Node as part of repair
+ signals Complete=True" path silently uncounted — closed{success}
didn't fire, the open gauge leaked, and dashboards under-reported
successful external remediations.
Fix: fire the business metrics on first observation of Complete=True
(branch 4) or first operator-delete (branch 2), regardless of whether
the Node still exists. The Node-PATCH side effect (the
ReleaseTaintRemoved event) stays gated on `changed` — accurate when
there's nothing to scrub.
Implementation:
* New annotation nvsentinel.dgxc.nvidia.com/close-recorded gates
idempotency across reconciles.
* recordClose split into recordCloseMetrics (counter + open-1 + age)
and stampCloseRecorded (annotation patch). Event emission is now
inline in the dispatch branches so its message can be branch-specific.
* stampCloseRecorded patches extrrObj in-place (rather than a
DeepCopy) so the rv writeback flows back to the caller — branch 2's
subsequent finalizer-removal Update would otherwise hit 409 conflict.
* annotationValueTrue constant consolidates the "true" annotation
marker literal across apply-abandoned and close-recorded.
Adds an envtest that simulates the Maestro-terminated-Node flow:
Released=True, Node deleted, Complete=True observed. Asserts both
closed{success} and external_response{success} fire and the
close-recorded annotation is stamped, plus re-reconciles don't
double-count.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Contributor
Merging this branch will decrease overall coverage
Coverage by fileChanged files (no unit tests)
Please note that the "Total", "Covered", and "Missed" counts above refer to code statements instead of lines of code. The value in brackets refers to the test coverage of that file in the old version of the code. Changed unit test files
|
Contributor
Merging this branch changes the coverage (2 decrease, 2 increase)
Coverage by fileChanged files (no unit tests)
Please note that the "Total", "Covered", and "Missed" counts above refer to code statements instead of lines of code. The value in brackets refers to the test coverage of that file in the old version of the code. Changed unit test files
|
natherz97
reviewed
Jun 26, 2026
natherz97
reviewed
Jun 26, 2026
natherz97
reviewed
Jun 26, 2026
natherz97
reviewed
Jun 26, 2026
If the target Node does not exist when reconcileApply runs, transition the
ExtRR to NVSentinelOwnershipReleased=False with reason NodeNotFound
instead of retrying. There is no useful case where the Node would appear
on a later reconcile, and the retry path made the controller logic more
complex without buying observability — the terminal Released=False is
itself the signal.
Drops the apply-abandoned annotation, the applyAbandonAfter threshold,
the nodeMissingRequeue cadence, and the abandonApply helper. The
released{node_not_found} metric label replaces released{apply_abandoned}.
The cleanup path is unaffected: reconcileCleanup already no-ops when the
Node is missing, so an operator delete or external Complete=True signal
still drives the finalizer cleanup cleanly.
Adds a terminal-state timestamp to ExternalRemediationRequestStatus, matching the sibling janitor CRs (RebootNode, TerminateNode, GPUReset). The reconciler will use this in the next commit to short-circuit dispatch on completed ExtRRs and to gate the close-bucket metric emission, replacing the close-recorded annotation. Regenerated: data-models/pkg/protos/external_remediation.pb.go distros/kubernetes/nvsentinel/charts/janitor/crds/external_remediation.crd.yaml
Adopt the sibling janitor controllers' pattern: Status.CompletionTime is
the canonical terminal-state signal, set in every "controller is done"
branch and used by dispatch to short-circuit subsequent reconciles to a
CheckUnlock-only pass.
Changes:
* Add SetCompletionTime() helper on the ExtRR wrapper, matching the
RebootNode/TerminateNode/GPUReset pattern (with *timestamppb.Timestamp
in place of *metav1.Time because the ExtRR status type is proto-generated).
* Add markCompletionTime() reconciler helper that status-subresource patches
Status.CompletionTime if not already set, with the standard
refresh-before-patch + reflect-back idempotency dance.
* Dispatch grows a top-level short-circuit: CompletionTime != nil →
CheckUnlock once, return. Subsumes the steady-state default branch for
completed ExtRRs.
* Replace closeRecordedAnnotation with Status.CompletionTime as the
gate for the close-bucket metrics (closed + external_response + open-1
+ age histogram). The annotation, annotationValueTrue constant, and
stampCloseRecorded helper are removed.
* SetCompletionTime called in transitionToReleaseFailure (NodeNotFound
terminal), reconcileCleanupAfterComplete (Complete=True scrub), and
reconcileCleanupOnDeletion (operator-initiated cleanup).
* The explicit CheckUnlock in reconcileCleanupAfterComplete is removed —
the next reconcile's dispatch short-circuit handles it. The CheckUnlock
in reconcileCleanupOnDeletion stays because the finalizer-drop must
happen on the same reconcile.
Move LockNode from reconcileApply up into dispatch, so it gates every active reconcile loop the same way RebootNode/TerminateNode/GPUReset do. Three top-level paths now: - DeletionTimestamp set: LockNode (requeue on fail) → cleanup-on-deletion - CompletionTime set: CheckUnlock (requeue on retry) → return - Active: LockNode (requeue on fail) → dispatch by condition Also fix CheckUnlock-return-value handling in reconcileCleanupOnDeletion: previously the retry signal was discarded, forcing us to wait for the next controller relist if the lease delete failed transiently. Now we requeue with the lock-retry cadence (per Nathan's review feedback). The functional difference vs the prior narrow-lock design is resilience to a lease being manually deleted mid-flight — the next reconcile re-acquires it instead of running unlocked. Pattern consistency with the sibling janitor controllers is the primary win.
Move LockNode/CheckUnlock out of dispatch and up to the Reconcile entry point, matching the sibling janitor controllers' layout exactly: lock once at the top of every non-init reconcile, unlock once on the completed path. The deletion path still acquires the lock (cleanup-on-deletion mutates the Node), so the lock predicate is "not completed OR being deleted" rather than just "not completed". dispatch becomes pure condition routing — no lock awareness, callers own the lock lifecycle. Resolves Nathan's review comment about ExtRR's lock shape diverging from the sibling controllers.
Three coverage additions matching the sibling controllers' depth of test
for the CompletionTime lifecycle:
* Direct unit tests for the wrapper's SetCompletionTime helper:
allocates Status when nil, stamps on existing Status, idempotent
when CompletionTime is already set.
* Assert Status.CompletionTime is stamped on the Complete=True scrub
path (existing coverage only checked it on NodeNotFound and the
metrics-firing test).
* New spec for the dispatch CompletionTime short-circuit: drives a
CR through apply + Complete=True, then asserts subsequent reconciles
release the lease via CheckUnlock, do not refire the closed{success}
counter, and emit no further events.
Also touch up a stale comment in the "delete before apply ran" test
that still referenced the pre-NodeNotFound retry-forever apply path.
Contributor
Author
|
@coderabbitai review |
Contributor
✅ Action performedReview finished.
|
Contributor
There was a problem hiding this comment.
Actionable comments posted: 5
Caution
Some comments are outside the diff and can’t be posted inline due to platform limitations.
⚠️ Outside diff range comments (2)
distros/kubernetes/nvsentinel/charts/janitor/templates/clusterrole.yaml (1)
152-158: 🔒 Security & Privacy | 🟠 Major | ⚡ Quick winRemove
deletefrom the ExtRR resource verbs.The controller only needs to update status/finalizers and let garbage collection finish deletion after the finalizer is dropped. Leaving
deletehere allows the janitor service account to remove arbitraryexternalremediationrequests, which is broader than necessary.Suggested change
verbs: - get - list - watch - update - patch - - delete🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@distros/kubernetes/nvsentinel/charts/janitor/templates/clusterrole.yaml` around lines 152 - 158, Remove the unnecessary delete verb from the ExtRR permissions in the ClusterRole rules for the janitor controller. Keep only the verbs needed by the controller’s reconciliation flow, especially update and patch for status/finalizers, and leave deletion to garbage collection after finalizers are cleared. Use the ClusterRole rule for externalremediationrequests to locate and adjust the verbs list.janitor/pkg/controller/externalremediationrequest_controller.go (1)
365-382: 🩺 Stability & Availability | 🟠 Major | ⚡ Quick winPersist the terminal node-not-found state atomically.
transitionToReleaseFailurewritesNVSentinelOwnershipReleased=FalseandCompletionTimein two separate patches. If the second patch fails after the first succeeds, the next reconcile no longer enters the failure path,dispatchfalls through its default case, and this ExtRR stays incomplete with the lock still held. Set both fields in the same status write, or explicitly re-terminalizeReleased=False && CompletionTime==nilon the next reconcile.🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@janitor/pkg/controller/externalremediationrequest_controller.go` around lines 365 - 382, The terminal node-not-found handling in transitionToReleaseFailure is split across transitionReleased and markCompletionTime, which can leave the ExtRR partially updated if the second write fails. Update the logic around transitionReleased/markCompletionTime so NVSentinelOwnershipReleased and CompletionTime are persisted atomically in one status update, or make dispatch/transitionReleased explicitly re-enter the failure path when Released is false and CompletionTime is still nil.
🧹 Nitpick comments (1)
janitor/api/v1alpha1/external_remediation_register_test.go (1)
191-237: 📐 Maintainability & Code Quality | 🔵 Trivial | ⚡ Quick winCollapse these
SetCompletionTimecases into a table-driven test.These three tests cover one function across multiple scenarios, so they should follow the repo's table-driven style to make future cases easier to add and maintain. As per coding guidelines, "Write table-driven tests when testing multiple scenarios in Go code."
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@janitor/api/v1alpha1/external_remediation_register_test.go` around lines 191 - 237, The three SetCompletionTime tests should be consolidated into one table-driven test to match the repo’s Go testing style. Refactor TestExternalRemediationRequest_SetCompletionTime_AllocatesStatus, TestExternalRemediationRequest_SetCompletionTime_StampsOnExistingStatus, and TestExternalRemediationRequest_SetCompletionTime_IdempotentWhenAlreadySet into a single subtest loop that covers each scenario for ExternalRemediationRequest.SetCompletionTime. Keep the existing assertions for nil Status allocation, preserving existing Conditions, and not overwriting an already set CompletionTime.Source: Coding guidelines
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In
`@distros/kubernetes/nvsentinel/charts/janitor/crds/external_remediation.crd.yaml`:
- Around line 163-166: The generated external_remediation CRD is missing the
repo-required Apache 2.0 license preamble. Update the CRD generation path for
this YAML so the emitted output includes the standard Apache 2.0 header before
the generated marker, and ensure the generator used for this CRD preserves that
header on future regenerations.
In `@janitor/pkg/controller/externalremediationrequest_controller.go`:
- Around line 678-682: The full-slice merge in patchStatusConditions can clobber
ExternalRemediationComplete=True and leave the object in a stale released state.
Update patchStatusConditions to use a conflict-detecting status update with
retry or SSA-style field ownership instead of MergeFrom on the entire conditions
list, and keep the reconcile path anchored around ExternalRemediationComplete so
concurrent writes are preserved rather than overwritten.
- Around line 424-430: The close-metrics emission in
externalremediationrequest_controller’s cleanup flow is not guarded by the
idempotency fence, so retries can double-count when markCompletionTime fails.
Move the metric calls in the external response and cleanup branches so they
happen only after a successful first-time CompletionTime stamp, and use
CompletionTime as the gate before calling recordCloseMetrics and IncExtRRTotal
in the relevant reconciliation paths.
In `@janitor/pkg/metrics/extrr_metrics.go`:
- Around line 46-51: The ExtRRTotal phase documentation is out of sync with the
labels emitted by `transitionToReleaseFailure`, since `released` now records
`node_not_found` instead of `apply_abandoned`. Update the phase comment in
`extrr_metrics.go` to match the actual metric contract used by `ExtRRTotal` and
`transitionToReleaseFailure`, keeping the `released` outcome list aligned with
the emitted labels.
In `@tests/external_remediation_test.go`:
- Around line 91-94: The cleanup in the ExternalRemediationRequest tests is too
broad because `helpers.DeleteAllCRs()` removes every object of that GVK in the
shared cluster. Update each affected `t.Cleanup` path in
`TestExternalRemediation`, `TestExternalRemediationNodeRecovery`, and the other
listed blocks to delete only the current test’s `crName` (using the existing
`client`/`ctx` flow), and if the test currently relies on teardown completion,
wait with `helpers.WaitForExtRRGone` for that specific object instead of
sweeping all CRs. Keep `helpers.ScrubExtRRStateFromNode` as the node-state
cleanup, but scope the CR deletion to the test-created resource only.
---
Outside diff comments:
In `@distros/kubernetes/nvsentinel/charts/janitor/templates/clusterrole.yaml`:
- Around line 152-158: Remove the unnecessary delete verb from the ExtRR
permissions in the ClusterRole rules for the janitor controller. Keep only the
verbs needed by the controller’s reconciliation flow, especially update and
patch for status/finalizers, and leave deletion to garbage collection after
finalizers are cleared. Use the ClusterRole rule for externalremediationrequests
to locate and adjust the verbs list.
In `@janitor/pkg/controller/externalremediationrequest_controller.go`:
- Around line 365-382: The terminal node-not-found handling in
transitionToReleaseFailure is split across transitionReleased and
markCompletionTime, which can leave the ExtRR partially updated if the second
write fails. Update the logic around transitionReleased/markCompletionTime so
NVSentinelOwnershipReleased and CompletionTime are persisted atomically in one
status update, or make dispatch/transitionReleased explicitly re-enter the
failure path when Released is false and CompletionTime is still nil.
---
Nitpick comments:
In `@janitor/api/v1alpha1/external_remediation_register_test.go`:
- Around line 191-237: The three SetCompletionTime tests should be consolidated
into one table-driven test to match the repo’s Go testing style. Refactor
TestExternalRemediationRequest_SetCompletionTime_AllocatesStatus,
TestExternalRemediationRequest_SetCompletionTime_StampsOnExistingStatus, and
TestExternalRemediationRequest_SetCompletionTime_IdempotentWhenAlreadySet into a
single subtest loop that covers each scenario for
ExternalRemediationRequest.SetCompletionTime. Keep the existing assertions for
nil Status allocation, preserving existing Conditions, and not overwriting an
already set CompletionTime.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: CHILL
Plan: Enterprise
Run ID: 5f2de93c-98a1-4a36-9a35-44190bdc5536
⛔ Files ignored due to path filters (1)
data-models/pkg/protos/external_remediation.pb.gois excluded by!**/*.pb.go
📒 Files selected for processing (15)
data-models/protobufs/external_remediation.protodistros/kubernetes/nvsentinel/charts/janitor/crds/external_remediation.crd.yamldistros/kubernetes/nvsentinel/charts/janitor/templates/clusterrole.yamljanitor/api/v1alpha1/external_remediation_register.gojanitor/api/v1alpha1/external_remediation_register_test.gojanitor/main.gojanitor/pkg/condition/condition.gojanitor/pkg/condition/condition_test.gojanitor/pkg/controller/externalremediationrequest_controller.gojanitor/pkg/controller/externalremediationrequest_controller_test.gojanitor/pkg/metrics/extrr_metrics.gojanitor/pkg/webhook/v1alpha1/janitor_webhook.gojanitor/pkg/webhook/v1alpha1/janitor_webhook_test.gotests/external_remediation_test.gotests/helpers/kube.go
🚧 Files skipped from review as they are similar to previous changes (5)
- janitor/pkg/webhook/v1alpha1/janitor_webhook.go
- janitor/pkg/condition/condition.go
- janitor/pkg/webhook/v1alpha1/janitor_webhook_test.go
- janitor/pkg/condition/condition_test.go
- tests/helpers/kube.go
…ound change Two issues found by CI: 1. markCompletionTime only wrote back Status after the status subresource patch, not ResourceVersion. Any subsequent Update on extrrObj (specifically the finalizer-removal Update in reconcileCleanupOnDeletion) then carried a stale rv and got a 409 Conflict from the apiserver. Fix: write back extrrObj.ResourceVersion from the patched object alongside Status. 2. Two tests — "adds the cleanup finalizer and initial Unknown conditions" and "is idempotent on re-reconcile" — used ExtRRs pointing at non-existent nodes and expected conditions to remain Unknown after 3 reconciles. Before the NodeNotFound-terminal change, missing nodes caused a requeue without condition transitions. Now they immediately set Released=False, breaking the assertions. Fix: create the nodes so the tests exercise the intended steady state (init → apply → Released=True awaiting). The "fresh ExtRR" test also now uses a single reconcile (init pass only) since it is specifically testing initialization, not the full apply.
Replaces apply_abandoned (removed) with node_not_found in the metric contract comment, matching the actual label emitted by transitionToReleaseFailure.
Contributor
Merging this branch will decrease overall coverage
Coverage by fileChanged files (no unit tests)
Please note that the "Total", "Covered", and "Missed" counts above refer to code statements instead of lines of code. The value in brackets refers to the test coverage of that file in the old version of the code. |
natherz97
reviewed
Jun 30, 2026
natherz97
approved these changes
Jun 30, 2026
8 of 18 tasks
Contributor
|
This pull request has been automatically locked since it has been closed for 90 days with no further activity. Please open a new pull request for related changes. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to subscribe to this conversation on GitHub.
Already have an account?
Sign in.
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Second in the ExternalRemediationRequest (ExtRR) series. Builds on #1376 (foundation: CRD + types + RBAC + scheme registration) by adding the controller that drives the node-coordination state machine. Until something starts creating ExtRR objects (a follow-up change wiring fault-remediation), this controller only acts when an operator hand-applies one.
What's included
State machine (6 dispatch branches)
NVSentinelOwnershipReleased=Unknown+ExternalRemediationComplete=Unknownon a fresh ExtRR.managed=falselabel, then transitionNVSentinelOwnershipReleased=True.ExternalRemediationComplete=True, remove the taint + label, then drop the finalizer so the apiserver can garbage-collect the object.ExternalRemediationComplete=Falsedoes NOT close the ExtRR. The node stays released until an operator deletes the object or the external system retries withTrue. This asymmetry is the core of the ADR-040 contract.kubectl delete extrrtriggers the same node cleanup before the finalizer drops, so operators can reclaim stalled nodes.Supporting pieces
janitor/pkg/condition— adapter between the protoConditionmessage andmeta.Conditionso existing controller-runtime helpers (meta.SetStatusCondition,meta.IsStatusConditionTrue) work against ExtRR status.janitor/pkg/metrics/err_metrics.go— three Prometheus series in thenvsentinel_external_remediation_namespace: lifecycle transitions counter, currently-open gauge, age-at-close histogram.janitor/main.go— manager registers the new reconciler alongside the existing RebootNode / TerminateNode / GPUReset controllers.Things deferred to follow-ups
EXTERNAL_REMEDIATIONrecommended action (a separate change).managed=false.Test Plan
🤖 Generated with Claude Code
Summary by CodeRabbit
New Features
Bug Fixes
Tests