[Defilan]

What

On --default-fsgroup=0 (the OpenShift preset), model-cache-prep now chowns the cache root to the uid the model downloader actually runs as, not to the literal 100:100. That uid is the InferenceService's spec.podSecurityContext.runAsUser when one is set, and otherwise comes from a new operator flag, --init-container-user (chart value controllerManager.initContainer.user, default 101).

fsGroup	Before	After
> 0 (default 102)	chown 0:<fsGroup> /models && chmod g+rwX /models	unchanged
<= 0	chown 100:100 /models && chmod 770 /models	chown <uid>:0 /models && chmod 770 /models

Under --model-cache-prep=never, the prefetch Job's runAsUser also comes from the same flag. Before, it was hardcoded to 101.

Why

Fixes #2204

The default init image (curlimages/curl:8.18.0) runs as uid 101, gid 102. Where nothing applies the pod fsGroup (for example a static PV on a driver that does not apply it), the old 100:100 / 770 root was not writable by the downloader, so it could not create its cache key directory. The operator cannot read an image's user without a registry call, so the uid is now a flag instead of a guess. This is option 1 from the issue.

How

• Effective uid. initContainerSecurityContext already copies a pod-level runAsUser onto the downloader. The new downloaderUID reads that same function, so the prep's owner can't drift from what the downloader runs as. When no runAsUser is set, the flag supplies the uid.
• Group and mode: <uid>:0, 770.
  • Group 0 is the group OpenShift runs SCC-assigned arbitrary uids under. A downloader or serving container whose uid comes from the namespace range can still reach the root through the group bits, without the operator knowing that uid.
  • 770 is the mode the old prep already set. On upgrade, only the owner of an existing root changes, and other users gain no access.
• Validation.
  • ValidateInitContainerUser accepts 1 to 2147483647 (the API server's runAsUser range). The operator refuses to start otherwise.
  • 0 is rejected. Under never, the value is the runAsUser next to runAsNonRoot, and the kubelet refuses uid 0 there. A root downloader also needs no chown of a root-owned mount.
  • The chart schema enforces the same range.
• Wiring. cmd/main.go passes the flag to both reconcilers that render the prep: InferenceServiceReconciler (target and draft storage) and ModelReconciler (prefetch Job). A zero field, as in test literals, resolves to the default.
• Chart. The value is rendered with int64. Helm decodes values-file numbers as float64, so without the cast an OpenShift-range uid rendered as --init-container-user=1.00068e+09. The new chart test caught this.
• Diagnosis. The ModelCacheNotWritable remedy used to name only --model-cache-prep, which on this profile is already in the position it recommends. It now also says that with the prep on and --default-fsgroup=0, the prep chowns the root to --init-container-user (chart controllerManager.initContainer.user) unless spec.podSecurityContext.runAsUser is set, so that flag must be the uid the init image runs as. The diagnosis cannot tell from the pod whether the operator disabled fsGroup (an OpenShift SCC injects one into the pod either way), so it stays one remedy string that covers both causes.
• Unchanged.
  • fsGroup > 0 installs are untouched: the existing cache-prep tripwire hashes for the serving Deployment and the prefetch Job are the same.
  • The golden init commands (internal/controller/testdata/initcmd_*) pin only the downloader command, not the prep, so nothing there changed or needed regenerating.
  • The MetalNode verifier still runs as the constant 101. It writes no cache, and any non-root numeric uid satisfies its runAsNonRoot.
• Docs.
  • docs/MODEL-CACHE.md gets a new section, "What the prep sets the cache root to", with an upgrade note.
  • docs/site/guides/openshift-install.md gets a new section, "The model cache on the preset".
  • The values-openshift.yaml comments, values.yaml, and the chart README row are updated.

Upgrade note

With fsGroup <= 0 and modelCache.prep: always:

• A cache root an older operator prepared as 100:100 mode 770 is re-owned to <uid>:0 mode 770 on the next prep run, meaning the next start of a cache-backed serving pod or prefetch Job.
• The chown is not recursive.
• Cache-backed serving pods on that profile roll once, because the prep command is part of the pod template.
• Anything that relied on uid or group 100 loses access to the root. --init-container-user=100 keeps uid 100 as the owner, with group 0.

Known limits

• In shared cache mode with fsGroup <= 0, two InferenceServices whose downloaders run as different uids (different spec.podSecurityContext.runAsUser) each chown the shared root to their own uid when they start. The root's owner therefore follows whichever pod started last. With mode 770 and group 0, the other service keeps access only through group 0. That holds under OpenShift's arbitrary-uid model, where containers run with gid 0, but not in general. The old 100:100 prep had the same single-owner limitation. This PR does not change it.

Not covered: OpenShift static-PV run

I could not run OpenShift here, so the test-e2e-openshift static-PV case from the issue's acceptance list is not covered. That job still installs with modelCache.enabled=false (MicroShift-in-container has no default storage class), so the prep never runs on an SCC cluster in CI.

Two things would cover it:

1. A test-e2e-openshift variant that creates a static hostPath/local PV on a driver that applies no fsGroup, installs with values-openshift.yaml and the cache enabled, and binds the service account to an SCC that admits the prep (uid 0 with CHOWN/FOWNER added). It would then assert that the root is 101:0 mode 770 and that the InferenceService reaches Ready.
2. Or a manual run of the same steps, recorded on the issue.

The OpenShift guide says that restricted-v2 does not admit the prep (it requires a uid from the namespace range and allows adding only NET_BIND_SERVICE). I took that from the SCC's definition and have not observed it on a cluster.

Mutation checks

Each production edit below was applied alone, the named tests were run, and the edit was reverted. Failure lines are verbatim.

1. The prep goes back to the literal "chown 100:100 /models && chmod 770 /models":

   cache_prep_owner_test.go:90: single-file prep = "chown 100:100 /models && chmod 770 /models", want "chown 1001:0 /models && chmod 770 /models"
   cache_prep_owner_test.go:98: multi-file prep = "chown 100:100 /models && chmod 770 /models", want "chown 2000:0 /models && chmod 770 /models"
   cache_prep_owner_test.go:132: target prep = "chown 100:100 /models && chmod 770 /models", want "chown 1001:0 /models && chmod 770 /models"
   cache_prep_owner_test.go:161: prefetch prep = "chown 100:100 /models && chmod 770 /models", want "chown 1001:0 /models && chmod 770 /models"
   [FAIL] cache prep init container (#855) [It] fsGroup<=0 case: prep chowns to the configured init container user with group 0, mode 770 (#2204)
   

2. The serving target site passes DefaultInitContainerUser instead of the reconciler field:

   cache_prep_owner_test.go:132: target prep = "chown 101:0 /models && chmod 770 /models", want "chown 1001:0 /models && chmod 770 /models"
   cache_prep_owner_test.go:138: draft prep "chown 1001:0 /models && chmod 770 /models" rendered next to the target prep "chown 101:0 /models && chmod 770 /models": the two chown the shared root to different owners
   

3. The draft site does the same:

   cache_prep_owner_test.go:138: draft prep "chown 101:0 /models && chmod 770 /models" rendered next to the target's "chown 1001:0 /models && chmod 770 /models": the draft path chowns the shared root to a different owner
   

   (This failure message was reworded after the run. The assertion is unchanged.)
4. The prefetch Job's storage call does the same:

   cache_prep_owner_test.go:161: prefetch prep = "chown 101:0 /models && chmod 770 /models", want "chown 1001:0 /models && chmod 770 /models"
   

5. The prefetch Job's never security context keeps the default uid:

   cache_prep_owner_test.go:184: container model-downloader runAsUser = 101, want 1001
   cache_prep_owner_test.go:184: container prefetch-done runAsUser = 101, want 1001
   

6. downloaderUID ignores the pod runAsUser:

   cache_prep_owner_test.go:90: single-file prep = "chown 1001:0 /models && chmod 770 /models", want "chown 2000:0 /models && chmod 770 /models"
   cache_prep_owner_test.go:98: multi-file prep = "chown 1001:0 /models && chmod 770 /models", want "chown 2000:0 /models && chmod 770 /models"
   

7. Only the multi-file branch passes the raw flag instead of the effective uid:

   cache_prep_owner_test.go:98: multi-file prep = "chown 1001:0 /models && chmod 770 /models", want "chown 2000:0 /models && chmod 770 /models"
   

8. ValidateInitContainerUser accepts 0:

   cache_prep_owner_test.go:202: ValidateInitContainerUser(0) = nil, want an error
   

9. An unset reconciler field is not defaulted (uid <= 0 changed to uid < 0):

   cache_prep_owner_test.go:132: target prep = "chown 0:0 /models && chmod 770 /models", want "chown 101:0 /models && chmod 770 /models"
   cache_prep_policy_test.go:298: serving downloader uid = 101, prefetch uid = 0: a serving pod could not write into what the prefetch created
   

10. Chart: the int64 cast is dropped:

    - renders a large uid from a values file as an integer
        Expected to contain:
            - --init-container-user=1000680000
        Actual: ... --init-container-user=1.00068e+09
    

11. Chart: the schema minimum is lowered to 0:

    - rejects uid 0
        - asserts[0] `failedTemplate` fail
    

12. Chart: the flag line is removed from the template. Four tests fail: renders the default init image's uid, renders a configured uid for a mirrored init image, renders a large uid from a values file as an integer, and renders the uid next to the OpenShift preset's disabled fsGroup.
13. The ModelCacheNotWritable remedy goes back to naming only --model-cache-prep (the file restored from the first commit, which still had the old wording):

    model_transfer_diag_test.go:249: ModelCacheNotWritable message does not name "--init-container-user": Model transfer failed: mkdir: can't create directory '/models/k': Permission denied (node node-a). the downloader cannot write to the model cache. The
    

    The same test also reports that --default-fsgroup=0 and controllerManager.initContainer.user are missing.

AI assistance: this change was AI-assisted. I reviewed the diffs and supervised the build.

Checklist

• Tests added/updated
• make test passes locally (via make gate)
• make lint passes locally (via make gate, which runs make lint-all)
• Commit messages follow conventional commits
• All commits are signed off (git commit -s) per DCO
• AI assistance (if any) is disclosed above, per CONTRIBUTING.md
• Documentation updated (if user-facing change)

make test-chart passes (24 suites, 263 tests), run with helm-unittest v1.1.1 installed into a scratch HELM_PLUGINS directory.

[codecov]

Codecov Report

❌ Patch coverage is 62.85714% with 13 lines in your changes missing coverage. Please review.

Files with missing lines	Patch %	Lines
cmd/main.go	0.00%	13 Missing ⚠️

📢 Thoughts on this report? Let us know!