[Defilan]

What

Four follow-ups from the review of #2185 (#2158), all about staging the model cache on FUSE storage (BlobFuse and the like), where flock does not exclude across nodes:

• Single-file models on FUSE (Model cache: single-file cached models still stage on FUSE under a lock that does not hold #2254). Serving pods no longer stage single-file cached models on a FUSE cache. They still start from a warm cache. Missing or unusable files get the same ModelCacheLockUnsupported refusal and condition as multi-file models. FUSE detection now sits next to cacheDirLock in one prologue (cacheStagingPrologue) that both builders use.
• One prefetch stager per cache directory and claim (Model cache: two prefetch stagers for the same source can write one FUSE cache at once #2250). The Model controller creates a prefetch Job only while no other prefetch Job, or a pod left over from one, writing the same cache directory on the same claim is still active. A Model that has to wait keeps its entry Pending, gets a new PrefetchWaiting=True condition (reason PrefetchWaitingForStager) naming the Job and Model it waits for, and requeues every 15 s.
• One source for the remedy (Model cache: keep the ModelCacheLockUnsupported remedy in one place #2251). Each source kind has one ModelCacheLockUnsupported remedy constant (prefetch, s3://, and local, which is new). The init script prints the remedy for its source. The ModelTransferHealthy condition takes the remedy from the termination message, so the log and the condition cannot disagree.
• Three smaller items (Model cache: smaller follow-ups from #2185 (OnChange FUSE log, stager backfill, temp file sweep) #2255):
  1. On FUSE, an OnChange serving pod that keeps its cached copy now logs Upstream changed for <file>; a serving pod does not stage on a FUSE cache, so it kept the cached copy. Before, it logged an error followed by "Revalidation unreachable".
  2. The FUSE prefetch stager now backfills the .llmkube-size record.
  3. Leftover llmkube_atomic_write temp files are swept.

Fixes #2250
Fixes #2254
Fixes #2251
Fixes #2255

Why

• Model cache: single-file cached models still stage on FUSE under a lock that does not hold #2254: mkcache in the single-file builder took cacheDirLock on any filesystem. On a BlobFuse RWX claim, two pods on different nodes could both hold the lock and curl -C - into the same partial. That is the truncated-file race from [BUG] Concurrent multi-file staging into a shared ReadWriteMany cache can publish a truncated file that later pods accept #2158, and fix: validate multi-file cache staging on shared volumes #2185 had closed it only for multi-file models.
• Model cache: two prefetch stagers for the same source can write one FUSE cache at once #2250: The cache directory is keyed on the source alone, and on FUSE a stager skips flock. Two Models with prefetch: true on the same source and the same claim each got their own Job, and the two Jobs wrote the same partials. Foreground deletion and podReplacementPolicy only order pods within one Job, not across Jobs.
• Model cache: keep the ModelCacheLockUnsupported remedy in one place #2251: The remedy was written twice in different words, once in the script and once in the diagnosis. The pod log and the condition would have drifted the next time prefetch support changed.
• Model cache: smaller follow-ups from #2185 (OnChange FUSE log, stager backfill, temp file sweep) #2255:
  1. The old log read as two failures, and "unreachable" was wrong because the origin had answered.
  2. Before this change, a pre-fix: validate multi-file cache staging on shared volumes #2185 FUSE cache warned "no valid size record" on every start, with no end.
  3. A pod that died between mktemp and mv left its temp file on the claim for good, because the sweeps only delete *.tmp.

How

Shared prologue. cacheStagingPrologue(remedy) detects FUSE (stat -f -c %t, magic 65735546). It defines two shell functions:

• llmkube_fuse_reader: true for a serving pod on FUSE.
• llmkube_stage_allowed <file>: prints the refusal and returns 1 for such a pod.

Off FUSE, it runs mkdir and takes the lock. A stager on FUSE runs only mkdir.

The single-file branches use it this way:

• Every transfer starts with llmkube_stage_allowed.
• The non-transfer writes (rm -f "$MODEL_PATH.tmp", the OnChange debris sweep) are skipped by unlessFUSEReader.
• remoteRevalidateScript takes a cached flag. With the flag set, a FUSE reader keeps a copy that passes the local checks, and otherwise refuses. Uncached callers render byte-identical output.

Single-file stagers. The single-file downloader now carries LLMKUBE_PREFETCH_STAGER=false, as the multi-file one already did. As a result:

• buildPrefetchJob flips that variable to true for single-file Jobs too, so they are stagers.
• They get podReplacementPolicy: Failed.
• deletePrefetchJob deletes them in the foreground.

TestDeletePrefetchJobPropagation has been updated to match. Its old case asserted background deletion for single-file prefetch Jobs. The background case now covers a Job built without the stager switch, which is what an older operator produced.

Serialization (#2250).

• startPrefetchTarget calls activeStager right before Create. activeStager lists the namespace's app.kubernetes.io/component=model-prefetch Jobs and Pods through APIReader (uncached).
• It compares the claim (the model-cache volume) and CACHE_DIR (the model-downloader env) from the pod spec. Jobs built by older operators are therefore recognised too.
• A Job counts as active if it has not finished, is being deleted, or still has active or terminating pods.
• A new ModelReconciler.stagerStartMu covers both the check and the Create. Without it, two of the four workers could both see "no stager" and both create one.
• No new RBAC: jobs and pods list are already granted. A re-verification that has to wait keeps its entry Succeeded, so consumers keep serving.
• I chose serializing over refusing because the second Model then works without any user action. Its stager finds a warm cache and completes.

Remedy (#2251). modelTransferSignature gained remedyFrom. For ModelCacheLockUnsupported, the remedy comes from cacheLockRemedyIn(msg), which looks up whichever known remedy constant appears in the termination message. If the log tail cut the remedy off, it falls back to a pointer at the log. To carry the remedy, findModelTransferFailure and matchModelTransferFailure now return a modelTransferFailure struct.

#2255:

1. The multi-file OnChange branches check llmkube_fuse_reader before they try to stage. fuseKeptCachedEcho distinguishes "unreachable" (no size from the origin) from "upstream changed".
2. The backfill condition is now ! llmkube_fuse_reader on the cache path, so the stager writes the record. The emptyDir path is unchanged.
3. Temp-file changes:
   • llmkube_atomic_write now uses <dest>.llmkube-tmp.XXXXXX and removes its temp file when the write fails.
   • The prologue sweeps *.llmkube-tmp.??????, plus the two unambiguous older names *.llmkube-size.?????? and *.sha256-rejected.??????, once they are more than 10 minutes old (-mmin +10).
   • The age limit exists because a serving pod on FUSE writes stamps without the directory lock, so a younger file may be a write in progress.
   • Only pods that may write the cache run the sweep.
   • Old <file>.sha256.XXXXXX stamp temp files are not swept, because a repository file could match that pattern. The docs say to delete them by hand.

Consumers and rollout impact.

• The init command changes for every cache-backed single-file and multi-file workload, and for uncached multi-file workloads (the new temp-file name in atomicWriteFn). Those pod templates roll once on upgrade. All 20 changed goldens in internal/controller/testdata/ were regenerated and read.
• The #2057 tripwire, TestCachePrepAlwaysTemplatesUnchanged, pins the default single-file Deployment and prefetch Job template hashes. Those pins move in the commits that change the template on purpose: Model cache: single-file cached models still stage on FUSE under a lock that does not hold #2254 (prologue and stager env) and the Model cache: smaller follow-ups from #2185 (OnChange FUSE log, stager backfill, temp file sweep) #2255 sweep. Each move has a reason comment, and docs/MODEL-CACHE.md now has an upgrade note.
• Prefetch Jobs that already exist keep their fingerprint and are not replaced.
• The InferenceService side reads a waiting entry as Pending with the message that names the blocker.
• The test suite needed GC stand-ins. envtest has no garbage collector, so foreground-deleted stagers and Jobs left over from earlier specs (same source, same claim, same namespace) stayed around. I added finishJobDeletion and an AfterEach collectPrefetchJobs to the two prefetch Describes. No assertion was weakened.

Testing. make gate passes on the final commits (regen, drift, fmt, vet, lint-all, full test suite).

• The script behavior is tested by running the scripts the storage builders generate. The containers come from buildModelStorageConfig and buildPrefetchJob, run with their own env and a stub stat -f.
• The shell tests need GNU or BusyBox stat -c and flock, so they skip on macOS. I cross-compiled the test binary and ran every shell-executing test under BusyBox (curlimages/curl:8.18.0, the init image) and under Debian (golang:1.27.3). Both pass.

Mutation checks. For each, I reverted the production change and confirmed the new test fails. Output below is quoted as captured. Some line numbers moved in later commits.

• Model cache: keep the ModelCacheLockUnsupported remedy in one place #2251: I restored the static remedy in modelTransferSignatures. TestCacheLockRemedyLogMatchesCondition/{http,s3} failed:

  model_storage_fuse_test.go:381: the condition's remedy differs from the one the downloader printed:
      log remedy: set spec.modelCache.prefetch: true on every InferenceService using this claim, or use storage with cross-node flock support
      condition:  Model transfer failed: ERROR: ModelCacheLockUnsupported: ... . for HTTP/HF sources, set spec.modelCache.prefetch: true on every service using the claim; s3:// sources cannot prefetch, ...
  

• Model cache: single-file cached models still stage on FUSE under a lock that does not hold #2254: I put back mkcache := mkdir -p "$CACHE_DIR" || exit 1; + cacheDirLock and dropped the llmkube_stage_allowed guard. All 18 FUSE cases of TestSingleFileFUSEStaging and all 7 cases of TestSingleFileFUSEWarmCache failed:

  model_storage_fuse_test.go:412: a serving pod staged on FUSE: Downloading model...
  model_storage_fuse_test.go:412: the FUSE stager must not rely on flock
  model_storage_fuse_test.go:560: a serving pod created a lock file on FUSE
  model_storage_fuse_test.go:551: a serving pod on FUSE issued 1 downloads
  

  I also dropped the single-file LLMKUBE_PREFETCH_STAGER=false env:

  model_storage_fuse_test.go:412: a serving pod staged on FUSE: Downloading model...
  model_storage_fuse_test.go:412: a single-file prefetch stager must wait for a failed pod before replacing it
  

• Model cache: smaller follow-ups from #2185 (OnChange FUSE log, stager backfill, temp file sweep) #2255 item 1: I removed the FUSE-reader branch from the multi-file OnChange refresh. TestMultiFileFUSEWarmCache/OnChange/*/{offline,changed,changed-recorded} failed:

  model_storage_fuse_test.go:173: log does not say "Upstream changed for model.safetensors; a serving pod does not stage on a FUSE cache, so it kept the cached copy":
      ERROR: ModelCacheLockUnsupported: FUSE cache cannot guarantee cross-node file locking for staging model.safetensors; ...
      Revalidation unreachable for model.safetensors; kept cached copy
  model_storage_fuse_test.go:173: the origin answered, but the log calls it unreachable:
  

• Model cache: smaller follow-ups from #2185 (OnChange FUSE log, stager backfill, temp file sweep) #2255 item 2: I restored the cache_fs != FUSE backfill condition. TestMultiFileFUSEStagerBackfillsSizeRecord failed:

  model_storage_fuse_test.go:622: read .../model.safetensors.llmkube-size: open .../model.safetensors.llmkube-size: no such file or directory
  

• Model cache: smaller follow-ups from #2185 (OnChange FUSE log, stager backfill, temp file sweep) #2255 item 3:
  • I removed the sweep from the prologue. TestCacheAtomicWriteDebrisSweep failed: model_storage_fuse_test.go:707: model.gguf.llmkube-tmp.AbC123: removed=false, want true
  • I removed the rm -f on a failed write. TestAtomicWriteRemovesItsTempOnFailure/mv-fails-true failed: model_storage_fuse_test.go:747: temp files left behind: [.../record.llmkube-tmp.BnlcEK]
  • I reverted the temp-file name to $1.XXXXXX. TestAtomicWriteDebrisMatchesTheSweep failed: model_storage_fuse_test.go:788: the sweep left the writer's temp files: [- model.safetensors.sha256.Ldfhhl]
• Model cache: two prefetch stagers for the same source can write one FUSE cache at once #2250:
  • I disabled the guard (if false && blocker != ""). All three serialization specs failed. For example: [FAILED] a second stager on the same cache directory and claim must not be created Expected <bool>: false to be true
  • The same disabled guard against the reported Model cache: two prefetch stagers for the same source can write one FUSE cache at once #2250 scenario (two Models on a multi-file hf://org/repo source with spec.files, sharing one claim): serializes stagers of a multi-file hf:// source shared on a claim failed with [FAILED] a second multi-file stager on the same cache directory and claim must not be created Expected <bool>: false to be true (model_prefetch_serialize_test.go:177)
  • I removed stagerStartMu. Only the concurrency spec failed:

    [FAILED] only one stager may write the cache directory on the claim
    Expected <[]string | len:2, cap:2>: ["concurrent-first-prefetch", "concurrent-second-prefetch"]
    to consist of <[]string | len:1, cap:1>: ["concurrent-first-prefetch"]
    

  • I dropped DeletionTimestamp from prefetchJobActive. waits for a stager Job that is still being deleted failed at model_prefetch_serialize_test.go:172.

Not covered.

• The waiting Model polls every 15 s, because the blocking Job's events reach only its own Model.
• A stager that never finishes (unschedulable, stalled) holds the next one indefinitely. The PrefetchWaiting message names it, and its own Model reports why.
• make test-e2e-rbac was not run: the change adds no new resource, verb or API group.

AI assistance: this change was AI-assisted. I reviewed the diffs and supervised the build.

Checklist

• Tests added/updated
• make test passes locally
• make lint passes locally
• Commit messages follow conventional commits
• All commits are signed off (git commit -s) per DCO
• AI assistance (if any) is disclosed above, per CONTRIBUTING.md
• Documentation updated (if user-facing change)

[codecov]

Codecov Report

❌ Patch coverage is 91.83673% with 12 lines in your changes missing coverage. Please review.

Files with missing lines	Patch %	Lines
internal/controller/model_prefetch.go	85.52%	11 Missing ⚠️
internal/controller/model_transfer_diag.go	94.73%	1 Missing ⚠️

📢 Thoughts on this report? Let us know!