Feature Request: --fit should be authoritative: fix fit.cpp failure handling, per-device validation, probe OOM risk, and UMA accounting #22592

Prerequisites

• I am running the latest code. Mention the version if possible as well.
• I carefully followed the README.md.
• I searched using keywords relevant to my issue to make sure that I am creating a new issue that is not already open (or closed).
• I reviewed the Discussions, and have a new and useful enhancement to share.

Feature Description

Summary

--fit currently behaves like a best-effort heuristic rather than an authoritative safety contract.

When a user enables --fit, I would expect the startup path to guarantee one of two outcomes:

1. parameters were fitted to the actual memory constraints that real initialization will use, or
2. startup stops clearly before attempting an unsafe load.

At the moment there are several paths where this contract can break:

• fit can continue from invalid or unknown device-memory information,
• fit failure can be logged while model loading still proceeds,
• fit mutates caller-owned params before success is known,
• fit measures an earlier lifecycle stage than real startup,
• multi-device fitting can accept a global memory target without validating each device after the candidate change,
• the fit probe itself can be the first heavy allocation/OOM point,
• UMA / iGPU systems can be handled with a dGPU-style split host/device accounting model even though both sides draw from the same physical memory pool.

I am opening this as one consolidated issue because these are not isolated user-facing symptoms. They all point to the same implementation-level problem in common/fit.cpp: --fit should only report success after the final candidate configuration has been validated against the real constraints it is supposed to protect.

Expected behavior

With --fit on:

1. device memory must be measured reliably, or fit should fail explicitly;
2. fit failure or internal fit error must be authoritative for the startup path;
3. params should only be committed after fit succeeds;
4. the accepted params should remain valid for the real initialization path, including warmup / first decode memory where relevant;
5. multi-device fitting should validate every device independently, not only summed memory;
6. shared-memory devices should be accounted as shared-memory devices, not as independent host + device buckets;
7. the fit preflight itself should not require a large unsafe probe-load before it can decide that the request is impossible.

Current failure classes

1. 0/0 device-memory fallback can make fit use the wrong budget

In common_get_device_memory_data(...), if a non-host backend reports:

free == 0 && total == 0

fit currently falls back to host memory for that device.

There is already a related closed issue referenced in the code comment:

• related: #18577 / #18587

That fallback was introduced around #18577/#18587 to handle devices that report 0 memory, but the generic fallback still looks wrong as a fit contract.

If a device cannot report memory, that should mean:

• device budget is unknown / invalid for authoritative fit,
• not "use host RAM as the device budget".

This is especially risky because it can make --fit look active while it is no longer fitting to the real device constraint.

2. Fit failure must stop startup, not merely warn

common_fit_params(...) has a structured status:

• success
• failure
• error

But the user-visible behavior shows cases where fit failure is logged and the model load still proceeds.

Example failure shape from a current report:

common_init_result: fitting params to device memory
common_fit_params: failed to fit params to free device memory: llama_params_fit is not implemented for SPLIT_MODE_TENSOR, abort
common_fit_params: fitting params to free memory took 0.00 seconds
llama_model_load_from_file_impl: ...

Related report:

• #22476 shows common_fit_params failing during startup and model loading continuing afterwards.

That means the startup fit path is not acting as a hard safety gate. If fit is enabled and fitting fails, real initialization should not continue unless the user explicitly opted into a best-effort / advisory mode.

3. Fit should be transactional

common/fit.cpp appears to mutate output params while still exploring possible solutions, for example:

• cparams->n_ctx
• mparams->n_gpu_layers
• tensor_split
• tensor_buft_overrides
• mparams.tensor_split
• mparams.tensor_buft_overrides

A failed fit should not leave partially mutated params behind.

Expected contract:

• success: commit fitted params,
• failure/error: leave input params unchanged.

4. Fit measures a pre-warmup lifecycle stage

The fit probe constructs a model/context and uses the memory breakdown from that stage.

Real startup later performs additional work such as warmup / decode / synchronization / backend-specific allocation paths.

So there is a structural mismatch:

• fit approves based on a cheaper earlier stage,
• real startup can still allocate more memory later.

Related failure shapes:

• #18066 reports that llama-fit-params prints a fitting result, but the real llama-cli run later fails allocating KV cache on one CUDA device. It was later partially addressed by #18070, but it still documents the mismatch class.
• #19980 reports another fit/startup lifecycle mismatch where later mmproj GPU memory is not covered by the fit decision.

This may overlap with multi-device validation issues, but the lifecycle mismatch is independently important.

5. Multi-device context fitting uses global memory reduction before per-device validation

In the multi-device context-reduction path, fit computes a global memory target and reduces context based on summed memory.

It also applies a heuristic correction for multi-device granularity, but after choosing the reduced context it can log:

entire model should be fit across devices by reducing context

without re-measuring the final candidate and checking every device against its own target.

For multi-device loading, the real constraint is not only:

sum(used memory) <= sum(free memory)

It must also be:

for every device:
    projected_used_on_this_device <= target_for_this_device

This matters for:

• heterogeneous GPU setups,
• uneven tensor/layer placement,
• MoE models,
• RPC + local GPU setups,
• APU + dGPU setups,
• KV/cache placement imbalance.

6. The fit probe itself can become the first OOM point

--fit is meant to reduce OOM risk, but the current measurement path first performs a probe-load:

mparams_copy.no_alloc  = true;
mparams_copy.use_mmap  = false;
mparams_copy.use_mlock = false;

llama_model_load_from_file(...)
llama_init_from_model(...)

This is still a potentially heavy model/context construction path before fit has decided whether the configuration is safe.

That is risky on:

• CPU-only systems,
• low-RAM systems,
• APUs / UMA systems,
• host-memory-pressure situations,
• large initial-weight cases.

I provided an additional initial-weights OOM log for this failure class.

The important principle is:

--fit should not need to risk OOM first in order to decide how to avoid OOM.

7. Vulkan / ROCm iGPU UMA needs shared-memory accounting, not dGPU-style split accounting

On integrated GPUs / APUs, device memory and host memory are often backed by the same physical RAM pool.

However, the fit path records device buffers and host buffers separately. That can be valid for discrete GPUs, where VRAM and RAM are genuinely separate. It is not a safe model for UMA devices.

Failure shape:

fit checks Vulkan0 / ROCm0 device-side budget
real run also uses Vulkan_Host / ROCm_Host / CPU buffers
both ultimately consume the same physical RAM pool
fit approves because the accounting model treats them as separate buckets

Related context:

• #19104 shows a Vulkan UMA device reporting uma: 1 and very large free Vulkan memory on AMD Strix Halo. This is useful topology context, not a direct --fit OOM report.
• #21801 shows a current ROCm / iGPU-style fit failure report with --fit on.

I provided additional Vulkan iGPU OOM and ROCm iGPU OOM logs for this class.

The key point is:

for UMA devices, fit should validate one shared physical-memory budget,
not independent host + device budgets.

Why I think this should be one issue first

There are multiple symptoms, but the common root is that --fit does not currently behave as an authoritative contract.

Splitting this immediately into many issues makes it easy to fix one symptom while leaving the overall contract broken.

A better target would be:

common/fit.cpp should only report success after final candidate params are validated
against the real device / host / shared-memory constraints that startup will use.

You maintainers can still split this into child issues later, but one master issue gives a clearer view of the common design problem.

Concrete fix checklist

Suggested implementation-level checklist:

• Treat free == 0 && total == 0 device memory as unknown/invalid for fit, not as host-memory fallback.
• Make fit failure/error authoritative for startup when --fit on is requested.
• Make fitting transactional: mutate local copies, commit only on success.
• Revalidate final candidate params before returning success.
• In multi-device mode, validate every device independently after context reduction / placement changes.
• Account for warmup / first decode / backend runtime allocations, or add explicit margins.
• Add UMA-aware accounting for Vulkan / ROCm / integrated GPUs.
• Improve logs so users can see whether fit is authoritative, advisory, failed, or skipped.

Happy to refine this list if needed.

Logs I added so far

Commit // OS // hardware
b97ebdc commit used on Ubuntu 26.04. and AMD gfx1151

I did use a context much higher than "correct" to ensure the undesired behavior takes place.

• Vulkan iGPU (AMD gfx1151) / UMA OOM with --fit on;
  llama-fit-128k-times-10-vulkan_gfx1151.log
  oom_report_vulkan_run_gfx1151.txt
• ROCm iGPU (AMD gfx1151) / UMA OOM with --fit on;
  llama-fit-128k-times-10_rocm.log
• initial-weights / probe-load OOM;
  oom_report_weights_only_vulkan_gfx1151.txt
  weights_only_OON-vulkan.log
• device-memory 0/0 fallback behavior, with reference to #18577 / #18587. Verified as well but lost log. Recreation, dev-container with broken GPU pass-through.

Related issues / reports

• #22476 — common_fit_params logs a fit failure during startup and startup continues into model loading.
• #21801 — current ROCm / iGPU-style --fit on OOM report.
• #18066 / #18070 — llama-fit-params reported a fitted configuration, but real load later OOMed on CUDA KV allocation; partially addressed, still useful for the mismatch class.
• #18577 / #18587 — origin/context for the 0/0 -> host memory fallback; current concern is that this remains unsafe as an authoritative fit rule.
• #19104 — Vulkan UMA / AMD Strix Halo topology context; useful for shared-memory accounting discussion, not a direct --fit OOM report.
• #19980 — later mmproj GPU memory can sit outside the fit decision, another lifecycle-mismatch report.
• #19646 — CPU-only/system-memory accounting report; related to the broader memory-budget contract.

Minimal expected outcome

At minimum, --fit on should not continue as if fitting succeeded when:

• device memory is unknown,
• fitting failed,
• fitting errored,
• final candidate params were not validated,
• or the accounting model does not match the physical memory topology.

If --fit cannot provide that guarantee for a backend or topology, it should fail explicitly instead of presenting the run as fitted.

Motivation

"Fix" the behavior and make llama.cpp even better!

Possible Implementation

I'll provide a draft PR later on - once I'm ready - to try tackle the issue together. I think i have found the fixes already. Sadly a lot of testing will be required.