feat(server): align props, slots, metrics, health, and slot persistence #1440

Parent

• epic(server): close llama-server b10621 compatibility gaps #1431

Problem

mlxcel's /props and /slots responses expose only a small subset of b10621 state. Slot save/restore/erase operations, fail-on-no-slot behavior, detailed generation parameters, and lifecycle/metrics compatibility are missing or incomplete. Health responses also diverge while a model is loading.

Scope

• Expand /props to expose compatible model path/alias, chat template, capabilities, modalities, build metadata, sleep state, and resolved defaults.
• Expand /slots to expose compatible slot state and parameters while redacting secrets and sensitive prompt content where b10621 does.
• Implement slot save, restore, and erase routes with atomic persistence and model/tokenizer compatibility checks.
• Implement fail_on_no_slot behavior and compatible queued/no-slot responses.
• Align metrics names, labels, enablement, and reset/lifecycle behavior for claimed b10621 metrics.
• Align /health and /v1/health status codes/envelopes during loading, ready, error, and shutdown states.
• Align idle-sleep and wake-up lifecycle behavior for --sleep-idle-seconds, including health, metrics, queued requests, and model-resource state.

Acceptance criteria

• Golden-schema tests cover props, idle/active/queued slots, model loading, and health transitions.
• Slot persistence survives restart, rejects mismatched model/tokenizer state, and cannot escape its storage root.
• Saturation tests distinguish queueing from fail-fast behavior.
• Metrics tests validate enable/disable gates and bounded label cardinality.
• Authentication policy for every observability and slot route is covered by integration tests.
• Idle sleep, wake-up, disabled sentinel, active-request suppression, and shutdown-race behavior are covered by lifecycle tests.

Validation

• Run concurrent requests across all configured slots and verify save/restore continuity with a real checkpoint.

Remaining after the first implementation PR

--sleep-idle-seconds is the only surface still deferred: a truthful implementation must tear down and reload the serving worker's model resources on idle, which reaches into every worker loop (batch scheduler, legacy, XLA, diffusion) and interacts with the model-pool lifecycle #1438 introduces. The manifest entry stays deferred against this issue; /props truthfully reports is_sleeping: false. Two recorded divergences also stay open here: slot save/restore persists the token stream rather than KV state, and the params blocks of GET /props / GET /slots omit b10621 keys mlxcel has no analogue for.