[JFMGC-8] Build failover solution for model serving infrastructure

[yakavaliou]

How it works:

1. You configure several JointFM deployments that serve the same model (e.g. Chevron + research) in one list: JOINTFM_DEPLOYMENT_IDS.
2. On startup / first forecast, the client checks health on each of those deployments.
   • If one is unreachable → log it and continue with the healthy ones.
   • If healthy ones don’t match on model/checkpoint → stop (don’t mix checkpoints for the day’s data).
3. For each forecast/predict request, the client picks the next deployment in turn (round-robin) and calls it.
4. If that call fails in a “deployment is down / warming / gateway” way (e.g. network error, 470, 502/503/504):
   • Log which deployment failed
   • Try the next deployment ID in the list
   • Do not permanently switch to a dedicated backup — just skip the bad one for that request and keep using the pool
5. If all configured deployments fail → the request fails (same as today with one dead deployment).

---

Note

Medium Risk
Changes how hosted predictions are routed and retried in production; mis-pooled deployments are guarded by checkpoint checks, but behavior differs from single-deployment clients and deserves rollout validation.

Overview
Adds hosted multi-deployment failover via a new JOINTFM_DEPLOYMENT_IDS selector (comma-separated, ≥2 unique IDs, same checkpoint; exclusive with other deployment selectors). Settings build an instances pool on JointFMSettings; YAML/env samples and API docs document the option.

Introduces JointFMInstancePool for round-robin predict/forecast traffic, per-request retry across untried peers on transient errors (network/JointFMRequestError and configured retry HTTP codes such as 470/5xx), and probe_all_health that tolerates unreachable peers but fails on model_version/checkpoint_version mismatch and sets sample batching cap to the minimum max_sample_count across healthy peers.

JointFMClient routes health, predict, and sample-batched forecasts through the pool when multiple instances are configured, uses fail-fast transport per peer (max_attempts=1), and gates pool traffic on pooled health before resolving model version. Public exports include JointFMInstancePool and JointFMInstanceSettings.

^{Reviewed by Cursor Bugbot for commit c3ba633. Bugbot is set up for automated code reviews on this repo. Configure here.}

[cursor]

Cursor Bugbot has reviewed your changes using default effort and found 1 potential issue.

^{❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, have a team admin enable autofix in the Cursor dashboard.}

^{Reviewed by Cursor Bugbot for commit c3ba633. Configure here.}

[cursor]

Pool failover breaks sample batching

Medium Severity

post_json treats every 470 as a peer outage and keeps failover, overwriting last_error on each attempt. Oversized sample requests surface as 470 with INPUT_SIZE_EXCEEDED, so a later network failure can replace that capacity error. forecast only special-cases JointFMHTTPStatusError, so the batching fallback never sees the cap and the request fails instead of batching.

Additional Locations (1)

• src/jointfm_client/client.py#L320-L328

 

^{Reviewed by Cursor Bugbot for commit c3ba633. Configure here.}

[shackmann]

This is an awesome contribution Yahor, thanks! Could we maybe make one small conceptual change?

Say I request 10K samples but the model has a capacity of 5K, then we were requesting 5K and then again 5K from the same instance. With several models, we not only have failover but we can speed up operations. Say we have 2 instances that can generate 5K samples, we can now run the requests in parallel, using both instances at the same time. Would that be possible?

[mariusvilkas]

I am not very familiar with the jointfm-client, but code does make sense to me.
Is there a way to call client to just get a status on the pool instances - like instance is offline or not reachable or something like that. Otherwise,
LGTM