[JFMGC-8] Build failover solution for model serving infrastructure

[yakavaliou]

Add optional client-side failover via JOINTFM_BACKUP_DEPLOYMENT_ID (same-checkpoint backup deployment).

On transport or 502/503/504 failures, switch once to the backup for the client lifetime after confirming matching model_version from health.

Note

Medium Risk
Changes the production inference path when the primary deployment is unavailable; safeguards (hosted-only backup, same model_version, sticky switch) limit wrong-routing risk but misconfiguration could still send traffic to an unintended deployment.

Overview
Adds optional hosted failover via JOINTFM_BACKUP_DEPLOYMENT_ID (documented in .env.sample and config.sample.yaml). Settings resolve a second DataRobot deployment’s predict URL, reject backup when using local service or when it matches the primary, and expose JOINTFM_BACKUP_DEPLOYMENT_ID_ENV on the public API.

JointFMClient routes all predict traffic (including forecast sample batching) through _post_predict_json. On transport failures or 502/503/504 from the primary—and only when a backup is configured—it switches predict_url/health_url to the backup for the rest of the client instance, re-probes health, and refuses failover if the backup’s model_version does not match the version already pinned from the primary. Successful primary calls never touch the backup.

Tests cover backup URL loading, invalid backup config, and health/predict failover plus model-version mismatch on backup.

^{Reviewed by Cursor Bugbot for commit 0d026e0. Bugbot is set up for automated code reviews on this repo. Configure here.}

[cursor]

Cursor Bugbot has reviewed your changes using default effort and found 1 potential issue.

^{❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, have a team admin enable autofix in the Cursor dashboard.}

^{Reviewed by Cursor Bugbot for commit 0d026e0. Configure here.}

[cursor]

Sticky failover before verify

High Severity

Failover calls _activate_backup before the backup health probe and _ensure_backup_model_version succeed. On a version mismatch or unreachable backup, _using_backup stays true and URLs already point at the backup, so the client never returns to the primary and later calls keep failing or use the wrong deployment.

Suggested change

	self._ensure_backup_model_version(metadata, pinned_model_version)
	primary_predict_url = self.predict_url
	primary_health_url = self.health_url
	primary_health_metadata = self._health_metadata
	primary_sample_batch_cap = self._sample_batch_cap
	self._activate_backup()
	try:
	metadata = self._probe_health(cache=True)
	self._ensure_backup_model_version(metadata, pinned_model_version)
	except Exception:
	self.predict_url = primary_predict_url
	self.health_url = primary_health_url
	self._using_backup = False
	self._health_metadata = primary_health_metadata
	self._sample_batch_cap = primary_sample_batch_cap
	raise

Additional Locations (1)

• src/jointfm_client/client.py#L177-L180

 

^{Triggered by team rule: suggestion rule}

^{Reviewed by Cursor Bugbot for commit 0d026e0. Configure here.}

[yakavaliou]

As discussed with Stefan, instead of switching to a backup instance when the main instance is unavailable, I’ll try a different approach in my follow-up PR: maintaining a list of instances and distributing the load among them. If some instances are unavailable, we’ll simply log that information.