Skip to content

DXP-569: check LLM providers when validators are created, not when listing them - #1839

Open
jmlago wants to merge 1 commit into
v0.124-devfrom
legatus/dxp-569
Open

jmlago wants to merge 1 commit into
v0.124-devfrom
legatus/dxp-569

Conversation

@jmlago

@jmlago jmlago commented Nov 19, 2026

Copy link
Copy Markdown
Member

Ticket: DXP-569

What changed

sim_getProvidersAndModels sent a test prompt ("respond with ok") to every stored provider/model each time it was called, with a 20 s timeout per probe. That made the provider list take many seconds to load, and the frontend hung while it loaded. As the ticket suggests, the check now runs when a validator is created instead.

  • Backend, backend/protocol_rpc/endpoints.py
    • get_providers_and_models returns the stored providers and makes no LLM calls. It no longer needs genvm_manager.
    • create_validator and update_validator take genvm_manager. They run check_provider_is_available for the validator's own provider before writing to the DB, so no transaction stays open during the LLM call. The response now includes llm_provider_available: bool.
    • create_random_validator(s) is unchanged: it already filters out unavailable providers when it picks them.
  • Backend, rpc_methods.py: injects genvm_manager into sim_createValidator and sim_updateValidator, and drops it from sim_getProvidersAndModels.
  • Frontend, stores/node.ts: after create, clone or update, if llm_provider_available === false, it shows a warn notification: "LLM provider check failed: Validator saved, but provider/model did not answer a test prompt…".

Assumptions

  • The check warns, it does not block. The validator is saved even if the probe fails. The integration tests (tests/integration/icontracts/tests/test_error_llm.py) create validators with invalid or unknown API keys on purpose and expect sim_createValidator to succeed. Rejecting those would break them, and it would also break anyone who sets up a validator before their key is ready.
  • New response field name: llm_provider_available, which is added to the validator dict that create and update return. Clients that don't know it can ignore it.
  • sim_updateValidator probes on every call, including stake-only edits. This keeps the logic simple, and an update is an explicit user action.
  • sim_replaceValidators (atomic batch replace) is not probed, so bulk setups stay fast.
  • sim_getProvidersAndModels still returns is_available / is_model_available, now always true (as get_all_dict already set them). The frontend fields and the "(potential config issue)" labels stay for compatibility, but they no longer trigger from that listing.
  • backend/protocol_rpc/fastapi_endpoint_generator.py is left untouched. Nothing imports it, and its registrations were already out of sync with the current endpoint signatures.

How it was tested

  • Backend unit tests (Python 3.12 container, same deps as CI): pytest tests/unit -n auto. Result: 1892 passed, 14 skipped. ruff check and black are clean.
    • New: test_get_providers_and_models_does_not_probe_llms, plus test_create_validator_probes_provider_without_blocking and test_update_validator_probes_provider_without_blocking (each parametrised over an available and an unavailable provider; the validator is saved in both cases).
    • Updated: test_simulator_sessions.py and test_rpc_methods.py for the new genvm_manager argument.
  • Frontend (Node 24 container, CI steps): npm run type-check passes. vitest run runs 224 tests, all passed, including 3 new store tests for the warning on create and update. ESLint and Prettier are clean on the changed files.
  • Not run: integration/E2E against a live stack.

🤖 Generated with Claude Code

…hen listing

sim_getProvidersAndModels sent a test prompt to every provider/model on each
call, so the provider list took many seconds to load and the frontend hung.
It now returns the stored providers without any LLM call.

sim_createValidator and sim_updateValidator now probe the validator's own
provider (before touching the DB) and return `llm_provider_available`. The
validator is saved either way, so validators with broken keys can still be
created on purpose; the frontend shows a warning when the probe fails.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@coderabbitai

coderabbitai Bot commented Nov 19, 2026

Copy link
Copy Markdown
Contributor

Important

Review skipped

Auto reviews are disabled on base/target branches other than the default branch.

Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

Run configuration

  • Source mode: Auto
  • Repository: Unavailable
  • Branch: Unavailable
  • File: No configuration file from this repository applied
  • Commit: Unavailable
  • Layers: defaults
  • Review profile: CHILL
  • Plan: Advanced
  • Run ID: 3f07cb3d-c498-4636-b34b-0c8587e0b7d4

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

  • Trigger review
  • Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

Share

@danieljrc888

Copy link
Copy Markdown
Contributor

Reviewed at 6f85e7b. The change is sound: listing providers no longer sends a paid test prompt to every provider on each call, the startup path (validators_init) is untouched, and create/update stay behind the hosted guard. No blocking defects. Follow-ups, preferably in this PR:

  1. backend/database_handler/llm_providers.py:90-91 now hardcodes is_available and is_model_available to True, but the frontend (ValidatorModal.vue, ProviderItem.vue, ProviderModal.vue, stores/node.ts) and genlayer-cli (src/commands/localnet/validators.ts) still branch on them, so a provider with a missing key looks healthy until after the validator is saved, and the CLI never warns. Either compute a cheap value with no network call (is the configured key variable set and non-empty) or remove the dead branches and the fields.
  2. update_validator runs the provider check before confirming the validator exists (endpoints.py, the check precedes registry.update_validator): a wrong address costs one paid LLM call and up to 20 seconds before the not-found error. Look the validator up first, and skip the check when the provider did not change (stake-only edits).
  3. Add tests for clone and for rpc_methods.create_validator forwarding genvm_manager.

@jmlago

jmlago commented Nov 19, 2026

Copy link
Copy Markdown
Member Author

Legatus audit: Approved (commit 6f85e7b)

No blocking findings. The change removes LLM probes from provider listing and checks the selected provider during validator creation/update, warning on failure.

Verified against origin/v0.124-dev:

  • Backend unit tests: 1,887 passed, 14 skipped.
  • Frontend tests: 225 passed.
  • Main CI test remains running; advisory E2E is requested and queued.

Audit worktree removed. No changes pushed or GitHub comments posted.

@jmlago

jmlago commented Nov 19, 2026

Copy link
Copy Markdown
Member Author

/run-e2e all

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants