Skip to content

fix: allow arbitrary model names for all providers - #1559

Merged
MuncleUscles merged 3 commits into
mainfrom
fix/open-model-registration
Mar 27, 2026
Merged

MuncleUscles merged 3 commits into
mainfrom
fix/open-model-registration

Conversation

@MuncleUscles

@MuncleUscles MuncleUscles commented Mar 27, 2026 •

Copy link
Copy Markdown
Member

Summary

  • Remove hardcoded model enum constraints from providers_schema.json (both backend and frontend copies). Plugin constraints (const) are preserved — they correctly enforce the right plugin per provider.
  • Add fallback template logic in get_default_provider_for(): when no exact (provider, model) match exists in default_providers/, it clones the plugin/config from any existing default for that provider and substitutes the requested model.

Why

Hardcoded model enums prevent adding new models (e.g. nvidia/nemotron-3-super-120b-a12b:free on openrouter) without a Studio release. Providers regularly add and deprecate models — the schema shouldn't gate which model strings are valid. The fallback template approach lets any model that works with the provider's API be registered immediately.

Files changed

  • backend/node/create_nodes/providers_schema.json — removed model enums from all provider then blocks
  • frontend/src/assets/schemas/providers_schema.json — same (kept in sync)
  • backend/node/create_nodes/providers.py — added fallback provider template in get_default_provider_for()

Test plan

  • Existing default providers still load and validate correctly (no schema changes to plugin constraints or config shapes)
  • Adding a validator with a non-default model string for an existing provider (e.g. openrouter) succeeds and uses the correct plugin/API config
  • Adding a validator with a model for a provider that has no defaults still raises ValueError
  • Frontend validator config modal accepts arbitrary model strings

Summary by CodeRabbit

  • Refactor

    • Provider selection now falls back to an existing provider template and reuses its configuration when no exact default match is found.
  • Bug Fixes

    • Removed restrictive model enumeration checks across multiple provider integrations, allowing broader model choices while keeping plugin-level validation.
  • Chores

    • CI workflows updated to use an OpenRouter-based provider configuration for integration and load tests.
  • Tests

    • Integration tests made configurable via environment variables to target different providers/models.

@coderabbitai

coderabbitai Bot commented Mar 27, 2026 •

Copy link
Copy Markdown
Contributor
📝 Walkthrough

Walkthrough

When no exact default provider/model is found, get_default_provider_for() falls back to cloning any provider with the same provider value and overriding only model. Provider-specific model enum validations were removed in backend/frontend schemas. CI workflows and integration tests were updated to use OpenRouter env vars and workflow vars for provider/model.

Changes

Cohort / File(s) Summary
Provider fallback logic
backend/node/create_nodes/providers.py
Add fallback in get_default_provider_for() to find any entry matching provider, clone its config and plugin_config (shallow copy of dicts when present), and override only model when no exact default exists; existing exact-match and multiple-defaults behavior unchanged.
Backend schema constraints
backend/node/create_nodes/providers_schema.json
Remove properties.model.enum constraints from multiple conditional allOf branches (provider→plugin branches); keep plugin discriminators and other required fields.
Frontend schema constraints
frontend/src/assets/schemas/providers_schema.json
Delete provider-specific model enum validations in conditional branches for the same provider/plugin combinations, leaving model less strictly validated while preserving plugin const checks.
CI workflow env changes
.github/workflows/backend_integration_tests_pr.yml, .github/workflows/load-test-oha.yml
Switch .env and test setup from OpenAI key to OpenRouter variables (OPENROUTERAPIKEY, LLM_PROVIDER, LLM_MODEL); update VALIDATORS_CONFIG_JSON; test steps set TEST_PROVIDER/TEST_PROVIDER_MODEL from workflow vars.
Integration test adjustments
tests/integration/test_llm_providers_registry.py
Make test provider/model configurable via TEST_PROVIDER/TEST_PROVIDER_MODEL env vars; derive API key env var per provider; use dynamic provider/model values throughout provider create/update/delete flow.

Estimated code review effort

🎯 4 (Complex) | ⏱️ ~45 minutes

Possibly related PRs

Suggested reviewers

  • cristiam86

Poem

🐇 I nibble configs in a moonlit byte,
I copy a provider and set its light,
Schemas loosened so models can roam,
I hop through workflows, make the tests feel home —
Hooray, a rabbit cheers for flexible code!

🚥 Pre-merge checks | ✅ 2 | ❌ 1

❌ Failed checks (1 warning)

Check name Status Explanation Resolution
Docstring Coverage ⚠️ Warning Docstring coverage is 0.00% which is insufficient. The required threshold is 80.00%. Write docstrings for the functions missing them to satisfy the coverage threshold.
✅ Passed checks (2 passed)
Check name Status Explanation
Title check ✅ Passed The title clearly and concisely describes the main change: allowing arbitrary model names for all providers instead of hardcoded enum constraints.
Description check ✅ Passed The PR description covers the main changes, rationale, affected files, and a test plan; however, some template sections like 'Decisions made', 'Checking tips', and 'User facing release notes' are incomplete or missing.

✏️ Tip: You can configure your own custom pre-merge checks in the settings.

✨ Finishing Touches
📝 Generate docstrings
  • Create stacked PR
  • Commit on current branch
🧪 Generate unit tests (beta)
  • Create PR with unit tests
  • Commit unit tests in branch fix/open-model-registration

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands and usage tips.

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 1

Caution

Some comments are outside the diff and can’t be posted inline due to platform limitations.

⚠️ Outside diff range comments (1)
backend/node/create_nodes/providers.py (1)

99-123: ⚠️ Potential issue | 🟠 Major

Return a cloned instance instead of the cached object to prevent state leaks.

The exact-match path (line 123) returns the cached instance directly, while the fallback creates a new one. Callers mutate the returned object—specifically at backend/protocol_rpc/endpoints.py:436 and backend/protocol_rpc/validators_init.py:129—which corrupts the shared cache.

Extract the cloning logic into a helper function and apply it to both paths:

Suggested fix
+import copy
@@
 def get_default_provider_for(provider: str, model: str) -> LLMProvider:
+    def _clone(template: LLMProvider, selected_model: str) -> LLMProvider:
+        return LLMProvider(
+            provider=template.provider,
+            model=selected_model,
+            config=copy.deepcopy(template.config),
+            plugin=template.plugin,
+            plugin_config=copy.deepcopy(template.plugin_config),
+        )
+
     llm_providers = get_default_providers()
@@
         if provider_templates:
             template = provider_templates[0]
-            return LLMProvider(
-                provider=provider,
-                model=model,
-                config=template.config.copy() if template.config else {},
-                plugin=template.plugin,
-                plugin_config=(
-                    template.plugin_config.copy() if template.plugin_config else {}
-                ),
-            )
+            return _clone(template, model)
@@
-    return matches[0]
+    return _clone(matches[0], model)
🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed.

In `@backend/node/create_nodes/providers.py` around lines 99 - 123, The
get_default_provider_for function currently returns a cached LLMProvider
instance for exact matches which allows callers (e.g., code paths at
backend/protocol_rpc/endpoints.py:436 and
backend/protocol_rpc/validators_init.py:129) to mutate shared state; extract the
cloning logic used in the fallback into a small helper (e.g.,
clone_llm_provider(template: LLMProvider) -> LLMProvider) that returns a new
LLMProvider with copied config and plugin_config, and use that helper both when
returning the fallback template and when returning the single exact match
(replace returning matches[0] with returning clone_llm_provider(matches[0])) so
callers always receive a fresh instance.
🤖 Prompt for all review comments with AI agents
Verify each finding against the current code and only fix it if needed.

Inline comments:
In `@backend/node/create_nodes/providers.py`:
- Around line 107-110: The fallback selects provider_templates[0] from
llm_providers which is nondeterministic; make selection deterministic by sorting
the filtered list (provider_templates) by the model name before picking the
first element. Update the logic around provider_templates / template (the
filtered list created from llm_providers where lp.provider == provider) to sort
by the model attribute (e.g., lp.model or the appropriate model field) and then
assign template = provider_templates[0] from the sorted list.

---

Outside diff comments:
In `@backend/node/create_nodes/providers.py`:
- Around line 99-123: The get_default_provider_for function currently returns a
cached LLMProvider instance for exact matches which allows callers (e.g., code
paths at backend/protocol_rpc/endpoints.py:436 and
backend/protocol_rpc/validators_init.py:129) to mutate shared state; extract the
cloning logic used in the fallback into a small helper (e.g.,
clone_llm_provider(template: LLMProvider) -> LLMProvider) that returns a new
LLMProvider with copied config and plugin_config, and use that helper both when
returning the fallback template and when returning the single exact match
(replace returning matches[0] with returning clone_llm_provider(matches[0])) so
callers always receive a fresh instance.
🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

  • Push a commit to this branch (recommended)
  • Create a new PR with the fixes

ℹ️ Review info
⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Pro

Run ID: 698ad116-b56b-4040-af2f-e1878aeab4c7

📥 Commits

Reviewing files that changed from the base of the PR and between fcbf0fa and 231efc9.

📒 Files selected for processing (3)
  • backend/node/create_nodes/providers.py
  • backend/node/create_nodes/providers_schema.json
  • frontend/src/assets/schemas/providers_schema.json
💤 Files with no reviewable changes (2)
  • backend/node/create_nodes/providers_schema.json
  • frontend/src/assets/schemas/providers_schema.json

Comment on lines +107 to +110
# Fallback: use any existing provider config as template for the same provider
provider_templates = [lp for lp in llm_providers if lp.provider == provider]
if provider_templates:
template = provider_templates[0]

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Potential issue | 🟠 Major

🧩 Analysis chain

🏁 Script executed:

#!/bin/bash
# Verify whether providers have multiple default templates with divergent config/plugin_config.
python - <<'PY'
import json, pathlib
from collections import defaultdict

base = pathlib.Path("backend/node/create_nodes/default_providers")
rows = defaultdict(list)

for f in sorted(base.glob("*.json")):
    data = json.loads(f.read_text())
    rows[data["provider"]].append((f.name, data.get("model"), data.get("config"), data.get("plugin_config")))

for provider, items in rows.items():
    if len(items) < 2:
        continue
    signatures = {
        (json.dumps(cfg, sort_keys=True), json.dumps(pcfg, sort_keys=True))
        for _, _, cfg, pcfg in items
    }
    if len(signatures) > 1:
        print(f"{provider}: {len(items)} templates with differing config/plugin_config")
        for name, model, _, _ in items:
            print(f"  - {name} (model={model})")
PY

Repository: genlayerlabs/genlayer-studio

Length of output: 586


🏁 Script executed:

cat -n backend/node/create_nodes/providers.py | head -150 | tail -100

Repository: genlayerlabs/genlayer-studio

Length of output: 4479


Make fallback template selection deterministic.

Line 110 picks provider_templates[0], but llm_providers is derived from unsorted defaults. This causes the fallback to select different templates across runs when multiple defaults exist for the same provider. The verification shows 10 OpenAI templates with diverging configurations, confirming this is a real scenario.

Sort the filtered templates by model name to ensure consistent selection:

Suggested fix
-    provider_templates = [lp for lp in llm_providers if lp.provider == provider]
+    provider_templates = sorted(
+        (lp for lp in llm_providers if lp.provider == provider),
+        key=lambda lp: lp.model,
+    )
     if provider_templates:
         template = provider_templates[0]
🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed.

In `@backend/node/create_nodes/providers.py` around lines 107 - 110, The fallback
selects provider_templates[0] from llm_providers which is nondeterministic; make
selection deterministic by sorting the filtered list (provider_templates) by the
model name before picking the first element. Update the logic around
provider_templates / template (the filtered list created from llm_providers
where lp.provider == provider) to sort by the model attribute (e.g., lp.model or
the appropriate model field) and then assign template = provider_templates[0]
from the sorted list.

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 2

🤖 Prompt for all review comments with AI agents
Verify each finding against the current code and only fix it if needed.

Inline comments:
In @.github/workflows/backend_integration_tests_pr.yml:
- Around line 54-60: The current printf lines that append OPENROUTERAPIKEY and
VALIDATORS_CONFIG_JSON interpolate shell variables directly (OPENROUTERAPIKEY,
LLM_PROVIDER, LLM_MODEL) which can corrupt values with shell metacharacters;
change the appending to use safe quoting or a heredoc so variables are written
verbatim (e.g., use a quoted printf format with the variable as a single
argument or use a <<'EOF' heredoc when writing the multi-line
VALIDATORS_CONFIG_JSON) to avoid word-splitting and expansion and ensure the
exact values for OPENROUTERAPIKEY and the VALIDATORS_CONFIG_JSON payload
(including LLM_PROVIDER and LLM_MODEL) are preserved.

In @.github/workflows/load-test-oha.yml:
- Around line 74-81: The printf lines that append OPENROUTERAPIKEY and
VALIDATORS_CONFIG_JSON interpolate shell variables (OPENROUTERAPIKEY,
LLM_PROVIDER, LLM_MODEL) unsafely; replace them with a safe, literal heredoc or
a printf with %s and proper quoting so values are not treated as shell/meta
characters — e.g., use a single-quoted heredoc (or printf '%s\n' with escaped
content) to append OPENROUTERAPIKEY and VALIDATORS_CONFIG_JSON to .env, keeping
the sed removal lines intact and ensuring the variables are expanded exactly
once into the heredoc body.
🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

  • Push a commit to this branch (recommended)
  • Create a new PR with the fixes

ℹ️ Review info
⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Pro

Run ID: 6873089c-603f-44e1-a66c-567cf519733f

📥 Commits

Reviewing files that changed from the base of the PR and between 231efc9 and 909928c.

📒 Files selected for processing (2)
  • .github/workflows/backend_integration_tests_pr.yml
  • .github/workflows/load-test-oha.yml

Comment on lines +54 to +60
# Inject OpenRouter API key
sed -i "/^OPENROUTERAPIKEY *= *'<add_your_openrouter_api_key_here>'$/d" .env
printf "\nOPENROUTERAPIKEY='$OPENROUTERAPIKEY'\n" >> .env

# Override validators config with configured provider/model
sed -i "/^VALIDATORS_CONFIG_JSON=/,/^]'/d" .env
printf "\nVALIDATORS_CONFIG_JSON='[\n {\"stake\": 100, \"provider\": \"$LLM_PROVIDER\", \"model\": \"$LLM_MODEL\", \"amount\": 5}\n]'\n" >> .env

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Potential issue | 🟡 Minor

Shell variable expansion may corrupt values containing special characters.

Directly interpolating $OPENROUTERAPIKEY in printf without proper escaping is unsafe. API keys commonly contain shell metacharacters ($, `, \, !) that will be interpreted by bash, potentially truncating or corrupting the value.

Similarly, $LLM_MODEL values like deepseek/deepseek-v3.2 are safe, but future models with special chars (e.g., @preset/...) could break.

🛡️ Suggested fix using heredoc or proper quoting
-          # Inject OpenRouter API key
-          sed -i "/^OPENROUTERAPIKEY *= *'<add_your_openrouter_api_key_here>'$/d" .env
-          printf "\nOPENROUTERAPIKEY='$OPENROUTERAPIKEY'\n" >> .env
-
-          # Override validators config with configured provider/model
-          sed -i "/^VALIDATORS_CONFIG_JSON=/,/^]'/d" .env
-          printf "\nVALIDATORS_CONFIG_JSON='[\n  {\"stake\": 100, \"provider\": \"$LLM_PROVIDER\", \"model\": \"$LLM_MODEL\", \"amount\": 5}\n]'\n" >> .env
+          # Inject OpenRouter API key (use printf %s to avoid interpretation)
+          sed -i "/^OPENROUTERAPIKEY=/d" .env
+          printf '\nOPENROUTERAPIKEY=%s\n' "'${OPENROUTERAPIKEY}'" >> .env
+
+          # Override validators config with configured provider/model
+          sed -i "/^VALIDATORS_CONFIG_JSON=/,/^]'/d" .env
+          cat >> .env <<EOF
+
+VALIDATORS_CONFIG_JSON='[
+  {"stake": 100, "provider": "${LLM_PROVIDER}", "model": "${LLM_MODEL}", "amount": 5}
+]'
+EOF
🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed.

In @.github/workflows/backend_integration_tests_pr.yml around lines 54 - 60, The
current printf lines that append OPENROUTERAPIKEY and VALIDATORS_CONFIG_JSON
interpolate shell variables directly (OPENROUTERAPIKEY, LLM_PROVIDER, LLM_MODEL)
which can corrupt values with shell metacharacters; change the appending to use
safe quoting or a heredoc so variables are written verbatim (e.g., use a quoted
printf format with the variable as a single argument or use a <<'EOF' heredoc
when writing the multi-line VALIDATORS_CONFIG_JSON) to avoid word-splitting and
expansion and ensure the exact values for OPENROUTERAPIKEY and the
VALIDATORS_CONFIG_JSON payload (including LLM_PROVIDER and LLM_MODEL) are
preserved.

Comment on lines +74 to +81
run: |
# Inject OpenRouter API key
sed -i "/^OPENROUTERAPIKEY *= *'<add_your_openrouter_api_key_here>'$/d" .env
printf "\nOPENROUTERAPIKEY='$OPENROUTERAPIKEY'\n" >> .env

# Override validators config with configured provider/model
sed -i "/^VALIDATORS_CONFIG_JSON=/,/^]'/d" .env
printf "\nVALIDATORS_CONFIG_JSON='[\n {\"stake\": 100, \"provider\": \"$LLM_PROVIDER\", \"model\": \"$LLM_MODEL\", \"amount\": 5}\n]'\n" >> .env

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Potential issue | 🟡 Minor

Same shell expansion concern as in backend_integration_tests_pr.yml.

The printf statements directly interpolate shell variables, which risks corruption if the API key or model names contain shell metacharacters. Consider applying the same heredoc/quoting approach suggested for the other workflow file.

🛡️ Suggested fix
        run: |
-          # Inject OpenRouter API key
-          sed -i "/^OPENROUTERAPIKEY *= *'<add_your_openrouter_api_key_here>'$/d" .env
-          printf "\nOPENROUTERAPIKEY='$OPENROUTERAPIKEY'\n" >> .env
-
-          # Override validators config with configured provider/model
-          sed -i "/^VALIDATORS_CONFIG_JSON=/,/^]'/d" .env
-          printf "\nVALIDATORS_CONFIG_JSON='[\n  {\"stake\": 100, \"provider\": \"$LLM_PROVIDER\", \"model\": \"$LLM_MODEL\", \"amount\": 5}\n]'\n" >> .env
+          # Inject OpenRouter API key
+          sed -i "/^OPENROUTERAPIKEY=/d" .env
+          printf '\nOPENROUTERAPIKEY=%s\n' "'${OPENROUTERAPIKEY}'" >> .env
+
+          # Override validators config with configured provider/model
+          sed -i "/^VALIDATORS_CONFIG_JSON=/,/^]'/d" .env
+          cat >> .env <<EOF
+
+VALIDATORS_CONFIG_JSON='[
+  {"stake": 100, "provider": "${LLM_PROVIDER}", "model": "${LLM_MODEL}", "amount": 5}
+]'
+EOF
📝 Committable suggestion

‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.

Suggested change
run: |
# Inject OpenRouter API key
sed -i "/^OPENROUTERAPIKEY *= *'<add_your_openrouter_api_key_here>'$/d" .env
printf "\nOPENROUTERAPIKEY='$OPENROUTERAPIKEY'\n" >> .env
# Override validators config with configured provider/model
sed -i "/^VALIDATORS_CONFIG_JSON=/,/^]'/d" .env
printf "\nVALIDATORS_CONFIG_JSON='[\n {\"stake\": 100, \"provider\": \"$LLM_PROVIDER\", \"model\": \"$LLM_MODEL\", \"amount\": 5}\n]'\n" >> .env
run: |
# Inject OpenRouter API key
sed -i "/^OPENROUTERAPIKEY=/d" .env
printf '\nOPENROUTERAPIKEY=%s\n' "'${OPENROUTERAPIKEY}'" >> .env
# Override validators config with configured provider/model
sed -i "/^VALIDATORS_CONFIG_JSON=/,/^]'/d" .env
cat >> .env <<EOF
VALIDATORS_CONFIG_JSON='[
{"stake": 100, "provider": "${LLM_PROVIDER}", "model": "${LLM_MODEL}", "amount": 5}
]'
EOF
🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed.

In @.github/workflows/load-test-oha.yml around lines 74 - 81, The printf lines
that append OPENROUTERAPIKEY and VALIDATORS_CONFIG_JSON interpolate shell
variables (OPENROUTERAPIKEY, LLM_PROVIDER, LLM_MODEL) unsafely; replace them
with a safe, literal heredoc or a printf with %s and proper quoting so values
are not treated as shell/meta characters — e.g., use a single-quoted heredoc (or
printf '%s\n' with escaped content) to append OPENROUTERAPIKEY and
VALIDATORS_CONFIG_JSON to .env, keeping the sed removal lines intact and
ensuring the variables are expanded exactly once into the heredoc body.

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 3

🤖 Prompt for all review comments with AI agents
Verify each finding against the current code and only fix it if needed.

Inline comments:
In `@tests/integration/test_llm_providers_registry.py`:
- Around line 9-16: The current lookup silently defaults unknown provider_name
to "OPENAIKEY"; update the logic around API_KEY_MAP and api_key_env_var to fail
fast: after defining API_KEY_MAP and before using api_key_env_var, check if
provider_name is a key in API_KEY_MAP and if not raise a clear exception (or
call pytest.skip/pytest.fail as appropriate) indicating unsupported
TEST_PROVIDER; use the symbols API_KEY_MAP, provider_name, and api_key_env_var
to locate and replace the defaulting behavior so unknown providers no longer
silently map to OPENAIKEY.
- Around line 25-39: The test finds target_provider_id via next(..., None) which
can be None; before calling post_request_localhost(payload("sim_deleteProvider",
target_provider_id)).json() add a guard: check if target_provider_id is None and
skip the deletion (or assert expected absence) to avoid sending an invalid ID;
update the test around target_provider_id and the post_request_localhost call to
only perform the delete when target_provider_id is not None.
- Around line 6-16: Add explicit type hints for the module-level variables:
annotate provider_name and model_name as str, API_KEY_MAP as Dict[str, str], and
api_key_env_var as str; use the existing symbols provider_name, model_name,
API_KEY_MAP, api_key_env_var to locate the assignments and update them with the
proper typing so the module complies with the project’s type-hinting guidelines.
🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

  • Push a commit to this branch (recommended)
  • Create a new PR with the fixes

ℹ️ Review info
⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Pro

Run ID: 102e5b03-ebef-4915-95a3-0443fd908f32

📥 Commits

Reviewing files that changed from the base of the PR and between 909928c and 23c8f58.

📒 Files selected for processing (1)
  • tests/integration/test_llm_providers_registry.py

Comment on lines +6 to 16
provider_name = os.environ.get("TEST_PROVIDER", "openai")
model_name = os.environ.get("TEST_PROVIDER_MODEL", "gpt-4o")

API_KEY_MAP = {
"openrouter": "OPENROUTERAPIKEY",
"openai": "OPENAIKEY",
"anthropic": "ANTHROPIC_API_KEY",
"google": "GEMINI_API_KEY",
}
api_key_env_var = API_KEY_MAP.get(provider_name, "OPENAIKEY")

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Potential issue | 🟡 Minor

🧩 Analysis chain

🏁 Script executed:

#!/bin/bash
python - <<'PY'
import ast
from pathlib import Path

path = next(Path(".").rglob("test_llm_providers_registry.py"))
tree = ast.parse(path.read_text())

for node in tree.body:
    if isinstance(node, ast.Assign):
        targets = [t.id for t in node.targets if isinstance(t, ast.Name)]
        if targets:
            print(f"Line {node.lineno}: untyped assignment -> {', '.join(targets)}")
PY

Repository: genlayerlabs/genlayer-studio

Length of output: 245


🏁 Script executed:

sed -n '6,16p' tests/integration/test_llm_providers_registry.py

Repository: genlayerlabs/genlayer-studio

Length of output: 407


Add type hints for module-level variables to comply with coding guidelines.

Lines 6, 7, 9, and 15 are untyped assignments. Type hints are required for all Python code per project guidelines.

Suggested refactor
-provider_name = os.environ.get("TEST_PROVIDER", "openai")
-model_name = os.environ.get("TEST_PROVIDER_MODEL", "gpt-4o")
+provider_name: str = os.environ.get("TEST_PROVIDER", "openai")
+model_name: str = os.environ.get("TEST_PROVIDER_MODEL", "gpt-4o")

-API_KEY_MAP = {
+API_KEY_MAP: dict[str, str] = {
     "openrouter": "OPENROUTERAPIKEY",
     "openai": "OPENAIKEY",
     "anthropic": "ANTHROPIC_API_KEY",
     "google": "GEMINI_API_KEY",
 }
-api_key_env_var = API_KEY_MAP.get(provider_name, "OPENAIKEY")
+api_key_env_var: str = API_KEY_MAP.get(provider_name, "OPENAIKEY")
📝 Committable suggestion

‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.

Suggested change
provider_name = os.environ.get("TEST_PROVIDER", "openai")
model_name = os.environ.get("TEST_PROVIDER_MODEL", "gpt-4o")
API_KEY_MAP = {
"openrouter": "OPENROUTERAPIKEY",
"openai": "OPENAIKEY",
"anthropic": "ANTHROPIC_API_KEY",
"google": "GEMINI_API_KEY",
}
api_key_env_var = API_KEY_MAP.get(provider_name, "OPENAIKEY")
provider_name: str = os.environ.get("TEST_PROVIDER", "openai")
model_name: str = os.environ.get("TEST_PROVIDER_MODEL", "gpt-4o")
API_KEY_MAP: dict[str, str] = {
"openrouter": "OPENROUTERAPIKEY",
"openai": "OPENAIKEY",
"anthropic": "ANTHROPIC_API_KEY",
"google": "GEMINI_API_KEY",
}
api_key_env_var: str = API_KEY_MAP.get(provider_name, "OPENAIKEY")
🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed.

In `@tests/integration/test_llm_providers_registry.py` around lines 6 - 16, Add
explicit type hints for the module-level variables: annotate provider_name and
model_name as str, API_KEY_MAP as Dict[str, str], and api_key_env_var as str;
use the existing symbols provider_name, model_name, API_KEY_MAP, api_key_env_var
to locate the assignments and update them with the proper typing so the module
complies with the project’s type-hinting guidelines.

Comment on lines +9 to 16
API_KEY_MAP = {
"openrouter": "OPENROUTERAPIKEY",
"openai": "OPENAIKEY",
"anthropic": "ANTHROPIC_API_KEY",
"google": "GEMINI_API_KEY",
}
api_key_env_var = API_KEY_MAP.get(provider_name, "OPENAIKEY")

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Potential issue | 🟠 Major

Fail fast for unsupported TEST_PROVIDER values.

Line 15 silently defaults unknown providers to "OPENAIKEY", which can hide bad CI/local configuration and produce misleading failures.

Proposed fix
 API_KEY_MAP = {
     "openrouter": "OPENROUTERAPIKEY",
     "openai": "OPENAIKEY",
     "anthropic": "ANTHROPIC_API_KEY",
     "google": "GEMINI_API_KEY",
 }
-api_key_env_var = API_KEY_MAP.get(provider_name, "OPENAIKEY")
+api_key_env_var = API_KEY_MAP.get(provider_name)
+if api_key_env_var is None:
+    raise ValueError(f"Unsupported TEST_PROVIDER: {provider_name}")
📝 Committable suggestion

‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.

Suggested change
API_KEY_MAP = {
"openrouter": "OPENROUTERAPIKEY",
"openai": "OPENAIKEY",
"anthropic": "ANTHROPIC_API_KEY",
"google": "GEMINI_API_KEY",
}
api_key_env_var = API_KEY_MAP.get(provider_name, "OPENAIKEY")
API_KEY_MAP = {
"openrouter": "OPENROUTERAPIKEY",
"openai": "OPENAIKEY",
"anthropic": "ANTHROPIC_API_KEY",
"google": "GEMINI_API_KEY",
}
api_key_env_var = API_KEY_MAP.get(provider_name)
if api_key_env_var is None:
raise ValueError(f"Unsupported TEST_PROVIDER: {provider_name}")
🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed.

In `@tests/integration/test_llm_providers_registry.py` around lines 9 - 16, The
current lookup silently defaults unknown provider_name to "OPENAIKEY"; update
the logic around API_KEY_MAP and api_key_env_var to fail fast: after defining
API_KEY_MAP and before using api_key_env_var, check if provider_name is a key in
API_KEY_MAP and if not raise a clear exception (or call pytest.skip/pytest.fail
as appropriate) indicating unsupported TEST_PROVIDER; use the symbols
API_KEY_MAP, provider_name, and api_key_env_var to locate and replace the
defaulting behavior so unknown providers no longer silently map to OPENAIKEY.

Comment on lines +25 to 39
target_provider_id = next(
(
provider["id"]
for provider in providers_and_models
if provider["model"] == "gpt-4o"
and provider["provider"] == "openai"
if provider["model"] == model_name
and provider["provider"] == provider_name
and provider["plugin"] == "openai-compatible"
),
None,
)

# Delete it
response = post_request_localhost(
payload("sim_deleteProvider", gpt4o_provider_id)
payload("sim_deleteProvider", target_provider_id)
).json()

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Potential issue | 🟠 Major

Guard delete when no existing provider is found.

next(..., None) can return None, and Line 38 then attempts deletion with an invalid ID. This breaks the flow for first-time provider/model combinations.

Proposed fix
     target_provider_id = next(
         (
             provider["id"]
             for provider in providers_and_models
             if provider["model"] == model_name
             and provider["provider"] == provider_name
             and provider["plugin"] == "openai-compatible"
         ),
         None,
     )

     # Delete it
-    response = post_request_localhost(
-        payload("sim_deleteProvider", target_provider_id)
-    ).json()
-    assert has_success_status(response)
+    if target_provider_id is not None:
+        response = post_request_localhost(
+            payload("sim_deleteProvider", target_provider_id)
+        ).json()
+        assert has_success_status(response)
🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed.

In `@tests/integration/test_llm_providers_registry.py` around lines 25 - 39, The
test finds target_provider_id via next(..., None) which can be None; before
calling post_request_localhost(payload("sim_deleteProvider",
target_provider_id)).json() add a guard: check if target_provider_id is None and
skip the deletion (or assert expected absence) to avoid sending an invalid ID;
update the test around target_provider_id and the post_request_localhost call to
only perform the delete when target_provider_id is not None.

@MuncleUscles
MuncleUscles merged commit 6650875 into main Mar 27, 2026
23 checks passed
@MuncleUscles
MuncleUscles deleted the fix/open-model-registration branch March 27, 2026 23:45
@github-actions

Copy link
Copy Markdown
Contributor

🎉 This PR is included in version 0.112.2 🎉

The release is available on GitHub release

Your semantic-release bot 📦🚀

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant