Delegate embedding model discovery to contextual-orchestrator

CHANGELOG.d/2.20.1-orchestrator-temperature-negotiation.md

@@ -0,0 +1,8 @@
+# 2.20.1 — Provider temperature capability negotiation
+
+- Compose now pins ContextualWisdomLab/contextual-orchestrator#791 so providers
+  that accept only their default sampling temperature can retry the same
+  request without the unsupported optional field, alongside the orchestrator's
+  embedding-capability selection contract.
+- The pin remains an immutable upstream commit and does not copy provider
+  behavior or credentials into LineageWeave.

CHANGELOG.d/2.20.2-orchestrator-owned-embedding.md

@@ -0,0 +1,9 @@
+# 2.20.2
+
+- Delegate embedding-model selection to contextual-orchestrator, retain the
+  resolved model as database provenance, and keep provider model selectors out
+  of the LineageWeave backend process.
+- Discard the legacy embedding selector at the Compose bootstrap so automatic
+  capability discovery remains the only model-selection path.
+- Keep the backward-compatible embedding client name aligned with the
+  orchestrator client's batch and per-request provenance contract.

backend/app/config.py

@@ -129,7 +129,9 @@ def load_settings() -> Settings:
         ],
         orchestrator_base_url=os.environ.get("ORCHESTRATOR_BASE_URL", ""),
         orchestrator_api_key=os.environ.get("ORCHESTRATOR_API_KEY", ""),
-        embedding_model=os.environ.get("LLM_GATEWAY_EMBEDDING_MODEL", "").strip(),
+        # contextual-orchestrator owns embedding model discovery. Keep the
+        # empty compatibility value for older callers; never read a selector.
+        embedding_model="",
         valkey_url=os.environ.get("VALKEY_URL", "redis://localhost:16379/0"),
         searxng_base_url=os.environ.get("SEARXNG_BASE_URL", ""),
         tepp_transport_url=os.environ.get("TEPP_TRANSPORT_URL", ""),

backend/app/post_content_queue.py

@@ -3,8 +3,8 @@
 from __future__ import annotations
 
 import hashlib
-from datetime import timedelta
 from dataclasses import dataclass
+from datetime import timedelta
 from typing import Any
 
 import asyncpg
@@ -48,7 +48,7 @@ async def post_content_is_complete(
     conn: asyncpg.Connection,
     post_id: str,
     *,
-    embedding_model_code: str,
+    embedding_model_code: str | None,
     require_structure: bool = False,
 ) -> bool:
     """Require configured semantic, structure, and region evidence before ready."""
@@ -61,8 +61,35 @@ async def post_content_is_complete(
                         where unit.post_id = $1
                    )
                and (
-                   $2 = ''
-                       or (
+                   (
+                       nullif($2::text, '') is null
+                       and (
+                           not exists(
+                               select 1
+                                 from post_content_unit unit
+                                 left join post_content_embedding embedding
+                                   on embedding.post_content_unit_id = unit.post_content_unit_id
+                                where unit.post_id = $1
+                                  and embedding.post_content_embedding_id is null
+                           )
+                           and not exists(
+                               select 1
+                                 from post_content_unit unit
+                                 join post_content_image image
+                                   on image.post_content_unit_id = unit.post_content_unit_id
+                                 join post_content_image_region region
+                                   on region.post_content_image_id = image.post_content_image_id
+                                 left join post_content_image_region_embedding embedding
+                                   on embedding.post_content_image_region_id = region.post_content_image_region_id
+                                where unit.post_id = $1
+                                  and region.description_status_code = 'described'
+                                  and embedding.post_content_image_region_embedding_id is null
+                           )
+                       )
+                   )
+                   or (
+                       nullif($2::text, '') is not null
+                       and (
                            not exists(
                                select 1
                                  from post_content_unit unit
@@ -88,6 +115,7 @@ async def post_content_is_complete(
                            )
                        )
                    )
+                   )
                and (
                        not $3::boolean
                        or not exists(

docker-compose.yml

@@ -161,9 +161,6 @@ services:
       # LLM_GATEWAY_API_KEY in the orchestrator's private env file.
       ORCHESTRATOR_BASE_URL: ${ORCHESTRATOR_BASE_URL:-http://orchestrator:8000}
       ORCHESTRATOR_API_KEY: ${ORCHESTRATOR_API_KEY:-${CONTEXTUAL_ORCHESTRATOR_TOKEN:-lineageweave-orchestrator-dev-only}}
-      # This is a non-secret embedding contract. Provider credentials remain
-      # private to contextual-orchestrator and are never injected into backend.
-      LLM_GATEWAY_EMBEDDING_MODEL: ${LLM_GATEWAY_EMBEDDING_MODEL:-text-embedding-3-large}
       SEARXNG_BASE_URL: http://searxng:8080
       TEPP_TRANSPORT_URL: ${TEPP_TRANSPORT_URL:-}
       TEPP_API_KEY: ${TEPP_API_KEY:-}

docker/contextual-orchestrator/Dockerfile

@@ -3,9 +3,11 @@ FROM python:3.12-slim@sha256:423ed6ab25b1921a477529254bfeeabf5855151dc2c3141699a
 WORKDIR /app
 
 # Reuse the upstream implementation without copying it into LineageWeave.
-# Pin the runtime to a reviewed immutable upstream commit; model selection,
-# structured synthesis, and reasoning policy stay in contextual-orchestrator.
-ADD https://github.com/ContextualWisdomLab/contextual-orchestrator/archive/7df051ac2b929e5910071ac1848d0447c5d6744e.tar.gz /tmp/contextual-orchestrator.tar.gz
+# Pin the runtime to ContextualWisdomLab/contextual-orchestrator#791's merged
+# integration head until its remaining protected dependency stack merges;
+# this keeps model selection, structured synthesis, and reasoning policy in
+# contextual-orchestrator itself.
+ADD https://github.com/ContextualWisdomLab/contextual-orchestrator/archive/4c31c3549537ba4813fe25e2bf4efc3e5b56aa3f.tar.gz /tmp/contextual-orchestrator.tar.gz
 RUN mkdir /tmp/contextual-orchestrator \
     && tar -xzf /tmp/contextual-orchestrator.tar.gz --strip-components=1 -C /tmp/contextual-orchestrator \
     && cp -R /tmp/contextual-orchestrator/contextual_orchestrator /app/contextual_orchestrator \

docker/contextual-orchestrator/start.py

@@ -7,11 +7,19 @@
 
 from __future__ import annotations
 
+import json
 import os
 import sys
-import json
 from pathlib import Path
 
+_PROVIDER_CREDENTIAL_NAMES = (
+    "BYTEZ_API_KEY",
+    "NVIDIA_NIM_API_KEY",
+    "NVIDIA_NIM_API_KEY_SUB",
+    "OPENROUTER_API_KEY",
+    "OPENAI_API_KEY",
+)
+
 
 def _pop_first_env(*names: str) -> str:
     """Read the first configured alias without leaving credentials in the environment."""
@@ -22,11 +30,22 @@ def _pop_first_env(*names: str) -> str:
     return ""
 
 
+def _pop_provider_credentials() -> dict[str, str]:
+    """Move every configured provider key out of the process environment."""
+    return {
+        name: value
+        for name in _PROVIDER_CREDENTIAL_NAMES
+        if (value := os.environ.pop(name, "").strip())
+    }
+
+
 def main() -> None:
     """Register the provider credential and delegate to the upstream server."""
-    provider_key = _pop_first_env("LLM_GATEWAY_API_KEY", "LLM_API_KEY", "NVIDIA_NIM_API_KEY")
+    provider_credentials = _pop_provider_credentials()
+    gateway_key = _pop_first_env("LLM_GATEWAY_API_KEY", "LLM_API_KEY")
+    provider_key = gateway_key or next(iter(provider_credentials.values()), "")
     if not provider_key:
-        raise SystemExit("LLM_GATEWAY_API_KEY or LLM_API_KEY is required to start the real LLM service")
+        raise SystemExit("at least one supported provider credential is required to start the real LLM service")
     auth_token = os.environ.get("CONTEXTUAL_ORCHESTRATOR_TOKEN", "").strip()
     if not auth_token:
         raise SystemExit("CONTEXTUAL_ORCHESTRATOR_TOKEN is required to start the authenticated LLM service")
@@ -56,35 +75,30 @@ def main() -> None:
         agent["base_url"] = provider_url
         agent["credential_key"] = "LLM_GATEWAY_API_KEY"
         agent.setdefault("provider_protocol", "auto")
-    embedding_model = os.environ.get("LLM_GATEWAY_EMBEDDING_MODEL", "").strip()
-    if embedding_model:
-        embedding_agents = [
-            agent
-            for agent in agents["agents"]
-            if "embedding" in agent.get("tags", [])
-        ]
-        if embedding_agents:
-            for agent in embedding_agents:
-                agent["model"] = embedding_model
-        else:
-            agents["agents"].append(
-                {
-                    "id": "llm_gateway_embedding_agent",
-                    "model": embedding_model,
-                    "base_url": provider_url,
-                    "credential_key": "LLM_GATEWAY_API_KEY",
-                    "provider_protocol": "auto",
-                    "tags": ["embedding"],
-                    "priority": 0,
-                }
-            )
+    # A legacy embedding selector must never reach the upstream server. Model
+    # capability discovery and selection belong to contextual-orchestrator.
+    os.environ.pop("LLM_GATEWAY_EMBEDDING_MODEL", None)
     agents_path.write_text(json.dumps(agents), encoding="utf-8")
 
     from contextual_orchestrator.credentials import register_credential
 
-    register_credential("NVIDIA_NIM_API_KEY", provider_key)
-    register_credential("LLM_GATEWAY_API_KEY", provider_key)
+    registered_names: set[str] = set()
+
+    def register(name: str, value: str) -> None:
+        if name not in registered_names:
+            register_credential(name, value)
+            registered_names.add(name)
+
+    if gateway_key:
+        register("NVIDIA_NIM_API_KEY", provider_credentials.get("NVIDIA_NIM_API_KEY", gateway_key))
+        register("LLM_GATEWAY_API_KEY", gateway_key)
+    else:
+        register("LLM_GATEWAY_API_KEY", provider_key)
+    for name, value in provider_credentials.items():
+        register(name, value)
     del provider_key
+    del gateway_key
+    del provider_credentials
     sys.argv = [
         "contextual_orchestrator",
         "--serve",

docs/adr/0030-external-llm-gateway-environment.md

@@ -21,9 +21,9 @@ These credentials are not interchangeable. The first is the local service
 boundary; the second is the provider credential.
 
 The backend must not load `~/.env` wholesale. Compose injects the provider
-credential and URL only into contextual-orchestrator; the backend receives
-only its internal orchestrator credential and the non-secret embedding model
-identifier. This prevents an unrelated application process from holding the
+credential and URL only into
+contextual-orchestrator; the backend receives only its internal orchestrator
+credential. This prevents an unrelated application process from holding the
 provider secret while preserving the single LLM/Vision boundary.
 
 ## Decision
@@ -64,13 +64,12 @@ local score, summary, extraction, or answer when the gateway is unavailable.
   `host.docker.internal:8080` text gateway and `host.docker.internal:18082`
   Vision gateway when `LINEAGEWEAVE_ALLOW_LOCAL_LLM_HTTP=1`; arbitrary local
   HTTP ports remain rejected.
-- `LLM_GATEWAY_MODEL` and `VISION_MODEL` are not selected by LineageWeave.
-  When they are blank, contextual-orchestrator resolves the registered agent
-  model, so a local or provider-specific model name cannot leak into this
-  application or be assumed available on an external gateway.
-- `LLM_GATEWAY_EMBEDDING_MODEL` is the explicit allowlisted semantic embedding
-  model. If it is absent, contextual-orchestrator rejects embedding work
-  instead of returning its standalone eight-dimensional heuristic vector.
+- `LLM_GATEWAY_MODEL`, `VISION_MODEL`, and
+  `LLM_GATEWAY_EMBEDDING_MODEL` are not LineageWeave runtime settings. The
+  contextual-orchestrator receives blank capability agents and owns discovery,
+  capability selection, and model provenance for text, vision, and embedding
+  work. A provider-specific model name is never selected or assumed available
+  by this application.
 - `LLM_API_KEY`, `LLM_API_GATEWAY`, and `LLM_GATEWAY_URL` are compatibility
   aliases only; `LLM_GATEWAY_API_KEY` and `LLM_GATEWAY_API_URL` are the
   canonical names for

docs/adr/0083-orchestrator-runtime-commit-pin.md

@@ -10,14 +10,19 @@ The orchestrator's `auto` reasoning mode is an internal routing decision and
 must not be forwarded as an upstream provider `reasoning_effort` value. The
 runtime also must discover provider models from the configured gateway rather
 than requiring `LLM_GATEWAY_MODEL`, and structured requests must remain
-multi-agent.
+multi-agent. Some provider deployments accept only their default sampling
+temperature; retrying the same rejected value cannot recover that capability
+mismatch.
 
 ## Decision
 
 `docker/contextual-orchestrator/Dockerfile` pins the downloaded archive to
-commit `7df051ac2b929e5910071ac1848d0447c5d6744e`. The pin remains explicit
-and immutable until the reviewed upstream change is superseded; it is not a
-moving `main` reference and it is not a LineageWeave monkey patch.
+commit `4c31c3549537ba4813fe25e2bf4efc3e5b56aa3f`, the merged integration
+head of ContextualWisdomLab/contextual-orchestrator#791. That PR combines
+the temperature negotiation in #779 with the orchestrator-owned embedding
+selection in #789 and remains stacked on #765. The pin remains explicit and
+immutable until the remaining protected stack merges; it is not a moving `main` reference
+and it is not a LineageWeave monkey patch.
 
 The runtime contract is:
 
@@ -33,6 +38,13 @@ The runtime contract is:
   reconciliation prompt; independent VISION worker evidence is retained instead.
 - A provider 4xx is reported as a failed orchestration attempt, never as a
   successful empty semantic result.
+- When HTTP 400/422 explicitly proves that `temperature` is unsupported, the
+  same endpoint is retried once with only that optional field omitted. Invalid
+  values and unrelated 4xx responses remain fail-closed.
+- Embedding requests may omit `model`; the orchestrator selects only an enabled
+  embedding-capable agent and returns the resolved model for provenance.
+- An omitted embedding model fails with `embedding_unavailable` when no eligible
+  agent exists; LineageWeave never supplies a heuristic or provider fallback.
 - An empty seed model is expanded from the configured gateway `/v1/models`
   endpoint; embedding-only rows are not added to the chat agent pool.
 - `json_object`, `json_schema`, and Responses JSON formats run conduct plus
@@ -44,4 +56,5 @@ The runtime contract is:
   implementation.
 - Rebuilding the image is required after the upstream pin changes.
 - Protected-branch review and merge remain external gates; this pin does not
-  bypass upstream review.
+ bypass ContextualWisdomLab/contextual-orchestrator#765 or
+ ContextualWisdomLab/contextual-orchestrator#779, #789, or #791.

docs/adr/0124-orchestrator-owned-embedding-model-provenance.md

@@ -0,0 +1,49 @@
+# ADR 0124: Orchestrator-owned embedding model discovery and provenance
+
+- Status: Accepted
+- Date: 2026-08-20
+- Scope: LineageWeave embedding ingestion and post-content backfill
+- Figma: N/A
+
+## Context
+
+LineageWeave must send embedding work through contextual-orchestrator. The
+orchestrator can select an embedding-capable agent when the request omits a
+model, but LineageWeave still read `LLM_GATEWAY_EMBEDDING_MODEL` and treated a
+local model string as a prerequisite. That made the consumer contradict the
+paper-grounded model boundary in ADR 0076 and the shared agent contract.
+
+## Decision
+
+1. LineageWeave does not read or expose a provider embedding model selector.
+2. The embedding request omits `model` when no explicit compatibility value is
+   supplied; contextual-orchestrator resolves the capability and returns the
+   served model code.
+3. Every persisted unit or visual-region embedding stores that returned model
+   code in the normalized embedding table. If the boundary returns no model
+   code, the vector is not persisted as provenance-bearing evidence and the
+   ingestion job remains incomplete for retry.
+4. Completeness checks in automatic mode require an embedding for every
+   embeddable unit and described image region, regardless of model code. An
+   explicit legacy model code, when supplied by a lower-level caller, retains
+   exact-model completeness matching but is not a runtime configuration path.
+5. The upstream contextual-orchestrator capability-selection contract is a
+   dependency of this consumer change. No monkey patch, provider SDK, local
+   model ranking, or sentinel model name is permitted.
+
+## Consequences
+
+- Runtime configuration contains only the internal orchestrator boundary; the
+  provider credentials remain in `~/.env` at the orchestrator service boundary.
+- Existing rows retain their actual `embedding_model_code`; no backfill uses a
+  guessed or synthetic model code.
+- A temporarily incomplete orchestrator response fails closed for evidence,
+  while the source body remains available for retry.
+
+## References
+
+ContextualWisdomLab. (2026). *contextual-orchestrator capability selection
+contract* (PR #789). https://github.com/ContextualWisdomLab/contextual-orchestrator/pull/789
+
+ContextualWisdomLab. (2026). *ADR 0076: Paper-grounded model policy*.
+`docs/adr/0076-paper-grounded-model-policy.md`.