Skip to content

Keep an offline endpoint's models in /v1/models, flagged endpoint_offline #35

Open

Description

@barelyworkingcode

User story

As an operator, I want relayLLM to keep an offline endpoint's models in its model list, flagged offline, so that relay can answer "endpoint unavailable" instead of "model not found" during a short outage.

Acceptance criteria

  • GET /v1/models keeps each last-known model of an offline endpoint, with endpoint_offline set to the endpoint name.
  • GET /v1/models keeps each modelMap key whose target is such a model, with the same endpoint_offline value.
  • Managed and virtual rows never carry endpoint_offline.
  • The offline mark clears within the existing 15 s probe window.

What I'd notice

A clearer error during a brief endpoint outage, once relay reads the flag.

Context

barelyworkingcode/relay#497 adds the relay half: relay reads endpoint_offline and answers 503 naming the endpoint. Today relayLLM drops an offline endpoint's rows, so relay cannot tell offline from absent. The wire shape is in the ## Contract of barelyworkingcode/relay#497. Found while working barelyworkingcode/relay#497.

Tasks

  • Keep last-known rows and their modelMap keys with endpoint_offline.
  • Test both kinds of row and the clear.
  • Real-app check against relay main with relay#497 merged.

Activity

  1. added
    needs-reviewAwaiting owner review before work starts
    from-devIdea found during development, not asked for by the owner
    on Oct 20, 2026
  2. barelyworkingcode commented on Oct 20, 2026

    @barelyworkingcode
    OwnerAuthor

    The relay half merged in barelyworkingcode/relay#522: relay reads endpoint_offline and answers 503 naming the endpoint.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    from-devIdea found during development, not asked for by the ownerneeds-reviewAwaiting owner review before work starts

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions