Skip to content

feat(practice): ask and score with the cloud model chosen in Settings - #83

Draft
ukez15 wants to merge 20 commits into
mainfrom
issue/82-practice-cloud-models
Draft

ukez15 wants to merge 20 commits into
mainfrom
issue/82-practice-cloud-models

Conversation

@ukez15

@ukez15 ukez15 commented Oct 27, 2026

Copy link
Copy Markdown
Owner

Closes #82

Brief (approved with the recommendations): docs/atlas-gotcha/practice-cloud-models.md.

Stacked on #79. This branch starts from #79's branch, so until #79 merges the diff also shows #79's commits. This PR's own change is the last commit, cabc6ae, and the brief. It's opened as a draft now so CI can check the backend, which can't build in the cloud session.

What changed

  • New ChatModelChoice.For(owner, options) (Application/Providers): the user's chat provider and model from Settings, else the configured default (Ollama). Live answers (AnswerGenerator) and practice both use it now, so they can't disagree. Live-answer behaviour is unchanged.
  • PracticeService asks and scores with that provider and model, and sends the key the app already hands to the session when it starts (ISessionKeyStore). "Ollama only in this wave" is gone.
  • Missing or rejected key: the provider's message ("Add your Anthropic API key in Settings.") arrives as the question stream's error event, or as the existing 503 at scoring. Practice never falls back to Ollama. After a scoring failure the answer stays open, so you can retry.
  • Document budget: a cloud question gets the full 6,000 characters; the local model keeps 2,000 (QuestionDocumentBudgetFor, from feat(practice): run mock interviews in the overlay, with live text and a mic gate #79).
  • Privacy: with a cloud model, practice sends that provider the document excerpts, the earlier questions and the spoken answer's text (never audio), the same as live answers. With the default (Ollama), nothing leaves the machine.

Tests (written first):

  • Unit tests:
    • local default with no key;
    • cloud model with the session key for both the question and the score;
    • full budget for a cloud question;
    • missing key with no fallback, and the question stays closed;
    • missing key at scoring keeps the answer.
  • One API test runs practice against a stand-in cloud provider, with a key and without one, and checks the key never appears in the response.

Not verifiable in the cloud session: the backend tests run in CI only. The real Anthropic, OpenAI and Google calls need the owner's key on Windows.

Checklist

  • Spec/acceptance criteria in specs/ are met (issue Practice: use the cloud model chosen in Settings #82's criteria)
  • Tests written first; coverage ≥ 75% for touched packages (CI checks this; no UI change)
  • No paid provider enabled by default; no data leaves the machine by default
  • No API keys, transcript text or document content in logs
  • src/ui has no electron imports
  • Conventional commit title

Generated by Claude Code

ukez15 and others added 20 commits October 26, 2026 19:08
Records that v1 is feature-complete on Windows, which epics stay open, the FluentAssertions ignore rule, the first real release.yml run on the new Actions majors, the deny-licenses deprecation, and the garbled @dependabot command gotcha. Removes PROGRESS.md, which asked to be deleted once read.

Refs #72
Mock interview sessions now show the practice panel in the existing overlay window instead of the answer assistant, with a mic status line so a mic that isn't heard is visible. The main window shows 'Practising in the overlay.' and the attempt history, reloaded on focus. Capture status polling moves into a shared useCaptureStatus hook.

Refs #74
The packaged app spawned InterviewCopilot.Api without a working folder, so ASP.NET never loaded the appsettings.json next to the exe. Its CORS allow-list was then empty and every UI request from app://interview-copilot was rejected, which the UI reports as 'Can't reach the InterviewCopilot backend'. The health probe isn't subject to CORS, so no error showed.

Closes #75
connect-src listed http://127.0.0.1:5480 but not ws://127.0.0.1:5480. Under CSP Level 3 an http: source doesn't match a ws: URL, so the session events socket was blocked: no live transcript in mock interviews and no auto-answers in the live overlay, with nothing shown to the user.

Refs #77
After scoring, 'Your answer' shows the exact text that was marked. The overlay follows new text: in a mock interview it pauses while the user has scrolled up, and on a live call it always jumps to the newest text. Brief updated with the owner's decisions.

Refs #74
…open

Whisper kept transcribing the mic between questions, competing with Ollama for the CPU, and late segments spoken before the question counted as the answer. Frames are now numbered on arrival so segment times are true stream positions; a capture gate drops mock-interview candidate audio while no question is open; the question returns answerFromMs and both the collector and the overlay ignore speech that started earlier. The worker ends a segment at a gap instead of stitching audio from two moments.

Refs #78
Importing the Audio test namespace into the practice tests made InMemoryPreferences ambiguous (CS0104).

Refs #78
A reconnected stream must start its segment times at 0, as before. The sink now forgets a session's frame counters when its stream ends, which also stops them growing without bound.

Refs #78
…question

The local model reads the whole prompt before it writes. At about 41 tokens/s on the
owner's machine, 6,000 document characters took about 37 s. Scoring and the live
assistant keep 6,000.

Refs #78
POST practice/next now streams server-sent events like live answers: "delta" for
each piece of the question, then "done" with the question and answerFromMs, or
"error" if the model fails part-way. Ownership, mode and live checks still answer
404, 400 and 409 before the stream starts.

The question opens for the answer only once it is complete, and a skipped question
closes as soon as the next one is asked, so the mic rests while it is written. The
relay loop is shared with live answers in LlmEventStream.

Refs #78
Cloud models read prompts fast, so they keep the full 6,000-character document
budget. Practice runs on Ollama in this wave, so today every question keeps the
2,000 limit; the rule is in place for when practice can use cloud models.

Refs #78
"Done answering" often comes as the candidate stops talking, while the last
sentence is still in the speech buffer or in Whisper. The worker now records how
far each speaker's stream is transcribed, and finishing waits, with the question
still open so the mic keeps listening, until transcription reaches the moment of
the click or 5 s pass. Deepgram sessions have nothing queued locally and don't wait.

Refs #78
Practice now picks the chat provider and model the way live answers do, through
a shared ChatModelChoice, and sends the key handed to the session. Ollama stays
the default with no key. A missing or rejected key shows the provider's message
with no fallback; a cloud question gets the full document budget.

Closes #82
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Practice: use the cloud model chosen in Settings

2 participants