Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
19 changes: 17 additions & 2 deletions usertest/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -101,6 +101,20 @@ tasks to give. Without it, a link gets `calendar`. The code is in `/etc/usertest
without speech recognition (a browser other than Chrome or Edge), the
session still starts, typed only, and records the screen without sound.

The page is in English or Dutch (Nederlands), picked with the buttons at the
top, by `&lang=nl` in the invite link, or remembered from a previous visit.
The choice switches the page's texts, the speech recognizer, the
moderator's voice (a `nl-NL` voice, else `nl-BE`, else the browser's default)
and the language the moderator speaks, also mid-session: switching back to
English restores all of it from the next line on. Session plans stay in
English; each turn tells Claude the language in a `[Language]` line, so the
cached system prompt does not change. The Atomic app in the second window
stays in English: the data-browser has no Dutch translation yet (it has
English, Spanish, French and German). To add a language, see the top of
`page/i18n.js`. Checked on 2026-09-29 with a stub in place of the Claude API
and a macOS Dutch voice; not yet with a Dutch-speaking tester, so the speech
recognition of Dutch is not verified.

The moderator (`moderator/server.mjs`) asks Claude (`claude-opus-5`, effort
`low`, server-side refusal fallback on) for the next line, following the
interview script in `moderator/script.md` and the session's plan: short spoken questions, mostly
Expand Down Expand Up @@ -139,9 +153,10 @@ Measured on 2026-09-28 on session 1 (about 11 minutes): 34 s, about 2,000
input and 2,400 output tokens, 6 findings.

Per session, `/var/lib/usertest-sessions/<id>/` holds `meta.json` (with
`input`, `voice` or `typed` for how the session started, and `typedTurns`),
`input`, `voice` or `typed` for how the session started, `typedTurns`,
`lang`, the current language, and `langChanges`, each switch with its time),
`transcript.jsonl` (both sides, with the log lines each turn saw, token
usage, and `input: "typed"` or `"voice"` per tester turn), `screen-NNN.jpg` per turn, `recording.webm`, and after the analysis
usage, and `input: "typed"` or `"voice"` and `lang` per tester turn), `screen-NNN.jpg` per turn, `recording.webm`, and after the analysis
`findings.json`, `findings.md` and, when filed, `filed.json`. Limits: the invite code on every request, at
most 120 turns per session and 20 sessions per UTC day. The moderator keeps
sessions in memory, so restarting it ends the sessions in progress.
Expand Down
5 changes: 3 additions & 2 deletions usertest/moderator/analyze.mjs
Original file line number Diff line number Diff line change
Expand Up @@ -30,7 +30,7 @@ const MODEL = process.env.ANALYSIS_MODEL ?? 'claude-opus-5';
const REPO = process.env.GITHUB_FINDINGS_REPO ?? 'ontola/usertest-findings';
const MAX_SHOTS = 12;

const INSTRUCTIONS = `You analyze one remote usability test of Atomic (a personal data app) and its drive apps, plugins that import data from services like Google Calendar, GitHub, Notion and Clockify. A voice moderator (also Claude) ran the session; you get its transcript (speech-to-text, so expect recognition errors; turns marked TESTER (typed) were typed by the tester instead), the app's log lines each turn saw, and some screenshots of the tester's screen.
const INSTRUCTIONS = `You analyze one remote usability test of Atomic (a personal data app) and its drive apps, plugins that import data from services like Google Calendar, GitHub, Notion and Clockify. A voice moderator (also Claude) ran the session; you get its transcript (speech-to-text, so expect recognition errors; turns marked TESTER (typed) were typed by the tester instead), the app's log lines each turn saw, and some screenshots of the tester's screen. The session may be in another language than English, or switch language midway (marked "(lang ...)"); write the findings in English anyway, translating the few words you quote.

Find what should change. Two kinds:
- "app": a problem in Atomic or a drive app: confusion, a dead end, a wrong expectation, an error, missing feedback. Give "repo": "atomic-plugins" for a drive app (integrations/<plugin>/, the catalog) and "atomic-server" for the host (drive, navigation, Integrations page, settings, tables).
Expand Down Expand Up @@ -64,7 +64,8 @@ function transcriptText(entries) {
);
} else if (e.role === 'moderator')
lines.push(`${time} MODERATOR: ${e.say || '(stayed silent)'}`);
else if (e.event) lines.push(`${time} (${e.event})`);
else if (e.event)
lines.push(`${time} (${e.event}${e.lang ? ` ${e.lang}` : ''})`);
}

return lines.join('\n');
Expand Down
2 changes: 1 addition & 1 deletion usertest/moderator/script.md
Original file line number Diff line number Diff line change
Expand Up @@ -7,7 +7,7 @@ Find out where people get stuck, what they expect, and what confuses them. The p
## How to speak

- One or two short sentences per turn, then stop. No lists, markdown, emoji, URLs or code: everything is spoken.
- Speak English only, even if the tester answers in another language: the voice and the speech recognizer are set to English. If they speak another language, kindly ask them to continue in English.
- Speak only the language named in the [Language] line of the latest turn: the tester chose it on the page, and the voice and the speech recognizer are set to it. It is English unless the tester picked another. When it changes mid-session, switch at once, without commenting on it. The session plan below is written in English; say its tasks naturally in the session's language, but keep names of apps, buttons and screens as the tester sees them (the Atomic app itself may stay in English). If the tester speaks a different language than [Language], kindly ask them to continue in that language, or to switch the language at the top of their page.
- Ask open questions: "What do you expect to happen?", "What are you looking for now?", "What do you make of that?" Never ask leading questions.
- Encourage thinking aloud. If they fall silent (you get "(silence)"), gently ask what they are doing or thinking.
- If they ask you to wait, say "Sure" once, then answer [WAIT] to every "(silence)" until they speak again.
Expand Down
57 changes: 52 additions & 5 deletions usertest/moderator/server.mjs
Original file line number Diff line number Diff line change
Expand Up @@ -17,6 +17,12 @@
* `typedTurns`. meta.json's `input` says how the session started: `voice`,
* or `typed` when the page had no microphone or speech recognition.
*
* The tester picks the language on the page (English by default, see
* LANGUAGES) and can switch mid-session (`POST /sessions/<id>/lang`). Every
* turn tells Claude the language to speak in a `[Language]` line, so the
* system prompt (script and plan, in English) stays the same and cached.
* meta.json keeps the current `lang` and every switch in `langChanges`.
*
* The endpoints spend API money and are public, so every request needs the
* invite code (USERTEST_CODE, sent as `x-usertest-code`), a session takes at
* most MAX_TURNS turns, and at most MAX_SESSIONS_PER_DAY sessions start per
Expand Down Expand Up @@ -54,6 +60,18 @@ const MAX_TURNS = 120;
const MAX_SESSIONS_PER_DAY = 20;
const MAX_CHUNK = 32 * 1024 * 1024;
const MAX_LOG_LINES = 15;
/** The page's languages (../page/i18n.js), by the BCP 47 tag it sends. */
const LANGUAGES = {
'en-US': {
name: 'English',
lost: 'Sorry, I lost my train of thought. Could you tell me what you are doing now?',
},
'nl-NL': {
name: 'Dutch (Nederlands)',
lost: 'Sorry, ik ben de draad even kwijt. Kun je vertellen wat je nu aan het doen bent?',
},
};
const DEFAULT_LANG = 'en-US';

if (!CODE || CODE.length < 12)
throw new Error('USERTEST_CODE must be set (at least 12 characters)');
Expand Down Expand Up @@ -89,7 +107,7 @@ const DEFAULT_PLAN = 'calendar';
if (!PLANS[DEFAULT_PLAN])
throw new Error(`sessions/${DEFAULT_PLAN}.md is missing`);
const client = new Anthropic();
/** id -> { dir, meta, started, plan, cursor, clients, messages, turns, done, analyzed } */
/** id -> { dir, meta, lang, langChanged, started, plan, cursor, clients, messages, turns, done, analyzed } */
const sessions = new Map();

mkdirSync(SESSIONS_DIR, { recursive: true });
Expand Down Expand Up @@ -173,11 +191,13 @@ async function nextLine(session, said, screenshot, typed) {
typeof screenshot === 'string' && /^[A-Za-z0-9+/=]+$/.test(screenshot)
? screenshot
: undefined;
const language = LANGUAGES[session.lang];
const opening =
session.meta.input === 'typed'
? '(the session starts now; the tester has no working microphone or speech recognition and types their answers)'
: '(the session starts now)';
const content = [
`[Language] ${language.name}${session.langChanged ? '. The tester just switched to it: speak it from now on.' : ''}`,
log.length ? `[Log]\n${log.join('\n')}` : '[Log]\n(nothing new)',
`[Tester${typed ? ', typed' : ''}]\n${session.turns === 0 ? opening : heard}`,
].join('\n\n');
Expand All @@ -199,10 +219,12 @@ async function nextLine(session, said, screenshot, typed) {
join(session.dir, `screen-${String(session.turns).padStart(3, '0')}.jpg`),
Buffer.from(shot, 'base64'),
);
session.langChanged = false;
record(session, {
role: 'tester',
said: heard,
input: typed ? 'typed' : 'voice',
lang: session.lang,
log,
screenshot: !!shot,
});
Expand Down Expand Up @@ -248,8 +270,7 @@ async function nextLine(session, said, screenshot, typed) {
const wait = !done && /^\[WAIT\]$/.test(text);
const say = wait
? ''
: text.replace('[END]', '').replace('[WAIT]', '').trim() ||
'Sorry, I lost my train of thought. Could you tell me what you are doing now?';
: text.replace('[END]', '').replace('[WAIT]', '').trim() || language.lost;

session.messages.push({ role: 'assistant', content: wait ? '[WAIT]' : say });
session.turns++;
Expand Down Expand Up @@ -333,13 +354,18 @@ createServer(async (req, res) => {
const dir = join(SESSIONS_DIR, id);
mkdirSync(dir);
const started = new Date().toISOString();
const lang = Object.hasOwn(LANGUAGES, body.lang)
? body.lang
: DEFAULT_LANG;
const session = {
dir,
meta: {
id,
started,
name: String(body.name ?? '').slice(0, 80),
lang: String(body.lang ?? '').slice(0, 20),
lang,
// Every switch after the start: [{ t, lang }].
langChanges: [],
plan,
// How the page started: `typed` without a microphone or speech
// recognition. Typed turns are counted either way.
Expand All @@ -348,6 +374,8 @@ createServer(async (req, res) => {
userAgent: req.headers['user-agent'],
model: MODEL,
},
lang,
langChanged: false,
started,
plan,
cursor: started,
Expand All @@ -364,7 +392,7 @@ createServer(async (req, res) => {
}

const match = url.pathname.match(
/^\/sessions\/([\w-]+)\/(turn|recording|end|news)$/,
/^\/sessions\/([\w-]+)\/(turn|recording|end|news|lang)$/,
);
const session = match && sessions.get(match[1]);
if (!session) return reply(404, { error: 'Unknown session' });
Expand Down Expand Up @@ -408,6 +436,25 @@ createServer(async (req, res) => {
return reply(204);
}

if (req.method === 'POST' && match[2] === 'lang') {
const { lang } = JSON.parse((await readBody(req, 1024)).toString());
if (!Object.hasOwn(LANGUAGES, lang))
return reply(400, {
error: `Unknown language; known: ${Object.keys(LANGUAGES).join(', ')}`,
});

if (lang !== session.lang) {
session.lang = lang;
session.langChanged = true;
session.meta.lang = lang;
session.meta.langChanges.push({ t: new Date().toISOString(), lang });
writeMeta(session);
record(session, { role: 'page', event: 'lang', lang });
}

return reply(200, { lang });
}

if (req.method === 'POST' && match[2] === 'end') {
record(session, { role: 'page', event: 'end' });
session.done = true;
Expand Down
Loading
Loading