Skip to content

usertest: sample data for sessions without a provider account (#196) - #239

Merged
michielbdejong merged 1 commit into
mainfrom
claude/issue-196-usertest-sample-data
Oct 1, 2026
Merged

michielbdejong merged 1 commit into
mainfrom
claude/issue-196-usertest-sample-data

Conversation

@michielbdejong

Copy link
Copy Markdown
Contributor

Closes #196.

A tester can now try Money, Google Calendar, GitHub issues, Clockify and Notion in a moderated session without a real account. Everything they see is invented. None of the apps' source changes, so VERSIONS is unchanged: a fresh catalog.mjs build gives byte-identical modules for all five existing apps, compared against origin/main's catalog.mjs.

The design, and why

Money needs no account, only a file, so it gets the sample statements #196 asked for:

  • integrations/money/fixtures/usertest/generate.mjs writes three files for Acme Studio, account NL00BANK0000000000, with invented counterparties:
    • acme-studio-2026-08.mt940: 1 Aug – 15 Sep, 31 bookings.
    • acme-studio-2026-08.camt053.xml: the same bookings, plus a pending entry that is not imported.
    • acme-studio-2026-09.mt940: 1 – 30 Sep, 25 bookings. Ten of them overlap the first file by bank reference, so 15 are new.
  • The bookings include a refund, a USD payment booked in EUR, and a seven-line direct-debit description.
  • catalog.mjs copies the files to samples/money/ on the catalog host. Caddy serves /samples/* as an attachment.
  • A plan's new Sample files line names its files. The moderator lists them with /plans, and the session page shows them in a "Sample files" card, before and during the session.
  • money.md now offers the sample first, and the tester's own export only as an option. The privacy question is gone from the first turn.

For the OAuth providers, the smallest fit was the mock proxy's existing stateful fixtures. These are the same ones the lanes' e2e tests run against.

  • catalog.mjs builds a second entry per app: "Google Calendar (sample data)", "GitHub issues (sample data)", "Clockify timesheets (sample data)" and "Notion (sample data)".
  • Each entry bundles the app's own built module with usertest/sample-data/proxy.mjs. Only store.proxy is replaced, by the fixture running in the app's frame. The drive, rows, row access and review steps are the real host's.
  • The fixtures are re-seeded with realistic invented data: Acme Studio's calendar around the install day, acme-studio/* repositories (the existing user-testing scenario), Alex Sample's Clockify fortnight, and Notion's Roadmap and Reading list.
  • The sample account starts connected, so each app takes its fresh-mount path straight to its picker or first sync. A yellow line above the app says the data is invented.
  • State survives a remount. The seed, the connected flag and each write the app sends are kept as JSON in one resource under the App (the place a view may always write). On load they are replayed with Date fixed at each write's original time, so ETags and timestamps come out identical.
  • Four new plans (calendar-sample, issue-tracker-sample, timesheets-sample, notion-sample) say in "For the moderator only" that the data is invented. So "no sign-in", "not my data" or "couldn't open it on GitHub" are expected, not findings. sessions/README.md explains when to use them.

I did not seed the test drive server-side and did not add a mode to the apps. Either would need app or host changes, and the host is frozen. This approach needs neither.

Checks run (2026-10-01, pin a12b74a6783b)

  • node usertest/e2e/samples.mjs: 4/4. It builds the catalog, serves it, and starts the pinned atomic-server. Then it opens /app/dev-drive the way the page does, installs each "(sample data)" entry from Integrations, imports, finds an invented record, reloads, and finds it again.
  • node --test usertest/sample-data/samples.test.mjs: 6/6. Covers reads, writes, and replay after a remount: calendar PATCH and ETag, a stale If-Match that still gets 412, GitHub create and close with identical dates, and a Clockify PUT.
  • node usertest/e2e/run.mjs: 19/19. The new check: money's sample links, none for notion, and the card stays visible during the session.
  • vitest for money: 175/175, including the new usertest-samples.test.ts. The files match their generator, both readers parse and reconcile them, and the overlap gives 15 new identities.
  • oxlint and oxfmt --check on integrations are clean.
  • caddy validate passes with Caddy 2.6.2, the version the droplet runs.
  • check-live.mjs could not reach the droplet from the cloud session (HTTP 403 from the proxy).

Lanes

None of the existing lanes covers this. The sample wrapper lives in usertest/, and lanes.mjs lets a tooling lane name only paths under integrations/tooling/, so I added no lane. The real-host check is usertest/e2e/samples.mjs. Like run.mjs, it is not in CI. Money's new vitest does run in CI, through money's certification step.

Not covered / known limits

  • Sample apps can't test signing in to the provider; the non--sample plans and a real account still cover that. Links to provider pages ("Open in Notion", an issue's GitHub page) lead nowhere.
  • Not verified on the droplet or with a real tester.
  • App finding, not fixed here: in the issue-tracker app, connecting by picking an existing connection in the host's consent bar (no page reload) leaves the repository picker loading forever. controller.connect() doesn't call listRepositories(). The Notion app likewise doesn't start its first sync on that path. I found this while building the fake proxy, worked around it with "starts connected", and left the apps alone.
  • Moneybird isn't in the user-testing catalog, so it has no sample entry. Todoist has no drive app.
  • The GitHub sandbox-repository decision in usertest: synthetic sample data for sessions that shouldn't need a real account (money first) #196 is asked on Agent coordination board (Claude sessions: rules, status, hand-offs) #227. The -sample plan covers the no-account case in the meantime.

Deploy needs (for Michiel or the usertest-deploy workflow)

Rows from docs/agents/usertest-droplet.md:

  • catalog.mjs → check-live.mjs → deploy.sh. This brings the new catalog entries, samples/, the page and the Caddyfile. deploy.sh restarts Caddy itself, and now also ships samples.
  • Restart the moderator (moderator/run.sh): the plans changed and server.mjs now lists sample files. This ends sessions in progress, so warn on Agent coordination board (Claude sessions: rules, status, hand-offs) #227 first.
  • No atomic-server restart, and no VERSIONS bumps.

🤖 Generated with Claude Code

https://claude.ai/code/session_01Qt6FnBHXdJWx9GEKuAqVBo


Generated by Claude Code

Money: synthetic Acme Studio bank statements (MT940 Aug, overlapping
MT940 Sep, camt.053 Aug) generated in integrations/money/fixtures/usertest/,
served from the catalog host's samples/money/ and linked on the session
page; money.md uses them first.

Calendar, GitHub issues, Clockify and Notion: catalog.mjs builds a
"<App> (sample data)" entry per app, the same built app with store.proxy
answered in the frame by the mock proxy's provider fixture, already
connected, with writes persisted under the App and replayed on remount.
New -sample session plans say the data is invented so the analysis does
not log it as a finding.

Checks: money vitest for the statements, node --test for the sample
accounts, usertest/e2e/samples.mjs against the pinned atomic-server, and
run.mjs for the page's sample-file links.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Qt6FnBHXdJWx9GEKuAqVBo
@michielbdejong
michielbdejong merged commit 9da7555 into main Oct 1, 2026
19 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

usertest: synthetic sample data for sessions that shouldn't need a real account (money first)

2 participants