Skip to content

Add containerised tier-2 acceptance tests against a real remoteStorage server - #6

Merged
michielbdejong merged 4 commits into
mainfrom
claude/acceptance-tests
Sep 2, 2026
Merged

michielbdejong merged 4 commits into
mainfrom
claude/acceptance-tests

Conversation

@michielbdejong

Copy link
Copy Markdown
Member

Summary

Pilots a reusable acceptance-testing setup: a single containerised command that runs the same way on a laptop and in CI, and drives the real remoteStorage connect-and-sync flow rather than mocking it.

  • Tier 1 (existing, untouched): npm run test:smoke — fast, no backend.

  • Tier 2 (new): make accept / npm run test:accept — both just run scripts/accept.sh, which brings up three containers via docker-compose.acceptance.yml:

    • app — litewrite built and served (docker/app.Dockerfile)
    • remotestorage — a real armadietto server (remotestorage/armadietto-monolithic, password-auth "classic" variant, matching litewrite's OAuth flow)
    • tests — the official mcr.microsoft.com/playwright image (browsers pre-installed, so nothing tries to download them)

    All three image tags are pinned by digest, not just a mutable tag.

  • tests/acceptance/ covers, each against real server state rather than sleeps:

    • connecting to a remoteStorage account through the actual widget UI + OAuth round trip
    • creating a document and confirming it's persisted server-side (verified via a direct HTTP call to the storage API, not just trusting the UI)
    • a fresh browser session (no local cache) recovering the document from the server
    • an edit in one browser context appearing in a second, via remoteStorage.js's real background sync
  • .github/workflows/acceptance.yml runs the same make accept on PRs and uploads the Playwright report on failure. deploy.yml is untouched.

  • CLAUDE.md documents both tiers, where things live, the deploy target, and repo conventions.

Verification

Ran make accept end to end in the dev session and watched all 4 scenarios pass twice in a row from a clean build. Then deliberately pointed the suite at a wrong remoteStorage URL and confirmed it fails loudly (make exits non-zero with a clear ECONNREFUSED, not a silent pass) before reverting. Tier-1 smoke tests and lint (standard) still pass unmodified.

One infra note: the tests and app build steps optionally trust $NODE_EXTRA_CA_CERTS if it's set on the host (via a no-op-by-default volume mount / build secret) — harmless for a normal laptop or GitHub Actions, but needed to validate this behind this session's own TLS-inspecting sandbox proxy.

Test plan

  • make accept — all 4 acceptance scenarios pass
  • Re-ran make accept a second time from a clean build — still green
  • Pointed REMOTESTORAGE_URL at a wrong port and confirmed make accept fails loudly, then reverted
  • npm run test:smoke and npm run lint still pass

Generated by Claude Code

claude added 4 commits August 31, 2026 16:36
…e server

Tier 1 (npm run test:smoke) stays as the fast, backend-less Playwright
suite. This adds tier 2: `make accept` / `npm run test:accept` bring up
litewrite, a real armadietto remoteStorage server, and Playwright as
three docker-compose services, then drive the actual widget connect,
create/persist, reload-recovery, and cross-context sync flows through
the real OAuth round trip - no mocks, no third-party accounts.

Image tags are pinned by digest (node:20-alpine, the official
mcr.microsoft.com/playwright image so browsers don't need downloading,
and remotestorage/armadietto-monolithic). Verified locally: all four
scenarios pass end to end, and pointing the tests at a wrong storage URL
makes the suite fail loudly rather than silently passing.

Adds .github/workflows/acceptance.yml to run the same command on PRs,
and a CLAUDE.md documenting both tiers.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_014gnL38EmFRmq4MDicaJQ9B
CI failed with remotestorage never reporting healthy in time - most
likely starved of CPU/disk while the ~750MB Playwright image was being
pulled and extracted concurrently. Widen the healthcheck budget
(2s/2s/30 -> 3s/5s/40) and have accept.sh print `docker compose logs`
before tearing down on any failure, so the next red run shows the
containers' own output instead of just "is unhealthy".

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_014gnL38EmFRmq4MDicaJQ9B
The previous CI failure's own log dump showed both app and remotestorage
had actually started fine by the time compose gave up waiting on
healthchecks - so the healthcheck timeout bump alone didn't address the
real cause. The ~750MB Playwright image was still being pulled and
extracted concurrently with the healthcheck window, competing for the
runner's disk/CPU. Pulling it up front, before app/remotestorage start,
removes that contention instead of just widening the timeout further.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_014gnL38EmFRmq4MDicaJQ9B
Pre-pulling the tests image didn't change anything: both prior CI
failures showed app and remotestorage still exhausting all 40 retries
in almost exactly the same ~128s, which rules out disk/CPU contention
as the cause - a real slowdown wouldn't reproduce that precisely twice.
That points to the healthcheck itself failing outright rather than
timing out.

Likely culprit: `wget http://localhost:8000/` can resolve to ::1 first
on a dual-stack host. GitHub Actions runners are dual-stack; this
sandbox apparently isn't, which is why it never reproduced locally. The
services bind 0.0.0.0 (IPv4 only), so an IPv6-first resolution would
fail every probe. Also have the failure-path log dump include
`compose ps` and use `--timestamps`, so if this isn't it, the next
failure is diagnosable in one round instead of another guess.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_014gnL38EmFRmq4MDicaJQ9B
@michielbdejong
michielbdejong merged commit cce0519 into main Sep 2, 2026
2 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants