diff --git a/.context/decisions/0030-compute-pages-follow-adr-0010-phase-3.md b/.context/decisions/0030-compute-pages-follow-adr-0010-phase-3.md index 675967bc..2c76bc16 100644 --- a/.context/decisions/0030-compute-pages-follow-adr-0010-phase-3.md +++ b/.context/decisions/0030-compute-pages-follow-adr-0010-phase-3.md @@ -80,6 +80,21 @@ The captured fixtures in `test/fixtures/compute/` gain a `GET /compute/me` captu - **Shared copy.** Seven keys were added to nemar-cli's contract and transcribed (`compute_page.*`, `compute_access.granted.link`), and `compute_access.granted.body` no longer promises submission from the website: jobs are submitted with the command line. - **Still open from the hand-off above:** replacing `compute-terms.ts` and `COMPUTE_REQUESTS_ENABLED` with `access.current_terms_version` and `access.requests_open` is a follow-up; `/compute` already reads `requests_open` (falling back to `terms_published`) for its own "request" link. +## Addendum: the job page (phase E3, website#496, 2026-11-07) + +`/compute/jobs/` reads `GET /compute/me/jobs/:id` and, on request, the log, the result links and the cancel (nemar-cli Epic D, phase D3). The decisions that are expensive to reverse: + +- **Bearer result links are minted only by a click and live only in the DOM.** The server-rendered page carries no link and calls neither the log nor the results client (a source scan fails the build otherwise). "Get download links" and "Renew links" are the only callers; a link reaches an `href` only through `safeResultHref` (`https:`, or `http:` on loopback for the local backend), with `rel="noopener noreferrer"` and no referrer, and is never written to storage, the URL, the console, an attribute other than `href` or the clipboard. A page restored from the back-forward cache clears the list, and rows are replaced when the links' own expiry passes. The backend's `notice` about keeping them private is shown as served, not retyped. +- **The log is inert text, and hostile characters are shown, not removed.** ANSI formatting is stripped as formatting; everything else goes through `displaySafeBody` (ADR 0029), so a bare carriage return becomes a line break (nothing can be overwritten) and a control, override, isolate or zero-width character is written out (`\x1b`, `\u202e`). A sequence that never ends is not stripped and swallows nothing after it. Lines are capped at 2,000 characters. The backend serves the FIRST 1 MiB with `X-Nemar-Truncated: true`, so a long log shows its beginning and says so (nemar-cli#2094 asks for a tail). The proxy forwards that one header and its constant has one reader. +- **The page claims only what the record carries.** The job has three times (created, started, ended) and no per-stage times (nemar-cli#2097), so the timeline shows a time on three steps and a step it cannot vouch for reads "not known", never "done"; Archiving is never claimed for a failed or canceled job. No sentence says whose fault a failure was (the user view serves no attribution) or promises a refund. The 17 failure codes, the cancel outcomes, the refusal statuses, the header and the bounds are transcribed and drift-tested (`test/compute-job-contract-drift.test.ts`); the wording is website-owned (`compute-copy.ts`, `compute-failure.ts`), and the cancel dialog repeats the CLI's own charge rule. +- **Polling is cheap by construction.** The first read is 10 seconds after load; an unchanged answer doubles the delay to 60 (the backend refreshes a live job from the cluster at most once a minute, so asking faster adds load and no freshness), a change resets it, a fifth error in a row stops it, a hidden tab pauses it and a visible one refreshes at once, an ended or unknown state stops it for good, and it stops after two hours with a Resume button. The schedule is pure (`nextPoll`, an injected clock); `compute-job-live.ts` is the one job module that reads a clock or holds a timer, which `test/compute-dashboard-ui.test.ts` names as its single exception. +- **The retention date is derived until the server states it.** `ended_at` plus `limits.results_days` gives "kept until", with the backend's own strict `now > ended + days` rule; the 410 stays authoritative and the button stays until the backend says so (nemar-cli#2096 asks for the date). A page never promises a charge beyond the dialog's sentence and the refreshed record. +- **One 404, never an oracle.** An absent, foreign or malformed id is one page with status 404 (a malformed id without calling the API), the headers `Cache-Control: private, no-store` and `Referrer-Policy: no-referrer` are set before any return, and `/compute` joins `/oauth` in `routeBypassesEdgeCache`. + +Dev-server note: `bun run dev` (Vite) answers a URL that ends in `.txt` (here, a results cursor that is a path ending in `.txt`, passed as `?after=`) as a static file lookup and 500s. It was checked against a deployed build and the effect is dev-only, so there is no code change; the local seed uses `.dat` outputs. + +Review round (same PR): the job read and the cancel carry named deadlines longer than the backend work behind them (`COMPUTE_JOB_READ_TIMEOUT_MS` 12 s, `COMPUTE_CANCEL_TIMEOUT_MS` 20 s, against a cluster read of about 8 s), so a slow cluster shows the stored record with `stale: true` or the cancel's own outcome instead of a false error. What the page holds of the result links is a pure state machine (`compute-job-results-state.ts`), tested for the rules that matter: rows are dropped after `expires_at`, a back-forward-cache restore drops every link and brings the "Get download links" button back (it is derived from the job, not from what was shown last), and a 410 never returns the button. A plain-http result link is accepted only when the page itself is on a loopback host. Logs write out the three implicit direction marks too (an option of `displaySafeBody`; mail rendering is unchanged). + ## Addendum: the dataset page dialog (phase E5, website#498, 2026-11-07) The dataset page's "Compute instructions" dialog now describes `nemar compute` instead of an S3 prefix and a Neuroscience Gateway flow that never shipped. The decisions that are expensive to reverse: @@ -96,5 +111,5 @@ Receipts for this addendum: website#498, #493; nemar-cli#2021, #2024, #2090, #20 ## Receipts -- website#493 (epic), website#494 (phase E1), website#495 (phase E2), website#498 (phase E5); nemar-cli#2021 (Epic D), #2071 (phase D1, commit 3a869903). +- website#493 (epic), website#494 (phase E1), website#495 (phase E2), website#496 (phase E3), website#498 (phase E5); nemar-cli#2094 to #2097; nemar-cli#2021 (Epic D), #2071 (phase D1, commit 3a869903). - ADRs 0006, 0007, 0010, 0014, 0015, 0029; nemar-cli ADRs 0040, 0042, 0118. diff --git a/.context/local-backend.md b/.context/local-backend.md index 03cf6a6e..9ef63f53 100644 --- a/.context/local-backend.md +++ b/.context/local-backend.md @@ -94,3 +94,20 @@ Worked on 2026-11-07. Differences from production: the compute switch reads `off exact production environment), so Reconciliation shows its paused banner; no mail is sent, so a decision answers `email_sent: false` and the row says the person could not be emailed. Browser windows cannot be narrower than about 600 px in the Chrome tools; check 375 px in devtools. + +## The job page (website#496) + +`--seed-compute-jobs` (with `--seed-compute` and `LOCAL_BACKEND_JOBS=1`) makes the job page's jobs for +the account `localjobs` (sign in with the `jobs` cookie) through the real services and the Tapis +stand-in: a finished job (three outputs, a log), a running one, one to cancel, a failed job with a +hostile log, one with a log over 1 MiB, one with 1,002 outputs (a second page of links), one whose +results expire in two days, one whose results are past the 30 days (the real 410), a running job of +another account (the 404 check) and a job that advances by itself every 20 seconds (queued, running, +finished). Ids are printed and written to `.local-backend/jobs.json`. The seed runs the backend with +`ENVIRONMENT=development` (compute only runs in `production` or `development`); mail stays fenced. +The seed takes about 40 seconds, and the watch job starts advancing when it ends: open it at once. +Result links point at the local object store on `127.0.0.1`, which is why `safeResultHref` accepts +`http:` on loopback. Not reachable here: a `stale: true` job (needs the stand-in down at read time; +covered by the captured fixture) and a real 5xx. + +`bun run dev` (Vite) 500s a request URL that ends in `.txt`, such as a results cursor `?after=out/f.txt`; this is dev-server only (a deployed build was checked), so the seed names its outputs `.dat`. diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml index da73b676..f868e9e1 100644 --- a/.github/workflows/ci.yml +++ b/.github/workflows/ci.yml @@ -127,6 +127,11 @@ jobs: # shared/contract/compute-user.ts and compute.ts: the job states, front doors, usage groups, # page bounds and read-refusal words the /compute page branches on (phase E2). Same `dev` # pairing, strict, and red until the compute epic reaches nemar-cli `dev`. + # test/compute-job-contract-drift.test.ts is an ELEVENTH, against shared/compute-state.ts, + # shared/contract/compute-submit.ts and compute-responses.ts: the failure codes, the refusal + # statuses, the truncated-log header, the bounds and the cancel outcomes the job page + # (phase E3) branches on. Same `dev` pairing, strict, and red until the compute epic reaches + # nemar-cli `dev`. # test/compute-admin-contract-drift.test.ts is a TENTH: the compute state machine, the decision # words and the bounds the admin Compute tab branches on (compute-access-state.ts, # compute-state.ts, contract/compute-access.ts, contract/compute-admin.ts). Same `dev` pairing, @@ -141,6 +146,8 @@ jobs: /shared/contract/account-copy.ts /shared/contract/compute-access.ts /shared/contract/compute-admin.ts + /shared/contract/compute-responses.ts + /shared/contract/compute-submit.ts /shared/contract/compute-terms.ts /shared/contract/compute-user.ts /shared/contract/compute.ts @@ -160,6 +167,8 @@ jobs: NEMAR_CLI_ACCOUNT_COPY: nemar-cli-contract/shared/contract/account-copy.ts NEMAR_CLI_COMPUTE_ACCESS_CONTRACT: nemar-cli-contract/shared/contract/compute-terms.ts NEMAR_CLI_COMPUTE_CONTRACT: nemar-cli-contract/shared/contract/compute.ts + NEMAR_CLI_COMPUTE_RESPONSES_CONTRACT: nemar-cli-contract/shared/contract/compute-responses.ts + NEMAR_CLI_COMPUTE_SUBMIT_CONTRACT: nemar-cli-contract/shared/contract/compute-submit.ts NEMAR_CLI_COMPUTE_STATE: nemar-cli-contract/shared/compute-state.ts NEMAR_CLI_COMPUTE_USER_CONTRACT: nemar-cli-contract/shared/contract/compute-user.ts NEMAR_CLI_COMPUTE_ADMIN_CONTRACT: nemar-cli-contract/shared/contract/compute-admin.ts diff --git a/AGENTS.md b/AGENTS.md index b76133f2..d52b7b98 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -77,6 +77,7 @@ src/ upload.astro upload/success.astro upload flow (dropzone + BIDS pre-check + full BIDS validation in a worker + direct-to-storage PUTs) settings.astro account: name/email/ORCID/GitHub/profile self-service; CLI keys card; compute access card compute/index.astro compute dashboard (app host, ADR 0030): access, balances, usage chart per pool, jobs with Load more, queue waits; each read and panel fails alone + compute/jobs/[id].astro one compute job (app host, ADR 0030 E3): timeline, resources and charge, failure sentence, log and result links on request (bearer links minted only by a click), cancel; private no-store, one 404 for absent, foreign or malformed ids; the page the results-ready mail links to admin/publication-requests.astro admin-only (role=admin; 404s for others) admin/mail.astro admin/mail/{[id],queues,away}.astro admin Mail tab: tickets, queues, away windows (ADR 0029); email is escaped text only admin/compute.astro admin Compute tab (website#497, ADR 0030): Requests (grant, deny, suspend, revoke by id, each through ComputeDecisionDialog) and read-only Pools, Usage, Flagged jobs, Reconciliation; a section the backend does not serve says so and links its nemar-cli issue; nothing owner-only or cost-bearing is offered @@ -98,6 +99,8 @@ src/ compute-view.ts compute-labels.ts compute-badges.ts pure view models (panelsFor, access, balances, queue, jobRowView, usage), the page's short labels (sentences live in account-copy.ts), job-state badges compute-format.ts compute-chart.ts integer milli-SU formatting (`suFixed`: the CLI's three-decimal digits; `formatSu`: the shortest exact decimal, the same rule as nemar-cli's `formatSu`) and the usage chart geometry (nice ceiling from the stacked total, hand-rolled SVG, ADR 0007) compute-jobs-dom.ts builds appended job rows with createElement/textContent only + compute-job-view.ts compute-logs.ts compute-failure.ts compute-copy.ts the job page's pure modules (E3): timeline, canCancel, retention window, results rows, cancel outcome words and the polling schedule (`nextPoll`, no clock); `prepareLog` (ANSI stripped, hostile characters written out, long lines capped); one sentence per failure code; the website-owned sentences + compute-job-dom.ts compute-job-page.ts compute-job-live.ts the job page's browser side: nodes from the same view models (createElement and textContent only), the controller (poll, log, results, cancel), and the ONE job module that reads a clock or holds a timer compute-admin-api.ts admin Compute client (browser-safe): access list and decisions by id, pools, jobs by flag, unmatched accounting rows, access report, status, the badge count; { error, code } reader, decision failure classes, no POST retry compute-admin-contract.ts hand copy of the closed lists the Compute tab branches on (drift-tested, test/compute-admin-contract-drift.test.ts, strict via NEMAR_CLI_COMPUTE_ADMIN_CONTRACT) compute-admin-view.ts pure view models for the Compute tab: request row, allowed buttons, decision and failure sentences, pool and job rows, parameters, pager @@ -128,7 +131,7 @@ src/ eeg-viewer.css global signal-viewer styles (runtime-built DOM), imported by BidsTree and the embed test/ fixtures/ qa-aggregates, qa-file-dataqual, qa-hed-summary (Phase 3) - fixtures/compute/ captured from the real nemar-cli routes, never hand-written: `NEMAR_CLI_DIR= bun scripts/capture-compute-fixtures.mjs` (access, E1) and `bun scripts/capture-compute-read-fixtures.mjs` (me, usage, jobs, errors; E2); `fixtures/compute/cli-help/` is the real `nemar ... --help-all` text the dialog's commands are checked against: `NEMAR_CLI_DIR= bun scripts/capture-compute-cli-help.mjs` (E5) + fixtures/compute/ captured from the real nemar-cli routes, never hand-written: `NEMAR_CLI_DIR= bun scripts/capture-compute-fixtures.mjs` (access, E1), `bun scripts/capture-compute-read-fixtures.mjs` (me, usage, jobs, errors; E2) and `bun scripts/capture-compute-job-fixtures.mjs` (job, log, results, cancel, refusals; E3, on the backend's lifecycle fixture and Tapis stand-in); `fixtures/compute/cli-help/` is the real `nemar ... --help-all` text the dialog's commands are checked against: `NEMAR_CLI_DIR= bun scripts/capture-compute-cli-help.mjs` (E5) public/ static logos + brain hero assets (og/ cards are generated, gitignored) public/brand/ GENERATED press logo kit (SVG, PDF, transparent PNG; color, reverse, black, white); rebuild with `bun scripts/generate-brand-kit.mjs`, never hand-edit public/vendor/bids-validator// GENERATED worker bundle of the BIDS validator + SHA256SUMS + provenance + licenses (ADR 0028); never hand-edited diff --git a/CHANGELOG.md b/CHANGELOG.md index 457d6749..3760a346 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -15,6 +15,16 @@ The release pull request from `staging` to `main` moves those entries under the ### Added +- **The compute job page** (website#496, epic website#493 phase E3; nemar-cli epic #2021 phase D3, website ADR 0030 addendum). + `/compute/jobs/` shows one job: its state on a timeline that claims only what the record carries (three times), its resources and charge (the same three-decimal digits as `nemar compute status`), a plain sentence for each of the 17 failure codes, its log on request, its result links on request (renewable), and a cancel for a live job. + A live job refreshes itself every 10 seconds, backing off to 60 while nothing changes, pausing in a hidden tab and stopping at an ended state. + The log is shown as inert text with hostile characters written out (`\x1b`, `\u202e`) rather than silently dropped; a log cut at 1 MiB says so and points at `nemar compute logs` (the end of a long log cannot be read yet: nemar-cli#2094). + Result links are bearer links: viewing the page mints none, only the button does, and they are never stored, logged or copied. + The page is the one the "results ready" email links to, answers the same 404 for an absent, foreign or malformed id, and is never edge-cached (`/compute` joins `/oauth` in `routeBypassesEdgeCache`). + The API proxy forwards one more response header, `X-Nemar-Truncated`. + Backend gaps filed: nemar-cli#2094 (log tail), #2095 (links as attachments), #2096 (state when results are deleted), #2097 (per-stage times). + `scripts/capture-compute-job-fixtures.mjs` captures nine fixture files from the real router, and `scripts/local-backend.mjs --seed-compute-jobs` seeds jobs through the real services and the Tapis stand-in. + - **The `/compute` dashboard** (website#495, epic website#493 phase E2; nemar-cli epic #2021 phase D2, website ADR 0030 addendum). A signed-in person sees their compute access, CPU and GPU balances (granted, held, charged, available, in the same three-decimal service units `nemar compute balance` prints), a daily usage chart per pool stacked by partition (hand-rolled SVG, themed by new `--chart-*` tokens, with the numbers in a table beside it), their jobs from every front door with "Load more", and the platform's recent queue waits ("Not enough data" with the count when fewer than five jobs started). A person without active access sees why and how to ask, not an error; a suspended or ended account keeps its history read-only; one failed read shows one notice that says the figures are missing, not zero. diff --git a/scripts/capture-compute-job-fixtures.mjs b/scripts/capture-compute-job-fixtures.mjs new file mode 100644 index 00000000..fef18085 --- /dev/null +++ b/scripts/capture-compute-job-fixtures.mjs @@ -0,0 +1,537 @@ +/** + * Capture the compute JOB PAGE fixtures from the REAL nemar-cli backend routes (website ADR 0030 and + * its phase E3 addendum; epic nemarOrg/website#493, issue #496). + * + * NEMAR_CLI_DIR= \ + * bun scripts/capture-compute-job-fixtures.mjs + * + * Captured from nemar-cli commit 27eecff9 (Epic D, `feature/issue-2021-epic-compute-users`; phases D1 + * to D4 and the copy keys are in it). The checkout needs `bun install` at its root and in `backend/`. + * + * Nothing is stood in for. The script runs outside `bun test` (step 0 of the phase plan: it can) on the + * backend's own lifecycle fixture (`backend/test/helpers/compute-lifecycle-fixture.ts`): real migrated + * SQLite, Miniflare's R2 for the results and records buckets, the real local S3 interface the signer + * addresses, and Epic A's Tapis stand-in (a real HTTP server built from Tapis's published + * specifications; no response of it was recorded from a real tenant). The REAL compute router + * (`makeRoutesApp`) is called with real hashed API keys, and the bodies are written unchanged. + * Two kinds of job are read: + * - rows written by `insertJob` with fixed times (the states, the failure codes, a recorded cancel): + * these rerun to the same bytes; + * - jobs taken through the real lifecycle (`makeJob`, `submitComputeJob`, `advanceComputeJob`, + * `playLauncher`, which plays the launcher's last step and is documented in the fixture): these + * carry fresh ids, times and presigned signatures on every run, so a rerun rewrites those files. + * Tests therefore read the identifiers out of the fixture, never a literal. + * + * Writes under test/fixtures/compute/: + * job-states.json GET /compute/me/jobs/:id, one job in each of the ten states + * job-failure-codes.json the same, one failed job per failure code (17) + * job-cancel-requested.json a running job with a cancel recorded + * job-stale.json a live job whose cluster read failed: `stale: true` + * job-lifecycle.json a job read at running and again at finished, through the real lifecycle + * logs.json GET /compute/jobs/:id/logs: live, ended (records copy), empty, hostile, + * over 1 MiB (headers and length only), not available yet + * results.json GET and POST renew of a finished job, a second page by cursor (a job with + * more outputs than one page; the first page is summarized, not stored), + * the expired and the not-finished refusals + * cancel.json POST /compute/jobs/:id/cancel: each outcome the route gives + * errors-job.json job_not_found for a foreign, an absent and a malformed id, on every + * job route; the credential refusals + * + * Every presigned address points at the local object-store stand-in and carries a test signature; the + * emails are @example.org; no body contains an identity. + */ + +import { mkdir, writeFile } from "node:fs/promises"; +import { dirname, join, resolve } from "node:path"; +import { fileURLToPath } from "node:url"; + +const NEMAR_CLI_DIR = process.env.NEMAR_CLI_DIR; +if (!NEMAR_CLI_DIR) { + console.error("Set NEMAR_CLI_DIR to a checkout of nemar-cli's Epic D branch."); + process.exit(1); +} +const root = resolve(NEMAR_CLI_DIR); +const here = dirname(fileURLToPath(import.meta.url)); +const OUT = resolve(here, "..", "test", "fixtures", "compute"); + +const lifecycle = await import(join(root, "backend/test/helpers/compute-lifecycle-fixture.ts")); +const { makeRoutesApp } = await import(join(root, "backend/test/helpers/compute-routes-app.ts")); +const { insertJob, personWithKey } = await import( + join(root, "backend/test/helpers/compute-user-seed.ts") +); +const { submitComputeJob, advanceComputeJob } = await import( + join(root, "backend/src/services/compute-lifecycle.ts") +); +const { hashApiKey } = await import(join(root, "backend/src/services/token.ts")); +const { COMPUTE_STATES, COMPUTE_FAILURE_CODES } = await import( + join(root, "shared/compute-state.ts") +); +const { COMPUTE_LOG_MAX_BYTES, COMPUTE_RESULTS_PAGE_MAX } = await import( + join(root, "shared/contract/compute-submit.ts") +); +const { startLifecycleFixture, makeJob, playLauncher, reread, setTapisTimes, MINUTE } = lifecycle; + +const COMMIT = "27eecff9"; +const COMMAND = "NEMAR_CLI_DIR= bun scripts/capture-compute-job-fixtures.mjs"; + +const provenance = (route, request, note) => ({ + command: COMMAND, + nemar_cli_commit: COMMIT, + route, + request, + note, +}); + +const fx = await startLifecycleFixture(); +await fx.reset(); +const handle = makeRoutesApp(); +const db = fx.harness.db; + +let keySeq = 0; +async function keyFor(accountId) { + const key = `capture-job-key-${++keySeq}-0123456789abcdef0123456789abcdef`; + db.query("INSERT INTO tokens (user_id, api_key_hash, api_key_prefix) VALUES (?, ?, ?)").run( + accountId, + await hashApiKey(key), + key.slice(0, 8), + ); + return key; +} + +/** The headers a page or a test reads; the rest are transport. */ +const KEPT_HEADERS = [ + "content-type", + "cache-control", + "x-content-type-options", + "x-nemar-truncated", +]; + +/** One request through the real router; `text` bodies are kept as text, JSON as parsed. */ +async function call(method, path, key, over = {}) { + const headers = {}; + if (key) headers.Authorization = `Bearer ${key}`; + const res = await handle( + new Request(`http://api.test/compute${path}`, { method, headers }), + fx.env(over.env ?? {}), + ); + const bytes = new Uint8Array(await res.arrayBuffer()); + const text = new TextDecoder().decode(bytes); + const kept = {}; + for (const name of KEPT_HEADERS) { + const v = res.headers.get(name); + if (v !== null) kept[name] = v; + } + let body; + if ((res.headers.get("content-type") ?? "").includes("json")) body = JSON.parse(text); + return { status: res.status, headers: kept, bytes, text, body }; +} + +/** The shape every fixture stores for a JSON answer: the status and the body, unchanged. */ +const asJson = ({ status, body }) => ({ status, body }); +/** And for a log: the status, the kept headers and the text. */ +const asText = ({ status, headers, text }) => ({ + status, + headers, + ...(status === 200 ? { text } : { body: JSON.parse(text) }), +}); + +// --- Insert-based jobs: fixed times, so the bytes rerun the same ----------------------------------- + +const owner = await personWithKey(db, { access: "active" }); +const ownerKey = owner.key; +const T = { + created: "2026-11-05 09:00:00", + started: "2026-11-05 09:10:00", + ended: "2026-11-05 09:50:00", +}; +const readJob = async (id, key = ownerKey) => call("GET", `/me/jobs/${id}`, key); + +const jobStates = { + _provenance: provenance( + "GET /compute/me/jobs/:id", + "one job in each of the ten states, by id", + "Real route over rows written with fixed times. A live job has no Tapis job, so nothing is refreshed and `stale` is absent. Times are the database's `YYYY-MM-DD HH:MM:SS` UTC. The route serves no name, no fault attribution and no hold.", + ), +}; +for (const state of COMPUTE_STATES) { + const terminal = ["finished", "failed", "cancelled"].includes(state); + const ran = ["running", "archiving", "finished", "failed"].includes(state); + const id = insertJob(db, { + account_id: owner.id, + state, + created_at: T.created, + started_at: ran ? T.started : null, + ended_at: terminal ? T.ended : null, + charged_milli: state === "finished" ? 2_500 : state === "failed" ? 1_250 : 0, + partition: "shared", + memory_mb: state === "finished" ? 8192 : null, + }); + jobStates[state] = asJson(await readJob(id)); +} + +const jobFailureCodes = { + _provenance: provenance( + "GET /compute/me/jobs/:id", + "one failed job per failure code (COMPUTE_FAILURE_CODES, 17)", + "Real route over rows written with fixed times. The user view carries `failure_code` only: `failure_detail` and `platform_failure` are not served.", + ), +}; +for (const code of COMPUTE_FAILURE_CODES) { + const id = insertJob(db, { + account_id: owner.id, + state: "failed", + failure_code: code, + created_at: T.created, + started_at: T.started, + ended_at: T.ended, + charged_milli: 500, + }); + jobFailureCodes[code] = asJson(await readJob(id)); +} + +const cancelRequestedId = insertJob(db, { + account_id: owner.id, + state: "running", + created_at: T.created, + started_at: T.started, + cancel_requested_at: "2026-11-05 09:20:00", +}); +const jobCancelRequested = { + _provenance: provenance( + "GET /compute/me/jobs/:id", + "a running job with a cancel recorded", + "Real route over a row written with fixed times: `cancel_requested` is true while the state is still live.", + ), + running: asJson(await readJob(cancelRequestedId)), +}; + +// --- Lifecycle jobs: the real services, the Tapis stand-in, the launcher's last step played ----------- + +const HOSTILE_LOG = [ + "plain line", + "\u001b[31mred\u001b[0m and \u001b[2K cleared", + "link \u001b]8;;https://example.org\u0007text\u001b]8;;\u0007 end", + "progress 10%\rprogress 50%\rprogress 100%", + "nul \u0000 bell \u0007 next-line \u0085 end", + "override ‮gnp.exe and isolate ⁦x⁩ and zero​width", + ' & "quotes" back\\slash', + "Traceback (most recent call last):", + ' File "main.py", line 3, in ', + "ZeroDivisionError: division by zero", + "", +].join("\n"); + +const TRACEBACK = [ + "Traceback (most recent call last):", + ' File "main.py", line 3, in ', + " print(1 / 0)", + "ZeroDivisionError: division by zero", + "", +].join("\n"); + +async function owned(over = {}) { + const made = await makeJob(fx, over); + return { ...made, key: await keyFor(made.accountId) }; +} + +async function submitted(over = {}) { + const j = await owned(over); + const sub = await submitComputeJob(j.env, j.job.id); + if (!sub.ok) throw new Error(`could not submit: ${sub.outcome}`); + const uuid = (await reread(j.env, j.job.id)).tapis_uuid; + return { ...j, uuid }; +} + +async function running(over = {}) { + const j = await submitted(over); + fx.tapis.advance(j.uuid, "QUEUED"); + fx.tapis.advance(j.uuid, "RUNNING"); + await advanceComputeJob(j.env, j.job.id); + return j; +} + +async function finished(outputs, log) { + const j = await running(); + if (log !== undefined) fx.tapis.setOutput(j.uuid, "tapisjob.out", log); + await playLauncher(fx, j.env, await reread(j.env, j.job.id), j.uuid, { outputs }); + fx.tapis.advance(j.uuid, "FINISHED"); + setTapisTimes(fx, j.uuid, 30 * MINUTE, 0); + await advanceComputeJob(j.env, j.job.id); + const job = await reread(j.env, j.job.id); + if (job.state !== "finished") throw new Error(`the job is ${job.state}, not finished`); + return j; +} + +// A job seen at running, then at finished. +const lifecycleJob = await running(); +const lifecycleRunning = asJson(await readJob(lifecycleJob.job.id, lifecycleJob.key)); +const lifecycleLog = await call("GET", `/jobs/${lifecycleJob.job.id}/logs`, lifecycleJob.key); +fx.tapis.setOutput(lifecycleJob.uuid, "tapisjob.out", "step one\nstep two\n"); +const logLive = asText(await call("GET", `/jobs/${lifecycleJob.job.id}/logs`, lifecycleJob.key)); +await playLauncher( + fx, + lifecycleJob.env, + await reread(lifecycleJob.env, lifecycleJob.job.id), + lifecycleJob.uuid, + { + outputs: [ + { path: "report.txt", content: "the report\n" }, + { path: "figures/plot.json", content: '{"x":[1,2,3]}' }, + { path: "table.csv", content: "a,b\n1,2\n" }, + ], + }, +); +fx.tapis.advance(lifecycleJob.uuid, "FINISHED"); +setTapisTimes(fx, lifecycleJob.uuid, 30 * MINUTE, 0); +await advanceComputeJob(lifecycleJob.env, lifecycleJob.job.id); +const lifecycleFinished = asJson(await readJob(lifecycleJob.job.id, lifecycleJob.key)); +const jobLifecycle = { + _provenance: provenance( + "GET /compute/me/jobs/:id", + "one job through makeJob, submitComputeJob, QUEUED, RUNNING, FINISHED", + "Real route over a job taken through the real lifecycle against the Tapis stand-in; the launcher's last step (its files and `declared.json`) is played by the backend's own helper. Ids and times are of the run that wrote this file.", + ), + job_id: lifecycleJob.job.id, + running: lifecycleRunning, + finished: lifecycleFinished, +}; + +// A stale read: a live job whose stored read is older than a minute and a cluster that cannot answer. +const staleJob = await submitted(); +db.query( + "UPDATE compute_jobs SET tapis_synced_at = datetime('now', '-10 minutes') WHERE id = ?", +).run(staleJob.job.id); +fx.tapis.down(); +const staleRead = asJson(await readJob(staleJob.job.id, staleJob.key)); +fx.tapis.up(); +const jobStale = { + _provenance: provenance( + "GET /compute/me/jobs/:id", + "a submitted job whose last cluster read is ten minutes old, with the Tapis stand-in down", + "Real route: the refresh failed, so the stored record is served with `stale: true`.", + ), + job_id: staleJob.job.id, + submitted: staleRead, +}; + +// --- Logs --------------------------------------------------------------------------------------------- + +const logsHostileJob = await running(); +fx.tapis.setOutput(logsHostileJob.uuid, "tapisjob.out", HOSTILE_LOG); +const logHostile = asText( + await call("GET", `/jobs/${logsHostileJob.job.id}/logs`, logsHostileJob.key), +); + +const logsEmptyJob = await running(); +fx.tapis.setOutput(logsEmptyJob.uuid, "tapisjob.out", ""); +const logEmpty = asText(await call("GET", `/jobs/${logsEmptyJob.job.id}/logs`, logsEmptyJob.key)); + +const logsBigJob = await running(); +fx.tapis.setOutput(logsBigJob.uuid, "tapisjob.out", "x".repeat(COMPUTE_LOG_MAX_BYTES + 4096)); +const big = await call("GET", `/jobs/${logsBigJob.job.id}/logs`, logsBigJob.key); +const logTruncated = { + status: big.status, + headers: big.headers, + byte_length: big.bytes.length, + note: "Headers and length only: the body is the first 1 MiB of 'x', which the test rebuilds.", +}; + +const notStarted = await owned(); +const logNotAvailable = asText( + await call("GET", `/jobs/${notStarted.job.id}/logs`, notStarted.key), +); + +// An ended job: the log is read from the records copy the lifecycle made when the job ended. +const endedWithLog = await finished([{ path: "report.txt", content: "the report\n" }], TRACEBACK); +const logEnded = asText(await call("GET", `/jobs/${endedWithLog.job.id}/logs`, endedWithLog.key)); + +const logs = { + _provenance: provenance( + "GET /compute/jobs/:id/logs", + "a running job (from Tapis), an empty log, a hostile log, a log over 1 MiB, a job that has not started, an ended job", + "Real route. The body is the job's own bytes as `text/plain`, `nosniff`, `no-store`; `X-Nemar-Truncated: true` only when cut at 1 MiB. The route serves the FIRST 1 MiB and has no `tail` (nemar-cli#2094). The hostile log's bytes are written to the Tapis stand-in's output and read back through the route.", + ), + live: logLive, + live_before_output: asText(lifecycleLog), + ended: logEnded, + empty: logEmpty, + hostile: logHostile, + truncated: logTruncated, + not_available: logNotAvailable, +}; + +// --- Results ------------------------------------------------------------------------------------------ + +const resultsGet = asJson( + await call("GET", `/jobs/${lifecycleJob.job.id}/results`, lifecycleJob.key), +); +const resultsRenew = asJson( + await call("POST", `/jobs/${lifecycleJob.job.id}/results/renew`, lifecycleJob.key), +); + +// More outputs than one page: the first page is summarized, the second is stored whole. +const OUTPUTS = COMPUTE_RESULTS_PAGE_MAX + 2; +const manyOutputs = Array.from({ length: OUTPUTS }, (_, i) => ({ + path: `out/file-${String(i).padStart(5, "0")}.txt`, + content: `n${i}\n`, +})); +const many = await finished(manyOutputs); +const firstPage = await call("GET", `/jobs/${many.job.id}/results`, many.key); +const cursor = firstPage.body.next_after; +const secondPage = await call( + "GET", + `/jobs/${many.job.id}/results?after=${encodeURIComponent(cursor)}`, + many.key, +); +const resultsManyFirst = { + status: firstPage.status, + summary: { + file_count: firstPage.body.files.length, + total: firstPage.body.total, + truncated: firstPage.body.truncated, + next_after: firstPage.body.next_after, + bearer_links: firstPage.body.bearer_links, + notice: firstPage.body.notice, + expires_at: firstPage.body.expires_at, + first_path: firstPage.body.files[0].path, + last_path: firstPage.body.files.at(-1).path, + note: "The files array (1,000 presigned links) is elided; the second page below is stored whole.", + }, +}; + +const notFinished = await running(); +const resultsBadState = asJson( + await call("GET", `/jobs/${notFinished.job.id}/results`, notFinished.key), +); + +const expiredJob = await finished([{ path: "report.txt", content: "the report\n" }]); +db.query("UPDATE compute_jobs SET ended_at = datetime('now', '-40 days') WHERE id = ?").run( + expiredJob.job.id, +); +const resultsExpired = asJson( + await call("GET", `/jobs/${expiredJob.job.id}/results`, expiredJob.key), +); +const resultsExpiredRenew = asJson( + await call("POST", `/jobs/${expiredJob.job.id}/results/renew`, expiredJob.key), +); + +const failedJob = await owned(); +db.query( + "UPDATE compute_jobs SET state = 'failed', failure_code = 'runtime_error', ended_at = datetime('now') WHERE id = ?", +).run(failedJob.job.id); +const resultsFailed = asJson(await call("GET", `/jobs/${failedJob.job.id}/results`, failedJob.key)); + +const results = { + _provenance: provenance( + "GET|POST /compute/jobs/:id/results[/renew]", + "a finished job with three outputs; a job with more outputs than one page (cursor); a job that is not finished; a job whose retention passed; a failed job", + "Real routes. Both verbs mint NEW 7-day presigned links (a renewal is the same handler). Presigned addresses point at the local object-store stand-in and carry test signatures that expire. `bearer_links` and `notice` come from the route. Files are in path order. The retention rule is the backend's: expired once `now > ended_at + results_days` (the job in `expired` had its `ended_at` moved 40 days back).", + ), + job_id: lifecycleJob.job.id, + get: resultsGet, + renew: resultsRenew, + many_job_id: many.job.id, + many_first_page: resultsManyFirst, + many_second_page: asJson(secondPage), + bad_state: resultsBadState, + expired: resultsExpired, + expired_renew: resultsExpiredRenew, + failed: resultsFailed, +}; + +// --- Cancel ------------------------------------------------------------------------------------------- + +const toCancel = await running(); +const cancelRunning = asJson(await call("POST", `/jobs/${toCancel.job.id}/cancel`, toCancel.key)); +const cancelAgain = asJson(await call("POST", `/jobs/${toCancel.job.id}/cancel`, toCancel.key)); +const afterCancel = asJson(await readJob(toCancel.job.id, toCancel.key)); +// The cluster cannot be reached: the intent is recorded and the job is still live. +const cancelPending = await running(); +fx.tapis.down(); +const cancelClusterDown = asJson( + await call("POST", `/jobs/${cancelPending.job.id}/cancel`, cancelPending.key), +); +const cancelClusterDownAgain = asJson( + await call("POST", `/jobs/${cancelPending.job.id}/cancel`, cancelPending.key), +); +const readClusterDown = asJson(await readJob(cancelPending.job.id, cancelPending.key)); +fx.tapis.up(); +const neverStarted = await owned(); +const cancelNeverStarted = asJson( + await call("POST", `/jobs/${neverStarted.job.id}/cancel`, neverStarted.key), +); +const cancelEnded = asJson( + await call("POST", `/jobs/${endedWithLog.job.id}/cancel`, endedWithLog.key), +); +const cancel = { + _provenance: provenance( + "POST /compute/jobs/:id/cancel", + "a running job twice, a running job while the cluster cannot be reached, a job that never started, a finished job", + "Real route, no body, no key. A cancel never refuses by state: an ended job answers 200 `already_ended` with `cancelled: false`. The Tapis stand-in carries out the cancel of a running job at once, so the state is already `cancelled`; with the stand-in down the intent is recorded and the job stays live (`cluster_down`).", + ), + running: cancelRunning, + running_again: cancelAgain, + running_read_after: afterCancel, + cluster_down: cancelClusterDown, + cluster_down_again: cancelClusterDownAgain, + cluster_down_read_after: readClusterDown, + never_started: cancelNeverStarted, + finished: cancelEnded, +}; + +// --- Refusals ----------------------------------------------------------------------------------------- + +const stranger = await personWithKey(db, { access: "active" }); +const strangerKey = stranger.key; +const pending = await personWithKey(db, { status: "pending", access: "none" }); +const ABSENT = "job_000000000000000000000000"; +const jobRoutes = [ + ["GET", (id) => `/me/jobs/${id}`], + ["GET", (id) => `/jobs/${id}/logs`], + ["GET", (id) => `/jobs/${id}/results`], + ["POST", (id) => `/jobs/${id}/results/renew`], + ["POST", (id) => `/jobs/${id}/cancel`], +]; +const notFound = {}; +for (const [method, path] of jobRoutes) { + const label = `${method} ${path("")}`; + notFound[label] = { + foreign: asJson(await call(method, path(lifecycleJob.job.id), strangerKey)), + absent: asJson(await call(method, path(ABSENT), ownerKey)), + malformed: asJson(await call(method, path("nope"), ownerKey)), + }; +} +const errorsJob = { + _provenance: provenance( + "the job routes", + "a foreign job, an absent id and a malformed id on every job route; no credential; an account that is not approved", + "Real refusals. A foreign, an absent and a malformed id answer the same bytes on every route (no route is an oracle). The 401 and 403 are authMiddleware's own shapes.", + ), + job_not_found: notFound, + no_credential: asJson(await call("GET", `/me/jobs/${ABSENT}`, null)), + account_not_approved: asJson(await call("GET", `/me/jobs/${ABSENT}`, pending.key)), + invalid_after: asJson( + await call("GET", `/jobs/${many.job.id}/results?after=${"x".repeat(2000)}`, many.key), + ), +}; + +// --- Write -------------------------------------------------------------------------------------------- + +const files = { + "job-states.json": jobStates, + "job-failure-codes.json": jobFailureCodes, + "job-cancel-requested.json": jobCancelRequested, + "job-stale.json": jobStale, + "job-lifecycle.json": jobLifecycle, + "logs.json": logs, + "results.json": results, + "cancel.json": cancel, + "errors-job.json": errorsJob, +}; +await mkdir(OUT, { recursive: true }); +for (const [name, body] of Object.entries(files)) { + await writeFile(join(OUT, name), `${JSON.stringify(body, null, 2)}\n`); +} +console.log(`wrote ${Object.keys(files).length} files under ${OUT}`); +await fx.dispose(); +process.exit(0); diff --git a/scripts/local-backend.mjs b/scripts/local-backend.mjs index a8f7ba86..d18e1f3a 100644 --- a/scripts/local-backend.mjs +++ b/scripts/local-backend.mjs @@ -37,6 +37,16 @@ * PUBLIC_API_BASE_URL has the `@nemar.cpending` mock for that screen). The * loopback guards below are unchanged. * + * --seed-compute-jobs with --seed-compute and LOCAL_BACKEND_JOBS=1, also makes the job page's + * (website phase E3, #496) jobs for `localjobs` through the REAL services and the + * Tapis stand-in: a finished job (three outputs, a log), a running one, a second + * running one to cancel, a hostile log, a log over 1 MiB, a failed job, a finished + * job with 1,002 outputs, one whose results expire soon and one whose results + * expired, a job that is queued and then advances by itself (queued, running, + * finished, every 20 s), and a running job of another account (the 404 check). The + * ids are printed and written to `.local-backend/jobs.json`. No route is stood in + * for; the launcher's last step is played by the backend's own helper. + * * It also seeds what the admin Compute tab (website#497) reviews: an owner and a * non-owner admin who are signed in, a queue of people who asked for compute * access (varied countries, a purpose full of markup, a service account, the @@ -113,6 +123,7 @@ const { default: dailyDoc } = await import( // Job rows need the backend's lifecycle fixture (Miniflare's R2 and the Tapis stand-in). const WITH_JOBS = process.env.LOCAL_BACKEND_JOBS === "1"; +const SEED_COMPUTE_JOBS = process.argv.includes("--seed-compute-jobs"); const lifecycle = WITH_JOBS ? await import(join(root, "backend/test/helpers/compute-lifecycle-fixture.ts")) : null; @@ -138,7 +149,9 @@ const STAMPS = { requested: "2026-10-20 09:30:00", granted: "2026-10-24 14:05:00 const db = fx ? fx.harness.db : freshDb(); const env = { ...(fx ? fx.env() : { DB: realD1(db) }), - ENVIRONMENT: "test", + // Compute runs only in `production` or `development` (the backend's switch), so the job page's seed + // needs `development`; both are non-production and fenced from real mail. + ENVIRONMENT: SEED_COMPUTE_JOBS ? "development" : "test", // Decisions are bookkeeping and read no switch, but minting an observer run does. COMPUTE_ENABLED: "1", API_BASE_URL: `http://localhost:${PORT}`, @@ -253,6 +266,9 @@ const PERSONAS = [ { who: "noorcid", compute_access: "none", identity: false }, ]; const SEED_COMPUTE = process.argv.includes("--seed-compute"); +if (SEED_COMPUTE_JOBS && !(SEED_COMPUTE && WITH_JOBS)) { + refuse("--seed-compute-jobs needs --seed-compute and LOCAL_BACKEND_JOBS=1."); +} if (SEED_COMPUTE) { const active = { compute_access: "active", @@ -263,6 +279,8 @@ if (SEED_COMPUTE) { { who: "rich", ...active }, { who: "frozen", ...active }, { who: "overrun", ...active }, + // The job page's account (--seed-compute-jobs): no other live jobs, so the concurrency limit has room. + ...(SEED_COMPUTE_JOBS ? [{ who: "jobs", ...active }] : []), // Asks, then is granted through the real admin routes below. { who: "granted", compute_access: "requested", compute_access_requested_at: STAMPS.requested }, ); @@ -359,11 +377,12 @@ if (server.hostname !== "127.0.0.1") { refuse(`the server bound ${server.hostname}, not 127.0.0.1.`); } - /** The compute dashboard's data (--seed-compute): the backend's own seed helpers and ledger services. */ async function seedComputeData() { const seeds = await import(join(root, "backend/test/helpers/compute-user-seed.ts")); - const { fundAccount, fundPlatform } = await import(join(root, "backend/test/helpers/compute-seed.ts")); + const { fundAccount, fundPlatform } = await import( + join(root, "backend/test/helpers/compute-seed.ts") + ); const { placeHold } = await import(join(root, "backend/src/services/compute-ledger.ts")); const { COMPUTE_STATES } = await import(join(root, "shared/compute-state.ts")); const { insertJob, jobIdFor, chargeAt, correctionAt, sqliteAt } = seeds; @@ -445,7 +464,9 @@ async function seedComputeData() { pool: i % 3 === 0 ? "gpu" : "cpu", gpus: i % 3 === 0 ? 1 : 0, created_at: ago(1, i), - started_at: ["running", "archiving", "finished", "failed"].includes(state) ? ago(1, i - 1) : null, + started_at: ["running", "archiving", "finished", "failed"].includes(state) + ? ago(1, i - 1) + : null, ended_at: terminal ? ago(1, i - 2) : null, cancel_requested_at: state === "running" ? ago(0, 1) : null, }); @@ -507,7 +528,10 @@ async function seedComputeData() { const accessRes = await asOwner("/admin/compute/access/grant/localgranted"); if (!accessRes.ok) throw new Error(`grant failed: ${accessRes.status} ${await accessRes.text()}`); const grantedId = idOf("granted"); - for (const [pool, amount] of [["cpu", 25_000], ["gpu", 10_000]]) { + for (const [pool, amount] of [ + ["cpu", 25_000], + ["gpu", 10_000], + ]) { const res = await asOwner(`/admin/compute/pools/by-id/${grantedId}/grant`, { pool, amount_milli: amount, @@ -517,6 +541,163 @@ async function seedComputeData() { } if (SEED_COMPUTE) await seedComputeData(); +/** The job page's jobs (--seed-compute-jobs): the backend's services, the Tapis stand-in, no route stood in. */ +const seededJobs = {}; +async function seedComputeJobs() { + const { createJob } = await import(join(root, "backend/src/services/compute-jobs.ts")); + const { fundAccount } = await import(join(root, "backend/test/helpers/compute-seed.ts")); + const { submitComputeJob, advanceComputeJob } = await import( + join(root, "backend/src/services/compute-lifecycle.ts") + ); + const { launchableSpec, playLauncher, reread, setTapisTimes, MINUTE } = lifecycle; + const idOf = (name) => db.query("SELECT id FROM users WHERE username = ?").get(`local${name}`).id; + const rich = idOf("jobs"); + const frozen = idOf("granted"); + await fundAccount(realD1(db), rich, 400_000, "cpu"); + const make = async (account, label) => { + const made = await createJob(env, { + source: "cli", + accountId: account, + appId: "nemar-launch", + appVersion: "1.0.0", + spec: launchableSpec(), + }); + if (!made.ok) throw new Error(`could not make ${label}: ${made.code}`); + seededJobs[label] = made.job.id; + return made.job.id; + }; + const submit = async (id) => { + const sub = await submitComputeJob(env, id); + if (!sub.ok) throw new Error(`could not submit ${id}: ${sub.outcome}`); + return (await reread(env, id)).tapis_uuid; + }; + const toRunning = async (id) => { + const uuid = await submit(id); + fx.tapis.advance(uuid, "QUEUED"); + fx.tapis.advance(uuid, "RUNNING"); + await advanceComputeJob(env, id); + return uuid; + }; + const finish = async (id, uuid, plan) => { + await playLauncher(fx, env, await reread(env, id), uuid, plan); + fx.tapis.advance(uuid, "FINISHED"); + setTapisTimes(fx, uuid, 30 * MINUTE, 0); + await advanceComputeJob(env, id); + }; + const outputs = [ + { path: "report.txt", content: "the report\n" }, + { path: "figures/plot.json", content: '{"x":[1,2,3]}' }, + { path: "table.csv", content: "a,b\n1,2\n" }, + ]; + + const finished = await make(rich, "finished"); + const u1 = await toRunning(finished); + fx.tapis.setOutput(u1, "tapisjob.out", "step one\nstep two\ndone\n"); + await finish(finished, u1, { outputs }); + + const running = await make(rich, "running"); + fx.tapis.setOutput(await toRunning(running), "tapisjob.out", "starting\nworking...\n"); + + const toCancel = await make(rich, "cancel"); + fx.tapis.setOutput(await toRunning(toCancel), "tapisjob.out", "this one will be canceled\n"); + + // Hostile and huge logs are on ENDED jobs (read from the records copy), which also keeps the + // number of live jobs under the concurrency limit. + const hostile = await make(rich, "hostile_log"); + const uh = await toRunning(hostile); + fx.tapis.setOutput( + uh, + "tapisjob.out", + [ + "plain line", + "\u001b[31mred\u001b[0m and \u001b[2K cleared", + "link \u001b]8;;https://example.org\u0007text\u001b]8;;\u0007 end", + "progress 10%\rprogress 50%\rprogress 100%", + "nul \u0000 bell \u0007 next-line \u0085 end", + "override \u202egnp.exe and isolate \u2066x\u2069 and zero\u200bwidth", + ' & "q" back\\slash', + "x".repeat(5000), + ].join("\n"), + ); + await finish(hostile, uh, { outputs: [], failureCode: "runtime_error" }); + + const big = await make(rich, "big_log"); + const lines = Array.from({ length: 40_000 }, (_, i) => `line ${i + 1} ${"y".repeat(20)}`).join( + "\n", + ); + const ub = await toRunning(big); + fx.tapis.setOutput(ub, "tapisjob.out", lines); + await finish(big, ub, { outputs: [], failureCode: "runtime_error" }); + + const failed = await make(rich, "failed"); + const u2 = await toRunning(failed); + fx.tapis.setOutput( + u2, + "tapisjob.out", + 'Traceback (most recent call last):\n File "main.py", line 3\nZeroDivisionError: division by zero\n', + ); + await finish(failed, u2, { outputs: [], failureCode: "runtime_error" }); + + const many = await make(rich, "many_results"); + const u3 = await toRunning(many); + await finish(many, u3, { + outputs: Array.from({ length: 1002 }, (_, i) => ({ + // `.dat`, not `.txt`: `bun run dev` answers a URL that ends in `.txt` (the cursor of the second page + // is the last path) as a static file lookup and 500s; a deployed build is not known to. + path: `out/file-${String(i).padStart(5, "0")}.dat`, + content: `n${i}\n`, + })), + }); + + for (const [label, daysAgo] of [ + ["expiring", 28], + ["expired", 31], + ]) { + const id = await make(rich, label); + const uuid = await toRunning(id); + await finish(id, uuid, { outputs }); + db.query("UPDATE compute_jobs SET ended_at = datetime('now', ?) WHERE id = ?").run( + `-${daysAgo} days`, + id, + ); + } + + const foreign = await make(frozen, "foreign"); + await toRunning(foreign); + + // A job that advances by itself every 20 s: submitted, queued, running, finished. + const watch = await make(rich, "watch"); + const watchUuid = await submit(watch); + let step = 0; + const timer = setInterval(async () => { + try { + step += 1; + if (step === 1) fx.tapis.advance(watchUuid, "QUEUED"); + if (step === 2) { + fx.tapis.advance(watchUuid, "RUNNING"); + fx.tapis.setOutput(watchUuid, "tapisjob.out", "watch job running\n"); + } + if (step === 3) { + fx.tapis.setOutput(watchUuid, "tapisjob.out", "watch job running\nwatch job done\n"); + await playLauncher(fx, env, await reread(env, watch), watchUuid, { outputs }); + fx.tapis.advance(watchUuid, "FINISHED"); + setTimeout(() => {}, 0); + } + // The backend refreshes a live job from the stand-in at most once a minute per read: age the + // stored read so the next page read picks the change up. + db.query( + "UPDATE compute_jobs SET tapis_synced_at = datetime('now', '-5 minutes') WHERE id = ?", + ).run(watch); + if (step >= 3) clearInterval(timer); + } catch (err) { + console.error(`watch job step failed: ${err instanceof Error ? err.message : err}`); + clearInterval(timer); + } + }, 20_000); + timer.unref?.(); +} +if (SEED_COMPUTE_JOBS) await seedComputeJobs(); + const cookies = {}; for (const p of [...PERSONAS, ...ADMIN_PERSONAS]) cookies[p.who] = await signIn(`${p.who}@nemar.test`); @@ -593,6 +774,9 @@ if (fx) { const outDir = join(import.meta.dir, "..", ".local-backend"); await mkdir(outDir, { recursive: true }); await writeFile(join(outDir, "cookies.json"), `${JSON.stringify(cookies, null, 2)}\n`); +if (SEED_COMPUTE_JOBS) { + await writeFile(join(outDir, "jobs.json"), `${JSON.stringify(seededJobs, null, 2)}\n`); +} console.log(`nemar-cli backend on http://localhost:${PORT} (in-memory database; restart resets)`); console.log(""); @@ -603,6 +787,13 @@ console.log( "Session cookies (paste as `nemar_session` for localhost:4321; also in .local-backend/cookies.json):", ); for (const [who, cookie] of Object.entries(cookies)) console.log(` ${who.padEnd(14)} ${cookie}`); +if (SEED_COMPUTE_JOBS) { + console.log(""); + console.log( + "Job page jobs (also in .local-backend/jobs.json), open /compute/jobs/ as `jobs`:", + ); + for (const [label, id] of Object.entries(seededJobs)) console.log(` ${label.padEnd(14)} ${id}`); +} console.log(""); console.log( "Admin Compute tab: sign in as `owner` or `adminself` (a non-owner administrator whose own", diff --git a/src/components/ComputeJobActions.astro b/src/components/ComputeJobActions.astro new file mode 100644 index 00000000..0c2bf6a3 --- /dev/null +++ b/src/components/ComputeJobActions.astro @@ -0,0 +1,31 @@ +--- +/** + * The Actions section of a job page (website ADR 0030 phase E3, #496): Cancel, for a live job only. + * It opens the shared `ConfirmDialog` (danger tone); the confirmed cancel is one `POST` made by + * `src/lib/compute-job-page.ts`, which shows the backend's outcome in words. The status line is where + * focus returns, because the button disappears after a successful cancel. + */ +import { JOB_COPY } from "../lib/compute-copy"; +import ConfirmDialog from "./ConfirmDialog.astro"; + +interface Props { + canCancel: boolean; +} +const { canCancel } = Astro.props; +const L = JOB_COPY.cancel; +--- + +
+ + + +
diff --git a/src/components/ComputeJobLogs.astro b/src/components/ComputeJobLogs.astro new file mode 100644 index 00000000..f66a15de --- /dev/null +++ b/src/components/ComputeJobLogs.astro @@ -0,0 +1,64 @@ +--- +/** + * The Log section of a job page (website ADR 0030 phase E3, #496). Nothing is read at page load: a log + * read can be a cluster read, so "Show log" asks for it. The text is the job's own output and is + * attacker-controlled: it is prepared by `src/lib/compute-logs.ts` (formatting removed, hostile + * characters written out, long lines capped) and set as `textContent` of the one `
` by
+ * `src/lib/compute-job-page.ts`. The backend returns the FIRST 1 MiB only (nemar-cli#2094 asks for a
+ * tail), so a cut log says so and points at the command line.
+ */
+import { JOB_COPY } from "../lib/compute-copy";
+
+interface Props {
+  jobId: string;
+}
+const { jobId } = Astro.props;
+const L = JOB_COPY.logs;
+---
+
+
+

+ +

+ + + + +

+ +

+ +
+ + diff --git a/src/components/ComputeJobResults.astro b/src/components/ComputeJobResults.astro new file mode 100644 index 00000000..6e0392e8 --- /dev/null +++ b/src/components/ComputeJobResults.astro @@ -0,0 +1,111 @@ +--- +/** + * The Results section of a job page (website ADR 0030 phase E3, #496). + * + * Viewing the page mints NOTHING and the server-rendered HTML carries no link: the download links are + * seven-day BEARER links, minted by the backend only when "Get download links" (or "Renew links") is + * pressed, written into this section by `src/lib/compute-job-page.ts` as `href` attributes with + * `rel="noopener noreferrer"`, and never stored, logged, copied or put in a URL. The backend's own + * `notice` about keeping them private is shown as served. + */ +import { JOB_COPY } from "../lib/compute-copy"; +import type { ResultsPanelView } from "../lib/compute-job-view"; + +interface Props { + jobId: string; + panel: ResultsPanelView; +} +const { jobId, panel } = Astro.props; +const L = JOB_COPY.results; +--- + +
+

{panel.headline}

+

+ +

+ + +

+ {L.allLead}nemar compute results {jobId} --download <dir>{L.allTail} +

+
+ + diff --git a/src/components/ComputeJobTimeline.astro b/src/components/ComputeJobTimeline.astro new file mode 100644 index 00000000..eb8ac235 --- /dev/null +++ b/src/components/ComputeJobTimeline.astro @@ -0,0 +1,145 @@ +--- +/** + * The state timeline and the details list of a job page (website ADR 0030 phase E3, #496). Both come + * from `jobDetailView` in `src/lib/compute-job-view.ts`; the browser rebuilds the same nodes from the + * same view (`src/lib/compute-job-dom.ts`), so the styles are GLOBAL and prefixed `.compute-job` (Astro + * scoped styles do not reach script-built nodes). Keep the class names in step with that module. + * + * The timeline claims only what the record carries: three times, no per-stage times. A step that is + * not known reads "not known" in words and carries a dash, never a check mark. The current step has a + * shape and the word, not a color alone. + */ +import { JOB_COPY } from "../lib/compute-copy"; +import type { JobDetailView } from "../lib/compute-job-view"; +import StatusIcon from "./admin/StatusIcon.astro"; + +interface Props { + view: JobDetailView; +} + +const { view } = Astro.props; +--- + +
    + {view.timeline.steps.map((step) => ( +
  1. + + + {step.label}{" "} + ({step.statusWord}) + {step.time && ( + + {step.time.label ? `${step.time.label} ` : ""} + + + )} + +
  2. + ))} +
+ + + diff --git a/src/lib/auth.test.ts b/src/lib/auth.test.ts index 4fe0df43..1549f0b4 100644 --- a/src/lib/auth.test.ts +++ b/src/lib/auth.test.ts @@ -33,6 +33,11 @@ describe("safeRedirectPath", () => { expect(safeRedirectPath("/path\nwith\nnewline")).toBe("/"); expect(safeRedirectPath("/path\rwith\rcr")).toBe("/"); }); + it("keeps a job page path, which the mail link and a signed-out visit return to", () => { + const job = "/compute/jobs/job_0123456789abcdef01234567"; + expect(safeRedirectPath(job)).toBe(job); + expect(safeRedirectPath(decodeURIComponent(encodeURIComponent(job)))).toBe(job); + }); it("guards against URL-encoded protocol-relative URLs", () => { expect(safeRedirectPath("/%2F%2Fevil.com")).toBe("/"); expect(safeRedirectPath("/%5C%5Cevil.com")).toBe("/"); diff --git a/src/lib/compute-api-job.test.ts b/src/lib/compute-api-job.test.ts new file mode 100644 index 00000000..cf96a411 --- /dev/null +++ b/src/lib/compute-api-job.test.ts @@ -0,0 +1,534 @@ +/** + * The compute client's job-page calls (phase E3, website#496). Every success case parses a body + * captured from the real routes by + * + * NEMAR_CLI_DIR= \ + * bun scripts/capture-compute-job-fixtures.mjs + * + * Transport conditions no route can be made to produce (a 502 with no JSON, a refused connection) use a + * real `Response`, as the repo's rule allows for failure paths. + */ +import { afterEach, describe, expect, it, vi } from "vitest"; +import cancel from "../../test/fixtures/compute/cancel.json"; +import errorsJob from "../../test/fixtures/compute/errors-job.json"; +import jobCancelRequested from "../../test/fixtures/compute/job-cancel-requested.json"; +import jobFailureCodes from "../../test/fixtures/compute/job-failure-codes.json"; +import jobLifecycle from "../../test/fixtures/compute/job-lifecycle.json"; +import jobStale from "../../test/fixtures/compute/job-stale.json"; +import jobStates from "../../test/fixtures/compute/job-states.json"; +import logs from "../../test/fixtures/compute/logs.json"; +import meActive from "../../test/fixtures/compute/me-active-with-usage.json"; +import results from "../../test/fixtures/compute/results.json"; +import { + COMPUTE_CANCEL_TIMEOUT_MS, + COMPUTE_JOB_READ_TIMEOUT_MS, + COMPUTE_LOG_TIMEOUT_MS, + ComputeApiError, + cancelComputeJob, + getComputeJob, + getComputeLogs, + getComputeResults, + isJobNotFound, + parseComputeCancel, + parseComputeJob, + parseComputeMe, + parseComputeResults, + renewComputeResults, +} from "./compute-api"; +import { COMPUTE_FAILURE_CODES, COMPUTE_JOB_STATES } from "./compute-contract"; + +const JOB_ID = "job_0123456789abcdef01234567"; + +function answering(res: Response) { + const seen: { url: string; init: RequestInit }[] = []; + const fn = (async (url: string, init: RequestInit) => { + seen.push({ url, init }); + return res; + }) as unknown as typeof fetch; + return { fn, seen }; +} + +const json = (body: unknown, status = 200): Response => + new Response(JSON.stringify(body), { status, headers: { "content-type": "application/json" } }); + +async function failure(promise: Promise): Promise { + try { + await promise; + } catch (err) { + expect(err).toBeInstanceOf(ComputeApiError); + return err as ComputeApiError; + } + throw new Error("expected the call to fail"); +} + +describe("GET /compute/me/jobs/:id", () => { + it("parses a captured job in each of the ten states", () => { + for (const state of COMPUTE_JOB_STATES) { + const read = parseComputeJob(jobStates[state].body); + expect(read.job.state).toBe(state); + expect(read.job.resources.nodes).toBe(1); + expect(read.stale).toBe(false); + } + }); + + it("keeps the memory figure when set and null when not", () => { + expect(parseComputeJob(jobStates.finished.body).job.resources.memory_mb).toBe(8192); + expect(parseComputeJob(jobStates.running.body).job.resources.memory_mb).toBeNull(); + }); + + it("parses a failed job for each of the failure codes", () => { + for (const code of COMPUTE_FAILURE_CODES) { + const body = (jobFailureCodes as unknown as Record)[code].body; + const read = parseComputeJob(body); + expect([code, read.job.failure_code]).toEqual([code, code]); + expect(read.job.state).toBe("failed"); + } + }); + + it("reads a recorded cancel on a live job", () => { + const read = parseComputeJob(jobCancelRequested.running.body); + expect([read.job.state, read.job.cancel_requested]).toEqual(["running", true]); + }); + + it("reads stale: true, and a missing stale as false", () => { + expect(parseComputeJob(jobStale.submitted.body).stale).toBe(true); + const { stale: _drop, ...rest } = jobStates.running.body; + expect(parseComputeJob(rest).stale).toBe(false); + }); + + it("parses the lifecycle job at running and at finished", () => { + expect(parseComputeJob(jobLifecycle.running.body).job.id).toBe(jobLifecycle.job_id); + const done = parseComputeJob(jobLifecycle.finished.body).job; + expect([done.state, done.charged_milli > 0]).toEqual(["finished", true]); + }); + + it("copies only the keys the page reads", () => { + const body = structuredClone(jobStates.running.body) as Record; + (body.job as Record).tenant_pool = "platform"; + (body.job as Record).failure_detail = "oom"; + ((body.job as Record).resources as Record).hold_milli = 9; + body.extra = "x"; + const read = parseComputeJob(body); + expect(Object.keys(read.job).sort()).toEqual( + [ + "app_id", + "app_version", + "cancel_requested", + "charged_milli", + "created_at", + "ended_at", + "failure_code", + "id", + "kind", + "name", + "partition", + "pool", + "resources", + "source", + "started_at", + "state", + ].sort(), + ); + expect(Object.keys(read.job.resources).sort()).toEqual( + ["cores_per_node", "gpus", "memory_mb", "minutes", "nodes"].sort(), + ); + expect(Object.keys(read).sort()).toEqual(["job", "stale"]); + }); + + it("reads an optional name through, and none as null", () => { + expect(parseComputeJob(jobStates.running.body).job.name).toBeNull(); + const body = structuredClone(jobStates.running.body) as { job: Record }; + body.job.name = "my analysis"; + expect(parseComputeJob(body).job.name).toBe("my analysis"); + }); + + it("rejects a missing key or a wrong type as a shape error, never zeros", () => { + type Loose = { job: Record & { resources: Record } }; + const bad = (mutate: (b: Loose) => void) => { + const b = structuredClone(jobStates.running.body) as unknown as Loose; + mutate(b); + return () => parseComputeJob(b); + }; + const without = (o: Record, key: string) => { + Reflect.deleteProperty(o, key); + }; + expect(bad((b) => without(b.job, "resources"))).toThrow(ComputeApiError); + expect(bad((b) => without(b.job.resources, "minutes"))).toThrow(ComputeApiError); + expect( + bad((b) => { + b.job.resources.nodes = "1"; + }), + ).toThrow(ComputeApiError); + expect( + bad((b) => { + b.job.charged_milli = 1.5; + }), + ).toThrow(ComputeApiError); + expect( + bad((b) => { + b.job.cancel_requested = 0; + }), + ).toThrow(ComputeApiError); + expect( + bad((b) => { + b.job.name = 5; + }), + ).toThrow(ComputeApiError); + expect(() => parseComputeJob(null)).toThrow(ComputeApiError); + expect(() => parseComputeJob({ job: jobStates.running.body.job, stale: "yes" })).toThrow( + ComputeApiError, + ); + }); + + it("asks for the job by id, with the SSR cookie, and sends nothing else", async () => { + const { fn, seen } = answering(json(jobStates.running.body)); + const read = await getComputeJob(JOB_ID, { fetch: fn, cookieHeader: "nemar_session=abc" }); + expect(read.job.state).toBe("running"); + expect(seen[0].url).toMatch(new RegExp(`/compute/me/jobs/${JOB_ID}$`)); + expect(seen[0].init.method).toBe("GET"); + expect((seen[0].init.headers as Record).Cookie).toBe("nemar_session=abc"); + }); + + it("refuses an id that is not a job id before sending anything", () => { + const { fn, seen } = answering(json({})); + for (const bad of ["", "nope", "job_XYZ", `${JOB_ID}/../logs`, `${JOB_ID}?x=1`]) { + expect(() => getComputeJob(bad, { fetch: fn })).toThrow(RangeError); + } + expect(seen).toEqual([]); + }); +}); + +describe("GET /compute/jobs/:id/logs", () => { + const live = logs.live; + + function logResponse(entry: { status: number; headers: Record; text?: string }) { + return new Response(entry.text ?? "", { status: entry.status, headers: entry.headers }); + } + + it("returns the text and no truncation for a log under 1 MiB", async () => { + const { fn, seen } = answering(logResponse(live)); + const log = await getComputeLogs(JOB_ID, { fetch: fn }); + expect(log).toEqual({ text: live.text, truncated: false }); + expect((seen[0].init.headers as Record).Accept).toBe("text/plain"); + }); + + it("reads X-Nemar-Truncated: true from the captured headers", async () => { + const t = logs.truncated; + expect(t.headers["x-nemar-truncated"]).toBe("true"); + const { fn } = answering( + new Response("x".repeat(1024), { status: t.status, headers: t.headers }), + ); + expect((await getComputeLogs(JOB_ID, { fetch: fn })).truncated).toBe(true); + }); + + it("treats any other value of the header, or none, as not truncated", async () => { + for (const value of ["false", "1", "TRUE", ""]) { + const { fn } = answering(new Response("a", { headers: { "X-Nemar-Truncated": value } })); + expect((await getComputeLogs(JOB_ID, { fetch: fn })).truncated).toBe(false); + } + }); + + it("returns an empty log as an empty string, not an error", async () => { + const { fn } = answering(logResponse(logs.empty)); + expect(await getComputeLogs(JOB_ID, { fetch: fn })).toEqual({ text: "", truncated: false }); + }); + + it("hands the hostile log back byte for byte (the page escapes it, the client does not)", async () => { + const { fn } = answering(logResponse(logs.hostile)); + const log = await getComputeLogs(JOB_ID, { fetch: fn }); + expect(log.text).toBe(logs.hostile.text); + expect(log.text).toContain("\u001b[31m"); + expect(log.text).toContain("", + "http://evil.example/a", + "file:///etc/passwd", + "//evil.example/a", + "not a url", + "", + "vbscript:x", + "blob:https://x/y", + ]) { + expect(safeResultHref(bad), bad).toBeNull(); + } + }); + + it("escapes the path and the checksum and formats the size", () => { + const row = resultRow({ + path: "a‮/b\u0000.txt", + size: 11, + sha256: "ab".repeat(32), + url: "javascript:1", + }); + expect(row.path).toBe("a\\u202e/b\\x00.txt"); + expect(row.size).toBe("11 B"); + expect(row.shaShort).toBe("abababababab"); + expect(row.sha.length).toBe(64); + expect(row.href).toBeNull(); + }); + + it("offers 100 more rows from the held page, then the next page by cursor, then nothing", () => { + const base = { total: 2500, nextAfter: "out/x" as string | null }; + expect(resultsPaging({ ...base, held: 1000, shown: 100, offset: 0 })).toEqual({ + showMore: RESULT_ROWS_STEP, + nextPageRemaining: null, + }); + expect(resultsPaging({ ...base, held: 1000, shown: 950, offset: 0 }).showMore).toBe(50); + expect(resultsPaging({ ...base, held: 1000, shown: 1000, offset: 0 })).toEqual({ + showMore: 0, + nextPageRemaining: 1500, + }); + expect( + resultsPaging({ ...base, held: 1000, shown: 1000, offset: 1000 }).nextPageRemaining, + ).toBe(500); + expect(resultsPaging({ total: 3, nextAfter: null, held: 3, shown: 3, offset: 0 })).toEqual({ + showMore: 0, + nextPageRemaining: null, + }); + }); + + it("knows when links have expired, and treats an unreadable date as expired", () => { + const at = "2026-11-14T15:35:07.000Z"; + const ms = Date.parse(at); + expect(linksExpired(at, ms - 1)).toBe(false); + expect(linksExpired(at, ms)).toBe(true); + expect(linksExpired("garbage", ms)).toBe(true); + }); + + it("says when a link outlives the files", () => { + const at = "2026-11-14T15:35:07.000Z"; + expect(linksOutliveFiles(at, Date.parse(at) - 1)).toBe(true); + expect(linksOutliveFiles(at, Date.parse(at) + 1)).toBe(false); + expect(linksOutliveFiles(at, null)).toBe(false); + }); +}); + +describe("cancelOutcome", () => { + const fx = (k: keyof typeof cancelFx) => + cancelOutcome(parseComputeCancel((cancelFx[k] as { body: unknown }).body)); + + it("words every captured answer", () => { + expect(fx("running")).toMatchObject({ kind: "canceled", poll: false }); + expect(fx("running_again").kind).toBe("canceled"); + expect(fx("never_started").kind).toBe("canceled"); + expect(fx("cluster_down")).toMatchObject({ kind: "recorded", poll: true }); + expect(fx("cluster_down_again").kind).toBe("recorded"); + const ended = fx("finished"); + expect(ended.kind).toBe("ended"); + expect(ended.sentence).toBe("The job had already ended (finished); nothing was canceled."); + expect(ended.poll).toBe(false); + }); + + it("words the rows the stand-in cannot make", () => { + const live = { state: "running", outcome: "already_cancelled", cancelled: true }; + expect(cancelOutcome(live).kind).toBe("already_recorded"); + expect(cancelOutcome({ ...live, outcome: "cancel_requested" }).kind).toBe("recorded"); + // An older Worker has no `cancelled`: the outcome decides. + expect(cancelOutcome({ ...live, outcome: "cancel_requested", cancelled: null }).kind).toBe( + "recorded", + ); + expect(cancelOutcome({ state: "failed", outcome: "already_ended", cancelled: null }).kind).toBe( + "ended", + ); + expect( + cancelOutcome({ state: null, outcome: "already_ended", cancelled: false }).sentence, + ).toContain("unknown"); + expect( + cancelOutcome({ state: "running", outcome: "something_new", cancelled: true }).kind, + ).toBe("unknown"); + }); + + it("always asks for the job to be read again and promises no amount", () => { + for (const k of ["running", "cluster_down", "finished"] as const) { + expect(fx(k).refresh).toBe(true); + expect(fx(k).sentence).not.toMatch(/refund|SU|charged/i); + } + }); +}); + +describe("nextPoll", () => { + const read = (state: string, changed = false): PollEvent => ({ kind: "read", state, changed }); + const drive = (events: PollEvent[], start = 0, random = 0.5) => { + let state: PollState = initialPoll(start); + const steps = []; + for (const e of events) { + const out = nextPoll(state, e, start + 1000, random); + state = out.next; + steps.push(out.step); + } + return { steps, state }; + }; + + it("starts at 10 s and doubles to 60 s while nothing changes", () => { + const { steps } = drive(Array(6).fill(read("running"))); + expect(steps.map((s) => (s as { delayMs: number }).delayMs)).toEqual([ + 20_000, 40_000, 60_000, 60_000, 60_000, 60_000, + ]); + expect(initialPoll(0).delayMs).toBe(POLL_BASE_MS); + }); + + it("resets to 10 s on a change", () => { + const { steps } = drive([ + read("running"), + read("running"), + read("archiving", true), + read("archiving"), + ]); + expect(steps.map((s) => (s as { delayMs: number }).delayMs)).toEqual([ + 20_000, 40_000, 10_000, 20_000, + ]); + }); + + it("stops on a terminal state, whatever the schedule before it was", () => { + for (const state of ["finished", "failed", "cancelled"]) { + for (const changed of [true, false]) { + const { steps } = drive([read("running"), read("running", true), read(state, changed)]); + expect(steps[2], `${state} ${changed}`).toEqual({ action: "stop", reason: "terminal" }); + } + } + }); + + it("stays stopped: no event restarts a stopped poller", () => { + let state = nextPoll(initialPoll(0), read("finished", true), 5).next; + for (const e of [ + read("running", true), + { kind: "visible", sinceLastReadMs: 99_999 } as PollEvent, + { kind: "error" } as PollEvent, + ]) { + const out = nextPoll(state, e, 10); + expect(out.step).toEqual({ action: "stop", reason: "terminal" }); + state = out.next; + } + }); + + it("stops on a state this build does not know", () => { + expect(nextPoll(initialPoll(0), read("weird"), 5).step).toEqual({ + action: "stop", + reason: "unknown_state", + }); + }); + + it("keeps polling an unreported job: it may resume", () => { + expect(nextPoll(initialPoll(0), read("unreported"), 5).step.action).toBe("wait"); + }); + + it("doubles on an error, and the fifth error in a row stops", () => { + const { steps } = drive(Array(POLL_ERROR_LIMIT).fill({ kind: "error" } as PollEvent)); + expect(steps.slice(0, 4).map((s) => (s as { delayMs: number }).delayMs)).toEqual([ + 20_000, 40_000, 60_000, 60_000, + ]); + expect(steps[4]).toEqual({ action: "stop", reason: "errors" }); + }); + + it("counts only consecutive errors: a good read clears them", () => { + const e = { kind: "error" } as PollEvent; + const { steps } = drive([e, e, e, e, read("running", true), e, e, e, e]); + expect(steps.every((s) => s.action === "wait")).toBe(true); + }); + + it("stops on a 401 and on a not-found", () => { + expect(nextPoll(initialPoll(0), { kind: "error", fatal: "session" }, 5).step).toEqual({ + action: "stop", + reason: "session", + }); + expect(nextPoll(initialPoll(0), { kind: "error", fatal: "not_found" }, 5).step).toEqual({ + action: "stop", + reason: "not_found", + }); + }); + + it("pauses when the tab is hidden, and refreshes at once on becoming visible after 5 s", () => { + expect(nextPoll(initialPoll(0), { kind: "hidden" }, 5).step).toEqual({ action: "pause" }); + expect(nextPoll(initialPoll(0), { kind: "visible", sinceLastReadMs: 5_001 }, 5).step).toEqual({ + action: "refresh_now", + }); + expect( + nextPoll(initialPoll(0), { kind: "visible", sinceLastReadMs: 5_000 }, 5).step.action, + ).toBe("wait"); + }); + + it("restarts the schedule at 10 s after a pause", () => { + let state = drive([read("running"), read("running"), read("running")]).state; + expect(state.delayMs).toBe(60_000); + state = nextPoll(state, { kind: "visible", sinceLastReadMs: 60_000 }, 5).next; + expect(state.delayMs).toBe(POLL_BASE_MS); + }); + + it("stops two hours after load, at exactly the boundary and not before", () => { + const s = initialPoll(1000); + expect(nextPoll(s, read("running"), 1000 + POLL_HARD_STOP_MS - 1).step.action).toBe("wait"); + expect(nextPoll(s, read("running"), 1000 + POLL_HARD_STOP_MS).step).toEqual({ + action: "stop", + reason: "hard_stop", + }); + expect( + nextPoll(s, { kind: "visible", sinceLastReadMs: 1e9 }, 1000 + POLL_HARD_STOP_MS).step.action, + ).toBe("stop"); + }); + + it("moves each delay by at most 15 percent", () => { + for (const [random, expected] of [ + [0, 8_500], + [0.5, 10_000], + [0.999999, 11_500], + ] as const) { + const out = nextPoll(initialPoll(0), read("running", true), 5, random); + expect((out.step as { delayMs: number }).delayMs).toBe(expected); + } + expect(POLL_MAX_MS).toBe(60_000); + }); +}); diff --git a/src/lib/compute-job-view.ts b/src/lib/compute-job-view.ts new file mode 100644 index 00000000..96799a3f --- /dev/null +++ b/src/lib/compute-job-view.ts @@ -0,0 +1,675 @@ +/** + * Pure view models for the job page `/compute/jobs/` (website ADR 0030 phase E3, issue #496). The + * server template, the browser script and the tests read one place. Nothing here decides what a state + * MEANS (the backend does); it decides what the page shows for each state it was told, and it claims + * only what the record carries: three times (created, started, ended), no per-stage times + * (nemar-cli#2097), no results expiry (nemar-cli#2096), no log tail (nemar-cli#2094), result links that + * download rather than render (nemar-cli#2095). + * + * Rules every function keeps: + * - A backend string passes through `displaySafeLine` and is rendered as a text node by the callers. + * - A state outside the closed ten is shown as its escaped word and nothing else. + * - No function reads a clock: `nowMs` and the random number are arguments. + */ +import { ACCOUNT_COPY } from "./account-copy"; +import { + ComputeApiError, + type ComputeCancel, + type ComputeJobRead, + type ComputeResultFile, +} from "./compute-api"; +import { type JobBadge, jobBadge, sourceLabel } from "./compute-badges"; +import { + COMPUTE_CANCEL_OUTCOMES, + isComputeJobState, + isTerminalComputeState, +} from "./compute-contract"; +import { JOB_COPY } from "./compute-copy"; +import { type FailureView, failureView } from "./compute-failure"; +import { + dbTimeToIso, + formatDuration, + formatEpochUtcMinute, + formatUtcMinute, + formatUtcSecond, + parseDbTime, +} from "./compute-format"; +import { COMPUTE_LABELS } from "./compute-labels"; +import { chargeText } from "./compute-view"; +import { formatBytes } from "./format"; +import { displaySafeLine } from "./mail-display"; +import type { IconName } from "./status-icons"; + +const DAY_MS = 86_400_000; + +// --- State --------------------------------------------------------------------------------------- + +/** A known, ended state: the record will not change again. */ +export function isTerminal(state: string): boolean { + return isComputeJobState(state) && isTerminalComputeState(state); +} + +/** A known state that has not ended: the page keeps asking. An unknown state is neither. */ +export function isLive(state: string): boolean { + return isComputeJobState(state) && !isTerminalComputeState(state); +} + +/** + * Whether the page offers Cancel. The route refuses nothing by state (an ended job answers 200 + * `already_ended`), so this is the PAGE's own choice: a known live state, with no cancel already + * recorded. Every front door is offered it, partner included; the backend resolves the intent. + */ +export function canCancel(job: { state: string; cancel_requested: boolean }): boolean { + return isLive(job.state) && !job.cancel_requested; +} + +// --- Timeline ------------------------------------------------------------------------------------ + +export type StepStatus = "passed" | "current" | "pending" | "unknown"; + +export interface StepTime { + readonly label: string | null; + readonly text: string; + readonly iso: string | null; + readonly title: string; +} + +export interface TimelineStep { + readonly key: "created" | "submitted" | "queued" | "running" | "archiving" | "terminal"; + readonly label: string; + readonly status: StepStatus; + readonly statusWord: string; + readonly icon: IconName; + readonly time: StepTime | null; +} + +export interface TimelineView { + readonly steps: readonly TimelineStep[]; + /** The escaped raw word of a state this build does not know; the timeline is then empty. */ + readonly unknownState: string | null; + readonly unreported: boolean; +} + +const LIVE_INDEX: Readonly> = { + created: 0, + uploading: 0, + submitted: 1, + queued: 2, + running: 3, + archiving: 4, +}; + +const STEP_KEYS = ["created", "submitted", "queued", "running", "archiving"] as const; + +const STATUS_ICON: Readonly> = { + passed: "ok", + current: "progress", + pending: "neutral", + unknown: "flat", +}; + +function stepTime(value: string | null, label: string | null): StepTime | null { + if (value === null) return null; + return { + label, + text: formatUtcMinute(value), + iso: dbTimeToIso(value), + title: formatUtcSecond(value), + }; +} + +const STATE_LABEL = (state: string): string => + isComputeJobState(state) ? jobBadge(state).label : displaySafeLine(state); + +/** + * The line of stages for a job. It claims only what is known: the record has three times, and Tapis may + * skip states, so a stage the record cannot vouch for reads "not known", never "done". + */ +export function timeline(job: { + state: string; + created_at: string; + started_at: string | null; + ended_at: string | null; +}): TimelineView { + if (!isComputeJobState(job.state)) { + return { steps: [], unknownState: displaySafeLine(job.state), unreported: false }; + } + const state = job.state; + const started = job.started_at !== null; + const terminal = isTerminalComputeState(state); + const statuses: StepStatus[] = []; + let terminalStatus: StepStatus; + if (Object.hasOwn(LIVE_INDEX, state)) { + const at = LIVE_INDEX[state]; + for (let i = 0; i < STEP_KEYS.length; i++) { + statuses.push(i < at ? "passed" : i === at ? "current" : "pending"); + } + terminalStatus = "pending"; + } else if (state === "unreported") { + // A side state with no place on the line: through Running only if it started, else only Created. + for (let i = 0; i < STEP_KEYS.length; i++) { + statuses.push(i === 0 || (started && i <= 3) ? "passed" : "unknown"); + } + terminalStatus = "unknown"; + } else if (state === "finished") { + for (let i = 0; i < STEP_KEYS.length; i++) statuses.push("passed"); + terminalStatus = "current"; + } else { + // failed or cancelled: Archiving is never claimed. + for (let i = 0; i < STEP_KEYS.length; i++) { + statuses.push(i === 0 ? "passed" : i <= 3 && started ? "passed" : "unknown"); + } + terminalStatus = "current"; + } + const stateLabel = STATE_LABEL(state); + const steps: TimelineStep[] = STEP_KEYS.map((key, i) => { + const status = statuses[i]; + const label = + key === "created" && state === "uploading" ? STATE_LABEL("uploading") : STATE_LABEL(key); + let time: StepTime | null = null; + if (status === "passed" || status === "current") { + if (key === "created") time = stepTime(job.created_at, null); + else if (key === "running") time = stepTime(job.started_at, JOB_COPY.timeline.started); + } + return { + key, + label, + status, + statusWord: JOB_COPY.timeline.stepWord[status], + icon: STATUS_ICON[status], + time, + }; + }); + steps.push({ + key: "terminal", + label: terminal ? stateLabel : JOB_COPY.timeline.ended, + status: terminalStatus, + statusWord: JOB_COPY.timeline.stepWord[terminalStatus], + icon: terminal + ? state === "finished" + ? "ok" + : state === "failed" + ? "error" + : "neutral" + : STATUS_ICON[terminalStatus], + time: terminal ? stepTime(job.ended_at, null) : null, + }); + return { steps, unknownState: null, unreported: state === "unreported" }; +} + +// --- The whole page ------------------------------------------------------------------------------ + +export interface DetailRow { + readonly key: string; + readonly label: string; + readonly value: string; + /** Set for a time: the machine value and the tooltip. */ + readonly time?: { readonly iso: string | null; readonly title: string }; +} + +export interface JobDetailView { + readonly id: string; + /** A name the job was given (not served today, nemar-cli#2079); escaped. */ + readonly title: string | null; + readonly badge: JobBadge; + /** The state word, escaped. */ + readonly state: string; + readonly known: boolean; + readonly live: boolean; + readonly terminal: boolean; + /** True only for a live job with a cancel on record. */ + readonly cancelRequested: boolean; + readonly canCancel: boolean; + readonly stale: boolean; + readonly app: string; + readonly appVersion: string; + readonly frontDoor: string; + readonly timeline: TimelineView; + readonly rows: readonly DetailRow[]; + readonly charge: { readonly text: string; readonly kind: "terminal" | "live" | "unknown" }; + readonly failure: FailureView | null; + readonly unreportedNote: string | null; + readonly unitsNote: string; +} + +function poolLabel(pool: string): string { + return Object.hasOwn(COMPUTE_LABELS.pools, pool) + ? COMPUTE_LABELS.pools[pool as "cpu" | "gpu"] + : displaySafeLine(pool); +} + +export function jobDetailView(read: ComputeJobRead): JobDetailView { + const { job } = read; + const R = JOB_COPY.rows; + const known = isComputeJobState(job.state); + const live = isLive(job.state); + const terminal = isTerminal(job.state); + const charge = chargeText(job); + const rows: DetailRow[] = [ + { + key: "partition", + label: R.partition, + value: `${displaySafeLine(job.partition)} (${poolLabel(job.pool)})`, + }, + { key: "nodes", label: R.nodes, value: String(job.resources.nodes) }, + { key: "cores", label: R.cores, value: String(job.resources.cores_per_node) }, + ]; + if (job.resources.gpus > 0) { + rows.push({ key: "gpus", label: R.gpus, value: String(job.resources.gpus) }); + } + if (job.resources.memory_mb !== null) { + rows.push({ + key: "memory", + label: R.memory, + value: `${job.resources.memory_mb.toLocaleString("en-US")} MB`, + }); + } + rows.push({ + key: "runtime", + label: R.runTime, + value: `${R.upTo} ${formatDuration(job.resources.minutes * 60)}`, + }); + const timed = (key: string, label: string, value: string | null): void => { + if (value === null) return; + rows.push({ + key, + label, + value: formatUtcMinute(value), + time: { iso: dbTimeToIso(value), title: formatUtcSecond(value) }, + }); + }; + timed("created", R.created, job.created_at); + timed("started", R.started, job.started_at); + timed("ended", R.ended, job.ended_at); + return { + id: displaySafeLine(job.id), + title: job.name === null ? null : displaySafeLine(job.name), + badge: jobBadge(job.state), + state: displaySafeLine(job.state), + known, + live, + terminal, + cancelRequested: job.cancel_requested && live, + canCancel: canCancel(job), + stale: read.stale, + app: displaySafeLine(job.app_id), + appVersion: displaySafeLine(job.app_version), + frontDoor: sourceLabel(job.source), + timeline: timeline(job), + rows, + charge, + failure: job.failure_code === null ? null : failureView(job.failure_code, job.id), + unreportedNote: + job.state === "unreported" ? ACCOUNT_COPY["compute_page.unreported_note"] : null, + unitsNote: ACCOUNT_COPY["compute_page.units_note"], + }; +} + +// --- Results retention --------------------------------------------------------------------------- + +/** Three days or fewer left counts as "expiring"; website-owned. */ +export const RESULTS_EXPIRING_DAYS = 3; + +export type ResultsState = "not_ready" | "none" | "available" | "expiring" | "expired"; + +export interface ResultsWindow { + readonly state: ResultsState; + readonly untilMs: number | null; + readonly untilText: string | null; + readonly untilIso: string | null; + readonly daysLeft: number | null; + readonly days: number | null; +} + +/** + * Where a job's results stand, DERIVED from `ended_at` and `limits.results_days` until the server states + * a date (nemar-cli#2096). The rule is the backend's own (`resultsRetentionPassed`): expired once + * `now > ended_at + days`, so the instant itself is still available. The 410 stays authoritative: the + * page keeps its button in `expired` and lets the backend decide. + */ +export function resultsWindow(input: { + state: string; + endedAt: string | null; + resultsDays: number | null; + nowMs: number; +}): ResultsWindow { + const none = (state: ResultsState, days: number | null = null): ResultsWindow => ({ + state, + untilMs: null, + untilText: null, + untilIso: null, + daysLeft: null, + days, + }); + if (input.state === "failed" || input.state === "cancelled") return none("none"); + if (input.state !== "finished") return none("not_ready"); + const ended = input.endedAt === null ? null : parseDbTime(input.endedAt); + if (ended === null || input.resultsDays === null) return none("available", input.resultsDays); + const untilMs = ended + input.resultsDays * DAY_MS; + const left = untilMs - input.nowMs; + const state: ResultsState = + left < 0 ? "expired" : left <= RESULTS_EXPIRING_DAYS * DAY_MS ? "expiring" : "available"; + return { + state, + untilMs, + untilText: formatEpochUtcMinute(untilMs), + untilIso: new Date(untilMs).toISOString(), + daysLeft: left < 0 ? 0 : Math.ceil(left / DAY_MS), + days: input.resultsDays, + }; +} + +export interface ResultsPanelView { + readonly window: ResultsWindow; + readonly headline: string; + /** Whether the "Get download links" button shows. It stays in `expired`: the 410 decides. */ + readonly canGetLinks: boolean; +} + +export function resultsPanelView(input: { + state: string; + endedAt: string | null; + resultsDays: number | null; + nowMs: number; +}): ResultsPanelView { + const w = resultsWindow(input); + const C = JOB_COPY.results; + let headline: string; + switch (w.state) { + case "not_ready": + headline = C.notReady; + break; + case "none": + headline = C.none; + break; + case "expired": + headline = C.expiredDerived(w.days ?? 0); + break; + case "expiring": + headline = C.expiring(w.daysLeft ?? 0, w.untilText ?? ""); + break; + default: + headline = + w.untilText === null || w.days === null ? C.keptLimited : C.kept(w.days, w.untilText); + } + return { window: w, headline, canGetLinks: input.state === "finished" }; +} + +// --- Failures of the log, the results and the cancel --------------------------------------------- + +export interface FailureNotice { + readonly kind: "none" | "expired" | "not_ready" | "not_found" | "session" | "account" | "message"; + readonly sentence: string; +} + +function transport(err: unknown): FailureNotice { + const n = COMPUTE_LABELS.notice; + if (err instanceof ComputeApiError) { + if (err.code === "job_not_found") return { kind: "not_found", sentence: JOB_COPY.notFoundBody }; + if (err.kind === "session") return { kind: "session", sentence: JOB_COPY.live.sessionEnded }; + if (err.kind === "account") return { kind: "account", sentence: n.account }; + const reason = n[err.kind === "rejected" ? "rejected" : err.kind]; + return { + kind: "message", + sentence: err.sentence ? `${reason} ${displaySafeLine(err.sentence)}` : reason, + }; + } + return { kind: "message", sentence: n.shape }; +} + +/** What the results panel says for a failed links call. A 410 is the expired state; a 409 means no files. */ +export function resultsFailure( + err: unknown, + resultsDays: number | null, + source = "", +): FailureNotice { + if (err instanceof ComputeApiError) { + if (err.code === "results_expired") { + return { kind: "expired", sentence: JOB_COPY.results.expired410(resultsDays) }; + } + if (err.code === "bad_state") { + // A partner job's files are the person's only once NEMAR has checked the job; say so, as the log does. + const sentence = + source === "partner" ? JOB_COPY.results.badStatePartner : JOB_COPY.results.badState; + return { kind: "not_ready", sentence }; + } + } + return transport(err); +} + +/** What the log panel says for a failed log call. */ +export function logFailure(err: unknown, source: string): FailureNotice { + if (err instanceof ComputeApiError) { + if (err.code === "logs_not_available") { + return { + kind: "none", + sentence: source === "partner" ? JOB_COPY.logs.nonePartner : JOB_COPY.logs.none, + }; + } + if (err.code === "storage_unavailable") { + return { kind: "message", sentence: JOB_COPY.logs.unavailable }; + } + } + return transport(err); +} + +// --- Result rows --------------------------------------------------------------------------------- + +/** Rows shown at first and added by each "Show more". */ +export const RESULT_ROWS_STEP = 100; + +const LOOPBACK = new Set(["127.0.0.1", "localhost", "[::1]"]); + +/** Whether a host name is this machine. The page asks it of ITS OWN host (`location.hostname`). */ +export function isLoopbackHost(hostname: string): boolean { + return LOOPBACK.has(hostname); +} + +/** + * The only way a result address reaches an `href`: `https:`, or `http:` on a loopback host when + * `allowLoopback` is true (the local backend's object store, for the browser check). The page sets that + * flag from its own host, so a deployed page never accepts a plain-http link, loopback or not. + * `javascript:`, `data:` and anything else render no link. + */ +export function safeResultHref(url: string, allowLoopback = false): string | null { + let parsed: URL; + try { + parsed = new URL(url); + } catch { + return null; + } + if (parsed.protocol === "https:") return parsed.href; + if (allowLoopback && parsed.protocol === "http:" && LOOPBACK.has(parsed.hostname)) { + return parsed.href; + } + return null; +} + +export interface ResultRow { + readonly path: string; + readonly size: string; + readonly shaShort: string; + readonly sha: string; + /** Null renders "link unavailable". */ + readonly href: string | null; +} + +export function resultRow(file: ComputeResultFile, allowLoopback = false): ResultRow { + return { + path: displaySafeLine(file.path), + size: formatBytes(file.size), + shaShort: displaySafeLine(file.sha256.slice(0, 12)), + sha: displaySafeLine(file.sha256), + href: safeResultHref(file.url, allowLoopback), + }; +} + +export interface ResultsPaging { + /** How many more rows the next "Show more" adds from the page already held. */ + readonly showMore: number; + /** Files beyond the held page that another call would fetch; null when there is nothing more. */ + readonly nextPageRemaining: number | null; +} + +/** + * What the list offers after `shown` of `held` rows are visible. The held page is up to 1,000 files; + * only when it is used up and the backend gave a cursor is a next call offered. + */ +export function resultsPaging(p: { + held: number; + shown: number; + offset: number; + total: number; + nextAfter: string | null; +}): ResultsPaging { + const showMore = Math.max(0, Math.min(RESULT_ROWS_STEP, p.held - p.shown)); + const remaining = p.total - (p.offset + p.held); + return { + showMore, + nextPageRemaining: showMore === 0 && p.nextAfter !== null && remaining > 0 ? remaining : null, + }; +} + +/** True once the links' own expiry has passed; an unreadable date counts as expired (never keep a link we cannot date). */ +export function linksExpired(expiresAtIso: string, nowMs: number): boolean { + const at = Date.parse(expiresAtIso); + return Number.isNaN(at) || at <= nowMs; +} + +/** Whether the links outlive the files (a link minted late): the retention date is before the link expiry. */ +export function linksOutliveFiles(expiresAtIso: string, untilMs: number | null): boolean { + const at = Date.parse(expiresAtIso); + return untilMs !== null && !Number.isNaN(at) && at > untilMs; +} + +// --- Cancel -------------------------------------------------------------------------------------- + +export interface CancelOutcomeView { + readonly kind: "recorded" | "canceled" | "already_recorded" | "ended" | "unknown"; + readonly sentence: string; + /** The job is always read again: the answer may have changed its state. */ + readonly refresh: true; + /** Keep polling for the cluster to carry it out. */ + readonly poll: boolean; +} + +/** + * The words for the backend's cancel answer. The codes are the backend's; the sentences are the page's + * and promise nothing about a charge beyond what the dialog already said and the refreshed record shows. + */ +export function cancelOutcome(answer: ComputeCancel): CancelOutcomeView { + const C = JOB_COPY.cancel; + const known = (COMPUTE_CANCEL_OUTCOMES as readonly string[]).includes(answer.outcome); + const ended = answer.outcome === "already_ended" || answer.cancelled === false; + if (ended) { + const state = answer.state === null ? "unknown" : STATE_LABEL(answer.state).toLowerCase(); + return { kind: "ended", sentence: C.ended(state), refresh: true, poll: false }; + } + if (!known) return { kind: "unknown", sentence: C.unknown, refresh: true, poll: true }; + if (answer.state === "cancelled") { + return { kind: "canceled", sentence: C.canceled, refresh: true, poll: false }; + } + if (answer.outcome === "already_cancelled") { + return { kind: "already_recorded", sentence: C.alreadyRecorded, refresh: true, poll: true }; + } + return { kind: "recorded", sentence: C.recorded, refresh: true, poll: true }; +} + +// --- Polling ------------------------------------------------------------------------------------- + +export const POLL_BASE_MS = 10_000; +export const POLL_MAX_MS = 60_000; +export const POLL_ERROR_LIMIT = 5; +export const POLL_HARD_STOP_MS = 2 * 3_600_000; +export const POLL_JITTER = 0.15; +/** On becoming visible, a read older than this is refreshed at once. */ +export const POLL_STALE_AFTER_MS = 5_000; + +export type StopReason = + | "terminal" + | "unknown_state" + | "errors" + | "hard_stop" + | "session" + | "not_found"; + +export interface PollState { + /** The un-jittered delay last scheduled. */ + readonly delayMs: number; + readonly errors: number; + /** When the page loaded (or Resume was pressed). */ + readonly sinceMs: number; + readonly stopped: StopReason | null; +} + +export function initialPoll(nowMs: number): PollState { + return { delayMs: POLL_BASE_MS, errors: 0, sinceMs: nowMs, stopped: null }; +} + +export type PollEvent = + | { readonly kind: "read"; readonly state: string; readonly changed: boolean } + | { readonly kind: "error"; readonly fatal?: "session" | "not_found" } + | { readonly kind: "hidden" } + | { readonly kind: "visible"; readonly sinceLastReadMs: number }; + +export type PollStep = + | { readonly action: "wait"; readonly delayMs: number } + | { readonly action: "refresh_now" } + | { readonly action: "pause" } + | { readonly action: "stop"; readonly reason: StopReason }; + +const jitter = (base: number, random: number): number => + Math.round(base * (1 + (random * 2 - 1) * POLL_JITTER)); + +/** + * The poller's whole schedule, pure (the browser module only holds the timer and the abort controller). + * The first read comes `POLL_BASE_MS` after load; an unchanged answer doubles the delay up to + * `POLL_MAX_MS` (the backend refreshes a live job from the cluster at most once a minute, so asking + * faster adds load and no freshness) and a changed one resets it; an error doubles it too, and the fifth + * in a row stops; a terminal or unknown state, a 401, a not-found and the two-hour mark stop for good; + * a hidden tab pauses and a visible one refreshes at once if the last read is stale. `random` is in + * [0, 1) and moves each delay by at most 15 percent. Once stopped it stays stopped. + */ +export function nextPoll( + prev: PollState, + event: PollEvent, + nowMs: number, + random = 0.5, +): { step: PollStep; next: PollState } { + const stop = (reason: StopReason): { step: PollStep; next: PollState } => ({ + step: { action: "stop", reason }, + next: { ...prev, stopped: prev.stopped ?? reason }, + }); + if (prev.stopped !== null) return { step: { action: "stop", reason: prev.stopped }, next: prev }; + if (nowMs - prev.sinceMs >= POLL_HARD_STOP_MS) return stop("hard_stop"); + switch (event.kind) { + case "hidden": + return { step: { action: "pause" }, next: prev }; + case "visible": { + const next = { ...prev, delayMs: POLL_BASE_MS }; + return event.sinceLastReadMs > POLL_STALE_AFTER_MS + ? { step: { action: "refresh_now" }, next } + : { step: { action: "wait", delayMs: jitter(POLL_BASE_MS, random) }, next }; + } + case "error": { + if (event.fatal) return stop(event.fatal); + const errors = prev.errors + 1; + if (errors >= POLL_ERROR_LIMIT) + return { ...stop("errors"), next: { ...prev, errors, stopped: "errors" } }; + const base = Math.min(prev.delayMs * 2, POLL_MAX_MS); + return { + step: { action: "wait", delayMs: jitter(base, random) }, + next: { ...prev, errors, delayMs: base }, + }; + } + case "read": { + if (!isComputeJobState(event.state)) return stop("unknown_state"); + if (isTerminalComputeState(event.state)) return stop("terminal"); + const base = event.changed ? POLL_BASE_MS : Math.min(prev.delayMs * 2, POLL_MAX_MS); + return { + step: { action: "wait", delayMs: jitter(base, random) }, + next: { ...prev, errors: 0, delayMs: base }, + }; + } + } +} diff --git a/src/lib/compute-logs.test.ts b/src/lib/compute-logs.test.ts new file mode 100644 index 00000000..751237b1 --- /dev/null +++ b/src/lib/compute-logs.test.ts @@ -0,0 +1,213 @@ +/** + * The job log made safe to show (phase E3, website#496; ADR 0029). The inputs are the real log the + * backend served for a hostile job (`test/fixtures/compute/logs.json`, captured by + * `scripts/capture-compute-job-fixtures.mjs`) and hand-built strings for the cases a route cannot be + * made to serve (a 1 MiB single line, a sequence cut at the end). + */ +import { describe, expect, it } from "vitest"; +import logs from "../../test/fixtures/compute/logs.json"; +import { + LOG_EARLIER_STEP, + LOG_LINE_MAX, + LOG_TAIL_DEFAULT, + logWindow, + prepareLog, + stripAnsi, +} from "./compute-logs"; + +const ESC = "\u001b"; + +describe("stripAnsi", () => { + it("removes color and cursor sequences and keeps the visible text", () => { + expect(stripAnsi(`${ESC}[31mred${ESC}[0m and ${ESC}[2K${ESC}[1;32mgreen${ESC}[m`)).toBe( + "red and green", + ); + expect(stripAnsi(`a${ESC}[10;20Hb${ESC}[?25lc`)).toBe("abc"); + }); + + it("removes an operating-system command (a hyperlink) ended by BEL or by ESC backslash", () => { + expect(stripAnsi(`${ESC}]8;;https://example.org\u0007text${ESC}]8;;\u0007 end`)).toBe( + "text end", + ); + expect(stripAnsi(`${ESC}]0;title${ESC}\\after`)).toBe("after"); + }); + + it("removes the two-character forms and the single-byte CSI", () => { + expect(stripAnsi(`x${ESC}7y${ESC}8z${ESC}cw`)).toBe("xyzw"); + expect(stripAnsi(`x${ESC}(By`)).toBe("xy"); + expect(stripAnsi("x\u009b31my")).toBe("xy"); + }); + + it("keeps the text that follows a stripped sequence (the text is the point)", () => { + expect(stripAnsi(`${ESC}[31mALERT${ESC}[0m: disk full`)).toBe("ALERT: disk full"); + }); + + it("does not swallow text after a sequence that never ends", () => { + // An unterminated OSC or CSI would, in a terminal, eat what follows. Here nothing is hidden. + expect(stripAnsi(`start ${ESC}]8;;never ends\nnext line`)).toBe( + `start ${ESC}]8;;never ends\nnext line`, + ); + expect(stripAnsi(`tail ${ESC}[3`)).toBe(`tail ${ESC}[3`); + expect(stripAnsi(`tail ${ESC}`)).toBe(`tail ${ESC}`); + }); + + it("leaves a log with no escape unchanged", () => { + const plain = 'Traceback (most recent call last):\n File "x.py", line 1\nValueError: no\n'; + expect(stripAnsi(plain)).toBe(plain); + }); +}); + +describe("prepareLog", () => { + it("turns a bare carriage return into a line break so an overwrite hides nothing", () => { + expect(prepareLog("progress 10%\rprogress 50%\rdone").lines).toEqual([ + "progress 10%", + "progress 50%", + "done", + ]); + }); + + it("treats CRLF as one line break", () => { + expect(prepareLog("one\r\ntwo\r\n").lines).toEqual(["one", "two"]); + }); + + it("writes out NUL, BEL and NEL as visible escapes", () => { + const { lines } = prepareLog("a\u0000b\u0007c\u0085d"); + expect(lines).toEqual(["a\\x00b\\x07c\\x85d"]); + }); + + it("writes out a right-to-left override, an isolate and a zero-width space", () => { + const { lines } = prepareLog("a‮b⁦c⁩d​e"); + expect(lines).toEqual(["a\\u202eb\\u2066c\\u2069d\\u200be"]); + }); + + it("writes out the three implicit direction marks too (LRM, RLM, ALM), unlike mail", () => { + expect(prepareLog("a\u200eb\u200fc\u061cd").lines).toEqual(["a\\u200eb\\u200fc\\u061cd"]); + const joined = prepareLog("x\u200e\u200f\u061cy").lines.join(""); + expect(joined).not.toMatch(/[\u200e\u200f\u061c]/); + }); + + it("writes out a stray ESC as a visible escape", () => { + expect(prepareLog(`x${ESC}\u00e9`).lines).toEqual(["x\\x1b\u00e9"]); + }); + + it("keeps tabs and newlines", () => { + expect(prepareLog("a\tb\nc").lines).toEqual(["a\tb", "c"]); + }); + + it("changes nothing that markup could use: angle brackets, ampersands, quotes, backslashes", () => { + const hostile = + " & 'q' back\\slash"; + expect(prepareLog(hostile).lines).toEqual([hostile]); + }); + + it("leaves an ordinary traceback identical after the pass", () => { + const traceback = [ + "Traceback (most recent call last):", + ' File "main.py", line 3, in ', + " print(1 / 0)", + "ZeroDivisionError: division by zero", + ]; + expect(prepareLog(`${traceback.join("\n")}\n`).lines).toEqual(traceback); + expect(prepareLog(traceback.join("\n")).lines).toEqual(traceback); + }); + + it("handles the real log the backend served for a hostile job", () => { + const { lines } = prepareLog(logs.hostile.text); + const joined = lines.join("\n"); + // Formatting is gone, the visible words stay. + expect(joined).toContain("red and cleared"); + expect(joined).toContain("link text end"); + expect(joined).not.toContain(ESC); + // The progress bar became lines, nothing was overwritten. + expect(lines).toContain("progress 10%"); + expect(lines).toContain("progress 50%"); + expect(lines).toContain("progress 100%"); + // The hostile characters are visible, and the markup is untouched. + expect(joined).toContain("nul \\x00 bell \\x07 next-line \\x85 end"); + expect(joined).toContain( + "override \\u202egnp.exe and isolate \\u2066x\\u2069 and zero\\u200bwidth", + ); + expect(joined).toContain(' &'); + // No control or bidirectional character survives. + // biome-ignore lint/suspicious/noControlCharactersInRegex: the point of the check. + expect(joined).not.toMatch(/[\u0000-\u0008\u000b-\u001f\u007f-\u009f​-‏‪-‮⁦-⁩]/); + }); + + it("caps a 1 MiB single line at the line limit and says how much was cut", () => { + const line = "x".repeat(1024 * 1024); + const { lines } = prepareLog(line); + expect(lines.length).toBe(1); + expect(lines[0].startsWith("x".repeat(LOG_LINE_MAX))).toBe(true); + expect(lines[0]).toBe( + `${"x".repeat(LOG_LINE_MAX)}[line cut: ${1024 * 1024 - LOG_LINE_MAX} more characters]`, + ); + }); + + it("leaves a line of exactly the limit alone and cuts one character over", () => { + expect(prepareLog("y".repeat(LOG_LINE_MAX)).lines).toEqual(["y".repeat(LOG_LINE_MAX)]); + expect(prepareLog("y".repeat(LOG_LINE_MAX + 1)).lines).toEqual([ + `${"y".repeat(LOG_LINE_MAX)}[line cut: 1 more characters]`, + ]); + }); + + it("does not cut a surrogate pair in half", () => { + const text = `${"a".repeat(LOG_LINE_MAX - 1)}\u{1F600}\u{1F600}`; + const [line] = prepareLog(text).lines; + expect(line.includes("\ud83d[")).toBe(false); + expect(line).toMatch(/^a+\[line cut: 2 more characters\]$/); + }); + + it("counts the cut in whole characters, not code units", () => { + const text = `${"a".repeat(LOG_LINE_MAX)}\u{1F600}\u{1F600}\u{1F600}`; + expect(prepareLog(text).lines[0]).toMatch(/\[line cut: 3 more characters\]$/); + }); + + it("gives no lines for an empty log, and one empty line for a lone newline", () => { + expect(prepareLog("").lines).toEqual([]); + expect(prepareLog("\n").lines).toEqual([""]); + }); + + it("handles text that is not valid UTF-8 after decoding (U+FFFD stays)", () => { + const decoded = new TextDecoder().decode(new Uint8Array([0x61, 0xff, 0xfe, 0x62])); + expect(prepareLog(decoded).lines).toEqual(["a��b"]); + }); +}); + +describe("logWindow", () => { + const lines = Array.from({ length: 5431 }, (_, i) => `line ${i + 1}`); + + it("shows the last 200 lines by default and counts what is hidden", () => { + const w = logWindow(lines, LOG_TAIL_DEFAULT); + expect(w.total).toBe(5431); + expect(w.shown).toBe(200); + expect(w.hidden).toBe(5231); + expect(w.text.split("\n").length).toBe(200); + expect(w.text.startsWith("line 5232\n")).toBe(true); + expect(w.text.endsWith("line 5431")).toBe(true); + }); + + it("adds 1,000 earlier lines per step and stops at the start", () => { + const first = logWindow(lines, LOG_TAIL_DEFAULT); + const second = logWindow(lines, first.shown + LOG_EARLIER_STEP); + expect(second.shown).toBe(1200); + expect(second.hidden).toBe(4231); + const all = logWindow(lines, 99999); + expect([all.shown, all.hidden]).toEqual([5431, 0]); + expect(all.text.startsWith("line 1\n")).toBe(true); + }); + + it("shows a short log whole", () => { + const w = logWindow(["a", "b"], 200); + expect([w.text, w.hidden, w.shown, w.total]).toEqual(["a\nb", 0, 2, 2]); + }); + + it("shows an empty log as empty", () => { + expect(logWindow([], 200)).toEqual({ text: "", total: 0, shown: 0, hidden: 0 }); + }); + + it("refuses a count that is not a positive whole number", () => { + for (const bad of [0, -1, 1.5, Number.NaN]) { + expect(() => logWindow(["a"], bad)).toThrow(RangeError); + } + }); +}); diff --git a/src/lib/compute-logs.ts b/src/lib/compute-logs.ts new file mode 100644 index 00000000..326c3f96 --- /dev/null +++ b/src/lib/compute-logs.ts @@ -0,0 +1,94 @@ +/** + * A job's log made safe to show (website ADR 0030 phase E3, issue #496; ADR 0029). Pure: no DOM, no + * network. + * + * The log is the job's own output, so anything it prints can be in it: an escape sequence that rewrites + * a terminal, a carriage return that overwrites the line before it, a right-to-left override that + * reorders what follows, a zero-width character that hides a word, markup. The page shows it with + * `textContent` in a `
`, so markup is already inert; this module makes the rest visible:
+ *
+ *  1. ANSI escape sequences are removed as FORMATTING (color, cursor moves, hyperlinks), keeping the
+ *     text between them. A sequence that never ends is NOT removed and eats nothing: it is written out
+ *     in step 2 with the text after it, so a program cannot hide the rest of its log behind one.
+ *  2. Everything else goes through `displaySafeBody` (the Mail tab's set, nemar-cli's escape set):
+ *     `\r\n` and a bare `\r` become line breaks (nothing can be overwritten), newline and tab stay,
+ *     and every other control, bidirectional character (the three implicit marks too: a program's log
+ *     has no right-to-left prose to protect, so they are written out here, unlike in mail), zero-width
+ *     and invisible character is WRITTEN OUT (`\x1b`, `‮`), which shows that something was there instead of cleaning it away
+ *     silently (the CLI drops them; ADR 0029 shows them).
+ *  3. Lines are capped at 2,000 characters with a visible count of what was cut, so a 1 MiB single line
+ *     cannot freeze the page.
+ *
+ * `<`, `>`, `&`, quotes and backslashes are never altered: an ordinary traceback survives unchanged.
+ */
+import { displaySafeBody } from "./mail-display";
+
+/** The longest line shown; a longer one ends with a visible note of how much was cut. */
+export const LOG_LINE_MAX = 2000;
+/** How many of the last lines show before "Show earlier lines". */
+export const LOG_TAIL_DEFAULT = 200;
+/** How many earlier lines each press of "Show earlier lines" adds. */
+export const LOG_EARLIER_STEP = 1000;
+
+// Terminated sequences only. Bodies may not contain a line break, so an OSC that never ends cannot
+// reach past its own line.
+// biome-ignore lint/suspicious/noControlCharactersInRegex: matching ESC and BEL is the point.
+const OSC = /\u001b\][^\u0007\u001b\n]*(?:\u0007|\u001b\\)/g;
+// biome-ignore lint/suspicious/noControlCharactersInRegex: matching ESC and the C1 CSI is the point.
+const CSI = /(?:\u001b\[|\u009b)[0-?]*[ -/]*[@-~]/g;
+// ESC, any intermediate bytes, one final byte: `ESC 7`, `ESC c`, `ESC ( B`. The introducers of a string
+// (`[` CSI, `]` OSC, P DCS, X SOS, ^ PM, _ APC) are not final bytes: when their long form did not match
+// they stay, and are written out below with the text after them.
+// biome-ignore lint/suspicious/noControlCharactersInRegex: matching ESC is the point.
+const TWO_CHARACTER = /\u001b[ -/]*[0-?@-OQ-WYZ\\`-~]/g;
+
+/** Remove complete ANSI escape sequences, keeping the text between them. */
+export function stripAnsi(text: string): string {
+  // Order matters: the long forms first, or `ESC [` would be taken for a two-character form.
+  return text.replace(OSC, "").replace(CSI, "").replace(TWO_CHARACTER, "");
+}
+
+function capLine(line: string): string {
+  if (line.length <= LOG_LINE_MAX) return line;
+  let cut = LOG_LINE_MAX;
+  // Never end on half of a surrogate pair.
+  const last = line.charCodeAt(cut - 1);
+  if (last >= 0xd800 && last <= 0xdbff) cut -= 1;
+  const rest = [...line.slice(cut)].length;
+  return `${line.slice(0, cut)}[line cut: ${rest} more characters]`;
+}
+
+export interface PreparedLog {
+  /** The lines to show, safe to put in a text node. No trailing empty line for a final newline. */
+  readonly lines: readonly string[];
+}
+
+/** A log made safe: formatting removed, hostile characters written out, long lines capped. */
+export function prepareLog(text: string): PreparedLog {
+  if (text === "") return { lines: [] };
+  const lines = displaySafeBody(stripAnsi(text), { escapeMarks: true }).split("\n");
+  if (lines.length > 1 && lines[lines.length - 1] === "") lines.pop();
+  else if (lines.length === 1 && lines[0] === "") return { lines: [] };
+  return { lines: lines.map(capLine) };
+}
+
+export interface LogWindow {
+  /** The last `shown` lines, joined with newlines. */
+  readonly text: string;
+  readonly total: number;
+  readonly shown: number;
+  /** Earlier lines not shown. */
+  readonly hidden: number;
+}
+
+/** The last `want` lines of a prepared log (all of it when it is shorter). */
+export function logWindow(lines: readonly string[], want: number): LogWindow {
+  if (!Number.isInteger(want) || want < 1) throw new RangeError("a window is 1 or more lines");
+  const shown = Math.min(want, lines.length);
+  return {
+    text: lines.slice(lines.length - shown).join("\n"),
+    total: lines.length,
+    shown,
+    hidden: lines.length - shown,
+  };
+}
diff --git a/src/lib/compute-view.ts b/src/lib/compute-view.ts
index e9f13252..894d91e3 100644
--- a/src/lib/compute-view.ts
+++ b/src/lib/compute-view.ts
@@ -333,6 +333,32 @@ export interface JobRowView {
   readonly chargeKind: "terminal" | "live" | "unknown";
 }
 
+/**
+ * What a job's charge reads, for the table and the job page alike so the two cannot disagree. A
+ * finished, failed or canceled job prints a true `0.000`; a live one reads "Not charged yet" until the
+ * backend reports an amount; a state this build does not know prints its amount when it has one and
+ * "Not known" when it has none.
+ */
+export function chargeText(job: { state: string; charged_milli: number }): {
+  readonly text: string;
+  readonly kind: JobRowView["chargeKind"];
+} {
+  const L = COMPUTE_LABELS.jobs;
+  if (isComputeJobState(job.state) && isTerminalComputeState(job.state)) {
+    return { text: suFixed(job.charged_milli), kind: "terminal" };
+  }
+  if (isComputeJobState(job.state)) {
+    return {
+      text: job.charged_milli > 0 ? suFixed(job.charged_milli) : L.notChargedYet,
+      kind: "live",
+    };
+  }
+  return {
+    text: job.charged_milli > 0 ? suFixed(job.charged_milli) : L.chargeUnknown,
+    kind: "unknown",
+  };
+}
+
 /**
  * One job as cells of data. The SSR template and the browser script are two thin renderers over this.
  * A finished job with no charge prints a true `0.000`; a live job with none reads "Not charged yet",
@@ -340,21 +366,9 @@ export interface JobRowView {
  * has one and "Not known" when it has none (a claim of zero or of "not yet" would be a guess).
  */
 export function jobRowView(job: ComputeJob): JobRowView {
-  const L = COMPUTE_LABELS.jobs;
   const known = isComputeJobState(job.state);
   const terminal = known && isTerminalComputeState(job.state);
-  let charged: string;
-  let chargeKind: JobRowView["chargeKind"];
-  if (terminal) {
-    charged = suFixed(job.charged_milli);
-    chargeKind = "terminal";
-  } else if (known) {
-    charged = job.charged_milli > 0 ? suFixed(job.charged_milli) : L.notChargedYet;
-    chargeKind = "live";
-  } else {
-    charged = job.charged_milli > 0 ? suFixed(job.charged_milli) : L.chargeUnknown;
-    chargeKind = "unknown";
-  }
+  const { text: charged, kind: chargeKind } = chargeText(job);
   const badge = jobBadge(job.state);
   return {
     id: displaySafeLine(job.id),
diff --git a/src/lib/mail-display.test.ts b/src/lib/mail-display.test.ts
index dbdf87c8..fc3fabe9 100644
--- a/src/lib/mail-display.test.ts
+++ b/src/lib/mail-display.test.ts
@@ -65,6 +65,12 @@ describe("displaySafeLine", () => {
     expect(displaySafeLine(input)).toBe(expected);
   });
 
+  it("writes out the three direction marks too when asked, and keeps newline and tab", () => {
+    expect(displaySafeBody("a\u200eb\u200fc\u061cd\n\te", { escapeMarks: true })).toBe(
+      "a\\u200eb\\u200fc\\u061cd\n\te",
+    );
+  });
+
   it("leaves ordinary text, including non-Latin scripts and emoji, alone", () => {
     const text = "Re: Zugang für Daten – ورود به سامانه 研究";
     expect(displaySafeLine(text)).toBe(text);
@@ -94,6 +100,9 @@ describe("the wider escape set (mirrors nemar-cli display-name)", () => {
   it("keeps exactly newline, tab, LRM, RLM and ALM in a body", () => {
     expect(displaySafeBody("a\n\tb\u200ec\u200fd\u061ce")).toBe("a\n\tb\u200ec\u200fd\u061ce");
     expect(displaySafeBody("a\u200bb\u200cc\u2066d")).toBe("a\\u200bb\\u200cc\\u2066d");
+    // The default is unchanged by the option: marks stay unless asked.
+    expect(displaySafeBody("a\u200eb", {})).toBe("a\u200eb");
+    expect(displaySafeBody("a\u200eb", { escapeMarks: false })).toBe("a\u200eb");
   });
 
   it("writes a line's escapes the way a body does, apart from newline and tab", () => {
diff --git a/src/lib/mail-display.ts b/src/lib/mail-display.ts
index f5a0b911..13ef6e46 100644
--- a/src/lib/mail-display.ts
+++ b/src/lib/mail-display.ts
@@ -35,6 +35,9 @@ const UNSAFE_CHARACTERS =
 /** What a body keeps besides ordinary text: newline, tab and the three implicit direction marks. */
 const BODY_KEPT: ReadonlySet = new Set(["\n", "\t", "\u200e", "\u200f", "\u061c"]);
 
+/** What a body keeps when the direction marks are written out too: newline and tab only. */
+const BODY_KEPT_NO_MARKS: ReadonlySet = new Set(["\n", "\t"]);
+
 function escapeCharacter(c: string): string {
   if (c === "\n") return "\\n";
   if (c === "\r") return "\\r";
@@ -52,12 +55,14 @@ export function displaySafeLine(text: string): string {
 
 /**
  * A message body: line endings normalized to `\n`, newlines and tabs kept, everything else in
- * the set written out except the three direction marks.
+ * the set written out except the three direction marks. `escapeMarks` writes those out too: a
+ * program's log has no right-to-left prose to protect (the job page passes it; mail does not).
  */
-export function displaySafeBody(text: string): string {
+export function displaySafeBody(text: string, options: { escapeMarks?: boolean } = {}): string {
+  const kept = options.escapeMarks === true ? BODY_KEPT_NO_MARKS : BODY_KEPT;
   return text
     .replace(/\r\n?/g, "\n")
-    .replace(UNSAFE_CHARACTERS, (c) => (BODY_KEPT.has(c) ? c : escapeCharacter(c)));
+    .replace(UNSAFE_CHARACTERS, (c) => (kept.has(c) ? c : escapeCharacter(c)));
 }
 
 // ---------------------------------------------------------------------------
diff --git a/src/middleware.test.ts b/src/middleware.test.ts
index b5bc625c..33c48122 100644
--- a/src/middleware.test.ts
+++ b/src/middleware.test.ts
@@ -2378,6 +2378,47 @@ describe("the authorization page's headers", () => {
   });
 });
 
+describe("the compute pages never enter the edge cache (website#496)", () => {
+  const JOB = "/compute/jobs/job_0123456789abcdef01234567";
+
+  it("bypasses /compute and everything under it, and nothing that only starts with it", () => {
+    for (const path of ["/compute", "/compute/", JOB, "/compute/jobs"]) {
+      expect(routeBypassesEdgeCache(path), path).toBe(true);
+    }
+    for (const path of ["/computex", "/compute-x", "/discover", "/"]) {
+      expect(routeBypassesEdgeCache(path), path).toBe(false);
+    }
+  });
+
+  it("does not open the cache for a signed-out request to a job page, whatever the page says", async () => {
+    const opened: string[] = [];
+    const storage = {
+      open: async (name: string) => {
+        opened.push(name);
+        return { match: async () => undefined, put: async () => {} };
+      },
+    } as unknown as CacheStorage;
+    const make = (url: string, cookie: boolean) =>
+      ({
+        request: new Request(url),
+        locals: { runtime: { caches: storage } },
+        cookies: { has: () => cookie, get: () => undefined },
+      }) as unknown as APIContext;
+    const publicPage = async () =>
+      new Response("x", { headers: { "Cache-Control": "public, max-age=60" } });
+    for (const origin of [`https://${APP_HOST}`, "http://localhost:4321"]) {
+      for (const cookie of [false, true]) {
+        for (const path of ["/compute", JOB]) {
+          const res = (await onRequest(make(`${origin}${path}`, cookie), publicPage)) as Response;
+          expect(res.status, `${origin}${path} ${cookie}`).toBe(200);
+          expect(res.headers.get("x-nemar-cache"), `${origin}${path}`).toBeNull();
+        }
+      }
+    }
+    expect(opened).toEqual([]);
+  });
+});
+
 // Compute access on the session user (website ADR 0030). The bodies are the
 // real `GET /auth/me` answers captured for each state by
 // scripts/capture-compute-fixtures.mjs (nemar-cli 3a869903).
diff --git a/src/middleware.ts b/src/middleware.ts
index dba101f3..c32e389b 100644
--- a/src/middleware.ts
+++ b/src/middleware.ts
@@ -132,14 +132,21 @@ export function isOidcAuthorizeRoute(pathname: string): boolean {
 }
 
 /**
- * Routes whose responses never enter the edge cache, whatever they say: everything under `/oauth`.
- * Each answer there is per visitor and some carry a one-time code; the page sends `no-store`, and
- * this keeps the cache from even being consulted for them.
+ * Routes whose responses never enter the edge cache, whatever they say: everything under `/oauth`
+ * and everything under `/compute`. Each answer there is per visitor; `/oauth` carries one-time codes
+ * and a job page can carry bearer result links after a click (website#496). The page sends
+ * `no-store`, and this keeps the cache from even being consulted for them, so even a signed-out
+ * redirect or a mistaken header cannot put one in the shared cache.
  *
  * Exported for the middleware unit tests.
  */
 export function routeBypassesEdgeCache(pathname: string): boolean {
-  return pathname === "/oauth" || pathname.startsWith("/oauth/");
+  return (
+    pathname === "/oauth" ||
+    pathname.startsWith("/oauth/") ||
+    pathname === "/compute" ||
+    pathname.startsWith("/compute/")
+  );
 }
 
 /** Base connect-src for every route: same-site APIs plus the raw README host. */
diff --git a/src/pages/api/v1/[...path].ts b/src/pages/api/v1/[...path].ts
index d06d3619..a7747222 100644
--- a/src/pages/api/v1/[...path].ts
+++ b/src/pages/api/v1/[...path].ts
@@ -1,6 +1,7 @@
 import type { APIRoute } from "astro";
 import { apiBase, copySetCookies } from "../../../lib/api-base";
 import { apiFetch } from "../../../lib/api-fetch";
+import { COMPUTE_TRUNCATED_HEADER } from "../../../lib/compute-contract";
 import { isAuthorizationProxyPath, isSafeProxyPath } from "../../../lib/proxy-path";
 
 /**
@@ -118,6 +119,10 @@ const proxy: APIRoute = async ({ params, request }) => {
   const contentType = upstream.headers.get("content-type");
   if (contentType) respHeaders.set("Content-Type", contentType);
   respHeaders.set("Cache-Control", "no-store");
+  // The compute log route marks a log cut at 1 MiB with this header (website#496). It is the ONE
+  // upstream header besides the content type that the job page reads, and this is its only reader.
+  const truncated = upstream.headers.get(COMPUTE_TRUNCATED_HEADER);
+  if (truncated !== null) respHeaders.set(COMPUTE_TRUNCATED_HEADER, truncated);
   copySetCookies(upstream, respHeaders);
 
   return new Response(upstream.body, { status: upstream.status, headers: respHeaders });
diff --git a/src/pages/compute/jobs/[id].astro b/src/pages/compute/jobs/[id].astro
new file mode 100644
index 00000000..e3251abd
--- /dev/null
+++ b/src/pages/compute/jobs/[id].astro
@@ -0,0 +1,400 @@
+---
+/**
+ * `/compute/jobs/`: one job of the signed-in person, its state as it changes, its resources and
+ * charge, its log on request, its result links on request (renewable) and a cancel for a live job
+ * (website ADR 0030 phase E3; epic nemarOrg/website#493, issue #496). App host only. This is the page
+ * the "results ready" email links to (`computeJobPageUrl` in nemar-cli), so it must be safe to open from
+ * a mail client: the server-rendered HTML names no result and carries no link.
+ *
+ * Order matters here, and `test/compute-job-ui.test.ts` pins it:
+ *  1. `Cache-Control: private, no-store` and `Referrer-Policy: no-referrer` are set BEFORE any return,
+ *     so the 404 and the notices carry them too (ADR 0021 keeps a page-set no-referrer);
+ *  2. a signed-out visitor goes to login with `next=` this path;
+ *  3. an id that is not `job_` and 24 hex digits is the SAME 404 as an unknown one, without calling the
+ *     API; a job that is absent or somebody else's is that 404 too, so the page is no oracle;
+ *  4. the job is read over the service binding with the visitor's cookie;
+ *  5. only for a finished job, `GET /compute/me` for `limits.results_days` (its own try/catch: a failure
+ *     degrades one line to "kept for a limited time" and never blocks the page).
+ * This frontmatter never imports the log or result clients: viewing a job mints nothing.
+ */
+import ComputeBadge from "../../../components/ComputeBadge.astro";
+import ComputeJobActions from "../../../components/ComputeJobActions.astro";
+import ComputeJobLogs from "../../../components/ComputeJobLogs.astro";
+import ComputeJobResults from "../../../components/ComputeJobResults.astro";
+import ComputeJobTimeline from "../../../components/ComputeJobTimeline.astro";
+import ComputePanel from "../../../components/ComputePanel.astro";
+import ComputePanelNotice from "../../../components/ComputePanelNotice.astro";
+import Base from "../../../layouts/Base.astro";
+import { apiFetch } from "../../../lib/api-fetch";
+import { getSession } from "../../../lib/auth";
+import {
+  ComputeApiError,
+  type ComputeInit,
+  type ComputeJobRead,
+  getComputeJob,
+  getComputeMe,
+} from "../../../lib/compute-api";
+import { COMPUTE_JOB_ID_RE } from "../../../lib/compute-contract";
+import { JOB_COPY } from "../../../lib/compute-copy";
+import { nowMs } from "../../../lib/compute-job-live";
+import { jobDetailView, resultsPanelView } from "../../../lib/compute-job-view";
+import { COMPUTE_LABELS } from "../../../lib/compute-labels";
+import { noticeFor } from "../../../lib/compute-view";
+
+// 1. Before any return.
+Astro.response.headers.set("Cache-Control", "private, no-store");
+Astro.response.headers.set("Referrer-Policy", "no-referrer");
+
+const id = Astro.params.id ?? "";
+const path = `/compute/jobs/${id}`;
+
+// 2. Signed out: login, then back here.
+const session = getSession(Astro.locals);
+if (!session) {
+  return Astro.redirect(`/login?next=${encodeURIComponent(path)}`);
+}
+
+const cookieHeader = Astro.request.headers.get("cookie") ?? undefined;
+const init: ComputeInit = { cookieHeader, fetch: apiFetch };
+
+type Outcome =
+  | { kind: "ok"; read: ComputeJobRead }
+  | { kind: "not_found" }
+  | { kind: "not_available" }
+  | { kind: "account"; sentence: string | null }
+  | { kind: "failed"; error: unknown };
+
+// 3 and 4.
+let outcome: Outcome;
+if (!COMPUTE_JOB_ID_RE.test(id)) {
+  outcome = { kind: "not_found" };
+} else {
+  try {
+    outcome = { kind: "ok", read: await getComputeJob(id, init) };
+  } catch (error) {
+    console.error(
+      `[compute-job] read failed: ${error instanceof ComputeApiError ? `${error.kind} ${error.status}` : String(error)}`,
+    );
+    if (error instanceof ComputeApiError && error.code === "job_not_found") {
+      outcome = { kind: "not_found" };
+    } else if (error instanceof ComputeApiError && error.kind === "session") {
+      return Astro.redirect(`/login?next=${encodeURIComponent(path)}`);
+    } else if (error instanceof ComputeApiError && error.kind === "account") {
+      outcome = { kind: "account", sentence: error.sentence ?? null };
+    } else if (
+      error instanceof ComputeApiError &&
+      error.kind === "unavailable" &&
+      error.status === 404
+    ) {
+      // A 404 with no job_not_found code is the release gate: an API without the compute routes.
+      outcome = { kind: "not_available" };
+    } else {
+      outcome = { kind: "failed", error };
+    }
+  }
+}
+if (outcome.kind === "not_found") Astro.response.status = 404;
+
+// 5. The retention policy, only for a finished job.
+let resultsDays: number | null = null;
+if (outcome.kind === "ok" && outcome.read.job.state === "finished") {
+  try {
+    resultsDays = (await getComputeMe(init)).resultsDays;
+  } catch (error) {
+    console.error(
+      `[compute-job] policy read failed: ${error instanceof ComputeApiError ? `${error.kind} ${error.status}` : String(error)}`,
+    );
+  }
+}
+
+const view = outcome.kind === "ok" ? jobDetailView(outcome.read) : null;
+const results =
+  outcome.kind === "ok"
+    ? resultsPanelView({
+        state: outcome.read.job.state,
+        endedAt: outcome.read.job.ended_at,
+        resultsDays,
+        nowMs: nowMs(),
+      })
+    : null;
+const source = outcome.kind === "ok" ? outcome.read.job.source : "";
+const L = JOB_COPY;
+---
+
+
+  
+

{L.back}

+ + {outcome.kind === "not_found" && ( +
+

{L.notFoundTitle}

+

{L.notFoundBody}

+
+ )} + + {outcome.kind === "not_available" && ( +
+

{L.notAvailableTitle}

+

{L.notAvailableBody}

+
+ )} + + {outcome.kind === "account" && ( +
+

{L.pageTitle}

+

{COMPUTE_LABELS.access.forbidden}

+ {outcome.sentence &&

{outcome.sentence}

} +
+ )} + + {outcome.kind === "failed" && ( + <> +

{L.pageTitle}

+ + + )} + + {view && results && outcome.kind === "ok" && ( +
+
+

+ {L.jobHeading} {view.id} +

+ {view.title &&

{view.title}

} +

+ {L.app}: {view.app} {view.appVersion} + {L.frontDoor}: {view.frontDoor} +

+

+ + + + +

+ + + +

+
+ + + + + + +
+ {view.rows.map((row) => ( +
+
{row.label}
+
+ {row.time ? : row.value} +
+
+ ))} +
+
{L.rows.charged}
+
{view.charge.text}
+
+
+

{view.unitsNote}

+
+ + + + + + + + + + + + + + +
+ )} +
+ + + + + diff --git a/test/api-proxy-compute-log.test.ts b/test/api-proxy-compute-log.test.ts new file mode 100644 index 00000000..bd952296 --- /dev/null +++ b/test/api-proxy-compute-log.test.ts @@ -0,0 +1,76 @@ +/** + * The same-origin proxy and the compute log's `X-Nemar-Truncated` header (website#496). The upstream is + * a real binding answering what the real log route answers (headers captured in + * `test/fixtures/compute/logs.json`). + */ +import { describe, expect, it } from "vitest"; +import { COMPUTE_TRUNCATED_HEADER } from "../src/lib/compute-contract"; +import { type ApiBinding, runWithRequestContext } from "../src/lib/request-context"; +import { GET } from "../src/pages/api/v1/[...path]"; +import logs from "./fixtures/compute/logs.json"; + +const JOB = "job_0123456789abcdef01234567"; + +async function through(upstream: Response): Promise { + const binding: ApiBinding = { fetch: async () => upstream }; + const request = new Request(`https://test.nemar.org/api/v1/compute/jobs/${JOB}/logs`); + return (await runWithRequestContext({ apiBinding: binding, visitorIp: "198.51.100.7" }, () => + GET({ params: { path: `compute/jobs/${JOB}/logs` }, request } as unknown as Parameters< + typeof GET + >[0]), + )) as Response; +} + +describe("the proxy and the truncated-log header", () => { + it("forwards the header the real route sets on a log cut at 1 MiB", async () => { + const t = logs.truncated; + const res = await through(new Response("x".repeat(64), { status: 200, headers: t.headers })); + expect(res.headers.get(COMPUTE_TRUNCATED_HEADER)).toBe("true"); + expect(await res.text()).toBe("x".repeat(64)); + }); + + it("sets nothing when the upstream sent none", async () => { + const res = await through( + new Response(logs.live.text, { status: 200, headers: logs.live.headers }), + ); + expect(res.headers.has(COMPUTE_TRUNCATED_HEADER)).toBe(false); + expect(await res.text()).toBe(logs.live.text); + expect(res.headers.get("content-type")).toBe("text/plain; charset=utf-8"); + }); + + it("still answers no-store when the upstream says public", async () => { + const res = await through( + new Response("a", { + headers: { "Cache-Control": "public, max-age=60", "X-Nemar-Truncated": "true" }, + }), + ); + expect(res.headers.get("cache-control")).toBe("no-store"); + }); + + it("copies Set-Cookie as before and no other upstream header", async () => { + const upstream = new Response("a", { headers: { "X-Secret-Upstream": "1", Etag: "abc" } }); + upstream.headers.append("Set-Cookie", "a=1; Path=/"); + upstream.headers.append("Set-Cookie", "b=2; Path=/"); + const res = await through(upstream); + expect(res.headers.getSetCookie()).toEqual(["a=1; Path=/", "b=2; Path=/"]); + expect(res.headers.has("x-secret-upstream")).toBe(false); + expect(res.headers.has("etag")).toBe(false); + }); + + it("is the only reader of the header in the source", async () => { + const files = import.meta.glob("/src/**/*.{ts,astro}", { + query: "?raw", + import: "default", + eager: true, + }) as Record; + const readers = Object.entries(files) + .filter(([, src]) => + /x-nemar-truncated/i.test(src.replace(/\/\*[\s\S]*?\*\/|\/\/[^\n]*/g, "")), + ) + .map(([name]) => name); + // The literal is spelled in the contract transcription only; every reader imports the constant. + expect( + readers.filter((n) => !n.endsWith("compute-contract.ts") && !n.endsWith(".test.ts")), + ).toEqual([]); + }); +}); diff --git a/test/compute-contract-drift.test.ts b/test/compute-contract-drift.test.ts index 3ada178a..a3db86bb 100644 --- a/test/compute-contract-drift.test.ts +++ b/test/compute-contract-drift.test.ts @@ -2,8 +2,12 @@ import { describe, expect, it } from "vitest"; import { extractNumber, extractNumberMap, + extractProduct, extractRegexSource, extractSatisfiedArray, + extractSatisfiedNumberMap, + extractString, + extractStringMap, } from "./compute-contract-drift"; import { extractStringArray } from "./mail-contract-drift"; @@ -28,7 +32,37 @@ describe("the extractors", () => { ); }); + it("read the maps, strings and products the job page's contract files use", () => { + const map = [ + "export const STATUS = {", + " a: 403,", + " // 404: a note with a colon: and a number 12", + " b_c: 404,", + "} as const satisfies Record;", + ].join("\n"); + expect(extractSatisfiedNumberMap(map, "STATUS")).toEqual({ a: 403, b_c: 404 }); + const sentences = [ + "export const SENT = {", + " ...OTHER,", + ' one: "First sentence.",', + " two:", + ' "Second; with a semicolon.",', + "} as const satisfies Partial>;", + ].join("\n"); + expect(extractStringMap(sentences, "SENT")).toEqual({ + one: "First sentence.", + two: "Second; with a semicolon.", + }); + expect(extractString('export const H = "X-Nemar-Truncated";', "H")).toBe("X-Nemar-Truncated"); + expect(extractProduct("export const MAX = 1024 * 1024;", "MAX")).toBe(1_048_576); + expect(extractProduct("export const MAX = 1000;", "MAX")).toBe(1000); + }); + it("answer undefined for a computed value, never a guess", () => { + expect(extractProduct("export const N = make(5);", "N")).toBeUndefined(); + expect(extractSatisfiedNumberMap("export const M = build();", "M")).toBeUndefined(); + expect(extractStringMap("export const M = build();", "M")).toBeUndefined(); + expect(extractString("export const S = `a${b}`;", "S")).toBeUndefined(); expect(extractNumber("export const N = 5 * 5;", "N")).toBeUndefined(); expect(extractStringArray("export const L = make();", "L")).toBeUndefined(); expect(extractNumberMap("export const M = build();", "M")).toBeUndefined(); diff --git a/test/compute-contract-drift.ts b/test/compute-contract-drift.ts index 21bd77c6..8462a957 100644 --- a/test/compute-contract-drift.ts +++ b/test/compute-contract-drift.ts @@ -71,3 +71,47 @@ export function extractTargets(source: string, name: string): Record 0 ? out : undefined; } + +/** `export const NAME = { a: 403, // note\n b: 401 } as const satisfies Record<...>;` -> the numbers. */ +export function extractSatisfiedNumberMap( + source: string, + name: string, +): Record | undefined { + const m = stripComments(source).match( + new RegExp(`export\\s+const\\s+${name}\\s*=\\s*\\{([^}]*)\\}\\s*as\\s+const\\s+satisfies[^;]*;`), + ); + if (!m) return undefined; + const out: Record = {}; + for (const entry of m[1].matchAll(/(\w+)\s*:\s*(\d+)/g)) out[entry[1]] = Number(entry[2]); + return Object.keys(out).length > 0 ? out : undefined; +} + +/** `export const NAME = { a: "text", b:\n "more" } as const ...;` -> the strings (spread entries are skipped). */ +export function extractStringMap(source: string, name: string): Record | undefined { + const stripped = stripComments(source); + const start = stripped.search(new RegExp(`export\\s+const\\s+${name}\\s*=\\s*\\{`)); + if (start < 0) return undefined; + const end = stripped.indexOf("} as const", start); + if (end < 0) return undefined; + const out: Record = {}; + for (const m of stripped.slice(start, end).matchAll(/(\w+)\s*:\s*"((?:[^"\\]|\\.)*)"/g)) { + out[m[1]] = m[2]; + } + return Object.keys(out).length > 0 ? out : undefined; +} + +/** `export const NAME = "text";` -> the string. */ +export function extractString(source: string, name: string): string | undefined { + return stripComments(source).match( + new RegExp(`export\\s+const\\s+${name}\\s*=\\s*"((?:[^"\\\\]|\\\\.)*)"\\s*;`), + )?.[1]; +} + +/** `export const NAME = 1024 * 1024;` -> 1048576 (a product of whole numbers; anything else is `undefined`). */ +export function extractProduct(source: string, name: string): number | undefined { + const m = stripComments(source).match( + new RegExp(`export\\s+const\\s+${name}\\s*=\\s*(\\d+(?:\\s*\\*\\s*\\d+)*)\\s*;`), + ); + if (!m) return undefined; + return m[1].split("*").reduce((a, n) => a * Number(n.trim()), 1); +} diff --git a/test/compute-dashboard-ui.test.ts b/test/compute-dashboard-ui.test.ts index ab37099e..fa7b84d0 100644 --- a/test/compute-dashboard-ui.test.ts +++ b/test/compute-dashboard-ui.test.ts @@ -47,6 +47,18 @@ const ALL_COMPUTE: Record = { ...Object.fromEntries(LIB_FILES), }; +/** The only Compute file that may read the clock: the job page's poller (`test/compute-job-ui.test.ts` pins that nothing else of the job page does). */ +const CLOCK_EXCEPTION = "/src/lib/compute-job-live.ts"; + +/** Files whose purpose is to hold the page's sentences: website-owned wording (website#496). */ +const SENTENCE_FILES = [ + "compute-labels.ts", + "compute-copy.ts", + "compute-failure.ts", + // The one backend sentence transcribed by hand (`upload_expired`), drift-tested. + "compute-contract.ts", +]; + /** Source with comments removed and runs of whitespace collapsed, so a formatter's line breaks do not matter. */ function flat(src: string): string { return src @@ -116,6 +128,9 @@ describe("the page", () => { it("asks for the server's default window and never builds a date from a clock", () => { expect(PAGE).toContain("getComputeUsage(init)"); for (const [name, src] of Object.entries(ALL_COMPUTE)) { + // The one named exception (website#496): the job page's poller holds the timer and reads the + // clock; its schedule is pure and tested with an injected clock (compute-job-view.test.ts). + if (name === CLOCK_EXCEPTION) continue; expect(code(src), name).not.toMatch(/Date\.now\(|new Date\(\)/); } }); @@ -304,7 +319,7 @@ describe("copy", () => { it("types no long sentence at a call site: those live in compute-labels.ts or the copy table", () => { const offenders: string[] = []; for (const [name, src] of Object.entries(ALL_COMPUTE)) { - if (name.endsWith("compute-labels.ts")) continue; + if (SENTENCE_FILES.some((f) => name.endsWith(f))) continue; const stripped = code(src); for (const m of stripped.matchAll(/(["'`])((?:\\.|(?!\1)[^\\\n]){40,})\1/g)) { // A message thrown for a developer (a RangeError) is not text a visitor reads. diff --git a/test/compute-job-contract-drift.test.ts b/test/compute-job-contract-drift.test.ts new file mode 100644 index 00000000..02a39b43 --- /dev/null +++ b/test/compute-job-contract-drift.test.ts @@ -0,0 +1,150 @@ +/** + * Drift check between `src/lib/compute-contract.ts` and nemar-cli's job-route vocabulary (website ADR + * 0030 phase E3 addendum; epic nemarOrg/website#493, issue #496). + * + * Compared as TEXT, like the other contract drift tests: + * - `shared/compute-state.ts`: the launcher, platform and submitter failure codes (the job page has + * one sentence for each) and the status every refusal answers with; + * - `shared/contract/compute-submit.ts`: the truncated-log header, the log and result-page bounds and + * the one refusal sentence this site transcribes (`upload_expired`); + * - `shared/contract/compute-responses.ts`: the cancel outcomes. + * + * CI sparse-checks out nemar-cli `dev` and points `NEMAR_CLI_COMPUTE_STATE`, + * `NEMAR_CLI_COMPUTE_SUBMIT_CONTRACT` and `NEMAR_CLI_COMPUTE_RESPONSES_CONTRACT` at the files. When a + * variable is set it is strict: a missing file FAILS. Without them this looks for a sibling checkout + * and skips visibly when there is none. + * + * EXPECTED RED on a phase PR into the epic branch until the compute epic reaches nemar-cli `dev`. + */ + +import { describe, expect, it } from "vitest"; +import { + COMPUTE_CANCEL_OUTCOMES, + COMPUTE_JOB_ROUTE_STATUS, + COMPUTE_LAUNCHER_FAILURE_CODES, + COMPUTE_LOG_MAX_BYTES, + COMPUTE_PLATFORM_FAILURE_CODES, + COMPUTE_RESULTS_PAGE_MAX, + COMPUTE_SUBMITTER_FAILURE_CODES, + COMPUTE_TRUNCATED_HEADER, + COMPUTE_UPLOAD_EXPIRED_SENTENCE, +} from "../src/lib/compute-contract"; +import { + extractNumber, + extractProduct, + extractSatisfiedNumberMap, + extractString, + extractStringMap, +} from "./compute-contract-drift"; +import { compareLists, extractStringArray } from "./mail-contract-drift"; + +async function nodeFs(): Promise<{ + existsSync(path: string): boolean; + readFileSync(path: string, encoding: string): string; +}> { + const specifier = "node:fs"; + return await import(/* @vite-ignore */ specifier); +} + +async function nodeProcess(): Promise<{ cwd(): string; env: Record }> { + const specifier = "node:process"; + return await import(/* @vite-ignore */ specifier); +} + +const fs = await nodeFs(); +const proc = await nodeProcess(); + +function resolveFromCwd(maybeRelative: string): string { + return maybeRelative.startsWith("/") ? maybeRelative : `${proc.cwd()}/${maybeRelative}`; +} + +interface Source { + readonly envName: string; + readonly candidates: readonly string[]; +} + +function locate(src: Source): { path: string | undefined; strict: boolean } { + const fromEnv = proc.env[src.envName]; + if (fromEnv) return { path: resolveFromCwd(fromEnv), strict: true }; + const found = src.candidates + .map((rel) => decodeURIComponent(new URL(rel, import.meta.url).pathname)) + .find((p) => fs.existsSync(p)); + return { path: found, strict: false }; +} + +const STATE = locate({ + envName: "NEMAR_CLI_COMPUTE_STATE", + candidates: ["../../nemar-cli/shared/compute-state.ts"], +}); +const SUBMIT = locate({ + envName: "NEMAR_CLI_COMPUTE_SUBMIT_CONTRACT", + candidates: ["../../nemar-cli/shared/contract/compute-submit.ts"], +}); +const RESPONSES = locate({ + envName: "NEMAR_CLI_COMPUTE_RESPONSES_CONTRACT", + candidates: ["../../nemar-cli/shared/contract/compute-responses.ts"], +}); + +const SHOULD_SKIP = [STATE, SUBMIT, RESPONSES].some((s) => !s.strict && !s.path); +if (SHOULD_SKIP) { + console.info( + "[compute-job drift] skipped: no nemar-cli checkout found beside this one and the NEMAR_CLI_COMPUTE_* variables are unset.", + ); +} + +function read(src: { path: string | undefined }): string { + expect( + src.path && fs.existsSync(src.path), + `${src.path} is missing: the nemar-cli checkout step failed, or nemar-cli dev does not carry the compute epic yet`, + ).toBe(true); + return fs.readFileSync(src.path as string, "utf8"); +} + +describe.skipIf(SHOULD_SKIP)("the job page's vocabulary agrees with nemar-cli", () => { + it("has the same launcher failure codes in the same order", () => { + const theirs = extractStringArray(read(STATE), "LAUNCHER_FAILURE_CODES"); + expect( + compareLists("LAUNCHER_FAILURE_CODES", COMPUTE_LAUNCHER_FAILURE_CODES, theirs), + ).toBeNull(); + }); + + it("has the same platform failure codes", () => { + const theirs = extractStringArray(read(STATE), "PLATFORM_FAILURE_CODES"); + expect( + compareLists("PLATFORM_FAILURE_CODES", COMPUTE_PLATFORM_FAILURE_CODES, theirs), + ).toBeNull(); + }); + + it("has the same submitter failure codes", () => { + const theirs = extractStringArray(read(STATE), "SUBMITTER_FAILURE_CODES"); + expect( + compareLists("SUBMITTER_FAILURE_CODES", COMPUTE_SUBMITTER_FAILURE_CODES, theirs), + ).toBeNull(); + }); + + it("has the same status for every job-route refusal", () => { + const theirs = extractSatisfiedNumberMap(read(STATE), "COMPUTE_REFUSAL_STATUS"); + expect(theirs, "COMPUTE_REFUSAL_STATUS not found").toBeDefined(); + for (const [code, status] of Object.entries(COMPUTE_JOB_ROUTE_STATUS)) { + expect([code, theirs?.[code]]).toEqual([code, status]); + } + }); + + it("has the same truncated-log header and the same bounds", () => { + const source = read(SUBMIT); + expect(extractString(source, "COMPUTE_TRUNCATED_HEADER")).toBe(COMPUTE_TRUNCATED_HEADER); + expect(extractProduct(source, "COMPUTE_LOG_MAX_BYTES")).toBe(COMPUTE_LOG_MAX_BYTES); + expect(extractNumber(source, "COMPUTE_RESULTS_PAGE_MAX")).toBe(COMPUTE_RESULTS_PAGE_MAX); + }); + + it("has the same sentence for an upload that expired", () => { + const theirs = extractStringMap(read(SUBMIT), "COMPUTE_SUBMIT_SENTENCES"); + expect(theirs, "COMPUTE_SUBMIT_SENTENCES not found").toBeDefined(); + expect(theirs?.upload_expired).toBe(COMPUTE_UPLOAD_EXPIRED_SENTENCE); + }); + + it("has the same cancel outcomes", () => { + const theirs = extractStringArray(read(RESPONSES), "COMPUTE_CANCEL_OUTCOMES"); + expect(compareLists("COMPUTE_CANCEL_OUTCOMES", COMPUTE_CANCEL_OUTCOMES, theirs)).toBeNull(); + }); +}); diff --git a/test/compute-job-ui.test.ts b/test/compute-job-ui.test.ts new file mode 100644 index 00000000..f87a996c --- /dev/null +++ b/test/compute-job-ui.test.ts @@ -0,0 +1,456 @@ +/** + * Wiring guards for the job page (website ADR 0030 phase E3; epic nemarOrg/website#493, issue #496). + * + * Source-level assertions, like `compute-dashboard-ui.test.ts`: Astro files have no rendering harness in + * this repo, so reading the source is the honest cheap check; the browser check against the real local + * backend covers the rest. The behavior of the pieces has its own tests: `compute-job-view.test.ts`, + * `compute-logs.test.ts`, `compute-failure.test.ts`, `compute-api-job.test.ts`. + */ +import { describe, expect, it } from "vitest"; +import ACTIONS from "../src/components/ComputeJobActions.astro?raw"; +import LOGS from "../src/components/ComputeJobLogs.astro?raw"; +import RESULTS from "../src/components/ComputeJobResults.astro?raw"; +import TIMELINE from "../src/components/ComputeJobTimeline.astro?raw"; +import PAGE from "../src/pages/compute/jobs/[id].astro?raw"; + +const libs = import.meta.glob("/src/lib/compute-*.ts", { + query: "?raw", + import: "default", + eager: true, +}) as Record; + +/** Every file that belongs to the job page. */ +const JOB_LIBS = [ + "compute-job-view.ts", + "compute-job-dom.ts", + "compute-job-live.ts", + "compute-job-page.ts", + "compute-job-results-state.ts", + "compute-logs.ts", + "compute-failure.ts", + "compute-copy.ts", +]; +const JOB: Record = { + "pages/compute/jobs/[id].astro": PAGE, + "ComputeJobActions.astro": ACTIONS, + "ComputeJobLogs.astro": LOGS, + "ComputeJobResults.astro": RESULTS, + "ComputeJobTimeline.astro": TIMELINE, + ...Object.fromEntries( + JOB_LIBS.map((name) => { + const src = libs[`/src/lib/${name}`]; + if (src === undefined) throw new Error(`${name} is missing`); + return [name, src]; + }), + ), +}; + +/** + * Source with its comments removed and its string, template and regular-expression literals LEFT + * ALONE, so a `//` inside `"http://..."` or a pattern is never taken for a comment (a scan built on a + * comment-stripping regex missed code after one). A small scanner, not a parser: enough for these files. + */ +function stripJsComments(src: string): string { + let out = ""; + let i = 0; + // A `/` starts a regular expression when the last significant character cannot end an operand. + const startsRegex = (): boolean => { + const before = out.trimEnd(); + if (before === "") return true; + const last = before[before.length - 1]; + return "(,=:[!&|?{};+-*%<>~^".includes(last) || /\b(return|typeof|case|in|of)$/.test(before); + }; + while (i < src.length) { + const c = src[i]; + const next = src[i + 1]; + if (c === "/" && next === "/") { + while (i < src.length && src[i] !== "\n") i++; + } else if (c === "/" && next === "*") { + const end = src.indexOf("*/", i + 2); + i = end < 0 ? src.length : end + 2; + } else if (c === '"' || c === "'" || c === "`") { + let j = i + 1; + while (j < src.length && src[j] !== c) j += src[j] === "\\" ? 2 : 1; + out += src.slice(i, j + 1); + i = j + 1; + } else if (c === "/" && startsRegex()) { + let j = i + 1; + let inClass = false; + while (j < src.length && (src[j] !== "/" || inClass)) { + if (src[j] === "\\") j++; + else if (src[j] === "[") inClass = true; + else if (src[j] === "]") inClass = false; + j++; + } + out += src.slice(i, j + 1); + i = j + 1; + } else { + out += c; + i++; + } + } + return out; +} + +/** The scripts and the frontmatter of an Astro file (the code), as one string. */ +function astroCode(src: string): string { + const parts: string[] = []; + const front = /^---\n([\s\S]*?)\n---/.exec(src); + if (front) parts.push(front[1]); + for (const m of src.matchAll(/]*>([\s\S]*?)<\/script>/g)) parts.push(m[1]); + return parts.map(stripJsComments).join("\n"); +} + +/** What a scan reads: comments removed, `