Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion AGENTS.md
Original file line number Diff line number Diff line change
Expand Up @@ -646,7 +646,7 @@ and what is historical. The entries worth knowing by name:
| `nemar dataset publish` | request, status, resend |
| `nemar dataset` (access) | request-access, access, invite, collaborators |
| `nemar sandbox` | training run, status, reset — required before uploading |
| `nemar admin` | users (`--kind`), approve, revoke, role, kind (account kind, owner-only — ADR 0048), keys (create/list/revoke a service/test account's API keys, owner-only), notify, s3, repo, ci, doi, publish, revert, make-public, delete-dataset, bulk-delete, reindex, hed-sweep, data-integrity-sweep, recording-stats-sweep, signal-defaults-sweep, zarr-fidelity-sweep, import-issue-triage, import-coverage, import-weekly, doctor, summary, notice, email-preferences, backfill-names, backfill-usernames, duplicates, e2e-test |
| `nemar admin` | users (`--kind`), approve, revoke, role, kind (account kind, owner-only — ADR 0048), keys (create/list/revoke a service/test account's API keys, owner-only), notify, s3, repo, ci, doi, publish, revert, make-public, delete-dataset, bulk-delete, reindex, hed-sweep, data-integrity-sweep, recording-stats-sweep, signal-defaults-sweep, zarr-fidelity-sweep, import-issue-triage, import-coverage, import-weekly, doctor, summary, notice, email-preferences, backfill-names, backfill-usernames, duplicates, e2e-test, docs (fetch a documentation page as Markdown, gated ones included) |
| `nemar admin import*` | OpenNeuro import, status, rollback, retry, verify, recover (issue #754, epic #967) |
| `nemar admin fleet` | drift, enforce, revalidate — governance across dataset repos (epic #713) |
| `nemar admin exemplar` | create, status, remint-dois — the staging exemplar fleet |
Expand Down
26 changes: 26 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -13,6 +13,32 @@ what merged, and this file says what it meant.
Newest first. Dates are the tag's publication date, UTC. Backfilled from 0.9.16 onward;
earlier releases are described only by their generated notes.

## Unreleased

### Added

- **`nemar admin docs <path...>` reads documentation pages, including the gated ones, without
a browser.** `nemarOrg/docs` is private at source and `docs.nemar.org` is the retrieval
surface, so a checkout is no longer how anyone opens an operations runbook. The command
trades the stored API key at `POST /auth/docs/cli-session` for a fifteen-minute, read-only,
documentation-scoped session and sends only that onward, so the long-lived key never reaches
the documentation host or its logs. Pages come back as Markdown on stdout and diagnostics on
stderr, so `nemar admin docs cli/commands > page.md` yields the page. Pass several paths at
once: they share one session, which matters because the mint sits in the strict per-address
rate-limit bucket. There is deliberately no flag that prints the session value, since a
credential on stdout is one in a shell history.

### Fixed

- **Rate limits now apply to the `/nemar` spelling of every route.** The API is mounted twice,
at `/` and at `/nemar`, and the limiter matched paths against the full request path, so the
strict per-address floor on the authentication routes (`/auth/login`, `/auth/code/request`,
`/auth/keys`, the device-flow routes) matched nothing when the prefix was used and those
requests fell to the general bucket. Both spellings are bucketed alike now. Low impact in
practice, since Cloudflare's own per-address ceilings and the per-email limit on code
requests both still applied, and this API is read-only for anyone without a key; recorded
because it is a real change in how those routes are throttled.

## 0.10.3 - 2026-09-10

### Security
Expand Down
19 changes: 17 additions & 2 deletions backend/src/lib/not-found.ts
Original file line number Diff line number Diff line change
Expand Up @@ -30,9 +30,24 @@ export function notFoundBody(method: string, path: string) {
}

/** The notFound handler for `api` and `app` alike, and the response any route
* that wants to look unrouted must return. */
* that wants to look unrouted must return.
*
* `no-store` is set HERE rather than at the one route that needs it, and that
* placement is the whole point. `routes/auth-docs.ts` declares that every
* answer it gives is a credential or a refusal about one and so may never sit
* in a shared cache -- but its two 404s returned this response, which carried
* no cache header, so the declared invariant was untrue (found reviewing
* #1384). Adding the header at the call site instead would have made that
* route's 404 differ from a genuinely unrouted one by exactly one header,
* destroying the parity this file exists to preserve. Setting it for every 404
* keeps the two byte-identical AND makes the invariant hold.
*
* Safe as a blanket rule: a 404 is never a document worth reusing, and the
* same URL is frequently a real page for the next caller, who may be an admin.
* That is the identical reasoning the docs gate's own `notFound()` already
* applies in `nemarOrg/docs`. */
export function notFoundResponse(
c: Context<{ Bindings: Bindings; Variables: Variables }>,
): Response {
return c.json(notFoundBody(c.req.method, c.req.path), 404);
return c.json(notFoundBody(c.req.method, c.req.path), 404, { "Cache-Control": "no-store" });
}
43 changes: 42 additions & 1 deletion backend/src/middleware/rateLimit.ts
Original file line number Diff line number Diff line change
Expand Up @@ -180,8 +180,22 @@ const AUTH_PATHS = [
// and cannot be brute-forced at any rate a bucket would help with. Note the
// comment on MAX_REQUESTS below -- shared CF egress IPs are why that bucket
// is 500 rather than 100 in the first place.
//
// `/auth/docs/cli-session` IS here, and unlike the two above it is called by
// a person's own machine rather than by a Worker, so the per-IP key is a real
// per-caller key for once. An admin mints one session per fifteen minutes of
// reading, so ten a minute is far above any honest use. The CLI takes several
// paths in one invocation partly so that a reading session costs one mint
// rather than one per page.
//
// The stronger claim -- that a stolen admin key cannot be turned into a
// stream of docs credentials from one address -- was written here first and
// was FALSE, because `/nemar/auth/docs/cli-session` matched nothing in this
// list. See `__normalizeMountPath` above, which is what makes it true, and
// note that it was true of no entry in this list before that.
"/auth/docs/grant",
"/auth/docs/exchange",
"/auth/docs/cli-session",
// NOT an /auth path, and deliberately in this list anyway (ADR 0042, #1253):
// POST /users/me/upload-access/request spends a live GitHub API call on the
// shared installation token for every attempt, and a refused one writes
Expand Down Expand Up @@ -249,11 +263,38 @@ export interface __BucketSelection {
maxRequests: number;
}

/**
* The API sub-app is mounted TWICE in `backend/src/index.ts`: at `/` and at
* `/nemar`. Hono hands middleware the FULL request path, not the path relative
* to the mount, so every path rule in this file saw `/nemar/auth/login` for one
* of the two spellings and matched none of them.
*
* FOUND BY REVIEW OF THE DOCS CLI ROUTE (#1384), and it was never specific to
* that route: every entry in `AUTH_PATHS` was reachable at 500/min (or, with a
* bearer, the 1000/min token bucket and the admin bypass beyond it) by
* prefixing `/nemar`. `/auth/login`, `/auth/code/request`, `/auth/keys` and the
* device-flow routes all sat behind a strict floor that one extra path segment
* stepped over. Measured, not inferred: middleware inside the prefixed mount
* reports `c.req.path` as `/nemar/auth/docs/cli-session`, and both spellings
* reach the same handler.
*
* Normalizing here rather than at each call site is deliberate: this is the one
* place that turns a path into a bucket, so a rule added later gets the fix for
* free. The data and zarr regexes benefit too, for the same reason.
*/
const MOUNT_PREFIX = "/nemar";

export function __normalizeMountPath(path: string): string {
if (path === MOUNT_PREFIX) return "/";
return path.startsWith(`${MOUNT_PREFIX}/`) ? path.slice(MOUNT_PREFIX.length) : path;
}

export function __selectBucket(
path: string,
rawPath: string,
authHeader: string | undefined,
ip: string,
): __BucketSelection {
const path = __normalizeMountPath(rawPath);
if (AUTH_PATHS.some((p) => path === p || path.startsWith(`${p}/`) || path.startsWith(`${p}?`))) {
return { keyKind: "auth-ip", rawKey: ip, maxRequests: AUTH_MAX_REQUESTS };
}
Expand Down
97 changes: 94 additions & 3 deletions backend/src/routes/auth-docs.ts
Original file line number Diff line number Diff line change
Expand Up @@ -2,9 +2,10 @@
* Docs admin gate routes (epic #1336 phase 0, issue #1338), mounted under the
* same `/auth` prefix as the other auth families.
*
* POST /auth/docs/grant - website (session): mint a one-time code
* POST /auth/docs/exchange - docs Pages Function: trade it for a cookie
* GET /auth/docs/verify - docs Pages Function: check that cookie
* POST /auth/docs/grant - website (session): mint a one-time code
* POST /auth/docs/exchange - docs Pages Function: trade it for a cookie
* GET /auth/docs/verify - docs Pages Function: check that cookie
* POST /auth/docs/cli-session - CLI (API key): mint the same credential direct
*
* WHY THIS EXISTS. `docs.nemar.org/admin/*` answered 200 to anyone. The
* Cloudflare Access app the docs repo described covers that Pages project's
Expand All @@ -23,6 +24,22 @@
* credential of its own, obtained once through a code that dies in sixty
* seconds.
*
* WHY THE CLI ROUTE SKIPS ALL OF THAT (epic #1336 phase 3, issue #1341). The
* three-hop handoff exists because a browser's proof of identity lives on one
* host and the credential has to be set on another. A CLI has no such split: it
* holds a long-lived API key, and `api.nemar.org` is the one host that can
* validate it. So `cli-session` is the whole handoff collapsed into a single
* authenticated call, with the mint-time gates intact.
*
* What it does NOT change is the separation ADR 0056 exists for: the API key
* never reaches `docs.nemar.org` or its logs. The caller trades it here for a
* short-lived, docs-scoped, read-only value and sends only that onward, as a
* `Cookie` header the gate already knows how to read. Nothing in the docs repo
* needed a change to accept it, which was worth checking rather than assuming:
* the Pages Function reads the credential from `Cookie` and presents it to
* `verify` in `X-Docs-Session`, so a client that sets the cookie header itself
* enters through the same door a browser does.
*
* `grant` is the only route here that a browser reaches, and only via the
* website's server-side render. `exchange` and `verify` are server-to-server
* calls from a Cloudflare Pages Function, which is why neither reads a cookie:
Expand All @@ -37,17 +54,21 @@ import { z } from "zod";
import {
DOCS_ADMIN_ROLES,
DOCS_AUTH_MESSAGES,
DOCS_CLI_SESSION_TTL_SECONDS,
DOCS_GRANT_TTL_SECONDS,
DOCS_SESSION_HEADER,
DOCS_SESSION_TTL_SECONDS,
type DocsCliSessionResponse,
type DocsExchangeResponse,
type DocsGrantResponse,
type DocsVerifyResponse,
} from "../../../shared/contract/docs-auth.js";
import { notFoundResponse } from "../lib/not-found";
import { resolveBearerUser } from "../middleware/auth";
import { webSessionMiddleware } from "../middleware/webSession";
import { isActiveAccountStatus } from "../services/account-tier";
import {
DOCS_CLI_MINT_INSERT_SQL,
DOCS_GRANT_INSERT_SQL,
DOCS_GRANT_PRUNE_SQL,
DOCS_MINT_CONSUME_SQL,
Expand Down Expand Up @@ -212,6 +233,76 @@ authDocsRoutes.post(
},
);

/**
* Mint a docs session for a CLI caller holding an admin API key.
*
* Reached by `nemar admin docs`, and by anything else that can present a key.
* The response body carries the credential, so it is `no-store` like every
* other answer in this file.
*
* BEARER ONLY, AND THE COOKIE IS DELIBERATELY NOT ACCEPTED. `authMiddleware`
* would take either, and taking either here would be a mistake: the app cookie
* is ambient in a browser, so a cookie-authenticated POST to this route is a
* cross-site request away from minting a docs credential for whoever is signed
* in. The browser already has a path to one -- `grant`, which checks `Origin`
* before it checks anything else. This route resolves the bearer itself rather
* than mounting `authMiddleware`, precisely so the cookie path is absent rather
* than merely discouraged.
*/
authDocsRoutes.post("/docs/cli-session", async (c) => {
const bearer = await resolveBearerUser(c);
if (bearer.kind === "refused") {
// `resolveBearerUser` builds its own refusals (malformed header, dead key,
// inactive account) and they are the right answers, but they are answers
// ABOUT a credential, so they get this file's cache rule too. Rebuilt
// rather than mutated: a Response's headers are not guaranteed mutable,
// and a silently-dropped header here would be invisible.
const refused = new Response(bearer.response.body, bearer.response);
refused.headers.set("Cache-Control", "no-store");
return refused;
}
if (bearer.kind === "absent") {
return c.json(
{ error: "unauthenticated", message: DOCS_AUTH_MESSAGES.unauthenticated },
401,
NO_STORE,
);
}

const user = bearer.user;
// Same disguise `grant` gives a signed-in non-admin, for the same reason and
// with the same caveat: it buys parity with the other entrances, not secrecy.
// See the comment on `grant` before building anything on top of it.
if (!isDocsAdmin(user.role)) {
return notFoundResponse(c);
}

const cookieIdRaw = generateCookieId();
const cookieIdHash = await hashCookieId(cookieIdRaw);
// Unlike `exchange`, these two DO describe the caller: a CLI on someone's
// machine talks to this host directly, with no Pages Function in between.
const userAgent = c.req.header("User-Agent") ?? null;
const ipHash = await hashIp(clientIp(c));

const result = await c.env.DB.prepare(DOCS_CLI_MINT_INSERT_SQL)
.bind(cookieIdHash, DOCS_CLI_SESSION_TTL_SECONDS, userAgent, ipHash, user.id)
.run();

// Zero rows means the account stopped qualifying between the bearer lookup
// and this write. Rare, and `not_found` keeps the answer indistinguishable
// from the non-admin case above rather than announcing the race.
if ((result.meta?.changes ?? 0) === 0) {
return notFoundResponse(c);
}

const body: DocsCliSessionResponse = {
session: cookieIdRaw,
max_age_seconds: DOCS_CLI_SESSION_TTL_SECONDS,
username: user.username || null,
};
return c.json(body, 200, NO_STORE);
});

/**
* Check a docs session value. Called by the Pages Function on every gated
* request.
Expand Down
59 changes: 59 additions & 0 deletions backend/src/services/docs-auth.ts
Original file line number Diff line number Diff line change
Expand Up @@ -146,6 +146,65 @@ export const DOCS_MINT_INSERT_SQL = `INSERT INTO web_sessions
AND u.status IN ${ACTIVE_ACCOUNT_STATUS_SQL_LIST}
AND u.deleted_at IS NULL`;

/**
* Mint a docs session straight from an account, for the CLI. Binds:
* cookieIdHash, ttlSeconds, userAgent, ipHash, userId.
*
* THERE IS NO GRANT ON THIS PATH, and the reason is worth stating rather than
* leaving as an asymmetry someone later "fixes". The grant exists because the
* browser flow crosses two hosts: the website proves the session and the docs
* host mints the cookie, so a one-time code has to travel between them through
* a URL. A CLI call crosses nothing. The caller presents a long-lived API key
* to the one host that can validate it and gets the credential back in the same
* response, so there is no window to protect and nothing to put in a URL.
*
* WHAT IS NOT DROPPED is every gate the grant path applies at mint time: role,
* status and `deleted_at` are re-read in the statement that writes. BUT THE
* REASON IS WEAKER HERE THAN THERE, and the first draft of this comment claimed
* otherwise, so be precise. {@link DOCS_MINT_INSERT_SQL} closes a real window:
* a grant row is up to sixty seconds old and carries only `user_id`, so without
* the re-read an account demoted inside that minute still mints. On this path
* `resolveBearerUser` re-reads role and status from `users` in the same request,
* microseconds earlier, and the route refuses a non-admin before reaching this
* statement. So these gates close no window the caller leaves open; they are
* redundant by construction.
*
* They are kept anyway, for two reasons worth stating rather than leaving to be
* rediscovered. The statement then holds its own contract -- "mints or inserts
* nothing" -- independently of who runs it, so a second caller added later
* cannot weaken it by forgetting a check. And the two mint paths stay
* symmetrical, which is what makes it obvious that a rule added to one belongs
* in the other. Proving this layer needs a test that runs the statement
* DIRECTLY: removing either gate alone leaves the route's behavior unchanged,
* because the other layer absorbs it (measured, not assumed -- see
* `backend/test/docs-auth-routes.test.ts`).
*
* `auth_method` is the literal `'api_key'` rather than a value copied from
* somewhere, because that is the truth about this session: the identity behind
* it was proven by a key, not by ORCID or an email code. The column is
* free-text (0050, 0083) and no route surfaces it, so this widens no response
* shape.
*
* `scope` is hard-coded `'docs'` and `remember` is hard-coded 0, both for the
* same reason they are in {@link DOCS_MINT_INSERT_SQL}: a bindings mistake must
* not be able to turn this into an app session.
*/
export const DOCS_CLI_MINT_INSERT_SQL = `INSERT INTO web_sessions
(user_id, cookie_id_hash, remember, expires_at, user_agent, ip_hash, auth_method, scope)
SELECT u.id,
?,
0,
datetime('now', '+' || ? || ' seconds'),
?,
?,
'api_key',
'docs'
FROM users u
WHERE u.id = ?
AND u.role IN ${DOCS_ADMIN_ROLES_SQL_LIST}
AND u.status IN ${ACTIVE_ACCOUNT_STATUS_SQL_LIST}
AND u.deleted_at IS NULL`;

/**
* Consume the grant, in the same `db.batch()` as {@link DOCS_MINT_INSERT_SQL}.
* Binds: codeHash, cookieIdHash.
Expand Down
11 changes: 9 additions & 2 deletions backend/src/services/web-session.ts
Original file line number Diff line number Diff line change
Expand Up @@ -341,8 +341,15 @@ export async function findSessionByCookieId(
}

/** How a session was established. Stored on `web_sessions.auth_method`
* (0050) so /auth/me and admin tooling can distinguish the flows. */
export type AuthMethod = "email_code" | "orcid";
* (0050) so /auth/me and admin tooling can distinguish the flows.
*
* `api_key` is written only by the CLI docs mint (`DOCS_CLI_MINT_INSERT_SQL`,
* epic #1336 phase 3), which spells the literal in SQL rather than binding a
* value from here. It is listed anyway: nothing surfaces this column today, so
* the omission was invisible, but migration 0050 says the column exists for
* "/auth/me and admin tooling", and the first reader to take that up would
* type it as this union and silently fail to model a real value. */
export type AuthMethod = "email_code" | "orcid" | "api_key";

/** The cookie a caller will set, plus the not-yet-executed INSERT that makes
* it valid. Split out of `issueSession` so a caller can put the session row
Expand Down
Loading
Loading