Repository navigation
Phase 1: approval grants upload access - #1258
neuromechanist merged 9 commits into
Conversation
Migration 0075 establishes status='approved' iff service_access=1.
(a) CLI signups an admin already approved get the upload grant they were
always meant to have (#1249); keyed on service_access=0 so it is a
no-op for the rows production fixed by hand on 2026-09-05.
(b) ORCID web signups auto-approved with no admin review drop to the tier
they actually hold: 'verified' when the email is verified, else
'pending'. ORCID proves the person, the email code proves the inbox,
and the base tier needs both, so orcid_verified does not substitute.
Apply only alongside phase 2, which teaches the middleware and the
website that 'verified' is active.
Part of #1250, phase #1251.
Both approve routes now write service_access, service_access_granted_at and service_access_granted_by in the same statement as the status flip, so an admin's one action means what it says (#1249). Revoke clears all three. Both are audit-logged with the grant recorded explicitly. An account that is already 'approved' but holds no grant is repaired with a 200 and a note rather than 409'd into a dead end. GET /admin/users returns signup_source, service_access, the grant stamp and the identity columns a username-less web row is identified by. The gate's 403 message pointed at a settings feature that has never existed; it now names admin approval and the support page. The 'error' string is unchanged because the CLI matches on it. Part of #1250, phase #1251.
nemar admin users renders a Tier column (upload / browse), falls back to the email plus the numeric id when username is NULL and a dash when the GitHub handle is, instead of printing the literal 'null', and gains --no-upload-access and --awaiting-approval. nemar auth status / whoami print an Upload access line, cached from /users/me on --refresh so the command stays usable offline; a backend that omits the field leaves the cache alone rather than reporting a granted account as ungranted. Part of #1250, phase #1251.
Phase 1 (PR #1258) is ahead in the merge queue and keeps 0040. Until that merge is rebased in, 0040 does not exist on this branch, so the ADR index's gapless-from-0001 check fails here by construction -- 'ADR numbers are unique and gapless from 0001' in test/adr-index.unit.test.ts. It goes green with the rebase onto the epic branch; nothing else in the file changed.
Review round on #1258. - ADR 0040 gains a Phasing section. Phase 1 is the vocabulary, migration 0075, approval as the single writer and admin visibility; Phase 2 is everything that makes 'verified' usable (middleware, login, retrieve-key, sandbox, the pending->verified transition and the dashboard mapping). The 'pay once' bullet now says 'once Phase 2 ships the verify step'. Same correction in upload-gate.ts's header. - 0075's header records how wrangler executes a migration file: one /query POST on the remote path, whose transactional scope wrangler does not guarantee and whose result handling assumes a partial failure is possible. Idempotency, not atomicity, is what makes that safe, and the header says why. It also records the one shape 0075 does not reach (a non-approved owner/admin grandfathered by 0062). - 'ADR 0010' is ambiguous here: this repo has its own unrelated ADR 0010. Every citation of the tiered-access one now says 'website ADR 0010'. - draft-1023-service-access-endpoints-plan.md gets the STATUS: HISTORICAL banner and a README row; it proposes the separate grant/revoke endpoints ADR 0040 rejected. Part of #1250, phase #1251.
Three audit writes could turn a succeeded operation into a reported failure, which is worse than the missing audit row it would be reporting. - regrantUploadAccess: the grant UPDATE and its audit insert are now one db.batch(), the same shape (and for the same reason) as the tombstone batch further down the file. Unbatched, a throwing audit insert 500'd an admin whose grant HAD landed, and the retry then 409'd 'already approved' — a dead end for an operation that succeeded. On a batch failure nothing is granted and the error says so, so the retry works. - finalizeApproval: the audit insert cannot join the UPDATE in a batch, because email_sent is only known after a notification attempt that must follow the commit. So it takes the role-change route's shape instead: a failed audit write is logged as an AUDIT GAP, never propagated. The 200 still reports the grant. - revoke: same, and more sharply. By that point tokens, IAM, GitHub, S3 permissions, status and service_access have all changed and cannot be undone; a 500 would read as 'the user still has access'. The repair audit row also now carries granted_by_id. Tests drive this by dropping the audit_log table (real engine, real failure) and asserting the grant is absent on the batched path and present on the two logged ones. Also asserts the eligibility 400s leave service_access at 0, and that the listing publishes the grant timestamp. Part of #1250, phase #1251.
Upload access is three states, not two. A CLI talking to a backend deployed before #1251 (or mid-rollout) gets no service_access field, and coercing that absence to 'no' tells an uploader they cannot upload. - GET /admin/users is now validated against a shared contract schema (adminUserListItemSchema) instead of cast, so a drifted row fails loudly rather than rendering coerced junk. service_access is optional there on purpose and carries no default. - nemar admin users renders 'unknown' for such a row, both tier filters select only an explicit browse, and a footer counts what was unreported. - nemar admin approve reads the tier off the response instead of hardcoding 'upload'; an unconfirmed grant is called out. It also stops warning about a notification email on the repair path, where none was owed or attempted. - nemar auth status: a --refresh that fails for anything but 401/403 now puts the reason on the failure line and reports the tier as unknown rather than presenting a cached 'granted' as the current answer. - Help text: --no-upload-access matches pending and revoked rows too, so it says 'every account without the upload grant'; the tier legend notes CLI key and sandbox arrive in Phase 2. Fixtures for the absent-key cases are written out in full and guarded, because the absence is the whole point of them. test/api-export-surface.unit.test.ts pins uploadTierOf as an intentional post-split addition. Part of #1250, phase #1251.
📦 Package PreviewYour package is ready to test! bun install -g nemar-cli@PR1258Version: 🤖 Updated on each push. |
Review round 1 — resolutionAll 18 findings addressed across four commits ( A. Writes that follow the invariant write
B. CLI tri-state
C. Tests
Beyond the list, three fault-injection tests drop the real D. Migration13. Checked The same header now also records the one shape 0075 does not reach: 0062 grandfathered Addendum (comment/ADR reviewer)
Verification
Every new assertion was watched fail by a single-line perturbation, reverted before the next: the tri-state collapsed to two, the approve Tier line hardcoded, the refresh-failure branch disabled, the reason dropped from the failure line, rule (b)'s
|
9d726e6
into
feature/issue-1250-epic-account-tiers
Phase 1 (PR #1258) is ahead in the merge queue and keeps 0040. Until that merge is rebased in, 0040 does not exist on this branch, so the ADR index's gapless-from-0001 check fails here by construction -- 'ADR numbers are unique and gapless from 0001' in test/adr-index.unit.test.ts. It goes green with the rebase onto the epic branch; nothing else in the file changed.
Phase 1 (PR #1258) is ahead in the merge queue and keeps 0040. Until that merge is rebased in, 0040 does not exist on this branch, so the ADR index's gapless-from-0001 check fails here by construction -- 'ADR numbers are unique and gapless from 0001' in test/adr-index.unit.test.ts. It goes green with the rebase onto the epic branch; nothing else in the file changed.
* feat: cite DOI uploaders by real name Replaces users.username with given_name/family_name as the DOI attribution across every mint path: publication approval, admin concept-DOI, exemplar re-mint, DOI metadata refresh, and the LLM enrichment DOI sync. The DataCurator contributor now emits "Family, Given" with givenName/familyName children and the ORCID nameIdentifier; a username never reaches DataCite, EZID, or Zenodo. Publishing is blocked with a typed owner_name_missing reason when the owner has no citable name, on both the request and the admin publish and concept-DOI paths, rather than silently dropping attribution. OpenNeuro imports and exemplars are exempt. Author matching now compares both name parts independently, so an author list written "Given Family" is recognised. * feat: source the signup name from ORCID GET /auth/orcid-name is a pre-signup lookup alongside check-username and check-github: the CLI reads the public ORCID record right after the ORCID prompt and asks for a given/family name ONLY when the record hides one. A pre-flight GET rather than a flag on the signup response, because signup is the call that creates the account. The signup insert still reads ORCID server-side and the record wins; the client-supplied pair is a fallback, never an override. Adds an ORCID_PUB_API_BASE binding so the public-record reads can be pointed at a mirror, or at a local server in tests. * feat: add admin ORCID name backfill POST /admin/users/backfill-names fills NULL researcher names from each account's public ORCID record, dry run by default, with a per-user outcome. Half a published name is reported and NOT written: a row with one part set still blocks publishing while looking done. CLI: nemar admin backfill-names [--apply] [--limit n] [--json]. Registered from its own admin domain file rather than users.ts, and the admin route inventory pin is updated to match. * test: give the golden publish owner a real name The approve flow now refuses an owner with no given/family name, so the step-machinery fixture has to seed one. Intentional fixture update; the gate itself is covered in uploader-real-name-doi.test.ts. * docs: record the real-name DOI decision ADR on why the uploader is cited by given/family name or not at all, and why there is no username fallback. Numbered 0040, which phase 1 of the same epic also claims on its branch; the index test fails loudly on the duplicate, so whichever lands second gets renumbered. Also lists nemar admin backfill-names in the AGENTS.md command map. * docs: renumber ADR to 0041 Phase 1 (PR #1258) is ahead in the merge queue and keeps 0040. Until that merge is rebased in, 0040 does not exist on this branch, so the ADR index's gapless-from-0001 check fails here by construction -- 'ADR numbers are unique and gapless from 0001' in test/adr-index.unit.test.ts. It goes green with the rebase onto the epic branch; nothing else in the file changed. * refactor: carry one uploader identity end to end DataCiteEnrichment holds a single uploader?: UploaderIdentity | null instead of a name/given/family/orcid quartet that bidsToDataCite had to re-derive and re-trim. authorMatchesUploader takes the identity, and its whole-name fallback branch -- unreachable once the object always carries both parts -- is gone. createConceptDoi enforces the rule itself via uploaderRequired rather than trusting three route guards, mirroring the Zenodo branch, and a repo-supplied enrichment file can no longer override who the depositor is. requiresUploaderName is typed from the shared dataset source enum. Tests: name parts are XML-escaped; a short part still must not match inside a longer word (restored alongside the family-overlap case). Review items 7, 8, 12, 15, 22, 25. * fix: never strip an existing DOI curator The enrichment DOI sync and the admin metadata refresh both rebuild the whole DataCite record from live DB state. For an already-published dataset whose owner has no researcher name -- most of the catalogue until the backfill runs -- that record has no DataCurator, and pushing it silently deletes the curator from a permanent DOI while logging success. Both now SKIP with a distinct log and a typed outcome (doi_sync / metadata_refresh), surfaced in the bulk reindex result. The per-version refresh inherits the skip, so version DOIs are safe too. Also: the publish walk-back to 'blocked' no longer strands the request at 'approving' behind a 500 if the UPDATE throws; fetchOrcidName is bounded by a 10s timeout now that the backfill loops it up to 100 times; the backfill's trailing count is non-fatal and reports remaining: null; and the concept-DOI name guard runs after the already-minted check so a re-invocation reports the no-op, not a name error. Review items 1, 2, 3, 5, 6, 23. * feat: share the publication vocabulary, fix the advice One z.enum in shared/contract for the publication block reason, the backfill outcome, the ORCID iD pattern, and the ORCID lookup status, replacing four hand-duplicated copies. The admin test-fixture route now validates an ORCID as an ORCID instead of max(19). GET /auth/orcid-name reports found / no_public_name / lookup_failed, so the CLI stops telling a user their record hides their name when ORCID is merely down. Signup reports researcher_name: recorded|missing, because only the insert knows whether the name actually landed. The fix-it text now reaches the terminal: the client prefers the body's `message` when it carries a block_reason, and hints are keyed on the reason so "Fix the CI issues" is no longer shown for a missing name. GET /publish/status returns the message regardless of repoName, which is repo-independent for owner_name_missing. Every message points at the route that EXISTS: make the ORCID name public and sign in again. PATCH /auth/profile rejects name edits until Phase 3 (#1253), so "set it in Settings" was a dead end. Review items 4, 9, 10, 11, 18, 19, 24. * test: pin what reaches EZID from the untested paths Route tests for the concept metadata refresh, its per-version loop, and the exemplar re-mint, driving real Hono routes against local Bun.serve GitHub and EZID upstreams and asserting on the DataCite XML the Worker would send: real name present, username absent, and nothing sent at all when a nameless owner would have stripped the curator. Adds an NEMAR_EZID_API_URL test seam mirroring NEMAR_GITHUB_API_URL. Also: exemption cases at the approve and concept-DOI gates plus a direct requiresUploaderName test; the doi.ts uploaderRequired and Zenodo refusals; the enrichment sync's guard against real rows (with a note on why the LLM-fronted entry point is not driven); the bulk-reindex surfacing of a skip; and the CLI client actually showing the fix-it text on all three refusal shapes. Review items 13, 14, 16, 17, and the testable half of 1. * docs: record the backfill and self-service gaps in ADR 0041 The ADR now says that nemar admin backfill-names --apply must be run against production by an admin before existing owners can publish (there is no cron), that an owner whose ORCID record hides their name has no self-service fix until Phase 3 (#1253), that the owner dashboard cannot yet explain the block (nemarOrg/website#304), and that version DOIs are minted without a curator (#1261). Receipts corrected: DOI permanence is an AGENTS.md core principle, not something ADR 0007 states. Review items 20, 21. * test: cover DOI attribution on tree-free paths The metadata-refresh happy path needs a real getTreeAtRef, and manifest-small-root-files.test.ts installs a process-wide mock.module on services/github that returns an empty tree and cannot be undone (mock.restore() and re-registering the real namespace both fail). A refresh test asserting on rebuilt XML therefore passed alone and failed in a full run. The real-name-in / username-nowhere assertions now ride the concept mint and the exemplar re-mint, which need no tree; the refresh's own skip behaviour needs none either and is asserted directly. The gap and its cause are written down in the file so the next person does not rediscover it. GitHub is intercepted by patching globalThis.fetch per call rather than via the URL override, which puts this file at the front of the chain regardless of what an earlier file left behind. * fix: a deliberate DOI-sync skip is not a failure extractEnrichmentSubErrors folded the doi_sync skip into the sub-error list, and runEnrichmentForDataset treats any non-empty list as ok:false -- so both reindex routes reported status: failed and logged "enrichment failed" for every dataset whose owner has no name yet, which until the backfill runs is most of the catalogue, burying the real failures. Skips now travel separately (extractEnrichmentSkips) and ride an ok result as EnrichmentRunResult.warnings, surfaced under warnings on both the single and bulk reindex responses. "failed" is reserved for errors. The new test file also covers the skip wiring inside enrichDataset. The real entry point still cannot be driven: not because of the LLM (the Claude call already takes its base URL from ANTHROPIC_BASE_URL, so a local server suffices) but because manifest-small-root-files.test.ts installs a permanent process-wide mock.module on services/github whose getTreeAtRef returns an empty tree, and enrichDataset returns "No README found" long before the DOI sync. The reason and the two attempted escapes are written into the file, alongside source-level guards that the skip sits before the EZID write and on the else-branch of the sync. Also: nemar admin backfill-names prints the warning behind remaining=unknown, and exits non-zero when the remainder is unknown. Review items 26, 27, 28.
Closes #1251
Part of #1250
Summary
Makes admin approval the single writer of upload access, and gives admins a way to see who holds it.
Since #1013 phase 1 shipped the
service_accessgate without its grant,nemar admin approvemoved a user tostatus='approved'and unlocked nothing (#1249), while ORCID web sign-ups reached the same status with no admin involved at all. This establishes the invariantstatus='approved'iffservice_access=1and keeps it.ADR 0040
Written as
.context/decisions/0040-approval-is-the-single-writer-of-upload-access.mdand added to the index (note: the brief said ADR 0037, but 0037-0039 already exist andtest/adr-index.unit.test.tsenforces gapless numbering, so this is 0040). It fixes the status vocabulary (pending/verified/approved/revoked), records that sandbox training stays CLI-only, that admins act once per uploader at the upload request, and supersedes the "auto-approve ORCID sign-ups tostatus='approved'" half of website ADR 0010. The two-tier idea and the export-control review behind the grant survive.Migration 0075
status='approved' AND service_access=0 AND signup_source='cli'gets the grant,service_access_granted_byNULL (a system grant). Keyed onservice_access=0, so it is a no-op for the rows production fixed by hand on 2026-09-05.status='approved' AND service_access=0 AND signup_source='web'moves toverifiedwhenemail_verified=1, elsepending, andapproved_atis cleared. Per the mid-task correction,orcid_verifieddoes not count: of the 19 production rows in this population all are ORCID-verified and one is email-verified, so 18 land atpending.Both statements also carry
deleted_at IS NULL(not in the issue text for (b); a tombstone is force-flipped torevokedso it cannot match anyway, and the guard keeps a future tombstone shape from being resurrected).Deploy order: apply only alongside phase 2. On its own, (b) moves every web row out of
approvedanduserStatusForDashboardreports both landing tiers as "pending" with nothing on the page to act on. This is called out in the migration header and the ADR.Backend
service_access,service_access_granted_atandservice_access_granted_byin the same statement as the status flip;verifiedwas already approvable, and thepending-for-ORCID-web allowance is unchanged.approvedwithservice_access=0is repaired: 200 with anote, audited asuser_upload_access_granted, rather than 409. A user who is approved and granted still 409s.service_access_cleared.GET /admin/usersreturnssignup_source,service_access,service_access_granted_at,given_name,family_name,orcid,email_verified.SERVICE_ACCESS_ERROR.messagenow names admin approval and https://nemar.org/support instead of a settings feature that has never existed. Theerrorstring is unchanged (the CLI matches on it).CLI
nemar admin usersrenders aTier:column (upload/browse), falls back to the email plus(no username, id N)whenusernameis NULL and a dash whengithub_usernameis, and adds--no-upload-accessand--awaiting-approval(filtered client-side; the listing is unpaginated, so no new query param).--awaiting-approvalis refused when combined with a conflicting status flag.nemar admin approvesays the approval grants upload access and prints the resulting tier.nemar auth status/whoamiprintUpload access: grantedorUpload access: not granted (one-time admin approval; see https://nemar.org/support), cached from/users/meon--refreshso the command stays usable offline; a never-refreshed account reportsunknownrather than guessing, and a backend that omits the field leaves the cache alone.Testing
bun testfrom the worktree root: 4511 pass, 97 skip, 50 fail. All 50 failures are the live-backend suites (test/api.test.ts,test/cli.test.ts,test/e2e-upload.test.ts), which needtest/.env.test(TEST_ADMIN_API_KEY/TEST_USER_API_KEY) — that file is gitignored and absent from this worktree. Verified pre-existing: stashing every change and runningtest/api.test.ts test/cli.test.tson the untouched base tree gives the same 48 failures.bun run lintandbun run typecheckare clean;biome checkreports nothing on the 13 files touched.New tests, all real-engine and no mocks:
backend/test/approval-grants-upload-migration.test.ts— 0075 against a real bun:sqlite DB with every prior migration applied, followingservice-access-migration.test.ts. Covers both rules, the ORCID-is-not-email split, idempotency, and that a hand-fixed row keeps its original grant timestamp. One test asserts the invariant over the whole seeded catalog rather than row by row.backend/test/approval-upload-grant-route.test.ts— real Hono dispatch through auth + admin middleware over realD1: approve by username and by id, the audit rows, the repair path, the still-409 case, revoke clearing the stamps, and the listing reporting the tier.backend/test/upload-gate-message-route.test.ts— the 403 body read offPOST /datasets/:id/upload-urls, not off the constant.test/admin-users-tier-cli.test.tsandtest/auth-status-upload-access.test.ts— real subprocess CLI runs against a real local HTTP server viaTEST_API_URL, followingtest/hed-sweep-cli.test.ts.Every new assertion was watched fail: the migration's two rules were mutated one at a time; the grant write, the stamp clearing, the listing column and the repair path were each removed in turn; the CLI's null handling, tier column and tier filter were each reverted; the gate message was restored to its old text; and
auth status's granted branch and its "only cache what the server sent" guard were each broken. Each perturbation failed the expected tests and only those, and each was reverted before the next.Not in this PR
Phase 2 items (token at verification, middleware change, ORCID finalize change, sandbox channel split) and phase 3 items (request endpoint, username PATCH) are deliberately out of scope. ORCID finalize therefore still auto-approves new web sign-ups, so 0075 establishes the invariant and every new web sign-up re-breaks it until phase 2 lands; the approve routes' repair path covers such a row in the meantime, and the ADR records the gap.