Repository navigation
Description
Goal
Stand up a durable staging stack so cross-cutting changes (S3 bucket policy, Object Lock, DOI minting, publication orchestration, data/zarr planes, website) can be validated end-to-end before touching production.
- Website:
test.nemar.org(newnemar-website-testPages project) - API/data/zarr:
api-test/data-test/zarr-test.nemar.org, all served by the existingnemar-api-devworker - Storage: a dedicated
nemar-devS3 bucket (isolated from prodnemar) - Data: a small curated fleet of
xx-prefixed exemplar copies of real datasets, publishable through the full pipeline with EZID sandbox DOIs (doi:10.5072/FK2)
Full approved plan: ~/.claude/plans/clever-snuggling-sketch.md (kept in the epic worktree as .context/ notes).
Why
Today nemar-api-dev shares the prod S3 bucket and the nemarDatasets GitHub org with overlapping dataset-ID allocation, so a dev action can clobber real prod data. There is no safe place to exercise the pipelines the other epics (#896, #885, #837) will change.
Prod-safety principle
Almost every code change is a no-op or narrowing on prod (gated by ENVIRONMENT !== "production" + a new is_exemplar D1 flag + defaulted params). The only substantive new prod behavior is the config-gated, outbound-only webhook forwarder in Phase 5.
Phases
- Prod-safety foundation —
is_exemplarmigration 0057;xx090001+ID partition; env-conditional sandbox size cap; non-prod on-import 403; webhook dev-range gate. Deployable to prod as no-ops. - Dedicated
nemar-devbucket + AWS + secret hygiene — Object Lock bucket,nemar-worker-devIAM (zeroiam:*), secret audit/rotation,S3_BUCKETflip + inline manifests. - Custom domains + hostname dispatch generalization + host-aware session cookie.
xxpublish/DOI/reindex gate relaxation + landing-URL & bytes_url parameterization.- Exemplar creation endpoint + CLI nm to xx clone tool + prod webhook forwarder + central-workflow bucket param.
- Website staging — separate
nemar-website-testPages project, env-aware data-plane fetches, noindex. - Dev cron + publish the fleet + E2E validation.
Coordination (do not disrupt in-flight epics)
- Epic: Centralize secret management in self-hosted Infisical (single source of truth for Cloudflare, GitHub, AWS, local) #885 (secrets-infisical, resting): this epic ADDS new dev secrets/principal (
nemar-devAWS keys,EZID_SANDBOX_*,DEV_WEBHOOK_MIRROR_URL) — feed them into the Epic: Centralize secret management in self-hosted Infisical (single source of truth for Cloudflare, GitHub, AWS, local) #885 inventory at cutover. State kept in.claude/epic.local.md; NOT touched by this epic (this epic uses a dedicated named state doc). - Epic: Platform hardening + website↔CLI contract (arch-review follow-ups, 2026-07) #896 (arch-hardening, active): shares
publication-orchestrator.ts/data-router.ts/dispatch.ts. Phases 4 and 5 are gated until Epic: Platform hardening + website↔CLI contract (arch-review follow-ups, 2026-07) #896's P0s (Phase 1: Release/deploy safety — serialize deploy-backend + gate on tests #897, S3 upload of files >5 GB fails silently: buildS3RemoteArgs omits partsize (git-annex single-part PUT hits S3 5 GB cap) #886) land; rebase the shared-file edits.
Prod-safety register (summary)
| Leak vector | Fence |
|---|---|
| Shared S3 bucket | S3_BUCKET=nemar-dev + dev AWS principal that cannot address arn:aws:s3:::nemar |
| Shared IAM namespace | dev principal has zero iam:* |
| Shared GitHub org repo collisions | ID partition (dev floor 90001 / prod ceiling 89999); on-import 403 in dev |
| Shared GitHub App (one webhook URL) | prod dev-range gate + outbound-only config-gated mirror |
| Prod DOI minting from dev | ENVIRONMENT gates + prod DOI secrets absent from dev |
| Search-engine leakage | host-gated X-Robots-Tag + dynamic robots.txt |
Activity
Phase 1: prod-safety foundation for staging (#931)
Phase 2: point dev worker at nemar-dev bucket (#932)
Phase 3: staging custom domains + host-fork generalization (#933)
Phase 5: DOI version-mint reconciliation (#900) (#940)
Address PR review findings
Phase 4: xx gate relaxation + landing/bytes URL params (#927) (#945)
Merge pull request #956 from nemarOrg/feature/issue-923-epic-test-sta…
neuromechanist commentedon Jul 18, 2026
Epic complete, merged to
devin PR #956 (commit 2206f27).Staging is live and in use:
test.nemar.orgplusapi-test/data-test/zarr-test.nemar.orgon the dev worker, backed by the dedicatednemar-devbucket andnemar-db-dev. All seven curated exemplars are published with sandbox DOIs (10.5072/FK2) and render across every staging surface.All 7 phases shipped (#924, #925, #926, #927, #928, #929, #930). The cross-phase review found a critical hole no per-phase review could see:
deleteDatasetCascadehad no environment or ID-band guard, so a non-production worker could delete a real production GitHub repository through four manual endpoints. Fixed and verified live.Follow-ups tracked separately: #955 (exemplar creation is not retry-safe after a partial failure), #957 (request-driven email flows are not environment-gated), nemarOrg/website#155 (staging deploy workflow fix), and the two cross-repo central-workflow changes documented in
.context/phase5-cross-repo-owner-deploys.md.