Skip to content

Epic: Staging environment (test.nemar.org) + exemplar test-dataset fleet #923

Closed
7 of 7 issues completed

Description

@neuromechanist

Goal

Stand up a durable staging stack so cross-cutting changes (S3 bucket policy, Object Lock, DOI minting, publication orchestration, data/zarr planes, website) can be validated end-to-end before touching production.

  • Website: test.nemar.org (new nemar-website-test Pages project)
  • API/data/zarr: api-test / data-test / zarr-test.nemar.org, all served by the existing nemar-api-dev worker
  • Storage: a dedicated nemar-dev S3 bucket (isolated from prod nemar)
  • Data: a small curated fleet of xx-prefixed exemplar copies of real datasets, publishable through the full pipeline with EZID sandbox DOIs (doi:10.5072/FK2)

Full approved plan: ~/.claude/plans/clever-snuggling-sketch.md (kept in the epic worktree as .context/ notes).

Why

Today nemar-api-dev shares the prod S3 bucket and the nemarDatasets GitHub org with overlapping dataset-ID allocation, so a dev action can clobber real prod data. There is no safe place to exercise the pipelines the other epics (#896, #885, #837) will change.

Prod-safety principle

Almost every code change is a no-op or narrowing on prod (gated by ENVIRONMENT !== "production" + a new is_exemplar D1 flag + defaulted params). The only substantive new prod behavior is the config-gated, outbound-only webhook forwarder in Phase 5.

Phases

  1. Prod-safety foundation — is_exemplar migration 0057; xx090001+ ID partition; env-conditional sandbox size cap; non-prod on-import 403; webhook dev-range gate. Deployable to prod as no-ops.
  2. Dedicated nemar-dev bucket + AWS + secret hygiene — Object Lock bucket, nemar-worker-dev IAM (zero iam:*), secret audit/rotation, S3_BUCKET flip + inline manifests.
  3. Custom domains + hostname dispatch generalization + host-aware session cookie.
  4. xx publish/DOI/reindex gate relaxation + landing-URL & bytes_url parameterization.
  5. Exemplar creation endpoint + CLI nm to xx clone tool + prod webhook forwarder + central-workflow bucket param.
  6. Website staging — separate nemar-website-test Pages project, env-aware data-plane fetches, noindex.
  7. Dev cron + publish the fleet + E2E validation.

Coordination (do not disrupt in-flight epics)

Prod-safety register (summary)

Leak vector Fence
Shared S3 bucket S3_BUCKET=nemar-dev + dev AWS principal that cannot address arn:aws:s3:::nemar
Shared IAM namespace dev principal has zero iam:*
Shared GitHub org repo collisions ID partition (dev floor 90001 / prod ceiling 89999); on-import 403 in dev
Shared GitHub App (one webhook URL) prod dev-range gate + outbound-only config-gated mirror
Prod DOI minting from dev ENVIRONMENT gates + prod DOI secrets absent from dev
Search-engine leakage host-gated X-Robots-Tag + dynamic robots.txt

Activity

  1. added a commit that references this issue on Jul 18, 2026
  2. neuromechanist commented on Jul 18, 2026

    @neuromechanist
    ContributorAuthor

    Epic complete, merged to dev in PR #956 (commit 2206f27).

    Staging is live and in use: test.nemar.org plus api-test / data-test / zarr-test.nemar.org on the dev worker, backed by the dedicated nemar-dev bucket and nemar-db-dev. All seven curated exemplars are published with sandbox DOIs (10.5072/FK2) and render across every staging surface.

    All 7 phases shipped (#924, #925, #926, #927, #928, #929, #930). The cross-phase review found a critical hole no per-phase review could see: deleteDatasetCascade had no environment or ID-band guard, so a non-production worker could delete a real production GitHub repository through four manual endpoints. Fixed and verified live.

    Follow-ups tracked separately: #955 (exemplar creation is not retry-safe after a partial failure), #957 (request-driven email flows are not environment-gated), nemarOrg/website#155 (staging deploy workflow fix), and the two cross-repo central-workflow changes documented in .context/phase5-cross-repo-owner-deploys.md.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    featureNew feature or enhancement

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions