Skip to content

Release v0.8.0: SCCN Cloudflare migration (Phases 1-10) - #355

Merged
neuromechanist merged 14 commits into
mainfrom
dev
Apr 26, 2026
Merged

neuromechanist merged 14 commits into
mainfrom
dev

Conversation

@neuromechanist

Copy link
Copy Markdown
Contributor

Summary

Cuts NEMAR over from the personal Cloudflare account (yooz-labs / shirazi-10f) to the SCCN institutional account. Closes Epic #314 phases 1-10. Phase 11 (old-account cleanup) stays on a 30-90 day timer post-merge per #332.

This is the release PR; merging will:

  • Auto-deploy the SCCN config to the production worker on push to `main`.
  • Auto-tag `v0.8.0` and publish `nemar-cli@latest` to npm via `auto-tag.yml` + `npm-publish.yml`.

What ships

Phase PR Outcome
1 #322 `docs.nemar.org` live on SCCN Pages (`nemar-docs` project)
2 #323 `dashboard.nemar.org/citations/` live on SCCN Pages
3 #324 SCCN D1 + Vectorize + worker script created
4 #325 All 16 worker secrets mirrored to SCCN
5–6 #326, #327 SCCN config landed (`backend/wrangler-sccn.toml`)
7 #345 Email sender/reply-to env-configurable; SCCN sends from `noreply@nemar.org`
8 #351 `api.nemar.org` bound as Workers Custom Domain on SCCN worker
9 #352 CLI default API URL flipped to `api.nemar.org` + silent migration shim for stored configs
— #353 `/health` and `/` report version from root `package.json` (single source of truth)
10 #354 CI/CD flipped to SCCN: `deploy-backend.yml` (-c wrangler-sccn.toml), `docs.yml` (nemar-docs project), `catalog-sync.yml`, `test.yml`, dataset-repo webhook templates

Verification (already completed on dev)

  • `api.nemar.org/health` returns `version: 0.8.0-dev` from SCCN worker (will be `0.8.0` post-merge).
  • `api.nemar.org/datasets` byte-identical to `nemar-api.sccn-org.workers.dev/datasets`.
  • D1 prod parity verified pre-cutover (only 1-row catalog drift on `ds006963`, self-heals via catalog-sync cron).
  • All 16 worker secrets mirrored, including matching `ENCRYPTION_KEY` for the user-revoke decrypt path.
  • `wrangler@4 deploy --dry-run -c wrangler-sccn.toml` (top-level + `--env dev`) binds correctly to SCCN `nemar-db` (`009b1a44-...`) and `nemar-db-dev` (`017d8f4e-...`).
  • Test on `dev`: installed `nemar-cli@dev = 0.8.0-dev.613`, `dataset list` returned 27 pages from SCCN dev.
  • `/test/config-migrate-api-url.test.ts` covers the legacy-URL rewrite shim (no mocks, real temp config dir).

Out of scope, follow-up

  1. Dataset-repo CI re-roll — once this PR merges and SCCN prod has the new `backend/src/services/github.ts` webhook templates, run `nemar admin ci add --all` to commit updated workflow files to all ~100 `nm*` repos. Until then, existing dataset repos still POST DOI/enrich webhooks to the personal-account read-only buffer.
  2. GitHub org webhooks on `nemarDatasets` — repoint org-level webhooks to `api.nemar.org/webhooks/...` via `gh api -X PATCH`.
  3. Phase 11 (Phase 11: Old account cleanup and deprecation #332) — personal-account decommission, scheduled 30-90 days post-cutover.

Test plan

  • After merge, watch `Deploy Backend` workflow on push to `main`. Confirm SCCN prod redeploys (`https://api.nemar.org/health\` reports `version: 0.8.0`).
  • Watch `auto-tag.yml` create `v0.8.0` tag.
  • Watch `npm-publish.yml` publish `nemar-cli@latest = 0.8.0` to npm.
  • `bun install -g nemar-cli@latest` and confirm `nemar --version` reports `0.8.0`.
  • Trigger `Catalog Sync` (`gh workflow run catalog-sync.yml --ref main`) and confirm both prod and dev syncs return non-zero `records_synced` against SCCN endpoints.
  • Run dataset-repo CI re-roll (`nemar admin ci add --all`), then trigger one `workflow_dispatch` of `llm-enrichment` on a re-rolled repo and confirm SCCN `/webhooks/llm-enrich` accepts the payload.

* Add SCCN wrangler config and update docs site_url

- Add backend/wrangler-sccn.toml for SCCN Cloudflare account
  with new D1, Vectorize, and Workers AI bindings
- Update mkdocs.yml site_url to docs.nemar.org
- SCCN D1 prod: 009b1a44-a385-4ecf-812d-ec8341587cb5
- SCCN D1 dev: 017d8f4e-a591-4a3d-944b-01b5ffe5101d

* Fix dev API URL in SCCN wrangler config

* Address PR review: fix stale TODOs, secrets list, API URL

* Polish comments per review: clarify cfman, ENCRYPTION_KEY, header
- Add migration 0019: description column on users table
- Update wrangler-sccn.toml to use cfman as primary tool
- Update CLAUDE.md with cfman accounts and SCCN infrastructure
- cfman accounts: neuromechanist (personal), sccn (SCCN)
Brings to dev from main:
- Maintenance mode middleware (#342)
- MAINTENANCE_MODE=read-only flip (prod only; dev [env.dev.vars] stays off)
- Upload resume fix (#335)
- S3 initremote fix on resume (#337)
- PR-merge workflow, HED detection, bulk CI deploy (#338)
- npm publish fixes (Node 20, npm@11, OIDC)
- GitHub Release auto-tag
Sync main to dev: maintenance mode + hotfixes
* Make sender/reply-to env-configurable for SCCN

* Address PR review: test, validation, docs

* Fix stale test: update nm999999 past MAX_NUMBER cap

* Fix stale CLI test: update nm999999 past MAX_NUMBER cap
* Phase 8: bind api.nemar.org custom domain on SCCN worker

Adds Workers Custom Domain route for api.nemar.org and updates
API_BASE_URL var to match. Cloudflare auto-manages the proxy DNS
record and TLS cert; no separate DNS step needed.

Tested:
- curl https://api.nemar.org/health -> 200 (worker version 0.1.0)
- curl https://api.nemar.org/datasets -> 200, response identical to
  https://nemar-api.sccn-org.workers.dev/datasets
- workers.dev URL still answers (kept as buffer)

Refs #329

* Phase 8 review fixes: drop stale pre-cutover comments

- wrangler-sccn.toml header no longer warns against production use;
  this is now the canonical SCCN config.
- CLAUDE.md SCCN API URL line points to api.nemar.org with the
  workers.dev URL noted as fallback.

Refs #329
* chore: bump version to 0.8.0-dev

* Phase 9: flip CLI default API URL to api.nemar.org

Source changes:
- src/lib/api.ts, src/lib/config.ts, src/commands/auth.ts: DEFAULT_API_URL
  is now https://api.nemar.org. IS_DEV_BUILD detection (workers.dev
  substring) is unchanged and still flips for CI-injected dev builds.
- src/lib/config.ts: new migrateApiUrl() runs on module load and silently
  rewrites stored apiUrl entries that match the legacy default. Custom URLs
  (workers.dev, self-hosted) are untouched. The legacy redirect on the
  personal Cloudflare worker keeps stale configs working through Phase 11.

Test:
- test/config-migrate-api-url.test.ts: real integration test against a temp
  config dir, covers legacy rewrite, idempotent rewrite of already-new URL,
  and pass-through of dev/custom URLs.
- test/setup.ts default updated to api.nemar.org.

CI:
- .github/workflows/npm-publish.yml: dev-branch sed pattern updated to
  source from api.nemar.org and target nemar-api-dev.sccn-org.workers.dev
  (SCCN dev worker, not the personal one). Without this dev npm publishes
  would silently keep pointing at api.nemar.org.

Docs:
- scripts/generate-docs.ts: API reference base URL updated.

Refs #330

* Phase 9 review fixes

- Export DEFAULT_API_URL from src/lib/config and use it in auth.ts
  (was: duplicate hardcoded literal). Eliminates drift risk on the
  next URL change. Build dedupes the literal too.
- Add idempotency note on migrateApiUrl() per reviewer ask.
- Test: cover account with no apiUrl field (legacy-flat migration
  result) to guard the optional-chaining guard.

Refs #330
* fix: backend reports actual version on /health and /

- Stop hardcoding 0.1.0 in backend/src/index.ts; import version from
  backend/package.json. Single source of truth.
- Bump backend/package.json from 0.1.0 to 0.8.0-dev so it tracks the
  CLI version (CLI and backend ship together).
- bump-version.sh now keeps backend/package.json in sync, so future
  bumps update both files atomically.
- Refresh the index.ts header doc comment: production route is
  api.nemar.org (SCCN), dev is nemar-api-dev.sccn-org, and the personal
  api.osc.earth/nemar URL is the read-only buffer.

* fix: bump-version restore_versions handles backend pkg too

Reviewer note: the three failure paths restored only root package.json,
leaving backend/package.json modified if a bump aborted mid-way. Factor
a restore_versions() helper that touches both files so the working tree
is never half-rolled-back.

* refactor: backend reads root package.json as version source

Backend was importing ../package.json (backend/package.json), with the
bump script syncing the two manifests. That left two write paths to
keep aligned. Wire the runtime to import the repo-root package.json
directly (../../package.json from backend/src/index.ts), making the
CLI manifest the single source of truth.

backend/package.json stays (it is private/never published, but wrangler
and bun tooling expect it). bump-version.sh keeps it in lockstep and
now also asserts backend.version == root.version post-bump, so any
drift introduced by a hand-edit fails the bump loudly.

Wrangler dry-run + tsc clean with the new import path.
api.nemar.org is canonical post Phase 8; the personal api.osc.earth
worker is the read-only buffer scheduled for sunset in Phase 11.
Reorder the entries so the doc reflects current reality.
* Phase 10: flip CI/CD to SCCN account (#331)

- catalog-sync.yml: prod/dev URLs to api.nemar.org and SCCN dev worker
- test.yml: TEST_API_URL to SCCN dev (5 jobs)
- deploy-backend.yml: pass -c wrangler-sccn.toml so D1 + worker target SCCN
- docs.yml: pages project nemar-cli -> nemar-docs (SCCN)
- backend/services/github.ts: dataset-repo webhook templates to api.nemar.org
- scripts/hallu-sync.sh: API_BASE default to api.nemar.org

Dry-runs (wrangler@4 with SCCN token) bind to SCCN prod and dev
D1 + Vectorize correctly. Existing dataset repos still carry legacy
webhook URLs in committed workflow files; will re-roll via
nemar admin ci add as a follow-up so DOI/enrich webhooks land on
SCCN D1 instead of the personal-account read-only buffer.

* Address PR review: drop task refs, fix legacy default

- test/e2e-sandbox.test.ts: TEST_API_URL default to SCCN dev so local
  invocations stop hitting the personal-account read-only buffer
- backend/src/index.ts: rewrite the dual-mount comment so /nemar reads
  as the legacy buffer route, not "production"
- workflow comments: strip "Phase 10 / Epic #314" annotations from
  catalog-sync, deploy-backend (x2), and docs per CLAUDE.md no-task-refs
- src/lib/config.ts: wrap migrateApiUrl write in try/catch + log
  (previously a Conf write failure aborted CLI startup with raw stack)
- backend/services/email.ts: warn once when FROM_EMAIL is unset and
  resolveEmailConfig falls back to the legacy osc.earth default; on
  accounts that don't verify osc.earth this masks a deploy bug
- test/workflow-yaml.test.ts: assert webhook templates target
  api.nemar.org and never api.osc.earth; URL regression now caught
- test/api.test.ts: assert /health version equals root package.json
  (drift between root and backend manifests was masked by toBeDefined)
- test/setup.ts: seed NEMAR_CONFIG_DIR to a tmpdir before any import
  so a future parent-process import of src/lib/config.ts can't capture
  the developer's real config path
- comment cleanup: drop Phase/Epic task refs from wrangler-sccn.toml
  and src/lib/config.ts; rewrite legacy wrangler.toml header to flag
  read-only buffer role; tighten version-import comment in index.ts
- email.ts: collapse 3-line concatenated string to single template literal
- config.ts: split long console.error across 4 lines for biome formatter
@neuromechanist

Copy link
Copy Markdown
Contributor Author

Comprehensive review fixes

Four reviewers ran (code-reviewer, comment-analyzer, silent-failure-hunter, pr-test-analyzer). Verdict from code-reviewer: "no blocking cross-cutting issues; ship it." Findings addressed in commits `0b8d65d` + `bd62d16`:

Must-fix (silent-failure HIGH + test crit-9/6/7)

  • `src/lib/config.ts` — wrapped `migrateApiUrl` write in try/catch with logged warning. Previously, a Conf write failure (EACCES, ENOSPC, FS race) aborted CLI startup with a raw stack trace. Now logs and continues; the legacy redirect is the safety net.
  • `test/workflow-yaml.test.ts` — added assertion that all webhook templates target `api.nemar.org` and never `api.osc.earth`. URL regression in `backend/src/services/github.ts` is now caught.
  • `test/api.test.ts` — `/health` test now asserts `data.version === rootPkg.version` (was `toBeDefined()`). Drift between root and backend manifests would have previously masked.
  • `test/setup.ts` — seeds `NEMAR_CONFIG_DIR` to a tmpdir before any import so a future parent-process import of `src/lib/config.ts` can't capture a developer's real config path.

Should-fix (silent-failure MEDIUM)

  • `backend/src/services/email.ts` — `resolveEmailConfig` now logs once per worker instance when `FROM_EMAIL` is unset and falls back to the legacy default. On accounts where osc.earth is not Resend-verified, every email would silently fail; the warning surfaces in Workers Logs.

Project convention (CLAUDE.md "no task references in code")

Verified, no action needed (positive findings from code-reviewer)

  • Wrangler/esbuild inlines JSON imports at bundle time; `/health` version-import is durable.
  • `migrateApiUrl()` exact-match guard means custom URLs (workers.dev, dev, self-hosted) untouched.
  • `bump-version.sh` dual-package sync + drift assertion correct (loud-fail, restore_versions on all 3 paths).
  • `MAINTENANCE_MODE` correctly omitted from `wrangler-sccn.toml`; middleware defaults unset to "off".
  • All 4 wrangler invocations in `deploy-backend.yml` pass `-c wrangler-sccn.toml`; accidental personal-account deploy is impossible from CI.

Open follow-ups (not blocking this PR)

CI deploy-dev failed because backend/package.json pinned wrangler@3.91.0
which does not parse \`import ... with { type: "json" }\` (introduced in
PR #353). Bump to wrangler 4.85.0 to match the version used for local
dry-runs and the version recommended by wrangler's own deprecation
warning.

api-test was racing deploy-dev: the tightened /health assertion compared
against the locally-checked-out package.json version while the deployed
worker on dev still ran the previous version. Replace the exact-match
check with a semver shape regex - still catches version=undefined
(the original test gap) without locking to current build.
@neuromechanist
neuromechanist merged commit 190bf54 into main Apr 26, 2026
18 checks passed

This branch was previously deployed

2 inactive deployments
npm — 1153e53b Deployed Apr 27, 2026 by neuromechanist via publish #626
development — 1153e53b Deployed Apr 26, 2026 by neuromechanist via deploy-dev #174
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant