Skip to content

Reorganize S3 bucket to subdirectory layout - #100

Merged
neuromechanist merged 2 commits into
devfrom
71-extensible-s3-bucket-structure-with-subdirectories-for-objects-versions-archives-and-analysis
Feb 7, 2026
Merged

neuromechanist merged 2 commits into
devfrom
71-extensible-s3-bucket-structure-with-subdirectories-for-objects-versions-archives-and-analysis

Conversation

@neuromechanist

@neuromechanist neuromechanist commented Feb 6, 2026 •

Copy link
Copy Markdown
Contributor

Summary

  • Reorganize S3 bucket from flat {datasetId}/{files} to subdirectories: objects/, version/, archives/
  • Update all backend S3 path construction (generateDatasetUploadUrls, generateStagingUrls, manifest paths)
  • Add generate-archive.yml GitHub Actions workflow for async zip creation via repository_dispatch
  • Add generate_archive step to publication orchestrator (non-blocking, after s3_lock)
  • Update CLI URL registration in dataset.ts and sandbox.ts to include /objects/ prefix
  • Replace old nemar-api.shirazi-10f.workers.dev URL with https://api.osc.earth/nemar across codebase
  • Add getArchiveUrl() S3 utility for presigned archive download URLs
  • Add migration script (scripts/migrate-s3-structure.sh) for nm000103-107
  • Add cleanup script (scripts/cleanup-test-datasets.sh) for test datasets

Target S3 Structure

s3://nemar/{datasetId}/
    objects/      # git-annex content-addressed blobs
    version/      # manifests (v1.0.0.json)
    archives/     # downloadable zip snapshots (v1.0.0.zip)

Files Changed

File Change
backend/src/services/s3.ts S3 path construction, new getArchiveUrl()
backend/src/routes/admin.ts generate_archive step in publish orchestrator
backend/src/services/github.ts Archive workflow template, triggerArchiveGeneration(), API URL fix
src/commands/dataset.ts URL registration with /objects/ prefix
src/commands/sandbox.ts URL registration with /objects/ prefix
CLAUDE.md Backend infrastructure docs, S3 structure
docs/reference/api.md API URL update
test/setup.ts Test API URL update
scripts/migrate-s3-structure.sh New: migrate existing datasets
scripts/cleanup-test-datasets.sh New: cleanup test data

Test plan

  • bun test passes (221 tests, 0 failures)
  • Deploy backend and verify new upload stores files under {id}/objects/
  • Verify manifest operations use {id}/version/ path
  • Run cleanup script on test datasets (--dry-run first)
  • Run migration script on nm000103-107 (--dry-run first)
  • Verify git annex get works from migrated repos
  • Test archive workflow triggers correctly

Closes #71

Move data files under objects/, manifests under version/,
add archives/ for zip snapshots. Update all S3 path
construction in backend, CLI URL registration, and
workflow templates. Add generate-archive workflow for
async zip creation via GitHub Actions. Update API URL
from old workers.dev to api.osc.earth/nemar. Include
migration and cleanup scripts for existing datasets.

Closes #71
- Fix incomplete archive: fail workflow on partial git-annex get,
  add verification step, validate client_payload inputs
- Fix silent success: log warning when getVersionTag fails in
  generate_archive step, record error in progress
- Narrow S3 Object Lock to objects/ prefix only (exclude
  version manifests and archives for correctability)
- Remove dead code: redundant response.status !== 204 check
- Simplify dataset.ts destructuring
- Update JSDoc: complete 13-step orchestrator listing, accurate
  module and function descriptions, fix stale step numbering
- Lift workflow variables to job-level env
- Fix stale manifests/ reference in plan.md
@neuromechanist

Copy link
Copy Markdown
Contributor Author

PR Review Summary

Ran 4 specialized review agents (code-reviewer, silent-failure-hunter, comment-analyzer, code-simplifier) in parallel. All findings addressed in e33d1c4.

Critical (2)

  1. Incomplete archive generation (github.ts workflow template) - git annex get . || echo silently continued on partial data retrieval, producing corrupt zip archives with dangling symlinks. Fixed: removed fallback, added explicit verification step that fails the workflow if any files are missing.

  2. Silent success on version tag failure (admin.ts:2765-2769) - When getVersionTag returned an error Response, generate_archive was marked as successfully completed without triggering anything. On resume, the step would be permanently skipped. Fixed: added else branch that logs a warning and records the error in updateProgress.

Important (3)

  1. S3 Object Lock scope too broad (s3.ts:276) - applyObjectLock used ${datasetId}/ prefix, which with the new subdirectory layout now locks manifests (version/) and archives (archives/) alongside data blobs. Locking manifests prevents corrections; locking archives prevents regeneration. Fixed: narrowed to ${datasetId}/objects/.

  2. git-annex fileprefix not updated (git-annex.ts:446) - configureS3Remote still uses fileprefix=${datasetId}/ without /objects. Not an immediate bug since presigned URLs are used for uploads, but would break native git annex copy --to. Noted for future work.

  3. Dead code in HTTP status check (github.ts:729) - !response.ok && response.status !== 204 is redundant since 204 is in the 200-299 range (already .ok). Simplified to if (!response.ok).

Suggestions (5)

  1. Stale inline comment "Step 6: Notify user" updated to "Step 13" after generate_archive insertion.
  2. JSDoc listed 7 of 12 publication steps; expanded to all 13.
  3. Workflow template repeated DATASET_ID/VERSION in 3 separate steps; lifted to job-level env: and added client_payload validation step.
  4. Unnecessary object-then-destructure in dataset.ts; simplified to const { s3_config, s3_prefix } = datasetInfo.
  5. Stale manifests/v*.json reference in plan.md updated to version/v*.json.

Verification

  • All 221 tests pass (0 failures)
  • Biome formatting clean

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant