Repository navigation
Reorganize S3 bucket to subdirectory layout - #100
Conversation
Move data files under objects/, manifests under version/, add archives/ for zip snapshots. Update all S3 path construction in backend, CLI URL registration, and workflow templates. Add generate-archive workflow for async zip creation via GitHub Actions. Update API URL from old workers.dev to api.osc.earth/nemar. Include migration and cleanup scripts for existing datasets. Closes #71
- Fix incomplete archive: fail workflow on partial git-annex get, add verification step, validate client_payload inputs - Fix silent success: log warning when getVersionTag fails in generate_archive step, record error in progress - Narrow S3 Object Lock to objects/ prefix only (exclude version manifests and archives for correctability) - Remove dead code: redundant response.status !== 204 check - Simplify dataset.ts destructuring - Update JSDoc: complete 13-step orchestrator listing, accurate module and function descriptions, fix stale step numbering - Lift workflow variables to job-level env - Fix stale manifests/ reference in plan.md
PR Review SummaryRan 4 specialized review agents (code-reviewer, silent-failure-hunter, comment-analyzer, code-simplifier) in parallel. All findings addressed in e33d1c4. Critical (2)
Important (3)
Suggestions (5)
Verification
|
Summary
{datasetId}/{files}to subdirectories:objects/,version/,archives/generateDatasetUploadUrls,generateStagingUrls, manifest paths)generate-archive.ymlGitHub Actions workflow for async zip creation viarepository_dispatchgenerate_archivestep to publication orchestrator (non-blocking, afters3_lock)dataset.tsandsandbox.tsto include/objects/prefixnemar-api.shirazi-10f.workers.devURL withhttps://api.osc.earth/nemaracross codebasegetArchiveUrl()S3 utility for presigned archive download URLsscripts/migrate-s3-structure.sh) for nm000103-107scripts/cleanup-test-datasets.sh) for test datasetsTarget S3 Structure
Files Changed
backend/src/services/s3.tsgetArchiveUrl()backend/src/routes/admin.tsgenerate_archivestep in publish orchestratorbackend/src/services/github.tstriggerArchiveGeneration(), API URL fixsrc/commands/dataset.ts/objects/prefixsrc/commands/sandbox.ts/objects/prefixCLAUDE.mddocs/reference/api.mdtest/setup.tsscripts/migrate-s3-structure.shscripts/cleanup-test-datasets.shTest plan
bun testpasses (221 tests, 0 failures){id}/objects/{id}/version/pathgit annex getworks from migrated reposCloses #71