Skip to content

Lazy chunked dir tree + drop catalog-row SSR fetch - #79

Merged
neuromechanist merged 3 commits into
mainfrom
feature/issue-78-lazy-dir-pagination
May 27, 2026
Merged

neuromechanist merged 3 commits into
mainfrom
feature/issue-78-lazy-dir-pagination

Conversation

@neuromechanist

Copy link
Copy Markdown
Contributor

Summary

Closes #78.

  • Chunked dir rendering. renderTopLevel now emits only the first 50 directory rows on initial paint and appends a "Show next 50 (X of N shown)" footer when more remain. Click reveals the next 50 in place and updates the counter; the footer self-removes when all rows are visible. on005509 paints 50 sub-* rows + button instead of all 330+ at once. Files at the root (README, dataset_description.json, code/, …) still render in full — only the directory list is capped.
  • Drop the api.nemar.org/datasets/<id> SSR fetch from the dataset detail page. With the post-#717 backfill (130/130 datasets at neuroschema 1.1), metadata.json carries license / modalities / tasks / authors natively — the catalog-row's CSV-split fallbacks never trigger. Saves one api.nemar.org call per detail render and removes a contributor to the rate-limit pressure that triggered the 429 earlier today. getDataset() stays in lib/api.ts because dataset/[id]/collaborators.astro still uses it.

The tree's <details> JSON listing endpoint, per-subdirectory lazy expand, and the discover / index pages' use of api.nemar.org/datasets are unchanged.

Test plan

  • bun run typecheck — 0 errors / 0 warnings
  • bun run lint — biome clean
  • bun run test — 372 tests pass (5 new chunking + 4 new renderDirChunkRows cases)
  • bun run build — clean
  • Local dev verification on on005509 (330+ subjects): first paint shows 50 dirs + "Show next 50 (50 of 333 shown)". Clicking through reveals the rest in chunks of 50; footer removed after the final reveal. Per-subdirectory expand still fetches data.nemar.org/<id>/<v>/<path>/?format=json.
  • performance.getEntriesByType('resource') confirms no api.nemar.org/datasets/on005509 browser request.
  • Post-deploy /browse against the Pages preview URL (cold isolate).

@cloudflare-workers-and-pages

cloudflare-workers-and-pages Bot commented May 27, 2026 •

Copy link
Copy Markdown

Deploying nemar-website with  Cloudflare Pages  Cloudflare Pages

Latest commit: 92432fa
Status: ✅  Deploy successful!
Preview URL: https://dc9f2f74.nemar-website.pages.dev
Branch Preview URL: https://feature-issue-78-lazy-dir-pa.nemar-website.pages.dev

View logs

@neuromechanist

Copy link
Copy Markdown
Contributor Author

Review pass — 3 sonnet agents

code-reviewer (pr-review-toolkit:code-reviewer)

  • Important / a11y: <ul data-dir-list> had no live region — screen readers wouldn't announce chunk reveals. Fixed in 92432fa: added aria-live="polite" to the dir list.
  • Important / code quality: enrichedMetadata uses ?? [] on fields the NeuroschemaDataset type already declares as non-optional string[]/Author[]. Deferred to Tighten enrichedMetadata to drop redundant ?? [] guards #80 — needs curl verification against live metadata.json responses before tightening.

silent-failure-hunter (pr-review-toolkit:silent-failure-hunter)

  • High / stuck button: if renderDirChunkRows returned rendered: 0 (data-rendered desync) the footer stayed live with no visible effect. Fixed in 92432fa: added if (result.rendered === 0) { console.warn(...); footer.remove(); return } guard.
  • High / missing cache: !listing || !list early-returned silently when the WeakMap cache was empty. Fixed in 92432fa: now console.errors + removes the footer so the user isn't left with a dead button.
  • Medium / text-node displacement: button label silently stopped updating if a browser extension displaced the firstChild text node. Fixed in 92432fa: added a console.warn in the else branch.

pr-test-analyzer (pr-review-toolkit:pr-test-analyzer)

  • Nice-to-have / tail-clamp: renderDirChunkRows had no test for non-zero fromIndex clamping (the actual final-chunk path live-verified via puppeteer on on005509). Added in 92432fa: two new cases pin the tail-clamp + the rendered: 0 contract the silent-failure fix relies on.
  • Click handler extraction: explicitly rejected — 30 lines of DOM plumbing, puppeteer (50 → 100 → ... → 333, footer removed) is the right test level.

374/374 tests green, lint clean, typecheck clean. Follow-up: #80 (drop ?? [] guards once metadata.json contract is confirmed).

@neuromechanist
neuromechanist merged commit d3636a0 into main May 27, 2026
1 check passed
@neuromechanist
neuromechanist deleted the feature/issue-78-lazy-dir-pagination branch June 21, 2026 22:50
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Lazy-paginate the dir tree (50 visible + 50 prefetched) + drop catalog-row SSR fetch

1 participant