Skip to content

Lazy-paginate the dir tree (50 visible + 50 prefetched) + drop catalog-row SSR fetch #78

Closed

Description

@neuromechanist

Problems

Two distinct things triggered today on https://ww2.nemar.org/dataset/on005509:

  1. 330 subject directories paint at once. Post-Pivot to canonical sources: drop summary.json + page-bundle + partial endpoints #76 the tree's all collapsed <details> rows, so it isn't broken per se, but the DOM is full of detail that most users never expand. User-reported observation: "shocker is that people usually don't want that."

  2. Each dataset detail SSR still hits api.nemar.org/datasets/:id for a catalog row whose fields are all now redundant with metadata.json (data.nemar.org). Post-backfill, the fallbacks the catalog row provided (modalities/tasks/authors CSV) are never triggered.

Proposal

(a) Chunked dir rendering. The JSON dir listing arrives in one fetch (all 330 in memory). Render only the first 50 dir rows visible at first paint. Pre-render the next 50 hidden via requestIdleCallback. A "Show next 50 (X of N)" button reveals the hidden chunk and schedules the next prefetch. At any moment the DOM holds at most 50 visible + 50 pre-rendered hidden = "only 50 ahead of the user." Root files (typically <20 entries on a BIDS dataset: README, dataset_description.json, code/, derivatives/, etc.) stay rendered in full.

(b) Drop the catalog-row SSR fetch. enrichedMetadata in src/pages/dataset/[id].astro currently reads catalogRow?.modalities / tasks / authors as fallback when metadata.recording_modality / tasks / authors is empty. With 130/130 datasets at neuroschema 1.1, those fallbacks aren't triggered anymore. Drop getDataset(id) from the SSR fan-out; trust metadata.json as the single source. Saves one api.nemar.org call per dataset detail render.

Scope

  • src/lib/render-dir-listing.ts — renderTopLevel gains a dirChunkSize option (default 50); add a renderDirChunk export that materializes rows for a slice. Emit a "Show next 50" button after the first chunk when more dirs remain.
  • src/pages/dataset/[id].astro — drop the getDataset call + catalogRow variable; simplify enrichedMetadata. Update the client <script> block: on button click, reveal pre-rendered chunk + schedule next prefetch via requestIdleCallback.
  • src/lib/render-dir-listing.test.ts — pin the chunked behavior (50 visible by default; "show next" button visible when more remain; button absent when total < 50).

What stays the same

  • data.nemar.org/<id>/<v>/?format=json is still one fetch, returns all entries. We slice on the client.
  • File rows at the root (README, dataset_description.json, etc.) render in full — no pagination.
  • Per-subdirectory lazy fetch behavior is unchanged.
  • Discover/index pages still call api.nemar.org/datasets (data.nemar.org has no list endpoint).

Acceptance

  • First paint of on005509 renders ~50 dir rows + the root files + a "Show next 50 (50 of 330)" button.
  • Clicking "Show next 50" reveals the next 50 instantly (pre-rendered) and updates the button to "Show next 50 (100 of 330)".
  • Background prefetch fires after each reveal so the next chunk is ready before the user clicks again.
  • Per-subdirectory lazy expand on click still works (separate code path).
  • enrichedMetadata builds without catalogRow; getDataset import removed from [id].astro.
  • No api.nemar.org calls on the dataset detail page beyond getQaAggregates.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions