Skip to content

Prepare TAPR sources and reproducible historical refreshes - #47

Draft
adpena wants to merge 13 commits into
feat/dataset-contractsfrom
codex/tapr-header-crosswalks
Draft

adpena wants to merge 13 commits into
feat/dataset-contractsfrom
codex/tapr-header-crosswalks

Conversation

@adpena

@adpena adpena commented Oct 30, 2026 •

Copy link
Copy Markdown
Owner

TAPR preparation depended on narrowed spreadsheets and separate year-specific scripts. This change uses one strict native reader and staged producer, preserving publisher codes, original labels, canonical Python field names, identity roles and source annotations. It retires two prior producer implementations.

Current 2024 Student/Staff sources preserve all 709 columns across both levels. Retained PROF keeps 407 campus and 420 district columns, including 38/42 codes absent from the selected three-family union. Graduation profiles retain 42 metrics per scope for both 2024 and 2025. Field periods come from selected Config or frozen catalog metadata; mixed, undocumented and unresolved meanings remain explicit.

The refreshed 2025 Staff observations contain 106 campus and 118 district columns, adding 15/17 fields to the pinned inputs without dropping a prior code. Four source-only Config branches register Student and Staff separately for campus and district preparation. The same Staff resource moves into referenced runtime schemas and its proposal path is removed; Student mappings come from the existing reviewed tuples and exact retained headers/dictionaries. Historical composite contracts, prior Config entries/options, enrichment bindings, prepared defaults and release 0.0.126 remain unchanged. These contracts cover the selected source tables, not every annual TAPR family.

One typed HTML dictionary extractor retains ordered duplicates, complete definitions, independently unknown components and source anchors. The existing dictionary_records contract remains the sole occurrence authority. Only a complete occurrence can document a field; partial occurrences are never combined, and conflicting known components fail. Researchers retain the existing code/label/field mapping APIs. An offline dictionary command reads capped, explicitly decoded bytes, binds their actual SHA and length to the acquisition receipt, and records append-only provenance.

The historical refresh CLI acquires or replays explicit public GET/POST controls, including repeated names and source order. New acquisitions require explicit SHA-pinned metadata. Data requests also validate the registered dataset, exact configured year and edition before HTTP; references use an explicit documentation section. Config parsing reuses one canonical captured-byte loader, preserves ordinary fallback behavior, and never supplies scientific policies or periods to observation metadata. Selected documentation closures are verified before requests, and optional redirect blocking keeps a request at its literal URL. Immutable observations retain original dates, bytes, annual references and predecessor history. Native keyed comparison separates structural drift, raw-token changes and exact decimal-spelling relations, with bounded entity examples and truncation flags. Scientific statuses remain unbound unless an explicit field policy is selected. Shared codes or values do not establish renames or methodological comparability. No command activates Config.

The existing staged producer accepts explicit processing dependencies keyed by selected dataset ID. It verifies their typed closures before reading source tables, copies them through the same provenance store, and includes them in each selected derivation. This retains corrections without placing processing IDs in acquisition-only metadata. Annual eligibility now comes from exact registered Config year/version checks instead of a fixed 2024/2025 allowlist; generic Config fallback semantics stay unchanged. No real 2026 input was added.

Validation at d27553c1:

  • Ruff, typing, dependency checks and 837 tests passed on each Python 3.14.10 and 3.15.2, with one optional external-PostgreSQL skip each. Explicit matching post-commit receipts and the normal pre-push hook passed.
  • Independent adversarial review closed four dictionary issues covering input bounds, parsed-byte provenance, block-text spacing and ambiguous class attributes. Separate final reviews cleared the acquisition change, append-only Student label correction, and eight-path source-registration/producer change. The final reviewed bytes match the committed source. Actual retained Staff headers and ordered dictionary observations were checked separately.
  • Two canonical offline 2025 Staff comparisons matched all 9,084 campus and 1,208 district keys. All 948,652 shared cells are lexically identical; the 15/17 additional columns remain preserved. Original local-observation dates remain unknown where originally unknown, and exact missing metadata dependencies were restored without replacing records. Scientific policies stayed unbound; this is source-observation evidence, not measurement-comparability or release qualification.
  • Two fresh 2025 Student observations are byte-identical to the pinned inputs: 243 campus and 242 district columns, with 9,084/1,208 data records. A linked append-only annotation corrects their observation labels while preserving the original receipts, dates and bytes; no redundant value comparison or reacquisition was needed.
  • Five independent offline synthetic probes passed for cross-scope correction closure, invalid processing dependency rejection before source reads, and cleanup after a second-scope staging failure. Both actual 2025 source producers then completed for both scopes: Student has 9,084×243 campus and 1,208×242 district values; Staff has 9,084×106 and 1,208×118. Native Parquet readback equality and closed derivation graphs passed, including the explicit Student correction. Python 3.15 consumed all four exact files in eight batches capped at 4,096 rows, without the entity engine. Independent actual-output review passed. These are staged source preparations; publication, catalog schema3 and app adoption remain separate.

Other families and further annual refreshes remain separate work. The branch inherits #46's catalog projection and read-only SQLite fixes through the same original commits. Draft stacked on #46; distribution still requires a new versioned bundle and consumer qualification. The app continues using immutable release 0.0.126.

@adpena adpena changed the title Preserve publisher header crosswalks in lazy dataset contracts Prepare complete TAPR profile sources with verifiable provenance Oct 31, 2026
@adpena adpena changed the title Prepare complete TAPR profile sources with verifiable provenance Prepare TAPR profiles with complete source columns and provenance Oct 31, 2026
@adpena adpena changed the title Prepare TAPR profiles with complete source columns and provenance Prepare TAPR sources with field metadata and provenance Oct 31, 2026
@adpena adpena changed the title Prepare TAPR sources with field metadata and provenance Prepare TAPR sources and reproducible historical refreshes Oct 31, 2026

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant