Repository navigation
Conversation
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Raw TAPR preparation previously decoded entire CSV inputs, retained full campus and district tables, and read the complete Parquet output back into memory. Replace that path with one logical-record scanner and byte-admitted native Arrow batches. The public eager API collects the same batches, preserving exact strings, reviewed headers and annotations, source hashes, strict identities and provenance.
Validate every selected raw source before writers or archives, then stage scopes sequentially with one admitted batch per Parquet row group and exact bounded readback. The producer accepts one physical table per source. Existing eager multi-table APIs remain available explicitly; the historical two-table 2025 combined profiles are rejected by bounded preparation. Documentation uses the existing separate student/staff profiles and fresh output directories. Scientific interpretation, defaults and published snapshots are unchanged.
Stacked on #56. Independent review cleared the implementation and the narrow corrective test/documentation migration. The initial full check on
41f8233failed on obsolete combined-table/eager-writer fixtures; those failures remain recorded. The corrective commit changes only tests and documentation, preserving the three core implementation modules.Validation on corrected commit
4d0522bdcdbc41cb4570d0a0050ea3a9c8160d8d: lockedscripts/checkand explicit matching receipt verification passed on Python 3.14.10 and 3.15.2. Each run passed Ruff, ty, deptry and 1,051 tests, with one optional external-PostgreSQL skip and one expected optional-dependency warning.Synthetic narrow/wide and adversarial coverage verifies the admission and exact-read contracts. Actual School Progress processing, peak RSS, throughput and production capacity remain separately gated; this PR makes no resource or data-release qualification claim.