Skip to content

Select source metadata without inheriting legacy interpretation #51

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Draft
adpena wants to merge 2 commits into
base: codex/tapr-prepared-selection
Choose a base branch
from
Draft

Select source metadata without inheriting legacy interpretation #51

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
bd63995
Select complete metadata for explicit source readers
adpena Nov 1, 2026
6f31a0b
Pin default reader contracts separately from source profiles
adpena Nov 1, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
19 changes: 18 additions & 1 deletion docs/config.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -100,7 +100,7 @@ Supported metadata fields:
| `published_at`, `revised_at` | Known publisher dates or timezone-aware ISO timestamps |
| `license`, `citation`, `identifiers` | Verified reuse/citation information and identifiers such as DOIs |
| `notes` | Source-specific definitions, caveats, suppression rules, or other research context |
| `provenance_ids` | SHA-256 identifiers of immutable acquisition/processing receipts |
| `provenance_ids` | SHA-256 identifiers of immutable acquisition receipts; typed processing dependencies belong in processing inputs |
| `references` | Per-version dictionary, glossary, masking, column schema, or other documentation with URL, availability/review status, exact review hash, and acquisition receipts |
| `metrics` | Physical field, unit, and optional per-metric reference period |
| `value_policy`, `estimation` | Source token interpretation and a separately versioned estimation assumption |
Expand Down Expand Up @@ -392,6 +392,23 @@ source rows, for example
`cfg.load_df(..., reader_profile="source")` explicitly reads the full raw edition;
the default reader continues to use its exact retained prepared workbook.
Raw source preparation cannot run inside a retained-format release build.
An explicit profile may declare a complete `metadata` record. Selecting it with
`cfg.resolve_dataset(..., reader_profile="source")` replaces the base
`DatasetMetadata` through the canonical parser; no base fields or references
are merged. Its version must match the declared source version, and references
belong only to that record, not a second profile-level `references` list.
Header lookup, source preparation and build recording select this same record.
Calls without `reader_profile` retain the base metadata and reader behavior.

The 2024 full PROF profiles select raw-only metadata: unknown period, no metric
definitions, no estimation and no applied value policy. Complete raw strings,
including masking-looking tokens, remain uninterpreted. The retained annual
dictionary, glossary and masking references provide context, not blanket
qualification of an indicator's scientific meaning. A catalog with no selected
policy rejects policy interpretation; it does not synthesize a raw-token
policy or numeric status. Legacy default metadata and its methodology gates
remain unchanged. The fresh source observation IDs retain their genuine HTTP
dates; the older local observations are not rewritten or newly qualified.
Source identities and historical names are lossless period-specific attributes,
not additional metric keys or automatic `.meta` hydration. No current geometry
roster limits source coverage. Unknown field units and reference periods remain
Expand Down
1 change: 1 addition & 0 deletions provenance/records/0b020c2d0e06b020297ec1df4b8e451103865c472525af1b094c3984bd0821c6.json
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
{"artifact":{"filename":"2024-c-PROF-dictionary.html","sha256":"3d661f216f46b631760f9490b815c71799adbdfeacd85250fe9bae824aaff7af","size_bytes":126748},"dataset":"campus_tapr_student_staff_profile","downloaded_at":"2026-10-31T19:21:17.694388+00:00","kind":"acquisition","method":"download","parameters":{"acquisition_client_sha256":"8933e2750a70ed2e68a86066e9088546fd7405ddeb2b1ae9dd613303b05c0fc1","audit":"advanced_prof_supporting_columns","documented_query_controls":[["_service","marykay"],["_program","perfrept.perfmast.sas"],["_debug","0"],["ccyy","2024"],["prgopt","2024/tapr/Advanced Download/getdata_2024.sas"],["sumlev","C"],["setpick","PROF"],["ddic","Y"]],"family":"PROF","http_method":"GET","http_status":200,"max_bytes":2097152,"no_dataset_download":true,"publisher_edition":"2024","role":"data_dictionary","scope":"campus","source_link_lines":[416,427],"source_navigation_record_id":"7b83f1a304d4f7339c10f48989995c5e95221ee4465e280376b14374d421e74a","timeout_seconds":20},"recorded_at":"2026-10-31T19:21:17.694415+00:00","requested_url":"https://rptsvr1.tea.texas.gov/cgi/sas/broker?_service=marykay&_program=perfrept.perfmast.sas&_debug=0&ccyy=2024&prgopt=2024%2Ftapr%2FAdvanced+Download%2Fgetdata_2024.sas&sumlev=C&setpick=PROF&ddic=Y","requested_year":2024,"resolved_url":"https://rptsvr1.tea.texas.gov/cgi/sas/broker?_service=marykay&_program=perfrept.perfmast.sas&_debug=0&ccyy=2024&prgopt=2024%2Ftapr%2FAdvanced+Download%2Fgetdata_2024.sas&sumlev=C&setpick=PROF&ddic=Y","resolved_year":2024,"response_headers":{"content-type":"text/html; charset=iso-8859-1"},"schema_version":1,"section":"reference","source_metadata":{"identifiers":[],"metrics":{},"notes":"Bounded metadata-only observation from exact documented advanced PROF dictionary link. No data rows, original historical acquisition, scientific equivalence, period or units inferred.","period":{"kind":"unknown"},"provenance_ids":[],"publisher":"Texas Education Agency","references":[],"source_url":"https://rptsvr1.tea.texas.gov/perfreport/tapr/2024/Advance%20Download/download-data-adv.html","title":"2024 Campus Profile Data Elements","version":"2024"}}
1 change: 1 addition & 0 deletions provenance/records/20c59ad1a7a6328db3a98790eb36884caac9ab54d8dc1119adb72d530d1287ef.json
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
{"artifact":{"filename":"2024-d-PROF-dictionary.html","sha256":"799c7c204a5aeff6613aa9143774441e061f85efed50aa247f363636529eea05","size_bytes":130071},"dataset":"district_tapr_student_staff_profile","downloaded_at":"2026-10-31T19:21:18.071600+00:00","kind":"acquisition","method":"download","parameters":{"acquisition_client_sha256":"8933e2750a70ed2e68a86066e9088546fd7405ddeb2b1ae9dd613303b05c0fc1","audit":"advanced_prof_supporting_columns","documented_query_controls":[["_service","marykay"],["_program","perfrept.perfmast.sas"],["_debug","0"],["ccyy","2024"],["prgopt","2024/tapr/Advanced Download/getdata_2024.sas"],["sumlev","D"],["setpick","PROF"],["ddic","Y"]],"family":"PROF","http_method":"GET","http_status":200,"max_bytes":2097152,"no_dataset_download":true,"publisher_edition":"2024","role":"data_dictionary","scope":"district","source_link_lines":[416,427],"source_navigation_record_id":"7b83f1a304d4f7339c10f48989995c5e95221ee4465e280376b14374d421e74a","timeout_seconds":20},"recorded_at":"2026-10-31T19:21:18.071643+00:00","requested_url":"https://rptsvr1.tea.texas.gov/cgi/sas/broker?_service=marykay&_program=perfrept.perfmast.sas&_debug=0&ccyy=2024&prgopt=2024%2Ftapr%2FAdvanced+Download%2Fgetdata_2024.sas&sumlev=D&setpick=PROF&ddic=Y","requested_year":2024,"resolved_url":"https://rptsvr1.tea.texas.gov/cgi/sas/broker?_service=marykay&_program=perfrept.perfmast.sas&_debug=0&ccyy=2024&prgopt=2024%2Ftapr%2FAdvanced+Download%2Fgetdata_2024.sas&sumlev=D&setpick=PROF&ddic=Y","resolved_year":2024,"response_headers":{"content-type":"text/html; charset=iso-8859-1"},"schema_version":1,"section":"reference","source_metadata":{"identifiers":[],"metrics":{},"notes":"Bounded metadata-only observation from exact documented advanced PROF dictionary link. No data rows, original historical acquisition, scientific equivalence, period or units inferred.","period":{"kind":"unknown"},"provenance_ids":[],"publisher":"Texas Education Agency","references":[],"source_url":"https://rptsvr1.tea.texas.gov/perfreport/tapr/2024/Advance%20Download/download-data-adv.html","title":"2024 District Profile Data Elements","version":"2024"}}
1 change: 1 addition & 0 deletions provenance/records/5c564e395e98296e0bfe6388f6b7a51743e68832e04db17ec458130f4bfcfbc1.json
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
{"artifact":{"filename":"2024-campus-advanced-prof.csv","sha256":"36649862475a49e69d739cb938fceaef14de581d146f7862c13dfc95229fd7fa","size_bytes":12231481},"dataset":"campus_tapr_student_staff_profile","downloaded_at":"2026-10-31T20:08:43.051845+00:00","kind":"acquisition","method":"download","parameters":{"acquisition_client_sha256":"cabdd13086afac5414a0bc0cb50aa41098d9871c48be0d494c04008323883a9f","form_content_type":null,"form_controls":[["_service","marykay"],["_program","perfrept.perfmast.sas"],["_debug","0"],["prgopt","2024/tapr/Advanced Download/getdata_2024.sas"],["ccyy","2024"],["sumlev","C"],["setpick","PROF"]],"form_url":"https://rptsvr1.tea.texas.gov/cgi/sas/broker","http_method":"GET","http_status":200,"transport_limits":{"attempts":1,"max_bytes":33554432,"max_retry_wait":0,"timeout":20}},"recorded_at":"2026-10-31T20:08:43.051899+00:00","requested_url":"https://rptsvr1.tea.texas.gov/cgi/sas/broker?_service=marykay&_program=perfrept.perfmast.sas&_debug=0&prgopt=2024%2Ftapr%2FAdvanced+Download%2Fgetdata_2024.sas&ccyy=2024&sumlev=C&setpick=PROF","requested_year":2024,"resolved_url":"https://rptsvr1.tea.texas.gov/cgi/sas/broker?_service=marykay&_program=perfrept.perfmast.sas&_debug=0&prgopt=2024%2Ftapr%2FAdvanced+Download%2Fgetdata_2024.sas&ccyy=2024&sumlev=C&setpick=PROF","resolved_year":2024,"response_headers":{"content-disposition":"attachment; filename=CAMPPROF.csv","content-type":"text/comma-separated-values"},"schema_version":1,"section":"data_sources","source_metadata":{"download_url":"https://rptsvr1.tea.texas.gov/cgi/sas/broker?_service=marykay&_program=perfrept.perfmast.sas&_debug=0&prgopt=2024%2Ftapr%2FAdvanced+Download%2Fgetdata_2024.sas&ccyy=2024&sumlev=C&setpick=PROF","identifiers":[],"metrics":{},"notes":"Exact documented public advanced-PROF controls. Edition 2024 is distinct from the unknown mixed-table measurement period. No units, metric bindings or value policy inherited. Retained historical copies remain independent; shared code/label/value does not establish annual equivalence. Annual documents provide context only.","period":{"kind":"unknown"},"provenance_ids":[],"publisher":"Texas Education Agency","references":[{"provenance_ids":["7b83f1a304d4f7339c10f48989995c5e95221ee4465e280376b14374d421e74a"],"reference_id":"advanced_form","reviewed_sha256":"c4db3109431c2b2042c9f4f08035047c51e99d46ae41681b9f7b59b4a3dad7bd","role":"source_notes","status":"available","title":"Retained official 2024 Advanced Download form; exact request controls","url":"https://rptsvr1.tea.texas.gov/perfreport/tapr/2024/Advance%20Download/download-data-adv.html"},{"provenance_ids":["0b020c2d0e06b020297ec1df4b8e451103865c472525af1b094c3984bd0821c6"],"reference_id":"source_dictionary","reviewed_sha256":"3d661f216f46b631760f9490b815c71799adbdfeacd85250fe9bae824aaff7af","role":"data_dictionary","status":"available","title":"2024 Campus Profile publisher Data Elements; retained current HTML observation","url":"https://rptsvr1.tea.texas.gov/cgi/sas/broker?_service=marykay&_program=perfrept.perfmast.sas&_debug=0&ccyy=2024&prgopt=2024%2Ftapr%2FAdvanced+Download%2Fgetdata_2024.sas&sumlev=C&setpick=PROF&ddic=Y"},{"provenance_ids":["0c7bf8fcad671a0fe5181bfe59a6c63d6a7682d167a3e87e9361f14f8912fabb"],"reference_id":"glossary","reviewed_sha256":"e92631bfd01e9e2e5f4197bfe996069822c852dd5db423fe3faeea5ab7aa62e8","role":"glossary","status":"available","title":"2023-2024 TAPR Glossary; contextual reference, not blanket period binding","url":"https://rptsvr1.tea.texas.gov/perfreport/tapr/2024/glossary.pdf"},{"provenance_ids":["27c1d3cebf8added00184ca8a86910dbfc47a1264d7fd216adb25499d1906595"],"reference_id":"masking","reviewed_sha256":"f85b2ef340b07b721e12717fc0dc0f7f5e8290a022391e89de8b65664bdb860a","role":"masking_methodology","status":"available","title":"2023-2024 TAPR Masking Rules; applicability remains field-specific and unbound","url":"https://rptsvr1.tea.texas.gov/perfreport/tapr/2024/masking.html"}],"source_url":"https://rptsvr1.tea.texas.gov/perfreport/tapr/2024/Advance%20Download/download-data-adv.html","title":"2024 TAPR Campus Profile; new advanced-PROF raw HTTP observation","version":"2024"}}
1 change: 1 addition & 0 deletions provenance/records/7b83f1a304d4f7339c10f48989995c5e95221ee4465e280376b14374d421e74a.json
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
{"artifact":{"filename":"2024-advanced.html","sha256":"c4db3109431c2b2042c9f4f08035047c51e99d46ae41681b9f7b59b4a3dad7bd","size_bytes":37342},"dataset":"tapr_2024_download_navigation","downloaded_at":"2026-10-31T17:01:43.135353+00:00","kind":"acquisition","method":"download","parameters":{"acquisition_client_sha256":"91066d8b0507fbb2bb22bc6e18a03ce773bb0d215dc3dbd72fed374cbc95d4ad","audit":"download_family_coverage","edition":"2024","http_method":"GET","http_status":200,"max_bytes":2097152,"role":"download_navigation","scope":"small official navigation/reference metadata; no statewide dataset download","timeout_seconds":15},"recorded_at":"2026-10-31T17:01:43.135376+00:00","requested_url":"https://rptsvr1.tea.texas.gov/perfreport/tapr/2024/Advance%20Download/download-data-adv.html","requested_year":null,"resolved_url":"https://rptsvr1.tea.texas.gov/perfreport/tapr/2024/Advance%20Download/download-data-adv.html","resolved_year":2024,"response_headers":{"content-type":"text/html; charset=iso-8859-1"},"schema_version":1,"section":"reference","source_metadata":{"identifiers":[],"metrics":{},"notes":"Current bounded metadata acquisition only. No dataset rows or original historical acquisition are inferred.","period":{"kind":"unknown"},"provenance_ids":[],"references":[],"title":"Official TEA TAPR 2024 download_navigation","version":"2024"}}
1 change: 1 addition & 0 deletions provenance/records/d39ff6de82aae86f03f0d9821b86440f2191abbb0f53e1a00909601b80754e8d.json
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
{"artifact":{"filename":"campus-crosswalk.json","sha256":"54cc99ffc8ce29f7ea9b80f88d03c0a4c7c2b345846707e81bb897c5a02c4de0","size_bytes":190439},"dataset":"campus_tapr_student_staff_profile:schema","downloaded_at":null,"kind":"acquisition","method":"local_observation","parameters":{},"recorded_at":"2026-10-31T16:47:20.610394+00:00","requested_url":null,"requested_year":2024,"resolved_url":null,"resolved_year":2024,"response_headers":{},"schema_version":1,"section":"reference","source_metadata":{"identifiers":[],"metrics":{},"period":{"kind":"unknown"},"provenance_ids":[],"references":[],"title":"TEA Data full TAPR source/header contract"}}
1 change: 1 addition & 0 deletions provenance/records/e206f265a486d928a815ed9ebf24c99655813c29b0bc37bc31fa74ba9604021e.json
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
{"artifact":{"filename":"2024-district-advanced-prof.csv","sha256":"dc0b51f65f5a311df77798b0091b7d66830f4e43b6b28d7dad635e47e2b01e0d","size_bytes":1909138},"dataset":"district_tapr_student_staff_profile","downloaded_at":"2026-10-31T20:08:43.335579+00:00","kind":"acquisition","method":"download","parameters":{"acquisition_client_sha256":"cabdd13086afac5414a0bc0cb50aa41098d9871c48be0d494c04008323883a9f","form_content_type":null,"form_controls":[["_service","marykay"],["_program","perfrept.perfmast.sas"],["_debug","0"],["prgopt","2024/tapr/Advanced Download/getdata_2024.sas"],["ccyy","2024"],["sumlev","D"],["setpick","PROF"]],"form_url":"https://rptsvr1.tea.texas.gov/cgi/sas/broker","http_method":"GET","http_status":200,"transport_limits":{"attempts":1,"max_bytes":33554432,"max_retry_wait":0,"timeout":20}},"recorded_at":"2026-10-31T20:08:43.335610+00:00","requested_url":"https://rptsvr1.tea.texas.gov/cgi/sas/broker?_service=marykay&_program=perfrept.perfmast.sas&_debug=0&prgopt=2024%2Ftapr%2FAdvanced+Download%2Fgetdata_2024.sas&ccyy=2024&sumlev=D&setpick=PROF","requested_year":2024,"resolved_url":"https://rptsvr1.tea.texas.gov/cgi/sas/broker?_service=marykay&_program=perfrept.perfmast.sas&_debug=0&prgopt=2024%2Ftapr%2FAdvanced+Download%2Fgetdata_2024.sas&ccyy=2024&sumlev=D&setpick=PROF","resolved_year":2024,"response_headers":{"content-disposition":"attachment; filename=DISTPROF.csv","content-type":"text/comma-separated-values"},"schema_version":1,"section":"data_sources","source_metadata":{"download_url":"https://rptsvr1.tea.texas.gov/cgi/sas/broker?_service=marykay&_program=perfrept.perfmast.sas&_debug=0&prgopt=2024%2Ftapr%2FAdvanced+Download%2Fgetdata_2024.sas&ccyy=2024&sumlev=D&setpick=PROF","identifiers":[],"metrics":{},"notes":"Exact documented public advanced-PROF controls. Edition 2024 is distinct from the unknown mixed-table measurement period. No units, metric bindings or value policy inherited. Retained historical copies remain independent; shared code/label/value does not establish annual equivalence. Annual documents provide context only.","period":{"kind":"unknown"},"provenance_ids":[],"publisher":"Texas Education Agency","references":[{"provenance_ids":["7b83f1a304d4f7339c10f48989995c5e95221ee4465e280376b14374d421e74a"],"reference_id":"advanced_form","reviewed_sha256":"c4db3109431c2b2042c9f4f08035047c51e99d46ae41681b9f7b59b4a3dad7bd","role":"source_notes","status":"available","title":"Retained official 2024 Advanced Download form; exact request controls","url":"https://rptsvr1.tea.texas.gov/perfreport/tapr/2024/Advance%20Download/download-data-adv.html"},{"provenance_ids":["20c59ad1a7a6328db3a98790eb36884caac9ab54d8dc1119adb72d530d1287ef"],"reference_id":"source_dictionary","reviewed_sha256":"799c7c204a5aeff6613aa9143774441e061f85efed50aa247f363636529eea05","role":"data_dictionary","status":"available","title":"2024 District Profile publisher Data Elements; retained current HTML observation","url":"https://rptsvr1.tea.texas.gov/cgi/sas/broker?_service=marykay&_program=perfrept.perfmast.sas&_debug=0&ccyy=2024&prgopt=2024%2Ftapr%2FAdvanced+Download%2Fgetdata_2024.sas&sumlev=D&setpick=PROF&ddic=Y"},{"provenance_ids":["0c7bf8fcad671a0fe5181bfe59a6c63d6a7682d167a3e87e9361f14f8912fabb"],"reference_id":"glossary","reviewed_sha256":"e92631bfd01e9e2e5f4197bfe996069822c852dd5db423fe3faeea5ab7aa62e8","role":"glossary","status":"available","title":"2023-2024 TAPR Glossary; contextual reference, not blanket period binding","url":"https://rptsvr1.tea.texas.gov/perfreport/tapr/2024/glossary.pdf"},{"provenance_ids":["27c1d3cebf8added00184ca8a86910dbfc47a1264d7fd216adb25499d1906595"],"reference_id":"masking","reviewed_sha256":"f85b2ef340b07b721e12717fc0dc0f7f5e8290a022391e89de8b65664bdb860a","role":"masking_methodology","status":"available","title":"2023-2024 TAPR Masking Rules; applicability remains field-specific and unbound","url":"https://rptsvr1.tea.texas.gov/perfreport/tapr/2024/masking.html"}],"source_url":"https://rptsvr1.tea.texas.gov/perfreport/tapr/2024/Advance%20Download/download-data-adv.html","title":"2024 TAPR District Profile; new advanced-PROF raw HTTP observation","version":"2024"}}
8 changes: 4 additions & 4 deletions teadata/manifest.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -979,11 +979,11 @@ def __exit__(self, *_args: Any) -> None:
if self._token is not None:
ACTIVE_BUILD.reset(self._token)

def before_read(self, source: ResolvedDataset) -> dict[str, Any]:
def before_read(self, source: ResolvedDataset, *, reader_profile: str | None = None) -> dict[str, Any]:
if processor_identity() != self.processor_sha256:
raise ContractError("Processing source changed during the build")
expected = self.cfg.resolve_dataset(
source.dataset, source.requested_year, section=source.section
source.dataset, source.requested_year, section=source.section, reader_profile=reader_profile
)
if expected.to_dict() != source.to_dict():
raise ContractError("Source configuration changed during the build")
Expand Down Expand Up @@ -1059,7 +1059,7 @@ def record_edition(self, key: EditionKey, source: ResolvedDataset, *, artifacts:
"""
if processor_identity() != self.processor_sha256:
raise ContractError("Processing source changed during the build")
expected = self.cfg.resolve_dataset(source.dataset, source.requested_year, section=source.section)
expected = self.cfg.resolve_dataset(source.dataset, source.requested_year, section=source.section, reader_profile="source")
if expected.to_dict() != source.to_dict() or key.dataset != source.dataset or key.edition != source.metadata.version:
raise ContractError("Edition registration differs from selected Config source")
if key.identity in self.edition_entries:
Expand Down Expand Up @@ -1108,7 +1108,7 @@ def leaf_contract(self, key: EditionKey, source: ResolvedDataset, *, native_sche

if processor_identity() != self.processor_sha256 or key.dataset != source.dataset or key.edition != source.metadata.version:
raise ContractError("Leaf contract differs from selected build source")
if self.cfg.resolve_dataset(source.dataset, source.requested_year, section=source.section).to_dict() != source.to_dict():
if self.cfg.resolve_dataset(source.dataset, source.requested_year, section=source.section, reader_profile="source").to_dict() != source.to_dict():
raise ContractError("Source configuration changed during the build")
resource = self.cfg.column_contract(source.dataset, source.requested_year, reader_profile="source")
if resource is None:
Expand Down
Loading