Hyperalign and Plot_Coords Updates#1
Merged
Merged
Conversation
…e of input's first element (list of arrays)
…on holding and updating plot with each array's data
…-tools into K_working * 'K_working' of https://bitbucket.org/cdl-dartmouth/hyper-tools: edited resize--> return m added reduceD_list
…the same data strcuture
…ts generate blank, frozen plot windows
…ridis, inferno, plasma, magma)
…work correctly); checking resize portion of align_lists
Member
|
There are some conflicts that need to be resolved before approving the pull request. First, Untitled.ipynb should either be renamed or deleted. Second, it looks like hyperalign.py and plot_coords.py conflict (e.g. because they are out of sync with the versions on the main branch). They need to be resolved using the merge tool or something similar. |
Member
Author
|
@andrewheusser @jeremymanning @TomHaoChang do me a favor and try to break things!! I think it's looking pretty good... (comparing the matlab and python outputs is a good reference) |
Member
|
i think everything is now resolved... |
Member
|
closing pull request. |
jeremymanning
added a commit
that referenced
this pull request
Jul 17, 2026
…ot warn) Release-review blocker #1 (critical). trust=False was warning-then- unpickling, so trust=True was a warning-suppression flag, not a security boundary. Now _unpickle_bytes REFUSES (raises HypertoolsTrustError) any remote payload unless trust=True. This single chokepoint covers all three remote-pickle entry paths -- extension-based (.pkl/.geo/...), content-sniffed magic byte, and extensionless protocol-0 (ASCII) -- and the extensionless sniff path now lets the trust refusal propagate instead of relabeling it 'corrupted'. New tests/test_pickle_trust_boundary.py proves, against a real loopback server, that a malicious __reduce__ payload NEVER executes under trust=False (refusal happens before deserialization), that trust=True opts in, and that local files, numeric npz, and built-in datasets are unaffected. Updated the load-sources tests that asserted the old warn-and-continue behavior. load() trust= docstring rewritten. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
jeremymanning
added a commit
that referenced
this pull request
Jul 17, 2026
…ializing Release-review blocker #1 (built-in integrity). Every built-in dataset is now checked against a hard-coded SHA-256 BEFORE it is unpickled, and every cache hit is validated. A download counts as success only when its bytes match the pin (so a rate-limit HTML page served with status 200 is retried, not cached as the dataset). A persistent mismatch -- corrupt or tampered download, poisoned cache, or a changed upstream file -- is a HARD error that removes the unverified file, never a silent redownload- and-reparse. sotus (loaded via datawrangler) is exempt. Interim hashes pin the current hosted pickle files; a verified non-executable conversion bundle (npz/parquet/json.gz, all round-trip checked) has been produced for Dropbox re-hosting, after which each entry swaps to its new URL + hash and the .npz/.parquet path stops unpickling entirely. 6 offline integrity tests (stubbed downloads). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
jeremymanning
added a commit
that referenced
this pull request
Jul 18, 2026
… — completes blocker #1 Release-review blocker #1 (dataset re-hosting, Jeremy's chosen path). The 15 DATA datasets are now hosted on Dropbox as non-executable .npz / .parquet / .json.gz and read with allow_pickle=False -- the built-in data path never unpickles. EXAMPLE_DATA points at the new URLs; _REHOSTED maps each to its reconstruction (npz list/array, parquet, or json.gz text corpus); _EXAMPLE_DATA_SHA256 pins the converted-file hashes and every download/cache-hit is verified before read. Verified: each Dropbox upload's SHA-256 matches the file built locally; each dataset reconstructs to the EXACT value hyp.load returned from its former pickle (values + dtype + columns; datasaurus DataFrames keep cols ['x','y'], its unused per-frame index is not preserved). The old Google Drive pickle ids stay live so hypertools <1.0 keeps loading. The fitted sklearn *_model Pipelines remain Drive pickles, hash-verified before unpickling (skops re-hosting is a follow-up). +2 tests: re-hosted npz refuses object arrays (allow_pickle=False proof); every re-hosted dataset is on the non-pickle path. Full suite 2357 passed. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
jeremymanning
added a commit
that referenced
this pull request
Jul 18, 2026
Addresses Jeremy's second release review (2026-07-18). Blocker #1 (sotus re-host) is code-ready pending its Dropbox upload; everything else lands here. - #2 (blocker) headless docs build: RTD now provisions a pinned Chrome for kaleido (.readthedocs.yaml post_install: plotly_get_chrome), a new `docs-clean` CI job builds the docs from a pristine `git archive` with `sphinx -W -E -a` (asserting docs/auto_examples is untracked) so a missing browser or any gallery-execution failure fails before it reaches RTD, and conf.py's fallback comment no longer overstates its coverage (it only guards a missing plotly import, not a missing Chrome). - #3 datasaurus indexes: hyp.load returns raw data, so each frame's original contiguous global-row-range index is part of the public result -- restore it from an in-package constant (_DATASAURUS_INDEX_STARTS) instead of a fresh RangeIndex. Add an immutable compatibility baseline (tests/data/rehosted_compat_baseline.json) + equivalence test covering every re-hosted dataset (values/index/columns/dtype/ordering), proven equal frame-by-frame to the pre-1.0 original. - #4 trust docstrings: load_source/_parse_payload now state that remote unpickling is REFUSED (raises HypertoolsTrustError), not warned. - #5 changelog: drop "small ... base install"; match the corrected README. - #6 atomic + concurrency-safe downloads: stream into a private temp file, verify its SHA-256, then os.replace into the cache (atomic rename); add a best-effort per-dataset filelock. A reader never sees a partial/unverified file, and an interrupted download leaves no corrupt cache entry. Real thread/atomicity tests added. - #7 merge strategy: CONTRIBUTING.md documents squash-merge (no history rewrite) so the branch's large-binary history never enters dev-1.0. - #8 sdist smoke: the release-qualification job now installs + smoke-tests the sdist in its own fresh venv too, not only the wheel. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
jeremymanning
added a commit
that referenced
this pull request
Jul 18, 2026
…endent legacy-equivalence evidence Third review's two moderate findings (blocker #1 sotus awaits its Dropbox upload): - #2 compatibility checks could silently disappear behind skips. The matrix's dataset tests skip on a download failure so unrelated PRs aren't blocked by a transient outage -- but that means a green matrix run could mean "every hosted-dataset check skipped". Added a HYPERTOOLS_REQUIRE_DATASETS=1 mode that turns any download/load failure into a HARD failure, a dedicated `dataset-gate` CI job that sets it and reports the exact count validated (16), and a release-gate test that cannot pass by skipping. - #3 the compat baseline is generated from the current loader, so on its own it only prevents future drift. Added tests/data/rehosted_legacy_provenance.json: independent, frozen evidence recording each dataset's retired legacy source, that artifact's SHA-256, and the canonical hash of what PRE-1.0 hyp.load returned -- reconstructed directly from the legacy artifacts via scripts/gen_legacy_provenance.py (all 16 proven to match the baseline). A new test asserts the baseline still equals this frozen legacy hash, so a regression can no longer be blessed by regenerating the baseline. The baseline generator now refuses to overwrite without an explicit --force acknowledgement. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
jeremymanning
added a commit
that referenced
this pull request
Jul 18, 2026
…ocker #1 sotus.json.gz was uploaded to Dropbox (sha256 verified byte-identical to the built+verified file). sotus now loads through the SAME integrity-checked, non-executable, atomic-download path as wiki/nips -- no datawrangler, no pickle. - EXAMPLE_DATA['sotus'] -> the Dropbox URL; _REHOSTED['sotus']='jsongz_strlist' (a flat list of the 29 speech strings); _EXAMPLE_DATA_SHA256['sotus'] pinned to 20068fb4...; a new _parse_rehosted branch reads gzip+json with no pickle. - removed _load_sotus_corpus and the datawrangler special case in _load_example_data; corrected the sotus and _integrity_ok docstrings. - tightened test_all_downloadable_builtins_are_pinned to require EVERY built-in be pinned and assert no delegated (datawrangler-zoo:) source remains; added test_every_non_model_builtin_is_pinned_and_non_pickle (Jeremy's regression: every downloadable non-model built-in is BOTH hash-pinned AND non-pickle). - committed tests/data/rehosted_conversion_manifest.json (the checksummed conversion record, now including sotus) + a test binding its converted_sha256 to the loader pins. sotus's legacy artifact sha is null (its pre-1.0 origin was the datawrangler corpus, not a hypertools-hosted artifact; equivalence is anchored by the canonical content hash in the legacy-provenance manifest). Verified: fresh hyp.load('sotus') downloads from Dropbox, integrity-checks (sha 20068fb4), and returns the 29 speeches from a gzip file (magic 1f8b, not a pickle). Full suite 2381 passed / 0 failed. Every built-in dataset is now hash-pinned and non-executable. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
jeremymanning
added a commit
that referenced
this pull request
Jul 18, 2026
…w findings) Runtime is unchanged (sotus is already fixed); this hardens the verification tooling around it. - #1 (High) authenticate-before-deserialize: scripts/gen_legacy_provenance.py now verifies each retired legacy artifact against an immutable, reviewed EXPECTED_LEGACY_SHA256 map BEFORE it is ever unpickled, and aborts on any mismatch -- so re-running the documented procedure can't be tricked into deserializing whatever a retired URL happens to serve. Uses a private tempfile.mkdtemp workdir (no predictable .legacy_*.tmp repo files), never overwrites the trust anchors, and carries a prominent "run only in an isolated environment; this unpickles legacy artifacts" warning. - #3 sotus provenance is no longer circular: the generator had recorded hyp.load('sotus') -- now the CURRENT json loader -- as "legacy" evidence. sotus is instead recorded as a DOCUMENTED evidence exception (independent_evidence=false, legacy_canonical_hash=null, with a note that its datawrangler-corpus origin is not independently recoverable). The provenance test asserts the 15 independently-verified datasets match the baseline AND that sotus is explicitly marked as an exception, not silently claimed as proof. - #2 dataset-gate is now a genuine host-availability gate: it uses NO dataset cache and clears ~/hypertools_data first, so every dataset is fetched FRESH from its host (the loader accepts a hash-valid cache hit without contacting the host, so a cached job proved integrity, not reachability). The release-gate message now says "loaded" (accurate whether cached or fresh); the CI job's fresh download is what proves availability. - #4 dropped the now-redundant `set(L._REHOSTED) | {'sotus'}` (sotus is a member) and the stale "migration in flight" comments. EXPECTED_LEGACY_SHA256 matches the committed provenance artifact hashes, so the authenticated generator reproduces the manifest with no drift. Full suite 2381 passed / 0 failed. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
jeremymanning
added a commit
that referenced
this pull request
Jul 18, 2026
…ocs -W filter (5th-review lows) Two low-severity hardening items from the fifth review (no runtime change): - #1 exact coverage instead of a >= count: the baseline and legacy-provenance coverage tests now assert set-equality with set(L._REHOSTED) (a MISSING or a STALE entry both fail), and the independent-evidence set must be EXACTLY set(L._REHOSTED) - {'sotus'} rather than `len(indep) >= 15` -- so a dataset can't quietly lose its independent evidence while a stale entry keeps the count. scripts/gen_legacy_provenance.py also asserts set(LEGACY_SOURCE) == set(EXPECTED_LEGACY_SHA256) so the source map and the trust-anchor map can't drift out of sync. - #2 the docs -W-gate warning filter (which drops only sphinx-gallery's transient external searchindex.js fetch-failure warning) moved from an inline conf.py closure to a side-effect-free docs/_gallery_log_filter.py with a unit test (tests/test_docs_gallery_log_filter.py). The test confirms only the intended warning is dropped and real warnings are kept, and GUARDS the matched marker against sphinx-gallery's own source, so a future wording change fails loudly instead of silently disabling the filter. Full suite 2389 passed / 0 failed. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
jeremymanning
added a commit
that referenced
this pull request
Jul 21, 2026
…order Addresses a follow-up release-gate review (2 High + 1 Medium + hardening). - .github/workflows/test.yml: add `tags: ['v*']` to the push trigger. A `branches` filter WITHOUT a `tags` filter means tag pushes never trigger the workflow, so the release-gate job (keyed on refs/tags/) would never run on the v1.0.0 tag. (High #1) - All 68 published notebooks are now gated (High #2). docs/auto_examples/*.ipynb are gitignored and GENERATED at build time (not the "53 committed" the review assumed), so they can't be scanned in a bare checkout. Coverage now at both layers: the gate helper scans the tracked-tutorials + tracked-gallery UNION and asserts >=15 tutorial coverage; and the docs-clean job (which builds the gallery) release-gates every generated docs/auto_examples/*.ipynb on master/tags, asserting the PyPI spec (>=50 present). - RELEASE_CHECKLIST.md: reorder so the irreversible PyPI upload happens LAST -- build locally -> push master -> wait for green master CI -> tag the green commit -> wait for green tag CI -> THEN twine upload the already-verified artifacts. (Medium) - Hardening (tests/test_release_readiness_gate.py): README image ref must equal `v` + pyproject version EXACTLY (not any semver); CHANGELOG top version must equal pyproject version (new always-on check too); validate the CHANGELOG date with datetime.date.fromisoformat (rejects 2026-99-99, which the regex allowed). - docs/tutorials/stock_forecasting.ipynb: drop the "hypertools 1.0 preview" prose from the install comment (released flagship docs should not say "preview"). - RELEASE_CHECKLIST wording: "retargets committed cells" (docs/conf.py generates the gallery), and the enforcement table reflects all of the above. Full suite 2477 passed / 12 skipped / 0 failed. Release gate verified: hard-fails on the current dev state and passes the version-match; green after the checklist flips. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
new features:
-both functions accept lists of numpy arrays
-plot_coords features perceptually uniform color mapping
-hyperalign adds 1 and 2 columns of zeros to 1D and 2D arrays, respectively, before aligning
can now easily use the two together:
import plot_coords as plot
import hyperalign as hyp
plot.plot_coords(hyp.hyperalign(x))