loading: key precompile cache file names on the package's source paths - #63293
Closed
IanButterworth wants to merge 4 commits into
Closed
IanButterworth wants to merge 4 commits into
IanButterworth wants to merge 4 commits into
Conversation
The precompile cache file slug is derived from the active project path (plus julia binary, sysimage, flags, cpu target and preferences), on the assumption that one project path holds one set of package versions. That breaks when distinct environments share a path, e.g. apps that each mount their project at `/work` in a container while sharing a depot: different versions of the same package, and the same version compiled against different dependencies, map to the same file name and overwrite each other. Alternating loads recompile every time, and concurrent precompilation hits "Required dependency ... failed to load from a cache file" and "Module ... is missing from the cache" when a file is replaced between the staleness check and the load. Hash the manifest contents into the slug as well, so environments that resolve different versions get different cache files. Environments without a manifest, and the pidfile path (which passes an empty project), are unchanged. Cache file lookup scans the directory, so the naming change does not affect loading of existing files. A version switch within one project now leaves the previous version's cache file alongside the new one until the per-package LRU evicts it, instead of overwriting it, which also makes switching back free. Fixes JuliaLang#63268 Assisted-by: Claude Code (Opus 5) Assisted-by: Codex (gpt-6-astra)
IanButterworth
added
packages
Keying cache file names on the manifest means a version change leaves the previous version's file behind until the per-package LRU evicts it. Once the source of such a file has been removed from every depot (as `Pkg.gc` does after its collect delay) the file can never load again, so drop it at the next cache write for the package instead of letting it occupy an LRU slot. Only sources under a depot are judged, and only files that have not been loaded for a week, since loading touches the file: a process with another view of the depots (another container with its own package sources) keeps the files it uses fresh. Assisted-by: Claude Code (Opus 5) Assisted-by: Codex (gpt-6-astra)
IanButterworth
marked this pull request as draft
September 20, 2026 17:12
…manifest Hash the source path of the package and of every module its cache requires, as resolved in the current environment, into the cache file name instead of the whole manifest. The name then changes only when the package or something in its dependency closure changes version or location, which are exactly the recompiles that must not overwrite another environment's file. Unrelated manifest edits no longer move a package to a new file, so a dev'd package keeps overwriting its one file while other packages are added or updated around it. The required modules are read back from the `.ji` header alongside the preferences blob. Assisted-by: Claude Code (Opus 5) Assisted-by: Codex (gpt-6-astra)
IanButterworth
changed the title
With the package's source paths in the key, the active project path adds nothing: environments that resolve the same versions produce interchangeable cache files, so let them share one instead of each writing a copy. Assisted-by: Claude Code (Opus 5) Assisted-by: Codex (gpt-6-astra)
IanButterworth
removed
backport 1.10
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes #63268
If we go ahead with this, we might want a more conservative fix for 1.13/1.10
Claude:
Cache file names are keyed on the active project path, so environments that share a path but resolve different versions (e.g. apps each mounting their project at
/workin a container with a shared depot) overwrite each other's files. Alternating loads recompile every time, and concurrent precompilation fails with "Required dependency ... failed to load from a cache file" or "Module ... is missing from the cache".Change
.jiheader, like the preferences blob) instead of the project path. Registry packages live atpackages/Name/<uuid+tree-hash slug>/, so the path identifies the version.Pkg.gc) and that nothing has loaded for a week (loading touches the file). Sources outside a depot are not judged.Why not hostname or the manifest
--network=hostor fixed hostnames still collide, as do concurrent CI jobs on one runner using the same workspace path; macOS hostnames change with the network.Test
Switches one project's manifest between two depot-resident versions of a package with a dev'd dependent: both versions' files coexist, switching back needs no recompile, and a version whose source was removed has its file dropped once a week old. Fails on nightly at the second load. The two-container case itself needs Linux mount namespaces.
Reviewed by Codex (gpt-6-astra); its finding that a source missing from this depot stack is not necessarily removed led to the one-week condition.