Enable root-relative ILLink task cache sharing

docs/design/tools/illink/task-cache.md

@@ -1,4 +1,4 @@
-# ILLink task cache: v1 decisions
+# ILLink task cache decisions
 
 These are design decisions. Eligible task invocations compute a content-based key, restore a cached output directory on a hit, or run ILLink and store successful results.
 
@@ -7,13 +7,15 @@ The feature is experimental. Its configuration and behavior may change or be rem
 ## Scope and configuration
 
 - Cache complete ILLink invocations in the MSBuild task, not in the linker command-line implementation.
-- Keep v1 simple. Reuse results at the same input/output paths; defer cross-worktree reuse and retain absolute paths in cache identity.
+- Share eligible results across clones/worktrees with equivalent relative layouts. The cache directory was already shareable; root-relative identity does not change cache location or enable caching.
+- Accept physical `SourceRoots` with stable `MappedPath` metadata (`/_/`, `/_1/`, etc.). Represent known paths under these roots by a tagged logical-root/relative-path identity; retain tagged absolute identities for external paths. Require distinct physical/logical roots and consistent mappings for nested roots. Reject missing or invalid mappings and run uncached.
+- Do not change compiler, symbol-preservation, or caching defaults. For an eligible cache invocation with one physical root (including consistent nested mappings), use that root as the child process's working directory and render known in-root paths relative to it. Resolve original relative inputs against the caller's working directory first; retain absolute external paths. Do not change process-wide cwd or task inputs. Disjoint-root invocations retain their original execution and use logical key mapping only. Disabled or bypassed caching leaves execution unchanged.
 - Enable caching only through the `ILLINK_EXPERIMENTAL_CACHE` environment variable set to `true` (case-insensitive). Unset, empty, or `false` disables caching; invalid boolean values disable caching with a diagnostic. Disabled caching performs no cache I/O, even if a cache directory already exists.
 - Use the optional `ILLINK_EXPERIMENTAL_CACHE_PATH` environment variable for the directory override. Unset or empty selects the platform default. Selecting a directory does not enable caching, and path values are never interpreted as booleans.
 - Use platform-specific default directories: `$XDG_CACHE_HOME/illink` on Linux/other XDG Unix, falling back to `$HOME/.cache/illink`; `$HOME/Library/Caches/illink` on macOS; `LocalApplicationData/illink` on Windows.
 - Resolve defaults inside `ILLinkCache.TryCreate` through a private helper, not in targets. Resolve relative overrides against the current working directory.
 - Ignore relative `XDG_CACHE_HOME` values. If no usable user location is available, skip caching with a diagnostic rather than fall back to the working directory or shared temporary storage.
-- Read configuration from the task process's environment on each invocation. Set it before starting MSBuild; do not change process-wide environment settings during parallel builds. Reused build processes must receive the intended environment. There are no cache task parameters, MSBuild property wiring, or linker command-line switches.
+- Read enablement and directory configuration from the task process's environment on each invocation. Set it before starting MSBuild; do not change process-wide environment settings during parallel builds. Reused build processes must receive the intended environment. `SourceRoots` is a task input for identity, not another enablement setting or a linker command-line switch.
 - Do not promise stability or migration support for the on-disk format.
 
 ## API shape
@@ -27,7 +29,7 @@ internal sealed class ILLinkCache
 }
 ```
 
-- The task owns input identity and execution. Its private `TryComputeCacheKey` combines command-line and response-file arguments with input-content and toolchain identity; inability to compute a key bypasses caching with a diagnostic.
+- The task owns input identity and execution. Its private `TryComputeCacheKey` combines structured arguments with input-content and toolchain identity; inability to compute a shareable key bypasses caching with a diagnostic.
 - `TryCreate` resolves the cache directory and returns null with a diagnostic if initialization cannot proceed.
 - `TryRestore` combines lookup and restoration. True means complete restoration. Missing entries and expected I/O, access, or invalid-cache-data failures return false with appropriate logging and fall back to normal linking. Copies overwrite existing files and are not rolled back if a later copy fails. Unexpected exceptions propagate.
 - `Store` is best-effort. Lock contention and existing entries are normal skips; expected cache I/O failures are logged without failing a successful link. Unexpected exceptions propagate.
@@ -55,15 +57,22 @@ internal sealed class ILLinkCache
 
 ## Input identity and eligibility
 
-- Use SHA-256 over a versioned, length-delimited description. Preserve argument ordering and absolute paths, including the working and output directories.
-- Hash the contents of all supplied assemblies, reference assemblies, and root descriptors. Include adjacent PDBs, MDBs, configuration files, and satellite assemblies, even when symbol emission is disabled. Follow assembly metadata file entries for linked resources and additional modules. Optional-file additions and removals change the key; timestamps alone do not.
+- Use SHA-256 over a versioned, length-delimited description. The key domain is `v3`, distinct from legacy same-path and key-only mapping keys; the entry/manifest storage format remains `v1`. Legacy keys are not read or migrated. Include the execution mode, effective working directory, and exact generated execution arguments for root-relative invocations.
+- Generate execution and key arguments through the same response-file generator. Map only known path fields: assembly/reference/descriptor paths, supported extra-argument paths, linker/tool/input inventory paths, working directory, and output directory. Preserve argument ordering, filenames, relative layout, duplicate-reference precedence, and ordered search directories. Do not replace substrings in raw command lines. Bypass overridden command generation or working directories that differ from the modeled invocation, file-shaped root names, and assemblies whose names differ from their filenames.
+- Sort the input inventory by logical identity, not physical path. Hash the full contents of all supplied assemblies, reference assemblies, root descriptors, adjacent portable PDBs and configuration files, and satellite assemblies. Optional-file additions and removals change the key; timestamps alone do not.
+- Cache only supported path-independent symbol behavior. Inspect input CodeView paths and portable-PDB document paths, including embedded portable PDBs and satellites. Accept relative non-escaping names and conventional synthetic `/_N/` paths. Reject physical absolute paths, native PDBs, MDBs, malformed symbols, and unsupported debug-directory kinds. When external portable PDBs are present on potentially emitted assemblies and symbols are enabled, require the caller to explicitly enable `PreserveSymbolPaths`; otherwise destination-dependent emission runs uncached. Even symbol-stripping requests undergo input-path checks because copy actions can retain original bytes.
+- Inspect reference-assembly metadata and hash reference sidecars, but do not apply output-symbol restrictions to references explicitly assigned `skip`; their symbols are not emitted.
+- Reject literal physical source-root prefixes in emitted-input PE bytes, auxiliary input files, and portable-PDB custom-debug blobs, using UTF-8/UTF-16 spellings. This also catches ordinary compiler-generated physical `CallerFilePath` strings. These checks are conservative and do not constitute a general proof about opaque or compressed application data; callers remain responsible for the documented stable-input and environment contract.
+- Bypass additional modules and linked resources in this first shared-cache scope. Normal linking remains responsible for handling them; do not approximate their resolution or cwd-dependent resource behavior.
+- Reject path-shaped assembly references in metadata and assembly names in external XML. These can make resolver probes escape the declared top-level search-directory inventory. Parse external descriptors, substitutions, and link-attribute XML with external entities disabled; unreadable or malformed XML runs normally, uncached.
 - Hash the entire linker assembly on every invocation, without loading it or using its module version ID (MVID). Content changes invalidate the cache even when the MVID, file length, and timestamp are unchanged. Do not scan its directory. Bundled dependencies and configuration are not fingerprinted: changes to Cecil or deployment configuration require a cleared/isolated cache or disabled caching unless the linker binary also changes.
 - Continue hashing the task assembly and selected dotnet host executable. Do not enumerate or hash the host's installed hostfxr/shared-runtime trees: doing so makes lookup cost scale with every installed framework/version, including ones unrelated to the link. Runtime/tool files supplied as linker inputs remain content-hashed.
 - Defer identifying the runtime that executes ILLink; do not include a runtime version or framework description in the key. The task runs in MSBuild, so its CoreLib informational version and framework description are not reliable identities for the child linker's runtime. The runtimeconfig describes requested frameworks and roll-forward policy, not the exact resolved CoreLib. Finding that CoreLib would require resolved-runtime information from the caller, host-resolution plumbing, or an additional process; defer that complexity and startup cost.
 - This deliberately accepts that runtime installation changes not represented by other keyed inputs may reuse an entry; callers needing invalidation for those changes must clear or isolate the cache.
 - Include platform, architecture, and culture. Do not enumerate or hash inherited environment variables, or bypass caching based on them. This deliberately keeps v1 simple rather than maintaining a partial list of runtime/loader settings. Environment changes are reflected only when they change other keyed inputs, such as the selected host path or culture. Callers must disable caching when inherited settings introduce otherwise untracked dependencies, affect outputs, or require tool-execution side effects (for example, startup hooks or profilers).
-- Bypass caching for non-whitespace `ExtraArgs`, custom steps/data, dependency-dump options, and explicit `ToolTask.EnvironmentVariables` overrides. These are caller-supplied inputs that v1 does not model in its key and can introduce undeclared dependencies, external outputs, or side effects. These invocations still run the linker normally; no extra-argument parsing or runtime-specific exceptions are supported.
+- Bypass caching for arbitrary `ExtraArgs`, custom steps/data, dependency-dump options, and explicit `ToolTask.EnvironmentVariables` overrides. These can introduce undeclared dependencies, external outputs, or side effects. The supported `ExtraArgs` shape for runtime's library builds is `--ignore-link-attributes true` followed by zero or more `--link-attributes FILE`, `--substitutions FILE`, or `-d DIRECTORY` pairs. Hash every referenced XML file and every top-level `.dll`, `.exe`, and `.winmd` candidate in each search directory, including supported assembly sidecars. Preserve directory order in the arguments and leave linker resolution unchanged; adding, changing, or removing candidates invalidates the key, including shadowed candidates. An unreadable directory or unsupported candidate bypasses caching. All other SDK options passed through `ExtraArgs` remain subject to the bypass.
 - Log and bypass caching if required files cannot be read or an input assembly cannot be inspected, including PE images without managed metadata. Normal linking determines whether those inputs are acceptable. Input files must remain stable during hashing and linking.
+- Runtime's `eng/illink.targets` supplies its repository root with logical identity `/_/` to per-library, shared-framework, and out-of-band tasks. SDK publish targets forward initialized `@(SourceRoot)` items. This wiring does not normalize upstream assemblies or turn symbol preservation on. Runtime's current raw `ExtraArgs` include options outside the supported grammar and still bypass caching; structured runtime argument integration is separate.
 
 ## Entry layout
 
@@ -82,6 +91,7 @@ internal sealed class ILLinkCache
 - Purge requires an explicit cache root and UTC cutoff. Only published SHA-256 entry directories in `v1` are considered; delete entries whose last-used time is strictly older than the cutoff and retain equality. Leave staging directories and other layouts alone. Report per-entry maintenance errors and continue, then return nonzero if maintenance failed.
 - For a missing, malformed, non-UTC, or unreadable marker, report the fallback and use the entry directory's creation time instead. This may purge a recently used entry or retain a stale one if restoring a cache snapshot resets directory creation times. Do not migrate or repair the marker.
 - Purge must not run during any build using the cache. Do not add purge locks or change reader locking; coordination with active readers/writers remains a separate follow-up, as does protection against manual deletion.
+- Root-independent keys can make different worktrees target the same entry. Existing immutable publication and per-key writer locks support this; simultaneous misses still link independently in distinct output directories. Live eviction would require reader/exclusive-delete coordination, not just the writer mutex or usage timestamps.
 - CI may record build start after restoring a job-private cache, run linking, then purge against that timestamp before uploading the next snapshot. Preserve tool diagnostics but do not fail the build for maintenance errors. Only purge once all users have exited; incremental skips and unreached work in failed builds do not refresh usage. See the [CI recipe](../../../tools/illink/illink-tasks.md#ci-purge-ordering). This does not enable an ILLink cache in CI automatically.
 - Attempt best-effort cleanup of the current attempt's unpublished staging directory. Log cleanup I/O/access failures without failing a successful link; do not scan or delete other attempts' staging directories. Correctness must tolerate leftovers after crashes.
 

docs/tools/illink/illink-tasks.md

@@ -58,6 +58,15 @@ Eligible invocations use a key covering arguments, input-file and sidecar conten
 assembly, the selected dotnet host executable, and the entire ILLink assembly.
 File contents are hashed on every invocation. Linker changes invalidate the cache even when
 the module version ID (MVID), file length, and timestamp are unchanged.
+`SourceRoots` with stable `MappedPath` metadata allow equivalent relative layouts in different
+clones/worktrees to share a key. Only known path arguments and input identities are mapped;
+external paths remain absolute and argument/search-directory order is preserved.
+For eligible invocations with one physical root (including consistent nested mappings),
+the task runs ILLink from that root with known in-root paths rendered relative to it.
+Original relative inputs are resolved against the caller's working directory first.
+External paths remain absolute. This does not change process-wide cwd, mutate task inputs,
+or normalize bytes embedded in inputs or outputs. Disjoint-root invocations retain their
+original execution; disabled or bypassed caching leaves execution unchanged.
 Bundled dependencies and configuration are not fingerprinted. Changes to those files require
 a cleared/isolated cache or disabled caching unless the linker binary also changes.
 
@@ -67,13 +76,28 @@ clear or isolate the cache when that distinction is required.
 Cache hits replace the output directory without running ILLink.
 Directories are created in the cache only when storing a successful result. Hits do not replay warnings or other linker diagnostics.
 
-Caching is bypassed with a diagnostic for non-whitespace `ExtraArgs`, custom steps/data,
-dependency-dump options, and explicit task environment overrides. Options supplied through
-`ExtraArgs` remain unsupported for caching; these invocations still run the linker normally.
+Caching is bypassed with a diagnostic for arbitrary `ExtraArgs`, custom steps/data,
+dependency-dump options, and explicit task environment overrides. The only supported
+`ExtraArgs` shape is `--ignore-link-attributes true` followed by zero or more
+`--link-attributes FILE`, `--substitutions FILE`, or `-d DIRECTORY` pairs. Referenced files
+and search-directory assembly candidates are content-hashed, including shadowed candidates.
+Other options still run the linker normally, uncached.
 Inherited environment variables are not tracked; disable caching
 when they affect outputs or dependencies beyond the keyed inputs, or require tool-execution
 side effects such as startup hooks or profiling. Unreadable inputs or linker files also
-fall back to normal linking. See the
+fall back to normal linking.
+
+Cache-enabled invocations that cannot meet the supported path-independence contract run
+uncached and store nothing, rather than create path-specific entries. Missing/invalid root
+mappings, physical symbol paths, unsupported symbol formats, additional modules, and linked
+resources bypass caching. Path-shaped assembly references and external XML assembly names
+also bypass caching because they can escape the declared search-directory inventory.
+Portable PDB emission requires already-normalized input paths and
+explicit `PreserveSymbolPaths=true`. Neither symbol preservation nor deterministic compiler
+paths are enabled automatically. Copy actions also require shareable input paths.
+The SDK forwards `@(SourceRoot)`; runtime library targets supply the repository root without
+changing how inputs are compiled. Current runtime raw arguments still include unsupported
+options and bypass caching; structured runtime argument integration is separate. See the
 [cache design](../../design/tools/illink/task-cache.md) for identity and eligibility details.
 
 ### Cache maintenance
@@ -166,6 +190,24 @@ paths.
 
 A list of XML [descriptors](data-formats.md#descriptor-format) files specifying trimmer roots at a granular level.
 
+### SourceRoots
+
+Physical absolute roots with stable `MappedPath` metadata for experimental cache identity.
+For example:
+
+```xml
+<ItemGroup>
+  <LinkerSourceRoot Include="$(RepositoryRoot)" MappedPath="/_/" />
+</ItemGroup>
+```
+
+Pass `SourceRoots="@(LinkerSourceRoot)"` to the task. The current shared-cache scope accepts
+the conventional `/_/`, `/_1/`, etc. logical prefixes; nested mappings must preserve relative
+layout. Eligible invocations with one physical root execute from it using relative in-root
+paths. This parameter does not enable caching, change compiler settings, or remap embedded
+paths in linker outputs. Without usable mappings a cache-enabled invocation runs normally,
+uncached.
+
 ## ILLink Task Customization
 
 The trimmer can be invoked as an MSBuild task, `ILLink`. We recommend not using the task directly, because the SDK has built-in logic that handles computing the right set of reference assemblies as inputs, incremental trimming, and similar logic. If you would like to use the [advanced options](illink-options.md), you can invoke the msbuild task directly and pass any extra arguments like this: