Skip to content

Prototype evaluation filesystem timestamp invalidation - #12

Draft
OvesN wants to merge 14 commits into
dev/veronikao/eval-observer-native-comparisonfrom
dev/veronikao/eval-observer-timestamp-invalidation
Draft

OvesN wants to merge 14 commits into
dev/veronikao/eval-observer-native-comparisonfrom
dev/veronikao/eval-observer-timestamp-invalidation

Conversation

@OvesN

@OvesN OvesN commented Aug 28, 2026

Copy link
Copy Markdown
Owner

Purpose

Stacked on #9 (dev/veronikao/eval-observer-native-comparison).

This PR adds an analysis-only filesystem timestamp invalidation prototype over the evaluation observations captured by #9. It asks whether validating the observed filesystem state is cheap enough to justify further persistent-cache work.

It does not add cache lookup, cache admission, serialization, persistence, evaluated-project materialization, concurrency, eviction, or result reuse.

Current mechanism

After a successful evaluation, the prototype builds a snapshot from observed:

  • project and imported sources;
  • file reads and typed existence probes;
  • consumed last-write timestamps;
  • complete directory enumerations;
  • directories traversed by globs;
  • ordered upward-search candidates and the selected result.

Each entry stores its canonical path, consumed timestamp, required file/directory existence predicates, and observation source. The snapshot also stores every path component checked for reparse points.

Capture fails closed for incomplete or conflicting observations, failed filesystem operations, unsupported providers or metadata/enumerations, missing traversal evidence, unstable project sources, non-canonical paths, and reparse-point traversal.

A reuse check first revalidates every stored path component for reparse points, then compares each stored timestamp and typed existence predicate. A mismatch returns Changed; an I/O or malformed-snapshot failure returns Failed. Only a complete match returns Valid.

This remains timestamp-based validation: timestamp-preserving content changes and the validation-to-materialization race are not solved. Environment variables, global properties, toolset selection, SDK resolution, Registry/process state, and other non-filesystem inputs require separate cache-key fields or versioned dependency contracts. SDK-resolver filesystem dependencies are intentionally deferred to the resolver contract.

Decision gate

The continuation target is complete unchanged validation below 10% of fresh evaluation time on real projects. Current-head performance must be measured with normal, non-reparse checkout paths; this description intentionally makes no stale performance claim. Missing the target means optimizing metadata probes before considering persistence.

Run the benchmark

Use disposable, restored checkouts in ordinary directories (not symlinks or junctions):

  • OrchardCore e3f8acb327a95f1dec6e75cefccaef2ad5eefb45
  • Roslyn 0f82fdec3c901702ec7fc3f0e9a813330a903ec9

From this PR checkout:

.\build.cmd -msbuildEngine dotnet -v quiet

$OrchardRoot = 'C:\src\OrchardCore'
$RoslynRoot = 'C:\src\roslyn'
$ResultsRoot = 'C:\benchmark-results\evaluation-timestamp'
$SdkVersion = (Get-Content .\global.json -Raw | ConvertFrom-Json).tools.dotnet

$env:MSBUILD_EVALUATION_TIMESTAMP_BENCHMARK_SDK_ROOT =
    (Resolve-Path ".\.dotnet\sdk\$SdkVersion").Path
$env:MSBUILD_EVALUATION_TIMESTAMP_BENCHMARK_PROJECTS = @(
    "$OrchardRoot\src\OrchardCore\OrchardCore\OrchardCore.csproj"
    "$OrchardRoot\src\OrchardCore.Cms.Web\OrchardCore.Cms.Web.csproj"
    "$RoslynRoot\src\Workspaces\Core\Portable\Microsoft.CodeAnalysis.Workspaces.csproj"
) -join [IO.Path]::PathSeparator
Remove-Item Env:\MSBUILD_EVALUATION_TIMESTAMP_BENCHMARK_MUTATIONS -ErrorAction SilentlyContinue

.\src\MSBuild.Benchmarks\Run-Benchmarks.ps1 `
    -Filter '*RealWorldEvaluationFilesystemTimestampBenchmark.*' `
    -Framework net11.0 `
    -LaunchCount 3 `
    -ArtifactsPath "$ResultsRoot\normal"

Open:

C:\benchmark-results\evaluation-timestamp\normal\net11.0\results\MSBuild.Benchmarks.RealWorldEvaluationFilesystemTimestampBenchmark-report-github.md

The decision row is ValidValidation; compare its mean with FreshEvaluation for each project. Full stale-mutation commands and interpretation are in documentation/specs/proposed/evaluation-filesystem-timestamp-invalidation-prototype-report.md.

@OvesN
OvesN force-pushed the dev/veronikao/eval-observer-native-comparison branch from 8216c14 to 0281b11 Compare September 1, 2026 20:03
@OvesN
OvesN force-pushed the dev/veronikao/eval-observer-timestamp-invalidation branch from 72b1ac1 to 1ffaeaf Compare September 2, 2026 01:40
OvesN and others added 4 commits September 2, 2026 05:06
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
@OvesN
OvesN force-pushed the dev/veronikao/eval-observer-timestamp-invalidation branch from 1ffaeaf to 5827473 Compare September 2, 2026 03:06
OvesN and others added 9 commits September 2, 2026 07:34
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Batch short benchmark operations, rely on natural glob timestamp changes, and report three-launch OrchardCore and Roslyn results against fresh reevaluation.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

# Conflicts:
#	src/Build.UnitTests/Definition/ProjectEvaluationContext_Tests.cs
Remove unreachable admission scaffolding, preserve filesystem provider provenance, tighten fail-closed validation, and align the real-world benchmark documentation.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant