JIT: gate inversion of bottom-tested no-IV loops on a hoisting benefit by AndyAyersMS · Pull Request #130368 · dotnet/runtime

AndyAyersMS · 2026-07-08T17:53:41Z

Behind DOTNET_JitLoopInversionRequireBenefitForBottomTested (off by default) for now.

When a loop is already bottom-tested (it has an exiting BBJ_COND latch that is not the recognized IV test) and AnalyzeIteration finds no IV, optTryInvertWhileLoop still inverts today, adding a duplicated zero-trip test plus a peeled iteration. Loop cloning can't follow up (it also needs the IV), so on arm64 this is often pure overhead — see #130045, the collateral from #129868.

This gates that case: only invert when the block we would duplicate (condBlock) carries a benefit worth bottom-testing — a call, or an indirection whose base is not assigned in the loop (a loop-invariant, hoistable load). The base-assignment check piggybacks on the existing size-limit walk, so there's no extra loop traversal.

The point of the gate is that the good and bad cases are otherwise structurally identical (multi-exit loop, exiting-cond latch, no recognized IV):

Dictionary.Enumerator.MoveNext — condition _index < _dictionary._count; _count is an invariant field load → kept.
BigIntegerCalculator.Compare — --i >= 0, pure arithmetic → skipped.
SpanHelpers.SequenceEqual / NonPackedContainsValueType — a[i], loop-variant → skipped.

SPMI (benchmarks.run, checked)

With the config enabled:

arm64: the [Perf] Linux/x64: 4 Regressions on 6/26/2026 6:28:17 PM +00:00 #130045 methods drop back to their pre-JIT: only skip loop inversion when the IV test is the bottom test #129868 size (e.g. Compare 172→116, SequenceEqual 320→292, NonPackedContainsValueType 200→168 / 456→424 / 432→404); ~197 other inversions are preserved that a straight revert would lose.
x64: Dictionary.Enumerator.MoveNext stays inverted (146 bytes, unchanged), so the JIT: only skip loop inversion when the IV test is the bottom test #129868 win is kept.
Replay is assert-clean (69,558 arm64 / 49,164 x64 contexts); with the config off, codegen is identical to main.

Local timing doesn't reproduce the regression on Apple Silicon, so posting EgorBot arm64 runs below for the key cases.

Note

This change and PR description were produced with GitHub Copilot CLI.

Under JitLoopInversionRequireBenefitForBottomTested (off by default), skip inverting an already bottom-tested loop with no recognized IV unless the duplicated condition holds a call or a loop-invariant, hoistable load. Targets the arm64 regressions in dotnet#130045 while keeping the Dictionary Enumerator.MoveNext inversion. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

dotnet-policy-service · 2026-07-08T17:54:50Z

Tagging subscribers to this area: @JulieLeeMSFT, @jakobbotsch
See info in area-owners.md if you want to be subscribed.

AndyAyersMS · 2026-07-08T17:56:32Z

@EgorBot -arm64 --envvars DOTNET_JitLoopInversionRequireBenefitForBottomTested:1

using System;
using System.Numerics;
using BenchmarkDotNet.Attributes;

public class Bench
{
    private readonly int[] _a = new int[512];
    private readonly int[] _b = new int[512];
    private BigInteger _x, _y;

    [GlobalSetup]
    public void Setup()
    {
        for (int i = 0; i < _a.Length; i++) { _a[i] = i; _b[i] = i; }
        byte[] bytes = new byte[259];
        new Random(42).NextBytes(bytes);
        bytes[^1] &= 0x7f;
        _x = new BigInteger(bytes);
        _y = new BigInteger(bytes);
    }

    [Benchmark]
    public bool Span_SequenceEqual() => _a.AsSpan().SequenceEqual(_b);

    [Benchmark]
    public int Span_IndexOf_NotFound() => _a.AsSpan().IndexOf(-1);

    [Benchmark]
    public int BigInteger_CompareTo() => _x.CompareTo(_y);

    [Benchmark]
    public bool BigInteger_Equals() => _x.Equals(_y);
}

Base is main (the env var names a config it doesn't have, so it runs the current #129868 codegen); the PR side has the gate enabled — so the diff isolates the fix on these #130045 cases.

Note

Comment generated with GitHub Copilot CLI.

Copilot

Pull request overview

This PR adds an (opt-in) heuristic gate to optTryInvertWhileLoop so that when a loop is already bottom-tested and AnalyzeIteration recognizes no IV, loop inversion is only performed if duplicating the condition block appears to have a “benefit” (call or potentially hoistable memory load). It also introduces a new JIT config knob to enable this gate.

Changes:

Track “exiting cond latch seen” separately from “IV-test latch” and only apply the new gate in the bottom-tested + no-IV case.
Classify condBlock for calls/indirections and use the existing loop size walk to conservatively detect stores to locals used in indirection address expressions.
Add JitLoopInversionRequireBenefitForBottomTested config (default off).

Reviewed changes

Copilot reviewed 2 out of 2 changed files in this pull request and generated 3 comments.

File	Description
`src/coreclr/jit/optimizer.cpp`	Adds the benefit-based gate and condition classification for bottom-tested no-IV loops.
`src/coreclr/jit/jitconfigvalues.h`	Introduces the new release config switch controlling the gate.

AndyAyersMS · 2026-07-09T01:37:16Z

+    bool         condHasCall     = false;
+    bool         condHasIndir    = false;
+    bool         condBaseStored  = false;
+    BitVecTraits condTraits(lvaCount, this);
+    BitVec       condIndirBaseLocals(BitVecOps::MakeEmpty(&condTraits));
+    if (checkBenefit)
+    {
+        assert(analyzedIteration);
+


Good catch — deferred the allocation: condIndirAddrLocals is now initialized to BitVecOps::UninitVal() and only MakeEmpty-d inside if (checkBenefit), so nothing is allocated when the config is off. (reply via Copilot CLI)

AndyAyersMS · 2026-07-09T01:37:17Z

+    // Classify condBlock here and record the locals used as bases of its indirections; the size-check
+    // walk below marks whether any such base is assigned in the loop (making the load loop-variant).


Fixed — reworded to "locals appearing in its indirection address expressions", and renamed condIndirBaseLocals→condIndirAddrLocals / m_baseLocals→m_addrLocals to match what CollectLocals actually does. (reply via Copilot CLI)

AndyAyersMS · 2026-07-09T01:37:18Z

+// When set, skip inverting a loop that is already bottom-tested (has an exiting BBJ_COND latch) and
+// has no recognized induction variable, unless the loop-continuation test that would be duplicated
+// contains a call or a loop-invariant (hence hoistable) memory load.


Softened — the comment now says "a memory load whose address has no loop-varying local" and drops the loop-invariant/hoistable claim. (reply via Copilot CLI)

AndyAyersMS · 2026-07-08T20:33:11Z

Re-running on server arm64 (Cobalt 100) — the earlier -arm64 landed on an Apple M4, which doesn't share the #130045 microarchitecture.

@EgorBot -linux_arm64 --envvars DOTNET_JitLoopInversionRequireBenefitForBottomTested:1

using System;
using System.Numerics;
using BenchmarkDotNet.Attributes;

public class Bench
{
    private readonly int[] _a = new int[512];
    private readonly int[] _b = new int[512];
    private BigInteger _x, _y;

    [GlobalSetup]
    public void Setup()
    {
        for (int i = 0; i < _a.Length; i++) { _a[i] = i; _b[i] = i; }
        byte[] bytes = new byte[259];
        new Random(42).NextBytes(bytes);
        bytes[^1] &= 0x7f;
        _x = new BigInteger(bytes);
        _y = new BigInteger(bytes);
    }

    [Benchmark]
    public bool Span_SequenceEqual() => _a.AsSpan().SequenceEqual(_b);

    [Benchmark]
    public int Span_IndexOf_NotFound() => _a.AsSpan().IndexOf(-1);

    [Benchmark]
    public int BigInteger_CompareTo() => _x.CompareTo(_y);

    [Benchmark]
    public bool BigInteger_Equals() => _x.Equals(_y);
}

Base is main (env var names a config it lacks, so it runs the current #129868 codegen); PR side has the gate enabled.

Note

Comment generated with GitHub Copilot CLI.

AndyAyersMS · 2026-07-08T21:11:20Z

Adding the two biggest #130045 regressions (LastIndexOfAnyExcept("ßäöüÄÖÜ"), ContainsKeyTrue<int,int>.IDictionary) plus two anchors, across server arm64 (Cobalt) and x64. Note: the fleet has no Windows Zen4, so Windows here is Turin (Zen5); Linux x64 is Genoa (Zen4).

@EgorBot -linux_arm64 -ubuntu24_azure_genoa -windows_x64 --envvars DOTNET_JitLoopInversionRequireBenefitForBottomTested:1

using System;
using System.Buffers;
using System.Collections.Generic;
using System.Linq;
using System.Numerics;
using System.Runtime.CompilerServices;
using BenchmarkDotNet.Attributes;

public class Bench
{
    private SearchValues<char> _searchValues;
    private char[] _textExcept;

    private int[] _found;
    private Dictionary<int, int> _dictionary;

    private readonly int[] _a = new int[512];
    private readonly int[] _b = new int[512];
    private BigInteger _x, _y;

    [GlobalSetup]
    public void Setup()
    {
        _searchValues = SearchValues.Create("ßäöüÄÖÜ");
        _textExcept = new string('ß', 256).ToCharArray();
        _textExcept[128] = '\n';

        _found = Enumerable.Range(0, 512).ToArray();
        _dictionary = _found.ToDictionary(k => k, k => k);

        for (int i = 0; i < _a.Length; i++) { _a[i] = i; _b[i] = i; }
        byte[] bytes = new byte[259];
        new Random(42).NextBytes(bytes);
        bytes[^1] &= 0x7f;
        _x = new BigInteger(bytes);
        _y = new BigInteger(bytes);
    }

    [Benchmark]
    public int LastIndexOfAnyExcept() => _textExcept.AsSpan().LastIndexOfAnyExcept(_searchValues);

    [Benchmark]
    public bool ContainsKeyTrue_IDictionary() => ContainsKey(_dictionary);

    [MethodImpl(MethodImplOptions.NoInlining)]
    private bool ContainsKey(IDictionary<int, int> collection)
    {
        bool result = false;
        var found = _found;
        for (int i = 0; i < found.Length; i++)
            result ^= collection.ContainsKey(found[i]);
        return result;
    }

    [Benchmark]
    public bool Span_SequenceEqual() => _a.AsSpan().SequenceEqual(_b);

    [Benchmark]
    public int BigInteger_CompareTo() => _x.CompareTo(_y);
}

Base is main (env var names a config it lacks, running the current #129868 codegen); PR side has the gate enabled.

Note

Comment generated with GitHub Copilot CLI.

AndyAyersMS · 2026-07-08T21:12:53Z

Also on Ampere (Neoverse-N1) — the core class behind the arm64-ubuntu lab queue, distinct from Cobalt's N2.

@EgorBot -ubuntu24_azure_ampere --envvars DOTNET_JitLoopInversionRequireBenefitForBottomTested:1

using System;
using System.Buffers;
using System.Collections.Generic;
using System.Linq;
using System.Numerics;
using System.Runtime.CompilerServices;
using BenchmarkDotNet.Attributes;

public class Bench
{
    private SearchValues<char> _searchValues;
    private char[] _textExcept;

    private int[] _found;
    private Dictionary<int, int> _dictionary;

    private readonly int[] _a = new int[512];
    private readonly int[] _b = new int[512];
    private BigInteger _x, _y;

    [GlobalSetup]
    public void Setup()
    {
        _searchValues = SearchValues.Create("ßäöüÄÖÜ");
        _textExcept = new string('ß', 256).ToCharArray();
        _textExcept[128] = '\n';

        _found = Enumerable.Range(0, 512).ToArray();
        _dictionary = _found.ToDictionary(k => k, k => k);

        for (int i = 0; i < _a.Length; i++) { _a[i] = i; _b[i] = i; }
        byte[] bytes = new byte[259];
        new Random(42).NextBytes(bytes);
        bytes[^1] &= 0x7f;
        _x = new BigInteger(bytes);
        _y = new BigInteger(bytes);
    }

    [Benchmark]
    public int LastIndexOfAnyExcept() => _textExcept.AsSpan().LastIndexOfAnyExcept(_searchValues);

    [Benchmark]
    public bool ContainsKeyTrue_IDictionary() => ContainsKey(_dictionary);

    [MethodImpl(MethodImplOptions.NoInlining)]
    private bool ContainsKey(IDictionary<int, int> collection)
    {
        bool result = false;
        var found = _found;
        for (int i = 0; i < found.Length; i++)
            result ^= collection.ContainsKey(found[i]);
        return result;
    }

    [Benchmark]
    public bool Span_SequenceEqual() => _a.AsSpan().SequenceEqual(_b);

    [Benchmark]
    public int BigInteger_CompareTo() => _x.CompareTo(_y);
}

Base is main (env var names a config it lacks, running the current #129868 codegen); PR side has the gate enabled.

Note

Comment generated with GitHub Copilot CLI.

AndyAyersMS · 2026-07-08T23:48:38Z

@EgorBot -ubuntu24_azure_ampere --envvars DOTNET_JitLoopInversionRequireBenefitForBottomTested:1 --filter "ContainsKeyTrueInt32IDictionary"

Real perf-repo benchmark this time (random ValuesGenerator keys). Base is main (env var names a config it lacks → current #129868 codegen); PR side has the gate enabled.

Note

Comment generated with GitHub Copilot CLI.

AndyAyersMS · 2026-07-09T00:46:03Z

Retry of the ContainsKeyTrue<Int32,Int32>.IDictionary filter run (prior attempt returned NA on the PR side with a bare BDN RuntimeError — no exception, and it reproduces neither in a local osx-arm64 run of the same benchmark under the config nor in SPMI replay, so it looks like a transient harness hiccup).

@EgorBot -ubuntu24_azure_ampere --envvars DOTNET_JitLoopInversionRequireBenefitForBottomTested:1 --filter "ContainsKeyTrueInt32IDictionary"

Base is main (env var names a config it lacks → current #129868 codegen); PR side has the gate enabled.

Note

Comment generated with GitHub Copilot CLI.

AndyAyersMS · 2026-07-09T01:31:34Z

EgorBot validation summary

Ran the #130045 cases across the hardware classes behind the perf lab, comparing main (current #129868 codegen) vs this PR with the gate enabled (--envvars DOTNET_JitLoopInversionRequireBenefitForBottomTested:1; main ignores the unknown config, so it serves as the regressed baseline).

Hardware → lab mapping: Ampere N1 = arm64-ubuntu, Cobalt N2 = arm64-azlinux, Genoa Zen4 = Linux/x64, Apple M4 = not a lab target.

The two biggest regressions recover to their lab baselines

Benchmark (hardware)	lab baseline → regressed	EgorBot: main → PR
`ContainsKeyTrue<int,int>.IDictionary` (Ampere N1)	3.36 µs → 4.85 µs	5.09 µs → 3.42 µs (−33%)
`SearchValuesCharTests.LastIndexOfAnyExcept` (Genoa Zen4)	87.8 ns → 131.9 ns	140.1 ns → 99.8 ns (−29%)

The PR numbers land right on the pre-#129868 lab baselines.

Full matrix (fix effect on the benchmark's Mean; negative = faster)

Benchmark	Apple M4	Ampere N1	Cobalt N2	Genoa Zen4
`BigInteger.CompareTo`	−44%	−49%	−36%	−21%
`Span<int>.SequenceEqual`	~0%	−15%	−6.6%	+4.4%
`LastIndexOfAnyExcept`	—	~0%	~0%	−29%
`ContainsKeyTrue.IDictionary`	—	−33%	~0%	~0%
`Span<int>.IndexOf` (not found)	+6%	—	+2%	—

Reading it

Every benchmark recovers on the hardware where it actually regressed: ContainsKeyTrue and SequenceEqual on Ampere N1, LastIndexOfAnyExcept on Zen4, BigInteger.CompareTo everywhere.
The give-back is small and localized to Span scans on cores where that inversion is genuinely beneficial (SequenceEqual +4% on Zen4, IndexOf +2–6% on M4/N2).
ContainsKeyTrue/LastIndexOfAnyExcept show ~0% on non-matching cores because they never regressed there (LastIndexOfAnyExcept was an x64-only regression; it improved on x86).

Notes: ContainsKeyTrue and LastIndexOfAnyExcept use --filter against the real perf-repo benchmarks; the rest are inline repros. One --filter run returned a transient NA on the PR side (no exception; not reproducible locally or in SPMI) and passed on retry.

Note

Comment generated with GitHub Copilot CLI.

Defer the benefit-gate bitvec allocation to when the config is on, correct the "indirection address locals" wording, and soften the config comment. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Copilot

Pull request overview

Copilot reviewed 2 out of 2 changed files in this pull request and generated 2 comments.

+            void CollectLocals(GenTree* tree)
+            {
+                if (tree->OperIsLocal())
+                {
+                    BitVecOps::AddElemD(m_traits, *m_addrLocals, tree->AsLclVarCommon()->GetLclNum());
+                }
+                tree->VisitOperands([&](GenTree* op) -> GenTree::VisitResult {
+                    CollectLocals(op);
+                    return GenTree::VisitResult::Continue;
+                });
+            }


+// When set, skip inverting a loop that is already bottom-tested (has an exiting BBJ_COND latch) and
+// has no recognized induction variable, unless the loop-continuation test that would be duplicated
+// contains a call or a memory load whose address has no loop-varying local.


Copilot AI review requested due to automatic review settings July 8, 2026 17:53

github-actions Bot added the area-CodeGen-coreclr CLR JIT compiler in src/coreclr/src/jit and related components such as SuperPMI label Jul 8, 2026

Copilot started reviewing on behalf of AndyAyersMS July 8, 2026 17:54 View session

dotnet-policy-service Bot assigned AndyAyersMS Jul 8, 2026

EgorBot mentioned this pull request Jul 8, 2026

Benchmarks for dotnet/runtime#130368 (for @AndyAyersMS) EgorBot/Benchmarks#294

Open

Copilot AI reviewed Jul 8, 2026

View reviewed changes

EgorBot mentioned this pull request Jul 8, 2026

Benchmarks for dotnet/runtime#130368 (for @AndyAyersMS) EgorBot/Benchmarks#295

Open

EgorBot mentioned this pull request Jul 8, 2026

Benchmarks for dotnet/runtime#130368 (for @AndyAyersMS) EgorBot/Benchmarks#296

Open

EgorBot mentioned this pull request Jul 8, 2026

Benchmarks for dotnet/runtime#130368 (for @AndyAyersMS) EgorBot/Benchmarks#297

Open

EgorBot mentioned this pull request Jul 8, 2026

Benchmarks for dotnet/runtime#130368 (for @AndyAyersMS) EgorBot/Benchmarks#298

Open

EgorBot mentioned this pull request Jul 9, 2026

Benchmarks for dotnet/runtime#130368 (for @AndyAyersMS) EgorBot/Benchmarks#299

Open

build-analysis Bot mentioned this pull request Jul 9, 2026

The Operation will be canceled. The next steps may not contain expected logs. dotnet/dnceng#3008

Open

3 tasks

JIT: address review feedback; clang-format

b016750

Defer the benefit-gate bitvec allocation to when the config is on, correct the "indirection address locals" wording, and soften the config comment. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Copilot AI review requested due to automatic review settings July 9, 2026 01:39

Copilot started reviewing on behalf of AndyAyersMS July 9, 2026 01:40 View session

Copilot AI reviewed Jul 9, 2026

View reviewed changes

		// Classify condBlock here and record the locals used as bases of its indirections; the size-check
		// walk below marks whether any such base is assigned in the loop (making the load loop-variant).

Uh oh!

Conversation

AndyAyersMS commented Jul 8, 2026

SPMI (benchmarks.run, checked)

Uh oh!

dotnet-policy-service Bot commented Jul 8, 2026

Uh oh!

AndyAyersMS commented Jul 8, 2026

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

AndyAyersMS Jul 9, 2026

Choose a reason for hiding this comment

Uh oh!

AndyAyersMS Jul 9, 2026

Choose a reason for hiding this comment

Uh oh!

AndyAyersMS Jul 9, 2026

Choose a reason for hiding this comment

Uh oh!

AndyAyersMS commented Jul 8, 2026

Uh oh!

AndyAyersMS commented Jul 8, 2026

Uh oh!

AndyAyersMS commented Jul 8, 2026

Uh oh!

AndyAyersMS commented Jul 8, 2026

Uh oh!

AndyAyersMS commented Jul 9, 2026

Uh oh!

AndyAyersMS commented Jul 9, 2026

EgorBot validation summary

The two biggest regressions recover to their lab baselines

Full matrix (fix effect on the benchmark's Mean; negative = faster)

Reading it

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants