Methods JITed during startup

Repro:

Build and run Console.WriteLine("Hello world".Replace('e', 'a'));
Collect list of JITed methods using e.g. perfview

Actual result: String.Replace is JITed in the foreground

Expected result: R2R version of String.Replace is, no JITing of String.Replace observed.

Could this be caused by the use of Vector<T> in string.Replace(char, char), and at runtime it's detected that AVX(2) is available, so the method gets JITed?

@brianrob

Cc @brianrob

crossgen with --singlemethodname Replace --singlemethodtypename System.String --singlemethodindex 3 --verbose:

Info: Method `string.Replace(char,char)` was not compiled because `[S.P.CoreLib]System.Numerics.Vector`1<uint16>` requires runtime JIT

--instruction-set:avx2 fixes it

Could this be caused by the use of Vector in string.Replace(char, char), and at runtime it's detected that AVX(2) is available, so the method gets JITed?

Yes, it is probably related. However, we should have enough flexibility in crossgen now to make it possible to precompile this successfully (as long as the app is running on current hardware at least).

Related: dotnet/designs#173 (and dotnet/core#7131) I assume we're going to default to Avx2 so it kinds of fixes itself

@trylek

@trylek @dotnet/crossgen-contrib @richlander . Yeah we have been discussing compiling with avx2 by default.

@EgorBo

As @EgorBo raises, it seems like the base question on why the R2R code cannot be used as-is is the basic question. If all non-AVX2 code is thrown out, then that R2R code is useless, right?

Ideally, the following is true:

All R2R code is used (not just in weird scenarios, but commonly).
R2R code is optimized for modern hardware.
R2R code includes high-value (currently) JIT-only optimizations, like guarded de-virtualization, at least for known high-value methods (call-sites or targets).
The JIT optimizes high-value methods at runtime (tiered comp, dynamic PGO, ...).

As we all know, we are not doing this today. For this conversation, it seems like the first topic is the concern.

@EgorBo

As @EgorBo raises, it seems like the base question on why the R2R code cannot be used as-is is the basic question. If all non-AVX2 code is thrown out, then that R2R code is useless, right?

Ideally, the following is true:

All R2R code is used (not just in weird scenarios, but commonly).

R2R code is optimized for modern hardware.

R2R code includes high-value (currently) JIT-only optimizations, like guarded de-virtualization, at least for known high-value methods (call-sites or targets).

The JIT optimizes high-value methods at runtime (tiered comp, dynamic PGO, ...).

As we all know, we are not doing this today. For this conversation, it seems like the first topic is the concern.

From my understanding the main problem that Vector<T> behaves differently in R2R'd code and Tier1 because it's basically of different widths for them, consider the following example:

// This method is Jitted (uses AVX2)
void Method1()
{
    Console.WriteLine(Method2(new Vector<long>(1)));
    // Expected: {2,2,2,2}
    // Actual: {2,2,1,1} or {2,2,0,0} or {2,2,garbage values}
}

// This method is R2R'd with SSE2
Vector<long> Method2(Vector<long> v)
{
    return v * 2;  // {v.long0 * 2, v.long1 * 2 }
}

So we don't allow this to happen and when we see Vector on a non-trimmed path and don't have AVX2 we fallback to JIT.
However it won't happen in e.g.

if (Sse2.IsSupported)
{
}
else if (Vector.IsHardwareAccelerated)
{
 Vector<T>
}

Right. That's my understanding, too. To my mind, our approach doesn't make sense. Why bother R2R-compiling methods that won't be used on most hardware? We should either compile those methods for the most common hardware (not SSE2) or not compile them at all.

non-trimmed path

trimmed? Do you mean tiered?

Neither am I, the root issue here can be fixed if we bump minimal baseline for R2R (dotnet/designs#173) do we plan to do it in 7.0?

@ivdiazsa

We're not making any more significant changes in 7.0, as you likely know.

We had a fair number of conversations about this topic over the last few months. Here's the basic context/problem:

AVX2 has a benefit at startup, but it isn't as high as one might expect (~2%). The real win is for steady-state, which is already solved via the JIT.
There is a win for AVX2 machines, however, the loss on non-AVX2 machines is expected to much greater due to unintuitive side-effects. In short:
- All SIMD R2R code is SSE2 today. There are a relatively small number of SIMD-dependent methods whose R2R methods need to be rejected on AVX2-capable machines. The reason is that we need APIs to always return the same answer, and we cannot guarantee this unless we use the same underlying functionality throughout process lifetime for the method implementation (the generated native code). This is particularly relevant for floating point math (per my understanding). This requires early jitting of high-quality code for those methods, which has both time and space costs.
- A key point (again per my understanding) is that most vector code doesn't have this characteristic, so most R2R vector code does not need to be rejected and can safely upgrade from SSE2 R2R code to AVX2 JIT code across process lifetime.
- If we consider the reverse (where all R2R code is compiled to AVX2), then all SIMD methods need to rejects on non-AVX2 machines and jitted at first use. That makes the cost disproportionate on non-AVX2 machines relatively to the gain on AVX2. Also, it's likely that AVX2 chips are better and more able to adapt.
- Last, a number of emulators (or maybe all of them) don't support AVX2. The same is true for the lowest end chips on sale for the cheapest laptops.
- Compiling for both SSE2 and AVX2 for the methods where it matters is (A) a lot of work since we don't have that capability to do (in either the R2R format or in crossgen), and (B) would bloat R2R images. Neither are attractive and don't feel worth it.
We have not done extensive measurements here, so a lot of this is conjecture. We should do more extensive measurements so that we can make better decisions. This change is not an "if" but "when". Actual measurements would help us choose the "when" with confidence.

Did I get that right? @ivdiazsa @mangod9 @davidwrighton @tannergooding

@gfoidl

Btw, String.Replace can be quickly fixed by adding Vector128 path (should also help with small inputs) cc @gfoidl

There are a relatively small number of SIMD-dependent methods whose R2R methods need to be rejected on AVX2-capable machines

IIRC, corelib is special and ignores and works under the presumption that we "do the right thing" and ensure all paths behave identically. While user code has the presumption that the two paths could differ in behavior and so will always throw out the R2R implementation if any path attempted to use an incompatible higher ISA. -- That is, SSE3, SSSE3, SSE4.1, and SSE4.2 are "encoding compatible with" SSE2 and so the if (Isa.IsSupported) checks get emitted in R2R as actual dynamic checks. AVX, AVX2, FMA, BMI1, BMI2, and others are "encoding incompatible" and so the if (Isa.IsSupported) checks will force R2R to throw out the code.

There are a few things to then keep in mind for corelib code:

SIMD is used in almost all our core code paths, often indirectly due to things like ROSpan<T> using it internally
Vector itself is always "encoding incompatible" and so any code path that uses Vector<T> will not have any R2R code
Vector128 is the "common" size that all platforms support (x86, x64, Arm64, etc.)
Vector128 will include WASM in the future, could include Arm32 if we added the support, etc
Vector256 is, today, only supported by x86/x64
Vector64 is, today, only supported by Arm64

x86-x64-v3 is the "level" that is the "baseline" for Azure, AWS, and more. It supports AVX/AVX2/BMI1/BMI2/FMA/LZCNT/MOVBE and has been around since ~2013 for Intel, ~2015 for AMD, and is reported by Steam Hardware Survey to be in 87.78% of reporting hardware.

x86-x64-v2 is the "level" that most emulators (including the x64 on Arm64 emulators provided by Microsoft/Apple) support. It supports SSE3/SSSE3/SSE4.1/SSE4.2/POPCNT/CX16 and has been around since ~2008 for Intel, ~2009 for AMD, and is reported by Steam Hardware Survey to be in 98.74% of reporting hardware.

x86-x64-v1 is the "level" we currently target. It supports SSE/SSE2/CMOV/CX8 and has been around since x64 was first introduced in 2003. For 32-bit it's been around since around 2000 and is reported by Steam Hardware Survey to be in 100% of reporting hardware.

The discussion around targeting AVX/AVX2 is really one around targeting x86-x64-v3. Since targeting it would only negatively impact startup for a likely minority of customers, I think its still worth considering making our baseline and if we, for whatever reason think that this is too much, the x86-x64-v2 is a reasonable alternative.

There is also a consideration that this impacts managed code but not native. Continuing to have the JIT, GC, and other native bits target a 20 year old baseline computer has some negative downsides with the main upside being that devs can "xcopy" their bits onto a flash drive and then run it on any computer.

Even Arm64 has the consideration that armv8.0-a is "missing" some fairly important instructions (like the atomics instruction set) and this is a major benefit that Apple M1 sees since it targets an armv8.5-a baseline instead.

I expect we could see some reasonable perf gains if we experimented with allowing a "less-portable" version of the JIT/GC that targeted a higher baseline (for both x64 and Arm64). I expect that this would be a net benefit for a majority of cases with it being "worse" only for a small subset of users with hardware that more than 15 years old (at which point, they likely aren't running a modern OS and likely have many other considerations around general app usage).

@gfoidl

Btw, String.Replace can be quickly fixed by adding Vector128 path (should also help with small inputs) cc @gfoidl

is this something which can be done in 7? Otherwise we could move this broader decision around avx2 to 8.

We'll take another run in .NET 8.

we have enabled avx2 for the new "composite" container images in 8. Is there any other specific work required here?

@mangod9

we have enabled avx2 for the new "composite" container images in 8. Is there any other specific work required here?

Just tried Console.WriteLine("Hello world".Replace('e', 'a')); on the latest daily SDK:

   1: JIT compiled System.Runtime.CompilerServices.CastHelpers:StelemRef(System.Array,long,System.Object) [Tier1, IL size=88, code size=93]
   2: JIT compiled System.Runtime.CompilerServices.CastHelpers:LdelemaRef(System.Array,long,ulong) [Tier1, IL size=44, code size=44]
   3: JIT compiled System.SpanHelpers:IndexOfNullCharacter(ulong) [Instrumented Tier0, IL size=805, code size=1160]
   4: JIT compiled System.Guid:TryFormatCore[ushort](System.Span`1[ushort],byref,int) [Tier0, IL size=894, code size=892]
   5: JIT compiled System.Guid:FormatGuidVector128Utf8(System.Guid,bool) [Tier0, IL size=331, code size=584]
   6: JIT compiled System.HexConverter:AsciiToHexVector128(System.Runtime.Intrinsics.Vector128`1[ubyte],System.Runtime.Intrinsics.Vector128`1[ubyte]) [Tier0, IL size=78, code size=359]
   7: JIT compiled System.Runtime.Intrinsics.Vector128:ShuffleUnsafe(System.Runtime.Intrinsics.Vector128`1[ubyte],System.Runtime.Intrinsics.Vector128`1[ubyte]) [Tier0, IL size=41, code size=50]
   8: JIT compiled System.Number:UInt32ToDecChars[ushort](ulong,uint) [Instrumented Tier0, IL size=114, code size=315]
   9: JIT compiled System.ArgumentOutOfRangeException:ThrowIfNegative[int](int,System.String) [Tier0, IL size=22, code size=50]
  10: JIT compiled System.Text.Unicode.Utf16Utility:GetPointerToFirstInvalidChar(ulong,int,byref,byref) [Instrumented Tier0, IL size=994, code size=1296]
  11: JIT compiled System.Text.Ascii:NarrowUtf16ToAscii(ulong,ulong,ulong) [Instrumented Tier0, IL size=491, code size=792]
  12: JIT compiled System.SpanHelpers:IndexOfNullByte(ulong) [Instrumented Tier0, IL size=844, code size=1540]
  13: JIT compiled System.PackedSpanHelpers:IndexOf[System.SpanHelpers+DontNegate`1[short]](byref,short,int) [Instrumented Tier0, IL size=698, code size=1911]
  14: JIT compiled System.PackedSpanHelpers:PackSources(System.Runtime.Intrinsics.Vector256`1[short],System.Runtime.Intrinsics.Vector256`1[short]) [Tier0, IL size=13, code size=52]
  15: JIT compiled System.PackedSpanHelpers:ComputeFirstIndexOverlapped(byref,byref,byref,System.Runtime.Intrinsics.Vector256`1[ubyte]) [Tier0, IL size=52, code size=124]
  16: JIT compiled System.PackedSpanHelpers:FixUpPackedVector256Result(System.Runtime.Intrinsics.Vector256`1[ubyte]) [Tier0, IL size=22, code size=42]
  17: JIT compiled System.SpanHelpers:LastIndexOfValueType[short,System.SpanHelpers+DontNegate`1[short]](byref,short,int) [Instrumented Tier0, IL size=963, code size=2111]
  18: JIT compiled System.Text.Ascii:WidenAsciiToUtf16(ulong,ulong,ulong) [Instrumented Tier0, IL size=604, code size=1784]
  19: JIT compiled (dynamicClass):InvokeStub_EventAttribute.set_Level(System.Object,System.Object,ulong) [FullOpts, IL size=25, code size=27]
  20: JIT compiled (dynamicClass):InvokeStub_EventAttribute.set_Message(System.Object,System.Object,ulong) [FullOpts, IL size=25, code size=28]
  21: JIT compiled (dynamicClass):InvokeStub_EventAttribute.set_Task(System.Object,System.Object,ulong) [FullOpts, IL size=25, code size=27]
  22: JIT compiled (dynamicClass):InvokeStub_EventAttribute.set_Opcode(System.Object,System.Object,ulong) [FullOpts, IL size=25, code size=27]
  23: JIT compiled (dynamicClass):InvokeStub_EventAttribute.set_Version(System.Object,System.Object,ulong) [FullOpts, IL size=25, code size=28]
  24: JIT compiled (dynamicClass):InvokeStub_EventAttribute.set_Keywords(System.Object,System.Object,ulong) [FullOpts, IL size=25, code size=28]
  25: JIT compiled System.Number:Int64ToHexChars[ushort](ulong,ulong,int,int) [Instrumented Tier0, IL size=67, code size=313]
  26: JIT compiled System.SpanHelpers:IndexOfAnyInRangeUnsignedNumber[ushort,System.SpanHelpers+DontNegate`1[ushort]](byref,ushort,ushort,int) [Tier0, IL size=142, code size=190]
  27: JIT compiled System.PackedSpanHelpers:IndexOfAnyInRange[System.SpanHelpers+DontNegate`1[short]](byref,short,short,int) [Instrumented Tier0, IL size=659, code size=1801]
  28: JIT compiled Program:<Main>$(System.String[]) [Tier0, IL size=20, code size=57]
  29: JIT compiled System.ArgumentOutOfRangeException:ThrowIfNegativeOrZero[int](int,System.String) [Tier0, IL size=36, code size=63]
Hallo world

I don't see string.Replace but I remember we used to have 4 or 5 functions jitted for a hello world 🤔 (tracked in #85791 so presumably @mangod9 this one can be closed)

#85791 is marked as Future. Surprised that System.ArgumentOutOfRangeException:ThrowIfNegativeOrZero is being JITted.

Surprised that System.ArgumentOutOfRangeException:ThrowIfNegativeOrZero is being JITted.

public static void ThrowIfNegativeOrZero<T>(T value, [CallerArgumentExpression(nameof(value))] string? paramName = null)
    where T : INumberBase<T>
{
    if (T.IsNegative(value) || T.IsZero(value))
        ThrowNegativeOrZero(paramName, value);
}

I guess it's the same SVM issue so presumably it will be fixed with #87438

hmm, makes sense

Navigation Menu

Repository navigation

Methods JITed during startup #67398

13 remaining items

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Methods JITed during startup #67398

Description

Activity

gfoidl commented on Mar 31, 2022

danmoseley commented on Mar 31, 2022

EgorBo commented on Mar 31, 2022

jkotas commented on Mar 31, 2022

EgorBo commented on Mar 31, 2022

mangod9 commented on Mar 31, 2022

richlander commented on Apr 2, 2022

EgorBo commented on Jun 26, 2022

richlander commented on Jun 26, 2022

13 remaining items

EgorBo commented on Jul 28, 2022

richlander commented on Jul 30, 2022

EgorBo commented on Jul 30, 2022

tannergooding commented on Aug 1, 2022

mangod9 commented on Aug 3, 2022

richlander commented on Aug 5, 2022

mangod9 commented on Jul 3, 2023

EgorBo commented on Jul 3, 2023

mangod9 commented on Jul 3, 2023

EgorBo commented on Jul 3, 2023

mangod9 commented on Jul 3, 2023

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions