Universal x64 JIT for crossgen2/ILC — Strategy

Goal

Replace the two cross-targeting x64 JITs shipped with crossgen2/ILC
(clrjit_unix_x64_<host> and clrjit_win_x64_<host>) with a single
clrjit_universal_x64_<host> that decides OS-dependent behavior at runtime.
The in-proc clrjit (runtime) build must stay compile-time specialized (no
runtime OS branches), exactly like arm/arm64 today.

How arm/arm64 already do this (the template)

• src/coreclr/inc/targetosarch.h: TargetOS::IsWindows/IsUnix/IsApplePlatform
  are compile-time const bool when TARGET_WINDOWS/TARGET_UNIX is defined,
  but become runtime static bool when neither is defined
  (TARGET_OS_RUNTIMEDETERMINED). The "universal" cmake OS sets neither.
• jit/target.h #undefs TARGET_UNIX/WINDOWS/OSX/APPLE so JIT code cannot use
  them directly; it must use TargetOS::*.
• CILJit::setTargetOS (ee_il_dll.cpp) sets the runtime values; VM/crossgen call
  it before compiling.
• ABI knobs that vary (arm hardfp/softfp) use CONFIGURABLE_ARM_ABI:
  a compile-time const bool compFeatureHfa becomes a runtime variable.
• Runtime-variable register masks already exist for APX/EVEX: rbmAllInt,
  rbmIntCalleeTrash, rbmAllFloat, ... are instance fields set in
  compiler.cpp init from RBM_*_INIT + runtime capability checks.
  RBM_ALLINT/etc. are macros that call get_RBM_*().

The x64-specific gap

TargetOS is already runtime-capable; the problem is x64 ABI is selected by the
compile-time macro UNIX_AMD64_ABI (and its negation WINDOWS_AMD64_ABI),
which drives, in targetamd64.h and jit.h:

• Register sets / masks: RBM_INT_CALLEE_SAVED, RBM_INT_CALLEE_TRASH_INIT,
  RBM_FLT_CALLEE_SAVED, RBM_FLT_CALLEE_TRASH_INIT, arg regs (REG_ARG_0…),
  REG_VAR_ORDER_*, callee-save counts CNT_CALLEE_*, LAST_FP_ARGREG.
• Feature constants: FEATURE_MULTIREG_ARGS/RET/… (0 on win, 1 on unix),
  MULTIREG_HAS_SECOND_GC_RET, FEATURE_PARTIAL_SIMD_CALLEE_SAVE,
  FEATURE_IMPLICIT_BYREFS sizing, MAX_*_MULTIREG_BYTES, MAX_RET_REG_COUNT,
  ETW_EBP_FRAMED, MIN_ARG_AREA_FOR_CALL, profiler arg regs.
• Struct classification (SysV eightbyte classifier) — Unix-only code.

Scale (jit/ tree): UNIX_AMD64_ABI 184 refs / 81 #if sites; WINDOWS_AMD64_ABI 28;
FEATURE_MULTIREG_RET 78; FEATURE_PARTIAL_SIMD_CALLEE_SAVE 75;
MULTIREG_HAS_SECOND_GC_RET 29; ETW_EBP_FRAMED 29.

The hard part: several of these feed compile-time-constant array sizes and enum
values (e.g. MAX_RET_REG_COUNT, MULTIREG_HAS_SECOND_GC_RET), so they can't
become plain runtime variables without either (a) sizing arrays for the max
(union of both ABIs) or (b) keeping the constant at the max and bounding at
runtime.

Proposed strategy (staged)

Stage 0 — Build scaffolding (no behavior change)

• Add CONFIGURABLE_AMD64_ABI compile define, set only on a new
  clrjit_universal_x64_<host> target in jit/CMakeLists.txt.
• Add cmake OS universal+x64 handling in clrdefinitions.cmake
  (set_target_definitions_to_custom_os_and_arch): define TARGET_AMD64/64BIT but
  neither TARGET_UNIX nor TARGET_WINDOWS nor UNIX_AMD64_ABI → forces
  TARGET_OS_RUNTIMEDETERMINED and the configurable ABI path.
• Keep the existing clrjit_unix_x64 / clrjit_win_x64 targets building until
  the universal one is proven, then delete them and flip
  JitConfigProvider.GetTargetSpec to emit universal_x64.

Stage 1 — Make ABI a runtime property

• Introduce a single runtime predicate compUnixAmd64Abi() /
  Compiler-level (or global, since immutable per-process) boolean initialized
  from TargetOS::IsUnix in setTargetOS, mirroring CONFIGURABLE_ARM_ABI's
  compFeatureHfa. Under the non-configurable native build it is a
  const bool derived from UNIX_AMD64_ABI.

Stage 2 — Convert register/mask definitions to runtime

• Extend the existing rbm*/cnt* init block in compiler.cpp to select the
  Unix vs Windows base sets by compUnixAmd64Abi() (union feeds APX/EVEX ORs as
  today). Convert RBM_INT_CALLEE_SAVED, RBM_FLT_CALLEE_SAVED,
  RBM_FLT_CALLEE_TRASH_INIT, REG_ARG_*, REG_VAR_ORDER_*, CNT_CALLEE_*,
  LAST_FP_ARGREG to get_*() accessors backed by instance fields (native build
  keeps them as const/#define via the non-configurable branch).

Does this ripple into other architectures? (No, with one bounded caveat.)

• RBM masks are defined per-arch in targetXXX.h; shared code (lsra,
  compiler, codegencommon) only references the macro name. Whether
  RBM_INT_CALLEE_SAVED expands to a constant or get_RBM_INT_CALLEE_SAVED() is
  transparent to consumers, so converting x64 is done entirely in
  targetamd64.h + compiler.{h,cpp}. arm64/arm/x86/loong/riscv headers are
  untouched and keep plain-constant #defines.
• Proven precedent: targetamd64.h already defines RBM_ALLINT,
  RBM_ALLFLOAT, RBM_INT_CALLEE_TRASH, RBM_FLT_CALLEE_TRASH as get_*()
  accessors (APX/EVEX) while every other arch keeps them constant — that
  conversion touched zero other architectures. This work converts the remaining
  x64 masks (RBM_INT_CALLEE_SAVED, RBM_FLT_CALLEE_SAVED, arg/order/count) the
  same way.
• Caveat: get_*() is a Compiler member and cannot be used in a compile-time
  constant context (file-scope static const, constexpr, array size). If a
  shared consumer uses a newly-runtime mask that way, that site is reworked
  to compute at runtime. This edits shared code but stays behavior-identical and
  constant-folds on other arches (their macros remain constants) — no
  retargeting or per-arch behavior change. The APX/EVEX conversion already flushed
  out these sites for RBM_ALLINT/RBM_INT_CALLEE_TRASH, so the remaining set is
  bounded and analogous.

Stage 3 — Convert feature flags & sizing

• For booleans (FEATURE_MULTIREG_*, MULTIREG_HAS_SECOND_GC_RET,
  FEATURE_PARTIAL_SIMD_CALLEE_SAVE, ETW_EBP_FRAMED,
  FEATURE_MULTIREG_STRUCT_PROMOTE): under CONFIGURABLE_AMD64_ABI make them
  runtime predicates; native build keeps #define constants.
• For array-sizing constants (MAX_RET_REG_COUNT, MAX_*_MULTIREG_BYTES,
  MAX_ARG_REG_COUNT, MAX_MULTIREG_COUNT): in the configurable build fix them
  to the union max (Unix values) and rely on runtime feature predicates to
  gate behavior; verify no correctness dependence on the smaller Windows values.

Stage 4 — Guard remaining #ifdef UNIX_AMD64_ABI code

• Convert each of the 81 #if sites: pure-codegen branches → if (compUnixAmd64Abi()); Unix-only helpers (SysV classifier) compiled
  unconditionally in the configurable build and called only on the Unix path.
• Windows-only (varargs shadow, WINDOWS_AMD64_ABI) likewise runtime-gated.

Stage 5 — Validation

• Diff crossgen2 output (R2R images) for a representative assembly built with the
  old per-OS JITs vs the universal JIT for both target OSes — require identical
  codegen (SuperPMI replay is the ideal gate: collect on both OS targets, replay
  with the universal JIT, expect zero diffs).
• Build both the native clrjit (compile-time) and clrjit_universal_x64
  (runtime) to confirm the dual-mode macros compile both ways.

Key files

• src/coreclr/jit/CMakeLists.txt (targets, defines)
• src/coreclr/clrdefinitions.cmake (set_target_definitions_to_custom_os_and_arch)
• src/coreclr/inc/targetosarch.h (already runtime-ready)
• src/coreclr/jit/target.h, targetamd64.h, targetamd64.cpp, jit.h
• src/coreclr/jit/compiler.cpp/.h (rbm/cnt init + accessors)
• src/coreclr/jit/ee_il_dll.cpp (setTargetOS)
• src/coreclr/tools/Common/JitInterface/JitConfigProvider.cs (GetTargetSpec)

GCInfo encoder (linked into the JIT binary)

The JIT links a per-(os,arch) gcinfo static lib: gcinfo_unix_x64 /
gcinfo_win_x64 (arm64 uses gcinfo_universal_arm64). Built by
create_gcinfo_lib in src/coreclr/gcinfo/CMakeLists.txt, which calls
set_target_definitions_to_custom_os_and_arch — so the unix lib is compiled with
TARGET_UNIX+UNIX_AMD64_ABI, the win lib with TARGET_WINDOWS.

Findings from auditing the encoder for x64 OS-dependence

• inc/gcinfotypes.h AMD64GcInfoEncoding: all constants (normalization,
  ENCBASE, HAS_FIXED_STACK_PARAMETER_SCRATCH_AREA, stack-base-reg encode) are
  OS-independent — identical for Unix and Windows x64.
• gcinfo/gcinfoencoder.cpp, arraylist.cpp, simplerhash.cpp,
  inc/gcinfoencoder.h, inc/gcinfo.h, gcinfohelpers.h: zero
  UNIX_AMD64_ABI/WINDOWS_AMD64_ABI/TARGET_UNIX/TARGET_WINDOWS uses. The
  only OS branch is one runtime check (TargetOS::IsApplePlatform && TargetArchitecture::IsArm64) at gcinfoencoder.cpp:573 — already runtime,
  arm64-only, compile-time-false for x64.
• JIT-side GC logic (jit/gcencode.cpp, jit/gcinfo.cpp, jit/jitgcinfo.h):
  zero OS/ABI conditionals; register numbers/EnC area are passed through from
  JIT computations (which live in the JIT proper, covered by the main strategy).

Consequence / required work

The x64 gcinfo encoder is effectively already OS-agnostic — the two libs
differ only by target #defines the gcinfo code never consumes, so they are
functionally identical object code. Unlike the JIT proper, there is little/no
#ifdef conversion to do here. The concrete work is:

1. Build wiring: add create_gcinfo_lib(TARGET gcinfo_universal_x64 OS universal ARCH x64) and link it into clrjit_universal_x64 (the
   unix/win x64 gcinfo libs can be retired with the corresponding JITs).
   OS universal ARCH x64 correctly yields TARGET_AMD64/64BIT with neither
   TARGET_UNIX nor TARGET_WINDOWS ⇒ targetosarch.h runtime-determined TargetOS,
   and no UNIX_AMD64_ABI.
2. Guardrail: enforce that the encoder never introduces compile-time OS
   macros for x64 — any future OS-dependent behavior must use runtime
   TargetOS::* (as the arm64-apple assert already does), so the single
   universal lib stays correct for both target OSes.
3. Validation: binary-diff gcinfo_unix_x64/gcinfo_win_x64 today to
   confirm they are identical (sanity), then confirm gcinfo_universal_x64
   links and R2R output is bit-identical vs the per-OS libs via SuperPMI replay.

Net: the gcinfo effort is small relative to the JIT proper, but it IS required —
we must produce and link a runtime-determined (universal) gcinfo build rather
than a per-OS one, exactly the same "swap to runtime TargetOS" model, and keep
the encoder free of compile-time x64 OS macros going forward.

Risk / cost

Large: ~180 macro refs, register enums, array sizing. Must not regress native
JIT codegen or perf. Best landed incrementally behind the existing per-OS JITs.