[RyuJIT/arm32][GCStress=C] jaggedarr_cs_do.cmd failure

[BruceForstall]

https://ci.dot.net/job/dotnet_coreclr/job/master/job/jitstress/job/arm_cross_checked_windows_nt_gcstress0xc_tst/18/artifact/bin/tests/Windows_NT.arm.Checked/Smarty.run.0/Smarty.0.fail.smrt/*view*/
https://ci.dot.net/job/dotnet_coreclr/job/master/job/jitstress/job/arm_cross_checked_windows_nt_gcstress0xc_jitstress1_tst/16/artifact/bin/tests/Windows_NT.arm.Checked/Smarty.run.0/Smarty.0.fail.smrt/*view*/

This failure is fairly consistent. I can reproduce it locally most of the time. Note that the test is seeded with a random number, which doesn't help reproducibility.

[BruceForstall]

The failure is in ProcessJagged3DArray().

The issue appears to be that the JIT generates an illegal byref -- a byref pointer that does not point anywhere within the associated object. Thus, when a GC happens and the base object is moved, that byref is not updated, leaving it pointing to garbage.

We have an array object -- a gcref -- and need to generate an address expression for an element in the array, something like:

  array object + index * element size + 8

where 8 is the offset to the first array element: offset zero is the MethodTable pointer and offset 4 is the array element count.

ARM doesn't have an addressing mode that can express this in one instruction (like an x86 LEA), so we generate two instructions, essentially:

// r0 = array object, r6 = computed index offset. We mark r4 as a byref.
add r4, r0, r6

// r4 - 32 is the offset of the object we care about. Then we load the array element.
// In this case, the loaded element is a gcref, so r4 becomes a gcref.
ldr r4, [r4-32]

We get this math because the user code uses "a[i - 10]", which is essentially "a + (i - 10) * 4 + 8" for element size 4, or "a + i * 4 - 40 + 8", or "a + i * 4 - 32". In the above code, r6 is i * 4.

The problem here, is that add r4,r0,r6, or a + i * 4, which we report as a byref, can point well beyond the end of the array.

Note that for this hole to occur, we need an indexing expression like this that allows a byref to point beyond the end of an array. An expression such as "a[i - X]" might be required. Then, for gcstress to catch it, we need the first occurrence of indexing to point beyond the end of the array, since we only force a GC the first time an instruction is encountered. After that, we replace the original instruction and run "normally". The function (or region) also needs to be fully interruptible.