C backend: a memory-bound loop's speed depends on unrelated code (flipping BLK_SHR in a program that never forks makes it 10-18% faster) #1648

The same program gets 10-18% faster when I flip BLK_SHR in its emitted C, although
it never forks (so the flag only changes which runtime code is compiled around the
loops). Same result, more instructions, fewer cycles: clang schedules the hot loop
differently.

matvec_schedule.bend multiplies a 50,257 x 768 matrix by a vector, 38 times, through
bend-ml-tensor-array@0.1.11.0 from BendHub (op 0: Mat.matmul_nt with one row, op 1:
Bands.matvec), on one thread pinned to a P-core (taskset -c 3). ms for the 38 products:

                    bend 2.0.37                     main e5527d5b (bun bend2/main.ts)
                    as built     BLK_SHR 1           as built     BLK_SHR 1
op 0, 3 rounds      751-762      677-680             774-804      661-667
op 1, 3 rounds      781-794      696-706             823-873      684-686

Every run prints checksum=90.09511.

perf stat, op 0, 2.0.37:
                    as built     BLK_SHR 1
instructions        19.25 G      20.79 G
cycles               4.13 G       3.58 G
LLC load misses      404 k        29.5 k

Both spend their time in spin_47 (the dot product, four partial sums). perf annotate,
the loads and the add of one step:

as built                                 BLK_SHR 1
movss (%r11,%rcx,4),%xmm3     4.77%      movss (%r10,%r15,4),%xmm2    10.08%
mulss (%rbx,%r10,4),%xmm3     3.24%      mulss (%r10,%r8,4),%xmm4     15.27%
addss %xmm6,%xmm3            41.75%      addss %xmm7,%xmm4             7.17%

As built, the loop waits on one addss right after its load. With the flag, both loads
index off one base register and the wait is spread. -falign-loops=64 does not change
it. A smaller Base-only program with the same loop shape shows no gap, so it depends
on the code around the loop.

I don't think this is a bug in Bend's semantics: it's clang's schedule of the C Bend
emits. But the emitted C decides it, so maybe the backend could steer it, e.g. by
emitting a loop's array accesses from one hoisted base pointer (the form the faster
build ended up with), so a memory-bound loop's speed stops depending on unrelated code.

Where it bit me: I compare every change in bend-ml and llama.bend against the previous
build. In a variant of this program the gap was 21%, and in llama.bend not keeping
the per-layer activations moved one-thread time by 4-6% with the hot loop's
instructions unchanged, only shifted 16 bytes (-falign-loops=64 closed most of that
one). Swings of this size are the size
of what I'm measuring, so I now check every small gap with counters and the hot loop's
disassembly before calling it a cost.