[Performance] Streaming orchestrator design space (a2a3/a5/Host/HW)

[hw-native-sys-bot]

Platform

a2a3 (Ascend 910B/C hardware)

Runtime Variant

tensormap_and_ringbuffer

Summary

Tracking issue for the a2a3 tensormap_and_ringbuffer orchestrator. Captures a PTO2_ORCH_PROFILING=1 measurement baseline of the per-task submit hot path, side-by-side numbers for the auto-dependency (paged_attention_unroll) and manual-dependency (paged_attention_unroll_manual_scope) variants on the same workload, and the architectural background needed to interpret them.

The intent of this issue is characterization, not regression reporting. Concrete evolution directions are posted as separate comments.

Related: #545 (overall runtime perf tracking — scheduler / dispatch / runtime-wrapping side), #849, #902.

Git Commit ID

60742ff

CANN Version

9.0.0 (V100R001C10SPC001B250)

Driver Version

26.0.rc1 (ascendhal_version 7.35.23)

Host Platform

Linux (aarch64)

Reproduction

# 1. Enable orchestrator + tensormap profiling
#    (src/a2a3/runtime/tensormap_and_ringbuffer/runtime/pto_runtime2_types.h)
#      #define PTO2_ORCH_PROFILING        1
#      #define PTO2_TENSORMAP_PROFILING   1
# 2. Rebuild the a2a3 onboard runtime
python3 -m venv --system-site-packages .venv && source .venv/bin/activate
pip install --no-build-isolation -e .
python simpler_setup/build_runtimes.py --platforms a2a3

# 3. Run the auto-dep variant on a free a2a3 device
task-submit --device auto --device-num 1 --run "source .venv/bin/activate && \
    python -m pytest \
        tests/st/a2a3/tensormap_and_ringbuffer/paged_attention_unroll/test_paged_attention_unroll.py \
        --platform a2a3 --device \$TASK_DEVICE -v -s --log-level v9 --enable-l2-swimlane 4"
# Inspect device log for the '=== Orchestrator Profiling' block:
ls -lt $HOME/ascend/log/debug/device-<id>/ | head -3

# 4. Repeat with the manual-scope variant
task-submit --device auto --device-num 1 --run "source .venv/bin/activate && \
    python -m pytest \
        examples/a2a3/tensormap_and_ringbuffer/paged_attention_unroll_manual_scope/test_paged_attention_unroll.py \
        --platform a2a3 --device \$TASK_DEVICE -v -s --log-level v9 --enable-l2-swimlane 4"

Both tests use the same Case1 parameters (batch=256, num_heads=16, kv_head_num=1, head_dim=128, block_size=128, context_len=8192, max_model_len=32768, dtype=bfloat16, aicpu_thread_num=4, block_dim=24).

Additional Context

1. What the orchestrator does

Single AICPU thread (Thread 3 of 4 on a2a3). Per runtime/pto_orchestrator.h:11-26: executes the user orchestration function, allocates intermediate buffers from the GM heap (heap ring), submits tasks via PTO2OrchestratorState::submit_task (real work), submit_dummy_task (barrier), or alloc_tensors (inline-complete buffer carve-out), builds the dependency graph through TensorMap or explicit_deps, and manages scopes via PTO2_SCOPE. It is not an executor — it writes records into shared memory and pushes opaque slot-state pointers onto a single SPSC wiring queue; the scheduler threads do all fanout wiring, list maintenance, and watermark advancement.

2. Two components: runtime interface and user code

The runtime is split into a function-pointer ops table and the user .so that calls it through that table. There is no link dependency: AICPU Thread 3 dumps the orchestration .so to a temp file, dlopens it, looks up aicpu_orchestration_config and aicpu_orchestration_entry, the runtime populates rt->ops, and the user code calls back via current_runtime()->ops->submit_task (etc.).

• Runtime interface: src/a2a3/runtime/tensormap_and_ringbuffer/orchestration/pto_orchestration_api.h — opaque struct PTO2Runtime { const PTO2RuntimeOps *ops; PTO2ScopeMode pending_scope_mode; }, the PTO2RuntimeOps function table (submit_task, scope_begin/end, alloc_tensors, submit_dummy_task, logging, set_tensor_data/get_tensor_data, fatal handling), inline rt_submit_*_task wrappers, the Arg builder, and PTO2_SCOPE(...) RAII.
• User code: examples/.../paged_attention_orch.cpp exports exactly aicpu_orchestration_entry (and _config). Includes only pto_orchestration_api.h. Zero runtime symbol dependencies.

3. Six phases of one task submission

All happen inside submit_task_common (runtime/pto_orchestrator.cpp:507-688), instrumented by CYCLE_COUNT_LAP calls:

1. Alloc (g_orch_alloc_cycle) — task ring slot allocation, heap ring packed-output allocation, prefetch_payload, bind_buffers, task_state.store(PENDING), scope_tasks_push. Blocks on back-pressure here.
2. sync_tensormap (g_orch_sync_cycle) — acquire-load fc.last_task_alive, possibly cleanup_retired every PTO2_TENSORMAP_CLEANUP_INTERVAL=64 retired tasks.
3. lookup+dep (g_orch_lookup_cycle) — explicit_deps loop (retire-skip + fanin append) and compute_task_fanin (pto_dep_compute.h:81-129): per non-OUTPUT tensor, Step A creator retention from tensor->owner_task_id, then Step B tensor_map.lookup — hash-bucket walk + check_overlap cascade (L1 byte-range, L2 hyper-rectangle, L3 conservative-OTHER). INOUT+COVERED removes the entry. Inline fanin builder caps at PTO2_FANIN_INLINE_CAP, spills to fanin_pool above that.
4. tensormap_ins (g_orch_insert_cycle) — register_task_outputs (pto_dep_compute.h:140-154): for each INOUT and OUTPUT_EXISTING, allocate entry from free list or bump region, 64-byte copy_from_tensor memcpy, hash-bucket and per-task chain prepends.
5. param_copy (g_orch_args_cycle) — task.task_id / kernel_id[3] / packed_buffer_* writes, per-producer fanout_count++, fanin metadata into payload, and payload.init(args, result, alloc_result, layout) — the actual tensor + scalar deep copy into the GM payload that AICore will read.
6. fanin+ready (g_orch_fanin_cycle) — single SPSC push of the slot-state pointer to sched->wiring.queue. The actual fanout wiring (lock + dep_pool prepend + early-finished check + ready-queue push) is deferred to scheduler thread 0's drain_wiring_queue (scheduler/pto_scheduler.h:671-711).

scope_end cost is tracked separately (g_orch_scope_end_cycle) — iterates scope tasks and calls scheduler->on_scope_end to release the +1 scope reference on each producer's fanout_count.

Raw device-log excerpts on the 1280-task Case1 workload (verbatim, paths anonymized):

# paged_attention_unroll (auto-dep, PTO2_SCOPE())
Thread 3: === Orchestrator Profiling: 1280 tasks, total=646.960us ===
Thread 3:   task+heap_alloc: 171.280us (26.5%)  work=171.280us wait=0.000us  atomics=1280
Thread 3:   sync_tensormap : 56.400us (8.7%)
Thread 3:   lookup+dep     : 80.720us (12.5%)
Thread 3:   tensormap_ins  : 70.520us (10.9%)
Thread 3:   param_copy     : 235.600us (36.4%)  atomics=2048
Thread 3:   fanin+ready    : 32.440us (5.0%)  work=32.440us wait=0.000us
Thread 3:   avg/task       : 0.505us
Thread 3: === TensorMap Lookup Stats ===
Thread 3:   lookups        : 3584, inserts: 768
Thread 3:   chain walked   : total=11, avg=0.0, max=1
Thread 3:   overlap checks : 0, hits=0 (0.0%)
Thread 3: PTO2 total submitted tasks = 1280, already executed 1256 tasks

# paged_attention_unroll_manual_scope (manual deps, PTO2_SCOPE(PTO2ScopeMode::MANUAL))
Thread 3: === Orchestrator Profiling: 1280 tasks, total=486.820us ===
Thread 3:   task+heap_alloc: 161.580us (33.2%)  work=161.580us wait=0.000us  atomics=1280
Thread 3:   sync_tensormap : 33.080us (6.8%)
Thread 3:   lookup+dep     : 19.360us (4.0%)
Thread 3:   tensormap_ins  : 3.520us (0.7%)
Thread 3:   param_copy     : 252.120us (51.8%)  atomics=2048
Thread 3:   fanin+ready    : 17.160us (3.5%)  work=17.160us wait=0.000us
Thread 3:   avg/task       : 0.380us
Thread 3: === TensorMap Lookup Stats ===
Thread 3:   lookups        : 256, inserts: 0
Thread 3:   chain walked   : total=0, avg=0.0, max=0
Thread 3:   overlap checks : 0, hits=0 (0.0%)
Thread 3: PTO2 total submitted tasks = 1280, already executed 1051 tasks

4. TensorMap cost and sequentiality

The orchestrator is one thread, so steps 2/3/4 run strictly in order with no pipelining: N tasks back-to-back cost N×(sync + lookup + insert). The TensorMap is private to the orchestrator (no atomics), so there's no contention — but also no parallelism. Auto-dep mode runs compute_task_fanin and register_task_outputs; manual-dep mode short-circuits both (pto_dep_compute.h:84-86 and :142-144). That's why the manual run shows inserts=0 and lookups=256 (only one trivial lookup per submit, no chain walked, no overlap check).

Side-by-side per-phase breakdown on the same 1280-task Case1 workload, same commit, both runs on a free a2a3 device under task-submit exclusive lock:

Phase	unroll (auto-dep)	unroll_manual_scope	Δ
total (sum of phases)	646.96 us	486.82 us	-24.8%
avg/task	0.505 us	0.380 us	-24.8%
task+heap_alloc	171.28 us (26.5%)	161.58 us (33.2%)	-5.7%
sync_tensormap	56.40 us (8.7%)	33.08 us (6.8%)	-41.3%
lookup+dep	80.72 us (12.5%)	19.36 us (4.0%)	-76.0%
tensormap_ins	70.52 us (10.9%)	3.52 us (0.7%)	-95.0%
param_copy	235.60 us (36.4%)	252.12 us (51.8%)	+7.0%
fanin+ready	32.44 us (5.0%)	17.16 us (3.5%)	-47.1%

TensorMap-internal stats:

Stat	unroll (auto-dep)	unroll_manual_scope
lookups	3584	256
inserts	768	0
chain walked total / avg / max	11 / 0.0 / 1	0 / 0.0 / 0
overlap checks / hits	0 / 0	0 / 0

Observations:

• param_copy stays the largest single phase in both runs (36-52%) — the unavoidable Tensor + scalar deep copy into the GM payload, tensormap-independent.
• Eliminating TensorMap collapses two phases (lookup+dep + tensormap_ins) from a combined 151.24 us (23.4%) to 22.88 us (4.7%), a 128.36 us / 19% absolute drop on total.
• sync_tensormap does not go to zero in manual mode because it is unconditional (pto_orchestrator.cpp:574-578) — it still loads last_task_alive and may run cleanup_retired even with zero inserts.
• fanin+ready drops 47% — orch is pushing fewer slot states through the SPSC wiring queue because explicit deps don't synthesize redundant edges.
• task+heap_alloc is essentially flat — the ring buffers are uncontested at this 1280-task / 16K-window scale.
• The 3584 lookups in the auto run come from the 4-tensor-input per submit × 1280 submits × ~0.7 non-OUTPUT-tensor fraction. Even with max chain length of 1 and zero overlap checks (this workload's tensors don't share base addrs in the same hash bucket — unsurprising for paged attention), the per-lookup cost is non-trivial because lookup always pays the hash + cache-line fetch + iterate cost.

5. Orch ↔ scheduler interaction factors

Two distinct cost regimes for orch↔sched data sharing on the hot path:

• Factor A — Write-once-by-orch, read-by-sched payload (capacity-bound, small): PTO2TaskDescriptor, PTO2TaskPayload, and the per-slot PTO2TaskSlotState payload/task pointers are written once by the orchestrator at step 5, read once by the scheduler at dispatch (and again by AICore at execute). No coherence ping-pong: orch dirties, sched reads. Per-task payload is bounded (≤16 tensors + ≤16 scalars), and active payload working set fits in L2 by design — task_window_size is small, default 16384/ring, often as small as 16. Cost: bandwidth only.
• Factor B — Shared coordination cache lines (coherence-bound, big): lines that both threads write or one writes and the other reads on the hot path force snoop / invalidation round-trips. The hot lines are:
  • PTO2RingFlowControl (pto_shared_memory.h:54-76) — current_task_index and last_task_alive are on separate 64B-aligned lines on purpose, so each is single-writer.
  • PTO2TaskSlotState (pto_runtime2_types.h:316-398) — single 64B line carrying both orch-managed (task_state=PENDING, bind_buffers, active_mask, fanout_count increments on producers) and sched-managed (task_state=COMPLETED/CONSUMED, fanout_refcount, fanin_refcount, fanin_count) fields. Pings on every fan-out edge.
  • sched->wiring.queue slot lines — orch pushes (step 6), sched batch-pops (drain_wiring_queue). One ping per submit.

Factor B is what scales with fan-out density. The recent deferred-wiring redesign (orch no longer takes fanout_lock per producer, no longer allocates dep_pool) was a targeted cut on this; the remaining orch-side coherence work is the per-producer fanout_count++ increment. Auto-dep mode pays more on this dimension too, because every discovered TensorMap edge becomes a fanout_count++ on the producer slot's hot cache line. Manual mode discovers fewer redundant edges → fewer pings.

---

Evolution directions are discussed in follow-up comments below.

[hw-native-sys-bot]

Direction 1 — Optimizing orchestration on the current AICPU deployment

Audience. Runtime developers working on the a2a3 onboard AICPU runtime, with the current Thread-3-orchestrator / Threads-0..2-scheduler layout fixed and within the existing pto_orchestrator.cpp / pto_tensormap.h / pto_runtime2_types.h surface.

Headline. Today's largest single phase is param_copy — but it's a fixed cost (the user's Tensor + scalar data has to be moved into the GM payload). The phase that decides whether the orchestrator stays fast as workloads get richer is TensorMap (sync + lookup + insert): it's structurally serial on Thread 3, and its share of total grows directly with DAG density. The biggest multiplier on either phase is not what code each phase runs — it's how the orchestrator's cache lines interact with the schedulers on the shared AICPU cluster.

---

1. TensorMap is structurally serial and on a growing curve

Steps 2 / 3 / 4 of submit_task_common — sync_tensormap, lookup+dep, tensormap_ins — run strictly in order on Thread 3 with no pipelining. Each lookup depends on prior inserts having landed. There is no contention (TensorMap is orch-private, no atomics) but also no parallelism: N tasks pay N × (sync + lookup + insert) in absolute serial time.

What matters is not whether TensorMap is currently bigger than something else, but how each phase's share of total scales with workload shape:

Workload	Tasks	avg/task	sync+look+ins (% of total)	param_copy (% of total)	chain walks	overlap checks
Case1 (B=256, H=16, block=128)	1280	0.505 us	32.0%	36.4%	11	0
Case2 (B=64, H=64, block=64)	576	0.475 us	40.8%	32.7%	226	192 (100%)
Case3 (B=64, H=64, head_dim=256)	576	0.601 us	34.2%	32.2%	248	192 (100%)

chain walks = total entries traversed across all lookup() calls in the bucket linked list (each step is one cache-line read to follow next_in_bucket). overlap checks = number of times a lookup() matched buffer.addr on a chain entry and triggered the check_overlap() L1-byte-range → L2-hyper-rectangle → L3-conservative cascade in pto_tensormap.h:220-321 (tens of ARM instructions + several cache-line accesses per call). These two metrics measure two independent cost dimensions of lookup: pointer-chase length vs. overlap-arithmetic firing rate.

Two facts read straight from the table:

• param_copy's share is stable at 32-36% across all three workloads — it's a fixed lower bound, the user's Tensor metadata + scalar values being deep-copied into payload.tensors[16] / payload.scalars[16] so AICore can read them. It's the dominant phase today, but its share won't grow.
• sync+look+ins's share rises from 32% (Case1) to 40.8% (Case2) just by halving block_size. More importantly, the per-submit TensorMap arithmetic gets dramatically richer: chain walks grow 20× (11 → 226), overlap checks go from never firing to firing on every probe with 100% hit rate. The check_overlap cascade is non-trivial — pto_tensormap.h:220-321 per call. This is the future shape: more tensor reuse → longer bucket chains → more overlap work → wider serial Thread-3 critical path.

So the framing is: param_copy is today's ceiling for orch throughput; sync+look+ins is the variable that decides whether the orchestrator stays fast as workloads get richer.

Concrete optimizations worth investigating, roughly ascending in effort:

• Short-circuit sync_tensormap when the ring's entry count is zero. Saves the unconditional last_task_alive.load and the cleanup_retired check entirely. Should drop today's 33-56 us to ~0 in manual-scope or sparse-DAG workloads. Trivial — gate at PTO2TensorMap::sync_tensormap entry on next_entry_idx - free_num == 0.
• MRU-cache the per-tensor lookup result for the immediately-following submit. When task N+1's first INPUT tensor is task N's first OUTPUT (the dominant pattern in unrolled kernels — QK output is SF input, SF output is PV input), the lookup is hitting the same bucket head we just inserted into. A 1-deep MRU shortcut bypasses the hash + chain walk for this case.
• Batch overlap checks across multiple INPUT tensors of the same submit. The 4 INPUT tensors of a typical kernel almost always overlap-check independently. An L1-byte-range pre-filter applied to the whole batch could reject most pairs without entering the L2 hyper-rectangle path.
• Perfect-hash the bucket array when DAG is statically known. Manual-scope's set_dependencies(...) already declares the dep graph at orch-time. A future hint could let the runtime pre-compute a perfect-hash bucket layout for that scope — zero chain walks by construction.

---

2. Today's headline cost is param_copy — work that must happen, but is happening on the wrong critical path

param_copy is 32-36% of total in every workload measured, and is the largest single phase on Case1 manual mode at 51.8%. It's not waste — it's the user's Tensor metadata + scalar values being deep-copied into payload.tensors[16] / payload.scalars[16] in GM so that AICore can read them. The payload.init call at pto_orchestrator.cpp:652 is doing the actual user-data shuffle the kernel needs.

But it's happening serially on Thread 3, blocking all subsequent submits behind it. Cuts to attempt:

• NEON-vectorize the Tensor struct copy. sizeof(Tensor)=128 (2 cache lines per arg, statically asserted at pto_tensormap.h:324). 16 tensors → 32 lines per submit. A pair of 64-byte stores using ld1 {v0-v3.16b} / st1 {v0-v3.16b} is materially faster than the default compiler-generated copy on the AICPU CPU, especially since most fields aren't read by the orchestrator (only AICore reads them).
• Pre-validate Arg once and skip per-submit re-validation. Some validation today is repeated per submit even when the user's Arg is reused via reset(). A "frozen" Arg flag could skip overlap and tag checks.
• Move payload.init off the critical path when block_dim=1. For single-block tasks, there is no scheduler-side argument fan-out; the orch could write payload directly into a per-core slot rather than the shared payload array, removing a snoop on the slot's cache line.

---

3. The biggest multiplier on both phases is the orch↔sched cache interaction on shared AICPU cluster lines

Two distinct regimes — only one is cheap:

• Factor A — write-once-by-orch, read-once-by-sched payload (capacity-bound, small). task_descriptors[slot], task_payloads[slot], payload pointers, kernel_ids, packed_buffer_base/end — orch dirties, sched reads, no further writes. With task_window_size=16384 per ring and 4 rings, the active payload working set fits in L2 by design. Cost is pure bandwidth; no snoop cost. Don't optimize this — it's already cheap, and shrinking task_window further would invite back-pressure stalls instead.

• Factor B — shared coordination cache lines (coherence-bound, large). Lines that both threads write force a snoop / invalidation round-trip on every transition. On a2a3 AICPU clusters with separate L1/L2 per core, each ping is tens-to-hundreds of cycles, paid on the critical path. The hot ones are:

  • PTO2RingFlowControl::current_task_index / last_task_alive — already on separate 64B-aligned cache lines, each single-writer. Good.
  • PTO2TaskSlotState — one 64B line carrying orch-managed task_state=PENDING, bind_buffers, active_mask, total_required_subtasks, AND sched-managed task_state=COMPLETED/CONSUMED, fanout_refcount, fanin_refcount, fanin_count. Every fan-out edge bounces this line between orch L1 and sched L1.
  • sched->wiring.queue slot lines — orch pushes (step 6), sched batch-pops in drain_wiring_queue.
  • Producer slots' fanout_count field — orch increments at step 5 for each fanin edge; this is on the same 64B line sched will read at completion.

The direct evidence is fanin+ready's 47% drop between auto and manual (32.44 us → 17.16 us) on Case1. Manual mode produces fewer redundant fanin edges, which means fewer producer slot-state lines get dirtied, which means less snoop traffic on the orch↔sched boundary. Same physical phase, same code path — only the cache traffic changed.

Concrete optimizations:

• Split PTO2TaskSlotState into two 64-byte lines: orch-owned and sched-owned. Today's struct (pto_runtime2_types.h:316-398) mixes both writers into one cache line by accident of size. With orch-only fields on line A and sched-only fields on line B, each line is single-writer; the shared atomics (task_state, fanin_refcount) move to a small third line read by both but written infrequently.
• Batch fanin emissions into a single store-buffer flush per submit. The orch currently increments fanout_count per producer in a loop at step 5. With N fanin edges, that's N separate cache-line touches on producer slots, often spread across different addresses. A precomputed delta-vector applied as a single batched-store sequence would let the CPU's store buffer coalesce.
• Make the wiring queue's pop side read-only-ish. Today's drain_wiring_queue does CAS on dequeue_pos. On a SPSC queue (one orch pushing, one sched popping) this is overkill — a relaxed-load + release-store would suffice and avoid the RMW on the same line orch is pushing to.

---

4. A5 platform pressures — what shifts on the next-gen silicon

All numbers above are measured on a2a3. A5 changes three things that all push orch harder; combined, they roughly 2× the relative pressure on orch before any code change.

Hardware deltas (from src/a5/platform/include/common/platform_config.h and stated AICPU clock):

Quantity	a2a3	a5	Δ
AICore count (PLATFORM_MAX_BLOCKDIM = AIC; AIV = 2×)	24 AIC + 48 AIV = 72	36 AIC + 72 AIV = 108	+50% consumer cores
AICPU threads (PLATFORM_MAX_AICPU_THREADS)	4 (1 cluster × 4)	7 (4 clusters × 2 minus 1 reserved)	same-ish total, 4× more clusters
AICPU clock	baseline	75% of a2a3	each orch cycle is ~1.33× longer

The three pressures, each amplifying the other:

1. Per-orch-task wall-clock rises ~33% just from frequency. Today's 0.505 μs/task on a2a3 projects to ~0.67 μs/task on a5 with no other change. Case1's 647 μs total becomes ~862 μs on a5 at the same code.

2. AICore consumption demand rises 50%. With 50% more AIC and 50% more AIV, sched/AICore can drain tasks 50% faster (capacity-wise). Orch must produce 50% more tasks per unit wallclock to avoid AICore underutilization.

3. Combined throughput gap = 1.5 / 0.75 = 2.0×. Orch on a5 is squeezed twice as hard relative to compute demand. The slack that today lets a2a3's orch finish 98% of submissions by the time sched drains (PTO2 total submitted tasks = 1280, already executed 1256 tasks in the Case1 device log) shrinks substantially. The "future bottleneck" identified in §1 — TensorMap going from 32% to 41% of total just by halving block_size — arrives on a5 sooner because orch has less headroom to absorb growth.

Cross-cluster coherence cost on a5 specifically.

a2a3's 4 AICPU threads share a single cluster's L2; the orch↔sched cache pings in §3's Factor B are intra-cluster snoops. a5's threads are split across 4 clusters with ~2 threads each (orch on one cluster, sched threads distributed across all four). This makes Factor B materially worse:

• PTO2TaskSlotState lines (containing both orch-written fanout_count and sched-written task_state / fanout_refcount) now bounce across cluster L2 boundaries instead of within one shared L2. Cross-cluster snoops cost more than intra-cluster snoops on every coherent processor.
• Producer slots written by sched on cluster A and read by orch on cluster B incur per-access fabric latency that didn't exist on a2a3.
• The 47% drop in fanin+ready between auto-dep and manual-scope on a2a3 (§3) becomes proportionally more valuable on a5 — fewer redundant fanin edges means fewer cross-cluster snoops, and each avoided snoop is more expensive.

What this means concretely:

• The §1 optimizations (TensorMap short-circuit, MRU cache, batched overlap, perfect-hash) move from "nice future improvements" to prerequisites for keeping a5 AICore from idling. The 2× pressure absorbs the slack that today makes them optional.
• The §3 cache-line split (PTO2TaskSlotState into orch-owned / sched-owned 64B lines) becomes especially valuable because cross-cluster ping is much more expensive than intra-cluster ping. The same code change recovers more wallclock on a5.
• Cluster placement of orch and sched-0 matters: putting them on the same cluster keeps the SPSC wiring queue's producer-consumer ping intra-cluster. The wiring queue's careful per-side cache-line layout (alignas(64) on head_ and tail_, shown in comment 2 §6) already supports this — it just needs to be enforced at thread-affinity setup.
• Hot-tensor MRU caching (the §1 "task N+1's input is task N's output" shortcut) wins more on a5 because the alternative — re-fetching producer state from a cross-cluster L2 — is slower than on a2a3.

A5 doesn't change which optimizations are right. It changes when they become mandatory. Most of §1-3's list is "useful today, mandatory on a5".

---

Where to start

• 30 minutes: short-circuit sync_tensormap on empty per-ring map.
• One day: split PTO2TaskSlotState into orch-owned / sched-owned cache lines. High priority on a5 due to cross-cluster snoop cost (§4).
• One week: NEON-vectorize payload.init and the Tensor struct copy; build the MRU lookup cache.
• A5 readiness checkpoint: re-run the PTO2_ORCH_PROFILING=1 measurement matrix (Case1/2/3) on a5 silicon once available and compare; the 2× pressure analysis predicts ~1100-1200 μs orch on Case1 a5 if no changes land, vs. 647 μs on a2a3.

Each of these is independently landable, independently measurable against the baseline above, and none of them changes the orchestrator's deployment model — which is the topic of comment 2 below.

[hw-native-sys-bot]

Direction 2 — Moving streaming orchestration off the AICPU (Host-side, over UB)

Audience. Anyone considering whether the tensormap_and_ringbuffer orchestrator (the streaming Thread-3 design — not host_build_graph's build-then-launch model) should move from AICPU to Host CPU. The premise is that the same task model — ring buffers, TensorMap, scope, streaming per-task submits — stays intact; only Thread 3's execution location changes.

Setup of the bus. Host CPU and AICPU both access on-device HBM through normal load/store instructions. AICPU reaches HBM through on-chip paths with very high bandwidth headroom (current paged_attention workloads use ~8.5 GB/s of capacity that's many tens of GB/s). Host reaches HBM through UB (Unified Bus) with the following physical limits I'll take as input for this analysis:

• UB ld/st latency: 500 ns per transaction
• UB bandwidth: 100 Gbps = 12.5 GB/s
• HBM is uncacheable from Host — each store goes as a separate UB transaction, no write-back caching, no L1 absorption

The last bullet is the load-bearing one. Everything below derives from it.

---

1. The right question isn't per-submit latency, it's whether the producer keeps up with the consumer

The streaming orchestrator is the producer in a producer-consumer pipeline. AICore (via sched on AICPU) is the consumer. What matters end-to-end is not how long each submit_task takes, but whether orch's sustained production rate ≥ sched/AICore's sustained consumption rate. If yes, orch is never on the critical path and any orch slowdown is invisible behind sched/AICore execution time.

So the analysis has two layers:

1. How fast can Host orch produce tasks — bounded by UB latency × per-task HBM transactions, modulated by CPU outstanding-request concurrency.
2. What's the per-task consumption rate of sched/AICore — this we don't have measured here; for paged-attention class workloads it's typically orders of magnitude slower than orch.

This comment quantifies (1) precisely and notes that (2) is the deciding factor for end-to-end impact.

---

2. What state is HBM-resident vs orch-private

submit_task_common touches two qualitatively different memory regions:

Orch-private (cacheable on Host's L1/L2/L3, no UB involvement):

• TensorMap entry pool, buckets, free list, task_entry_heads[ring]
• scope_tasks[], scope_begins[], scope_stack_top
• Local PTO2FaninBuilder during a submit

HBM-resident (every read/write goes over UB at 500 ns):

• PTO2RingFlowControl::current_task_index, last_task_alive
• task_descriptors[], task_payloads[], slot_states[] — the ring contents themselves
• sched->wiring.queue (buffer slots + head_)
• Producer slots' fanout_count fields

For orch on Host, the partition becomes load-bearing: orch-private state moves into Host CPU's own cache hierarchy (faster than AICPU's on-chip caches — bigger L2/L3, wider OoO, SIMD). HBM-resident state is uncacheable to Host CPU — every access is a 500 ns UB transaction.

---

3. Per-task HBM write traffic

Counting cache-line writes that orch must commit to HBM on each submit:

Region	Lines written per task
PTO2TaskDescriptor	1
PTO2TaskPayload header (~576 B)	9
tensors[] (4 active × 2 lines per Tensor)	8
scalars[]	1
PTO2TaskSlotState (own)	1
current_task_index publish	1
Producer slots' fanout_count++ (avg 2 fanin per task)	2
SPSC wiring queue: buffer slot + head_ release-store	2
Per task total	~25 cache lines

For 1280 tasks (Case1 paged_attention auto-dep baseline): 32,000 cache-line transactions crossing UB.

---

4. The actual throughput limit on Host is not UB bandwidth

Under uncacheable HBM, each cache-line write is a 500 ns UB transaction. UB advertises 12.5 GB/s peak, but that requires enough concurrent transactions in flight to saturate the pipe. The CPU's outstanding-write capacity is the binding constraint:

• 64 B per transaction / 500 ns = 128 MB/s per single-issue stream
• To hit UB's 12.5 GB/s peak: ~100 concurrent transactions
• Modern Host CPUs typically have MSHR / store-buffer depth K = 8-16 outstanding

So Host's effective HBM write throughput is:

K (CPU outstanding-request depth)	Effective throughput
8	1.0 GB/s
16	2.0 GB/s
100 (would-saturate-UB)	12.5 GB/s

The first two are realistic; the third is not achievable from a single CPU thread. UB's 12.5 GB/s ceiling is almost entirely unused; the bottleneck is the CPU's ability to issue uncached writes.

This also means: making UB faster doesn't help. Adding a second UB link doesn't help. Wider HBM doesn't help. The only way to make Host orch faster is to reduce the number of cache lines it has to write.

---

5. Total Host orch projection

Three independent resource pipelines, all running concurrently across in-flight submits:

Pipeline	Computation	Host time
Payload streaming (32,000 line writes at K=8 throughput)	32,000 × 64 B / 1 GB/s	~2,000 μs
fanout_count++ (uncached read + write = 1 μs/op, K=8 overlap)	2560 ops × 1 μs / 8	~320 μs
Local compute (orch-private, Host-cached)	TensorMap + scope mgmt, Host L2/L3 hit	~100 μs
Critical path = max		~2,000 μs

vs AICPU baseline 647 μs → Host naive port ≈ 3× slower

At K=16 (more aggressive Host CPU): ~1,000 μs, ~1.5× slower.

The payload streaming pipeline dominates and saturates the CPU's outstanding-request depth. The fanout_count++ and local compute pipelines sit in the slack.

---

6. SPSC wiring queue is not a bottleneck — verified from code

I initially worried that the wiring queue push (sched->wiring.queue.push(...)) would be a contested RMW on a single line, serialized across all submits. Looking at the actual code (scheduler/pto_scheduler.h:437-505), it is a textbook lock-free SPSC:

bool push(PTO2TaskSlotState *item) {
    uint64_t h = head_.load(std::memory_order_relaxed);          // own line, hot in L1
    uint64_t next_h = h + 1;
    if (next_h - tail_cached_ > mask_) {                          // check local cache first
        tail_cached_ = tail_.load(std::memory_order_acquire);     // refresh only on full
        if (next_h - tail_cached_ > mask_) return false;
    }
    buffer_[h & mask_] = item;                                    // plain store
    head_.store(next_h, std::memory_order_release);              // release-store, not RMW
    return true;
}

The struct layout (alignas(64) on each of head_, tail_cached_, tail_, head_cached_) places producer and consumer state on disjoint cache lines. Producer (orch) only writes head_ and tail_cached_; consumer (sched thread 0) only writes tail_ and head_cached_. Zero atomic RMW. No CAS. No contested writes on the producer side.

On Host over UB, each push is 2 uncached stores: the slot pointer and the head_ release-store. Per push: 2 × 500 ns = 1 μs (or 500 ns if write-combined within UB protocol). For 1280 pushes: ~640 μs - 1280 μs serial, but they're independent of payload streaming and can run in the same CPU outstanding-request budget. Folded into the same 2000 μs critical path above.

This corrected a real mistake I made earlier in this thread — assuming the wiring queue was a contested RMW choke point. It's not.

---

7. fanout_count++ is also not an atomic RMW

PTO2TaskSlotState::fanout_count is a plain int32_t (pto_runtime2_types.h:319). The increment loop in submit_task_common (pto_orchestrator.cpp:636-641) does plain non-atomic producer->fanout_count++. Safe because only orch writes it (sched reads fanout_count only after orch's scope_end, by which point all increments are finalized).

But on uncacheable HBM from Host, even a non-atomic increment costs 1 μs (500 ns read + 500 ns write) because each access is a UB transaction. The lines may also be co-touched by sched (other fields on the same PTO2TaskSlotState line), but the cost on Host is dominated by the uncached property, not by cross-agent coherence.

This is the 320 μs in the table above. Real cost, but small compared to the payload streaming dominator.

---

8. Where Host wins: orch-private work

The phases comment 1 identified as the future bottleneck on AICPU (TensorMap lookup, insert, dep computation) are entirely orch-private. On Host:

• TensorMap entry pool (pool_size × 128 B = 8 MB) fits in Host L2 / L3 — on AICPU it spills past L2
• lookup's hash + chain-walk runs at Host CPU speed (wider OoO, faster clock, SIMD-able overlap check)
• check_overlap's L1/L2/L3 cascade (pto_tensormap.h:220-321) benefits from Host's bigger register file and OoO
• Multi-core: TensorMap can be sharded by ring → 4 rings × 4 threads = 4× parallelism on the "future bottleneck" phase

The Case2 (block=64) data from comment 1 — where TensorMap chain walks grew from 11 to 226 and overlap checks fired 100% — represents exactly the workloads where Host has the largest local-compute advantage. Host orch is conservatively 2× faster on these phases, possibly 4×. On richly fan-out DAGs the orch-private speedup can fully offset the HBM-write slowdown.

User-level parallel_for becomes meaningful (originally tracked as #545 Subtask 1; consolidated here as a concrete Host-multi-core optimization with no AICPU equivalent). Orchestration code containing parallel_for(0, N, body) constructs is forced to serialize on AICPU because Thread 3 is the only orch core. On Host, with proper data-dependence analysis, loop iterations that have no loop-carried dependences can be dispatched across orch threads — independent iterations build sub-DAGs in parallel, then merge through the wiring queue. This is iteration-level parallelism on top of runtime-internal sharding mentioned above; the two compose: ring-sharded TensorMap × N-way parallel_for can give multiplicative speedup on the dependency-discovery critical path. AICPU has neither lever; Host has both.

---

9. Pipeline-ability: does 3× slower orch starve the consumer?

Orch's job in the pipeline is to keep the wiring queue and ring buffer non-empty so AICore is never idle waiting for next task. The question is whether Host orch's slower production rate (~2 ms for 1280 tasks ≈ 1.5 μs per task) stays ahead of sched/AICore's consumption rate.

We don't have sched/AICore consumption measured in this thread. But two anchor points:

• For paged-attention-class workloads on a2a3 with 24 AIC + 48 AIV: per-task AICore execution is typically tens to hundreds of μs (small batched matmul, softmax, vector ops). Even at 50 μs/task averaged over 72 cores: 1280 tasks consume in ~890 μs of compute (back-of-envelope).
• On AICPU today, the Profiling output shows 1256 of 1280 tasks already FIN'd by orch-done time (647 μs). So orch and consumer are running nearly in lockstep — small slack.

If sched/AICore consumes 1280 tasks in ~890 μs (close to the AICPU orch's 647 μs), and Host orch produces them in ~2000 μs (3× slower), then orch becomes the binding rate: end-to-end wallclock grows from ~890 μs (AICore-bound) to ~2000 μs (Host-orch-bound). That's a real 2× wallclock regression for this workload class.

For workloads where AICore compute per task is bigger (large matmul, deep softmax, dense ops):

• AICore execution: many ms total
• Host orch ~2 ms is hidden behind it
• No end-to-end regression

For workloads where AICore execution is even smaller (lightweight broadcast ops, point-wise tensor moves):

• AICore execution: hundreds of μs
• Host orch becomes the wallclock bottleneck → regression worse than 2×

The deciding factor is "average AICore per-task time vs. ~1.5 μs Host orch per-task production". Workloads on the "lightweight task" end of the spectrum will see real Host-orch wallclock cost.

---

10. The architectural fix: reduce per-task HBM traffic, not the protocol

Going back to section 4: the bottleneck is Host CPU outstanding-request depth × UB latency, with effective throughput ~1 GB/s. The only way to make Host orch fast is to shrink the per-task HBM write count from ~25 lines to something dramatically smaller.

Hierarchical decomposition is the architectural lever:

• Host side runs: control flow, TensorMap, dep derivation, scope management. Output of a submit is a compact description: (kernel_ids[3], compact_tensor_refs, scalar_values[16], producer_task_ids[4]). Maybe 150 B per submit.
• AICPU stub runs: receives the compact descriptions, performs the per-task HBM writes locally (AICPU's HBM access is cheap), pushes to wiring queue.

Numbers under this model:

Resource pipeline	Host-on-UB load	Host time
Description streaming (150 B/task × 1280, K=8)	192 KB / 1 GB/s	~192 μs
fanout_count++ and other contested writes	(moved to AICPU stub — now local)	0
Local compute (TensorMap on Host)	benefits from L3 cache + multi-core	~100 μs
Critical path		~300 μs

vs AICPU baseline 647 μs → Host-with-decomp ≈ 2× faster than current AICPU, on top of recovering the comment-1 TensorMap-scaling concern.

The seam to cut along is already mostly cleanly laid out in the codebase: compute_task_fanin and register_task_outputs (pto_dep_compute.h) are header-only templates that emit producer IDs via a runtime_emit callback. That callback is the natural Host↔AICPU boundary. Host implementation calls them with TensorMap living on Host; AICPU stub receives emitted producer IDs and skips TensorMap entirely.

---

11. Summary

Under the user-supplied physical UB constraints (500 ns ld/st, 12.5 GB/s, uncacheable from Host):

Deployment	Per-task HBM writes per task	Critical path	vs AICPU baseline (647 μs)
AICPU today	(HBM access is on-chip, cheap)	647 μs (measured)	1.0×
Host, naive port (current submit_task on Host)	25 lines	~2,000 μs	~3× slower
Host with cache-line-budget optimization (split PTO2TaskSlotState, etc.)	22 lines	~1,700 μs	~2.6× slower
Host + hierarchical decomp (only descriptions cross UB)	~3 lines	~300 μs	~2× faster

Three actionable takeaways:

1. Bandwidth is not the bottleneck on Host; the CPU's outstanding-request depth is. So protocol redesign on top of the existing per-task write traffic gets nowhere — every "smarter" version of submit_task_common running on Host still has to push ~25 lines through a ~1 GB/s effective pipe.

2. The wiring queue and fanout_count++ are not contested RMWs — both are plain stores with appropriate memory ordering. They are not the bottleneck and don't need redesign. (This corrects a wrong direction in earlier optimization analyses on this thread.)

3. The only architectural lever that pays back the UB cost is reducing per-task UB traffic. Hierarchical decomposition along the compute_task_fanin / register_task_outputs seam reduces traffic by ~10×, brings Host orch below AICPU baseline, and simultaneously addresses the comment-1 TensorMap-scaling concern via Host's multi-core + larger cache.

Direction 3 below asks the orthogonal question: if the workload genuinely demands more than this architecture can provide, what would purpose-built hardware for orchestration look like?

[hw-native-sys-bot]

Direction 3 — Purpose-built hardware for streaming orchestration

Audience. Anyone with input on chip-definition for future generations: if the orchestration workload identified in comments 1-2 is on the long-term roadmap, what hardware structures would purpose-build for it? This comment is not a deployment recommendation — it's a sketch of what the relevant compute domain should look like if drawn from the workload's actual data-flow patterns.

The proposal in one sentence. Build a dedicated "orchestration compute domain" with three categories of fast register files that match the three distinct memory-access patterns identified in comments 1-2 — instruction (orch code), streaming data (orch→sched payload that is written-once-read-once-then-recycled), and communication (small atomic state read/written by both orch and sched).

This comment evaluates that proposal along five dimensions: sizing (does it fit in realistic silicon?), per-task speedup (what does it buy?), TensorMap handling (the hardest unsolved piece), architectural boundary (where does it sit and how does it connect?), and target workloads (where this design actually pays off).

---

1. Mapping the three register categories to the runtime's actual data structures

Category	Maps to in tensormap_and_ringbuffer	Pattern
Instruction	aicpu_orchestration_entry (user .so's instruction stream) — hot inner loop typically < 1 KB, full SO < 1 MB	Read-only after load; written once at init
Streaming data	PTO2TaskPayload (tensors[16] + scalars[16] + header), PTO2TaskDescriptor, current_task_index publish	Written once by orch, read once by sched, then slot recycled. No long-term storage need.
Communication	PTO2TaskSlotState::{task_state, fanout_count, fanin_refcount, fanout_refcount}, PTO2RingFlowControl::{current_task_index, last_task_alive}, SPSC queue head_/tail_	Small bytes; atomic reads/writes by both orch and sched; bounces continuously

The streaming-data category is the largest by volume but the simplest semantically — the data has a strict producer-consumer life cycle and never needs to be stored long-term. Today's runtime stores it in HBM because there's no other persistent shared memory, but HBM is overkill for a write-once-read-once data path.

The communication category is small by volume but is the source of all the cache coherence pain in comments 1 (Factor B) and 2 (uncached UB transactions). It's small enough to fit in a dedicated multi-ported register file with hardware-supported atomics, sidestepping the general cache-coherence protocol entirely.

---

2. Sizing — does it fit in realistic on-chip silicon?

Based on the workload data measured in comments 1-2:

Instruction SRAM — minimal:

• Hot inner loop of orchestration: a few hundred bytes (the loop body of paged_attention_unroll's bn loop)
• Whole orch .so text: tens of KB to ~1 MB
• Budget: 64 KB instruction scratchpad. Fully resident; no misses possible.

Streaming data SRAM — the bulk of the cost:

• Per-task footprint (from comment 2 §3): 25 cache lines = 1.6 KB
• Active in-flight tasks: bounded by ring window. Default PTO2_TASK_WINDOW=16384 × PTO2_MAX_RING_DEPTH=4 = up to 64K slots, but realistic is 1K-5K
• Budget: 16 MB streaming SRAM would cover 10K simultaneous in-flight tasks. Comfortable margin.
• This is the "passthrough" buffer — orch writes one end, sched reads the other end, no HBM involved on the hot path

Communication register file — small but high-port-count:

• Per slot comm state: fanout_count (4B) + task_state (4B) + fanin_refcount (4B) + fanout_refcount (4B) + fanin_count (4B) + active_mask (8B) ≈ 32B per slot
• Per ring control: current_task_index + last_task_alive + SPSC head_/tail_ ≈ 32B per ring
• For 16K-slot rings × 4 rings: 16K × 4 × 32B = 2 MB total comm state
• Budget: 4 MB multi-ported register file with hardware atomic ALU

Total on-chip footprint: ~20 MB SRAM. On a 7nm process this is ~30-50 mm² of silicon — roughly the silicon cost of 3-5 AIC cores. Not free, but in the same order as a single workload class's compute area. Realistic if the workload demand justifies it.

---

3. Per-task performance projection

Per-task work decomposed into the three categories with their projected hardware latency:

Phase (from comment 1)	Today on AICPU	On dedicated HW
task+heap_alloc (~171 μs / 1280 = 134 ns/task)	cache-coherent HBM access	comm register file: ~1-2 cycles for current_task_index++, ~1 cycle for last_task_alive read
sync_tensormap (44 ns/task)	local + 1 HBM read	comm register file: ~1 cycle for last_task_alive read
lookup+dep (63 ns/task)	local TensorMap walk	(see §4 — this is the open question)
tensormap_ins (55 ns/task)	local TensorMap insert	(see §4)
param_copy (184 ns/task)	~1.5 KB write to HBM via cache	streaming SRAM: 25 cache lines × ~1 cycle = ~25 cycles (~12 ns @ 2 GHz)
fanin+ready (25 ns/task)	atomic RMW + SPSC push	comm register file: hardware atomic, ~1 cycle

Excluding TensorMap (see §4):

• Per-task fixed cost on dedicated HW: ~30-50 cycles = ~15-25 ns
• vs AICPU baseline 505 ns/task → ~20-30× speedup on non-TensorMap work

For 1280 tasks: ~20-30 μs total orch time vs current 647 μs. Orch is no longer perceptible against any realistic AICore execution time. The pipeline-fill concern from comment 2 disappears.

---

4. The hard part — what to do with TensorMap

TensorMap is not a clean fit for the three-register-file model:

• It's a 8 MB hash table (pool_size × 128B) with chained buckets, not a simple ring buffer
• lookup does pointer-chase across the chain + check_overlap arithmetic (L1 byte-range → L2 hyper-rectangle → L3 conservative, ~tens of ARM instructions per call)
• insert is an O(1) linked-list update but on a structure that needs to support concurrent lookups
• It's the phase comment 1 identified as the future bottleneck — exactly the workload this hardware most needs to accelerate

Three architectural options, each with very different cost/benefit:

Option A — Run TensorMap on a general-purpose CPU core within the orch domain.

• Add 1-4 small general-purpose cores to the orch domain
• TensorMap lives in their L1/L2 (fed by the streaming SRAM)
• Per-lookup cost: similar to today's AICPU per-lookup but with bigger L1 and possibly multi-threaded
• Gain over AICPU: ~2-4× from cache + OoO
• Loss vs the rest of this proposal: TensorMap stays serial (single core does it for each submit), so it bounds total orch throughput
• Simplest to design, weakest performance ceiling

Option B — Dedicated TensorMap engine (CAM + comparator array).

• Content-addressable memory keyed on buffer.addr
• Array of parallel comparators that run check_overlap on all bucket entries simultaneously
• 1-cycle hash + 1-cycle parallel overlap check
• Per-lookup cost: ~2-5 cycles regardless of chain length
• Per-insert cost: ~2-5 cycles
• Gain: ~50-100× over AICPU TensorMap
• Cost: significant silicon for CAM + N-way comparator array; the overlap arithmetic in check_overlap is non-trivial (stride checks, ref-shape derivation, per-dim segment intersection) — turning that into parallel hardware is a major engineering effort
• Hardest to design, highest performance ceiling

Option C — Software-defined TensorMap on a fast CPU core within the orch domain, but the hash table lives in dedicated SRAM (not HBM).

• Same algorithm, but the 8 MB pool is in low-latency SRAM next to the CPU
• L1/L2 misses become 1-cycle SRAM hits
• The check_overlap arithmetic still runs in ALU (no hardware acceleration of overlap)
• Per-lookup cost: ~5-15 cycles
• Gain: ~10-30× over AICPU
• The pragmatic middle ground — keeps software flexibility, eliminates memory latency, leaves the door open to add hardware acceleration of check_overlap later if it remains the bottleneck

Recommendation: start with Option C. It captures most of the gain at a fraction of the silicon cost. The CAM-based Option B is worth designing only if TensorMap workload patterns stabilize across multiple product generations.

---

5. Architectural boundary questions

This proposal is silent on several integration questions that determine whether the design is buildable:

Q1: Where does the orch domain physically sit?

• On-chip alongside AICore: tight integration, AICore can directly read the streaming SRAM as task dispatch payload, comm registers are visible to sched logic that triggers AICore. Tight but high pressure on chip area.
• Separate chiplet with high-bandwidth interconnect: more area budget, but the link to AICore becomes the new bottleneck and may reintroduce coherence pain.
• Replace AICPU entirely — orch domain subsumes the four AICPU threads' role. Probably the cleanest design.

Q2: How does sched (or what was sched) fit in?

• Today: 3 AICPU threads scan core completion registers and dispatch new tasks.
• In this proposal: the streaming-data + comm-register pattern already encodes producer→consumer + atomic-state semantics. The sched function decomposes into:
  • Pop next ready task from streaming SRAM → dispatch to AICore (HW-managed FIFO)
  • On AICore completion → update comm register fanout_refcount (HW-managed atomic)
  • Watermark advance (last_task_alive) → HW counter
• It's plausible the entire scheduler logic becomes a small state machine + the comm register file, with no general-purpose cores needed for sched.

Q3: How does user orch code reach this domain?

• ISA-compatible with a general CPU: existing orch source compiles unchanged. Easy adoption. Maybe 2-4 cores in the domain running C++ orch code at high frequency.
• Dedicated DSL with intrinsic submit/scope operations: 10× faster but requires recompiling user kernels and limits expressibility.
• General CPU + intrinsic instructions for the hot operations: the practical middle ground. rt_submit_task becomes a single instruction (or short sequence) that operates on streaming SRAM + comm registers directly.

Q4: How many orch domain instances per chip?

• Today: 1 orchestrator thread per AICPU. Comment 1 §1 identifies single-thread serialization as the main TensorMap bottleneck.
• In this proposal: N orch domain instances (one per ring? one per workload class?) could run in parallel, each with own streaming SRAM + comm register file.
• Sharing AICore cluster as the consumer: needs a shared dispatch arbiter between orch domains.
• Plausibly 2-4 instances per chip for multi-workload serving scenarios.

---

6. Where this design actually pays off

Connecting back to the workload data:

Wins big:

• Comment 1's future bottleneck workloads (Case2-like, where TensorMap chain walks grow 20× and overlap checks fire 100%). Option B or C TensorMap acceleration is the direct counter. Without this hardware, that bottleneck keeps growing with model complexity.
• High-throughput inference serving where many small requests share the chip. Orch per-task time directly impacts time-to-first-token; sub-microsecond orch is meaningful.
• Multi-chip / multi-domain dispatch where the orch domain coordinates across many AICore clusters. Comm register file's HW atomics scale better than cache-coherent state across chiplets.

Wins less (orch already invisible):

• Today's Case1 paged_attention (orch 647 μs out of 15 s wallclock — orch is rounding error). Hardware would speed up orch from 0.005% to 0.0003% of wallclock — not worth the silicon.
• Compute-bound workloads with very long per-task AICore time (large training matmuls). Orch overhead is hidden behind compute regardless.

Wins least / no win:

• Workloads where AICore is bandwidth-bound on HBM. The orch domain doesn't help with HBM bandwidth into AICore. Faster orch just means AICore stalls earlier.
• Workloads dominated by user code outside submit_task (heavy control flow, get_tensor_data polling). Dedicated hardware accelerates submit_task internals but the user-code path is still general-purpose CPU.

Wins partially:

• Comment 2's hierarchical decomposition (Host + AICPU stub). The hardware in this comment effectively replaces the AICPU stub with a faster purpose-built unit. Host still does the high-level orchestration; the dedicated domain handles the streaming + comm. This is the most likely actual deployment if both comments' directions are adopted.

---

Summary

The user's three-register-file architecture maps cleanly to the three memory-access patterns I identified in comments 1-2:

• Instruction → cheap and easy: 64 KB scratchpad, fully resident
• Streaming data → the bulk of the silicon cost: ~16 MB SRAM, replaces HBM-mediated producer-consumer with on-die FIFO
• Communication → small but architecturally crucial: ~4 MB multi-ported register file with hardware atomics, eliminates Factor B coherence cost entirely

Total silicon: ~20 MB SRAM ≈ 3-5 AIC cores' area. Per-task orch time drops from 505 ns to ~15-25 ns excluding TensorMap (~20-30× speedup); with Option C TensorMap (software in SRAM-backed CPU) → ~50-80 ns total per task. Orch becomes ≤ 0.01% of typical workload wallclock and exits all critical paths.

The unresolved design question is TensorMap: software-on-CPU-with-SRAM (Option C) is the pragmatic choice; a full hardware engine (Option B) only justifies its silicon if TensorMap workload patterns stay stable across product generations.

This hardware doesn't replace comments 1 and 2 — it makes them moot for the per-submit critical path. Comment 1's software optimizations remain useful for the years before this hardware ships; comment 2's hierarchical decomposition is the natural way to interface this hardware with a general-purpose host that handles model serving and user control flow. The three directions stack: software optimization buys time, deployment flexibility buys options, hardware redesign sets the ceiling.