Add: expose L3/L4 host scheduling swimlane

[TaoZQY]

Summary

• Add a fixed, non-variadic host-span C ABI to the process-global logger and resolve it before local chip children are forked. The bridge reuses SIMPLER_HOST_STRACE and remains disabled for unsupported topologies.
• Emit l3.graph_build, l3.submit, l3.dispatch, l3.frame_submit, l3.activate, and l3.complete with run, task-slot, worker, dispatch, and endpoint attributes.
• Add strace_timing.py --swimlane to render real OS pid/tid lanes, process/thread labels, and submit-to-dispatch flow arrows without changing the established --trace-out view.
• Keep raw clk=dev timestamps in unalignedDeviceSpans instead of putting unrelated device and host clocks on one visible Chrome Trace axis, which avoids an empty multi-day Perfetto viewport without inventing a clock offset.
• Document the markers and CLI, and add coverage for graph failures, real lane identities, flow correlation, legacy-output compatibility, single-frame completion, and prepared-frame activation.

Usage

python -m simpler_setup.tools.strace_timing path/to/log \
  --swimlane host_swimlane.json

Open host_swimlane.json in Perfetto or chrome://tracing.

Testing

• pytest tests/ut/py/test_strace_timing.py tests/ut/py/test_worker/test_host_worker.py -q — 226 passed
• test_scheduler with host tracing enabled — 63 passed
• test_scheduler builds with SIMPLER_HOST_STRACE=0
• pytest examples/workers/l3/child_memory -q --platform a2a3sim — 1 passed

Fixes #1708

[coderabbitai]

Important

Review skipped

Auto incremental reviews are disabled on this repository.

Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Pro Plus

Run ID: 8e452ec1-f081-45f8-a992-16ea98fafa67

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

• 🔍 Trigger review

Note

Reviews paused

It looks like this branch is under active development. To avoid overwhelming you with review comments due to an influx of new commits, CodeRabbit has automatically paused this review. You can configure this behavior by changing the reviews.auto_review.auto_pause_after_reviewed_commits setting.

Use the following commands to manage reviews:

• @coderabbitai resume to resume automatic reviews.
• @coderabbitai review to trigger a single review.

Use the checkboxes below for quick actions:

• ▶️ Resume reviews
• 🔍 Trigger review

📝 Walkthrough

Walkthrough

The PR adds a host-span ABI and logger bridge, instruments hierarchical execution, preloads the logger before extension loading, publishes runtime binaries atomically, and adds filtered host swimlane JSON output with real PID/TID lanes and unaligned device spans.

Changes

Host tracing and swimlane output

Layer / File(s)	Summary
Host-span ABI and sink bridge src/common/log/..., python/bindings/..., tests/ut/cpp/...	Defines the span ABI, weak sink declaration, bounded encoding, logger emission, RAII helpers, and Python binding functions.
Runtime initialization and graph-build tracing python/simpler/..., .claude/rules/project-layout.md, tests/ut/py/test_worker/test_host_worker.py	Preloads libsimpler_log.so globally, centralizes logger initialization, gates host spans on sink availability, and emits l3.graph_build.
Hierarchical scheduler span instrumentation src/common/hierarchical/..., tests/ut/cpp/...	Adds l3.submit, l3.dispatch, l3.complete, l3.frame_submit, and l3.activate spans with execution metadata and tests.
Atomic runtime artifact publication simpler_setup/runtime_compiler.py, tests/ut/py/test_runtime_builder.py	Stages runtime binaries, applies processing before atomic replacement, and removes temporary files on failure.
Filtered host swimlane generation simpler_setup/tools/..., tests/ut/py/test_strace_timing.py, docs/dfx/host-trace.md, simpler_setup/tools/README.md	Adds legacy-span filtering, real PID/TID lanes, dispatch flow events, unaligned device-span output, and the --swimlane CLI option.

Estimated code review effort: 4 (Complex) | ~60 minutes

Sequence Diagram(s)

sequenceDiagram
  participant PythonWorker
  participant HierarchicalScheduler
  participant HostLogger
  participant StraceTiming
  PythonWorker->>HostLogger: emit l3.graph_build
  HierarchicalScheduler->>HostLogger: emit submit, dispatch, frame, activation, completion spans
  HostLogger-->>StraceTiming: write STRACE records
  StraceTiming-->>StraceTiming: filter spans and build PID/TID lanes
  StraceTiming-->>StraceTiming: write flow events and unalignedDeviceSpans

Loading

Possibly related PRs

• hw-native-sys/simpler#1691: Both changes modify host logging and STRACE parsing.
• hw-native-sys/simpler#1466: Both modify hierarchical orchestration and worker-dispatch paths.
• hw-native-sys/simpler#1379: Both modify hierarchical orchestration and concurrent request execution paths.

Poem

A rabbit watches host spans run,
Across real lanes beneath the sun.
Submit, dispatch, complete in flight,
Device clocks keep their separate light.
Atomic binaries land just right.

🚥 Pre-merge checks | ✅ 4 | ❌ 1

❌ Failed checks (1 warning)

Check name	Status	Explanation	Resolution
Docstring Coverage	⚠️ Warning	Docstring coverage is 23.91% which is insufficient. The required threshold is 80.00%.	Write docstrings for the functions missing them to satisfy the coverage threshold.

✅ Passed checks (4 passed)

Check name	Status	Explanation
Title check	✅ Passed	The title clearly identifies the main change: exposing the L3/L4 host scheduling swimlane.
Description check	✅ Passed	The description directly explains the host tracing, swimlane output, compatibility behavior, documentation, and test coverage.
Linked Issues check	✅ Passed	The changes implement the linked issue objectives, including host spans, real PID/TID lanes, flow arrows, device-span separation, gating, and preload behavior.
Out of Scope Changes check	✅ Passed	The changes support the host swimlane feature and its required logger loading, atomic publication, documentation, and test coverage.

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}

[coderabbitai]

Actionable comments posted: 4

🤖 Prompt for all review comments with AI agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@python/simpler/worker.py`:
- Around line 6475-6482: Update Worker.init() so
_initialize_simpler_log(binaries) and _bind_host_span_sink() run before any
local forks for every hierarchical worker, including L4 parents without
device_ids. If the sink bind returns false, quietly disable host tracing instead
of raising; set self._host_trace_enabled only when the bind succeeds.

In `@simpler_setup/tools/strace_timing.py`:
- Line 414: Rename the loop variable in the span.attrs iteration to avoid
shadowing the imported field symbol and resolve Ruff F402, updating its
references within the loop accordingly.

In `@src/common/hierarchical/worker_manager.cpp`:
- Around line 466-475: Capture the dispatch trace metadata before the worker is
notified: in the dispatch flow around cv_.notify_one(), read and store the run
ID, callable hash, and attributes from ring_ while d.task_slot is still valid.
After unlocking, emit l3.dispatch using only those captured values, preserving
the existing timing and notification behavior.

In `@src/common/log/host_log.cpp`:
- Around line 223-235: Update simpler_log_emit_host_span to bound the encoded
span->name and span->attributes fields so each formatted STRACE record remains
within PIPE_BUF, including truncation before logging. Escape newline and
field-delimiter characters using the format expected by the trace parser, while
preserving the existing validation and metadata fields.

🪄 Autofix

Fix all unresolved CodeRabbit comments on this PR:

• Push a commit to this branch (recommended)
• Create a new PR with the fixes

---

ℹ️ Review info

⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Pro Plus

Run ID: c8861214-fbc6-497d-9951-312cbf8715f6

📥 Commits

Reviewing files that changed from the base of the PR and between 764f456 and 49281b0.

📒 Files selected for processing (17)

• docs/dfx/host-trace.md
• python/bindings/CMakeLists.txt
• python/bindings/task_interface.cpp
• python/simpler/task_interface.py
• python/simpler/worker.py
• simpler_setup/tools/README.md
• simpler_setup/tools/strace_timing.py
• src/common/hierarchical/host_trace.cpp
• src/common/hierarchical/host_trace.h
• src/common/hierarchical/orchestrator.cpp
• src/common/hierarchical/worker_manager.cpp
• src/common/log/host_log.cpp
• src/common/log/include/common/host_span.h
• tests/ut/cpp/CMakeLists.txt
• tests/ut/cpp/hierarchical/test_scheduler.cpp
• tests/ut/py/test_strace_timing.py
• tests/ut/py/test_worker/test_host_worker.py

[TaoZQY]

@coderabbitai review

[coderabbitai]

✅ Action performed

Review finished.

Note: CodeRabbit is an incremental review system and does not re-review already reviewed commits. This command is applicable only when automatic reviews are paused.

[coderabbitai]

Note

GitHub couldn't provide a complete incremental comparison for this pull request, so CodeRabbit is performing a full review instead. This review may take a little longer.

[coderabbitai]

Actionable comments posted: 2

🧹 Nitpick comments (1)

simpler_setup/tools/strace_timing.py (1)

425-436: 🎯 Functional Correctness | 🔵 Trivial | 💤 Low value

Thread labels depend on which span is parsed first.

_host_thread_name uses a single representative span per (pid, tid), chosen by parse order. If a thread emits an l3.* span without a role attribute first, the fallback at Line 431 labels the lane worker even when the thread is the facade or the scheduler.

Consider selecting the representative from the first span that carries a role attribute, and falling back to parse order only when no span on that lane has a role.

♻️ Proposed change

     for pid, tid in host_threads:
-        representative = next(span for span in host_spans if span.pid == pid and span.tid == tid)
+        lane_spans = [span for span in host_spans if span.pid == pid and span.tid == tid]
+        representative = next(
+            (span for span in lane_spans if "role" in attrs_by_id[id(span)]),
+            lane_spans[0],
+        )

Also applies to: 477-487

🤖 Prompt for AI Agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@simpler_setup/tools/strace_timing.py` around lines 425 - 436, Update the
representative-span selection for each (pid, tid) lane before calling
_host_thread_name: prefer the first span carrying a role attribute, while
retaining parse-order selection when no span has a role. Ensure facade and
scheduler roles take precedence over inferred l3.* worker labels regardless of
span parse order.

🤖 Prompt for all review comments with AI agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@simpler_setup/runtime_compiler.py`:
- Around line 347-348: Add a narrow, documented S603 noqa to the subprocess.run
call in _strip, stating that strip_bin comes from the controlled toolchain
selection; no direct change is needed at simpler_setup/runtime_compiler.py lines
347-348 beyond this suppression. Add documented S603 and S607 noqa exceptions to
the fixed test compiler invocation in tests/ut/py/test_runtime_builder.py lines
631-635, noting its test-only toolchain dependency.

In `@src/common/log/include/common/host_span.h`:
- Around line 35-50: Update the test capture sink’s implementation file to
define SIMPLER_LOG_HOST_SPAN_IMPL before including common/host_span.h, so its
simpler_log_emit_host_span definition is emitted strong. Document that test
sinks must set this macro, consistent with the existing host_log.cpp
implementation contract.

---

Nitpick comments:
In `@simpler_setup/tools/strace_timing.py`:
- Around line 425-436: Update the representative-span selection for each (pid,
tid) lane before calling _host_thread_name: prefer the first span carrying a
role attribute, while retaining parse-order selection when no span has a role.
Ensure facade and scheduler roles take precedence over inferred l3.* worker
labels regardless of span parse order.

🪄 Autofix

Fix all unresolved CodeRabbit comments on this PR:

• Push a commit to this branch (recommended)
• Create a new PR with the fixes

---

ℹ️ Review info

⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Pro Plus

Run ID: 6311d561-68de-49da-837e-5fdba48f82d0

📥 Commits

Reviewing files that changed from the base of the PR and between 89f15eb and 966d864.

📒 Files selected for processing (23)

• .claude/rules/project-layout.md
• docs/dfx/host-trace.md
• python/bindings/CMakeLists.txt
• python/bindings/task_interface.cpp
• python/simpler/__init__.py
• python/simpler/_log.py
• python/simpler/_log_preload.py
• python/simpler/task_interface.py
• python/simpler/worker.py
• simpler_setup/runtime_compiler.py
• simpler_setup/tools/README.md
• simpler_setup/tools/strace_timing.py
• src/common/hierarchical/orchestrator.cpp
• src/common/hierarchical/worker_manager.cpp
• src/common/log/host_log.cpp
• src/common/log/include/common/host_span.h
• src/common/log/include/common/host_span_scope.h
• tests/ut/cpp/CMakeLists.txt
• tests/ut/cpp/a5/test_host_log_off.cpp
• tests/ut/cpp/hierarchical/test_scheduler.cpp
• tests/ut/py/test_runtime_builder.py
• tests/ut/py/test_strace_timing.py
• tests/ut/py/test_worker/test_host_worker.py

🚧 Files skipped from review as they are similar to previous changes (8)

• src/common/hierarchical/orchestrator.cpp
• simpler_setup/tools/README.md
• tests/ut/py/test_strace_timing.py
• python/simpler/worker.py
• tests/ut/cpp/hierarchical/test_scheduler.cpp
• tests/ut/py/test_worker/test_host_worker.py
• src/common/hierarchical/worker_manager.cpp
• docs/dfx/host-trace.md

[TaoZQY]

@coderabbitai review

[coderabbitai]

✅ Action performed

Review finished.

Note: CodeRabbit is an incremental review system and does not re-review already reviewed commits. This command is applicable only when automatic reviews are paused.

[coderabbitai]

Note

GitHub couldn't provide a complete incremental comparison for this pull request, so CodeRabbit is performing a full review instead. This review may take a little longer.

[coderabbitai]

Actionable comments posted: 3

🧹 Nitpick comments (1)

simpler_setup/tools/strace_timing.py (1)

462-487: 🚀 Performance & Scalability | 🔵 Trivial | ⚡ Quick win

Group the host spans once instead of rescanning per lane.

Line 466 rescans host_spans for every pid. Line 478 rescans it for every thread. Both are O(pids × spans) and O(threads × spans). A long capture contains many spans and many worker threads, so the cost grows quadratically.

♻️ Proposed refactor to a single grouping pass

-    host_pids = sorted({span.pid for span in host_spans})
-    host_threads = sorted({(span.pid, span.tid) for span in host_spans})
+    spans_by_pid: dict = {}
+    first_by_thread: dict = {}
+    for span in host_spans:
+        spans_by_pid.setdefault(span.pid, []).append(span)
+        first_by_thread.setdefault((span.pid, span.tid), span)
+    host_pids = sorted(spans_by_pid)
+    host_threads = sorted(first_by_thread)
 
     for pid in host_pids:
-        process_spans = [span for span in host_spans if span.pid == pid]
+        process_spans = spans_by_pid[pid]
         role = "host" if any(span.name.startswith("l3.") for span in process_spans) else "chip child"
@@
     for pid, tid in host_threads:
-        representative = next(span for span in host_spans if span.pid == pid and span.tid == tid)
+        representative = first_by_thread[(pid, tid)]

🤖 Prompt for AI Agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@simpler_setup/tools/strace_timing.py` around lines 462 - 487, Refactor the
host metadata generation around host_pids and host_threads to build a single
grouping of host_spans by (pid, tid), reusing it for both process and thread
metadata. Derive each process’s spans and role from that grouped data, and
obtain each thread representative without repeated scans of host_spans; preserve
the existing event names and _host_thread_name behavior.

🤖 Prompt for all review comments with AI agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@simpler_setup/tools/strace_timing.py`:
- Around line 503-520: Update the submit/dispatch correlation around the submits
and dispatches collections to retain all submit spans for each _flow_key rather
than overwriting repeated task-slot entries. For each dispatch, select the
latest matching submit whose start timestamp is at or before the dispatch
timestamp, and use that submit as the source while preserving the existing
unmatched-dispatch behavior.
- Around line 522-548: The task dispatch events in the dispatch-event
construction should use a numeric or valid hexadecimal flow id instead of the
descriptive string built as flow_id. Preserve that descriptive identifier by
adding it to the event args, and ensure both the start and finish events reuse
the same valid id.

In `@tests/ut/py/test_runtime_builder.py`:
- Around line 636-638: Move the noqa suppression for S607 from the
subprocess.run statement line to the diagnostic line containing the "gcc"
argument, or collapse the call so the suppression covers both diagnostics;
preserve the existing subprocess.run behavior and S603 suppression.

---

Nitpick comments:
In `@simpler_setup/tools/strace_timing.py`:
- Around line 462-487: Refactor the host metadata generation around host_pids
and host_threads to build a single grouping of host_spans by (pid, tid), reusing
it for both process and thread metadata. Derive each process’s spans and role
from that grouped data, and obtain each thread representative without repeated
scans of host_spans; preserve the existing event names and _host_thread_name
behavior.

🪄 Autofix

Fix all unresolved CodeRabbit comments on this PR:

• Push a commit to this branch (recommended)
• Create a new PR with the fixes

---

ℹ️ Review info

⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Pro Plus

Run ID: dbf09d94-043f-4f91-be92-1016a6683cc2

📥 Commits

Reviewing files that changed from the base of the PR and between 60312e4 and a8bec15.

📒 Files selected for processing (23)

• .claude/rules/project-layout.md
• docs/dfx/host-trace.md
• python/bindings/CMakeLists.txt
• python/bindings/task_interface.cpp
• python/simpler/__init__.py
• python/simpler/_log.py
• python/simpler/_log_preload.py
• python/simpler/task_interface.py
• python/simpler/worker.py
• simpler_setup/runtime_compiler.py
• simpler_setup/tools/README.md
• simpler_setup/tools/strace_timing.py
• src/common/hierarchical/orchestrator.cpp
• src/common/hierarchical/worker_manager.cpp
• src/common/log/host_log.cpp
• src/common/log/include/common/host_span.h
• src/common/log/include/common/host_span_scope.h
• tests/ut/cpp/CMakeLists.txt
• tests/ut/cpp/a5/test_host_log_off.cpp
• tests/ut/cpp/hierarchical/test_scheduler.cpp
• tests/ut/py/test_runtime_builder.py
• tests/ut/py/test_strace_timing.py
• tests/ut/py/test_worker/test_host_worker.py

🚧 Files skipped from review as they are similar to previous changes (20)

• tests/ut/cpp/CMakeLists.txt
• python/simpler/init.py
• python/bindings/CMakeLists.txt
• python/simpler/worker.py
• src/common/hierarchical/orchestrator.cpp
• .claude/rules/project-layout.md
• src/common/log/include/common/host_span.h
• simpler_setup/tools/README.md
• python/simpler/_log_preload.py
• tests/ut/py/test_strace_timing.py
• tests/ut/py/test_worker/test_host_worker.py
• src/common/log/include/common/host_span_scope.h
• src/common/hierarchical/worker_manager.cpp
• tests/ut/cpp/a5/test_host_log_off.cpp
• docs/dfx/host-trace.md
• src/common/log/host_log.cpp
• tests/ut/cpp/hierarchical/test_scheduler.cpp
• python/bindings/task_interface.cpp
• python/simpler/task_interface.py
• python/simpler/_log.py

[ChaoWao]

Re-review of 009b7c86

Re-reviewed against the new merge-base 8a82bf9e. Disclosure: some of this
overlaps a refactor I had prototyped locally, so treat my read of
_log_preload.py / host_span_scope.h / place_binary as less independent than
the rest.

What got better

• The sink binding is now the right shape. Passing the entry-point address
  from Python (host_span_sink_address → _bind_host_span_sink) beats both
  dlsym(RTLD_DEFAULT) and a weak symbol: Python already holds the CDLL, and
  because the address is passed explicitly there is no load-order constraint —
  binding no longer has to precede the _task_interface import. A stale .so
  lacking the symbol yields AttributeError → address 0 → quiet disable.
• _host_spans_active() as a function rather than a module constant, so it
  re-evaluates after a later bind. Correct.
• Field encoding (encode_host_span_field) protects the record framing from
  %, [, ] and non-printables, and keeps space/= only where they are
  grammar.
• HostLogTest.ForkedProcessesEmitWholePipeRecords turns the fork-atomicity
  invariant from a comment into a test. That one is worth having on its own.
• The data race I would have raised in enqueue_dispatch is gone: in
  submit_dispatch the admission_mu_ guard covers the whole function and
  trace_run / trace_hash are captured while it is held.

Must fix

1. legacy_spans() is an allow-list and silently drops an existing marker.
simpler_setup/tools/strace_timing.py:142 keeps only simpler_run /
simpler_run.*. simpler_prewarm.build (emitted from
runtime_maker.cpp, reachable via ChipWorker.init(prewarm_config=…)) matches
neither branch, so it disappears from the TPOT table, the tree, the rounds table
and --trace-out — which the PR description says stays unchanged. Reproduced
with a two-record log: the hid=0 bucket present before is absent after.

The intent is to exclude the new family, so invert it to a deny-list on the new
prefix; that is also future-proof for any other marker family.

2. by_name() selects by stream position. strace_timing.py:108 uses
m.setdefault(s.name, s) over self.spans, which is in file order — so "first"
means first in the stream, not earliest by ts. Stream order carries no
information (records are stamped before the write and interleave across threads
and forked children), so this should select by minimum ts.

Latent today, because names happen to be unique within one (pid, inv) — the
device phases come from a static table and task_slot_0…15 are separately
spelled. It stops being latent the moment a per-task family reaches
group_invocations, which is exactly what removing the allow-list in (1) would
do.

3. The parent's log level is never seeded. _initialize_simpler_log has one
caller — python/simpler/task_interface.py:1338, inside ChipWorker.init, which
runs in the forked child. worker.py never calls it, and _log.py binds the sink
at import but never calls simpler_log_init. So the parent's HostLogger stays
at its constructor default TIMING and the Python threshold does not reach it:
setting the simpler logger to NUL does not silence parent-side spans.

Should fix

4. Two atomicity premises in one file. src/common/log/host_log.cpp:45
reasons from _POSIX_PIPE_BUF (512) and budgets span fields to 256 bytes;
:211 says "a record of at most PIPE_BUF bytes reaches a shared pipe
indivisibly" with a 2048-byte buffer. The intent looks deliberate — the portable
512 floor for machine-parsed span records, best-effort for human log lines — but
nothing says so, so a reader cannot tell which bound governs where.

5. _log.py's handler discards a good value.

try:
    from _task_interface import DEFAULT_LOG_THRESHOLD as _NATIVE_DEFAULT
    from _task_interface import _bind_host_span_sink
    _bind_host_span_sink(_host_span_sink_address(_host_log_handle))
except (ImportError, AttributeError):
    _NATIVE_DEFAULT = 25

If the first import succeeds and the second fails — an extension predating
_bind_host_span_sink — the handler overwrites the successfully imported native
constant with the hardcoded 25. It conflates "no extension" with "extension lacks
this symbol".

6. The ordering comment now over-claims. _log.py states the preload is
ordered before the _task_interface import "so the logger entry point is bound
into the extension before any worker can fork". With address passing that
constraint no longer exists; the invariant that actually holds is only "bind
before the first span is emitted".

Consider

7. sink_slot() is per module, not per process. It is a function-local static
in an inline function, and with -fvisibility=hidden on the extension plus
-fno-gnu-unique on the logger each including module gets its own slot. Only
_task_interface and test_scheduler include the header today, so nothing
breaks — but the docstrings say "process-global", which will mislead whoever first
makes host_runtime.so emit a span.

8. The encoder protects record framing, not k=v framing. It runs on the
already-joined attribute string, so it cannot tell a separator space from a space
inside a value: error_message=foo bar encodes unchanged and then parses as
error_message=foo plus a stray bar. Unreachable today (all attributes are
numeric or enum), but the 192-byte cap sits on the same open k=v payload that is
meant to be the extension point, so the first free-text attribute meets both
limits at once.

9. The race fix lengthened a critical section. emit — and
trace_dispatch_attrs's ostringstream — now run while admission_mu_ is held.
Correctness was the right trade; the cheaper shape is to capture the two scalars
under the lock and format/write outside it.

Question

What is the per-record cost on the default build? SIMPLER_HOST_STRACE defaults
on and HostLogger's default threshold is TIMING, so a stock build now pays
roughly four write(2) calls per task serialized on the logger mutex, where every
pre-existing host marker was per chip-run. A before/after on an L3 example with a
non-trivial task count would settle whether that needs addressing before merge.

---

I have not built or run the suites against this head, so everything above is from
reading the diff plus the surrounding files. Happy to rebuild and run if that
would help.

[TaoZQY]

@ChaoWao Addressed the review points in the updated single commit:

• legacy_spans() now denies only the new l3.* namespace, so existing families such as simpler_prewarm.build remain in TPOT/tree/rounds/--trace-out.
• Invocation.by_name() now selects the minimum timestamp rather than file order.
• hierarchical Worker.init() seeds the parent logger with the snapshotted Python threshold before its first fork; the regression test pins NUL propagation and call ordering.
• split the native-default and optional sink-binder imports, and corrected the preload/binding comments.
• clarified the portable 512-byte machine-record atomicity budget versus the 2048-byte formatting buffer and best-effort human log records.
• corrected docs/binding text to describe an extension-local/DSO-local sink slot connected to the process-global logger.
• l3.dispatch captures run/hash while admission is locked, then formats and emits after releasing admission_mu_.

I left item 8 unchanged because all current attributes are numeric/enums; there is no free-text k=v producer yet. The first free-text attribute should introduce value-level encoding rather than encoding an already-joined payload.

Validation: 1340 Python UT passed (6 skipped), 93 non-hardware C++ tests passed, pre-commit passed, and both one-device and two-device L3 a2a3sim scenes passed.

For the performance question, local hardware precheck could not identify the silicon, so I did not bypass it. I compared merge-base 8a82bf9e with this head on the same host using an a2a3sim L3 workload (10 child tasks/round, 2 warmups + 15 measured rounds, 5 fresh processes/version). Median-of-process-medians was 23.28 ms baseline vs 24.04 ms current (+3.3%, about 76.7 us/task), while process medians overlapped broadly: 21.76–27.56 ms baseline and 22.37–26.11 ms current. The paired directions were mixed, so this does not show a stable reproducible regression, but it is noisy host-simulation data rather than a hardware conclusion. A same-device onboard comparison remains the definitive follow-up when an identifiable device is available.

[TaoZQY]

@ChaoWao Follow-up with an onboard L3 before/after measurement on the current head.

Setup

• Baseline: 50c06606 (main at the comparison point)
• PR: 9e72937e
• Workload: the existing per_task_runtime_env vector-add kernel under a real Worker(level=3), with 64 independent L2 child submissions per worker.run
• Same NPU device (9) for every process
• Order-balanced runs: baseline → PR and PR → baseline
• Per version/direction: 20 warmups + 50 measured rounds; outputs validated after every process
• Default host tracing/logging configuration (no tracing or log-level overrides)
• Both task-submit jobs completed with exit code 0:
  • task_20260812_004308_364535012115
  • task_20260812_004347_3687788815

The architecture precheck could not query npu-smi because DCMI returned -8005; this run therefore explicitly bypassed that precheck and relied on the successful a2a3 workload execution.

Results

Order	Build	Mean (ms/run)	Median (ms/run)	Stddev (ms)
baseline → PR	baseline	87.518	74.785	49.889
baseline → PR	PR	88.711	75.686	54.410
PR → baseline	PR	81.728	78.134	12.846
PR → baseline	baseline	76.058	75.907	2.752
pooled (100 samples/build)	baseline	81.788	75.160	35.792
pooled (100 samples/build)	PR	85.219	76.943	39.685

The order-balanced pooled median delta is +1.783 ms/run (+2.37%). With 64 child tasks per run, that is a rough end-to-end estimate of +27.9 us per child task.

The PR logs also confirm the exact marker volume. In each 70-round PR process (20 warmup + 50 measured), they contain:

• l3.graph_build: 70
• l3.dispatch: 4,480
• l3.frame_submit: 4,480
• l3.complete: 4,480
• l3.activate: 0

This independent/immediate-dispatch workload therefore emits 193 host-span records per run (1 + 64 * 3), giving a rough measured estimate of 9.24 us per emitted record from the pooled median delta. l3.activate is only emitted for staged/prepare-only activation and is not exercised by this workload, so a four-record staged path was not directly measured; a linear extrapolation from this result would be about 37 us/task, not an observed number.

The means are more outlier-sensitive than the medians here, but both order directions show a positive median delta (+1.2% and +2.9%). So the default-on tracing cost is measurable on this L3 workload, although it is small relative to the full task latency.