feat(runtime): Add persistent L3 distributed execution

[high-cloud]

Summary

• add opt-in DistributedCompiledProgram.prepare(persistent=True) execution backed by one long-running L3 Worker.run()
• let generated HOST orchestration accept an internal domain provider so CommDomains can be retained per compiled program
• restore retained windows to fresh zero-filled semantics before reuse
• preserve the synchronous request boundary, TaskArgs lifetime, per-request cleanup, error propagation, and final domain release
• document the lifecycle and keep English and Chinese developer docs aligned

Motivation

DeepSeek decode repeatedly enters Worker.run() and allocates/releases the same communication domains. For small decode DAGs, that host-side lifecycle can cost more than the effective device work. Persistent mode moves the L3 orchestration and physical CommDomain allocation to the prepared-worker lifetime while keeping normal dispatch as the default.

Compatibility and limitations

• default prepare() behavior is unchanged
• persistent mode requires regenerated HOST orchestration artifacts with the internal _domain_provider hook
• different compiled programs retain isolated domains even when generated names match
• no Simpler source changes are included
• the current implementation still relies on Simpler's private drain and Worker cleanup APIs; a future public request-boundary API should encapsulate this sequence

Validation

• native worktree build: passed
• full unit suite: 7397 passed, 2 skipped
• focused runtime/codegen tests: 122 passed
• ruff check and ruff format --check: passed
• git diff --check: passed
• clang-tidy was attempted; the local clang-tidy 18 differs from the repository's expected 21 and reports pre-existing unchecked-optional-access findings across distributed_codegen.cpp

Relate

#2069

[gemini-code-assist]

Warning

You have reached your daily quota limit. Please wait up to 24 hours and I will start processing your requests again!

[coderabbitai]

Important

Review skipped

Auto incremental reviews are disabled on this repository.

Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Pro

Run ID: e09e0937-3260-4863-8a1b-b958136fb355

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

• 🔍 Trigger review

📝 Walkthrough

Walkthrough

Persistent prepared workers gain an opt-in persistent=True mode that queues dispatches, retains and zeroes communication domains, and cleans up on close. HOST orchestration code generation accepts a domain provider, with runtime, codegen, documentation, and lifecycle tests updated accordingly.

Changes

Persistent L3 execution

Layer / File(s)	Summary
Persistent worker dispatch python/pypto/runtime/distributed_runner.py	DistributedWorker adds queued persistent dispatch, per-program retained domains, synchronous zeroing and drain boundaries, error propagation, and shutdown cleanup.
Prepared-worker API and integration python/pypto/ir/distributed_compiled_program.py, tests/ut/runtime/test_distributed_worker.py	prepare() accepts and forwards persistent=True; tests cover forwarding, lifecycle, reuse, isolation, cleanup, and errors.
Domain-provider code generation and validation src/codegen/distributed/distributed_codegen.cpp, tests/ut/codegen/distributed/test_host_orch_distributed.py, docs/en/dev/06-persistent-l3.md, docs/zh-cn/dev/06-persistent-l3.md	Generated HOST entries accept _domain_provider, communication-domain scopes use it when available, and documentation describes persistent lifecycle, fresh-window semantics, isolation, serialization, and runtime dependencies.

Estimated code review effort: 4 (Complex) | ~60 minutes

Sequence Diagram(s)

sequenceDiagram
  participant Caller
  participant DistributedWorker
  participant HostOrchestrator
  participant CommDomain
  Caller->>DistributedWorker: run(compiled, ...)
  DistributedWorker->>HostOrchestrator: dispatch through persistent queue
  HostOrchestrator->>CommDomain: retain or allocate via _domain_provider
  HostOrchestrator-->>DistributedWorker: complete after drain
  DistributedWorker-->>Caller: return result or propagate error
  Caller->>DistributedWorker: close()
  DistributedWorker->>CommDomain: release retained domains

Loading

Poem

A rabbit hops through queues of light,
Domains stay warm through every flight.
Zeroed windows, clean and bright,
Providers guide the orchestration right.
Close the worker—leases take flight!

🚥 Pre-merge checks | ✅ 4 | ❌ 1

❌ Failed checks (1 warning)

Check name	Status	Explanation	Resolution
Docstring Coverage	⚠️ Warning	Docstring coverage is 22.64% which is insufficient. The required threshold is 80.00%.	Write docstrings for the functions missing them to satisfy the coverage threshold.

✅ Passed checks (4 passed)

Check name	Status	Explanation
Title check	✅ Passed	The title clearly summarizes the main change: adding persistent L3 distributed execution.
Description check	✅ Passed	The description is directly related to the changeset and matches the persistent execution work.
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}

[gemini-code-assist]

Warning

You have reached your daily quota limit. Please wait up to 24 hours and I will start processing your requests again!

[coderabbitai]

Actionable comments posted: 1

🤖 Prompt for all review comments with AI agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@src/codegen/distributed/distributed_codegen.cpp`:
- Around line 290-294: Update the generated function body for functions whose
level is at least 3 to forward _domain_provider in direct
orchestrator-to-orchestrator calls. Reuse the existing _domain_provider
parameter emitted by the signature construction near func->name_, ensuring
nested host orchestrators preserve the caller’s communication-domain scope
instead of allocating a transient domain.

🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

• Push a commit to this branch (recommended)
• Create a new PR with the fixes

---

ℹ️ Review info

⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Pro

Run ID: 1a8d3056-b25b-435d-bc94-39ea22bdf577

📥 Commits

Reviewing files that changed from the base of the PR and between 2b3f017 and 2972d5a.

📒 Files selected for processing (7)

• docs/en/dev/06-persistent-l3.md
• docs/zh-cn/dev/06-persistent-l3.md
• python/pypto/ir/distributed_compiled_program.py
• python/pypto/runtime/distributed_runner.py
• src/codegen/distributed/distributed_codegen.cpp
• tests/ut/codegen/distributed/test_host_orch_distributed.py
• tests/ut/runtime/test_distributed_worker.py

[chatgpt-codex-connector]

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 2972d5a029

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

[YunjiQin]

Approve.

设计合理、CI 全绿、测试覆盖到位(prepare flag 透传、缺 hook 拒绝、复用+清零、TaskArgs 生命周期、多程序隔离、错误传播、错误等待 outer cleanup)。codegen 门控用 level_.value_or(AIV) >= 3 双重判断,默认路径行为不变,向后兼容。

对持久模式下'主线程内存原语 vs 后台常驻 Worker.run'的跨线程担忧,已核对 runtime 实现后消解:malloc/free/copy_to 走 control mailbox 控制面(mailbox_mu_ 全程持锁),与 scope/drain/ring 任务面互不相干,不存在记账污染;且请求间后台线程阻塞于 queue.get、子 worker 静止,close() 中的 owned-tensor 回收命中空闲态。

低危收尾建议(非阻塞,可后续跟进):

• close() 里将 _stop_persistent_dispatcher() 移到 _close_owned_tensors() 之前,使正确性不依赖'请求间后台空闲'这条隐式不变量;或在文档/注释中显式声明该不变量。
• 依赖 Simpler 私有 drain / Worker cleanup API 属临时方案,PR 描述已声明,待未来公开 request-boundary API 封装。