Skip to content

test(a09): 降低分发校验内存占用并隔离构建租约 - #878

Merged
LIghtJUNction merged 5 commits into
wip/rust-core-go-extensionsfrom
a09/distribution-verifier-build-lease
Oct 12, 2026
Merged

LIghtJUNction merged 5 commits into
wip/rust-core-go-extensionsfrom
a09/distribution-verifier-build-lease

Conversation

@LIghtJUNction

@LIghtJUNction LIghtJUNction commented Oct 12, 2026 •

Copy link
Copy Markdown
Member

开发主线交接(2026-10-12)

开发默认主线现为 wip/rust-core-go-extensions;旧单体 main 与 traditional 冻结于 3948a0a1ee1cefdd016a7aa45a11851545bb4b85。分支交接不是发布、部署或验收通过。原问题、源码基线、测试与未完成项保留;旧单体路径仅作历史证据,后续先在微服务源码重新确认并按职责移植,不回灌旧数据库/权限/资金实现。

主线/S11 跟踪:#895。Extore 商品同步:#894,未验收,禁止部署,尤其禁止合并/推送 Extore main。

本 PR 的候选实现和已有验证范围保持原样;不因主线交接自动合并或标记验收完成。

S11 review update — 2026-10-12

Existing PR updated in place. Still Draft; neither A09 nor S11 is complete.

New commit: 478f658cd7d550f7813f12e8a13630d0c8afb89e, parent fbfd531c91d367fd3475c75ece7ec323c35dec0f. The update changes only tests/architecture_completion/build_lease.py and adds test_build_lease_descendants.py (+253/-13). The distribution verifier and the ten original lease tests are unchanged.

Reproduced defect and repair

The original lease runner waited only for its direct child. A launcher could exit while a worker still ran, or close inherited file descriptors before the wrapper died, allowing a second heavy build to acquire the slot prematurely.

A private, short-lived Linux subreaper now retains the lock and waits for local descendants. It runs in a fresh Python interpreter and separate session. Commands travel over a private pipe, not supervisor arguments or receipts; receipts contain hashes/error types rather than command secrets. It preserves the direct command's exit status and never scans or kills unrelated processes. This remains a test utility, not an additional product program.

Checks actually run for this update

Environment: Linux, Python 3.13.5, selected exact source files in /mnt/data/s11-review; not the original project workspace or its mise toolchain.

python -W error::DeprecationWarning -B -m unittest discover \
  -s tests/architecture_completion -p 'test_build_lease*.py' -v
  • Original source: all ten original tests passed; all three new process-tree regression cases failed (TimeoutError not raised).
  • Final source: 15 unique tests passed, no skips, exit code 0. Includes wrapper death, launcher exit, closed inherited descriptors, private spawn failure, signed signal status and an unrelated child left untouched.
  • An intermediate fork-based candidate emitted a multithreaded-fork deprecation warning. It was replaced by the fresh-interpreter runner; the final run treats that warning as an error. The intermediate result is not final acceptance.
  • Final Python AST and trailing-whitespace checks passed.
  • Remote final blobs match the tested local bytes: helper 35175e1050d97095ee4de08124f4fc40970dca6b; new tests c66ac226c6cab74efe33d4138236c8b28cc678ec. The original test blob remains 336b13df1c31a36d74b8c742cd9f677abe31d2bf.
  • The existing branch was advanced without force, with the expected previous head checked, and PR head read back.

Limits retained

This protects cooperative local descendant processes. It does not protect against killing the supervisor itself, supervise jobs submitted to an external Docker daemon, prove database writer exclusion, or enforce CPU/memory limits. Do not use its receipts as evidence for those properties.

No Rust/Go/Web build, original-host mise execution, PostgreSQL, actual Docker/native install, snapshot migration, live provider operation, release or deployment was run in this S11 review. Distribution measurements below are historical A09 evidence, not rerun by S11. Full architecture integration remains blocked on the domain contracts and required environment; see #675.


Initial A09 candidate — historical scope and evidence

Submit the verified offsite A09 code candidate only, not a claim that the full A09 architecture task is complete.

  • Stream and bound distribution archive verification rather than retaining decompressed package data.
  • Validate component resource inventory, binary headers/architecture, manifest size and duplicate JSON keys.
  • Add regression tests for malformed/missing/extra content and byte-identical archive compatibility.
  • Add a test-only, cooperative Linux build lease with independent Cargo output directories and process-exit checks. This does not implement a PostgreSQL writer fence or recovery CAS.

The initial four blob IDs at fbfd531c91d367fd3475c75ece7ec323c35dec0f matched the delivered A09 patch exactly. Based on WIP commit a28f06e88b7adf3701937f2a4018689bdc96ad85 (the modified verifier was unchanged from the earlier review baseline). The S11 update above supersedes the initial lease helper.

Checks performed before PR

Offsite sandbox (Python 3.13.5):

  • 21 distribution regression tests: PASS.
  • 10 build lease/process tests: PASS.
  • 11 synthetic component/platform archive byte comparisons: PASS.
  • 192 MiB synthetic archive verification peak RSS: about 270 MiB before vs 18 MiB after; with 96 MiB address-space limit, old failed and new passed. This benchmark concerns the verifier only.

These checks used an offsite Python environment, not the required host's pinned mise toolchain. No real product, database, container or native installation tests were run here. The original failing runs and methodology are recorded in the A09 handoff, not committed to the source tree.

Remaining A09 work — BLOCKED / PENDING

  • Isolated PostgreSQL resource allocations and ten owners' frozen interface contracts.
  • Core schema maintenance, encrypted backup/independent restore, dead-holder CAS, Core-specific pgroll evaluation.
  • Final cold Docker/native package builds, distribution install/upgrade/rollback tests and bounded 1C1G measurements.

S11 must obtain the original workspace's private handoff and rerun the required combined checks. No production deployment, release, funds, or database mutation is authorized.

Target remains the active WIP branch rather than main; this is not a production-ready architecture.


A09 recovery increment (2026-10-12)

Pushed as commit 92d8f8e on the same PR branch, preserving the earlier offsite candidate and the S11 descendant build-lease fix.

Six new test-only files under tests/architecture_completion/:

  • backup.py and test_backup.py: bounded AES-256-GCM authenticated synthetic dumps; separate private key; source/target binding, digest, nonce, size, exclusive outputs and corruption tests.
  • pg_fixture.py and test_pg_fixture.py: owned ephemeral local Docker PostgreSQL fixture contracts; per-agent independent databases and roles; strict source/target identity, no external DSN, no arbitrary cleanup or automatic retry of unknown mutations.
  • run_postgres_recovery.py: synthetic dump → encryption → full authentication → independent empty-target restore → catalog/data/permission checks. The runner is not a Core migration or production recovery tool.
  • README.md: test commands and explicit limitations.

Rechecked here on Python 3.13.5 with cryptography 46.0.4: 18 crypto tests and 18 mocked Docker/permissions tests passed using the packaged owned-source snapshot and the earlier lease helper. Their six Git blob SHA-1 IDs exactly match the original frozen A09 handoff; no other files were modified by this commit.

BLOCKED: no local Docker socket, PostgreSQL, pgroll, nFPM or mise in this execution environment; no real Docker/PostgreSQL backup/restore, Core writer fence, dead-holder CAS, full-scheme migration, actual package install or 1C1G load test was run. The overall A09 work remains incomplete and this PR remains Draft. S11 must re-run the real integrated acceptance against the frozen final source. No deployment or merge was performed.

Protect package verification against unbounded member allocations, invalid
binary headers, and malformed or incomplete manifests. Add test-only Linux
process build leases with independent Cargo targets and fail-closed timeout.

A09 remains blocked for real PG, recovery, pgroll and deployment acceptance.
S11 reproduced early slot release after a launcher exits or its wrapper is
killed while workers remain alive. Supervise the local process tree in a
fresh interpreter and private session; keep the kernel lock until adopted
children exit, even when workers close inherited file descriptors.

Add five regressions, retaining all ten existing lease tests. On the prior
source the three process-tree regressions fail; the final source passes all
15 tests on Linux/Python 3.13.5. Exact-byte hashes checked separately.

This updates the existing A09 candidate only. No Core/Go/Web wiring, PG,
Docker daemon jobs, original-host toolchain, migration or S11 completion
is claimed. No workflow, deployment or release changes. [skip ci]
Add authenticated private AES-GCM backup helpers, an owned per-task Docker
PostgreSQL fixture and a one-shot synthetic dump/independent restore probe.
Exercise denial rules, source/target binding, damaged archive rejection,
role scope, and no replay after ambiguous restore.

No real Docker/PG or Core schema qualification is claimed. Keep this A09
increment on the draft PR for S11's final integrated acceptance.
Synchronize wip/rust-core-go-extensions at 6c31b08 into PR #878 without rewriting history. Preserve all three A09 commits and all seven new development-mainline commits. The two sides change disjoint paths; no historical main implementation is imported.

This is source synchronization, not A09 or S11 acceptance. Refs #895.
Keep the source snapshot workflow manual and read-only, preserving self-contained source bundles without automatic Git writes. Build the actual Go host even when an independent check failed, as the Rust stream tests already continue on non-cancellation. Exercise the current file-only Compose contract instead of removed environment settings.

Resolve the migration relation OID before switching to the restricted runtime identity. Check table and column writes through that OID and retain the lack of schema access; do not grant new runtime privileges.

Local final sources: 10 workflow checks, 11 fixture checks and 19 core boundary checks passed; one Compose-client check skipped because Docker is unavailable. Real Docker/PostgreSQL and the full Rust/Go pipeline still require CI. The unrelated local-release test attempt timed out and is not counted as passing. No product deployment, release or live database operation.

Continue existing PR #878; refs #895.

Copy link
Copy Markdown
Member Author

S11 同步与 CI 修复 — 2026-10-12

继续更新本 PR,没有新建 PR、合并 PR 或部署。

已推送

  • 3eaa716ceeee51a68a775b8f116a2a793a3e100e:将当前开发主线 wip/rust-core-go-extensions@6c31b08c4af8647f8f0d0fa63dbeabdd2cb4f7dc 同步到本分支。使用保留历史的双父提交,非 rebase、非强推。原 A09 三个提交及开发主线七个新增提交全部保留。旧 main / traditional@3948a0a1 已冻结,没有回灌旧单体实现。
  • 7b81ce1b5f64716b38d44f2b453e628ea7d910bd:追加五文件修复,所有上传 blob 均与最终本地测试文件的 Git blob hash 一致。

修复依据与改动

当前主线失败运行 #38242743997 的 workflow/source 检查先失败,随后实际 Go host 构建被跳过,而 Rust 测试仍继续,导致 LMM_RELAY_EXTENSION_BIN 缺失。另有 Docker/RPC 测试报 permission denied for schema core_migration。

  1. 快照工作流改成手动、只读,移除自动 Git 写入;保留含主线与前置 PR 的独立源码 bundle。现有禁止发布/部署的检查不放宽。
  2. Go host 构建与后续 Rust 测试采用一致的非取消条件;PR 检查覆盖工作流文件变化。独立 Core 检查改用现有 TOML/Compose 测试,删除已失效的环境变量断言,补充资源上限检查。
  3. 迁移权限断言在切换运行身份前取得表 OID,切换后通过 OID 检查权限。保留无 schema 访问权,并增加列写权限检查;没有新增 GRANT、放宽角色权限或修改产品数据库实现。

本次实际验证

Linux,Python 3.13.5,Node v22.16.0;不是原主机 mise 工具链。修改前四个已有文件与固定 6c31b08c 源码产物逐字节一致,快照工作流也来自该版本。

node --test scripts/workflow-topology.test.mjs
python -B scripts/test_core_rpc_fixture.py -v
python -B scripts/test-core-boundaries.py
  • 当前主线拓扑问题复现:原 8 项中 6 通过、2 失败。
  • 最终文件复测:工作流 10 通过;fixture 11 通过、1 跳过;Core 职责检查 19 通过。合计 40 通过、1 跳过,未重复累计。
  • 跳过原因:本地缺少 Docker Compose 客户端。权限 SQL 的本地测试是源码断言,不是 PostgreSQL 执行证据。
  • Python AST、YAML 解析、差异空白检查通过。另一次无关 test-local-release-tests.py 尝试超时,未计为通过,也未据此删除或跳过 CI 测试。

最终提交已经触发微内核 CI #38264218436 和 Linux 打包 CI #38264218405;本条记录时二者仍在运行,未取得新提交完整 CI 通过结果。

A09 与 S11 仍未整体验收。真实 Docker/PostgreSQL、最终 Rust/Go 组合、业务前置项集成、迁移及安装/回滚仍需完成。跟踪 #895。

Copy link
Copy Markdown
Member Author

补充已终结的 CI 结果:job 114862160342 于 14:47 UTC 失败,实际源码为 a4317db5ee3dd07ca388c6d47b64e75ca04f43f4(7b81ce1b 合入 158c8ecd)。两处可定位阻塞:

  1. scripts/test_core_rpc_fixture.py:249 收到 Compose 字符串 "268435456",与整数 268435456 比较失败。建议严格规范化数值表示,保留精确 256 MiB 和其余资源/权限断言;此失败不证明容器没有内存上限。
  2. scripts/test-core-rpc-docker.py 的权限探针已提前解析 core_migration 的 OID,但仍在 SET SESSION AUTHORIZATION lmm_core_runtime 后按名称解析 core_charging.requests,先触发 schema 权限拒绝。建议同样在切换前取得该关系 OID,继续验证表级与列级禁写,不为通过测试扩大 runtime 权限。

这是日志/源码核查,未本地复跑;不能由探针失败推断实际 RPC 产品缺陷。arm64 DBus 链接失败仍是另一项阻塞;成功的 Rust/Go/PG 子步骤不等于整体 A09/S11 已通过。

LIghtJUNction added a commit that referenced this pull request Oct 12, 2026
…ip ci]

Review and adapt PR #878 topics fbfd531, 478f658, 92d8f8e and 7b81ce1, authored by LIghtJUNction. Bound tar metadata and preserve private recovery storage policy. 24 verifier, 14 builder, 52 architecture, 12 RPC fixture and 10 workflow tests pass on host. Actual container/Core recovery remains unverified; no CI dispatch or deployment.
@LIghtJUNction
LIghtJUNction merged commit ebf44a8 into wip/rust-core-go-extensions Oct 12, 2026
1 of 3 checks passed
@LIghtJUNction

Copy link
Copy Markdown
Member Author

已合入微服务主线,GitHub 已读回 MERGED,合并提交 ebf44a8 保留本机验证的 9831bd9 及加强:tar 元数据大小/递归边界、same-filesystem 硬链接反例、Compose 字符串内存值、统一 Backups 内分离私钥。16 个主题文件均已审查;源快照改为手动只读,未触发远程 CI。实际主机 Python 3.14.7 下 verifier24、builder14、architecture52、RPC fixture12 与 workflow10 项通过,旧实现负对照也记录。没有把 mocked Docker/权限测试标成真实容器、Core 恢复或资金验收;完整 A09/S11 仍未完成,未部署生产。

@LIghtJUNction
LIghtJUNction deleted the a09/distribution-verifier-build-lease branch November 5, 2026 07:31
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant