Skip to content

Implement Milestone 3: execution core - #11

Open
PePoDev wants to merge 21 commits into
m2/catalog-and-studiofrom
m3/execution-core
Open

PePoDev wants to merge 21 commits into
m2/catalog-and-studiofrom
m3/execution-core

Conversation

@PePoDev

@PePoDev PePoDev commented Oct 26, 2026

Copy link
Copy Markdown
Owner

Summary

Implements Milestone 3 (execution core) by following docs/superpowers/plans/2026-10-27-execution-core.md, which is also in this PR. The first commit adds the plan; each of the other 20 commits is one plan task, in plan order.

Stacked on #10 (Milestone 2). Review and merge #10 first; this PR's base is m2/catalog-and-studio.

  • Tasks 1-2: graceful API shutdown, and multi-stage release images that run as uid 10001.
  • Tasks 3-7:
    • domain run rules: states, transitions, aggregation, retry classification and run plans
    • runtime input resolution and output validation
    • the executor contract (plugin-sdk) and checked registry (plugin-host)
    • the mock executor
    • the trusted shell executor (process-group kill, env allowlist, bounded logs)
  • Tasks 8-14:
    • runs, node runs, events and workers with tenant isolation
    • worker-only SECURITY DEFINER functions, fenced by worker id and lease token
    • one SQL routine for every node outcome (retry with backoff, exactly-once release, transitive skip, run settlement)
    • the run API (start, inspect, cancel, rerun)
    • admin opt-in for the shell executor
    • leader-elected lease recovery
    • replayable SSE (Last-Event-ID, gapless per-run sequence, LISTEN/NOTIFY fan-out)
    • the worker process (capacity, heartbeats, deadline, user cancel and shutdown handled separately, log draining)
  • Tasks 15-17:
    • a Run button and input form in the studio, plus a recent-runs and logs drawer
    • a live run page with graph overlays, node details and logs
    • organization settings for the shell executor with a trusted-user warning
  • Task 18: a worker service in Compose (non-root, read-only, healthcheck), CI scaling to 3 workers, and an extended end-to-end smoke test.
  • Tasks 19-20: fixes for issues an adversarial review found:
    • The shell executor now needs the operator switch CORQTY_SHELL_EXECUTOR=enabled before any organization admin can enable it.
    • SSE streams are capped (256 per process, 8 per user).
    • Scratch directories are kept until the outcome is reported.
    • Node inputs appear in live events, and live views keep reconnecting and stay accurate.

Notes

  • The shell executor runs as the shared worker user. Enabling CORQTY_SHELL_EXECUTOR is for single-tenant or fully trusted deployments only (see README).
  • Run, node-input and node-output limits are 1 MiB each, and logs are limited to 10 MiB per node run; these stay fixed until Milestone 5.

Test plan

  • cargo fmt --check, cargo clippy --workspace --all-targets -- -D warnings
  • cargo test --workspace (176 tests)
  • Linux shell executor contract tests in rust:1.96-bookworm (15 tests)
  • pnpm web:lint, pnpm web:test (120 tests), pnpm web:build
  • CORQTY_WORKER_CAPACITY=1 CORQTY_SHELL_EXECUTOR=enabled docker compose up --build --wait --scale worker=3, then the smoke script with -ExpectedWorkers 3. It covers runs end to end, parallel branches on different workers, failure skipping, SSE through the Next.js rewrite, and the shell JSON round trip.
  • For every plan task, its section applied literally to the parent commit reproduces that commit byte for byte.
  • CI green on Linux

🤖 Generated with Claude Code

PePoDev and others added 21 commits October 27, 2026 01:50
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…main

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…rker-only functions

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant