feat(scripts): concurrent runs, results, progress and cancel (#1843)

[cjimti]

Seven tickets and one filter fix on the managed-script run path, touching run_script, manage_script (help, validate, get_run, and the new cancel_run), platform.query, platform.call, trino_export / api_export / graphql_export inside a run, the run worker, POST /api/v1/portal/scripts/{id}/runs, the new POST /api/v1/portal/scripts/{id}/runs/{runID}/cancel, and dev/start.sh. Eight changes:

1. Make the script run worker's concurrency and run timeout configurable (default 4 concurrent runs, 15m timeout) #1843 Runs execute concurrently, admitted by the replica's load.
2. Scripts: synchronous runs over HTTP and a return value (platform.return) so a run can hand data back #1845 platform.result hands a value back, and the portal run route can wait for it.
3. Script runs: progress (platform.progress), live log and cancel #1847 platform.progress, the log so far on a running run, and cancel.
4. platform.query rows lose SELECT column order (keys come back alphabetized) #1852 Query rows keep the SELECT's column order.
5. manage_script validate reports an f-string for a string literal "f" #1853 validate's Python-ism checks read code, not string contents or comments.
6. Record trino_export / api_export assets written inside a run as run outputs #1854 Export tools inside a run are run outputs and version one asset per name.
7. fix(dev): probe_bind reports a port busy on lingering TIME_WAIT #1841 probe_bind sets SO_REUSEADDR.
8. The runs listing's run_status filter admits skipped_overlap (previously unfilterable) and canceled.

1. Concurrent runs (#1843)

A replica executed one run at a time; its capacity was its replica count. The worker now launches each claimed run on its own goroutine under its own context, and decides before every claim whether to take another (internal/platform/scriptadmit, internal/platform/scriptexec/worker.go).

• Adaptive (default).
  • A run is claimed while fewer than min_concurrency (1) are executing, or while fewer than max_concurrency (16) are executing and the replica has headroom.
  • Headroom means memory under max_memory_percent (70) of the container's cgroup limit or GOMEMLIMIT, and CPU under max_cpu_percent (75) of the cgroup quota or GOMAXPROCS.
  • Memory is Go runtime memory from runtime/metrics; CPU is getrusage over a one-second window (internal/procload). The runtime's /cpu/classes metrics are not used: they update only at GC.
  • A measurement that cannot be taken (no memory limit, CPU on Windows) never refuses.
  • Nothing is refused or failed for want of room: runs stay queued for this replica or another (SKIP LOCKED).
• Shedding. Past shed_memory_percent (90), the worker stops its most recently started run and requeues it with a memory-pressure reason. That spends the platform's retry budget, not the script's. The last run is never shed.
• Fixed. concurrency: N claims N at once; 1 reproduces the previous worker. A value that is neither adaptive nor a whole number fails config validation at startup.
• Ceilings.
  • run_timeout defaults to 15m (was a fixed 10m); max_steps, max_query_rows and result_max_bytes are configurable too.
  • manage_script help reports them under limits.
  • The claim lease is derived from the timeout (scriptexec.LeaseFor, timeout plus 5m), so a long run is not claimed twice.
• Metrics. script_run_admission_refusals_total{reason=ceiling|memory|cpu} (counted when the queue held work) and script_run_queue_wait_seconds, beside the existing script_runs_running.

scripts:
  worker:
    concurrency: adaptive
    max_concurrency: 16
    min_concurrency: 1
    max_memory_percent: 70
    max_cpu_percent: 75
    shed_memory_percent: 90
    run_timeout: 15m

run_script and run_draft stay refused from inside a run. The stated reason changes from "one run at a time" to what still holds: a run waiting on a run it queued holds a worker slot, and waiters can hold every slot a replica admits.

2. platform.result and waiting over HTTP (#1845)

• The binding.
  • platform.result(value) sets one JSON value on the run: capped at result_max_bytes (1 MiB), set once, stored in script_runs.result.
  • A second call, a non-JSON value, or an oversized one fails the run with the reason.
  • The binding is result, not return: return is a Starlark keyword and platform.return(...) does not parse.
• Where it appears. run_script, get_run and drafts carry it as result.
• Waiting over HTTP.
  • POST /api/v1/portal/scripts/{id}/runs?wait=N holds the request up to N seconds (capped at 300, the same cap as run_script's wait_seconds).
  • A run that finishes in time answers 200 with the run detail. Otherwise, and without wait, the route answers 202 with the run id as before.
  • run_script and the route share one wait, runcontrol.AwaitRun.

3. Progress, live log, cancel (#1847)

• Live progress and log.
  • platform.progress(message, done=None, total=None) records the latest report in memory (internal/platform/scriptlive).
  • Every two seconds the runner writes the latest report and the log printed so far to the run row, fenced on the run's lease. The write is RunStore.RecordProgress; an unchanged snapshot leaves the row alone.
  • The live log is the run's log itself, one buffer the interpreter prints into and the reporter snapshots.
• Cancel. manage_script command=cancel_run and POST /api/v1/portal/scripts/{id}/runs/{runID}/cancel are allowed to whoever may read the run: owner, administrator, requester.
  • RunStore.CancelRun locks the row, finishes a pending run as canceled in the same statement (never claimed), or stamps cancel_requested_at/_by on a running one.
  • It returns the status the run had, which the surfaces word through runcontrol.CancelMessage.
  • The worker holding a running run reads the flag back from its next progress write and cancels that run's context. The run ends canceled within seconds, keeping the outputs it wrote.
  • A worker that claims a run whose previous worker died after a cancel request finishes it canceled without executing.
  • A progress write refused because the lease moved stops the stale execution.
• The new status. canceled is terminal: not retried, not mailed, swept by retention. cancel_run is classified as a write for the draft write barrier (internal/toolwrite) and refused inside a run.
• Migration 000153_script_run_live adds result, progress_message/done/total/at and cancel_requested_at/by, and extends the status check with canceled. The down migration records canceled rows as failed.

4. SELECT column order (#1852)

Rows arrived as JSON objects and were converted with sorted keys, so exporting query rows alphabetized the CSV columns. starlarkconv.RowsToStarlark / ResultToStarlark build each row in the order the result's columns names them, with any key the columns do not name following in sorted order. platform.query and platform.call of a query tool both use it.

5. validate reads code (#1853)

The lexical checks moved to internal/platform/scriptlex and now run over CodeOnly(source). That is the source with string contents and comments blanked, keeping every offset so line numbers stay exact. A string literal "f", r["f"], SELECT datetime ... or a comment mentioning open( are no longer reported. The credential scan still reads the raw source, because a pasted key sits inside a string.

6. Export tools inside a run (#1854)

• Recorded as outputs. A file trino_export, api_export or graphql_export writes inside a run is recorded on the run's outputs through the output writer's lease-fenced RecordToolOutput. The record carries name, asset and version (or managed resource), format, rows, bytes, and a new tool field. It does not claim the name for platform.export.
• One asset per name.
  • A named export inside a run takes platform.export's identity: the first run creates the script's asset for that name, and each later run writes its next version.
  • The platform hands each export tool ExportUserContext.RunOutputKey (producedby.RunOutputKey, nil outside a run). One helper, toolkit.PersistRunAsset, does the lookup, insert, version and first-write race for all three tools.
  • The key format is written once, script.OutputIdentityKey, and the platform export writer uses it too.
• Unchanged. resource= still lands in a managed file, an explicit idempotency_key keeps its meaning, and every call outside a run still creates a new asset.
• Version reporting. Each tool's result now reports asset_version.

7. probe_bind (#1841)

The probe bound without SO_REUSEADDR, which every listener the stack starts sets. So a port whose only sockets were in TIME_WAIT after make dev-stop read as busy for about thirty seconds.

Checked on darwin, for four cases (TIME_WAIT only, a live same-user listener, a Docker listener on a specific address, a Docker listener on the wildcard), against net.Listen: the probe with SO_REUSEADDR answers the same as the Go server's own bind; the plain bind was stricter in three.

Portal

The run history shows a running run's progress under its status (120 of 500 · entities) and "Stopping" once a cancel is requested. An open run re-reads itself every 3 s while in flight, with Cancel run / Stop run, the log so far, and a Result block. canceled is counted in the summary line. Outputs written by an export tool say via <tool>.

Readers of this state

The portal JSON contract changed on portalRun (progress, cancel_requested), portalRunDetail (result, cancel_requested_by), the run route (a 200 carrying portalRunDetail), the new cancelResponse, and script.RunOutput.tool. Readers checked:

• ui/src/api/portal/hooks/scripts.ts: ScriptRun, ScriptRunDetail, ScriptRunOutput, useScriptRun (live refetch), useCancelScriptRun.
• ui/src/pages/scripts/ScriptRunHistory.tsx: progress line, run control, result.
• ui/src/pages/scripts/runFormat.ts: progressText, the canceled count, outputLink naming the tool.
• ui/src/mocks/handlers/scripts.ts and ui/src/mocks/data/scripts.ts: the cancel route and a run result, held to the swagger routes by route-conformance.test.ts.

internal/apidocs is regenerated.

Decomposition and the import allowlist

The package budgets required splitting the new code into packages:

• internal/procload: process memory and CPU against the container's limits.
• internal/platform/scriptadmit: the admission policy and the scripts.worker capacity section, inlined into pkg/platform.ScriptsWorkerConfig.
• internal/platform/scriptlex: the lexical checks, moved out of scriptrun.
• internal/platform/scriptlive: the run's live log, progress and result.
• internal/platform/runcontrol: waiting on and canceling a run.
• internal/httpserver/scripthttp/transferwords: the owner-transfer wording, moved out of scripthttp.

pkg/script stays under its exported-surface budget: CancelRun returns the prior status rather than a new outcome type.

The 16 new allowlist edges are these packages and their callers, plus pkg/platform -> internal/producedby and internal/producedby -> pkg/script for the run output key:

internal/httpserver/scripthttp -> internal/httpserver/scripthttp/transferwords
internal/httpserver/scripthttp -> internal/platform/runcontrol
internal/httpserver/scripthttp/transferwords -> internal/producedview
internal/platform/runcontrol -> pkg/script
internal/platform/scriptadmit -> internal/procload
internal/platform/scriptexec -> internal/platform/scriptadmit
internal/platform/scriptexec -> internal/platform/scriptlive
internal/platform/scriptlayer -> internal/platform/runcontrol
internal/platform/scriptlive -> internal/platform/starlarkconv
internal/platform/scriptlive -> pkg/script
internal/platform/scriptrun -> internal/platform/scriptlex
internal/platform/scriptrun -> internal/platform/scriptlive
internal/platform/scriptwiring -> internal/platform/scriptadmit
internal/producedby -> pkg/script
pkg/platform -> internal/platform/scriptadmit
pkg/platform -> internal/producedby

Behavior that changed for existing tests

• TestIntegration_TwoRunsReadingOneRevisionCannotBothWrite asserted the worker executes two queued runs in order. They now run at once, so the test asserts the invariant instead: exactly one writes the state revision and the other fails naming it.
• TestValidate_WarningsDoNotBlock relied on datetime inside a comment producing a warning, which is the false positive manage_script validate reports an f-string for a string literal "f" #1853 removes.

Evidence

• Acceptance. 16 criteria passed through the real MCP client and REST surface on the local dev stack: test/acceptance/issue_{1843,1845,1847,1852,1853,1854}_test.go, recorded at build/<n>/acceptance.jsonl.
  • Make the script run worker's concurrency and run timeout configurable (default 4 concurrent runs, 15m timeout) #1843: three queued runs overlapped.
  • Script runs: progress (platform.progress), live log and cancel #1847: a running run was canceled 2.05 s after the request.
  • Record trino_export / api_export assets written inside a run as run outputs #1854: the second run wrote version 2 of the same asset.
• fix(dev): probe_bind reports a port busy on lingering TIME_WAIT #1841. make dev came up immediately after make dev-stop with 67 sockets in TIME_WAIT on the stack's ports. test/gates/probe_bind_test.go fails against the old probe.
• Real Postgres. Migration 000153, RecordProgress, CancelRun for each prior status, the result and the progress columns (internal/platform/scriptstore/runlive_realdb_integration_test.go).
• make verify passed.

Closes #1841
Closes #1843
Closes #1845
Closes #1847
Closes #1852
Closes #1853
Closes #1854

[codecov]

Codecov Report

❌ Patch coverage is 94.55217% with 59 lines in your changes missing coverage. Please review.
✅ Project coverage is 91.92%. Comparing base (245cca8) to head (512b3ad).
⚠️ Report is 1 commits behind head on main.

Files with missing lines	Patch %	Lines
internal/platform/scriptlive/scriptlive.go	92.38%	5 Missing and 3 partials ⚠️
internal/platform/scriptrun/host.go	64.28%	3 Missing and 2 partials ⚠️
pkg/platform/platform.go	16.66%	5 Missing ⚠️
pkg/toolkits/graphql/export.go	87.50%	3 Missing and 2 partials ⚠️
pkg/toolkits/trino/export.go	88.63%	2 Missing and 3 partials ⚠️
internal/platform/scriptstore/runs.go	95.12%	2 Missing and 2 partials ⚠️
internal/procload/procload.go	94.28%	2 Missing and 2 partials ⚠️
pkg/toolkits/apigateway/export.go	93.33%	2 Missing and 2 partials ⚠️
internal/platform/scriptexec/runner.go	94.23%	2 Missing and 1 partial ⚠️
internal/platform/scriptwiring/scriptwiring.go	75.00%	3 Missing ⚠️
... and 7 more

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #1857      +/-   ##
==========================================
+ Coverage   91.87%   91.92%   +0.04%     
==========================================
  Files         941      953      +12     
  Lines       90680    91547     +867     
==========================================
+ Hits        83314    84150     +836     
- Misses       4815     4834      +19     
- Partials     2551     2563      +12     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
• 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.