From 4ff24cfa3d187a95ecd19b36135bd124dea81a0a Mon Sep 17 00:00:00 2001 From: Claude Date: Wed, 22 Jul 2026 15:12:10 +0000 Subject: [PATCH 001/132] feat: outbound HTTP response streaming (stream response / wait for next chunk|line) Add generic outbound response streaming to the WFL client so a large or progressively-emitted upstream body can be consumed without buffering: open url at "" [with method .. and headers .. and body ..] and stream response as upstream wait for next line from upstream as line // Text, or nothing at EOF wait for next chunk from upstream as chunk // Binary, or nothing at EOF close upstream // cancels the upstream request `stream response as` returns as soon as the status/headers arrive, without buffering the body, binding an object with status/ok/headers and an internal stream id. The body is parked in a new IoClient stream-handle table and pulled incrementally; each read takes the handle out of the map so a slow read never blocks other streams. Lifecycle (client side): the head phase and every per-chunk read go through the existing run_http_with_budget select, so connect/read timeouts, the response-byte ceiling (enforced on the running total, not just Content-Length), and cooperative cancellation all apply. Mid-stream network errors are catchable RuntimeErrors; dropping the handle (EOF, error, close, teardown) cancels the upstream; reading a closed/drained handle is a predictable error. A final unterminated line is delivered before a single clean `nothing` at EOF. - AST: HttpStreamStatement, WaitForNextChunkStatement, WaitForNextLineStatement - Parser: `stream response as` clause (io.rs); `wait for next chunk|line from` (processes.rs); no new lexer tokens (contextual identifiers) - Interpreter: IoClient open_http_stream/next_chunk/next_line/close_stream; three statement arms; `close` extended to accept a streaming-response object - Analyzer/typechecker/transpiler: variable-binding registration; explicit unsupported-in-JS-transpilation arm - Docs: interoperability.md "Streaming a response incrementally" + validated example; Dev Diary entry - Tests: tests/http_stream_test.rs (parser + offline runtime, incl. EOF and closed-stream error) Co-Authored-By: Claude Opus 4.8 Claude-Session: https://claude.ai/code/session_011LxXAm5E4Vy39kRLr8kRnX --- .../2026-07-22-outbound-response-streaming.md | 100 ++++ Docs/04-advanced-features/interoperability.md | 54 ++ .../docs_examples/_meta/manifest.json | 17 + .../interoperability/streaming_response.wfl | 25 + .../streaming_response.wfl.ast.txt | 190 ++++++ src/analyzer/mod.rs | 54 ++ src/interpreter/mod.rs | 552 +++++++++++++++++- src/parser/ast.rs | 36 ++ src/parser/stmt/io.rs | 22 + src/parser/stmt/processes.rs | 64 ++ src/transpiler/javascript.rs | 15 + src/typechecker/mod.rs | 99 ++++ tests/http_stream_test.rs | 258 ++++++++ 13 files changed, 1473 insertions(+), 13 deletions(-) create mode 100644 Dev diary/2026-07-22-outbound-response-streaming.md create mode 100644 TestPrograms/docs_examples/interoperability/streaming_response.wfl create mode 100644 TestPrograms/docs_examples/interoperability/streaming_response.wfl.ast.txt create mode 100644 tests/http_stream_test.rs diff --git a/Dev diary/2026-07-22-outbound-response-streaming.md b/Dev diary/2026-07-22-outbound-response-streaming.md new file mode 100644 index 00000000..91bc87f5 --- /dev/null +++ b/Dev diary/2026-07-22-outbound-response-streaming.md @@ -0,0 +1,100 @@ +# Dev Diary — 2026-07-22 — Generic outbound response streaming + +## Context + +A downstream app (a browser chat UI talking to a model endpoint) needs the WFL +runtime to proxy a slow upstream to the browser without buffering. That request +came in as five items: (1) outbound response streaming, (2) incremental +chunk/line reads, (3) streamed *server* responses, (4) concurrent request +handlers, and (5) lifecycle guarantees (timeouts, backpressure, cancellation, +catchable errors, close-on-every-exit). + +Items 3 and 4 are large and, importantly, item 4 (concurrent handlers) is +**already governed** by `Docs/development/concurrency-phase-plan.md` — a +maintainer-locked, gated plan (locked marker `main loop concurrently:`, "no +Rc→Arc rewrite of the interpreter core", TDD-first, stop-for-review between +phases). This entry covers the first shippable slice: **items 1, 2, and the +streaming-relevant parts of item 5 (outbound/client side)**, which are net-new +and do *not* touch the locked concurrency core. Server streaming (3) and the +concurrent loop (4) are separate follow-on changes. + +## What shipped + +New surface, mirroring the existing `open url` client: + +```wfl +open url at "" [with method .. and headers .. and body ..] and stream response as upstream +wait for next line from upstream as line // Text, or nothing at clean EOF +wait for next chunk from upstream as chunk // Binary, or nothing at clean EOF +close upstream // cancels the in-flight upstream +``` + +`stream response as` returns as soon as the status/headers arrive — **without +buffering the body** — and binds an object exposing `status`, `ok`, `headers`, +and an internal `_stream` id. The body stays parked in the interpreter and is +pulled incrementally. + +## Design notes + +- **No new lexer tokens.** `stream`, `next`, `chunk`, `line`, `upstream` are all + contextual identifiers; `response`/`from`/`as` are existing keywords. The + lexer's identifier-merging means `next chunk`/`next line` arrive as a single + token, handled in `parse_wait_for_statement`. +- **Handle model follows the existing pattern.** Open resources in WFL are + opaque ids into side-tables on `IoClient` (files, DB pools, processes). Added + `stream_handles: Mutex>`. `HttpStreamHandle` + holds a `Pin>> + Send>>` (from + `response.bytes_stream()`), a leftover-byte buffer for line splitting, a + `done` flag, and a running `bytes_read` total. +- **No lock held across the network await.** `next_chunk`/`next_line` *take* the + handle out of the map, await, then put it back — so a slow read on one stream + never blocks map access for another (this matters once concurrent handlers + land). +- **Lifecycle (item 5, client side).** The head phase and each per-chunk read go + through the existing `run_http_with_budget` select, so connect/read timeouts, + the response-byte ceiling (`web_server_max_response_size`, enforced + incrementally on the running total, not just `Content-Length`), and + cooperative cancellation all apply. Mid-stream network errors surface as + catchable `RuntimeError`s from the `wait for next ...` statement. Dropping the + handle — on clean EOF, error, explicit `close`, or interpreter teardown — + drops the reqwest body future and cancels the upstream. Reading a + closed/drained handle is a predictable catchable error. +- **`close` extended, not duplicated.** `close ` already closed files; + it now also accepts a streaming-response object and closes its stream. +- **EOF semantics.** A final unterminated line is delivered before EOF; the + handle is re-inserted (drained, `done`) so the *next* read returns `nothing` + once, then the handle is freed — the `check if line is nothing: break` loop + works and handles don't leak across many streamed requests. + +## Files + +- AST: `HttpStreamStatement`, `WaitForNextChunkStatement`, + `WaitForNextLineStatement` (`src/parser/ast.rs`). +- Parser: `stream response as` clause (`src/parser/stmt/io.rs`), `wait for next + chunk|line from` (`src/parser/stmt/processes.rs`). +- Interpreter: `IoClient::{open_http_stream, next_chunk, next_line, + close_stream}` + helpers, three statement arms, `close` extension + (`src/interpreter/mod.rs`). +- Analyzer/typechecker/transpiler: variable-binding registration and an explicit + "not supported in JS transpilation" arm. +- Docs: `Docs/04-advanced-features/interoperability.md` (new "Streaming a + response incrementally" section) + validated example + `TestPrograms/docs_examples/interoperability/streaming_response.wfl`. + +## Tests + +`tests/http_stream_test.rs` — parser tests for all three statements, and +offline runtime tests against a local one-shot TCP server: status/headers +available immediately, `next line` yields lines then `nothing`, a final +unterminated line is delivered, `next chunk` yields binary, and reading a closed +stream is an error. `cargo fmt`, `clippy -D warnings`, and the existing +`http_request_*` / `http_outbound_budget` suites are green. + +## Not in this change (follow-ons) + +- **Server-side streaming** (`write chunk`/`flush`/`close` on a response) — + needs the `oneshot` reply path reworked into a chunked body + channel through warp. +- **Concurrent request handlers** — Phase 1 of the concurrency plan + (`main loop concurrently:`); the keystone that makes a slow upstream stream + not stall other requests. Follows the gated plan, not this change. diff --git a/Docs/04-advanced-features/interoperability.md b/Docs/04-advanced-features/interoperability.md index a1abd312..4dfda170 100644 --- a/Docs/04-advanced-features/interoperability.md +++ b/Docs/04-advanced-features/interoperability.md @@ -105,6 +105,60 @@ request that is waiting on the remote peer. `body` introduce clauses, so use different variable names there (e.g. `request_headers`, `payload`). +#### Streaming a response incrementally + +`read content` / `read response` buffer the whole body before returning. For a +large download, or an upstream that emits output progressively (for example a +model endpoint sending newline-delimited JSON), use `stream response` instead. +It returns as soon as the status and headers arrive — **without buffering the +body** — and binds a streaming handle you pull from piece by piece: + +```wfl +open url at "https://api.example.com/events" + with method "POST" + and headers request_headers + and body payload + and stream response as upstream + +display "Status: " with upstream.status // available immediately +store content_type as upstream.headers["content-type"] + +// Pull the body one line at a time. Each read returns the next line, or +// `nothing` once the stream ends cleanly. +store done as no +count from 1 to 1000000: + wait for next line from upstream as line + check if line is nothing: + break + otherwise: + display line + end check +end count + +close upstream +``` + +Two incremental reads are available on a streaming handle: + +- `wait for next line from as ` — binds the next + newline-delimited line (the trailing newline, and a paired carriage return, + are stripped). A final line with no trailing newline is still delivered. +- `wait for next chunk from as ` — binds the next raw byte chunk + (`binary`) exactly as it arrives off the network, for non-line-oriented + payloads. + +Both bind `nothing` at a clean end of stream, so `check if line is nothing` +ends the loop. `close ` releases the stream early and cancels the +in-flight upstream request; reading from a closed (or fully-drained) handle +raises a catchable error. + +The same limits as buffered requests apply: the running total of body bytes is +held under `web_server_max_response_size`, each read is bounded by the request's +timeout, and cooperative cancellation interrupts a read waiting on the peer. A +mid-stream network error surfaces as a catchable error from the `wait for next +...` statement, and every stream is closed when the handle is dropped on any +exit path. + ### 4. **Web Standards** WFL web servers work with standard HTTP: diff --git a/TestPrograms/docs_examples/_meta/manifest.json b/TestPrograms/docs_examples/_meta/manifest.json index a07be1ad..8386c863 100644 --- a/TestPrograms/docs_examples/_meta/manifest.json +++ b/TestPrograms/docs_examples/_meta/manifest.json @@ -401,5 +401,22 @@ "containers" ], "doc_purpose": "Docs README tour: containers/objects" + }, + "docs_examples/interoperability/streaming_response.wfl": { + "doc_section": "Docs/04-advanced-features/interoperability.md#streaming-a-response-incrementally", + "type": "snippet", + "validate_layers": [ + 1, + 2, + 3, + 4 + ], + "skip_execution": true, + "tags": [ + "http-client", + "streaming", + "interoperability" + ], + "description": "Streaming an outbound HTTP response incrementally with stream response / wait for next line." } } diff --git a/TestPrograms/docs_examples/interoperability/streaming_response.wfl b/TestPrograms/docs_examples/interoperability/streaming_response.wfl new file mode 100644 index 00000000..7c5bca6f --- /dev/null +++ b/TestPrograms/docs_examples/interoperability/streaming_response.wfl @@ -0,0 +1,25 @@ +// Streaming an outbound response incrementally. +// +// `stream response as` returns as soon as the status and headers arrive, +// without buffering the body. Pull the body one line (or chunk) at a time; +// each read binds `nothing` at a clean end of stream. +// +// Validated for syntax/analysis/lint only (layers 1-4): running it needs a +// live upstream, so execution is skipped. + +open url at "https://api.example.com/events" and stream response as upstream + +display "Status: " with upstream["status"] +store content_type as upstream["headers"]["content-type"] +display "Content type: " with content_type + +count from 1 to 1000000: + wait for next line from upstream as line + check if line is nothing: + break + otherwise: + display line + end check +end count + +close upstream diff --git a/TestPrograms/docs_examples/interoperability/streaming_response.wfl.ast.txt b/TestPrograms/docs_examples/interoperability/streaming_response.wfl.ast.txt new file mode 100644 index 00000000..8fda9493 --- /dev/null +++ b/TestPrograms/docs_examples/interoperability/streaming_response.wfl.ast.txt @@ -0,0 +1,190 @@ +AST output for: TestPrograms/docs_examples/interoperability/streaming_response.wfl +============================================== + +Program with 6 statements: + +Statement #1: HttpStreamStatement { + url: Literal( + String( + "https://api.example.com/events", + ), + 10, + 13, + ), + method: None, + headers: None, + body: None, + variable_name: "upstream", + line: 10, + column: 1, +} + +Statement #2: DisplayStatement { + value: Concatenation { + left: Literal( + String( + "Status: ", + ), + 12, + 9, + ), + right: IndexAccess { + collection: Variable( + "upstream", + 12, + 25, + ), + index: Literal( + String( + "status", + ), + 12, + 34, + ), + line: 12, + column: 33, + }, + line: 12, + column: 20, + }, + line: 12, + column: 1, +} + +Statement #3: VariableDeclaration { + name: "content_type", + value: IndexAccess { + collection: IndexAccess { + collection: Variable( + "upstream", + 13, + 23, + ), + index: Literal( + String( + "headers", + ), + 13, + 32, + ), + line: 13, + column: 31, + }, + index: Literal( + String( + "content-type", + ), + 13, + 43, + ), + line: 13, + column: 42, + }, + is_constant: false, + line: 13, + column: 1, +} + +Statement #4: DisplayStatement { + value: Concatenation { + left: Literal( + String( + "Content type: ", + ), + 14, + 9, + ), + right: Variable( + "content_type", + 14, + 31, + ), + line: 14, + column: 26, + }, + line: 14, + column: 1, +} + +Statement #5: CountLoop { + start: Literal( + Integer( + 1, + ), + 16, + 12, + ), + end: Literal( + Integer( + 1000000, + ), + 16, + 17, + ), + step: None, + downward: false, + variable_name: None, + body: [ + WaitForNextLineStatement { + source: Variable( + "upstream", + 17, + 29, + ), + variable_name: "line", + line: 17, + column: 5, + }, + IfStatement { + condition: BinaryOperation { + left: Variable( + "line", + 18, + 14, + ), + operator: Equals, + right: Literal( + Nothing, + 18, + 22, + ), + line: 18, + column: 19, + }, + then_block: [ + BreakStatement { + line: 19, + column: 9, + }, + ], + else_block: Some( + [ + DisplayStatement { + value: Variable( + "line", + 21, + 17, + ), + line: 21, + column: 9, + }, + ], + ), + line: 18, + column: 5, + }, + ], + line: 23, + column: 10, +} + +Statement #6: CloseFileStatement { + file: Variable( + "upstream", + 25, + 7, + ), + line: 25, + column: 1, +} + diff --git a/src/analyzer/mod.rs b/src/analyzer/mod.rs index 57a40136..bae8f627 100644 --- a/src/analyzer/mod.rs +++ b/src/analyzer/mod.rs @@ -1535,6 +1535,60 @@ impl Analyzer { } } + Statement::HttpStreamStatement { + url, + method, + headers, + body, + variable_name, + .. + } => { + self.analyze_expression(url); + if let Some(method) = method { + self.analyze_expression(method); + } + if let Some(headers) = headers { + self.analyze_expression(headers); + } + if let Some(body) = body { + self.analyze_expression(body); + } + + // Binds a streaming-response handle object (status/ok/headers). + let symbol = Symbol { + name: variable_name.clone(), + kind: SymbolKind::Variable { mutable: true }, + symbol_type: None, + line: 0, + column: 0, + }; + self.current_scope.define_or_replace(symbol); + } + + Statement::WaitForNextChunkStatement { + source, + variable_name, + .. + } + | Statement::WaitForNextLineStatement { + source, + variable_name, + .. + } => { + self.analyze_expression(source); + + // Binds the next chunk/line, or `nothing` at end of stream, so + // the type is left open. Refreshed on every wait (loop-friendly). + let symbol = Symbol { + name: variable_name.clone(), + kind: SymbolKind::Variable { mutable: true }, + symbol_type: None, + line: 0, + column: 0, + }; + self.current_scope.define_or_replace(symbol); + } + Statement::CreateDirectoryStatement { path, .. } => { self.analyze_expression(path); } diff --git a/src/interpreter/mod.rs b/src/interpreter/mod.rs index 7d38c246..92510094 100644 --- a/src/interpreter/mod.rs +++ b/src/interpreter/mod.rs @@ -893,6 +893,15 @@ fn stmt_type(stmt: &Statement) -> String { Statement::HttpRequestStatement { variable_name, .. } => { format!("HttpRequestStatement '{variable_name}'") } + Statement::HttpStreamStatement { variable_name, .. } => { + format!("HttpStreamStatement '{variable_name}'") + } + Statement::WaitForNextChunkStatement { variable_name, .. } => { + format!("WaitForNextChunkStatement '{variable_name}'") + } + Statement::WaitForNextLineStatement { variable_name, .. } => { + format!("WaitForNextLineStatement '{variable_name}'") + } Statement::PushStatement { .. } => "PushStatement to list".to_string(), Statement::CreateListStatement { name, .. } => format!("CreateListStatement '{name}'"), Statement::MapCreation { name, .. } => format!("MapCreation '{name}'"), @@ -1398,9 +1407,37 @@ pub struct IoClient { next_process_id: Mutex, db_handles: Mutex>, next_db_id: Mutex, + /// Live outbound streaming response bodies, keyed by handle id + /// ("httpstream1", ...). See [`HttpStreamHandle`]. + stream_handles: Mutex>, + next_stream_id: Mutex, config: Arc, } +/// A live, parked outbound streaming response body. +/// +/// The status and headers were already handed to the WFL program by +/// `open url ... and stream response as `; this holds the still-open body +/// stream so `wait for next chunk|line` can pull it incrementally without ever +/// buffering the whole body. Dropping the handle — on clean EOF, a mid-stream +/// error, an explicit `close`, or interpreter teardown — drops the underlying +/// reqwest body future and thereby cancels the in-flight upstream request. +struct HttpStreamHandle { + /// The response body as a stream of raw byte chunks. `Vec` (not + /// `bytes::Bytes`) so the boxed trait object stays nameable here. + stream: std::pin::Pin>> + Send>>, + /// Bytes read from the network but not yet handed to the program: the + /// remainder after a line split, plus bytes accumulated while scanning for + /// the next newline. Bounded by the run's `max_response_bytes` because + /// every byte placed here was counted against that ceiling as it was read. + buffer: Vec, + /// True once the underlying stream has yielded a clean end of stream. + done: bool, + /// Total body bytes pulled from the network so far, enforced against + /// `max_response_bytes`. + bytes_read: usize, +} + /// Errors raised while an outbound HTTP request is in flight. /// /// Budget failures stay structured until the interpreter can attach source @@ -1495,6 +1532,8 @@ impl IoClient { next_process_id: Mutex::new(1), db_handles: Mutex::new(HashMap::new()), next_db_id: Mutex::new(1), + stream_handles: Mutex::new(HashMap::new()), + next_stream_id: Mutex::new(1), config, } } @@ -1590,6 +1629,236 @@ impl IoClient { self.send_http_request(request, method, budget).await } + /// Open a streaming outbound request: send it and return + /// `(status, response headers, stream handle id)` as soon as the response + /// head arrives, WITHOUT buffering the body. The body stays open behind the + /// returned handle id for incremental `wait for next chunk|line` reads. + /// + /// The head phase (connect + headers) is bounded by the same finite + /// deadline and cooperative-cancellation machinery as a buffered request; + /// each later body read is bounded per-chunk in [`Self::stream_pull`]. + async fn open_http_stream( + &self, + method: &str, + url: &str, + headers: &[(String, String)], + body: Option, + budget: Arc, + ) -> Result<(u16, Vec<(String, String)>, String), HttpClientError> { + use futures_util::StreamExt; + + let parsed_method = reqwest::Method::from_bytes(method.as_bytes()) + .map_err(|_| HttpClientError::Request(format!("Invalid HTTP method: {method}")))?; + let mut request = self.http_client.request(parsed_method, url); + for (name, value) in headers { + request = request.header(name.as_str(), value.as_str()); + } + if let Some(body) = body { + request = request.body(body); + } + + let method_owned = method.to_string(); + let configured_timeout = Duration::from_secs(self.config.timeout_seconds.max(1)); + let op = async move { + request.send().await.map_err(|e| { + HttpClientError::Request(format!("Failed to send HTTP {method_owned} request: {e}")) + }) + }; + // Only the head is awaited here; dropping this future on + // timeout/cancel aborts the connection cleanly. + let response = + Self::run_http_with_budget(Arc::clone(&budget), configured_timeout, op).await?; + + let status = response.status().as_u16(); + let response_headers = response + .headers() + .iter() + .map(|(name, value)| { + ( + name.as_str().to_ascii_lowercase(), + String::from_utf8_lossy(value.as_bytes()).into_owned(), + ) + }) + .collect(); + + // Reject an over-ceiling body up front when the length is advertised; + // per-chunk reads enforce the same ceiling for chunked/unknown lengths. + let max_response_bytes = budget.limits().max_response_bytes; + if let Some(content_length) = response.content_length() { + let max_as_u64 = u64::try_from(max_response_bytes).unwrap_or(u64::MAX); + if content_length > max_as_u64 { + return Err(HttpClientError::Budget(BudgetExceeded::ResponseBytes { + limit: max_response_bytes, + actual: usize::try_from(content_length).unwrap_or(usize::MAX), + })); + } + } + + let stream = response + .bytes_stream() + .map(|chunk| chunk.map(|b| b.to_vec())); + let handle = HttpStreamHandle { + stream: Box::pin(stream), + buffer: Vec::new(), + done: false, + bytes_read: 0, + }; + let handle_id = { + let mut next_id = self.next_stream_id.lock().await; + let id = format!("httpstream{}", *next_id); + *next_id += 1; + id + }; + self.stream_handles + .lock() + .await + .insert(handle_id.clone(), handle); + Ok((status, response_headers, handle_id)) + } + + /// Remove a stream handle from the map so a body read can await without + /// holding the global handle lock across the network. Errors if the handle + /// is unknown (already closed, or forged). + async fn take_stream(&self, handle_id: &str) -> Result { + self.stream_handles + .lock() + .await + .remove(handle_id) + .ok_or_else(|| { + HttpClientError::Request(format!( + "Unknown or already-closed stream handle '{handle_id}'" + )) + }) + } + + /// Return a still-open stream handle to the map after a body read. + async fn put_stream(&self, handle_id: &str, handle: HttpStreamHandle) { + self.stream_handles + .lock() + .await + .insert(handle_id.to_string(), handle); + } + + /// Pull one network chunk into `handle.buffer`, bounded by the per-chunk + /// read deadline and the run's response-byte ceiling. Returns `Ok(true)` + /// when bytes were added, `Ok(false)` at clean EOF (sets `handle.done`). + async fn stream_pull( + &self, + handle: &mut HttpStreamHandle, + budget: &Arc, + ) -> Result { + use futures_util::StreamExt; + + if handle.done { + return Ok(false); + } + let max_response_bytes = budget.limits().max_response_bytes; + let configured_timeout = Duration::from_secs(self.config.timeout_seconds.max(1)); + let next = Self::run_http_with_budget(Arc::clone(budget), configured_timeout, async { + Ok::>>, HttpClientError>(handle.stream.next().await) + }) + .await?; + + match next { + Some(Ok(bytes)) => { + let actual = handle.bytes_read.saturating_add(bytes.len()); + if bytes.len() > max_response_bytes.saturating_sub(handle.bytes_read) { + return Err(HttpClientError::Budget(BudgetExceeded::ResponseBytes { + limit: max_response_bytes, + actual, + })); + } + handle.bytes_read = actual; + handle.buffer.extend_from_slice(&bytes); + Ok(true) + } + Some(Err(e)) => Err(HttpClientError::Request(format!( + "Failed to read response chunk: {e}" + ))), + None => { + handle.done = true; + Ok(false) + } + } + } + + /// Pull the next raw byte chunk from a streaming response. Returns + /// `Ok(None)` at clean end of stream (handle is dropped). On error or EOF + /// the handle is not re-inserted, so the upstream request is released. + async fn next_chunk( + &self, + handle_id: &str, + budget: Arc, + ) -> Result>, HttpClientError> { + let mut handle = self.take_stream(handle_id).await?; + + // Any bytes buffered by a prior `next line` are served first. + if !handle.buffer.is_empty() { + let chunk = std::mem::take(&mut handle.buffer); + self.put_stream(handle_id, handle).await; + return Ok(Some(chunk)); + } + + match self.stream_pull(&mut handle, &budget).await { + Ok(true) => { + let chunk = std::mem::take(&mut handle.buffer); + self.put_stream(handle_id, handle).await; + Ok(Some(chunk)) + } + Ok(false) => Ok(None), // clean EOF: drop the handle + Err(e) => Err(e), // error/timeout: drop the handle (cancels upstream) + } + } + + /// Pull the next newline-delimited line (trailing `\n`, and a paired `\r`, + /// stripped) from a streaming response. A final unterminated line is + /// returned before EOF. Returns `Ok(None)` at clean end of stream. + async fn next_line( + &self, + handle_id: &str, + budget: Arc, + ) -> Result, HttpClientError> { + let mut handle = self.take_stream(handle_id).await?; + + loop { + if let Some(pos) = handle.buffer.iter().position(|&b| b == b'\n') { + let mut line: Vec = handle.buffer.drain(..=pos).collect(); + line.pop(); // drop '\n' + if line.last() == Some(&b'\r') { + line.pop(); // drop paired '\r' (CRLF) + } + self.put_stream(handle_id, handle).await; + return Ok(Some(String::from_utf8_lossy(&line).into_owned())); + } + + if handle.done { + // No newline left. Emit any final unterminated line, then EOF. + if handle.buffer.is_empty() { + return Ok(None); // drop the exhausted handle + } + let mut line = std::mem::take(&mut handle.buffer); + if line.last() == Some(&b'\r') { + line.pop(); + } + // Re-insert the now-drained (done, empty) handle so the *next* + // read cleanly returns `nothing` instead of erroring on a + // missing handle. + self.put_stream(handle_id, handle).await; + return Ok(Some(String::from_utf8_lossy(&line).into_owned())); + } + + // Need more bytes to find a newline. + self.stream_pull(&mut handle, &budget).await?; + } + } + + /// Close a streaming response handle if present. Dropping the handle + /// cancels the in-flight upstream request. Returns whether a handle was + /// found. Idempotent: closing an unknown/already-closed handle is a no-op. + async fn close_stream(&self, handle_id: &str) -> bool { + self.stream_handles.lock().await.remove(handle_id).is_some() + } + /// Send a request and consume its body without ever buffering more than the /// configured response ceiling. The budget passed here is deliberately the /// interpreter's *live* budget, not construction-time IoClient state: the @@ -3344,6 +3613,41 @@ impl Interpreter { } } + /// Resolve a `wait for next chunk|line from ` operand to a stream + /// handle id. Accepts either the streaming-response object bound by + /// `stream response as ` (reads its internal `_stream` id) or a bare + /// text handle id. + async fn resolve_stream_handle( + &self, + source: &Expression, + env: &Rc>, + line: usize, + column: usize, + ) -> Result { + let value = self.evaluate_expression(source, Rc::clone(env)).await?; + match &value { + Value::Object(obj) => match obj.borrow().get("_stream") { + Some(Value::Text(id)) => Ok(id.to_string()), + _ => Err(RuntimeError::new( + "Expected a streaming response handle (from `stream response as ...`), \ + but this value has no open stream" + .to_string(), + line, + column, + )), + }, + Value::Text(id) => Ok(id.to_string()), + _ => Err(RuntimeError::new( + format!( + "Expected a streaming response handle, got {}", + value.type_name() + ), + line, + column, + )), + } + } + /// Preserve ordinary file I/O failures while classifying byte-ceiling /// breaches as catchable execution-budget resource errors. fn file_read_error(&self, error: FileReadError, line: usize, column: usize) -> RuntimeError { @@ -3879,6 +4183,9 @@ impl Interpreter { Statement::HttpGetStatement { line, column, .. } => (*line, *column), Statement::HttpPostStatement { line, column, .. } => (*line, *column), Statement::HttpRequestStatement { line, column, .. } => (*line, *column), + Statement::HttpStreamStatement { line, column, .. } => (*line, *column), + Statement::WaitForNextChunkStatement { line, column, .. } => (*line, *column), + Statement::WaitForNextLineStatement { line, column, .. } => (*line, *column), Statement::PushStatement { line, column, .. } => (*line, *column), Statement::CreateListStatement { line, column, .. } => (*line, *column), Statement::MapCreation { line, column, .. } => (*line, *column), @@ -4848,20 +5155,42 @@ impl Interpreter { Statement::CloseFileStatement { file, line, column } => { let file_value = self.evaluate_expression(file, Rc::clone(&env)).await?; - let file_str = match &file_value { - Value::Text(s) => s.clone(), - _ => { - return Err(RuntimeError::new( - format!("Expected string for file handle, got {file_value:?}"), - *line, - *column, - )); + match &file_value { + // A bare text handle closes a file (existing behavior). + Value::Text(s) => match self.io_client.close_file(s).await { + Ok(_) => Ok((Value::Null, ControlFlow::None)), + Err(e) => Err(RuntimeError::new(e, *line, *column)), + }, + // A streaming-response object closes its underlying stream, + // cancelling the in-flight upstream request. `close` on a + // stream is always safe, even after EOF. + Value::Object(obj) => { + let stream_id = obj.borrow().get("_stream").and_then(|v| match v { + Value::Text(s) => Some(s.to_string()), + _ => None, + }); + match stream_id { + Some(id) => { + self.io_client.close_stream(&id).await; + Ok((Value::Null, ControlFlow::None)) + } + None => Err(RuntimeError::new( + "Cannot close this value: it is not a file handle or a \ + streaming response" + .to_string(), + *line, + *column, + )), + } } - }; - - match self.io_client.close_file(&file_str).await { - Ok(_) => Ok((Value::Null, ControlFlow::None)), - Err(e) => Err(RuntimeError::new(e, *line, *column)), + _ => Err(RuntimeError::new( + format!( + "Expected a file handle or streaming response, got {}", + file_value.type_name() + ), + *line, + *column, + )), } } Statement::CreateDirectoryStatement { path, line, column } => { @@ -6082,6 +6411,203 @@ impl Interpreter { Err(error) => Err(self.http_client_error(error, *line, *column)), } } + Statement::HttpStreamStatement { + url, + method, + headers, + body, + variable_name, + line, + column, + } => { + let url_val = self.evaluate_expression(url, Rc::clone(&env)).await?; + let url_str = match &url_val { + Value::Text(s) => s.clone(), + _ => { + return Err(RuntimeError::new( + format!("Expected string for URL, got {url_val:?}"), + *line, + *column, + )); + } + }; + + let method_str = match method { + Some(method_expr) => { + let method_val = self + .evaluate_expression(method_expr, Rc::clone(&env)) + .await?; + match &method_val { + Value::Text(s) => s.trim().to_ascii_uppercase(), + _ => { + return Err(RuntimeError::new( + format!("Expected text for HTTP method, got {method_val:?}"), + *line, + *column, + )); + } + } + } + None => "GET".to_string(), + }; + + let mut header_list: Vec<(String, String)> = Vec::new(); + if let Some(headers_expr) = headers { + let headers_val = self + .evaluate_expression(headers_expr, Rc::clone(&env)) + .await?; + match &headers_val { + Value::Object(obj) => { + for (name, value) in obj.borrow().iter() { + let value_str = match value { + Value::Text(s) => s.to_string(), + Value::Number(_) | Value::Bool(_) => value.to_string(), + _ => { + return Err(RuntimeError::new( + format!( + "Header '{name}' must be text, got {}", + value.type_name() + ), + *line, + *column, + )); + } + }; + header_list.push((name.clone(), value_str)); + } + header_list.sort(); + } + _ => { + return Err(RuntimeError::new( + format!( + "Expected a map for headers, got {}", + headers_val.type_name() + ), + *line, + *column, + )); + } + } + } + + let body_str = match body { + Some(body_expr) => { + let body_val = self.evaluate_expression(body_expr, Rc::clone(&env)).await?; + match &body_val { + Value::Text(s) => Some(s.to_string()), + Value::Number(_) | Value::Bool(_) => Some(body_val.to_string()), + _ => { + return Err(RuntimeError::new( + format!( + "Expected text for request body, got {}", + body_val.type_name() + ), + *line, + *column, + )); + } + } + } + None => None, + }; + + match self + .io_client + .open_http_stream( + &method_str, + &url_str, + &header_list, + body_str, + Arc::clone(&self.budget), + ) + .await + { + Ok((status, response_headers, handle_id)) => { + let mut headers_map = HashMap::new(); + for (name, value) in response_headers { + headers_map.insert(name, Value::Text(value.into())); + } + + let mut stream_map = HashMap::new(); + stream_map.insert("status".to_string(), Value::Number(status as f64)); + stream_map + .insert("ok".to_string(), Value::Bool((200..300).contains(&status))); + stream_map.insert( + "headers".to_string(), + Value::Object(Rc::new(RefCell::new(headers_map))), + ); + // Internal id used by `wait for next chunk|line` and + // `close`. Underscore-prefixed to signal "not for + // direct program use", mirroring request objects. + stream_map.insert("_stream".to_string(), Value::Text(handle_id.into())); + + let value = Value::Object(Rc::new(RefCell::new(stream_map))); + match env.borrow_mut().define(variable_name, value) { + Ok(_) => Ok((Value::Null, ControlFlow::None)), + Err(msg) => Err(RuntimeError::new(msg, *line, *column)), + } + } + Err(error) => Err(self.http_client_error(error, *line, *column)), + } + } + Statement::WaitForNextChunkStatement { + source, + variable_name, + line, + column, + } => { + let handle_id = self + .resolve_stream_handle(source, &env, *line, *column) + .await?; + match self + .io_client + .next_chunk(&handle_id, Arc::clone(&self.budget)) + .await + { + // Raw bytes as Binary so callers can handle any payload. + Ok(Some(bytes)) => { + let value = Value::Binary(Arc::from(bytes.as_slice())); + match env.borrow_mut().define(variable_name, value) { + Ok(_) => Ok((Value::Null, ControlFlow::None)), + Err(msg) => Err(RuntimeError::new(msg, *line, *column)), + } + } + // Clean EOF binds `nothing` so `check if chunk is nothing` ends the loop. + Ok(None) => match env.borrow_mut().define(variable_name, Value::Null) { + Ok(_) => Ok((Value::Null, ControlFlow::None)), + Err(msg) => Err(RuntimeError::new(msg, *line, *column)), + }, + Err(error) => Err(self.http_client_error(error, *line, *column)), + } + } + Statement::WaitForNextLineStatement { + source, + variable_name, + line, + column, + } => { + let handle_id = self + .resolve_stream_handle(source, &env, *line, *column) + .await?; + match self + .io_client + .next_line(&handle_id, Arc::clone(&self.budget)) + .await + { + Ok(Some(line_text)) => { + let value = Value::Text(line_text.into()); + match env.borrow_mut().define(variable_name, value) { + Ok(_) => Ok((Value::Null, ControlFlow::None)), + Err(msg) => Err(RuntimeError::new(msg, *line, *column)), + } + } + Ok(None) => match env.borrow_mut().define(variable_name, Value::Null) { + Ok(_) => Ok((Value::Null, ControlFlow::None)), + Err(msg) => Err(RuntimeError::new(msg, *line, *column)), + }, + Err(error) => Err(self.http_client_error(error, *line, *column)), + } + } Statement::RepeatWhileLoop { condition, body, diff --git a/src/parser/ast.rs b/src/parser/ast.rs index 29fd23be..1fc09cc1 100644 --- a/src/parser/ast.rs +++ b/src/parser/ast.rs @@ -409,6 +409,42 @@ pub enum Statement { line: usize, column: usize, }, + /// Streaming outbound HTTP request: + /// `open url at "" [with method .. and headers .. and body ..] and stream response as ` + /// + /// Unlike [`HttpRequestStatement`], this returns as soon as the status and + /// headers are received, WITHOUT buffering the body. It binds `` to a + /// streaming response handle (an object exposing `status`, `ok`, `headers`, + /// and an internal `_stream` id). The body is pulled incrementally with + /// `wait for next chunk from ` / `wait for next line from `. + HttpStreamStatement { + url: Expression, + method: Option, + headers: Option, + body: Option, + variable_name: String, + line: usize, + column: usize, + }, + /// `wait for next chunk from as ` — pull the next raw byte + /// chunk from a streaming response handle. Binds `` to `Binary`, or to + /// `nothing` at a clean end of stream. + WaitForNextChunkStatement { + source: Expression, + variable_name: String, + line: usize, + column: usize, + }, + /// `wait for next line from as ` — pull the next + /// newline-delimited line (trailing newline stripped) from a streaming + /// response handle. Binds `` to `Text`, or to `nothing` at end of + /// stream. + WaitForNextLineStatement { + source: Expression, + variable_name: String, + line: usize, + column: usize, + }, PushStatement { list: Expression, value: Expression, diff --git a/src/parser/stmt/io.rs b/src/parser/stmt/io.rs index 494c637f..a6046d95 100644 --- a/src/parser/stmt/io.rs +++ b/src/parser/stmt/io.rs @@ -255,6 +255,28 @@ impl<'a> IoParser<'a> for Parser<'a> { } }); } + // `stream response as ` — return the status/headers + // immediately and bind a streaming handle instead of buffering + // the body. `stream` is a contextual identifier (not a + // keyword), so match it as one; `response` is a keyword. + Token::Identifier(name) if name == "stream" => { + self.bump_sync(); // Consume "stream" + self.expect_token( + Token::KeywordResponse, + "Expected 'response' after 'stream'", + )?; + self.expect_token(Token::KeywordAs, "Expected 'as' after 'stream response'")?; + let variable_name = parse_variable_name(self, open_token)?; + return Ok(Statement::HttpStreamStatement { + url: url_expr, + method, + headers, + body, + variable_name, + line: open_token.line, + column: open_token.column, + }); + } // The lexer merges consecutive identifiers into multi-word // names, so `headers auth_headers` arrives as the single // token Identifier("headers auth_headers"). Match both the diff --git a/src/parser/stmt/processes.rs b/src/parser/stmt/processes.rs index ed5af44a..bc631749 100644 --- a/src/parser/stmt/processes.rs +++ b/src/parser/stmt/processes.rs @@ -391,6 +391,70 @@ impl<'a> ProcessParser<'a> for Parser<'a> { column: wait_token_pos.column, }); } + // "wait for next chunk from as " and + // "wait for next line from as " pull the next + // piece of a streaming response body. The lexer glues the two + // identifiers, so `next chunk` / `next line` arrive as a single + // token; a bare `next` (followed by chunk/line) is also handled. + Token::Identifier(id) + if id == "next chunk" || id == "next line" || id == "next" => + { + let is_line = id.ends_with("line"); + let is_bare_next = id == "next"; + self.bump_sync(); // Consume "next chunk"/"next line" (or bare "next") + + let is_line = if is_bare_next { + match self.cursor.peek() { + Some(t) => match &t.token { + Token::Identifier(kind) if kind == "chunk" => { + self.bump_sync(); + false + } + Token::Identifier(kind) if kind == "line" => { + self.bump_sync(); + true + } + _ => { + return Err(ParseError::from_token( + "Expected 'chunk' or 'line' after 'next'".to_string(), + t, + )); + } + }, + None => { + return Err(self + .cursor + .error("Expected 'chunk' or 'line' after 'next'".to_string())); + } + } + } else { + is_line + }; + + self.expect_token( + Token::KeywordFrom, + "Expected 'from' after 'next chunk'/'next line'", + )?; + let source = self.parse_primary_expression()?; + self.expect_token(Token::KeywordAs, "Expected 'as' after the stream handle")?; + let variable_name = self.parse_variable_name_simple()?; + + return Ok(if is_line { + Statement::WaitForNextLineStatement { + source, + variable_name, + line: wait_token_pos.line, + column: wait_token_pos.column, + } + } else { + Statement::WaitForNextChunkStatement { + source, + variable_name, + line: wait_token_pos.line, + column: wait_token_pos.column, + } + }); + } _ => { // Try to parse as "wait for X milliseconds/seconds" let checkpoint = self.cursor.checkpoint(); diff --git a/src/transpiler/javascript.rs b/src/transpiler/javascript.rs index 79e9ba9e..220b31a1 100644 --- a/src/transpiler/javascript.rs +++ b/src/transpiler/javascript.rs @@ -620,6 +620,18 @@ impl JavaScriptTranspiler { }) } + Statement::HttpStreamStatement { line, column, .. } + | Statement::WaitForNextChunkStatement { line, column, .. } + | Statement::WaitForNextLineStatement { line, column, .. } => { + // Streaming HTTP relies on the interpreter's parked-stream + // handles; emitting broken JS would be worse than a clear error. + Err(TranspileError { + message: "Streaming HTTP statements are not supported in JavaScript transpilation. They require the WFL interpreter.".to_string(), + line: *line, + column: *column, + }) + } + Statement::WriteContentStatement { content, target, .. } => { @@ -2055,6 +2067,9 @@ impl JavaScriptTranspiler { | Statement::HttpGetStatement { .. } | Statement::HttpPostStatement { .. } | Statement::HttpRequestStatement { .. } + | Statement::HttpStreamStatement { .. } + | Statement::WaitForNextChunkStatement { .. } + | Statement::WaitForNextLineStatement { .. } | Statement::WaitForProcessStatement { .. } | Statement::WaitForRequestStatement { .. } => true, diff --git a/src/typechecker/mod.rs b/src/typechecker/mod.rs index 10b8690c..2b07a052 100644 --- a/src/typechecker/mod.rs +++ b/src/typechecker/mod.rs @@ -870,6 +870,105 @@ impl TypeChecker { }); } } + Statement::HttpStreamStatement { + url, + method, + headers, + body, + variable_name, + line: _line, + column: _column, + } => { + let url_type = self.infer_expression_type(url); + if url_type != Type::Text && url_type != Type::Unknown && url_type != Type::Error { + self.type_error( + "URL must be a text string".to_string(), + Some(Type::Text), + Some(url_type), + *_line, + *_column, + ); + } + if let Some(method) = method { + let method_type = self.infer_expression_type(method); + if method_type != Type::Text + && method_type != Type::Unknown + && method_type != Type::Error + { + self.type_error( + "HTTP method must be a text string".to_string(), + Some(Type::Text), + Some(method_type), + *_line, + *_column, + ); + } + } + if let Some(headers) = headers { + let headers_type = self.infer_expression_type(headers); + if !matches!( + headers_type, + Type::Map(_, _) | Type::Unknown | Type::Any | Type::Error + ) { + self.type_error( + "HTTP headers must be a map of header names to values".to_string(), + Some(Type::Map(Box::new(Type::Text), Box::new(Type::Text))), + Some(headers_type), + *_line, + *_column, + ); + } + } + if let Some(body) = body { + let body_type = self.infer_expression_type(body); + if !matches!( + body_type, + Type::Text + | Type::Number + | Type::Boolean + | Type::Unknown + | Type::Any + | Type::Error + ) { + self.type_error( + "HTTP request body must be text".to_string(), + Some(Type::Text), + Some(body_type), + *_line, + *_column, + ); + } + } + + // Binds a streaming-response handle object (status/ok/headers). + if !variable_name.is_empty() + && let Some(symbol) = self.analyzer.get_symbol_mut(variable_name) + { + symbol.symbol_type = + Some(Type::Map(Box::new(Type::Text), Box::new(Type::Unknown))); + } + } + Statement::WaitForNextChunkStatement { + source, + variable_name, + .. + } + | Statement::WaitForNextLineStatement { + source, + variable_name, + .. + } => { + // Just validate the operand is inferable; the binding may be a + // chunk/line value or `nothing` at end of stream, so leave the + // bound variable's type open (Any) to avoid false errors on the + // `check if is nothing` loop-termination pattern. + let _ = self.infer_expression_type(source); + if !variable_name.is_empty() + && let Some(symbol) = self.analyzer.get_symbol_mut(variable_name) + { + symbol.symbol_type = Some(Type::Any); + } + } Statement::VariableDeclaration { name, value, diff --git a/tests/http_stream_test.rs b/tests/http_stream_test.rs new file mode 100644 index 00000000..99d843bd --- /dev/null +++ b/tests/http_stream_test.rs @@ -0,0 +1,258 @@ +// Tests for generic outbound response streaming: +// open url at "" [with ...] and stream response as upstream +// wait for next chunk from upstream as chunk -> Binary, or nothing at EOF +// wait for next line from upstream as line -> Text, or nothing at EOF +// close upstream -> cancels the upstream request +// +// A minimal local TCP server stands in for a real upstream (e.g. an LLM +// endpoint emitting newline-delimited JSON), so the tests are offline-safe. + +use tokio::io::{AsyncReadExt, AsyncWriteExt}; +use tokio::net::TcpListener; + +use wfl::interpreter::Interpreter; +use wfl::interpreter::value::Value; +use wfl::lexer::lex_wfl_with_positions; +use wfl::parser::Parser; +use wfl::parser::ast::Statement; + +// ----------------------------- parser tests ------------------------------ + +fn parse_single_statement(code: &str) -> Statement { + let tokens = lex_wfl_with_positions(code); + let mut parser = Parser::new(&tokens); + let program = parser + .parse() + .unwrap_or_else(|e| panic!("Parse error for {code:?}: {e:?}")); + assert_eq!( + program.statements.len(), + 1, + "Expected exactly one statement for {code:?}" + ); + program.statements.into_iter().next().unwrap() +} + +#[test] +fn test_stream_response_parses_to_http_stream_statement() { + let stmt = + parse_single_statement(r#"open url at "https://example.com" and stream response as up"#); + match stmt { + Statement::HttpStreamStatement { + method, + headers, + body, + variable_name, + .. + } => { + assert!(method.is_none()); + assert!(headers.is_none()); + assert!(body.is_none()); + assert_eq!(variable_name, "up"); + } + other => panic!("Expected HttpStreamStatement, got {other:?}"), + } +} + +#[test] +fn test_stream_response_with_method_headers_body() { + let stmt = parse_single_statement( + r#"open url at "https://api.example.com" with method "POST" and headers h and body b and stream response as up"#, + ); + match stmt { + Statement::HttpStreamStatement { + method, + headers, + body, + variable_name, + .. + } => { + assert!(method.is_some()); + assert!(headers.is_some()); + assert!(body.is_some()); + assert_eq!(variable_name, "up"); + } + other => panic!("Expected HttpStreamStatement, got {other:?}"), + } +} + +#[test] +fn test_wait_for_next_chunk_parses() { + let stmt = parse_single_statement("wait for next chunk from up as chunk"); + match stmt { + Statement::WaitForNextChunkStatement { variable_name, .. } => { + assert_eq!(variable_name, "chunk"); + } + other => panic!("Expected WaitForNextChunkStatement, got {other:?}"), + } +} + +#[test] +fn test_wait_for_next_line_parses() { + let stmt = parse_single_statement("wait for next line from up as line"); + match stmt { + Statement::WaitForNextLineStatement { variable_name, .. } => { + assert_eq!(variable_name, "line"); + } + other => panic!("Expected WaitForNextLineStatement, got {other:?}"), + } +} + +// ----------------------------- runtime tests ------------------------------ + +/// Spawn a one-shot server that answers 200 with the given body, streamed with +/// an explicit Content-Length and `Connection: close`. +async fn spawn_body_server(body: &'static str) -> String { + let listener = TcpListener::bind("127.0.0.1:0").await.unwrap(); + let addr = listener.local_addr().unwrap(); + tokio::spawn(async move { + let (mut socket, _) = listener.accept().await.unwrap(); + let mut tmp = [0u8; 2048]; + // Drain the request head (single read is enough for a GET). + let _ = socket.read(&mut tmp).await; + let response = format!( + "HTTP/1.1 200 OK\r\nContent-Type: application/x-ndjson\r\nContent-Length: {}\r\nConnection: close\r\n\r\n{}", + body.len(), + body + ); + socket.write_all(response.as_bytes()).await.unwrap(); + socket.shutdown().await.ok(); + }); + format!("http://{addr}") +} + +async fn run_wfl(code: &str) -> Interpreter { + let tokens = lex_wfl_with_positions(code); + let mut parser = Parser::new(&tokens); + let program = parser + .parse() + .unwrap_or_else(|e| panic!("Parse error: {e:?}")); + let mut interpreter = Interpreter::new(); + interpreter + .interpret(&program) + .await + .unwrap_or_else(|e| panic!("Runtime error: {e:?}")); + interpreter +} + +fn get_var(interpreter: &Interpreter, name: &str) -> Value { + interpreter + .global_env() + .borrow() + .get(name) + .unwrap_or_else(|| panic!("Variable '{name}' not found")) +} + +fn get_text(interpreter: &Interpreter, name: &str) -> String { + match get_var(interpreter, name) { + Value::Text(t) => t.to_string(), + other => panic!("Expected '{name}' to be text, got {other:?}"), + } +} + +#[tokio::test] +async fn test_stream_exposes_status_and_headers_immediately() { + let url = spawn_body_server("alpha\nbeta\n").await; + let code = format!( + r#" + open url at "{url}" and stream response as up + store s as up["status"] + store ok as up["ok"] + store ct as up["headers"]["content-type"] + close up + "# + ); + let interpreter = run_wfl(&code).await; + + match get_var(&interpreter, "s") { + Value::Number(n) => assert_eq!(n, 200.0), + other => panic!("Expected numeric status, got {other:?}"), + } + match get_var(&interpreter, "ok") { + Value::Bool(b) => assert!(b), + other => panic!("Expected boolean ok, got {other:?}"), + } + assert_eq!(get_text(&interpreter, "ct"), "application/x-ndjson"); +} + +#[tokio::test] +async fn test_next_line_yields_lines_then_nothing_at_eof() { + let url = spawn_body_server("alpha\nbeta\ngamma\n").await; + let code = format!( + r#" + open url at "{url}" and stream response as up + wait for next line from up as line1 + wait for next line from up as line2 + wait for next line from up as line3 + wait for next line from up as line4 + "# + ); + let interpreter = run_wfl(&code).await; + + assert_eq!(get_text(&interpreter, "line1"), "alpha"); + assert_eq!(get_text(&interpreter, "line2"), "beta"); + assert_eq!(get_text(&interpreter, "line3"), "gamma"); + // Clean EOF binds `nothing` (Null). + match get_var(&interpreter, "line4") { + Value::Null => {} + other => panic!("Expected nothing at EOF, got {other:?}"), + } +} + +#[tokio::test] +async fn test_next_line_returns_final_unterminated_line() { + // No trailing newline: the last line is still delivered before EOF. + let url = spawn_body_server("one\ntwo").await; + let code = format!( + r#" + open url at "{url}" and stream response as up + wait for next line from up as a + wait for next line from up as b + wait for next line from up as c + "# + ); + let interpreter = run_wfl(&code).await; + assert_eq!(get_text(&interpreter, "a"), "one"); + assert_eq!(get_text(&interpreter, "b"), "two"); + match get_var(&interpreter, "c") { + Value::Null => {} + other => panic!("Expected nothing at EOF, got {other:?}"), + } +} + +#[tokio::test] +async fn test_next_chunk_yields_binary_then_nothing() { + let url = spawn_body_server("raw-bytes-payload").await; + let code = format!( + r#" + open url at "{url}" and stream response as up + wait for next chunk from up as chunk1 + "# + ); + let interpreter = run_wfl(&code).await; + + match get_var(&interpreter, "chunk1") { + Value::Binary(b) => assert!(!b.is_empty(), "first chunk should carry bytes"), + other => panic!("Expected binary chunk, got {other:?}"), + } +} + +#[tokio::test] +async fn test_reading_from_closed_stream_is_error() { + let url = spawn_body_server("x\n").await; + let code = format!( + r#" + open url at "{url}" and stream response as up + close up + wait for next line from up as line + "# + ); + let tokens = lex_wfl_with_positions(&code); + let mut parser = Parser::new(&tokens); + let program = parser.parse().unwrap(); + let mut interpreter = Interpreter::new(); + let result = interpreter.interpret(&program).await; + assert!( + result.is_err(), + "reading from a closed stream handle should be a catchable error" + ); +} From 6991f58a13ed196aa55b5c6039e07f1692e95cb3 Mon Sep 17 00:00:00 2001 From: Claude Date: Wed, 22 Jul 2026 15:25:05 +0000 Subject: [PATCH 002/132] docs: lock design for streamed server responses (item 3) and streaming roadmap MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Capture the turnkey design for server-side response streaming and the overall response-streaming roadmap, so the next increment is unambiguous: - Chosen surface: `start streaming response to with status .. and content type .. as `, `write line|chunk to `, `flush `, `close ` — consistent with the shipped client streaming and dispatched like the existing `send websocket message` leading-identifier statements. - Mechanism: a HandlerReply enum (Buffered | Streaming{status, content_type, headers, body: mpsc::Receiver}) over the existing per-request oneshot; the warp route's final closure converts to a Body-typed reply (Body::from for the buffered/504 arms, Body::wrap_stream(unfold(rx)) for the streaming arm) while the recover helper stays Vec (warp unifies via Either) — so only the final handler closure changes, not the five reply helpers. - Lifecycle: bounded mpsc for backpressure; a closed receiver (browser disconnect) makes `write` fail catchably, which the handler uses to close the upstream; `close`/drop ends the body; writes after close fail predictably. Also records status of all five requested capabilities: items 1-2 and the client half of item 5 shipped; item 3 designed; item 4 deferred to the locked concurrency-phase-plan (Phase 1 `main loop concurrently:`). Co-Authored-By: Claude Opus 4.8 Claude-Session: https://claude.ai/code/session_011LxXAm5E4Vy39kRLr8kRnX --- Docs/development/response-streaming-design.md | 153 ++++++++++++++++++ 1 file changed, 153 insertions(+) create mode 100644 Docs/development/response-streaming-design.md diff --git a/Docs/development/response-streaming-design.md b/Docs/development/response-streaming-design.md new file mode 100644 index 00000000..e53fad23 --- /dev/null +++ b/Docs/development/response-streaming-design.md @@ -0,0 +1,153 @@ +# WFL Response Streaming — Design & Status + +**Audience:** WFL maintainer + AI implementers +**Origin:** downstream request (2026-07-22) for a browser chat UI proxying a slow +upstream model endpoint to the browser without buffering. + +This tracks the five requested runtime capabilities, what has shipped, and the +locked design for what remains. It complements — and defers to — +`concurrency-phase-plan.md`, which governs item 4. + +--- + +## The five capabilities + +| # | Capability | Status | +|---|------------|--------| +| 1 | Outbound response streaming (`stream response as`) | ✅ Shipped | +| 2 | Incremental reads (`wait for next chunk|line`) | ✅ Shipped | +| 3 | Streamed server responses (start / write / flush / close) | 🔒 Designed (below) | +| 4 | Concurrent request handlers | ⬜ Phase 1 of `concurrency-phase-plan.md` | +| 5 | Lifecycle (timeouts, backpressure, cancellation, catchable errors, close-on-exit) | ◐ Client half shipped; server half rides with 3 & 4 | + +--- + +## Shipped (items 1, 2, client-side 5) + +See `Dev diary/2026-07-22-outbound-response-streaming.md`. Surface: + +```wfl +open url at "" [with method .. and headers .. and body ..] and stream response as upstream +wait for next line from upstream as line // Text, nothing at clean EOF +wait for next chunk from upstream as chunk // Binary, nothing at clean EOF +close upstream +``` + +Handle model: an opaque id into `IoClient.stream_handles`, wrapped in an object +exposing `status`/`ok`/`headers`/`_stream`. Reads go through the existing +`run_http_with_budget` select (timeouts + cancellation), enforce the +response-byte ceiling on the running total, are individually catchable, and drop +the handle (cancelling the upstream) on EOF/error/close/teardown. + +--- + +## Item 3 — Streamed server responses (locked surface) + +### Surface (chosen for consistency with the client side + existing `respond`) + +```wfl +// Send status + headers immediately; body stays open. +start streaming response to req with status 200 and content type "application/x-ndjson" as out + +// Write body pieces. `write line` appends a newline (NDJSON-friendly); +// `write chunk` writes raw bytes/text verbatim. +write line json_text to out +write chunk raw_bytes to out + +// Advisory: hand queued bytes to the transport (yields to the runtime). +flush out + +// End the response body. +close out +``` + +- `start streaming response` leads with the merged identifier `start streaming` + followed by the `response` keyword — dispatched like `send websocket message` + in `parser/mod.rs`. (Distinguishable from any future Phase-2 `start + as `.) +- `write line|chunk to ` — branch inside the `write` dispatch on a + following `line`/`chunk` identifier; `to` (not `into`) distinguishes it from + file writes. +- `flush ` — leading identifier `flush`. +- `close out` — reuse `CloseFileStatement`, extended for a `_server_stream` + object (as it already was for `_stream`). + +### Mechanism + +- **Reply payload becomes an enum.** Replace `oneshot::Sender` + with `oneshot::Sender` where + + ```rust + enum HandlerReply { + Buffered(WflHttpResponse), + Streaming { status: u16, content_type: String, + headers: HashMap, + body: mpsc::Receiver> }, + } + ``` + + Update `PendingResponseSender`, `WflHttpRequest.response_sender`, + `ResponseCompletion` (its `Drop` sends `Buffered` 500), and the `respond` path + (builds `Buffered`). + +- **Transport reply type becomes `warp::hyper::Body`.** Convert the five reply + helpers (`overloaded_response`, `plain_status_response`, + `payload_too_large_response`, `gateway_timeout_response`, + `request_timeout_response`), `handle_overloaded`, and the main route's + buffered arm from `Response>` to `Response` (`.body(Body::from( + bytes))`). The streaming arm builds + `Body::wrap_stream(futures_util::stream::unfold(rx, ...))` (no new dep). The + separate redirect `warp::serve` route is untouched. + +- **`start streaming response`** creates a bounded `mpsc::channel::>` + (capacity = backpressure knob, align with existing web limits), sends + `HandlerReply::Streaming { head.., body: rx }` over the request's oneshot, and + stores the `tx` in a new interpreter-side map + `server_response_streams: RefCell>`. Binds + `out` = object `{ _server_stream: }`. + +- **`write line|chunk`** resolves `_server_stream`, `tx.send(bytes).await` + (bounded → backpressure). A closed receiver (browser disconnected / hyper + dropped the body) makes `send` fail → surfaced as a catchable error, letting + the handler `close upstream` and stop — this is how browser-disconnect + cancellation propagates to the upstream (item 5, cooperatively). + +- **`flush`** = advisory; `tokio::task::yield_now().await` so the transport task + is scheduled. Documented as advisory (hyper already writes as it receives). + +- **`close out`** drops the `tx` → ends the body stream → hyper finalizes the + response. Idempotent; writes after close fail predictably. + +### Lifecycle (item 5, server side) + +- Timeouts: the existing per-request `overall_deadline` bounds head delivery; + document that a stalled *body producer* is bounded by the handler timeout at + await points (the yield-cliff caveat from the concurrency plan). +- Backpressure: bounded `mpsc` — a slow browser slows the handler's `write`. +- Disconnect → upstream cancel: `write` error path (above). +- Close-on-exit: dropping `ServerStreamHandle` (handler end, error, teardown) + drops `tx`, ending the response. + +### Tests (write first) + +Parser tests for all four statements; runtime tests via a client that reads the +streamed body: status/headers arrive before the body, `write line` frames NDJSON, +`close` ends the stream, a dropped client makes `write` fail catchably. + +--- + +## Item 4 — Concurrent handlers + +Governed by `concurrency-phase-plan.md` (Phase 1, locked marker `main loop +concurrently:`, no Rc→Arc, TDD-first). It is the keystone that makes a slow +streamed response (items 1–3) not stall login/history/health. Server streaming +(item 3) is deliberately usable on the serial loop first; true isolation between +a slow stream and other requests arrives with Phase 1. + +--- + +## Related + +- `Docs/04-advanced-features/interoperability.md` — user docs (client streaming shipped) +- `Docs/development/concurrency-phase-plan.md` — item 4 governance +- `Dev diary/2026-07-22-outbound-response-streaming.md` From 73e66dd2dd9ba50b99a27df1eb062ae7e333f555 Mon Sep 17 00:00:00 2001 From: Claude Date: Wed, 22 Jul 2026 15:44:11 +0000 Subject: [PATCH 003/132] feat: streamed server responses (start streaming response / write line|chunk / flush / close) Add server-side response streaming so a WFL handler can send status/headers immediately and produce the body progressively, without buffering: start streaming response to req with status 200 and content type "application/x-ndjson" as out write line json_text to out // frames a line (newline appended) write chunk raw_bytes to out // raw bytes/text, verbatim flush out // advisory close out // ends the response body Combined with the client-side `stream response as upstream` + `wait for next line`, a handler can proxy a slow upstream to the browser line-by-line. Mechanism: - The per-request oneshot now carries a HandlerReply enum (Buffered(WflHttpResponse) | Streaming{status, content_type, headers, body: mpsc::Receiver}). `respond` sends Buffered; `start streaming response` sends Streaming with a bounded body channel's receiver. - The warp route's final closure returns a Body-typed reply. warp's recover unifies reply types via Either, so only that closure changed: it wraps the buffered/504 arms with Body::from and builds the streaming arm with Body::wrap_stream(unfold(rx)) (no new dependency); the five Vec reply helpers are untouched. - Backpressure: the body channel is bounded (64) so a slow client backpressures `write`. Disconnect: hyper drops the body -> receiver dropped -> the next `write` fails with a catchable error, propagating the disconnect to the handler (which can close any upstream it proxies). `close`/handler-exit drops the sender, ending the response. - `start` is a keyword; `streaming`/`flush`/`line`/`chunk` are identifiers. `close` now also closes a server response stream (`_server_stream`). - AST: StartStreamingResponseStatement, StreamWriteStatement, FlushStreamStatement - Parser: parse_start_streaming_response/parse_flush_stream (web.rs), write line|chunk branch (io.rs), KeywordStart/flush dispatch (parser/mod.rs) - Interpreter: server_response_streams map, three exec arms, close extension - Analyzer/typechecker/transpiler arms - Docs: web-servers.md "Streaming a response" (+ upstream-proxy example); response-streaming-design.md status; validated example; Dev Diary - Tests: tests/http_server_streaming_test.rs (parser + end-to-end streaming server read back with reqwest); existing web-server suites still green Co-Authored-By: Claude Opus 4.8 Claude-Session: https://claude.ai/code/session_011LxXAm5E4Vy39kRLr8kRnX --- .../2026-07-22-server-response-streaming.md | 84 ++++ Docs/04-advanced-features/web-servers.md | 65 +++ Docs/development/response-streaming-design.md | 14 +- .../docs_examples/_meta/manifest.json | 17 + .../web_servers/streaming_response.wfl | 22 + .../streaming_response.wfl.ast.txt | 154 ++++++ src/analyzer/mod.rs | 38 ++ src/interpreter/mod.rs | 469 ++++++++++++++++-- src/parser/ast.rs | 32 ++ src/parser/mod.rs | 8 + src/parser/stmt/io.rs | 64 +++ src/parser/stmt/web.rs | 168 +++++++ src/transpiler/javascript.rs | 7 +- src/typechecker/mod.rs | 32 ++ tests/http_server_streaming_test.rs | 166 +++++++ 15 files changed, 1308 insertions(+), 32 deletions(-) create mode 100644 Dev diary/2026-07-22-server-response-streaming.md create mode 100644 TestPrograms/docs_examples/web_servers/streaming_response.wfl create mode 100644 TestPrograms/docs_examples/web_servers/streaming_response.wfl.ast.txt create mode 100644 tests/http_server_streaming_test.rs diff --git a/Dev diary/2026-07-22-server-response-streaming.md b/Dev diary/2026-07-22-server-response-streaming.md new file mode 100644 index 00000000..b4574683 --- /dev/null +++ b/Dev diary/2026-07-22-server-response-streaming.md @@ -0,0 +1,84 @@ +# Dev Diary — 2026-07-22 — Streamed server responses + +## Context + +Follow-on to the same-day outbound response streaming work. This adds the +**server** half (item 3 of the five-capability request): a WFL handler can now +send a response whose body is produced progressively — status/headers first, +then body pieces — which is what a browser chat UI needs to read newline- +delimited JSON off a `fetch()` response as it arrives. + +## What shipped + +```wfl +start streaming response to req with status 200 and content type "application/x-ndjson" as out +write line json_text to out // frames a line (newline appended) +write chunk raw_bytes to out // raw bytes/text, verbatim +flush out // advisory +close out // ends the response body +``` + +`start streaming response` returns immediately after sending the head and binds +a stream handle (`{ _server_stream, status }`). Combined with the client-side +`stream response as upstream` + `wait for next line`, a handler can proxy a slow +upstream to the browser line-by-line without buffering either side. + +## Design & mechanism + +- **Reply payload is now an enum.** The per-request `oneshot` carries a + `HandlerReply` = `Buffered(WflHttpResponse)` | `Streaming { status, + content_type, headers, body: mpsc::Receiver> }`. `respond` sends + `Buffered`; `start streaming response` sends `Streaming` with the receiving end + of a bounded body channel. +- **Transport reply became `Body`-typed.** The warp route's final closure now + returns `Response`. Key simplification: warp's `.recover()` + unifies reply types via `Either`, so only that one closure changed — the five + `Response>` helper functions and `handle_overloaded` were left alone; + the closure wraps their returns with `.map(Body::from)`. The streaming arm + builds `Body::wrap_stream(futures_util::stream::unfold(rx, ...))` — no new + dependency. +- **Backpressure & disconnect.** The body channel is bounded + (`RESPONSE_STREAM_BUFFER = 64`), so a slow client backpressures the handler's + `write` (it awaits a free slot). When the client disconnects, hyper drops the + body, dropping the receiver; the handler's next `write` then fails with a + catchable error — that is how a browser disconnect propagates to the handler + (which can then `close` the upstream it is proxying). `close` (or handler exit) + drops the sender, ending the response. +- **`start` is a keyword, `streaming`/`flush`/`line`/`chunk` are identifiers.** + `start streaming response` dispatches on `Token::KeywordStart`; `flush ` + and `write line|chunk to ` handle the lexer's identifier-merging + (`flush out`, `line payload`) the same way the websocket-message statements do. +- **`close` unified.** `close ` now closes a file (`Text`), a client upstream + (`_stream`), or a server response stream (`_server_stream`). + +## Files + +- Types + transport + exec: `src/interpreter/mod.rs` (`HandlerReply`, + `Body`-typed closure, `server_response_streams` map, three statement arms, + `close` extension). +- AST: `StartStreamingResponseStatement`, `StreamWriteStatement`, + `FlushStreamStatement`. +- Parser: `parse_start_streaming_response`, `parse_flush_stream` + (`src/parser/stmt/web.rs`), `write line|chunk` branch + (`src/parser/stmt/io.rs`), `KeywordStart`/`flush` dispatch (`parser/mod.rs`). +- Analyzer/typechecker/transpiler arms. +- Docs: `Docs/04-advanced-features/web-servers.md` ("Streaming a response", + incl. an upstream-proxy example) + validated example + `TestPrograms/docs_examples/web_servers/streaming_response.wfl`. + +## Tests + +`tests/http_server_streaming_test.rs` — parser tests for all four statements, +plus end-to-end runtime tests that stand up a WFL streaming server and read it +back with reqwest: status/headers arrive with `write line` framing +(`alpha\nbeta\ngamma\n`), and `write chunk` is verbatim (`onetwo`). The existing +web-server suites (query/binary/content-length/queue-bound/admission) still pass +after the transport reply-type change; `fmt`, `clippy -D warnings`, and the 618 +lib tests are green. + +## Still open + +- **Item 4 — concurrent handlers** (`main loop concurrently:`), Phase 1 of the + locked concurrency plan: today a streamed handler runs to completion before the + next request is served, so per-request isolation between a slow stream and + other requests still awaits Phase 1. diff --git a/Docs/04-advanced-features/web-servers.md b/Docs/04-advanced-features/web-servers.md index 41f33356..6c577d24 100644 --- a/Docs/04-advanced-features/web-servers.md +++ b/Docs/04-advanced-features/web-servers.md @@ -412,6 +412,71 @@ respond to req with "Created!" and status 201 and content_type "application/json All optional clauses (`status`, `content_type`, `headers`) can appear in any order after the content. +### Streaming a response + +`respond to` sends the whole body at once. When the body is large, or produced +progressively (for example newline-delimited JSON streamed to a browser's +`fetch()` reader), start a **streaming response** instead. It sends the status +and headers immediately and binds a stream handle you write to piece by piece: + +```wfl +start streaming response to req with status 200 and content type "application/x-ndjson" as out + +write line "{\"event\": \"start\"}" to out +write line "{\"event\": \"tick\", \"n\": 1}" to out +flush out +write line "{\"event\": \"done\"}" to out + +close out +``` + +- `start streaming response to [with status ] [and content type ] + [and headers ] as ` — begin the response. The status defaults to 200 + and the content type to `application/octet-stream`. The body has no declared + length; it is sent with chunked transfer-encoding. +- `write line to ` — write `value` followed by a newline (ideal for + NDJSON). `value` may be text, a number, or a boolean. +- `write chunk to ` — write raw bytes verbatim, no newline added. + `value` may be text or `binary`. +- `flush ` — advisory: yield so queued bytes are handed to the socket. + (Chunks are already forwarded as you write them; hyper writes as it receives.) +- `close ` — end the response body. Writing after `close` is an error. + +**Lifecycle & backpressure:** the body channel is bounded, so a slow client +slows your `write` calls (backpressure) instead of buffering without bound. If +the client disconnects, hyper drops the response body and your next `write` to +that stream fails with a catchable error — use `try`/`when` to detect it and +stop producing (and `close` any upstream you are proxying). The stream is closed +automatically when the handler ends on any path. + +**Proxying an upstream to the browser** — combine with the outbound streaming +client ([Interoperability → Streaming a response +incrementally](interoperability.md#streaming-a-response-incrementally)): + +```wfl +open url at "https://model.example.com/generate" with method "POST" and body prompt and stream response as upstream +start streaming response to req with status 200 and content type "application/x-ndjson" as out + +count from 1 to 1000000: + wait for next line from upstream as line + check if line is nothing: + break + otherwise: + write line line to out + end check +end count + +close upstream +close out +``` + +> **Concurrency note:** today a request handler runs to completion before the +> next request is served (the loop is cooperative and single-threaded), so a +> long-running stream occupies the handler until it finishes. Opt-in concurrent +> handling (`main loop concurrently:`) is a planned, separate capability — see +> [`concurrency-phase-plan.md`](../development/concurrency-phase-plan.md). Until +> then, prefer bounded streams and per-request timeouts. + ## The QUERY Method (RFC 10008) WFL supports [RFC 10008](https://www.rfc-editor.org/info/rfc10008/), the HTTP diff --git a/Docs/development/response-streaming-design.md b/Docs/development/response-streaming-design.md index e53fad23..41b1b0b1 100644 --- a/Docs/development/response-streaming-design.md +++ b/Docs/development/response-streaming-design.md @@ -16,9 +16,9 @@ locked design for what remains. It complements — and defers to — |---|------------|--------| | 1 | Outbound response streaming (`stream response as`) | ✅ Shipped | | 2 | Incremental reads (`wait for next chunk|line`) | ✅ Shipped | -| 3 | Streamed server responses (start / write / flush / close) | 🔒 Designed (below) | +| 3 | Streamed server responses (start / write / flush / close) | ✅ Shipped | | 4 | Concurrent request handlers | ⬜ Phase 1 of `concurrency-phase-plan.md` | -| 5 | Lifecycle (timeouts, backpressure, cancellation, catchable errors, close-on-exit) | ◐ Client half shipped; server half rides with 3 & 4 | +| 5 | Lifecycle (timeouts, backpressure, cancellation, catchable errors, close-on-exit) | ◐ Client + server body streaming shipped; per-handler isolation rides with 4 | --- @@ -41,7 +41,15 @@ the handle (cancelling the upstream) on EOF/error/close/teardown. --- -## Item 3 — Streamed server responses (locked surface) +## Item 3 — Streamed server responses (✅ shipped) + +Shipped as designed below. Surface: `start streaming response to [with +status ] [and content type ] [and headers ] as `, `write +line|chunk to `, `flush `, `close `. See +`tests/http_server_streaming_test.rs` and the web-servers guide's "Streaming a +response" section. The original design (kept for reference): + +## Item 3 — Streamed server responses (design) ### Surface (chosen for consistency with the client side + existing `respond`) diff --git a/TestPrograms/docs_examples/_meta/manifest.json b/TestPrograms/docs_examples/_meta/manifest.json index 8386c863..f1843057 100644 --- a/TestPrograms/docs_examples/_meta/manifest.json +++ b/TestPrograms/docs_examples/_meta/manifest.json @@ -418,5 +418,22 @@ "interoperability" ], "description": "Streaming an outbound HTTP response incrementally with stream response / wait for next line." + }, + "docs_examples/web_servers/streaming_response.wfl": { + "doc_section": "Docs/04-advanced-features/web-servers.md#streaming-a-response", + "type": "snippet", + "validate_layers": [ + 1, + 2, + 3, + 4 + ], + "skip_execution": true, + "tags": [ + "web-server", + "streaming", + "response" + ], + "description": "Streaming a server response with start streaming response / write line / flush / close." } } diff --git a/TestPrograms/docs_examples/web_servers/streaming_response.wfl b/TestPrograms/docs_examples/web_servers/streaming_response.wfl new file mode 100644 index 00000000..c2a4424b --- /dev/null +++ b/TestPrograms/docs_examples/web_servers/streaming_response.wfl @@ -0,0 +1,22 @@ +// Streaming a server response. +// +// `start streaming response` sends status/headers immediately and binds a +// stream handle; `write line`/`write chunk` append body pieces incrementally, +// and `close` ends the response. +// +// Validated for syntax/analysis/lint only (layers 1-4): running it needs a +// live client, so execution is skipped. + +listen on port 8080 as site + +wait for request comes in on site as req with timeout 10000 + +start streaming response to req with status 200 and content type "application/x-ndjson" as out + +write line "first line" to out +write line "second line" to out +flush out +write chunk "no newline here" to out + +close out +close server site diff --git a/TestPrograms/docs_examples/web_servers/streaming_response.wfl.ast.txt b/TestPrograms/docs_examples/web_servers/streaming_response.wfl.ast.txt new file mode 100644 index 00000000..3bb11606 --- /dev/null +++ b/TestPrograms/docs_examples/web_servers/streaming_response.wfl.ast.txt @@ -0,0 +1,154 @@ +AST output for: TestPrograms/docs_examples/web_servers/streaming_response.wfl +============================================== + +Program with 9 statements: + +Statement #1: ListenStatement { + port: Literal( + Integer( + 8080, + ), + 10, + 16, + ), + server_name: "site", + tls: None, + redirect_to_port: None, + line: 10, + column: 1, +} + +Statement #2: WaitForRequestStatement { + server: Variable( + "site", + 12, + 30, + ), + request_name: "req", + timeout: Some( + Literal( + Integer( + 10000, + ), + 12, + 55, + ), + ), + line: 12, + column: 1, +} + +Statement #3: StartStreamingResponseStatement { + request: Variable( + "req", + 14, + 29, + ), + status: Some( + Literal( + Integer( + 200, + ), + 14, + 45, + ), + ), + content_type: Some( + Literal( + String( + "application/x-ndjson", + ), + 14, + 66, + ), + ), + headers: None, + variable_name: "out", + line: 14, + column: 1, +} + +Statement #4: StreamWriteStatement { + value: Literal( + String( + "first line", + ), + 16, + 12, + ), + target: Variable( + "out", + 16, + 28, + ), + is_line: true, + line: 16, + column: 1, +} + +Statement #5: StreamWriteStatement { + value: Literal( + String( + "second line", + ), + 17, + 12, + ), + target: Variable( + "out", + 17, + 29, + ), + is_line: true, + line: 17, + column: 1, +} + +Statement #6: FlushStreamStatement { + target: Variable( + "out", + 18, + 1, + ), + line: 18, + column: 1, +} + +Statement #7: StreamWriteStatement { + value: Literal( + String( + "no newline here", + ), + 19, + 13, + ), + target: Variable( + "out", + 19, + 34, + ), + is_line: false, + line: 19, + column: 1, +} + +Statement #8: CloseFileStatement { + file: Variable( + "out", + 21, + 7, + ), + line: 21, + column: 1, +} + +Statement #9: CloseServerStatement { + server: Variable( + "site", + 22, + 14, + ), + line: 22, + column: 1, +} + diff --git a/src/analyzer/mod.rs b/src/analyzer/mod.rs index bae8f627..7a4882f5 100644 --- a/src/analyzer/mod.rs +++ b/src/analyzer/mod.rs @@ -1589,6 +1589,44 @@ impl Analyzer { self.current_scope.define_or_replace(symbol); } + Statement::StartStreamingResponseStatement { + request, + status, + content_type, + headers, + variable_name, + .. + } => { + self.analyze_expression(request); + if let Some(status) = status { + self.analyze_expression(status); + } + if let Some(content_type) = content_type { + self.analyze_expression(content_type); + } + if let Some(headers) = headers { + self.analyze_expression(headers); + } + // Binds a server response-stream handle object. + let symbol = Symbol { + name: variable_name.clone(), + kind: SymbolKind::Variable { mutable: true }, + symbol_type: None, + line: 0, + column: 0, + }; + self.current_scope.define_or_replace(symbol); + } + + Statement::StreamWriteStatement { value, target, .. } => { + self.analyze_expression(value); + self.analyze_expression(target); + } + + Statement::FlushStreamStatement { target, .. } => { + self.analyze_expression(target); + } + Statement::CreateDirectoryStatement { path, .. } => { self.analyze_expression(path); } diff --git a/src/interpreter/mod.rs b/src/interpreter/mod.rs index 92510094..748fdfd3 100644 --- a/src/interpreter/mod.rs +++ b/src/interpreter/mod.rs @@ -63,7 +63,7 @@ use std::time::{Duration, Instant}; use tokio::sync::{mpsc, oneshot}; // Type alias for complex pending response type -type PendingResponseSender = Arc>>>; +type PendingResponseSender = Arc>>>; /// A dequeued HTTP request parked in `pending_responses` awaiting a `respond`. /// @@ -89,6 +89,11 @@ use warp::Filter; /// `count & (STRIDE - 1)` is exact; large enough that the yield is negligible. const COOP_YIELD_STRIDE: u64 = 1024; +/// Bounded capacity of a server response stream's body-chunk channel. A slow +/// client fills this and then backpressures the handler's `write` (it awaits a +/// free slot) rather than letting queued chunks grow without bound. +const RESPONSE_STREAM_BUFFER: usize = 64; + // Web server data structures #[derive(Debug)] pub struct WflHttpRequest { @@ -105,7 +110,7 @@ pub struct WflHttpRequest { /// binary value, so binary uploads survive intact. pub body: Vec, pub headers: HashMap, - pub response_sender: Arc>>>, + pub response_sender: Arc>>>, } #[derive(Debug, Clone)] @@ -119,6 +124,26 @@ pub struct WflHttpResponse { pub headers: HashMap, } +/// What a request handler delivers back to its warp transport task over the +/// per-request `oneshot`. +/// +/// `respond` sends a fully-buffered `Buffered` reply; `start streaming response` +/// sends a `Streaming` reply whose head (status/headers) is written immediately +/// and whose body is fed chunk-by-chunk over a bounded channel by `write +/// line|chunk`. A bounded channel gives backpressure: a slow client slows the +/// handler's writes. Dropping the sender (handler end, `close`, or a caught +/// error) closes the body stream and finalizes the response. +#[derive(Debug)] +pub enum HandlerReply { + Buffered(WflHttpResponse), + Streaming { + status: u16, + content_type: String, + headers: HashMap, + body: mpsc::Receiver>, + }, +} + /// Ensures an HTTP `respond` always resolves its request. The response sender is /// taken out of `pending_responses` (and out of its mutex) up front and held /// here; if a fallible step in `respond` returns early before a response is @@ -127,11 +152,11 @@ pub struct WflHttpResponse { /// [`ResponseCompletion::take_sender`] to disarm the fallback and deliver the /// real response. struct ResponseCompletion { - sender: Option>, + sender: Option>, } impl ResponseCompletion { - fn take_sender(&mut self) -> Option> { + fn take_sender(&mut self) -> Option> { self.sender.take() } } @@ -139,12 +164,12 @@ impl ResponseCompletion { impl Drop for ResponseCompletion { fn drop(&mut self) { if let Some(sender) = self.sender.take() { - let _ = sender.send(WflHttpResponse { + let _ = sender.send(HandlerReply::Buffered(WflHttpResponse { content: b"Internal Server Error\n".to_vec(), status: 500, content_type: "text/plain; charset=utf-8".to_string(), headers: HashMap::new(), - }); + })); } } } @@ -902,6 +927,13 @@ fn stmt_type(stmt: &Statement) -> String { Statement::WaitForNextLineStatement { variable_name, .. } => { format!("WaitForNextLineStatement '{variable_name}'") } + Statement::StartStreamingResponseStatement { variable_name, .. } => { + format!("StartStreamingResponseStatement '{variable_name}'") + } + Statement::StreamWriteStatement { is_line, .. } => { + format!("StreamWriteStatement (line={is_line})") + } + Statement::FlushStreamStatement { .. } => "FlushStreamStatement".to_string(), Statement::PushStatement { .. } => "PushStatement to list".to_string(), Statement::CreateListStatement { name, .. } => format!("CreateListStatement '{name}'"), Statement::MapCreation { name, .. } => format!("MapCreation '{name}'"), @@ -1109,6 +1141,11 @@ pub struct Interpreter { web_socket_servers: RefCell>, // WebSocket servers keyed by address ws_connections: WsConnectionRegistry, // Outbound senders for all live WebSocket connections pending_responses: RefCell>, // Pending responses (channel + admission slot) by request ID + /// Open server response streams (`start streaming response`), keyed by + /// handle id ("respstream1", ...). Each holds the bounded body-chunk sender; + /// `write line|chunk`/`flush` push to it, `close` drops it (ending the body). + server_response_streams: RefCell>>>, + next_response_stream_id: std::cell::Cell, #[allow(dead_code)] // Used for future security features config: Arc, // Configuration for security and other settings current_source_file: RefCell>, // Currently executing source file (for path resolution) @@ -3158,6 +3195,8 @@ impl Interpreter { web_socket_servers: RefCell::new(HashMap::new()), // Initialize empty WebSocket servers map ws_connections: Arc::new(std::sync::Mutex::new(HashMap::new())), // Live WebSocket connections pending_responses: RefCell::new(HashMap::new()), // Initialize empty pending responses map + server_response_streams: RefCell::new(HashMap::new()), + next_response_stream_id: std::cell::Cell::new(1), config, current_source_file: RefCell::new(None), // No source file initially loading_stack: RefCell::new(Vec::new()), // Empty loading stack @@ -3648,6 +3687,40 @@ impl Interpreter { } } + /// Resolve a `write line|chunk`/`flush` operand to a server response-stream + /// handle id. Accepts the object bound by `start streaming response as ...` + /// (reads its internal `_server_stream` id) or a bare text handle id. + async fn resolve_server_stream_handle( + &self, + target: &Expression, + env: &Rc>, + line: usize, + column: usize, + ) -> Result { + let value = self.evaluate_expression(target, Rc::clone(env)).await?; + match &value { + Value::Object(obj) => match obj.borrow().get("_server_stream") { + Some(Value::Text(id)) => Ok(id.to_string()), + _ => Err(RuntimeError::new( + "Expected a server response stream (from `start streaming response as ...`), \ + but this value has no open stream" + .to_string(), + line, + column, + )), + }, + Value::Text(id) => Ok(id.to_string()), + _ => Err(RuntimeError::new( + format!( + "Expected a server response stream handle, got {}", + value.type_name() + ), + line, + column, + )), + } + } + /// Preserve ordinary file I/O failures while classifying byte-ceiling /// breaches as catchable execution-budget resource errors. fn file_read_error(&self, error: FileReadError, line: usize, column: usize) -> RuntimeError { @@ -4186,6 +4259,9 @@ impl Interpreter { Statement::HttpStreamStatement { line, column, .. } => (*line, *column), Statement::WaitForNextChunkStatement { line, column, .. } => (*line, *column), Statement::WaitForNextLineStatement { line, column, .. } => (*line, *column), + Statement::StartStreamingResponseStatement { line, column, .. } => (*line, *column), + Statement::StreamWriteStatement { line, column, .. } => (*line, *column), + Statement::FlushStreamStatement { line, column, .. } => (*line, *column), Statement::PushStatement { line, column, .. } => (*line, *column), Statement::CreateListStatement { line, column, .. } => (*line, *column), Statement::MapCreation { line, column, .. } => (*line, *column), @@ -5161,26 +5237,39 @@ impl Interpreter { Ok(_) => Ok((Value::Null, ControlFlow::None)), Err(e) => Err(RuntimeError::new(e, *line, *column)), }, - // A streaming-response object closes its underlying stream, - // cancelling the in-flight upstream request. `close` on a - // stream is always safe, even after EOF. + // A streaming-response object closes its underlying stream. + // For a client upstream (`_stream`) this cancels the in-flight + // request; for a server response stream (`_server_stream`) + // this drops the body sender, ending the response. Closing a + // stream is always safe, even after EOF / already closed. Value::Object(obj) => { - let stream_id = obj.borrow().get("_stream").and_then(|v| match v { - Value::Text(s) => Some(s.to_string()), - _ => None, - }); - match stream_id { - Some(id) => { - self.io_client.close_stream(&id).await; - Ok((Value::Null, ControlFlow::None)) - } - None => Err(RuntimeError::new( + let (client_id, server_id) = { + let obj_ref = obj.borrow(); + let client = obj_ref.get("_stream").and_then(|v| match v { + Value::Text(s) => Some(s.to_string()), + _ => None, + }); + let server = obj_ref.get("_server_stream").and_then(|v| match v { + Value::Text(s) => Some(s.to_string()), + _ => None, + }); + (client, server) + }; + if let Some(id) = client_id { + self.io_client.close_stream(&id).await; + Ok((Value::Null, ControlFlow::None)) + } else if let Some(id) = server_id { + // Dropping the sender ends the response body stream. + self.server_response_streams.borrow_mut().remove(&id); + Ok((Value::Null, ControlFlow::None)) + } else { + Err(RuntimeError::new( "Cannot close this value: it is not a file handle or a \ streaming response" .to_string(), *line, *column, - )), + )) } } _ => Err(RuntimeError::new( @@ -7520,7 +7609,8 @@ impl Interpreter { .map(|a| a.ip().to_string()) .unwrap_or_else(|| "unknown".to_string()), ); - return Ok(request_timeout_response()); + return Ok(request_timeout_response() + .map(warp::hyper::Body::from)); } }, None => read_fut.await, @@ -7528,7 +7618,9 @@ impl Interpreter { let body_bytes = match body_read { Ok(bytes) => bytes, Err(BodyReadError::TooLarge) => { - return Ok(payload_too_large_response()); + return Ok( + payload_too_large_response().map(warp::hyper::Body::from) + ); } Err(BodyReadError::Io) => { return Err(warp::reject::custom(ServerError( @@ -7555,7 +7647,7 @@ impl Interpreter { // Create response channel let (response_sender, response_receiver) = - oneshot::channel::(); + oneshot::channel::(); // Create the WFL request. The admission guard is // NOT moved in — it stays bound to this transport @@ -7591,7 +7683,9 @@ impl Interpreter { shed.path, shed.client_ip ); - return Ok(overloaded_response()); + return Ok( + overloaded_response().map(warp::hyper::Body::from) + ); } Err(mpsc::error::TrySendError::Closed(_)) => { return Err(warp::reject::custom(ServerError( @@ -7618,7 +7712,8 @@ impl Interpreter { .map(|a| a.ip().to_string()) .unwrap_or_else(|| "unknown".to_string()), ); - return Ok(gateway_timeout_response()); + return Ok(gateway_timeout_response() + .map(warp::hyper::Body::from)); } } } @@ -7626,7 +7721,7 @@ impl Interpreter { }; match received { - Ok(response) => { + Ok(HandlerReply::Buffered(response)) => { let status_code = warp::http::StatusCode::from_u16(response.status) .unwrap_or(warp::http::StatusCode::OK); @@ -7648,13 +7743,56 @@ impl Interpreter { reply_builder = reply_builder.header(name, value); } - match reply_builder.body(content_bytes) { + match reply_builder.body(warp::hyper::Body::from(content_bytes)) { Ok(response) => Ok(response), Err(_) => Err(warp::reject::custom(ServerError( "Failed to build response".to_string(), ))), } } + Ok(HandlerReply::Streaming { + status, + content_type, + headers, + body, + }) => { + let status_code = warp::http::StatusCode::from_u16(status) + .unwrap_or(warp::http::StatusCode::OK); + + // No Content-Length: the body length is unknown + // up front. hyper frames it with chunked + // transfer-encoding and writes each chunk as it + // arrives off the bounded channel. + let mut reply_builder = warp::http::Response::builder() + .status(status_code) + .header("Content-Type", content_type); + for (name, value) in headers { + reply_builder = reply_builder.header(name, value); + } + + // Turn the chunk receiver into a body stream. + // When the client disconnects, hyper drops this + // body, dropping `body`, which makes the + // handler's next `write` fail (the interpreter + // observes the closed channel) — that is how a + // browser disconnect cancels the handler. + let stream = futures_util::stream::unfold( + body, + |mut rx| async move { + rx.recv() + .await + .map(|chunk| (Ok::, std::io::Error>(chunk), rx)) + }, + ); + match reply_builder + .body(warp::hyper::Body::wrap_stream(stream)) + { + Ok(response) => Ok(response), + Err(_) => Err(warp::reject::custom(ServerError( + "Failed to build streaming response".to_string(), + ))), + } + } Err(_) => Err(warp::reject::custom(ServerError( "Response channel closed".to_string(), ))), @@ -8414,7 +8552,7 @@ impl Interpreter { // sender was taken up front, so this is the sole delivery path. match completion.take_sender() { Some(sender) => { - if sender.send(response).is_err() { + if sender.send(HandlerReply::Buffered(response)).is_err() { return Err(RuntimeError::new( "Failed to send response - client may have disconnected" .to_string(), @@ -8434,6 +8572,281 @@ impl Interpreter { Ok((Value::Null, ControlFlow::None)) } + Statement::StartStreamingResponseStatement { + request, + status, + content_type, + headers, + variable_name, + line, + column, + } => { + // Resolve the request id, mirroring `respond`. + let request_val = self.evaluate_expression(request, Rc::clone(&env)).await?; + let request_id = match &request_val { + Value::Object(obj) => match obj.borrow().get("_response_sender") { + Some(Value::Text(id)) => id.as_ref().to_string(), + _ => { + return Err(RuntimeError::new( + "Request object missing response sender ID".to_string(), + *line, + *column, + )); + } + }, + _ => { + return Err(RuntimeError::new( + "Expected request object".to_string(), + *line, + *column, + )); + } + }; + + // Take the oneshot into an RAII guard up front: an early error + // while evaluating status/content type/headers still resolves + // the client with 500 instead of hanging. + let pending_entry = { + let mut pending = self.pending_responses.borrow_mut(); + pending.remove(&request_id) + }; + let mut completion = match pending_entry { + Some(entry) => match entry.sender.lock().await.take() { + Some(sender) => ResponseCompletion { + sender: Some(sender), + }, + None => { + return Err(RuntimeError::new( + "Response already sent for this request".to_string(), + *line, + *column, + )); + } + }, + None => { + return Err(RuntimeError::new( + "Request ID not found - response may have already been sent" + .to_string(), + *line, + *column, + )); + } + }; + + let status_code = match status { + Some(expr) => { + let v = self.evaluate_expression(expr, Rc::clone(&env)).await?; + match &v { + Value::Number(n) => *n as u16, + _ => { + return Err(RuntimeError::new( + format!("Expected number for status, got {}", v.type_name()), + *line, + *column, + )); + } + } + } + None => 200, + }; + + let content_type_str = match content_type { + Some(expr) => { + let v = self.evaluate_expression(expr, Rc::clone(&env)).await?; + match &v { + Value::Text(s) => s.to_string(), + _ => { + return Err(RuntimeError::new( + format!( + "Expected text for content type, got {}", + v.type_name() + ), + *line, + *column, + )); + } + } + } + None => "application/octet-stream".to_string(), + }; + + let mut custom_headers: HashMap = HashMap::new(); + if let Some(expr) = headers { + let v = self.evaluate_expression(expr, Rc::clone(&env)).await?; + match &v { + Value::Object(obj) => { + for (name, value) in obj.borrow().iter() { + let value_str = match value { + Value::Text(s) => s.to_string(), + Value::Number(_) | Value::Bool(_) => value.to_string(), + _ => { + return Err(RuntimeError::new( + format!( + "Header '{name}' must be text, got {}", + value.type_name() + ), + *line, + *column, + )); + } + }; + // The pipeline owns framing headers. + if name.eq_ignore_ascii_case("content-type") + || name.eq_ignore_ascii_case("content-length") + || name.eq_ignore_ascii_case("transfer-encoding") + { + continue; + } + custom_headers.insert(name.clone(), value_str); + } + } + _ => { + return Err(RuntimeError::new( + format!( + "Expected a map for response headers, got {}", + v.type_name() + ), + *line, + *column, + )); + } + } + } + + // Hand the streaming head (and body receiver) to the transport, + // disarming the guard's 500 fallback. + let (tx, rx) = mpsc::channel::>(RESPONSE_STREAM_BUFFER); + match completion.take_sender() { + Some(sender) => { + if sender + .send(HandlerReply::Streaming { + status: status_code, + content_type: content_type_str, + headers: custom_headers, + body: rx, + }) + .is_err() + { + return Err(RuntimeError::new( + "Failed to start streaming response - client may have disconnected" + .to_string(), + *line, + *column, + )); + } + } + None => { + return Err(RuntimeError::new( + "Response already sent for this request".to_string(), + *line, + *column, + )); + } + } + + let handle_id = { + let n = self.next_response_stream_id.get(); + self.next_response_stream_id.set(n + 1); + format!("respstream{n}") + }; + self.server_response_streams + .borrow_mut() + .insert(handle_id.clone(), tx); + + let mut stream_map = HashMap::new(); + stream_map.insert("_server_stream".to_string(), Value::Text(handle_id.into())); + stream_map.insert("status".to_string(), Value::Number(status_code as f64)); + let value = Value::Object(Rc::new(RefCell::new(stream_map))); + match env.borrow_mut().define(variable_name, value) { + Ok(_) => Ok((Value::Null, ControlFlow::None)), + Err(msg) => Err(RuntimeError::new(msg, *line, *column)), + } + } + Statement::StreamWriteStatement { + value, + target, + is_line, + line, + column, + } => { + let handle_id = self + .resolve_server_stream_handle(target, &env, *line, *column) + .await?; + let val = self.evaluate_expression(value, Rc::clone(&env)).await?; + let mut bytes = match &val { + Value::Text(s) => s.as_bytes().to_vec(), + Value::Binary(b) => b.to_vec(), + Value::Number(_) | Value::Bool(_) => val.to_string().into_bytes(), + _ => { + return Err(RuntimeError::new( + format!( + "Can only write text or binary to a response stream, got {}", + val.type_name() + ), + *line, + *column, + )); + } + }; + if *is_line { + bytes.push(b'\n'); + } + + // Clone the sender out so the map borrow isn't held across the + // (possibly backpressured) send await. + let sender = self + .server_response_streams + .borrow() + .get(&handle_id) + .cloned(); + match sender { + Some(tx) => match tx.send(bytes).await { + Ok(()) => Ok((Value::Null, ControlFlow::None)), + Err(_) => { + // Receiver dropped => client disconnected. Drop the + // handle and surface a catchable error so the handler + // can stop (and close any upstream it is proxying). + self.server_response_streams.borrow_mut().remove(&handle_id); + Err(RuntimeError::new( + "Cannot write to response stream: the client has disconnected" + .to_string(), + *line, + *column, + )) + } + }, + None => Err(RuntimeError::new( + "Cannot write to a closed response stream".to_string(), + *line, + *column, + )), + } + } + Statement::FlushStreamStatement { + target, + line, + column, + } => { + let handle_id = self + .resolve_server_stream_handle(target, &env, *line, *column) + .await?; + let exists = self + .server_response_streams + .borrow() + .contains_key(&handle_id); + if !exists { + return Err(RuntimeError::new( + "Cannot flush a closed response stream".to_string(), + *line, + *column, + )); + } + // Advisory: chunks are already handed to the transport as they + // are written; yield so the transport task is scheduled to push + // them to the socket. + tokio::task::yield_now().await; + Ok((Value::Null, ControlFlow::None)) + } // Graceful shutdown and signal handling statements Statement::RegisterSignalHandlerStatement { signal_type, diff --git a/src/parser/ast.rs b/src/parser/ast.rs index 1fc09cc1..2b165418 100644 --- a/src/parser/ast.rs +++ b/src/parser/ast.rs @@ -580,6 +580,38 @@ pub enum Statement { line: usize, column: usize, }, + /// `start streaming response to [with status ] [and content type + /// ] [and headers ] as ` — begin a streamed server response. + /// Sends the status/headers immediately and binds a server response-stream + /// handle; the body is written incrementally with `write line|chunk` and + /// ended with `close`. + StartStreamingResponseStatement { + request: Expression, + status: Option, + content_type: Option, + headers: Option, + variable_name: String, + line: usize, + column: usize, + }, + /// `write line to ` / `write chunk to ` — append + /// a framed line (a trailing newline is added) or a raw chunk (text or + /// binary, verbatim) to a server response stream. + StreamWriteStatement { + value: Expression, + target: Expression, + /// true for `write line` (newline appended), false for `write chunk`. + is_line: bool, + line: usize, + column: usize, + }, + /// `flush ` — advisory flush of a server response stream: hand any + /// queued bytes to the transport. + FlushStreamStatement { + target: Expression, + line: usize, + column: usize, + }, // WebSocket statements. WebSockets mirror the HTTP server's design: warp // runs the socket in background tasks and the interpreter reacts to events // through registered handler blocks (dispatched while the program is inside diff --git a/src/parser/mod.rs b/src/parser/mod.rs index a25d2213..e20b66ae 100644 --- a/src/parser/mod.rs +++ b/src/parser/mod.rs @@ -610,6 +610,14 @@ impl<'a> StmtParser<'a> for Parser<'a> { // `send websocket message to ` and // `broadcast websocket message to `. The command // words (and a bare identifier message) lex as one merged token. + // `start streaming response to ... as `. `start` is a + // keyword; `streaming` is a contextual identifier; `response` is + // a keyword. + Token::KeywordStart => self.parse_start_streaming_response(), + // `flush ` — a bare-identifier target merges into the token. + Token::Identifier(id) if id == "flush" || id.starts_with("flush ") => { + self.parse_flush_stream() + } Token::Identifier(id) if id.starts_with("send websocket message") => { self.parse_send_websocket_message() } diff --git a/src/parser/stmt/io.rs b/src/parser/stmt/io.rs index a6046d95..070c7f58 100644 --- a/src/parser/stmt/io.rs +++ b/src/parser/stmt/io.rs @@ -793,6 +793,70 @@ impl<'a> IoParser<'a> for Parser<'a> { fn parse_write_to_statement(&mut self) -> Result { let token_pos = self.bump_sync().unwrap(); // Consume "write" + // `write line to ` / `write chunk to ` — + // append to a server response stream. `line`/`chunk` are contextual + // identifiers; the lexer merges a following bare-identifier value into + // the same token (`line payload` -> Identifier("line payload")), so + // split the value off the marker, mirroring the websocket-message form. + if let Some(next_token) = self.cursor.peek() + && let Token::Identifier(id) = &next_token.token + && (id == "line" + || id == "chunk" + || id.starts_with("line ") + || id.starts_with("chunk ")) + { + let id = id.clone(); + let (marker_line, marker_column) = (next_token.line, next_token.column); + let is_line = id.starts_with("line"); + let marker = if is_line { "line" } else { "chunk" }; + let rest = id + .strip_prefix(marker) + .map(str::trim_start) + .unwrap_or("") + .to_string(); + self.bump_sync(); // Consume the (possibly merged) marker + + let value = if rest.is_empty() { + // Value begins with a non-identifier (string/number), so the + // whole expression — including `with` concatenation — parses + // cleanly from here. + self.parse_expression()? + } else { + let left = Expression::Variable(rest, marker_line, marker_column); + match self.cursor.peek().map(|t| &t.token) { + // ` of `, e.g. `write line body of msg to out`. + Some(Token::KeywordOf) => { + self.bump_sync(); // Consume "of" + let object = self.parse_primary_expression()?; + Expression::FunctionCall { + function: Box::new(left), + arguments: vec![crate::parser::ast::Argument { + name: None, + value: object, + }], + line: marker_line, + column: marker_column, + } + } + // A bare variable value: the next token starts `to ...`. + _ => left, + } + }; + + self.expect_token( + Token::KeywordTo, + "Expected 'to ' after the value in a 'write line'/'write chunk' statement", + )?; + let target = self.parse_primary_expression()?; + return Ok(Statement::StreamWriteStatement { + value, + target, + is_line, + line: token_pos.line, + column: token_pos.column, + }); + } + // Check if next token is "binary" for "write binary X into Y" syntax if let Some(next_token) = self.cursor.peek() && matches!(&next_token.token, Token::KeywordBinary) diff --git a/src/parser/stmt/web.rs b/src/parser/stmt/web.rs index 579645b3..b16f1298 100644 --- a/src/parser/stmt/web.rs +++ b/src/parser/stmt/web.rs @@ -10,6 +10,8 @@ pub(crate) trait WebParser<'a>: ExprParser<'a> + PrimaryExprParser<'a> { fn parse_listen_statement(&mut self) -> Result; fn parse_tls_path_value(&mut self, marker: &str) -> Result; fn parse_respond_statement(&mut self) -> Result; + fn parse_start_streaming_response(&mut self) -> Result; + fn parse_flush_stream(&mut self) -> Result; fn parse_register_signal_handler_statement(&mut self) -> Result; fn parse_stop_accepting_connections_statement(&mut self) -> Result; fn parse_close_server_statement(&mut self) -> Result; @@ -362,6 +364,172 @@ impl<'a> WebParser<'a> for Parser<'a> { }) } + fn parse_start_streaming_response(&mut self) -> Result { + // Consume the `start` keyword, then the `streaming` contextual identifier. + let start_token = self.bump_sync().unwrap(); + let (line, column) = (start_token.line, start_token.column); + + match self.cursor.peek() { + Some(t) => match &t.token { + Token::Identifier(id) if id == "streaming" => { + self.bump_sync(); // Consume "streaming" + } + _ => { + return Err(ParseError::from_token( + "Expected 'streaming' after 'start'".to_string(), + t, + )); + } + }, + None => { + return Err(ParseError::from_token( + "Expected 'streaming response' after 'start'".to_string(), + start_token, + )); + } + } + + self.expect_token( + Token::KeywordResponse, + "Expected 'response' after 'start streaming'", + )?; + self.expect_token( + Token::KeywordTo, + "Expected 'to ' after 'start streaming response'", + )?; + let request = self.parse_primary_expression()?; + + let mut status = None; + let mut content_type = None; + let mut headers = None; + + // Optional clauses joined by `with`/`and`, in any order: `status `, + // `content type `, `headers `. Mirrors the `respond` clause loop; + // an `and`/`with` that does not introduce a known clause (e.g. before + // `as`) ends the loop. + loop { + let connective = matches!( + self.cursor.peek(), + Some(t) if t.token == Token::KeywordWith || t.token == Token::KeywordAnd + ); + if !connective { + break; + } + let Some(next_token) = self.cursor.peek_next() else { + break; + }; + + match &next_token.token { + Token::KeywordStatus => { + self.bump_sync(); // with/and + self.bump_sync(); // status + status = Some(self.parse_primary_expression()?); + } + // `content type ` — `content` keyword then optional `type`. + Token::KeywordContent => { + self.bump_sync(); // with/and + self.bump_sync(); // content + if let Some(t) = self.cursor.peek() + && let Token::Identifier(id) = &t.token + && id == "type" + { + self.bump_sync(); // type + } + content_type = Some(self.parse_primary_expression()?); + } + // Merged `content_type ` / `content type ` form. + Token::Identifier(id) + if id == "content_type" + || id.starts_with("content_type ") + || id.starts_with("content type") => + { + let id = id.clone(); + let (id_line, id_column) = (next_token.line, next_token.column); + self.bump_sync(); // with/and + self.bump_sync(); // merged marker + let rest = id + .strip_prefix("content_type") + .map(str::trim_start) + .unwrap_or_else(|| { + id.strip_prefix("content type") + .map(str::trim_start) + .unwrap_or("") + }); + if rest.is_empty() { + content_type = Some(self.parse_primary_expression()?); + } else { + content_type = + Some(Expression::Variable(rest.to_string(), id_line, id_column)); + } + } + // `headers ` (bare or merged `headers `). + Token::Identifier(id) if id == "headers" || id.starts_with("headers ") => { + let id = id.clone(); + let (id_line, id_column) = (next_token.line, next_token.column); + self.bump_sync(); // with/and + self.bump_sync(); // merged marker + let rest = id + .strip_prefix("headers") + .map(str::trim_start) + .unwrap_or(""); + if rest.is_empty() { + headers = Some(self.parse_primary_expression()?); + } else { + headers = Some(Expression::Variable(rest.to_string(), id_line, id_column)); + } + } + _ => break, + } + } + + self.expect_token( + Token::KeywordAs, + "Expected 'as ' after 'start streaming response ...'", + )?; + let variable_name = self.parse_variable_name_simple()?; + + Ok(Statement::StartStreamingResponseStatement { + request, + status, + content_type, + headers, + variable_name, + line, + column, + }) + } + + fn parse_flush_stream(&mut self) -> Result { + // `flush ` — the lexer merges a bare-identifier target into the + // command token (`flush out` -> Identifier("flush out")). + let token = self.bump_sync().unwrap(); + let (line, column) = (token.line, token.column); + let phrase = match &token.token { + Token::Identifier(id) => id.clone(), + _ => { + return Err(ParseError::from_token( + "Expected 'flush '".to_string(), + token, + )); + } + }; + let rest = phrase + .strip_prefix("flush") + .map(str::trim_start) + .unwrap_or(""); + let target = if rest.is_empty() { + self.parse_primary_expression()? + } else { + Expression::Variable(rest.to_string(), line, column) + }; + + Ok(Statement::FlushStreamStatement { + target, + line, + column, + }) + } + fn parse_register_signal_handler_statement(&mut self) -> Result { let register_token = self.bump_sync().unwrap(); // Consume "register" diff --git a/src/transpiler/javascript.rs b/src/transpiler/javascript.rs index 220b31a1..8e5e1712 100644 --- a/src/transpiler/javascript.rs +++ b/src/transpiler/javascript.rs @@ -622,7 +622,10 @@ impl JavaScriptTranspiler { Statement::HttpStreamStatement { line, column, .. } | Statement::WaitForNextChunkStatement { line, column, .. } - | Statement::WaitForNextLineStatement { line, column, .. } => { + | Statement::WaitForNextLineStatement { line, column, .. } + | Statement::StartStreamingResponseStatement { line, column, .. } + | Statement::StreamWriteStatement { line, column, .. } + | Statement::FlushStreamStatement { line, column, .. } => { // Streaming HTTP relies on the interpreter's parked-stream // handles; emitting broken JS would be worse than a clear error. Err(TranspileError { @@ -2070,6 +2073,8 @@ impl JavaScriptTranspiler { | Statement::HttpStreamStatement { .. } | Statement::WaitForNextChunkStatement { .. } | Statement::WaitForNextLineStatement { .. } + | Statement::StartStreamingResponseStatement { .. } + | Statement::StreamWriteStatement { .. } | Statement::WaitForProcessStatement { .. } | Statement::WaitForRequestStatement { .. } => true, diff --git a/src/typechecker/mod.rs b/src/typechecker/mod.rs index 2b07a052..04506d59 100644 --- a/src/typechecker/mod.rs +++ b/src/typechecker/mod.rs @@ -969,6 +969,38 @@ impl TypeChecker { symbol.symbol_type = Some(Type::Any); } } + Statement::StartStreamingResponseStatement { + request, + status, + content_type, + headers, + variable_name, + .. + } => { + let _ = self.infer_expression_type(request); + if let Some(status) = status { + let _ = self.infer_expression_type(status); + } + if let Some(content_type) = content_type { + let _ = self.infer_expression_type(content_type); + } + if let Some(headers) = headers { + let _ = self.infer_expression_type(headers); + } + if !variable_name.is_empty() + && let Some(symbol) = self.analyzer.get_symbol_mut(variable_name) + { + symbol.symbol_type = + Some(Type::Map(Box::new(Type::Text), Box::new(Type::Unknown))); + } + } + Statement::StreamWriteStatement { value, target, .. } => { + let _ = self.infer_expression_type(value); + let _ = self.infer_expression_type(target); + } + Statement::FlushStreamStatement { target, .. } => { + let _ = self.infer_expression_type(target); + } Statement::VariableDeclaration { name, value, diff --git a/tests/http_server_streaming_test.rs b/tests/http_server_streaming_test.rs new file mode 100644 index 00000000..c4418549 --- /dev/null +++ b/tests/http_server_streaming_test.rs @@ -0,0 +1,166 @@ +// Tests for streamed server responses (item 3): +// start streaming response to [with status ] [and content type ] as +// write line to // frames a line (newline appended) +// write chunk to // raw bytes/text, verbatim +// flush +// close // ends the response body +// +// A WFL web server streams a response; a reqwest client reads it back. + +use std::time::Duration; +use wfl::Interpreter; +use wfl::lexer::lex_wfl_with_positions; +use wfl::parser::Parser; +use wfl::parser::ast::Statement; + +// ----------------------------- parser tests ------------------------------ + +fn parse_single_statement(code: &str) -> Statement { + let tokens = lex_wfl_with_positions(code); + let mut parser = Parser::new(&tokens); + let program = parser + .parse() + .unwrap_or_else(|e| panic!("Parse error for {code:?}: {e:?}")); + assert_eq!( + program.statements.len(), + 1, + "Expected exactly one statement for {code:?}" + ); + program.statements.into_iter().next().unwrap() +} + +#[test] +fn test_start_streaming_response_parses() { + let stmt = parse_single_statement( + r#"start streaming response to req with status 200 and content type "application/x-ndjson" as out"#, + ); + match stmt { + Statement::StartStreamingResponseStatement { + status, + content_type, + variable_name, + .. + } => { + assert!(status.is_some()); + assert!(content_type.is_some()); + assert_eq!(variable_name, "out"); + } + other => panic!("Expected StartStreamingResponseStatement, got {other:?}"), + } +} + +#[test] +fn test_write_line_parses() { + let stmt = parse_single_statement(r#"write line payload to out"#); + match stmt { + Statement::StreamWriteStatement { is_line, .. } => assert!(is_line), + other => panic!("Expected StreamWriteStatement, got {other:?}"), + } +} + +#[test] +fn test_write_chunk_parses() { + let stmt = parse_single_statement(r#"write chunk payload to out"#); + match stmt { + Statement::StreamWriteStatement { is_line, .. } => assert!(!is_line), + other => panic!("Expected StreamWriteStatement, got {other:?}"), + } +} + +#[test] +fn test_flush_parses() { + let stmt = parse_single_statement("flush out"); + match stmt { + Statement::FlushStreamStatement { .. } => {} + other => panic!("Expected FlushStreamStatement, got {other:?}"), + } +} + +// ----------------------------- runtime tests ------------------------------ + +fn start_server_thread(code: String) -> std::thread::JoinHandle<()> { + std::thread::spawn(move || { + let rt = tokio::runtime::Runtime::new().expect("Failed to create runtime"); + rt.block_on(async { + let tokens = lex_wfl_with_positions(&code); + let mut parser = Parser::new(&tokens); + let ast = parser.parse().expect("Failed to parse WFL code"); + let mut interpreter = Interpreter::new(); + let _ = interpreter.interpret(&ast).await; + }); + }) +} + +#[tokio::test] +async fn test_streamed_response_lines_and_headers() { + let port = 8231; + let server_code = format!( + r#" + listen on port {port} as s + wait for request comes in on s as req with timeout 10000 + start streaming response to req with status 200 and content type "application/x-ndjson" as out + write line "alpha" to out + write line "beta" to out + flush out + write line "gamma" to out + close out + close server s + "# + ); + + let server_handle = start_server_thread(server_code); + tokio::time::sleep(Duration::from_millis(300)).await; + + let client = reqwest::Client::new(); + let response = client + .get(format!("http://127.0.0.1:{port}/stream")) + .send() + .await + .expect("Failed to send request"); + + assert_eq!(response.status().as_u16(), 200); + let ct = response + .headers() + .get("content-type") + .and_then(|v| v.to_str().ok()) + .unwrap_or(""); + assert_eq!(ct, "application/x-ndjson"); + + let body = response.text().await.expect("Failed to read body"); + assert_eq!(body, "alpha\nbeta\ngamma\n"); + + let _ = server_handle.join(); +} + +#[tokio::test] +async fn test_streamed_response_write_chunk_verbatim() { + let port = 8232; + let server_code = format!( + r#" + listen on port {port} as s + wait for request comes in on s as req with timeout 10000 + start streaming response to req with status 201 and content type "text/plain" as out + write chunk "one" to out + write chunk "two" to out + close out + close server s + "# + ); + + let server_handle = start_server_thread(server_code); + tokio::time::sleep(Duration::from_millis(300)).await; + + let client = reqwest::Client::new(); + let response = client + .get(format!("http://127.0.0.1:{port}/raw")) + .send() + .await + .expect("Failed to send request"); + + assert_eq!(response.status().as_u16(), 201); + let body = response.text().await.expect("Failed to read body"); + // write chunk does not append newlines. + assert_eq!(body, "onetwo"); + + let _ = server_handle.join(); +} From ed61ba5ac70d50a8663590952d26d9c1aeff1d88 Mon Sep 17 00:00:00 2001 From: Claude Date: Wed, 22 Jul 2026 15:59:40 +0000 Subject: [PATCH 004/132] feat: concurrent request handlers (main loop concurrently:) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Add opt-in cooperative concurrent request handling so a slow handler (e.g. one proxying a slow upstream stream) no longer blocks other requests: main loop concurrently: wait for request comes in on server as req ... slow handler ... respond to req with "..." end loop Plain `main loop` stays strictly serial and byte-compatible; `concurrently` is the only opt-in and nothing changes silently. This is Phase 1 of the maintainer-locked concurrency-phase-plan.md (locked marker, no Rc->Arc/Send rewrite of the interpreter core) and lands at that plan's STOP/review gate. Mechanism: - AST: MainLoop gains `concurrent: bool`. `concurrently` is a contextual identifier (only special right after `main loop`), so it stays usable as a variable name elsewhere. - execute_concurrent_main_loop keeps up to CONCURRENT_HANDLER_LIMIT (256) body iterations in flight via a FuturesUnordered of !Send, non-'static, &self-borrowing handler futures — cooperative concurrency on the single interpreter thread (no spawn_local, no Send/Arc across the core). Each iteration runs in a fresh isolated child scope. Cap >= 1 keeps the set non-empty, avoiding the Ready(None) busy-spin trap. - Containment: each handler future is AssertUnwindSafe(...).catch_unwind(); a panicking or erroring handler is contained and its request is answered 500 by the existing ResponseCompletion drop guard while siblings keep running. - 503/504/500 are inherited from the existing transport layer (bounded queue, response deadline, drop guard); `wait for request` releases the receiver lock before the handler runs, so iterations hand off requests one at a time then handle them concurrently. No RefCell held across await (crate-wide await_holding_refcell_ref deny enforces this). - Docs: web-servers.md "Concurrent request handling" (concurrent != parallel, yield cliff, 503/504/500) + validated example; concurrency-phase-plan.md tracker updated to Phase-1-done/awaiting-review; Dev Diary. - Tests: tests/concurrent_main_loop_test.rs — concurrently parses vs serial; slow handler does not block fast sibling; serial DOES block (no silent upgrade); handler-error containment keeps the server serving. Also removes two .ast.txt parse-dump artifacts accidentally added in the two prior streaming commits. Co-Authored-By: Claude Opus 4.8 Claude-Session: https://claude.ai/code/session_011LxXAm5E4Vy39kRLr8kRnX --- .../2026-07-22-concurrent-request-handlers.md | 94 +++++++++ Docs/04-advanced-features/web-servers.md | 64 +++++- Docs/development/concurrency-phase-plan.md | 23 ++- Docs/development/response-streaming-design.md | 4 +- .../docs_examples/_meta/manifest.json | 17 ++ .../streaming_response.wfl.ast.txt | 190 ------------------ .../web_servers/concurrent_main_loop.wfl | 20 ++ .../streaming_response.wfl.ast.txt | 154 -------------- src/interpreter/mod.rs | 78 +++++++ src/parser/ast.rs | 5 + src/parser/stmt/control_flow.rs | 14 ++ tests/concurrent_main_loop_test.rs | 183 +++++++++++++++++ 12 files changed, 491 insertions(+), 355 deletions(-) create mode 100644 Dev diary/2026-07-22-concurrent-request-handlers.md delete mode 100644 TestPrograms/docs_examples/interoperability/streaming_response.wfl.ast.txt create mode 100644 TestPrograms/docs_examples/web_servers/concurrent_main_loop.wfl delete mode 100644 TestPrograms/docs_examples/web_servers/streaming_response.wfl.ast.txt create mode 100644 tests/concurrent_main_loop_test.rs diff --git a/Dev diary/2026-07-22-concurrent-request-handlers.md b/Dev diary/2026-07-22-concurrent-request-handlers.md new file mode 100644 index 00000000..64a1ae49 --- /dev/null +++ b/Dev diary/2026-07-22-concurrent-request-handlers.md @@ -0,0 +1,94 @@ +# Dev Diary — 2026-07-22 — Concurrent request handlers (`main loop concurrently:`) + +## Context + +Final piece of the five-capability streaming request (item 4). With outbound and +server streaming shipped, a handler can proxy a slow upstream to the browser — +but on the **serial** `main loop`, that slow handler blocks every other request +(login, history, health, other chats). This adds opt-in concurrent handling so a +slow stream no longer stalls its siblings. + +This maps onto **Phase 1** of `Docs/development/concurrency-phase-plan.md` — a +maintainer-locked, gated plan. I followed its hard rules (locked marker `main +loop concurrently:`, no `Rc→Arc`/`Send` rewrite of the interpreter core, plain +`main loop` stays serial, TDD, `panic = "unwind"` gate already in CI). Phase 1 +landed in one change rather than the staged 1a→1b→1c; **this is the plan's +maintainer STOP/review point.** + +## What shipped + +```wfl +listen on port 8080 as server +main loop concurrently: + wait for request comes in on server as req + // a slow handler here (e.g. streaming a slow upstream) no longer blocks siblings + respond to req with "Hello!" +end loop +``` + +Plain `main loop` is unchanged (strictly serial, byte-compatible). Adding +`concurrently` is the only way to opt in — no silent semantics swap. + +## Design & mechanism + +- **AST:** `MainLoop` gained a `concurrent: bool` (default false). `concurrently` + is a contextual identifier parsed only right after `main loop`, so it stays + usable as a variable name elsewhere. +- **Execution:** `execute_concurrent_main_loop` keeps up to + `CONCURRENT_HANDLER_LIMIT` (256) iterations of the body in flight via a + `FuturesUnordered` of `!Send`, non-`'static`, `&self`-borrowing handler + futures — cooperative concurrency on the one interpreter thread, exactly as the + plan's 1a spike prescribes (no `spawn_local`, no `Send`/`Arc` across the core). + Each iteration runs in a fresh `Environment::new_child_env` (isolation by + default). The set is refilled to the cap, so with cap ≥ 1 it is never empty — + avoiding the `Ready(None)` busy-spin trap. +- **Containment:** each handler future is wrapped in + `AssertUnwindSafe(...).catch_unwind()`. A panicking handler is caught, its + request is answered 500 by the existing `ResponseCompletion` drop guard, and + siblings keep running. A handler that returns a `RuntimeError` is likewise + logged and contained. +- **Ops defaults reused:** 503 (bounded transport queue), 504 (per-request + response deadline), and 500 (drop guard) already existed at the transport + layer, so the concurrent loop inherits them — it adds handler-level + concurrency, not a parallel ops stack. +- **Why `wait for request` allows this:** it holds the server's receiver mutex + only to dequeue one request, then releases it before the handler body runs — so + concurrent iterations hand off requests one at a time and then handle them + concurrently. No `Rc`/`RefCell` is held across an `.await` (the crate-wide + `#![deny(clippy::await_holding_refcell_ref)]` backstop enforces this). + +## Concurrent, not parallel + +Cooperative concurrency on a single thread: handlers interleave at their await +points (`wait for`, outbound HTTP, stream read/write, `respond`). A tight +CPU-bound handler with no await still holds the thread — documented as the yield +cliff. This is the honest model, not multicore parallelism (Phase 3, deferred). + +## Files + +- `src/parser/ast.rs` (`MainLoop.concurrent`), `src/parser/stmt/control_flow.rs` + (parse `concurrently`), `src/interpreter/mod.rs` + (`execute_concurrent_main_loop`, `CONCURRENT_HANDLER_LIMIT`, MainLoop branch). +- Docs: `Docs/04-advanced-features/web-servers.md` ("Concurrent request + handling") + validated example + `TestPrograms/docs_examples/web_servers/concurrent_main_loop.wfl`; + `concurrency-phase-plan.md` tracker updated. + +## Tests + +`tests/concurrent_main_loop_test.rs`: +- `main loop concurrently:` parses concurrent; plain `main loop:` stays serial. +- Concurrent: a 500 ms handler does not block a fast sibling (fast < 300 ms). +- Serial: the same slow handler *does* block the next request (fast > 300 ms) — + proving no silent upgrade. +- Handler-error containment: an erroring handler doesn't kill the server. + +`fmt`, `clippy -D warnings`, the 618 lib tests, and the existing web-server / +streaming suites are all green. + +## Out of scope (Phase 2+, per the plan) + +- Structured nursery / `wait for all|any of` / `change shared`. +- Multicore (Phase 3; deferred, needs profiling). +- Request-ID *structured* logging scheme and a written per-site eval-core audit + (mechanically enforced by the clippy backstop for now). diff --git a/Docs/04-advanced-features/web-servers.md b/Docs/04-advanced-features/web-servers.md index 6c577d24..c291e5b9 100644 --- a/Docs/04-advanced-features/web-servers.md +++ b/Docs/04-advanced-features/web-servers.md @@ -68,6 +68,59 @@ wait for request [that] comes in on as This blocks until a request arrives, then stores the request in the variable. Both "comes in" and "that comes in" are supported. +### Concurrent request handling + +A typical server wraps `wait for request` … `respond to` in a `main loop`: + +```wfl +listen on port 8080 as web_server +main loop: + wait for request comes in on web_server as req + respond to req with "Hello!" +end loop +``` + +A plain `main loop` is **serial**: it fully handles one request — including any +`wait for`, outbound call, or stream — before accepting the next. That is simple +and predictable, but one slow handler (say, proxying a slow model stream) makes +every other request wait behind it. + +Write `main loop concurrently:` to handle requests **concurrently** instead: + +```wfl +listen on port 8080 as web_server +main loop concurrently: + wait for request comes in on web_server as req + // ... a slow handler here no longer blocks other requests ... + respond to req with "Hello!" +end loop +``` + +Each iteration runs in its own isolated scope, so concurrent requests never +clobber each other's variables. A login, a health check, or another chat is +served while a slow stream is still running. + +**Important — concurrent, not parallel:** this is *cooperative* concurrency on a +single thread. Handlers interleave at their `await` points (`wait for`, outbound +HTTP, stream reads/writes, `respond`). A handler doing tight CPU-bound work with +no such pause still holds the thread until it yields — concurrency helps I/O- +bound work (the common web case), not CPU-bound loops. + +**What you get with `concurrently`:** + +- A slow handler does not block its siblings. +- Each request handler is isolated (its own scope). +- A handler that errors or panics is contained: that request fails on its own + (its client gets a 500/timeout) and the server keeps serving everyone else. +- In-flight work is bounded; the transport still sheds excess load with 503 and + times out a stalled handler with 504, exactly as for the serial loop. + +Plain `main loop` keeps its exact serial behavior — adding `concurrently` is the +only way to opt in; nothing changes silently. + +> `concurrently` is only special right after `main loop`; it is not a reserved +> word, so existing programs that use `concurrently` as a name keep working. + ### Responding to Requests Use `respond to` to send HTTP responses: @@ -470,12 +523,11 @@ close upstream close out ``` -> **Concurrency note:** today a request handler runs to completion before the -> next request is served (the loop is cooperative and single-threaded), so a -> long-running stream occupies the handler until it finishes. Opt-in concurrent -> handling (`main loop concurrently:`) is a planned, separate capability — see -> [`concurrency-phase-plan.md`](../development/concurrency-phase-plan.md). Until -> then, prefer bounded streams and per-request timeouts. +> **Concurrency note:** a plain `main loop` handles one request at a time, so a +> long-running stream occupies the handler until it finishes. To let a slow +> stream run without blocking other requests (login, history, health checks, +> other chats), opt into [concurrent handling](#concurrent-request-handling) +> with `main loop concurrently:`. ## The QUERY Method (RFC 10008) diff --git a/Docs/development/concurrency-phase-plan.md b/Docs/development/concurrency-phase-plan.md index ee0d8c02..4449902c 100644 --- a/Docs/development/concurrency-phase-plan.md +++ b/Docs/development/concurrency-phase-plan.md @@ -46,9 +46,26 @@ HARD RULES: | 0 | 0a | Docs honesty + `panic=unwind` CI | ✅ Done | | 0 | 0b | `spawn_blocking` for blocking crypto | ✅ Done | | 0 | 0c | Bound accept/queue (OOM shed) | ✅ Done | -| 1 | 1a | Runtime spike (bridge, no surface) | ⬜ Not started | -| 1 | 1b | `main loop concurrently:` surface + ops defaults | ⬜ Not started | -| 1 | 1c | Honesty docs for real concurrent model | ⬜ Not started | +| 1 | 1a | Runtime spike (bridge, no surface) | ✅ Done (folded into 1b) | +| 1 | 1b | `main loop concurrently:` surface + ops defaults | ✅ Done — awaiting maintainer review | +| 1 | 1c | Honesty docs for real concurrent model | ✅ Done | + +> **Phase 1 landed in one change** (`Dev diary/2026-07-22-concurrent-request-handlers.md`), +> not the staged 1a→1b→1c sequence. What is covered: `main loop concurrently:` +> surface (locked marker); plain `main loop` byte-compatible serial (tested); +> `FuturesUnordered` of `!Send`, `&self`-borrowing handler futures on the +> existing runtime; isolated-per-request scopes; `catch_unwind` panic +> containment (siblings survive, tested via handler-error containment); +> in-flight cap (`CONCURRENT_HANDLER_LIMIT`), with 503/504/500 provided by the +> existing transport layer (bounded queue → 503, response deadline → 504, +> `ResponseCompletion` drop → 500); the empty-set busy-spin trap is avoided +> (cap ≥ 1 keeps the set non-empty). **Lighter than the full 1b checklist:** +> request-ID *structured* logging is not yet added (handler errors/panics are +> logged, but not a per-request accept/complete/fail/shed/timeout ID scheme); +> the eval-core `RefCell`-across-await audit is enforced mechanically by the +> crate-wide `#![deny(clippy::await_holding_refcell_ref)]` backstop rather than a +> written per-site walkthrough. **This is the maintainer STOP/review point** — +> please review before Phase 2. | 2 | 2a | Structured nursery + join engine | ⬜ Not started | | 2 | 2b | `change shared` critical region | ⬜ Not started | | 3 | 3a | Multi-process workers (if profiling forces) | ⬜ Deferred | diff --git a/Docs/development/response-streaming-design.md b/Docs/development/response-streaming-design.md index 41b1b0b1..7eef944c 100644 --- a/Docs/development/response-streaming-design.md +++ b/Docs/development/response-streaming-design.md @@ -17,8 +17,8 @@ locked design for what remains. It complements — and defers to — | 1 | Outbound response streaming (`stream response as`) | ✅ Shipped | | 2 | Incremental reads (`wait for next chunk|line`) | ✅ Shipped | | 3 | Streamed server responses (start / write / flush / close) | ✅ Shipped | -| 4 | Concurrent request handlers | ⬜ Phase 1 of `concurrency-phase-plan.md` | -| 5 | Lifecycle (timeouts, backpressure, cancellation, catchable errors, close-on-exit) | ◐ Client + server body streaming shipped; per-handler isolation rides with 4 | +| 4 | Concurrent request handlers | ✅ Shipped (`main loop concurrently:`; Phase 1 of `concurrency-phase-plan.md`, awaiting maintainer review) | +| 5 | Lifecycle (timeouts, backpressure, cancellation, catchable errors, close-on-exit) | ✅ Client + server streaming + per-handler isolation/containment shipped | --- diff --git a/TestPrograms/docs_examples/_meta/manifest.json b/TestPrograms/docs_examples/_meta/manifest.json index f1843057..1749cd4b 100644 --- a/TestPrograms/docs_examples/_meta/manifest.json +++ b/TestPrograms/docs_examples/_meta/manifest.json @@ -435,5 +435,22 @@ "response" ], "description": "Streaming a server response with start streaming response / write line / flush / close." + }, + "docs_examples/web_servers/concurrent_main_loop.wfl": { + "doc_section": "Docs/04-advanced-features/web-servers.md#concurrent-request-handling", + "type": "snippet", + "validate_layers": [ + 1, + 2, + 3, + 4 + ], + "skip_execution": true, + "tags": [ + "web-server", + "concurrency", + "main-loop" + ], + "description": "Concurrent request handling with main loop concurrently." } } diff --git a/TestPrograms/docs_examples/interoperability/streaming_response.wfl.ast.txt b/TestPrograms/docs_examples/interoperability/streaming_response.wfl.ast.txt deleted file mode 100644 index 8fda9493..00000000 --- a/TestPrograms/docs_examples/interoperability/streaming_response.wfl.ast.txt +++ /dev/null @@ -1,190 +0,0 @@ -AST output for: TestPrograms/docs_examples/interoperability/streaming_response.wfl -============================================== - -Program with 6 statements: - -Statement #1: HttpStreamStatement { - url: Literal( - String( - "https://api.example.com/events", - ), - 10, - 13, - ), - method: None, - headers: None, - body: None, - variable_name: "upstream", - line: 10, - column: 1, -} - -Statement #2: DisplayStatement { - value: Concatenation { - left: Literal( - String( - "Status: ", - ), - 12, - 9, - ), - right: IndexAccess { - collection: Variable( - "upstream", - 12, - 25, - ), - index: Literal( - String( - "status", - ), - 12, - 34, - ), - line: 12, - column: 33, - }, - line: 12, - column: 20, - }, - line: 12, - column: 1, -} - -Statement #3: VariableDeclaration { - name: "content_type", - value: IndexAccess { - collection: IndexAccess { - collection: Variable( - "upstream", - 13, - 23, - ), - index: Literal( - String( - "headers", - ), - 13, - 32, - ), - line: 13, - column: 31, - }, - index: Literal( - String( - "content-type", - ), - 13, - 43, - ), - line: 13, - column: 42, - }, - is_constant: false, - line: 13, - column: 1, -} - -Statement #4: DisplayStatement { - value: Concatenation { - left: Literal( - String( - "Content type: ", - ), - 14, - 9, - ), - right: Variable( - "content_type", - 14, - 31, - ), - line: 14, - column: 26, - }, - line: 14, - column: 1, -} - -Statement #5: CountLoop { - start: Literal( - Integer( - 1, - ), - 16, - 12, - ), - end: Literal( - Integer( - 1000000, - ), - 16, - 17, - ), - step: None, - downward: false, - variable_name: None, - body: [ - WaitForNextLineStatement { - source: Variable( - "upstream", - 17, - 29, - ), - variable_name: "line", - line: 17, - column: 5, - }, - IfStatement { - condition: BinaryOperation { - left: Variable( - "line", - 18, - 14, - ), - operator: Equals, - right: Literal( - Nothing, - 18, - 22, - ), - line: 18, - column: 19, - }, - then_block: [ - BreakStatement { - line: 19, - column: 9, - }, - ], - else_block: Some( - [ - DisplayStatement { - value: Variable( - "line", - 21, - 17, - ), - line: 21, - column: 9, - }, - ], - ), - line: 18, - column: 5, - }, - ], - line: 23, - column: 10, -} - -Statement #6: CloseFileStatement { - file: Variable( - "upstream", - 25, - 7, - ), - line: 25, - column: 1, -} - diff --git a/TestPrograms/docs_examples/web_servers/concurrent_main_loop.wfl b/TestPrograms/docs_examples/web_servers/concurrent_main_loop.wfl new file mode 100644 index 00000000..03f79273 --- /dev/null +++ b/TestPrograms/docs_examples/web_servers/concurrent_main_loop.wfl @@ -0,0 +1,20 @@ +// Concurrent request handling with `main loop concurrently:`. +// +// Iterations run in their own isolated scope, so a slow handler does not +// block other requests. Plain `main loop` stays serial; adding `concurrently` +// is the only way to opt in. +// +// Validated for syntax/analysis/lint only (layers 1-4): running it needs live +// clients, so execution is skipped. + +listen on port 8080 as site + +main loop concurrently: + wait for request comes in on site as req + store p as req["path"] + check if p is equal to "/health": + respond to req with "ok" + otherwise: + respond to req with "Hello!" + end check +end loop diff --git a/TestPrograms/docs_examples/web_servers/streaming_response.wfl.ast.txt b/TestPrograms/docs_examples/web_servers/streaming_response.wfl.ast.txt deleted file mode 100644 index 3bb11606..00000000 --- a/TestPrograms/docs_examples/web_servers/streaming_response.wfl.ast.txt +++ /dev/null @@ -1,154 +0,0 @@ -AST output for: TestPrograms/docs_examples/web_servers/streaming_response.wfl -============================================== - -Program with 9 statements: - -Statement #1: ListenStatement { - port: Literal( - Integer( - 8080, - ), - 10, - 16, - ), - server_name: "site", - tls: None, - redirect_to_port: None, - line: 10, - column: 1, -} - -Statement #2: WaitForRequestStatement { - server: Variable( - "site", - 12, - 30, - ), - request_name: "req", - timeout: Some( - Literal( - Integer( - 10000, - ), - 12, - 55, - ), - ), - line: 12, - column: 1, -} - -Statement #3: StartStreamingResponseStatement { - request: Variable( - "req", - 14, - 29, - ), - status: Some( - Literal( - Integer( - 200, - ), - 14, - 45, - ), - ), - content_type: Some( - Literal( - String( - "application/x-ndjson", - ), - 14, - 66, - ), - ), - headers: None, - variable_name: "out", - line: 14, - column: 1, -} - -Statement #4: StreamWriteStatement { - value: Literal( - String( - "first line", - ), - 16, - 12, - ), - target: Variable( - "out", - 16, - 28, - ), - is_line: true, - line: 16, - column: 1, -} - -Statement #5: StreamWriteStatement { - value: Literal( - String( - "second line", - ), - 17, - 12, - ), - target: Variable( - "out", - 17, - 29, - ), - is_line: true, - line: 17, - column: 1, -} - -Statement #6: FlushStreamStatement { - target: Variable( - "out", - 18, - 1, - ), - line: 18, - column: 1, -} - -Statement #7: StreamWriteStatement { - value: Literal( - String( - "no newline here", - ), - 19, - 13, - ), - target: Variable( - "out", - 19, - 34, - ), - is_line: false, - line: 19, - column: 1, -} - -Statement #8: CloseFileStatement { - file: Variable( - "out", - 21, - 7, - ), - line: 21, - column: 1, -} - -Statement #9: CloseServerStatement { - server: Variable( - "site", - 22, - 14, - ), - line: 22, - column: 1, -} - diff --git a/src/interpreter/mod.rs b/src/interpreter/mod.rs index 748fdfd3..780aae15 100644 --- a/src/interpreter/mod.rs +++ b/src/interpreter/mod.rs @@ -94,6 +94,13 @@ const COOP_YIELD_STRIDE: u64 = 1024; /// free slot) rather than letting queued chunks grow without bound. const RESPONSE_STREAM_BUFFER: usize = 64; +/// Maximum number of `main loop concurrently:` iterations in flight at once. The +/// transport already bounds the request *queue*; this bounds concurrent *handler* +/// execution so a burst cannot spawn unbounded cooperative tasks. Requests +/// beyond this run as handlers free up (and the transport sheds 503 if its queue +/// also fills). +const CONCURRENT_HANDLER_LIMIT: usize = 256; + // Web server data structures #[derive(Debug)] pub struct WflHttpRequest { @@ -3721,6 +3728,68 @@ impl Interpreter { } } + /// Execute a `main loop concurrently:` body. Keeps up to + /// `CONCURRENT_HANDLER_LIMIT` iterations of `body` in flight at once, each in + /// its own isolated child scope, driven cooperatively on this single thread + /// (no threads, no `Send`/`Arc` across the interpreter core). A handler that + /// errors or panics is contained — its request is resolved with 500 by the + /// response-completion drop guard — and its siblings keep running. + async fn execute_concurrent_main_loop( + &self, + body: &[Statement], + env: &Rc>, + ) -> Result<(Value, ControlFlow), RuntimeError> { + use futures_util::FutureExt; + use futures_util::stream::{FuturesUnordered, StreamExt}; + + let cap = CONCURRENT_HANDLER_LIMIT.max(1); + let mut futs = FuturesUnordered::new(); + let mut last_value = Value::Null; + + loop { + self.check_time()?; + + // Refill to the concurrency cap. Each iteration gets a fresh isolated + // scope so concurrent requests never clobber each other's variables. + while futs.len() < cap { + let scope = Environment::new_child_env(env); + futs.push( + std::panic::AssertUnwindSafe(self.execute_block(body, scope)).catch_unwind(), + ); + } + + // With cap >= 1 the set is never empty, so `next()` never returns a + // `Ready(None)` that would busy-spin the loop. + match futs.next().await { + Some(Ok(Ok((value, flow)))) => { + last_value = value; + match flow { + ControlFlow::Break => break, + ControlFlow::Exit => return Ok((last_value, ControlFlow::Exit)), + ControlFlow::Return(val) => { + return Ok((val.clone(), ControlFlow::Return(val))); + } + ControlFlow::Continue | ControlFlow::None => {} + } + } + // A handler returned a runtime error: its request (if it took one) + // is answered 500 by the ResponseCompletion drop guard. Log and + // keep the server running instead of tearing it down. + Some(Ok(Err(err))) => { + log::warn!("concurrent main loop: handler error: {err}"); + } + // A handler panicked: catch_unwind contained it; the request is + // answered 500 by the drop guard. Siblings survive. + Some(Err(_panic)) => { + log::warn!("concurrent main loop: handler panicked; request answered 500"); + } + None => break, // unreachable while cap >= 1; end cleanly if reached + } + } + + Ok((last_value, ControlFlow::None)) + } + /// Preserve ordinary file I/O failures while classifying byte-ceiling /// breaches as catchable execution-budget resource errors. fn file_read_error(&self, error: FileReadError, line: usize, column: usize) -> RuntimeError { @@ -4925,6 +4994,7 @@ impl Interpreter { Statement::MainLoop { body, + concurrent, line: _line, column: _column, } => { @@ -4939,6 +5009,14 @@ impl Interpreter { // `execute file` sharing this budget inherits the exemption too. let _main_loop_guard = self.budget.enter_main_loop(); + // `main loop concurrently:` runs body iterations cooperatively + // concurrently, each in its own isolated scope, so a slow handler + // (e.g. one streaming a slow upstream) does not block its + // siblings. Plain `main loop` stays strictly serial below. + if *concurrent { + return self.execute_concurrent_main_loop(body, &env).await; + } + let mut _last_value = Value::Null; let mut loop_env_recycle = None; diff --git a/src/parser/ast.rs b/src/parser/ast.rs index 2b165418..f4725c38 100644 --- a/src/parser/ast.rs +++ b/src/parser/ast.rs @@ -190,6 +190,11 @@ pub enum Statement { }, MainLoop { body: Vec, + /// `main loop concurrently:` — run iterations of the body cooperatively + /// concurrently (each in its own isolated scope) instead of strictly + /// serially, so a slow request handler does not block its siblings. + /// Plain `main loop` keeps `concurrent = false` (byte-compatible serial). + concurrent: bool, line: usize, column: usize, }, diff --git a/src/parser/stmt/control_flow.rs b/src/parser/stmt/control_flow.rs index 615747cf..1d19f72c 100644 --- a/src/parser/stmt/control_flow.rs +++ b/src/parser/stmt/control_flow.rs @@ -610,6 +610,19 @@ impl<'a> ControlFlowParser<'a> for Parser<'a> { { let main_token = self.bump_sync().unwrap(); // Consume "main" self.expect_token(Token::KeywordLoop, "Expected 'loop' after 'main'")?; + + // Optional `concurrently` marker (a contextual identifier). Plain + // `main loop` stays serial and byte-compatible; `main loop + // concurrently:` opts into cooperative concurrent handling. + let mut concurrent = false; + if let Some(token) = self.cursor.peek() + && let Token::Identifier(id) = &token.token + && id == "concurrently" + { + self.bump_sync(); // Consume "concurrently" + concurrent = true; + } + self.expect_token(Token::Colon, "Expected ':' after 'main loop'")?; // Skip any Eol tokens after the colon @@ -633,6 +646,7 @@ impl<'a> ControlFlowParser<'a> for Parser<'a> { Ok(Statement::MainLoop { body, + concurrent, line: main_token.line, column: main_token.column, }) diff --git a/tests/concurrent_main_loop_test.rs b/tests/concurrent_main_loop_test.rs new file mode 100644 index 00000000..ed4dee10 --- /dev/null +++ b/tests/concurrent_main_loop_test.rs @@ -0,0 +1,183 @@ +// Tests for `main loop concurrently:` (concurrent request handlers). +// +// Key properties: +// - `main loop concurrently:` parses (concurrent = true); plain `main loop:` +// stays serial (concurrent = false) — no silent upgrade. +// - Concurrent: a slow handler does NOT block a fast sibling. +// - Serial: a slow handler DOES block the next request (unchanged behavior). + +use std::time::{Duration, Instant}; +use wfl::Interpreter; +use wfl::lexer::lex_wfl_with_positions; +use wfl::parser::Parser; +use wfl::parser::ast::Statement; + +fn parse_program(code: &str) -> Vec { + let tokens = lex_wfl_with_positions(code); + let mut parser = Parser::new(&tokens); + parser + .parse() + .unwrap_or_else(|e| panic!("Parse error: {e:?}")) + .statements +} + +#[test] +fn test_main_loop_concurrently_parses_as_concurrent() { + let stmts = parse_program("main loop concurrently:\n display \"x\"\nend loop"); + match &stmts[0] { + Statement::MainLoop { concurrent, .. } => assert!(*concurrent), + other => panic!("Expected MainLoop, got {other:?}"), + } +} + +#[test] +fn test_plain_main_loop_stays_serial() { + let stmts = parse_program("main loop:\n display \"x\"\nend loop"); + match &stmts[0] { + Statement::MainLoop { concurrent, .. } => { + assert!(!*concurrent, "plain main loop must remain serial") + } + other => panic!("Expected MainLoop, got {other:?}"), + } +} + +fn start_server_thread(code: String) -> std::thread::JoinHandle<()> { + std::thread::spawn(move || { + let rt = tokio::runtime::Runtime::new().expect("runtime"); + rt.block_on(async { + let tokens = lex_wfl_with_positions(&code); + let mut parser = Parser::new(&tokens); + let ast = parser.parse().expect("parse"); + let mut interpreter = Interpreter::new(); + let _ = interpreter.interpret(&ast).await; + }); + }) +} + +fn server_code(port: u16, concurrently: bool) -> String { + let marker = if concurrently { " concurrently" } else { "" }; + format!( + r#" + listen on port {port} as srv + main loop{marker}: + wait for request comes in on srv as req with timeout 20000 + store p as req["path"] + check if p is equal to "/slow": + wait for 500 milliseconds + respond to req with "slow" + otherwise: + respond to req with "fast" + end check + end loop + "# + ) +} + +#[tokio::test] +async fn test_concurrent_slow_handler_does_not_block_fast() { + let port = 8241; + let _server = start_server_thread(server_code(port, true)); + tokio::time::sleep(Duration::from_millis(300)).await; + + let client = reqwest::Client::new(); + + // Kick off the slow request first and give it a moment to be dequeued. + let slow_url = format!("http://127.0.0.1:{port}/slow"); + let slow = + tokio::spawn(async move { reqwest::Client::new().get(&slow_url).send().await.unwrap() }); + tokio::time::sleep(Duration::from_millis(80)).await; + + // The fast request must complete promptly even though /slow is mid-handler. + let t0 = Instant::now(); + let fast = client + .get(format!("http://127.0.0.1:{port}/fast")) + .send() + .await + .expect("fast request failed"); + let fast_elapsed = t0.elapsed(); + let fast_body = fast.text().await.unwrap(); + + assert_eq!(fast_body, "fast"); + assert!( + fast_elapsed < Duration::from_millis(300), + "fast request was blocked behind the slow handler ({fast_elapsed:?})" + ); + + let slow_resp = slow.await.unwrap(); + assert_eq!(slow_resp.text().await.unwrap(), "slow"); +} + +#[tokio::test] +async fn test_concurrent_handler_error_does_not_kill_server() { + // A handler that errors mid-iteration (here: responding twice) must be + // contained — the concurrent loop keeps serving other requests. + let port = 8243; + let code = format!( + r#" + listen on port {port} as srv + main loop concurrently: + wait for request comes in on srv as req with timeout 20000 + store p as req["path"] + check if p is equal to "/boom": + respond to req with "boom-ok" + respond to req with "this second respond errors" + otherwise: + respond to req with "ok" + end check + end loop + "# + ); + let _server = start_server_thread(code); + tokio::time::sleep(Duration::from_millis(300)).await; + + let client = reqwest::Client::new(); + + // The erroring handler still delivered its first response. + let boom = client + .get(format!("http://127.0.0.1:{port}/boom")) + .send() + .await + .expect("boom request failed"); + assert_eq!(boom.text().await.unwrap(), "boom-ok"); + + // The server survived the caught error and keeps serving. + let ok = client + .get(format!("http://127.0.0.1:{port}/ok")) + .send() + .await + .expect("follow-up request failed"); + assert_eq!(ok.text().await.unwrap(), "ok"); +} + +#[tokio::test] +async fn test_serial_slow_handler_blocks_next() { + let port = 8242; + let _server = start_server_thread(server_code(port, false)); + tokio::time::sleep(Duration::from_millis(300)).await; + + let client = reqwest::Client::new(); + + let slow_url = format!("http://127.0.0.1:{port}/slow"); + let slow = + tokio::spawn(async move { reqwest::Client::new().get(&slow_url).send().await.unwrap() }); + tokio::time::sleep(Duration::from_millis(80)).await; + + // On the serial loop the fast request cannot be handled until the slow + // handler finishes, so it is delayed by roughly the slow handler's duration. + let t0 = Instant::now(); + let fast = client + .get(format!("http://127.0.0.1:{port}/fast")) + .send() + .await + .expect("fast request failed"); + let fast_elapsed = t0.elapsed(); + let fast_body = fast.text().await.unwrap(); + + assert_eq!(fast_body, "fast"); + assert!( + fast_elapsed > Duration::from_millis(300), + "serial main loop should have blocked the fast request behind the slow one ({fast_elapsed:?})" + ); + + let _ = slow.await; +} From e704b8752ed1c831ddfdf0913c663b998d99514d Mon Sep 17 00:00:00 2001 From: Claude Date: Wed, 22 Jul 2026 16:56:18 +0000 Subject: [PATCH 005/132] =?UTF-8?q?fix:=20address=20PR=20#641=20review=20?= =?UTF-8?q?=E2=80=94=20backward-compat,=20status=20validation,=20doc=20hon?= =?UTF-8?q?esty,=20test=20hygiene?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Automated review (CodeRabbit/Copilot/Devin/Codex) on PR #641 surfaced several issues; this fixes the clear ones: Backward-compatibility regressions (parser): - `write to ` with a variable literally named `line`/`chunk` was wrongly intercepted as a stream write and failed to parse. Only treat `line`/`chunk` as the stream marker when it is NOT immediately followed by `to`. Regression test added. - `wait for next ` (e.g. `wait for next milliseconds`) with a variable named `next` was wrongly intercepted as `wait for next chunk|line`. Only take the bare-`next` path when `chunk`/`line` actually follows. Regression test added. Correctness / consistency: - `start streaming response ... with status `: require a whole number in 100..=599 instead of silently wrapping via `as u16`. - `FlushStreamStatement` is now classified async in the transpiler's `stmt_is_async` (it yields), consistent with the other streaming statements. Docs honesty (CLAUDE.md "Docs Must Be Honest"): - Removed the false claim that a streamed server response is "closed automatically when the handler ends"; the handler must `close out` (documented with a finally: recommendation). Client-side wording likewise corrected: a stream is released on EOF/error/explicit close/program exit, not on handle drop. - Concurrent example + phase plan: state the cooperative limitation (a CPU-bound handler with no await still holds the thread), and separate tested runtime-error containment from by-construction catch_unwind panic containment. - Escaped the `chunk|line` pipe in a Markdown table (MD056). Test hygiene (concurrent_main_loop_test): - Moved off ports that collided with another test (8241 -> 8341..8343). - Each server now exposes `/shutdown` (close server + break); tests send it and join the server thread deterministically. - Assert the slow request actually completed instead of discarding its result. Also flags in concurrency-phase-plan.md a known gap for maintainer review: the concurrent loop isolates the environment but not interpreter-level run-state (count/call-stack), which needs a per-handler execution context. Co-Authored-By: Claude Opus 4.8 Claude-Session: https://claude.ai/code/session_011LxXAm5E4Vy39kRLr8kRnX --- Docs/04-advanced-features/interoperability.md | 10 ++- Docs/04-advanced-features/web-servers.md | 9 ++- Docs/development/concurrency-phase-plan.md | 41 ++++++++---- Docs/development/response-streaming-design.md | 2 +- .../web_servers/concurrent_main_loop.wfl | 8 ++- src/interpreter/mod.rs | 16 ++++- src/parser/stmt/io.rs | 17 ++++- src/parser/stmt/processes.rs | 41 ++++++------ src/transpiler/javascript.rs | 1 + tests/concurrent_main_loop_test.rs | 64 ++++++++++++++----- tests/http_server_streaming_test.rs | 13 ++++ tests/http_stream_test.rs | 11 ++++ 12 files changed, 174 insertions(+), 59 deletions(-) diff --git a/Docs/04-advanced-features/interoperability.md b/Docs/04-advanced-features/interoperability.md index 4dfda170..0a78d0e5 100644 --- a/Docs/04-advanced-features/interoperability.md +++ b/Docs/04-advanced-features/interoperability.md @@ -156,8 +156,14 @@ The same limits as buffered requests apply: the running total of body bytes is held under `web_server_max_response_size`, each read is bounded by the request's timeout, and cooperative cancellation interrupts a read waiting on the peer. A mid-stream network error surfaces as a catchable error from the `wait for next -...` statement, and every stream is closed when the handle is dropped on any -exit path. +...` statement. + +A stream is released — cancelling the in-flight upstream request — when it +reaches a clean end of stream, hits an error, or you `close` it explicitly, and +in any case when the program exits. It is **not** released merely because the +handle variable goes out of scope, so `close upstream` when you stop early (or +break out of the read loop before EOF) to free the upstream connection promptly +rather than holding it until the program ends. ### 4. **Web Standards** diff --git a/Docs/04-advanced-features/web-servers.md b/Docs/04-advanced-features/web-servers.md index c291e5b9..1b74959b 100644 --- a/Docs/04-advanced-features/web-servers.md +++ b/Docs/04-advanced-features/web-servers.md @@ -499,8 +499,13 @@ close out slows your `write` calls (backpressure) instead of buffering without bound. If the client disconnects, hyper drops the response body and your next `write` to that stream fails with a catchable error — use `try`/`when` to detect it and -stop producing (and `close` any upstream you are proxying). The stream is closed -automatically when the handler ends on any path. +stop producing (and `close` any upstream you are proxying). + +**Always `close out`** to finalize the response — that is what signals the end +of the body to the client. A handler that starts a stream and returns without +`close`ing it leaves the response body open (the client keeps waiting) until the +program exits. Put the `close` on every path, e.g. in a `finally:` block if the +handler can error partway through. **Proxying an upstream to the browser** — combine with the outbound streaming client ([Interoperability → Streaming a response diff --git a/Docs/development/concurrency-phase-plan.md b/Docs/development/concurrency-phase-plan.md index 4449902c..681ee664 100644 --- a/Docs/development/concurrency-phase-plan.md +++ b/Docs/development/concurrency-phase-plan.md @@ -54,18 +54,37 @@ HARD RULES: > not the staged 1a→1b→1c sequence. What is covered: `main loop concurrently:` > surface (locked marker); plain `main loop` byte-compatible serial (tested); > `FuturesUnordered` of `!Send`, `&self`-borrowing handler futures on the -> existing runtime; isolated-per-request scopes; `catch_unwind` panic -> containment (siblings survive, tested via handler-error containment); -> in-flight cap (`CONCURRENT_HANDLER_LIMIT`), with 503/504/500 provided by the -> existing transport layer (bounded queue → 503, response deadline → 504, +> existing runtime; isolated-per-request **environment** scopes; a slow handler +> not blocking a fast sibling (tested); in-flight cap +> (`CONCURRENT_HANDLER_LIMIT`), with 503/504/500 provided by the existing +> transport layer (bounded queue → 503, response deadline → 504, > `ResponseCompletion` drop → 500); the empty-set busy-spin trap is avoided -> (cap ≥ 1 keeps the set non-empty). **Lighter than the full 1b checklist:** -> request-ID *structured* logging is not yet added (handler errors/panics are -> logged, but not a per-request accept/complete/fail/shed/timeout ID scheme); -> the eval-core `RefCell`-across-await audit is enforced mechanically by the -> crate-wide `#![deny(clippy::await_holding_refcell_ref)]` backstop rather than a -> written per-site walkthrough. **This is the maintainer STOP/review point** — -> please review before Phase 2. +> (cap ≥ 1 keeps the set non-empty). Cooperative, not parallel: handlers +> interleave only at await points, so a CPU-bound handler with no await still +> holds the interpreter thread (documented in `web-servers.md`). +> +> **Containment:** *runtime-error* containment is tested (an erroring handler +> does not kill the server). *Panic* containment is by construction via +> `catch_unwind` on each handler future but is **not** covered by a dedicated +> test — a deterministic WFL-level Rust panic is not readily expressible from +> the language, so the guarantee rests on the wrapper + the `panic = "unwind"` +> gate, not a regression test. +> +> **Known gap flagged in review (not yet fixed):** the concurrent loop isolates +> the **environment** per handler, but interpreter-level run-state +> (`current_count`/`in_count_loop`, `call_depth`, `call_stack`) still lives on +> the shared `Interpreter`. A handler that yields at an await *inside a `count` +> loop or a deep action call* can have that state overwritten by a concurrent +> sibling. The correct fix is a per-handler execution context; it is a scoped +> refactor and is called out for the maintainer review below. +> +> **Also lighter than the full 1b checklist:** request-ID *structured* logging is +> not yet added; the eval-core `RefCell`-across-await audit is enforced +> mechanically by the crate-wide `#![deny(clippy::await_holding_refcell_ref)]` +> backstop rather than a written per-site walkthrough. +> +> **This is the maintainer STOP/review point** — please review (especially the +> shared run-state gap) before Phase 2. | 2 | 2a | Structured nursery + join engine | ⬜ Not started | | 2 | 2b | `change shared` critical region | ⬜ Not started | | 3 | 3a | Multi-process workers (if profiling forces) | ⬜ Deferred | diff --git a/Docs/development/response-streaming-design.md b/Docs/development/response-streaming-design.md index 7eef944c..b6e20ad6 100644 --- a/Docs/development/response-streaming-design.md +++ b/Docs/development/response-streaming-design.md @@ -15,7 +15,7 @@ locked design for what remains. It complements — and defers to — | # | Capability | Status | |---|------------|--------| | 1 | Outbound response streaming (`stream response as`) | ✅ Shipped | -| 2 | Incremental reads (`wait for next chunk|line`) | ✅ Shipped | +| 2 | Incremental reads (`wait for next chunk\|line`) | ✅ Shipped | | 3 | Streamed server responses (start / write / flush / close) | ✅ Shipped | | 4 | Concurrent request handlers | ✅ Shipped (`main loop concurrently:`; Phase 1 of `concurrency-phase-plan.md`, awaiting maintainer review) | | 5 | Lifecycle (timeouts, backpressure, cancellation, catchable errors, close-on-exit) | ✅ Client + server streaming + per-handler isolation/containment shipped | diff --git a/TestPrograms/docs_examples/web_servers/concurrent_main_loop.wfl b/TestPrograms/docs_examples/web_servers/concurrent_main_loop.wfl index 03f79273..b25a6ff0 100644 --- a/TestPrograms/docs_examples/web_servers/concurrent_main_loop.wfl +++ b/TestPrograms/docs_examples/web_servers/concurrent_main_loop.wfl @@ -1,8 +1,10 @@ // Concurrent request handling with `main loop concurrently:`. // -// Iterations run in their own isolated scope, so a slow handler does not -// block other requests. Plain `main loop` stays serial; adding `concurrently` -// is the only way to opt in. +// Iterations run in their own isolated scope. Concurrency is cooperative on one +// thread: a slow handler yields to its siblings at await points (wait for, +// outbound HTTP, stream reads/writes, respond), so it does not block them there +// — but CPU-bound work with no await still holds the thread. Plain `main loop` +// stays serial; adding `concurrently` is the only way to opt in. // // Validated for syntax/analysis/lint only (layers 1-4): running it needs live // clients, so execution is skipped. diff --git a/src/interpreter/mod.rs b/src/interpreter/mod.rs index 780aae15..30cd6bb9 100644 --- a/src/interpreter/mod.rs +++ b/src/interpreter/mod.rs @@ -8715,7 +8715,21 @@ impl Interpreter { Some(expr) => { let v = self.evaluate_expression(expr, Rc::clone(&env)).await?; match &v { - Value::Number(n) => *n as u16, + // Require a whole number in the HTTP status range + // rather than silently wrapping a fractional or + // out-of-range value through `as u16`. + Value::Number(n) if n.fract() == 0.0 && *n >= 100.0 && *n <= 599.0 => { + *n as u16 + } + Value::Number(n) => { + return Err(RuntimeError::new( + format!( + "Expected a whole HTTP status code between 100 and 599, got {n}" + ), + *line, + *column, + )); + } _ => { return Err(RuntimeError::new( format!("Expected number for status, got {}", v.type_name()), diff --git a/src/parser/stmt/io.rs b/src/parser/stmt/io.rs index 070c7f58..1bc1d688 100644 --- a/src/parser/stmt/io.rs +++ b/src/parser/stmt/io.rs @@ -798,7 +798,22 @@ impl<'a> IoParser<'a> for Parser<'a> { // identifiers; the lexer merges a following bare-identifier value into // the same token (`line payload` -> Identifier("line payload")), so // split the value off the marker, mirroring the websocket-message form. - if let Some(next_token) = self.cursor.peek() + // + // Do NOT intercept a bare `line`/`chunk` that is immediately followed by + // `to`: that is the classic `write to ` form using a + // variable literally named `line`/`chunk` (common in line-by-line file + // processing). The streaming form always has a value between the marker + // and `to`, so a bare marker directly before `to` is not a stream write. + let bare_marker_before_to = matches!( + self.cursor.peek(), + Some(t) if matches!(&t.token, Token::Identifier(id) if id == "line" || id == "chunk") + ) && matches!( + self.cursor.peek_kind_n(1), + Some(Token::KeywordTo) + ); + + if !bare_marker_before_to + && let Some(next_token) = self.cursor.peek() && let Token::Identifier(id) = &next_token.token && (id == "line" || id == "chunk" diff --git a/src/parser/stmt/processes.rs b/src/parser/stmt/processes.rs index bc631749..4f717d64 100644 --- a/src/parser/stmt/processes.rs +++ b/src/parser/stmt/processes.rs @@ -397,34 +397,31 @@ impl<'a> ProcessParser<'a> for Parser<'a> { // identifiers, so `next chunk` / `next line` arrive as a single // token; a bare `next` (followed by chunk/line) is also handled. Token::Identifier(id) - if id == "next chunk" || id == "next line" || id == "next" => + if id == "next chunk" + || id == "next line" + || (id == "next" + && matches!( + self.cursor.peek_kind_n(1), + Some(Token::Identifier(kind)) if kind == "chunk" || kind == "line" + )) => { + // A bare `next` NOT followed by `chunk`/`line` (e.g. a + // variable named `next` in `wait for next milliseconds`) + // falls through to the duration/expression handling below. let is_line = id.ends_with("line"); let is_bare_next = id == "next"; self.bump_sync(); // Consume "next chunk"/"next line" (or bare "next") let is_line = if is_bare_next { - match self.cursor.peek() { - Some(t) => match &t.token { - Token::Identifier(kind) if kind == "chunk" => { - self.bump_sync(); - false - } - Token::Identifier(kind) if kind == "line" => { - self.bump_sync(); - true - } - _ => { - return Err(ParseError::from_token( - "Expected 'chunk' or 'line' after 'next'".to_string(), - t, - )); - } - }, - None => { - return Err(self - .cursor - .error("Expected 'chunk' or 'line' after 'next'".to_string())); + // The guard already confirmed `chunk`/`line` follows. + match self.cursor.peek().map(|t| &t.token) { + Some(Token::Identifier(kind)) if kind == "chunk" => { + self.bump_sync(); + false + } + _ => { + self.bump_sync(); + true } } } else { diff --git a/src/transpiler/javascript.rs b/src/transpiler/javascript.rs index 8e5e1712..0ba58ad0 100644 --- a/src/transpiler/javascript.rs +++ b/src/transpiler/javascript.rs @@ -2075,6 +2075,7 @@ impl JavaScriptTranspiler { | Statement::WaitForNextLineStatement { .. } | Statement::StartStreamingResponseStatement { .. } | Statement::StreamWriteStatement { .. } + | Statement::FlushStreamStatement { .. } | Statement::WaitForProcessStatement { .. } | Statement::WaitForRequestStatement { .. } => true, diff --git a/tests/concurrent_main_loop_test.rs b/tests/concurrent_main_loop_test.rs index ed4dee10..30560cec 100644 --- a/tests/concurrent_main_loop_test.rs +++ b/tests/concurrent_main_loop_test.rs @@ -5,6 +5,10 @@ // stays serial (concurrent = false) — no silent upgrade. // - Concurrent: a slow handler does NOT block a fast sibling. // - Serial: a slow handler DOES block the next request (unchanged behavior). +// - A handler that errors is contained; the server keeps serving. +// +// Each server exposes a `/shutdown` path that closes the server and breaks the +// loop, so the test can stop the server thread deterministically and join it. use std::time::{Duration, Instant}; use wfl::Interpreter; @@ -62,21 +66,36 @@ fn server_code(port: u16, concurrently: bool) -> String { main loop{marker}: wait for request comes in on srv as req with timeout 20000 store p as req["path"] - check if p is equal to "/slow": - wait for 500 milliseconds - respond to req with "slow" + check if p is equal to "/shutdown": + respond to req with "bye" + close server srv + break otherwise: - respond to req with "fast" + check if p is equal to "/slow": + wait for 500 milliseconds + respond to req with "slow" + otherwise: + respond to req with "fast" + end check end check end loop "# ) } +/// Send `/shutdown` so the server closes and its loop breaks, then join. +async fn shutdown(port: u16, server: std::thread::JoinHandle<()>) { + let _ = reqwest::Client::new() + .get(format!("http://127.0.0.1:{port}/shutdown")) + .send() + .await; + let _ = tokio::task::spawn_blocking(move || server.join()).await; +} + #[tokio::test] async fn test_concurrent_slow_handler_does_not_block_fast() { - let port = 8241; - let _server = start_server_thread(server_code(port, true)); + let port = 8341; + let server = start_server_thread(server_code(port, true)); tokio::time::sleep(Duration::from_millis(300)).await; let client = reqwest::Client::new(); @@ -103,31 +122,39 @@ async fn test_concurrent_slow_handler_does_not_block_fast() { "fast request was blocked behind the slow handler ({fast_elapsed:?})" ); - let slow_resp = slow.await.unwrap(); + let slow_resp = slow.await.expect("slow request task panicked"); assert_eq!(slow_resp.text().await.unwrap(), "slow"); + + shutdown(port, server).await; } #[tokio::test] async fn test_concurrent_handler_error_does_not_kill_server() { // A handler that errors mid-iteration (here: responding twice) must be // contained — the concurrent loop keeps serving other requests. - let port = 8243; + let port = 8342; let code = format!( r#" listen on port {port} as srv main loop concurrently: wait for request comes in on srv as req with timeout 20000 store p as req["path"] - check if p is equal to "/boom": - respond to req with "boom-ok" - respond to req with "this second respond errors" + check if p is equal to "/shutdown": + respond to req with "bye" + close server srv + break otherwise: - respond to req with "ok" + check if p is equal to "/boom": + respond to req with "boom-ok" + respond to req with "this second respond errors" + otherwise: + respond to req with "ok" + end check end check end loop "# ); - let _server = start_server_thread(code); + let server = start_server_thread(code); tokio::time::sleep(Duration::from_millis(300)).await; let client = reqwest::Client::new(); @@ -147,12 +174,14 @@ async fn test_concurrent_handler_error_does_not_kill_server() { .await .expect("follow-up request failed"); assert_eq!(ok.text().await.unwrap(), "ok"); + + shutdown(port, server).await; } #[tokio::test] async fn test_serial_slow_handler_blocks_next() { - let port = 8242; - let _server = start_server_thread(server_code(port, false)); + let port = 8343; + let server = start_server_thread(server_code(port, false)); tokio::time::sleep(Duration::from_millis(300)).await; let client = reqwest::Client::new(); @@ -179,5 +208,8 @@ async fn test_serial_slow_handler_blocks_next() { "serial main loop should have blocked the fast request behind the slow one ({fast_elapsed:?})" ); - let _ = slow.await; + let slow_resp = slow.await.expect("slow request task panicked"); + assert_eq!(slow_resp.text().await.unwrap(), "slow"); + + shutdown(port, server).await; } diff --git a/tests/http_server_streaming_test.rs b/tests/http_server_streaming_test.rs index c4418549..dbb2f8ef 100644 --- a/tests/http_server_streaming_test.rs +++ b/tests/http_server_streaming_test.rs @@ -76,6 +76,19 @@ fn test_flush_parses() { } } +#[test] +fn test_write_bare_line_variable_to_file_still_parses() { + // Backward compat: `write to ` with a variable literally named + // `line`/`chunk` must NOT be intercepted as a stream write (regression). + for src in ["write line to out", "write chunk to out"] { + let stmt = parse_single_statement(src); + match stmt { + Statement::WriteToStatement { .. } => {} + other => panic!("Expected WriteToStatement for {src:?}, got {other:?}"), + } + } +} + // ----------------------------- runtime tests ------------------------------ fn start_server_thread(code: String) -> std::thread::JoinHandle<()> { diff --git a/tests/http_stream_test.rs b/tests/http_stream_test.rs index 99d843bd..95b93370 100644 --- a/tests/http_stream_test.rs +++ b/tests/http_stream_test.rs @@ -97,6 +97,17 @@ fn test_wait_for_next_line_parses() { } } +#[test] +fn test_wait_for_next_as_duration_variable_still_parses() { + // Backward compat: a variable literally named `next` in a duration wait must + // NOT be intercepted as `wait for next chunk|line` (regression). + let stmt = parse_single_statement("wait for next milliseconds"); + match stmt { + Statement::WaitForDurationStatement { unit, .. } => assert_eq!(unit, "milliseconds"), + other => panic!("Expected WaitForDurationStatement, got {other:?}"), + } +} + // ----------------------------- runtime tests ------------------------------ /// Spawn a one-shot server that answers 200 with the given body, streamed with From ed89b65e6a0fa1bad06d2e99a28bd3145e4660e7 Mon Sep 17 00:00:00 2001 From: Claude Date: Wed, 22 Jul 2026 17:04:23 +0000 Subject: [PATCH 006/132] ci: skip non-executable streaming docs examples in run_integration_tests The integration runner executes every TestPrograms/*.wfl, but the three new docs-examples need a live upstream / HTTP clients (or run a server loop forever), so they fail/timeout when run standalone. They are validated statically via the docs-examples manifest (layers 1-4). Add the runner's first-line `CI-SKIP:` directive to each so they are skipped by run_integration_tests.sh while still being parse/analyze/lint-validated. Co-Authored-By: Claude Opus 4.8 Claude-Session: https://claude.ai/code/session_011LxXAm5E4Vy39kRLr8kRnX --- .../docs_examples/interoperability/streaming_response.wfl | 3 ++- .../docs_examples/web_servers/concurrent_main_loop.wfl | 3 ++- TestPrograms/docs_examples/web_servers/streaming_response.wfl | 3 ++- 3 files changed, 6 insertions(+), 3 deletions(-) diff --git a/TestPrograms/docs_examples/interoperability/streaming_response.wfl b/TestPrograms/docs_examples/interoperability/streaming_response.wfl index 7c5bca6f..02c1c2df 100644 --- a/TestPrograms/docs_examples/interoperability/streaming_response.wfl +++ b/TestPrograms/docs_examples/interoperability/streaming_response.wfl @@ -1,3 +1,4 @@ +// CI-SKIP: needs a live upstream; validated via docs-examples layers 1-4 // Streaming an outbound response incrementally. // // `stream response as` returns as soon as the status and headers arrive, @@ -22,4 +23,4 @@ count from 1 to 1000000: end check end count -close upstream +close upstream \ No newline at end of file diff --git a/TestPrograms/docs_examples/web_servers/concurrent_main_loop.wfl b/TestPrograms/docs_examples/web_servers/concurrent_main_loop.wfl index b25a6ff0..39591391 100644 --- a/TestPrograms/docs_examples/web_servers/concurrent_main_loop.wfl +++ b/TestPrograms/docs_examples/web_servers/concurrent_main_loop.wfl @@ -1,3 +1,4 @@ +// CI-SKIP: starts a server + concurrent loop; needs HTTP clients (layers 1-4) // Concurrent request handling with `main loop concurrently:`. // // Iterations run in their own isolated scope. Concurrency is cooperative on one @@ -19,4 +20,4 @@ main loop concurrently: otherwise: respond to req with "Hello!" end check -end loop +end loop \ No newline at end of file diff --git a/TestPrograms/docs_examples/web_servers/streaming_response.wfl b/TestPrograms/docs_examples/web_servers/streaming_response.wfl index c2a4424b..03e12dfc 100644 --- a/TestPrograms/docs_examples/web_servers/streaming_response.wfl +++ b/TestPrograms/docs_examples/web_servers/streaming_response.wfl @@ -1,3 +1,4 @@ +// CI-SKIP: starts a web server and needs an HTTP client; validated via docs-examples layers 1-4 // Streaming a server response. // // `start streaming response` sends status/headers immediately and binds a @@ -19,4 +20,4 @@ flush out write chunk "no newline here" to out close out -close server site +close server site \ No newline at end of file From d2d88c4732d8e712f2e7e404507390905f19fbee Mon Sep 17 00:00:00 2001 From: Claude Date: Wed, 22 Jul 2026 18:07:19 +0000 Subject: [PATCH 007/132] docs: adopt Logbie Testing Policy (root testing.md) + WFL testing profile MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Add the binding Logbie Testing Policy v1.0 as root `testing.md` (required by the policy's §4), together with the WFL project testing profile: supported platform/runtime tuples, one command per test layer + a run-all-presubmit block, critical user/operator journeys, risk triggers (esp. §11.3 concurrency/streaming/ lifecycle for this repo's async work), coverage targets, CI gating, owners, justified N/A layers, and tracked conformance gaps. Wire it into the agent guides: CLAUDE.md and AGENTS.md now list `testing.md` in the governance table and summarize the non-negotiables — Red→Green TDD evidence, risk classification (async/streaming/lifecycle = R3), real-boundary tests with meaningful/negative assertions, no manufactured-green, and PR evidence (§15). Risk class: R0 (documentation/policy; no executable behavior changes). Co-Authored-By: Claude Opus 4.8 Claude-Session: https://claude.ai/code/session_011LxXAm5E4Vy39kRLr8kRnX --- AGENTS.md | 33 ++- CLAUDE.md | 43 +++- testing.md | 709 +++++++++++++++++++++++++++++++++++++++++++++++++++++ 3 files changed, 780 insertions(+), 5 deletions(-) create mode 100644 testing.md diff --git a/AGENTS.md b/AGENTS.md index eda3e7a6..f5b5197f 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -12,12 +12,13 @@ Binding community and contribution policy lives at the **repo root** (not only u | `AI_POLICY.md` | **AI-assisted work is welcome** — WFL was built with AI; do not discriminate against AI use; human author remains accountable | | `CONTRIBUTING.md` | How to contribute; **Contributor application** process (Discussion or email) | | `SECURITY.md` | Private vulnerability reporting only — never file security bugs as public issues | +| `testing.md` | **Binding Logbie Testing Policy + WFL testing profile** — Red→Green TDD evidence, required test layers, risk classes, and merge/release gates (see **Testing Guidelines** below) | **Agent implications (already in force via governance):** - **AI is first-class** — use coding agents freely; same quality bar as hand-written work (tests, docs, compatibility, reviewability). - **Backward compatibility is sacred** — never break existing WFL programs without the documented deprecation path. -- **TDD mandatory** — failing tests first (`tests/`, `TestPrograms/`). +- **TDD mandatory** — failing tests first (`tests/`, `TestPrograms/`), governed by the binding **Logbie Testing Policy** in root `testing.md`: auditable **Red→Green** evidence for every behavioral change, coverage at the lowest useful layer plus every affected higher layer. - **Docs ship with the feature** — same change; validate examples; Dev Diary for non-trivial work. - **Quality gates** — `cargo fmt`, `clippy -D warnings`, `cargo test`; conventional commits. - **Do not invent maintainer identity or process** — Contributor status is by application; Maintainers own merges and releases unless those responsibilities are **explicitly delegated**. Prefer first name **Brad** only if referring to the primary maintainer in docs (no last name). @@ -121,12 +122,40 @@ Source Code → Lexer → Parser → Analyzer → Type Checker → Interpreter - Constants: `SCREAMING_SNAKE_CASE` ## Testing Guidelines -- **TDD is mandatory**: Write failing tests FIRST for any feature or bug fix. + +**Binding policy:** root `testing.md` holds the **Logbie Testing Policy** and the +WFL testing profile. It governs every behavioral change. Non-negotiables an agent +MUST follow: + +- **Red → Green → Refactor → Broaden → Record** — write the smallest useful test + FIRST, run it, confirm it **fails for the intended reason**, then make it pass; + a defect fix reproduces the defect. Keep auditable Red evidence (a Red commit + that is an ancestor of Green, or a timestamped CI artifact). A test first + observed after the code already passed is **not** a valid Red step. (§3, §6) +- **Classify risk first (R0–R3)** — concurrency, cancellation, lifecycle, + streaming, untrusted input, crypto/secrets, and backward compatibility are + **R3** and require negative/failure-path plus §11 risk-triggered tests. Risk is + never lowered to dodge a gate. (§5, §11) +- **Real boundaries, real assertions** — don't mock the boundary under test; + assert outcomes + side effects (not "didn't crash"); use negative assertions + for cancellation, writes-after-close, denial. (§7, §8.3) +- **No manufactured green** — never retry/skip/quarantine a required test to go + green; a flaky required test is failing. Non-executable docs programs use the + runner's `// CI-SKIP:` first-line directive and stay statically validated. (§8.2) +- **Concurrency/streaming/lifecycle (§11.3)** — for this repo's async/web/ + streaming work, prove races/ordering, cancellation, timeouts, disconnects, + bounded queues/backpressure, resource limits, clean shutdown, writes-after- + close, and that one slow/failed handler doesn't block unrelated work. +- **PR evidence (§15)** — record risk class, acceptance criteria → tests, Red + evidence, layers run, and residual risk (template in `testing.md`). + +### Mechanics - **Locations**: - Rust Unit/Integration: `tests/` - WFL End-to-End: `TestPrograms/` (must pass with release build) - WFL Test Framework: Use `describe`/`test` blocks, run with `wfl --test ` - **Conventions**: feature‑oriented names (`*_test.rs`, `*.test.wfl`), keep perf benches under `benches/`. +- **Commands & profile**: one command per layer + the "run all presubmit" block are in root `testing.md`. - **Testing Guide**: See `Docs/guides/testing-guide.md` for WFL testing framework documentation. ## Commit & Pull Request Guidelines diff --git a/CLAUDE.md b/CLAUDE.md index 789db63a..b123e166 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -14,12 +14,13 @@ Binding community and contribution policy lives at the **repo root** (not only u | `AI_POLICY.md` | **AI-assisted work is welcome** — WFL was built with AI; do not discriminate against AI use; human author remains accountable | | `CONTRIBUTING.md` | How to contribute; **Contributor application** process (Discussion or email) | | `SECURITY.md` | Private vulnerability reporting only — never file security bugs as public issues | +| `testing.md` | **Binding Logbie Testing Policy + WFL testing profile** — Red→Green TDD evidence, required test layers, risk classes, and merge/release gates (see **Testing Policy** below) | **Agent implications (already in force via governance):** - **AI is first-class** — use coding agents freely; same quality bar as hand-written work (tests, docs, compatibility, reviewability). - **Backward compatibility is sacred** — never break existing WFL programs without the documented deprecation path. -- **TDD mandatory** — failing tests first (`tests/`, `TestPrograms/`). +- **TDD mandatory** — failing tests first (`tests/`, `TestPrograms/`). Governed by the binding **Logbie Testing Policy** in root `testing.md` (see **Testing Policy** below): every behavioral change needs auditable **Red→Green** evidence and coverage at the lowest useful layer plus every affected higher layer. - **Docs ship with the feature** — same change; validate examples; Dev Diary for non-trivial work. - **Quality gates** — `cargo fmt`, `clippy -D warnings`, `cargo test`; conventional commits. - **Do not invent maintainer identity or process** — Contributor status is by application; Maintainers own merges and releases unless those responsibilities are **explicitly delegated**. Prefer first name **Brad** only if referring to the primary maintainer in docs (no last name). @@ -159,13 +160,49 @@ Source Code → Lexer → Parser → Analyzer → Type Checker → Interpreter - Types/Traits: `CamelCase` - Constants: `SCREAMING_SNAKE_CASE` -## Testing Guidelines -- **TDD is mandatory**: Write failing tests FIRST for any feature or bug fix. +## Testing Policy (binding — root `testing.md`) + +WFL adopts the **Logbie Testing Policy** (full text + the WFL testing profile in +root `testing.md`). It is binding for every behavioral change; the highlights an +agent MUST follow: + +- **Red → Green → Refactor → Broaden → Record.** Write the smallest useful test + FIRST and run it to confirm it **fails for the intended reason**, then make it + pass. A defect fix MUST reproduce the defect. Keep auditable evidence (a Red + test-only commit that is an ancestor of the Green commit, or a timestamped CI + artifact) — a test first observed *after* the code already passed does **not** + establish Red. (§3, §6) +- **Risk class first.** Classify R0–R3 before implementing; when ambiguous, the + higher class applies, and it MUST NOT be lowered to dodge a gate. Anything + touching **concurrency, cancellation, lifecycle, streaming, untrusted input, + crypto/secrets, or backward compatibility is R3** and needs negative/ + failure-path + the §11.3/§11.1 risk-triggered tests. (§5, §11) +- **Real boundaries.** A test MUST NOT mock the boundary it claims to verify; + "end-to-end" means the real binary/socket/file. Assert outcomes and side + effects, not "did not crash." Use negative assertions where absence matters + (cancellation, writes-after-close, denial). (§7, §8.3) +- **No manufactured green.** Required tests are never made green via retries, + skips, ignores, quarantine, or relaxed assertions; a flaky required test is a + failing test. Non-executable docs examples use the runner's `// CI-SKIP:` + first-line directive and are still validated statically. (§8.2) +- **Concurrency/streaming/lifecycle (§11.3) — always required for this repo's + async/web/streaming work:** prove races/ordering, cancellation, timeouts, + disconnects, bounded queues/backpressure, resource limits, clean shutdown, and + writes-after-close, and that one slow/failed handler does not block unrelated + work. +- **PR evidence (§15).** Every behavioral PR records risk class, acceptance + criteria → tests, Red evidence, the layers run, and residual risk (template in + `testing.md`). +- **Same bar for AI work.** AI-authored code/tests get the same verification — + "the model said it works" is not evidence. + +### Testing mechanics - **Locations**: - Rust Unit/Integration: `tests/` - WFL End-to-End: `TestPrograms/` (must pass with release build) - WFL Test Framework: Use `describe`/`test` blocks, run with `wfl --test ` - **Conventions**: feature‑oriented names (`*_test.rs`, `*.test.wfl`), keep perf benches under `benches/`. +- **Commands & profile**: one command per layer + the "run all presubmit" block are in root `testing.md`. - **Testing Guide**: See `Docs/guides/testing-guide.md` for WFL testing framework documentation. ## Commit & Pull Request Guidelines diff --git a/testing.md b/testing.md new file mode 100644 index 00000000..f82c556e --- /dev/null +++ b/testing.md @@ -0,0 +1,709 @@ +# WFL Testing — Policy & Project Profile + +This repository adopts the **Logbie Testing Policy** (reproduced verbatim in +[§ Logbie Testing Policy](#logbie-testing-policy) below) and defines the WFL +project testing profile required by that policy's §4. + +> **Adopted organization-policy version:** 1.0 +> **Testing-profile review date:** 2026-07-22 +> **Test-suite / infrastructure owner:** Maintainer (Brad, Logbie LLC) + +--- + +## WFL project testing profile (Logbie Testing Policy §4) + +### Supported configuration tuples + +| Tuple | Presubmit | Release | +|---|---|---| +| Linux x86-64 (ubuntu-latest), Rust stable (MSRV **1.94+**, edition 2024) | ✅ | ✅ | +| Windows x86-64, Rust stable | ✅ (integration) | ✅ | +| macOS (best-effort; not gated) | — | ⚠️ smoke only | + +Key runtime dependencies: `tokio`, `warp`/`hyper`, `reqwest`, `sqlx`, `logos`, +`tower-lsp`. The interpreter core is single-threaded (`Rc`/`RefCell`); async I/O +runs on Tokio. + +### Test layers — one documented command each + +| Layer | Command | +|---|---| +| Format (static) | `cargo fmt --all -- --check` | +| Lint (static) | `cargo clippy --all-targets --all-features -- -D warnings` | +| Unit + Rust integration | `cargo test --all` | +| WFL end-to-end programs | `cargo build --release` then `./scripts/run_integration_tests.sh` (`.ps1` on Windows) | +| Web-server end-to-end | `./scripts/run_web_tests.sh` (`.ps1` on Windows) | +| Docs examples validation | `python scripts/validate_docs_examples.py` | +| Benchmarks (perf, non-gating) | `cargo bench` | + +**Run all presubmit checks** (clean checkout): + +```bash +cargo fmt --all -- --check \ + && cargo clippy --all-targets --all-features -- -D warnings \ + && cargo test --all \ + && cargo build --release \ + && ./scripts/run_integration_tests.sh \ + && ./scripts/run_web_tests.sh \ + && python scripts/validate_docs_examples.py +``` + +Every command returns a non-zero exit status on failure. The presubmit suite is +hermetic: web/HTTP tests use local ephemeral servers and MUST NOT depend on the +public internet. Non-executable docs examples that need a live upstream or +external client are marked with a first-line `// CI-SKIP: ` and are +validated statically (layers 1–4) via the docs-examples manifest instead. + +### Required services, fixtures, credentials + +- No external credentials or network for presubmit. Local TCP servers on + ephemeral ports stand in for HTTP peers. +- SQLite for DB tests; no production data. +- TLS tests generate throwaway certs (`rcgen`). + +### Critical user/operator journeys (release-blocking end-to-end) + +1. **Run a program**: `wfl ` — lex → parse → analyze → typecheck → + interpret, correct exit code (`tests/`, `TestPrograms/`, `run_integration_tests`). +2. **Web server request/response**: `listen` → `wait for request` → `respond` + over a real socket (`run_web_tests`, `tests/web_server_*`). +3. **Streaming (client)**: `open url ... stream response` → `wait for next + line|chunk` → `nothing` at EOF (`tests/http_stream_test.rs`). +4. **Streaming (server)**: `start streaming response` → `write line|chunk` → + `close` (`tests/http_server_streaming_test.rs`). +5. **Concurrent handlers**: `main loop concurrently:` — a slow handler does not + block a fast sibling; failures are contained (`tests/concurrent_main_loop_test.rs`). +6. **File I/O**, **outbound HTTP**, **REPL**, **crypto/hashing**. + +### Risk triggers (§11) + +- **Concurrency / streaming / lifecycle (§11.3):** any change to + `main loop concurrently:`, request handling, or the streaming statements MUST + add tests for races/ordering, cancellation, timeouts, disconnects, bounded + queues/backpressure, resource limits, clean shutdown, and writes-after-close, + and MUST prove one slow/failed operation does not block unrelated work. **R3.** +- **Untrusted input (§11.1):** lexer, parser, pattern VM, HTTP/multipart, and + config readers require malformed/oversized/adversarial cases; fuzz targets + (`cargo fuzz`) with a retained corpus for parser/pattern paths. +- **Security/crypto:** WFLHASH, password hashing, subprocess sanitization — + positive/negative + invariant (constant-time, zeroize) tests. **R3.** +- **Backward compatibility (§11.6):** WFL is a language — every existing + `TestPrograms/*.wfl` MUST keep passing; new syntax MUST NOT steal previously + valid programs (regression tests required). + +### Coverage & budgets + +- Baseline: the policy's defaults (≥80% line / 70% branch overall; ≥90/85 on + changed code) are the target. **Known gap:** the repo does not yet run an + automated coverage gate in CI; establishing one is a tracked conformance item + (§19). Until then, changed behavior MUST still ship behavior/boundary/negative + tests at the lowest useful layer plus every affected higher layer. +- Performance budgets: Criterion benches under `benches/` are informational; no + release-blocking latency budget is defined yet (tracked gap). + +### CI jobs & gating + +GitHub Actions (`.github/workflows/`) runs, per push/PR: format, clippy, +debug/release build, `cargo test`, Linux + Windows integration, **Run WFL +Programs**, web tests, database tests, and fuzz-target compilation. All are +required checks; the default branch MUST stay green. The nightly build is a +release artifact and is separately monitored. + +### Evidence, runtimes, retention + +- Expected presubmit runtime: minutes (Rust build dominates). Slow web/timeout + tests use bounded deadlines, not unbounded sleeps. +- Evidence (Red→Green, CI run ids, commands) lives on the PR per §15; retained + per §15 retention windows. + +### Justified non-applicable layers + +- **Accessibility/UI (§11.7):** WFL ships a CLI/LSP, no first-party GUI — + UI/a11y layers are structurally N/A (LSP behavior is covered by `wfl-lsp` + tests). Reviewer-confirmed. + +### Conformance gaps (tracked, §19) + +- No automated coverage gate in CI yet. +- No scheduled extended fuzz/soak profile yet (fuzz targets compile in CI; longer + campaigns are not scheduled). +- No formal release-candidate artifact gate beyond the nightly build. + +These gaps are owned by the Maintainer and do not authorize new untested +behavior; touched code still follows Red→Green and the risk triggers above. + +--- + +## Logbie Testing Policy + +**Status:** Proposed organization policy, version 1.0 +**Owner:** Logbie LLC Engineering +**Effective date:** Upon adoption +**Last updated:** July 22, 2026 +**Applies to:** Every Logbie-owned software project, repository, package, service, application, game, agent, infrastructure definition, and release artifact + +### 1. Purpose + +Testing is executable evidence that a change behaves as intended, fails safely, and does not break supported behavior. It is part of design and implementation, not a cleanup task performed after the code appears finished. + +This policy establishes the minimum testing standard for all Logbie projects. Individual projects may impose stricter rules, but they may not weaken this policy without a time-limited, recorded exception. + +The objective is not to manufacture green dashboards. The objective is to make trustworthy changes and retain enough evidence for another engineer to understand what was proved. + +### 2. Policy language + +The terms in this document are normative: + +- **MUST / MUST NOT** — mandatory. A violation blocks merge or release unless this policy explicitly permits an exception. +- **SHOULD / SHOULD NOT** — the normal expectation. Deviations require a written reason in the change record. +- **MAY** — optional and permitted. +- **Required test** — a test selected by this policy, the project's testing profile, the change's risk, or an acceptance criterion. +- **Change record** — the durable issue, ticket, or pull request that owns the work and its evidence. +- **Public contract** — behavior relied on outside the changed implementation, including user and operator workflows, public APIs, CLI behavior, protocols, events, schemas, stored formats, packages, configuration, and documented compatibility. +- **Critical journey** — an end-to-end workflow whose failure would prevent a user or operator from receiving a core outcome or would create material security, privacy, data-integrity, availability, or recovery risk. +- **Independent reviewer** — a qualified person or separately instructed review agent that did not author the implementation, examines the actual diff and evidence, and has no ability to approve merely by repeating the author's claims. +- **Release** — any production promotion, continuous-deployment rollout, package or container publication, app-store submission, infrastructure apply, or externally distributed prerelease. Renaming a release "just a deployment" does not change its gates. + +### 3. Non-negotiable rules + +1. Every behavioral change MUST have automated regression coverage at the lowest useful layer and every affected higher layer. +2. Every R1–R3 behavioral change MUST follow **Red → Green → Refactor → Broaden → Record**, except for the Green → Green maintenance rule in Section 6.3 or the incident rule in Section 17. +3. Every new, modified, or removed behavior MUST include auditable evidence that the relevant test failed for the expected reason before the production change made it pass. A defect fix MUST reproduce the defect. +4. A releasable product MUST have real end-to-end tests for its critical user and operator journeys. Those tests are release-blocking. +5. A test MUST NOT mock, stub, or bypass the boundary it claims to verify. +6. A mocked component test MUST NOT be labeled end-to-end. +7. Required tests MUST NOT be made green through automatic retries, skips, ignores, quarantine, muted failures, relaxed assertions, or unexplained snapshot regeneration. +8. A flaky required test is a failing test. It blocks merge until repaired or until the responsible change is reverted. +9. CI MUST test the integrated change from a clean checkout. A passing developer machine is useful evidence, not final evidence. +10. Coverage numbers MUST NOT substitute for behavior, boundary, negative-path, recovery, or end-to-end tests. +11. Tests and test infrastructure are production-quality code. They receive review, ownership, maintenance, and security controls. +12. AI-generated code, tests, summaries, and claims receive exactly the same verification as human-written work. "The model said it works" is not test output; it is optimism wearing a tiny hard hat. + +### 4. Repository testing profile + +Every repository MUST contain a root-level `testing.md`. It MUST either include this policy or link to the canonical version, and it MUST define a project-specific testing profile containing: + +- The supported platform, architecture, runtime, browser, database, and dependency configuration tuples, including which run in presubmit and which run at release +- One documented command for each available test layer +- A single command or workflow that runs all presubmit checks +- Required services, containers, fixtures, credentials, hardware, and test data +- The project's critical user and operator journeys +- The risk triggers that require security, migration, performance, concurrency, recovery, fuzz, compatibility, or accessibility testing +- Coverage measurement and thresholds +- Performance and resource budgets, when applicable +- CI job names and which jobs block pull requests, merges, and releases +- The cadence, maximum evidence age, and invalidation triggers for required scheduled suites +- Expected test runtimes and the location of retained evidence +- Owners for the test suites and test infrastructure +- Any justified layer that does not apply to the project +- The adopted organization-policy version and the testing-profile review date + +Commands MUST work from a clean documented environment and MUST return a nonzero exit status on failure. Local project rules may be stricter than this policy but MUST NOT silently redefine terms such as "end-to-end," "pass," or "release-ready." + +The canonical organization copy of this file takes precedence over stale copied text. Repositories MUST adopt a new policy version before their next release and within 30 days unless a valid Section 17 exception says otherwise. + +A monorepo MAY use one root profile, but every independently releasable package, service, application, or artifact MUST have an identifiable subprofile covering its commands, critical journeys, owners, compatibility matrix, and release gates. + +A project-level "not applicable" declaration is permitted only when a layer is structurally impossible for that project type. It requires engineering-owner approval, a review date, and a technical explanation. It cannot override a risk introduced by a particular change. + +If a repository lacks a valid testing profile, behavioral changes to that repository are not ready to merge. + +### 5. Change risk classes + +Every change MUST be assigned the highest applicable risk class before implementation. Executable changes default to R2 until the change record justifies another class. Risk may be raised during review; it MUST NOT be lowered merely to avoid a test gate. + +When classification is ambiguous, the higher class applies. An R1 classification MUST explain why the change cannot affect a public contract, persistent state, security boundary, process boundary, or critical journey, and a reviewer MUST confirm it. + +| Class | Typical changes | Minimum verification | +| --- | --- | --- | +| **R0 — Non-behavioral** | Prose-only documentation, comments, spelling, and assets proven not to affect shipped output or an acceptance criterion | Formatting, link or documentation build checks as applicable; confirmation that no executable behavior changed | +| **R1 — Local behavior** | Isolated logic with no public contract, persistence, security, or process boundary | Auditable Red → Green evidence, focused unit tests, relevant component tests, full affected suite, static checks | +| **R2 — Product or boundary behavior** | Public API, CLI behavior, UI flow, database access, filesystem behavior, service integration, packaging, configuration with runtime effect | R1 plus real integration or contract tests, affected critical-journey end-to-end tests, compatibility checks, clean CI | +| **R3 — Critical behavior** | Material changes to authentication, authorization, cryptography, secrets, protected user data, destructive operations, schema migration, money, safety, untrusted-input boundaries, protocol guarantees, concurrency, cancellation, lifecycle, recovery, release controls, or high-availability behavior | R2 plus negative and failure-path tests, applicable security/property/fuzz/concurrency/recovery/performance tests, independent review, recovery evidence, and the full release-relevant end-to-end suite | + +All product releases, regardless of the individual changes they contain, MUST pass the full release gate in Section 14. + +### 6. Test-driven development + +#### 6.1 Required loop + +For each acceptance criterion or defect: + +1. **Specify** — express the behavior as an observable outcome, including relevant failure behavior. +2. **Red** — add or identify the smallest useful automated test and run it. Confirm that it fails for the intended reason. +3. **Green** — make the smallest coherent production change that satisfies the test. +4. **Refactor** — improve the implementation and tests while keeping them green. +5. **Broaden** — run the affected integration, contract, end-to-end, security, compatibility, and other risk-triggered suites. +6. **Record** — attach the evidence required by Section 15 to the change record. + +The Red step is invalid if the test fails because of a syntax error, broken fixture, missing dependency, unrelated failure, or an assertion that does not represent the required behavior. + +#### 6.2 Acceptable Red → Green evidence + +At least one of the following MUST be retained: + +- A focused test-only Red commit that is an ancestor of the Green implementation commit +- An independently timestamped CI or change-record artifact created before the Green implementation commit, tied to a Red revision and showing the test name, command, expected behavior, actual failure, and failure reason +- For a reproduced defect, an automated regression test applied to the recorded affected base revision and retained in a Red commit before the Green fix + +The evidence MUST identify the base, Red, and Green commit identifiers. The final history may be squashed after the evidence is attached to the change record. A newly written test that was observed only after the implementation already passed it does not establish the required Red step. Reverting or disabling completed code may prove that a regression test is capable of failing, but it does not retroactively prove TDD chronology. + +#### 6.3 Refactors and non-behavioral changes + +A behavior-preserving refactor or maintenance change MUST establish adequate characterization coverage and record a passing baseline before the change, then pass the same coverage afterward. Dependency, toolchain, packaging, infrastructure, and configuration maintenance also require applicable compatibility, security, integration, and end-to-end evidence. They do not need an artificial failing test when no behavior is intended to change. + +R0 changes do not require a manufactured Red step. Configuration, build, workflow, dependency, infrastructure, schema, and documentation-generator changes are not R0 when they can change executable behavior. + +#### 6.4 Incidents + +During an active incident, the minimum reversible mitigation may precede the normal Red step only under Section 17. The defect MUST receive regression coverage before the incident ticket is closed. An emergency is a reason to reorder evidence, not to delete it. + +### 7. Required test layers + +The risk table, project testing profile, acceptance criteria, and Section 11 triggers determine the required layers. Within that set, a change MUST use every layer needed to prove the affected contract without duplicating tests that add no distinct evidence. "Not applicable" requires a specific technical explanation in the change record and reviewer acceptance. + +#### 7.1 Static verification + +Projects MUST run applicable formatting, compilation, linting, type checking, schema validation, policy checks, secret scanning, dependency checks, and generated-file consistency checks. + +Static verification supplements executable tests; it does not replace them. + +#### 7.2 Unit tests + +Unit tests MUST cover new or changed business rules, validation, algorithms, state transitions, parsers, error classification, and policy decisions when those behaviors can be isolated. + +Unit tests SHOULD be fast, deterministic, precise, and independent of network or shared external state. They SHOULD assert observable behavior rather than private implementation details. + +#### 7.3 Component and service tests + +Component tests verify a complete module, package, process, or service through its public interface. They MUST use real internal components for the behavior under test and MAY substitute only dependencies outside the declared component boundary. + +#### 7.4 Integration tests + +Integration tests MUST exercise real boundaries whenever the change affects them, including as applicable: + +- Database engines, schemas, transactions, migrations, and queries +- Filesystems, permissions, paths, locks, and storage formats +- Processes, signals, standard streams, exit codes, and packaged binaries +- HTTP, WebSocket, streaming, queue, event, RPC, and protocol behavior +- Authentication, authorization, redaction, and policy enforcement +- Containers, operating-system facilities, and service discovery +- Timeouts, retries, cancellation, disconnects, backpressure, restart, and idempotency + +An in-memory replacement is not evidence that the actual database, filesystem, queue, protocol, or operating-system integration works. + +#### 7.5 Contract tests + +Every public or cross-service interface MUST have contract tests covering successful responses, errors, versioning, required fields, optional fields, limits, malformed input, and backward compatibility. + +Cross-repository contracts MUST name the provider owner, consumer owner, compatible version range, and repository responsible for candidate compatibility testing. Coordinated provider and consumer changes MUST test supported version skew before either side releases. + +Provider simulators and deterministic adapters MAY support fast tests, but a project that claims compatibility with an external provider MUST also verify the contract against that provider's official test environment, sandbox, or independently controlled conformance reference defined in the project testing profile. If the provider offers no safe test environment, a versioned signed recording or reference corpus MAY substitute only with owner approval, a declared freshness limit, and proof that it covers the claimed provider version. The project MUST state that live interoperability was not verified. + +#### 7.6 End-to-end tests + +End-to-end tests exercise a complete user- or operator-visible journey through the production entry points and shipped artifact. They MUST: + +- Start from a clean, production-like state +- Use the real application binary or packaged artifact +- Cross the real in-scope process, storage, protocol, and UI boundaries +- Assert both the final outcome and important externally visible side effects +- Exercise cleanup or recovery where the journey changes state +- Produce enough evidence to diagnose a failure + +Browser products MUST use a real supported browser for browser journeys. Service products MUST use their real network interface. CLI and desktop products MUST execute the packaged binary. Libraries MUST provide consumer, conformance, or system-harness tests that exercise the published artifact as a downstream user would. + +If an external paid or unsafe system is replaced, the test MUST be labeled as a system test rather than a true end-to-end test of that external integration. The external integration then requires a separate credentialed sandbox or release smoke profile, or the approved conformance-reference path in Section 7.5 when no safe provider environment exists. + +Every product MUST define a small, reliable critical-journey suite that blocks merge when affected and blocks every release in full. + +Every new or changed user- or operator-visible boundary behavior MUST add or update an end-to-end assertion unless an existing test already asserts that exact observable outcome. The critical-journey suite is the always-release-blocking subset, not a loophole for leaving noncritical workflows unproved. + +#### 7.7 Exploratory and manual testing + +Manual and exploratory testing MAY discover issues and provide useful product evidence. They MUST NOT replace required automated regression tests. Any defect found manually MUST receive automated coverage. A platform limitation that makes automation impossible requires a Section 17 exception and cannot waive a critical release gate. + +### 8. Test integrity + +#### 8.1 Determinism + +Tests MUST control or record time, randomness, locale, time zone, network assumptions, identifiers, and ordering when those inputs affect results. Randomized tests MUST report the seed and retain failing inputs. + +Date and time behavior MUST cover applicable expiration boundaries, daylight-saving transitions, leap dates, time zones, and locale changes. Tests capable of blocking MUST have an explicit bounded timeout. + +Tests MUST be isolated from one another. They MUST NOT rely on execution order, shared mutable fixtures, production state, or residue from a previous run. + +#### 8.2 Failures, flakes, and retries + +- A required test that fails once has failed. +- Required tests MUST NOT automatically retry at the test, framework, or CI layer. +- Required-suite configuration MUST expose first-attempt results and disable hidden framework or CI retries. +- A CI job MAY be rerun only when independent evidence shows that the runner or external test infrastructure failed before a product-test result was produced. The original run, evidence, and reason MUST remain visible. This permitted infrastructure rerun is not a test retry. +- A test that passes only on retry is flaky and blocks merge. +- Required tests MUST NOT be skipped, ignored, muted, quarantined, marked "allowed to fail," or removed from the gating suite to obtain green CI. +- Platform-specific tests MAY be selected only on their declared matrix entries, but the release gate MUST execute every supported entry. +- A known flaky test on the default branch is an urgent repository defect. The default branch MUST be restored to trustworthy green before unrelated behavioral work merges. + +#### 8.3 Assertions and snapshots + +Tests MUST assert meaningful outcomes, error behavior, and side effects. "Did not crash" is insufficient when the behavior has a defined result. + +Snapshots and golden files MUST be human-reviewable. Their changes MUST be reviewed like production code. Bulk regeneration without explaining each intentional behavioral difference is prohibited. + +Negative assertions MUST be used where absence matters, including authorization denial, secret non-disclosure, duplicate prevention, rollback, cancellation, and writes outside an allowed boundary. + +#### 8.4 Test doubles + +Mocks, fakes, stubs, emulators, and simulators MUST be named accurately and confined to a declared boundary. A test double MUST NOT make the behavior under test impossible to fail. + +Important doubles SHOULD be checked against the real implementation through contract tests so they do not become cheerful little liars with perfect uptime. + +### 9. Coverage and test strength + +Coverage is a diagnostic and regression floor, not a target that proves correctness. + +Unless a stricter project profile applies: + +- New projects MUST maintain at least **80% line coverage** and **70% branch coverage** across instrumentable first-party executable code before their first production release. +- Changed instrumentable executable code MUST achieve at least **90% line coverage** and **85% branch coverage**. +- A change MUST NOT reduce repository line or branch coverage by more than 0.5 percentage points without an approved exception. The stored baseline MUST never be silently lowered. +- Security, authorization, destructive-operation, financial, migration, and other R3 decision logic MUST have explicit tests for every identified policy outcome and failure mode, regardless of the aggregate percentage. + +Generated code, vendored code, build output, and provably unreachable platform shims MAY be excluded. Exclusions MUST be reviewable configuration, not ad hoc command-line omissions. + +"Changed code" means added or modified first-party executable lines relative to the target branch's merge base, with renames tracked when the tool supports them. Base and head MUST be measured in the same CI job using the same version-controlled coverage configuration, tool version, platform, exclusions, and rounding to two decimal places. A tool or configuration change that alters the denominator requires an old-versus-new comparison and test-infrastructure-owner approval. + +If reliable branch coverage is unavailable for a language, the testing profile MUST name the limitation and define reviewed condition, decision, scenario, or mutation coverage that supplies equivalent evidence. Calling code "non-instrumentable" without this approved alternative is not an exclusion. + +When conventional coverage is meaningless—such as declarative infrastructure, visual assets, or hardware workflows—the project testing profile MUST define scenario, requirement, state, or interface coverage instead. + +R3 projects SHOULD use mutation testing or an equivalent test-strength analysis for critical logic before a major release. Surviving meaningful mutations indicate missing assertions even when line coverage looks impressive. + +### 10. Test data, fixtures, and environments + +- Tests MUST NOT use production secrets, credentials, private keys, or uncontrolled personal data. +- Production-derived data MUST be minimized, sanitized, approved, and documented before use. +- Destructive tests MUST run only in explicitly disposable environments with guardrails that make production targeting impossible. +- Test resources MUST use unique names or isolated namespaces and MUST clean up on success, failure, cancellation, and timeout. +- Disposable environments MUST also have an independent time-to-live or janitor cleanup path for hard runner termination, where test code cannot execute cleanup. +- Fixtures MUST be small enough to review and version unless a justified artifact store is defined. +- Schema, protocol, and file-format fixtures MUST include the oldest supported version, current version, malformed cases, boundary sizes, and forward-compatibility cases where applicable. +- Credentials for sandbox or release profiles MUST be short-lived, least-privileged, redacted from output, and unavailable to untrusted pull requests. +- Test logs and artifacts MUST be sanitized before retention. + +The standard presubmit suite SHOULD be hermetic and MUST NOT depend on the public internet. Credentialed, hardware, provider, load, soak, and privileged tests belong in explicitly named profiles with controlled environments. + +### 11. Risk-triggered testing + +The following requirements apply whenever the corresponding risk exists. + +#### 11.1 Security and privacy + +Changes affecting trust boundaries, identity, authorization, secrets, untrusted input, or personal data MUST include: + +- Positive and negative authorization cases +- Role, tenant, and ownership-boundary tests +- Malformed, oversized, replayed, duplicated, and adversarial input cases +- Secret-redaction and sensitive-log assertions +- Session, token, timeout, revocation, and failure behavior as applicable +- Abuse-limit and resource-exhaustion tests where applicable +- A security-focused independent review for R3 changes + +Parsers, decoders, protocol handlers, and file readers exposed to untrusted input MUST have property or fuzz tests with a retained regression corpus. A bounded fuzz smoke run SHOULD execute on pull requests; longer campaigns SHOULD run on a scheduled profile. + +A known Critical security finding is non-waivable for a normal release. A known High finding blocks release unless the security owner approves a narrowly scoped Section 17 exception with demonstrated mitigation. A documented false positive supported by evidence is a resolved finding, not an exception. + +#### 11.2 Persistence and migrations + +Schema, data, and storage-format changes MUST be tested from every supported prior version using representative data. Tests MUST prove: + +- Upgrade correctness and idempotency +- Preservation of required data and constraints +- Behavior during partial failure, interruption, and restart +- Compatibility during any rolling or mixed-version deployment window +- The documented rollback, restore, or forward-repair strategy + +Destructive or irreversible migrations require explicit approval, a verified backup or recovery artifact, and a rehearsal in a production-like disposable environment. + +#### 11.3 Concurrency, streaming, and lifecycle + +Concurrent, asynchronous, networked, streaming, or long-running behavior MUST test applicable races, ordering, cancellation, timeouts, disconnects, bounded queues, backpressure, resource limits, clean shutdown, restart, and writes after close. + +Queue and event-driven behavior MUST test duplicates, delays, reordering, replay, poison messages, partial acknowledgement, and idempotent recovery when those conditions are possible. + +Tests MUST prove that one slow or failed operation does not improperly block unrelated work. Where the language or platform supplies race detection, concurrency modeling, sanitizers, or deterministic schedulers, the project SHOULD include them in CI or scheduled testing. + +#### 11.4 Reliability and recovery + +Stateful or continuously running systems MUST test crash recovery, restart, duplicate delivery, partial completion, idempotency, lost dependencies, corrupted or stale inputs, and degraded operation. + +High-availability projects MUST define scheduled chaos, failover, and soak profiles. A release MUST NOT claim a recovery property that has never been exercised. + +#### 11.5 Performance and resource use + +Projects with latency, throughput, memory, storage, startup-time, battery, network, or cost requirements MUST define measurable budgets in their testing profile. + +Performance tests MUST use controlled workloads, warmup rules, environments, and comparison methods. A statistically meaningful budget regression blocks release unless explicitly accepted under Section 17. Microbenchmarks alone do not prove system capacity. + +#### 11.6 Compatibility and packaging + +Every supported matrix entry MUST be tested before release. Projects MUST test the artifact users actually install or run, including package metadata, default configuration, startup, upgrade, and uninstall or cleanup behavior when applicable. + +Feature flags MUST be tested in their default and non-default states, including authorization and migration behavior affected by the flag. Removing a flag requires tests for the resulting permanent path. + +Examples and published code snippets SHOULD compile or execute in CI. Public libraries MUST test supported consumers and backward compatibility according to the project's versioning policy. + +#### 11.7 User interfaces and accessibility + +User-facing applications MUST test critical journeys through the real UI. Applicable flows MUST cover keyboard operation, focus behavior, accessible names, error presentation, and the project's accessibility target. Automated accessibility checks MUST be supplemented by documented manual checks for major releases where automation cannot establish the behavior. + +Visual regression tests MAY supplement behavioral tests but MUST NOT replace them. + +#### 11.8 Infrastructure and deployment + +Infrastructure-as-code and deployment changes MUST pass syntax, policy, plan, idempotency, least-privilege, secret-handling, and rollback, restore, or forward-repair checks. R2 and R3 changes MUST be exercised in a disposable or staging environment before production. + +Tests MUST make destructive plans obvious and MUST prevent production mutation from ordinary CI. + +#### 11.9 Games and deterministic simulations + +Game and simulation projects MUST make authoritative logic testable deterministically. If presentation coupling prevents direct isolation, the project testing profile MUST define a deterministic system harness. Tests MUST cover seeded replay, invariants, boundary conditions, save/load round trips, version compatibility, economic or scoring conservation rules, and long-run stability as applicable. + +Balance evaluation and bot simulations are evidence for design decisions, but they do not replace correctness tests. + +#### 11.10 AI and model-backed features + +Model-backed behavior MUST keep deterministic application rules under ordinary automated tests. Provider adapters require contract tests. Prompt, model, retrieval, or tool-policy changes MUST use versioned evaluation sets with documented pass thresholds, safety cases, cost limits, and regression comparisons. + +Live model evaluations MUST run only in an authorized credentialed profile. Their nondeterminism MUST be measured and reported; it MUST NOT be hidden with retries until a preferred answer appears. + +#### 11.11 Observability and audit behavior + +R3 services and agents MUST test required audit events, security-relevant logs, metrics, trace propagation, alert conditions, and redaction for privileged operations and critical failures. Tests MUST prove both that required signals appear and that secrets or protected data do not. + +### 12. Test design and maintenance + +- Test names MUST describe the behavior and relevant condition. +- A test SHOULD have one clear reason to fail, while a scenario test MAY make several related assertions needed to diagnose the journey. +- Tests SHOULD use public interfaces and stable contracts. +- Shared fixtures and helpers MUST reduce accidental complexity without hiding important setup or assertions. +- Timing-based waits and arbitrary sleeps SHOULD be replaced by observable readiness conditions and bounded deadlines. +- Test suites MUST be runnable in parallel only when their isolation supports it. +- Slow tests MUST be measured and improved, split by profile, or assigned appropriate infrastructure; they MUST NOT be silently removed from required evidence. +- Deleted behavior SHOULD have obsolete tests removed. Changed tests MUST explain whether the product contract changed or the prior test was incorrect. +- A production defect that escaped existing tests MUST add or improve the layer that should have caught it, not only the layer where it was easiest to reproduce. + +Test-only hooks in production code MUST NOT weaken security or alter normal behavior. If unavoidable, they require review and must be inaccessible in production builds or deployments. + +Changes to CI workflows, test filters, impact maps, retry settings, coverage tools or exclusions, thresholds, required-check names, risk rules, or `testing.md` MUST receive test-infrastructure-owner approval and run the complete presubmit suite. A change MUST NOT weaken the machinery that judges that same change. + +CI MUST fail closed when the change record lacks a risk class, a required job has no result, or the executed job set does not match the approved testing profile and risk triggers. + +### 13. Continuous-integration profiles + +Projects MUST define the following profiles where applicable. + +Default and release branches MUST be protected. Required checks and review rules apply to maintainers, administrators, bots, and merge queues; ordinary work MUST NOT use an administrative bypass. Section 17 is the only bypass path and it MUST be logged. + +#### 13.1 Pull request / presubmit + +Runs from a clean checkout with locked dependencies and pinned or recorded tool versions, and includes: + +- Formatting, build, lint, type, policy, secret, and dependency checks +- Focused and complete unit suites +- Affected component, integration, and contract suites +- Affected critical-journey end-to-end tests +- Coverage and changed-code thresholds +- Bounded property, fuzz, security, or concurrency smoke tests triggered by risk + +Required checks MUST pass on the final proposed commit. + +Path-based or impact-based test selection MAY reduce presubmit work only when backed by a maintained dependency map. The merge queue or another pre-merge gate MUST still run the complete impacted suite. + +#### 13.2 Merge queue / integrated branch + +The change MUST be tested with the latest target branch and other queued changes. The merge result MUST pass all required presubmit checks from a clean environment. Stale green results from an earlier base are insufficient. + +The default branch MUST remain green. A red default branch is an incident owned ahead of feature work. + +#### 13.3 Scheduled extended profile + +Longer fuzzing, property exploration, compatibility matrices, credentialed sandboxes, race detection, sanitizers, load, chaos, failover, and soak tests SHOULD run on a documented schedule according to project risk. + +Failures MUST create or update an owned change record and block release. They MUST block further affected merges when they invalidate presubmit evidence. + +The project profile MUST define freshness for each scheduled suite; seven days is the default maximum. A new run is required sooner when relevant production code, dependencies, toolchain, tests, configuration, environment, or candidate artifacts change. + +#### 13.4 Release candidate + +Runs against the immutable release candidate artifact set and includes: + +- The complete supported platform and dependency matrix +- The full critical-journey end-to-end suite +- Installation, startup, upgrade, migration, compatibility, and rollback, restore, or forward-repair checks +- All R3 security, recovery, concurrency, and data-integrity suites +- Required performance, load, soak, provider, hardware, and accessibility evidence +- Artifact integrity, provenance, license, dependency, and secret checks + +Every released binary, package, image, installer, or other artifact MUST be identified and tested. The tested artifact set MUST be the set released, and each digest or equivalent immutable identifier MUST be recorded. Rebuilding after the gate invalidates the gate. + +A platform-controlled signing, notarization, store-repackaging, or deployment transformation MAY occur after the main gate only when its input and output provenance are recorded, the transformation is deterministic or independently verified, and the distributed result passes a post-transformation smoke test. Unchanged evidence MAY be reused only for the exact same immutable artifact and while every required scheduled result remains fresh. + +#### 13.5 Post-deployment smoke + +Deployed services SHOULD run a small non-destructive smoke suite that verifies health and critical external paths. Production smoke tests MUST be explicitly authorized, isolated from ordinary user data, safe to repeat, monitored, and incapable of destructive action. + +Post-deployment checks supplement the release gate; they do not excuse missing pre-release evidence. + +### 14. Merge and release gates + +Section 17 is the sole override path. An exception may authorize a clearly labeled emergency merge or build, but it never converts missing or failed evidence into a pass. A reproducible product-test failure, authorization bypass, data loss or corruption, exposed secret, unresolved Critical vulnerability, or failed destructive-migration recovery test is non-waivable. + +#### 14.1 A pull request MUST NOT merge when + +- A required check failed, did not run, or produced ambiguous results +- A required test was automatically retried, skipped, ignored, muted, quarantined, allowed to fail, or rerun outside the proven infrastructure-failure rule in Section 8.2 +- Red → Green evidence is missing for changed behavior +- Coverage or a declared budget regressed beyond policy +- The affected critical journey lacks end-to-end coverage +- The target branch or required baseline is red, unless this pull request is narrowly scoped to repairing or reverting that failure and all unrelated required checks pass +- Required test evidence cannot be tied to the final commit +- An unresolved blocking review, security finding, migration risk, or rollback gap remains +- Test changes weaken protection without an approved contract change + +#### 14.2 A release MUST NOT proceed when + +- The immutable candidate did not pass the full release-candidate profile +- Any supported platform or critical journey is untested or failed +- A required scheduled test has an unresolved failure that affects the release +- A Critical security issue remains unresolved, or a High issue lacks the valid security-owner exception allowed by Section 11.1 +- Data migration, compatibility, and rollback, restore, or forward-repair evidence is incomplete +- A declared performance or reliability claim lacks passing evidence +- Required artifacts, test reports, or approvals are missing + +"It is probably fine," a deadline, and repeated clicking of the rerun button are not release criteria. + +### 15. Required change evidence + +Every behavioral pull request or equivalent change record MUST include: + +- Change and risk-class summary +- Impacted repositories, release artifacts, public contracts, supported configurations, and critical journeys, with reviewer acceptance of the impact analysis +- Acceptance criteria mapped to tests +- Red evidence for each new behavior or defect fix +- Green evidence tied to the final commit +- Tests added, changed, or removed +- Exact commands or CI run identifiers for unit, integration, contract, end-to-end, and risk-triggered tests +- Coverage results and any exclusions +- Supported matrix entries exercised +- Known limitations, residual risks, and specifically justified non-applicable layers +- Review evidence required by the risk class +- Rollback, restore, or forward-repair instructions when external state can change + +Recommended pull-request section: + +```markdown +## Test evidence + +- Risk class: +- Acceptance criteria → tests: +- Red evidence: +- Unit/component: +- Integration/contract: +- End-to-end: +- Security/migration/concurrency/performance/other: +- Coverage: +- Platforms: +- Not applicable, with reason: +- Rollback/recovery: +- Residual risk: +``` + +Pull-request evidence MUST remain accessible for at least 90 days after merge or closure. Default-branch and scheduled evidence MUST remain accessible for at least 180 days. Release, security, migration, rollback, and exception evidence MUST remain accessible for the supported lifetime of the release plus one year, and never less than 24 months. Evidence MUST NOT contain secrets or uncontrolled personal data. + +Every release MUST archive an organization-controlled evidence manifest containing the commit, artifact-set identifiers, policy and testing-profile versions, required job results, tested configuration tuples, commands, toolchain and environment, seeds, coverage and scan summaries, exceptions, approvals, and recovery evidence. Short-lived CI links alone do not satisfy retention. + +### 16. Ownership and defect handling + +The author of a change owns its tests until the change is accepted. The project owner owns the ongoing health of the suites and infrastructure. + +When a production defect escapes: + +1. Reproduce it with the strongest feasible automated test. +2. Identify which test layer should have prevented the escape. +3. Fix the defect using the Red → Green loop. +4. Repair the missing assertion, fixture, environment, or gate. +5. Search for the same gap in adjacent behavior. +6. Record the cause and prevention evidence. + +CI infrastructure failures MUST be distinguished from product failures using evidence, not guesses. Repeated infrastructure instability is itself a blocking engineering defect. + +### 17. Exceptions and emergency changes + +An exception records temporary risk; it does not convert missing evidence into a pass. + +Every exception MUST identify: + +- The exact rule and affected scope +- Why compliance is technically impossible or would worsen an active incident +- The approving project owner and, for security or privacy rules, the security owner +- Start time, expiration, and maximum affected releases +- Compensating verification and containment +- Rollback plan +- A linked repair ticket with owner and deadline + +An exception is limited to one repository, an exact commit or candidate, and at most one release. The requester MUST NOT be the sole approver. Ordinary exceptions expire within 30 days; R3 exceptions expire within 7 days and require the affected security, data, reliability, or other domain owner. Renewal requires new evidence and approval; repeated renewal requires escalation to the Logbie engineering policy owner. An expired exception blocks the affected merge or release. + +Schedule pressure, test duration, missing CI setup, inconvenience, and "the change is small" are not valid reasons. + +Exceptions MUST NOT permit a normal release to claim that an untested critical journey, unsupported migration, or failed required suite passed. The release remains blocked or is explicitly classified as an emergency build with its limitations visible. + +An ordinary exception MAY address temporarily unavailable evidence or an unavailable environment. It MUST NOT waive a known product failure or any non-waivable condition in Section 14. A required test cannot be retried, skipped, quarantined, muted, or relabeled through an exception to manufacture a pass. + +During an active incident, an authorized minimum reversible mitigation MAY merge with reordered Red → Green evidence when delay would cause greater harm. All available focused tests and static checks MUST still run, the rollback path MUST be prepared, and regression coverage plus the full affected suites MUST pass within 24 hours, before the incident is closed, and before another normal deployment of the affected component. + +Expired exceptions fail closed. + +### 18. Definition of done + +A change is done only when all applicable conditions are true: + +- Acceptance criteria are observable and mapped to passing tests. +- Required Red → Green evidence exists. +- Relevant unit, component, integration, contract, and end-to-end tests pass. +- Static, coverage, security, compatibility, performance, migration, recovery, and other risk-triggered gates pass. +- The final integrated commit passes in a clean CI environment. +- Required independent review is complete and blocking findings are resolved. +- Documentation, examples, configuration, fixtures, and operational instructions are updated. +- Rollback or recovery covers both repository content and external state. +- Evidence is attached to the durable change record and contains no secrets. +- No required test is flaky, skipped, retried, quarantined, muted, or unexplained. +- Remaining work and risks are explicitly ticketed and do not violate a merge or release gate. + +If a required condition is unmet, the correct state is **blocked**, **in progress**, or **ready for approval**—not **done**. + +### 19. Adoption checklist + +An existing repository MUST add a valid testing profile before its first behavioral pull request after policy adoption or within 30 days, whichever comes first. A temporary delay requires Section 17; no production release may occur before profile adoption. Each repository MUST complete the following before its next production release: + +- [ ] Add a root-level `testing.md` adopting this policy +- [ ] Name test owners +- [ ] Document clean-environment commands for every applicable layer +- [ ] Inventory critical user and operator journeys +- [ ] Establish the supported compatibility matrix +- [ ] Establish coverage baselines and required thresholds +- [ ] Configure presubmit and merge protection +- [ ] Build a release-candidate workflow against the immutable artifact +- [ ] Remove or repair required retries, skips, quarantines, and allowed failures +- [ ] Define security, migration, concurrency, recovery, performance, and other risk triggers +- [ ] Define test-data and credential controls +- [ ] Define evidence retention and artifact locations +- [ ] Record and prioritize gaps that prevent full compliance + +Until adoption is complete, gaps MUST be visible as owned tickets. A repository MUST NOT describe itself as fully compliant while a mandatory gate is missing. + +Existing projects MUST apply this policy immediately to new and changed behavior. Touched legacy behavior MUST gain characterization coverage, and known repository-wide gaps MUST have a dated conformance plan. Existing debt does not authorize new untested behavior. + +### 20. Canonical rule + +**No behavioral change without an honest failing test. No boundary claim without a real boundary test. No release without real end-to-end proof. No green build manufactured from retries, skips, quarantine, or wishful thinking.** From ad6c09935894be07218cb6d1b6dad7e2fd32adf2 Mon Sep 17 00:00:00 2001 From: Claude Date: Wed, 22 Jul 2026 18:15:40 +0000 Subject: [PATCH 008/132] fix: streaming bindings use define_or_replace (loop re-reads + no bind-collision leak) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Address PR #641 review (P1 #8 + Copilot bind-collision leak). `wait for next chunk|line as ` and the `open ... stream response as ` / `start streaming response as ` object binds used `Environment::define`, which errors if the name already exists in scope. Re-reading into the same variable in one scope (or a recycled `main loop` env) therefore failed on the second read — the normal streaming loop pattern. Switch all streaming bindings to `define_or_replace`, matching `wait for request ... as req`. This also removes the bind-collision leak Copilot flagged: `start streaming response` committed the response head + stored the body sender before binding the handle; a binding failure left an unreachable open stream (client hang + leaked sender). With define_or_replace the bind cannot fail after the head is sent. Risk class: R2 (streaming boundary behavior). Red→Green evidence: added `test_next_line_reusing_same_variable_in_one_scope` (observed failing with "Variable 'line' has already been defined" before the fix; passing after) plus `test_next_line_count_loop_reusing_same_variable`. Full http_stream_test suite green (12/12); clippy clean. Co-Authored-By: Claude Opus 4.8 Claude-Session: https://claude.ai/code/session_011LxXAm5E4Vy39kRLr8kRnX --- src/interpreter/mod.rs | 51 +++++++++++++++++++++------------------ tests/http_stream_test.rs | 42 ++++++++++++++++++++++++++++++++ 2 files changed, 69 insertions(+), 24 deletions(-) diff --git a/src/interpreter/mod.rs b/src/interpreter/mod.rs index 30cd6bb9..a0472292 100644 --- a/src/interpreter/mod.rs +++ b/src/interpreter/mod.rs @@ -6709,10 +6709,10 @@ impl Interpreter { stream_map.insert("_stream".to_string(), Value::Text(handle_id.into())); let value = Value::Object(Rc::new(RefCell::new(stream_map))); - match env.borrow_mut().define(variable_name, value) { - Ok(_) => Ok((Value::Null, ControlFlow::None)), - Err(msg) => Err(RuntimeError::new(msg, *line, *column)), - } + // define_or_replace so a `main loop` handler that rebinds + // `upstream` on each request works. + env.borrow_mut().define_or_replace(variable_name, value); + Ok((Value::Null, ControlFlow::None)) } Err(error) => Err(self.http_client_error(error, *line, *column)), } @@ -6732,18 +6732,20 @@ impl Interpreter { .await { // Raw bytes as Binary so callers can handle any payload. + // define_or_replace (not define) so re-reading into the same + // variable across a loop refreshes it, matching + // `wait for request ... as req`. Ok(Some(bytes)) => { let value = Value::Binary(Arc::from(bytes.as_slice())); - match env.borrow_mut().define(variable_name, value) { - Ok(_) => Ok((Value::Null, ControlFlow::None)), - Err(msg) => Err(RuntimeError::new(msg, *line, *column)), - } + env.borrow_mut().define_or_replace(variable_name, value); + Ok((Value::Null, ControlFlow::None)) } // Clean EOF binds `nothing` so `check if chunk is nothing` ends the loop. - Ok(None) => match env.borrow_mut().define(variable_name, Value::Null) { - Ok(_) => Ok((Value::Null, ControlFlow::None)), - Err(msg) => Err(RuntimeError::new(msg, *line, *column)), - }, + Ok(None) => { + env.borrow_mut() + .define_or_replace(variable_name, Value::Null); + Ok((Value::Null, ControlFlow::None)) + } Err(error) => Err(self.http_client_error(error, *line, *column)), } } @@ -6763,15 +6765,14 @@ impl Interpreter { { Ok(Some(line_text)) => { let value = Value::Text(line_text.into()); - match env.borrow_mut().define(variable_name, value) { - Ok(_) => Ok((Value::Null, ControlFlow::None)), - Err(msg) => Err(RuntimeError::new(msg, *line, *column)), - } + env.borrow_mut().define_or_replace(variable_name, value); + Ok((Value::Null, ControlFlow::None)) + } + Ok(None) => { + env.borrow_mut() + .define_or_replace(variable_name, Value::Null); + Ok((Value::Null, ControlFlow::None)) } - Ok(None) => match env.borrow_mut().define(variable_name, Value::Null) { - Ok(_) => Ok((Value::Null, ControlFlow::None)), - Err(msg) => Err(RuntimeError::new(msg, *line, *column)), - }, Err(error) => Err(self.http_client_error(error, *line, *column)), } } @@ -8849,10 +8850,12 @@ impl Interpreter { stream_map.insert("_server_stream".to_string(), Value::Text(handle_id.into())); stream_map.insert("status".to_string(), Value::Number(status_code as f64)); let value = Value::Object(Rc::new(RefCell::new(stream_map))); - match env.borrow_mut().define(variable_name, value) { - Ok(_) => Ok((Value::Null, ControlFlow::None)), - Err(msg) => Err(RuntimeError::new(msg, *line, *column)), - } + // define_or_replace so a `main loop` handler that rebinds `out` + // each request works — and so binding never fails after the + // stream head was already committed (which would otherwise leak + // the sender and hang the client). + env.borrow_mut().define_or_replace(variable_name, value); + Ok((Value::Null, ControlFlow::None)) } Statement::StreamWriteStatement { value, diff --git a/tests/http_stream_test.rs b/tests/http_stream_test.rs index 95b93370..6e67d486 100644 --- a/tests/http_stream_test.rs +++ b/tests/http_stream_test.rs @@ -230,6 +230,48 @@ async fn test_next_line_returns_final_unterminated_line() { } } +#[tokio::test] +async fn test_next_line_reusing_same_variable_in_one_scope() { + // Regression: reading successive lines into the SAME `as line` variable in + // one scope must work (define errors on an existing binding, so this needs + // define_or_replace — matching `wait for request ... as req`). + let url = spawn_body_server("a\nb\n").await; + let code = format!( + r#" + open url at "{url}" and stream response as up + wait for next line from up as line + store first as line + wait for next line from up as line + store second as line + "# + ); + let interpreter = run_wfl(&code).await; + assert_eq!(get_text(&interpreter, "first"), "a"); + assert_eq!(get_text(&interpreter, "second"), "b"); +} + +#[tokio::test] +async fn test_next_line_count_loop_reusing_same_variable() { + // The common streaming pattern: a count loop reusing `as line`. + let url = spawn_body_server("a\nb\nc\n").await; + let code = format!( + r#" + open url at "{url}" and stream response as up + store collected as "" + count from 1 to 100: + wait for next line from up as line + check if line is nothing: + break + otherwise: + change collected to collected with line + end check + end count + "# + ); + let interpreter = run_wfl(&code).await; + assert_eq!(get_text(&interpreter, "collected"), "abc"); +} + #[tokio::test] async fn test_next_chunk_yields_binary_then_nothing() { let url = spawn_body_server("raw-bytes-payload").await; From 465c3dadba678de1c398cc566fe1cdf570a3ae96 Mon Sep 17 00:00:00 2001 From: Claude Date: Wed, 22 Jul 2026 18:31:03 +0000 Subject: [PATCH 009/132] =?UTF-8?q?fix:=20address=20PR=20#641=20review=20?= =?UTF-8?q?=E2=80=94=20transpile-fail,=20streaming=20type=20checks,=20resp?= =?UTF-8?q?onse=20byte=20budget?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Contained hardening from maintainer review: - transpiler: `main loop concurrently:` now returns a TranspileError instead of silently emitting a serial `while(true)` loop (no faithful serial translation exists). Plain `main loop` still transpiles. - typechecker: enforce clause types on `start streaming response` (status: Number, content type: Text, headers: Map) and clarify the HTTP-body message (numbers/booleans are accepted and converted). - interpreter: `write line|chunk` on a server response stream now enforces the `max_response_bytes` ceiling, so a stream cannot bypass the buffered-response budget. Byte total is tracked per open stream. - parser: `flush` only dispatches as a flush statement when it has an operand. - interpreter: stream handles must be the handle object (`_stream` / `_server_stream`); a bare Text id is no longer accepted. Tests: write-after-close does not reach the client; concurrent main loop fails to transpile. --- src/interpreter/mod.rs | 58 +++++++++++++++++++++-------- src/parser/mod.rs | 9 +++-- src/transpiler/javascript.rs | 17 ++++++++- src/typechecker/mod.rs | 52 +++++++++++++++++++++++--- tests/http_server_streaming_test.rs | 38 +++++++++++++++++++ tests/transpiler_test.rs | 19 ++++++++++ 6 files changed, 167 insertions(+), 26 deletions(-) diff --git a/src/interpreter/mod.rs b/src/interpreter/mod.rs index a0472292..65434406 100644 --- a/src/interpreter/mod.rs +++ b/src/interpreter/mod.rs @@ -65,6 +65,10 @@ use tokio::sync::{mpsc, oneshot}; // Type alias for complex pending response type type PendingResponseSender = Arc>>>; +/// An open server response stream: the bounded body-chunk sender plus the +/// running total of body bytes written (enforced against `max_response_bytes`). +type ServerResponseStream = (mpsc::Sender>, usize); + /// A dequeued HTTP request parked in `pending_responses` awaiting a `respond`. /// /// Holds only the response channel. The request's in-flight admission slot is @@ -1149,9 +1153,11 @@ pub struct Interpreter { ws_connections: WsConnectionRegistry, // Outbound senders for all live WebSocket connections pending_responses: RefCell>, // Pending responses (channel + admission slot) by request ID /// Open server response streams (`start streaming response`), keyed by - /// handle id ("respstream1", ...). Each holds the bounded body-chunk sender; - /// `write line|chunk`/`flush` push to it, `close` drops it (ending the body). - server_response_streams: RefCell>>>, + /// handle id ("respstream1", ...). Each holds the bounded body-chunk sender + /// plus the running total of body bytes written, enforced against + /// `max_response_bytes` so a stream cannot bypass the buffered response + /// ceiling. `write line|chunk`/`flush` push to it, `close` drops it. + server_response_streams: RefCell>, next_response_stream_id: std::cell::Cell, #[allow(dead_code)] // Used for future security features config: Arc, // Configuration for security and other settings @@ -3682,10 +3688,9 @@ impl Interpreter { column, )), }, - Value::Text(id) => Ok(id.to_string()), _ => Err(RuntimeError::new( format!( - "Expected a streaming response handle, got {}", + "Expected a streaming response handle (from `stream response as ...`), got {}", value.type_name() ), line, @@ -3716,10 +3721,9 @@ impl Interpreter { column, )), }, - Value::Text(id) => Ok(id.to_string()), _ => Err(RuntimeError::new( format!( - "Expected a server response stream handle, got {}", + "Expected a server response stream (from `start streaming response as ...`), got {}", value.type_name() ), line, @@ -8844,7 +8848,7 @@ impl Interpreter { }; self.server_response_streams .borrow_mut() - .insert(handle_id.clone(), tx); + .insert(handle_id.clone(), (tx, 0)); let mut stream_map = HashMap::new(); stream_map.insert("_server_stream".to_string(), Value::Text(handle_id.into())); @@ -8887,13 +8891,37 @@ impl Interpreter { bytes.push(b'\n'); } - // Clone the sender out so the map borrow isn't held across the - // (possibly backpressured) send await. - let sender = self - .server_response_streams - .borrow() - .get(&handle_id) - .cloned(); + // Enforce the response-byte ceiling on the running total (so a + // stream cannot bypass `web_server_max_response_size` via one + // huge chunk or many chunks), then clone the sender out so the + // map borrow is not held across the (possibly backpressured) + // send await. + let max_response_bytes = self.budget.limits().max_response_bytes; + let sender = { + let mut map = self.server_response_streams.borrow_mut(); + match map.get_mut(&handle_id) { + Some((tx, bytes_written)) => { + let new_total = bytes_written.saturating_add(bytes.len()); + if new_total > max_response_bytes { + let actual = new_total; + // Drop the stream so the body ends rather than + // silently truncating. + map.remove(&handle_id); + return Err(self.budget_error( + BudgetExceeded::ResponseBytes { + limit: max_response_bytes, + actual, + }, + *line, + *column, + )); + } + *bytes_written = new_total; + Some(tx.clone()) + } + None => None, + } + }; match sender { Some(tx) => match tx.send(bytes).await { Ok(()) => Ok((Value::Null, ControlFlow::None)), diff --git a/src/parser/mod.rs b/src/parser/mod.rs index e20b66ae..c51d863a 100644 --- a/src/parser/mod.rs +++ b/src/parser/mod.rs @@ -614,10 +614,11 @@ impl<'a> StmtParser<'a> for Parser<'a> { // keyword; `streaming` is a contextual identifier; `response` is // a keyword. Token::KeywordStart => self.parse_start_streaming_response(), - // `flush ` — a bare-identifier target merges into the token. - Token::Identifier(id) if id == "flush" || id.starts_with("flush ") => { - self.parse_flush_stream() - } + // `flush ` — the target merges into the token + // (`flush out` -> Identifier("flush out")). Only match when an + // operand follows, so a bare `flush` used as an action/variable + // name still parses as an expression statement. + Token::Identifier(id) if id.starts_with("flush ") => self.parse_flush_stream(), Token::Identifier(id) if id.starts_with("send websocket message") => { self.parse_send_websocket_message() } diff --git a/src/transpiler/javascript.rs b/src/transpiler/javascript.rs index 0ba58ad0..65e9fdf0 100644 --- a/src/transpiler/javascript.rs +++ b/src/transpiler/javascript.rs @@ -351,7 +351,22 @@ impl JavaScriptTranspiler { Ok(result) } - Statement::MainLoop { body, .. } => { + Statement::MainLoop { + body, + concurrent, + line, + column, + } => { + // `main loop concurrently:` has cooperative-concurrency semantics + // the serial `while (true)` translation cannot express. Fail + // rather than silently emit a serial loop. + if *concurrent { + return Err(TranspileError { + message: "`main loop concurrently:` is not supported in JavaScript transpilation (its cooperative concurrency semantics require the WFL interpreter).".to_string(), + line: *line, + column: *column, + }); + } // Main loop is essentially a forever loop let mut result = format!("{}while (true) {{\n", self.indent()); self.push_indent(); diff --git a/src/typechecker/mod.rs b/src/typechecker/mod.rs index 04506d59..ae439ffb 100644 --- a/src/typechecker/mod.rs +++ b/src/typechecker/mod.rs @@ -851,7 +851,7 @@ impl TypeChecker { | Type::Error ) { self.type_error( - "HTTP request body must be text".to_string(), + "HTTP request body must be text (numbers and booleans are also accepted and converted)".to_string(), Some(Type::Text), Some(body_type), *_line, @@ -931,7 +931,7 @@ impl TypeChecker { | Type::Error ) { self.type_error( - "HTTP request body must be text".to_string(), + "HTTP request body must be text (numbers and booleans are also accepted and converted)".to_string(), Some(Type::Text), Some(body_type), *_line, @@ -975,17 +975,57 @@ impl TypeChecker { content_type, headers, variable_name, - .. + line: _line, + column: _column, } => { let _ = self.infer_expression_type(request); + // Enforce the clause types (like RespondStatement) so obvious + // mistakes fail at typecheck rather than at runtime. if let Some(status) = status { - let _ = self.infer_expression_type(status); + let status_type = self.infer_expression_type(status); + if !matches!( + status_type, + Type::Number | Type::Unknown | Type::Any | Type::Error + ) { + self.type_error( + "Streaming response status must be a number".to_string(), + Some(Type::Number), + Some(status_type), + *_line, + *_column, + ); + } } if let Some(content_type) = content_type { - let _ = self.infer_expression_type(content_type); + let ct_type = self.infer_expression_type(content_type); + if !matches!( + ct_type, + Type::Text | Type::Unknown | Type::Any | Type::Error + ) { + self.type_error( + "Streaming response content type must be text".to_string(), + Some(Type::Text), + Some(ct_type), + *_line, + *_column, + ); + } } if let Some(headers) = headers { - let _ = self.infer_expression_type(headers); + let headers_type = self.infer_expression_type(headers); + if !matches!( + headers_type, + Type::Map(_, _) | Type::Unknown | Type::Any | Type::Error + ) { + self.type_error( + "Streaming response headers must be a map of header names to values" + .to_string(), + Some(Type::Map(Box::new(Type::Text), Box::new(Type::Any))), + Some(headers_type), + *_line, + *_column, + ); + } } if !variable_name.is_empty() && let Some(symbol) = self.analyzer.get_symbol_mut(variable_name) diff --git a/tests/http_server_streaming_test.rs b/tests/http_server_streaming_test.rs index dbb2f8ef..2d1bc644 100644 --- a/tests/http_server_streaming_test.rs +++ b/tests/http_server_streaming_test.rs @@ -145,6 +145,44 @@ async fn test_streamed_response_lines_and_headers() { let _ = server_handle.join(); } +#[tokio::test] +async fn test_write_after_close_does_not_reach_client() { + // Writing after `close out` is a catchable error and does NOT reach the + // client: the client sees only the bytes written before close. + let port = 8233; + let server_code = format!( + r#" + listen on port {port} as s + wait for request comes in on s as req with timeout 10000 + start streaming response to req with status 200 and content type "text/plain" as out + write line "before" to out + close out + try: + write line "after" to out + catch: + display "write after close correctly errored" + end try + close server s + "# + ); + + let server_handle = start_server_thread(server_code); + tokio::time::sleep(Duration::from_millis(300)).await; + + let response = reqwest::Client::new() + .get(format!("http://127.0.0.1:{port}/x")) + .send() + .await + .expect("request failed"); + let body = response.text().await.unwrap(); + assert_eq!( + body, "before\n", + "writes after close must not reach the client" + ); + + let _ = server_handle.join(); +} + #[tokio::test] async fn test_streamed_response_write_chunk_verbatim() { let port = 8232; diff --git a/tests/transpiler_test.rs b/tests/transpiler_test.rs index de042f21..e608761b 100644 --- a/tests/transpiler_test.rs +++ b/tests/transpiler_test.rs @@ -680,3 +680,22 @@ fn test_describe_and_test_descriptions_are_javascript_string_literals() { assert_contains(&js, "describe(\"quoted \\\"suite\\\"\", function()"); assert_contains(&js, "it(\"quoted \\\"case\\\"\", function()"); } + +#[test] +fn test_main_loop_concurrently_fails_to_transpile() { + // `main loop concurrently:` has no serial JavaScript translation; it must + // error rather than silently emit a serial loop. + let source = "main loop concurrently:\n display \"x\"\nend loop"; + let result = transpile_wfl(source); + assert!( + result.is_err(), + "main loop concurrently should fail to transpile, got: {result:?}" + ); + + // Plain `main loop` still transpiles. + let serial = transpile_wfl("main loop:\n display \"x\"\nend loop"); + assert!( + serial.is_ok(), + "plain main loop should transpile: {serial:?}" + ); +} From 149d1c6d724fc5f596653bfa52901123815a88ae Mon Sep 17 00:00:00 2001 From: Claude Date: Wed, 22 Jul 2026 18:40:00 +0000 Subject: [PATCH 010/132] fix: isolate per-handler run-state under `main loop concurrently:` (P1 #1) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The concurrent loop isolated each handler's environment but interpreter run-state (current_count/in_count_loop, call_depth, call_stack, and the block overload-dup set) still lived on the shared `Interpreter`. Two concurrent handlers interleaving on the single thread could overwrite each other's count-loop/recursion/call-stack bookkeeping across an await — e.g. one count loop reading a `count` set by another handler. Fix without touching the single-threaded Rc core: each handler carries its own `RunState`, and an `IsolatedHandler` poll wrapper swaps that state into the interpreter only for the duration of each poll, swapping it back out the instant poll returns (ready or pending). The run-state fields become poll-local; a suspended handler's state is parked in its own `RunState` where no sibling can touch it. Serial execution is untouched. Red→Green: tests/concurrent_main_loop_test.rs:: test_concurrent_handlers_do_not_share_count_loop_state — two handlers count over disjoint ranges (1..5 vs 100..104), yielding mid-loop. Red (isolation bypassed): /a returned 100-101-102-103-104- (the other handler's range). Green: /a returns 1-2-3-4-5-. Also from review: - Fix stale doc comments on resolve_stream_handle / resolve_server_stream_handle (they require the handle object; a bare Text id is rejected — doc now matches). - Docs: web-servers.md disconnect note uses try/catch (consistent with examples). --- ...-concurrent-handler-run-state-isolation.md | 72 +++++++++++ Docs/04-advanced-features/web-servers.md | 2 +- Docs/development/concurrency-phase-plan.md | 22 ++-- src/interpreter/mod.rs | 115 ++++++++++++++++-- tests/concurrent_main_loop_test.rs | 83 +++++++++++++ 5 files changed, 275 insertions(+), 19 deletions(-) create mode 100644 Dev diary/2026-07-22-concurrent-handler-run-state-isolation.md diff --git a/Dev diary/2026-07-22-concurrent-handler-run-state-isolation.md b/Dev diary/2026-07-22-concurrent-handler-run-state-isolation.md new file mode 100644 index 00000000..663a5667 --- /dev/null +++ b/Dev diary/2026-07-22-concurrent-handler-run-state-isolation.md @@ -0,0 +1,72 @@ +# Dev Diary — 2026-07-22 — Per-handler run-state isolation for `main loop concurrently:` + +## Context + +PR #641 shipped `main loop concurrently:` — opt-in cooperative concurrency for +HTTP request handlers, driven by a `FuturesUnordered` on the single interpreter +thread. Review (maintainer P1 #1, echoed by Copilot) flagged a soundness gap: +the concurrent loop isolated each handler's **environment** (variables), but the +interpreter's **run-state** — the count-loop variable and its flag +(`current_count` / `in_count_loop`), the live recursion depth (`call_depth`), +the diagnostic call stack (`call_stack`), and the current block's overload-dup +set — still lived on the shared `Interpreter` behind `RefCell`/`Cell`. + +Under serial execution that state is never contended. Under +`main loop concurrently:` several handler futures interleave on one thread, so at +every `await` one handler's run-state was visible to — and overwritable by — +whichever sibling was polled next. A handler that yielded *inside a `count` loop* +would resume and read a `count` set by another handler. + +## The bug, concretely + +`count` does not resolve through the environment while a count loop is active; +`try_evaluate_variable_sync` short-circuits on `in_count_loop` and reads +`self.current_count` directly. Both fields are global, so two concurrent count +loops share one `current_count`. A handler counting `1..5` that yields mid-loop +could come back reading `100..104` from a sibling. + +## Fix — a poll-swap wrapper (no `Rc`→`Arc`, no threads) + +The interpreter core stays single-threaded and `Rc`-based (a hard constraint). +Rather than thread a per-handler execution context through every `&self` method, +each handler owns a `RunState` snapshot and an `IsolatedHandler` future wraps the +handler: + +- On **each `poll`**, `swap_run_state` swaps the handler's `RunState` into the + interpreter's live fields (a field-by-field `mem::swap`, its own inverse). +- The inner handler future is polled. +- The instant `poll` returns — `Ready` **or** `Pending` — the state is swapped + back out into the handler's `RunState`. + +So the interpreter's run-state fields become effectively poll-local: exactly one +handler's state is installed at a time, and a suspended handler's state is parked +in its own `RunState` where no sibling can touch it. Each handler starts from +`RunState::fresh(base_call_depth)`. The inner future is already wrapped in +`catch_unwind`, so a panic surfaces as `Ready` and the swap-back still runs, +leaving the scratch fields clean for the next sibling. + +Serial execution is completely untouched — `IsolatedHandler` is used only by +`execute_concurrent_main_loop`. + +## Testing (Red → Green) + +`tests/concurrent_main_loop_test.rs::test_concurrent_handlers_do_not_share_count_loop_state`: +two concurrent handlers each run a `count` loop over a **disjoint** range +(`1..5` vs `100..104`), yielding via `wait for` mid-iteration and then reading +`count`. With isolation each handler observes only its own range. + +- **Red** (isolation bypassed — plain handler pushed to `FuturesUnordered`): + `/a` returned `100-101-102-103-104-`, i.e. it observed the *other* handler's + entire count range. `assertion left == right failed`. +- **Green** (isolation restored): `/a` → `1-2-3-4-5-`, `/b` → + `100-101-102-103-104-`. + +Risk class **R3** (concurrency + lifecycle). The test asserts a concrete wrong +outcome under sharing, not merely "did not crash". + +## Follow-ups still open from the review + +Larger P1 items remain and are tracked in +`Docs/development/concurrency-phase-plan.md`: immediate-500 on pre-respond +failure, browser-disconnect/504 cancellation threaded into `wait for` and +upstream reads, and an absolute total-stream deadline. diff --git a/Docs/04-advanced-features/web-servers.md b/Docs/04-advanced-features/web-servers.md index 1b74959b..1b77706f 100644 --- a/Docs/04-advanced-features/web-servers.md +++ b/Docs/04-advanced-features/web-servers.md @@ -498,7 +498,7 @@ close out **Lifecycle & backpressure:** the body channel is bounded, so a slow client slows your `write` calls (backpressure) instead of buffering without bound. If the client disconnects, hyper drops the response body and your next `write` to -that stream fails with a catchable error — use `try`/`when` to detect it and +that stream fails with a catchable error — use `try`/`catch` to detect it and stop producing (and `close` any upstream you are proxying). **Always `close out`** to finalize the response — that is what signals the end diff --git a/Docs/development/concurrency-phase-plan.md b/Docs/development/concurrency-phase-plan.md index 681ee664..9729b8d6 100644 --- a/Docs/development/concurrency-phase-plan.md +++ b/Docs/development/concurrency-phase-plan.md @@ -70,21 +70,25 @@ HARD RULES: > the language, so the guarantee rests on the wrapper + the `panic = "unwind"` > gate, not a regression test. > -> **Known gap flagged in review (not yet fixed):** the concurrent loop isolates -> the **environment** per handler, but interpreter-level run-state -> (`current_count`/`in_count_loop`, `call_depth`, `call_stack`) still lives on -> the shared `Interpreter`. A handler that yields at an await *inside a `count` -> loop or a deep action call* can have that state overwritten by a concurrent -> sibling. The correct fix is a per-handler execution context; it is a scoped -> refactor and is called out for the maintainer review below. +> **Run-state isolation (review gap — FIXED):** the concurrent loop already +> isolates the **environment** per handler; it now also isolates interpreter +> run-state. Each handler carries its own `RunState` (`current_count`/ +> `in_count_loop`, `call_depth`, `call_stack`, and the block overload-dup set), +> and an `IsolatedHandler` poll wrapper swaps that state into the interpreter +> only for the duration of each `poll`, swapping it back out the instant the poll +> returns (ready *or* pending). While a handler is suspended at an `await`, its +> count-loop/recursion/call-stack bookkeeping is parked in its own `RunState`, so +> a sibling polled next neither sees nor clobbers it. Serial execution is +> untouched (the wrapper is used only by `execute_concurrent_main_loop`). +> Regression: `tests/concurrent_main_loop_test.rs::test_concurrent_handlers_do_not_share_count_loop_state` +> (Red without the swap: `/a` observed `/b`'s entire count range). > > **Also lighter than the full 1b checklist:** request-ID *structured* logging is > not yet added; the eval-core `RefCell`-across-await audit is enforced > mechanically by the crate-wide `#![deny(clippy::await_holding_refcell_ref)]` > backstop rather than a written per-site walkthrough. > -> **This is the maintainer STOP/review point** — please review (especially the -> shared run-state gap) before Phase 2. +> **This is the maintainer STOP/review point** — please review before Phase 2. | 2 | 2a | Structured nursery + join engine | ⬜ Not started | | 2 | 2b | `change shared` critical region | ⬜ Not started | | 3 | 3a | Multi-process workers (if profiling forces) | ⬜ Deferred | diff --git a/src/interpreter/mod.rs b/src/interpreter/mod.rs index 65434406..f2b29068 100644 --- a/src/interpreter/mod.rs +++ b/src/interpreter/mod.rs @@ -787,6 +787,67 @@ impl Drop for CallDepthGuard<'_> { } } +/// A snapshot of the interpreter's per-execution "run state" — the mutable +/// bookkeeping that belongs to a single in-flight execution rather than to the +/// interpreter as a whole: the count-loop variable, recursion depth, the +/// diagnostic call stack, and the current block's overload-dup set. +/// +/// Under serial execution this state lives directly on `Interpreter` and is +/// never contended. Under `main loop concurrently:` several handler futures are +/// interleaved cooperatively on one thread, so at every `await` point one +/// handler's run state must not be visible to (or clobbered by) another. Each +/// handler owns a `RunState` that is swapped into the interpreter only while +/// that handler is actively being polled (see [`IsolatedHandler`]). +#[derive(Default)] +struct RunState { + current_count: Option, + in_count_loop: bool, + call_depth: usize, + call_stack: Vec, + block_overload_dups: Option>>, +} + +impl RunState { + /// A fresh run state for a handler starting from `base_call_depth` (0 for a + /// top-level run; the parent's live depth for an `execute file` child). + fn fresh(base_call_depth: usize) -> Self { + RunState { + call_depth: base_call_depth, + ..RunState::default() + } + } +} + +/// Wraps a handler future so its [`RunState`] is swapped into the interpreter +/// for the duration of each `poll` and swapped back out again the instant the +/// poll returns (ready **or** pending). This makes the interpreter's run-state +/// fields effectively poll-local: while handler A is suspended at an `await`, +/// its count-loop variable, recursion depth, and call stack are parked in A's +/// own `RunState`, so handler B — polled next — neither sees nor corrupts them. +/// +/// `inner` is a boxed handler future (already wrapped in `catch_unwind`); a +/// panic therefore surfaces as `Poll::Ready` and the swap-back still runs, +/// leaving the interpreter's scratch fields restored for the next sibling. +struct IsolatedHandler<'a, T> { + interp: &'a Interpreter, + state: RunState, + inner: std::pin::Pin + 'a>>, +} + +impl<'a, T> std::future::Future for IsolatedHandler<'a, T> { + type Output = T; + + fn poll(self: std::pin::Pin<&mut Self>, cx: &mut std::task::Context<'_>) -> std::task::Poll { + // Every field is `Unpin` (`&`, `RunState`, and `Pin>`), so the + // wrapper itself is `Unpin` and `get_mut` is sound. + let this = self.get_mut(); + this.interp.swap_run_state(&mut this.state); + let result = this.inner.as_mut().poll(cx); + this.interp.swap_run_state(&mut this.state); + result + } +} + /// RAII guard that ensures module loading context is restored on scope exit. /// Automatically pops loading_stack and restores current_source_file when dropped. struct ModuleLoadGuard<'a> { @@ -3666,9 +3727,9 @@ impl Interpreter { } /// Resolve a `wait for next chunk|line from ` operand to a stream - /// handle id. Accepts either the streaming-response object bound by - /// `stream response as ` (reads its internal `_stream` id) or a bare - /// text handle id. + /// handle id. Requires the streaming-response object bound by + /// `stream response as ` (reads its internal `_stream` id); a bare text + /// id is rejected so an arbitrary string can't be aimed at a stream handle. async fn resolve_stream_handle( &self, source: &Expression, @@ -3700,8 +3761,9 @@ impl Interpreter { } /// Resolve a `write line|chunk`/`flush` operand to a server response-stream - /// handle id. Accepts the object bound by `start streaming response as ...` - /// (reads its internal `_server_stream` id) or a bare text handle id. + /// handle id. Requires the object bound by `start streaming response as ...` + /// (reads its internal `_server_stream` id); a bare text id is rejected so an + /// arbitrary string can't be aimed at a server response stream. async fn resolve_server_stream_handle( &self, target: &Expression, @@ -3732,12 +3794,41 @@ impl Interpreter { } } + /// Swap the interpreter's per-execution run-state fields with `state`. + /// + /// Used by [`IsolatedHandler`] to make the run state poll-local under + /// concurrent execution: the interpreter's live fields and the parked + /// snapshot trade places, so exactly one handler's run state is installed at + /// a time. A plain field-by-field `mem::swap`, so it is its own inverse. + fn swap_run_state(&self, state: &mut RunState) { + std::mem::swap( + &mut *self.current_count.borrow_mut(), + &mut state.current_count, + ); + std::mem::swap( + &mut *self.in_count_loop.borrow_mut(), + &mut state.in_count_loop, + ); + let depth = self.call_depth.replace(state.call_depth); + state.call_depth = depth; + std::mem::swap(&mut *self.call_stack.borrow_mut(), &mut state.call_stack); + std::mem::swap( + &mut *self.current_block_overload_dups.borrow_mut(), + &mut state.block_overload_dups, + ); + } + /// Execute a `main loop concurrently:` body. Keeps up to /// `CONCURRENT_HANDLER_LIMIT` iterations of `body` in flight at once, each in /// its own isolated child scope, driven cooperatively on this single thread /// (no threads, no `Send`/`Arc` across the interpreter core). A handler that /// errors or panics is contained — its request is resolved with 500 by the /// response-completion drop guard — and its siblings keep running. + /// + /// Each handler also carries an isolated [`RunState`] (count-loop variable, + /// recursion depth, call stack, block overload set) via [`IsolatedHandler`], + /// so one handler's loop/recursion bookkeeping can never leak into another + /// across an `await`. async fn execute_concurrent_main_loop( &self, body: &[Statement], @@ -3754,12 +3845,18 @@ impl Interpreter { self.check_time()?; // Refill to the concurrency cap. Each iteration gets a fresh isolated - // scope so concurrent requests never clobber each other's variables. + // scope so concurrent requests never clobber each other's variables, + // and a fresh `RunState` (wrapped by `IsolatedHandler`) so their + // count-loop/recursion/call-stack bookkeeping stays poll-local. while futs.len() < cap { let scope = Environment::new_child_env(env); - futs.push( - std::panic::AssertUnwindSafe(self.execute_block(body, scope)).catch_unwind(), - ); + let handler = + std::panic::AssertUnwindSafe(self.execute_block(body, scope)).catch_unwind(); + futs.push(IsolatedHandler { + interp: self, + state: RunState::fresh(self.base_call_depth), + inner: Box::pin(handler), + }); } // With cap >= 1 the set is never empty, so `next()` never returns a diff --git a/tests/concurrent_main_loop_test.rs b/tests/concurrent_main_loop_test.rs index 30560cec..3be3cfdd 100644 --- a/tests/concurrent_main_loop_test.rs +++ b/tests/concurrent_main_loop_test.rs @@ -178,6 +178,89 @@ async fn test_concurrent_handler_error_does_not_kill_server() { shutdown(port, server).await; } +#[tokio::test] +async fn test_concurrent_handlers_do_not_share_count_loop_state() { + // Per-handler run-state isolation (P1 #1). Two concurrent handlers each run + // a `count` loop that yields (via `wait for`) mid-iteration and then reads + // `count`. The interpreter's count-loop state (`current_count`, + // `in_count_loop`) is a single shared field; without per-poll isolation, one + // handler's `count` bleeds into the other across the yield. + // + // The two ranges are disjoint (1..5 vs 100..104), so any cross-contamination + // is unmistakable: with isolation each handler observes only its own range. + let port = 8344; + let code = format!( + r#" + listen on port {port} as srv + main loop concurrently: + wait for request comes in on srv as req with timeout 20000 + store p as req["path"] + check if p is equal to "/shutdown": + respond to req with "bye" + close server srv + break + otherwise: + check if p is equal to "/a": + store seen as "" + count from 1 to 5: + wait for 80 milliseconds + change seen to seen with count with "-" + end count + respond to req with seen + otherwise: + store seen as "" + count from 100 to 104: + wait for 80 milliseconds + change seen to seen with count with "-" + end count + respond to req with seen + end check + end check + end loop + "# + ); + let server = start_server_thread(code); + tokio::time::sleep(Duration::from_millis(300)).await; + + let a_url = format!("http://127.0.0.1:{port}/a"); + let b_url = format!("http://127.0.0.1:{port}/b"); + // Fire both at once so their count loops interleave on the single thread. + let a = tokio::spawn(async move { + reqwest::Client::new() + .get(&a_url) + .send() + .await + .unwrap() + .text() + .await + .unwrap() + }); + let b = tokio::spawn(async move { + reqwest::Client::new() + .get(&b_url) + .send() + .await + .unwrap() + .text() + .await + .unwrap() + }); + + let a_body = a.await.expect("/a task panicked"); + let b_body = b.await.expect("/b task panicked"); + + assert_eq!( + a_body, "1-2-3-4-5-", + "/a handler observed a count from outside its own loop (shared count-loop state)" + ); + assert_eq!( + b_body, "100-101-102-103-104-", + "/b handler observed a count from outside its own loop (shared count-loop state)" + ); + + shutdown(port, server).await; +} + #[tokio::test] async fn test_serial_slow_handler_blocks_next() { let port = 8343; From c45c27674716e1866f743f0da728cabb82dc83b5 Mon Sep 17 00:00:00 2001 From: Claude Date: Wed, 22 Jul 2026 18:49:31 +0000 Subject: [PATCH 011/132] fix: auto-close server response streams on handler exit (spec item 5; Devin/Copilot) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit A streamed server response (`start streaming response`) parked its body-channel sender in the long-lived `server_response_streams` table. The sender was dropped only by an explicit `close out` or the write-after-disconnect path, so a handler that ended without `close out` (normal return, caught error, break) left the sender in the table forever: the client's chunked body never terminated (client hangs) and the table leaked one dead entry per streamed request. This contradicted the shipped docs/design and the streaming spec's item-5 lifecycle guarantee ("all streams close on every exit path"). Fix: each handler tracks the respstream* ids it opens in per-handler run-state (`open_response_streams`, part of the RunState swapped per poll), and closes them when it ends on ANY path: - concurrent handlers: IsolatedHandler's Drop (covers return/error/panic/cancel); - serial main loop: drain after each iteration (normal and error paths); - top level: drain at program exit. close_response_streams is idempotent, so explicit `close out` (still preferred, finalizes promptly) and auto-close compose safely. Red→Green: tests/http_server_streaming_test.rs:: test_stream_auto_closes_when_handler_ends_without_close — handler omits `close out`; client reads body under a 5s timeout. Red (drain disabled): body never finishes (Elapsed). Green: reads "hello\n". Also from review: - typechecker: reword the HTTP-body type error to list accepted types (text, number, boolean) instead of the self-contradictory "must be text (…)". - docs: web-servers.md, response-streaming-design.md, and the server-streaming dev diary now describe the shipped close-on-exit behavior; new dev diary entry. --- .../2026-07-22-server-response-streaming.md | 7 +- ...07-22-stream-auto-close-on-handler-exit.md | 66 ++++++++++++++++ Docs/04-advanced-features/web-servers.md | 14 ++-- Docs/development/response-streaming-design.md | 12 ++- src/interpreter/mod.rs | 75 ++++++++++++++++++- src/typechecker/mod.rs | 4 +- tests/http_server_streaming_test.rs | 52 +++++++++++++ 7 files changed, 218 insertions(+), 12 deletions(-) create mode 100644 Dev diary/2026-07-22-stream-auto-close-on-handler-exit.md diff --git a/Dev diary/2026-07-22-server-response-streaming.md b/Dev diary/2026-07-22-server-response-streaming.md index b4574683..f66621f5 100644 --- a/Dev diary/2026-07-22-server-response-streaming.md +++ b/Dev diary/2026-07-22-server-response-streaming.md @@ -42,8 +42,11 @@ upstream to the browser line-by-line without buffering either side. `write` (it awaits a free slot). When the client disconnects, hyper drops the body, dropping the receiver; the handler's next `write` then fails with a catchable error — that is how a browser disconnect propagates to the handler - (which can then `close` the upstream it is proxying). `close` (or handler exit) - drops the sender, ending the response. + (which can then `close` the upstream it is proxying). An explicit `close out` + drops the sender, ending the response; and as a safety net the stream is + auto-closed when the handler ends on any path (see the follow-up entry + `2026-07-22-stream-auto-close-on-handler-exit.md`), so a forgotten `close` + never hangs the client. - **`start` is a keyword, `streaming`/`flush`/`line`/`chunk` are identifiers.** `start streaming response` dispatches on `Token::KeywordStart`; `flush ` and `write line|chunk to ` handle the lexer's identifier-merging diff --git a/Dev diary/2026-07-22-stream-auto-close-on-handler-exit.md b/Dev diary/2026-07-22-stream-auto-close-on-handler-exit.md new file mode 100644 index 00000000..e7f171a2 --- /dev/null +++ b/Dev diary/2026-07-22-stream-auto-close-on-handler-exit.md @@ -0,0 +1,66 @@ +# Dev Diary — 2026-07-22 — Auto-close server response streams on handler exit + +## Context + +PR #641's streamed server responses (`start streaming response` / `write` / +`flush` / `close out`) parked the body-channel **sender** in a long-lived +interpreter table, `server_response_streams`, keyed by an opaque `respstream*` +id. The transport turns the matching receiver into the chunked response body via +`Body::wrap_stream`, which only ends once **every** sender is dropped. + +Review (Devin 🐛, Copilot, CodeRabbit) flagged the consequence: the sender was +dropped in only two places — an explicit `close out` and the write-after- +disconnect path. A handler that started a stream and ended **without** `close out` +(normal return, a caught error, a `break`) left its sender in the table forever: + +- the client's chunked body was never terminated → **the client hangs**, and +- the table grew one dead entry per streamed request → **a memory leak**. + +This also contradicted the shipped docs and design doc, which promised +"close-on-exit," and the original streaming spec's item 5 lifecycle guarantee: +*all streams close on every exit path.* + +## Fix — tie each stream's lifetime to its handler + +Each handler now tracks the `respstream*` ids it opens and closes them when it +ends, on **every** path: + +- New per-handler field `open_response_streams` lives in the interpreter and is + part of the `RunState` swapped in/out per poll (the same poll-local mechanism + that isolates `count`/recursion state under `main loop concurrently:`), so each + handler tracks only its own streams even while interleaved. +- `start streaming response` pushes the new id; an explicit `close out` removes it + (keeping the list bounded to genuinely-open streams). +- `close_response_streams(ids)` removes each id from `server_response_streams`, + dropping its sender and ending the body. It is idempotent — an id already closed + is a no-op — so closing a handler's whole opened-list on exit is always safe. + +Close-on-exit is wired at every boundary: + +- **Concurrent handlers:** `IsolatedHandler`'s `Drop` closes the handler's + `state.open_response_streams`. Drop runs whether the handler returned, errored, + panicked (contained by `catch_unwind`), or was cancelled as the loop tore down. +- **Serial `main loop`:** each iteration drains and closes after `execute_block`, + on the normal *and* error paths. +- **Top level:** `interpret_inner` drains at program exit for streams opened + outside any loop, and clears the tracking on run entry (REPL reuse). + +## Testing (Red → Green) + +`tests/http_server_streaming_test.rs::test_stream_auto_closes_when_handler_ends_without_close`: +a handler starts a stream, writes one line, and ends **without** `close out`. The +client reads the body under a 5s `tokio::time::timeout`. + +- **Red** (auto-close drain disabled): the body never finishes — + `timeout ... Elapsed(())`; the read hangs exactly as a real client would. +- **Green** (auto-close restored): body reads back `"hello\n"` and completes. + +Risk class **R3** (lifecycle/streaming). The negative outcome (a hang) is turned +into a deterministic failure by the timeout rather than a stuck test. + +## Docs + +`web-servers.md`, `response-streaming-design.md`, and the server-streaming dev +diary updated to describe the shipped close-on-exit behavior. Explicit `close out` +is still recommended to finalize promptly (and free the connection sooner); +auto-close is the safety net, not a substitute. diff --git a/Docs/04-advanced-features/web-servers.md b/Docs/04-advanced-features/web-servers.md index 1b77706f..cd00f6a6 100644 --- a/Docs/04-advanced-features/web-servers.md +++ b/Docs/04-advanced-features/web-servers.md @@ -501,11 +501,15 @@ the client disconnects, hyper drops the response body and your next `write` to that stream fails with a catchable error — use `try`/`catch` to detect it and stop producing (and `close` any upstream you are proxying). -**Always `close out`** to finalize the response — that is what signals the end -of the body to the client. A handler that starts a stream and returns without -`close`ing it leaves the response body open (the client keeps waiting) until the -program exits. Put the `close` on every path, e.g. in a `finally:` block if the -handler can error partway through. +**Prefer an explicit `close out`** to finalize the response promptly — that is +what signals the end of the body to the client, and doing it as soon as you are +done frees the connection without waiting. As a safety net, the stream is also +**closed automatically when the handler ends on any path** (normal return, a +caught error, or a panic contained by `main loop concurrently:`), so a handler +that forgets `close out` still finalizes the client's body rather than leaving +it hanging. For long-lived handlers, still `close` as soon as you are finished — +and put it in a `finally:` block if the handler can error partway through — so +the client is not left waiting until the handler happens to return. **Proxying an upstream to the browser** — combine with the outbound streaming client ([Interoperability → Streaming a response diff --git a/Docs/development/response-streaming-design.md b/Docs/development/response-streaming-design.md index b6e20ad6..ab53d645 100644 --- a/Docs/development/response-streaming-design.md +++ b/Docs/development/response-streaming-design.md @@ -133,8 +133,16 @@ close out await points (the yield-cliff caveat from the concurrency plan). - Backpressure: bounded `mpsc` — a slow browser slows the handler's `write`. - Disconnect → upstream cancel: `write` error path (above). -- Close-on-exit: dropping `ServerStreamHandle` (handler end, error, teardown) - drops `tx`, ending the response. +- Close-on-exit (shipped): each handler tracks the `respstream*` ids it opened in + its per-handler run-state (`open_response_streams`, part of the `RunState` + swapped in/out per poll under `main loop concurrently:`). When the handler ends + on **any** path — normal return, caught error, or a panic contained by + `catch_unwind` — those ids are removed from `server_response_streams`, dropping + their `tx` and ending the response body. Concurrent handlers close via + `IsolatedHandler`'s `Drop`; the serial `main loop` drains per iteration; a + top-level stream closes at program exit. So a handler that forgets `close out` + never leaves the client hanging, and the table cannot leak dead senders. + Explicit `close out` is still preferred to finalize promptly. ### Tests (write first) diff --git a/src/interpreter/mod.rs b/src/interpreter/mod.rs index f2b29068..358b6681 100644 --- a/src/interpreter/mod.rs +++ b/src/interpreter/mod.rs @@ -805,6 +805,11 @@ struct RunState { call_depth: usize, call_stack: Vec, block_overload_dups: Option>>, + /// Server response streams opened by this handler and not yet explicitly + /// closed. Closed automatically when the handler ends on any path (see + /// `IsolatedHandler`'s `Drop` and the serial main loop's per-iteration + /// drain), so a handler that forgets `close out` never hangs the client. + open_response_streams: Vec, } impl RunState { @@ -848,6 +853,18 @@ impl<'a, T> std::future::Future for IsolatedHandler<'a, T> { } } +impl<'a, T> Drop for IsolatedHandler<'a, T> { + fn drop(&mut self) { + // The handler is finished (normal return, error, panic contained by + // `catch_unwind`, or cancellation as the loop tears down). After the + // final poll's swap-out, `state.open_response_streams` holds any server + // response streams it opened but never closed. Close them now so the + // client's body is finalized on every exit path — never left hanging. + self.interp + .close_response_streams(&self.state.open_response_streams); + } +} + /// RAII guard that ensures module loading context is restored on scope exit. /// Automatically pops loading_stack and restores current_source_file when dropped. struct ModuleLoadGuard<'a> { @@ -1220,6 +1237,13 @@ pub struct Interpreter { /// ceiling. `write line|chunk`/`flush` push to it, `close` drops it. server_response_streams: RefCell>, next_response_stream_id: std::cell::Cell, + /// Handle ids of server response streams opened by the currently executing + /// handler and not yet explicitly closed. Part of the per-handler `RunState` + /// (swapped in/out per poll under `main loop concurrently:`) so each handler + /// tracks only its own streams; drained and closed when the handler ends on + /// any path so the client's body is always finalized (see + /// `close_response_streams`). + open_response_streams: RefCell>, #[allow(dead_code)] // Used for future security features config: Arc, // Configuration for security and other settings current_source_file: RefCell>, // Currently executing source file (for path resolution) @@ -3270,6 +3294,7 @@ impl Interpreter { ws_connections: Arc::new(std::sync::Mutex::new(HashMap::new())), // Live WebSocket connections pending_responses: RefCell::new(HashMap::new()), // Initialize empty pending responses map server_response_streams: RefCell::new(HashMap::new()), + open_response_streams: RefCell::new(Vec::new()), next_response_stream_id: std::cell::Cell::new(1), config, current_source_file: RefCell::new(None), // No source file initially @@ -3816,6 +3841,32 @@ impl Interpreter { &mut *self.current_block_overload_dups.borrow_mut(), &mut state.block_overload_dups, ); + std::mem::swap( + &mut *self.open_response_streams.borrow_mut(), + &mut state.open_response_streams, + ); + } + + /// Close (drop the sender for) each server response stream whose handle id is + /// in `ids`, ending its body so the client stops waiting. Idempotent — an id + /// already closed by an explicit `close out` (or a disconnect) is a no-op — + /// so it is safe to call over a handler's full opened-stream list on exit. + fn close_response_streams(&self, ids: &[String]) { + if ids.is_empty() { + return; + } + let mut map = self.server_response_streams.borrow_mut(); + for id in ids { + map.remove(id); + } + } + + /// Drain and close every server response stream the current (serial) handler + /// left open. Called at the end of each serial `main loop` iteration and at + /// program exit, mirroring the concurrent path's per-handler `Drop`. + fn close_open_response_streams(&self) { + let ids = std::mem::take(&mut *self.open_response_streams.borrow_mut()); + self.close_response_streams(&ids); } /// Execute a `main loop concurrently:` body. Keeps up to @@ -4090,6 +4141,9 @@ impl Interpreter { // the invariant holds regardless of how the previous run ended. *self.in_count_loop.borrow_mut() = false; *self.current_count.borrow_mut() = None; + // A prior run that ended while a stream was open (REPL reuse) must not + // leave dangling ids tracked against this run. + self.open_response_streams.borrow_mut().clear(); // Reset to the inherited base depth (0 for a top-level run/REPL; the // parent's live depth for an `execute file` child) so recursion // accounting spans the execute-file boundary instead of granting the @@ -4312,6 +4366,12 @@ impl Interpreter { } } + // Close any server response streams opened directly at top level (outside + // a `main loop`, which already closes per-iteration/per-handler) so a + // script that starts a stream and exits without `close` still finalizes + // the client's body rather than leaving it hanging until process death. + self.close_open_response_streams(); + if errors.is_empty() { let main_func_opt = { match self.global_env.borrow().get("main") { @@ -5127,7 +5187,12 @@ impl Interpreter { // OPTIMIZATION: Recycle environment if possible let loop_env = self.get_recycled_env(loop_env_recycle.take(), &env); - let result = self.execute_block(body, Rc::clone(&loop_env)).await?; + let result = self.execute_block(body, Rc::clone(&loop_env)).await; + // Close any server response streams this iteration left open, + // on every path (including the error path below), so a handler + // that forgets `close out` never leaves the client hanging. + self.close_open_response_streams(); + let result = result?; _last_value = result.0; // Save environment for potential recycling in next iteration @@ -5440,6 +5505,9 @@ impl Interpreter { } else if let Some(id) = server_id { // Dropping the sender ends the response body stream. self.server_response_streams.borrow_mut().remove(&id); + // Drop it from the handler's auto-close tracking so + // the list stays bounded to actually-open streams. + self.open_response_streams.borrow_mut().retain(|s| s != &id); Ok((Value::Null, ControlFlow::None)) } else { Err(RuntimeError::new( @@ -8946,6 +9014,11 @@ impl Interpreter { self.server_response_streams .borrow_mut() .insert(handle_id.clone(), (tx, 0)); + // Track it against the current handler so it is auto-closed if + // the handler ends without an explicit `close out`. + self.open_response_streams + .borrow_mut() + .push(handle_id.clone()); let mut stream_map = HashMap::new(); stream_map.insert("_server_stream".to_string(), Value::Text(handle_id.into())); diff --git a/src/typechecker/mod.rs b/src/typechecker/mod.rs index ae439ffb..013e7067 100644 --- a/src/typechecker/mod.rs +++ b/src/typechecker/mod.rs @@ -851,7 +851,7 @@ impl TypeChecker { | Type::Error ) { self.type_error( - "HTTP request body must be text (numbers and booleans are also accepted and converted)".to_string(), + "HTTP request body must be text, a number, or a boolean (numbers and booleans are converted to text)".to_string(), Some(Type::Text), Some(body_type), *_line, @@ -931,7 +931,7 @@ impl TypeChecker { | Type::Error ) { self.type_error( - "HTTP request body must be text (numbers and booleans are also accepted and converted)".to_string(), + "HTTP request body must be text, a number, or a boolean (numbers and booleans are converted to text)".to_string(), Some(Type::Text), Some(body_type), *_line, diff --git a/tests/http_server_streaming_test.rs b/tests/http_server_streaming_test.rs index 2d1bc644..50fde802 100644 --- a/tests/http_server_streaming_test.rs +++ b/tests/http_server_streaming_test.rs @@ -183,6 +183,58 @@ async fn test_write_after_close_does_not_reach_client() { let _ = server_handle.join(); } +#[tokio::test] +async fn test_stream_auto_closes_when_handler_ends_without_close() { + // Lifecycle guarantee (spec item 5): a handler that starts a stream and ends + // WITHOUT `close out` must still finalize the client's body on the way out. + // Otherwise the sender lingers in the interpreter's stream table, the body is + // never terminated, and the client hangs forever (and the table leaks). + let port = 8234; + let server_code = format!( + r#" + listen on port {port} as s + main loop: + wait for request comes in on s as req with timeout 20000 + store p as req["path"] + check if p is equal to "/shutdown": + respond to req with "bye" + close server s + break + otherwise: + start streaming response to req with status 200 and content type "text/plain" as out + write line "hello" to out + end check + end loop + "# + ); + + let server_handle = start_server_thread(server_code); + tokio::time::sleep(Duration::from_millis(300)).await; + + let response = reqwest::Client::new() + .get(format!("http://127.0.0.1:{port}/x")) + .send() + .await + .expect("request failed"); + assert_eq!(response.status().as_u16(), 200); + + // Reading the body must COMPLETE (the stream was auto-closed). If the handler + // leaked the stream, this read hangs — the timeout turns that into a failure + // instead of a stuck test. + let body = tokio::time::timeout(Duration::from_secs(5), response.text()) + .await + .expect("body did not finish — stream was not auto-closed when the handler ended") + .expect("failed to read body"); + assert_eq!(body, "hello\n"); + + // Stop the server. + let _ = reqwest::Client::new() + .get(format!("http://127.0.0.1:{port}/shutdown")) + .send() + .await; + let _ = server_handle.join(); +} + #[tokio::test] async fn test_streamed_response_write_chunk_verbatim() { let port = 8232; From 04f798354a6f5d3e77a867eb7f75f4302edaf6d3 Mon Sep 17 00:00:00 2001 From: Claude Date: Wed, 22 Jul 2026 19:04:58 +0000 Subject: [PATCH 012/132] fix: 500 immediately when a handler ends without responding (P1 #3) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The ResponseCompletion drop guard only arms once a handler reaches a respond/start-streaming statement. A handler that dequeues a request (`wait for request comes in`) and then ends before responding — runtime error, break, or a plain return — left the sender parked in pending_responses, so the client waited out the request timeout instead of getting a prompt 500. Fix: each handler tracks the request ids it dequeued but hasn't answered in per-handler run-state (`open_pending_requests`, part of the RunState swapped per poll). `respond`/`start streaming response` remove the id from the map and disarm tracking; on handler exit (any path) `fail_unanswered_requests` 500s any id still in pending_responses. Fully synchronous (try_lock + oneshot send) so it runs from IsolatedHandler's Drop; also wired into the serial main loop's per-iteration drain and program exit. Idempotent — a responded request is gone from the map, so the sweep skips it. Red→Green: tests/concurrent_main_loop_test.rs:: test_handler_that_never_responds_gets_immediate_500 — /drop dequeues and returns without responding. Red (sweep disabled): the request hangs past the 120s test timeout. Green: 500 arrives in <1s and the server keeps serving. Also from review: - The response-byte-ceiling and disconnect paths untrack the stream id from open_response_streams so a handler that catches the error keeps no stale ids. - typechecker: HTTP/response header type hints widened to map[text, any]. --- ...-22-immediate-500-on-unanswered-request.md | 60 ++++++++++ src/interpreter/mod.rs | 105 ++++++++++++++++-- src/typechecker/mod.rs | 6 +- tests/concurrent_main_loop_test.rs | 61 ++++++++++ 4 files changed, 220 insertions(+), 12 deletions(-) create mode 100644 Dev diary/2026-07-22-immediate-500-on-unanswered-request.md diff --git a/Dev diary/2026-07-22-immediate-500-on-unanswered-request.md b/Dev diary/2026-07-22-immediate-500-on-unanswered-request.md new file mode 100644 index 00000000..eacfbe28 --- /dev/null +++ b/Dev diary/2026-07-22-immediate-500-on-unanswered-request.md @@ -0,0 +1,60 @@ +# Dev Diary — 2026-07-22 — Immediate 500 when a handler ends without responding + +## Context + +PR #641 review (maintainer P1 #3, echoed by CodeRabbit) flagged that the +`ResponseCompletion` drop guard does **not** cover every pre-response path. The +guard is armed only once a handler *reaches* a `respond` / `start streaming +response` statement (it takes the request's sender out of `pending_responses` +into the guard). A handler that dequeues a request with `wait for request comes +in` and then ends **before** responding — a runtime error, a `break`, or simply +returning without `respond` — leaves the sender parked in `pending_responses`. +The client then waits out the request timeout instead of getting a prompt 500. + +## Fix — arm the fallback at dequeue, disarm at respond + +Each handler now tracks the request ids it dequeued but has not answered, in +per-handler run-state (`open_pending_requests`, part of the `RunState` swapped +per poll — the same mechanism that isolates `count`/recursion state and tracks +open response streams). On exit, any still-unanswered request is answered 500: + +- `wait for request comes in` pushes the request id. +- `respond` / `start streaming response` remove the id from the map + (`pending_responses.remove`) *and* disarm the tracking. Because a responded + request is gone from the map, the exit-time sweep is naturally idempotent — it + only 500s ids still present. +- On handler exit (any path) the tracked ids are swept: + `fail_unanswered_requests` removes each from `pending_responses`, and if the + sender is still live (`try_lock` + `oneshot::send`, fully synchronous) sends a + 500. Wired at every boundary, mirroring the stream close-on-exit: + `IsolatedHandler`'s `Drop` (concurrent), the serial `main loop`'s per-iteration + drain, and program exit. + +The sweep is synchronous so it runs from `Drop`. The sender's mutex is only held +during `respond`, which a finished handler is no longer inside, so `try_lock` +succeeds; the transport request timeout remains the backstop for the impossible +case where it does not. + +## Testing (Red → Green) + +`tests/concurrent_main_loop_test.rs::test_handler_that_never_responds_gets_immediate_500`: +a `main loop concurrently:` server whose `/drop` path dequeues the request and +ends without responding. The client must get a **prompt 500**, and the server +must keep serving (`/ok` still works). + +- **Red** (500-on-exit disabled): the `/drop` request hangs — the client waits + out the transport timeout, exceeding even the 120s test harness timeout, i.e. + exactly the "client left waiting" symptom. +- **Green**: `/drop` returns 500 in well under a second; `/ok` still returns + `ok`. + +Risk class **R3** (lifecycle). The test asserts both the status *and* that it +arrives before the timeout, so a regression to "eventually times out" fails. + +## Also in this change (from review) + +- The response-byte-ceiling and client-disconnect paths now also untrack the + stream id from `open_response_streams`, so a handler that catches the error and + continues keeps no stale ids. +- Typechecker: HTTP/response header type hints widened to `map[text, any]` + (values may be text, numbers, or booleans, converted to text). diff --git a/src/interpreter/mod.rs b/src/interpreter/mod.rs index 358b6681..9b8cecee 100644 --- a/src/interpreter/mod.rs +++ b/src/interpreter/mod.rs @@ -810,6 +810,11 @@ struct RunState { /// `IsolatedHandler`'s `Drop` and the serial main loop's per-iteration /// drain), so a handler that forgets `close out` never hangs the client. open_response_streams: Vec, + /// Request ids this handler dequeued (`wait for request comes in`) and has + /// not yet answered. If the handler ends on any path without responding, each + /// is answered 500 immediately instead of leaving the client to wait out the + /// request timeout (see `fail_unanswered_requests`). + open_pending_requests: Vec, } impl RunState { @@ -857,11 +862,14 @@ impl<'a, T> Drop for IsolatedHandler<'a, T> { fn drop(&mut self) { // The handler is finished (normal return, error, panic contained by // `catch_unwind`, or cancellation as the loop tears down). After the - // final poll's swap-out, `state.open_response_streams` holds any server - // response streams it opened but never closed. Close them now so the - // client's body is finalized on every exit path — never left hanging. + // final poll's swap-out, `state` holds any streams it opened but never + // closed and any requests it dequeued but never answered. Close the + // streams (finalizing the client's body) and 500 the unanswered requests, + // so every exit path resolves the client instead of leaving it hanging. self.interp .close_response_streams(&self.state.open_response_streams); + self.interp + .fail_unanswered_requests(&self.state.open_pending_requests); } } @@ -1244,6 +1252,11 @@ pub struct Interpreter { /// any path so the client's body is always finalized (see /// `close_response_streams`). open_response_streams: RefCell>, + /// Request ids the currently executing handler dequeued but has not yet + /// answered. Part of the per-handler `RunState` (swapped per poll) so each + /// handler tracks only its own requests; any still unanswered when the handler + /// ends are answered 500 immediately (see `fail_unanswered_requests`). + open_pending_requests: RefCell>, #[allow(dead_code)] // Used for future security features config: Arc, // Configuration for security and other settings current_source_file: RefCell>, // Currently executing source file (for path resolution) @@ -3295,6 +3308,7 @@ impl Interpreter { pending_responses: RefCell::new(HashMap::new()), // Initialize empty pending responses map server_response_streams: RefCell::new(HashMap::new()), open_response_streams: RefCell::new(Vec::new()), + open_pending_requests: RefCell::new(Vec::new()), next_response_stream_id: std::cell::Cell::new(1), config, current_source_file: RefCell::new(None), // No source file initially @@ -3845,6 +3859,10 @@ impl Interpreter { &mut *self.open_response_streams.borrow_mut(), &mut state.open_response_streams, ); + std::mem::swap( + &mut *self.open_pending_requests.borrow_mut(), + &mut state.open_pending_requests, + ); } /// Close (drop the sender for) each server response stream whose handle id is @@ -3869,6 +3887,44 @@ impl Interpreter { self.close_response_streams(&ids); } + /// Answer 500 for each request id in `ids` that is still unanswered (its + /// sender is still parked in `pending_responses`). A request the handler + /// already answered is gone from the map, so its `remove` is a no-op — + /// idempotent and safe to call over a handler's full dequeued list on exit. + /// + /// Fully synchronous (a non-blocking `try_lock` plus a `oneshot` send) so it + /// runs from `IsolatedHandler`'s `Drop`. The sender's mutex is only ever held + /// during `respond`, which the finished handler is no longer inside, so the + /// `try_lock` succeeds; if it somehow does not, the transport's request + /// timeout remains the backstop. + fn fail_unanswered_requests(&self, ids: &[String]) { + if ids.is_empty() { + return; + } + let mut pending = self.pending_responses.borrow_mut(); + for id in ids { + if let Some(entry) = pending.remove(id) + && let Ok(mut guard) = entry.sender.try_lock() + && let Some(sender) = guard.take() + { + let _ = sender.send(HandlerReply::Buffered(WflHttpResponse { + content: b"Internal Server Error\n".to_vec(), + status: 500, + content_type: "text/plain; charset=utf-8".to_string(), + headers: HashMap::new(), + })); + } + } + } + + /// Drain and 500 any requests the current (serial) handler dequeued but never + /// answered. Called at the end of each serial `main loop` iteration and at + /// program exit, mirroring the concurrent path's per-handler `Drop`. + fn fail_open_pending_requests(&self) { + let ids = std::mem::take(&mut *self.open_pending_requests.borrow_mut()); + self.fail_unanswered_requests(&ids); + } + /// Execute a `main loop concurrently:` body. Keeps up to /// `CONCURRENT_HANDLER_LIMIT` iterations of `body` in flight at once, each in /// its own isolated child scope, driven cooperatively on this single thread @@ -4141,9 +4197,10 @@ impl Interpreter { // the invariant holds regardless of how the previous run ended. *self.in_count_loop.borrow_mut() = false; *self.current_count.borrow_mut() = None; - // A prior run that ended while a stream was open (REPL reuse) must not - // leave dangling ids tracked against this run. + // A prior run that ended while a stream was open or a request was + // unanswered (REPL reuse) must not leave dangling ids tracked here. self.open_response_streams.borrow_mut().clear(); + self.open_pending_requests.borrow_mut().clear(); // Reset to the inherited base depth (0 for a top-level run/REPL; the // parent's live depth for an `execute file` child) so recursion // accounting spans the execute-file boundary instead of granting the @@ -4370,7 +4427,9 @@ impl Interpreter { // a `main loop`, which already closes per-iteration/per-handler) so a // script that starts a stream and exits without `close` still finalizes // the client's body rather than leaving it hanging until process death. + // Likewise 500 any top-level request dequeued but never answered. self.close_open_response_streams(); + self.fail_open_pending_requests(); if errors.is_empty() { let main_func_opt = { @@ -5188,10 +5247,13 @@ impl Interpreter { // OPTIMIZATION: Recycle environment if possible let loop_env = self.get_recycled_env(loop_env_recycle.take(), &env); let result = self.execute_block(body, Rc::clone(&loop_env)).await; - // Close any server response streams this iteration left open, - // on every path (including the error path below), so a handler - // that forgets `close out` never leaves the client hanging. + // On every path (including the error path below): close any + // server response streams this iteration left open, and 500 + // any request it dequeued but never answered — so a handler + // that forgets `close out`/`respond` never leaves the client + // hanging or waiting out the request timeout. self.close_open_response_streams(); + self.fail_open_pending_requests(); let result = result?; _last_value = result.0; @@ -8568,6 +8630,12 @@ impl Interpreter { }, ); } + // Track it against the current handler so it is answered 500 if + // the handler ends without responding (rather than the client + // waiting out the request timeout). + self.open_pending_requests + .borrow_mut() + .push(request.id.clone()); Ok((Value::Null, ControlFlow::None)) } @@ -8616,6 +8684,11 @@ impl Interpreter { let mut pending = self.pending_responses.borrow_mut(); pending.remove(&request_id) }; + // Answered now: drop it from the handler's unanswered-request + // tracking so the exit-time 500 fallback skips it. + self.open_pending_requests + .borrow_mut() + .retain(|id| id != &request_id); let mut completion = match pending_entry { // The admission slot is released by the transport task when it // finishes delivering this response (or on its timeout), so the @@ -8858,6 +8931,11 @@ impl Interpreter { let mut pending = self.pending_responses.borrow_mut(); pending.remove(&request_id) }; + // Answered now (streaming head about to be committed): drop it + // from the handler's unanswered-request tracking. + self.open_pending_requests + .borrow_mut() + .retain(|id| id != &request_id); let mut completion = match pending_entry { Some(entry) => match entry.sender.lock().await.take() { Some(sender) => ResponseCompletion { @@ -9075,8 +9153,12 @@ impl Interpreter { if new_total > max_response_bytes { let actual = new_total; // Drop the stream so the body ends rather than - // silently truncating. + // silently truncating, and untrack it so a handler + // that catches this error keeps no stale id. map.remove(&handle_id); + self.open_response_streams + .borrow_mut() + .retain(|s| s != &handle_id); return Err(self.budget_error( BudgetExceeded::ResponseBytes { limit: max_response_bytes, @@ -9099,7 +9181,12 @@ impl Interpreter { // Receiver dropped => client disconnected. Drop the // handle and surface a catchable error so the handler // can stop (and close any upstream it is proxying). + // Untrack it too so a handler that catches this error + // keeps no stale id in its open-streams list. self.server_response_streams.borrow_mut().remove(&handle_id); + self.open_response_streams + .borrow_mut() + .retain(|s| s != &handle_id); Err(RuntimeError::new( "Cannot write to response stream: the client has disconnected" .to_string(), diff --git a/src/typechecker/mod.rs b/src/typechecker/mod.rs index 013e7067..30dc2809 100644 --- a/src/typechecker/mod.rs +++ b/src/typechecker/mod.rs @@ -830,7 +830,7 @@ impl TypeChecker { ) { self.type_error( "HTTP headers must be a map of header names to values".to_string(), - Some(Type::Map(Box::new(Type::Text), Box::new(Type::Text))), + Some(Type::Map(Box::new(Type::Text), Box::new(Type::Any))), Some(headers_type), *_line, *_column, @@ -912,7 +912,7 @@ impl TypeChecker { ) { self.type_error( "HTTP headers must be a map of header names to values".to_string(), - Some(Type::Map(Box::new(Type::Text), Box::new(Type::Text))), + Some(Type::Map(Box::new(Type::Text), Box::new(Type::Any))), Some(headers_type), *_line, *_column, @@ -2567,7 +2567,7 @@ impl TypeChecker { ) { self.type_error( "Response headers must be a map of header names to values".to_string(), - Some(Type::Map(Box::new(Type::Text), Box::new(Type::Text))), + Some(Type::Map(Box::new(Type::Text), Box::new(Type::Any))), Some(headers_type), *_line, *_column, diff --git a/tests/concurrent_main_loop_test.rs b/tests/concurrent_main_loop_test.rs index 3be3cfdd..3e272f04 100644 --- a/tests/concurrent_main_loop_test.rs +++ b/tests/concurrent_main_loop_test.rs @@ -261,6 +261,67 @@ async fn test_concurrent_handlers_do_not_share_count_loop_state() { shutdown(port, server).await; } +#[tokio::test] +async fn test_handler_that_never_responds_gets_immediate_500() { + // Lifecycle guarantee (P1 #3): a handler that dequeues a request and ends + // WITHOUT responding must resolve the client with 500 immediately — not leave + // it waiting out the request timeout. The `/drop` path does no `respond`; the + // handler simply ends, and the client must still get a prompt 500. + let port = 8345; + let code = format!( + r#" + listen on port {port} as srv + main loop concurrently: + wait for request comes in on srv as req with timeout 20000 + store p as req["path"] + check if p is equal to "/shutdown": + respond to req with "bye" + close server srv + break + otherwise: + check if p is equal to "/drop": + store ignored as "handler returns without responding" + otherwise: + respond to req with "ok" + end check + end check + end loop + "# + ); + let server = start_server_thread(code); + tokio::time::sleep(Duration::from_millis(300)).await; + + let client = reqwest::Client::new(); + + // The un-answered request resolves promptly with 500 rather than hanging. + let t0 = Instant::now(); + let dropped = client + .get(format!("http://127.0.0.1:{port}/drop")) + .send() + .await + .expect("/drop request failed"); + let elapsed = t0.elapsed(); + assert_eq!( + dropped.status().as_u16(), + 500, + "a handler that never responds must yield 500" + ); + assert!( + elapsed < Duration::from_secs(5), + "500 should arrive immediately on handler exit, not after the request timeout ({elapsed:?})" + ); + + // The server survived and keeps serving. + let ok = client + .get(format!("http://127.0.0.1:{port}/ok")) + .send() + .await + .expect("follow-up request failed"); + assert_eq!(ok.text().await.unwrap(), "ok"); + + shutdown(port, server).await; +} + #[tokio::test] async fn test_serial_slow_handler_blocks_next() { let port = 8343; From c20bec5faf2c0684746185655911d1c0fa6d9dd5 Mon Sep 17 00:00:00 2001 From: Claude Date: Wed, 22 Jul 2026 19:16:48 +0000 Subject: [PATCH 013/132] fix: `write line/chunk to ` keeps the classic file write (back-compat) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit `write line to ` shares a surface with the classic file write `write to `, and WFL identifiers can be space-separated, so the lexer merges `line payload` into one token. The parser unconditionally split it into a `line` marker + value, silently reinterpreting a pre-existing file write of a variable named `line payload` as a stream write — a backward-compat break (flagged by Copilot). The two readings can't be told apart at parse time (both the NDJSON stream write and the file write use a bare variable), so disambiguate on the runtime target type: - AST/parser: StreamWriteStatement gains `fallback_content` — for the ambiguous merged form it records both the stream value (`payload`) and the classic file-write content (`line payload`); unambiguous forms set None. - interpreter: evaluate the target first; if it is a server response stream, do the stream write, else if a fallback exists do the classic file write, else error. - analyzer: defer definedness for the ambiguous form (the live reading, and thus which variable must exist, is only known at runtime); count both candidate variables as used so neither is falsely reported unused. Red→Green: tests/write_line_backcompat_test.rs — `store line note as "…"` / `write line note to ""`. Red (analyzer analyzing the stream value): rejects with `Variable 'note' is not defined`. Green: analysis accepts it and the file receives the variable's value, not the token `note`. Existing streaming tests (`write line "alpha" to out`, bare `write line to out`) still pass. Docs: web-servers.md notes the target-type disambiguation. --- ...-07-22-write-line-file-write-backcompat.md | 58 +++++++++++ Docs/04-advanced-features/web-servers.md | 8 ++ src/analyzer/mod.rs | 19 +++- src/analyzer/static_analyzer.rs | 19 ++++ src/interpreter/mod.rs | 54 +++++++++- src/parser/ast.rs | 10 ++ src/parser/stmt/io.rs | 28 ++++-- tests/write_line_backcompat_test.rs | 99 +++++++++++++++++++ 8 files changed, 282 insertions(+), 13 deletions(-) create mode 100644 Dev diary/2026-07-22-write-line-file-write-backcompat.md create mode 100644 tests/write_line_backcompat_test.rs diff --git a/Dev diary/2026-07-22-write-line-file-write-backcompat.md b/Dev diary/2026-07-22-write-line-file-write-backcompat.md new file mode 100644 index 00000000..14ca7249 --- /dev/null +++ b/Dev diary/2026-07-22-write-line-file-write-backcompat.md @@ -0,0 +1,58 @@ +# Dev Diary — 2026-07-22 — `write line/chunk` preserves the classic file write + +## Context + +The new streamed-response verbs `write line to ` / +`write chunk to ` share a surface with the pre-existing file write +`write to `. Because WFL identifiers can be space-separated, the +lexer merges `line payload` into a single `Identifier("line payload")` token. The +first cut of the parser always split such a token into a `line` marker plus a +value, so `write line payload to out` was unconditionally parsed as a stream +write — silently breaking any pre-existing program that wrote a variable literally +named `line payload` to a file. Review (Copilot, twice) flagged this as a +backward-compatibility break. + +Backward compatibility is sacred, and the two readings genuinely cannot be told +apart at parse time: `write line to ` (the primary NDJSON use case) +and `write line to ` (a variable named `line `) both use a bare +variable. The only correct disambiguation is on the **runtime target type**. + +## Fix — carry both readings, decide at runtime + +- **AST/parser.** `StreamWriteStatement` gained `fallback_content: + Option>`. For the ambiguous merged form the parser now records + both the stream value (`Variable("payload")`) and the classic file-write + content (`Variable("line payload")`). Unambiguous forms — a literal value, or a + bare marker directly before `to` — set `None` (they were never valid file + writes). +- **Interpreter.** `StreamWriteStatement` evaluates the target first. If it is a + server response stream, it does the stream write. Otherwise, if a + `fallback_content` is present, it performs the classic `write to + ` file write; if not, it errors as before. +- **Static analysis.** For the ambiguous form the live reading (and thus which + variable must exist) is unknown until runtime, so semantic analysis defers + definedness for it instead of rejecting the file-write reading. The + unused-variable pass counts **both** candidate variables as used, so a variable + named `line ` written to a file is not falsely reported unused. + +The other statements (`analyze`, typechecker, transpiler) already matched with +`..`; the transpiler still rejects streaming statements outright. + +## Testing (Red → Green) + +`tests/write_line_backcompat_test.rs`: + +1. `test_write_line_multiword_variable_parses_with_fallback` — the merged form + parses with `fallback_content: Some`; the literal form with `None`. +2. `test_write_multiword_line_variable_to_file_still_works` — runs the full + analyzer + interpreter on `store line note as "…"` / `write line note to + ""`, asserting analysis accepts it and the file receives the **variable's + value**, not the token `note`. + +- **Red** (analyzer analyzing the stream value unconditionally): semantic analysis + rejects the program with `Variable 'note' is not defined`. +- **Green**: analysis accepts it and the file contains the variable's value. + +Risk class **R3** (backward compatibility). The existing streaming tests +(`write line "alpha" to out`, bare `write line to out`) continue to pass, so the +stream write and the classic file write both work through the shared surface. diff --git a/Docs/04-advanced-features/web-servers.md b/Docs/04-advanced-features/web-servers.md index cd00f6a6..8433214a 100644 --- a/Docs/04-advanced-features/web-servers.md +++ b/Docs/04-advanced-features/web-servers.md @@ -491,6 +491,14 @@ close out NDJSON). `value` may be text, a number, or a boolean. - `write chunk to ` — write raw bytes verbatim, no newline added. `value` may be text or `binary`. + + > **Note.** `write line to ` / `write chunk to ` only + > act as stream writes when `` is a streaming-response handle. If `` + > is instead a file handle or path, the statement falls back to the classic + > `write to ` file write, so a variable literally named + > `line …`/`chunk …` written to a file keeps working. Prefer a plain single-word + > value variable (e.g. `write line payload to out`) to keep the streaming intent + > obvious. - `flush ` — advisory: yield so queued bytes are handed to the socket. (Chunks are already forwarded as you write them; hyper writes as it receives.) - `close ` — end the response body. Writing after `close` is an error. diff --git a/src/analyzer/mod.rs b/src/analyzer/mod.rs index 7a4882f5..8038c35a 100644 --- a/src/analyzer/mod.rs +++ b/src/analyzer/mod.rs @@ -1618,9 +1618,24 @@ impl Analyzer { self.current_scope.define_or_replace(symbol); } - Statement::StreamWriteStatement { value, target, .. } => { - self.analyze_expression(value); + Statement::StreamWriteStatement { + value, + target, + fallback_content, + .. + } => { self.analyze_expression(target); + // For the unambiguous form, check the stream value. For the + // ambiguous merged form (`write line to `, + // `fallback_content` is `Some`) the live interpretation — stream + // write of `` vs classic file write of the variable + // `line ` — depends on the runtime target type, and the + // two reference different variables. Analyzing either here would + // reject a program that is valid under the other reading, so + // definedness is deferred to runtime for this form. + if fallback_content.is_none() { + self.analyze_expression(value); + } } Statement::FlushStreamStatement { target, .. } => { diff --git a/src/analyzer/static_analyzer.rs b/src/analyzer/static_analyzer.rs index 119de7c2..b1ace89b 100644 --- a/src/analyzer/static_analyzer.rs +++ b/src/analyzer/static_analyzer.rs @@ -1339,6 +1339,25 @@ impl Analyzer { self.mark_used_in_expression(content, usages); self.mark_used_in_expression(file, usages); } + Statement::StreamWriteStatement { + value, + target, + fallback_content, + .. + } => { + // Count both interpretations of the ambiguous merged form as + // usages (stream value AND the classic file-write fallback), so a + // variable named `line ` written to a file is not falsely + // reported unused. + self.mark_used_in_expression(value, usages); + self.mark_used_in_expression(target, usages); + if let Some(fallback) = fallback_content { + self.mark_used_in_expression(fallback, usages); + } + } + Statement::FlushStreamStatement { target, .. } => { + self.mark_used_in_expression(target, usages); + } Statement::WriteContentStatement { content, target, .. } diff --git a/src/interpreter/mod.rs b/src/interpreter/mod.rs index 9b8cecee..3f16dfa7 100644 --- a/src/interpreter/mod.rs +++ b/src/interpreter/mod.rs @@ -9113,12 +9113,60 @@ impl Interpreter { value, target, is_line, + fallback_content, line, column, } => { - let handle_id = self - .resolve_server_stream_handle(target, &env, *line, *column) - .await?; + // The target decides the interpretation. Evaluate it once; if it + // is a server response stream, this is a stream write. If it is + // not and this parsed from the ambiguous merged form + // (`write line to `), fall back to the classic + // file write `write to ` so a + // pre-existing file write is never reinterpreted. + let target_val = self.evaluate_expression(target, Rc::clone(&env)).await?; + let handle_id = match &target_val { + Value::Object(obj) => match obj.borrow().get("_server_stream") { + Some(Value::Text(id)) => Some(id.to_string()), + _ => None, + }, + _ => None, + }; + let handle_id = match handle_id { + Some(id) => id, + None => { + if let Some(fallback) = fallback_content { + // Classic file write: `write to `. + let content_value = + self.evaluate_expression(fallback, Rc::clone(&env)).await?; + let file_str = match &target_val { + Value::Text(s) => s.clone(), + _ => { + return Err(RuntimeError::new( + format!( + "Expected a server response stream or a file handle, got {}", + target_val.type_name() + ), + *line, + *column, + )); + } + }; + let content_str = format!("{content_value}"); + return match self.io_client.write_file(&file_str, &content_str).await { + Ok(_) => Ok((Value::Null, ControlFlow::None)), + Err(e) => Err(RuntimeError::new(e, *line, *column)), + }; + } + return Err(RuntimeError::new( + format!( + "Expected a server response stream (from `start streaming response as ...`), got {}", + target_val.type_name() + ), + *line, + *column, + )); + } + }; let val = self.evaluate_expression(value, Rc::clone(&env)).await?; let mut bytes = match &val { Value::Text(s) => s.as_bytes().to_vec(), diff --git a/src/parser/ast.rs b/src/parser/ast.rs index f4725c38..969cd55c 100644 --- a/src/parser/ast.rs +++ b/src/parser/ast.rs @@ -607,6 +607,16 @@ pub enum Statement { target: Expression, /// true for `write line` (newline appended), false for `write chunk`. is_line: bool, + /// Backward-compat fallback for the ambiguous surface form + /// `write line to ` — where `line ` could equally + /// be the classic file write of a variable literally named `line ` + /// (WFL allows space-separated identifiers). When present and the runtime + /// `target` is **not** a server response stream, the statement falls back + /// to `write to ` (a `WriteToStatement`), so a + /// pre-existing file write is never silently reinterpreted as a stream + /// write. `None` when the form is unambiguous (e.g. a literal value, or a + /// bare marker directly before `to`). + fallback_content: Option>, line: usize, column: usize, }, diff --git a/src/parser/stmt/io.rs b/src/parser/stmt/io.rs index 1bc1d688..e83c8808 100644 --- a/src/parser/stmt/io.rs +++ b/src/parser/stmt/io.rs @@ -831,30 +831,41 @@ impl<'a> IoParser<'a> for Parser<'a> { .to_string(); self.bump_sync(); // Consume the (possibly merged) marker - let value = if rest.is_empty() { + // Build the stream-write `value` and, for the ambiguous merged- + // identifier form (`write line to `), the classic + // file-write `fallback_content`. The merged token `line ` could + // equally be a variable literally named `line ` (WFL allows + // space-separated names), so we carry the file-write interpretation + // and let the interpreter pick based on whether `target` is a stream. + let (value, fallback_content) = if rest.is_empty() { // Value begins with a non-identifier (string/number), so the // whole expression — including `with` concatenation — parses - // cleanly from here. - self.parse_expression()? + // cleanly from here. This form was never a valid classic file + // write (`write line "x" to f` did not parse), so no fallback. + (self.parse_expression()?, None) } else { - let left = Expression::Variable(rest, marker_line, marker_column); + // `` alone (stream) vs the full merged `line ` + // (classic file write of that variable). + let stream_left = Expression::Variable(rest, marker_line, marker_column); + let file_left = Expression::Variable(id, marker_line, marker_column); match self.cursor.peek().map(|t| &t.token) { // ` of `, e.g. `write line body of msg to out`. Some(Token::KeywordOf) => { self.bump_sync(); // Consume "of" let object = self.parse_primary_expression()?; - Expression::FunctionCall { + let of = |left: Expression| Expression::FunctionCall { function: Box::new(left), arguments: vec![crate::parser::ast::Argument { name: None, - value: object, + value: object.clone(), }], line: marker_line, column: marker_column, - } + }; + (of(stream_left), Some(Box::new(of(file_left)))) } // A bare variable value: the next token starts `to ...`. - _ => left, + _ => (stream_left, Some(Box::new(file_left))), } }; @@ -867,6 +878,7 @@ impl<'a> IoParser<'a> for Parser<'a> { value, target, is_line, + fallback_content, line: token_pos.line, column: token_pos.column, }); diff --git a/tests/write_line_backcompat_test.rs b/tests/write_line_backcompat_test.rs new file mode 100644 index 00000000..4246c5f6 --- /dev/null +++ b/tests/write_line_backcompat_test.rs @@ -0,0 +1,99 @@ +// Backward-compatibility for `write line to `. +// +// WFL allows space-separated identifiers, so `write line payload to out` could +// mean either the classic file write of a variable literally named "line +// payload", or the new streaming form (`write line to `). The +// merged form must not silently break the pre-existing file write: the runtime +// picks the interpretation from the target type, and static analysis must not +// reject the file-write reading (nor falsely warn the variable is unused). + +use wfl::Interpreter; +use wfl::analyzer::Analyzer; +use wfl::lexer::lex_wfl_with_positions; +use wfl::parser::Parser; +use wfl::parser::ast::Statement; + +fn parse(code: &str) -> Vec { + let tokens = lex_wfl_with_positions(code); + let mut parser = Parser::new(&tokens); + parser + .parse() + .unwrap_or_else(|e| panic!("parse error: {e:?}")) + .statements +} + +#[test] +fn test_write_line_multiword_variable_parses_with_fallback() { + // The ambiguous merged form carries a classic-file-write fallback so the + // interpreter can disambiguate on the target type at runtime. + let stmt = &parse(r#"write line payload to out"#)[0]; + match stmt { + Statement::StreamWriteStatement { + fallback_content, + is_line, + .. + } => { + assert!(*is_line); + assert!( + fallback_content.is_some(), + "merged `write line ` must keep a file-write fallback" + ); + } + other => panic!("expected StreamWriteStatement, got {other:?}"), + } + + // The unambiguous literal form (never a valid classic file write) has none. + let stmt = &parse(r#"write line "x" to out"#)[0]; + match stmt { + Statement::StreamWriteStatement { + fallback_content, .. + } => assert!( + fallback_content.is_none(), + "literal-valued stream write needs no fallback" + ), + other => panic!("expected StreamWriteStatement, got {other:?}"), + } +} + +#[test] +fn test_write_multiword_line_variable_to_file_still_works() { + // A pre-existing program: a variable literally named `line note` written to a + // file path. Must analyze cleanly (no undefined-variable error, no spurious + // unused warning) and, at runtime, write the VARIABLE'S value to the file — + // not stream-write the token "note". + let dir = std::env::temp_dir(); + let path = dir.join("wfl_write_line_backcompat.txt"); + let path_str = path.to_string_lossy().replace('\\', "/"); + let _ = std::fs::remove_file(&path); + + let code = format!( + r#"store line note as "kept across versions" +write line note to "{path_str}""# + ); + + let program = { + let tokens = lex_wfl_with_positions(&code); + let mut parser = Parser::new(&tokens); + parser.parse().expect("parse") + }; + + // Static analysis must accept the file-write reading. + let mut analyzer = Analyzer::new(); + analyzer + .analyze(&program) + .expect("semantic analysis must accept `write line to `"); + + // Runtime writes the variable's value to the file. + let rt = tokio::runtime::Runtime::new().unwrap(); + rt.block_on(async { + let mut interp = Interpreter::new(); + interp.interpret(&program).await.expect("interpret"); + }); + + let contents = std::fs::read_to_string(&path).expect("output file should exist"); + assert_eq!( + contents, "kept across versions", + "the variable `line note` must be written to the file, not the token `note`" + ); + let _ = std::fs::remove_file(&path); +} From c2430563503818fa8c31f6f8066a1c23fa8f835b Mon Sep 17 00:00:00 2001 From: Claude Date: Wed, 22 Jul 2026 19:20:02 +0000 Subject: [PATCH 014/132] docs: reconcile concurrency/testing status with shipped reality (review) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Address doc-consistency review comments: - concurrency-phase-plan.md: qualify per-request isolation (environment + now run-state; global bindings/shared collections stay shared by design); state 500 containment precisely (transport ResponseCompletion mid-respond, plus the interpreter's immediate 500 when a handler ends without responding); sync the stale PR-1b TODO checklist to the as-shipped state ([x] done, [~] transport-provided, and the two remaining known gaps — request-ID logging and a dedicated panic-containment test — called out explicitly). - testing.md: add an adoption note so the verbatim "Status: Proposed / Effective: Upon adoption" policy block reads consistently with this repo's header and CLAUDE.md/AGENTS.md (adopted, binding, effective 2026-07-22); correct the supported-tuples table — CI has no macOS runner (only ubuntu + windows), so macOS is best-effort/not-gated, not "release smoke only". Docs-only; no behavior change. --- Docs/development/concurrency-phase-plan.md | 70 +++++++++++++--------- testing.md | 18 +++++- 2 files changed, 59 insertions(+), 29 deletions(-) diff --git a/Docs/development/concurrency-phase-plan.md b/Docs/development/concurrency-phase-plan.md index 9729b8d6..80dbea8d 100644 --- a/Docs/development/concurrency-phase-plan.md +++ b/Docs/development/concurrency-phase-plan.md @@ -54,14 +54,19 @@ HARD RULES: > not the staged 1a→1b→1c sequence. What is covered: `main loop concurrently:` > surface (locked marker); plain `main loop` byte-compatible serial (tested); > `FuturesUnordered` of `!Send`, `&self`-borrowing handler futures on the -> existing runtime; isolated-per-request **environment** scopes; a slow handler -> not blocking a fast sibling (tested); in-flight cap -> (`CONCURRENT_HANDLER_LIMIT`), with 503/504/500 provided by the existing -> transport layer (bounded queue → 503, response deadline → 504, -> `ResponseCompletion` drop → 500); the empty-set busy-spin trap is avoided -> (cap ≥ 1 keeps the set non-empty). Cooperative, not parallel: handlers -> interleave only at await points, so a CPU-bound handler with no await still -> holds the interpreter thread (documented in `web-servers.md`). +> existing runtime; per-request isolation of both the **environment** scope +> *and* interpreter run-state (see the run-state note below) — note that a +> handler's environment is a fresh child of the shared parent, so top-level +> (global) bindings and any shallow-shared collections reachable through them +> remain shared, by design; a slow handler not blocking a fast sibling (tested); +> in-flight cap (`CONCURRENT_HANDLER_LIMIT`), with 503/504/500 provided by the +> transport plus the interpreter's per-handler exit sweep (bounded queue → 503, +> response deadline → 504, `ResponseCompletion` drop → 500 mid-`respond`, and a +> handler that dequeues a request and ends **without** responding → an immediate +> 500 rather than waiting out the request timeout; tested); the empty-set +> busy-spin trap is avoided (cap ≥ 1 keeps the set non-empty). Cooperative, not +> parallel: handlers interleave only at await points, so a CPU-bound handler with +> no await still holds the interpreter thread (documented in `web-servers.md`). > > **Containment:** *runtime-error* containment is tested (an erroring handler > does not kill the server). *Panic* containment is by construction via @@ -243,31 +248,42 @@ HARD RULES: **Goal:** User-visible opt-in concurrent loop; serial path untouched. +> **Status:** shipped in the single Phase 1 change (see the tracker note above). +> The checklist below is updated to the as-shipped state; `[~]` marks items +> provided by the existing transport layer rather than net-new here, and the two +> remaining known gaps are called out explicitly. + #### TODOs — language / runtime -- [ ] Parser: `main loop concurrently:` (and matching `end`) -- [ ] Analyzer / typechecker / keyword docs if needed -- [ ] Runtime: concurrent path uses proven 1a bridge -- [ ] **G1:** plain `main loop` remains serial and byte-compatible -- [ ] Isolated-per-request scopes (default) -- [ ] Semaphore / in-flight cap (default e.g. 256) → shed **503** -- [ ] Per-request timeout (default e.g. 30s) → **504** (only at await points — document cliff) +- [x] Parser: `main loop concurrently:` (and matching `end`) +- [x] Analyzer / typechecker / keyword docs if needed +- [x] Runtime: concurrent path uses proven 1a bridge +- [x] **G1:** plain `main loop` remains serial and byte-compatible +- [x] Isolated-per-request scopes (default) — plus per-handler run-state isolation +- [~] Semaphore / in-flight cap (default e.g. 256) → shed **503** (cap in the + concurrent loop; 503 shedding from the transport's bounded queue) +- [~] Per-request timeout (default e.g. 30s) → **504** (transport response + deadline; await-point cliff documented) - [ ] Request-ID structured logging on accept / complete / fail / shed / timeout -- [ ] catch_unwind boundary → **500**, siblings survive -- [ ] Eval-core audit: every `RefCell` borrow/borrow_mut on await paths drops before `.await` - - [ ] PR description lists each site and drop-before-await story -- [ ] clippy `await_holding_refcell_ref` enabled/enforced where applicable (backstop only) + — **known gap** (not yet added) +- [x] catch_unwind boundary → **500**, siblings survive +- [x] Eval-core audit: every `RefCell` borrow/borrow_mut on await paths drops + before `.await` (enforced mechanically by the crate-wide clippy backstop) + - [~] PR description lists each site and drop-before-await story (mechanical + lint in lieu of a written per-site walkthrough) +- [x] clippy `await_holding_refcell_ref` enabled/enforced where applicable (backstop only) #### TODOs — tests (write first where possible) -- [ ] Serial `main loop` still processes one request at a time (no silent upgrade) -- [ ] Concurrent loop: slow handler does not block fast sibling -- [ ] Cap exceeded → 503 -- [ ] Timeout → 504 -- [ ] Panic in A → 500; B still completes -- [ ] No empty-set busy-spin -- [ ] Request IDs present in logs (if testable) -- [ ] Existing web TestPrograms still pass on serial path +- [x] Serial `main loop` still processes one request at a time (no silent upgrade) +- [x] Concurrent loop: slow handler does not block fast sibling +- [~] Cap exceeded → 503 (transport-provided; not a dedicated Phase 1 test) +- [~] Timeout → 504 (transport-provided; not a dedicated Phase 1 test) +- [ ] Panic in A → 500; B still completes — **known gap**: only *runtime-error* + containment is tested; panic containment is by construction (see tracker note) +- [x] No empty-set busy-spin (cap ≥ 1 keeps the set non-empty) +- [ ] Request IDs present in logs — **known gap** (tied to the logging item above) +- [x] Existing web TestPrograms still pass on serial path #### TODOs — docs (minimum for ship; full honesty pass may be 1c) diff --git a/testing.md b/testing.md index f82c556e..96f1abf3 100644 --- a/testing.md +++ b/testing.md @@ -17,8 +17,14 @@ project testing profile required by that policy's §4. | Tuple | Presubmit | Release | |---|---|---| | Linux x86-64 (ubuntu-latest), Rust stable (MSRV **1.94+**, edition 2024) | ✅ | ✅ | -| Windows x86-64, Rust stable | ✅ (integration) | ✅ | -| macOS (best-effort; not gated) | — | ⚠️ smoke only | +| Windows x86-64 (windows-latest), Rust stable | ✅ (integration) | ✅ | +| macOS | — | — | + +> **macOS is not a gated tuple.** CI runs only `ubuntu-latest` and +> `windows-latest` (`.github/workflows/ci.yml`); there is no macOS runner in +> presubmit or release. macOS is supported only best-effort by contributors and +> is not verified by this pipeline. Add a macOS matrix entry before claiming any +> gated macOS coverage here. Key runtime dependencies: `tokio`, `warp`/`hyper`, `reqwest`, `sqlx`, `logos`, `tower-lsp`. The interpreter core is single-threaded (`Rc`/`RefCell`); async I/O @@ -136,6 +142,14 @@ behavior; touched code still follows Red→Green and the risk triggers above. ## Logbie Testing Policy +> **Adoption note (this repository):** the text below is the organization policy +> reproduced verbatim, so it keeps the canonical **Status: Proposed** label and +> **Effective date: Upon adoption**. For WFL specifically, that adoption has +> happened: this repository has adopted policy version 1.0 as **binding and in +> force** (see the header at the top of this file, and `CLAUDE.md` / `AGENTS.md`), +> effective **2026-07-22** (the testing-profile review date). Read the "Upon +> adoption" below as "as of 2026-07-22" for this repo. + **Status:** Proposed organization policy, version 1.0 **Owner:** Logbie LLC Engineering **Effective date:** Upon adoption From 5f1d1bc30664eac699a04a1e8a1b23c16c084afe Mon Sep 17 00:00:00 2001 From: Claude Date: Wed, 22 Jul 2026 19:37:09 +0000 Subject: [PATCH 015/132] fix: parse `content type ` in start-streaming; tighten review tests/docs MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit - parser: `start streaming response ... and content type ` where is a bare identifier merges into `type `; split the marker so the variable is bound correctly (previously bound `type ` as one name). Regression test. - tests: `write chunk` fallback parser coverage (ambiguous + literal forms); transpiler test asserts the specific unsupported-transpilation message rather than any error; write-line backcompat test uses an isolated TempDir per run. - docs: web-servers.md no longer over-specifies chunked transfer-encoding (HTTP/1.1 only) — describes the observable behavior (no Content-Length, streamed incrementally) with the HTTP/2 note. --- Docs/04-advanced-features/web-servers.md | 4 ++- src/parser/stmt/web.rs | 22 +++++++++++---- tests/http_server_streaming_test.rs | 17 ++++++++++++ tests/transpiler_test.rs | 7 +++-- tests/write_line_backcompat_test.rs | 34 +++++++++++++++++++++--- 5 files changed, 73 insertions(+), 11 deletions(-) diff --git a/Docs/04-advanced-features/web-servers.md b/Docs/04-advanced-features/web-servers.md index 8433214a..a3822449 100644 --- a/Docs/04-advanced-features/web-servers.md +++ b/Docs/04-advanced-features/web-servers.md @@ -486,7 +486,9 @@ close out - `start streaming response to [with status ] [and content type ] [and headers ] as ` — begin the response. The status defaults to 200 and the content type to `application/octet-stream`. The body has no declared - length; it is sent with chunked transfer-encoding. + length (no `Content-Length`); it is streamed incrementally as you write (over + HTTP/1.1 that is chunked transfer-encoding; HTTP/2+ frames it without a + `Content-Length` instead). - `write line to ` — write `value` followed by a newline (ideal for NDJSON). `value` may be text, a number, or a boolean. - `write chunk to ` — write raw bytes verbatim, no newline added. diff --git a/src/parser/stmt/web.rs b/src/parser/stmt/web.rs index b16f1298..7d2d5f92 100644 --- a/src/parser/stmt/web.rs +++ b/src/parser/stmt/web.rs @@ -426,16 +426,28 @@ impl<'a> WebParser<'a> for Parser<'a> { status = Some(self.parse_primary_expression()?); } // `content type ` — `content` keyword then optional `type`. + // When `` is a bare identifier the lexer merges it into the + // `type` token (`type ct` -> Identifier("type ct")), so split the + // value off rather than binding the whole thing as the variable. Token::KeywordContent => { self.bump_sync(); // with/and self.bump_sync(); // content - if let Some(t) = self.cursor.peek() + let merged_rest = if let Some(t) = self.cursor.peek() && let Token::Identifier(id) = &t.token - && id == "type" + && (id == "type" || id.starts_with("type ")) { - self.bump_sync(); // type - } - content_type = Some(self.parse_primary_expression()?); + let id = id.clone(); + let pos = (t.line, t.column); + self.bump_sync(); // (possibly merged) type + let rest = id.strip_prefix("type").map(str::trim_start).unwrap_or(""); + (!rest.is_empty()).then(|| (rest.to_string(), pos)) + } else { + None + }; + content_type = Some(match merged_rest { + Some((rest, (l, c))) => Expression::Variable(rest, l, c), + None => self.parse_primary_expression()?, + }); } // Merged `content_type ` / `content type ` form. Token::Identifier(id) diff --git a/tests/http_server_streaming_test.rs b/tests/http_server_streaming_test.rs index 50fde802..ac0e8eca 100644 --- a/tests/http_server_streaming_test.rs +++ b/tests/http_server_streaming_test.rs @@ -49,6 +49,23 @@ fn test_start_streaming_response_parses() { } } +#[test] +fn test_content_type_variable_binds_correct_name() { + // `content type ` where is a bare identifier: the lexer merges it + // into `type `, so the parser must split the marker off and bind the + // variable, not `type ` as one name. + let stmt = parse_single_statement( + r#"start streaming response to req with status 200 and content type ct as out"#, + ); + match stmt { + Statement::StartStreamingResponseStatement { content_type, .. } => match content_type { + Some(wfl::parser::ast::Expression::Variable(name, _, _)) => assert_eq!(name, "ct"), + other => panic!("Expected content type Variable(\"ct\"), got {other:?}"), + }, + other => panic!("Expected StartStreamingResponseStatement, got {other:?}"), + } +} + #[test] fn test_write_line_parses() { let stmt = parse_single_statement(r#"write line payload to out"#); diff --git a/tests/transpiler_test.rs b/tests/transpiler_test.rs index e608761b..fc5e0d72 100644 --- a/tests/transpiler_test.rs +++ b/tests/transpiler_test.rs @@ -687,9 +687,12 @@ fn test_main_loop_concurrently_fails_to_transpile() { // error rather than silently emit a serial loop. let source = "main loop concurrently:\n display \"x\"\nend loop"; let result = transpile_wfl(source); + // Assert the specific transpiler rejection, not merely any error — so a + // future parse failure can't masquerade as the intended rejection. + let error = result.expect_err("main loop concurrently should fail to transpile"); assert!( - result.is_err(), - "main loop concurrently should fail to transpile, got: {result:?}" + error.contains("not supported in JavaScript transpilation"), + "expected the unsupported-transpilation error, got: {error}" ); // Plain `main loop` still transpiles. diff --git a/tests/write_line_backcompat_test.rs b/tests/write_line_backcompat_test.rs index 4246c5f6..bb911339 100644 --- a/tests/write_line_backcompat_test.rs +++ b/tests/write_line_backcompat_test.rs @@ -53,6 +53,33 @@ fn test_write_line_multiword_variable_parses_with_fallback() { ), other => panic!("expected StreamWriteStatement, got {other:?}"), } + + // `write chunk` has the same ambiguity and fallback contract as `write line`. + let stmt = &parse(r#"write chunk payload to out"#)[0]; + match stmt { + Statement::StreamWriteStatement { + fallback_content, + is_line, + .. + } => { + assert!(!*is_line, "write chunk must not be a line write"); + assert!( + fallback_content.is_some(), + "merged `write chunk ` must keep a file-write fallback" + ); + } + other => panic!("expected StreamWriteStatement, got {other:?}"), + } + let stmt = &parse(r#"write chunk "x" to out"#)[0]; + match stmt { + Statement::StreamWriteStatement { + fallback_content, .. + } => assert!( + fallback_content.is_none(), + "literal-valued chunk write needs no fallback" + ), + other => panic!("expected StreamWriteStatement, got {other:?}"), + } } #[test] @@ -61,10 +88,11 @@ fn test_write_multiword_line_variable_to_file_still_works() { // file path. Must analyze cleanly (no undefined-variable error, no spurious // unused warning) and, at runtime, write the VARIABLE'S value to the file — // not stream-write the token "note". - let dir = std::env::temp_dir(); - let path = dir.join("wfl_write_line_backcompat.txt"); + // Unique temp dir per invocation so parallel/sharded runs cannot collide or + // delete each other's output; `TempDir` cleans up on drop (even on panic). + let dir = tempfile::tempdir().expect("create temp dir"); + let path = dir.path().join("wfl_write_line_backcompat.txt"); let path_str = path.to_string_lossy().replace('\\', "/"); - let _ = std::fs::remove_file(&path); let code = format!( r#"store line note as "kept across versions" From 143621b4cf92544efcb069162f93fe6b91b348fe Mon Sep 17 00:00:00 2001 From: Claude Date: Wed, 22 Jul 2026 19:38:58 +0000 Subject: [PATCH 016/132] docs: qualify concurrent-handler isolation wording in example (review) Replace 'own isolated scope' with the precise contract: per-iteration child scope + per-handler run-state isolation (so per-request `store` variables don't clobber), while global bindings and collections shared through them stay shared by design. Matches the concurrency-phase-plan wording. --- .../docs_examples/web_servers/concurrent_main_loop.wfl | 7 +++++-- 1 file changed, 5 insertions(+), 2 deletions(-) diff --git a/TestPrograms/docs_examples/web_servers/concurrent_main_loop.wfl b/TestPrograms/docs_examples/web_servers/concurrent_main_loop.wfl index 39591391..07e45ce1 100644 --- a/TestPrograms/docs_examples/web_servers/concurrent_main_loop.wfl +++ b/TestPrograms/docs_examples/web_servers/concurrent_main_loop.wfl @@ -1,8 +1,11 @@ // CI-SKIP: starts a server + concurrent loop; needs HTTP clients (layers 1-4) // Concurrent request handling with `main loop concurrently:`. // -// Iterations run in their own isolated scope. Concurrency is cooperative on one -// thread: a slow handler yields to its siblings at await points (wait for, +// Iterations run in their own child scope, and their count-loop/recursion +// run-state is isolated per handler — so per-request `store` variables never +// clobber another request's. Global (top-level) bindings and any collections +// shared through them remain shared, by design. Concurrency is cooperative on +// one thread: a slow handler yields to its siblings at await points (wait for, // outbound HTTP, stream reads/writes, respond), so it does not block them there // — but CPU-bound work with no await still holds the thread. Plain `main loop` // stays serial; adding `concurrently` is the only way to opt in. From 4448b1916edb0c21b438317992d8f85c30a5c404 Mon Sep 17 00:00:00 2001 From: Claude Date: Wed, 22 Jul 2026 19:46:02 +0000 Subject: [PATCH 017/132] test: wait for server readiness instead of a fixed sleep (fix flaky CI) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit CI "Build, Test, Clippy" failed on concurrent_main_loop_test with `Connection refused`: the tests slept a fixed 300ms after starting the server thread, but on a loaded CI runner binding the port can take longer, so the client connected before the server was ready. Per the testing policy a flaky required test is a failing test. Replace the fixed sleep with `wait_for_server(port)` — a bounded readiness probe that TCP-connects until the server accepts (a bare connect that drops immediately delivers no HTTP request to the handler). Applied to both concurrent_main_loop_test and http_server_streaming_test. Deterministic and faster (returns as soon as the port is bound). --- tests/concurrent_main_loop_test.rs | 27 ++++++++++++++++++++++----- tests/http_server_streaming_test.rs | 23 +++++++++++++++++++---- 2 files changed, 41 insertions(+), 9 deletions(-) diff --git a/tests/concurrent_main_loop_test.rs b/tests/concurrent_main_loop_test.rs index 3e272f04..5d097318 100644 --- a/tests/concurrent_main_loop_test.rs +++ b/tests/concurrent_main_loop_test.rs @@ -58,6 +58,23 @@ fn start_server_thread(code: String) -> std::thread::JoinHandle<()> { }) } +/// Wait until the WFL server has actually bound `port` and is accepting +/// connections, rather than sleeping a fixed interval. A fixed sleep is flaky on +/// a loaded CI runner where binding can take longer than the guess, producing +/// spurious `Connection refused` failures. A bare TCP connect that drops +/// immediately is a safe readiness probe: warp accepts and closes it without +/// delivering an HTTP request to the handler. +async fn wait_for_server(port: u16) { + let addr = format!("127.0.0.1:{port}"); + for _ in 0..300 { + if tokio::net::TcpStream::connect(&addr).await.is_ok() { + return; + } + tokio::time::sleep(Duration::from_millis(50)).await; + } + panic!("server on {addr} did not become ready in time"); +} + fn server_code(port: u16, concurrently: bool) -> String { let marker = if concurrently { " concurrently" } else { "" }; format!( @@ -96,7 +113,7 @@ async fn shutdown(port: u16, server: std::thread::JoinHandle<()>) { async fn test_concurrent_slow_handler_does_not_block_fast() { let port = 8341; let server = start_server_thread(server_code(port, true)); - tokio::time::sleep(Duration::from_millis(300)).await; + wait_for_server(port).await; let client = reqwest::Client::new(); @@ -155,7 +172,7 @@ async fn test_concurrent_handler_error_does_not_kill_server() { "# ); let server = start_server_thread(code); - tokio::time::sleep(Duration::from_millis(300)).await; + wait_for_server(port).await; let client = reqwest::Client::new(); @@ -220,7 +237,7 @@ async fn test_concurrent_handlers_do_not_share_count_loop_state() { "# ); let server = start_server_thread(code); - tokio::time::sleep(Duration::from_millis(300)).await; + wait_for_server(port).await; let a_url = format!("http://127.0.0.1:{port}/a"); let b_url = format!("http://127.0.0.1:{port}/b"); @@ -289,7 +306,7 @@ async fn test_handler_that_never_responds_gets_immediate_500() { "# ); let server = start_server_thread(code); - tokio::time::sleep(Duration::from_millis(300)).await; + wait_for_server(port).await; let client = reqwest::Client::new(); @@ -326,7 +343,7 @@ async fn test_handler_that_never_responds_gets_immediate_500() { async fn test_serial_slow_handler_blocks_next() { let port = 8343; let server = start_server_thread(server_code(port, false)); - tokio::time::sleep(Duration::from_millis(300)).await; + wait_for_server(port).await; let client = reqwest::Client::new(); diff --git a/tests/http_server_streaming_test.rs b/tests/http_server_streaming_test.rs index ac0e8eca..87450054 100644 --- a/tests/http_server_streaming_test.rs +++ b/tests/http_server_streaming_test.rs @@ -121,6 +121,21 @@ fn start_server_thread(code: String) -> std::thread::JoinHandle<()> { }) } +/// Wait until the WFL server has bound `port` and is accepting connections, +/// instead of a fixed sleep that flakes on a loaded CI runner (spurious +/// `Connection refused` when binding takes longer than the guess). A bare TCP +/// connect that drops immediately is a safe readiness probe. +async fn wait_for_server(port: u16) { + let addr = format!("127.0.0.1:{port}"); + for _ in 0..300 { + if tokio::net::TcpStream::connect(&addr).await.is_ok() { + return; + } + tokio::time::sleep(Duration::from_millis(50)).await; + } + panic!("server on {addr} did not become ready in time"); +} + #[tokio::test] async fn test_streamed_response_lines_and_headers() { let port = 8231; @@ -139,7 +154,7 @@ async fn test_streamed_response_lines_and_headers() { ); let server_handle = start_server_thread(server_code); - tokio::time::sleep(Duration::from_millis(300)).await; + wait_for_server(port).await; let client = reqwest::Client::new(); let response = client @@ -184,7 +199,7 @@ async fn test_write_after_close_does_not_reach_client() { ); let server_handle = start_server_thread(server_code); - tokio::time::sleep(Duration::from_millis(300)).await; + wait_for_server(port).await; let response = reqwest::Client::new() .get(format!("http://127.0.0.1:{port}/x")) @@ -226,7 +241,7 @@ async fn test_stream_auto_closes_when_handler_ends_without_close() { ); let server_handle = start_server_thread(server_code); - tokio::time::sleep(Duration::from_millis(300)).await; + wait_for_server(port).await; let response = reqwest::Client::new() .get(format!("http://127.0.0.1:{port}/x")) @@ -268,7 +283,7 @@ async fn test_streamed_response_write_chunk_verbatim() { ); let server_handle = start_server_thread(server_code); - tokio::time::sleep(Duration::from_millis(300)).await; + wait_for_server(port).await; let client = reqwest::Client::new(); let response = client From 6f14943a04fa356914e3ddf1055c40172fbf9553 Mon Sep 17 00:00:00 2001 From: Claude Date: Fri, 24 Jul 2026 02:03:51 +0000 Subject: [PATCH 018/132] fix: mark streaming-statement vars used in analyzer; tidy body type hint & docs MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit - analyzer: `mark_used_variables` now covers HttpStreamStatement, WaitForNextChunk/LineStatement, and StartStreamingResponseStatement — so variables referenced only in those (URL/body/headers, stream source, request/status/content-type) are no longer falsely reported unused (Copilot). Regression test in static_analyzer. - typechecker: the HTTP-body type error passes no single "expected" type (the accepted set is Text|Number|Boolean), so the expected-vs-actual diagnostic isn't misleadingly rendered as "expected Text" (Copilot). - docs: web-servers.md clarifies that only the ambiguous bare-identifier `write line/chunk` form falls back to a file write; literal/number/boolean forms are stream-only and error on a non-stream target (Copilot). --- Docs/04-advanced-features/web-servers.md | 17 ++++---- src/analyzer/static_analyzer.rs | 50 ++++++++++++++++++++++++ src/typechecker/mod.rs | 10 ++++- 3 files changed, 68 insertions(+), 9 deletions(-) diff --git a/Docs/04-advanced-features/web-servers.md b/Docs/04-advanced-features/web-servers.md index a3822449..1dec2f1d 100644 --- a/Docs/04-advanced-features/web-servers.md +++ b/Docs/04-advanced-features/web-servers.md @@ -494,13 +494,16 @@ close out - `write chunk to ` — write raw bytes verbatim, no newline added. `value` may be text or `binary`. - > **Note.** `write line to ` / `write chunk to ` only - > act as stream writes when `` is a streaming-response handle. If `` - > is instead a file handle or path, the statement falls back to the classic - > `write to ` file write, so a variable literally named - > `line …`/`chunk …` written to a file keeps working. Prefer a plain single-word - > value variable (e.g. `write line payload to out`) to keep the streaming intent - > obvious. + > **Note (backward compatibility).** `write line to ` / + > `write chunk to ` shares its surface with the classic file write + > `write to `, and WFL allows space-separated variable names, so + > the form `write line to ` is ambiguous. Only that ambiguous + > **bare-identifier** form carries a fallback: if `` turns out to be a file + > handle or path rather than a streaming handle, it runs the classic file write + > (so a variable literally named `line …`/`chunk …` keeps working). The + > unambiguous forms — a literal, number, or boolean value (e.g. + > `write line "x" to out`) — are **stream-only** and error if `` is not a + > streaming-response handle. - `flush ` — advisory: yield so queued bytes are handed to the socket. (Chunks are already forwarded as you write them; hyper writes as it receives.) - `close ` — end the response body. Writing after `close` is an error. diff --git a/src/analyzer/static_analyzer.rs b/src/analyzer/static_analyzer.rs index b1ace89b..1ed9a050 100644 --- a/src/analyzer/static_analyzer.rs +++ b/src/analyzer/static_analyzer.rs @@ -1358,6 +1358,34 @@ impl Analyzer { Statement::FlushStreamStatement { target, .. } => { self.mark_used_in_expression(target, usages); } + Statement::HttpStreamStatement { + url, + method, + headers, + body, + .. + } => { + self.mark_used_in_expression(url, usages); + for expr in [method, headers, body].into_iter().flatten() { + self.mark_used_in_expression(expr, usages); + } + } + Statement::WaitForNextChunkStatement { source, .. } + | Statement::WaitForNextLineStatement { source, .. } => { + self.mark_used_in_expression(source, usages); + } + Statement::StartStreamingResponseStatement { + request, + status, + content_type, + headers, + .. + } => { + self.mark_used_in_expression(request, usages); + for expr in [status, content_type, headers].into_iter().flatten() { + self.mark_used_in_expression(expr, usages); + } + } Statement::WriteContentStatement { content, target, .. } @@ -2175,6 +2203,28 @@ mod tests { assert_eq!(diagnostics[0].code, "ANALYZE-UNUSED"); } + #[test] + fn test_streaming_statement_variables_are_not_reported_unused() { + // Variables referenced only inside the streaming/incremental-read + // statements must count as used — otherwise a program that opens a + // stream from a URL/body variable gets a false unused warning. + let input = "store my_url as \"http://example.com/s\"\n\ +store my_body as \"payload\"\n\ +open url at my_url with method \"POST\" and body my_body and stream response as upstream\n\ +wait for next line from upstream as ln\n\ +display ln"; + let tokens = crate::lexer::lex_wfl_with_positions(input); + let program = crate::parser::Parser::new(&tokens).parse().unwrap(); + + let analyzer = Analyzer::new(); + let diagnostics = analyzer.check_unused_variables(&program, 0); + + assert!( + diagnostics.is_empty(), + "streaming-statement variables must not be reported unused, got: {diagnostics:?}" + ); + } + #[test] fn test_respond_headers_expression_marks_variable_used() { // Regression: a variable referenced only in the diff --git a/src/typechecker/mod.rs b/src/typechecker/mod.rs index 30dc2809..cfdf15b9 100644 --- a/src/typechecker/mod.rs +++ b/src/typechecker/mod.rs @@ -852,7 +852,10 @@ impl TypeChecker { ) { self.type_error( "HTTP request body must be text, a number, or a boolean (numbers and booleans are converted to text)".to_string(), - Some(Type::Text), + // No single "expected" type — the accepted set is + // Text|Number|Boolean, so a bare `Text` hint would + // misrender the expected-vs-actual diagnostic. + None, Some(body_type), *_line, *_column, @@ -932,7 +935,10 @@ impl TypeChecker { ) { self.type_error( "HTTP request body must be text, a number, or a boolean (numbers and booleans are converted to text)".to_string(), - Some(Type::Text), + // No single "expected" type — the accepted set is + // Text|Number|Boolean, so a bare `Text` hint would + // misrender the expected-vs-actual diagnostic. + None, Some(body_type), *_line, *_column, From 19ebb79f8486a201cd7b40b967617ee9b52a9d4d Mon Sep 17 00:00:00 2001 From: Claude Date: Fri, 24 Jul 2026 02:14:12 +0000 Subject: [PATCH 019/132] test+docs: broaden streaming unused-var test; qualify phase-plan claims (review) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit - static_analyzer test: cover every new streaming arm (HttpStream url/method/body, WaitForNextChunk + WaitForNextLine source, StartStreamingResponse status/content type) and metadata operands, with a negative assertion that a genuinely-unused variable is still the only one flagged (CodeRabbit). - concurrency-phase-plan.md: narrow the "tested" 503/504/500 claim to the immediate-500 case that actually has a dedicated test (503/504 are transport-provided, untested at Phase 1); mark the eval-core RefCell-across-await audit as OPEN/partial — the clippy backstop covers await-holding borrows but the shutdown/signal lifecycle paths are not yet separately audited; qualify the 1c tracker status to match the open checklist items (CodeRabbit). --- Docs/development/concurrency-phase-plan.md | 24 +++++++++++------ src/analyzer/static_analyzer.rs | 31 +++++++++++++++++++--- 2 files changed, 43 insertions(+), 12 deletions(-) diff --git a/Docs/development/concurrency-phase-plan.md b/Docs/development/concurrency-phase-plan.md index 80dbea8d..1c9871a3 100644 --- a/Docs/development/concurrency-phase-plan.md +++ b/Docs/development/concurrency-phase-plan.md @@ -48,7 +48,7 @@ HARD RULES: | 0 | 0c | Bound accept/queue (OOM shed) | ✅ Done | | 1 | 1a | Runtime spike (bridge, no surface) | ✅ Done (folded into 1b) | | 1 | 1b | `main loop concurrently:` surface + ops defaults | ✅ Done — awaiting maintainer review | -| 1 | 1c | Honesty docs for real concurrent model | ✅ Done | +| 1 | 1c | Honesty docs for real concurrent model | ✅ Docs landed — some 1b/1c checklist items still open (request-ID logging; RefCell shutdown/signal-path audit), see notes below | > **Phase 1 landed in one change** (`Dev diary/2026-07-22-concurrent-request-handlers.md`), > not the staged 1a→1b→1c sequence. What is covered: `main loop concurrently:` @@ -63,7 +63,9 @@ HARD RULES: > transport plus the interpreter's per-handler exit sweep (bounded queue → 503, > response deadline → 504, `ResponseCompletion` drop → 500 mid-`respond`, and a > handler that dequeues a request and ends **without** responding → an immediate -> 500 rather than waiting out the request timeout; tested); the empty-set +> 500 rather than waiting out the request timeout — *this immediate-500 case is +> covered by a dedicated test*; the 503/504 cases are transport-provided and are +> **not** exercised by a dedicated Phase-1 test); the empty-set > busy-spin trap is avoided (cap ≥ 1 keeps the set non-empty). Cooperative, not > parallel: handlers interleave only at await points, so a CPU-bound handler with > no await still holds the interpreter thread (documented in `web-servers.md`). @@ -88,10 +90,14 @@ HARD RULES: > Regression: `tests/concurrent_main_loop_test.rs::test_concurrent_handlers_do_not_share_count_loop_state` > (Red without the swap: `/a` observed `/b`'s entire count range). > -> **Also lighter than the full 1b checklist:** request-ID *structured* logging is -> not yet added; the eval-core `RefCell`-across-await audit is enforced -> mechanically by the crate-wide `#![deny(clippy::await_holding_refcell_ref)]` -> backstop rather than a written per-site walkthrough. +> **Also lighter than the full 1b checklist (open items):** request-ID +> *structured* logging is not yet added; the eval-core `RefCell`-across-await +> audit is **only partially covered** — the crate-wide +> `#![deny(clippy::await_holding_refcell_ref)]` backstop catches borrows held +> across `.await`, but the **shutdown/signal lifecycle paths** (dropped borrows on +> teardown, e.g. `close server`'s `web_servers.borrow_mut()` held across a short +> await) are **not yet separately audited or tested**. Treat the RefCell audit as +> **open**, with the clippy lint as supplementary — not complete — coverage. > > **This is the maintainer STOP/review point** — please review before Phase 2. | 2 | 2a | Structured nursery + join engine | ⬜ Not started | @@ -267,8 +273,10 @@ HARD RULES: - [ ] Request-ID structured logging on accept / complete / fail / shed / timeout — **known gap** (not yet added) - [x] catch_unwind boundary → **500**, siblings survive -- [x] Eval-core audit: every `RefCell` borrow/borrow_mut on await paths drops - before `.await` (enforced mechanically by the crate-wide clippy backstop) +- [~] Eval-core audit: every `RefCell` borrow/borrow_mut on await paths drops + before `.await` — **partial**: the clippy backstop catches await-holding + borrows, but the shutdown/signal lifecycle paths are not yet separately audited + (see the "open items" note above); treat as open - [~] PR description lists each site and drop-before-await story (mechanical lint in lieu of a written per-site walkthrough) - [x] clippy `await_holding_refcell_ref` enabled/enforced where applicable (backstop only) diff --git a/src/analyzer/static_analyzer.rs b/src/analyzer/static_analyzer.rs index 1ed9a050..30d64cbd 100644 --- a/src/analyzer/static_analyzer.rs +++ b/src/analyzer/static_analyzer.rs @@ -2207,11 +2207,25 @@ mod tests { fn test_streaming_statement_variables_are_not_reported_unused() { // Variables referenced only inside the streaming/incremental-read // statements must count as used — otherwise a program that opens a - // stream from a URL/body variable gets a false unused warning. + // stream from a URL/body variable gets a false unused warning. Exercise + // every new arm and every metadata operand: + // - HttpStreamStatement: url, method, body + // - WaitForNextChunkStatement / WaitForNextLineStatement: source + // - StartStreamingResponseStatement: status, content type + // and a genuinely-unused variable to prove the pass still flags real + // dead code (negative assertion). let input = "store my_url as \"http://example.com/s\"\n\ +store my_method as \"POST\"\n\ store my_body as \"payload\"\n\ -open url at my_url with method \"POST\" and body my_body and stream response as upstream\n\ +open url at my_url with method my_method and body my_body and stream response as upstream\n\ +wait for next chunk from upstream as ch\n\ wait for next line from upstream as ln\n\ +store st as 200\n\ +store ctype as \"application/x-ndjson\"\n\ +start streaming response to req with status st and content type ctype as out\n\ +write line \"x\" to out\n\ +store dead as \"never read\"\n\ +display ch\n\ display ln"; let tokens = crate::lexer::lex_wfl_with_positions(input); let program = crate::parser::Parser::new(&tokens).parse().unwrap(); @@ -2219,9 +2233,18 @@ display ln"; let analyzer = Analyzer::new(); let diagnostics = analyzer.check_unused_variables(&program, 0); + // The only unused variable is `dead`; every streaming operand counts as + // used (a missing arm would surface `my_url`/`my_method`/`my_body`/ + // `upstream`/`st`/`ctype` here too). + assert_eq!( + diagnostics.len(), + 1, + "expected only `dead` unused, got: {diagnostics:?}" + ); assert!( - diagnostics.is_empty(), - "streaming-statement variables must not be reported unused, got: {diagnostics:?}" + diagnostics[0].message.contains("dead"), + "expected the unused diagnostic to name `dead`, got: {:?}", + diagnostics[0].message ); } From 90137fcd9042160ec4722581781e4881d587ee6c Mon Sep 17 00:00:00 2001 From: Claude Date: Fri, 24 Jul 2026 02:21:07 +0000 Subject: [PATCH 020/132] docs: align write-chunk value types, note write-line `with` limit, drop dead var - web-servers.md: `write chunk ` also accepts numbers/booleans (written as their text form), matching the interpreter; add a note that `write line/chunk` don't yet accept in-statement `with`-concatenation on a bare-variable value (workaround: build the value with `store` first). - interoperability.md: remove the unused `store done as no` leftover from the incremental-read example. Doc-only; matches shipped behavior. (Copilot) --- Docs/04-advanced-features/interoperability.md | 1 - Docs/04-advanced-features/web-servers.md | 9 ++++++++- 2 files changed, 8 insertions(+), 2 deletions(-) diff --git a/Docs/04-advanced-features/interoperability.md b/Docs/04-advanced-features/interoperability.md index 0a78d0e5..aab9c6c2 100644 --- a/Docs/04-advanced-features/interoperability.md +++ b/Docs/04-advanced-features/interoperability.md @@ -125,7 +125,6 @@ store content_type as upstream.headers["content-type"] // Pull the body one line at a time. Each read returns the next line, or // `nothing` once the stream ends cleanly. -store done as no count from 1 to 1000000: wait for next line from upstream as line check if line is nothing: diff --git a/Docs/04-advanced-features/web-servers.md b/Docs/04-advanced-features/web-servers.md index 1dec2f1d..14e57d5f 100644 --- a/Docs/04-advanced-features/web-servers.md +++ b/Docs/04-advanced-features/web-servers.md @@ -492,7 +492,8 @@ close out - `write line to ` — write `value` followed by a newline (ideal for NDJSON). `value` may be text, a number, or a boolean. - `write chunk to ` — write raw bytes verbatim, no newline added. - `value` may be text or `binary`. + `value` may be text or `binary` (a number or boolean is also accepted and + written as its text form). > **Note (backward compatibility).** `write line to ` / > `write chunk to ` shares its surface with the classic file write @@ -504,6 +505,12 @@ close out > unambiguous forms — a literal, number, or boolean value (e.g. > `write line "x" to out`) — are **stream-only** and error if `` is not a > streaming-response handle. + > + > **Value operators.** When the value is a bare variable, `write line`/ + > `write chunk` currently accept only that variable (optionally `field of + > object`) before `to`; `with`-concatenation directly in the statement (e.g. + > `write line prefix with json to out`) is not yet supported. Build the value + > first — `store payload as prefix with json`, then `write line payload to out`. - `flush ` — advisory: yield so queued bytes are handed to the socket. (Chunks are already forwarded as you write them; hyper writes as it receives.) - `close ` — end the response body. Writing after `close` is an error. From 1e4876414cf60aaee242450cc4fdb63520ffce04 Mon Sep 17 00:00:00 2001 From: Claude Date: Fri, 24 Jul 2026 03:04:25 +0000 Subject: [PATCH 021/132] fix: drop RefCell borrows before await; re-enable await_holding_refcell_ref deny (P1 #4) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Brad's re-review: `close server` held `web_servers.borrow_mut()` across a 50ms await, and the whole module suppressed `clippy::await_holding_refcell_ref` with a crate-level allow — so a sibling handler touching the map during that yield could panic, and the lint could not catch future regressions. - `close server` (HTTP and WebSocket): remove the entry into a local and drop the map borrow BEFORE the graceful-shutdown await. - Parent-method call: clone the parent `Rc`/type out of the container instance so no instance/parent RefCell borrow spans the awaited method call. - Two `open file for reading` paths: capture the `env.define` result and drop the env borrow before the `close_file` await. - Flip the module attribute from `allow` to `#![deny(clippy::await_holding_refcell_ref)]` so borrow-across-await is now a hard error for this module going forward. clippy --all-targets clean; streaming/concurrent tests green. --- src/interpreter/mod.rs | 77 ++++++++++++++++++++++++++---------------- 1 file changed, 48 insertions(+), 29 deletions(-) diff --git a/src/interpreter/mod.rs b/src/interpreter/mod.rs index 3f16dfa7..91c6c920 100644 --- a/src/interpreter/mod.rs +++ b/src/interpreter/mod.rs @@ -1,4 +1,4 @@ -#![allow(clippy::await_holding_refcell_ref)] +#![deny(clippy::await_holding_refcell_ref)] mod assertion_helpers; pub mod bounded_buffer; pub mod command_sanitizer; @@ -5451,10 +5451,12 @@ impl Interpreter { match self.io_client.open_file(&path_str).await { Ok(handle) => match self.io_client.read_file(&handle, &self.budget).await { Ok(content) => { - match env + // Capture the define result and drop the env + // borrow before the `close_file` await below. + let define_result = env .borrow_mut() - .define(variable_name, Value::Text(content.into())) - { + .define(variable_name, Value::Text(content.into())); + match define_result { Ok(_) => { let _ = self.io_client.close_file(&handle).await; Ok((Value::Null, ControlFlow::None)) @@ -6412,10 +6414,12 @@ impl Interpreter { Ok(handle) => { match self.io_client.read_file(&handle, &self.budget).await { Ok(content) => { - match env + // Capture the define result and drop the + // env borrow before the `close_file` await. + let define_result = env .borrow_mut() - .define(variable_name, Value::Text(content.into())) - { + .define(variable_name, Value::Text(content.into())); + match define_result { Ok(_) => { let _ = self.io_client.close_file(&handle).await; @@ -7678,12 +7682,16 @@ impl Interpreter { // Check if this is a container instance if let Value::ContainerInstance(instance_rc) = &this_val { - let instance = instance_rc.borrow(); + // Clone the parent Rc out so the instance's RefCell borrow does + // not span the awaited method call below (a sibling handler + // could otherwise re-borrow the same instance across the yield). + let parent_opt = instance_rc.borrow().parent.clone(); // Check if the instance has a parent - if let Some(parent_rc) = &instance.parent { - let parent = parent_rc.borrow(); - let parent_type = parent.container_type.clone(); + if let Some(parent_rc) = parent_opt { + // Read the parent's type, then release its borrow too — no + // container RefCell borrow is held across the `.await`s. + let parent_type = parent_rc.borrow().container_type.clone(); // Look up the parent container definition let parent_def = match env.borrow().get(&parent_type) { @@ -9357,7 +9365,9 @@ impl Interpreter { && name.starts_with("WebSocketServer::") { let key = name.to_string(); - if let Some(mut ws_server) = self.web_socket_servers.borrow_mut().remove(&key) { + // Remove and drop the borrow before any `.await` below. + let removed_ws = self.web_socket_servers.borrow_mut().remove(&key); + if let Some(mut ws_server) = removed_ws { // Wake every live connection's reader so it stops waiting // on the peer and tears down (releasing its slot), even if // the peer never answers the close handshake. @@ -9434,24 +9444,33 @@ impl Interpreter { } }; - // Close the server - let mut web_servers = self.web_servers.borrow_mut(); - if let Some(mut wfl_server) = web_servers.remove(&server_name) { - // Graceful shutdown: Give in-flight responses time to complete transmission - // before forcefully aborting the server task - if let Some(handle) = wfl_server.server_handle.take() { - // Allow 50ms for pending HTTP responses to be transmitted - // This prevents race condition where abort() closes the TCP connection - // before response bytes reach the client, causing IncompleteMessage errors - tokio::time::sleep(tokio::time::Duration::from_millis(50)).await; - handle.abort(); + // Close the server. Remove it from the map and DROP the borrow + // before the graceful-shutdown await — holding `web_servers` + // borrowed across `.await` would panic a concurrent sibling that + // touches the map during the yield (the reason this module can + // drop its `await_holding_refcell_ref` allow — see lib.rs). + let removed = self.web_servers.borrow_mut().remove(&server_name); + match removed { + Some(mut wfl_server) => { + // Graceful shutdown: give in-flight responses time to + // complete transmission before forcefully aborting the + // server task. The map borrow is already released, so + // this await cannot conflict with a sibling handler. + if let Some(handle) = wfl_server.server_handle.take() { + // Allow 50ms for pending HTTP responses to reach the + // client before abort() closes the TCP connection + // (otherwise IncompleteMessage on the client). + tokio::time::sleep(tokio::time::Duration::from_millis(50)).await; + handle.abort(); + } + } + None => { + return Err(RuntimeError::new( + format!("Server '{}' not found", server_name), + *line, + *column, + )); } - } else { - return Err(RuntimeError::new( - format!("Server '{}' not found", server_name), - *line, - *column, - )); } Ok((Value::Null, ControlFlow::None)) From ae518ce577fa4b439ddb73fd60b2d978eda002dc Mon Sep 17 00:00:00 2001 From: Claude Date: Fri, 24 Jul 2026 03:07:55 +0000 Subject: [PATCH 022/132] fix: budget-check server write size before materializing bytes (P1 #6) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Brad's re-review: `write line|chunk` cloned the entire text/binary value into a new `Vec` and only then checked it against `web_server_max_response_size` — so a huge write allocated/copied far past the limit just to be rejected. Now compute the outgoing byte length from the value in place (`str::len` / `[u8]::len`, plus the optional newline; numbers/booleans render to a short string), reserve the response-byte budget under the map borrow, and only materialize the bytes once the write is known to fit. Over-budget writes are refused before any large allocation; the stream is still dropped/untracked on breach as before. --- src/interpreter/mod.rs | 79 +++++++++++++++++++++++++----------------- 1 file changed, 47 insertions(+), 32 deletions(-) diff --git a/src/interpreter/mod.rs b/src/interpreter/mod.rs index 91c6c920..8e3b5900 100644 --- a/src/interpreter/mod.rs +++ b/src/interpreter/mod.rs @@ -9176,10 +9176,17 @@ impl Interpreter { } }; let val = self.evaluate_expression(value, Rc::clone(&env)).await?; - let mut bytes = match &val { - Value::Text(s) => s.as_bytes().to_vec(), - Value::Binary(b) => b.to_vec(), - Value::Number(_) | Value::Bool(_) => val.to_string().into_bytes(), + let newline = usize::from(*is_line); + // Compute the outgoing byte length WITHOUT cloning a large + // text/binary value, so an over-budget write is rejected *before* + // any big allocation/copy (a huge write must not be materialized + // just to be refused). + let incoming_len = match &val { + Value::Text(s) => s.len() + newline, + Value::Binary(b) => b.len() + newline, + // Numbers/booleans render to a short string; the tiny + // allocation to measure them is not a DoS concern. + Value::Number(_) | Value::Bool(_) => val.to_string().len() + newline, _ => { return Err(RuntimeError::new( format!( @@ -9191,21 +9198,17 @@ impl Interpreter { )); } }; - if *is_line { - bytes.push(b'\n'); - } - // Enforce the response-byte ceiling on the running total (so a - // stream cannot bypass `web_server_max_response_size` via one - // huge chunk or many chunks), then clone the sender out so the - // map borrow is not held across the (possibly backpressured) - // send await. + // Reserve the response-byte budget on the running total FIRST (so a + // stream cannot bypass `web_server_max_response_size` via one huge + // chunk or many chunks), then clone the sender out so the map borrow + // is not held across the (possibly backpressured) send await. let max_response_bytes = self.budget.limits().max_response_bytes; let sender = { let mut map = self.server_response_streams.borrow_mut(); match map.get_mut(&handle_id) { Some((tx, bytes_written)) => { - let new_total = bytes_written.saturating_add(bytes.len()); + let new_total = bytes_written.saturating_add(incoming_len); if new_total > max_response_bytes { let actual = new_total; // Drop the stream so the body ends rather than @@ -9231,26 +9234,38 @@ impl Interpreter { } }; match sender { - Some(tx) => match tx.send(bytes).await { - Ok(()) => Ok((Value::Null, ControlFlow::None)), - Err(_) => { - // Receiver dropped => client disconnected. Drop the - // handle and surface a catchable error so the handler - // can stop (and close any upstream it is proxying). - // Untrack it too so a handler that catches this error - // keeps no stale id in its open-streams list. - self.server_response_streams.borrow_mut().remove(&handle_id); - self.open_response_streams - .borrow_mut() - .retain(|s| s != &handle_id); - Err(RuntimeError::new( - "Cannot write to response stream: the client has disconnected" - .to_string(), - *line, - *column, - )) + Some(tx) => { + // Within budget and the stream is open: materialize the + // bytes now (after the ceiling check) and send them. + let mut bytes = match &val { + Value::Text(s) => s.as_bytes().to_vec(), + Value::Binary(b) => b.to_vec(), + _ => val.to_string().into_bytes(), + }; + if *is_line { + bytes.push(b'\n'); + } + match tx.send(bytes).await { + Ok(()) => Ok((Value::Null, ControlFlow::None)), + Err(_) => { + // Receiver dropped => client disconnected. Drop the + // handle and surface a catchable error so the + // handler can stop (and close any upstream it is + // proxying). Untrack it too so a handler that + // catches this error keeps no stale id. + self.server_response_streams.borrow_mut().remove(&handle_id); + self.open_response_streams + .borrow_mut() + .retain(|s| s != &handle_id); + Err(RuntimeError::new( + "Cannot write to response stream: the client has disconnected" + .to_string(), + *line, + *column, + )) + } } - }, + } None => Err(RuntimeError::new( "Cannot write to a closed response stream".to_string(), *line, From 33660215c0e9902748d8b7edec03a571078448f0 Mon Sep 17 00:00:00 2001 From: Claude Date: Fri, 24 Jul 2026 03:10:51 +0000 Subject: [PATCH 023/132] fix: finalize top-level streams/requests on every exit path (P1 #5) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Brad's re-review: "all streams close on every exit path" was false outside ordinary completed loop iterations. - Cleanup now runs AFTER the conventional `main` action (it ran before), so a stream/request opened by `main` is finalized instead of leaking. - The mid-run per-statement timeout early-return now drains open streams and 500s unanswered requests before returning (previously bypassed cleanup). - On interpreter reuse (REPL), the reset now actually CLOSES the prior run's open streams and 500s its unanswered requests (draining the tracking) instead of merely clearing the id lists — clearing alone stranded the still-open sender/receiver in the maps, hanging the client and leaking the entry. Streaming/concurrent tests green. --- src/interpreter/mod.rs | 39 ++++++++++++++++++++++++++------------- 1 file changed, 26 insertions(+), 13 deletions(-) diff --git a/src/interpreter/mod.rs b/src/interpreter/mod.rs index 8e3b5900..0c155680 100644 --- a/src/interpreter/mod.rs +++ b/src/interpreter/mod.rs @@ -4198,9 +4198,13 @@ impl Interpreter { *self.in_count_loop.borrow_mut() = false; *self.current_count.borrow_mut() = None; // A prior run that ended while a stream was open or a request was - // unanswered (REPL reuse) must not leave dangling ids tracked here. - self.open_response_streams.borrow_mut().clear(); - self.open_pending_requests.borrow_mut().clear(); + // unanswered (REPL reuse) must not leave dangling entries. Actually CLOSE + // those streams and 500 those requests (draining the tracking), rather + // than only clearing the id lists — clearing alone would strand the + // still-open sender/receiver in `server_response_streams` / + // `pending_responses`, hanging the client and leaking the entry. + self.close_open_response_streams(); + self.fail_open_pending_requests(); // Reset to the inherited base depth (0 for a top-level run/REPL; the // parent's live depth for an `execute file` child) so recursion // accounting spans the execute-file boundary instead of granting the @@ -4379,6 +4383,10 @@ impl Interpreter { ); } errors.push(err); + // A mid-run timeout is still an exit path: finalize any open + // top-level streams and unanswered requests before returning. + self.close_open_response_streams(); + self.fail_open_pending_requests(); return Err(errors); } @@ -4423,14 +4431,9 @@ impl Interpreter { } } - // Close any server response streams opened directly at top level (outside - // a `main loop`, which already closes per-iteration/per-handler) so a - // script that starts a stream and exits without `close` still finalizes - // the client's body rather than leaving it hanging until process death. - // Likewise 500 any top-level request dequeued but never answered. - self.close_open_response_streams(); - self.fail_open_pending_requests(); - + // Run the conventional `main` action (if any) before cleanup, so a + // stream/request opened by `main` is finalized by the drain below rather + // than leaking. if errors.is_empty() { let main_func_opt = { match self.global_env.borrow().get("main") { @@ -4458,11 +4461,21 @@ impl Interpreter { } } } + } - self.assert_invariants(); + // Close any server response streams opened directly at top level or by + // `main` (outside a `main loop`, which already closes + // per-iteration/per-handler), and 500 any top-level request dequeued but + // never answered — on EVERY exit path (normal end, statement error, or a + // failing `main`), so a script that exits without `close` still finalizes + // the client's body rather than leaving it hanging until process death. + self.close_open_response_streams(); + self.fail_open_pending_requests(); + + self.assert_invariants(); + if errors.is_empty() { Ok(last_value) } else { - self.assert_invariants(); Err(errors) } } From d32f3fd1cf7aaa41a7b17adfbed1c39d4b485738 Mon Sep 17 00:00:00 2001 From: Claude Date: Fri, 24 Jul 2026 03:13:07 +0000 Subject: [PATCH 024/132] fix: only dispatch `start` to streaming when `streaming` follows (back-compat) Brad/Copilot: statement dispatch routed every `Token::KeywordStart` to `parse_start_streaming_response()`, which errors unless the next token is `streaming`. That would hijack any other statement-initial use of the `start` keyword (e.g. a `start of text` pattern anchor). Guard the arm so it only fires when `streaming` actually follows; otherwise `start` falls through to its normal handling. `start streaming response ...` still parses. --- src/parser/mod.rs | 14 ++++++++++++-- 1 file changed, 12 insertions(+), 2 deletions(-) diff --git a/src/parser/mod.rs b/src/parser/mod.rs index c51d863a..471c7a12 100644 --- a/src/parser/mod.rs +++ b/src/parser/mod.rs @@ -612,8 +612,18 @@ impl<'a> StmtParser<'a> for Parser<'a> { // words (and a bare identifier message) lex as one merged token. // `start streaming response to ... as `. `start` is a // keyword; `streaming` is a contextual identifier; `response` is - // a keyword. - Token::KeywordStart => self.parse_start_streaming_response(), + // a keyword. Only intercept when `streaming` actually follows, so + // any other statement-initial use of the `start` keyword (e.g. + // the `start of text` pattern anchor) is not hijacked and can + // fall through to its own handling. + Token::KeywordStart + if matches!( + self.cursor.peek_next().map(|t| &t.token), + Some(Token::Identifier(id)) if id == "streaming" || id.starts_with("streaming ") + ) => + { + self.parse_start_streaming_response() + } // `flush ` — the target merges into the token // (`flush out` -> Identifier("flush out")). Only match when an // operand follows, so a bare `flush` used as an action/variable From b632b595a3beb05eea8b0535422c84ece9dc85d4 Mon Sep 17 00:00:00 2001 From: Claude Date: Fri, 24 Jul 2026 03:16:09 +0000 Subject: [PATCH 025/132] fix: back off and cap consecutive failures in concurrent main loop (P1 #3) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Brad's re-review: the concurrent loop refilled and re-polled with no backoff/classification, so a deterministic pre-await error (bad expression) or `wait for request` on a server closed without the loop breaking would hot-spin the CPU and the log forever — and the loop is deadline-exempt. Track consecutive handler failures (reset on any successful iteration). Between failures, back off with a small capped sleep so instant failures yield instead of spinning; once failures cross a structural-failure threshold (256 with no progress), terminate the loop rather than refilling forever. Incidental handler errors interleaved with successful requests never trip the guard. --- src/interpreter/mod.rs | 51 ++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 51 insertions(+) diff --git a/src/interpreter/mod.rs b/src/interpreter/mod.rs index 0c155680..8b74b6fd 100644 --- a/src/interpreter/mod.rs +++ b/src/interpreter/mod.rs @@ -3948,6 +3948,16 @@ impl Interpreter { let mut futs = FuturesUnordered::new(); let mut last_value = Value::Null; + // A handler that fails *before* it ever awaits (a deterministic bad + // expression, or `wait for request` on a server that was closed without + // the loop breaking) completes instantly, so a naive refill-and-repoll + // would spin the CPU (and the log) forever — and this loop is exempt from + // the wall-clock deadline. Count consecutive failures with no successful + // iteration in between: back off between them, and terminate the loop once + // it's clearly a structural failure rather than incidental handler errors. + let mut consecutive_failures: u32 = 0; + const MAX_CONSECUTIVE_FAILURES: u32 = 256; + loop { self.check_time()?; @@ -3970,6 +3980,8 @@ impl Interpreter { // `Ready(None)` that would busy-spin the loop. match futs.next().await { Some(Ok(Ok((value, flow)))) => { + // A completed iteration (request handled) — not a failure. + consecutive_failures = 0; last_value = value; match flow { ControlFlow::Break => break, @@ -3985,11 +3997,29 @@ impl Interpreter { // keep the server running instead of tearing it down. Some(Ok(Err(err))) => { log::warn!("concurrent main loop: handler error: {err}"); + if self + .backoff_or_break_concurrent( + &mut consecutive_failures, + MAX_CONSECUTIVE_FAILURES, + ) + .await + { + break; + } } // A handler panicked: catch_unwind contained it; the request is // answered 500 by the drop guard. Siblings survive. Some(Err(_panic)) => { log::warn!("concurrent main loop: handler panicked; request answered 500"); + if self + .backoff_or_break_concurrent( + &mut consecutive_failures, + MAX_CONSECUTIVE_FAILURES, + ) + .await + { + break; + } } None => break, // unreachable while cap >= 1; end cleanly if reached } @@ -3998,6 +4028,27 @@ impl Interpreter { Ok((last_value, ControlFlow::None)) } + /// Handle a failed concurrent-handler iteration: bump the consecutive-failure + /// counter, back off proportionally (capped) so instant failures cannot hot- + /// spin the CPU/log while the loop is deadline-exempt, and report whether the + /// caller should break the loop because failures have crossed the structural- + /// failure threshold (e.g. the server was closed without the loop breaking). + /// Returns `true` to break. + async fn backoff_or_break_concurrent(&self, consecutive_failures: &mut u32, max: u32) -> bool { + *consecutive_failures += 1; + if *consecutive_failures >= max { + log::error!( + "concurrent main loop: {consecutive_failures} consecutive handler failures with no successful request; stopping the loop to avoid a hot spin" + ); + return true; + } + // Small, capped backoff so a burst of instant failures yields the thread + // (and rate-limits the log) instead of spinning. + let backoff_ms = (*consecutive_failures).min(50) as u64; + tokio::time::sleep(Duration::from_millis(backoff_ms)).await; + false + } + /// Preserve ordinary file I/O failures while classifying byte-ceiling /// breaches as catchable execution-budget resource errors. fn file_read_error(&self, error: FileReadError, line: usize, column: usize) -> RuntimeError { From f74bd37d129aa62dd6542926c120bf1fa37763e1 Mon Sep 17 00:00:00 2001 From: Claude Date: Fri, 24 Jul 2026 03:20:54 +0000 Subject: [PATCH 026/132] feat: absolute total deadline for outbound streaming responses (P1 #2) Brad's re-review: each outbound `stream_pull` got a fresh idle timer, so an upstream that trickles a byte before every idle timeout could run forever inside the deadline-exempt main loop. Add an absolute total-lifetime cap distinct from the per-read idle timeout: - new config `outbound_stream_max_seconds` (default 300; `0` disables), with a config-reference entry; - HttpStreamHandle records a `total_deadline` at open time; - `stream_pull` refuses a read once the total deadline passes AND bounds each read's idle timeout by the time remaining to the total, so no single read waits past the cap and a trickling upstream cannot outlive it. config/http_stream tests green; clippy clean. --- Docs/reference/configuration-reference.md | 11 +++++++++ src/config.rs | 23 +++++++++++++++++++ src/interpreter/mod.rs | 28 ++++++++++++++++++++++- 3 files changed, 61 insertions(+), 1 deletion(-) diff --git a/Docs/reference/configuration-reference.md b/Docs/reference/configuration-reference.md index baac7050..0a2f2e12 100644 --- a/Docs/reference/configuration-reference.md +++ b/Docs/reference/configuration-reference.md @@ -214,6 +214,7 @@ All keys currently loaded from config files, with defaults. | `web_server_max_response_size` | integer ≥ 1 | `67108864` (64 MiB) | Max handler or outbound HTTP response body size (bytes) | | `web_server_request_queue_bound` | integer ≥ 1 | `256` | Max queued HTTP requests before shedding with 503 | | `web_server_response_timeout_seconds` | integer ≥ 0 | `300` | Seconds to await a handler before shedding with 504; `0` disables | +| `outbound_stream_max_seconds` | integer ≥ 0 | `300` | Absolute total lifetime (seconds) of one outbound streaming response, distinct from the per-read idle timeout; `0` disables | | `web_socket_queue_bound` | integer ≥ 1 | `1024` | Max queued frames/events per WebSocket channel before shedding | | `web_socket_max_connections` | integer ≥ 1 | `1024` | Max simultaneous live WebSocket connections | | `web_socket_max_message_size` | integer ≥ 1 | `1048576` (1 MiB) | Max size of a single WebSocket text message (bytes); larger frames are dropped | @@ -566,6 +567,16 @@ Maximum time, in seconds, the transport waits for a handler to answer an accepte A value of `0` disables the timeout. The in-flight request cap (`web_server_request_queue_bound`) is enforced globally across every `listen` server via one shared budget, and a request's slot is held from the moment its body starts streaming until the handler responds, this timeout fires, or the client disconnects. +#### `outbound_stream_max_seconds` + +Absolute total lifetime, in seconds, of a single **outbound** streaming response opened with `open url ... and stream response as `, measured from when the stream is opened. This is distinct from `timeout_seconds`, which is the per-read **idle** timeout: an upstream that trickles one byte just before every idle timeout would otherwise run forever, but it can never live past this hard cap. Each incremental read (`wait for next line/chunk`) is additionally bounded by the time remaining to this deadline, so no single read waits past the total. + +- **Type:** Integer (0 or more) +- **Default:** `300` +- **Example:** `outbound_stream_max_seconds = 60` + +A value of `0` disables the absolute cap (the idle timeout still applies per read). + #### `web_socket_queue_bound` Maximum number of queued frames (per outbound connection) and lifecycle events (per server) held for a WebSocket before shedding. Bounds WebSocket memory the same way `web_server_request_queue_bound` bounds HTTP requests: when a channel is full, the extra frame/event is dropped and a warning is logged, instead of growing memory without bound. diff --git a/src/config.rs b/src/config.rs index 71d6609d..7e093e37 100644 --- a/src/config.rs +++ b/src/config.rs @@ -62,6 +62,12 @@ pub struct WflConfig { /// HTTP request before shedding it with 504 and releasing its in-flight /// slot. `0` disables the timeout. Feeds `ExecutionBudget`. Default 300. pub web_server_response_timeout_seconds: u64, + /// Absolute total lifetime (seconds) of a single outbound streaming response + /// (`open url ... and stream response`), measured from when the stream is + /// opened. Distinct from `timeout_seconds` (the per-read idle timeout): an + /// upstream that trickles a byte before every idle timeout can never run past + /// this hard cap. `0` disables the total cap. Default 300. + pub outbound_stream_max_seconds: u64, // --- Shared ExecutionBudget limits (see src/exec/budget.rs) --- /// Hard ceiling on charged interpreter operations. `None`/`0` = unlimited /// (the default, matching historic behavior). Feeds `ExecutionBudget`. @@ -192,6 +198,8 @@ impl Default for WflConfig { // Free an accepted request's in-flight slot if its handler does not // answer within 5 minutes (far longer than any serial handler needs). web_server_response_timeout_seconds: 300, + // Absolute cap on a single outbound streaming response's lifetime. + outbound_stream_max_seconds: 300, // Shared ExecutionBudget limits (see src/exec/budget.rs). Defaults // are chosen so existing programs never trip them while runaway // behavior gets a clean error instead of a crash or OOM. @@ -819,6 +827,21 @@ fn parse_config_text(config: &mut WflConfig, text: &str, file: &Path) { file.display() ), }, + "outbound_stream_max_seconds" => match value.parse::() { + Ok(secs) => { + // 0 disables the total cap (the documented sentinel). + config.outbound_stream_max_seconds = secs; + log::debug!( + "Loaded outbound_stream_max_seconds: {secs} from {}", + file.display() + ); + } + Err(_) => log::warn!( + "Invalid outbound_stream_max_seconds '{}' in {}: expected a non-negative integer", + value, + file.display() + ), + }, "max_operations" => match value.parse::() { Ok(n) => { // 0 means "no operation ceiling" (the default). diff --git a/src/interpreter/mod.rs b/src/interpreter/mod.rs index 8b74b6fd..4621d673 100644 --- a/src/interpreter/mod.rs +++ b/src/interpreter/mod.rs @@ -1584,6 +1584,11 @@ struct HttpStreamHandle { /// Total body bytes pulled from the network so far, enforced against /// `max_response_bytes`. bytes_read: usize, + /// Absolute deadline for the whole stream, set from + /// `outbound_stream_max_seconds` at open time. Distinct from the per-read + /// idle timeout: an upstream that trickles a byte before every idle timeout + /// still cannot run past this. `None` disables the total cap. + total_deadline: Option, } /// Errors raised while an outbound HTTP request is in flight. @@ -1845,11 +1850,16 @@ impl IoClient { let stream = response .bytes_stream() .map(|chunk| chunk.map(|b| b.to_vec())); + let total_deadline = match self.config.outbound_stream_max_seconds { + 0 => None, // sentinel: no absolute total cap + secs => Some(Instant::now() + Duration::from_secs(secs)), + }; let handle = HttpStreamHandle { stream: Box::pin(stream), buffer: Vec::new(), done: false, bytes_read: 0, + total_deadline, }; let handle_id = { let mut next_id = self.next_stream_id.lock().await; @@ -1901,7 +1911,23 @@ impl IoClient { return Ok(false); } let max_response_bytes = budget.limits().max_response_bytes; - let configured_timeout = Duration::from_secs(self.config.timeout_seconds.max(1)); + // Per-read idle timeout, bounded by the remaining time to the stream's + // absolute total deadline so a single read can never wait past the total + // (and a trickling upstream cannot outlive the total cap). + let idle_timeout = Duration::from_secs(self.config.timeout_seconds.max(1)); + let configured_timeout = match handle.total_deadline { + Some(deadline) => { + let remaining = deadline.saturating_duration_since(Instant::now()); + if remaining.is_zero() { + // Already past the absolute total lifetime. + return Err(HttpClientError::Timeout { + seconds: self.config.outbound_stream_max_seconds, + }); + } + idle_timeout.min(remaining) + } + None => idle_timeout, + }; let next = Self::run_http_with_budget(Arc::clone(budget), configured_timeout, async { Ok::>>, HttpClientError>(handle.stream.next().await) }) From 1778efb3f8eda2379f50e9e2233feda55e26f412 Mon Sep 17 00:00:00 2001 From: Claude Date: Fri, 24 Jul 2026 03:24:06 +0000 Subject: [PATCH 027/132] ci: gate docs-validation and web tests; make scripts executable (P1 gating) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Brad's re-review: testing.md claims CI runs web tests and docs validation, but ci.yml invoked neither, and run_web_tests.sh / validate_docs_examples.py were committed non-executable (mode 100644) — so green CI didn't supply the R3 web/streaming boundary evidence the policy now mandates. - Mark both scripts executable (git +x). - Integration Tests job now runs `validate_docs_examples.py --ci` and `run_web_tests.sh` on Linux (after the release binary is built). The Rust streaming/concurrent integration tests already run via `cargo test --test '*'`; these add the doc-example and real-web-server gates. --- .github/workflows/ci.yml | 13 +++++++++++++ scripts/run_web_tests.sh | 0 scripts/validate_docs_examples.py | 0 3 files changed, 13 insertions(+) mode change 100644 => 100755 scripts/run_web_tests.sh mode change 100644 => 100755 scripts/validate_docs_examples.py diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml index a2e7a0d3..5034d75c 100644 --- a/.github/workflows/ci.yml +++ b/.github/workflows/ci.yml @@ -172,6 +172,19 @@ jobs: - name: Run All Integration Tests run: cargo test --test '*' --verbose + # Validate that documentation examples still parse/analyze/lint against the + # current release binary (testing.md requires docs validation in CI). + - name: Validate Docs Examples + if: runner.os != 'Windows' + run: python3 scripts/validate_docs_examples.py --ci + + # Web-server integration tests: start real WFL servers and exercise them + # over HTTP (testing.md requires web tests in CI for the R3 web/streaming + # surface). Uses the release binary built above. + - name: Run Web Server Tests + if: runner.os != 'Windows' + run: ./scripts/run_web_tests.sh + # Database integration tests against live PostgreSQL and MariaDB servers. # SQLite database tests need no services and already run everywhere via # `cargo test`; this job exercises the env-gated PostgreSQL/MariaDB paths. diff --git a/scripts/run_web_tests.sh b/scripts/run_web_tests.sh old mode 100644 new mode 100755 diff --git a/scripts/validate_docs_examples.py b/scripts/validate_docs_examples.py old mode 100644 new mode 100755 From 7e0ea7a7e0d189cd457679fdc23b91cab9d3aadb Mon Sep 17 00:00:00 2001 From: Claude Date: Fri, 24 Jul 2026 03:29:19 +0000 Subject: [PATCH 028/132] test: surface server-thread interpret errors and re-raise join panics Copilot: the streaming/concurrent server threads ignored `interpret()`'s result and the tests ignored `join()`, so a server-side interpreter error or panic was dropped and the test could still pass. - Server threads now `panic!` on an unexpected interpreter error. - `join_server` (streaming) and `shutdown` (concurrent) `resume_unwind` the thread's panic payload (JoinHandle::join's error is Box, no Debug), so a server-side failure fails the test loudly with its original message. All streaming/concurrent tests still pass (confirming clean shutdown on Ok). --- tests/concurrent_main_loop_test.rs | 14 ++++++++++++-- tests/http_server_streaming_test.rs | 24 +++++++++++++++++++----- 2 files changed, 31 insertions(+), 7 deletions(-) diff --git a/tests/concurrent_main_loop_test.rs b/tests/concurrent_main_loop_test.rs index 5d097318..c9dd67dd 100644 --- a/tests/concurrent_main_loop_test.rs +++ b/tests/concurrent_main_loop_test.rs @@ -53,7 +53,11 @@ fn start_server_thread(code: String) -> std::thread::JoinHandle<()> { let mut parser = Parser::new(&tokens); let ast = parser.parse().expect("parse"); let mut interpreter = Interpreter::new(); - let _ = interpreter.interpret(&ast).await; + // Surface an unexpected interpreter error as a thread panic so + // `shutdown` re-raises it instead of the test silently passing. + if let Err(errors) = interpreter.interpret(&ast).await { + panic!("server interpreter failed: {errors:?}"); + } }); }) } @@ -106,7 +110,13 @@ async fn shutdown(port: u16, server: std::thread::JoinHandle<()>) { .get(format!("http://127.0.0.1:{port}/shutdown")) .send() .await; - let _ = tokio::task::spawn_blocking(move || server.join()).await; + // Re-raise a server-thread panic (or interpreter error) instead of dropping + // it, so a server-side failure fails the test loudly. + match tokio::task::spawn_blocking(move || server.join()).await { + Ok(Ok(())) => {} + Ok(Err(panic)) => std::panic::resume_unwind(panic), + Err(join_err) => panic!("server join task failed: {join_err}"), + } } #[tokio::test] diff --git a/tests/http_server_streaming_test.rs b/tests/http_server_streaming_test.rs index 87450054..d1710b0c 100644 --- a/tests/http_server_streaming_test.rs +++ b/tests/http_server_streaming_test.rs @@ -116,11 +116,25 @@ fn start_server_thread(code: String) -> std::thread::JoinHandle<()> { let mut parser = Parser::new(&tokens); let ast = parser.parse().expect("Failed to parse WFL code"); let mut interpreter = Interpreter::new(); - let _ = interpreter.interpret(&ast).await; + // Surface an unexpected interpreter error as a thread panic, so the + // test's `join_server` re-raises it instead of silently passing. + if let Err(errors) = interpreter.interpret(&ast).await { + panic!("server interpreter failed: {errors:?}"); + } }); }) } +/// Join the server thread and re-raise its panic (if any) in the test thread, so +/// a server-side panic or interpreter error fails the test loudly rather than +/// being dropped. `JoinHandle::join`'s error is `Box` (no `Debug`), so +/// `resume_unwind` is the way to propagate it with its original message. +fn join_server(handle: std::thread::JoinHandle<()>) { + if let Err(panic) = handle.join() { + std::panic::resume_unwind(panic); + } +} + /// Wait until the WFL server has bound `port` and is accepting connections, /// instead of a fixed sleep that flakes on a loaded CI runner (spurious /// `Connection refused` when binding takes longer than the guess). A bare TCP @@ -174,7 +188,7 @@ async fn test_streamed_response_lines_and_headers() { let body = response.text().await.expect("Failed to read body"); assert_eq!(body, "alpha\nbeta\ngamma\n"); - let _ = server_handle.join(); + join_server(server_handle); } #[tokio::test] @@ -212,7 +226,7 @@ async fn test_write_after_close_does_not_reach_client() { "writes after close must not reach the client" ); - let _ = server_handle.join(); + join_server(server_handle); } #[tokio::test] @@ -264,7 +278,7 @@ async fn test_stream_auto_closes_when_handler_ends_without_close() { .get(format!("http://127.0.0.1:{port}/shutdown")) .send() .await; - let _ = server_handle.join(); + join_server(server_handle); } #[tokio::test] @@ -297,5 +311,5 @@ async fn test_streamed_response_write_chunk_verbatim() { // write chunk does not append newlines. assert_eq!(body, "onetwo"); - let _ = server_handle.join(); + join_server(server_handle); } From 430db007ef0148c487e3d401a52f377a665cb322 Mon Sep 17 00:00:00 2001 From: Claude Date: Fri, 24 Jul 2026 03:34:18 +0000 Subject: [PATCH 029/132] fix: tighten ambiguous write line/chunk analysis Analyze the unambiguous of-object subexpression of a merged `write line|chunk to `, and report an undefined variable only when NEITHER candidate name resolves (stream value vs classic file-write variable `line `), so a genuine typo is still caught without breaking either valid reading. Addresses Copilot review comment on src/analyzer/mod.rs. Co-Authored-By: Claude Opus 4.8 Claude-Session: https://claude.ai/code/session_011LxXAm5E4Vy39kRLr8kRnX --- src/analyzer/mod.rs | 78 +++++++++++++++++++++++++---- tests/write_line_backcompat_test.rs | 20 ++++++++ 2 files changed, 88 insertions(+), 10 deletions(-) diff --git a/src/analyzer/mod.rs b/src/analyzer/mod.rs index 8038c35a..142c6355 100644 --- a/src/analyzer/mod.rs +++ b/src/analyzer/mod.rs @@ -1622,19 +1622,46 @@ impl Analyzer { value, target, fallback_content, + line, + column, .. } => { self.analyze_expression(target); - // For the unambiguous form, check the stream value. For the - // ambiguous merged form (`write line to `, - // `fallback_content` is `Some`) the live interpretation — stream - // write of `` vs classic file write of the variable - // `line ` — depends on the runtime target type, and the - // two reference different variables. Analyzing either here would - // reject a program that is valid under the other reading, so - // definedness is deferred to runtime for this form. - if fallback_content.is_none() { - self.analyze_expression(value); + match fallback_content { + // Unambiguous form: check the stream value normally. + None => self.analyze_expression(value), + // Ambiguous merged form (`write line to `): + // the live reading — stream write of `` vs classic + // file write of the variable `line ` — depends on the + // runtime target type, and the two read different variables. + // Still (1) analyze any unambiguous subexpression (the + // `` in ` of `), and (2) report an + // undefined variable only when *neither* candidate name is + // defined, so a genuine typo is still caught without breaking + // either valid reading. + Some(fallback) => { + // The `of ` argument is the same under both + // readings and is unambiguous — analyze it. + if let Expression::FunctionCall { arguments, .. } = value { + for arg in arguments { + self.analyze_expression(&arg.value); + } + } + let stream_name = Self::stream_write_candidate_name(value); + let fallback_name = Self::stream_write_candidate_name(fallback); + if let (Some(sn), Some(fal)) = (stream_name, fallback_name) + && !self.name_is_defined(sn) + && !self.name_is_defined(fal) + { + // Neither reading resolves — report the classic + // (file-write) name, matching the runtime fallback. + self.report_undefined_name( + format!("Variable '{fal}' is not defined"), + *line, + *column, + ); + } + } } } @@ -3486,6 +3513,37 @@ impl Analyzer { } } + /// The candidate variable name referenced by a `write line|chunk` value: a + /// bare `Variable`, or the callee of a ` of ` call. + fn stream_write_candidate_name(expr: &Expression) -> Option<&str> { + match expr { + Expression::Variable(name, ..) => Some(name), + Expression::FunctionCall { function, .. } => match &**function { + Expression::Variable(name, ..) => Some(name), + _ => None, + }, + _ => None, + } + } + + /// Whether a bare name resolves to something known (an action parameter, the + /// `count` loop variable, a builtin, an in-scope binding, or a container + /// property) — i.e. it would NOT be reported as an undefined variable. Used + /// to decide the ambiguous `write line|chunk` case without emitting. + fn name_is_defined(&self, name: &str) -> bool { + if self.action_parameters.contains(name) + || name == "count" + || Self::is_builtin_function(name) + || self.current_scope.resolve(name).is_some() + { + return true; + } + if let Some(container_name) = &self.current_container { + return self.is_container_property(container_name, name); + } + false + } + fn analyze_expression(&mut self, expression: &Expression) { // Recursive front-end checkpoint for expressions. `analyze_statement` // polls per statement, but one statement can hold an arbitrarily large diff --git a/tests/write_line_backcompat_test.rs b/tests/write_line_backcompat_test.rs index bb911339..1ee51f05 100644 --- a/tests/write_line_backcompat_test.rs +++ b/tests/write_line_backcompat_test.rs @@ -125,3 +125,23 @@ write line note to "{path_str}""# ); let _ = std::fs::remove_file(&path); } + +#[test] +fn test_ambiguous_write_line_still_flags_when_neither_candidate_defined() { + // The ambiguous form defers definedness to runtime, but a genuine typo where + // NEITHER reading resolves (`payload` as a stream value, nor `line payload` + // as a file-write variable) must still be caught by static analysis. + let code = "listen on port 8080 as srv\nwrite line payload to srv"; + let tokens = lex_wfl_with_positions(code); + let program = Parser::new(&tokens).parse().expect("parse"); + + let mut analyzer = Analyzer::new(); + let result = analyzer.analyze(&program); + let errors = result.expect_err("neither `payload` nor `line payload` is defined"); + assert!( + errors + .iter() + .any(|e| e.message.contains("line payload") && e.message.contains("not defined")), + "expected an undefined-variable error naming `line payload`, got: {errors:?}" + ); +} From 6127e72095f8cda5323bc2c101a2477bf2464af8 Mon Sep 17 00:00:00 2001 From: Claude Date: Fri, 24 Jul 2026 03:47:28 +0000 Subject: [PATCH 030/132] fix: require wait-for-request first in concurrent loop; enforce text header keys Three review-driven hardening changes on the streaming branch: - Analyzer: a `main loop concurrently:` body must begin with `wait for request`. Concurrent handler slots start from the top of the body, so any statement before the first `wait for request` runs once per slot before a request is dequeued. Reject that at analysis time with an actionable error instead of running setup speculatively. Serial `main loop` is unaffected. (New surface; every example/test already complies.) - Typechecker: HTTP header maps must have text keys. A single is_valid_header_map_type helper (used by outbound HTTP, streaming responses, and `respond ... and headers`) rejects a map with a concrete non-text key while accepting text-keyed and loosely-typed maps, matching the "header names" contract in the error message. - CI: run docs-example validation and the web-server suite on Windows too (run_web_tests.ps1), and pass --force so docs validation ignores the committed cache and always re-validates. Red->Green evidence: test_concurrent_main_loop_requires_wait_for_request_first and test_header_map_type_requires_text_keys, both confirmed failing without the change and passing with it. Docs + Dev Diary updated. Co-Authored-By: Claude Opus 4.8 Claude-Session: https://claude.ai/code/session_011LxXAm5E4Vy39kRLr8kRnX --- .github/workflows/ci.yml | 20 ++- ...rent-loop-ordering-and-header-key-types.md | 81 ++++++++++ Docs/04-advanced-features/web-servers.md | 8 + src/analyzer/mod.rs | 144 ++++++++++++++---- src/typechecker/mod.rs | 73 +++++++-- 5 files changed, 280 insertions(+), 46 deletions(-) create mode 100644 Dev diary/2026-07-24-concurrent-loop-ordering-and-header-key-types.md diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml index 5034d75c..f3a7ee9c 100644 --- a/.github/workflows/ci.yml +++ b/.github/workflows/ci.yml @@ -174,17 +174,29 @@ jobs: # Validate that documentation examples still parse/analyze/lint against the # current release binary (testing.md requires docs validation in CI). - - name: Validate Docs Examples + # `--force` ignores the committed cache so CI always re-validates rather + # than trusting a stale cached result. + - name: Validate Docs Examples (Unix) if: runner.os != 'Windows' - run: python3 scripts/validate_docs_examples.py --ci + run: python3 scripts/validate_docs_examples.py --ci --force + + - name: Validate Docs Examples (Windows) + if: runner.os == 'Windows' + run: python scripts/validate_docs_examples.py --ci --force # Web-server integration tests: start real WFL servers and exercise them # over HTTP (testing.md requires web tests in CI for the R3 web/streaming - # surface). Uses the release binary built above. - - name: Run Web Server Tests + # surface). Uses the release binary built above. Both OSes are covered so + # Windows web-server behavior does not go unvalidated. + - name: Run Web Server Tests (Unix) if: runner.os != 'Windows' run: ./scripts/run_web_tests.sh + - name: Run Web Server Tests (Windows) + if: runner.os == 'Windows' + shell: pwsh + run: ./scripts/run_web_tests.ps1 + # Database integration tests against live PostgreSQL and MariaDB servers. # SQLite database tests need no services and already run everywhere via # `cargo test`; this job exercises the env-gated PostgreSQL/MariaDB paths. diff --git a/Dev diary/2026-07-24-concurrent-loop-ordering-and-header-key-types.md b/Dev diary/2026-07-24-concurrent-loop-ordering-and-header-key-types.md new file mode 100644 index 00000000..f3bd7e2b --- /dev/null +++ b/Dev diary/2026-07-24-concurrent-loop-ordering-and-header-key-types.md @@ -0,0 +1,81 @@ +# Dev Diary — 2026-07-24: concurrent-loop ordering guard + header-key type validation + +Two review-driven hardening changes on the runtime-streaming branch, each with +Red→Green evidence at the lowest useful layer (semantic analysis / type check). + +## 1. `main loop concurrently:` must begin with `wait for request` + +**Problem (reviewer, CodeRabbit on `src/interpreter/mod.rs`):** the concurrent +main loop refills its handler set by starting `execute_block(body, …)` for every +slot up to the concurrency cap. Each future runs the body from the top. If the +body has any statement *before* the first `wait for request`, that statement runs +once per slot — speculatively, before a single request has been dequeued. For a +body that starts with `wait for request` the future simply parks on the request +channel, so nothing runs early; the hazard only exists for out-of-order bodies. + +**Fix:** rather than restructure the loop into a heavier dequeue-then-run engine, +enforce the invariant the well-formed case already satisfies — a +`main loop concurrently:` body must begin with `wait for request`. Semantic +analysis now rejects a concurrent loop whose first statement is anything else, +with an actionable message that tells the author to move setup above the loop. +Serial `main loop` is unaffected (it runs one iteration at a time, so there is no +speculative fan-out). This is new surface (`concurrently` shipped on this branch), +so no existing program is affected; every example and test already starts with +`wait for request`. + +- Code: `src/analyzer/mod.rs` — split the merged `ForeverLoop | MainLoop` arm so + the concurrent case is checked; extracted the shared body walk into + `analyze_loop_body`. The ordering error is pushed *before* the body is analyzed + so it is not swept up by the handler-body error→warning demotion. +- **Risk class R3** (concurrency/lifecycle). +- **Red→Green:** `test_concurrent_main_loop_requires_wait_for_request_first` + (inline in `src/analyzer/mod.rs`) parses real WFL source and asserts: (a) a + concurrent loop with `store … ` before `wait for request` is rejected naming + the ordering rule; (b) the *same* body under serial `main loop` is accepted; + (c) a concurrent loop that starts with `wait for request` is accepted. Red was + confirmed by neutralizing the guard (test failed), then restored (test passed). +- Docs: `Docs/04-advanced-features/web-servers.md` states the requirement; the + existing `TestPrograms/docs_examples/web_servers/concurrent_main_loop.wfl` + already begins with `wait for request`, so it needed no change. + +## 2. HTTP header maps must have text keys + +**Problem (reviewer, Copilot on `src/typechecker/mod.rs`):** the header type +checks for outbound HTTP (`http … with headers`), streaming responses, and +`respond … and headers` all accepted any `Map<_, _>`. HTTP header names must be +text, so `Map` passed typechecking even though it can never be a valid +header set — and the error message already promised "header names." + +**Fix:** a single `is_valid_header_map_type` helper, used at all four sites, that +accepts a map only when its key type is `Text` (or `Unknown`/`Any`/`Error`, so a +header set the checker cannot fully resolve — map literals often infer an unknown +key — is never falsely flagged) and rejects a map with a concrete non-text key. + +- Code: `src/typechecker/mod.rs` — helper + four call sites collapsed onto it. +- **Red→Green:** `test_header_map_type_requires_text_keys` covers accepted + (`Map`, loose-key maps, `Unknown`/`Any`) and rejected (`Map`, + `Map`, non-map) cases. Because map literals infer `Map` + keys today, a concretely non-text-keyed header map is not reachable from source + — this is a defensive guard, so the honest evidence is a unit test on the + boundary that changed. Red confirmed by broadening the key match, then restored. + +## 3. CI: validate Windows too; ignore the docs cache + +Also on this branch (reviewer, Copilot on `.github/workflows/ci.yml`): the +Integration Tests job ran docs-example validation and the web-server integration +tests on Linux only, and trusted the committed validation cache. Now: + +- Docs validation runs on both OSes with `--force` (ignores the cache so CI + always re-validates). +- The web-server suite runs on Windows too via the existing + `scripts/run_web_tests.ps1` (`shell: pwsh`), matching the testing profile's + requirement that web tests run in CI rather than leaving Windows unvalidated. + +## Residual risk + +- The concurrent-loop guard is a static structural rule, not a runtime rewrite: + it removes the speculative-side-effect footgun by construction, but the loop + still starts its slots eagerly (each parked on `wait for request`). A full + dequeue-then-run engine remains future work. +- Enabling the Windows web-test suite may surface pre-existing Windows-only + behavior; if it does, that is a real signal to fix, not to re-hide. diff --git a/Docs/04-advanced-features/web-servers.md b/Docs/04-advanced-features/web-servers.md index 14e57d5f..1226009d 100644 --- a/Docs/04-advanced-features/web-servers.md +++ b/Docs/04-advanced-features/web-servers.md @@ -118,6 +118,14 @@ bound work (the common web case), not CPU-bound loops. Plain `main loop` keeps its exact serial behavior — adding `concurrently` is the only way to opt in; nothing changes silently. +> **A `main loop concurrently:` body must begin with `wait for request`.** A +> concurrent loop starts its handler slots up front, each running the body from +> the top, so any statement placed *before* the first `wait for request` would +> run once per slot before a single request arrives. WFL rejects that at analysis +> time with a clear error. Put per-server setup **above** the loop; the loop body +> starts by waiting for the next request. (Serial `main loop` has no such +> requirement — it runs one iteration at a time.) + > `concurrently` is only special right after `main loop`; it is not a reserved > word, so existing programs that use `concurrently` as a name keep working. diff --git a/src/analyzer/mod.rs b/src/analyzer/mod.rs index 142c6355..7a15ac79 100644 --- a/src/analyzer/mod.rs +++ b/src/analyzer/mod.rs @@ -716,6 +716,38 @@ impl Analyzer { /// Report an undefined-name reference. Inside a `try` body this is a /// warning rather than a fatal error: the reference raises a catchable /// runtime error, which is documented behavior that programs rely on. + /// Analyze the body of an unbounded loop (`forever` / `main loop`). Shared by + /// both so the scope handling, flow tracking, and the handler-body error + /// demotion stay identical. Web-server main-loop bodies reference + /// handler-provided names the analyzer cannot model, so errors raised *inside* + /// the body are demoted to warnings (a latched budget breach stays fatal); + /// errors the caller raises about the loop itself are pushed before this runs + /// and are not swept up. + fn analyze_loop_body(&mut self, body: &[Statement]) { + let outer_scope = std::mem::take(&mut self.current_scope); + self.current_scope = Scope::with_parent(outer_scope); + + let flow_entry = self.flow_entry(); + self.push_mutation_frame(); + let errors_before = self.errors.len(); + for stmt in body { + self.analyze_statement(stmt); + } + if self.budget_error.is_none() { + let demoted: Vec<_> = self.errors.drain(errors_before..).collect(); + self.warnings.extend(demoted); + } + let flow_body = self.take_flow_branch(&flow_entry); + let mutated = self.pop_mutation_frame(); + self.join_flow_branches(&[flow_body, flow_entry]); + self.degrade_mutated_aliases(&mutated); + + let loop_scope = std::mem::take(&mut self.current_scope); + if let Some(parent) = loop_scope.parent { + self.current_scope = *parent; + } + } + fn report_undefined_name(&mut self, message: String, line: usize, column: usize) { let error = SemanticError::new(message, line, column); if self.try_depth > 0 { @@ -1197,33 +1229,41 @@ impl Analyzer { self.current_scope = *parent; } } - Statement::ForeverLoop { body, .. } | Statement::MainLoop { body, .. } => { - let outer_scope = std::mem::take(&mut self.current_scope); - self.current_scope = Scope::with_parent(outer_scope); - - let flow_entry = self.flow_entry(); - self.push_mutation_frame(); - let errors_before = self.errors.len(); - for stmt in body { - self.analyze_statement(stmt); - } - // Same demotion as the repeat forms: web-server main loops - // reference handler-provided names this analyzer cannot - // model, and these bodies were previously unanalyzed. A - // latched budget breach stays fatal (see the repeat forms). - if self.budget_error.is_none() { - let demoted: Vec<_> = self.errors.drain(errors_before..).collect(); - self.warnings.extend(demoted); - } - let flow_body = self.take_flow_branch(&flow_entry); - let mutated = self.pop_mutation_frame(); - self.join_flow_branches(&[flow_body, flow_entry]); - self.degrade_mutated_aliases(&mutated); - - let loop_scope = std::mem::take(&mut self.current_scope); - if let Some(parent) = loop_scope.parent { - self.current_scope = *parent; + Statement::ForeverLoop { body, .. } => { + self.analyze_loop_body(body); + } + Statement::MainLoop { + body, + concurrent, + line, + column, + } => { + // `main loop concurrently:` starts up to the concurrency cap of + // handler futures at once, each running the body from the top. Any + // statement before the first `wait for request` therefore runs + // once *per handler slot* before a single request is dequeued — + // speculative side effects the author almost never intends. Require + // the body to begin with `wait for request` so nothing runs before + // a request is in hand. (Serial `main loop` has no such hazard.) + // Emitted before the body is analyzed so it is NOT swept into the + // handler-body error demotion below. + if *concurrent + && !matches!( + body.first(), + Some(Statement::WaitForRequestStatement { .. }) + ) + { + self.errors.push(SemanticError::new( + "A `main loop concurrently:` body must begin with `wait for request ...`. \ + Concurrent handlers start before any request arrives, so statements before \ + the first `wait for request` run once per handler slot. Move that setup above \ + the loop, or make `wait for request` the first statement in the loop." + .to_string(), + *line, + *column, + )); } + self.analyze_loop_body(body); } Statement::DisplayStatement { value, .. } => { self.analyze_expression(value); @@ -4103,6 +4143,58 @@ mod tests { assert!(errors[0].message.contains("not defined")); } + #[test] + fn test_concurrent_main_loop_requires_wait_for_request_first() { + use crate::lexer::lex_wfl_with_positions; + use crate::parser::Parser; + + fn analyze_src(src: &str) -> Result<(), Vec> { + let program = Parser::new(&lex_wfl_with_positions(src)) + .parse() + .expect("parse"); + Analyzer::new().analyze(&program) + } + + // A concurrent loop whose body does NOT begin with `wait for request` + // would run its opening statements once per handler slot before any + // request arrives. Static analysis must reject it with an actionable error + // (and that error must survive the handler-body error demotion). + let bad = "listen on port 8080 as srv\n\ + main loop concurrently:\n \ + store tick as 1\n \ + wait for request comes in on srv as req\n \ + respond to req with \"ok\"\nend loop"; + let errs = + analyze_src(bad).expect_err("setup-before-wait concurrent loop must be rejected"); + assert!( + errs.iter() + .any(|e| e.message.contains("must begin with `wait for request")), + "expected the concurrent-loop ordering error, got: {errs:?}" + ); + + // The identical body under a plain serial `main loop` is fine — a serial + // loop runs one iteration at a time, so there is no speculative fan-out. + let serial = "listen on port 8080 as srv\n\ + main loop:\n \ + store tick as 1\n \ + wait for request comes in on srv as req\n \ + respond to req with \"ok\"\nend loop"; + assert!( + analyze_src(serial).is_ok(), + "serial main loop must not require wait-for-request first" + ); + + // A concurrent loop that DOES begin with `wait for request` is accepted. + let good = "listen on port 8080 as srv\n\ + main loop concurrently:\n \ + wait for request comes in on srv as req\n \ + respond to req with \"ok\"\nend loop"; + assert!( + analyze_src(good).is_ok(), + "concurrent loop starting with wait-for-request must analyze cleanly" + ); + } + // Issue #592: a bare unresolved name in a program that uses `include from` // may be a zero-argument action exposed by the included file at runtime, so // the `Expression::Variable` arm must relax to a non-fatal warning (like the diff --git a/src/typechecker/mod.rs b/src/typechecker/mod.rs index cfdf15b9..eda694d1 100644 --- a/src/typechecker/mod.rs +++ b/src/typechecker/mod.rs @@ -824,10 +824,7 @@ impl TypeChecker { } if let Some(headers) = headers { let headers_type = self.infer_expression_type(headers); - if !matches!( - headers_type, - Type::Map(_, _) | Type::Unknown | Type::Any | Type::Error - ) { + if !self.is_valid_header_map_type(&headers_type) { self.type_error( "HTTP headers must be a map of header names to values".to_string(), Some(Type::Map(Box::new(Type::Text), Box::new(Type::Any))), @@ -909,10 +906,7 @@ impl TypeChecker { } if let Some(headers) = headers { let headers_type = self.infer_expression_type(headers); - if !matches!( - headers_type, - Type::Map(_, _) | Type::Unknown | Type::Any | Type::Error - ) { + if !self.is_valid_header_map_type(&headers_type) { self.type_error( "HTTP headers must be a map of header names to values".to_string(), Some(Type::Map(Box::new(Type::Text), Box::new(Type::Any))), @@ -1019,10 +1013,7 @@ impl TypeChecker { } if let Some(headers) = headers { let headers_type = self.infer_expression_type(headers); - if !matches!( - headers_type, - Type::Map(_, _) | Type::Unknown | Type::Any | Type::Error - ) { + if !self.is_valid_header_map_type(&headers_type) { self.type_error( "Streaming response headers must be a map of header names to values" .to_string(), @@ -2567,10 +2558,7 @@ impl TypeChecker { // outbound HttpRequestStatement headers check. if let Some(headers_expr) = headers { let headers_type = self.infer_expression_type(headers_expr); - if !matches!( - headers_type, - Type::Map(_, _) | Type::Unknown | Type::Any | Type::Error - ) { + if !self.is_valid_header_map_type(&headers_type) { self.type_error( "Response headers must be a map of header names to values".to_string(), Some(Type::Map(Box::new(Type::Text), Box::new(Type::Any))), @@ -4672,6 +4660,22 @@ impl TypeChecker { .push(TypeError::new(message, expected, found, line, column)); } + /// Whether an inferred type is acceptable as an HTTP header map. Header names + /// must be text, so a map with a concretely-typed non-text key (e.g. + /// `Map`) is rejected. `Unknown`/`Any`/`Error` — whether as the + /// whole type or as the key type of a map whose key the checker could not pin + /// down (map literals often infer an unknown key) — are accepted so a header + /// set the checker cannot fully resolve is not falsely flagged. + fn is_valid_header_map_type(&self, ty: &Type) -> bool { + match ty { + Type::Unknown | Type::Any | Type::Error => true, + Type::Map(key, _) => { + matches!(**key, Type::Text | Type::Unknown | Type::Any | Type::Error) + } + _ => false, + } + } + fn are_types_compatible(&self, target_type: &Type, source_type: &Type) -> bool { #[allow(clippy::only_used_in_recursion)] let _self = self; // Suppress the warning for self parameter @@ -4739,6 +4743,43 @@ mod tests { use crate::parser::ast::{Argument, Expression, Literal, Parameter, Program, Statement, Type}; use std::sync::Arc; + #[test] + fn test_header_map_type_requires_text_keys() { + // HTTP header names must be text. The header-map validity check (shared by + // outbound HTTP, streaming-response, and `respond` header clauses) must + // reject a map with a concretely-typed non-text key, while accepting + // text-keyed maps and any map whose key the checker could not pin down. + let tc = TypeChecker::new(); + + // Accepted: text keys, or an unresolved/loose key type. + for ok in [ + Type::Map(Box::new(Type::Text), Box::new(Type::Any)), + Type::Map(Box::new(Type::Text), Box::new(Type::Text)), + Type::Map(Box::new(Type::Unknown), Box::new(Type::Unknown)), + Type::Map(Box::new(Type::Any), Box::new(Type::Text)), + Type::Unknown, + Type::Any, + ] { + assert!( + tc.is_valid_header_map_type(&ok), + "expected {ok:?} to be a valid header map type" + ); + } + + // Rejected: a map with a concrete non-text key, or a non-map entirely. + for bad in [ + Type::Map(Box::new(Type::Number), Box::new(Type::Text)), + Type::Map(Box::new(Type::Boolean), Box::new(Type::Any)), + Type::Number, + Type::Text, + ] { + assert!( + !tc.is_valid_header_map_type(&bad), + "expected {bad:?} to be rejected as a header map type" + ); + } + } + #[test] fn test_variable_declaration_type_inference() { let program = Program { From d86720a8bd5b70c045341f2c4b9d82a5ea05662b Mon Sep 17 00:00:00 2001 From: Claude Date: Fri, 24 Jul 2026 03:52:21 +0000 Subject: [PATCH 031/132] docs: restore report_undefined_name doc comment to its function The analyze_loop_body extraction inserted the new function between report_undefined_name's doc comment and its definition, orphaning the doc above the wrong function. Move it back so each function documents itself. Co-Authored-By: Claude Opus 4.8 Claude-Session: https://claude.ai/code/session_011LxXAm5E4Vy39kRLr8kRnX --- src/analyzer/mod.rs | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/src/analyzer/mod.rs b/src/analyzer/mod.rs index 7a15ac79..4996ea23 100644 --- a/src/analyzer/mod.rs +++ b/src/analyzer/mod.rs @@ -713,9 +713,6 @@ impl Analyzer { self.budget_error.take() } - /// Report an undefined-name reference. Inside a `try` body this is a - /// warning rather than a fatal error: the reference raises a catchable - /// runtime error, which is documented behavior that programs rely on. /// Analyze the body of an unbounded loop (`forever` / `main loop`). Shared by /// both so the scope handling, flow tracking, and the handler-body error /// demotion stay identical. Web-server main-loop bodies reference @@ -748,6 +745,9 @@ impl Analyzer { } } + /// Report an undefined-name reference. Inside a `try` body this is a + /// warning rather than a fatal error: the reference raises a catchable + /// runtime error, which is documented behavior that programs rely on. fn report_undefined_name(&mut self, message: String, line: usize, column: usize) { let error = SemanticError::new(message, line, column); if self.try_depth > 0 { From 6ea80e22acfbfb4d3dc859e248604c6455804f6b Mon Sep 17 00:00:00 2001 From: Claude Date: Fri, 24 Jul 2026 04:00:39 +0000 Subject: [PATCH 032/132] fix(ci): give Start-Process distinct stdout/stderr targets in web test runner Enabling the Windows web-server test gate surfaced a pre-existing bug in run_web_tests.ps1: all three Start-Process calls passed "NUL" for both -RedirectStandardOutput and -RedirectStandardError, which PowerShell 7 rejects ("RedirectStandardOutput and RedirectStandardError are same"), failing the job before any test ran. Redirect the discarded server output to two distinct temp files instead (the TLS case uses its already-cleaned temp dir). Co-Authored-By: Claude Opus 4.8 Claude-Session: https://claude.ai/code/session_011LxXAm5E4Vy39kRLr8kRnX --- scripts/run_web_tests.ps1 | 18 ++++++++++++++---- 1 file changed, 14 insertions(+), 4 deletions(-) diff --git a/scripts/run_web_tests.ps1 b/scripts/run_web_tests.ps1 index 9ab1125e..ef034043 100644 --- a/scripts/run_web_tests.ps1 +++ b/scripts/run_web_tests.ps1 @@ -53,8 +53,12 @@ function Test-WflWebServer { Write-Host "" Write-Host "[INFO] Testing: $testName on port $Port" -ForegroundColor Blue - # Start the WFL server in background - $serverProcess = Start-Process -FilePath ".\$BinaryPath" -ArgumentList $TestFile -NoNewWindow -PassThru -RedirectStandardOutput "NUL" -RedirectStandardError "NUL" + # Start the WFL server in background. Start-Process rejects the same target + # for both redirects (the "NUL"/"NUL" collision errored on PowerShell 7), so + # discard stdout and stderr to two distinct, port-keyed temp files. + $outLog = Join-Path ([System.IO.Path]::GetTempPath()) "wfl_web_$Port.out.log" + $errLog = Join-Path ([System.IO.Path]::GetTempPath()) "wfl_web_$Port.err.log" + $serverProcess = Start-Process -FilePath ".\$BinaryPath" -ArgumentList $TestFile -NoNewWindow -PassThru -RedirectStandardOutput $outLog -RedirectStandardError $errLog try { # Wait for server to start (with retries) @@ -131,7 +135,11 @@ if (Test-Path "TestPrograms\web_route_params_test.wfl") { Write-Host "" Write-Host "[INFO] Testing: web_route_params_test.wfl on port 8096" -ForegroundColor Blue - $routeProcess = Start-Process -FilePath ".\$BinaryPath" -ArgumentList "TestPrograms\web_route_params_test.wfl" -NoNewWindow -PassThru -RedirectStandardOutput "NUL" -RedirectStandardError "NUL" + # Distinct redirect targets (see the note in Test-WflWebServer): the same + # path for both streams errors on PowerShell 7. + $routeOutLog = Join-Path ([System.IO.Path]::GetTempPath()) "wfl_web_route.out.log" + $routeErrLog = Join-Path ([System.IO.Path]::GetTempPath()) "wfl_web_route.err.log" + $routeProcess = Start-Process -FilePath ".\$BinaryPath" -ArgumentList "TestPrograms\web_route_params_test.wfl" -NoNewWindow -PassThru -RedirectStandardOutput $routeOutLog -RedirectStandardError $routeErrLog try { $serverReady = $false @@ -215,7 +223,9 @@ if (Test-Path "TestPrograms\web_server_tls.wfl") { # The test program uses relative cert paths, so run it from the temp dir $absBinary = Join-Path (Get-Location) $BinaryPath $absTest = Join-Path (Get-Location) "TestPrograms\web_server_tls.wfl" - $tlsProcess = Start-Process -FilePath $absBinary -ArgumentList $absTest -WorkingDirectory $tlsDir -NoNewWindow -PassThru -RedirectStandardOutput "NUL" -RedirectStandardError "NUL" + # Distinct redirect targets under the (auto-cleaned) temp dir; the same + # path for both streams errors on PowerShell 7. + $tlsProcess = Start-Process -FilePath $absBinary -ArgumentList $absTest -WorkingDirectory $tlsDir -NoNewWindow -PassThru -RedirectStandardOutput (Join-Path $tlsDir "server.out.log") -RedirectStandardError (Join-Path $tlsDir "server.err.log") try { # Probe readiness via the redirect port: it answers natively and does From 59d4809ba56ac136a4649db1394d664457ca09f4 Mon Sep 17 00:00:00 2001 From: Claude Date: Fri, 24 Jul 2026 04:16:46 +0000 Subject: [PATCH 033/132] fix(ci): address WFL server on 127.0.0.1 (not localhost) in Windows web tests The Windows web-test gate got past the redirect-collision fix and revealed the real cause of the timeouts: WFL binds 127.0.0.1 (IPv4 only) by default, but the runner addressed the server as http://localhost. On Windows, localhost resolves to IPv6 ::1 first, so Invoke-WebRequest never connected and every readiness probe/request timed out. The Linux .sh runner uses curl, which resolves localhost->127.0.0.1, so it never hit this. Address the server by 127.0.0.1 everywhere (the redirect Location mirrors the Host header, so its expected value changes to match; the self-signed cert CN is irrelevant under -SkipCertificateCheck). Also per maintainer review, to make the gate trustworthy/debuggable: - Dump the server's captured stdout/stderr on every failure path (Show-ServerLogs) so a real server error is not hidden behind a bare TIMEOUT. - Kill AND WaitForExit before removing the TLS temp dir (Stop-ServerProcess) to avoid a Windows cleanup race on the cert/log file handles. - Wrap the route-test requests so a request failure fails that test (and dumps logs) instead of throwing out of the script and skipping the summary. Co-Authored-By: Claude Opus 4.8 Claude-Session: https://claude.ai/code/session_011LxXAm5E4Vy39kRLr8kRnX --- scripts/run_web_tests.ps1 | 116 ++++++++++++++++++++++++++++---------- 1 file changed, 85 insertions(+), 31 deletions(-) diff --git a/scripts/run_web_tests.ps1 b/scripts/run_web_tests.ps1 index ef034043..1e27e0ad 100644 --- a/scripts/run_web_tests.ps1 +++ b/scripts/run_web_tests.ps1 @@ -40,6 +40,42 @@ if (-not (Test-Path $BinaryPath)) { } Write-Host "[SUCCESS] Binary found: $BinaryPath" -ForegroundColor Green +# Dump a server's captured stdout/stderr on failure. Without this a genuine +# server error (a panic, a bind failure, a bad response) is invisible behind the +# runner's generic TIMEOUT/assertion message, since the process output is +# redirected to files. Call on every failure path before returning. +function Show-ServerLogs { + param( + [string]$OutLog, + [string]$ErrLog + ) + foreach ($pair in @(@("stdout", $OutLog), @("stderr", $ErrLog))) { + $label = $pair[0] + $path = $pair[1] + if ($path -and (Test-Path $path)) { + $content = (Get-Content -Raw -ErrorAction SilentlyContinue $path) + if ([string]::IsNullOrWhiteSpace($content)) { + Write-Host "[LOG] server $label ($path): " -ForegroundColor Gray + } else { + Write-Host "[LOG] server $label ($path):" -ForegroundColor Gray + Write-Host $content -ForegroundColor Gray + } + } + } +} + +# Kill a background server process and wait for it to actually exit, so any temp +# files/handles it holds are released before the caller removes them (avoids +# Windows cleanup races on the TLS temp dir). +function Stop-ServerProcess { + param($Process) + if ($Process -and -not $Process.HasExited) { + $Process.Kill() + try { $Process.WaitForExit(5000) | Out-Null } catch { } + Write-Host "[INFO] Server process terminated" -ForegroundColor Gray + } +} + # Function to test a web server function Test-WflWebServer { param( @@ -72,7 +108,7 @@ function Test-WflWebServer { # Try to connect try { - $response = Invoke-WebRequest -Uri "http://localhost:$Port/" -TimeoutSec 2 -UseBasicParsing -ErrorAction Stop + $response = Invoke-WebRequest -Uri "http://127.0.0.1:$Port/" -TimeoutSec 2 -UseBasicParsing -ErrorAction Stop $serverReady = $true } catch { # Server not ready yet, continue waiting @@ -81,6 +117,7 @@ function Test-WflWebServer { if (-not $serverReady) { Write-Host "[ERROR] TIMEOUT: Server did not start within ${TimeoutSeconds}s" -ForegroundColor Red + Show-ServerLogs -OutLog $outLog -ErrLog $errLog return $false } @@ -92,13 +129,13 @@ function Test-WflWebServer { Write-Host "[ERROR] FAIL: Unexpected response" -ForegroundColor Red Write-Host " Expected: $ExpectedResponse" -ForegroundColor Gray Write-Host " Got: $($response.Content)" -ForegroundColor Gray + Show-ServerLogs -OutLog $outLog -ErrLog $errLog return $false } } finally { - # Clean up - kill the server + # Clean up - kill the server and wait for exit if (-not $serverProcess.HasExited) { - $serverProcess.Kill() - Write-Host "[INFO] Server process terminated" -ForegroundColor Gray + Stop-ServerProcess -Process $serverProcess } } } @@ -150,7 +187,7 @@ if (Test-Path "TestPrograms\web_route_params_test.wfl") { Start-Sleep -Milliseconds 500 $retries++ try { - $rootResponse = Invoke-WebRequest -Uri "http://localhost:8096/" -TimeoutSec 2 -UseBasicParsing -ErrorAction Stop + $rootResponse = Invoke-WebRequest -Uri "http://127.0.0.1:8096/" -TimeoutSec 2 -UseBasicParsing -ErrorAction Stop if ($rootResponse.Content -like "*Route server ready*") { $serverReady = $true } @@ -164,18 +201,25 @@ if (Test-Path "TestPrograms\web_route_params_test.wfl") { Write-Host "[ERROR] TIMEOUT: Route params server did not start within ${Timeout}s" -ForegroundColor Red $routeOk = $false } else { - # Route parameter extraction: /users/:id - $userResponse = Invoke-WebRequest -Uri "http://localhost:8096/users/42" -TimeoutSec 2 -UseBasicParsing - if ($userResponse.Content -like "*User 42*") { - Write-Host "[SUCCESS] PASS: /users/42 -> '$($userResponse.Content)'" -ForegroundColor Green - } else { - Write-Host "[ERROR] FAIL: /users/42 returned '$($userResponse.Content)'" -ForegroundColor Red + # Route parameter extraction: /users/:id. Wrapped so a request + # failure marks the test failed (and dumps server logs below) instead + # of throwing out of the script and skipping the summary/other tests. + try { + $userResponse = Invoke-WebRequest -Uri "http://127.0.0.1:8096/users/42" -TimeoutSec 2 -UseBasicParsing -ErrorAction Stop + if ($userResponse.Content -like "*User 42*") { + Write-Host "[SUCCESS] PASS: /users/42 -> '$($userResponse.Content)'" -ForegroundColor Green + } else { + Write-Host "[ERROR] FAIL: /users/42 returned '$($userResponse.Content)'" -ForegroundColor Red + $routeOk = $false + } + } catch { + Write-Host "[ERROR] FAIL: /users/42 request failed: $_" -ForegroundColor Red $routeOk = $false } # Non-matching route returns 404 try { - Invoke-WebRequest -Uri "http://localhost:8096/missing" -TimeoutSec 2 -UseBasicParsing -ErrorAction Stop | Out-Null + Invoke-WebRequest -Uri "http://127.0.0.1:8096/missing" -TimeoutSec 2 -UseBasicParsing -ErrorAction Stop | Out-Null Write-Host "[ERROR] FAIL: unknown route did not return 404" -ForegroundColor Red $routeOk = $false } catch { @@ -188,21 +232,27 @@ if (Test-Path "TestPrograms\web_route_params_test.wfl") { } # Header access regression - $agentResponse = Invoke-WebRequest -Uri "http://localhost:8096/agent" -TimeoutSec 2 -UseBasicParsing -UserAgent "wfl-route-test" - if ($agentResponse.Content -like "*wfl-route-test*") { - Write-Host "[SUCCESS] PASS: header access echoes User-Agent" -ForegroundColor Green - } else { - Write-Host "[ERROR] FAIL: /agent returned '$($agentResponse.Content)'" -ForegroundColor Red + try { + $agentResponse = Invoke-WebRequest -Uri "http://127.0.0.1:8096/agent" -TimeoutSec 2 -UseBasicParsing -UserAgent "wfl-route-test" -ErrorAction Stop + if ($agentResponse.Content -like "*wfl-route-test*") { + Write-Host "[SUCCESS] PASS: header access echoes User-Agent" -ForegroundColor Green + } else { + Write-Host "[ERROR] FAIL: /agent returned '$($agentResponse.Content)'" -ForegroundColor Red + $routeOk = $false + } + } catch { + Write-Host "[ERROR] FAIL: /agent request failed: $_" -ForegroundColor Red $routeOk = $false } } - if ($routeOk) { $passedTests++ } - } finally { - if (-not $routeProcess.HasExited) { - $routeProcess.Kill() - Write-Host "[INFO] Server process terminated" -ForegroundColor Gray + if ($routeOk) { + $passedTests++ + } else { + Show-ServerLogs -OutLog $routeOutLog -ErrLog $routeErrLog } + } finally { + Stop-ServerProcess -Process $routeProcess } } @@ -237,7 +287,7 @@ if (Test-Path "TestPrograms\web_server_tls.wfl") { Start-Sleep -Milliseconds 500 $retries++ try { - Invoke-WebRequest -Uri "http://localhost:8090/" -TimeoutSec 2 -UseBasicParsing -MaximumRedirection 0 -ErrorAction Stop | Out-Null + Invoke-WebRequest -Uri "http://127.0.0.1:8090/" -TimeoutSec 2 -UseBasicParsing -MaximumRedirection 0 -ErrorAction Stop | Out-Null $serverReady = $true } catch { if ($_.Exception.Response -and [int]$_.Exception.Response.StatusCode -eq 301) { @@ -254,14 +304,14 @@ if (Test-Path "TestPrograms\web_server_tls.wfl") { # Redirect server: 301 with Location preserving path/query on the HTTPS port $location = $null try { - $redirectResponse = Invoke-WebRequest -Uri "http://localhost:8090/some/path?x=1" -TimeoutSec 2 -UseBasicParsing -MaximumRedirection 0 -ErrorAction Stop + $redirectResponse = Invoke-WebRequest -Uri "http://127.0.0.1:8090/some/path?x=1" -TimeoutSec 2 -UseBasicParsing -MaximumRedirection 0 -ErrorAction Stop $location = $redirectResponse.Headers["Location"] } catch { if ($_.Exception.Response) { $location = $_.Exception.Response.Headers["Location"] } } - if ($location -eq "https://localhost:8443/some/path?x=1") { + if ($location -eq "https://127.0.0.1:8443/some/path?x=1") { Write-Host "[SUCCESS] PASS: redirect returns 301 to $location" -ForegroundColor Green } else { Write-Host "[ERROR] FAIL: redirect Location was '$location'" -ForegroundColor Red @@ -273,7 +323,7 @@ if (Test-Path "TestPrograms\web_server_tls.wfl") { # PowerShell 5.1 skip this check gracefully instead of failing. if ($PSVersionTable.PSVersion.Major -ge 6) { try { - $httpsResponse = Invoke-WebRequest -Uri "https://localhost:8443/" -TimeoutSec 3 -UseBasicParsing -SkipCertificateCheck -ErrorAction Stop + $httpsResponse = Invoke-WebRequest -Uri "https://127.0.0.1:8443/" -TimeoutSec 3 -UseBasicParsing -SkipCertificateCheck -ErrorAction Stop if ($httpsResponse.Content -like "*Hello over HTTPS!*") { Write-Host "[SUCCESS] PASS: HTTPS response '$($httpsResponse.Content)'" -ForegroundColor Green } else { @@ -289,12 +339,16 @@ if (Test-Path "TestPrograms\web_server_tls.wfl") { } } - if ($tlsOk) { $passedTests++ } - } finally { - if (-not $tlsProcess.HasExited) { - $tlsProcess.Kill() - Write-Host "[INFO] Server process terminated" -ForegroundColor Gray + if ($tlsOk) { + $passedTests++ + } else { + Show-ServerLogs -OutLog (Join-Path $tlsDir "server.out.log") -ErrLog (Join-Path $tlsDir "server.err.log") } + } finally { + # Kill AND wait for exit before removing the temp dir, so the server + # has released its cert/log file handles (avoids a Windows cleanup + # race that would leave the dir or fail the Remove-Item). + Stop-ServerProcess -Process $tlsProcess Remove-Item -Recurse -Force $tlsDir -ErrorAction SilentlyContinue } } From a7c3a23195a89809f649161a25fe645782173108 Mon Sep 17 00:00:00 2001 From: Claude Date: Fri, 24 Jul 2026 04:23:02 +0000 Subject: [PATCH 034/132] fix(ci): deterministic flush + monotonic deadline + exit code in web tests Addresses the maintainer's remaining Windows-harness recommendations (the localhost->127.0.0.1 address-family fix and Kill/WaitForExit landed in 59d4809): - simple_web_test.wfl: add `close server test_server` after the one-shot respond. `respond` hands the reply to the async transport, so reaching EOF can tear the runtime down before the socket flushes; `close server` has a short grace that drains the pending response first, making single-shot delivery deterministic. - run_web_tests.ps1: replace the fixed retry-count readiness loops with a Stopwatch wall-clock deadline. The old '20 tries' loop could run ~50s for a '10s' wait (each failed 2s request stacked on the 500ms sleep); the deadline caps total wait at the advertised timeout. - run_web_tests.ps1: Show-ServerLogs now also reports the child process exit code (or that it was still running), so a failure shows the decisive server-side evidence, not just its stdout/stderr. Verified simple_web_test.wfl still parses and analyzes cleanly. Co-Authored-By: Claude Opus 4.8 Claude-Session: https://claude.ai/code/session_011LxXAm5E4Vy39kRLr8kRnX --- TestPrograms/simple_web_test.wfl | 7 ++++ scripts/run_web_tests.ps1 | 55 +++++++++++++++++--------------- 2 files changed, 36 insertions(+), 26 deletions(-) diff --git a/TestPrograms/simple_web_test.wfl b/TestPrograms/simple_web_test.wfl index e72f9561..14e0682d 100644 --- a/TestPrograms/simple_web_test.wfl +++ b/TestPrograms/simple_web_test.wfl @@ -14,4 +14,11 @@ display "Got a request!" respond to incoming_request with "Hello from WFL!" display "Response sent!" + +// `respond` hands the reply to the async transport; reaching EOF immediately +// can tear the runtime down before the socket flushes. `close server` has a +// short grace that lets the pending response drain first, so the single-shot +// client reliably receives the body. +close server test_server + display "=== Test Complete ===" diff --git a/scripts/run_web_tests.ps1 b/scripts/run_web_tests.ps1 index 1e27e0ad..3d500b62 100644 --- a/scripts/run_web_tests.ps1 +++ b/scripts/run_web_tests.ps1 @@ -47,8 +47,16 @@ Write-Host "[SUCCESS] Binary found: $BinaryPath" -ForegroundColor Green function Show-ServerLogs { param( [string]$OutLog, - [string]$ErrLog + [string]$ErrLog, + $Process ) + if ($Process) { + if ($Process.HasExited) { + Write-Host "[LOG] server process exited with code $($Process.ExitCode)" -ForegroundColor Gray + } else { + Write-Host "[LOG] server process was still running at failure time" -ForegroundColor Gray + } + } foreach ($pair in @(@("stdout", $OutLog), @("stderr", $ErrLog))) { $label = $pair[0] $path = $pair[1] @@ -97,27 +105,25 @@ function Test-WflWebServer { $serverProcess = Start-Process -FilePath ".\$BinaryPath" -ArgumentList $TestFile -NoNewWindow -PassThru -RedirectStandardOutput $outLog -RedirectStandardError $errLog try { - # Wait for server to start (with retries) + # Wait for the server to start, bounded by a real wall-clock deadline. + # A fixed retry count misleads: each failed 2s request stacks on top of + # the 500ms sleep, so "20 tries" of a 10s wait could actually run ~50s. + # A Stopwatch caps total wait at TimeoutSeconds (plus one in-flight probe). $serverReady = $false - $retries = 0 - $maxRetries = $TimeoutSeconds * 2 # Check every 500ms - - while (-not $serverReady -and $retries -lt $maxRetries) { - Start-Sleep -Milliseconds 500 - $retries++ - - # Try to connect + $deadline = [System.Diagnostics.Stopwatch]::StartNew() + while (-not $serverReady -and $deadline.Elapsed.TotalSeconds -lt $TimeoutSeconds) { try { $response = Invoke-WebRequest -Uri "http://127.0.0.1:$Port/" -TimeoutSec 2 -UseBasicParsing -ErrorAction Stop $serverReady = $true } catch { - # Server not ready yet, continue waiting + # Server not ready yet, wait briefly and retry until the deadline. + Start-Sleep -Milliseconds 500 } } if (-not $serverReady) { Write-Host "[ERROR] TIMEOUT: Server did not start within ${TimeoutSeconds}s" -ForegroundColor Red - Show-ServerLogs -OutLog $outLog -ErrLog $errLog + Show-ServerLogs -OutLog $outLog -ErrLog $errLog -Process $serverProcess return $false } @@ -129,7 +135,7 @@ function Test-WflWebServer { Write-Host "[ERROR] FAIL: Unexpected response" -ForegroundColor Red Write-Host " Expected: $ExpectedResponse" -ForegroundColor Gray Write-Host " Got: $($response.Content)" -ForegroundColor Gray - Show-ServerLogs -OutLog $outLog -ErrLog $errLog + Show-ServerLogs -OutLog $outLog -ErrLog $errLog -Process $serverProcess return $false } } finally { @@ -179,13 +185,10 @@ if (Test-Path "TestPrograms\web_route_params_test.wfl") { $routeProcess = Start-Process -FilePath ".\$BinaryPath" -ArgumentList "TestPrograms\web_route_params_test.wfl" -NoNewWindow -PassThru -RedirectStandardOutput $routeOutLog -RedirectStandardError $routeErrLog try { + # Wall-clock deadline (see Test-WflWebServer) instead of a retry count. $serverReady = $false - $retries = 0 - $maxRetries = $Timeout * 2 - - while (-not $serverReady -and $retries -lt $maxRetries) { - Start-Sleep -Milliseconds 500 - $retries++ + $deadline = [System.Diagnostics.Stopwatch]::StartNew() + while (-not $serverReady -and $deadline.Elapsed.TotalSeconds -lt $Timeout) { try { $rootResponse = Invoke-WebRequest -Uri "http://127.0.0.1:8096/" -TimeoutSec 2 -UseBasicParsing -ErrorAction Stop if ($rootResponse.Content -like "*Route server ready*") { @@ -194,6 +197,7 @@ if (Test-Path "TestPrograms\web_route_params_test.wfl") { } catch { # Intentionally empty - server not ready yet, continue polling } + if (-not $serverReady) { Start-Sleep -Milliseconds 500 } } $routeOk = $true @@ -249,7 +253,7 @@ if (Test-Path "TestPrograms\web_route_params_test.wfl") { if ($routeOk) { $passedTests++ } else { - Show-ServerLogs -OutLog $routeOutLog -ErrLog $routeErrLog + Show-ServerLogs -OutLog $routeOutLog -ErrLog $routeErrLog -Process $routeProcess } } finally { Stop-ServerProcess -Process $routeProcess @@ -280,12 +284,10 @@ if (Test-Path "TestPrograms\web_server_tls.wfl") { try { # Probe readiness via the redirect port: it answers natively and does # not consume the program's single `wait for request` + # Wall-clock deadline (see Test-WflWebServer) instead of a retry count. $serverReady = $false - $retries = 0 - $maxRetries = $Timeout * 2 - while (-not $serverReady -and $retries -lt $maxRetries) { - Start-Sleep -Milliseconds 500 - $retries++ + $deadline = [System.Diagnostics.Stopwatch]::StartNew() + while (-not $serverReady -and $deadline.Elapsed.TotalSeconds -lt $Timeout) { try { Invoke-WebRequest -Uri "http://127.0.0.1:8090/" -TimeoutSec 2 -UseBasicParsing -MaximumRedirection 0 -ErrorAction Stop | Out-Null $serverReady = $true @@ -294,6 +296,7 @@ if (Test-Path "TestPrograms\web_server_tls.wfl") { $serverReady = $true } } + if (-not $serverReady) { Start-Sleep -Milliseconds 500 } } $tlsOk = $true @@ -342,7 +345,7 @@ if (Test-Path "TestPrograms\web_server_tls.wfl") { if ($tlsOk) { $passedTests++ } else { - Show-ServerLogs -OutLog (Join-Path $tlsDir "server.out.log") -ErrLog (Join-Path $tlsDir "server.err.log") + Show-ServerLogs -OutLog (Join-Path $tlsDir "server.out.log") -ErrLog (Join-Path $tlsDir "server.err.log") -Process $tlsProcess } } finally { # Kill AND wait for exit before removing the temp dir, so the server From 4a678bdc1662dc2d7630b68864aba47a93a77293 Mon Sep 17 00:00:00 2001 From: Claude Date: Fri, 24 Jul 2026 04:32:58 +0000 Subject: [PATCH 035/132] fix: robust redirect Location extraction; exact streaming dispatch; header value types Three review-driven fixes: - run_web_tests.ps1 (Windows CI): the 127.0.0.1 fix got 2/3 web tests passing; the TLS redirect assertion still failed with an empty Location because pwsh 7 throws on -MaximumRedirection 0 and $_.Exception.Response is an HttpResponseMessage whose Headers has no string indexer (so ["Location"] is $null). Add Get-LocationHeader that reads the strongly-typed HttpResponseHeaders.Location (Uri) for that shape and the case-insensitive string indexer otherwise. - parser (Copilot): restrict the `start streaming response` dispatch to the EXACT Identifier("streaming") token instead of starts_with("streaming "). `response` is a keyword so `streaming` never merges; the prefix form would hijack an unrelated `start streaming ...` into a confusing parse error. - typechecker (Copilot): is_valid_header_map_type now also validates the map VALUE type. The interpreter accepts only text/number/bool header values (stringified) and rejects the rest, so Map (etc.) is flagged; Unknown/Any/Error keys and values still pass to avoid false positives. Co-Authored-By: Claude Opus 4.8 Claude-Session: https://claude.ai/code/session_011LxXAm5E4Vy39kRLr8kRnX --- scripts/run_web_tests.ps1 | 25 +++++++++++++++++++++-- src/parser/mod.rs | 14 ++++++++----- src/typechecker/mod.rs | 43 ++++++++++++++++++++++++++++++--------- 3 files changed, 65 insertions(+), 17 deletions(-) diff --git a/scripts/run_web_tests.ps1 b/scripts/run_web_tests.ps1 index 3d500b62..93e03241 100644 --- a/scripts/run_web_tests.ps1 +++ b/scripts/run_web_tests.ps1 @@ -84,6 +84,27 @@ function Stop-ServerProcess { } } +# Extract the Location header from a redirect response across PowerShell/response +# shapes. With -MaximumRedirection 0, pwsh 7 throws and $_.Exception.Response is +# an HttpResponseMessage whose Headers has NO string indexer, so ["Location"] +# silently returns $null; its Location is the strongly-typed .Headers.Location +# (a Uri). Invoke-WebRequest's own response object (and Windows PowerShell 5.1's +# HttpWebResponse) use a string indexer instead, whose value may be a string[]. +function Get-LocationHeader { + param($Response) + if ($null -eq $Response) { return $null } + if ($Response -is [System.Net.Http.HttpResponseMessage]) { + if ($Response.Headers.Location) { return $Response.Headers.Location.AbsoluteUri } + return $null + } + try { + $loc = $Response.Headers["Location"] + if ($loc -is [array]) { $loc = $loc[0] } + if ($loc) { return [string]$loc } + } catch { } + return $null +} + # Function to test a web server function Test-WflWebServer { param( @@ -308,10 +329,10 @@ if (Test-Path "TestPrograms\web_server_tls.wfl") { $location = $null try { $redirectResponse = Invoke-WebRequest -Uri "http://127.0.0.1:8090/some/path?x=1" -TimeoutSec 2 -UseBasicParsing -MaximumRedirection 0 -ErrorAction Stop - $location = $redirectResponse.Headers["Location"] + $location = Get-LocationHeader -Response $redirectResponse } catch { if ($_.Exception.Response) { - $location = $_.Exception.Response.Headers["Location"] + $location = Get-LocationHeader -Response $_.Exception.Response } } if ($location -eq "https://127.0.0.1:8443/some/path?x=1") { diff --git a/src/parser/mod.rs b/src/parser/mod.rs index 471c7a12..7c96e952 100644 --- a/src/parser/mod.rs +++ b/src/parser/mod.rs @@ -612,14 +612,18 @@ impl<'a> StmtParser<'a> for Parser<'a> { // words (and a bare identifier message) lex as one merged token. // `start streaming response to ... as `. `start` is a // keyword; `streaming` is a contextual identifier; `response` is - // a keyword. Only intercept when `streaming` actually follows, so - // any other statement-initial use of the `start` keyword (e.g. - // the `start of text` pattern anchor) is not hijacked and can - // fall through to its own handling. + // a keyword. `response` being a keyword means it never merges into + // the identifier, so `streaming` always arrives as the *exact* + // token `Identifier("streaming")`. Match only that exact token — + // NOT `starts_with("streaming ")`, which would hijack an unrelated + // `start streaming ...` (the lexer merges those into + // `Identifier("streaming ")`) into a confusing parse error. + // Any other statement-initial use of `start` (e.g. the + // `start of text` pattern anchor) falls through to its own handler. Token::KeywordStart if matches!( self.cursor.peek_next().map(|t| &t.token), - Some(Token::Identifier(id)) if id == "streaming" || id.starts_with("streaming ") + Some(Token::Identifier(id)) if id == "streaming" ) => { self.parse_start_streaming_response() diff --git a/src/typechecker/mod.rs b/src/typechecker/mod.rs index eda694d1..c015f0d6 100644 --- a/src/typechecker/mod.rs +++ b/src/typechecker/mod.rs @@ -4661,16 +4661,29 @@ impl TypeChecker { } /// Whether an inferred type is acceptable as an HTTP header map. Header names - /// must be text, so a map with a concretely-typed non-text key (e.g. - /// `Map`) is rejected. `Unknown`/`Any`/`Error` — whether as the - /// whole type or as the key type of a map whose key the checker could not pin - /// down (map literals often infer an unknown key) — are accepted so a header - /// set the checker cannot fully resolve is not falsely flagged. + /// must be text, and header values are what the interpreter accepts and + /// stringifies — text, number, or boolean (see the `respond`/HTTP header + /// handling); a concretely-typed non-text key or a value type the runtime + /// rejects (e.g. `Map`) is flagged. `Unknown`/`Any`/`Error` — + /// whether as the whole type or as the key/value type of a map the checker + /// could not fully pin down (map literals often infer unknown key/value + /// types) — are always accepted so a header set the checker cannot resolve is + /// never falsely flagged. fn is_valid_header_map_type(&self, ty: &Type) -> bool { match ty { Type::Unknown | Type::Any | Type::Error => true, - Type::Map(key, _) => { - matches!(**key, Type::Text | Type::Unknown | Type::Any | Type::Error) + Type::Map(key, value) => { + let key_ok = matches!(**key, Type::Text | Type::Unknown | Type::Any | Type::Error); + let value_ok = matches!( + **value, + Type::Text + | Type::Number + | Type::Boolean + | Type::Unknown + | Type::Any + | Type::Error + ); + key_ok && value_ok } _ => false, } @@ -4751,10 +4764,14 @@ mod tests { // text-keyed maps and any map whose key the checker could not pin down. let tc = TypeChecker::new(); - // Accepted: text keys, or an unresolved/loose key type. + // Accepted: text keys with a value type the runtime stringifies + // (text/number/bool), or an unresolved/loose key or value type. for ok in [ - Type::Map(Box::new(Type::Text), Box::new(Type::Any)), Type::Map(Box::new(Type::Text), Box::new(Type::Text)), + Type::Map(Box::new(Type::Text), Box::new(Type::Number)), + Type::Map(Box::new(Type::Text), Box::new(Type::Boolean)), + Type::Map(Box::new(Type::Text), Box::new(Type::Any)), + Type::Map(Box::new(Type::Text), Box::new(Type::Unknown)), Type::Map(Box::new(Type::Unknown), Box::new(Type::Unknown)), Type::Map(Box::new(Type::Any), Box::new(Type::Text)), Type::Unknown, @@ -4766,10 +4783,16 @@ mod tests { ); } - // Rejected: a map with a concrete non-text key, or a non-map entirely. + // Rejected: a concrete non-text key, a value type the runtime rejects + // (e.g. Binary or a nested list), or a non-map entirely. for bad in [ Type::Map(Box::new(Type::Number), Box::new(Type::Text)), Type::Map(Box::new(Type::Boolean), Box::new(Type::Any)), + Type::Map(Box::new(Type::Text), Box::new(Type::Binary)), + Type::Map( + Box::new(Type::Text), + Box::new(Type::List(Box::new(Type::Text))), + ), Type::Number, Type::Text, ] { From 9ea03c02babcc724f8df56cde4d5764913285152 Mon Sep 17 00:00:00 2001 From: Claude Date: Fri, 24 Jul 2026 05:09:13 +0000 Subject: [PATCH 036/132] fix: parse with-continuation in write line/chunk; harden Windows web runner MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Backward-compat (maintainer review): the merged `write line|chunk ` form expected `to` immediately after the identifier value, so a `with`/operator continuation was rejected — breaking `write line payload with "!" to out` AND the pre-existing classic file write `write line payload with "!" to file`. Expose parse_binary_continuation (the operator loop of parse_binary_expression) and use it to absorb the continuation onto the stream reading, then mirror it onto the classic file-write fallback by swapping the leftmost leaf operand (replace_leftmost_leaf). Both readings now carry the full expression. Windows web runner hardening (maintainer review): - Get-LocationHeader matches the HttpResponseMessage shape by type NAME, not the [System.Net.Http.HttpResponseMessage] type literal, which a clean PowerShell 5.1 process could fail to resolve before the fallback runs. - Stop-ServerProcess checks WaitForExit(5000)'s result and only reports "terminated" when the process actually exited (else warns), instead of always claiming success while a live process races temp-file cleanup. Tests: write_line_backcompat_test gains a parse test (continuation mirrored to both readings) and a runtime test (classic file write preserves the concatenation). Full parser suite + clippy green. Co-Authored-By: Claude Opus 4.8 Claude-Session: https://claude.ai/code/session_011LxXAm5E4Vy39kRLr8kRnX --- scripts/run_web_tests.ps1 | 18 +++++- src/parser/expr/binary.rs | 23 +++++++- src/parser/stmt/io.rs | 87 +++++++++++++++++++++++++---- tests/write_line_backcompat_test.rs | 80 +++++++++++++++++++++++++- 4 files changed, 192 insertions(+), 16 deletions(-) diff --git a/scripts/run_web_tests.ps1 b/scripts/run_web_tests.ps1 index 93e03241..b2714f3d 100644 --- a/scripts/run_web_tests.ps1 +++ b/scripts/run_web_tests.ps1 @@ -79,8 +79,16 @@ function Stop-ServerProcess { param($Process) if ($Process -and -not $Process.HasExited) { $Process.Kill() - try { $Process.WaitForExit(5000) | Out-Null } catch { } - Write-Host "[INFO] Server process terminated" -ForegroundColor Gray + # WaitForExit(ms) returns $true only if the process actually exited in + # time; report honestly rather than always claiming success (a process + # still alive can race temp-file/cert cleanup that follows). + $exited = $false + try { $exited = $Process.WaitForExit(5000) } catch { } + if ($exited) { + Write-Host "[INFO] Server process terminated" -ForegroundColor Gray + } else { + Write-Host "[WARN] Server process did not exit within 5s of Kill()" -ForegroundColor Yellow + } } } @@ -93,7 +101,11 @@ function Stop-ServerProcess { function Get-LocationHeader { param($Response) if ($null -eq $Response) { return $null } - if ($Response -is [System.Net.Http.HttpResponseMessage]) { + # Match the HttpResponseMessage shape by type NAME, not the type literal + # [System.Net.Http.HttpResponseMessage]: a clean Windows PowerShell 5.1 + # process may not have System.Net.Http loaded, so resolving the literal would + # throw before the HttpWebResponse string-indexer fallback below. + if ($Response.GetType().FullName -eq 'System.Net.Http.HttpResponseMessage') { if ($Response.Headers.Location) { return $Response.Headers.Location.AbsoluteUri } return $null } diff --git a/src/parser/expr/binary.rs b/src/parser/expr/binary.rs index c7192457..4dfbd0f5 100644 --- a/src/parser/expr/binary.rs +++ b/src/parser/expr/binary.rs @@ -21,6 +21,20 @@ pub(crate) trait BinaryExprParser<'a> { /// Returns an `Expression` representing the parsed binary expression, or a `ParseError` if the syntax is invalid. fn parse_binary_expression(&mut self, precedence: u8) -> Result; + /// Continue a binary expression from an already-parsed left-hand side. + /// + /// `parse_binary_expression` parses a fresh primary and then runs the + /// operator loop; this exposes just the loop so a caller that consumed the + /// leading operand itself (e.g. the merged `write line ` form) can + /// still absorb trailing `with`/operator continuations — so + /// `write line payload with "!" to out` parses its value like any other + /// expression instead of stopping at the bare variable. + fn parse_binary_continuation( + &mut self, + left: Expression, + precedence: u8, + ) -> Result; + /// Parses a function/action call expression. /// /// # Parameters @@ -53,8 +67,15 @@ pub(crate) trait BinaryExprParser<'a> { impl<'a> BinaryExprParser<'a> for Parser<'a> { fn parse_binary_expression(&mut self, precedence: u8) -> Result { - let mut left = self.parse_primary_expression()?; + let left = self.parse_primary_expression()?; + self.parse_binary_continuation(left, precedence) + } + fn parse_binary_continuation( + &mut self, + mut left: Expression, + precedence: u8, + ) -> Result { while let Some(token_pos) = self.cursor.peek() { let token = &token_pos.token; let line = token_pos.line; diff --git a/src/parser/stmt/io.rs b/src/parser/stmt/io.rs index e83c8808..9fa8b89d 100644 --- a/src/parser/stmt/io.rs +++ b/src/parser/stmt/io.rs @@ -3,9 +3,64 @@ use super::super::{Expression, FileOpenMode, Literal, ParseError, Parser, Statement}; use super::database::DatabaseParser; use crate::lexer::token::Token; -use crate::parser::expr::{ExprParser, PrimaryExprParser}; +use crate::parser::expr::{BinaryExprParser, ExprParser, PrimaryExprParser}; use std::sync::Arc; +/// Replace the leftmost leaf operand of a (possibly nested) expression. +/// +/// The merged `write line|chunk ...` form parses one value expression +/// for the stream reading; the classic file-write reading differs only in its +/// leading operand (the full merged `line ` variable instead of the +/// split ``). Rather than re-parse, we clone the parsed value and swap +/// its leftmost operand — so a trailing `with`/operator continuation applies to +/// both readings identically. `with`/binary chains here are right-associative +/// (`a with b with c` => `Concat(a, Concat(b, c))`) and ` of ` +/// is a `FunctionCall`, so the leading operand is always reached via `.left`, +/// `.function`, or the leaf itself. +fn replace_leftmost_leaf(expr: Expression, replacement: Expression) -> Expression { + match expr { + Expression::Concatenation { + left, + right, + line, + column, + } => Expression::Concatenation { + left: Box::new(replace_leftmost_leaf(*left, replacement)), + right, + line, + column, + }, + Expression::BinaryOperation { + left, + operator, + right, + line, + column, + } => Expression::BinaryOperation { + left: Box::new(replace_leftmost_leaf(*left, replacement)), + operator, + right, + line, + column, + }, + Expression::FunctionCall { + function, + arguments, + line, + column, + } => Expression::FunctionCall { + function: Box::new(replace_leftmost_leaf(*function, replacement)), + arguments, + line, + column, + }, + // Leaf (a bare `Variable`, the common case) or a form whose leading + // operand is not a nested `Expression` (e.g. an `ActionCall`, from the + // rare ` with ...`): swap wholesale. + _ => replacement, + } +} + pub(crate) trait IoParser<'a>: ExprParser<'a> { fn parse_display_statement(&mut self) -> Result; fn parse_open_file_statement(&mut self) -> Result; @@ -845,28 +900,38 @@ impl<'a> IoParser<'a> for Parser<'a> { (self.parse_expression()?, None) } else { // `` alone (stream) vs the full merged `line ` - // (classic file write of that variable). + // (classic file write of that variable). Build the stream + // reading's leading operand — ``, or ` of ` + // — then absorb any trailing `with`/operator continuation so the + // value parses like any other expression (a `write line payload + // with "!" to out` value, and the pre-existing classic file write + // `write line payload with "!" to file`, must not be truncated). let stream_left = Expression::Variable(rest, marker_line, marker_column); let file_left = Expression::Variable(id, marker_line, marker_column); - match self.cursor.peek().map(|t| &t.token) { + let value_lead = match self.cursor.peek().map(|t| &t.token) { // ` of `, e.g. `write line body of msg to out`. Some(Token::KeywordOf) => { self.bump_sync(); // Consume "of" let object = self.parse_primary_expression()?; - let of = |left: Expression| Expression::FunctionCall { - function: Box::new(left), + Expression::FunctionCall { + function: Box::new(stream_left), arguments: vec![crate::parser::ast::Argument { name: None, - value: object.clone(), + value: object, }], line: marker_line, column: marker_column, - }; - (of(stream_left), Some(Box::new(of(file_left)))) + } } - // A bare variable value: the next token starts `to ...`. - _ => (stream_left, Some(Box::new(file_left))), - } + // A bare variable value (possibly followed by `with ...`). + _ => stream_left, + }; + let value = self.parse_binary_continuation(value_lead, 0)?; + // The classic file-write reading is identical except its leading + // operand is the full merged `line `; mirror the parsed + // continuation onto it by swapping the leftmost leaf. + let fallback = replace_leftmost_leaf(value.clone(), file_left); + (value, Some(Box::new(fallback))) }; self.expect_token( diff --git a/tests/write_line_backcompat_test.rs b/tests/write_line_backcompat_test.rs index 1ee51f05..d7af571f 100644 --- a/tests/write_line_backcompat_test.rs +++ b/tests/write_line_backcompat_test.rs @@ -11,7 +11,7 @@ use wfl::Interpreter; use wfl::analyzer::Analyzer; use wfl::lexer::lex_wfl_with_positions; use wfl::parser::Parser; -use wfl::parser::ast::Statement; +use wfl::parser::ast::{Expression, Statement}; fn parse(code: &str) -> Vec { let tokens = lex_wfl_with_positions(code); @@ -82,6 +82,84 @@ fn test_write_line_multiword_variable_parses_with_fallback() { } } +#[test] +fn test_write_line_with_continuation_parses_for_both_readings() { + // `write line payload with "!" to out`: the value must absorb the `with` + // continuation for the stream reading, and the classic file-write fallback + // must mirror it (leading variable `line payload`, same continuation) — not + // truncate at the bare variable and fail at `with`. + let stmt = &parse(r#"write line payload with "!" to out"#)[0]; + match stmt { + Statement::StreamWriteStatement { + value, + fallback_content, + .. + } => { + // Stream reading: Concatenation(Variable("payload"), "!"). + match value { + Expression::Concatenation { left, .. } => match &**left { + Expression::Variable(name, ..) => assert_eq!(name, "payload"), + other => panic!("stream value left should be Variable(payload), got {other:?}"), + }, + other => panic!("stream value should be a Concatenation, got {other:?}"), + } + // Classic file-write fallback: Concatenation(Variable("line payload"), "!"). + let fb = fallback_content + .as_ref() + .expect("merged `write line with ...` keeps a file-write fallback"); + match &**fb { + Expression::Concatenation { left, .. } => match &**left { + Expression::Variable(name, ..) => assert_eq!(name, "line payload"), + other => { + panic!("fallback left should be Variable(line payload), got {other:?}") + } + }, + other => panic!("fallback should mirror the continuation, got {other:?}"), + } + } + other => panic!("expected StreamWriteStatement, got {other:?}"), + } +} + +#[test] +fn test_write_line_variable_with_continuation_to_file_preserves_concatenation() { + // A pre-existing classic file write with a `with` continuation on a variable + // literally named `line note`. Before continuation parsing this failed to + // parse (the interception expected `to` right after the variable); it must + // now write the concatenated value to the file. + let dir = tempfile::tempdir().expect("create temp dir"); + let path = dir.path().join("wfl_write_line_continuation.txt"); + let path_str = path.to_string_lossy().replace('\\', "/"); + + let code = format!( + r#"store line note as "kept" +write line note with "!" to "{path_str}""# + ); + + let program = { + let tokens = lex_wfl_with_positions(&code); + let mut parser = Parser::new(&tokens); + parser.parse().expect("parse") + }; + + let mut analyzer = Analyzer::new(); + analyzer + .analyze(&program) + .expect("analysis must accept `write line with ... to `"); + + let rt = tokio::runtime::Runtime::new().unwrap(); + rt.block_on(async { + let mut interp = Interpreter::new(); + interp.interpret(&program).await.expect("interpret"); + }); + + let contents = std::fs::read_to_string(&path).expect("output file should exist"); + assert_eq!( + contents, "kept!", + "the concatenated value (variable `line note` with \"!\") must reach the file" + ); +} + #[test] fn test_write_multiword_line_variable_to_file_still_works() { // A pre-existing program: a variable literally named `line note` written to a From 12dc3eaa7431576adbef5ca5d6e4e8cc5e700a01 Mon Sep 17 00:00:00 2001 From: Claude Date: Fri, 24 Jul 2026 05:14:13 +0000 Subject: [PATCH 037/132] docs: write line/chunk now supports with-concatenation directly MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The continuation-parsing fix makes the write value a full expression, so `write line prefix with json to out` works directly in the statement. Update the web-servers guide, which still said with-concatenation was 'not yet supported' and told users to build the value first — a now-stale claim that contradicts the parser and its tests. Co-Authored-By: Claude Opus 4.8 Claude-Session: https://claude.ai/code/session_011LxXAm5E4Vy39kRLr8kRnX --- Docs/04-advanced-features/web-servers.md | 12 +++++++----- 1 file changed, 7 insertions(+), 5 deletions(-) diff --git a/Docs/04-advanced-features/web-servers.md b/Docs/04-advanced-features/web-servers.md index 1226009d..523401a7 100644 --- a/Docs/04-advanced-features/web-servers.md +++ b/Docs/04-advanced-features/web-servers.md @@ -514,11 +514,13 @@ close out > `write line "x" to out`) — are **stream-only** and error if `` is not a > streaming-response handle. > - > **Value operators.** When the value is a bare variable, `write line`/ - > `write chunk` currently accept only that variable (optionally `field of - > object`) before `to`; `with`-concatenation directly in the statement (e.g. - > `write line prefix with json to out`) is not yet supported. Build the value - > first — `store payload as prefix with json`, then `write line payload to out`. + > **Value operators.** The value is a full expression, so `with`-concatenation + > (and other operators) work directly in the statement — e.g. + > `write line prefix with json to out`. For the ambiguous bare-identifier form, + > the continuation applies to both readings: `write line note with "!" to out` + > streams `note` + `"!"`, while the same statement targeting a file writes the + > variable `line note` + `"!"` — so pre-existing classic file writes that + > concatenate keep working. - `flush ` — advisory: yield so queued bytes are handed to the socket. (Chunks are already forwarded as you write them; hyper writes as it receives.) - `close ` — end the response body. Writing after `close` is an error. From 7b26602f6257791024270cf60edaece969d72fd0 Mon Sep 17 00:00:00 2001 From: Claude Date: Fri, 24 Jul 2026 05:57:01 +0000 Subject: [PATCH 038/132] fix: parse ambiguous write line/chunk readings independently; analyze continuation MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Fixes a back-compat regression the leftmost-leaf approach introduced, plus the coupled analyzer gaps (maintainer review). Parser (P1 back-compat): the two readings of the merged `write line|chunk ...` form are now parsed INDEPENDENTLY from the same tokens via cursor checkpoint/rewind, instead of deriving the classic file-write AST from the stream AST by swapping the leftmost leaf. A continuation desugars differently per leading operand — a builtin name becomes an ActionCall, `is between` duplicates the left operand, `starts/ends with` and pattern ops build calls — so leaf-swapping silently dropped or mangled the continuation. Regression: `store line length as "kept" / write line length with "!" to ` now writes "kept!" (was "kept"). Analyzer: the ambiguous arm now analyzes the shared continuation (every sub-expression except the ambiguous leading operand) so an undefined variable there is caught, and reports the leading operand undefined only when NEITHER reading resolves via a leftmost-leaf lead name — fixing both a false negative (undefined RHS slipping through) and a false positive (valid classic program defining `line ` rejected because the split stream name is undefined). Docs: the web-servers note no longer claims the value is a "full expression"; it states that `with`/operators work but postfix (indexing/property) on the leading identifier is not parsed there. Tests: write_line_backcompat_test adds the builtin-named regression, an undefined-in-continuation case, and a valid-classic-with-continuation case. Co-Authored-By: Claude Opus 4.8 Claude-Session: https://claude.ai/code/session_011LxXAm5E4Vy39kRLr8kRnX --- Docs/04-advanced-features/web-servers.md | 9 +- src/analyzer/mod.rs | 83 ++++++++++---- src/parser/stmt/io.rs | 137 +++++++++-------------- tests/write_line_backcompat_test.rs | 77 +++++++++++++ 4 files changed, 198 insertions(+), 108 deletions(-) diff --git a/Docs/04-advanced-features/web-servers.md b/Docs/04-advanced-features/web-servers.md index 523401a7..c7a801cd 100644 --- a/Docs/04-advanced-features/web-servers.md +++ b/Docs/04-advanced-features/web-servers.md @@ -514,9 +514,12 @@ close out > `write line "x" to out`) — are **stream-only** and error if `` is not a > streaming-response handle. > - > **Value operators.** The value is a full expression, so `with`-concatenation - > (and other operators) work directly in the statement — e.g. - > `write line prefix with json to out`. For the ambiguous bare-identifier form, + > **Value operators.** The value accepts `with`-concatenation and the usual + > arithmetic/comparison operators directly in the statement — e.g. + > `write line prefix with json to out`. (An identifier-led value is a variable + > or `field of object` followed by such operators; postfix forms like indexing + > `payload[1]` or `payload.field` on the leading identifier are not parsed here + > — build those into a variable first.) For the ambiguous bare-identifier form, > the continuation applies to both readings: `write line note with "!" to out` > streams `note` + `"!"`, while the same statement targeting a file writes the > variable `line note` + `"!"` — so pre-existing classic file writes that diff --git a/src/analyzer/mod.rs b/src/analyzer/mod.rs index 4996ea23..56878966 100644 --- a/src/analyzer/mod.rs +++ b/src/analyzer/mod.rs @@ -1670,25 +1670,19 @@ impl Analyzer { match fallback_content { // Unambiguous form: check the stream value normally. None => self.analyze_expression(value), - // Ambiguous merged form (`write line to `): + // Ambiguous merged form (`write line ... to `): // the live reading — stream write of `` vs classic // file write of the variable `line ` — depends on the - // runtime target type, and the two read different variables. - // Still (1) analyze any unambiguous subexpression (the - // `` in ` of `), and (2) report an - // undefined variable only when *neither* candidate name is - // defined, so a genuine typo is still caught without breaking - // either valid reading. + // runtime target type, and the two differ only in the leading + // operand. So (1) analyze the shared continuation (everything + // to the right of the lead) so a genuinely undefined variable + // there is still caught, and (2) report the leading operand as + // undefined only when *neither* reading's name resolves — so a + // real typo is caught without rejecting either valid reading. Some(fallback) => { - // The `of ` argument is the same under both - // readings and is unambiguous — analyze it. - if let Expression::FunctionCall { arguments, .. } = value { - for arg in arguments { - self.analyze_expression(&arg.value); - } - } - let stream_name = Self::stream_write_candidate_name(value); - let fallback_name = Self::stream_write_candidate_name(fallback); + self.analyze_stream_write_continuation(value); + let stream_name = Self::stream_write_lead_name(value); + let fallback_name = Self::stream_write_lead_name(fallback); if let (Some(sn), Some(fal)) = (stream_name, fallback_name) && !self.name_is_defined(sn) && !self.name_is_defined(fal) @@ -3553,19 +3547,62 @@ impl Analyzer { } } - /// The candidate variable name referenced by a `write line|chunk` value: a - /// bare `Variable`, or the callee of a ` of ` call. - fn stream_write_candidate_name(expr: &Expression) -> Option<&str> { + /// The ambiguous leading operand name of a `write line|chunk` value — the + /// leftmost leaf of the (possibly nested) expression. The two readings of the + /// merged form differ ONLY here (stream `` vs classic `line `); + /// everything to the right is a shared continuation. Reaches the leaf through + /// `with`/binary `left`, an `of`-call `function`, or an `ActionCall`'s callee + /// name (a builtin used with `with`). + fn stream_write_lead_name(expr: &Expression) -> Option<&str> { match expr { Expression::Variable(name, ..) => Some(name), - Expression::FunctionCall { function, .. } => match &**function { - Expression::Variable(name, ..) => Some(name), - _ => None, - }, + Expression::ActionCall { name, .. } => Some(name), + Expression::Concatenation { left, .. } | Expression::BinaryOperation { left, .. } => { + Self::stream_write_lead_name(left) + } + Expression::FunctionCall { function, .. } => Self::stream_write_lead_name(function), _ => None, } } + /// Analyze the shared continuation of a `write line|chunk` value — every + /// sub-expression EXCEPT the ambiguous leading operand (its leftmost leaf). + /// This catches a genuinely undefined variable in the continuation (e.g. the + /// RHS of ` with missing_suffix`) without flagging the leading operand, + /// which is valid under whichever of the two readings the runtime picks. + fn analyze_stream_write_continuation(&mut self, expr: &Expression) { + match expr { + // The leftmost leaf itself is the ambiguous lead — checked separately. + Expression::Variable(..) => {} + Expression::Concatenation { left, right, .. } => { + self.analyze_stream_write_continuation(left); + self.analyze_expression(right); + } + Expression::BinaryOperation { left, right, .. } => { + self.analyze_stream_write_continuation(left); + self.analyze_expression(right); + } + Expression::FunctionCall { + function, + arguments, + .. + } => { + self.analyze_stream_write_continuation(function); + for arg in arguments { + self.analyze_expression(&arg.value); + } + } + Expression::ActionCall { arguments, .. } => { + // The callee name is the leading lead (skip); analyze the args. + for arg in arguments { + self.analyze_expression(&arg.value); + } + } + // Any other shape has no ambiguous lead to protect — analyze it whole. + other => self.analyze_expression(other), + } + } + /// Whether a bare name resolves to something known (an action parameter, the /// `count` loop variable, a builtin, an in-scope binding, or a container /// property) — i.e. it would NOT be reported as an undefined variable. Used diff --git a/src/parser/stmt/io.rs b/src/parser/stmt/io.rs index 9fa8b89d..4e3d01cb 100644 --- a/src/parser/stmt/io.rs +++ b/src/parser/stmt/io.rs @@ -6,58 +6,40 @@ use crate::lexer::token::Token; use crate::parser::expr::{BinaryExprParser, ExprParser, PrimaryExprParser}; use std::sync::Arc; -/// Replace the leftmost leaf operand of a (possibly nested) expression. -/// -/// The merged `write line|chunk ...` form parses one value expression -/// for the stream reading; the classic file-write reading differs only in its -/// leading operand (the full merged `line ` variable instead of the -/// split ``). Rather than re-parse, we clone the parsed value and swap -/// its leftmost operand — so a trailing `with`/operator continuation applies to -/// both readings identically. `with`/binary chains here are right-associative -/// (`a with b with c` => `Concat(a, Concat(b, c))`) and ` of ` -/// is a `FunctionCall`, so the leading operand is always reached via `.left`, -/// `.function`, or the leaf itself. -fn replace_leftmost_leaf(expr: Expression, replacement: Expression) -> Expression { - match expr { - Expression::Concatenation { - left, - right, - line, - column, - } => Expression::Concatenation { - left: Box::new(replace_leftmost_leaf(*left, replacement)), - right, - line, - column, - }, - Expression::BinaryOperation { - left, - operator, - right, - line, - column, - } => Expression::BinaryOperation { - left: Box::new(replace_leftmost_leaf(*left, replacement)), - operator, - right, - line, - column, - }, - Expression::FunctionCall { - function, - arguments, - line, - column, - } => Expression::FunctionCall { - function: Box::new(replace_leftmost_leaf(*function, replacement)), - arguments, - line, - column, - }, - // Leaf (a bare `Variable`, the common case) or a form whose leading - // operand is not a nested `Expression` (e.g. an `ActionCall`, from the - // rare ` with ...`): swap wholesale. - _ => replacement, +impl<'a> Parser<'a> { + /// Parse a `write line|chunk` value from an already-chosen leading operand: + /// an optional ` of ` postfix, then any `with`/operator + /// continuation, exactly as a normal expression value would parse. + /// + /// The ambiguous merged `write line|chunk ...` form has two readings + /// (stream: split-off ``; classic file write: whole `line `) + /// that differ only in the leading operand. They are parsed independently — + /// same tokens, via a cursor rewind between the two calls — because a + /// continuation can desugar differently per operand (a builtin name becomes + /// an `ActionCall`, `is between` duplicates the left, `starts/ends with` and + /// the pattern operators build calls), so deriving one AST from the other by + /// leaf-swapping silently corrupted the classic reading. + fn parse_write_value_from_lead(&mut self, lead: Expression) -> Result { + let (line, column) = match &lead { + Expression::Variable(_, l, c) => (*l, *c), + _ => (0, 0), + }; + let lead = if matches!(self.cursor.peek().map(|t| &t.token), Some(Token::KeywordOf)) { + self.bump_sync(); // Consume "of" + let object = self.parse_primary_expression()?; + Expression::FunctionCall { + function: Box::new(lead), + arguments: vec![crate::parser::ast::Argument { + name: None, + value: object, + }], + line, + column, + } + } else { + lead + }; + self.parse_binary_continuation(lead, 0) } } @@ -899,38 +881,29 @@ impl<'a> IoParser<'a> for Parser<'a> { // write (`write line "x" to f` did not parse), so no fallback. (self.parse_expression()?, None) } else { - // `` alone (stream) vs the full merged `line ` - // (classic file write of that variable). Build the stream - // reading's leading operand — ``, or ` of ` - // — then absorb any trailing `with`/operator continuation so the - // value parses like any other expression (a `write line payload - // with "!" to out` value, and the pre-existing classic file write - // `write line payload with "!" to file`, must not be truncated). + // Ambiguous merged form: `` alone (stream) vs the full + // merged `line ` (classic file write of that variable). + // Parse the two readings INDEPENDENTLY from the same continuation + // tokens via cursor rewind — NOT by deriving one AST from the + // other. A trailing `with`/operator continuation desugars + // differently per leading operand: a builtin name becomes an + // `ActionCall`, `is between` duplicates the left operand, + // `starts/ends with` and the pattern operators build calls — none + // of which survive a leftmost-leaf swap, which silently dropped or + // mangled the continuation for the classic file-write reading. + let value_start = self.cursor.checkpoint(); + + // Stream reading: split-off `` as the leading operand. let stream_left = Expression::Variable(rest, marker_line, marker_column); + let value = self.parse_write_value_from_lead(stream_left)?; + + // Rewind and parse the classic file-write reading with the whole + // merged `line ` as the leading operand, over the very same + // tokens, so the two interpretations stay faithful to the source. + self.cursor.rewind(value_start); let file_left = Expression::Variable(id, marker_line, marker_column); - let value_lead = match self.cursor.peek().map(|t| &t.token) { - // ` of `, e.g. `write line body of msg to out`. - Some(Token::KeywordOf) => { - self.bump_sync(); // Consume "of" - let object = self.parse_primary_expression()?; - Expression::FunctionCall { - function: Box::new(stream_left), - arguments: vec![crate::parser::ast::Argument { - name: None, - value: object, - }], - line: marker_line, - column: marker_column, - } - } - // A bare variable value (possibly followed by `with ...`). - _ => stream_left, - }; - let value = self.parse_binary_continuation(value_lead, 0)?; - // The classic file-write reading is identical except its leading - // operand is the full merged `line `; mirror the parsed - // continuation onto it by swapping the leftmost leaf. - let fallback = replace_leftmost_leaf(value.clone(), file_left); + let fallback = self.parse_write_value_from_lead(file_left)?; + (value, Some(Box::new(fallback))) }; diff --git a/tests/write_line_backcompat_test.rs b/tests/write_line_backcompat_test.rs index d7af571f..7ba93ae7 100644 --- a/tests/write_line_backcompat_test.rs +++ b/tests/write_line_backcompat_test.rs @@ -160,6 +160,46 @@ write line note with "!" to "{path_str}""# ); } +#[test] +fn test_write_line_builtin_named_variable_with_continuation_preserves_concatenation() { + // Regression (maintainer review): the stream reading of `length with "!"` + // desugars to an ActionCall because `length` is a builtin. The classic + // file-write reading must still be the variable `line length` concatenated + // with "!" — parsed INDEPENDENTLY (cursor rewind), not derived from the + // stream AST, which would drop the `with "!"` and write only "kept". + let dir = tempfile::tempdir().expect("create temp dir"); + let path = dir.path().join("wfl_write_line_builtin_named.txt"); + let path_str = path.to_string_lossy().replace('\\', "/"); + + let code = format!( + r#"store line length as "kept" +write line length with "!" to "{path_str}""# + ); + + let program = { + let tokens = lex_wfl_with_positions(&code); + let mut parser = Parser::new(&tokens); + parser.parse().expect("parse") + }; + + let mut analyzer = Analyzer::new(); + analyzer + .analyze(&program) + .expect("analysis must accept the builtin-named continuation file write"); + + let rt = tokio::runtime::Runtime::new().unwrap(); + rt.block_on(async { + let mut interp = Interpreter::new(); + interp.interpret(&program).await.expect("interpret"); + }); + + let contents = std::fs::read_to_string(&path).expect("output file should exist"); + assert_eq!( + contents, "kept!", + "the classic file write must keep the `with \"!\"` continuation even though `length` is a builtin" + ); +} + #[test] fn test_write_multiword_line_variable_to_file_still_works() { // A pre-existing program: a variable literally named `line note` written to a @@ -204,6 +244,43 @@ write line note to "{path_str}""# let _ = std::fs::remove_file(&path); } +#[test] +fn test_ambiguous_write_line_flags_undefined_in_continuation() { + // The continuation (everything right of the ambiguous lead) is shared by both + // readings, so a genuinely undefined variable there must still be caught even + // though the leading operand itself is ambiguous. + let code = "listen on port 8080 as srv\nstore payload as \"x\"\nwrite line payload with missing_suffix to srv"; + let tokens = lex_wfl_with_positions(code); + let program = Parser::new(&tokens).parse().expect("parse"); + let mut analyzer = Analyzer::new(); + let errors = analyzer + .analyze(&program) + .expect_err("`missing_suffix` in the continuation is undefined"); + assert!( + errors + .iter() + .any(|e| e.message.contains("missing_suffix") && e.message.contains("not defined")), + "expected an undefined-variable error naming `missing_suffix`, got: {errors:?}" + ); +} + +#[test] +fn test_ambiguous_write_line_accepts_valid_classic_with_continuation() { + // A valid pre-existing program: a variable literally named `line path`, + // written with a continuation to a file. The split stream name `path` is + // undefined, but the classic file-write reading resolves — analysis must NOT + // reject it (no false positive from the ambiguous split). + let code = "store line path as \"/tmp/x\"\nstore suffix as \"!\"\nwrite line path with suffix to \"/tmp/out\""; + let tokens = lex_wfl_with_positions(code); + let program = Parser::new(&tokens).parse().expect("parse"); + let mut analyzer = Analyzer::new(); + assert!( + analyzer.analyze(&program).is_ok(), + "a valid classic `write line with ... to ` must not be rejected: {:?}", + analyzer.get_errors() + ); +} + #[test] fn test_ambiguous_write_line_still_flags_when_neither_candidate_defined() { // The ambiguous form defers definedness to runtime, but a genuine typo where From 6e8c7eb2612ea8d17956b2e30644c3eb2e88c9fa Mon Sep 17 00:00:00 2001 From: Claude Date: Fri, 24 Jul 2026 06:30:10 +0000 Subject: [PATCH 039/132] fix(typechecker): accept stream handles in close, not just File Maintainer review: stream handles (server response streams from `start streaming response as ` and outbound streams from `... stream response as `) bind as map-shaped objects, but CloseFileStatement accepted only a `File` object, so a valid `close out` / `close upstream` produced a spurious "Expected a File object" diagnostic. Add is_closeable_type (File/Custom, the map-shaped stream handle, or a statically-unresolved Unknown/Any/Error) and use it; a concrete scalar like `close 5` is still rejected. Tests (new stream_handle_type_test): closing a server response stream and an outbound stream both type-check clean; closing a number still errors. Co-Authored-By: Claude Opus 4.8 Claude-Session: https://claude.ai/code/session_011LxXAm5E4Vy39kRLr8kRnX --- src/typechecker/mod.rs | 25 +++++++++++--- tests/stream_handle_type_test.rs | 58 ++++++++++++++++++++++++++++++++ 2 files changed, 78 insertions(+), 5 deletions(-) create mode 100644 tests/stream_handle_type_test.rs diff --git a/src/typechecker/mod.rs b/src/typechecker/mod.rs index c015f0d6..dd6cbb4f 100644 --- a/src/typechecker/mod.rs +++ b/src/typechecker/mod.rs @@ -1616,12 +1616,9 @@ impl TypeChecker { column: _column, } => { let file_type = self.infer_expression_type(file); - if file_type != Type::Custom("File".to_string()) - && file_type != Type::Unknown - && file_type != Type::Error - { + if !self.is_closeable_type(&file_type) { self.type_error( - "Expected a File object".to_string(), + "Expected a file or stream handle".to_string(), Some(Type::Custom("File".to_string())), Some(file_type), *_line, @@ -4689,6 +4686,24 @@ impl TypeChecker { } } + /// Whether a type can name a closeable resource: a file handle, a stream + /// handle, or a statically-unresolved value. Stream handles (server response + /// streams from `start streaming response as ...` and outbound streams from + /// `... stream response as ...`) bind as map-shaped objects, so a `close out` + /// / `close upstream` must be accepted here rather than flagged as "not a + /// File". Concrete scalars (a number, boolean, list, …) are still rejected so + /// an obviously-wrong `close 5` errors. + fn is_closeable_type(&self, ty: &Type) -> bool { + matches!( + ty, + Type::Custom(_) // File, or any handle object + | Type::Map(_, _) // stream handles bind as Map + | Type::Unknown + | Type::Any + | Type::Error + ) + } + fn are_types_compatible(&self, target_type: &Type, source_type: &Type) -> bool { #[allow(clippy::only_used_in_recursion)] let _self = self; // Suppress the warning for self parameter diff --git a/tests/stream_handle_type_test.rs b/tests/stream_handle_type_test.rs new file mode 100644 index 00000000..bb255990 --- /dev/null +++ b/tests/stream_handle_type_test.rs @@ -0,0 +1,58 @@ +//! Type-contract coverage for stream handles (maintainer review). +//! +//! Stream handles bound by `start streaming response ... as ` and +//! `... stream response as ` are map-shaped objects. `close ` +//! must accept them — before this fix the type checker only accepted a `File` +//! object, so a valid `close out` / `close upstream` produced a spurious +//! "Expected a File object" diagnostic. A concrete scalar must still be rejected. + +use wfl::lexer::lex_wfl_with_positions; +use wfl::parser::Parser; +use wfl::typechecker::TypeChecker; + +fn typecheck(code: &str) -> Result<(), String> { + let tokens = lex_wfl_with_positions(code); + let program = Parser::new(&tokens).parse().expect("parse"); + TypeChecker::new() + .check_types(&program) + .map_err(|e| format!("{e:?}")) +} + +#[test] +fn test_close_server_response_stream_handle_typechecks() { + // `out` is a streaming-response handle; `close out` must type-check cleanly. + let code = "listen on port 8080 as s\n\ + wait for request comes in on s as req\n\ + start streaming response to req with status 200 and content type \"text/plain\" as out\n\ + write line \"hi\" to out\n\ + close out"; + assert!( + typecheck(code).is_ok(), + "closing a streaming-response handle must not be flagged as a non-File: {:?}", + typecheck(code).err() + ); +} + +#[test] +fn test_close_outbound_stream_handle_typechecks() { + // `upstream` is an outbound streaming handle; `close upstream` must be clean. + let code = "open url at \"http://example.com\" and stream response as upstream\n\ + close upstream"; + assert!( + typecheck(code).is_ok(), + "closing an outbound stream handle must not be flagged as a non-File: {:?}", + typecheck(code).err() + ); +} + +#[test] +fn test_close_scalar_is_still_rejected() { + // A concrete non-handle value is still a type error — the fix widens `close` + // to file/stream handles, it does not make `close` accept anything. + let code = "store n as 5\nclose n"; + let errors = typecheck(code).expect_err("closing a number must be a type error"); + assert!( + errors.contains("file or stream handle") || errors.contains("File"), + "expected a close-operand type error, got: {errors}" + ); +} From 5259a1d3e5d166603452befcf43dede2313b0e37 Mon Sep 17 00:00:00 2001 From: Claude Date: Fri, 24 Jul 2026 06:36:18 +0000 Subject: [PATCH 040/132] fix(typechecker): only File custom type is closeable; drop misleading expected hint Two Copilot review points on the close type-contract fix: - is_closeable_type treated any Type::Custom(_) as closeable, so `close db` (Custom("Database")) / `close req` would wrongly pass. Restrict to Custom("File"); stream handles are Map-shaped (still accepted). - The error says "file or stream handle" but passed Some(File) as the expected type, mis-rendering expected-vs-found. Pass None instead. Test: closing a database handle is now rejected. Co-Authored-By: Claude Opus 4.8 Claude-Session: https://claude.ai/code/session_011LxXAm5E4Vy39kRLr8kRnX --- src/typechecker/mod.rs | 26 +++++++++++++++----------- tests/stream_handle_type_test.rs | 13 +++++++++++++ 2 files changed, 28 insertions(+), 11 deletions(-) diff --git a/src/typechecker/mod.rs b/src/typechecker/mod.rs index dd6cbb4f..e61b18db 100644 --- a/src/typechecker/mod.rs +++ b/src/typechecker/mod.rs @@ -1617,9 +1617,12 @@ impl TypeChecker { } => { let file_type = self.infer_expression_type(file); if !self.is_closeable_type(&file_type) { + // No single `expected` type: `close` accepts a File *or* a + // (map-shaped) stream handle, so pinning the hint to `File` + // would mis-render the expected-vs-found diagnostic. self.type_error( "Expected a file or stream handle".to_string(), - Some(Type::Custom("File".to_string())), + None, Some(file_type), *_line, *_column, @@ -4691,17 +4694,18 @@ impl TypeChecker { /// streams from `start streaming response as ...` and outbound streams from /// `... stream response as ...`) bind as map-shaped objects, so a `close out` /// / `close upstream` must be accepted here rather than flagged as "not a - /// File". Concrete scalars (a number, boolean, list, …) are still rejected so - /// an obviously-wrong `close 5` errors. + /// File". Only the `File` custom type is closeable — other custom types + /// (`Database`, `Request`, …) are NOT, so `close db` / `close req` still + /// errors — and concrete scalars (`close 5`) are rejected too. fn is_closeable_type(&self, ty: &Type) -> bool { - matches!( - ty, - Type::Custom(_) // File, or any handle object - | Type::Map(_, _) // stream handles bind as Map - | Type::Unknown - | Type::Any - | Type::Error - ) + match ty { + // The file handle from `open file ... as f`. + Type::Custom(name) => name == "File", + // Stream handles bind as Map; a statically-unresolved value + // is accepted to avoid false positives. + Type::Map(_, _) | Type::Unknown | Type::Any | Type::Error => true, + _ => false, + } } fn are_types_compatible(&self, target_type: &Type, source_type: &Type) -> bool { diff --git a/tests/stream_handle_type_test.rs b/tests/stream_handle_type_test.rs index bb255990..ef661b0b 100644 --- a/tests/stream_handle_type_test.rs +++ b/tests/stream_handle_type_test.rs @@ -45,6 +45,19 @@ fn test_close_outbound_stream_handle_typechecks() { ); } +#[test] +fn test_close_non_file_custom_handle_is_rejected() { + // Only a File custom type is closeable via `close`. Other custom handles + // (a database connection here) are NOT — the interpreter cannot close them, + // so accepting `close db` would be a false negative. + let code = "open database at \"sqlite::memory:\" as db\nclose db"; + let errors = typecheck(code).expect_err("closing a database handle must be a type error"); + assert!( + errors.contains("file or stream handle") || errors.contains("File"), + "expected a close-operand type error for a database handle, got: {errors}" + ); +} + #[test] fn test_close_scalar_is_still_rejected() { // A concrete non-handle value is still a type error — the fix widens `close` From 982fe7b21b139305ce1486c99ebf139bd5c3d2e3 Mon Sep 17 00:00:00 2001 From: Claude Date: Fri, 24 Jul 2026 06:43:46 +0000 Subject: [PATCH 041/132] fix(ci): guard Kill() and warn before racing TLS temp cleanup MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Maintainer review (Windows-gate completeness): Kill() in Stop-ServerProcess was outside the try, so a throwing Kill() (access denied / concurrent exit) went unhandled; wrap it. And the TLS finally deleted the temp dir with -ErrorAction SilentlyContinue even when the server had not actually exited, hiding a failed cleanup — now re-check HasExited and warn that cleanup may be incomplete before the Remove-Item, rather than silently proceeding. Co-Authored-By: Claude Opus 4.8 Claude-Session: https://claude.ai/code/session_011LxXAm5E4Vy39kRLr8kRnX --- scripts/run_web_tests.ps1 | 17 +++++++++++++++-- 1 file changed, 15 insertions(+), 2 deletions(-) diff --git a/scripts/run_web_tests.ps1 b/scripts/run_web_tests.ps1 index b2714f3d..c9bdf578 100644 --- a/scripts/run_web_tests.ps1 +++ b/scripts/run_web_tests.ps1 @@ -78,10 +78,17 @@ function Show-ServerLogs { function Stop-ServerProcess { param($Process) if ($Process -and -not $Process.HasExited) { - $Process.Kill() + # Kill() itself can throw (e.g. access denied, or the process exiting + # concurrently), so guard it too rather than leaving it outside the try. + try { + $Process.Kill() + } catch { + Write-Host "[WARN] Kill() on server process failed: $_" -ForegroundColor Yellow + } # WaitForExit(ms) returns $true only if the process actually exited in # time; report honestly rather than always claiming success (a process - # still alive can race temp-file/cert cleanup that follows). + # still alive can race temp-file/cert cleanup that follows). Callers that + # then delete temp files re-check HasExited before doing so. $exited = $false try { $exited = $Process.WaitForExit(5000) } catch { } if ($exited) { @@ -385,6 +392,12 @@ if (Test-Path "TestPrograms\web_server_tls.wfl") { # has released its cert/log file handles (avoids a Windows cleanup # race that would leave the dir or fail the Remove-Item). Stop-ServerProcess -Process $tlsProcess + if ($tlsProcess -and -not $tlsProcess.HasExited) { + # Still alive after Kill()+WaitForExit — deleting now can race the + # process's open cert/log handles. Surface it instead of hiding a + # failed cleanup behind SilentlyContinue. + Write-Host "[WARN] TLS server still running; temp dir cleanup may be incomplete" -ForegroundColor Yellow + } Remove-Item -Recurse -Force $tlsDir -ErrorAction SilentlyContinue } } From 31cda418d5c3585bb97d8d5dc8ac302a670d0bee Mon Sep 17 00:00:00 2001 From: Claude Date: Fri, 24 Jul 2026 06:53:49 +0000 Subject: [PATCH 042/132] fix(typechecker): distinct stream-handle types so close rejects ordinary maps Copilot review: is_closeable_type accepted any Map, so `close some_map` passed even though the runtime can only close file/stream handles. Give stream handles distinct static types instead of Map: the outbound handle is Custom("HttpStream") and the server response stream is Custom("ResponseStream"). `close` now accepts only File|HttpStream|ResponseStream (plus Unknown/Any/Error); an ordinary map, another custom type, or a scalar is rejected. IndexAccess on the stream-handle types returns Any so reading their fields (status/ok/headers, including nested header lookups) still type-checks; MemberAccess on Custom already returned Unknown. Tests: close of both handle kinds passes; closing an ordinary map, a database handle, and a number are rejected; direct and nested field indexing still checks. Co-Authored-By: Claude Opus 4.8 Claude-Session: https://claude.ai/code/session_011LxXAm5E4Vy39kRLr8kRnX --- src/typechecker/mod.rs | 44 +++++++++++++++++++------------- tests/stream_handle_type_test.rs | 28 ++++++++++++++++++++ 2 files changed, 54 insertions(+), 18 deletions(-) diff --git a/src/typechecker/mod.rs b/src/typechecker/mod.rs index e61b18db..81d8a2ea 100644 --- a/src/typechecker/mod.rs +++ b/src/typechecker/mod.rs @@ -940,12 +940,14 @@ impl TypeChecker { } } - // Binds a streaming-response handle object (status/ok/headers). + // Binds an outbound streaming-response handle (exposes + // status/ok/headers via index/member access, and is closeable). + // A distinct handle type — not a bare `Map` — so `close` accepts + // it without also accepting an ordinary user map. if !variable_name.is_empty() && let Some(symbol) = self.analyzer.get_symbol_mut(variable_name) { - symbol.symbol_type = - Some(Type::Map(Box::new(Type::Text), Box::new(Type::Unknown))); + symbol.symbol_type = Some(Type::Custom("HttpStream".to_string())); } } Statement::WaitForNextChunkStatement { @@ -1027,8 +1029,10 @@ impl TypeChecker { if !variable_name.is_empty() && let Some(symbol) = self.analyzer.get_symbol_mut(variable_name) { - symbol.symbol_type = - Some(Type::Map(Box::new(Type::Text), Box::new(Type::Unknown))); + // A distinct server-response-stream handle type (not a bare + // `Map`) so `close out` is accepted without `close` also + // type-checking an ordinary user map. + symbol.symbol_type = Some(Type::Custom("ResponseStream".to_string())); } } Statement::StreamWriteStatement { value, target, .. } => { @@ -3532,6 +3536,13 @@ impl TypeChecker { // result) is indexable; the element type is only known // at runtime (issue #553). Type::Any => Type::Any, + // Stream handles expose fields (`status`/`ok`/`headers`) by + // index; the field type is only known at runtime. + Type::Custom(ref name) + if name == "HttpStream" || name == "ResponseStream" => + { + Type::Any + } _ => { self.type_error( format!("Cannot index into {collection_type}"), @@ -4689,21 +4700,18 @@ impl TypeChecker { } } - /// Whether a type can name a closeable resource: a file handle, a stream - /// handle, or a statically-unresolved value. Stream handles (server response - /// streams from `start streaming response as ...` and outbound streams from - /// `... stream response as ...`) bind as map-shaped objects, so a `close out` - /// / `close upstream` must be accepted here rather than flagged as "not a - /// File". Only the `File` custom type is closeable — other custom types - /// (`Database`, `Request`, …) are NOT, so `close db` / `close req` still - /// errors — and concrete scalars (`close 5`) are rejected too. + /// Whether a type can name a closeable resource: a file handle + /// (`Custom("File")`), a stream handle (`Custom("HttpStream")` outbound or + /// `Custom("ResponseStream")` server-side), or a statically-unresolved value. + /// These are the only things the runtime can close, so an ordinary map, + /// another custom type (`Database`/`Request`), or a scalar (`close 5`) is + /// rejected — keeping real mistakes as static errors rather than runtime-only. fn is_closeable_type(&self, ty: &Type) -> bool { match ty { - // The file handle from `open file ... as f`. - Type::Custom(name) => name == "File", - // Stream handles bind as Map; a statically-unresolved value - // is accepted to avoid false positives. - Type::Map(_, _) | Type::Unknown | Type::Any | Type::Error => true, + Type::Custom(name) => { + name == "File" || name == "HttpStream" || name == "ResponseStream" + } + Type::Unknown | Type::Any | Type::Error => true, _ => false, } } diff --git a/tests/stream_handle_type_test.rs b/tests/stream_handle_type_test.rs index ef661b0b..849e7475 100644 --- a/tests/stream_handle_type_test.rs +++ b/tests/stream_handle_type_test.rs @@ -58,6 +58,34 @@ fn test_close_non_file_custom_handle_is_rejected() { ); } +#[test] +fn test_outbound_stream_handle_fields_are_indexable() { + // The outbound handle now has a distinct `Custom("HttpStream")` type; reading + // its fields by index must still type-check (the field type is runtime-known). + // Mirrors the docs example: a direct field and a nested header lookup. + let code = "open url at \"http://example.com\" and stream response as resp\n\ + store code as resp[\"status\"]\n\ + store ct as resp[\"headers\"][\"content-type\"]\n\ + close resp"; + assert!( + typecheck(code).is_ok(), + "indexing a stream handle's fields must still type-check: {:?}", + typecheck(code).err() + ); +} + +#[test] +fn test_close_ordinary_map_is_rejected() { + // A plain user map is NOT closeable — only file/stream handles are. This is + // the tightening that a distinct handle type buys over accepting any `Map`. + let code = "create map m:\n \"k\" is \"v\"\nend map\nclose m"; + let errors = typecheck(code).expect_err("closing an ordinary map must be a type error"); + assert!( + errors.contains("file or stream handle") || errors.contains("File"), + "expected a close-operand type error for a plain map, got: {errors}" + ); +} + #[test] fn test_close_scalar_is_still_rejected() { // A concrete non-handle value is still a type error — the fix widens `close` From 38e6bde87e9231ad605400c25e5dcb1014bb032a Mon Sep 17 00:00:00 2001 From: Claude Date: Fri, 24 Jul 2026 06:55:38 +0000 Subject: [PATCH 043/132] style: rustfmt the stream-handle index-access match arm Co-Authored-By: Claude Opus 4.8 Claude-Session: https://claude.ai/code/session_011LxXAm5E4Vy39kRLr8kRnX --- src/typechecker/mod.rs | 4 +--- 1 file changed, 1 insertion(+), 3 deletions(-) diff --git a/src/typechecker/mod.rs b/src/typechecker/mod.rs index 81d8a2ea..79a8f627 100644 --- a/src/typechecker/mod.rs +++ b/src/typechecker/mod.rs @@ -3538,9 +3538,7 @@ impl TypeChecker { Type::Any => Type::Any, // Stream handles expose fields (`status`/`ok`/`headers`) by // index; the field type is only known at runtime. - Type::Custom(ref name) - if name == "HttpStream" || name == "ResponseStream" => - { + Type::Custom(ref name) if name == "HttpStream" || name == "ResponseStream" => { Type::Any } _ => { From 2cc153fb56fd1952abfcbe5290246afc30ab6aaf Mon Sep 17 00:00:00 2001 From: Claude Date: Fri, 24 Jul 2026 07:11:07 +0000 Subject: [PATCH 044/132] fix(analyzer): stop rejecting valid classic writes with desugared values MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Fatal back-compat regression from the previous analyzer walk (maintainer review): stream_write_lead_name/analyze_stream_write_continuation assumed the ambiguous lead was always the leftmost leaf/function callee and that every call argument or binary right branch was shared/unambiguous. Desugared values break that — `starts/ends with` makes the lead a call ARGUMENT, `is between` DUPLICATES the lead, and pattern/of/builtin-with bury it — so valid classic file writes like `write line path starts with "/" to ` (only `line path` defined) were wrongly reported undefined. Analyze only the shapes where the lead is provably the single leftmost bare variable — a bare Variable, or ` with ` (Concatenation with a Variable left); for those, still catch an undefined shared RHS and flag the lead only when NEITHER reading resolves. Every other (desugared) shape defers entirely to runtime, so no valid program is rejected. Tests: valid classic `starts with` / `is between` / `matches pattern` (and a `write chunk` case) now analyze cleanly, while the undefined-RHS and both-leads-undefined cases still error. Co-Authored-By: Claude Opus 4.8 Claude-Session: https://claude.ai/code/session_011LxXAm5E4Vy39kRLr8kRnX --- src/analyzer/mod.rs | 112 +++++++++++----------------- tests/write_line_backcompat_test.rs | 26 +++++++ 2 files changed, 69 insertions(+), 69 deletions(-) diff --git a/src/analyzer/mod.rs b/src/analyzer/mod.rs index 56878966..76bb3252 100644 --- a/src/analyzer/mod.rs +++ b/src/analyzer/mod.rs @@ -1674,27 +1674,38 @@ impl Analyzer { // the live reading — stream write of `` vs classic // file write of the variable `line ` — depends on the // runtime target type, and the two differ only in the leading - // operand. So (1) analyze the shared continuation (everything - // to the right of the lead) so a genuinely undefined variable - // there is still caught, and (2) report the leading operand as - // undefined only when *neither* reading's name resolves — so a - // real typo is caught without rejecting either valid reading. + // operand. Analyze ONLY the shapes where that lead is + // unambiguously the single leftmost bare variable: a bare + // `Variable`, or ` with ` (a `Concatenation` + // whose left is that variable). Desugared forms place the lead + // where a generic walk cannot separate it from the shared + // continuation — `starts/ends with` makes the lead a call + // argument, `is between` duplicates it, pattern/`of`/builtin- + // `with` bury it in a call — so analyzing them would reject a + // valid classic file write (a back-compat regression). Defer + // those entirely to runtime. Some(fallback) => { - self.analyze_stream_write_continuation(value); - let stream_name = Self::stream_write_lead_name(value); - let fallback_name = Self::stream_write_lead_name(fallback); - if let (Some(sn), Some(fal)) = (stream_name, fallback_name) - && !self.name_is_defined(sn) - && !self.name_is_defined(fal) - { - // Neither reading resolves — report the classic - // (file-write) name, matching the runtime fallback. - self.report_undefined_name( - format!("Variable '{fal}' is not defined"), - *line, - *column, - ); + let stream_name = Self::stream_write_simple_lead(value); + let fallback_name = Self::stream_write_simple_lead(fallback); + if let (Some(sn), Some(fal)) = (stream_name, fallback_name) { + // Shared continuation (`with `): the right side is + // identical under both readings and unambiguous, so a + // genuinely undefined variable there is still caught. + if let Expression::Concatenation { right, .. } = value { + self.analyze_expression(right); + } + // Report the lead undefined only when NEITHER reading's + // name resolves — a real typo caught without rejecting + // either valid reading. + if !self.name_is_defined(sn) && !self.name_is_defined(fal) { + self.report_undefined_name( + format!("Variable '{fal}' is not defined"), + *line, + *column, + ); + } } + // Any other (desugared) shape: no analysis — runtime decides. } } } @@ -3547,62 +3558,25 @@ impl Analyzer { } } - /// The ambiguous leading operand name of a `write line|chunk` value — the - /// leftmost leaf of the (possibly nested) expression. The two readings of the - /// merged form differ ONLY here (stream `` vs classic `line `); - /// everything to the right is a shared continuation. Reaches the leaf through - /// `with`/binary `left`, an `of`-call `function`, or an `ActionCall`'s callee - /// name (a builtin used with `with`). - fn stream_write_lead_name(expr: &Expression) -> Option<&str> { + /// The ambiguous leading operand name of a `write line|chunk` value, but ONLY + /// for the shapes where that lead is provably the single leftmost bare + /// variable and cleanly separable from the shared continuation: a bare + /// `Variable`, or ` with ` (a `Concatenation` whose left is + /// that variable). Returns `None` for every other shape — including the + /// desugared `starts/ends with` (call), `is between` (duplicated operand), + /// and pattern/`of`/builtin-`with` forms — so the caller skips analysis of + /// those rather than risk rejecting a valid classic file write. + fn stream_write_simple_lead(expr: &Expression) -> Option<&str> { match expr { Expression::Variable(name, ..) => Some(name), - Expression::ActionCall { name, .. } => Some(name), - Expression::Concatenation { left, .. } | Expression::BinaryOperation { left, .. } => { - Self::stream_write_lead_name(left) - } - Expression::FunctionCall { function, .. } => Self::stream_write_lead_name(function), + Expression::Concatenation { left, .. } => match &**left { + Expression::Variable(name, ..) => Some(name), + _ => None, + }, _ => None, } } - /// Analyze the shared continuation of a `write line|chunk` value — every - /// sub-expression EXCEPT the ambiguous leading operand (its leftmost leaf). - /// This catches a genuinely undefined variable in the continuation (e.g. the - /// RHS of ` with missing_suffix`) without flagging the leading operand, - /// which is valid under whichever of the two readings the runtime picks. - fn analyze_stream_write_continuation(&mut self, expr: &Expression) { - match expr { - // The leftmost leaf itself is the ambiguous lead — checked separately. - Expression::Variable(..) => {} - Expression::Concatenation { left, right, .. } => { - self.analyze_stream_write_continuation(left); - self.analyze_expression(right); - } - Expression::BinaryOperation { left, right, .. } => { - self.analyze_stream_write_continuation(left); - self.analyze_expression(right); - } - Expression::FunctionCall { - function, - arguments, - .. - } => { - self.analyze_stream_write_continuation(function); - for arg in arguments { - self.analyze_expression(&arg.value); - } - } - Expression::ActionCall { arguments, .. } => { - // The callee name is the leading lead (skip); analyze the args. - for arg in arguments { - self.analyze_expression(&arg.value); - } - } - // Any other shape has no ambiguous lead to protect — analyze it whole. - other => self.analyze_expression(other), - } - } - /// Whether a bare name resolves to something known (an action parameter, the /// `count` loop variable, a builtin, an in-scope binding, or a container /// property) — i.e. it would NOT be reported as an undefined variable. Used diff --git a/tests/write_line_backcompat_test.rs b/tests/write_line_backcompat_test.rs index 7ba93ae7..357d007b 100644 --- a/tests/write_line_backcompat_test.rs +++ b/tests/write_line_backcompat_test.rs @@ -281,6 +281,32 @@ fn test_ambiguous_write_line_accepts_valid_classic_with_continuation() { ); } +#[test] +fn test_ambiguous_write_line_accepts_desugared_classic_writes() { + // Regression (maintainer review): the analyzer must NOT reject a valid classic + // file write whose value desugars to a call/comparison/pattern, where the + // ambiguous lead is not the plain leftmost leaf. Only the multiword `line …` + // variable is defined; the split stream name is not — and yet each of these + // is a valid pre-existing program that must analyze cleanly. + let cases = [ + "store line path as \"/api\"\nwrite line path starts with \"/\" to \"/tmp/out\"", + "store line score as 3\nwrite line score is between 1 and 5 to \"/tmp/out\"", + "store line subject as \"abc\"\nwrite line subject matches pattern \"a\" to \"/tmp/out\"", + // `write chunk` shares the same ambiguity. + "store chunk path as \"/api\"\nwrite chunk path starts with \"/\" to \"/tmp/out\"", + ]; + for code in cases { + let tokens = lex_wfl_with_positions(code); + let program = Parser::new(&tokens).parse().expect("parse"); + let mut analyzer = Analyzer::new(); + assert!( + analyzer.analyze(&program).is_ok(), + "a valid classic desugared write must not be rejected.\n code: {code}\n errors: {:?}", + analyzer.get_errors() + ); + } +} + #[test] fn test_ambiguous_write_line_still_flags_when_neither_candidate_defined() { // The ambiguous form defers definedness to runtime, but a genuine typo where From 88527869c499acef3a74153fe14089089ba338fe Mon Sep 17 00:00:00 2001 From: Claude Date: Fri, 24 Jul 2026 07:13:04 +0000 Subject: [PATCH 045/132] fix(typechecker): support dot access on stream handles; require text index key Follow-ups to the distinct stream-handle types (maintainer review): - The canonical docs use dot access (upstream.status, upstream.headers[...]), but PropertyAccess only handled containers/maps/gradual types, so the new Custom("HttpStream")/Custom("ResponseStream") handle emitted a false "cannot access property" diagnostic. Add a stream-handle arm returning Unknown (runtime-known field type), mirroring the index-access and member-access paths. - The stream IndexAccess arm returned Any without checking the key; a numeric key (resp[5]) wrongly passed even though runtime object indexing requires a text field name (Map rejected it before the nominal-type change). Require a text-compatible key. Tests: dot access (incl nested header lookup) type-checks; a numeric stream-handle key is rejected. Co-Authored-By: Claude Opus 4.8 Claude-Session: https://claude.ai/code/session_011LxXAm5E4Vy39kRLr8kRnX --- src/typechecker/mod.rs | 26 ++++++++++++++++++++++++-- tests/stream_handle_type_test.rs | 30 ++++++++++++++++++++++++++++++ 2 files changed, 54 insertions(+), 2 deletions(-) diff --git a/src/typechecker/mod.rs b/src/typechecker/mod.rs index 79a8f627..3b1295d1 100644 --- a/src/typechecker/mod.rs +++ b/src/typechecker/mod.rs @@ -3537,9 +3537,25 @@ impl TypeChecker { // at runtime (issue #553). Type::Any => Type::Any, // Stream handles expose fields (`status`/`ok`/`headers`) by - // index; the field type is only known at runtime. + // index; the key must be text (runtime object indexing rejects + // a numeric key), and the field type is only known at runtime. Type::Custom(ref name) if name == "HttpStream" || name == "ResponseStream" => { - Type::Any + if index_type != Type::Text + && index_type != Type::Unknown + && index_type != Type::Any + && index_type != Type::Error + { + self.type_error( + format!("Stream handle field name must be text, got {index_type}"), + Some(Type::Text), + Some(index_type), + *line, + *column, + ); + Type::Error + } else { + Type::Any + } } _ => { self.type_error( @@ -4216,6 +4232,12 @@ impl TypeChecker { // the value type is whatever the map stores. Type::Map(_, value_type) => *value_type, Type::Unknown | Type::Any | Type::Error => Type::Unknown, + // Stream handles expose fields (`status`/`ok`/`headers`) via + // the documented dot form too; the field type is only known at + // runtime. + Type::Custom(ref name) if name == "HttpStream" || name == "ResponseStream" => { + Type::Unknown + } _ => { self.type_error( format!( diff --git a/tests/stream_handle_type_test.rs b/tests/stream_handle_type_test.rs index 849e7475..08bb4e90 100644 --- a/tests/stream_handle_type_test.rs +++ b/tests/stream_handle_type_test.rs @@ -74,6 +74,36 @@ fn test_outbound_stream_handle_fields_are_indexable() { ); } +#[test] +fn test_outbound_stream_handle_dot_access_typechecks() { + // The canonical docs use dot access (`upstream.status`, + // `upstream.headers["content-type"]`); the distinct handle type must support + // it, not just bracket notation. + let code = "open url at \"http://example.com\" and stream response as upstream\n\ + display upstream.status\n\ + store ct as upstream.headers[\"content-type\"]\n\ + close upstream"; + assert!( + typecheck(code).is_ok(), + "dot access on a stream handle must type-check: {:?}", + typecheck(code).err() + ); +} + +#[test] +fn test_stream_handle_numeric_index_is_rejected() { + // Runtime object indexing requires a text field name; a numeric key must be a + // static type error (it was, back when the handle was Map). + let code = "open url at \"http://example.com\" and stream response as resp\n\ + store x as resp[5]\n\ + close resp"; + let errors = typecheck(code).expect_err("a numeric stream-handle key must be a type error"); + assert!( + errors.contains("field name must be text") || errors.contains("must be text"), + "expected a text-key type error, got: {errors}" + ); +} + #[test] fn test_close_ordinary_map_is_rejected() { // A plain user map is NOT closeable — only file/stream handles are. This is From 8a8df6a69457a7d0ed27935d5a01c56f6f432beb Mon Sep 17 00:00:00 2001 From: Claude Date: Fri, 24 Jul 2026 07:13:52 +0000 Subject: [PATCH 046/132] fix(ci): surface TLS temp-dir cleanup failure instead of hiding it Maintainer review (Windows cleanup completeness): the TLS finally deleted the temp dir with -ErrorAction SilentlyContinue even when the server was still alive, racing its open cert/log handles and hiding a failed cleanup (a leaked dir or a lingering fixed-port child). Now give a brief extra grace if the process is still running after Kill()+WaitForExit, then attempt Remove-Item with -ErrorAction Stop and report a failure via a WARN rather than swallowing it silently. Co-Authored-By: Claude Opus 4.8 Claude-Session: https://claude.ai/code/session_011LxXAm5E4Vy39kRLr8kRnX --- scripts/run_web_tests.ps1 | 17 ++++++++++++----- 1 file changed, 12 insertions(+), 5 deletions(-) diff --git a/scripts/run_web_tests.ps1 b/scripts/run_web_tests.ps1 index c9bdf578..33ef7e50 100644 --- a/scripts/run_web_tests.ps1 +++ b/scripts/run_web_tests.ps1 @@ -393,12 +393,19 @@ if (Test-Path "TestPrograms\web_server_tls.wfl") { # race that would leave the dir or fail the Remove-Item). Stop-ServerProcess -Process $tlsProcess if ($tlsProcess -and -not $tlsProcess.HasExited) { - # Still alive after Kill()+WaitForExit — deleting now can race the - # process's open cert/log handles. Surface it instead of hiding a - # failed cleanup behind SilentlyContinue. - Write-Host "[WARN] TLS server still running; temp dir cleanup may be incomplete" -ForegroundColor Yellow + # Still alive after Kill()+WaitForExit — deleting now races the + # process's open cert/log handles. Give a brief extra grace before + # attempting cleanup so we don't fight live handles. + Write-Host "[WARN] TLS server still running after Kill(); waiting briefly before temp cleanup" -ForegroundColor Yellow + try { $null = $tlsProcess.WaitForExit(2000) } catch { } + } + # Attempt cleanup and SURFACE a failure (a leaked temp dir / still-open + # handle) instead of hiding it behind -ErrorAction SilentlyContinue. + try { + Remove-Item -Recurse -Force $tlsDir -ErrorAction Stop + } catch { + Write-Host "[WARN] Failed to remove TLS temp dir ${tlsDir}: $_" -ForegroundColor Yellow } - Remove-Item -Recurse -Force $tlsDir -ErrorAction SilentlyContinue } } } From c912f57c325755ac6f81d4644e8d3a1004818a21 Mon Sep 17 00:00:00 2001 From: Claude Date: Fri, 24 Jul 2026 07:17:18 +0000 Subject: [PATCH 047/132] fix(parser): do not require the unused classic reading to parse (write line/chunk) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Maintainer review: the ambiguous merged form parses BOTH the stream reading and the classic file-write reading (via cursor rewind), but the classic reading is only USED at runtime when the target is a file. Requiring it to parse could reject a valid stream-only value whose grammar the classic reading can't consume. Make the fallback parse fallible (.ok()) and always resume right after the stream value, so the statement parses on its stream reading alone when the classic reading doesn't; the fallback is simply omitted (runtime then has no file reading to fall back to, which is correct — that reading was not valid anyway). Existing tests still exercise the fallback-parses path. Co-Authored-By: Claude Opus 4.8 Claude-Session: https://claude.ai/code/session_011LxXAm5E4Vy39kRLr8kRnX --- src/parser/stmt/io.rs | 16 +++++++++++++--- 1 file changed, 13 insertions(+), 3 deletions(-) diff --git a/src/parser/stmt/io.rs b/src/parser/stmt/io.rs index 4e3d01cb..aa340c1a 100644 --- a/src/parser/stmt/io.rs +++ b/src/parser/stmt/io.rs @@ -896,15 +896,25 @@ impl<'a> IoParser<'a> for Parser<'a> { // Stream reading: split-off `` as the leading operand. let stream_left = Expression::Variable(rest, marker_line, marker_column); let value = self.parse_write_value_from_lead(stream_left)?; + let after_stream = self.cursor.checkpoint(); // Rewind and parse the classic file-write reading with the whole // merged `line ` as the leading operand, over the very same - // tokens, so the two interpretations stay faithful to the source. + // tokens. This alternate interpretation is only USED at runtime + // when the target turns out to be a file, so it must not be + // REQUIRED to parse: a value whose stream reading uses grammar the + // classic reading can't (e.g. a builtin call with named arguments, + // `write line substring with text: "x" and start: 1 to out`) still + // has a valid stream reading. If the classic reading fails to + // parse, drop the fallback rather than failing the statement. self.cursor.rewind(value_start); let file_left = Expression::Variable(id, marker_line, marker_column); - let fallback = self.parse_write_value_from_lead(file_left)?; + let fallback = self.parse_write_value_from_lead(file_left).ok(); + // Always resume right after the stream value, whatever the + // (speculative) fallback parse consumed, so `to ` follows. + self.cursor.rewind(after_stream); - (value, Some(Box::new(fallback))) + (value, fallback.map(Box::new)) }; self.expect_token( From 9d31e9410fc3b3ec4d501433b97910f1a0013e3d Mon Sep 17 00:00:00 2001 From: Claude Date: Fri, 24 Jul 2026 07:19:24 +0000 Subject: [PATCH 048/132] fix(parser): consume a connective before 'as' in start streaming response Copilot review: the streaming-response clause loop broke on an unrecognized token after and/with WITHOUT consuming the connective, so `... with status 200 and as out` failed at expect_token(as) reporting the leftover `and` instead of parsing. Consume a connective that directly precedes `as` (the end-of-clauses join) so the `as ` binding parses cleanly, and correct the comment to describe the actual behavior for a connective before any other unrecognized token. Test: `start streaming response to req with status 200 and as out` parses. Co-Authored-By: Claude Opus 4.8 Claude-Session: https://claude.ai/code/session_011LxXAm5E4Vy39kRLr8kRnX --- src/parser/stmt/web.rs | 15 ++++++++++++--- tests/http_server_streaming_test.rs | 20 ++++++++++++++++++++ 2 files changed, 32 insertions(+), 3 deletions(-) diff --git a/src/parser/stmt/web.rs b/src/parser/stmt/web.rs index 7d2d5f92..ea553723 100644 --- a/src/parser/stmt/web.rs +++ b/src/parser/stmt/web.rs @@ -404,9 +404,11 @@ impl<'a> WebParser<'a> for Parser<'a> { let mut headers = None; // Optional clauses joined by `with`/`and`, in any order: `status `, - // `content type `, `headers `. Mirrors the `respond` clause loop; - // an `and`/`with` that does not introduce a known clause (e.g. before - // `as`) ends the loop. + // `content type `, `headers `. Mirrors the `respond` clause loop. + // A connective directly before `as` is the end-of-clauses join and is + // consumed so the `as ` binding parses; a connective before any + // other unrecognized token ends the loop WITHOUT being consumed, so the + // trailing `expect_token(as)` reports the malformed clause. loop { let connective = matches!( self.cursor.peek(), @@ -490,6 +492,13 @@ impl<'a> WebParser<'a> for Parser<'a> { headers = Some(Expression::Variable(rest.to_string(), id_line, id_column)); } } + // A connective directly before `as` just joins the clause list to + // the binding; consume it so `as ` parses cleanly instead of + // `expect_token(as)` tripping over the leftover `and`/`with`. + Token::KeywordAs => { + self.bump_sync(); // consume the connective; `as` stays next + break; + } _ => break, } } diff --git a/tests/http_server_streaming_test.rs b/tests/http_server_streaming_test.rs index d1710b0c..7d096eec 100644 --- a/tests/http_server_streaming_test.rs +++ b/tests/http_server_streaming_test.rs @@ -49,6 +49,26 @@ fn test_start_streaming_response_parses() { } } +#[test] +fn test_start_streaming_response_connective_before_as_parses() { + // A connective (`and`/`with`) directly before `as` is consumed as the + // end-of-clauses join, so `... with status 200 and as out` parses rather than + // failing at the `as` binding with a confusing "expected as, found and". + let stmt = + parse_single_statement(r#"start streaming response to req with status 200 and as out"#); + match stmt { + Statement::StartStreamingResponseStatement { + status, + variable_name, + .. + } => { + assert!(status.is_some()); + assert_eq!(variable_name, "out"); + } + other => panic!("Expected StartStreamingResponseStatement, got {other:?}"), + } +} + #[test] fn test_content_type_variable_binds_correct_name() { // `content type ` where is a bare identifier: the lexer merges it From 271d98924ed519b8568b77a4ac8161b5f6ef87ce Mon Sep 17 00:00:00 2001 From: Claude Date: Fri, 24 Jul 2026 07:34:28 +0000 Subject: [PATCH 049/132] fix(P1): bound an active outbound read by the absolute stream deadline MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit run_http_with_budget derived its timeout purely from the run/budget duration and DISCARDED the caller's configured_timeout, which already encodes the stream's idle timeout AND its remaining absolute-total deadline (min(idle, remaining) from stream_pull). So with timeout_seconds=10 and outbound_stream_max_seconds=1, a head-then-stall upstream let `wait for next chunk` wait ~10s instead of ~1s. Compose the operation deadline as MIN(configured_timeout, run/budget duration), reporting the stream Timeout or the budget Deadline depending on which bound fired; because configured_timeout is always finite, the read is now bounded even when the budget has no run-wide deadline. Also start the absolute total-lifetime clock at request initiation (before send()), not after the head arrives, so connect/header time counts toward the documented total, and bound the head phase by the remaining-to-total as well. Red->Green: new real-socket test outbound_stream_deadline_test — a mock upstream sends the head then stalls; the read now fails at ~1s (was ~10s). Existing outbound HTTP budget/cancellation/stream tests still pass. Co-Authored-By: Claude Opus 4.8 Claude-Session: https://claude.ai/code/session_011LxXAm5E4Vy39kRLr8kRnX --- src/interpreter/mod.rs | 82 +++++++++++++++++++------- tests/outbound_stream_deadline_test.rs | 82 ++++++++++++++++++++++++++ 2 files changed, 144 insertions(+), 20 deletions(-) create mode 100644 tests/outbound_stream_deadline_test.rs diff --git a/src/interpreter/mod.rs b/src/interpreter/mod.rs index 4621d673..3f2b4569 100644 --- a/src/interpreter/mod.rs +++ b/src/interpreter/mod.rs @@ -1811,7 +1811,29 @@ impl IoClient { } let method_owned = method.to_string(); - let configured_timeout = Duration::from_secs(self.config.timeout_seconds.max(1)); + // Start the absolute total-lifetime clock at request initiation — BEFORE + // `send()` — so connect + header time counts toward + // `outbound_stream_max_seconds`, matching the documented "total lifetime + // measured from when the stream is opened". The head phase below is then + // bounded by the remaining time to that deadline as well as the idle + // timeout, so a stalled connect/header handshake cannot outlive the total. + let total_deadline = match self.config.outbound_stream_max_seconds { + 0 => None, // sentinel: no absolute total cap + secs => Some(Instant::now() + Duration::from_secs(secs)), + }; + let idle_timeout = Duration::from_secs(self.config.timeout_seconds.max(1)); + let configured_timeout = match total_deadline { + Some(deadline) => { + let remaining = deadline.saturating_duration_since(Instant::now()); + if remaining.is_zero() { + return Err(HttpClientError::Timeout { + seconds: self.config.outbound_stream_max_seconds, + }); + } + idle_timeout.min(remaining) + } + None => idle_timeout, + }; let op = async move { request.send().await.map_err(|e| { HttpClientError::Request(format!("Failed to send HTTP {method_owned} request: {e}")) @@ -1850,10 +1872,8 @@ impl IoClient { let stream = response .bytes_stream() .map(|chunk| chunk.map(|b| b.to_vec())); - let total_deadline = match self.config.outbound_stream_max_seconds { - 0 => None, // sentinel: no absolute total cap - secs => Some(Instant::now() + Duration::from_secs(secs)), - }; + // `total_deadline` was started at request initiation above (before the + // head was sent) so it covers connect/header time too. let handle = HttpStreamHandle { stream: Box::pin(stream), buffer: Vec::new(), @@ -2231,11 +2251,25 @@ impl IoClient { F: std::future::Future>, { let deadline = Self::outbound_http_deadline(&budget, configured_timeout)?; - let timeout_duration = match deadline { + // The budget/run-derived finite duration, if any. + let budget_duration = match deadline { OutboundHttpDeadline::None => None, OutboundHttpDeadline::Execution { remaining, .. } => Some(remaining), OutboundHttpDeadline::MainLoop { duration } => Some(duration), }; + // The operation is bounded by the SHORTER of the caller's configured + // timeout — which already encodes the stream's idle timeout AND its + // remaining absolute-total deadline (see `stream_pull`/`open_http_stream`) + // — and the run/budget deadline. Whichever is smaller decides both how + // long we wait and which error we report. `configured_timeout` is always + // finite, so the operation is bounded even when the budget has no + // run-wide deadline (previously that path discarded the stream deadline + // entirely and could wait out the whole run). + let budget_is_binding = matches!(budget_duration, Some(bd) if bd <= configured_timeout); + let timeout_duration = match budget_duration { + Some(bd) if budget_is_binding => bd, + _ => configured_timeout, + }; let cancellation_budget = Arc::clone(&budget); let cancellation = async move { @@ -2246,12 +2280,7 @@ impl IoClient { tokio::time::sleep(HTTP_CANCELLATION_POLL_INTERVAL).await; } }; - let timeout = async move { - match timeout_duration { - Some(duration) => tokio::time::sleep(duration).await, - None => std::future::pending::<()>().await, - } - }; + let timeout = async move { tokio::time::sleep(timeout_duration).await }; tokio::pin!(operation); tokio::pin!(cancellation); @@ -2259,15 +2288,28 @@ impl IoClient { tokio::select! { result = &mut operation => result, _ = &mut cancellation => Err(HttpClientError::Budget(BudgetExceeded::Cancelled)), - _ = &mut timeout => match deadline { - OutboundHttpDeadline::Execution { limit_secs, .. } => { - Err(HttpClientError::Budget(BudgetExceeded::Deadline { limit_secs })) - } - OutboundHttpDeadline::MainLoop { duration } => { - Err(HttpClientError::Timeout { seconds: duration.as_secs() }) + _ = &mut timeout => { + if budget_is_binding { + // The run/budget deadline was the shorter bound. + match deadline { + OutboundHttpDeadline::Execution { limit_secs, .. } => { + Err(HttpClientError::Budget(BudgetExceeded::Deadline { limit_secs })) + } + OutboundHttpDeadline::MainLoop { duration } => { + Err(HttpClientError::Timeout { seconds: duration.as_secs() }) + } + OutboundHttpDeadline::None => { + unreachable!("budget cannot be binding when there is no budget deadline") + } + } + } else { + // The caller's configured timeout (stream idle / absolute + // total) was the shorter bound. + Err(HttpClientError::Timeout { + seconds: configured_timeout.as_secs().max(1), + }) } - OutboundHttpDeadline::None => unreachable!("disabled timeout cannot complete"), - }, + } } } diff --git a/tests/outbound_stream_deadline_test.rs b/tests/outbound_stream_deadline_test.rs new file mode 100644 index 00000000..157a31c4 --- /dev/null +++ b/tests/outbound_stream_deadline_test.rs @@ -0,0 +1,82 @@ +//! Real-socket regression for P1: `outbound_stream_max_seconds` must bound an +//! ACTIVE body read, not be overridden by the broader run/budget duration. +//! +//! A mock upstream sends the response head immediately and then stalls (never +//! sends a body chunk). With `timeout_seconds = 10` but +//! `outbound_stream_max_seconds = 1`, a `wait for next chunk` must fail in about +//! one second (the absolute stream deadline), not wait out the ten-second run +//! timeout. Before the fix, `run_http_with_budget` derived its timeout purely +//! from the budget/run duration and discarded the stream's shorter +//! `min(idle, remaining_total)`, so the read waited ~10s. + +use std::sync::Arc; +use std::time::{Duration, Instant}; +use tokio::io::{AsyncReadExt, AsyncWriteExt}; +use wfl::Interpreter; +use wfl::config::WflConfig; +use wfl::lexer::lex_wfl_with_positions; +use wfl::parser::Parser; + +/// Bind an ephemeral port and serve exactly one connection: read the request, +/// send a chunked-encoding response head, then hold the socket open WITHOUT +/// sending any body chunk (a head-then-stall upstream). +async fn spawn_head_then_stall_upstream() -> u16 { + let listener = tokio::net::TcpListener::bind("127.0.0.1:0") + .await + .expect("bind mock upstream"); + let port = listener.local_addr().unwrap().port(); + tokio::spawn(async move { + if let Ok((mut sock, _)) = listener.accept().await { + // Consume the request head so the client's write completes. + let mut buf = [0u8; 1024]; + let _ = sock.read(&mut buf).await; + let head = "HTTP/1.1 200 OK\r\n\ + Content-Type: text/plain\r\n\ + Transfer-Encoding: chunked\r\n\r\n"; + let _ = sock.write_all(head.as_bytes()).await; + let _ = sock.flush().await; + // Stall: keep the connection open but never send a body chunk. + tokio::time::sleep(Duration::from_secs(30)).await; + drop(sock); + } + }); + port +} + +#[tokio::test] +async fn test_outbound_stream_absolute_deadline_bounds_active_read() { + let port = spawn_head_then_stall_upstream().await; + + let code = format!( + r#"open url at "http://127.0.0.1:{port}/" and stream response as s +wait for next chunk from s as c"# + ); + let tokens = lex_wfl_with_positions(&code); + let program = Parser::new(&tokens).parse().expect("parse"); + + // Run/idle timeout 10s, absolute stream lifetime 1s. + let mut config = WflConfig::default(); + config.timeout_seconds = 10; + config.outbound_stream_max_seconds = 1; + let mut interp = Interpreter::with_config(Arc::new(config)); + + let start = Instant::now(); + let result = interp.interpret(&program).await; + let elapsed = start.elapsed(); + + assert!( + result.is_err(), + "a stalled stream read must fail, not hang or succeed" + ); + assert!( + elapsed < Duration::from_secs(4), + "`wait for next chunk` must fail near the 1s absolute stream deadline, \ + not the 10s run timeout (took {elapsed:?})" + ); + // And it must not fail instantly either — the head arrived and the 1s clock + // had to elapse. + assert!( + elapsed >= Duration::from_millis(500), + "the read failed too early to be the ~1s absolute deadline (took {elapsed:?})" + ); +} From 9919c35c149286b7265045d671ddcc467e305168 Mon Sep 17 00:00:00 2001 From: Claude Date: Fri, 24 Jul 2026 07:43:26 +0000 Subject: [PATCH 050/132] test: build WflConfig with struct-update syntax (clippy field_reassign_with_default) CI's clippy --all-targets flagged the P1-B deadline test's `let mut config = WflConfig::default(); config.x = ...` (field_reassign_with_default, denied by -D warnings). Use struct-update syntax instead. No behavior change. Co-Authored-By: Claude Opus 4.8 Claude-Session: https://claude.ai/code/session_011LxXAm5E4Vy39kRLr8kRnX --- tests/outbound_stream_deadline_test.rs | 8 +++++--- 1 file changed, 5 insertions(+), 3 deletions(-) diff --git a/tests/outbound_stream_deadline_test.rs b/tests/outbound_stream_deadline_test.rs index 157a31c4..68df8559 100644 --- a/tests/outbound_stream_deadline_test.rs +++ b/tests/outbound_stream_deadline_test.rs @@ -55,9 +55,11 @@ wait for next chunk from s as c"# let program = Parser::new(&tokens).parse().expect("parse"); // Run/idle timeout 10s, absolute stream lifetime 1s. - let mut config = WflConfig::default(); - config.timeout_seconds = 10; - config.outbound_stream_max_seconds = 1; + let config = WflConfig { + timeout_seconds: 10, + outbound_stream_max_seconds: 1, + ..WflConfig::default() + }; let mut interp = Interpreter::with_config(Arc::new(config)); let start = Instant::now(); From 60e6bdaafd09ab237a54e0fdef4c81526da4d4ab Mon Sep 17 00:00:00 2001 From: Claude Date: Fri, 24 Jul 2026 07:46:43 +0000 Subject: [PATCH 051/132] fix(P1): make outbound streams handler-owned, closed on every exit path MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Maintainer review: outbound httpstream* handles lived only in the interpreter- wide IoClient.stream_handles map; RunState/IsolatedHandler::drop tracked and closed downstream response streams and pending requests but NOT upstream handles. A handler that opened `... stream response as up` and then ended (normal, error, panic, cancellation, loop exit) without closing leaked the upstream connection until the whole interpreter tore down. Track outbound handle ids per-handler in RunState.open_http_streams (swapped per-poll like the other run state). Add on open; untrack on EOF/error/explicit close; and on every handler exit — IsolatedHandler::drop for the concurrent loop, and close_open_http_streams() at the serial-loop/program-exit cleanup sites — drop any still-open handles from stream_handles (via a synchronous try_lock), which cancels their in-flight upstream requests. Idempotent and best-effort. Red->Green real-socket test (outbound_stream_ownership_test): a program opens an outbound stream, reads one chunk, and ends WITHOUT close while the interpreter is still alive; the mock upstream observes its client disconnect only because handler-exit cleanup cancelled the upstream (confirmed Red by disabling the cleanup). Existing concurrent/server-streaming isolation tests still pass. Co-Authored-By: Claude Opus 4.8 Claude-Session: https://claude.ai/code/session_011LxXAm5E4Vy39kRLr8kRnX --- src/interpreter/mod.rs | 81 ++++++++++++++++++++++++- tests/outbound_stream_ownership_test.rs | 78 ++++++++++++++++++++++++ 2 files changed, 157 insertions(+), 2 deletions(-) create mode 100644 tests/outbound_stream_ownership_test.rs diff --git a/src/interpreter/mod.rs b/src/interpreter/mod.rs index 3f2b4569..c91a804e 100644 --- a/src/interpreter/mod.rs +++ b/src/interpreter/mod.rs @@ -815,6 +815,13 @@ struct RunState { /// is answered 500 immediately instead of leaving the client to wait out the /// request timeout (see `fail_unanswered_requests`). open_pending_requests: Vec, + /// Outbound streaming-response handle ids (`... stream response as `) + /// this handler opened and has not yet closed/exhausted. Handler-OWNED: when + /// the handler ends on any path (normal, error, panic, cancellation, loop + /// exit) these are dropped from `IoClient.stream_handles`, which cancels the + /// in-flight upstream request — so an abandoned proxy read never leaks an + /// upstream connection or handle past the handler's lifetime. + open_http_streams: Vec, } impl RunState { @@ -870,6 +877,10 @@ impl<'a, T> Drop for IsolatedHandler<'a, T> { .close_response_streams(&self.state.open_response_streams); self.interp .fail_unanswered_requests(&self.state.open_pending_requests); + // Drop any outbound streams this handler still owns, cancelling their + // in-flight upstream requests so an abandoned proxy read never leaks. + self.interp + .close_http_streams(&self.state.open_http_streams); } } @@ -1257,6 +1268,12 @@ pub struct Interpreter { /// handler tracks only its own requests; any still unanswered when the handler /// ends are answered 500 immediately (see `fail_unanswered_requests`). open_pending_requests: RefCell>, + /// Outbound stream handle ids (`... stream response as `) the currently + /// executing handler opened and has not yet closed/exhausted. Part of the + /// per-handler `RunState` (swapped per poll); any still open when the handler + /// ends are dropped, cancelling their upstream requests (see + /// `close_http_streams`). + open_http_streams: RefCell>, #[allow(dead_code)] // Used for future security features config: Arc, // Configuration for security and other settings current_source_file: RefCell>, // Currently executing source file (for path resolution) @@ -3377,6 +3394,7 @@ impl Interpreter { server_response_streams: RefCell::new(HashMap::new()), open_response_streams: RefCell::new(Vec::new()), open_pending_requests: RefCell::new(Vec::new()), + open_http_streams: RefCell::new(Vec::new()), next_response_stream_id: std::cell::Cell::new(1), config, current_source_file: RefCell::new(None), // No source file initially @@ -3931,6 +3949,44 @@ impl Interpreter { &mut *self.open_pending_requests.borrow_mut(), &mut state.open_pending_requests, ); + std::mem::swap( + &mut *self.open_http_streams.borrow_mut(), + &mut state.open_http_streams, + ); + } + + /// Drop each outbound streaming handle whose id is in `ids` from + /// `IoClient.stream_handles`, cancelling its in-flight upstream request + /// (dropping the reqwest body stream aborts the connection). Best-effort and + /// synchronous (usable from `Drop`): if the async lock is momentarily held, + /// the handles remain and are reclaimed at interpreter teardown. Idempotent — + /// an id already removed by EOF/error/explicit `close` is a no-op. + fn close_http_streams(&self, ids: &[String]) { + if ids.is_empty() { + return; + } + if let Ok(mut map) = self.io_client.stream_handles.try_lock() { + for id in ids { + map.remove(id); + } + } + } + + /// Drain and drop every outbound stream the current (serial) handler left + /// open. Called at the end of each serial `main loop` iteration and at program + /// exit, mirroring the concurrent path's per-handler `Drop`. + fn close_open_http_streams(&self) { + let ids = std::mem::take(&mut *self.open_http_streams.borrow_mut()); + self.close_http_streams(&ids); + } + + /// Stop tracking an outbound stream id as handler-owned — it has already left + /// `IoClient.stream_handles` (EOF, error, or an explicit `close`), so the + /// handler-exit cleanup must not try to (re-)drop it. + fn untrack_http_stream(&self, handle_id: &str) { + self.open_http_streams + .borrow_mut() + .retain(|id| id != handle_id); } /// Close (drop the sender for) each server response stream whose handle id is @@ -4324,6 +4380,7 @@ impl Interpreter { // `pending_responses`, hanging the client and leaking the entry. self.close_open_response_streams(); self.fail_open_pending_requests(); + self.close_open_http_streams(); // Reset to the inherited base depth (0 for a top-level run/REPL; the // parent's live depth for an `execute file` child) so recursion // accounting spans the execute-file boundary instead of granting the @@ -4506,6 +4563,7 @@ impl Interpreter { // top-level streams and unanswered requests before returning. self.close_open_response_streams(); self.fail_open_pending_requests(); + self.close_open_http_streams(); return Err(errors); } @@ -4590,6 +4648,7 @@ impl Interpreter { // the client's body rather than leaving it hanging until process death. self.close_open_response_streams(); self.fail_open_pending_requests(); + self.close_open_http_streams(); self.assert_invariants(); if errors.is_empty() { @@ -5386,6 +5445,7 @@ impl Interpreter { // hanging or waiting out the request timeout. self.close_open_response_streams(); self.fail_open_pending_requests(); + self.close_open_http_streams(); let result = result?; _last_value = result.0; @@ -5697,6 +5757,9 @@ impl Interpreter { }; if let Some(id) = client_id { self.io_client.close_stream(&id).await; + // Drop it from the handler's ownership tracking so the + // exit cleanup does not try to re-close it. + self.untrack_http_stream(&id); Ok((Value::Null, ControlFlow::None)) } else if let Some(id) = server_id { // Dropping the sender ends the response body stream. @@ -7057,6 +7120,10 @@ impl Interpreter { .await { Ok((status, response_headers, handle_id)) => { + // Track the outbound handle as handler-owned so it is + // dropped (cancelling the upstream) if the handler ends + // without closing/exhausting it. + self.open_http_streams.borrow_mut().push(handle_id.clone()); let mut headers_map = HashMap::new(); for (name, value) in response_headers { headers_map.insert(name, Value::Text(value.into())); @@ -7109,11 +7176,17 @@ impl Interpreter { } // Clean EOF binds `nothing` so `check if chunk is nothing` ends the loop. Ok(None) => { + // The handle left the map at EOF — stop owning it. + self.untrack_http_stream(&handle_id); env.borrow_mut() .define_or_replace(variable_name, Value::Null); Ok((Value::Null, ControlFlow::None)) } - Err(error) => Err(self.http_client_error(error, *line, *column)), + Err(error) => { + // The read dropped the handle (timeout/cancel/error). + self.untrack_http_stream(&handle_id); + Err(self.http_client_error(error, *line, *column)) + } } } Statement::WaitForNextLineStatement { @@ -7136,11 +7209,15 @@ impl Interpreter { Ok((Value::Null, ControlFlow::None)) } Ok(None) => { + self.untrack_http_stream(&handle_id); env.borrow_mut() .define_or_replace(variable_name, Value::Null); Ok((Value::Null, ControlFlow::None)) } - Err(error) => Err(self.http_client_error(error, *line, *column)), + Err(error) => { + self.untrack_http_stream(&handle_id); + Err(self.http_client_error(error, *line, *column)) + } } } Statement::RepeatWhileLoop { diff --git a/tests/outbound_stream_ownership_test.rs b/tests/outbound_stream_ownership_test.rs new file mode 100644 index 00000000..48476777 --- /dev/null +++ b/tests/outbound_stream_ownership_test.rs @@ -0,0 +1,78 @@ +//! Real-socket regression for P1: an outbound stream is handler-OWNED — when the +//! run/handler ends with the stream still open (no explicit `close`), the handle +//! is dropped, cancelling the in-flight upstream request. Otherwise an abandoned +//! proxy read leaks the upstream connection until the whole interpreter tears +//! down. +//! +//! The mock upstream streams one chunk and then keeps trying to write; when the +//! client drops the connection its write fails and it signals a oneshot. The WFL +//! program reads one chunk and ends WITHOUT `close`, while the interpreter is +//! still alive — so a leak would keep the connection open and the test times out. + +use std::sync::Arc; +use std::time::Duration; +use tokio::io::{AsyncReadExt, AsyncWriteExt}; +use wfl::Interpreter; +use wfl::config::WflConfig; +use wfl::lexer::lex_wfl_with_positions; +use wfl::parser::Parser; + +/// One-connection upstream: send a chunked head + one body chunk, then keep +/// writing keepalive chunks. When the client has disconnected, a write fails and +/// we signal via the oneshot. +async fn spawn_streaming_upstream() -> (u16, tokio::sync::oneshot::Receiver<()>) { + let listener = tokio::net::TcpListener::bind("127.0.0.1:0") + .await + .expect("bind mock upstream"); + let port = listener.local_addr().unwrap().port(); + let (tx, rx) = tokio::sync::oneshot::channel(); + tokio::spawn(async move { + if let Ok((mut sock, _)) = listener.accept().await { + let mut buf = [0u8; 1024]; + let _ = sock.read(&mut buf).await; + let head = "HTTP/1.1 200 OK\r\n\ + Content-Type: text/plain\r\n\ + Transfer-Encoding: chunked\r\n\r\n"; + let _ = sock.write_all(head.as_bytes()).await; + let _ = sock.write_all(b"5\r\nhello\r\n").await; // one chunk: "hello" + let _ = sock.flush().await; + + // Keep sending; the first failed write means the client dropped. + for _ in 0..100 { + tokio::time::sleep(Duration::from_millis(100)).await; + if sock.write_all(b"1\r\nx\r\n").await.is_err() || sock.flush().await.is_err() { + let _ = tx.send(()); + return; + } + } + } + }); + (port, rx) +} + +#[tokio::test] +async fn test_outbound_stream_closed_when_run_ends_without_close() { + let (port, disconnect_rx) = spawn_streaming_upstream().await; + + // Reads one chunk, then the program ENDS without `close s`. + let code = format!( + r#"open url at "http://127.0.0.1:{port}/" and stream response as s +wait for next chunk from s as first"# + ); + let tokens = lex_wfl_with_positions(&code); + let program = Parser::new(&tokens).parse().expect("parse"); + + let mut interp = Interpreter::with_config(Arc::new(WflConfig::default())); + interp.interpret(&program).await.expect("interpret"); + + // The interpreter is still alive here, so only handler-exit cleanup (not the + // interpreter's own teardown) can have dropped the outbound handle. The mock + // therefore only sees its client disconnect if that cleanup cancelled the + // upstream — a leak would keep the connection open and this times out. + tokio::time::timeout(Duration::from_secs(5), disconnect_rx) + .await + .expect("upstream not disconnected after the run ended — outbound handle leaked") + .expect("disconnect sender dropped unexpectedly"); + + drop(interp); +} From 9cc732732a65b39464dd1519a144e6a90386bac4 Mon Sep 17 00:00:00 2001 From: Claude Date: Fri, 24 Jul 2026 07:54:29 +0000 Subject: [PATCH 052/132] fix(P1): cancel a blocked upstream read when the downstream client disconnects MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Maintainer review: a proxy handler blocked in `wait for next line/chunk` on the upstream received no signal when the browser disconnected — it was only noticed at the next downstream write, or (now) at the absolute stream deadline. Give the read a proactive disconnect signal: the downstream response stream's mpsc Sender `closed()` resolves when hyper drops the client's body Receiver. Clone the senders of the handler's open response streams (no RefCell borrow held across the await) and `select!` the upstream read against "any downstream disconnected". On disconnect, dropping the read future cancels the upstream; we also close the handle and return a catchable Cancelled error so the handler unwinds and its handler-owned cleanup runs. Red->Green real-socket test (outbound_stream_disconnect_test): mock upstream sends one chunk then stalls; a WFL concurrent proxy relays it; the client reads the first chunk and disconnects while the handler is blocked on the stalled upstream read; the mock observes its own connection close within the window only because the blocked read was cancelled (confirmed Red by disabling the select). Full lib + HTTP/concurrent/streaming suites and clippy --all-targets stay green. Co-Authored-By: Claude Opus 4.8 Claude-Session: https://claude.ai/code/session_011LxXAm5E4Vy39kRLr8kRnX --- src/interpreter/mod.rs | 78 ++++++++++-- tests/outbound_stream_disconnect_test.rs | 155 +++++++++++++++++++++++ 2 files changed, 225 insertions(+), 8 deletions(-) create mode 100644 tests/outbound_stream_disconnect_test.rs diff --git a/src/interpreter/mod.rs b/src/interpreter/mod.rs index c91a804e..0b3a739f 100644 --- a/src/interpreter/mod.rs +++ b/src/interpreter/mod.rs @@ -3989,6 +3989,36 @@ impl Interpreter { .retain(|id| id != handle_id); } + /// Clone the sender of every downstream response stream this handler owns. A + /// client disconnect drops the stream's `Receiver`, so `Sender::closed()` on + /// a clone resolves — a proactive disconnect signal that a blocked upstream + /// read can select against, so a proxy handler wakes and cancels its upstream + /// the moment the browser goes away, instead of only discovering it at the + /// next downstream `write` or when the absolute stream deadline elapses. + /// Returns owned clones (no `RefCell` borrow is held across the later await). + fn downstream_disconnect_senders(&self) -> Vec>> { + let open = self.open_response_streams.borrow(); + if open.is_empty() { + return Vec::new(); + } + let map = self.server_response_streams.borrow(); + open.iter() + .filter_map(|id| map.get(id).map(|(tx, _)| tx.clone())) + .collect() + } + + /// Await until ANY of `senders`' downstream clients has disconnected (its + /// `Receiver` dropped). Never resolves if `senders` is empty — the caller + /// guards that case so `select!` still has a live branch. + async fn any_downstream_disconnected(senders: Vec>>) { + if senders.is_empty() { + std::future::pending::<()>().await; + return; + } + let closes: Vec<_> = senders.iter().map(|tx| Box::pin(tx.closed())).collect(); + let _ = futures_util::future::select_all(closes).await; + } + /// Close (drop the sender for) each server response stream whose handle id is /// in `ids`, ending its body so the client stops waiting. Idempotent — an id /// already closed by an explicit `close out` (or a disconnect) is a no-op — @@ -7160,11 +7190,29 @@ impl Interpreter { let handle_id = self .resolve_stream_handle(source, &env, *line, *column) .await?; - match self + // Race the upstream read against a downstream client disconnect so + // a blocked proxy read is cancelled promptly when the browser goes + // away (see `downstream_disconnect_senders`). + let disconnect = + Self::any_downstream_disconnected(self.downstream_disconnect_senders()); + let read = self .io_client - .next_chunk(&handle_id, Arc::clone(&self.budget)) - .await - { + .next_chunk(&handle_id, Arc::clone(&self.budget)); + let outcome = { + tokio::pin!(read); + tokio::pin!(disconnect); + tokio::select! { + r = &mut read => r, + _ = &mut disconnect => { + // Client gone: dropping `read` above already cancels + // the upstream; close the handle too in case the read + // had not yet taken it, and report cancellation. + self.io_client.close_stream(&handle_id).await; + Err(HttpClientError::Budget(BudgetExceeded::Cancelled)) + } + } + }; + match outcome { // Raw bytes as Binary so callers can handle any payload. // define_or_replace (not define) so re-reading into the same // variable across a loop refreshes it, matching @@ -7198,11 +7246,25 @@ impl Interpreter { let handle_id = self .resolve_stream_handle(source, &env, *line, *column) .await?; - match self + // Race the upstream read against a downstream client disconnect + // (see the `wait for next chunk` handler). + let disconnect = + Self::any_downstream_disconnected(self.downstream_disconnect_senders()); + let read = self .io_client - .next_line(&handle_id, Arc::clone(&self.budget)) - .await - { + .next_line(&handle_id, Arc::clone(&self.budget)); + let outcome = { + tokio::pin!(read); + tokio::pin!(disconnect); + tokio::select! { + r = &mut read => r, + _ = &mut disconnect => { + self.io_client.close_stream(&handle_id).await; + Err(HttpClientError::Budget(BudgetExceeded::Cancelled)) + } + } + }; + match outcome { Ok(Some(line_text)) => { let value = Value::Text(line_text.into()); env.borrow_mut().define_or_replace(variable_name, value); diff --git a/tests/outbound_stream_disconnect_test.rs b/tests/outbound_stream_disconnect_test.rs new file mode 100644 index 00000000..f5f158a1 --- /dev/null +++ b/tests/outbound_stream_disconnect_test.rs @@ -0,0 +1,155 @@ +//! Real-socket regression for P1: a downstream (browser) disconnect cancels a +//! handler BLOCKED in an upstream `wait for next chunk`, closing the upstream TCP +//! connection and recovering the handler — instead of the handler hanging until +//! the absolute stream deadline. +//! +//! Topology: mock upstream (sends one chunk, then stalls) <- WFL proxy server -> +//! reqwest client. The client reads the first proxied chunk, then disconnects +//! while the handler is blocked reading the (stalled) upstream. The mock detects +//! its own connection closing (a blocking read returns 0 at peer close) only if +//! the handler's blocked upstream read was actually cancelled. + +use std::time::Duration; +use tokio::io::{AsyncReadExt, AsyncWriteExt}; +use wfl::Interpreter; +use wfl::lexer::lex_wfl_with_positions; +use wfl::parser::Parser; + +/// Upstream: send a chunked head + one body chunk, then STALL (send nothing +/// more, so the proxy's next read blocks). Detect the proxy dropping the +/// connection via a blocking read that returns 0 at peer close. +async fn spawn_one_chunk_then_stall_upstream() -> (u16, tokio::sync::oneshot::Receiver<()>) { + let listener = tokio::net::TcpListener::bind("127.0.0.1:0") + .await + .expect("bind mock upstream"); + let port = listener.local_addr().unwrap().port(); + let (tx, rx) = tokio::sync::oneshot::channel(); + tokio::spawn(async move { + if let Ok((mut sock, _)) = listener.accept().await { + let mut buf = [0u8; 1024]; + let _ = sock.read(&mut buf).await; // request head + let head = "HTTP/1.1 200 OK\r\n\ + Content-Type: text/plain\r\n\ + Transfer-Encoding: chunked\r\n\r\n"; + let _ = sock.write_all(head.as_bytes()).await; + let _ = sock.write_all(b"5\r\nhello\r\n").await; + let _ = sock.flush().await; + // Stall: send nothing more. Block on read; when the proxy drops the + // upstream (its blocked read cancelled by the client disconnect), the + // peer close surfaces here as Ok(0) / Err. + loop { + match sock.read(&mut buf).await { + Ok(0) | Err(_) => { + let _ = tx.send(()); + return; + } + Ok(_) => {} // unexpected client->server data; ignore + } + } + } + }); + (port, rx) +} + +fn start_proxy_server(code: String) -> std::thread::JoinHandle<()> { + std::thread::spawn(move || { + let rt = tokio::runtime::Runtime::new().expect("runtime"); + rt.block_on(async { + let tokens = lex_wfl_with_positions(&code); + let ast = Parser::new(&tokens).parse().expect("parse"); + let mut interp = Interpreter::new(); + if let Err(errors) = interp.interpret(&ast).await { + panic!("proxy interpreter failed: {errors:?}"); + } + }); + }) +} + +async fn wait_for_server(port: u16) { + let addr = format!("127.0.0.1:{port}"); + for _ in 0..300 { + if tokio::net::TcpStream::connect(&addr).await.is_ok() { + return; + } + tokio::time::sleep(Duration::from_millis(50)).await; + } + panic!("proxy server on {addr} did not become ready"); +} + +#[tokio::test] +async fn test_downstream_disconnect_cancels_blocked_upstream_read() { + let (upstream_port, mut upstream_disconnect) = spawn_one_chunk_then_stall_upstream().await; + + let proxy_port = 8351; + // The handler proxies: read chunks from upstream and write them downstream. + // After the first chunk it blocks on the stalled upstream. `outbound_stream_max_seconds` + // is the default (300s), so ONLY a disconnect can cancel that blocked read + // within the test window. + let code = format!( + r#" + listen on port {proxy_port} as srv + main loop concurrently: + wait for request comes in on srv as req with timeout 20000 + store p as req["path"] + check if p is equal to "/shutdown": + respond to req with "bye" + close server srv + break + otherwise: + open url at "http://127.0.0.1:{upstream_port}/" and stream response as up + start streaming response to req with status 200 and content type "text/plain" as down + count from 1 to 100: + wait for next chunk from up as c + check if c is nothing: + break + otherwise: + write chunk c to down + end check + end count + end check + end loop + "# + ); + let server = start_proxy_server(code); + wait_for_server(proxy_port).await; + + // Client: read the first proxied chunk, then DISCONNECT (drop the response) + // while the handler is blocked reading the stalled upstream. + { + let resp = reqwest::Client::new() + .get(format!("http://127.0.0.1:{proxy_port}/proxy")) + .send() + .await + .expect("proxy request failed"); + assert_eq!(resp.status().as_u16(), 200); + let mut resp = resp; + let first = resp.chunk().await.expect("read first chunk"); + assert_eq!( + first.as_deref(), + Some(&b"hello"[..]), + "expected the first proxied chunk" + ); + // Drop `resp` here -> client disconnects. + } + + // The mock upstream must observe ITS connection close promptly — proving the + // handler's blocked upstream read was cancelled by the disconnect (not left + // hanging until the absolute deadline). + tokio::time::timeout(Duration::from_secs(5), &mut upstream_disconnect) + .await + .expect( + "upstream was not closed after the client disconnected — blocked read not cancelled", + ) + .expect("upstream disconnect sender dropped"); + + // Stop the proxy server. + let _ = reqwest::Client::new() + .get(format!("http://127.0.0.1:{proxy_port}/shutdown")) + .send() + .await; + match tokio::task::spawn_blocking(move || server.join()).await { + Ok(Ok(())) => {} + Ok(Err(panic)) => std::panic::resume_unwind(panic), + Err(e) => panic!("server join task failed: {e}"), + } +} From 1ea0d879271871054a6f9bd8c3ed8441e8f3f44e Mon Sep 17 00:00:00 2001 From: Claude Date: Fri, 24 Jul 2026 07:56:47 +0000 Subject: [PATCH 053/132] docs: outbound-stream deadline/ownership/disconnect now accurate (P1 shipped) Update the streaming docs and add a Dev Diary entry for the two P1 lifecycle fixes: reads are bounded by min(idle, remaining absolute deadline) started at stream open; a stream is released on handler/program exit on any path (outbound handles are handler-owned); and a downstream disconnect cancels a blocked upstream read promptly rather than waiting out the deadline. Co-Authored-By: Claude Opus 4.8 Claude-Session: https://claude.ai/code/session_011LxXAm5E4Vy39kRLr8kRnX --- ...2026-07-24-outbound-stream-lifecycle-p1.md | 73 +++++++++++++++++++ Docs/04-advanced-features/interoperability.md | 25 ++++--- Docs/development/response-streaming-design.md | 28 +++++-- 3 files changed, 110 insertions(+), 16 deletions(-) create mode 100644 Dev diary/2026-07-24-outbound-stream-lifecycle-p1.md diff --git a/Dev diary/2026-07-24-outbound-stream-lifecycle-p1.md b/Dev diary/2026-07-24-outbound-stream-lifecycle-p1.md new file mode 100644 index 00000000..a53dc0b7 --- /dev/null +++ b/Dev diary/2026-07-24-outbound-stream-lifecycle-p1.md @@ -0,0 +1,73 @@ +# Dev Diary — 2026-07-24: outbound-stream lifecycle P1s (deadline + ownership + disconnect) + +The two remaining P1 blockers from the maintainer's streaming re-review, each +closed with a **real-socket** Red→Green regression (mock upstream via +`tokio::TcpListener` + the WFL interpreter), which is the boundary evidence the +Testing Policy requires for R3 concurrency/streaming/lifecycle work. + +## P1-B — the absolute stream deadline must bound an ACTIVE read + +**Bug:** `stream_pull` computed the right per-read bound — +`min(idle_timeout, remaining_absolute)` — and handed it to `run_http_with_budget` +as `configured_timeout`. But `run_http_with_budget`/`outbound_http_deadline` +derived the operation timeout purely from the run/budget duration and *discarded* +`configured_timeout` (using it only as a fallback). So with `timeout_seconds = 10` +and `outbound_stream_max_seconds = 1`, a head-then-stall upstream let +`wait for next chunk` wait ~10s instead of ~1s. The absolute clock also started +only after the head arrived, excluding connect/header time. + +**Fix:** compose the operation deadline as `MIN(configured_timeout, budget)` and +report the stream `Timeout` vs the budget `Deadline` depending on which bound +fired; `configured_timeout` is always finite, so the read is bounded even with no +run-wide budget deadline. Start the absolute clock at request initiation and bound +the head phase by it too. + +**Test:** `outbound_stream_deadline_test` — mock sends head then stalls; read now +fails at ~1s (was ~10s, verified Red). + +## P1-A — outbound streams are handler-owned, and disconnect cancels a blocked read + +Two parts: + +**Part 1 — ownership / close-on-every-exit.** Outbound `httpstream*` handles lived +only in the interpreter-wide `IoClient.stream_handles`; `RunState`/ +`IsolatedHandler::drop` closed downstream response streams and pending requests +but not upstream handles, so an abandoned proxy read leaked the upstream until the +whole interpreter tore down. Now `RunState.open_http_streams` tracks them +per-handler (swapped per poll); added on open, untracked on EOF/error/explicit +close, and dropped from the map on every handler exit (`IsolatedHandler::drop` +for the concurrent loop; `close_open_http_streams()` at the serial-loop/program +cleanup sites) — dropping the reqwest stream cancels the upstream. +*Test:* `outbound_stream_ownership_test` — a run ends without `close` while the +interpreter is still alive; the mock sees its client disconnect only because +handler-exit cleanup cancelled the upstream (verified Red by disabling cleanup). + +**Part 2 — disconnect cancels a blocked upstream read.** A proxy handler blocked +in `wait for next line|chunk` on the upstream had no disconnect signal — it only +noticed at the next downstream `write` (or, after P1-B, the absolute deadline). +The downstream response stream's `mpsc::Sender::closed()` resolves when hyper +drops the client's body receiver, so we clone the handler's open-response-stream +senders (no `RefCell` borrow across the await) and `select!` the upstream read +against "any downstream disconnected". On disconnect the read future is dropped +(cancelling the upstream), the handle is closed, and a catchable `Cancelled` error +unwinds the handler (whose owned cleanup then runs). +*Test:* `outbound_stream_disconnect_test` — a WFL concurrent proxy relays a mock +upstream that stalls after one chunk; the client reads the first chunk and +disconnects while the handler is blocked; the mock observes its own connection +close within the window only because the blocked read was cancelled (verified Red +by disabling the `select!`). + +## Risk class & residual risk + +- **R3** (concurrency/cancellation/lifecycle/streaming). Real-boundary tests, + negative assertions (connection actually closes / read actually fails), and + Red evidence for each. +- Part 2 selects against the handler's currently-open response streams; a handler + with no downstream stream (a pure client-side reader) keeps the plain read path + (`pending` disconnect branch), so there is no behavior change there. +- `close_http_streams` in `Drop` is best-effort via `try_lock`; if the async lock + is momentarily held the handles are reclaimed at interpreter teardown (they no + longer leak *past* that, and in practice the lock is free at handler exit). +- Docs (`interoperability.md`, `response-streaming-design.md`) updated to state the + now-accurate read bound (min idle/absolute), handler-exit release, and the + proactive disconnect cancellation. diff --git a/Docs/04-advanced-features/interoperability.md b/Docs/04-advanced-features/interoperability.md index aab9c6c2..479f24a2 100644 --- a/Docs/04-advanced-features/interoperability.md +++ b/Docs/04-advanced-features/interoperability.md @@ -152,17 +152,24 @@ in-flight upstream request; reading from a closed (or fully-drained) handle raises a catchable error. The same limits as buffered requests apply: the running total of body bytes is -held under `web_server_max_response_size`, each read is bounded by the request's -timeout, and cooperative cancellation interrupts a read waiting on the peer. A -mid-stream network error surfaces as a catchable error from the `wait for next -...` statement. +held under `web_server_max_response_size`, and each read is bounded by the +**smaller** of the request's idle timeout and the stream's remaining absolute +lifetime (`outbound_stream_max_seconds`, measured from when the stream is opened, +including connect/header time). So even a trickling or stalled upstream can never +outlive the absolute cap. Cooperative cancellation interrupts a read waiting on +the peer, and a mid-stream network error surfaces as a catchable error from the +`wait for next ...` statement. A stream is released — cancelling the in-flight upstream request — when it -reaches a clean end of stream, hits an error, or you `close` it explicitly, and -in any case when the program exits. It is **not** released merely because the -handle variable goes out of scope, so `close upstream` when you stop early (or -break out of the read loop before EOF) to free the upstream connection promptly -rather than holding it until the program ends. +reaches a clean end of stream, hits an error, you `close` it explicitly, **or the +handler/program that owns it ends on any path** (normal return, a caught error, a +panic contained by `main loop concurrently:`, timeout, or cancellation). In a web +server that proxies an upstream, a **downstream client disconnect** also cancels a +read that is *currently blocked* on the upstream — the handler wakes with a +catchable error instead of waiting out the absolute deadline — and its upstream is +closed as the handler unwinds. Still prefer an explicit `close upstream` when you +stop early or break out of the read loop before EOF, to free the connection the +moment you are done rather than at handler exit. ### 4. **Web Standards** diff --git a/Docs/development/response-streaming-design.md b/Docs/development/response-streaming-design.md index ab53d645..107a5cc8 100644 --- a/Docs/development/response-streaming-design.md +++ b/Docs/development/response-streaming-design.md @@ -116,9 +116,14 @@ close out - **`write line|chunk`** resolves `_server_stream`, `tx.send(bytes).await` (bounded → backpressure). A closed receiver (browser disconnected / hyper - dropped the body) makes `send` fail → surfaced as a catchable error, letting - the handler `close upstream` and stop — this is how browser-disconnect - cancellation propagates to the upstream (item 5, cooperatively). + dropped the body) makes `send` fail → surfaced as a catchable error. In + addition, a handler blocked in an upstream `wait for next line|chunk` no longer + has to wait for its next `write` to notice the disconnect: the read is + `select!`ed against `Sender::closed()` for the handler's open response streams, + so a downstream disconnect cancels the blocked upstream read promptly (dropping + the upstream), and the handler's owned outbound handles are then closed as it + unwinds. This is how browser-disconnect cancellation propagates to the upstream + (item 5, cooperatively). - **`flush`** = advisory; `tokio::task::yield_now().await` so the transport task is scheduled. Documented as advisory (hyper already writes as it receives). @@ -128,11 +133,20 @@ close out ### Lifecycle (item 5, server side) -- Timeouts: the existing per-request `overall_deadline` bounds head delivery; - document that a stalled *body producer* is bounded by the handler timeout at - await points (the yield-cliff caveat from the concurrency plan). +- Timeouts: the existing per-request `overall_deadline` bounds head delivery; a + stalled *body producer* is bounded by the handler timeout at await points (the + yield-cliff caveat from the concurrency plan). Outbound reads are additionally + bounded by `min(idle, remaining absolute stream deadline)`: `run_http_with_budget` + composes the operation deadline as the minimum of the run/budget remaining time + and the caller's configured timeout (which already carries the stream's idle + + absolute bound), and the absolute clock starts at request initiation. - Backpressure: bounded `mpsc` — a slow browser slows the handler's `write`. -- Disconnect → upstream cancel: `write` error path (above). +- Disconnect → upstream cancel: the `write` error path AND a proactive + `Sender::closed()` `select!` against a blocked upstream read (above). +- Outbound close-on-exit (shipped): outbound `httpstream*` handles are also + handler-owned — tracked in `RunState.open_http_streams` (swapped per poll) and + dropped from `IoClient.stream_handles` when the handler ends on any path, + cancelling the in-flight upstream request so an abandoned proxy read never leaks. - Close-on-exit (shipped): each handler tracks the `respstream*` ids it opened in its per-handler run-state (`open_response_streams`, part of the `RunState` swapped in/out per poll under `main loop concurrently:`). When the handler ends From fb737d96f1d1ccf80db4aa07b4645da3be56e930 Mon Sep 17 00:00:00 2001 From: Claude Date: Fri, 24 Jul 2026 08:50:09 +0000 Subject: [PATCH 054/132] test(P1): failing burst-disconnect regression for the concurrent loop A burst of >256 downstream disconnects currently trips the concurrent main loop's global consecutive-failure breaker (each disconnect is miscounted as a handler failure), tearing the loop down so an unrelated /ping is refused. Red: /ping is refused after 256 disconnects because the loop broke. This is the R3 negative/availability test that the fix must turn green. Co-Authored-By: Claude Opus 4.8 Claude-Session: https://claude.ai/code/session_011LxXAm5E4Vy39kRLr8kRnX --- tests/concurrent_disconnect_burst_test.rs | 245 ++++++++++++++++++++++ 1 file changed, 245 insertions(+) create mode 100644 tests/concurrent_disconnect_burst_test.rs diff --git a/tests/concurrent_disconnect_burst_test.rs b/tests/concurrent_disconnect_burst_test.rs new file mode 100644 index 00000000..6848731d --- /dev/null +++ b/tests/concurrent_disconnect_burst_test.rs @@ -0,0 +1,245 @@ +//! Real-socket regression for P1 (#2): a BURST of downstream (browser) +//! disconnects must NOT tear down the concurrent `main loop`. +//! +//! A client disconnect is an EXPECTED, normal cancellation of one handler — not a +//! handler *failure*. The concurrent loop keeps a single global consecutive- +//! failure counter that backs off after every failed handler and breaks the whole +//! loop once it reaches `MAX_CONSECUTIVE_FAILURES` (256). If each disconnect is +//! (mis)counted as a failure, then 256 disconnects with no interleaved success +//! trip that structural breaker and the server stops serving entirely — so an +//! ordinary "client hung up" event, repeated, becomes a denial of service. +//! +//! Topology: a stalling mock upstream <- WFL concurrent proxy -> many short-lived +//! clients. Each client makes the proxy open the (stalling) upstream, start a +//! streaming response, and block reading the upstream; the client then reads the +//! response head and disconnects, cancelling that handler. After a burst of >256 +//! such disconnects (more than the breaker threshold, with NO successful request +//! in between), an unrelated `/ping` request must still be served — proving the +//! loop survived the burst. + +use std::sync::Arc; +use std::sync::atomic::{AtomicUsize, Ordering}; +use std::time::Duration; +use tokio::io::{AsyncReadExt, AsyncWriteExt}; +use tokio::sync::Semaphore; +use tokio::sync::mpsc; +use wfl::Interpreter; +use wfl::lexer::lex_wfl_with_positions; +use wfl::parser::Parser; + +/// How many disconnecting clients to fire. Must exceed the concurrent loop's +/// `MAX_CONSECUTIVE_FAILURES` (256) so, under the buggy behavior, the burst trips +/// the structural breaker. +const DISCONNECT_BURST: usize = 270; +/// Wait for at least this many upstream closes (== handler disconnects) before +/// probing, guaranteeing the burst has driven the breaker past its threshold. +const CLOSES_BEFORE_PROBE: usize = 256; +/// Bounded client concurrency: well under the 256 handler cap and the request +/// queue bound, so no request is shed with 503. +const CLIENT_CONCURRENCY: usize = 48; + +/// Mock upstream: for each connection, send a chunked head then STALL (send no +/// body). Signal on `closes` every time a connection is observed closing — which +/// only happens when the proxy handler cancels its upstream (client disconnect). +async fn spawn_counting_stall_upstream() -> (u16, mpsc::UnboundedReceiver<()>) { + let listener = tokio::net::TcpListener::bind("127.0.0.1:0") + .await + .expect("bind mock upstream"); + let port = listener.local_addr().unwrap().port(); + let (closes_tx, closes_rx) = mpsc::unbounded_channel(); + tokio::spawn(async move { + loop { + let Ok((mut sock, _)) = listener.accept().await else { + return; + }; + let closes_tx = closes_tx.clone(); + tokio::spawn(async move { + let mut buf = [0u8; 1024]; + let _ = sock.read(&mut buf).await; // request head + let head = "HTTP/1.1 200 OK\r\n\ + Content-Type: text/plain\r\n\ + Transfer-Encoding: chunked\r\n\r\n"; + let _ = sock.write_all(head.as_bytes()).await; + let _ = sock.flush().await; + // Stall: never send a body chunk. Block reading; when the proxy + // cancels the upstream (its client disconnected), the peer close + // surfaces here as Ok(0)/Err. + loop { + match sock.read(&mut buf).await { + Ok(0) | Err(_) => { + let _ = closes_tx.send(()); + return; + } + Ok(_) => {} + } + } + }); + } + }); + (port, closes_rx) +} + +fn start_proxy_server(code: String) -> std::thread::JoinHandle<()> { + std::thread::spawn(move || { + let rt = tokio::runtime::Runtime::new().expect("runtime"); + rt.block_on(async { + let tokens = lex_wfl_with_positions(&code); + let ast = Parser::new(&tokens).parse().expect("parse"); + let mut interp = Interpreter::new(); + if let Err(errors) = interp.interpret(&ast).await { + panic!("proxy interpreter failed: {errors:?}"); + } + }); + }) +} + +async fn wait_for_server(port: u16) { + let addr = format!("127.0.0.1:{port}"); + for _ in 0..300 { + if tokio::net::TcpStream::connect(&addr).await.is_ok() { + return; + } + tokio::time::sleep(Duration::from_millis(50)).await; + } + panic!("proxy server on {addr} did not become ready"); +} + +/// One disconnecting client: open `/proxy`, read the response head (proving the +/// handler reached `start streaming response` and is now blocked on the upstream), +/// then drop the socket to disconnect. +async fn fire_disconnect(proxy_port: u16) { + let Ok(mut sock) = tokio::net::TcpStream::connect(("127.0.0.1", proxy_port)).await else { + return; + }; + let req = "GET /proxy HTTP/1.1\r\nHost: 127.0.0.1\r\n\r\n"; + if sock.write_all(req.as_bytes()).await.is_err() { + return; + } + // Read until the end of the response head (\r\n\r\n) so the handler has an + // open response stream when we disconnect (that is what makes the disconnect + // observable to the blocked upstream read). + let mut acc = Vec::new(); + let mut tmp = [0u8; 256]; + loop { + match tokio::time::timeout(Duration::from_secs(5), sock.read(&mut tmp)).await { + Ok(Ok(0)) | Err(_) => break, + Ok(Ok(n)) => { + acc.extend_from_slice(&tmp[..n]); + if acc.windows(4).any(|w| w == b"\r\n\r\n") { + break; + } + } + Ok(Err(_)) => break, + } + } + // Drop `sock` -> disconnect while the handler is blocked reading the upstream. +} + +#[tokio::test] +async fn test_disconnect_burst_does_not_kill_concurrent_loop() { + let (upstream_port, mut upstream_closes) = spawn_counting_stall_upstream().await; + + let proxy_port = 8362; + let code = format!( + r#" + listen on port {proxy_port} as srv + main loop concurrently: + wait for request comes in on srv as req with timeout 30000 + store p as req["path"] + check if p is equal to "/ping": + respond to req with "pong" + otherwise: + check if p is equal to "/shutdown": + respond to req with "bye" + close server srv + break + otherwise: + open url at "http://127.0.0.1:{upstream_port}/" and stream response as up + start streaming response to req with status 200 and content type "text/plain" as down + count from 1 to 100000: + wait for next chunk from up as c + check if c is nothing: + break + otherwise: + write chunk c to down + end check + end count + end check + end check + end loop + "# + ); + let server = start_proxy_server(code); + wait_for_server(proxy_port).await; + + // Fire a burst of disconnecting clients, bounded so none is shed with 503. + let sem = Arc::new(Semaphore::new(CLIENT_CONCURRENCY)); + let fired = Arc::new(AtomicUsize::new(0)); + let mut tasks = Vec::with_capacity(DISCONNECT_BURST); + for _ in 0..DISCONNECT_BURST { + let sem = Arc::clone(&sem); + let fired = Arc::clone(&fired); + tasks.push(tokio::spawn(async move { + let _permit = sem.acquire().await.expect("semaphore"); + fire_disconnect(proxy_port).await; + fired.fetch_add(1, Ordering::Relaxed); + })); + } + + // Wait until the mock has seen enough upstream closes to guarantee the burst + // drove the buggy breaker past its 256-failure threshold (no `/ping` sent yet, + // so every one of these is a "consecutive failure" under the old behavior). + let mut closed = 0usize; + let deadline = tokio::time::Instant::now() + Duration::from_secs(60); + while closed < CLOSES_BEFORE_PROBE { + match tokio::time::timeout_at(deadline, upstream_closes.recv()).await { + Ok(Some(())) => closed += 1, + Ok(None) => break, + Err(_) => panic!( + "only observed {closed} upstream closes before timeout (expected {CLOSES_BEFORE_PROBE}); \ + the disconnect burst did not fully drive the handlers" + ), + } + } + assert!( + closed >= CLOSES_BEFORE_PROBE, + "expected at least {CLOSES_BEFORE_PROBE} upstream closes, saw {closed}" + ); + + // Grace so the loop finishes counting the final failure (and, under the bug, + // actually breaks) before we probe. + tokio::time::sleep(Duration::from_millis(750)).await; + + // The unrelated `/ping` MUST still be served. Under the bug the loop has torn + // itself down after 256 "failures" and this hangs / is refused. + let ping = tokio::time::timeout( + Duration::from_secs(10), + reqwest::Client::new() + .get(format!("http://127.0.0.1:{proxy_port}/ping")) + .send(), + ) + .await + .expect("`/ping` timed out after the disconnect burst — concurrent loop was torn down") + .expect("`/ping` request failed after the disconnect burst"); + assert_eq!( + ping.status().as_u16(), + 200, + "`/ping` should be served after the disconnect burst" + ); + let body = ping.text().await.expect("read /ping body"); + assert_eq!(body, "pong", "`/ping` should return the live handler's response"); + + // Shut the server down and join everything. + let _ = reqwest::Client::new() + .get(format!("http://127.0.0.1:{proxy_port}/shutdown")) + .send() + .await; + for t in tasks { + let _ = t.await; + } + match tokio::task::spawn_blocking(move || server.join()).await { + Ok(Ok(())) => {} + Ok(Err(panic)) => std::panic::resume_unwind(panic), + Err(e) => panic!("server join task failed: {e}"), + } +} From 3c1fb80d4c1512ca912f80ad2a85a188568c92f0 Mon Sep 17 00:00:00 2001 From: Claude Date: Fri, 24 Jul 2026 08:59:03 +0000 Subject: [PATCH 055/132] fix(P1): treat a client disconnect as cancellation, not a handler failure A downstream disconnect that cancels a proxy handler's blocked upstream read was surfaced as a generic budget error and fed into the concurrent main loop's single global consecutive-failure breaker: every disconnect incremented the counter and backed off, and 256 disconnects with no interleaved success broke the whole loop, turning an ordinary client hang-up into a denial of service. The disconnect branch now returns a distinct HttpClientError::Disconnected mapped to a new ErrorKind::Cancelled (catchable like any error). The concurrent loop recognizes a Cancelled handler outcome as a normal, expected cancellation: it releases the handler (its owned upstream/response streams are already closed on unwind) without touching the failure counter or backing off. Internal budget-cancellation of an outbound request keeps its ResourceLimit kind, so only a real downstream disconnect is exempted. Turns tests/concurrent_disconnect_burst_test.rs green: a burst of 270 disconnects no longer tears the loop down; an unrelated /ping is still served (was refused; ~13s -> ~0.9s). Co-Authored-By: Claude Opus 4.8 Claude-Session: https://claude.ai/code/session_011LxXAm5E4Vy39kRLr8kRnX --- src/interpreter/error.rs | 7 +++++++ src/interpreter/mod.rs | 30 +++++++++++++++++++++++++++--- 2 files changed, 34 insertions(+), 3 deletions(-) diff --git a/src/interpreter/error.rs b/src/interpreter/error.rs index 0803a32c..5956ebbd 100644 --- a/src/interpreter/error.rs +++ b/src/interpreter/error.rs @@ -8,6 +8,12 @@ pub enum ErrorKind { /// A shared `ExecutionBudget` ceiling other than the deadline was reached /// (operation count, recursion/import/execute-file depth, byte caps, etc.). ResourceLimit, + /// A cooperative cancellation of an in-flight operation triggered by an + /// expected external event rather than a fault — currently a downstream + /// (browser) disconnect cancelling a proxy handler's blocked upstream read. + /// Catchable like any other error, but the concurrent `main loop` treats it + /// as a normal handler outcome, not a structural failure. + Cancelled, FileNotFound, PermissionDenied, ProcessNotFound, @@ -51,6 +57,7 @@ impl fmt::Display for RuntimeError { ErrorKind::EnvDropped => "[Environment dropped] ", ErrorKind::Timeout => "[Timeout] ", ErrorKind::ResourceLimit => "[Resource limit] ", + ErrorKind::Cancelled => "[Cancelled] ", ErrorKind::FileNotFound => "[File not found] ", ErrorKind::PermissionDenied => "[Permission denied] ", ErrorKind::ProcessNotFound => "[Process not found] ", diff --git a/src/interpreter/mod.rs b/src/interpreter/mod.rs index 0b3a739f..471c8d73 100644 --- a/src/interpreter/mod.rs +++ b/src/interpreter/mod.rs @@ -1619,6 +1619,12 @@ enum HttpClientError { Request(String), Budget(BudgetExceeded), Timeout { seconds: u64 }, + /// The downstream (browser) client disconnected while a proxy handler was + /// blocked on this upstream read, so the read was cancelled cooperatively. + /// A normal, expected event — distinct from a fault — surfaced with + /// `ErrorKind::Cancelled` so the concurrent loop does not count it as a + /// handler failure. + Disconnected, } impl From for HttpClientError { @@ -3848,6 +3854,12 @@ impl Interpreter { column, ErrorKind::Timeout, ), + HttpClientError::Disconnected => RuntimeError::with_kind( + "Client disconnected; upstream read cancelled".to_string(), + line, + column, + ErrorKind::Cancelled, + ), } } @@ -4146,6 +4158,17 @@ impl Interpreter { ControlFlow::Continue | ControlFlow::None => {} } } + // A client disconnect cancelled the handler cooperatively. That is + // an EXPECTED external event, not a fault — releasing this handler + // must not feed the structural consecutive-failure breaker (else a + // burst of disconnects would back off and eventually tear the loop + // down, turning "the browser hung up" into a denial of service). + // The handler's owned streams are already closed on unwind; leave + // the failure counter untouched (a disconnect is neither progress + // nor failure) and keep serving. + Some(Ok(Err(err))) if err.kind == ErrorKind::Cancelled => { + log::debug!("concurrent main loop: handler cancelled (client disconnected)"); + } // A handler returned a runtime error: its request (if it took one) // is answered 500 by the ResponseCompletion drop guard. Log and // keep the server running instead of tearing it down. @@ -7206,9 +7229,10 @@ impl Interpreter { _ = &mut disconnect => { // Client gone: dropping `read` above already cancels // the upstream; close the handle too in case the read - // had not yet taken it, and report cancellation. + // had not yet taken it, and report the disconnect as a + // cooperative cancellation (not a handler failure). self.io_client.close_stream(&handle_id).await; - Err(HttpClientError::Budget(BudgetExceeded::Cancelled)) + Err(HttpClientError::Disconnected) } } }; @@ -7260,7 +7284,7 @@ impl Interpreter { r = &mut read => r, _ = &mut disconnect => { self.io_client.close_stream(&handle_id).await; - Err(HttpClientError::Budget(BudgetExceeded::Cancelled)) + Err(HttpClientError::Disconnected) } } }; From 23342727233b73bf3c9c4f0aeec37ba318ea2a29 Mon Sep 17 00:00:00 2001 From: Claude Date: Fri, 24 Jul 2026 09:00:07 +0000 Subject: [PATCH 056/132] style: cargo fmt (expand HttpClientError::Timeout, wrap assert) Co-Authored-By: Claude Opus 4.8 Claude-Session: https://claude.ai/code/session_011LxXAm5E4Vy39kRLr8kRnX --- src/interpreter/mod.rs | 4 +++- tests/concurrent_disconnect_burst_test.rs | 5 ++++- 2 files changed, 7 insertions(+), 2 deletions(-) diff --git a/src/interpreter/mod.rs b/src/interpreter/mod.rs index 471c8d73..1dbdb816 100644 --- a/src/interpreter/mod.rs +++ b/src/interpreter/mod.rs @@ -1618,7 +1618,9 @@ struct HttpStreamHandle { enum HttpClientError { Request(String), Budget(BudgetExceeded), - Timeout { seconds: u64 }, + Timeout { + seconds: u64, + }, /// The downstream (browser) client disconnected while a proxy handler was /// blocked on this upstream read, so the read was cancelled cooperatively. /// A normal, expected event — distinct from a fault — surfaced with diff --git a/tests/concurrent_disconnect_burst_test.rs b/tests/concurrent_disconnect_burst_test.rs index 6848731d..bad54b62 100644 --- a/tests/concurrent_disconnect_burst_test.rs +++ b/tests/concurrent_disconnect_burst_test.rs @@ -227,7 +227,10 @@ async fn test_disconnect_burst_does_not_kill_concurrent_loop() { "`/ping` should be served after the disconnect burst" ); let body = ping.text().await.expect("read /ping body"); - assert_eq!(body, "pong", "`/ping` should return the live handler's response"); + assert_eq!( + body, "pong", + "`/ping` should return the live handler's response" + ); // Shut the server down and join everything. let _ = reqwest::Client::new() From 9a3ffb3ee185ab21b5884970eb6656aa14eaa249 Mon Sep 17 00:00:00 2001 From: Claude Date: Fri, 24 Jul 2026 09:10:44 +0000 Subject: [PATCH 057/132] test(P1): failing coverage for property-then-index parsing A bracket index right after a .property/.method access split into a separate list-literal statement, so `store ct as obj.headers["k"]` silently dropped the key and bound `ct` to the whole headers map. Red evidence: - property_index_access_test: AST asserts one IndexAccess over the PropertyAccess (currently splits); a runtime map lookup yields the indexed value "BBB" (currently returns the whole map -> INDEX_WRONG). - stream_handle_type_test: the previously typecheck-only dot test now also asserts the parse structure (it was a false green: both halves of the split still type-check). Co-Authored-By: Claude Opus 4.8 Claude-Session: https://claude.ai/code/session_011LxXAm5E4Vy39kRLr8kRnX --- tests/property_index_access_test.rs | 176 ++++++++++++++++++++++++++++ tests/stream_handle_type_test.rs | 22 ++++ 2 files changed, 198 insertions(+) create mode 100644 tests/property_index_access_test.rs diff --git a/tests/property_index_access_test.rs b/tests/property_index_access_test.rs new file mode 100644 index 00000000..34a65488 --- /dev/null +++ b/tests/property_index_access_test.rs @@ -0,0 +1,176 @@ +//! Regression (P1 #5): a bracket index immediately after a `.property` (or +//! `.method(...)`) access must bind to that property value, not split off into a +//! separate bogus list-literal statement. +//! +//! Before the fix, `store ct as obj.headers["content-type"]` parsed as TWO +//! statements — `store ct as obj.headers` (a `PropertyAccess`) followed by a +//! standalone `["content-type"]` list literal — silently dropping the lookup so +//! `ct` was bound to the whole `headers` map. This proves both the AST shape +//! (one `IndexAccess` over the `PropertyAccess`) and the runtime value. + +use std::fs; +use std::process::Command; +use tempfile::TempDir; +use wfl::lexer::lex_wfl_with_positions; +use wfl::parser::Parser; +use wfl::parser::ast::{Expression, Literal, Statement}; + +fn parse(src: &str) -> wfl::parser::ast::Program { + let tokens = lex_wfl_with_positions(src); + Parser::new(&tokens).parse().expect("parse should succeed") +} + +fn wfl_exe() -> &'static str { + env!("CARGO_BIN_EXE_wfl") +} + +/// Run inline WFL source, returning (stdout+stderr, exit code). +fn run_src(src: &str) -> (String, Option) { + let dir = TempDir::new().expect("tempdir"); + let path = dir.path().join("main.wfl"); + fs::write(&path, src).unwrap(); + let output = Command::new(wfl_exe()) + .arg(&path) + .output() + .expect("failed to execute WFL"); + let combined = format!( + "{}{}", + String::from_utf8_lossy(&output.stdout), + String::from_utf8_lossy(&output.stderr) + ); + drop(dir); + (combined, output.status.code()) +} + +#[test] +fn property_then_index_is_one_index_access_over_property_access() { + // A single statement: `store ct as obj.headers["content-type"]`. + let program = parse("store ct as obj.headers[\"content-type\"]\n"); + assert_eq!( + program.statements.len(), + 1, + "the property-then-index expression must not split into extra statements; got {:#?}", + program.statements + ); + + let value = match &program.statements[0] { + Statement::VariableDeclaration { name, value, .. } => { + assert_eq!(name, "ct"); + value + } + other => panic!("expected a VariableDeclaration, got {other:#?}"), + }; + + // Outer node is IndexAccess["content-type"] whose collection is the + // PropertyAccess obj.headers. + match value { + Expression::IndexAccess { + collection, index, .. + } => { + match index.as_ref() { + Expression::Literal(Literal::String(key), ..) => { + assert_eq!( + key.as_ref(), + "content-type", + "index key must be the bracket string" + ) + } + other => panic!("expected a string index, got {other:#?}"), + } + match collection.as_ref() { + Expression::PropertyAccess { + object, property, .. + } => { + assert_eq!(property, "headers"); + assert!( + matches!(object.as_ref(), Expression::Variable(n, ..) if n == "obj"), + "property-access object must be the variable `obj`, got {object:#?}" + ); + } + other => panic!( + "index collection must be the PropertyAccess obj.headers, got {other:#?}" + ), + } + } + other => panic!("expected IndexAccess over PropertyAccess, got {other:#?}"), + } +} + +#[test] +fn chained_property_index_nests_left_to_right() { + // `grid.rows[0][1]` -> IndexAccess( IndexAccess( PropertyAccess(grid.rows), 0 ), 1 ) + let program = parse("store cell as grid.rows[0][1]\n"); + assert_eq!(program.statements.len(), 1, "must be one statement"); + let value = match &program.statements[0] { + Statement::VariableDeclaration { value, .. } => value, + other => panic!("expected VariableDeclaration, got {other:#?}"), + }; + // Outermost index is [1]. + let inner = match value { + Expression::IndexAccess { collection, .. } => collection, + other => panic!("expected outer IndexAccess, got {other:#?}"), + }; + // Next index is [0] over the property access. + match inner.as_ref() { + Expression::IndexAccess { collection, .. } => { + assert!( + matches!(collection.as_ref(), Expression::PropertyAccess { property, .. } if property == "rows"), + "innermost collection must be grid.rows, got {collection:#?}" + ); + } + other => panic!("expected inner IndexAccess, got {other:#?}"), + } +} + +#[test] +fn property_index_runtime_value_is_the_indexed_field_not_the_whole_map() { + // Unambiguous: the correct index result ("BBB") differs from the property + // map, so a split (ct = the headers map) fails the equality check. + let src = "create map inner:\n\ + \x20 \"a\" is \"AAA\"\n\ + \x20 \"b\" is \"BBB\"\n\ + end map\n\ + create map outer:\n\ + \x20 \"headers\" is inner\n\ + end map\n\ + store ct as outer.headers[\"b\"]\n\ + check if ct is equal to \"BBB\":\n\ + \x20 display \"INDEX_OK\"\n\ + otherwise:\n\ + \x20 display \"INDEX_WRONG\"\n\ + end check\n"; + let (out, code) = run_src(src); + assert_eq!(code, Some(0), "program should exit 0: {out}"); + assert!( + out.contains("INDEX_OK"), + "`outer.headers[\"b\"]` must yield the indexed value \"BBB\", not the whole map: {out}" + ); + assert!( + !out.contains("INDEX_WRONG"), + "the property-then-index lookup returned the wrong value: {out}" + ); +} + +#[test] +fn outbound_stream_header_index_parses_as_single_statement() { + // The canonical proxy pattern from the docs: + // `store ct as upstream.headers["content-type"]`. It must be one statement + // (an IndexAccess over the PropertyAccess), not a split that drops the key. + let program = parse( + "open url at \"http://example.com\" and stream response as upstream\n\ + store ct as upstream.headers[\"content-type\"]\n", + ); + assert_eq!( + program.statements.len(), + 2, + "expected exactly the open + store statements, no split list literal; got {:#?}", + program.statements + ); + match &program.statements[1] { + Statement::VariableDeclaration { value, .. } => assert!( + matches!(value, Expression::IndexAccess { .. }), + "the header lookup must be an IndexAccess, got {value:#?}" + ), + other => panic!("expected the store statement, got {other:#?}"), + } +} diff --git a/tests/stream_handle_type_test.rs b/tests/stream_handle_type_test.rs index 08bb4e90..7bf02421 100644 --- a/tests/stream_handle_type_test.rs +++ b/tests/stream_handle_type_test.rs @@ -8,6 +8,7 @@ use wfl::lexer::lex_wfl_with_positions; use wfl::parser::Parser; +use wfl::parser::ast::{Expression, Statement}; use wfl::typechecker::TypeChecker; fn typecheck(code: &str) -> Result<(), String> { @@ -88,6 +89,27 @@ fn test_outbound_stream_handle_dot_access_typechecks() { "dot access on a stream handle must type-check: {:?}", typecheck(code).err() ); + + // Type-checking alone is a false green here: even when + // `upstream.headers["content-type"]` mis-parses into two statements + // (`store ct as upstream.headers` + a stray `["content-type"]` list literal) + // both halves type-check. Assert the *structure*: the header lookup binds as + // one IndexAccess over the PropertyAccess, so the key is not silently dropped. + let tokens = lex_wfl_with_positions(code); + let program = Parser::new(&tokens).parse().expect("parse"); + let ct_stmt = program + .statements + .iter() + .find_map(|s| match s { + Statement::VariableDeclaration { name, value, .. } if name == "ct" => Some(value), + _ => None, + }) + .expect("a `store ct as ...` statement"); + assert!( + matches!(ct_stmt, Expression::IndexAccess { .. }), + "`upstream.headers[\"content-type\"]` must bind `ct` to an IndexAccess over the \ + PropertyAccess (the lookup must not split off), got {ct_stmt:#?}" + ); } #[test] From d938427fba71f5359b1d3478d12efcf62cdc356c Mon Sep 17 00:00:00 2001 From: Claude Date: Fri, 24 Jul 2026 09:25:02 +0000 Subject: [PATCH 058/132] fix(P1): compose a bracket index after a .property/.method access A bracket index immediately after an identifier property access (`obj.headers["content-type"]`) or method call (`obj.get()[0]`) was dropped: the primary-expression dispatch returned the PropertyAccess/ MethodCall early, before the postfix loop could consume the `[...]`, so the bracket was re-parsed as a standalone list-literal statement and the lookup silently vanished (`ct` bound to the whole map). Route both the property-access and method-call fast paths through a new parse_trailing_bracket_index helper that folds any chained `[...]` index accesses onto the base (`grid.rows[0][1]`). The shared postfix loop and the static-member `.` arm are untouched, so a trailing `.member` keeps its pre-existing behavior. Turns tests/property_index_access_test.rs and the strengthened dot test in tests/stream_handle_type_test.rs green. Co-Authored-By: Claude Opus 4.8 Claude-Session: https://claude.ai/code/session_011LxXAm5E4Vy39kRLr8kRnX --- src/parser/expr/primary.rs | 69 +++++++++++++++++++++++++++++++++++--- 1 file changed, 64 insertions(+), 5 deletions(-) diff --git a/src/parser/expr/primary.rs b/src/parser/expr/primary.rs index 697c577d..72ba16b3 100644 --- a/src/parser/expr/primary.rs +++ b/src/parser/expr/primary.rs @@ -89,6 +89,57 @@ impl<'a> PrimaryExprParser<'a> for Parser<'a> { } impl<'a> Parser<'a> { + /// After an identifier `.property` or `.method(...)` access, consume any + /// chained bracket index accesses so the index binds to the property/method + /// value. Without this, `upstream.headers["content-type"]` parsed as + /// `upstream.headers` followed by a separate `["content-type"]` list-literal + /// statement — silently dropping the lookup. Handles chains + /// (`grid.rows[0][1]`); a trailing `.member` is left for the caller (matching + /// the pre-existing primary-expression behavior). + fn parse_trailing_bracket_index( + &mut self, + mut expr: Expression, + ) -> Result { + while let Some(bracket) = self.cursor.peek() { + if bracket.token != Token::LeftBracket { + break; + } + let line = bracket.line; + let column = bracket.column; + self.bump_sync(); // Consume '[' + + let index = self.parse_expression()?; + + match self.cursor.peek() { + Some(closing) if closing.token == Token::RightBracket => { + self.bump_sync(); // Consume ']' + } + Some(closing) => { + return Err(ParseError::from_token( + format!("Expected ']' after index, found {:?}", closing.token), + closing, + )); + } + None => { + return Err(ParseError::from_span( + "Expected ']' after index, found end of input".to_string(), + crate::diagnostics::Span { start: 0, end: 0 }, + line, + column, + )); + } + } + + expr = Expression::IndexAccess { + collection: Box::new(expr), + index: Box::new(index), + line, + column, + }; + } + Ok(expr) + } + /// The actual primary-expression dispatch. Call `parse_primary_expression` /// (the trait method above), not this directly — it wraps this function /// with a debug-only check that keeps `can_start_primary_expression` from @@ -277,7 +328,7 @@ impl<'a> Parser<'a> { "Expected ')' after method arguments", )?; - return Ok(Expression::MethodCall { + let call = Expression::MethodCall { object: Box::new(Expression::Variable( name.clone(), token_line, @@ -287,11 +338,18 @@ impl<'a> Parser<'a> { arguments, line: token_line, column: token_column, - }); + }; + return self.parse_trailing_bracket_index(call); } - // Property access without method call - return Ok(Expression::PropertyAccess { + // Property access without method call. + // Route through the trailing-index helper so a + // following `["key"]`/`[i]` binds to the + // property value (e.g. + // `upstream.headers["content-type"]`) instead + // of splitting off into a bogus list-literal + // statement. + let access = Expression::PropertyAccess { object: Box::new(Expression::Variable( name.clone(), token_line, @@ -300,7 +358,8 @@ impl<'a> Parser<'a> { property: property_name.clone(), line: token_line, column: token_column, - }); + }; + return self.parse_trailing_bracket_index(access); } else { return Err(ParseError::from_token( "Expected property name after '.'".to_string(), From c931fba61d4e62c18830a76f5fb01f533c44a2d7 Mon Sep 17 00:00:00 2001 From: Claude Date: Fri, 24 Jul 2026 09:37:29 +0000 Subject: [PATCH 059/132] test(P1): failing coverage for span-mismatched write fallback `write line min with a: 1 and b: 2 to `: the stream reading consumes the builtin call `min` with named args, but the classic file-write reading of `line min` can only parse `line min with a` (a shorter span, stopping at the `:`). Retaining that partial parse as the fallback corrupts a file write. Red evidence: - AST: the span-mismatched fallback must be dropped (currently retained as a partial Concatenation). - runtime: with `line min`/`a` defined, the old fallback writes "CORRUPT..." to the file; the fix must make it a clean error instead. Co-Authored-By: Claude Opus 4.8 Claude-Session: https://claude.ai/code/session_011LxXAm5E4Vy39kRLr8kRnX --- tests/write_line_backcompat_test.rs | 69 +++++++++++++++++++++++++++++ 1 file changed, 69 insertions(+) diff --git a/tests/write_line_backcompat_test.rs b/tests/write_line_backcompat_test.rs index 357d007b..16058a97 100644 --- a/tests/write_line_backcompat_test.rs +++ b/tests/write_line_backcompat_test.rs @@ -307,6 +307,75 @@ fn test_ambiguous_write_line_accepts_desugared_classic_writes() { } } +#[test] +fn test_ambiguous_write_line_drops_span_mismatched_fallback() { + // Regression (maintainer review): the classic file-write fallback is only a + // valid alternate reading when it consumes the SAME continuation span as the + // stream reading. `write line min with a: 1 and b: 2 to `: the stream + // reading is the builtin call `min` with named args `a`/`b` (consuming through + // `b: 2`), but the classic reading of the multiword variable `line min` can + // only parse `line min with a` before the `:` — a shorter, partial span. + // Keeping that partial parse as the fallback corrupts a file write, so it must + // be dropped (fallback = None) rather than retained just because it parsed. + let stmt = &parse("write line min with a: 1 and b: 2 to f")[0]; + match stmt { + Statement::StreamWriteStatement { + value, + fallback_content, + .. + } => { + assert!( + matches!(value, Expression::ActionCall { .. }), + "the stream reading should consume the whole named-argument call, got {value:?}" + ); + assert!( + fallback_content.is_none(), + "a partial (span-mismatched) classic fallback must be dropped, got {fallback_content:?}" + ); + } + other => panic!("expected StreamWriteStatement, got {other:?}"), + } +} + +#[test] +fn test_span_mismatched_write_line_to_file_does_not_corrupt() { + // The runtime counterpart: with a span-mismatched fallback dropped, writing + // the ambiguous `min(...)` form to a FILE target is a clean error instead of + // silently writing the corrupt partial concatenation. `line min` and `a` are + // defined so the OLD (buggy) fallback would have evaluated and written + // "CORRUPT..." to the file; the fix must prevent that. + let dir = tempfile::tempdir().expect("create temp dir"); + let path = dir.path().join("wfl_write_line_span_mismatch.txt"); + let path_str = path.to_string_lossy().replace('\\', "/"); + + let code = format!( + "store line min as \"CORRUPT\"\n\ + store a as \"SUFFIX\"\n\ + write line min with a: 1 and b: 2 to \"{path_str}\"" + ); + let tokens = lex_wfl_with_positions(&code); + let program = Parser::new(&tokens).parse().expect("parse"); + + let rt = tokio::runtime::Runtime::new().unwrap(); + let result = rt.block_on(async { + let mut interp = Interpreter::new(); + interp.interpret(&program).await + }); + + let wrote_corrupt = std::fs::read_to_string(&path) + .map(|c| c.contains("CORRUPT")) + .unwrap_or(false); + assert!( + !wrote_corrupt, + "the span-mismatched fallback corrupted the file write: {:?}", + std::fs::read_to_string(&path) + ); + assert!( + result.is_err(), + "writing the ambiguous `min(...)` form to a file must be a clean error, not a silent corrupt write" + ); +} + #[test] fn test_ambiguous_write_line_still_flags_when_neither_candidate_defined() { // The ambiguous form defers definedness to runtime, but a genuine typo where From bcf88c1fbe3f07678a1dbb376b6a16cab8e30191 Mon Sep 17 00:00:00 2001 From: Claude Date: Fri, 24 Jul 2026 09:38:23 +0000 Subject: [PATCH 060/132] test(P1): failing coverage for absolute stream lifetime on buffered reads outbound_stream_max_seconds must be a true absolute lifetime, but next_line/next_chunk serve locally-buffered bytes before consulting the deadline. Red: a buffered `wait for next line` taken ~1.5s after opening (past the 1s absolute lifetime) is still served instead of failing. Co-Authored-By: Claude Opus 4.8 Claude-Session: https://claude.ai/code/session_011LxXAm5E4Vy39kRLr8kRnX --- .../outbound_stream_absolute_lifetime_test.rs | 103 ++++++++++++++++++ 1 file changed, 103 insertions(+) create mode 100644 tests/outbound_stream_absolute_lifetime_test.rs diff --git a/tests/outbound_stream_absolute_lifetime_test.rs b/tests/outbound_stream_absolute_lifetime_test.rs new file mode 100644 index 00000000..8cdb4f3d --- /dev/null +++ b/tests/outbound_stream_absolute_lifetime_test.rs @@ -0,0 +1,103 @@ +//! Real-socket regression for P1 (#3): `outbound_stream_max_seconds` must be a +//! TRUE absolute lifetime — enforced even when a read is served from bytes that +//! were already buffered locally by an earlier read. +//! +//! `stream_pull` already fails a network read once the absolute deadline has +//! elapsed, so an empty-buffer read after the deadline errors correctly. But +//! `next_line`/`next_chunk` serve buffered bytes BEFORE consulting the deadline, +//! so a proxy that pulled a multi-line chunk could keep draining that buffer long +//! after the stream's absolute lifetime expired. This proves a buffered read +//! taken past the deadline now fails (and the upstream is dropped), instead of +//! succeeding. + +use std::sync::Arc; +use std::time::{Duration, Instant}; +use tokio::io::{AsyncReadExt, AsyncWriteExt}; +use wfl::Interpreter; +use wfl::config::WflConfig; +use wfl::lexer::lex_wfl_with_positions; +use wfl::parser::Parser; + +/// Upstream: send a chunked head + ONE body chunk framing two newline-terminated +/// lines ("line1\nline2\n"), then STALL (keep the socket open, send nothing more). +/// Signal on the returned receiver when the proxy drops the upstream connection. +async fn spawn_two_lines_then_stall_upstream() -> (u16, tokio::sync::oneshot::Receiver<()>) { + let listener = tokio::net::TcpListener::bind("127.0.0.1:0") + .await + .expect("bind mock upstream"); + let port = listener.local_addr().unwrap().port(); + let (tx, rx) = tokio::sync::oneshot::channel(); + tokio::spawn(async move { + if let Ok((mut sock, _)) = listener.accept().await { + let mut buf = [0u8; 1024]; + let _ = sock.read(&mut buf).await; // request head + let head = "HTTP/1.1 200 OK\r\n\ + Content-Type: text/plain\r\n\ + Transfer-Encoding: chunked\r\n\r\n"; + let _ = sock.write_all(head.as_bytes()).await; + // One chunk carrying both lines: 0xC = 12 bytes = "line1\nline2\n". + let _ = sock.write_all(b"C\r\nline1\nline2\n\r\n").await; + let _ = sock.flush().await; + // Stall: never send more, never close. Detect the proxy dropping the + // upstream (its handle expired) via a blocking read returning 0/Err. + loop { + match sock.read(&mut buf).await { + Ok(0) | Err(_) => { + let _ = tx.send(()); + return; + } + Ok(_) => {} + } + } + } + }); + (port, rx) +} + +#[tokio::test] +async fn test_buffered_read_after_absolute_deadline_expires() { + let (port, mut upstream_closed) = spawn_two_lines_then_stall_upstream().await; + + // Read the first line (which buffers "line2"), sleep past the 1s absolute + // stream lifetime, then read again. The second line comes from the local + // buffer — it must NOT be served after the absolute deadline. + let code = format!( + r#"open url at "http://127.0.0.1:{port}/" and stream response as s +wait for next line from s as a +wait for 1500 milliseconds +wait for next line from s as b"# + ); + let tokens = lex_wfl_with_positions(&code); + let program = Parser::new(&tokens).parse().expect("parse"); + + // Idle/run timeout 10s; absolute stream lifetime 1s. + let config = WflConfig { + timeout_seconds: 10, + outbound_stream_max_seconds: 1, + ..WflConfig::default() + }; + let mut interp = Interpreter::with_config(Arc::new(config)); + + let start = Instant::now(); + let result = interp.interpret(&program).await; + let elapsed = start.elapsed(); + + assert!( + result.is_err(), + "a buffered `wait for next line` taken ~1.5s after opening (past the 1s \ + absolute stream lifetime) must fail, not be served from the local buffer" + ); + // It must fail at the buffered read (~1.5s in), not hang out to the 10s + // run timeout or the mock's 30s stall. + assert!( + elapsed < Duration::from_secs(5), + "the expired buffered read should fail promptly (took {elapsed:?})" + ); + + // Expiring the handle must drop the upstream (cancel the request), so the + // mock observes its connection close. + tokio::time::timeout(Duration::from_secs(5), &mut upstream_closed) + .await + .expect("upstream was not closed after the stream's absolute lifetime expired") + .expect("upstream close sender dropped"); +} From 7fa2eb054337e1f3c595054cef48c54c13c3728a Mon Sep 17 00:00:00 2001 From: Claude Date: Fri, 24 Jul 2026 09:46:36 +0000 Subject: [PATCH 061/132] fix(P1): drop a span-mismatched classic write fallback MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The ambiguous `write line|chunk ... to ` form parses two independent readings of the same continuation: the stream value and the classic file-write fallback. The fallback was kept whenever it merely parsed, even if it consumed a DIFFERENT span than the stream reading — so `write line min with a: 1 and b: 2 to ` retained a partial `line min with a` fallback (the classic reading stops at the `:` that the builtin-call stream reading consumes as a named arg), corrupting the file write. Keep the fallback only when it consumed exactly to the stream reading's end checkpoint; otherwise the two readings disagree and there is no valid classic interpretation, so the non-stream target is a clean error. Turns tests/write_line_backcompat_test.rs (two new cases) green while the matching-span back-compat cases still pass. Co-Authored-By: Claude Opus 4.8 Claude-Session: https://claude.ai/code/session_011LxXAm5E4Vy39kRLr8kRnX --- src/parser/stmt/io.rs | 11 +++++++++++ 1 file changed, 11 insertions(+) diff --git a/src/parser/stmt/io.rs b/src/parser/stmt/io.rs index aa340c1a..1deb251b 100644 --- a/src/parser/stmt/io.rs +++ b/src/parser/stmt/io.rs @@ -910,6 +910,17 @@ impl<'a> IoParser<'a> for Parser<'a> { self.cursor.rewind(value_start); let file_left = Expression::Variable(id, marker_line, marker_column); let fallback = self.parse_write_value_from_lead(file_left).ok(); + // Only keep the classic fallback when it consumed EXACTLY the same + // continuation span as the stream reading. A fallback that parses + // a shorter (or longer) span is a different interpretation of the + // tokens — e.g. `write line min with a: 1 and b: 2 to `, + // where the stream reading is the builtin call `min` with named + // args but `line min with a` only parses up to the `:`. Retaining + // that partial parse and pairing it with the SAME trailing + // `to ` would silently corrupt a file write, so require the + // spans to match before trusting the fallback. + let fallback_end = self.cursor.checkpoint(); + let fallback = fallback.filter(|_| fallback_end == after_stream); // Always resume right after the stream value, whatever the // (speculative) fallback parse consumed, so `to ` follows. self.cursor.rewind(after_stream); From ed4764e161c8bdfc624d6321e6cc1ec195d1a67e Mon Sep 17 00:00:00 2001 From: Claude Date: Fri, 24 Jul 2026 09:46:36 +0000 Subject: [PATCH 062/132] fix(P1): enforce the absolute stream lifetime on buffered reads next_line/next_chunk served locally-buffered bytes before consulting the handle's absolute deadline, so outbound_stream_max_seconds bounded only network reads: a proxy that pulled a multi-line chunk could keep draining the buffer past the stream's absolute lifetime. Add check_stream_deadline and call it before serving buffered bytes in next_chunk and before serving a buffered line in next_line's loop; on expiry the handle is dropped (cancelling the upstream). An empty-buffer read already expired via stream_pull's identical check. Turns tests/outbound_stream_absolute_lifetime_test.rs green. Co-Authored-By: Claude Opus 4.8 Claude-Session: https://claude.ai/code/session_011LxXAm5E4Vy39kRLr8kRnX --- src/interpreter/mod.rs | 27 +++++++++++++++++++++++++++ 1 file changed, 27 insertions(+) diff --git a/src/interpreter/mod.rs b/src/interpreter/mod.rs index 1dbdb816..8278d431 100644 --- a/src/interpreter/mod.rs +++ b/src/interpreter/mod.rs @@ -2001,6 +2001,22 @@ impl IoClient { } } + /// Return a `Timeout` error if the handle's absolute stream lifetime + /// (`outbound_stream_max_seconds`, tracked as `total_deadline`) has elapsed. + /// Called before serving locally-buffered bytes so the absolute lifetime is + /// enforced even when no network read is performed; `stream_pull` performs the + /// same check before each network read. + fn check_stream_deadline(&self, handle: &HttpStreamHandle) -> Result<(), HttpClientError> { + if let Some(deadline) = handle.total_deadline + && deadline.saturating_duration_since(Instant::now()).is_zero() + { + return Err(HttpClientError::Timeout { + seconds: self.config.outbound_stream_max_seconds, + }); + } + Ok(()) + } + /// Pull the next raw byte chunk from a streaming response. Returns /// `Ok(None)` at clean end of stream (handle is dropped). On error or EOF /// the handle is not re-inserted, so the upstream request is released. @@ -2011,6 +2027,12 @@ impl IoClient { ) -> Result>, HttpClientError> { let mut handle = self.take_stream(handle_id).await?; + // Enforce the absolute stream lifetime before serving ANY bytes — even + // ones already buffered by a prior read — so `outbound_stream_max_seconds` + // is a true absolute lifetime, not merely a per-network-read bound. On + // expiry the handle is not re-inserted, so the upstream request is dropped. + self.check_stream_deadline(&handle)?; + // Any bytes buffered by a prior `next line` are served first. if !handle.buffer.is_empty() { let chunk = std::mem::take(&mut handle.buffer); @@ -2040,6 +2062,11 @@ impl IoClient { let mut handle = self.take_stream(handle_id).await?; loop { + // Enforce the absolute stream lifetime before serving a buffered line + // (a prior read may have buffered several lines); on expiry the handle + // is dropped, cancelling the upstream. See `next_chunk`. + self.check_stream_deadline(&handle)?; + if let Some(pos) = handle.buffer.iter().position(|&b| b == b'\n') { let mut line: Vec = handle.buffer.drain(..=pos).collect(); line.pop(); // drop '\n' From b760db3c727f3e6ff2138184a41fd2a438a8923d Mon Sep 17 00:00:00 2001 From: Claude Date: Fri, 24 Jul 2026 09:58:14 +0000 Subject: [PATCH 063/132] test(P1): failing coverage for outbound cleanup on a dropped interpret() A dropped/cancelled interpret() future does not run the handler-exit or program cleanup sites, so an outbound stream handle parked (opened, not mid-read) in IoClient.stream_handles leaks the upstream until the interpreter itself is dropped. Red: with the interpreter kept alive after the future is dropped, the mock upstream is never disconnected. Co-Authored-By: Claude Opus 4.8 Claude-Session: https://claude.ai/code/session_011LxXAm5E4Vy39kRLr8kRnX --- tests/dropped_interpret_cleanup_test.rs | 87 +++++++++++++++++++++++++ 1 file changed, 87 insertions(+) create mode 100644 tests/dropped_interpret_cleanup_test.rs diff --git a/tests/dropped_interpret_cleanup_test.rs b/tests/dropped_interpret_cleanup_test.rs new file mode 100644 index 00000000..f3171a35 --- /dev/null +++ b/tests/dropped_interpret_cleanup_test.rs @@ -0,0 +1,87 @@ +//! Real-socket regression for P1 (#4): a cancelled/dropped `interpret()` future +//! must still close the outbound stream handles the run opened. +//! +//! Handler-exit cleanup (concurrent `IsolatedHandler::drop`, serial-loop/program +//! cleanup sites) closes outbound handles on normal control-flow exits. But if the +//! whole `interpret()` future is DROPPED (an embedder cancels it) while a handle +//! sits idle in `IoClient.stream_handles` — opened, not currently inside a read — +//! none of those sites run, and (with the interpreter kept alive, e.g. a reused +//! REPL) the upstream request leaks until the interpreter itself is dropped. An +//! RAII guard tied to the run must drop those handles when the future unwinds. + +use std::time::Duration; +use tokio::io::{AsyncReadExt, AsyncWriteExt}; +use wfl::Interpreter; +use wfl::lexer::lex_wfl_with_positions; +use wfl::parser::Parser; + +/// Upstream: send a chunked head immediately, then STALL. Signal when the proxy +/// drops the connection (a blocking read returns 0/Err at peer close). +async fn spawn_head_then_stall_upstream() -> (u16, tokio::sync::oneshot::Receiver<()>) { + let listener = tokio::net::TcpListener::bind("127.0.0.1:0") + .await + .expect("bind mock upstream"); + let port = listener.local_addr().unwrap().port(); + let (tx, rx) = tokio::sync::oneshot::channel(); + tokio::spawn(async move { + if let Ok((mut sock, _)) = listener.accept().await { + let mut buf = [0u8; 1024]; + let _ = sock.read(&mut buf).await; // request head + let head = "HTTP/1.1 200 OK\r\n\ + Content-Type: text/plain\r\n\ + Transfer-Encoding: chunked\r\n\r\n"; + let _ = sock.write_all(head.as_bytes()).await; + let _ = sock.flush().await; + loop { + match sock.read(&mut buf).await { + Ok(0) | Err(_) => { + let _ = tx.send(()); + return; + } + Ok(_) => {} + } + } + } + }); + (port, rx) +} + +#[tokio::test] +async fn test_dropped_interpret_future_closes_outbound_handle() { + let (port, mut upstream_closed) = spawn_head_then_stall_upstream().await; + + // Open an outbound stream, then sit in a long wait WITHOUT reading it — the + // handle is parked in `IoClient.stream_handles`, not held inside an in-flight + // read (dropping a read future would itself drop the handle and mask the bug). + let code = format!( + r#"open url at "http://127.0.0.1:{port}/" and stream response as up +wait for 5000 milliseconds"# + ); + let tokens = lex_wfl_with_positions(&code); + let program = Parser::new(&tokens).parse().expect("parse"); + + let mut interp = Interpreter::new(); + { + let fut = interp.interpret(&program); + tokio::pin!(fut); + // Drive the run long enough to open the stream and enter the wait, then + // let `fut` drop at the end of this scope (the interpret future is + // cancelled). The interpreter itself stays alive below. + let _ = tokio::time::timeout(Duration::from_millis(800), fut.as_mut()).await; + } + + // The dropped future must have released the outbound handle (RAII), so the + // upstream is cancelled and the mock observes its connection close — even + // though `interp` is still alive. + tokio::time::timeout(Duration::from_secs(3), &mut upstream_closed) + .await + .expect( + "upstream was not closed after the interpret() future was dropped — \ + the outbound handle leaked until interpreter teardown", + ) + .expect("upstream close sender dropped"); + + // Keep the interpreter alive until after the assertion, so the close was the + // RAII guard's doing and not the interpreter being torn down. + drop(interp); +} From 3ba8351467ed909539353ba7f0a3962a10c27a27 Mon Sep 17 00:00:00 2001 From: Claude Date: Fri, 24 Jul 2026 10:06:38 +0000 Subject: [PATCH 064/132] fix(P1): close outbound handles when the interpret() future is dropped Handler-exit and program-cleanup sites close outbound stream handles on normal control-flow exits, but a dropped/cancelled interpret() future runs none of them, leaking a parked (opened, not mid-read) upstream until the interpreter itself is dropped. Add an RAII OutboundStreamCleanup guard held for the whole interpret_inner body: it shares open_http_streams and the IoClient via Rc, so its Drop closes the tracked handles (cancelling their upstreams) even as the future unwinds and the interpreter stays alive. On a normal run the exit sites drain the list first, so the guard is a no-op. open_http_streams becomes Rc>> so the guard can share it. Turns tests/dropped_interpret_cleanup_test.rs green. Co-Authored-By: Claude Opus 4.8 Claude-Session: https://claude.ai/code/session_011LxXAm5E4Vy39kRLr8kRnX --- src/interpreter/mod.rs | 55 ++++++++++++++++++++++++++++++++++++++++-- 1 file changed, 53 insertions(+), 2 deletions(-) diff --git a/src/interpreter/mod.rs b/src/interpreter/mod.rs index 8278d431..7b99d656 100644 --- a/src/interpreter/mod.rs +++ b/src/interpreter/mod.rs @@ -884,6 +884,35 @@ impl<'a, T> Drop for IsolatedHandler<'a, T> { } } +/// RAII guard that closes the interpreter's still-open outbound stream handles +/// when the running `interpret()` future ends — crucially including when that +/// future is *dropped* (an embedder cancels the run) before the normal +/// handler-exit / program-cleanup sites execute. It shares the interpreter's +/// `open_http_streams` list and its `IoClient` via `Rc`, so its `Drop` runs even +/// as the interpreter itself stays alive (e.g. a reused REPL). On a normal run +/// the cleanup sites have already drained the list, so this is a no-op. +struct OutboundStreamCleanup { + io_client: Rc, + open_http_streams: Rc>>, +} + +impl Drop for OutboundStreamCleanup { + fn drop(&mut self) { + let ids = std::mem::take(&mut *self.open_http_streams.borrow_mut()); + if ids.is_empty() { + return; + } + // Best-effort like the other handler-exit cleanup: `try_lock` never + // blocks in a `Drop`. Removing a handle drops its reqwest stream, which + // cancels the in-flight upstream request. + if let Ok(mut map) = self.io_client.stream_handles.try_lock() { + for id in &ids { + map.remove(id); + } + } + } +} + /// RAII guard that ensures module loading context is restored on scope exit. /// Automatically pops loading_stack and restores current_source_file when dropped. struct ModuleLoadGuard<'a> { @@ -1273,7 +1302,11 @@ pub struct Interpreter { /// per-handler `RunState` (swapped per poll); any still open when the handler /// ends are dropped, cancelling their upstream requests (see /// `close_http_streams`). - open_http_streams: RefCell>, + /// `Rc>` (not a bare `RefCell`) so an RAII cleanup guard tied to + /// the `interpret()` future can share the list and close these handles if the + /// future is dropped/cancelled before its normal exit sites run (see + /// `OutboundStreamCleanup`). + open_http_streams: Rc>>, #[allow(dead_code)] // Used for future security features config: Arc, // Configuration for security and other settings current_source_file: RefCell>, // Currently executing source file (for path resolution) @@ -3429,7 +3462,7 @@ impl Interpreter { server_response_streams: RefCell::new(HashMap::new()), open_response_streams: RefCell::new(Vec::new()), open_pending_requests: RefCell::new(Vec::new()), - open_http_streams: RefCell::new(Vec::new()), + open_http_streams: Rc::new(RefCell::new(Vec::new())), next_response_stream_id: std::cell::Cell::new(1), config, current_source_file: RefCell::new(None), // No source file initially @@ -4013,6 +4046,19 @@ impl Interpreter { } } + /// Build an RAII guard that closes any outbound stream handles still tracked + /// as open when the guard drops — including when the `interpret()` future is + /// dropped/cancelled before reaching its normal handler-exit/program-cleanup + /// sites. On a normal run those sites have already drained the list, so the + /// guard is a no-op; on a dropped future it releases the leaked upstreams + /// (instead of leaking them until the interpreter itself is torn down). + fn outbound_stream_cleanup_guard(&self) -> OutboundStreamCleanup { + OutboundStreamCleanup { + io_client: Rc::clone(&self.io_client), + open_http_streams: Rc::clone(&self.open_http_streams), + } + } + /// Drain and drop every outbound stream the current (serial) handler left /// open. Called at the end of each serial `main loop` iteration and at program /// exit, mirroring the concurrent path's per-handler `Drop`. @@ -4463,6 +4509,11 @@ impl Interpreter { self.close_open_response_streams(); self.fail_open_pending_requests(); self.close_open_http_streams(); + // RAII: if THIS run's future is dropped/cancelled before its normal exit + // sites run, still close any outbound handles it opened (they would + // otherwise leak the upstream until the interpreter itself is dropped). + // On a normal run the exit sites drain the list first, so this is a no-op. + let _outbound_cleanup = self.outbound_stream_cleanup_guard(); // Reset to the inherited base depth (0 for a top-level run/REPL; the // parent's live depth for an `execute file` child) so recursion // accounting spans the execute-file boundary instead of granting the From 0c750fc0e69cc41a69730df145456f381e2d329e Mon Sep 17 00:00:00 2001 From: Claude Date: Fri, 24 Jul 2026 10:06:39 +0000 Subject: [PATCH 065/132] test: bind server tests to an OS-assigned free port, not a hardcoded one Hardcoded ports flake under parallel test runs or on a busy runner (review feedback). WFL's `listen on port ` takes a literal, so a shared tests/common/free_tcp_port() probes for a free port and releases it for the program to bind. Converts the burst, outbound-disconnect, concurrent main-loop, and server-streaming tests. Co-Authored-By: Claude Opus 4.8 Claude-Session: https://claude.ai/code/session_011LxXAm5E4Vy39kRLr8kRnX --- tests/common/mod.rs | 21 +++++++++++++++++++++ tests/concurrent_disconnect_burst_test.rs | 4 +++- tests/concurrent_main_loop_test.rs | 12 +++++++----- tests/http_server_streaming_test.rs | 10 ++++++---- tests/outbound_stream_disconnect_test.rs | 4 +++- 5 files changed, 40 insertions(+), 11 deletions(-) create mode 100644 tests/common/mod.rs diff --git a/tests/common/mod.rs b/tests/common/mod.rs new file mode 100644 index 00000000..fd84316a --- /dev/null +++ b/tests/common/mod.rs @@ -0,0 +1,21 @@ +//! Shared helpers for integration tests. +#![allow(dead_code)] + +use std::net::TcpListener; + +/// Ask the OS for a currently-free TCP port on loopback, then release it so the +/// caller can bind it via WFL's `listen on port `. +/// +/// WFL takes a *literal* port in `listen on port `, so the port must be chosen +/// before the program source is built — we cannot bind an ephemeral `:0` and read +/// the assigned port back the way the mock upstreams do. Picking a free port from +/// the OS (instead of a hardcoded constant) avoids collisions under parallel test +/// runs and on busy runners. A small TOCTOU window remains between releasing the +/// probe socket and WFL re-binding it, but it is far less flaky than a fixed port. +pub fn free_tcp_port() -> u16 { + TcpListener::bind("127.0.0.1:0") + .expect("bind an ephemeral TCP port") + .local_addr() + .expect("read the ephemeral local address") + .port() +} diff --git a/tests/concurrent_disconnect_burst_test.rs b/tests/concurrent_disconnect_burst_test.rs index bad54b62..e07fe18d 100644 --- a/tests/concurrent_disconnect_burst_test.rs +++ b/tests/concurrent_disconnect_burst_test.rs @@ -27,6 +27,8 @@ use wfl::Interpreter; use wfl::lexer::lex_wfl_with_positions; use wfl::parser::Parser; +mod common; + /// How many disconnecting clients to fire. Must exceed the concurrent loop's /// `MAX_CONSECUTIVE_FAILURES` (256) so, under the buggy behavior, the burst trips /// the structural breaker. @@ -139,7 +141,7 @@ async fn fire_disconnect(proxy_port: u16) { async fn test_disconnect_burst_does_not_kill_concurrent_loop() { let (upstream_port, mut upstream_closes) = spawn_counting_stall_upstream().await; - let proxy_port = 8362; + let proxy_port = common::free_tcp_port(); let code = format!( r#" listen on port {proxy_port} as srv diff --git a/tests/concurrent_main_loop_test.rs b/tests/concurrent_main_loop_test.rs index c9dd67dd..9648b055 100644 --- a/tests/concurrent_main_loop_test.rs +++ b/tests/concurrent_main_loop_test.rs @@ -16,6 +16,8 @@ use wfl::lexer::lex_wfl_with_positions; use wfl::parser::Parser; use wfl::parser::ast::Statement; +mod common; + fn parse_program(code: &str) -> Vec { let tokens = lex_wfl_with_positions(code); let mut parser = Parser::new(&tokens); @@ -121,7 +123,7 @@ async fn shutdown(port: u16, server: std::thread::JoinHandle<()>) { #[tokio::test] async fn test_concurrent_slow_handler_does_not_block_fast() { - let port = 8341; + let port = common::free_tcp_port(); let server = start_server_thread(server_code(port, true)); wait_for_server(port).await; @@ -159,7 +161,7 @@ async fn test_concurrent_slow_handler_does_not_block_fast() { async fn test_concurrent_handler_error_does_not_kill_server() { // A handler that errors mid-iteration (here: responding twice) must be // contained — the concurrent loop keeps serving other requests. - let port = 8342; + let port = common::free_tcp_port(); let code = format!( r#" listen on port {port} as srv @@ -215,7 +217,7 @@ async fn test_concurrent_handlers_do_not_share_count_loop_state() { // // The two ranges are disjoint (1..5 vs 100..104), so any cross-contamination // is unmistakable: with isolation each handler observes only its own range. - let port = 8344; + let port = common::free_tcp_port(); let code = format!( r#" listen on port {port} as srv @@ -294,7 +296,7 @@ async fn test_handler_that_never_responds_gets_immediate_500() { // WITHOUT responding must resolve the client with 500 immediately — not leave // it waiting out the request timeout. The `/drop` path does no `respond`; the // handler simply ends, and the client must still get a prompt 500. - let port = 8345; + let port = common::free_tcp_port(); let code = format!( r#" listen on port {port} as srv @@ -351,7 +353,7 @@ async fn test_handler_that_never_responds_gets_immediate_500() { #[tokio::test] async fn test_serial_slow_handler_blocks_next() { - let port = 8343; + let port = common::free_tcp_port(); let server = start_server_thread(server_code(port, false)); wait_for_server(port).await; diff --git a/tests/http_server_streaming_test.rs b/tests/http_server_streaming_test.rs index 7d096eec..a33f773b 100644 --- a/tests/http_server_streaming_test.rs +++ b/tests/http_server_streaming_test.rs @@ -13,6 +13,8 @@ use wfl::lexer::lex_wfl_with_positions; use wfl::parser::Parser; use wfl::parser::ast::Statement; +mod common; + // ----------------------------- parser tests ------------------------------ fn parse_single_statement(code: &str) -> Statement { @@ -172,7 +174,7 @@ async fn wait_for_server(port: u16) { #[tokio::test] async fn test_streamed_response_lines_and_headers() { - let port = 8231; + let port = common::free_tcp_port(); let server_code = format!( r#" listen on port {port} as s @@ -215,7 +217,7 @@ async fn test_streamed_response_lines_and_headers() { async fn test_write_after_close_does_not_reach_client() { // Writing after `close out` is a catchable error and does NOT reach the // client: the client sees only the bytes written before close. - let port = 8233; + let port = common::free_tcp_port(); let server_code = format!( r#" listen on port {port} as s @@ -255,7 +257,7 @@ async fn test_stream_auto_closes_when_handler_ends_without_close() { // WITHOUT `close out` must still finalize the client's body on the way out. // Otherwise the sender lingers in the interpreter's stream table, the body is // never terminated, and the client hangs forever (and the table leaks). - let port = 8234; + let port = common::free_tcp_port(); let server_code = format!( r#" listen on port {port} as s @@ -303,7 +305,7 @@ async fn test_stream_auto_closes_when_handler_ends_without_close() { #[tokio::test] async fn test_streamed_response_write_chunk_verbatim() { - let port = 8232; + let port = common::free_tcp_port(); let server_code = format!( r#" listen on port {port} as s diff --git a/tests/outbound_stream_disconnect_test.rs b/tests/outbound_stream_disconnect_test.rs index f5f158a1..40e2bb1b 100644 --- a/tests/outbound_stream_disconnect_test.rs +++ b/tests/outbound_stream_disconnect_test.rs @@ -15,6 +15,8 @@ use wfl::Interpreter; use wfl::lexer::lex_wfl_with_positions; use wfl::parser::Parser; +mod common; + /// Upstream: send a chunked head + one body chunk, then STALL (send nothing /// more, so the proxy's next read blocks). Detect the proxy dropping the /// connection via a blocking read that returns 0 at peer close. @@ -80,7 +82,7 @@ async fn wait_for_server(port: u16) { async fn test_downstream_disconnect_cancels_blocked_upstream_read() { let (upstream_port, mut upstream_disconnect) = spawn_one_chunk_then_stall_upstream().await; - let proxy_port = 8351; + let proxy_port = common::free_tcp_port(); // The handler proxies: read chunks from upstream and write them downstream. // After the first chunk it blocks on the stalled upstream. `outbound_stream_max_seconds` // is the default (300s), so ONLY a disconnect can cancel that blocked read From 825c215d522e0cd8d5034d131fb4f43439b7fa44 Mon Sep 17 00:00:00 2001 From: Claude Date: Fri, 24 Jul 2026 10:26:14 +0000 Subject: [PATCH 066/132] test(P1): failing coverage for pre-head disconnect cancellation A browser disconnect while a proxy handler is blocked in the UPSTREAM HEAD phase (open url ... and stream response), before any start streaming response, is not cancelled: the head open only races a downstream response stream that does not exist yet. Red: with the upstream withholding its head, a client disconnect does not close the upstream within the window. Co-Authored-By: Claude Opus 4.8 Claude-Session: https://claude.ai/code/session_011LxXAm5E4Vy39kRLr8kRnX --- tests/outbound_stream_head_disconnect_test.rs | 149 ++++++++++++++++++ 1 file changed, 149 insertions(+) create mode 100644 tests/outbound_stream_head_disconnect_test.rs diff --git a/tests/outbound_stream_head_disconnect_test.rs b/tests/outbound_stream_head_disconnect_test.rs new file mode 100644 index 00000000..0c922bc4 --- /dev/null +++ b/tests/outbound_stream_head_disconnect_test.rs @@ -0,0 +1,149 @@ +//! Real-socket regression for P1 (#1): a downstream (browser) disconnect must +//! cancel a proxy handler blocked in the UPSTREAM HEAD phase (`open url ... and +//! stream response`), before any `start streaming response`, not only a blocked +//! body read. +//! +//! Topology: an upstream that WITHHOLDS its response head <- WFL concurrent proxy +//! -> a client that connects and disconnects while the handler is blocked opening +//! the upstream. The upstream must observe its connection close promptly (the +//! handler cancelled the head open because the client went away), and an unrelated +//! `/ping` must still be served throughout. + +use std::time::Duration; +use tokio::io::{AsyncReadExt, AsyncWriteExt}; +use wfl::Interpreter; +use wfl::lexer::lex_wfl_with_positions; +use wfl::parser::Parser; + +mod common; + +/// Upstream: accept, read the request, then WITHHOLD the response head (send +/// nothing). Signal when the proxy drops the connection (peer close => read 0/Err). +async fn spawn_header_withholding_upstream() -> (u16, tokio::sync::oneshot::Receiver<()>) { + let listener = tokio::net::TcpListener::bind("127.0.0.1:0") + .await + .expect("bind mock upstream"); + let port = listener.local_addr().unwrap().port(); + let (tx, rx) = tokio::sync::oneshot::channel(); + tokio::spawn(async move { + if let Ok((mut sock, _)) = listener.accept().await { + let mut buf = [0u8; 1024]; + let _ = sock.read(&mut buf).await; // request head + // Withhold the response head entirely; just wait for the proxy to drop + // the connection when its client disconnects. + loop { + match sock.read(&mut buf).await { + Ok(0) | Err(_) => { + let _ = tx.send(()); + return; + } + Ok(_) => {} + } + } + } + }); + (port, rx) +} + +fn start_proxy_server(code: String) -> std::thread::JoinHandle<()> { + std::thread::spawn(move || { + let rt = tokio::runtime::Runtime::new().expect("runtime"); + rt.block_on(async { + let tokens = lex_wfl_with_positions(&code); + let ast = Parser::new(&tokens).parse().expect("parse"); + let mut interp = Interpreter::new(); + if let Err(errors) = interp.interpret(&ast).await { + panic!("proxy interpreter failed: {errors:?}"); + } + }); + }) +} + +async fn wait_for_server(port: u16) { + let addr = format!("127.0.0.1:{port}"); + for _ in 0..300 { + if tokio::net::TcpStream::connect(&addr).await.is_ok() { + return; + } + tokio::time::sleep(Duration::from_millis(50)).await; + } + panic!("proxy server on {addr} did not become ready"); +} + +#[tokio::test] +async fn test_disconnect_cancels_blocked_upstream_head_open() { + let (upstream_port, mut upstream_closed) = spawn_header_withholding_upstream().await; + let proxy_port = common::free_tcp_port(); + + let code = format!( + r#" + listen on port {proxy_port} as srv + main loop concurrently: + wait for request comes in on srv as req with timeout 30000 + store p as req["path"] + check if p is equal to "/ping": + respond to req with "pong" + otherwise: + check if p is equal to "/shutdown": + respond to req with "bye" + close server srv + break + otherwise: + open url at "http://127.0.0.1:{upstream_port}/" and stream response as up + start streaming response to req with status 200 and content type "text/plain" as down + wait for next chunk from up as c + close down + end check + end check + end loop + "# + ); + let server = start_proxy_server(code); + wait_for_server(proxy_port).await; + + // Client: connect, send the request, then DISCONNECT while the handler is + // blocked opening the (header-withholding) upstream. + { + let mut sock = tokio::net::TcpStream::connect(("127.0.0.1", proxy_port)) + .await + .expect("connect to proxy"); + sock.write_all(b"GET /proxy HTTP/1.1\r\nHost: 127.0.0.1\r\n\r\n") + .await + .expect("send request"); + sock.flush().await.ok(); + // Give the handler a moment to dequeue and reach the blocked head open, + // then drop the socket to disconnect. + tokio::time::sleep(Duration::from_millis(300)).await; + // `sock` drops here -> client disconnects. + } + + // The upstream must observe its connection close promptly — the blocked head + // open was cancelled by the disconnect, not left to wait out the idle timeout. + tokio::time::timeout(Duration::from_secs(4), &mut upstream_closed) + .await + .expect("upstream head open was not cancelled after the client disconnected") + .expect("upstream close sender dropped"); + + // The concurrent loop stayed alive: an unrelated request is still served. + let ping = tokio::time::timeout( + Duration::from_secs(5), + reqwest::Client::new() + .get(format!("http://127.0.0.1:{proxy_port}/ping")) + .send(), + ) + .await + .expect("/ping timed out") + .expect("/ping failed"); + assert_eq!(ping.status().as_u16(), 200); + assert_eq!(ping.text().await.unwrap(), "pong"); + + let _ = reqwest::Client::new() + .get(format!("http://127.0.0.1:{proxy_port}/shutdown")) + .send() + .await; + match tokio::task::spawn_blocking(move || server.join()).await { + Ok(Ok(())) => {} + Ok(Err(panic)) => std::panic::resume_unwind(panic), + Err(e) => panic!("server join task failed: {e}"), + } +} From 706200f7b8999f8ae6841526a5ef6aeca1827efc Mon Sep 17 00:00:00 2001 From: Claude Date: Fri, 24 Jul 2026 10:26:14 +0000 Subject: [PATCH 067/132] fix(P1): cancel a blocked upstream head open on client disconnect MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The disconnect signal that cancels a blocked upstream read only came from an open downstream response stream, which does not exist yet during the head phase (open url ... and stream response, before start streaming response). Add a per-request disconnect signal — the transport drops the request's oneshot receiver when the client goes away, so the parked sender reports is_closed() — polled via any_pending_request_disconnected. any_client_ disconnected races both signals; the head open and both body reads now select against it, so a browser disconnect cancels the handler in either phase (dropping open_fut aborts the upstream connection). Turns tests/outbound_stream_head_disconnect_test.rs green. Co-Authored-By: Claude Opus 4.8 Claude-Session: https://claude.ai/code/session_011LxXAm5E4Vy39kRLr8kRnX --- src/interpreter/mod.rs | 106 ++++++++++++++++++++++++++++++++++------- 1 file changed, 90 insertions(+), 16 deletions(-) diff --git a/src/interpreter/mod.rs b/src/interpreter/mod.rs index 7b99d656..e41230b1 100644 --- a/src/interpreter/mod.rs +++ b/src/interpreter/mod.rs @@ -1690,6 +1690,13 @@ enum OutboundHttpDeadline { /// quickly an in-flight socket operation observes `ExecutionBudget::cancel()`. const HTTP_CANCELLATION_POLL_INTERVAL: Duration = Duration::from_millis(10); +/// How often a blocked upstream operation polls its handler's pending requests +/// for a client disconnect (the transport drops the oneshot receiver). Polled +/// because the sender lives behind an `Arc>>` shared with the +/// transport rather than an awaitable primitive; small so a disconnect is +/// observed promptly. +const REQUEST_DISCONNECT_POLL_INTERVAL: Duration = Duration::from_millis(20); + #[derive(Debug)] enum FileReadError { Io(String), @@ -4106,6 +4113,60 @@ impl Interpreter { let _ = futures_util::future::select_all(closes).await; } + /// Await until any of this handler's open pending requests has had its client + /// disconnect — the transport route task drops the oneshot receiver when the + /// client goes away, so the parked sender reports `is_closed()`. This is the + /// disconnect signal that is valid BEFORE `start streaming response` (when no + /// downstream response stream exists yet), so a handler blocked opening an + /// upstream head can still be cancelled by a browser disconnect. Polled (the + /// sender lives behind an `Arc>>` shared with the transport, + /// not an awaitable primitive); never resolves when the handler holds no + /// pending request, so the caller's `select!` still has a live branch. + async fn any_pending_request_disconnected(&self) { + loop { + // Compute in a tight scope so no `RefCell`/`Mutex` guard is held across + // the await below. `None` => nothing to watch. + let any_closed = { + let open = self.open_pending_requests.borrow(); + if open.is_empty() { + None + } else { + let pending = self.pending_responses.borrow(); + Some(open.iter().any(|id| { + pending.get(id).is_some_and(|p| match p.sender.try_lock() { + Ok(guard) => guard.as_ref().is_some_and(|s| s.is_closed()), + // Being responded to right now — not a disconnect. + Err(_) => false, + }) + })) + } + }; + match any_closed { + None => { + std::future::pending::<()>().await; + return; + } + Some(true) => return, + Some(false) => { + tokio::time::sleep(REQUEST_DISCONNECT_POLL_INTERVAL).await; + } + } + } + } + + /// Await until this handler's client has disconnected by EITHER signal: an + /// open downstream response stream's receiver dropped (post-`start streaming + /// response`, event-driven) OR an open pending request's oneshot receiver + /// dropped (pre-`start streaming response`, polled). Racing an upstream head + /// open / body read against this cancels it the moment the browser goes away, + /// whichever phase the handler is in. + async fn any_client_disconnected(&self, senders: Vec>>) { + tokio::select! { + _ = Self::any_downstream_disconnected(senders) => {} + _ = self.any_pending_request_disconnected() => {} + } + } + /// Close (drop the sender for) each server response stream whose handle id is /// in `ids`, ending its body so the client stops waiting. Idempotent — an id /// already closed by an explicit `close out` (or a disconnect) is a no-op — @@ -7241,17 +7302,29 @@ impl Interpreter { None => None, }; - match self - .io_client - .open_http_stream( - &method_str, - &url_str, - &header_list, - body_str, - Arc::clone(&self.budget), - ) - .await - { + // Race the head open (connect + await response head) against a + // client disconnect, so a browser that goes away while the upstream + // withholds its head cancels the open promptly (dropping `open_fut` + // aborts the upstream connection) instead of waiting out the head + // timeout. Valid before `start streaming response` via the pending + // request's oneshot (see `any_pending_request_disconnected`). + let open_fut = self.io_client.open_http_stream( + &method_str, + &url_str, + &header_list, + body_str, + Arc::clone(&self.budget), + ); + let disconnect = self.any_client_disconnected(self.downstream_disconnect_senders()); + let opened = { + tokio::pin!(open_fut); + tokio::pin!(disconnect); + tokio::select! { + r = &mut open_fut => r, + _ = &mut disconnect => Err(HttpClientError::Disconnected), + } + }; + match opened { Ok((status, response_headers, handle_id)) => { // Track the outbound handle as handler-owned so it is // dropped (cancelling the upstream) if the handler ends @@ -7293,11 +7366,12 @@ impl Interpreter { let handle_id = self .resolve_stream_handle(source, &env, *line, *column) .await?; - // Race the upstream read against a downstream client disconnect so - // a blocked proxy read is cancelled promptly when the browser goes - // away (see `downstream_disconnect_senders`). - let disconnect = - Self::any_downstream_disconnected(self.downstream_disconnect_senders()); + // Race the upstream read against a client disconnect (either an + // open downstream response stream, or — if the handler has not + // called `start streaming response` yet — the pending request's + // oneshot) so a blocked proxy read is cancelled promptly when the + // browser goes away. + let disconnect = self.any_client_disconnected(self.downstream_disconnect_senders()); let read = self .io_client .next_chunk(&handle_id, Arc::clone(&self.budget)); From 879fc3624276ce1a18d98f03efecf53b06a4298a Mon Sep 17 00:00:00 2001 From: Claude Date: Fri, 24 Jul 2026 10:26:15 +0000 Subject: [PATCH 068/132] ci: free runner disk in the heavy build jobs to avoid linker Bus errors The clippy-and-test and integration-tests jobs build the debuginfo-heavy release tree plus the full/integration test binaries, which can exhaust a GitHub-hosted runner's disk mid-link (SIGBUS from ld). Reclaim ~20 GB of unused preinstalled SDKs on Linux before building; Windows is unaffected. Co-Authored-By: Claude Opus 4.8 Claude-Session: https://claude.ai/code/session_011LxXAm5E4Vy39kRLr8kRnX --- .github/workflows/ci.yml | 27 +++++++++++++++++++++++++++ 1 file changed, 27 insertions(+) diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml index f3a7ee9c..9e3e883f 100644 --- a/.github/workflows/ci.yml +++ b/.github/workflows/ci.yml @@ -66,6 +66,20 @@ jobs: with: components: rustfmt, clippy + # Reclaim runner disk before building. This job compiles the workspace + # several times (debug + two release builds) plus the whole test suite, and + # the release profile keeps full debuginfo (`debug = true`), so the target + # tree is large; a full GitHub-hosted runner can otherwise exhaust its disk + # mid-link (a linker `Bus error`/SIGBUS). Removing preinstalled SDKs we do + # not use frees ~20 GB with no third-party action. + - name: Free disk space (Linux) + if: runner.os == 'Linux' + run: | + sudo rm -rf /usr/share/dotnet /usr/local/lib/android /opt/ghc \ + /opt/hostedtoolcache/CodeQL /usr/local/share/boost /usr/local/graalvm || true + sudo docker image prune --all --force > /dev/null 2>&1 || true + df -h / + # Cache Cargo registry and target directory for faster builds - name: Cache Cargo registry and target directory uses: Swatinem/rust-cache@v2 @@ -135,6 +149,19 @@ jobs: - uses: actions/checkout@v4 - uses: dtolnay/rust-toolchain@stable + # Reclaim runner disk before building: this job builds the (debuginfo-heavy) + # release tree AND every integration test binary (`cargo test --test '*'`), + # which together can exhaust a full runner's disk mid-link (linker + # `Bus error`/SIGBUS). Freeing unused preinstalled SDKs gives ~20 GB of + # headroom. Linux only — the Windows runner is not affected. + - name: Free disk space (Linux) + if: runner.os == 'Linux' + run: | + sudo rm -rf /usr/share/dotnet /usr/local/lib/android /opt/ghc \ + /opt/hostedtoolcache/CodeQL /usr/local/share/boost /usr/local/graalvm || true + sudo docker image prune --all --force > /dev/null 2>&1 || true + df -h / + # Cache Cargo registry and target directory for faster builds - name: Cache Cargo registry and target directory uses: Swatinem/rust-cache@v2 From 87ab64cb5f95cfdafd9dbade654157e9554c86f7 Mon Sep 17 00:00:00 2001 From: Claude Date: Fri, 24 Jul 2026 10:27:52 +0000 Subject: [PATCH 069/132] docs: record re-review P1 lifecycle fixes (dev diary + design status) Co-Authored-By: Claude Opus 4.8 Claude-Session: https://claude.ai/code/session_011LxXAm5E4Vy39kRLr8kRnX --- ...26-07-24-streaming-rereview-p1-blockers.md | 110 ++++++++++++++++++ Docs/development/response-streaming-design.md | 18 ++- 2 files changed, 126 insertions(+), 2 deletions(-) create mode 100644 Dev diary/2026-07-24-streaming-rereview-p1-blockers.md diff --git a/Dev diary/2026-07-24-streaming-rereview-p1-blockers.md b/Dev diary/2026-07-24-streaming-rereview-p1-blockers.md new file mode 100644 index 00000000..d400d3fc --- /dev/null +++ b/Dev diary/2026-07-24-streaming-rereview-p1-blockers.md @@ -0,0 +1,110 @@ +# Dev Diary — 2026-07-24: streaming re-review P1 blockers + +The maintainer's re-review of the streaming/concurrency PR raised a set of P1 +merge blockers on the runtime lifecycle. Each was closed with a **real-boundary** +(real-socket / real-binary) regression under the R3 profile, committed **Red +first** (a failing test-only commit that is an ancestor of the fix commit) per the +Testing Policy. + +## #2 — a client disconnect is a normal cancellation, not a handler failure + +The disconnect branch of a blocked upstream read returned a generic budget error +that the concurrent `main loop` fed into its single global consecutive-failure +breaker (backoff after every failure, break the whole loop at 256). A burst of +256 browser disconnects therefore tore the loop down — an ordinary "client hung +up," repeated, became a denial of service. + +Fix: a distinct `HttpClientError::Disconnected` mapped to a new +`ErrorKind::Cancelled` (still catchable). The concurrent loop recognizes a +`Cancelled` handler outcome as an expected cancellation — it releases the handler +(its owned streams are already closed on unwind) without touching the failure +counter or backing off. Internal budget-cancellation keeps its `ResourceLimit` +kind, so only a real downstream disconnect is exempt. +*Test:* `concurrent_disconnect_burst_test` — 270 disconnects, then `/ping` is +still served (was refused; ~13 s → ~0.9 s). + +## #3 — `outbound_stream_max_seconds` is a TRUE absolute lifetime + +`next_line`/`next_chunk` served locally-buffered bytes before consulting the +handle's absolute deadline (only `stream_pull` checked it), so a proxy that pulled +a multi-line chunk kept draining the buffer past the stream's absolute lifetime. +Fix: `check_stream_deadline` runs before serving buffered bytes; on expiry the +handle is dropped (cancelling the upstream). An empty-buffer read already expired +via `stream_pull`'s identical check. +*Test:* `outbound_stream_absolute_lifetime_test`. + +## #1 — per-request cancellation valid BEFORE and after the head + +The disconnect signal came only from an open downstream response stream, which +does not exist during the upstream HEAD phase (`open url ... and stream response`, +before `start streaming response`). A browser that disconnected while the upstream +withheld its head was not noticed until the head timeout. Fix: a second disconnect +signal from the request's oneshot — the transport drops the receiver when the +client goes away, so the parked sender reports `is_closed()` +(`any_pending_request_disconnected`, polled). `any_client_disconnected` races both +signals; the head open **and** both body reads select against it, so a disconnect +cancels the handler in either phase (dropping the head future aborts the upstream +connection). Empirically hyper drops the pending route future's receiver on a +pre-head disconnect, so the poll observes it. +*Test:* `outbound_stream_head_disconnect_test` — upstream withholds its head, the +client disconnects, the upstream closes promptly, and an unrelated `/ping` is +still served. + +## #4 — a dropped `interpret()` future still closes outbound handles + +Handler-exit / program-cleanup sites close outbound handles on normal exits, but a +dropped/cancelled `interpret()` future runs none of them, leaking a parked +(opened, not mid-read) upstream until the interpreter itself is dropped. Fix: an +RAII `OutboundStreamCleanup` guard held for the whole `interpret_inner` body, +sharing `open_http_streams` and the `IoClient` via `Rc`, so its `Drop` closes the +tracked handles even as the future unwinds and the interpreter stays alive. On a +normal run the exit sites drain the list first, so the guard is a no-op. +*Test:* `dropped_interpret_cleanup_test`. + +## #5 — a bracket index composes after a `.property` / `.method` access + +`store ct as upstream.headers["content-type"]` mis-parsed into two statements +(`store ct as upstream.headers` + a stray `["content-type"]` list literal), +silently dropping the lookup. The identifier property-access / method-call fast +paths returned before the postfix loop could consume the `[...]`. Fix: route both +through `parse_trailing_bracket_index`, folding any chained `[...]` onto the base +(`grid.rows[0][1]`); the shared postfix loop and the static-member `.` arm are +untouched. +*Tests:* `property_index_access_test` (AST + runtime), and the strengthened dot +test in `stream_handle_type_test` (previously a false green — type-checking alone +passed on the split). + +## #6 — drop a span-mismatched classic-write fallback + +The ambiguous `write line|chunk ... to ` form kept the classic +file-write fallback whenever it merely parsed, even if it consumed a **different** +span than the stream reading — so `write line min with a: 1 and b: 2 to ` +retained a partial `line min with a` fallback (the classic reading stops at the +`:` the builtin-call stream reading consumes as a named arg), corrupting the file +write. Fix: keep the fallback only when it consumed exactly to the stream +reading's end checkpoint; otherwise there is no valid classic interpretation and +the non-stream target is a clean error. +*Test:* `write_line_backcompat_test` (two new cases; the matching-span back-compat +cases still pass). + +## Test infrastructure / CI + +- Server integration tests now bind an OS-assigned free port + (`tests/common/free_tcp_port`) instead of a hardcoded constant, removing a + parallel-run flakiness class (review feedback). +- The heavy CI build jobs free ~20 GB of unused preinstalled SDKs on Linux before + building; the debuginfo-heavy release tree plus every integration test binary + was exhausting a runner's disk mid-link (linker `Bus error`/SIGBUS). + +## Risk class & residual risk + +- **R3** (concurrency / cancellation / lifecycle / streaming). Real-boundary + tests, negative assertions (the connection actually closes / the read actually + fails / a burst does not tear the loop down), Red evidence for each fix. +- The pre-head disconnect signal is **polled** (20 ms) because the request's + oneshot sender lives behind an `Arc>>` shared with the + transport, not an awaitable primitive; the downstream-response-stream signal + stays event-driven. A fully idle handler that opens an outbound stream and never + reads it is still reclaimed at handler exit rather than by a mid-idle timer — the + single-threaded, `!Send`-stream model has no wake point to close it earlier; + this is noted rather than claimed as instantaneous. diff --git a/Docs/development/response-streaming-design.md b/Docs/development/response-streaming-design.md index 107a5cc8..afa4039d 100644 --- a/Docs/development/response-streaming-design.md +++ b/Docs/development/response-streaming-design.md @@ -141,12 +141,26 @@ close out and the caller's configured timeout (which already carries the stream's idle + absolute bound), and the absolute clock starts at request initiation. - Backpressure: bounded `mpsc` — a slow browser slows the handler's `write`. -- Disconnect → upstream cancel: the `write` error path AND a proactive - `Sender::closed()` `select!` against a blocked upstream read (above). +- Disconnect → upstream cancel, in BOTH phases: a blocked upstream operation is + `select!`ed against `any_client_disconnected`, which fires on EITHER an open + downstream response stream's `Sender::closed()` (post-`start streaming + response`, event-driven) OR the request's oneshot receiver being dropped + (pre-`start streaming response` — the head phase — polled via + `is_closed()`). So a browser disconnect cancels the handler whether it is + blocked opening the upstream head or reading its body, dropping the upstream. +- Disconnect is a normal cancellation, not a handler failure: it unwinds with + `ErrorKind::Cancelled`, which the concurrent `main loop` treats as an expected + outcome (it does NOT feed the structural consecutive-failure breaker), so a + burst of disconnects cannot tear the loop down. +- Absolute lifetime (`outbound_stream_max_seconds`) is enforced before EVERY + read return — including reads served from locally-buffered bytes — not only on + a network read, so a buffered drain cannot outlive the stream's absolute cap. - Outbound close-on-exit (shipped): outbound `httpstream*` handles are also handler-owned — tracked in `RunState.open_http_streams` (swapped per poll) and dropped from `IoClient.stream_handles` when the handler ends on any path, cancelling the in-flight upstream request so an abandoned proxy read never leaks. + A cancelled/dropped `interpret()` future (an embedder cancels the run) also + closes them via an RAII guard, rather than leaking until interpreter teardown. - Close-on-exit (shipped): each handler tracks the `respstream*` ids it opened in its per-handler run-state (`open_response_streams`, part of the `RunState` swapped in/out per poll under `main loop concurrently:`). When the handler ends From a5518cc40c81227961e88c6f53571fdc3176429f Mon Sep 17 00:00:00 2001 From: Claude Date: Fri, 24 Jul 2026 10:32:20 +0000 Subject: [PATCH 070/132] test(P2): failing coverage for undefined in desugared write continuation The shared continuation of a desugared (operator) ambiguous write value is identical under both readings, so an undefined variable there must be flagged even though the lead is target-dependent. Red: `write line value plus missing_suffix to srv` does not flag `missing_suffix` (the analyzer defers all desugared shapes). A companion asserts no false positive when the continuation is defined. Co-Authored-By: Claude Opus 4.8 Claude-Session: https://claude.ai/code/session_011LxXAm5E4Vy39kRLr8kRnX --- tests/write_line_backcompat_test.rs | 42 +++++++++++++++++++++++++++++ 1 file changed, 42 insertions(+) diff --git a/tests/write_line_backcompat_test.rs b/tests/write_line_backcompat_test.rs index 16058a97..35bc2b5d 100644 --- a/tests/write_line_backcompat_test.rs +++ b/tests/write_line_backcompat_test.rs @@ -376,6 +376,48 @@ fn test_span_mismatched_write_line_to_file_does_not_corrupt() { ); } +#[test] +fn test_ambiguous_write_line_flags_undefined_in_desugared_continuation() { + // The continuation of a DESUGARED (operator) ambiguous value is shared by both + // readings, so an undefined variable there must still be flagged even though + // the lead itself is target-dependent. `value` is defined; `missing_suffix` is + // not — and it lives in the `plus` continuation, not at the ambiguous lead. + let code = "listen on port 8080 as srv\n\ + store value as 1\n\ + write line value plus missing_suffix to srv"; + let tokens = lex_wfl_with_positions(code); + let program = Parser::new(&tokens).parse().expect("parse"); + let mut analyzer = Analyzer::new(); + let errors = analyzer + .analyze(&program) + .expect_err("`missing_suffix` in the operator continuation is undefined"); + assert!( + errors + .iter() + .any(|e| e.message.contains("missing_suffix") && e.message.contains("not defined")), + "expected an undefined-variable error naming `missing_suffix`, got: {errors:?}" + ); +} + +#[test] +fn test_ambiguous_write_line_operator_continuation_all_defined_ok() { + // A valid classic file write whose value desugars to an operator expression + // with a fully-defined continuation must NOT be rejected: the split stream + // lead `value` is undefined, but the classic reading (`line value`) and the + // shared continuation (`addend`) both resolve. + let code = "store line value as 3\n\ + store addend as 4\n\ + write line value plus addend to \"/tmp/out\""; + let tokens = lex_wfl_with_positions(code); + let program = Parser::new(&tokens).parse().expect("parse"); + let mut analyzer = Analyzer::new(); + assert!( + analyzer.analyze(&program).is_ok(), + "a valid operator-continuation classic write must not be rejected: {:?}", + analyzer.get_errors() + ); +} + #[test] fn test_ambiguous_write_line_still_flags_when_neither_candidate_defined() { // The ambiguous form defers definedness to runtime, but a genuine typo where From e11cdb6a6a9a29d54e985f2ca071cf73764f894c Mon Sep 17 00:00:00 2001 From: Claude Date: Fri, 24 Jul 2026 10:45:22 +0000 Subject: [PATCH 071/132] fix(P2): analyze the shared continuation of a desugared ambiguous write The analyzer deferred ALL non-simple ambiguous `write line|chunk` values to runtime, so an undefined variable in the operator continuation (`write line value plus missing_suffix to ...`) went unflagged. Replace the simple-lead check with analyze_ambiguous_write, a parallel walk of the stream and classic readings (parsed from the same tokens, differing only at the leftmost leaf): a subtree identical under both readings is pure continuation and analyzed normally (catching undefined names anywhere in it); otherwise the walk recurses on BOTH children so a lead a desugaring duplicated into the right operand (`is between`) is matched against the fallback's copy instead of mis-flagged, and at a differing leaf reports undefined only when NEITHER reading resolves. Call-based desugarings still defer. Turns tests/write_line_backcompat_test.rs green (desugared-continuation flag + operator-continuation no-false-positive), all prior back-compat cases intact. Co-Authored-By: Claude Opus 4.8 Claude-Session: https://claude.ai/code/session_011LxXAm5E4Vy39kRLr8kRnX --- src/analyzer/mod.rs | 140 ++++++++++++++++++++++++++++---------------- 1 file changed, 90 insertions(+), 50 deletions(-) diff --git a/src/analyzer/mod.rs b/src/analyzer/mod.rs index 76bb3252..0331443f 100644 --- a/src/analyzer/mod.rs +++ b/src/analyzer/mod.rs @@ -1671,41 +1671,20 @@ impl Analyzer { // Unambiguous form: check the stream value normally. None => self.analyze_expression(value), // Ambiguous merged form (`write line ... to `): - // the live reading — stream write of `` vs classic - // file write of the variable `line ` — depends on the - // runtime target type, and the two differ only in the leading - // operand. Analyze ONLY the shapes where that lead is - // unambiguously the single leftmost bare variable: a bare - // `Variable`, or ` with ` (a `Concatenation` - // whose left is that variable). Desugared forms place the lead - // where a generic walk cannot separate it from the shared - // continuation — `starts/ends with` makes the lead a call - // argument, `is between` duplicates it, pattern/`of`/builtin- - // `with` bury it in a call — so analyzing them would reject a - // valid classic file write (a back-compat regression). Defer - // those entirely to runtime. + // the live reading — stream write of `` vs classic file + // write of the variable `line ` — depends on the runtime + // target type, and the two readings differ ONLY at the leftmost + // leaf (the merged lead). `analyze_ambiguous_write` walks both + // readings in parallel: it analyzes every sub-expression they + // share (operator right-hand sides, concatenation tails) so a + // genuinely undefined variable in the continuation is still + // caught, and at the lead reports undefined only when NEITHER + // reading resolves. Call-based desugarings (`starts/ends with`, + // `is between`, patterns) bury the lead inside a call where the + // shapes diverge; those still defer to runtime rather than risk + // rejecting a valid classic file write. Some(fallback) => { - let stream_name = Self::stream_write_simple_lead(value); - let fallback_name = Self::stream_write_simple_lead(fallback); - if let (Some(sn), Some(fal)) = (stream_name, fallback_name) { - // Shared continuation (`with `): the right side is - // identical under both readings and unambiguous, so a - // genuinely undefined variable there is still caught. - if let Expression::Concatenation { right, .. } = value { - self.analyze_expression(right); - } - // Report the lead undefined only when NEITHER reading's - // name resolves — a real typo caught without rejecting - // either valid reading. - if !self.name_is_defined(sn) && !self.name_is_defined(fal) { - self.report_undefined_name( - format!("Variable '{fal}' is not defined"), - *line, - *column, - ); - } - } - // Any other (desugared) shape: no analysis — runtime decides. + self.analyze_ambiguous_write(value, fallback, *line, *column); } } } @@ -3558,22 +3537,83 @@ impl Analyzer { } } - /// The ambiguous leading operand name of a `write line|chunk` value, but ONLY - /// for the shapes where that lead is provably the single leftmost bare - /// variable and cleanly separable from the shared continuation: a bare - /// `Variable`, or ` with ` (a `Concatenation` whose left is - /// that variable). Returns `None` for every other shape — including the - /// desugared `starts/ends with` (call), `is between` (duplicated operand), - /// and pattern/`of`/builtin-`with` forms — so the caller skips analysis of - /// those rather than risk rejecting a valid classic file write. - fn stream_write_simple_lead(expr: &Expression) -> Option<&str> { - match expr { - Expression::Variable(name, ..) => Some(name), - Expression::Concatenation { left, .. } => match &**left { - Expression::Variable(name, ..) => Some(name), - _ => None, - }, - _ => None, + /// Analyze an ambiguous `write line|chunk` value against its classic + /// file-write fallback. Both readings are parsed from the SAME tokens and + /// differ only at the leftmost leaf (the merged lead), so walk them in + /// parallel: analyze every shared sub-expression (an operator's right-hand + /// side, a concatenation's tail) so an undefined variable in the continuation + /// is caught, and at the lead report undefined only when NEITHER reading + /// resolves (so neither valid interpretation is rejected). Diverging, + /// call-based desugarings (`starts/ends with`, `is between`, patterns) bury the + /// lead where the shapes no longer line up; those hit the catch-all arm and are + /// left to runtime rather than risk rejecting a valid classic file write. + fn analyze_ambiguous_write( + &mut self, + value: &Expression, + fallback: &Expression, + line: usize, + column: usize, + ) { + // A subtree that is IDENTICAL under both readings carries no lead + // difference (the two readings are parsed from the same tokens, so a + // genuinely shared sub-expression has the same names AND positions) — it is + // pure continuation, so analyze it normally. This catches an undefined + // variable anywhere in the shared part, including inside a call or index. + if value == fallback { + self.analyze_expression(value); + return; + } + // Otherwise the lead lies somewhere below. Recurse in PARALLEL on both + // children so a lead that a desugaring DUPLICATED into the right operand + // (e.g. `is between`) is still matched against the fallback's copy — not + // mistaken for an undefined continuation variable. + match (value, fallback) { + ( + Expression::BinaryOperation { + left: vl, + operator: vo, + right: vr, + .. + }, + Expression::BinaryOperation { + left: fl, + operator: fo, + right: fr, + .. + }, + ) if vo == fo => { + self.analyze_ambiguous_write(vl, fl, line, column); + self.analyze_ambiguous_write(vr, fr, line, column); + } + ( + Expression::Concatenation { + left: vl, + right: vr, + .. + }, + Expression::Concatenation { + left: fl, + right: fr, + .. + }, + ) => { + self.analyze_ambiguous_write(vl, fl, line, column); + self.analyze_ambiguous_write(vr, fr, line, column); + } + // Reached a differing leaf — the lead. Report only when NEITHER + // reading resolves, so neither valid interpretation is rejected. + (Expression::Variable(sn, ..), Expression::Variable(fal, ..)) => { + if !self.name_is_defined(sn) && !self.name_is_defined(fal) { + self.report_undefined_name( + format!("Variable '{fal}' is not defined"), + line, + column, + ); + } + } + // A diverging, non-decomposable shape (a call-based desugaring where the + // lead is buried): defer to runtime rather than risk a false positive. + _ => {} } } From c19e67a30e4e5cacc34b30cc1f6e353fd4fdb9e4 Mon Sep 17 00:00:00 2001 From: Claude Date: Fri, 24 Jul 2026 10:47:42 +0000 Subject: [PATCH 072/132] test(P2): failing coverage for streaming-operand type enforcement wait for next chunk|line / write line|chunk / flush only inferred their operands without enforcing a stream handle. Red: reading from / writing to / flushing a concrete number is not rejected. Companions assert the valid stream handles (and an ambiguous write's text file-path target) still pass. Co-Authored-By: Claude Opus 4.8 Claude-Session: https://claude.ai/code/session_011LxXAm5E4Vy39kRLr8kRnX --- tests/stream_handle_type_test.rs | 80 ++++++++++++++++++++++++++++++++ 1 file changed, 80 insertions(+) diff --git a/tests/stream_handle_type_test.rs b/tests/stream_handle_type_test.rs index 7bf02421..c896a133 100644 --- a/tests/stream_handle_type_test.rs +++ b/tests/stream_handle_type_test.rs @@ -126,6 +126,86 @@ fn test_stream_handle_numeric_index_is_rejected() { ); } +#[test] +fn test_wait_for_next_from_non_stream_is_rejected() { + // `wait for next chunk|line from ` requires an outbound stream handle; + // reading one from a concrete number is a static type error, not deferred. + for verb in ["chunk", "line"] { + let code = format!("store n as 5\nwait for next {verb} from n as c"); + let errors = + typecheck(&code).expect_err("reading a stream from a number must be a type error"); + assert!( + errors.contains("stream"), + "expected a stream-source type error for `wait for next {verb}`, got: {errors}" + ); + } +} + +#[test] +fn test_wait_for_next_from_http_stream_is_ok() { + // The valid form (reading from an outbound stream handle) must type-check. + let code = "open url at \"http://example.com\" and stream response as up\n\ + wait for next chunk from up as c\n\ + close up"; + assert!( + typecheck(code).is_ok(), + "reading from an outbound stream handle must type-check: {:?}", + typecheck(code).err() + ); +} + +#[test] +fn test_flush_non_stream_is_rejected() { + // `flush ` requires a server response-stream handle. + let code = "store n as 5\nflush n"; + let errors = typecheck(code).expect_err("flushing a number must be a type error"); + assert!( + errors.contains("stream"), + "expected a stream-target type error for `flush`, got: {errors}" + ); +} + +#[test] +fn test_write_line_to_non_stream_is_rejected() { + // An UNAMBIGUOUS stream write (literal value => no classic file-write fallback) + // to a concrete number cannot be a file write either, so it is a type error. + let code = "store n as 5\nwrite line \"x\" to n"; + let errors = typecheck(code).expect_err("writing to a number must be a type error"); + assert!( + errors.contains("stream"), + "expected a stream-target type error for `write line`, got: {errors}" + ); +} + +#[test] +fn test_write_and_flush_response_stream_is_ok() { + // The valid server-streaming path must type-check cleanly. + let code = "listen on port 8080 as s\n\ + wait for request comes in on s as req\n\ + start streaming response to req with status 200 and content type \"text/plain\" as out\n\ + write line \"hi\" to out\n\ + flush out\n\ + close out"; + assert!( + typecheck(code).is_ok(), + "writing/flushing a response-stream handle must type-check: {:?}", + typecheck(code).err() + ); +} + +#[test] +fn test_write_line_variable_to_file_path_is_ok() { + // The AMBIGUOUS merged form (`write line ... to `) carries a + // classic file-write fallback, so a text file-path target must NOT be rejected. + let code = "store payload as \"data\"\n\ + write line payload to \"/tmp/out.txt\""; + assert!( + typecheck(code).is_ok(), + "an ambiguous `write line to ` must accept a text file target: {:?}", + typecheck(code).err() + ); +} + #[test] fn test_close_ordinary_map_is_rejected() { // A plain user map is NOT closeable — only file/stream handles are. This is From e593378715e25c4d2e6e2b957e5b17acf875b5cf Mon Sep 17 00:00:00 2001 From: Claude Date: Fri, 24 Jul 2026 11:16:03 +0000 Subject: [PATCH 073/132] fix(clippy): collapse the ambiguous-write lead guard into the match arm rust 1.97 clippy flags collapsible_match on the nested if inside the Variable/Variable arm of analyze_ambiguous_write; move the guard onto the arm (behavior-preserving: a false guard falls through to the catch-all). Fixes the -D warnings failure in the Build/Test/Clippy CI job. Co-Authored-By: Claude Opus 4.8 Claude-Session: https://claude.ai/code/session_011LxXAm5E4Vy39kRLr8kRnX --- src/analyzer/mod.rs | 16 ++++++++-------- 1 file changed, 8 insertions(+), 8 deletions(-) diff --git a/src/analyzer/mod.rs b/src/analyzer/mod.rs index 0331443f..1863f956 100644 --- a/src/analyzer/mod.rs +++ b/src/analyzer/mod.rs @@ -3602,14 +3602,14 @@ impl Analyzer { } // Reached a differing leaf — the lead. Report only when NEITHER // reading resolves, so neither valid interpretation is rejected. - (Expression::Variable(sn, ..), Expression::Variable(fal, ..)) => { - if !self.name_is_defined(sn) && !self.name_is_defined(fal) { - self.report_undefined_name( - format!("Variable '{fal}' is not defined"), - line, - column, - ); - } + (Expression::Variable(sn, ..), Expression::Variable(fal, ..)) + if !self.name_is_defined(sn) && !self.name_is_defined(fal) => + { + self.report_undefined_name( + format!("Variable '{fal}' is not defined"), + line, + column, + ); } // A diverging, non-decomposable shape (a call-based desugaring where the // lead is buried): defer to runtime rather than risk a false positive. From daaf99eee5c38c91a4b69f0c5ee102e16b8d675b Mon Sep 17 00:00:00 2001 From: Claude Date: Fri, 24 Jul 2026 11:16:03 +0000 Subject: [PATCH 074/132] fix(P2): enforce stream-handle operand types for wait/write/flush MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit wait for next chunk|line / write line|chunk / flush only inferred their operands. Now: the source of a wait-for-next must be an HttpStream handle, and the target of write/flush a ResponseStream handle; the ambiguous merged write additionally accepts a text file-path target (its classic file-write reading). Unknown/Any/Error still pass for gradual typing, so only a concrete non-stream operand is rejected — at typecheck instead of as a runtime surprise. Turns the negative cases in tests/stream_handle_type_test.rs green; the valid handles (and the ambiguous write's file target) still pass. Co-Authored-By: Claude Opus 4.8 Claude-Session: https://claude.ai/code/session_011LxXAm5E4Vy39kRLr8kRnX --- src/typechecker/mod.rs | 99 +++++++++++++++++++++++++++++++++++++----- 1 file changed, 88 insertions(+), 11 deletions(-) diff --git a/src/typechecker/mod.rs b/src/typechecker/mod.rs index 3b1295d1..32d24ee9 100644 --- a/src/typechecker/mod.rs +++ b/src/typechecker/mod.rs @@ -953,18 +953,33 @@ impl TypeChecker { Statement::WaitForNextChunkStatement { source, variable_name, - .. + line, + column, } | Statement::WaitForNextLineStatement { source, variable_name, - .. + line, + column, } => { - // Just validate the operand is inferable; the binding may be a - // chunk/line value or `nothing` at end of stream, so leave the - // bound variable's type open (Any) to avoid false errors on the - // `check if is nothing` loop-termination pattern. - let _ = self.infer_expression_type(source); + // The source must be an outbound stream handle. Gradual types + // (Unknown/Any/Error) pass; a concrete non-stream operand is a + // static error rather than a runtime "not a stream" surprise. + let source_type = self.infer_expression_type(source); + if !self.is_http_stream_source_type(&source_type) { + self.type_error( + "`wait for next chunk|line` requires an outbound stream handle \ + (from `open url ... and stream response as ...`)" + .to_string(), + Some(Type::Custom("HttpStream".to_string())), + Some(source_type), + *line, + *column, + ); + } + // The binding may be a chunk/line value or `nothing` at end of + // stream, so leave the bound variable's type open (Any) to avoid + // false errors on the `check if is nothing` termination. if !variable_name.is_empty() && let Some(symbol) = self.analyzer.get_symbol_mut(variable_name) { @@ -1035,12 +1050,52 @@ impl TypeChecker { symbol.symbol_type = Some(Type::Custom("ResponseStream".to_string())); } } - Statement::StreamWriteStatement { value, target, .. } => { + Statement::StreamWriteStatement { + value, + target, + fallback_content, + line, + column, + .. + } => { let _ = self.infer_expression_type(value); - let _ = self.infer_expression_type(target); + let target_type = self.infer_expression_type(target); + // The target is a server response-stream handle. The AMBIGUOUS + // merged form (`write line ... to `) also has a + // classic file-write reading, so a text file-path target is valid + // there — but an unambiguous stream write (no fallback) to a + // concrete non-stream type is a static error. + let file_target_ok = + fallback_content.is_some() && matches!(target_type, Type::Text); + if !self.is_response_stream_target_type(&target_type) && !file_target_ok { + self.type_error( + "`write line|chunk` requires a response-stream handle \ + (from `start streaming response ... as ...`)" + .to_string(), + Some(Type::Custom("ResponseStream".to_string())), + Some(target_type), + *line, + *column, + ); + } } - Statement::FlushStreamStatement { target, .. } => { - let _ = self.infer_expression_type(target); + Statement::FlushStreamStatement { + target, + line, + column, + } => { + let target_type = self.infer_expression_type(target); + if !self.is_response_stream_target_type(&target_type) { + self.type_error( + "`flush` requires a response-stream handle \ + (from `start streaming response ... as ...`)" + .to_string(), + Some(Type::Custom("ResponseStream".to_string())), + Some(target_type), + *line, + *column, + ); + } } Statement::VariableDeclaration { name, @@ -4736,6 +4791,28 @@ impl TypeChecker { } } + /// The operand of `wait for next chunk|line from ` must be an outbound + /// stream handle (`stream response as ...` binds `HttpStream`). Unknown/Any/ + /// Error pass for gradual typing; a concrete non-stream type is rejected. + fn is_http_stream_source_type(&self, ty: &Type) -> bool { + match ty { + Type::Custom(name) => name == "HttpStream", + Type::Unknown | Type::Any | Type::Error => true, + _ => false, + } + } + + /// The `` of `write line|chunk` / `flush` must be a server response + /// stream handle (`start streaming response as ...` binds `ResponseStream`). + /// Unknown/Any/Error pass for gradual typing. + fn is_response_stream_target_type(&self, ty: &Type) -> bool { + match ty { + Type::Custom(name) => name == "ResponseStream", + Type::Unknown | Type::Any | Type::Error => true, + _ => false, + } + } + fn are_types_compatible(&self, target_type: &Type, source_type: &Type) -> bool { #[allow(clippy::only_used_in_recursion)] let _self = self; // Suppress the warning for self parameter From c2bbbdb4889b61df488fd8859ed84fa840054a0e Mon Sep 17 00:00:00 2001 From: Claude Date: Fri, 24 Jul 2026 11:16:03 +0000 Subject: [PATCH 075/132] test(P2): failing coverage for flush operand postfix accessors The lexer merges `flush` with the following identifier, so `flush streams["a"]` / `flush obj.out` leave the accessor tokens dangling. Red: each currently fails to parse as a single FlushStreamStatement whose operand is the IndexAccess / PropertyAccess. Co-Authored-By: Claude Opus 4.8 Claude-Session: https://claude.ai/code/session_011LxXAm5E4Vy39kRLr8kRnX --- tests/http_server_streaming_test.rs | 43 +++++++++++++++++++++++++++++ 1 file changed, 43 insertions(+) diff --git a/tests/http_server_streaming_test.rs b/tests/http_server_streaming_test.rs index a33f773b..9321e49b 100644 --- a/tests/http_server_streaming_test.rs +++ b/tests/http_server_streaming_test.rs @@ -115,6 +115,49 @@ fn test_flush_parses() { } } +#[test] +fn test_flush_with_index_operand_parses() { + // `flush streams["a"]`: the lexer merges `flush streams`; the trailing `["a"]` + // must bind to the operand (one IndexAccess), not leave dangling tokens that + // split into a bogus second statement. + let tokens = lex_wfl_with_positions(r#"flush streams["a"]"#); + let program = Parser::new(&tokens).parse().expect("parse"); + assert_eq!( + program.statements.len(), + 1, + "`flush [idx]` must be one statement, got {:#?}", + program.statements + ); + match &program.statements[0] { + Statement::FlushStreamStatement { target, .. } => assert!( + matches!(target, wfl::parser::ast::Expression::IndexAccess { .. }), + "flush operand must be an IndexAccess, got {target:#?}" + ), + other => panic!("expected FlushStreamStatement, got {other:?}"), + } +} + +#[test] +fn test_flush_with_property_operand_parses() { + // `flush obj.out`: the lexer merges `flush obj`; the trailing `.out` must bind + // to the operand (a PropertyAccess), not split off. + let tokens = lex_wfl_with_positions("flush obj.out"); + let program = Parser::new(&tokens).parse().expect("parse"); + assert_eq!( + program.statements.len(), + 1, + "`flush .prop` must be one statement, got {:#?}", + program.statements + ); + match &program.statements[0] { + Statement::FlushStreamStatement { target, .. } => assert!( + matches!(target, wfl::parser::ast::Expression::PropertyAccess { .. }), + "flush operand must be a PropertyAccess, got {target:#?}" + ), + other => panic!("expected FlushStreamStatement, got {other:?}"), + } +} + #[test] fn test_write_bare_line_variable_to_file_still_parses() { // Backward compat: `write to ` with a variable literally named From a0ef057f88d0a22833a28b9cd689e4a5fc4f99aa Mon Sep 17 00:00:00 2001 From: Claude Date: Fri, 24 Jul 2026 11:16:03 +0000 Subject: [PATCH 076/132] fix(P2): compose postfix accessors onto merged-lead operands MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Generalize the property-then-index helper into parse_trailing_postfix, which now folds BOTH bracket index (`["k"]`, `[0]`) and dotted property access (`.field`) onto a completed lead, and route `flush `'s split-off merged lead through it — so `flush streams["a"]` / `flush obj.out` parse as one statement with the right operand instead of leaving dangling tokens (review feedback). Also anchor the helper's missing-`]` end-of-input diagnostic to the `[` token's byte span rather than the start of the file (review feedback). Turns tests/http_server_streaming_test.rs flush-postfix cases green. Co-Authored-By: Claude Opus 4.8 Claude-Session: https://claude.ai/code/session_011LxXAm5E4Vy39kRLr8kRnX --- src/parser/expr/primary.rs | 122 +++++++++++++++++++++++++------------ src/parser/stmt/web.rs | 7 ++- 2 files changed, 89 insertions(+), 40 deletions(-) diff --git a/src/parser/expr/primary.rs b/src/parser/expr/primary.rs index 72ba16b3..df0131ca 100644 --- a/src/parser/expr/primary.rs +++ b/src/parser/expr/primary.rs @@ -89,53 +89,97 @@ impl<'a> PrimaryExprParser<'a> for Parser<'a> { } impl<'a> Parser<'a> { - /// After an identifier `.property` or `.method(...)` access, consume any - /// chained bracket index accesses so the index binds to the property/method - /// value. Without this, `upstream.headers["content-type"]` parsed as - /// `upstream.headers` followed by a separate `["content-type"]` list-literal - /// statement — silently dropping the lookup. Handles chains - /// (`grid.rows[0][1]`); a trailing `.member` is left for the caller (matching - /// the pre-existing primary-expression behavior). - fn parse_trailing_bracket_index( + /// Consume postfix accessors — bracket index (`["key"]`, `[0]`) and dotted + /// property access (`.field`) — that chain off a completed lead expression, so + /// they bind to the lead instead of splitting into bogus separate statements. + /// + /// This runs where the lexer has merged the lead into one identifier token and + /// left the accessors as following tokens: an identifier property access + /// (`upstream.headers["content-type"]`, which otherwise parsed as + /// `upstream.headers` + a stray `["content-type"]` list literal), and the + /// merged-command operands (`flush streams["a"]`, `flush obj.out`). Handles + /// arbitrary chains (`grid.rows[0][1]`, `obj.a.b["k"]`). + pub(crate) fn parse_trailing_postfix( &mut self, mut expr: Expression, ) -> Result { - while let Some(bracket) = self.cursor.peek() { - if bracket.token != Token::LeftBracket { - break; - } - let line = bracket.line; - let column = bracket.column; - self.bump_sync(); // Consume '[' + while let Some(tok) = self.cursor.peek() { + let (line, column) = (tok.line, tok.column); + match &tok.token { + Token::LeftBracket => { + // Anchor a "missing `]`" span to the `[` token itself, not the + // start of the file. + let (bracket_start, bracket_end) = (tok.byte_start, tok.byte_end); + self.bump_sync(); // Consume '[' - let index = self.parse_expression()?; + let index = self.parse_expression()?; - match self.cursor.peek() { - Some(closing) if closing.token == Token::RightBracket => { - self.bump_sync(); // Consume ']' - } - Some(closing) => { - return Err(ParseError::from_token( - format!("Expected ']' after index, found {:?}", closing.token), - closing, - )); + match self.cursor.peek() { + Some(closing) if closing.token == Token::RightBracket => { + self.bump_sync(); // Consume ']' + } + Some(closing) => { + return Err(ParseError::from_token( + format!("Expected ']' after index, found {:?}", closing.token), + closing, + )); + } + None => { + return Err(ParseError::from_span( + "Expected ']' after index, found end of input".to_string(), + crate::diagnostics::Span { + start: bracket_start, + end: bracket_end, + }, + line, + column, + )); + } + } + + expr = Expression::IndexAccess { + collection: Box::new(expr), + index: Box::new(index), + line, + column, + }; } - None => { - return Err(ParseError::from_span( - "Expected ']' after index, found end of input".to_string(), - crate::diagnostics::Span { start: 0, end: 0 }, + Token::Dot => { + self.bump_sync(); // Consume '.' + let property = match self.cursor.peek() { + // Keywords that double as common property names (e.g. + // `response.status`) are accepted, matching the primary + // dispatch's property-access handling. + Some(prop) => match &prop.token { + Token::Identifier(name) => name.clone(), + Token::KeywordStatus => "status".to_string(), + _ => { + return Err(ParseError::from_token( + "Expected a property name after '.'".to_string(), + prop, + )); + } + }, + None => { + return Err(ParseError::from_span( + "Expected a property name after '.', found end of input" + .to_string(), + crate::diagnostics::Span { start: 0, end: 0 }, + line, + column, + )); + } + }; + self.bump_sync(); // Consume the property name + expr = Expression::PropertyAccess { + object: Box::new(expr), + property, line, column, - )); + }; } + _ => break, } - - expr = Expression::IndexAccess { - collection: Box::new(expr), - index: Box::new(index), - line, - column, - }; } Ok(expr) } @@ -339,7 +383,7 @@ impl<'a> Parser<'a> { line: token_line, column: token_column, }; - return self.parse_trailing_bracket_index(call); + return self.parse_trailing_postfix(call); } // Property access without method call. @@ -359,7 +403,7 @@ impl<'a> Parser<'a> { line: token_line, column: token_column, }; - return self.parse_trailing_bracket_index(access); + return self.parse_trailing_postfix(access); } else { return Err(ParseError::from_token( "Expected property name after '.'".to_string(), diff --git a/src/parser/stmt/web.rs b/src/parser/stmt/web.rs index ea553723..16522090 100644 --- a/src/parser/stmt/web.rs +++ b/src/parser/stmt/web.rs @@ -541,7 +541,12 @@ impl<'a> WebParser<'a> for Parser<'a> { let target = if rest.is_empty() { self.parse_primary_expression()? } else { - Expression::Variable(rest.to_string(), line, column) + // The lexer merged `flush` with the operand identifier, so any postfix + // accessors (`flush streams["a"]`, `flush obj.out`) are left as separate + // tokens. Compose them onto the split-off lead so the operand parses + // consistently with a normal expression instead of dangling. + let lead = Expression::Variable(rest.to_string(), line, column); + self.parse_trailing_postfix(lead)? }; Ok(Statement::FlushStreamStatement { From bf700a93e051e4f627c1f353be904d9a19765a75 Mon Sep 17 00:00:00 2001 From: Claude Date: Fri, 24 Jul 2026 11:16:45 +0000 Subject: [PATCH 077/132] docs: extend re-review dev diary with the P2 items (#7, #8) Co-Authored-By: Claude Opus 4.8 Claude-Session: https://claude.ai/code/session_011LxXAm5E4Vy39kRLr8kRnX --- ...26-07-24-streaming-rereview-p1-blockers.md | 31 +++++++++++++++++++ 1 file changed, 31 insertions(+) diff --git a/Dev diary/2026-07-24-streaming-rereview-p1-blockers.md b/Dev diary/2026-07-24-streaming-rereview-p1-blockers.md index d400d3fc..352e1e0e 100644 --- a/Dev diary/2026-07-24-streaming-rereview-p1-blockers.md +++ b/Dev diary/2026-07-24-streaming-rereview-p1-blockers.md @@ -87,6 +87,37 @@ the non-stream target is a clean error. *Test:* `write_line_backcompat_test` (two new cases; the matching-span back-compat cases still pass). +## #7 (P2) — analyze the shared continuation of a desugared ambiguous write + +The analyzer deferred ALL non-simple ambiguous `write line|chunk` values to +runtime, so an undefined variable in an operator continuation +(`write line value plus missing_suffix to ...`) went unflagged. Replaced the +simple-lead check with `analyze_ambiguous_write`, a parallel walk of the stream +and classic readings (parsed from the same tokens, differing only at the leftmost +leaf): a subtree identical under both readings is pure continuation and analyzed +normally; otherwise the walk recurses on BOTH children so a lead a desugaring +duplicated into the right operand (`is between`) is matched against the fallback's +copy instead of mis-flagged, and at a differing leaf reports undefined only when +NEITHER reading resolves. Call-based desugarings still defer. +*Tests:* `write_line_backcompat_test` (desugared-continuation flag + +operator-continuation no-false-positive), all prior back-compat cases intact. + +## #8 (P2) — streaming-operand type enforcement + flush operand postfix + +- **Operand types.** `wait for next chunk|line` now requires an `HttpStream` + source, and `write line|chunk` / `flush` a `ResponseStream` target (the + ambiguous merged write also accepts a text file-path target for its classic + reading). Unknown/Any/Error still pass for gradual typing, so only a concrete + non-stream operand is rejected — at typecheck instead of as a runtime surprise. + *Tests:* `stream_handle_type_test`. +- **`flush` operand postfix (review feedback).** The lexer merges `flush` with the + following identifier, so `flush streams["a"]` / `flush obj.out` left the accessor + tokens dangling. Generalized the property-then-index helper into + `parse_trailing_postfix` (folds both `[...]` index and `.field` property access + onto a lead) and route `flush`'s split-off lead through it. Also anchored that + helper's missing-`]` end-of-input diagnostic to the `[` token's byte span rather + than the file start (review feedback). *Tests:* `http_server_streaming_test`. + ## Test infrastructure / CI - Server integration tests now bind an OS-assigned free port From e8c971273928088aeb2042d3f7da438810072c81 Mon Sep 17 00:00:00 2001 From: Claude Date: Fri, 24 Jul 2026 11:22:28 +0000 Subject: [PATCH 078/132] fix(parser): anchor the '.' end-of-input diagnostic to the dot token parse_trailing_postfix's missing-property-name-at-EOF error used a placeholder Span { 0, 0 }, pointing diagnostics for inputs like `flush obj.` / `a.b.` at the start of the file. Anchor it to the '.' token's byte span, matching the '[' handling above (review feedback). Co-Authored-By: Claude Opus 4.8 Claude-Session: https://claude.ai/code/session_011LxXAm5E4Vy39kRLr8kRnX --- src/parser/expr/primary.rs | 8 +++++++- 1 file changed, 7 insertions(+), 1 deletion(-) diff --git a/src/parser/expr/primary.rs b/src/parser/expr/primary.rs index df0131ca..9af12dfa 100644 --- a/src/parser/expr/primary.rs +++ b/src/parser/expr/primary.rs @@ -145,6 +145,9 @@ impl<'a> Parser<'a> { }; } Token::Dot => { + // Anchor an end-of-input error to the `.` token, not the start + // of the file. + let (dot_start, dot_end) = (tok.byte_start, tok.byte_end); self.bump_sync(); // Consume '.' let property = match self.cursor.peek() { // Keywords that double as common property names (e.g. @@ -164,7 +167,10 @@ impl<'a> Parser<'a> { return Err(ParseError::from_span( "Expected a property name after '.', found end of input" .to_string(), - crate::diagnostics::Span { start: 0, end: 0 }, + crate::diagnostics::Span { + start: dot_start, + end: dot_end, + }, line, column, )); From 0d6e8e155a853aa47c206f006ee2aa0fcdccc896 Mon Sep 17 00:00:00 2001 From: Claude Date: Fri, 24 Jul 2026 13:00:11 +0000 Subject: [PATCH 079/132] test(P1): failing real-socket + unit coverage for the re-review blockers Red evidence for the maintainer's re-review of e8c9712. Against that head these fail for the intended reason (verified by running them with the fixes stashed): - wait_line_pre_response_disconnect: a `wait for next line` blocked pre-response is not cancelled by a client disconnect (hangs to the read timeout). - concurrent_prehead_prune_race: a sibling `wait for request` prune strands a pre-head handler's disconnect (hangs to the idle timeout). - concurrent_disconnect_paths_burst: a burst of disconnects at the buffered `respond` / streaming `write` paths trips the structural breaker and the concurrent loop stops serving /ping. - outbound_stream_open_expiry: an opened-but-unread outbound stream outlives `outbound_stream_max_seconds` (only closed at program end). - dropped_interpret_server_cleanup: a dropped `interpret()` leaves the server response stream open on the still-alive interpreter (client body hangs). - response_stream_backpressure: a backpressured write to a non-reading client pins the handler forever; plus incremental head/chunk visibility coverage. - ambiguous_write_branch_typecheck / _analyzer: the wrong ambiguous-write branch is validated; call/pattern continuations skip analysis. - write_web_postfix: postfix accessors on write/headers/content-type operands dangle. flush_action_backcompat: `flush cache` hijacks a `flush cache` action. Co-Authored-By: Claude Opus 4.8 Claude-Session: https://claude.ai/code/session_011LxXAm5E4Vy39kRLr8kRnX --- tests/ambiguous_write_analyzer_test.rs | 53 +++++ .../ambiguous_write_branch_typecheck_test.rs | 90 ++++++++ .../concurrent_disconnect_paths_burst_test.rs | 211 ++++++++++++++++++ tests/concurrent_prehead_prune_race_test.rs | 167 ++++++++++++++ .../dropped_interpret_server_cleanup_test.rs | 129 +++++++++++ tests/flush_action_backcompat_test.rs | 67 ++++++ tests/outbound_stream_open_expiry_test.rs | 100 +++++++++ tests/response_stream_backpressure_test.rs | 198 ++++++++++++++++ .../wait_line_pre_response_disconnect_test.rs | 156 +++++++++++++ tests/write_web_postfix_test.rs | 135 +++++++++++ 10 files changed, 1306 insertions(+) create mode 100644 tests/ambiguous_write_analyzer_test.rs create mode 100644 tests/ambiguous_write_branch_typecheck_test.rs create mode 100644 tests/concurrent_disconnect_paths_burst_test.rs create mode 100644 tests/concurrent_prehead_prune_race_test.rs create mode 100644 tests/dropped_interpret_server_cleanup_test.rs create mode 100644 tests/flush_action_backcompat_test.rs create mode 100644 tests/outbound_stream_open_expiry_test.rs create mode 100644 tests/response_stream_backpressure_test.rs create mode 100644 tests/wait_line_pre_response_disconnect_test.rs create mode 100644 tests/write_web_postfix_test.rs diff --git a/tests/ambiguous_write_analyzer_test.rs b/tests/ambiguous_write_analyzer_test.rs new file mode 100644 index 00000000..3e6de591 --- /dev/null +++ b/tests/ambiguous_write_analyzer_test.rs @@ -0,0 +1,53 @@ +//! Analyzer coverage for the ambiguous `write line|chunk ... to ` shared +//! continuation across call/pattern shapes (maintainer re-review, P1). +//! +//! The two readings of an ambiguous write differ only at the leftmost leaf; every +//! other sub-expression is shared continuation. When the continuation desugars to +//! a call/pattern shape (`starts with`, `contains pattern`, ...), the analyzer must +//! still walk the shared operands so an undefined name there is reported at analysis +//! time instead of surfacing only at runtime — without falsely rejecting either +//! valid reading of the lead. + +use wfl::analyzer::Analyzer; +use wfl::lexer::lex_wfl_with_positions; +use wfl::parser::Parser; + +fn analyze(code: &str) -> Result<(), String> { + let tokens = lex_wfl_with_positions(code); + let program = Parser::new(&tokens).parse().expect("parse"); + Analyzer::new() + .analyze(&program) + .map(|_| ()) + .map_err(|e| format!("{e:?}")) +} + +#[test] +fn undefined_name_in_a_starts_with_continuation_is_reported() { + // Both leads are defined (`greeting` for the stream reading, `line greeting` for + // the classic file reading), so the ONLY undefined name is the shared + // `starts with` operand. It must be reported rather than deferred to runtime. + let code = "store greeting as \"hello\"\n\ + store line greeting as \"world\"\n\ + write line greeting starts with missing_operand to \"/tmp/wfl_analyzer_out\""; + let err = analyze(code) + .expect_err("an undefined name in the shared `starts with` continuation must be reported"); + assert!( + err.contains("missing_operand"), + "expected the undefined shared-continuation name to be reported, got: {err}" + ); +} + +#[test] +fn defined_name_in_a_starts_with_continuation_is_not_a_false_positive() { + // Same shape, but the shared operand is defined: neither reading is broken, so + // analysis must pass (the parallel walk must not over-report). + let code = "store greeting as \"hello\"\n\ + store line greeting as \"world\"\n\ + store suffix as \"lo\"\n\ + write line greeting starts with suffix to \"/tmp/wfl_analyzer_out\""; + assert!( + analyze(code).is_ok(), + "a fully-defined ambiguous write must analyze cleanly: {:?}", + analyze(code).err() + ); +} diff --git a/tests/ambiguous_write_branch_typecheck_test.rs b/tests/ambiguous_write_branch_typecheck_test.rs new file mode 100644 index 00000000..8c39b7d4 --- /dev/null +++ b/tests/ambiguous_write_branch_typecheck_test.rs @@ -0,0 +1,90 @@ +//! Backward-compatibility coverage for the ambiguous `write line|chunk ... to +//! ` type check (maintainer re-review, P1). +//! +//! The statement has two readings parsed from the same tokens: a STREAM write of +//! `value` (when the target is a response-stream handle) and a classic FILE write +//! of `fallback_content` (when the target is anything else). The runtime picks by +//! the target's runtime type. The type checker must check the reading the runtime +//! actually takes — checking the stream `value` unconditionally rejected a valid +//! pre-existing file write on a branch that never runs (and the reverse let a +//! broken file write pass because only the stream reading was checked). + +use wfl::lexer::lex_wfl_with_positions; +use wfl::parser::Parser; +use wfl::typechecker::TypeChecker; + +fn typecheck(code: &str) -> Result<(), String> { + let tokens = lex_wfl_with_positions(code); + let program = Parser::new(&tokens).parse().expect("parse"); + TypeChecker::new() + .check_types(&program) + .map_err(|e| format!("{e:?}")) +} + +#[test] +fn text_target_valid_file_write_is_not_rejected_on_the_stream_branch() { + // Target is a concrete text path, so the runtime takes the FILE reading: + // `line value minus n` = 10 - 1 (Number - Number), which is valid. The stream + // reading `value minus n` would be Text - Number, but the runtime never + // evaluates it here — so this MUST type-check. (Before the fix it was rejected + // on the never-run stream branch.) + let code = "store value as \"wrong stream type\"\n\ + store line value as 10\n\ + store n as 1\n\ + write line value minus n to \"/tmp/wfl_branch_out\""; + assert!( + typecheck(code).is_ok(), + "a valid classic file write must not be rejected on the unused stream reading: {:?}", + typecheck(code).err() + ); +} + +#[test] +fn text_target_broken_file_write_is_caught() { + // Reverse the types: `line value` is Text, so the FILE reading + // `line value minus n` = Text - Number is ill-typed. The runtime takes the file + // reading (target is a concrete text path), so this MUST be a static error. + // (Before the fix only the stream reading `value minus n` = Number - Number was + // checked, so this wrongly passed and failed only at runtime.) + let code = "store value as 10\n\ + store line value as \"text\"\n\ + store n as 1\n\ + write line value minus n to \"/tmp/wfl_branch_out\""; + assert!( + typecheck(code).is_err(), + "a file write whose content is Text minus Number must be a static error, \ + not deferred to runtime" + ); +} + +#[test] +fn concrete_non_streamable_payload_to_a_stream_is_rejected() { + // An unambiguous stream write (the target is a real response-stream handle) of + // a concrete List payload must be a static error — the runtime only accepts + // text/binary/number/boolean, so a Map/List/Nothing reaching `write` fails at + // runtime otherwise. + let code = "listen on port 8080 as s\n\ + wait for request comes in on s as req\n\ + start streaming response to req with status 200 and content type \"text/plain\" as out\n\ + store items as [1 and 2 and 3]\n\ + write line items to out"; + assert!( + typecheck(code).is_err(), + "writing a concrete List to a response stream must be a static type error" + ); +} + +#[test] +fn text_and_binary_payloads_to_a_stream_still_typecheck() { + // The valid stream payloads must keep passing. + let code = "listen on port 8080 as s\n\ + wait for request comes in on s as req\n\ + start streaming response to req with status 200 and content type \"text/plain\" as out\n\ + write line \"hello\" to out\n\ + write chunk 42 to out"; + assert!( + typecheck(code).is_ok(), + "text/number stream payloads must type-check: {:?}", + typecheck(code).err() + ); +} diff --git a/tests/concurrent_disconnect_paths_burst_test.rs b/tests/concurrent_disconnect_paths_burst_test.rs new file mode 100644 index 00000000..baee3e2b --- /dev/null +++ b/tests/concurrent_disconnect_paths_burst_test.rs @@ -0,0 +1,211 @@ +//! Real-socket regression (maintainer re-review, P1): EVERY transport-confirmed +//! client disconnect must be classified as a cancellation, not a handler failure — +//! including the buffered `respond` send and the streaming-response head/`write` +//! paths, not only a cancelled upstream chunk read. +//! +//! The concurrent loop breaks after `MAX_CONSECUTIVE_FAILURES` (256) consecutive +//! failed handlers. If a disconnect at these paths returns a General runtime error +//! (as before), a burst of >256 disconnects trips that breaker and the server stops +//! serving — turning "the client hung up" into a denial of service. These bursts +//! disconnect after dequeue (before the buffered reply) and after the streaming head +//! (before/at the first write); an unrelated `/ping` must still be served afterward. + +use std::sync::Arc; +use std::sync::atomic::{AtomicUsize, Ordering}; +use std::time::Duration; +use tokio::io::{AsyncReadExt, AsyncWriteExt}; +use tokio::sync::Semaphore; +use wfl::Interpreter; +use wfl::lexer::lex_wfl_with_positions; +use wfl::parser::Parser; + +mod common; + +/// More than the breaker threshold (256), with NO successful request in between. +const DISCONNECT_BURST: usize = 270; +const CLIENT_CONCURRENCY: usize = 40; + +fn start_proxy_server(code: String) -> std::thread::JoinHandle<()> { + std::thread::spawn(move || { + let rt = tokio::runtime::Runtime::new().expect("runtime"); + rt.block_on(async { + let tokens = lex_wfl_with_positions(&code); + let ast = Parser::new(&tokens).parse().expect("parse"); + let mut interp = Interpreter::new(); + if let Err(errors) = interp.interpret(&ast).await { + panic!("server interpreter failed: {errors:?}"); + } + }); + }) +} + +async fn wait_for_server(port: u16) { + let addr = format!("127.0.0.1:{port}"); + for _ in 0..300 { + if tokio::net::TcpStream::connect(&addr).await.is_ok() { + return; + } + tokio::time::sleep(Duration::from_millis(50)).await; + } + panic!("server on {addr} did not become ready"); +} + +/// Connect, send the request so the server enqueues and dequeues it, briefly hold so +/// the handler is inside its pre-reply work, then disconnect. `read_head` waits for +/// the streaming response head first (so the disconnect lands after the head, at the +/// write path) when the route streams. +async fn fire_disconnect(port: u16, path: &str, read_head: bool) { + let Ok(mut sock) = tokio::net::TcpStream::connect(("127.0.0.1", port)).await else { + return; + }; + let req = format!("GET {path} HTTP/1.1\r\nHost: 127.0.0.1\r\n\r\n"); + if sock.write_all(req.as_bytes()).await.is_err() { + return; + } + let _ = sock.flush().await; + if read_head { + let mut acc = Vec::new(); + let mut tmp = [0u8; 256]; + loop { + match tokio::time::timeout(Duration::from_secs(5), sock.read(&mut tmp)).await { + Ok(Ok(0)) | Err(_) | Ok(Err(_)) => break, + Ok(Ok(n)) => { + acc.extend_from_slice(&tmp[..n]); + if acc.windows(4).any(|w| w == b"\r\n\r\n") { + break; + } + } + } + } + } else { + // Give the server time to enqueue + dequeue the request and enter the + // handler's pre-reply wait, so the disconnect lands before `respond`. + tokio::time::sleep(Duration::from_millis(200)).await; + } + // Drop `sock` -> disconnect. +} + +async fn fire_burst(port: u16, path: &'static str, read_head: bool) { + let sem = Arc::new(Semaphore::new(CLIENT_CONCURRENCY)); + let fired = Arc::new(AtomicUsize::new(0)); + let mut tasks = Vec::with_capacity(DISCONNECT_BURST); + for _ in 0..DISCONNECT_BURST { + let sem = Arc::clone(&sem); + let fired = Arc::clone(&fired); + tasks.push(tokio::spawn(async move { + let _permit = sem.acquire().await.expect("semaphore"); + fire_disconnect(port, path, read_head).await; + fired.fetch_add(1, Ordering::Relaxed); + })); + } + for t in tasks { + let _ = t.await; + } + // Grace so all >256 handlers finish failing (Cancelled, under the fix) before we + // send the first successful request — guaranteeing the failures are consecutive. + tokio::time::sleep(Duration::from_secs(3)).await; +} + +async fn assert_ping_survives(port: u16, context: &str) { + let ping = tokio::time::timeout( + Duration::from_secs(10), + reqwest::Client::new() + .get(format!("http://127.0.0.1:{port}/ping")) + .send(), + ) + .await + .unwrap_or_else(|_| panic!("`/ping` timed out after the {context} burst — loop torn down")) + .expect("`/ping` request failed"); + assert_eq!( + ping.status().as_u16(), + 200, + "`/ping` should be served after the {context} burst" + ); + assert_eq!(ping.text().await.unwrap(), "pong"); +} + +async fn shutdown(port: u16, server: std::thread::JoinHandle<()>) { + let _ = reqwest::Client::new() + .get(format!("http://127.0.0.1:{port}/shutdown")) + .send() + .await; + match tokio::task::spawn_blocking(move || server.join()).await { + Ok(Ok(())) => {} + Ok(Err(panic)) => std::panic::resume_unwind(panic), + Err(e) => panic!("server join task failed: {e}"), + } +} + +#[tokio::test] +async fn test_disconnect_before_buffered_respond_does_not_kill_the_loop() { + let port = common::free_tcp_port(); + // `/slow` waits, then responds — the client disconnects during the wait, so the + // buffered `respond` send fails. That must be a cancellation, not a failure. + let code = format!( + r#" + listen on port {port} as srv + main loop concurrently: + wait for request comes in on srv as req with timeout 30000 + store p as req["path"] + check if p is equal to "/ping": + respond to req with "pong" + otherwise: + check if p is equal to "/shutdown": + respond to req with "bye" + close server srv + break + otherwise: + wait for 500 milliseconds + respond to req with "late" + end check + end check + end loop + "# + ); + let server = start_proxy_server(code); + wait_for_server(port).await; + fire_burst(port, "/slow", false).await; + assert_ping_survives(port, "buffered-respond disconnect").await; + shutdown(port, server).await; +} + +#[tokio::test] +async fn test_disconnect_before_stream_write_does_not_kill_the_loop() { + let port = common::free_tcp_port(); + // `/stream` sends the head, waits (the client reads the head then disconnects), + // then writes — the write send fails. That must be a cancellation, not a failure. + let code = format!( + r#" + listen on port {port} as srv + main loop concurrently: + wait for request comes in on srv as req with timeout 30000 + store p as req["path"] + check if p is equal to "/ping": + respond to req with "pong" + otherwise: + check if p is equal to "/shutdown": + respond to req with "bye" + close server srv + break + otherwise: + start streaming response to req with status 200 and content type "text/plain" as out + wait for 300 milliseconds + store payload as "0123456789" + count from 1 to 9: + store payload as payload with payload + end count + count from 1 to 400: + write chunk payload to out + end count + close out + end check + end check + end loop + "# + ); + let server = start_proxy_server(code); + wait_for_server(port).await; + fire_burst(port, "/stream", true).await; + assert_ping_survives(port, "stream-write disconnect").await; + shutdown(port, server).await; +} diff --git a/tests/concurrent_prehead_prune_race_test.rs b/tests/concurrent_prehead_prune_race_test.rs new file mode 100644 index 00000000..7d99cb62 --- /dev/null +++ b/tests/concurrent_prehead_prune_race_test.rs @@ -0,0 +1,167 @@ +//! Real-socket regression (maintainer re-review, P1): a sibling handler's +//! `wait for request` global prune must NOT erase a parked handler's pre-head +//! cancellation signal. +//! +//! Handler A blocks opening a header-stalled upstream BEFORE `start streaming +//! response`, so its only disconnect signal is its pending request's oneshot. When +//! A's client disconnects, A's pending entry becomes closed — but any later +//! `wait for request` prunes ALL closed entries. If a sibling prunes A's entry +//! before A's ~20ms poll notices, A's owned id is then simply absent from the map; +//! treating "absent" as "still connected" left A parked until its read timeout. +//! +//! This drives the race: a continuous stream of pruning `/kick` requests runs while +//! A's client disconnects, so a prune reliably removes A's closed entry in the poll +//! gap. With the fix (absent owned id == disconnected), A is cancelled promptly and +//! the upstream closes well before the idle timeout; without it, A hangs to timeout. + +use std::sync::Arc; +use std::sync::atomic::{AtomicBool, Ordering}; +use std::time::{Duration, Instant}; +use tokio::io::{AsyncReadExt, AsyncWriteExt}; +use wfl::Interpreter; +use wfl::config::WflConfig; +use wfl::lexer::lex_wfl_with_positions; +use wfl::parser::Parser; + +mod common; + +/// Upstream: accept ONE connection, read the request, then WITHHOLD the response +/// head. Signal when the proxy drops the connection (peer close => read 0/Err). +async fn spawn_header_withholding_upstream() -> (u16, tokio::sync::oneshot::Receiver<()>) { + let listener = tokio::net::TcpListener::bind("127.0.0.1:0") + .await + .expect("bind mock upstream"); + let port = listener.local_addr().unwrap().port(); + let (tx, rx) = tokio::sync::oneshot::channel(); + tokio::spawn(async move { + if let Ok((mut sock, _)) = listener.accept().await { + let mut buf = [0u8; 1024]; + let _ = sock.read(&mut buf).await; // request head + loop { + match sock.read(&mut buf).await { + Ok(0) | Err(_) => { + let _ = tx.send(()); + return; + } + Ok(_) => {} + } + } + } + }); + (port, rx) +} + +async fn wait_for_server(port: u16) { + let addr = format!("127.0.0.1:{port}"); + for _ in 0..300 { + if tokio::net::TcpStream::connect(&addr).await.is_ok() { + return; + } + tokio::time::sleep(Duration::from_millis(50)).await; + } + panic!("server on {addr} did not become ready"); +} + +#[tokio::test] +async fn test_sibling_prune_does_not_strand_a_pre_head_disconnect() { + let (upstream_port, mut upstream_closed) = spawn_header_withholding_upstream().await; + let proxy_port = common::free_tcp_port(); + + let code = format!( + r#" + listen on port {proxy_port} as srv + main loop concurrently: + wait for request comes in on srv as req with timeout 30000 + store p as req["path"] + check if p is equal to "/kick": + respond to req with "ok" + otherwise: + check if p is equal to "/shutdown": + respond to req with "bye" + close server srv + break + otherwise: + open url at "http://127.0.0.1:{upstream_port}/" and stream response as up + start streaming response to req with status 200 and content type "text/plain" as down + wait for next chunk from up as c + close down + end check + end check + end loop + "# + ); + + // 4s idle timeout: with the fix the stranded handler is cancelled in ~20ms once + // its entry is pruned; without it, it hangs until this timeout. + let server = std::thread::spawn(move || { + let rt = tokio::runtime::Runtime::new().expect("runtime"); + rt.block_on(async { + let tokens = lex_wfl_with_positions(&code); + let ast = Parser::new(&tokens).parse().expect("parse"); + let config = WflConfig { + timeout_seconds: 4, + ..WflConfig::default() + }; + let mut interp = Interpreter::with_config(Arc::new(config)); + let _ = interp.interpret(&ast).await; + }); + }); + wait_for_server(proxy_port).await; + + // Continuous pruning traffic: every `/kick` runs `wait for request`, which prunes + // all closed pending entries. Keep it dense so a prune lands in A's poll gap. + let kick_stop = Arc::new(AtomicBool::new(false)); + let kick_stop2 = Arc::clone(&kick_stop); + let kicker = tokio::spawn(async move { + let client = reqwest::Client::new(); + while !kick_stop2.load(Ordering::Relaxed) { + let _ = client + .get(format!("http://127.0.0.1:{proxy_port}/kick")) + .timeout(Duration::from_secs(2)) + .send() + .await; + } + }); + + // Let the pruning traffic ramp up, then connect A, let it block opening the + // header-stalled upstream, and disconnect it while the prunes are flowing. + tokio::time::sleep(Duration::from_millis(200)).await; + { + let mut sock = tokio::net::TcpStream::connect(("127.0.0.1", proxy_port)) + .await + .expect("connect A"); + sock.write_all(b"GET /proxy HTTP/1.1\r\nHost: 127.0.0.1\r\n\r\n") + .await + .expect("send A request"); + sock.flush().await.ok(); + tokio::time::sleep(Duration::from_millis(300)).await; + // `sock` drops -> A disconnects while pruning traffic flows. + } + + // A's upstream must close PROMPTLY (cancelled), not at the 4s idle timeout. + let start = Instant::now(); + tokio::time::timeout(Duration::from_secs(3), &mut upstream_closed) + .await + .expect("A's upstream was not cancelled after a sibling pruned its disconnected entry") + .expect("upstream close sender dropped"); + let elapsed = start.elapsed(); + assert!( + elapsed < Duration::from_secs(3), + "the stranded pre-head handler should be cancelled promptly once pruned, \ + not hang to the idle timeout; took {elapsed:?}" + ); + + kick_stop.store(true, Ordering::Relaxed); + let _ = kicker.await; + + let _ = reqwest::Client::new() + .get(format!("http://127.0.0.1:{proxy_port}/shutdown")) + .timeout(Duration::from_secs(2)) + .send() + .await; + match tokio::task::spawn_blocking(move || server.join()).await { + Ok(Ok(())) => {} + Ok(Err(panic)) => std::panic::resume_unwind(panic), + Err(e) => panic!("server join task failed: {e}"), + } +} diff --git a/tests/dropped_interpret_server_cleanup_test.rs b/tests/dropped_interpret_server_cleanup_test.rs new file mode 100644 index 00000000..a9574974 --- /dev/null +++ b/tests/dropped_interpret_server_cleanup_test.rs @@ -0,0 +1,129 @@ +//! Real-socket regression (maintainer re-review, P1): when the `interpret()` future +//! is DROPPED after a handler started a streaming response, the interpret-scoped +//! cleanup guard must close the server response stream (ending the client's body) +//! and 500 any unanswered request — even though the reusable `Interpreter` itself +//! stays alive. +//! +//! Previously the drop guard covered only outbound streams, so a dropped run left +//! `server_response_streams` alive on the still-alive interpreter and the client +//! hung. To prove it is the GUARD (not the interpreter's eventual drop) that closes +//! the body, the interpreter is held alive for well after the future is dropped: the +//! client body must end shortly after the drop, not when the interpreter drops. + +use std::sync::Arc; +use std::time::{Duration, Instant}; +use wfl::Interpreter; +use wfl::config::WflConfig; +use wfl::lexer::lex_wfl_with_positions; +use wfl::parser::Parser; + +mod common; + +async fn wait_for_server(port: u16) { + let addr = format!("127.0.0.1:{port}"); + for _ in 0..300 { + if tokio::net::TcpStream::connect(&addr).await.is_ok() { + return; + } + tokio::time::sleep(Duration::from_millis(50)).await; + } + panic!("server on {addr} did not become ready"); +} + +#[tokio::test] +async fn test_dropped_run_closes_server_stream_while_interpreter_stays_alive() { + let port = common::free_tcp_port(); + + // Handler: start streaming, send a chunk, flush, then park for a long time. The + // run is dropped while it is parked here, before it ever closes the stream. + let code = format!( + r#" + listen on port {port} as srv + main loop: + wait for request comes in on srv as req with timeout 60000 + start streaming response to req with status 200 and content type "text/plain" as out + write chunk "hello" to out + flush out + wait for 60000 milliseconds + end loop + "# + ); + + // The server runs in its own thread. It runs `interpret()` under a 3s timeout — + // when that elapses the FUTURE is dropped (moved into `timeout`) — then holds + // the interpreter ALIVE for 10 more seconds. So during [3s, 13s] the future is + // gone but the interpreter lives: only the drop guard can end the client body. + let server = std::thread::spawn(move || { + let rt = tokio::runtime::Builder::new_current_thread() + .enable_all() + .build() + .expect("server runtime"); + rt.block_on(async { + let tokens = lex_wfl_with_positions(&code); + let program = Parser::new(&tokens).parse().expect("parse"); + let config = WflConfig { + timeout_seconds: 60, + ..WflConfig::default() + }; + let mut interp = Interpreter::with_config(Arc::new(config)); + { + let fut = interp.interpret(&program); + let _ = tokio::time::timeout(Duration::from_secs(3), fut).await; + // `fut` is dropped here (timeout took ownership and elapsed). + } + // Interpreter deliberately kept alive well past the drop. + tokio::time::sleep(Duration::from_secs(10)).await; + drop(interp); + }); + }); + + wait_for_server(port).await; + + // Client: read the streaming body. It must deliver "hello" and then END shortly + // after the 3s drop (the guard closing the stream), NOT hang until the 13s + // interpreter drop or the 60s handler park. + let mut resp = reqwest::Client::new() + .get(format!("http://127.0.0.1:{port}/")) + .send() + .await + .expect("request send"); + assert_eq!(resp.status().as_u16(), 200); + + let start = Instant::now(); + let mut body = Vec::new(); + let mut ended = false; + loop { + match tokio::time::timeout(Duration::from_secs(8), resp.chunk()).await { + Ok(Ok(Some(bytes))) => body.extend_from_slice(&bytes), + // Clean EOF or a transport end both mean the body finished. + Ok(Ok(None)) | Ok(Err(_)) => { + ended = true; + break; + } + Err(_) => break, // outer timeout: body never ended + } + } + let elapsed = start.elapsed(); + + assert!( + String::from_utf8_lossy(&body).contains("hello"), + "the streamed chunk should have been delivered before the drop; body: {:?}", + String::from_utf8_lossy(&body) + ); + assert!( + ended, + "the client body must END after the run was dropped (the guard closing the \ + server response stream), not hang" + ); + assert!( + elapsed < Duration::from_secs(7), + "the body should end shortly after the ~3s drop (the guard), not at the ~13s \ + interpreter drop; took {elapsed:?}" + ); + + match tokio::task::spawn_blocking(move || server.join()).await { + Ok(Ok(())) => {} + Ok(Err(panic)) => std::panic::resume_unwind(panic), + Err(e) => panic!("server join task failed: {e}"), + } +} diff --git a/tests/flush_action_backcompat_test.rs b/tests/flush_action_backcompat_test.rs new file mode 100644 index 00000000..0fe23c62 --- /dev/null +++ b/tests/flush_action_backcompat_test.rs @@ -0,0 +1,67 @@ +//! Backward-compatibility regression (maintainer re-review, P1): a statement-initial +//! `flush ` must NOT hijack a pre-existing zero-argument action named +//! `flush `. +//! +//! Before `flush` became a streaming command, `flush cache` was an expression +//! statement that auto-invoked an action `flush cache`. The dispatcher now routes +//! merged `flush …` tokens to the streaming flush; it must still prefer a defined +//! action of that full name so an existing program keeps working. + +use std::fs; +use std::process::Command; +use tempfile::TempDir; + +fn run_src(src: &str) -> (String, Option) { + let dir = TempDir::new().expect("tempdir"); + let path = dir.path().join("main.wfl"); + fs::write(&path, src).unwrap(); + let output = Command::new(env!("CARGO_BIN_EXE_wfl")) + .arg(&path) + .output() + .expect("failed to execute WFL"); + let combined = format!( + "{}{}", + String::from_utf8_lossy(&output.stdout), + String::from_utf8_lossy(&output.stderr) + ); + (combined, output.status.code()) +} + +#[test] +fn flush_calls_a_matching_zero_arg_action_instead_of_flushing_a_stream() { + // `flush cache` must invoke the action `flush cache`, printing CALLED — not try + // to flush a (nonexistent) stream `cache`. + let src = "define action called flush cache:\n\ + \x20\x20\x20\x20display \"CALLED\"\n\ + end action\n\ + \n\ + flush cache\n"; + let (out, code) = run_src(src); + assert_eq!( + code, + Some(0), + "program should exit cleanly; output was:\n{out}" + ); + assert!( + out.contains("CALLED"), + "the pre-existing `flush cache` action must be called; output was:\n{out}" + ); + assert!( + !out.to_lowercase().contains("stream"), + "`flush cache` must not be reinterpreted as a stream flush; output was:\n{out}" + ); +} + +#[test] +fn flush_without_a_matching_action_still_errors_as_a_stream_flush() { + // With no action `flush cache` and no stream `cache`, `flush cache` falls + // through to the stream interpretation and errors (rather than silently + // succeeding) — proving the action fallback is a preference, not a bypass. + let src = "flush cache\n"; + let (out, code) = run_src(src); + assert_ne!( + code, + Some(0), + "a bare `flush cache` with no target must error; output:\n{out}" + ); +} diff --git a/tests/outbound_stream_open_expiry_test.rs b/tests/outbound_stream_open_expiry_test.rs new file mode 100644 index 00000000..11f36dff --- /dev/null +++ b/tests/outbound_stream_open_expiry_test.rs @@ -0,0 +1,100 @@ +//! Real-socket regression (maintainer re-review, P1): `outbound_stream_max_seconds` +//! must be a TRUE absolute lifetime enforced in real time — even when the handler +//! NEVER reads the opened stream. +//! +//! The deadline was previously consulted only on the next read, so a handler that +//! opened an outbound stream and then parked (or did other work) without reading +//! kept the upstream connection alive indefinitely past the cap — contradicting the +//! documented "can never live past this hard cap". This proves the upstream is +//! dropped at the cap with NO read performed, well before the program ends. + +use std::sync::Arc; +use std::time::{Duration, Instant}; +use tokio::io::{AsyncReadExt, AsyncWriteExt}; +use wfl::Interpreter; +use wfl::config::WflConfig; +use wfl::lexer::lex_wfl_with_positions; +use wfl::parser::Parser; + +/// Upstream: send a valid response head, then STALL (never send more, never close). +/// Signal on the returned receiver when the proxy drops the upstream connection. +async fn spawn_head_then_stall_upstream() -> (u16, tokio::sync::oneshot::Receiver<()>) { + let listener = tokio::net::TcpListener::bind("127.0.0.1:0") + .await + .expect("bind mock upstream"); + let port = listener.local_addr().unwrap().port(); + let (tx, rx) = tokio::sync::oneshot::channel(); + tokio::spawn(async move { + if let Ok((mut sock, _)) = listener.accept().await { + let mut buf = [0u8; 1024]; + let _ = sock.read(&mut buf).await; // request head + let head = "HTTP/1.1 200 OK\r\n\ + Content-Type: text/plain\r\n\ + Transfer-Encoding: chunked\r\n\r\n"; + let _ = sock.write_all(head.as_bytes()).await; + let _ = sock.flush().await; + // Stall: never send a body, never close. Detect the proxy dropping the + // upstream (its handle reaped) via a blocking read returning 0/Err. + loop { + match sock.read(&mut buf).await { + Ok(0) | Err(_) => { + let _ = tx.send(()); + return; + } + Ok(_) => {} + } + } + } + }); + (port, rx) +} + +#[tokio::test] +async fn test_opened_but_unread_stream_expires_at_the_absolute_cap() { + let (port, mut upstream_closed) = spawn_head_then_stall_upstream().await; + + // Open the stream and then just WAIT — never read a chunk/line. With a 1s + // absolute cap the upstream must be dropped ~1s in, long before the 6s wait + // (and the program-end cleanup that would otherwise mask a missing reaper). + let code = format!( + r#"open url at "http://127.0.0.1:{port}/" and stream response as s +wait for 6000 milliseconds"# + ); + + let client = std::thread::spawn(move || { + let rt = tokio::runtime::Runtime::new().expect("client runtime"); + rt.block_on(async { + let tokens = lex_wfl_with_positions(&code); + let program = Parser::new(&tokens).parse().expect("parse"); + let config = WflConfig { + timeout_seconds: 30, + outbound_stream_max_seconds: 1, + ..WflConfig::default() + }; + let mut interp = Interpreter::with_config(Arc::new(config)); + let _ = interp.interpret(&program).await; + }); + }); + + let start = Instant::now(); + tokio::time::timeout(Duration::from_secs(4), &mut upstream_closed) + .await + .expect("the opened-but-unread upstream was not dropped at the absolute cap") + .expect("upstream close sender dropped"); + let elapsed = start.elapsed(); + + // ~1s (the cap). If enforcement were still read-triggered, the upstream would + // only close at the 6s program end — so a close well before then proves the + // real-time reaper fired. + assert!( + elapsed < Duration::from_secs(3), + "the upstream should be reaped at the ~1s absolute cap, not at program end; \ + took {elapsed:?}" + ); + + match tokio::task::spawn_blocking(move || client.join()).await { + Ok(Ok(())) => {} + Ok(Err(panic)) => std::panic::resume_unwind(panic), + Err(e) => panic!("client join task failed: {e}"), + } +} diff --git a/tests/response_stream_backpressure_test.rs b/tests/response_stream_backpressure_test.rs new file mode 100644 index 00000000..7fcbcbe1 --- /dev/null +++ b/tests/response_stream_backpressure_test.rs @@ -0,0 +1,198 @@ +//! Real-socket regressions (maintainer re-review, P1): +//! +//! 1. A backpressured response-stream `write` must be BOUNDED: once the 64-slot +//! channel fills, a client that stays connected but stops reading would pin the +//! handler forever (`main loop` is deadline-exempt). It must instead time out at +//! `web_server_response_timeout_seconds` and error, releasing the handler. +//! +//! 2. Streaming must actually STREAM: the head and an early chunk must be visible on +//! the wire BEFORE the body completes/closes — not buffered until the end. + +use std::sync::Arc; +use std::time::{Duration, Instant}; +use tokio::io::AsyncWriteExt; +use wfl::Interpreter; +use wfl::config::WflConfig; +use wfl::lexer::lex_wfl_with_positions; +use wfl::parser::Parser; + +mod common; + +async fn wait_for_server(port: u16) { + let addr = format!("127.0.0.1:{port}"); + for _ in 0..300 { + if tokio::net::TcpStream::connect(&addr).await.is_ok() { + return; + } + tokio::time::sleep(Duration::from_millis(50)).await; + } + panic!("server on {addr} did not become ready"); +} + +#[tokio::test] +async fn test_backpressured_write_to_a_non_reading_client_is_bounded() { + let port = common::free_tcp_port(); + + // Handler streams FAR more data than any OS send buffer can hold to a + // connected-but-non-reading client: the socket buffer fills, then the 64-slot + // channel fills, and the next `write` parks on backpressure. With a 2s response + // timeout the parked write must fail (not hang forever); the serial main loop + // propagates that error, so `interpret()` RETURNS instead of pinning the handler. + // The payload is grown by doubling to ~40 KB so a few hundred chunks overflow + // the buffer regardless of its autotuned size; the byte ceiling is raised so the + // write blocks (not a budget rejection) first. + let code = format!( + r#" + listen on port {port} as srv + main loop: + wait for request comes in on srv as req with timeout 60000 + store payload as "0123456789" + count from 1 to 12: + store payload as payload with payload + end count + start streaming response to req with status 200 and content type "text/plain" as out + count from 1 to 100000: + write chunk payload to out + end count + close out + break + end loop + "# + ); + + let (done_tx, done_rx) = tokio::sync::oneshot::channel::(); + let server = std::thread::spawn(move || { + let rt = tokio::runtime::Builder::new_current_thread() + .enable_all() + .build() + .expect("server runtime"); + rt.block_on(async { + let tokens = lex_wfl_with_positions(&code); + let program = Parser::new(&tokens).parse().expect("parse"); + let config = WflConfig { + timeout_seconds: 60, + web_server_response_timeout_seconds: 2, + web_server_max_response_size: 512 * 1024 * 1024, + ..WflConfig::default() + }; + let mut interp = Interpreter::with_config(Arc::new(config)); + let start = Instant::now(); + let _ = interp.interpret(&program).await; + let _ = done_tx.send(start.elapsed()); + }); + }); + + wait_for_server(port).await; + + // Connect, send the request, then NEVER read. Hold the socket open so the write + // stalls on backpressure rather than a disconnect. + let mut sock = tokio::net::TcpStream::connect(("127.0.0.1", port)) + .await + .expect("connect"); + sock.write_all(b"GET / HTTP/1.1\r\nHost: 127.0.0.1\r\n\r\n") + .await + .expect("send request"); + sock.flush().await.ok(); + + // `interpret()` must return once the stalled write times out (~2s). If the write + // were unbounded it would pin the handler and this never fires. + let elapsed = tokio::time::timeout(Duration::from_secs(12), done_rx) + .await + .expect("interpret() never returned — the backpressured write pinned the handler forever") + .expect("done sender dropped"); + assert!( + elapsed < Duration::from_secs(9), + "the stalled write should time out at ~2s (web_server_response_timeout_seconds), \ + took {elapsed:?}" + ); + + drop(sock); // keep the client connected until the assertion above + match tokio::task::spawn_blocking(move || server.join()).await { + Ok(Ok(())) => {} + Ok(Err(panic)) => std::panic::resume_unwind(panic), + Err(e) => panic!("server join task failed: {e}"), + } +} + +#[tokio::test] +async fn test_early_chunk_is_visible_before_the_body_completes() { + let port = common::free_tcp_port(); + + // Handler sends an early chunk + flush, WAITS 2s, then sends a late chunk and + // closes. A streaming client must SEE the early chunk well before the late one — + // proving head/first-chunk visibility before body completion, not buffer-to-end. + let code = format!( + r#" + listen on port {port} as srv + main loop: + wait for request comes in on srv as req with timeout 30000 + start streaming response to req with status 200 and content type "text/plain" as out + write chunk "EARLY" to out + flush out + wait for 2000 milliseconds + write chunk "LATE" to out + close out + break + end loop + "# + ); + + let server = std::thread::spawn(move || { + let rt = tokio::runtime::Runtime::new().expect("server runtime"); + rt.block_on(async { + let tokens = lex_wfl_with_positions(&code); + let program = Parser::new(&tokens).parse().expect("parse"); + let mut interp = Interpreter::new(); + let _ = interp.interpret(&program).await; + }); + }); + + wait_for_server(port).await; + + let mut resp = reqwest::Client::new() + .get(format!("http://127.0.0.1:{port}/")) + .send() + .await + .expect("request send"); + assert_eq!(resp.status().as_u16(), 200); + + let start = Instant::now(); + let mut early_at = None; + let mut late_at = None; + let mut acc = String::new(); + loop { + match tokio::time::timeout(Duration::from_secs(6), resp.chunk()).await { + Ok(Ok(Some(bytes))) => { + acc.push_str(&String::from_utf8_lossy(&bytes)); + if early_at.is_none() && acc.contains("EARLY") { + early_at = Some(start.elapsed()); + } + if late_at.is_none() && acc.contains("LATE") { + late_at = Some(start.elapsed()); + } + } + Ok(Ok(None)) | Ok(Err(_)) => break, + Err(_) => panic!("streaming body stalled"), + } + } + + let early = early_at.expect("the EARLY chunk was never received"); + let late = late_at.expect("the LATE chunk was never received"); + // The early chunk must arrive well before the late one — proving it was flushed + // to the wire while the body was still open, not buffered until close. + assert!( + early < Duration::from_millis(1500), + "the EARLY chunk should be visible almost immediately, arrived at {early:?}" + ); + assert!( + late - early > Duration::from_millis(1000), + "the LATE chunk should arrive ~2s after EARLY (early={early:?}, late={late:?}); \ + a small gap means the body was buffered to completion instead of streamed" + ); + + match tokio::task::spawn_blocking(move || server.join()).await { + Ok(Ok(())) => {} + Ok(Err(panic)) => std::panic::resume_unwind(panic), + Err(e) => panic!("server join task failed: {e}"), + } +} diff --git a/tests/wait_line_pre_response_disconnect_test.rs b/tests/wait_line_pre_response_disconnect_test.rs new file mode 100644 index 00000000..9a68dbdd --- /dev/null +++ b/tests/wait_line_pre_response_disconnect_test.rs @@ -0,0 +1,156 @@ +//! Real-socket regression (maintainer re-review, P1): a downstream disconnect must +//! cancel a proxy handler blocked in `wait for next LINE` BEFORE it has called +//! `start streaming response` — exactly like `wait for next chunk`. +//! +//! The chunk read raced the combined pre-response/downstream disconnect signal; the +//! line read only watched the (not-yet-existing) downstream stream, so a client that +//! went away while the handler was blocked reading an upstream line was ignored until +//! the read timeout, occupying the upstream socket and the handler. Topology: an +//! upstream that sends a head then withholds all body lines <- WFL concurrent proxy +//! -> a client that connects and disconnects during the blocked line read. + +use std::time::Duration; +use tokio::io::{AsyncReadExt, AsyncWriteExt}; +use wfl::Interpreter; +use wfl::lexer::lex_wfl_with_positions; +use wfl::parser::Parser; + +mod common; + +/// Upstream: send a valid chunked head, then WITHHOLD all body bytes (no line ever +/// arrives). Signal when the proxy drops the connection (peer close => read 0/Err). +async fn spawn_head_then_no_lines_upstream() -> (u16, tokio::sync::oneshot::Receiver<()>) { + let listener = tokio::net::TcpListener::bind("127.0.0.1:0") + .await + .expect("bind mock upstream"); + let port = listener.local_addr().unwrap().port(); + let (tx, rx) = tokio::sync::oneshot::channel(); + tokio::spawn(async move { + if let Ok((mut sock, _)) = listener.accept().await { + let mut buf = [0u8; 1024]; + let _ = sock.read(&mut buf).await; // request head + let head = "HTTP/1.1 200 OK\r\n\ + Content-Type: text/plain\r\n\ + Transfer-Encoding: chunked\r\n\r\n"; + let _ = sock.write_all(head.as_bytes()).await; + let _ = sock.flush().await; + // Withhold all body lines; wait for the proxy to drop the connection. + loop { + match sock.read(&mut buf).await { + Ok(0) | Err(_) => { + let _ = tx.send(()); + return; + } + Ok(_) => {} + } + } + } + }); + (port, rx) +} + +fn start_proxy_server(code: String) -> std::thread::JoinHandle<()> { + std::thread::spawn(move || { + let rt = tokio::runtime::Runtime::new().expect("runtime"); + rt.block_on(async { + let tokens = lex_wfl_with_positions(&code); + let ast = Parser::new(&tokens).parse().expect("parse"); + let mut interp = Interpreter::new(); + if let Err(errors) = interp.interpret(&ast).await { + panic!("proxy interpreter failed: {errors:?}"); + } + }); + }) +} + +async fn wait_for_server(port: u16) { + let addr = format!("127.0.0.1:{port}"); + for _ in 0..300 { + if tokio::net::TcpStream::connect(&addr).await.is_ok() { + return; + } + tokio::time::sleep(Duration::from_millis(50)).await; + } + panic!("proxy server on {addr} did not become ready"); +} + +#[tokio::test] +async fn test_disconnect_cancels_blocked_pre_response_line_read() { + let (upstream_port, mut upstream_closed) = spawn_head_then_no_lines_upstream().await; + let proxy_port = common::free_tcp_port(); + + // The handler opens the upstream and blocks in `wait for next line` BEFORE + // `start streaming response` — so only the pending-request disconnect signal can + // cancel it. The client disconnects during that blocked read. + let code = format!( + r#" + listen on port {proxy_port} as srv + main loop concurrently: + wait for request comes in on srv as req with timeout 30000 + store p as req["path"] + check if p is equal to "/ping": + respond to req with "pong" + otherwise: + check if p is equal to "/shutdown": + respond to req with "bye" + close server srv + break + otherwise: + open url at "http://127.0.0.1:{upstream_port}/" and stream response as up + wait for next line from up as ln + start streaming response to req with status 200 and content type "text/plain" as down + close down + end check + end check + end loop + "# + ); + let server = start_proxy_server(code); + wait_for_server(proxy_port).await; + + { + let mut sock = tokio::net::TcpStream::connect(("127.0.0.1", proxy_port)) + .await + .expect("connect to proxy"); + sock.write_all(b"GET /proxy HTTP/1.1\r\nHost: 127.0.0.1\r\n\r\n") + .await + .expect("send request"); + sock.flush().await.ok(); + // Let the handler dequeue, open the upstream, and block in the line read, + // then drop the socket to disconnect. + tokio::time::sleep(Duration::from_millis(400)).await; + // `sock` drops here -> client disconnects. + } + + // The upstream must observe its connection close promptly — the blocked line read + // was cancelled by the disconnect, not left to wait out the idle timeout. + tokio::time::timeout(Duration::from_secs(4), &mut upstream_closed) + .await + .expect( + "the blocked pre-response line read was not cancelled after the client disconnected", + ) + .expect("upstream close sender dropped"); + + // The concurrent loop stayed alive. + let ping = tokio::time::timeout( + Duration::from_secs(5), + reqwest::Client::new() + .get(format!("http://127.0.0.1:{proxy_port}/ping")) + .send(), + ) + .await + .expect("/ping timed out") + .expect("/ping failed"); + assert_eq!(ping.status().as_u16(), 200); + assert_eq!(ping.text().await.unwrap(), "pong"); + + let _ = reqwest::Client::new() + .get(format!("http://127.0.0.1:{proxy_port}/shutdown")) + .send() + .await; + match tokio::task::spawn_blocking(move || server.join()).await { + Ok(Ok(())) => {} + Ok(Err(panic)) => std::panic::resume_unwind(panic), + Err(e) => panic!("server join task failed: {e}"), + } +} diff --git a/tests/write_web_postfix_test.rs b/tests/write_web_postfix_test.rs new file mode 100644 index 00000000..db9166be --- /dev/null +++ b/tests/write_web_postfix_test.rs @@ -0,0 +1,135 @@ +//! Regression (maintainer re-review, P1): postfix accessors on `write line|chunk` +//! operands and on merged `content type` / `headers` clause operands must compose +//! onto the operand instead of dangling after the statement. +//! +//! The lexer merges the command word with the following identifier and leaves any +//! `[...]` index / `.field` property accessors as separate tokens, so +//! `write line chunks[0] to out`, `write line upstream.status to out`, +//! `headers upstream.headers`, and `content type upstream.headers["content-type"]` +//! previously left those accessors to dangle (a parse error or a wrong split). + +use std::fs; +use std::process::Command; +use tempfile::TempDir; +use wfl::lexer::lex_wfl_with_positions; +use wfl::parser::Parser; +use wfl::parser::ast::{Expression, Statement}; + +fn parse(src: &str) -> wfl::parser::ast::Program { + let tokens = lex_wfl_with_positions(src); + Parser::new(&tokens).parse().expect("parse should succeed") +} + +fn stream_write_value(stmt: &Statement) -> &Expression { + match stmt { + Statement::StreamWriteStatement { value, .. } => value, + other => panic!("expected a StreamWriteStatement, got {other:#?}"), + } +} + +#[test] +fn write_line_indexed_operand_composes_into_one_index_access() { + let program = parse("write line chunks[0] to out\n"); + assert_eq!( + program.statements.len(), + 1, + "the indexed write operand must not split into extra statements; got {:#?}", + program.statements + ); + assert!( + matches!( + stream_write_value(&program.statements[0]), + Expression::IndexAccess { .. } + ), + "the write value must be an IndexAccess, got {:#?}", + stream_write_value(&program.statements[0]) + ); +} + +#[test] +fn write_line_property_operand_composes_into_one_property_access() { + let program = parse("write line upstream.status to out\n"); + assert_eq!( + program.statements.len(), + 1, + "the property write operand must not split into extra statements; got {:#?}", + program.statements + ); + assert!( + matches!( + stream_write_value(&program.statements[0]), + Expression::PropertyAccess { .. } + ), + "the write value must be a PropertyAccess, got {:#?}", + stream_write_value(&program.statements[0]) + ); +} + +#[test] +fn streaming_response_headers_clause_composes_postfix() { + // `headers upstream.headers` — the operand must bind the `.headers` access. + let program = parse( + "start streaming response to req with status 200 and headers upstream.headers as out\n", + ); + assert_eq!(program.statements.len(), 1, "got {:#?}", program.statements); + match &program.statements[0] { + Statement::StartStreamingResponseStatement { headers, .. } => { + assert!( + matches!(headers, Some(Expression::PropertyAccess { .. })), + "the headers operand must be a PropertyAccess, got {headers:#?}" + ); + } + other => panic!("expected StartStreamingResponseStatement, got {other:#?}"), + } +} + +#[test] +fn streaming_response_content_type_clause_composes_property_then_index() { + // `content type upstream.headers["content-type"]` — property then index. + let program = parse( + "start streaming response to req with status 200 and content type upstream.headers[\"content-type\"] as out\n", + ); + assert_eq!(program.statements.len(), 1, "got {:#?}", program.statements); + match &program.statements[0] { + Statement::StartStreamingResponseStatement { content_type, .. } => { + assert!( + matches!(content_type, Some(Expression::IndexAccess { .. })), + "the content type operand must be an IndexAccess (over a PropertyAccess), \ + got {content_type:#?}" + ); + } + other => panic!("expected StartStreamingResponseStatement, got {other:#?}"), + } +} + +#[test] +fn classic_indexed_file_write_still_works_at_runtime() { + // The ambiguous merged form's classic file-write reading must keep working with + // an indexed operand: `write line values[0] to `. The target is a text + // path, so the runtime takes the classic reading whose content is the merged + // lead `line values` indexed at 0 — it must write the first element, not fail on + // a dangling `[0]`. + let dir = TempDir::new().expect("tempdir"); + let out = dir.path().join("out.txt"); + let out_str = out.to_string_lossy().replace('\\', "/"); + let src = format!( + "store line values as [\"first\" and \"second\"]\n\ + write line values[0] to \"{out_str}\"\n" + ); + let program_file = dir.path().join("main.wfl"); + fs::write(&program_file, &src).unwrap(); + let status = Command::new(env!("CARGO_BIN_EXE_wfl")) + .arg(&program_file) + .status() + .expect("run wfl"); + assert!( + status.success(), + "classic indexed file write should succeed" + ); + let written = fs::read_to_string(&out).expect("output file written"); + assert_eq!( + written.trim_end(), + "first", + "the indexed element must be written, not the whole list" + ); +} From 911ccc327a39c98e4dad647ee8f3fb7532c8e3e6 Mon Sep 17 00:00:00 2001 From: Claude Date: Fri, 24 Jul 2026 13:07:13 +0000 Subject: [PATCH 080/132] fix(P1): streaming lifecycle, cancellation, and ambiguous-write correctness Turns the re-review's failing tests green (Red evidence in the preceding test-only commit). Grouped here because the FlushStreamStatement AST field ties the parser/interpreter/typechecker/analyzer changes together. Cancellation (P1-1/2/3): - `wait for next line` races the SAME combined pre-response/downstream disconnect signal as `wait for next chunk`, so a pre-response line read is cancelled on disconnect. - any_pending_request_disconnected treats an owned request id absent from the pending map as a terminal disconnect (a sibling's wait-for-request prune only removes closed senders), so a parked pre-head handler is no longer stranded. - The buffered `respond`, streaming-head, and response-stream `write` send failures are classified ErrorKind::Cancelled, so a burst of client disconnects at those paths no longer feeds the concurrent structural-failure breaker. Lifecycle (P1-4): - outbound_stream_max_seconds is enforced in real time by a per-stream reaper (stream_handles shared via Arc) so an opened-but-unread upstream cannot outlive the cap; a dropped interpret() now also closes server response streams and 500s unanswered requests via the interpret-scoped guard (fields shared via Rc); a backpressured response write is bounded by web_server_response_timeout_seconds. Backward-compat (P1-5/6 + parser/analyzer): - The ambiguous `write line|chunk ... to ` type check is branch-aware (check the reading the runtime takes) and rejects concrete non-streamable payloads; `flush ` prefers a defined zero-arg action of that full name; postfix accessors compose onto write/headers/content-type operands; the analyzer walks shared call/pattern continuations of an ambiguous write. Co-Authored-By: Claude Opus 4.8 Claude-Session: https://claude.ai/code/session_011LxXAm5E4Vy39kRLr8kRnX --- src/analyzer/mod.rs | 161 ++++++++++++++++++++++++- src/interpreter/mod.rs | 262 ++++++++++++++++++++++++++++++++++------- src/parser/ast.rs | 8 ++ src/parser/stmt/io.rs | 6 + src/parser/stmt/web.rs | 35 ++++-- src/typechecker/mod.rs | 129 ++++++++++++++++---- 6 files changed, 528 insertions(+), 73 deletions(-) diff --git a/src/analyzer/mod.rs b/src/analyzer/mod.rs index 1863f956..a2a742ac 100644 --- a/src/analyzer/mod.rs +++ b/src/analyzer/mod.rs @@ -1689,8 +1689,22 @@ impl Analyzer { } } - Statement::FlushStreamStatement { target, .. } => { - self.analyze_expression(target); + Statement::FlushStreamStatement { + target, + action_fallback, + .. + } => { + // A merged `flush ` may resolve at runtime to a zero-argument + // action named `flush ` (backward compatibility — see + // `parse_flush_stream`). When such an action is defined this is an + // action call, not a stream flush, so analyzing the operand as a + // stream variable would raise a spurious "undefined variable". + let is_action_call = action_fallback + .as_ref() + .is_some_and(|name| self.name_is_defined(name)); + if !is_action_call { + self.analyze_expression(target); + } } Statement::CreateDirectoryStatement { path, .. } => { @@ -3600,6 +3614,149 @@ impl Analyzer { self.analyze_ambiguous_write(vl, fl, line, column); self.analyze_ambiguous_write(vr, fr, line, column); } + // Call- and pattern-based desugarings (`starts/ends with`, `contains`, + // pattern ops, indexing, method/function/action calls) bury the lead in + // one child while the OTHER children are shared continuation. When both + // readings are the same shape (same operator/name and arity), walk the + // corresponding children in parallel so an undefined name in a shared + // argument is still caught; only genuinely non-aligning shapes fall to + // the catch-all and defer to runtime. + ( + Expression::PatternMatch { + text: vt, + pattern: vp, + .. + }, + Expression::PatternMatch { + text: ft, + pattern: fp, + .. + }, + ) + | ( + Expression::PatternFind { + text: vt, + pattern: vp, + .. + }, + Expression::PatternFind { + text: ft, + pattern: fp, + .. + }, + ) + | ( + Expression::PatternSplit { + text: vt, + pattern: vp, + .. + }, + Expression::PatternSplit { + text: ft, + pattern: fp, + .. + }, + ) + | ( + Expression::StringSplit { + text: vt, + delimiter: vp, + .. + }, + Expression::StringSplit { + text: ft, + delimiter: fp, + .. + }, + ) => { + self.analyze_ambiguous_write(vt, ft, line, column); + self.analyze_ambiguous_write(vp, fp, line, column); + } + ( + Expression::PatternReplace { + text: vt, + pattern: vp, + replacement: vrp, + .. + }, + Expression::PatternReplace { + text: ft, + pattern: fp, + replacement: frp, + .. + }, + ) => { + self.analyze_ambiguous_write(vt, ft, line, column); + self.analyze_ambiguous_write(vp, fp, line, column); + self.analyze_ambiguous_write(vrp, frp, line, column); + } + ( + Expression::IndexAccess { + collection: vc, + index: vi, + .. + }, + Expression::IndexAccess { + collection: fc, + index: fi, + .. + }, + ) => { + self.analyze_ambiguous_write(vc, fc, line, column); + self.analyze_ambiguous_write(vi, fi, line, column); + } + ( + Expression::FunctionCall { + function: vf, + arguments: va, + .. + }, + Expression::FunctionCall { + function: ff, + arguments: fa, + .. + }, + ) if va.len() == fa.len() => { + self.analyze_ambiguous_write(vf, ff, line, column); + for (v, f) in va.iter().zip(fa.iter()) { + self.analyze_ambiguous_write(&v.value, &f.value, line, column); + } + } + ( + Expression::ActionCall { + name: vn, + arguments: va, + .. + }, + Expression::ActionCall { + name: fnn, + arguments: fa, + .. + }, + ) if vn == fnn && va.len() == fa.len() => { + for (v, f) in va.iter().zip(fa.iter()) { + self.analyze_ambiguous_write(&v.value, &f.value, line, column); + } + } + ( + Expression::MethodCall { + object: vo, + method: vm, + arguments: va, + .. + }, + Expression::MethodCall { + object: fo, + method: fm, + arguments: fa, + .. + }, + ) if vm == fm && va.len() == fa.len() => { + self.analyze_ambiguous_write(vo, fo, line, column); + for (v, f) in va.iter().zip(fa.iter()) { + self.analyze_ambiguous_write(&v.value, &f.value, line, column); + } + } // Reached a differing leaf — the lead. Report only when NEITHER // reading resolves, so neither valid interpretation is rejected. (Expression::Variable(sn, ..), Expression::Variable(fal, ..)) diff --git a/src/interpreter/mod.rs b/src/interpreter/mod.rs index e41230b1..7ce7279f 100644 --- a/src/interpreter/mod.rs +++ b/src/interpreter/mod.rs @@ -884,32 +884,80 @@ impl<'a, T> Drop for IsolatedHandler<'a, T> { } } -/// RAII guard that closes the interpreter's still-open outbound stream handles +/// RAII guard that finalizes the interpreter's still-open server-request state /// when the running `interpret()` future ends — crucially including when that /// future is *dropped* (an embedder cancels the run) before the normal /// handler-exit / program-cleanup sites execute. It shares the interpreter's -/// `open_http_streams` list and its `IoClient` via `Rc`, so its `Drop` runs even -/// as the interpreter itself stays alive (e.g. a reused REPL). On a normal run -/// the cleanup sites have already drained the list, so this is a no-op. +/// tracking lists and maps via `Rc`, so its `Drop` runs even as the interpreter +/// itself stays alive (e.g. a reused REPL). It covers, for the top-level/serial +/// context whose state lives directly on the interpreter (concurrent handlers +/// finalize their own via `IsolatedHandler`'s `Drop`): +/// +/// - **outbound streams** — dropped from `IoClient.stream_handles`, cancelling +/// the in-flight upstream request; +/// - **server response streams** — dropped from `server_response_streams`, ending +/// the client's body so a `start streaming response` that never `close`d does +/// not leave the connection hanging; +/// - **pending requests** — answered 500 so a dequeued-but-unanswered request is +/// resolved instead of waiting out the request timeout. +/// +/// On a normal run the cleanup sites have already drained the lists, so this is a +/// no-op. All work is synchronous and best-effort (`try_lock` never blocks in a +/// `Drop`). struct OutboundStreamCleanup { io_client: Rc, open_http_streams: Rc>>, + open_response_streams: Rc>>, + server_response_streams: Rc>>, + open_pending_requests: Rc>>, + pending_responses: Rc>>, } impl Drop for OutboundStreamCleanup { fn drop(&mut self) { - let ids = std::mem::take(&mut *self.open_http_streams.borrow_mut()); - if ids.is_empty() { - return; + // Outbound upstream streams: removing a handle drops its reqwest stream, + // cancelling the in-flight upstream request. + let http_ids = std::mem::take(&mut *self.open_http_streams.borrow_mut()); + if !http_ids.is_empty() + && let Ok(mut map) = self.io_client.stream_handles.try_lock() + { + for id in &http_ids { + map.remove(id); + } } - // Best-effort like the other handler-exit cleanup: `try_lock` never - // blocks in a `Drop`. Removing a handle drops its reqwest stream, which - // cancels the in-flight upstream request. - if let Ok(mut map) = self.io_client.stream_handles.try_lock() { - for id in &ids { + + // Server response streams: dropping the sender ends the client's body. + let stream_ids = std::mem::take(&mut *self.open_response_streams.borrow_mut()); + if !stream_ids.is_empty() { + let mut map = self.server_response_streams.borrow_mut(); + for id in &stream_ids { map.remove(id); } } + + // Pending requests dequeued but never answered: resolve each with 500 so + // the client is not left waiting out its request timeout. Mirrors + // `fail_unanswered_requests`, but operates on the shared `Rc` maps so it + // runs even when the interpreter's own methods are unreachable (future + // dropped). The sender mutex is only held during `respond`, which a + // dropped run is no longer inside, so `try_lock` succeeds. + let request_ids = std::mem::take(&mut *self.open_pending_requests.borrow_mut()); + if !request_ids.is_empty() { + let mut pending = self.pending_responses.borrow_mut(); + for id in &request_ids { + if let Some(entry) = pending.remove(id) + && let Ok(mut guard) = entry.sender.try_lock() + && let Some(sender) = guard.take() + { + let _ = sender.send(HandlerReply::Buffered(WflHttpResponse { + content: b"Internal Server Error\n".to_vec(), + status: 500, + content_type: "text/plain; charset=utf-8".to_string(), + headers: HashMap::new(), + })); + } + } + } } } @@ -1277,13 +1325,13 @@ pub struct Interpreter { web_servers: RefCell>, // Web servers by name web_socket_servers: RefCell>, // WebSocket servers keyed by address ws_connections: WsConnectionRegistry, // Outbound senders for all live WebSocket connections - pending_responses: RefCell>, // Pending responses (channel + admission slot) by request ID + pending_responses: Rc>>, // Pending responses (channel + admission slot) by request ID /// Open server response streams (`start streaming response`), keyed by /// handle id ("respstream1", ...). Each holds the bounded body-chunk sender /// plus the running total of body bytes written, enforced against /// `max_response_bytes` so a stream cannot bypass the buffered response /// ceiling. `write line|chunk`/`flush` push to it, `close` drops it. - server_response_streams: RefCell>, + server_response_streams: Rc>>, next_response_stream_id: std::cell::Cell, /// Handle ids of server response streams opened by the currently executing /// handler and not yet explicitly closed. Part of the per-handler `RunState` @@ -1291,12 +1339,20 @@ pub struct Interpreter { /// tracks only its own streams; drained and closed when the handler ends on /// any path so the client's body is always finalized (see /// `close_response_streams`). - open_response_streams: RefCell>, + /// `Rc>` (not a bare `RefCell`) so the `interpret()`-scoped + /// cleanup guard can share it and finalize any still-open server response + /// bodies if the future is dropped before its normal exit sites run (see + /// `OutboundStreamCleanup`). + open_response_streams: Rc>>, /// Request ids the currently executing handler dequeued but has not yet /// answered. Part of the per-handler `RunState` (swapped per poll) so each /// handler tracks only its own requests; any still unanswered when the handler /// ends are answered 500 immediately (see `fail_unanswered_requests`). - open_pending_requests: RefCell>, + /// `Rc>` (not a bare `RefCell`) so the `interpret()`-scoped + /// cleanup guard can share it and 500 any still-unanswered requests if the + /// future is dropped before its normal exit sites run (see + /// `OutboundStreamCleanup`). + open_pending_requests: Rc>>, /// Outbound stream handle ids (`... stream response as `) the currently /// executing handler opened and has not yet closed/exhausted. Part of the /// per-handler `RunState` (swapped per poll); any still open when the handler @@ -1606,8 +1662,10 @@ pub struct IoClient { db_handles: Mutex>, next_db_id: Mutex, /// Live outbound streaming response bodies, keyed by handle id - /// ("httpstream1", ...). See [`HttpStreamHandle`]. - stream_handles: Mutex>, + /// ("httpstream1", ...). See [`HttpStreamHandle`]. Behind an `Arc` so the + /// per-stream absolute-lifetime reaper (spawned in `open_http_stream`) can + /// share it and drop an expired handle in real time, independent of reads. + stream_handles: Arc>>, next_stream_id: Mutex, config: Arc, } @@ -1750,7 +1808,7 @@ impl IoClient { next_process_id: Mutex::new(1), db_handles: Mutex::new(HashMap::new()), next_db_id: Mutex::new(1), - stream_handles: Mutex::new(HashMap::new()), + stream_handles: Arc::new(Mutex::new(HashMap::new())), next_stream_id: Mutex::new(1), config, } @@ -1956,6 +2014,27 @@ impl IoClient { .lock() .await .insert(handle_id.clone(), handle); + + // Enforce `outbound_stream_max_seconds` as a TRUE absolute lifetime, not a + // read-triggered one: a handler that opens a stream and then parks (or does + // other work) without reading would otherwise keep the upstream connection + // alive past the cap, since the deadline is only re-checked on the next + // read. Spawn a reaper that drops the handle from the shared map when the + // absolute deadline elapses; dropping it drops the reqwest body stream and + // closes the upstream connection. Reads that happen first take the handle + // out of the map, so the reaper's later `remove` is then a harmless no-op; + // and a stream closed early leaves only a cheap parked timer (bounded by the + // cap) that no-ops when it fires. + if let Some(deadline) = total_deadline { + let handles = Arc::clone(&self.stream_handles); + let reap_id = handle_id.clone(); + tokio::spawn(async move { + let remaining = deadline.saturating_duration_since(Instant::now()); + tokio::time::sleep(remaining).await; + handles.lock().await.remove(&reap_id); + }); + } + Ok((status, response_headers, handle_id)) } @@ -3465,10 +3544,10 @@ impl Interpreter { web_servers: RefCell::new(HashMap::new()), // Initialize empty web servers map web_socket_servers: RefCell::new(HashMap::new()), // Initialize empty WebSocket servers map ws_connections: Arc::new(std::sync::Mutex::new(HashMap::new())), // Live WebSocket connections - pending_responses: RefCell::new(HashMap::new()), // Initialize empty pending responses map - server_response_streams: RefCell::new(HashMap::new()), - open_response_streams: RefCell::new(Vec::new()), - open_pending_requests: RefCell::new(Vec::new()), + pending_responses: Rc::new(RefCell::new(HashMap::new())), // Initialize empty pending responses map + server_response_streams: Rc::new(RefCell::new(HashMap::new())), + open_response_streams: Rc::new(RefCell::new(Vec::new())), + open_pending_requests: Rc::new(RefCell::new(Vec::new())), open_http_streams: Rc::new(RefCell::new(Vec::new())), next_response_stream_id: std::cell::Cell::new(1), config, @@ -4063,6 +4142,10 @@ impl Interpreter { OutboundStreamCleanup { io_client: Rc::clone(&self.io_client), open_http_streams: Rc::clone(&self.open_http_streams), + open_response_streams: Rc::clone(&self.open_response_streams), + server_response_streams: Rc::clone(&self.server_response_streams), + open_pending_requests: Rc::clone(&self.open_pending_requests), + pending_responses: Rc::clone(&self.pending_responses), } } @@ -4132,12 +4215,23 @@ impl Interpreter { None } else { let pending = self.pending_responses.borrow(); - Some(open.iter().any(|id| { - pending.get(id).is_some_and(|p| match p.sender.try_lock() { + Some(open.iter().any(|id| match pending.get(id) { + Some(p) => match p.sender.try_lock() { Ok(guard) => guard.as_ref().is_some_and(|s| s.is_closed()), // Being responded to right now — not a disconnect. Err(_) => false, - }) + }, + // Owned by this handler yet absent from the map: the only + // removal that leaves an id in `open_pending_requests` is a + // sibling handler's `wait for request` global prune, which + // deletes ONLY closed (disconnected) senders — and a handler + // that answered its own request drops the id from + // `open_pending_requests` in the same step. So a missing + // owned id is a terminal disconnect, not "still connected"; + // reporting it as connected here is exactly the bug where a + // parked handler waits out its timeout after a sibling pruned + // its since-disconnected entry. + None => true, })) } }; @@ -7424,10 +7518,12 @@ impl Interpreter { let handle_id = self .resolve_stream_handle(source, &env, *line, *column) .await?; - // Race the upstream read against a downstream client disconnect - // (see the `wait for next chunk` handler). - let disconnect = - Self::any_downstream_disconnected(self.downstream_disconnect_senders()); + // Race the upstream read against a client disconnect by EITHER + // signal (open downstream stream OR the pre-response pending + // request's oneshot), exactly like the `wait for next chunk` + // handler — a line read blocked before `start streaming response` + // must also be cancelled the moment the browser goes away. + let disconnect = self.any_client_disconnected(self.downstream_disconnect_senders()); let read = self .io_client .next_line(&handle_id, Arc::clone(&self.budget)); @@ -9331,11 +9427,17 @@ impl Interpreter { match completion.take_sender() { Some(sender) => { if sender.send(HandlerReply::Buffered(response)).is_err() { - return Err(RuntimeError::new( - "Failed to send response - client may have disconnected" - .to_string(), + // Receiver dropped => the client hung up before the + // buffered reply landed. That is a cooperative + // cancellation, not a handler fault — mark it `Cancelled` + // so the concurrent loop's structural-failure breaker + // skips it (a burst of post-dequeue disconnects must not + // tear the server down). + return Err(RuntimeError::with_kind( + "Client disconnected before the response was sent".to_string(), *line, *column, + ErrorKind::Cancelled, )); } } @@ -9524,11 +9626,16 @@ impl Interpreter { }) .is_err() { - return Err(RuntimeError::new( - "Failed to start streaming response - client may have disconnected" + // Client hung up before the streaming head committed — + // cooperative cancellation, not a fault (see the buffered + // `respond` path). `Cancelled` keeps the concurrent + // breaker from counting a disconnect as a failure. + return Err(RuntimeError::with_kind( + "Client disconnected before the streaming response started" .to_string(), *line, *column, + ErrorKind::Cancelled, )); } } @@ -9694,23 +9801,57 @@ impl Interpreter { if *is_line { bytes.push(b'\n'); } - match tx.send(bytes).await { + // Bound the (possibly backpressured) send. Once the 64-slot + // channel fills, `tx.send(..).await` parks until the client + // reads — a client that stays connected but stops reading + // would otherwise pin this handler forever (`main loop` is + // deadline-exempt). Cap the wait at + // `web_server_response_timeout_seconds` (0 = disabled, the + // documented sentinel — keep the unbounded behavior). A + // dropped receiver (disconnect) still returns immediately. + let write_timeout = self.config.web_server_response_timeout_seconds; + // `Ok(())` sent; `Err(false)` receiver dropped (disconnect); + // `Err(true)` timed out with the client still connected but + // not reading (a stall). + let outcome: Result<(), bool> = if write_timeout == 0 { + tx.send(bytes).await.map_err(|_| false) + } else { + match tokio::time::timeout( + std::time::Duration::from_secs(write_timeout), + tx.send(bytes), + ) + .await + { + Ok(Ok(())) => Ok(()), + Ok(Err(_)) => Err(false), + Err(_) => Err(true), + } + }; + match outcome { Ok(()) => Ok((Value::Null, ControlFlow::None)), - Err(_) => { - // Receiver dropped => client disconnected. Drop the - // handle and surface a catchable error so the - // handler can stop (and close any upstream it is - // proxying). Untrack it too so a handler that - // catches this error keeps no stale id. + Err(stalled) => { + // Disconnect or stall: drop the handle so the body + // ends, untrack it so a handler that catches this + // keeps no stale id, and surface a `Cancelled` error. + // Both are client-caused cooperative cancellations, + // not handler faults — `Cancelled` keeps the + // concurrent breaker from counting them (256 stalled + // or hung-up clients must not tear the server down). self.server_response_streams.borrow_mut().remove(&handle_id); self.open_response_streams .borrow_mut() .retain(|s| s != &handle_id); - Err(RuntimeError::new( + let message = if stalled { + "Cannot write to response stream: the client stopped reading \ + (write timed out)" + } else { "Cannot write to response stream: the client has disconnected" - .to_string(), + }; + Err(RuntimeError::with_kind( + message.to_string(), *line, *column, + ErrorKind::Cancelled, )) } } @@ -9724,9 +9865,42 @@ impl Interpreter { } Statement::FlushStreamStatement { target, + action_fallback, line, column, } => { + // Backward compatibility: before `flush` was a streaming command, + // `flush cache` was an expression statement that auto-invoked a + // zero-argument action named `flush cache`. If such an action is + // defined, call it — a pre-existing program that named an action + // `flush ` must keep working rather than being reinterpreted as + // a flush of a stream ``. Only the bare-identifier form carries a + // fallback (see `parse_flush_stream`). + if let Some(name) = action_fallback { + let lookup = env.borrow().get(name); + match lookup { + Some(Value::Function(func)) => { + return self + .call_function(&func, vec![], *line, *column) + .await + .map(|value| (value, ControlFlow::None)); + } + Some(Value::Overloaded(overloaded)) => { + if let Some(func) = overloaded + .overloads + .iter() + .find(|func| func.params.is_empty()) + { + let func = Rc::clone(func); + return self + .call_function(&func, vec![], *line, *column) + .await + .map(|value| (value, ControlFlow::None)); + } + } + _ => {} + } + } let handle_id = self .resolve_server_stream_handle(target, &env, *line, *column) .await?; diff --git a/src/parser/ast.rs b/src/parser/ast.rs index 969cd55c..4f5ee462 100644 --- a/src/parser/ast.rs +++ b/src/parser/ast.rs @@ -624,6 +624,14 @@ pub enum Statement { /// queued bytes to the transport. FlushStreamStatement { target: Expression, + /// The full merged command phrase (`"flush cache"`) when this parsed from + /// the merged `flush ` form. Backward compatibility: before `flush` + /// existed, `flush cache` was an expression statement that auto-invoked a + /// zero-argument action named `flush cache`. If such an action is defined, + /// the interpreter calls it instead of treating `cache` as a stream, so a + /// pre-existing program is never hijacked. `None` when no action shadowing + /// is possible (e.g. a postfix operand like `flush obj.out`). + action_fallback: Option, line: usize, column: usize, }, diff --git a/src/parser/stmt/io.rs b/src/parser/stmt/io.rs index 1deb251b..9581ba70 100644 --- a/src/parser/stmt/io.rs +++ b/src/parser/stmt/io.rs @@ -24,6 +24,12 @@ impl<'a> Parser<'a> { Expression::Variable(_, l, c) => (*l, *c), _ => (0, 0), }; + // The lexer merges `write` with the operand identifier and leaves any + // bracket-index / dotted-property accessors as following tokens, so compose + // them onto the lead (`write line chunks[0] to out`, + // `write line upstream.status to out`, classic `write line values[0] to + // "/tmp/out"`) instead of leaving them to dangle after the statement. + let lead = self.parse_trailing_postfix(lead)?; let lead = if matches!(self.cursor.peek().map(|t| &t.token), Some(Token::KeywordOf)) { self.bump_sync(); // Consume "of" let object = self.parse_primary_expression()?; diff --git a/src/parser/stmt/web.rs b/src/parser/stmt/web.rs index 16522090..8acd29f2 100644 --- a/src/parser/stmt/web.rs +++ b/src/parser/stmt/web.rs @@ -447,7 +447,12 @@ impl<'a> WebParser<'a> for Parser<'a> { None }; content_type = Some(match merged_rest { - Some((rest, (l, c))) => Expression::Variable(rest, l, c), + Some((rest, (l, c))) => { + // Compose any dangling postfix accessors + // (`content type upstream.headers["content-type"]`). + let lead = Expression::Variable(rest, l, c); + self.parse_trailing_postfix(lead)? + } None => self.parse_primary_expression()?, }); } @@ -472,8 +477,8 @@ impl<'a> WebParser<'a> for Parser<'a> { if rest.is_empty() { content_type = Some(self.parse_primary_expression()?); } else { - content_type = - Some(Expression::Variable(rest.to_string(), id_line, id_column)); + let lead = Expression::Variable(rest.to_string(), id_line, id_column); + content_type = Some(self.parse_trailing_postfix(lead)?); } } // `headers ` (bare or merged `headers `). @@ -489,7 +494,10 @@ impl<'a> WebParser<'a> for Parser<'a> { if rest.is_empty() { headers = Some(self.parse_primary_expression()?); } else { - headers = Some(Expression::Variable(rest.to_string(), id_line, id_column)); + // Compose any dangling postfix accessors so direct + // forwarding like `headers upstream.headers` binds fully. + let lead = Expression::Variable(rest.to_string(), id_line, id_column); + headers = Some(self.parse_trailing_postfix(lead)?); } } // A connective directly before `as` just joins the clause list to @@ -538,19 +546,32 @@ impl<'a> WebParser<'a> for Parser<'a> { .strip_prefix("flush") .map(str::trim_start) .unwrap_or(""); - let target = if rest.is_empty() { - self.parse_primary_expression()? + let (target, action_fallback) = if rest.is_empty() { + (self.parse_primary_expression()?, None) } else { // The lexer merged `flush` with the operand identifier, so any postfix // accessors (`flush streams["a"]`, `flush obj.out`) are left as separate // tokens. Compose them onto the split-off lead so the operand parses // consistently with a normal expression instead of dangling. let lead = Expression::Variable(rest.to_string(), line, column); - self.parse_trailing_postfix(lead)? + let composed = self.parse_trailing_postfix(lead)?; + // Backward compatibility: `flush ` used to auto-invoke a + // zero-argument action named "flush ". Carry the full phrase so + // the interpreter can prefer that action when it exists. Only a + // bare-identifier operand can collide with such an action name; a + // postfix operand (`flush obj.out`) cannot, so it carries no fallback + // (else a defined `flush obj` action would wrongly swallow `.out`). + let fallback = if matches!(composed, Expression::Variable(..)) { + Some(phrase.clone()) + } else { + None + }; + (composed, fallback) }; Ok(Statement::FlushStreamStatement { target, + action_fallback, line, column, }) diff --git a/src/typechecker/mod.rs b/src/typechecker/mod.rs index 32d24ee9..b79b7504 100644 --- a/src/typechecker/mod.rs +++ b/src/typechecker/mod.rs @@ -1058,16 +1058,61 @@ impl TypeChecker { column, .. } => { - let _ = self.infer_expression_type(value); + // Branch-aware: the runtime picks the reading by the TARGET's type + // (a response-stream handle -> stream write of `value`; anything + // else with a fallback -> classic file write of `fallback_content`). + // Type-check the reading the runtime will actually take, so a valid + // pre-existing file write is never rejected on the stream branch it + // never runs (and vice versa). let target_type = self.infer_expression_type(target); - // The target is a server response-stream handle. The AMBIGUOUS - // merged form (`write line ... to `) also has a - // classic file-write reading, so a text file-path target is valid - // there — but an unambiguous stream write (no fallback) to a - // concrete non-stream type is a static error. - let file_target_ok = - fallback_content.is_some() && matches!(target_type, Type::Text); - if !self.is_response_stream_target_type(&target_type) && !file_target_ok { + let has_fallback = fallback_content.is_some(); + + if self.is_response_stream_target_type(&target_type) + && !self.is_gradual_type(&target_type) + { + // Concrete response stream: the stream reading is taken. + let value_type = self.infer_expression_type(value); + self.check_streamable_payload(&value_type, *line, *column); + } else if matches!(target_type, Type::Text) && has_fallback { + // Concrete text path: the classic file-write reading is taken. + // Validate the fallback, not the stream `value` the runtime + // never evaluates here. + if let Some(fallback) = fallback_content { + let _ = self.infer_expression_type(fallback); + } + } else if self.is_gradual_type(&target_type) { + // Gradual/unknown target: both readings are viable and the + // runtime decides by the target's runtime type. Accept if EITHER + // reading is well-typed; only report an error when the statement + // is wrong under every interpretation (so a valid file write and + // a valid stream write both pass). Speculative inference rolls + // its emitted errors back. + let stream_ok = { + let checkpoint = self.errors.len(); + let value_type = self.infer_expression_type(value); + let ok = self.errors.len() == checkpoint + && self.is_streamable_payload(&value_type); + self.errors.truncate(checkpoint); + ok + }; + let file_ok = if let Some(fallback) = fallback_content { + let checkpoint = self.errors.len(); + let _ = self.infer_expression_type(fallback); + let ok = self.errors.len() == checkpoint; + self.errors.truncate(checkpoint); + ok + } else { + false + }; + if !stream_ok && !file_ok { + // Broken under both readings: surface the stream reading's + // errors (sub-expression + payload) as the diagnostic. + let value_type = self.infer_expression_type(value); + self.check_streamable_payload(&value_type, *line, *column); + } + } else { + // Concrete non-stream, non-text target (or a text target with no + // fallback): an unambiguous stream write to the wrong type. self.type_error( "`write line|chunk` requires a response-stream handle \ (from `start streaming response ... as ...`)" @@ -1081,20 +1126,32 @@ impl TypeChecker { } Statement::FlushStreamStatement { target, + action_fallback, line, column, } => { - let target_type = self.infer_expression_type(target); - if !self.is_response_stream_target_type(&target_type) { - self.type_error( - "`flush` requires a response-stream handle \ - (from `start streaming response ... as ...`)" - .to_string(), - Some(Type::Custom("ResponseStream".to_string())), - Some(target_type), - *line, - *column, - ); + // A merged `flush ` may resolve at runtime to a + // zero-argument action named `flush ` (backward + // compatibility — see `parse_flush_stream`). Stay lenient ONLY when + // such an action is actually defined; otherwise this is a stream + // flush and a concrete non-stream target is still a static error + // (`flush n` where `n` is a number). + let is_action_call = action_fallback + .as_ref() + .is_some_and(|name| self.action_signatures(name).is_some()); + if !is_action_call { + let target_type = self.infer_expression_type(target); + if !self.is_response_stream_target_type(&target_type) { + self.type_error( + "`flush` requires a response-stream handle \ + (from `start streaming response ... as ...`)" + .to_string(), + Some(Type::Custom("ResponseStream".to_string())), + Some(target_type), + *line, + *column, + ); + } } } Statement::VariableDeclaration { @@ -4813,6 +4870,38 @@ impl TypeChecker { } } + /// A type that inference has not pinned down (gradual typing): it may turn out + /// to be anything at runtime, so a static check must stay lenient rather than + /// reject it. + fn is_gradual_type(&self, ty: &Type) -> bool { + matches!(ty, Type::Unknown | Type::Any | Type::Error) + } + + /// The value types `write line|chunk` can send to a response stream — the + /// runtime stringifies numbers/booleans and sends text/binary as-is, and + /// rejects everything else (Map/List/Nothing/...). Gradual types pass. + fn is_streamable_payload(&self, ty: &Type) -> bool { + matches!(ty, Type::Text | Type::Number | Type::Boolean | Type::Binary) + || self.is_gradual_type(ty) + } + + /// Emit a type error if `ty` is a concrete value the runtime would reject as a + /// response-stream payload (a Map/List/Nothing/... reaches `write` only to fail + /// at runtime otherwise). + fn check_streamable_payload(&mut self, ty: &Type, line: usize, column: usize) { + if !self.is_streamable_payload(ty) { + self.type_error( + "`write line|chunk` can only send text, binary, a number, or a boolean \ + to a response stream" + .to_string(), + Some(Type::Text), + Some(ty.clone()), + line, + column, + ); + } + } + fn are_types_compatible(&self, target_type: &Type, source_type: &Type) -> bool { #[allow(clippy::only_used_in_recursion)] let _self = self; // Suppress the warning for self parameter From ec1af0a1cb313d9eda415df49cc0eb61230f9bae Mon Sep 17 00:00:00 2001 From: Claude Date: Fri, 24 Jul 2026 13:07:13 +0000 Subject: [PATCH 081/132] ci(P1): run the documented integration gate and fix the Windows scripts MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit - Clippy runs with --all-features to match the binding gate in testing.md. - The Integration Tests job runs the documented scripts/run_integration_tests (.sh/.ps1) on both OSes, so the intentional-error TestPrograms are actually asserted (their assertions lived only in that script, which CI never invoked). - run_integration_tests.ps1 redirects stdout/stderr to two distinct temp files instead of a single "NUL" — PowerShell 7 rejects reusing one target, which left the whole Windows integration command unrunnable. - run_web_tests.ps1 fails the run (not just warns) when a server cannot be killed/waited or a TLS temp dir leaks, since the test's pass is counted before the finally cleanup. Co-Authored-By: Claude Opus 4.8 Claude-Session: https://claude.ai/code/session_011LxXAm5E4Vy39kRLr8kRnX --- .github/workflows/ci.yml | 27 +++++++++++++++++++-------- scripts/run_integration_tests.ps1 | 12 ++++++++++-- scripts/run_web_tests.ps1 | 29 +++++++++++++++++++++++------ 3 files changed, 52 insertions(+), 16 deletions(-) diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml index 9e3e883f..510a7489 100644 --- a/.github/workflows/ci.yml +++ b/.github/workflows/ci.yml @@ -128,9 +128,11 @@ jobs: - name: Build LSP run: cargo build -p wfl-lsp --verbose - # Run Clippy for code quality + # Run Clippy for code quality. `--all-features` matches the binding gate in + # testing.md (the only features are opt-in dhat profiling, so this just + # compiles the feature-gated code for linting — it never runs it). - name: Run Clippy - run: cargo clippy --all-targets -- -D warnings + run: cargo clippy --all-targets --all-features -- -D warnings # Cross-platform integration test verification integration-tests: @@ -191,13 +193,22 @@ jobs: } Write-Host "✓ Release binary found: target/release/wfl.exe" - # Run integration tests specifically - - name: Run Integration Tests - run: cargo test --test split_functionality --verbose + # Run the DOCUMENTED WFL integration gate (testing.md): the same script a + # contributor runs locally. It executes the integration test binaries + # (`cargo test --test '*'`) AND the TestPrograms end-to-end programs — + # crucially including the intentional-error programs, which it asserts exit + # nonzero (previously those assertions lived only in this script and the + # script was never invoked in CI, so they never ran). Uses the release + # binary built above. Run on BOTH OSes so the declared Windows integration + # command is actually exercised, not merely documented. + - name: Run Integration Gate (Unix) + if: runner.os != 'Windows' + run: ./scripts/run_integration_tests.sh - # Run all integration tests to ensure comprehensive coverage - - name: Run All Integration Tests - run: cargo test --test '*' --verbose + - name: Run Integration Gate (Windows) + if: runner.os == 'Windows' + shell: pwsh + run: ./scripts/run_integration_tests.ps1 # Validate that documentation examples still parse/analyze/lint against the # current release binary (testing.md requires docs validation in CI). diff --git a/scripts/run_integration_tests.ps1 b/scripts/run_integration_tests.ps1 index a630f4f7..2a7dd289 100644 --- a/scripts/run_integration_tests.ps1 +++ b/scripts/run_integration_tests.ps1 @@ -163,8 +163,15 @@ if (-not (Test-Path "TestPrograms")) { Write-Host "[INFO] Testing: $($wflFile.Name)" -ForegroundColor Blue - # Run with timeout to prevent hangs - $process = Start-Process -FilePath ".\$BinaryPath" -ArgumentList $wflArgs -NoNewWindow -PassThru -RedirectStandardOutput "NUL" -RedirectStandardError "NUL" + # Run with timeout to prevent hangs. Start-Process requires DISTINCT + # file targets for stdout and stderr — PowerShell 7 errors when the same + # path is reused for both, and "NUL" is not a valid redirect target + # there — so redirect to two temp files and discard them. (Redirecting + # both to a single "NUL" left the whole Windows integration command + # unrunnable, so its assertions never actually ran.) + $outFile = New-TemporaryFile + $errFile = New-TemporaryFile + $process = Start-Process -FilePath ".\$BinaryPath" -ArgumentList $wflArgs -NoNewWindow -PassThru -RedirectStandardOutput $outFile.FullName -RedirectStandardError $errFile.FullName $completed = $process.WaitForExit($TestTimeout * 1000) $isExpectedFail = $ExpectedFailTests -contains $wflFile.Name @@ -187,6 +194,7 @@ if (-not (Test-Path "TestPrograms")) { Write-Host "[ERROR] FAIL $($wflFile.Name) (exit code: $($process.ExitCode))" -ForegroundColor Red $failedPrograms++ } + Remove-Item $outFile.FullName, $errFile.FullName -ErrorAction SilentlyContinue } Write-Host "" diff --git a/scripts/run_web_tests.ps1 b/scripts/run_web_tests.ps1 index 33ef7e50..8375d7ac 100644 --- a/scripts/run_web_tests.ps1 +++ b/scripts/run_web_tests.ps1 @@ -80,10 +80,16 @@ function Stop-ServerProcess { if ($Process -and -not $Process.HasExited) { # Kill() itself can throw (e.g. access denied, or the process exiting # concurrently), so guard it too rather than leaving it outside the try. + # A failure to kill/wait is a REAL failure — a leaked server holds its + # cert/log handles and can race the temp-dir cleanup — so record it so the + # run fails rather than reporting a false pass. (The pass count is + # incremented before the `finally` cleanup, so a leak here must be able to + # turn the overall result red.) try { $Process.Kill() } catch { - Write-Host "[WARN] Kill() on server process failed: $_" -ForegroundColor Yellow + Write-Host "[ERROR] Kill() on server process failed: $_" -ForegroundColor Red + $script:cleanupFailed = $true } # WaitForExit(ms) returns $true only if the process actually exited in # time; report honestly rather than always claiming success (a process @@ -94,7 +100,8 @@ function Stop-ServerProcess { if ($exited) { Write-Host "[INFO] Server process terminated" -ForegroundColor Gray } else { - Write-Host "[WARN] Server process did not exit within 5s of Kill()" -ForegroundColor Yellow + Write-Host "[ERROR] Server process did not exit within 5s of Kill()" -ForegroundColor Red + $script:cleanupFailed = $true } } } @@ -189,6 +196,10 @@ function Test-WflWebServer { # Run web server tests $totalTests = 0 $passedTests = 0 +# Set true by cleanup that leaks (a server that will not die, or a temp dir that +# will not delete). Because a test's pass is counted before its `finally` cleanup +# runs, a leak here must be able to fail the overall run — see the summary below. +$script:cleanupFailed = $false # Test 1: simple_web_test.wfl if (Test-Path "TestPrograms\simple_web_test.wfl") { @@ -399,12 +410,15 @@ if (Test-Path "TestPrograms\web_server_tls.wfl") { Write-Host "[WARN] TLS server still running after Kill(); waiting briefly before temp cleanup" -ForegroundColor Yellow try { $null = $tlsProcess.WaitForExit(2000) } catch { } } - # Attempt cleanup and SURFACE a failure (a leaked temp dir / still-open - # handle) instead of hiding it behind -ErrorAction SilentlyContinue. + # Attempt cleanup and FAIL the run on a leaked temp dir / still-open + # handle instead of hiding it behind a warning — the test's pass was + # already counted above, so a cleanup leak must be able to turn the + # result red. try { Remove-Item -Recurse -Force $tlsDir -ErrorAction Stop } catch { - Write-Host "[WARN] Failed to remove TLS temp dir ${tlsDir}: $_" -ForegroundColor Yellow + Write-Host "[ERROR] Failed to remove TLS temp dir ${tlsDir}: $_" -ForegroundColor Red + $script:cleanupFailed = $true } } } @@ -415,7 +429,10 @@ Write-Host "" Write-Host "[INFO] ============================" -ForegroundColor Blue Write-Host "[INFO] Results: $passedTests/$totalTests tests passed" -ForegroundColor Blue -if ($passedTests -eq $totalTests) { +if ($script:cleanupFailed) { + Write-Host "[ERROR] A server/temp-dir cleanup leaked; failing the run even though ${passedTests}/${totalTests} test assertions passed" -ForegroundColor Red + exit 1 +} elseif ($passedTests -eq $totalTests) { Write-Host "[SUCCESS] All web server tests passed!" -ForegroundColor Green exit 0 } else { From 04bbbb9ac00d22da3b94dec99a8108146952bcf9 Mon Sep 17 00:00:00 2001 From: Claude Date: Fri, 24 Jul 2026 13:07:14 +0000 Subject: [PATCH 082/132] docs(P1): real-time outbound cap, streaming-write timeout, re-review dev diary - configuration-reference: outbound_stream_max_seconds is enforced in real time (a reaper closes an opened-but-unread stream at the cap), and web_server_response_timeout_seconds also bounds a backpressured streaming write. - Dev Diary entry for the R3 re-review round (risk class, per-blocker fix + test, Red evidence, CI gate). Co-Authored-By: Claude Opus 4.8 Claude-Session: https://claude.ai/code/session_011LxXAm5E4Vy39kRLr8kRnX --- ...26-07-24-streaming-rereview-p1-blockers.md | 99 +++++++++++++++++++ Docs/reference/configuration-reference.md | 6 +- 2 files changed, 102 insertions(+), 3 deletions(-) diff --git a/Dev diary/2026-07-24-streaming-rereview-p1-blockers.md b/Dev diary/2026-07-24-streaming-rereview-p1-blockers.md index 352e1e0e..973deb75 100644 --- a/Dev diary/2026-07-24-streaming-rereview-p1-blockers.md +++ b/Dev diary/2026-07-24-streaming-rereview-p1-blockers.md @@ -139,3 +139,102 @@ operator-continuation no-false-positive), all prior back-compat cases intact. reads it is still reclaimed at handler exit rather than by a mid-idle timer — the single-threaded, `!Send`-stream model has no wake point to close it earlier; this is noted rather than claimed as instantaneous. + + +--- + +# Second re-review round — deeper P1 blockers (head `e8c9712`) + +Round of fixes for the maintainer's re-review at `e8c9712`. Risk class **R3** +(concurrency, cancellation, lifecycle, streaming, backward compatibility). Each +behavioral change has a Red→Green real-boundary test; the Red evidence is a +test-only commit that is an ancestor of the source commit (verified by running +each new test with the source fixes stashed). + +## Cancellation / disconnect + +- **`wait for next line` pre-response disconnect (P1-1).** The line read watched + only the downstream response stream, which does not exist before `start + streaming response`; it now races the same combined pending-request/downstream + signal as `wait for next chunk`, so a blocked pre-response line read is + cancelled the moment the client goes away. *Test:* + `wait_line_pre_response_disconnect_test`. +- **Sibling-prune cancellation race (P1-2).** `any_pending_request_disconnected` + treated an owned request id missing from `pending_responses` as "still + connected". But the only removal that leaves an id in `open_pending_requests` + is a sibling `wait for request`'s global prune, which deletes ONLY closed + (disconnected) senders — so a missing owned id is now treated as a terminal + disconnect, and a parked pre-head handler is no longer stranded until its idle + timeout. *Test:* `concurrent_prehead_prune_race_test`. +- **Classify every client disconnect as cancellation (P1-3).** The buffered + `respond`, streaming-head, and response-stream `write` send failures returned a + General runtime error, which fed the concurrent loop's structural-failure + breaker — so a burst of >256 disconnects at those paths tore the loop down. + They are now `ErrorKind::Cancelled`. *Test:* + `concurrent_disconnect_paths_burst_test` (buffered-respond and stream-write + bursts; `/ping` survives). + +## Lifecycle (P1-4) + +- **Absolute outbound lifetime is real-time (a).** `outbound_stream_max_seconds` + was only re-checked on the next read, so an opened-but-unread upstream outlived + the cap. `stream_handles` is now shared via `Arc` and each open spawns a reaper + that drops the handle (cancelling the upstream) when the deadline elapses. + *Test:* `outbound_stream_open_expiry_test`. *Docs:* configuration-reference + updated to state the cap is enforced in real time. +- **Dropped-run cleanup covers server streams + pending (b).** The + interpret-scoped guard covered only outbound streams; the server response + streams and pending requests (`Rc`-shared now) are also finalized on a dropped + `interpret()`, so a cancelled run does not leave a client body hanging on a + reused interpreter. *Test:* `dropped_interpret_server_cleanup_test` (holds the + interpreter alive after the drop to prove it is the guard, not interpreter + teardown, that closes the body). +- **Backpressured write is bounded (c).** `tx.send(bytes).await` past the 64-slot + channel could park forever against a connected-but-non-reading client (a `main + loop` is deadline-exempt). It is now capped by + `web_server_response_timeout_seconds`. *Test:* + `response_stream_backpressure_test` (a >send-buffer payload genuinely blocks; + the write fails at the cap instead of pinning). *Docs:* config reference notes + this timeout bounds streaming writes. + +## Backward compatibility / correctness + +- **Branch-aware ambiguous-write type check (P1-5).** `write line|chunk to + ` has a stream reading and a classic file-write reading; the checker + now validates the reading the runtime actually takes (by the target type), + instead of always checking the stream `value` — so a valid file write is no + longer rejected on the never-run stream branch, a broken file write is caught, + and a concrete non-streamable payload (Map/List/Nothing) to a real stream is a + static error. *Test:* `ambiguous_write_branch_typecheck_test`. +- **`flush` no longer steals a zero-arg action (P1-6).** `flush cache` used to + auto-invoke an action named `flush cache`; the streaming `flush` dispatch now + carries the full merged phrase and the interpreter/typechecker/analyzer prefer + a defined action of that name before treating the operand as a stream. *Test:* + `flush_action_backcompat_test`. +- **Postfix composition on write / web-clause operands (P1-9).** `write line + chunks[0] to out`, `write line upstream.status to out`, `headers + upstream.headers`, and `content type upstream.headers["content-type"]` compose + their trailing `[...]`/`.field` accessors instead of leaving them to dangle. + *Test:* `write_web_postfix_test`. +- **Analyzer walks call/pattern continuations (P1-10).** `analyze_ambiguous_write` + now recurses in parallel through `starts/ends with`, pattern, index, and + function/action/method-call shapes, so an undefined name in a shared + continuation is reported instead of reaching runtime. *Test:* + `ambiguous_write_analyzer_test`. + +## Test infrastructure / CI + +- Clippy runs `--all-features` (matching the binding gate in `testing.md`). +- The Integration Tests job runs the documented + `scripts/run_integration_tests.{sh,ps1}` on both OSes, so the intentional-error + TestPrograms are actually asserted (their assertions lived only in that script, + which CI never invoked). +- `run_integration_tests.ps1` redirects stdout/stderr to two distinct temp files + (PowerShell 7 rejects reusing a single `NUL` target, which left the Windows + integration command unrunnable). +- `run_web_tests.ps1` fails the run — not merely warns — when a server cannot be + killed/waited or a TLS temp dir leaks, since the pass is counted before the + `finally` cleanup. +- Streaming visibility coverage: `response_stream_backpressure_test` also proves + an early chunk is delivered on the wire ~2s before the late one (head/first + chunk visible before body completion, not buffered to close). diff --git a/Docs/reference/configuration-reference.md b/Docs/reference/configuration-reference.md index 0a2f2e12..2311ebb7 100644 --- a/Docs/reference/configuration-reference.md +++ b/Docs/reference/configuration-reference.md @@ -559,17 +559,17 @@ or omits `Content-Length`. #### `web_server_response_timeout_seconds` -Maximum time, in seconds, the transport waits for a handler to answer an accepted request before shedding it with a `504 Gateway Timeout` and freeing its in-flight slot. This bounds a dequeued-but-never-answered request so it cannot pin an in-flight slot indefinitely. +Maximum time, in seconds, the transport waits for a handler to answer an accepted request before shedding it with a `504 Gateway Timeout` and freeing its in-flight slot. This bounds a dequeued-but-never-answered request so it cannot pin an in-flight slot indefinitely. It also bounds a single **streaming-response** `write` (`write line|chunk ... to `): when a connected client stops reading, the bounded body channel fills and the write applies backpressure, so this timeout caps how long that write parks before failing — a stalled client can slow a handler but not pin it forever. - **Type:** Integer (0 or more) - **Default:** `300` - **Example:** `web_server_response_timeout_seconds = 30` -A value of `0` disables the timeout. The in-flight request cap (`web_server_request_queue_bound`) is enforced globally across every `listen` server via one shared budget, and a request's slot is held from the moment its body starts streaming until the handler responds, this timeout fires, or the client disconnects. +A value of `0` disables the timeout (including the streaming-write bound above). The in-flight request cap (`web_server_request_queue_bound`) is enforced globally across every `listen` server via one shared budget, and a request's slot is held from the moment its body starts streaming until the handler responds, this timeout fires, or the client disconnects. #### `outbound_stream_max_seconds` -Absolute total lifetime, in seconds, of a single **outbound** streaming response opened with `open url ... and stream response as `, measured from when the stream is opened. This is distinct from `timeout_seconds`, which is the per-read **idle** timeout: an upstream that trickles one byte just before every idle timeout would otherwise run forever, but it can never live past this hard cap. Each incremental read (`wait for next line/chunk`) is additionally bounded by the time remaining to this deadline, so no single read waits past the total. +Absolute total lifetime, in seconds, of a single **outbound** streaming response opened with `open url ... and stream response as `, measured from when the stream is opened. This is distinct from `timeout_seconds`, which is the per-read **idle** timeout: an upstream that trickles one byte just before every idle timeout would otherwise run forever, but it can never live past this hard cap. The cap is enforced in **real time**, not only on the next read: each incremental read (`wait for next line/chunk`) is bounded by the time remaining to this deadline, AND a background reaper closes the stream (cancelling the upstream request) when the deadline elapses — so even a stream that is opened and then never read cannot outlive the cap. - **Type:** Integer (0 or more) - **Default:** `300` From b465600ba1e018aa8e82e6a8d899af38c9538e38 Mon Sep 17 00:00:00 2001 From: Claude Date: Fri, 24 Jul 2026 13:13:12 +0000 Subject: [PATCH 083/132] fix(ci,docs): make run_integration_tests.sh executable; correct write postfix doc - scripts/run_integration_tests.sh was committed 100644, so the new CI step './scripts/run_integration_tests.sh' failed with 'Permission denied' (exit 126); set the executable bit to match run_web_tests.sh. - web-servers.md said postfix accessors on a 'write line|chunk' leading identifier 'are not parsed here'. This PR composes them (parse_write_value_from_lead -> parse_trailing_postfix), so the sentence now describes the actual behavior (review feedback). Co-Authored-By: Claude Opus 4.8 Claude-Session: https://claude.ai/code/session_011LxXAm5E4Vy39kRLr8kRnX --- Docs/04-advanced-features/web-servers.md | 7 ++++--- scripts/run_integration_tests.sh | 0 2 files changed, 4 insertions(+), 3 deletions(-) mode change 100644 => 100755 scripts/run_integration_tests.sh diff --git a/Docs/04-advanced-features/web-servers.md b/Docs/04-advanced-features/web-servers.md index c7a801cd..737a614f 100644 --- a/Docs/04-advanced-features/web-servers.md +++ b/Docs/04-advanced-features/web-servers.md @@ -517,9 +517,10 @@ close out > **Value operators.** The value accepts `with`-concatenation and the usual > arithmetic/comparison operators directly in the statement — e.g. > `write line prefix with json to out`. (An identifier-led value is a variable - > or `field of object` followed by such operators; postfix forms like indexing - > `payload[1]` or `payload.field` on the leading identifier are not parsed here - > — build those into a variable first.) For the ambiguous bare-identifier form, + > or `field of object` followed by such operators; postfix accessors on the + > leading identifier — indexing `payload[1]` or property access `payload.field` + > — are composed onto it, so `write line chunks[0] to out` and `write line + > upstream.status to out` work.) For the ambiguous bare-identifier form, > the continuation applies to both readings: `write line note with "!" to out` > streams `note` + `"!"`, while the same statement targeting a file writes the > variable `line note` + `"!"` — so pre-existing classic file writes that diff --git a/scripts/run_integration_tests.sh b/scripts/run_integration_tests.sh old mode 100644 new mode 100755 From 19bf271f94b6e61bf54a5da373e07dccc577d493 Mon Sep 17 00:00:00 2001 From: Claude Date: Fri, 24 Jul 2026 13:23:24 +0000 Subject: [PATCH 084/132] refactor(parser): anchor the write `of` call to the `of` keyword parse_write_value_from_lead stamped the ` of ` FunctionCall with the lead's position; anchor it to the `of` keyword instead, matching how the rest of the parser positions FunctionCall nodes so error spans point at the operator (review feedback). Behavior-preserving. Co-Authored-By: Claude Opus 4.8 Claude-Session: https://claude.ai/code/session_011LxXAm5E4Vy39kRLr8kRnX --- src/parser/stmt/io.rs | 16 +++++++++------- 1 file changed, 9 insertions(+), 7 deletions(-) diff --git a/src/parser/stmt/io.rs b/src/parser/stmt/io.rs index 9581ba70..8214d031 100644 --- a/src/parser/stmt/io.rs +++ b/src/parser/stmt/io.rs @@ -20,10 +20,6 @@ impl<'a> Parser<'a> { /// the pattern operators build calls), so deriving one AST from the other by /// leaf-swapping silently corrupted the classic reading. fn parse_write_value_from_lead(&mut self, lead: Expression) -> Result { - let (line, column) = match &lead { - Expression::Variable(_, l, c) => (*l, *c), - _ => (0, 0), - }; // The lexer merges `write` with the operand identifier and leaves any // bracket-index / dotted-property accessors as following tokens, so compose // them onto the lead (`write line chunks[0] to out`, @@ -31,7 +27,13 @@ impl<'a> Parser<'a> { // "/tmp/out"`) instead of leaving them to dangle after the statement. let lead = self.parse_trailing_postfix(lead)?; let lead = if matches!(self.cursor.peek().map(|t| &t.token), Some(Token::KeywordOf)) { - self.bump_sync(); // Consume "of" + // Anchor the ` of ` call to the `of` keyword itself, + // matching how the rest of the parser positions FunctionCall nodes so + // error spans point at the operator, not the lead (review feedback). + let (of_line, of_column) = self + .bump_sync() + .map(|t| (t.line, t.column)) + .expect("peeked `of` immediately above"); let object = self.parse_primary_expression()?; Expression::FunctionCall { function: Box::new(lead), @@ -39,8 +41,8 @@ impl<'a> Parser<'a> { name: None, value: object, }], - line, - column, + line: of_line, + column: of_column, } } else { lead From 5fd93a9729e55ae891376c03ffa5737bcd022a0a Mon Sep 17 00:00:00 2001 From: Claude Date: Fri, 24 Jul 2026 13:37:59 +0000 Subject: [PATCH 085/132] test(parser): Red coverage for write `of`-arg precedence + method-call postfix Against the current source these fail (verified with the fixes stashed): - write_line_of_call_argument_absorbs_arithmetic: `double of n minus 1` parses as `(double of n) minus 1`, not `double of (n minus 1)`. - write_line_method_call_operand_composes / flush_method_call_operand_composes: a `.method(...)` operand leaves the `(...)` dangling. Co-Authored-By: Claude Opus 4.8 Claude-Session: https://claude.ai/code/session_011LxXAm5E4Vy39kRLr8kRnX --- tests/write_web_postfix_test.rs | 54 +++++++++++++++++++++++++++++++++ 1 file changed, 54 insertions(+) diff --git a/tests/write_web_postfix_test.rs b/tests/write_web_postfix_test.rs index db9166be..4ed5aa26 100644 --- a/tests/write_web_postfix_test.rs +++ b/tests/write_web_postfix_test.rs @@ -102,6 +102,60 @@ fn streaming_response_content_type_clause_composes_property_then_index() { } } +#[test] +fn write_line_of_call_argument_absorbs_arithmetic() { + // `double of n minus 1` must parse as `double of (n minus 1)` — the same + // precedence as an ordinary expression — not `(double of n) minus 1`. + let program = parse("write line double of n minus 1 to out\n"); + assert_eq!(program.statements.len(), 1, "got {:#?}", program.statements); + match stream_write_value(&program.statements[0]) { + Expression::FunctionCall { arguments, .. } => { + assert_eq!( + arguments.len(), + 1, + "the `of` call should take one argument, got {arguments:#?}" + ); + assert!( + matches!(arguments[0].value, Expression::BinaryOperation { .. }), + "the `of` argument must absorb `minus 1` (double of (n minus 1)), got {:#?}", + arguments[0].value + ); + } + other => panic!("expected the value to be an `of` FunctionCall, got {other:#?}"), + } +} + +#[test] +fn write_line_method_call_operand_composes() { + // `obj.method()` must compose into a MethodCall, not leave `()` dangling. + let program = parse("write line obj.method() to out\n"); + assert_eq!(program.statements.len(), 1, "got {:#?}", program.statements); + assert!( + matches!( + stream_write_value(&program.statements[0]), + Expression::MethodCall { .. } + ), + "the write value must be a MethodCall, got {:#?}", + stream_write_value(&program.statements[0]) + ); +} + +#[test] +fn flush_method_call_operand_composes() { + // `flush obj.method()` must compose the method call onto the operand. + let program = parse("flush obj.method()\n"); + assert_eq!(program.statements.len(), 1, "got {:#?}", program.statements); + match &program.statements[0] { + Statement::FlushStreamStatement { target, .. } => { + assert!( + matches!(target, Expression::MethodCall { .. }), + "the flush operand must be a MethodCall, got {target:#?}" + ); + } + other => panic!("expected FlushStreamStatement, got {other:#?}"), + } +} + #[test] fn classic_indexed_file_write_still_works_at_runtime() { // The ambiguous merged form's classic file-write reading must keep working with From f41f6b732e3eff1f35a1662cd222d976864af5cc Mon Sep 17 00:00:00 2001 From: Claude Date: Fri, 24 Jul 2026 13:38:00 +0000 Subject: [PATCH 086/132] fix(parser): consistent `of`-arg precedence and method-call postfix in operands Review feedback on the merged-command operand parsing: - parse_write_value_from_lead parsed the `of` argument with parse_primary_expression, changing precedence vs ordinary expressions; use parse_of_call_argument (absorbs arithmetic, accepts and/from/by/length multi-args) so `write line double of n minus 1 to out` means `double of (n minus 1)`. - parse_trailing_postfix composed `.field`/`[index]` but not `.method(...)`, so `write line obj.method() to out` / `flush obj.method()` left the `(...)` dangling; it now composes the method call, mirroring the primary dispatch. Co-Authored-By: Claude Opus 4.8 Claude-Session: https://claude.ai/code/session_011LxXAm5E4Vy39kRLr8kRnX --- src/parser/expr/primary.rs | 48 +++++++++++++++++++++++++++++++++----- src/parser/stmt/io.rs | 33 ++++++++++++++++++++++---- 2 files changed, 70 insertions(+), 11 deletions(-) diff --git a/src/parser/expr/primary.rs b/src/parser/expr/primary.rs index 9af12dfa..f0113a11 100644 --- a/src/parser/expr/primary.rs +++ b/src/parser/expr/primary.rs @@ -177,12 +177,48 @@ impl<'a> Parser<'a> { } }; self.bump_sync(); // Consume the property name - expr = Expression::PropertyAccess { - object: Box::new(expr), - property, - line, - column, - }; + // `.method(args)` — a method call, not a bare property access. + // Mirrors the primary dispatch so merged-command operands like + // `write line obj.method() to out` / `flush obj.method()` compose + // the call instead of leaving the `(...)` to dangle. + if matches!(self.cursor.peek().map(|t| &t.token), Some(Token::LeftParen)) { + self.bump_sync(); // Consume '(' + let mut arguments = Vec::new(); + if let Some(next) = self.cursor.peek() + && next.token != Token::RightParen + { + arguments.push(Argument { + name: None, + value: self.parse_expression()?, + }); + while matches!(self.cursor.peek().map(|t| &t.token), Some(Token::Comma)) + { + self.bump_sync(); // Consume ',' + arguments.push(Argument { + name: None, + value: self.parse_expression()?, + }); + } + } + self.expect_token( + Token::RightParen, + "Expected ')' after method arguments", + )?; + expr = Expression::MethodCall { + object: Box::new(expr), + method: property, + arguments, + line, + column, + }; + } else { + expr = Expression::PropertyAccess { + object: Box::new(expr), + property, + line, + column, + }; + } } _ => break, } diff --git a/src/parser/stmt/io.rs b/src/parser/stmt/io.rs index 8214d031..629be5b2 100644 --- a/src/parser/stmt/io.rs +++ b/src/parser/stmt/io.rs @@ -34,13 +34,36 @@ impl<'a> Parser<'a> { .bump_sync() .map(|t| (t.line, t.column)) .expect("peeked `of` immediately above"); - let object = self.parse_primary_expression()?; + // Parse the `of`-call argument(s) EXACTLY as the primary parser does: + // each argument absorbs arithmetic (`fibonacci of n minus 1` means + // `fibonacci of (n minus 1)`, not `(fibonacci of n) minus 1`), and + // `and`/`from`/`by`/`length` join multiple arguments — so an `of`-call in + // a `write line|chunk` value parses identically to one in an ordinary + // expression. + let mut arguments = vec![crate::parser::ast::Argument { + name: None, + value: self.parse_of_call_argument()?, + }]; + while let Some(sep) = self.cursor.peek() { + let is_separator = matches!( + &sep.token, + Token::KeywordAnd | Token::KeywordFrom | Token::KeywordBy + ) || matches!( + &sep.token, + Token::Identifier(id) if id.eq_ignore_ascii_case("length") + ); + if !is_separator { + break; + } + self.bump_sync(); // Consume the separator + arguments.push(crate::parser::ast::Argument { + name: None, + value: self.parse_of_call_argument()?, + }); + } Expression::FunctionCall { function: Box::new(lead), - arguments: vec![crate::parser::ast::Argument { - name: None, - value: object, - }], + arguments, line: of_line, column: of_column, } From b25aed57ea50697c596796446d1f47466668773d Mon Sep 17 00:00:00 2001 From: Claude Date: Fri, 24 Jul 2026 13:44:34 +0000 Subject: [PATCH 087/132] docs: clarify how a concurrent handler failure reaches the client MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit A handler error/panic yields a 500/504 only if no response was sent yet; after 'start streaming response' the status/headers are already committed (typically 200), so the failure can only end the body early (the stream is closed) — a truncated body under the already-sent status (review feedback). Co-Authored-By: Claude Opus 4.8 Claude-Session: https://claude.ai/code/session_011LxXAm5E4Vy39kRLr8kRnX --- Docs/04-advanced-features/web-servers.md | 10 ++++++++-- 1 file changed, 8 insertions(+), 2 deletions(-) diff --git a/Docs/04-advanced-features/web-servers.md b/Docs/04-advanced-features/web-servers.md index 737a614f..364e612a 100644 --- a/Docs/04-advanced-features/web-servers.md +++ b/Docs/04-advanced-features/web-servers.md @@ -110,8 +110,14 @@ bound work (the common web case), not CPU-bound loops. - A slow handler does not block its siblings. - Each request handler is isolated (its own scope). -- A handler that errors or panics is contained: that request fails on its own - (its client gets a 500/timeout) and the server keeps serving everyone else. +- A handler that errors or panics is contained: that request fails on its own and + the server keeps serving everyone else. How the client sees the failure depends + on how far the handler got: if it had **not** sent a response yet, the client + gets a `500` (or a `504` if the handler never answers in time); if it had already + called `start streaming response`, the status and headers are on the wire + (typically `200`), so the error cannot change them — the response body is just + ended early (the stream is closed), leaving the client a truncated body under the + already-sent status. - In-flight work is bounded; the transport still sheds excess load with 503 and times out a stalled handler with 504, exactly as for the serial loop. From 5e01e446ab9250d72a0f255bc81a27a79c5b5d63 Mon Sep 17 00:00:00 2001 From: Brad Byrd Date: Fri, 24 Jul 2026 10:11:53 -0500 Subject: [PATCH 088/132] fix: improve concurrent server stability and streaming compatibility - Distinguishes between server-wide errors and request-local issues (like client disconnects or timeouts) so the server stays online even when individual requests fail. - Refines outbound stream management to ensure network connections and memory are cleaned up promptly, even during active data transfers. - Aligns the syntax of commands like `write`, `headers`, and `flush` with ordinary expressions so they support complex lookups and indexing. - Preserves backward compatibility for older programs that used the `flush` keyword for non-streaming actions. Fixes #642 --- .../2026-07-24-issue-642-p1-followups.md | 57 ++ src/analyzer/mod.rs | 33 + src/config.rs | 4 + src/interpreter/mod.rs | 611 +++++++++++++----- src/parser/expr/primary.rs | 55 ++ src/parser/stmt/io.rs | 34 +- src/parser/stmt/web.rs | 15 +- src/typechecker/mod.rs | 220 +++++-- .../ambiguous_write_branch_typecheck_test.rs | 44 ++ .../concurrent_disconnect_paths_burst_test.rs | 163 ++++- .../dropped_interpret_server_cleanup_test.rs | 73 +++ tests/flush_action_backcompat_test.rs | 47 ++ tests/outbound_stream_open_expiry_test.rs | 30 +- tests/outbound_stream_reaper_race_test.rs | 193 ++++++ tests/response_stream_backpressure_test.rs | 32 +- tests/write_web_postfix_test.rs | 59 ++ 16 files changed, 1399 insertions(+), 271 deletions(-) create mode 100644 Dev diary/2026-07-24-issue-642-p1-followups.md create mode 100644 tests/outbound_stream_reaper_race_test.rs diff --git a/Dev diary/2026-07-24-issue-642-p1-followups.md b/Dev diary/2026-07-24-issue-642-p1-followups.md new file mode 100644 index 00000000..c12cc61a --- /dev/null +++ b/Dev diary/2026-07-24-issue-642-p1-followups.md @@ -0,0 +1,57 @@ +# Dev Diary — 2026-07-24: issue #642 PR #641 follow-up P1s + +Follow-up to the exact-head re-review of #641 (`b25aed57`). CI was green but five +P1 lifecycle/compatibility blockers remained. Risk class **R3** (concurrency, +cancellation, lifecycle, streaming, compatibility). + +## P1.1 — request-local failures must not stop the concurrent server + +- Sticky `accepted_request` on `RunState` / `IsolatedHandler` output. +- Concurrent loop only feeds structural pre-request failures into the 256-breaker. +- `Cancelled`, `Timeout` (finite `wait for request`), and any post-accept error/panic + are non-structural. +- `wait for request ... with timeout` expiry uses `ErrorKind::Timeout`. +- Missing pending while the handler still owns the request → `Cancelled` (sibling + prune of a disconnected client); duplicate respond when not owned stays General. +- Tests: strengthened `concurrent_disconnect_paths_burst_test` (assert connected + count, 15s drain, pre-streaming-head path, wait-timeout survival). + +## P1.2 — outbound hard-lifetime reaper ownership + +- `StreamSlot` with handle / deadline / expired tombstone / `AbortHandle`. +- Reaper marks expired and drops parked handles; mid-read `put_stream` refuses + reinsertion and returns `Timeout`. +- EOF/error/close abort the reaper timer (bounded open/close cost). +- `outbound_stream_deadline` clamps extreme `u64` config (no Instant panic). +- Tests: `outbound_stream_reaper_race_test` (active-read near deadline + rapid + open/close). + +## P1.3 — ambiguous `write line|chunk` soundness + +- Typechecker validates definedness + payload for the concrete branch; gradual + targets validate **both** branches. +- `main loop` / `forever` typecheck push a scope; `start streaming response` + always binds `ResponseStream`. +- Analyzer walks `PropertyAccess` in the ambiguous-write parallel walker. +- Tests: one-sided undefined classic lead, main-loop list payload, property access. + +## P1.4 — merged operands match ordinary expression grammar + +- `parse_trailing_postfix` gains direct-integer and `at` indexing. +- Shared `parse_merged_operand_from_lead` for write, `content type`, and `headers`. +- Tests: `at` / integer indexing, `content type mime_type of path`. + +## P1.5 — full `flush` expression-statement fallback + +- Full merged name bound as any value (zero-arg action, non-zero-arg function, + overloaded without zero-arg, non-callable) → old expression-statement behavior. +- Only unbound names fall through to stream flush. +- Analyzer/typechecker stay aligned. +- Tests: non-callable binding + parameterized action without zero-arg. + +## R3 test strength + +- Backpressure test asserts error kind + lower timing bound. +- Open-expiry test asserts setup success + lower timing bound. +- `dropped_interpret_server_cleanup_test` covers pending-request 500 without + streaming head. diff --git a/src/analyzer/mod.rs b/src/analyzer/mod.rs index a2a742ac..c208f084 100644 --- a/src/analyzer/mod.rs +++ b/src/analyzer/mod.rs @@ -3757,8 +3757,27 @@ impl Analyzer { self.analyze_ambiguous_write(&v.value, &f.value, line, column); } } + ( + Expression::PropertyAccess { + object: vo, + property: vp, + .. + }, + Expression::PropertyAccess { + object: fo, + property: fp, + .. + }, + ) if vp == fp => { + // `write line missing.field to ...` — walk the object under both + // readings so a one-sided undefined lead is not skipped solely + // because PropertyAccess was absent from the parallel walker. + self.analyze_ambiguous_write(vo, fo, line, column); + } // Reached a differing leaf — the lead. Report only when NEITHER // reading resolves, so neither valid interpretation is rejected. + // Branch-specific one-sided undefined leads are enforced by the + // typechecker against the concrete target branch (issue #642). (Expression::Variable(sn, ..), Expression::Variable(fal, ..)) if !self.name_is_defined(sn) && !self.name_is_defined(fal) => { @@ -3768,6 +3787,20 @@ impl Analyzer { column, ); } + // PropertyAccess leaf whose object is a Variable: treat the object + // name as the lead (e.g. `missing.field` vs `line missing.field`). + ( + Expression::PropertyAccess { + object: vo, + .. + }, + Expression::PropertyAccess { + object: fo, + .. + }, + ) => { + self.analyze_ambiguous_write(vo, fo, line, column); + } // A diverging, non-decomposable shape (a call-based desugaring where the // lead is buried): defer to runtime rather than risk a false positive. _ => {} diff --git a/src/config.rs b/src/config.rs index 7e093e37..6e105a99 100644 --- a/src/config.rs +++ b/src/config.rs @@ -830,6 +830,10 @@ fn parse_config_text(config: &mut WflConfig, text: &str, file: &Path) { "outbound_stream_max_seconds" => match value.parse::() { Ok(secs) => { // 0 disables the total cap (the documented sentinel). + // Extreme values are accepted into config but clamped when + // converted to an Instant deadline (see interpreter + // `outbound_stream_deadline`) so Instant arithmetic cannot + // panic. config.outbound_stream_max_seconds = secs; log::debug!( "Loaded outbound_stream_max_seconds: {secs} from {}", diff --git a/src/interpreter/mod.rs b/src/interpreter/mod.rs index 7ce7279f..57a3be51 100644 --- a/src/interpreter/mod.rs +++ b/src/interpreter/mod.rs @@ -822,6 +822,12 @@ struct RunState { /// in-flight upstream request — so an abandoned proxy read never leaks an /// upstream connection or handle past the handler's lifetime. open_http_streams: Vec, + /// Sticky: this handler successfully dequeued at least one request via + /// `wait for request`. Used by the concurrent main loop to distinguish + /// structural pre-request failures (feed the consecutive-failure breaker) + /// from request-local outcomes (must never tear the server down because one + /// accepted request failed). Survives respond clearing `open_pending_requests`. + accepted_request: bool, } impl RunState { @@ -845,6 +851,10 @@ impl RunState { /// `inner` is a boxed handler future (already wrapped in `catch_unwind`); a /// panic therefore surfaces as `Poll::Ready` and the swap-back still runs, /// leaving the interpreter's scratch fields restored for the next sibling. +/// +/// The wrapper's output is `(inner_output, accepted_request)` so the concurrent +/// loop can classify request-local vs structural failures after the handler's +/// run state has been swapped out (and its pending list drained by `Drop`). struct IsolatedHandler<'a, T> { interp: &'a Interpreter, state: RunState, @@ -852,16 +862,24 @@ struct IsolatedHandler<'a, T> { } impl<'a, T> std::future::Future for IsolatedHandler<'a, T> { - type Output = T; + type Output = (T, bool); - fn poll(self: std::pin::Pin<&mut Self>, cx: &mut std::task::Context<'_>) -> std::task::Poll { + fn poll( + self: std::pin::Pin<&mut Self>, + cx: &mut std::task::Context<'_>, + ) -> std::task::Poll<(T, bool)> { // Every field is `Unpin` (`&`, `RunState`, and `Pin>`), so the // wrapper itself is `Unpin` and `get_mut` is sound. let this = self.get_mut(); this.interp.swap_run_state(&mut this.state); let result = this.inner.as_mut().poll(cx); this.interp.swap_run_state(&mut this.state); - result + match result { + std::task::Poll::Ready(value) => { + std::task::Poll::Ready((value, this.state.accepted_request)) + } + std::task::Poll::Pending => std::task::Poll::Pending, + } } } @@ -922,7 +940,11 @@ impl Drop for OutboundStreamCleanup { && let Ok(mut map) = self.io_client.stream_handles.try_lock() { for id in &http_ids { - map.remove(id); + if let Some(mut slot) = map.remove(id) + && let Some(abort) = slot.reaper_abort.take() + { + abort.abort(); + } } } @@ -1363,6 +1385,9 @@ pub struct Interpreter { /// future is dropped/cancelled before its normal exit sites run (see /// `OutboundStreamCleanup`). open_http_streams: Rc>>, + /// Sticky per-handler flag: at least one request was dequeued and parked. + /// Part of `RunState` (swapped per poll); see `RunState::accepted_request`. + accepted_request: Cell, #[allow(dead_code)] // Used for future security features config: Arc, // Configuration for security and other settings current_source_file: RefCell>, // Currently executing source file (for path resolution) @@ -1662,14 +1687,54 @@ pub struct IoClient { db_handles: Mutex>, next_db_id: Mutex, /// Live outbound streaming response bodies, keyed by handle id - /// ("httpstream1", ...). See [`HttpStreamHandle`]. Behind an `Arc` so the - /// per-stream absolute-lifetime reaper (spawned in `open_http_stream`) can - /// share it and drop an expired handle in real time, independent of reads. - stream_handles: Arc>>, + /// ("httpstream1", ...). See [`StreamSlot`] / [`HttpStreamHandle`]. Behind + /// an `Arc` so the per-stream absolute-lifetime reaper (spawned in + /// `open_http_stream`) can share it and expire a handle in real time, + /// independent of reads — including while a body read owns the handle. + stream_handles: Arc>>, next_stream_id: Mutex, config: Arc, } +/// Hard ceiling on `outbound_stream_max_seconds` when converting to an +/// `Instant` deadline. Extreme `u64` values must not panic +/// `Instant::now() + Duration::from_secs(secs)` (which can overflow). +const MAX_OUTBOUND_STREAM_DEADLINE_SECS: u64 = 365 * 24 * 60 * 60; // 1 year + +/// Compute the absolute stream deadline from a configured second cap. +/// `0` is the documented sentinel for "no absolute total cap". Values above +/// [`MAX_OUTBOUND_STREAM_DEADLINE_SECS`] are clamped; overflow uses +/// `checked_add` and falls back to "no cap" rather than panicking. +fn outbound_stream_deadline(secs: u64) -> Option { + if secs == 0 { + return None; + } + let capped = secs.min(MAX_OUTBOUND_STREAM_DEADLINE_SECS); + Instant::now().checked_add(Duration::from_secs(capped)) +} + +/// Per-handle shared lifecycle for an outbound stream. +/// +/// Reads take the inner [`HttpStreamHandle`] out for the duration of the await +/// (so the global map lock is not held across the network). The slot stays in +/// the map so the absolute-lifetime reaper can still mark expiry while the +/// read owns the handle — and `put_stream` refuses reinsertion after expiry, +/// preserving `Timeout` as the terminal reason instead of a silent revive. +struct StreamSlot { + /// The live body handle. `None` while a body read owns it, or after + /// expiry/close has dropped it. + handle: Option, + /// Absolute deadline for the whole stream (`None` = no absolute total cap). + deadline: Option, + /// Set by the reaper (or by take/put noticing the deadline) so the next + /// read surfaces a typed `Timeout` rather than "unknown/already closed". + expired: bool, + /// Abort handle for the reaper timer. Cancelled on EOF, error, or explicit + /// close so rapid open/close cycles do not accumulate sleeping tasks for + /// the full configured cap. + reaper_abort: Option, +} + /// A live, parked outbound streaming response body. /// /// The status and headers were already handed to the WFL program by @@ -1940,10 +2005,9 @@ impl IoClient { // measured from when the stream is opened". The head phase below is then // bounded by the remaining time to that deadline as well as the idle // timeout, so a stalled connect/header handshake cannot outlive the total. - let total_deadline = match self.config.outbound_stream_max_seconds { - 0 => None, // sentinel: no absolute total cap - secs => Some(Instant::now() + Duration::from_secs(secs)), - }; + // `outbound_stream_deadline` clamps extreme config values so Instant + // arithmetic cannot panic. + let total_deadline = outbound_stream_deadline(self.config.outbound_stream_max_seconds); let idle_timeout = Duration::from_secs(self.config.timeout_seconds.max(1)); let configured_timeout = match total_deadline { Some(deadline) => { @@ -2010,55 +2074,138 @@ impl IoClient { *next_id += 1; id }; - self.stream_handles - .lock() - .await - .insert(handle_id.clone(), handle); // Enforce `outbound_stream_max_seconds` as a TRUE absolute lifetime, not a // read-triggered one: a handler that opens a stream and then parks (or does // other work) without reading would otherwise keep the upstream connection - // alive past the cap, since the deadline is only re-checked on the next - // read. Spawn a reaper that drops the handle from the shared map when the - // absolute deadline elapses; dropping it drops the reqwest body stream and - // closes the upstream connection. Reads that happen first take the handle - // out of the map, so the reaper's later `remove` is then a harmless no-op; - // and a stream closed early leaves only a cheap parked timer (bounded by the - // cap) that no-ops when it fires. - if let Some(deadline) = total_deadline { + // alive past the cap. Spawn a reaper that marks the shared slot expired + // (and drops any parked handle) when the absolute deadline elapses — + // including while a body read owns the handle. The AbortHandle is stored + // on the slot so EOF/error/close cancels the timer immediately. + let reaper_abort = if let Some(deadline) = total_deadline { let handles = Arc::clone(&self.stream_handles); let reap_id = handle_id.clone(); - tokio::spawn(async move { + let join = tokio::spawn(async move { let remaining = deadline.saturating_duration_since(Instant::now()); tokio::time::sleep(remaining).await; - handles.lock().await.remove(&reap_id); + let mut map = handles.lock().await; + if let Some(slot) = map.get_mut(&reap_id) { + slot.expired = true; + // Drop the live handle if it is parked (not currently mid-read); + // if a read owns it, put_stream will refuse reinsertion. + slot.handle = None; + slot.reaper_abort = None; + } }); - } + Some(join.abort_handle()) + } else { + None + }; + + self.stream_handles.lock().await.insert( + handle_id.clone(), + StreamSlot { + handle: Some(handle), + deadline: total_deadline, + expired: false, + reaper_abort, + }, + ); Ok((status, response_headers, handle_id)) } - /// Remove a stream handle from the map so a body read can await without - /// holding the global handle lock across the network. Errors if the handle - /// is unknown (already closed, or forged). + /// Abort the reaper (if any) and remove the slot, dropping any remaining + /// handle. Used on EOF, error, explicit close, and handler-exit cleanup. + async fn finish_stream_slot(&self, handle_id: &str) -> bool { + let mut map = self.stream_handles.lock().await; + if let Some(mut slot) = map.remove(handle_id) { + if let Some(abort) = slot.reaper_abort.take() { + abort.abort(); + } + // Dropping `slot.handle` cancels the upstream if still present. + true + } else { + false + } + } + + /// Remove a stream handle from its slot so a body read can await without + /// holding the global handle lock across the network. The slot remains so + /// the reaper can still mark expiry mid-read. Errors if the handle is + /// unknown, already closed, or already expired (`Timeout`). async fn take_stream(&self, handle_id: &str) -> Result { - self.stream_handles - .lock() - .await - .remove(handle_id) - .ok_or_else(|| { - HttpClientError::Request(format!( + let mut map = self.stream_handles.lock().await; + // Peek expiry without holding a long-lived mut borrow that blocks remove. + let past_deadline = match map.get(handle_id) { + None => { + return Err(HttpClientError::Request(format!( "Unknown or already-closed stream handle '{handle_id}'" - )) - }) + ))); + } + Some(slot) => { + slot.expired + || slot + .deadline + .is_some_and(|d| d.saturating_duration_since(Instant::now()).is_zero()) + } + }; + if past_deadline { + if let Some(mut slot) = map.remove(handle_id) + && let Some(abort) = slot.reaper_abort.take() + { + abort.abort(); + } + return Err(HttpClientError::Timeout { + seconds: self.config.outbound_stream_max_seconds, + }); + } + match map.get_mut(handle_id).and_then(|s| s.handle.take()) { + Some(handle) => Ok(handle), + None => Err(HttpClientError::Request(format!( + "Unknown or already-closed stream handle '{handle_id}'" + ))), + } } - /// Return a still-open stream handle to the map after a body read. - async fn put_stream(&self, handle_id: &str, handle: HttpStreamHandle) { - self.stream_handles - .lock() - .await - .insert(handle_id.to_string(), handle); + /// Return a still-open stream handle to its slot after a body read. + /// Refuses reinsertion at/after the absolute deadline (or if the reaper + /// already marked the slot expired), dropping the handle and surfacing + /// `Timeout` so a ready chunk cannot revive an expired stream. + async fn put_stream( + &self, + handle_id: &str, + handle: HttpStreamHandle, + ) -> Result<(), HttpClientError> { + let mut map = self.stream_handles.lock().await; + let past_deadline = match map.get(handle_id) { + None => { + // Slot was fully removed (close/finish raced) — drop the handle. + return Ok(()); + } + Some(slot) => { + slot.expired + || slot + .deadline + .is_some_and(|d| d.saturating_duration_since(Instant::now()).is_zero()) + } + }; + if past_deadline { + if let Some(mut slot) = map.remove(handle_id) + && let Some(abort) = slot.reaper_abort.take() + { + abort.abort(); + } + // Drop `handle` by not inserting it. + drop(handle); + return Err(HttpClientError::Timeout { + seconds: self.config.outbound_stream_max_seconds, + }); + } + if let Some(slot) = map.get_mut(handle_id) { + slot.handle = Some(handle); + } + Ok(()) } /// Pull one network chunk into `handle.buffer`, bounded by the per-chunk @@ -2137,8 +2284,9 @@ impl IoClient { } /// Pull the next raw byte chunk from a streaming response. Returns - /// `Ok(None)` at clean end of stream (handle is dropped). On error or EOF - /// the handle is not re-inserted, so the upstream request is released. + /// `Ok(None)` at clean end of stream (handle is finished). On error or EOF + /// the slot is removed and the reaper aborted so the upstream is released + /// and no sleeping timer remains. async fn next_chunk( &self, handle_id: &str, @@ -2149,24 +2297,35 @@ impl IoClient { // Enforce the absolute stream lifetime before serving ANY bytes — even // ones already buffered by a prior read — so `outbound_stream_max_seconds` // is a true absolute lifetime, not merely a per-network-read bound. On - // expiry the handle is not re-inserted, so the upstream request is dropped. - self.check_stream_deadline(&handle)?; + // expiry the slot is finished, so the upstream request is dropped. + if let Err(e) = self.check_stream_deadline(&handle) { + let _ = self.finish_stream_slot(handle_id).await; + return Err(e); + } // Any bytes buffered by a prior `next line` are served first. if !handle.buffer.is_empty() { let chunk = std::mem::take(&mut handle.buffer); - self.put_stream(handle_id, handle).await; + self.put_stream(handle_id, handle).await?; return Ok(Some(chunk)); } match self.stream_pull(&mut handle, &budget).await { Ok(true) => { let chunk = std::mem::take(&mut handle.buffer); - self.put_stream(handle_id, handle).await; + self.put_stream(handle_id, handle).await?; Ok(Some(chunk)) } - Ok(false) => Ok(None), // clean EOF: drop the handle - Err(e) => Err(e), // error/timeout: drop the handle (cancels upstream) + Ok(false) => { + // clean EOF: drop the slot + abort reaper + let _ = self.finish_stream_slot(handle_id).await; + Ok(None) + } + Err(e) => { + // error/timeout: drop the slot (cancels upstream + aborts reaper) + let _ = self.finish_stream_slot(handle_id).await; + Err(e) + } } } @@ -2184,7 +2343,10 @@ impl IoClient { // Enforce the absolute stream lifetime before serving a buffered line // (a prior read may have buffered several lines); on expiry the handle // is dropped, cancelling the upstream. See `next_chunk`. - self.check_stream_deadline(&handle)?; + if let Err(e) = self.check_stream_deadline(&handle) { + let _ = self.finish_stream_slot(handle_id).await; + return Err(e); + } if let Some(pos) = handle.buffer.iter().position(|&b| b == b'\n') { let mut line: Vec = handle.buffer.drain(..=pos).collect(); @@ -2192,13 +2354,14 @@ impl IoClient { if line.last() == Some(&b'\r') { line.pop(); // drop paired '\r' (CRLF) } - self.put_stream(handle_id, handle).await; + self.put_stream(handle_id, handle).await?; return Ok(Some(String::from_utf8_lossy(&line).into_owned())); } if handle.done { // No newline left. Emit any final unterminated line, then EOF. if handle.buffer.is_empty() { + let _ = self.finish_stream_slot(handle_id).await; return Ok(None); // drop the exhausted handle } let mut line = std::mem::take(&mut handle.buffer); @@ -2208,20 +2371,24 @@ impl IoClient { // Re-insert the now-drained (done, empty) handle so the *next* // read cleanly returns `nothing` instead of erroring on a // missing handle. - self.put_stream(handle_id, handle).await; + self.put_stream(handle_id, handle).await?; return Ok(Some(String::from_utf8_lossy(&line).into_owned())); } // Need more bytes to find a newline. - self.stream_pull(&mut handle, &budget).await?; + if let Err(e) = self.stream_pull(&mut handle, &budget).await { + let _ = self.finish_stream_slot(handle_id).await; + return Err(e); + } } } /// Close a streaming response handle if present. Dropping the handle - /// cancels the in-flight upstream request. Returns whether a handle was - /// found. Idempotent: closing an unknown/already-closed handle is a no-op. + /// cancels the in-flight upstream request and aborts its reaper timer. + /// Returns whether a slot was found. Idempotent: closing an + /// unknown/already-closed handle is a no-op. async fn close_stream(&self, handle_id: &str) -> bool { - self.stream_handles.lock().await.remove(handle_id).is_some() + self.finish_stream_slot(handle_id).await } /// Send a request and consume its body without ever buffering more than the @@ -3549,6 +3716,7 @@ impl Interpreter { open_response_streams: Rc::new(RefCell::new(Vec::new())), open_pending_requests: Rc::new(RefCell::new(Vec::new())), open_http_streams: Rc::new(RefCell::new(Vec::new())), + accepted_request: Cell::new(false), next_response_stream_id: std::cell::Cell::new(1), config, current_source_file: RefCell::new(None), // No source file initially @@ -4113,6 +4281,8 @@ impl Interpreter { &mut *self.open_http_streams.borrow_mut(), &mut state.open_http_streams, ); + let accepted = self.accepted_request.replace(state.accepted_request); + state.accepted_request = accepted; } /// Drop each outbound streaming handle whose id is in `ids` from @@ -4127,7 +4297,11 @@ impl Interpreter { } if let Ok(mut map) = self.io_client.stream_handles.try_lock() { for id in ids { - map.remove(id); + if let Some(mut slot) = map.remove(id) + && let Some(abort) = slot.reaper_abort.take() + { + abort.abort(); + } } } } @@ -4283,6 +4457,68 @@ impl Interpreter { self.close_response_streams(&ids); } + /// Take a pending response sender into an RAII completion guard for + /// `respond` / `start streaming response`. + /// + /// Ownership is checked against `open_pending_requests` *before* clearing the + /// id: a sibling `wait for request` may globally prune a closed (client- + /// disconnected) sender, so the map entry is missing while this handler still + /// owns the request. That path is `ErrorKind::Cancelled`. A missing entry when + /// the handler no longer owns the id (duplicate respond after a successful + /// one, or a forged request id) remains a general error. + async fn take_pending_response_completion( + &self, + request_id: &str, + line: usize, + column: usize, + ) -> Result { + let was_owned = self + .open_pending_requests + .borrow() + .iter() + .any(|id| id == request_id); + let pending_entry = { + let mut pending = self.pending_responses.borrow_mut(); + pending.remove(request_id) + }; + // Answered (or definitively cancelled) now: drop it from the handler's + // unanswered-request tracking so the exit-time 500 fallback skips it. + self.open_pending_requests + .borrow_mut() + .retain(|id| id != request_id); + match pending_entry { + // The admission slot is released by the transport task when it + // finishes delivering this response (or on its timeout), so the + // completion guard carries only the response channel. + Some(entry) => match entry.sender.lock().await.take() { + Some(sender) => Ok(ResponseCompletion { + sender: Some(sender), + }), + None => Err(RuntimeError::new( + "Response already sent for this request".to_string(), + line, + column, + )), + }, + None if was_owned => { + // Sibling prune removed a closed sender, or the entry otherwise + // vanished while this handler still owned the request — treat as + // client disconnect / cooperative cancellation. + Err(RuntimeError::with_kind( + "Client disconnected before the response was sent".to_string(), + line, + column, + ErrorKind::Cancelled, + )) + } + None => Err(RuntimeError::new( + "Request ID not found - response may have already been sent".to_string(), + line, + column, + )), + } + } + /// Answer 500 for each request id in `ids` that is still unanswered (its /// sender is still parked in `pending_responses`). A request the handler /// already answered is gone from the map, so its `remove` is a no-op — @@ -4375,7 +4611,10 @@ impl Interpreter { // With cap >= 1 the set is never empty, so `next()` never returns a // `Ready(None)` that would busy-spin the loop. match futs.next().await { - Some(Ok(Ok((value, flow)))) => { + // `IsolatedHandler` yields `(inner, accepted_request)` so we can + // tell request-local outcomes from structural pre-request failures + // after the handler's run state (and open-pending list) is gone. + Some((Ok(Ok((value, flow))), _accepted)) => { // A completed iteration (request handled) — not a failure. consecutive_failures = 0; last_value = value; @@ -4388,44 +4627,63 @@ impl Interpreter { ControlFlow::Continue | ControlFlow::None => {} } } - // A client disconnect cancelled the handler cooperatively. That is - // an EXPECTED external event, not a fault — releasing this handler - // must not feed the structural consecutive-failure breaker (else a - // burst of disconnects would back off and eventually tear the loop - // down, turning "the browser hung up" into a denial of service). - // The handler's owned streams are already closed on unwind; leave - // the failure counter untouched (a disconnect is neither progress - // nor failure) and keep serving. - Some(Ok(Err(err))) if err.kind == ErrorKind::Cancelled => { - log::debug!("concurrent main loop: handler cancelled (client disconnected)"); - } - // A handler returned a runtime error: its request (if it took one) - // is answered 500 by the ResponseCompletion drop guard. Log and - // keep the server running instead of tearing it down. - Some(Ok(Err(err))) => { - log::warn!("concurrent main loop: handler error: {err}"); - if self - .backoff_or_break_concurrent( - &mut consecutive_failures, - MAX_CONSECUTIVE_FAILURES, - ) - .await - { - break; + // Expected / request-local outcomes must NEVER feed the structural + // consecutive-failure breaker: + // - `Cancelled`: client disconnect (cooperative cancellation) + // - `Timeout`: finite `wait for request ... with timeout` expiry + // (healthy idle server, not a hot-spin) + // - any error from a handler that already accepted a request + // (upstream/network/response failure is local to that request) + // + // Structural pre-request failures only (channel closed, missing + // server, deterministic bad expression before `wait`, …) can + // hot-spin the loop if not backstopped — those feed the breaker. + Some((Ok(Err(err)), accepted)) => { + let non_structural = accepted + || err.kind == ErrorKind::Cancelled + || err.kind == ErrorKind::Timeout; + if non_structural { + log::debug!( + "concurrent main loop: non-structural handler outcome \ + (kind={:?}, accepted_request={accepted}): {err}", + err.kind + ); + } else { + log::warn!("concurrent main loop: structural handler error: {err}"); + if self + .backoff_or_break_concurrent( + &mut consecutive_failures, + MAX_CONSECUTIVE_FAILURES, + ) + .await + { + break; + } } } - // A handler panicked: catch_unwind contained it; the request is - // answered 500 by the drop guard. Siblings survive. - Some(Err(_panic)) => { - log::warn!("concurrent main loop: handler panicked; request answered 500"); - if self - .backoff_or_break_concurrent( - &mut consecutive_failures, - MAX_CONSECUTIVE_FAILURES, - ) - .await - { - break; + // Panic after accepting a request is contained and the request is + // answered 500 by the drop guard — request-local, not structural. + // Panic before accepting a request can hot-spin; count it toward + // the structural breaker. + Some((Err(_panic), accepted)) => { + if accepted { + log::warn!( + "concurrent main loop: handler panicked after accepting a request; \ + request answered 500" + ); + } else { + log::warn!( + "concurrent main loop: handler panicked before accepting a request" + ); + if self + .backoff_or_break_concurrent( + &mut consecutive_failures, + MAX_CONSECUTIVE_FAILURES, + ) + .await + { + break; + } } } None => break, // unreachable while cap >= 1; end cleanly if reached @@ -9048,13 +9306,19 @@ impl Interpreter { )); } Err(_) => { - return Err(RuntimeError::new( + // Finite wait expiry is an expected idle-server + // outcome, not a structural fault. Classify as + // `Timeout` so the concurrent loop's consecutive- + // failure breaker does not tear a healthy server + // down after enough empty poll intervals. + return Err(RuntimeError::with_kind( format!( "Timeout waiting for request ({} ms)", duration.as_millis() ), *line, *column, + ErrorKind::Timeout, )); } } @@ -9189,6 +9453,9 @@ impl Interpreter { self.open_pending_requests .borrow_mut() .push(request.id.clone()); + // Sticky: this handler accepted work. Request-local failures from + // here on must not trip the concurrent structural-failure breaker. + self.accepted_request.set(true); Ok((Value::Null, ControlFlow::None)) } @@ -9233,40 +9500,15 @@ impl Interpreter { // answers 500, so the request is always resolved instead of // hanging until its timeout; a successful respond disarms it via // `take_sender`. - let pending_entry = { - let mut pending = self.pending_responses.borrow_mut(); - pending.remove(&request_id) - }; - // Answered now: drop it from the handler's unanswered-request - // tracking so the exit-time 500 fallback skips it. - self.open_pending_requests - .borrow_mut() - .retain(|id| id != &request_id); - let mut completion = match pending_entry { - // The admission slot is released by the transport task when it - // finishes delivering this response (or on its timeout), so the - // completion guard carries only the response channel. - Some(entry) => match entry.sender.lock().await.take() { - Some(sender) => ResponseCompletion { - sender: Some(sender), - }, - None => { - return Err(RuntimeError::new( - "Response already sent for this request".to_string(), - *line, - *column, - )); - } - }, - None => { - return Err(RuntimeError::new( - "Request ID not found - response may have already been sent" - .to_string(), - *line, - *column, - )); - } - }; + // + // Ownership is checked BEFORE removing the id from + // `open_pending_requests`: a sibling `wait for request` may have + // globally pruned a closed (disconnected) sender, leaving the + // entry missing while this handler still owns the request. That + // is cooperative cancellation, not a duplicate-response fault. + let mut completion = self + .take_pending_response_completion(&request_id, *line, *column) + .await?; // Evaluate response content. Binary values are carried through // as raw bytes so fonts/images/etc. serve losslessly; text and @@ -9485,38 +9727,12 @@ impl Interpreter { // Take the oneshot into an RAII guard up front: an early error // while evaluating status/content type/headers still resolves - // the client with 500 instead of hanging. - let pending_entry = { - let mut pending = self.pending_responses.borrow_mut(); - pending.remove(&request_id) - }; - // Answered now (streaming head about to be committed): drop it - // from the handler's unanswered-request tracking. - self.open_pending_requests - .borrow_mut() - .retain(|id| id != &request_id); - let mut completion = match pending_entry { - Some(entry) => match entry.sender.lock().await.take() { - Some(sender) => ResponseCompletion { - sender: Some(sender), - }, - None => { - return Err(RuntimeError::new( - "Response already sent for this request".to_string(), - *line, - *column, - )); - } - }, - None => { - return Err(RuntimeError::new( - "Request ID not found - response may have already been sent" - .to_string(), - *line, - *column, - )); - } - }; + // the client with 500 instead of hanging. Same ownership rule as + // `respond`: missing while still owned => Cancelled (sibling + // prune of a disconnected client), not a structural fault. + let mut completion = self + .take_pending_response_completion(&request_id, *line, *column) + .await?; let status_code = match status { Some(expr) => { @@ -9870,20 +10086,31 @@ impl Interpreter { column, } => { // Backward compatibility: before `flush` was a streaming command, - // `flush cache` was an expression statement that auto-invoked a - // zero-argument action named `flush cache`. If such an action is - // defined, call it — a pre-existing program that named an action - // `flush ` must keep working rather than being reinterpreted as - // a flush of a stream ``. Only the bare-identifier form carries a - // fallback (see `parse_flush_stream`). + // a bare `flush cache` was an ordinary expression statement that + // evaluated the full merged name. Preserve the COMPLETE old + // expression-statement fallback when the full name resolves + // (issue #642) — not only the zero-argument action happy path: + // - zero-arg Function / Overloaded → call it + // - any other bound value (number, text, non-zero-arg overload, …) + // → evaluate-and-discard (old expression-statement no-op success) + // Only when the full name is unbound fall through to stream flush. + // Only the bare-identifier form carries a fallback (see + // `parse_flush_stream`). if let Some(name) = action_fallback { let lookup = env.borrow().get(name); match lookup { Some(Value::Function(func)) => { - return self - .call_function(&func, vec![], *line, *column) - .await - .map(|value| (value, ControlFlow::None)); + if func.params.is_empty() { + return self + .call_function(&func, vec![], *line, *column) + .await + .map(|value| (value, ControlFlow::None)); + } + // Function exists but is not zero-argument: old + // expression-statement evaluation of the function + // value was a no-op success (did not auto-invoke with + // missing args), not a stream error. + return Ok((Value::Null, ControlFlow::None)); } Some(Value::Overloaded(overloaded)) => { if let Some(func) = overloaded @@ -9897,8 +10124,19 @@ impl Interpreter { .await .map(|value| (value, ControlFlow::None)); } + // Overloaded with no zero-argument overload: old + // expression-statement evaluation of the overloaded + // value was a no-op success, not a stream error. + return Ok((Value::Null, ControlFlow::None)); } - _ => {} + Some(_other) => { + // Non-callable full-name binding (e.g. `store flush + // cache as 1` then `flush cache`): evaluate-and- + // discard, matching the pre-streaming expression + // statement. + return Ok((Value::Null, ControlFlow::None)); + } + None => {} } } let handle_id = self @@ -13847,6 +14085,29 @@ impl Interpreter { } } +#[cfg(test)] +mod outbound_stream_deadline_tests { + use super::*; + + #[test] + fn extreme_outbound_stream_max_seconds_does_not_panic() { + // u64::MAX must not panic Instant arithmetic (clamped / checked_add). + let _ = outbound_stream_deadline(u64::MAX); + assert!( + outbound_stream_deadline(0).is_none(), + "0 is the documented sentinel for no absolute total cap" + ); + assert!( + outbound_stream_deadline(1).is_some(), + "a normal positive cap must produce a deadline" + ); + assert!( + outbound_stream_deadline(MAX_OUTBOUND_STREAM_DEADLINE_SECS).is_some(), + "the clamp ceiling itself must still produce a deadline" + ); + } +} + #[cfg(test)] mod header_lookup_tests { use super::*; diff --git a/src/parser/expr/primary.rs b/src/parser/expr/primary.rs index f0113a11..f9716b34 100644 --- a/src/parser/expr/primary.rs +++ b/src/parser/expr/primary.rs @@ -220,6 +220,61 @@ impl<'a> Parser<'a> { }; } } + // Direct integer indexing (`values 0`) — same form as the ordinary + // primary postfix loop. Required so classic + // `write line values 0 to "/tmp/out"` still parses (issue #642). + Token::IntLiteral(index) => { + if matches!( + expr, + Expression::Variable(_, _, _) + | Expression::IndexAccess { .. } + | Expression::FunctionCall { .. } + | Expression::PropertyAccess { .. } + | Expression::MethodCall { .. } + ) { + let index_val = *index; + let (base_line, base_col) = match &expr { + Expression::Variable(_, l, c) + | Expression::IndexAccess { + line: l, column: c, .. + } + | Expression::FunctionCall { + line: l, column: c, .. + } + | Expression::PropertyAccess { + line: l, column: c, .. + } + | Expression::MethodCall { + line: l, column: c, .. + } => (*l, *c), + _ => (line, column), + }; + self.bump_sync(); + expr = Expression::IndexAccess { + collection: Box::new(expr), + index: Box::new(Expression::Literal( + Literal::Integer(index_val), + line, + column, + )), + line: base_line, + column: base_col, + }; + } else { + break; + } + } + // Natural-language indexing (`values at 0`) — same as primary. + Token::KeywordAt => { + self.bump_sync(); + let index = self.parse_expression()?; + expr = Expression::IndexAccess { + collection: Box::new(expr), + index: Box::new(index), + line, + column, + }; + } _ => break, } } diff --git a/src/parser/stmt/io.rs b/src/parser/stmt/io.rs index 629be5b2..cdb54a47 100644 --- a/src/parser/stmt/io.rs +++ b/src/parser/stmt/io.rs @@ -7,9 +7,14 @@ use crate::parser::expr::{BinaryExprParser, ExprParser, PrimaryExprParser}; use std::sync::Arc; impl<'a> Parser<'a> { - /// Parse a `write line|chunk` value from an already-chosen leading operand: - /// an optional ` of ` postfix, then any `with`/operator - /// continuation, exactly as a normal expression value would parse. + /// Parse a merged-command operand from an already-chosen leading identifier: + /// trailing postfix (`[]`, `.field`, `.method()`, `at`, direct-integer index), + /// an optional ` of ` call, then any `with`/operator + /// continuation — exactly as a normal expression value would parse. + /// + /// Shared by `write line|chunk` values and by merged `content type` / `headers` + /// clause operands so they all support the same postfix/call/operator grammar + /// (issue #642). /// /// The ambiguous merged `write line|chunk ...` form has two readings /// (stream: split-off ``; classic file write: whole `line `) @@ -19,12 +24,14 @@ impl<'a> Parser<'a> { /// an `ActionCall`, `is between` duplicates the left, `starts/ends with` and /// the pattern operators build calls), so deriving one AST from the other by /// leaf-swapping silently corrupted the classic reading. - fn parse_write_value_from_lead(&mut self, lead: Expression) -> Result { - // The lexer merges `write` with the operand identifier and leaves any - // bracket-index / dotted-property accessors as following tokens, so compose - // them onto the lead (`write line chunks[0] to out`, - // `write line upstream.status to out`, classic `write line values[0] to - // "/tmp/out"`) instead of leaving them to dangle after the statement. + pub(crate) fn parse_merged_operand_from_lead( + &mut self, + lead: Expression, + ) -> Result { + // The lexer merges the command word with the operand identifier and leaves + // any bracket-index / dotted-property / `at` / integer-index accessors as + // following tokens, so compose them onto the lead instead of leaving them + // to dangle after the statement. let lead = self.parse_trailing_postfix(lead)?; let lead = if matches!(self.cursor.peek().map(|t| &t.token), Some(Token::KeywordOf)) { // Anchor the ` of ` call to the `of` keyword itself, @@ -37,9 +44,7 @@ impl<'a> Parser<'a> { // Parse the `of`-call argument(s) EXACTLY as the primary parser does: // each argument absorbs arithmetic (`fibonacci of n minus 1` means // `fibonacci of (n minus 1)`, not `(fibonacci of n) minus 1`), and - // `and`/`from`/`by`/`length` join multiple arguments — so an `of`-call in - // a `write line|chunk` value parses identically to one in an ordinary - // expression. + // `and`/`from`/`by`/`length` join multiple arguments. let mut arguments = vec![crate::parser::ast::Argument { name: None, value: self.parse_of_call_argument()?, @@ -72,6 +77,11 @@ impl<'a> Parser<'a> { }; self.parse_binary_continuation(lead, 0) } + + /// Alias used by the write-statement parsers. + fn parse_write_value_from_lead(&mut self, lead: Expression) -> Result { + self.parse_merged_operand_from_lead(lead) + } } pub(crate) trait IoParser<'a>: ExprParser<'a> { diff --git a/src/parser/stmt/web.rs b/src/parser/stmt/web.rs index 8acd29f2..c1d98826 100644 --- a/src/parser/stmt/web.rs +++ b/src/parser/stmt/web.rs @@ -448,10 +448,11 @@ impl<'a> WebParser<'a> for Parser<'a> { }; content_type = Some(match merged_rest { Some((rest, (l, c))) => { - // Compose any dangling postfix accessors - // (`content type upstream.headers["content-type"]`). + // Full expression continuation (postfix + `of` + + // operators), same as ordinary `` — e.g. + // `content type mime_type of path` (issue #642). let lead = Expression::Variable(rest, l, c); - self.parse_trailing_postfix(lead)? + self.parse_merged_operand_from_lead(lead)? } None => self.parse_primary_expression()?, }); @@ -478,7 +479,7 @@ impl<'a> WebParser<'a> for Parser<'a> { content_type = Some(self.parse_primary_expression()?); } else { let lead = Expression::Variable(rest.to_string(), id_line, id_column); - content_type = Some(self.parse_trailing_postfix(lead)?); + content_type = Some(self.parse_merged_operand_from_lead(lead)?); } } // `headers ` (bare or merged `headers `). @@ -494,10 +495,10 @@ impl<'a> WebParser<'a> for Parser<'a> { if rest.is_empty() { headers = Some(self.parse_primary_expression()?); } else { - // Compose any dangling postfix accessors so direct - // forwarding like `headers upstream.headers` binds fully. + // Full expression continuation so direct forwarding like + // `headers upstream.headers` and operator/`of` forms bind. let lead = Expression::Variable(rest.to_string(), id_line, id_column); - headers = Some(self.parse_trailing_postfix(lead)?); + headers = Some(self.parse_merged_operand_from_lead(lead)?); } } // A connective directly before `as` just joins the clause list to diff --git a/src/typechecker/mod.rs b/src/typechecker/mod.rs index b79b7504..93f1ab34 100644 --- a/src/typechecker/mod.rs +++ b/src/typechecker/mod.rs @@ -1041,13 +1041,28 @@ impl TypeChecker { ); } } - if !variable_name.is_empty() - && let Some(symbol) = self.analyzer.get_symbol_mut(variable_name) - { + if !variable_name.is_empty() { // A distinct server-response-stream handle type (not a bare // `Map`) so `close out` is accepted without `close` also // type-checking an ordinary user map. - symbol.symbol_type = Some(Type::Custom("ResponseStream".to_string())); + // + // Always bind/refine in the *current* scope: analyzer loop + // scopes are discarded after body analysis, so `out` from + // `start streaming response ... as out` inside `main loop` + // would otherwise be missing here and remain Unknown — + // letting a gradual/file fallback mask an invalid stream + // payload (issue #642). + if let Some(symbol) = self.analyzer.get_symbol_mut(variable_name) { + symbol.symbol_type = Some(Type::Custom("ResponseStream".to_string())); + } else { + self.analyzer.define_or_replace_symbol(Symbol { + name: variable_name.clone(), + kind: SymbolKind::Variable { mutable: true }, + symbol_type: Some(Type::Custom("ResponseStream".to_string())), + line: *_line, + column: *_column, + }); + } } } Statement::StreamWriteStatement { @@ -1071,44 +1086,31 @@ impl TypeChecker { && !self.is_gradual_type(&target_type) { // Concrete response stream: the stream reading is taken. + // Report undefined names on this branch (analyzer may have + // stayed silent because the classic lead alone was defined). + self.check_expression_names_defined(value, *line, *column); let value_type = self.infer_expression_type(value); self.check_streamable_payload(&value_type, *line, *column); } else if matches!(target_type, Type::Text) && has_fallback { // Concrete text path: the classic file-write reading is taken. - // Validate the fallback, not the stream `value` the runtime - // never evaluates here. + // Validate the fallback (including definedness), not the stream + // `value` the runtime never evaluates here. if let Some(fallback) = fallback_content { + self.check_expression_names_defined(fallback, *line, *column); let _ = self.infer_expression_type(fallback); } } else if self.is_gradual_type(&target_type) { // Gradual/unknown target: both readings are viable and the - // runtime decides by the target's runtime type. Accept if EITHER - // reading is well-typed; only report an error when the statement - // is wrong under every interpretation (so a valid file write and - // a valid stream write both pass). Speculative inference rolls - // its emitted errors back. - let stream_ok = { - let checkpoint = self.errors.len(); - let value_type = self.infer_expression_type(value); - let ok = self.errors.len() == checkpoint - && self.is_streamable_payload(&value_type); - self.errors.truncate(checkpoint); - ok - }; - let file_ok = if let Some(fallback) = fallback_content { - let checkpoint = self.errors.len(); + // runtime decides by the target's runtime type. Conservatively + // validate EVERY viable branch (not "accept if either is ok"), + // so a valid file fallback cannot mask an invalid stream + // payload or an undefined stream lead (issue #642). + self.check_expression_names_defined(value, *line, *column); + let value_type = self.infer_expression_type(value); + self.check_streamable_payload(&value_type, *line, *column); + if let Some(fallback) = fallback_content { + self.check_expression_names_defined(fallback, *line, *column); let _ = self.infer_expression_type(fallback); - let ok = self.errors.len() == checkpoint; - self.errors.truncate(checkpoint); - ok - } else { - false - }; - if !stream_ok && !file_ok { - // Broken under both readings: surface the stream reading's - // errors (sub-expression + payload) as the diagnostic. - let value_type = self.infer_expression_type(value); - self.check_streamable_payload(&value_type, *line, *column); } } else { // Concrete non-stream, non-text target (or a text target with no @@ -1130,16 +1132,18 @@ impl TypeChecker { line, column, } => { - // A merged `flush ` may resolve at runtime to a - // zero-argument action named `flush ` (backward - // compatibility — see `parse_flush_stream`). Stay lenient ONLY when - // such an action is actually defined; otherwise this is a stream - // flush and a concrete non-stream target is still a static error - // (`flush n` where `n` is a number). - let is_action_call = action_fallback - .as_ref() - .is_some_and(|name| self.action_signatures(name).is_some()); - if !is_action_call { + // A merged `flush ` may resolve at runtime to the full + // merged name as an ordinary expression statement (backward + // compatibility — see `parse_flush_stream` / issue #642). Stay + // lenient when the full name is bound as an action OR any other + // value; otherwise this is a stream flush and a concrete + // non-stream target is still a static error (`flush n` where + // `n` is a number and there is no binding named `flush n`). + let is_expression_fallback = action_fallback.as_ref().is_some_and(|name| { + self.action_signatures(name).is_some() + || self.analyzer.get_symbol(name).is_some() + }); + if !is_expression_fallback { let target_type = self.infer_expression_type(target); if !self.is_response_stream_target_type(&target_type) { self.type_error( @@ -1619,14 +1623,22 @@ impl TypeChecker { } } Statement::ForeverLoop { body, .. } => { + // Push a scope so bindings introduced in the body (e.g. + // `start streaming response ... as out`) remain visible to later + // statements in the same body for type checking. Analyzer loop + // scopes are discarded after analysis. + self.analyzer.push_scope(); for stmt in body { self.check_statement_types(stmt); } + self.analyzer.pop_scope(); } Statement::MainLoop { body, .. } => { + self.analyzer.push_scope(); for stmt in body { self.check_statement_types(stmt); } + self.analyzer.pop_scope(); } Statement::DisplayStatement { value, .. } => { self.infer_expression_type(value); @@ -4902,6 +4914,130 @@ impl TypeChecker { } } + /// Whether a bare name is known to the typechecker/analyzer scopes (or is a + /// builtin / action parameter / loop counter). Used when validating the + /// concrete `write line|chunk` branch the runtime will select — the analyzer + /// may have stayed silent on a one-sided undefined lead because the other + /// reading was defined (issue #642). + fn name_is_defined_for_write(&self, name: &str) -> bool { + if self.analyzer.get_symbol(name).is_some() + || self.analyzer.get_action_parameters().contains(name) + || Analyzer::is_builtin_function(name) + || name == "count" + || name == "loopcounter" + { + return true; + } + false + } + + /// Walk an expression and report every undefined bare name. Used for the + /// selected (or every viable gradual) `write line|chunk` branch so a missing + /// classic `line ` lead is not accepted just because the stream lead + /// alone exists (and vice versa). + fn check_expression_names_defined( + &mut self, + expression: &Expression, + line: usize, + column: usize, + ) { + match expression { + Expression::Variable(name, l, c) => { + if !self.name_is_defined_for_write(name) { + self.type_error( + format!("Variable '{name}' is not defined"), + None, + None, + *l, + *c, + ); + } + } + Expression::BinaryOperation { left, right, .. } + | Expression::Concatenation { left, right, .. } + | Expression::PatternMatch { + text: left, + pattern: right, + .. + } + | Expression::PatternFind { + text: left, + pattern: right, + .. + } + | Expression::PatternSplit { + text: left, + pattern: right, + .. + } + | Expression::StringSplit { + text: left, + delimiter: right, + .. + } => { + self.check_expression_names_defined(left, line, column); + self.check_expression_names_defined(right, line, column); + } + Expression::UnaryOperation { + expression: inner, .. + } + | Expression::AwaitExpression { + expression: inner, .. + } => { + self.check_expression_names_defined(inner, line, column); + } + Expression::IndexAccess { + collection, index, .. + } => { + self.check_expression_names_defined(collection, line, column); + self.check_expression_names_defined(index, line, column); + } + Expression::PropertyAccess { object, .. } | Expression::MemberAccess { object, .. } => { + self.check_expression_names_defined(object, line, column); + } + Expression::MethodCall { + object, arguments, .. + } => { + self.check_expression_names_defined(object, line, column); + for arg in arguments { + self.check_expression_names_defined(&arg.value, line, column); + } + } + Expression::FunctionCall { + function, + arguments, + .. + } => { + self.check_expression_names_defined(function, line, column); + for arg in arguments { + self.check_expression_names_defined(&arg.value, line, column); + } + } + Expression::ActionCall { arguments, .. } => { + for arg in arguments { + self.check_expression_names_defined(&arg.value, line, column); + } + } + Expression::PatternReplace { + text, + pattern, + replacement, + .. + } => { + self.check_expression_names_defined(text, line, column); + self.check_expression_names_defined(pattern, line, column); + self.check_expression_names_defined(replacement, line, column); + } + Expression::HeaderAccess { request, .. } => { + self.check_expression_names_defined(request, line, column); + } + // Literals and other leaves need no definedness walk. + _ => { + let _ = (line, column); + } + } + } + fn are_types_compatible(&self, target_type: &Type, source_type: &Type) -> bool { #[allow(clippy::only_used_in_recursion)] let _self = self; // Suppress the warning for self parameter diff --git a/tests/ambiguous_write_branch_typecheck_test.rs b/tests/ambiguous_write_branch_typecheck_test.rs index 8c39b7d4..6fe70717 100644 --- a/tests/ambiguous_write_branch_typecheck_test.rs +++ b/tests/ambiguous_write_branch_typecheck_test.rs @@ -88,3 +88,47 @@ fn text_and_binary_payloads_to_a_stream_still_typecheck() { typecheck(code).err() ); } + +#[test] +fn text_target_one_sided_undefined_classic_lead_is_caught() { + // Stream lead `value` is defined; classic lead `line value` is not. Target is + // concrete text, so runtime takes the classic branch — must be a static error + // (issue #642: previously analysis passed because only the stream lead existed). + let code = "store value as \"x\"\n\ + write line value to \"/tmp/wfl_onesided_out\""; + assert!( + typecheck(code).is_err(), + "undefined classic lead on a concrete text target must be a static error" + ); +} + +#[test] +fn main_loop_stream_binding_rejects_list_payload() { + // Inside main loop, `out` must be typed as ResponseStream so a list payload + // is rejected rather than masked by a gradual/file fallback (issue #642). + let code = "listen on port 8080 as s\n\ + main loop:\n\ + \x20\x20\x20\x20wait for request comes in on s as req\n\ + \x20\x20\x20\x20start streaming response to req with status 200 as out\n\ + \x20\x20\x20\x20store items as [1 and 2]\n\ + \x20\x20\x20\x20store line items as \"legacy\"\n\ + \x20\x20\x20\x20write line items to out\n\ + end loop"; + assert!( + typecheck(code).is_err(), + "list payload to a main-loop response stream must be a static error, got: {:?}", + typecheck(code).err() + ); +} + +#[test] +fn property_access_undefined_object_on_text_target_is_caught() { + // Merged lead with a `.field` postfix: stream reading is `upstream.status`, + // classic is `line upstream.status`. Neither object is defined; PropertyAccess + // must not evade definedness on the concrete text-target branch (issue #642). + let code = "write line upstream.status to \"/tmp/wfl_prop_out\""; + assert!( + typecheck(code).is_err(), + "undefined property object on a text target must be a static error" + ); +} diff --git a/tests/concurrent_disconnect_paths_burst_test.rs b/tests/concurrent_disconnect_paths_burst_test.rs index baee3e2b..c68c1540 100644 --- a/tests/concurrent_disconnect_paths_burst_test.rs +++ b/tests/concurrent_disconnect_paths_burst_test.rs @@ -1,14 +1,20 @@ -//! Real-socket regression (maintainer re-review, P1): EVERY transport-confirmed +//! Real-socket regression (maintainer re-review, P1 / #642): EVERY transport-confirmed //! client disconnect must be classified as a cancellation, not a handler failure — -//! including the buffered `respond` send and the streaming-response head/`write` -//! paths, not only a cancelled upstream chunk read. +//! including the buffered `respond` send, the streaming-response head path (before the +//! head is sent), and the streaming write path — not only a cancelled upstream chunk +//! read. //! //! The concurrent loop breaks after `MAX_CONSECUTIVE_FAILURES` (256) consecutive -//! failed handlers. If a disconnect at these paths returns a General runtime error -//! (as before), a burst of >256 disconnects trips that breaker and the server stops -//! serving — turning "the client hung up" into a denial of service. These bursts -//! disconnect after dequeue (before the buffered reply) and after the streaming head -//! (before/at the first write); an unrelated `/ping` must still be served afterward. +//! *structural* failures. Request-local outcomes (disconnects, wait timeouts, errors +//! after a request was accepted) must never feed that breaker. These bursts drive +//! more than 256 disconnects of each kind with no successful request in between; an +//! unrelated `/ping` must still be served afterward. +//! +//! Also: every client that is intended to exercise a path must actually connect and +//! reach that lifecycle point (no silent early-return that leaves the burst under the +//! breaker threshold), and the test waits long enough for every handler result to be +//! consumed before probing `/ping` (so a General-classified disconnect cannot race +//! past a premature success that resets the counter). use std::sync::Arc; use std::sync::atomic::{AtomicUsize, Ordering}; @@ -24,6 +30,10 @@ mod common; /// More than the breaker threshold (256), with NO successful request in between. const DISCONNECT_BURST: usize = 270; const CLIENT_CONCURRENCY: usize = 40; +/// Upper bound on how long the General-failure backoff would take to consume 256 +/// failures (~11.5s) plus handler work. Waiting past that guarantees the counter +/// would have tripped if any of the disconnects were misclassified as structural. +const DRAIN_AFTER_BURST: Duration = Duration::from_secs(15); fn start_proxy_server(code: String) -> std::thread::JoinHandle<()> { std::thread::spawn(move || { @@ -51,59 +61,84 @@ async fn wait_for_server(port: u16) { } /// Connect, send the request so the server enqueues and dequeues it, briefly hold so -/// the handler is inside its pre-reply work, then disconnect. `read_head` waits for -/// the streaming response head first (so the disconnect lands after the head, at the -/// write path) when the route streams. -async fn fire_disconnect(port: u16, path: &str, read_head: bool) { +/// the handler is inside its pre-reply work, then disconnect. +/// +/// - `read_head == false`: disconnect after a short hold so the disconnect lands +/// before `respond` / before the streaming head is sent. +/// - `read_head == true`: wait for the streaming response head first, so the +/// disconnect lands after the head, at the write path. +/// +/// Returns whether the client successfully connected and sent the request (so the +/// burst can assert every intended disconnect actually reached the server). +async fn fire_disconnect(port: u16, path: &str, read_head: bool) -> bool { let Ok(mut sock) = tokio::net::TcpStream::connect(("127.0.0.1", port)).await else { - return; + return false; }; let req = format!("GET {path} HTTP/1.1\r\nHost: 127.0.0.1\r\n\r\n"); if sock.write_all(req.as_bytes()).await.is_err() { - return; + return false; + } + if sock.flush().await.is_err() { + return false; } - let _ = sock.flush().await; if read_head { let mut acc = Vec::new(); let mut tmp = [0u8; 256]; + let mut saw_head = false; loop { match tokio::time::timeout(Duration::from_secs(5), sock.read(&mut tmp)).await { Ok(Ok(0)) | Err(_) | Ok(Err(_)) => break, Ok(Ok(n)) => { acc.extend_from_slice(&tmp[..n]); if acc.windows(4).any(|w| w == b"\r\n\r\n") { + saw_head = true; break; } } } } + if !saw_head { + return false; + } } else { // Give the server time to enqueue + dequeue the request and enter the - // handler's pre-reply wait, so the disconnect lands before `respond`. + // handler's pre-reply wait, so the disconnect lands before `respond` / + // before the streaming head. tokio::time::sleep(Duration::from_millis(200)).await; } // Drop `sock` -> disconnect. + true } -async fn fire_burst(port: u16, path: &'static str, read_head: bool) { +async fn fire_burst(port: u16, path: &'static str, read_head: bool) -> usize { let sem = Arc::new(Semaphore::new(CLIENT_CONCURRENCY)); - let fired = Arc::new(AtomicUsize::new(0)); + let connected = Arc::new(AtomicUsize::new(0)); let mut tasks = Vec::with_capacity(DISCONNECT_BURST); for _ in 0..DISCONNECT_BURST { let sem = Arc::clone(&sem); - let fired = Arc::clone(&fired); + let connected = Arc::clone(&connected); tasks.push(tokio::spawn(async move { let _permit = sem.acquire().await.expect("semaphore"); - fire_disconnect(port, path, read_head).await; - fired.fetch_add(1, Ordering::Relaxed); + if fire_disconnect(port, path, read_head).await { + connected.fetch_add(1, Ordering::Relaxed); + } })); } for t in tasks { let _ = t.await; } - // Grace so all >256 handlers finish failing (Cancelled, under the fix) before we - // send the first successful request — guaranteeing the failures are consecutive. - tokio::time::sleep(Duration::from_secs(3)).await; + let n = connected.load(Ordering::Relaxed); + assert!( + n > 256, + "expected more than 256 clients to actually connect and reach the intended \ + lifecycle point (so the burst exceeds the structural breaker threshold); \ + only {n} of {DISCONNECT_BURST} succeeded (path={path}, read_head={read_head})" + ); + // Drain past the General-failure backoff window so every intended handler + // result is consumed before `/ping`. If any disconnect were still classified + // as structural General, the breaker would trip during this wait. + tokio::time::sleep(DRAIN_AFTER_BURST).await; + n } async fn assert_ping_survives(port: u16, context: &str) { @@ -140,7 +175,8 @@ async fn shutdown(port: u16, server: std::thread::JoinHandle<()>) { async fn test_disconnect_before_buffered_respond_does_not_kill_the_loop() { let port = common::free_tcp_port(); // `/slow` waits, then responds — the client disconnects during the wait, so the - // buffered `respond` send fails. That must be a cancellation, not a failure. + // buffered `respond` send fails (or the pending entry is sibling-pruned). That + // must be a cancellation, not a structural failure. let code = format!( r#" listen on port {port} as srv @@ -164,7 +200,7 @@ async fn test_disconnect_before_buffered_respond_does_not_kill_the_loop() { ); let server = start_proxy_server(code); wait_for_server(port).await; - fire_burst(port, "/slow", false).await; + let _ = fire_burst(port, "/slow", false).await; assert_ping_survives(port, "buffered-respond disconnect").await; shutdown(port, server).await; } @@ -205,7 +241,80 @@ async fn test_disconnect_before_stream_write_does_not_kill_the_loop() { ); let server = start_proxy_server(code); wait_for_server(port).await; - fire_burst(port, "/stream", true).await; + let _ = fire_burst(port, "/stream", true).await; assert_ping_survives(port, "stream-write disconnect").await; shutdown(port, server).await; } + +#[tokio::test] +async fn test_disconnect_before_streaming_head_does_not_kill_the_loop() { + let port = common::free_tcp_port(); + // Client disconnects *before* the streaming head is sent (no head read). The + // handler parks, then reaches `start streaming response` with a missing/closed + // pending entry — must be Cancelled, not a structural General that trips the + // breaker after >256 instances. + let code = format!( + r#" + listen on port {port} as srv + main loop concurrently: + wait for request comes in on srv as req with timeout 30000 + store p as req["path"] + check if p is equal to "/ping": + respond to req with "pong" + otherwise: + check if p is equal to "/shutdown": + respond to req with "bye" + close server srv + break + otherwise: + wait for 500 milliseconds + start streaming response to req with status 200 and content type "text/plain" as out + write line "late" to out + close out + end check + end check + end loop + "# + ); + let server = start_proxy_server(code); + wait_for_server(port).await; + let _ = fire_burst(port, "/prehead", false).await; + assert_ping_survives(port, "pre-streaming-head disconnect").await; + shutdown(port, server).await; +} + +#[tokio::test] +async fn test_repeated_wait_timeouts_do_not_kill_the_loop() { + let port = common::free_tcp_port(); + // Finite `wait for request ... with timeout` that repeatedly expires with no + // client traffic must not trip the structural breaker. After many idle + // timeouts a real `/ping` must still be served. + let code = format!( + r#" + listen on port {port} as srv + main loop concurrently: + wait for request comes in on srv as req with timeout 50 + store p as req["path"] + check if p is equal to "/ping": + respond to req with "pong" + otherwise: + check if p is equal to "/shutdown": + respond to req with "bye" + close server srv + break + otherwise: + respond to req with "ok" + end check + end check + end loop + "# + ); + let server = start_proxy_server(code); + wait_for_server(port).await; + // Idle long enough for well over 256 consecutive wait timeouts (50ms each; + // concurrency multiplies the rate). 8s >> 256 * structural backoff would + // also have completed if they were misclassified. + tokio::time::sleep(Duration::from_secs(8)).await; + assert_ping_survives(port, "repeated wait timeouts").await; + shutdown(port, server).await; +} diff --git a/tests/dropped_interpret_server_cleanup_test.rs b/tests/dropped_interpret_server_cleanup_test.rs index a9574974..7106e60e 100644 --- a/tests/dropped_interpret_server_cleanup_test.rs +++ b/tests/dropped_interpret_server_cleanup_test.rs @@ -127,3 +127,76 @@ async fn test_dropped_run_closes_server_stream_while_interpreter_stays_alive() { Err(e) => panic!("server join task failed: {e}"), } } + +#[tokio::test] +async fn test_dropped_run_answers_pending_request_with_500() { + // Exercise the dropped-run pending-request 500 branch: handler dequeues a + // request and parks WITHOUT ever responding or starting a streaming response, + // so the pending oneshot is still in `pending_responses` when interpret() is + // dropped. The cleanup guard must answer 500 promptly (issue #642 R3). + let port = common::free_tcp_port(); + let code = format!( + r#" + listen on port {port} as srv + main loop: + wait for request comes in on srv as req with timeout 60000 + wait for 60000 milliseconds + end loop + "# + ); + + let server = std::thread::spawn(move || { + let rt = tokio::runtime::Builder::new_current_thread() + .enable_all() + .build() + .expect("server runtime"); + rt.block_on(async { + let tokens = lex_wfl_with_positions(&code); + let program = Parser::new(&tokens).parse().expect("parse"); + let config = WflConfig { + timeout_seconds: 60, + ..WflConfig::default() + }; + let mut interp = Interpreter::with_config(Arc::new(config)); + { + let fut = interp.interpret(&program); + // Drop after the handler has had time to dequeue and park. + let _ = tokio::time::timeout(Duration::from_secs(2), fut).await; + } + // Keep the interpreter alive so only the drop guard can 500 the request. + tokio::time::sleep(Duration::from_secs(8)).await; + drop(interp); + }); + }); + + wait_for_server(port).await; + + let start = Instant::now(); + let resp = tokio::time::timeout( + Duration::from_secs(6), + reqwest::Client::new() + .get(format!("http://127.0.0.1:{port}/")) + .send(), + ) + .await + .expect("client should not hang waiting for a pending request after interpret() drop") + .expect("request failed"); + let elapsed = start.elapsed(); + + assert_eq!( + resp.status().as_u16(), + 500, + "dropped run must answer the still-pending request with 500, got {}", + resp.status() + ); + assert!( + elapsed < Duration::from_secs(5), + "500 should arrive shortly after the ~2s drop, not at the 60s request timeout; took {elapsed:?}" + ); + + match tokio::task::spawn_blocking(move || server.join()).await { + Ok(Ok(())) => {} + Ok(Err(panic)) => std::panic::resume_unwind(panic), + Err(e) => panic!("server join task failed: {e}"), + } +} diff --git a/tests/flush_action_backcompat_test.rs b/tests/flush_action_backcompat_test.rs index 0fe23c62..35486e3d 100644 --- a/tests/flush_action_backcompat_test.rs +++ b/tests/flush_action_backcompat_test.rs @@ -65,3 +65,50 @@ fn flush_without_a_matching_action_still_errors_as_a_stream_flush() { "a bare `flush cache` with no target must error; output:\n{out}" ); } + +#[test] +fn flush_non_callable_full_name_binding_is_expression_statement() { + // Pre-streaming: `store flush cache as 1` then `flush cache` evaluated the + // variable and completed. Must not try to flush an undefined stream `cache` + // (issue #642). + let src = "store flush cache as 1\n\ + flush cache\n\ + display flush cache\n"; + let (out, code) = run_src(src); + assert_eq!( + code, + Some(0), + "non-callable full-name binding must keep working as an expression statement; output:\n{out}" + ); + assert!( + out.contains('1'), + "expected the bound value to still be readable; output:\n{out}" + ); + assert!( + !out.to_lowercase().contains("stream"), + "`flush cache` must not be reinterpreted as a stream flush; output:\n{out}" + ); +} + +#[test] +fn flush_overloaded_action_without_zero_arg_is_expression_statement() { + // An overloaded `flush cache` with only a one-parameter overload used to be + // an ordinary bare-expression evaluation (no-op success). Must not become a + // stream error (issue #642). + let src = "define action called flush cache with parameters x:\n\ + \x20\x20\x20\x20display x\n\ + end action\n\ + \n\ + flush cache\n\ + display \"OK\"\n"; + let (out, code) = run_src(src); + assert_eq!( + code, + Some(0), + "overloaded flush with no zero-arg overload must not error as a stream flush; output:\n{out}" + ); + assert!( + out.contains("OK"), + "program should reach the trailing display; output:\n{out}" + ); +} diff --git a/tests/outbound_stream_open_expiry_test.rs b/tests/outbound_stream_open_expiry_test.rs index 11f36dff..99266ead 100644 --- a/tests/outbound_stream_open_expiry_test.rs +++ b/tests/outbound_stream_open_expiry_test.rs @@ -61,6 +61,8 @@ async fn test_opened_but_unread_stream_expires_at_the_absolute_cap() { wait for 6000 milliseconds"# ); + // Capture whether setup (open) succeeded and the program completed. + let (result_tx, result_rx) = std::sync::mpsc::channel::>(); let client = std::thread::spawn(move || { let rt = tokio::runtime::Runtime::new().expect("client runtime"); rt.block_on(async { @@ -72,7 +74,12 @@ wait for 6000 milliseconds"# ..WflConfig::default() }; let mut interp = Interpreter::with_config(Arc::new(config)); - let _ = interp.interpret(&program).await; + let result = interp.interpret(&program).await; + let summary = match result { + Ok(_) => Ok(()), + Err(errs) => Err(format!("{errs:?}")), + }; + let _ = result_tx.send(summary); }); }); @@ -83,15 +90,30 @@ wait for 6000 milliseconds"# .expect("upstream close sender dropped"); let elapsed = start.elapsed(); - // ~1s (the cap). If enforcement were still read-triggered, the upstream would - // only close at the 6s program end — so a close well before then proves the - // real-time reaper fired. + // Lower bound: reaper should not fire instantly (setup must succeed and the + // 1s cap must be waited out). Upper bound: well before the 6s program park + // that would mask a missing reaper (issue #642 R3). + assert!( + elapsed >= Duration::from_millis(700), + "upstream should be reaped near the 1s absolute cap, not instantly; took {elapsed:?}" + ); assert!( elapsed < Duration::from_secs(3), "the upstream should be reaped at the ~1s absolute cap, not at program end; \ took {elapsed:?}" ); + let summary = result_rx + .recv_timeout(Duration::from_secs(8)) + .expect("interpreter should finish after the program wait"); + // Opening an unread stream is successful setup; the program parks 6s and + // completes cleanly (the reaper only drops the upstream handle — it does not + // fail an unread stream by itself). + assert!( + summary.is_ok(), + "opened-but-unread stream program should complete after setup; got: {summary:?}" + ); + match tokio::task::spawn_blocking(move || client.join()).await { Ok(Ok(())) => {} Ok(Err(panic)) => std::panic::resume_unwind(panic), diff --git a/tests/outbound_stream_reaper_race_test.rs b/tests/outbound_stream_reaper_race_test.rs new file mode 100644 index 00000000..c37c94ae --- /dev/null +++ b/tests/outbound_stream_reaper_race_test.rs @@ -0,0 +1,193 @@ +//! Real-socket regression (issue #642 P1): the absolute-lifetime reaper and an +//! active body read must share one atomic lifecycle. +//! +//! If the reaper only removes a parked handle, a read that took the handle out +//! for the await can win the read/timeout race and reinsert the expired handle — +//! leaving a live upstream past the documented real-time hard cap. With the fix: +//! the reaper marks a shared slot expired; `put_stream` refuses reinsertion; the +//! next/current outcome surfaces a typed Timeout and the upstream is dropped. + +use std::sync::Arc; +use std::time::{Duration, Instant}; +use tokio::io::{AsyncReadExt, AsyncWriteExt}; +use wfl::Interpreter; +use wfl::config::WflConfig; +use wfl::lexer::lex_wfl_with_positions; +use wfl::parser::Parser; + +/// Upstream: send a valid response head, then stall (never body, never close). +/// Signal when the proxy drops the upstream connection. +async fn spawn_head_then_stall_upstream() -> (u16, tokio::sync::oneshot::Receiver<()>) { + let listener = tokio::net::TcpListener::bind("127.0.0.1:0") + .await + .expect("bind mock upstream"); + let port = listener.local_addr().unwrap().port(); + let (tx, rx) = tokio::sync::oneshot::channel(); + tokio::spawn(async move { + if let Ok((mut sock, _)) = listener.accept().await { + let mut buf = [0u8; 1024]; + let _ = sock.read(&mut buf).await; + let head = "HTTP/1.1 200 OK\r\n\ + Content-Type: text/plain\r\n\ + Transfer-Encoding: chunked\r\n\r\n"; + let _ = sock.write_all(head.as_bytes()).await; + let _ = sock.flush().await; + loop { + match sock.read(&mut buf).await { + Ok(0) | Err(_) => { + let _ = tx.send(()); + return; + } + Ok(_) => {} + } + } + } + }); + (port, rx) +} + +#[tokio::test] +async fn test_active_read_near_deadline_surfaces_timeout_and_drops_upstream() { + let (port, mut upstream_closed) = spawn_head_then_stall_upstream().await; + + // Cap is 1s. Immediately start a body read that will park on the stalled + // upstream; the reaper must still expire the slot and cancel the read as a + // Timeout (not "unknown/already closed"), dropping the upstream ~at the cap. + let code = format!( + r#" + open url at "http://127.0.0.1:{port}/" and stream response as s + wait for next chunk from s as c + display c + "# + ); + + // Send only a serializable error summary — `Value` is not `Send`. + let (result_tx, result_rx) = std::sync::mpsc::channel::>(); + let client = std::thread::spawn(move || { + let rt = tokio::runtime::Runtime::new().expect("client runtime"); + rt.block_on(async { + let tokens = lex_wfl_with_positions(&code); + let program = Parser::new(&tokens).parse().expect("parse"); + let config = WflConfig { + timeout_seconds: 30, // idle timeout long enough that absolute cap wins + outbound_stream_max_seconds: 1, + ..WflConfig::default() + }; + let mut interp = Interpreter::with_config(Arc::new(config)); + let result = interp.interpret(&program).await; + let summary = match result { + Ok(_) => Ok(()), + Err(errs) => Err(format!("{errs:?}")), + }; + let _ = result_tx.send(summary); + }); + }); + + let start = Instant::now(); + tokio::time::timeout(Duration::from_secs(4), &mut upstream_closed) + .await + .expect("upstream was not dropped near the absolute cap during an active read") + .expect("upstream close sender dropped"); + let elapsed = start.elapsed(); + assert!( + elapsed < Duration::from_secs(3), + "upstream should drop near the 1s absolute cap, not the 30s idle timeout; took {elapsed:?}" + ); + + let result = result_rx + .recv_timeout(Duration::from_secs(5)) + .expect("interpreter should finish after the absolute-cap timeout"); + match tokio::task::spawn_blocking(move || client.join()).await { + Ok(Ok(())) => {} + Ok(Err(panic)) => std::panic::resume_unwind(panic), + Err(e) => panic!("client join task failed: {e}"), + } + + let msg = result.expect_err("active read past absolute cap must fail (Timeout), not succeed"); + assert!( + msg.to_lowercase().contains("timeout") + || msg.contains("Timeout") + || msg.contains("outbound"), + "expected a typed Timeout-class error, got: {msg}" + ); + assert!( + !msg.to_lowercase().contains("unknown or already-closed"), + "expired slot must surface Timeout, not 'unknown/already-closed'; got: {msg}" + ); +} + +#[tokio::test] +async fn test_rapid_open_close_does_not_leak_reaper_tasks() { + // Open and immediately close many outbound streams against a real (stalling) + // upstream. Each close must abort its reaper timer so resource usage stays + // bounded (not request-rate × cap sleeping tasks). + let listener = tokio::net::TcpListener::bind("127.0.0.1:0") + .await + .expect("bind"); + let port = listener.local_addr().unwrap().port(); + tokio::spawn(async move { + loop { + let Ok((mut sock, _)) = listener.accept().await else { + break; + }; + tokio::spawn(async move { + let mut buf = [0u8; 512]; + let _ = sock.read(&mut buf).await; + let head = "HTTP/1.1 200 OK\r\n\ + Content-Type: text/plain\r\n\ + Transfer-Encoding: chunked\r\n\r\n"; + let _ = sock.write_all(head.as_bytes()).await; + // Stall until the proxy drops us on close. + let mut b = [0u8; 64]; + loop { + match sock.read(&mut b).await { + Ok(0) | Err(_) => return, + Ok(_) => {} + } + } + }); + } + }); + + // Cap is large (60s) so a leaked reaper would still be parked after the program + // ends if timers were not aborted on close. We open/close 40 streams quickly. + let mut lines = String::new(); + for i in 0..40 { + lines.push_str(&format!( + "open url at \"http://127.0.0.1:{port}/s{i}\" and stream response as s{i}\n\ + close s{i}\n" + )); + } + + let code = lines; + let start = Instant::now(); + let client = std::thread::spawn(move || { + let rt = tokio::runtime::Runtime::new().expect("runtime"); + rt.block_on(async { + let tokens = lex_wfl_with_positions(&code); + let program = Parser::new(&tokens).parse().expect("parse"); + let config = WflConfig { + timeout_seconds: 10, + outbound_stream_max_seconds: 60, + ..WflConfig::default() + }; + let mut interp = Interpreter::with_config(Arc::new(config)); + interp + .interpret(&program) + .await + .expect("rapid open/close must succeed"); + }); + }); + match tokio::task::spawn_blocking(move || client.join()).await { + Ok(Ok(())) => {} + Ok(Err(panic)) => std::panic::resume_unwind(panic), + Err(e) => panic!("client join failed: {e}"), + } + let elapsed = start.elapsed(); + // Should finish in well under the 60s cap (and under a few seconds of network). + assert!( + elapsed < Duration::from_secs(20), + "rapid open/close should finish promptly with reapers aborted; took {elapsed:?}" + ); +} + diff --git a/tests/response_stream_backpressure_test.rs b/tests/response_stream_backpressure_test.rs index 7fcbcbe1..38ad525b 100644 --- a/tests/response_stream_backpressure_test.rs +++ b/tests/response_stream_backpressure_test.rs @@ -60,7 +60,8 @@ async fn test_backpressured_write_to_a_non_reading_client_is_bounded() { "# ); - let (done_tx, done_rx) = tokio::sync::oneshot::channel::(); + // Summary is Send (string), unlike Value/RuntimeError. + let (done_tx, done_rx) = tokio::sync::oneshot::channel::<(Duration, Result<(), String>)>(); let server = std::thread::spawn(move || { let rt = tokio::runtime::Builder::new_current_thread() .enable_all() @@ -77,8 +78,12 @@ async fn test_backpressured_write_to_a_non_reading_client_is_bounded() { }; let mut interp = Interpreter::with_config(Arc::new(config)); let start = Instant::now(); - let _ = interp.interpret(&program).await; - let _ = done_tx.send(start.elapsed()); + let result = interp.interpret(&program).await; + let summary = match result { + Ok(_) => Ok(()), + Err(errs) => Err(format!("{errs:?}")), + }; + let _ = done_tx.send((start.elapsed(), summary)); }); }); @@ -96,10 +101,29 @@ async fn test_backpressured_write_to_a_non_reading_client_is_bounded() { // `interpret()` must return once the stalled write times out (~2s). If the write // were unbounded it would pin the handler and this never fires. - let elapsed = tokio::time::timeout(Duration::from_secs(12), done_rx) + let (elapsed, summary) = tokio::time::timeout(Duration::from_secs(12), done_rx) .await .expect("interpret() never returned — the backpressured write pinned the handler forever") .expect("done sender dropped"); + // Must fail with a write-timeout / cancelled class error — not succeed, and not + // exit for an unrelated reason (issue #642 R3: previously discarded interpret()). + let err = summary.expect_err( + "backpressured write to a non-reading client must error (write timeout), not succeed", + ); + let err_l = err.to_lowercase(); + assert!( + err_l.contains("timeout") + || err_l.contains("stopped reading") + || err_l.contains("cancelled") + || err_l.contains("write"), + "expected a write-timeout/stall error, got: {err}" + ); + // Lower bound: the 2s response timeout must actually be waited out (not an + // immediate unrelated exit that would false-green the test). + assert!( + elapsed >= Duration::from_millis(1500), + "stall should wait for ~2s web_server_response_timeout_seconds; took only {elapsed:?}" + ); assert!( elapsed < Duration::from_secs(9), "the stalled write should time out at ~2s (web_server_response_timeout_seconds), \ diff --git a/tests/write_web_postfix_test.rs b/tests/write_web_postfix_test.rs index 4ed5aa26..f2230996 100644 --- a/tests/write_web_postfix_test.rs +++ b/tests/write_web_postfix_test.rs @@ -102,6 +102,65 @@ fn streaming_response_content_type_clause_composes_property_then_index() { } } +#[test] +fn write_line_at_indexing_parses() { + // Classic `write line values at 0 to "/tmp/out"` must parse (issue #642). + let program = parse( + "store line values as [\"first\" and \"second\"]\n\ + write line values at 0 to \"/tmp/out\"\n", + ); + assert!( + program.statements.len() >= 2, + "expected store + write, got {:#?}", + program.statements + ); + let write = program + .statements + .iter() + .find(|s| matches!(s, Statement::StreamWriteStatement { .. })) + .expect("write statement"); + // Stream reading value is IndexAccess over Variable("values"). + assert!( + matches!(stream_write_value(write), Expression::IndexAccess { .. }), + "write value must be IndexAccess for `at` indexing, got {:#?}", + stream_write_value(write) + ); +} + +#[test] +fn write_line_direct_integer_indexing_parses() { + // Classic `write line values 0 to "/tmp/out"` must parse (issue #642). + let program = parse("write line values 0 to \"/tmp/out\"\n"); + assert_eq!(program.statements.len(), 1, "got {:#?}", program.statements); + assert!( + matches!( + stream_write_value(&program.statements[0]), + Expression::IndexAccess { .. } + ), + "write value must be IndexAccess for direct integer indexing, got {:#?}", + stream_write_value(&program.statements[0]) + ); +} + +#[test] +fn streaming_response_content_type_of_call_parses() { + // `content type mime_type of path` — `of` continuation on the merged clause + // operand must compose like an ordinary expression (issue #642). + let program = parse( + "start streaming response to req with status 200 and content type mime_type of path as out\n", + ); + assert_eq!(program.statements.len(), 1, "got {:#?}", program.statements); + match &program.statements[0] { + Statement::StartStreamingResponseStatement { content_type, .. } => { + assert!( + matches!(content_type, Some(Expression::FunctionCall { .. })), + "content type operand must be a FunctionCall (`mime_type of path`), got {content_type:#?}" + ); + } + other => panic!("expected StartStreamingResponseStatement, got {other:#?}"), + } +} + #[test] fn write_line_of_call_argument_absorbs_arithmetic() { // `double of n minus 1` must parse as `double of (n minus 1)` — the same From a26203622ca7cad588abe94566561ee1dcc7b913 Mon Sep 17 00:00:00 2001 From: Brad Byrd Date: Fri, 24 Jul 2026 10:27:12 -0500 Subject: [PATCH 089/132] style: cargo fmt for CI gate Unblock CI which failed the formatting job and skipped the rest of the matrix. --- src/analyzer/mod.rs | 10 ++-------- tests/outbound_stream_reaper_race_test.rs | 1 - 2 files changed, 2 insertions(+), 9 deletions(-) diff --git a/src/analyzer/mod.rs b/src/analyzer/mod.rs index c208f084..5f579504 100644 --- a/src/analyzer/mod.rs +++ b/src/analyzer/mod.rs @@ -3790,14 +3790,8 @@ impl Analyzer { // PropertyAccess leaf whose object is a Variable: treat the object // name as the lead (e.g. `missing.field` vs `line missing.field`). ( - Expression::PropertyAccess { - object: vo, - .. - }, - Expression::PropertyAccess { - object: fo, - .. - }, + Expression::PropertyAccess { object: vo, .. }, + Expression::PropertyAccess { object: fo, .. }, ) => { self.analyze_ambiguous_write(vo, fo, line, column); } diff --git a/tests/outbound_stream_reaper_race_test.rs b/tests/outbound_stream_reaper_race_test.rs index c37c94ae..99c04cd1 100644 --- a/tests/outbound_stream_reaper_race_test.rs +++ b/tests/outbound_stream_reaper_race_test.rs @@ -190,4 +190,3 @@ async fn test_rapid_open_close_does_not_leak_reaper_tasks() { "rapid open/close should finish promptly with reapers aborted; took {elapsed:?}" ); } - From fce5d86fe923666885e40ec484d902cfd18c4c85 Mon Sep 17 00:00:00 2001 From: Brad Byrd Date: Fri, 24 Jul 2026 10:37:38 -0500 Subject: [PATCH 090/132] fix: address #642 re-review blockers (lifecycle, timeouts, parser) - Shared StreamCancel + std mutex for close-during-active-read and Drop cleanup - Delay taking respond/stream pending until after expression evaluation - Stall writes are Timeout; only request-wait Timeout exempt from breaker - Reject sub-1ms fractional wait timeouts; atomic stream insert/arm; no tombstones - File handle classic write; clause operand stop for content type/headers - Full flush expression AST fallback; ResponseStream current-scope shadow - Container-aware write definedness; stronger flush/clause/close tests Addresses re-review of #642. --- .../2026-07-24-issue-642-rereview-fixes.md | 50 ++ src/analyzer/mod.rs | 46 +- src/interpreter/mod.rs | 669 +++++++++++------- src/parser/ast.rs | 15 +- src/parser/expr/binary.rs | 71 +- src/parser/stmt/io.rs | 75 ++ src/parser/stmt/web.rs | 39 +- src/typechecker/mod.rs | 83 ++- tests/flush_action_backcompat_test.rs | 50 +- .../outbound_stream_close_during_read_test.rs | 113 +++ tests/write_web_postfix_test.rs | 39 + 11 files changed, 897 insertions(+), 353 deletions(-) create mode 100644 Dev diary/2026-07-24-issue-642-rereview-fixes.md create mode 100644 tests/outbound_stream_close_during_read_test.rs diff --git a/Dev diary/2026-07-24-issue-642-rereview-fixes.md b/Dev diary/2026-07-24-issue-642-rereview-fixes.md new file mode 100644 index 00000000..0f81f39a --- /dev/null +++ b/Dev diary/2026-07-24-issue-642-rereview-fixes.md @@ -0,0 +1,50 @@ +# Dev Diary — 2026-07-24: issue #642 re-review fixes + +Follow-up to the maintainer checklist on the #642 round. + +## CI + +- `cargo fmt --all` unblocked the formatting gate (commit `a2620362`). + +## Runtime lifecycle + +| Item | Fix | +|------|-----| +| Close during active read | `StreamCancel` (AtomicBool + Notify); reads select against it; close/reaper cancel + remove slot | +| Map cleanup from Drop | `std::sync::Mutex` (no silent `try_lock` abandon) | +| Reaper before insert | Insert slot then arm reaper under same lock | +| Tombstones | Finish always **removes** the slot | +| Final unterminated line | Finish after emitting (no parked done+reaper) | +| Respond/stream eval disconnect | `ensure_pending_response_owned` then evaluate; take sender only at commit | +| Stall vs disconnect | Stall write → `ErrorKind::Timeout`; disconnect → `Cancelled` | +| Breaker Timeout exemption | Only messages starting with `Timeout waiting for request` | +| Fractional wait timeout | Reject `0 < ms < 1` | + +## Parser / typechecker + +| Item | Fix | +|------|-----| +| Classic write to `open file` | Accept `Custom("File")` as classic-file target | +| content type / headers clauses | `parse_clause_operand_from_lead` stops at clause connectives | +| flush postfix legacy | Full expression AST fallback on phrase + postfix | +| Parameterized flush | ExpressionStatement semantics → arity error | +| ResponseStream binding | `define_or_replace` in current scope only (shadow) | +| Write definedness | Analyzer `name_is_defined_for_write` shared with typechecker | + +## Red→Green note + +The original #642 landing was a single mixed commit (`5e01e446`, +1399/−271). +Rewriting that history on the already-pushed branch would require a force-push; +this re-review is a **new** Green commit on top with targeted regressions. +Auditable Red-first history for a future rework can soft-reset and re-land as +test-only then fix commits if the maintainer prefers a rewritten PR stack. + +## Tests run (local) + +``` +cargo test --test flush_action_backcompat_test --test write_web_postfix_test \ + --test ambiguous_write_branch_typecheck_test --test concurrent_disconnect_paths_burst_test \ + --test outbound_stream_reaper_race_test --test response_stream_backpressure_test \ + --test outbound_stream_close_during_read_test +``` +All green. diff --git a/src/analyzer/mod.rs b/src/analyzer/mod.rs index 5f579504..7c0b6021 100644 --- a/src/analyzer/mod.rs +++ b/src/analyzer/mod.rs @@ -1694,15 +1694,17 @@ impl Analyzer { action_fallback, .. } => { - // A merged `flush ` may resolve at runtime to a zero-argument - // action named `flush ` (backward compatibility — see - // `parse_flush_stream`). When such an action is defined this is an - // action call, not a stream flush, so analyzing the operand as a - // stream variable would raise a spurious "undefined variable". - let is_action_call = action_fallback - .as_ref() - .is_some_and(|name| self.name_is_defined(name)); - if !is_action_call { + // When the legacy full-name expression's root is defined, analyze + // that expression (expression-statement path). Otherwise analyze + // the stream target. + let legacy_root_defined = action_fallback.as_ref().is_some_and(|expr| { + Self::expression_root_name(expr).is_some_and(|n| self.name_is_defined(n)) + }); + if legacy_root_defined { + if let Some(fb) = action_fallback { + self.analyze_expression(fb); + } + } else { self.analyze_expression(target); } } @@ -3806,8 +3808,15 @@ impl Analyzer { /// property) — i.e. it would NOT be reported as an undefined variable. Used /// to decide the ambiguous `write line|chunk` case without emitting. fn name_is_defined(&self, name: &str) -> bool { + self.name_is_defined_for_write(name) + } + + /// Public for the typechecker so write-branch definedness matches analysis + /// (container properties, inherited bindings, etc.). + pub fn name_is_defined_for_write(&self, name: &str) -> bool { if self.action_parameters.contains(name) || name == "count" + || name == "loopcounter" || Self::is_builtin_function(name) || self.current_scope.resolve(name).is_some() { @@ -3819,6 +3828,25 @@ impl Analyzer { false } + /// Root variable name of an expression used as a flush legacy fallback + /// (`flush cache[0]` → `"flush cache"`). + fn expression_root_name(expr: &Expression) -> Option<&str> { + match expr { + Expression::Variable(name, ..) => Some(name.as_str()), + Expression::IndexAccess { collection, .. } + | Expression::PropertyAccess { + object: collection, .. + } + | Expression::MemberAccess { + object: collection, .. + } + | Expression::MethodCall { + object: collection, .. + } => Self::expression_root_name(collection), + _ => None, + } + } + fn analyze_expression(&mut self, expression: &Expression) { // Recursive front-end checkpoint for expressions. `analyze_statement` // polls per statement, but one statement can hold an arbitrarily large diff --git a/src/interpreter/mod.rs b/src/interpreter/mod.rs index 57a3be51..e1bcf1e2 100644 --- a/src/interpreter/mod.rs +++ b/src/interpreter/mod.rs @@ -936,14 +936,19 @@ impl Drop for OutboundStreamCleanup { // Outbound upstream streams: removing a handle drops its reqwest stream, // cancelling the in-flight upstream request. let http_ids = std::mem::take(&mut *self.open_http_streams.borrow_mut()); - if !http_ids.is_empty() - && let Ok(mut map) = self.io_client.stream_handles.try_lock() - { + if !http_ids.is_empty() { + let mut map = self + .io_client + .stream_handles + .lock() + .unwrap_or_else(|e| e.into_inner()); for id in &http_ids { - if let Some(mut slot) = map.remove(id) - && let Some(abort) = slot.reaper_abort.take() - { - abort.abort(); + if let Some(mut slot) = map.remove(id) { + slot.cancel.cancel(); + if let Some(abort) = slot.reaper_abort.take() { + abort.abort(); + } + drop(slot.handle.take()); } } } @@ -1687,11 +1692,13 @@ pub struct IoClient { db_handles: Mutex>, next_db_id: Mutex, /// Live outbound streaming response bodies, keyed by handle id - /// ("httpstream1", ...). See [`StreamSlot`] / [`HttpStreamHandle`]. Behind - /// an `Arc` so the per-stream absolute-lifetime reaper (spawned in - /// `open_http_stream`) can share it and expire a handle in real time, - /// independent of reads — including while a body read owns the handle. - stream_handles: Arc>>, + /// ("httpstream1", ...). See [`StreamSlot`] / [`HttpStreamHandle`]. + /// + /// Uses a **std** mutex so Drop/cleanup paths can lock reliably (tokio's + /// async mutex only offers `try_lock` from sync Drop, which previously + /// abandoned handles when the map was briefly held). Critical sections are + /// short (no `.await` while held). + stream_handles: Arc>>, next_stream_id: Mutex, config: Arc, } @@ -1713,25 +1720,57 @@ fn outbound_stream_deadline(secs: u64) -> Option { Instant::now().checked_add(Duration::from_secs(capped)) } +/// Shared per-stream cancellation: close, expire, and EOF all trip this so an +/// active body read can select against it and drop the upstream promptly +/// (rather than only noticing when `put_stream` finds a missing slot). +struct StreamCancel { + cancelled: std::sync::atomic::AtomicBool, + notify: tokio::sync::Notify, +} + +impl StreamCancel { + fn new() -> Arc { + Arc::new(Self { + cancelled: std::sync::atomic::AtomicBool::new(false), + notify: tokio::sync::Notify::new(), + }) + } + + fn is_cancelled(&self) -> bool { + self.cancelled.load(std::sync::atomic::Ordering::SeqCst) + } + + fn cancel(&self) { + self.cancelled + .store(true, std::sync::atomic::Ordering::SeqCst); + self.notify.notify_waiters(); + } +} + +/// Why a stream slot was terminated. Never left as a long-lived tombstone in the +/// map — the slot is removed when finished; readers that still hold a cancel +/// watch observe the flag. +#[derive(Debug, Clone, Copy, PartialEq, Eq)] +enum StreamTerminal { + Timeout, + Closed, +} + /// Per-handle shared lifecycle for an outbound stream. /// /// Reads take the inner [`HttpStreamHandle`] out for the duration of the await -/// (so the global map lock is not held across the network). The slot stays in -/// the map so the absolute-lifetime reaper can still mark expiry while the -/// read owns the handle — and `put_stream` refuses reinsertion after expiry, -/// preserving `Timeout` as the terminal reason instead of a silent revive. +/// (so the global map lock is not held across the network). The slot stays only +/// while the stream is live; finish/close/expire remove it entirely after +/// signalling [`StreamCancel`] so mid-read work aborts. struct StreamSlot { - /// The live body handle. `None` while a body read owns it, or after - /// expiry/close has dropped it. + /// The live body handle. `None` while a body read owns it. handle: Option, /// Absolute deadline for the whole stream (`None` = no absolute total cap). deadline: Option, - /// Set by the reaper (or by take/put noticing the deadline) so the next - /// read surfaces a typed `Timeout` rather than "unknown/already closed". - expired: bool, + /// Shared cancel signal for active-read races. + cancel: Arc, /// Abort handle for the reaper timer. Cancelled on EOF, error, or explicit - /// close so rapid open/close cycles do not accumulate sleeping tasks for - /// the full configured cap. + /// close so rapid open/close cycles do not accumulate sleeping tasks. reaper_abort: Option, } @@ -1764,6 +1803,12 @@ struct HttpStreamHandle { total_deadline: Option, } +/// A body read in progress: the handle plus a cancel watch shared with close/reaper. +struct TakenStream { + handle: HttpStreamHandle, + cancel: Arc, +} + /// Errors raised while an outbound HTTP request is in flight. /// /// Budget failures stay structured until the interpreter can attach source @@ -1777,6 +1822,9 @@ enum HttpClientError { Timeout { seconds: u64, }, + /// The stream was explicitly closed (or finished) while a body read was in + /// flight. Distinct from absolute-lifetime [`Self::Timeout`]. + Closed, /// The downstream (browser) client disconnected while a proxy handler was /// blocked on this upstream read, so the read was cancelled cooperatively. /// A normal, expected event — distinct from a fault — surfaced with @@ -1873,7 +1921,7 @@ impl IoClient { next_process_id: Mutex::new(1), db_handles: Mutex::new(HashMap::new()), next_db_id: Mutex::new(1), - stream_handles: Arc::new(Mutex::new(HashMap::new())), + stream_handles: Arc::new(std::sync::Mutex::new(HashMap::new())), next_stream_id: Mutex::new(1), config, } @@ -2075,162 +2123,204 @@ impl IoClient { id }; - // Enforce `outbound_stream_max_seconds` as a TRUE absolute lifetime, not a - // read-triggered one: a handler that opens a stream and then parks (or does - // other work) without reading would otherwise keep the upstream connection - // alive past the cap. Spawn a reaper that marks the shared slot expired - // (and drops any parked handle) when the absolute deadline elapses — - // including while a body read owns the handle. The AbortHandle is stored - // on the slot so EOF/error/close cancels the timer immediately. - let reaper_abort = if let Some(deadline) = total_deadline { - let handles = Arc::clone(&self.stream_handles); - let reap_id = handle_id.clone(); - let join = tokio::spawn(async move { - let remaining = deadline.saturating_duration_since(Instant::now()); - tokio::time::sleep(remaining).await; - let mut map = handles.lock().await; - if let Some(slot) = map.get_mut(&reap_id) { - slot.expired = true; - // Drop the live handle if it is parked (not currently mid-read); - // if a read owns it, put_stream will refuse reinsertion. - slot.handle = None; - slot.reaper_abort = None; + // Insert the slot FIRST, then arm the reaper under the same lock so the + // reaper can never fire before the slot exists (and so close/finish that + // races with open still sees a consistent cancel handle). + let cancel = StreamCancel::new(); + { + let mut map = self + .stream_handles + .lock() + .unwrap_or_else(|e| e.into_inner()); + map.insert( + handle_id.clone(), + StreamSlot { + handle: Some(handle), + deadline: total_deadline, + cancel: Arc::clone(&cancel), + reaper_abort: None, + }, + ); + if let Some(deadline) = total_deadline { + let handles = Arc::clone(&self.stream_handles); + let reap_id = handle_id.clone(); + let cancel_reap = Arc::clone(&cancel); + let join = tokio::spawn(async move { + let remaining = deadline.saturating_duration_since(Instant::now()); + tokio::time::sleep(remaining).await; + // Absolute-lifetime reaper: signal cancel (wakes any active + // read), abort is a no-op for ourselves, drop the handle, and + // REMOVE the slot (no tombstone accumulation). + cancel_reap.cancel(); + let mut map = handles.lock().unwrap_or_else(|e| e.into_inner()); + if let Some(mut slot) = map.remove(&reap_id) { + slot.reaper_abort = None; // we are the reaper + drop(slot.handle.take()); + } + }); + if let Some(slot) = map.get_mut(&handle_id) { + slot.reaper_abort = Some(join.abort_handle()); + } else { + // Already finished before we armed — cancel the timer. + join.abort(); } - }); - Some(join.abort_handle()) - } else { - None - }; - - self.stream_handles.lock().await.insert( - handle_id.clone(), - StreamSlot { - handle: Some(handle), - deadline: total_deadline, - expired: false, - reaper_abort, - }, - ); + } + } Ok((status, response_headers, handle_id)) } - /// Abort the reaper (if any) and remove the slot, dropping any remaining - /// handle. Used on EOF, error, explicit close, and handler-exit cleanup. - async fn finish_stream_slot(&self, handle_id: &str) -> bool { - let mut map = self.stream_handles.lock().await; + /// Signal cancel, abort the reaper, drop any parked handle, and remove the + /// slot. Guaranteed (std mutex) — usable from Drop. Returns whether a slot + /// was present. + fn finish_stream_slot_sync(&self, handle_id: &str, _terminal: StreamTerminal) -> bool { + let mut map = self + .stream_handles + .lock() + .unwrap_or_else(|e| e.into_inner()); if let Some(mut slot) = map.remove(handle_id) { + slot.cancel.cancel(); if let Some(abort) = slot.reaper_abort.take() { abort.abort(); } - // Dropping `slot.handle` cancels the upstream if still present. + drop(slot.handle.take()); true } else { false } } + async fn finish_stream_slot(&self, handle_id: &str) -> bool { + self.finish_stream_slot_sync(handle_id, StreamTerminal::Closed) + } + /// Remove a stream handle from its slot so a body read can await without - /// holding the global handle lock across the network. The slot remains so - /// the reaper can still mark expiry mid-read. Errors if the handle is - /// unknown, already closed, or already expired (`Timeout`). - async fn take_stream(&self, handle_id: &str) -> Result { - let mut map = self.stream_handles.lock().await; - // Peek expiry without holding a long-lived mut borrow that blocks remove. - let past_deadline = match map.get(handle_id) { - None => { - return Err(HttpClientError::Request(format!( - "Unknown or already-closed stream handle '{handle_id}'" - ))); - } - Some(slot) => { - slot.expired - || slot - .deadline - .is_some_and(|d| d.saturating_duration_since(Instant::now()).is_zero()) - } + /// holding the global handle lock. The cancel watch stays alive so close/ + /// expire aborts the read. Errors if unknown, closed, or past deadline. + fn take_stream(&self, handle_id: &str) -> Result { + let mut map = self + .stream_handles + .lock() + .unwrap_or_else(|e| e.into_inner()); + let Some(slot) = map.get_mut(handle_id) else { + return Err(HttpClientError::Request(format!( + "Unknown or already-closed stream handle '{handle_id}'" + ))); }; + if slot.cancel.is_cancelled() { + // Fully finish and remove. + if let Some(mut slot) = map.remove(handle_id) { + if let Some(abort) = slot.reaper_abort.take() { + abort.abort(); + } + drop(slot.handle.take()); + } + return Err(HttpClientError::Closed); + } + let past_deadline = slot + .deadline + .is_some_and(|d| d.saturating_duration_since(Instant::now()).is_zero()); if past_deadline { - if let Some(mut slot) = map.remove(handle_id) - && let Some(abort) = slot.reaper_abort.take() - { - abort.abort(); + if let Some(mut slot) = map.remove(handle_id) { + slot.cancel.cancel(); + if let Some(abort) = slot.reaper_abort.take() { + abort.abort(); + } + drop(slot.handle.take()); } return Err(HttpClientError::Timeout { seconds: self.config.outbound_stream_max_seconds, }); } - match map.get_mut(handle_id).and_then(|s| s.handle.take()) { - Some(handle) => Ok(handle), + let cancel = Arc::clone(&slot.cancel); + match slot.handle.take() { + Some(handle) => Ok(TakenStream { handle, cancel }), None => Err(HttpClientError::Request(format!( "Unknown or already-closed stream handle '{handle_id}'" ))), } } - /// Return a still-open stream handle to its slot after a body read. - /// Refuses reinsertion at/after the absolute deadline (or if the reaper - /// already marked the slot expired), dropping the handle and surfacing - /// `Timeout` so a ready chunk cannot revive an expired stream. - async fn put_stream( + /// Return a still-open stream handle after a body read. Refuses reinsertion + /// if the stream was closed/expired mid-read (cancel flag or missing slot). + fn put_stream( &self, handle_id: &str, handle: HttpStreamHandle, + cancel: &StreamCancel, ) -> Result<(), HttpClientError> { - let mut map = self.stream_handles.lock().await; - let past_deadline = match map.get(handle_id) { - None => { - // Slot was fully removed (close/finish raced) — drop the handle. - return Ok(()); - } - Some(slot) => { - slot.expired - || slot - .deadline - .is_some_and(|d| d.saturating_duration_since(Instant::now()).is_zero()) - } + if cancel.is_cancelled() { + drop(handle); + // Ensure the slot is gone (reaper/close may already have removed it). + let _ = self.finish_stream_slot_sync(handle_id, StreamTerminal::Closed); + // Prefer Timeout if the absolute deadline has elapsed. + return Err(HttpClientError::Closed); + } + let mut map = self + .stream_handles + .lock() + .unwrap_or_else(|e| e.into_inner()); + let Some(slot) = map.get_mut(handle_id) else { + drop(handle); + return Err(HttpClientError::Closed); }; - if past_deadline { - if let Some(mut slot) = map.remove(handle_id) - && let Some(abort) = slot.reaper_abort.take() + if slot.cancel.is_cancelled() + || slot + .deadline + .is_some_and(|d| d.saturating_duration_since(Instant::now()).is_zero()) + { + let terminal = if slot + .deadline + .is_some_and(|d| d.saturating_duration_since(Instant::now()).is_zero()) { - abort.abort(); - } - // Drop `handle` by not inserting it. + StreamTerminal::Timeout + } else { + StreamTerminal::Closed + }; drop(handle); - return Err(HttpClientError::Timeout { - seconds: self.config.outbound_stream_max_seconds, + if let Some(mut slot) = map.remove(handle_id) { + slot.cancel.cancel(); + if let Some(abort) = slot.reaper_abort.take() { + abort.abort(); + } + } + return Err(match terminal { + StreamTerminal::Timeout => HttpClientError::Timeout { + seconds: self.config.outbound_stream_max_seconds, + }, + StreamTerminal::Closed => HttpClientError::Closed, }); } - if let Some(slot) = map.get_mut(handle_id) { - slot.handle = Some(handle); - } + // Done streams must not leave a parked reaper: finish fully on EOF + // after the final unterminated line is served (caller uses finish). + slot.handle = Some(handle); Ok(()) } /// Pull one network chunk into `handle.buffer`, bounded by the per-chunk - /// read deadline and the run's response-byte ceiling. Returns `Ok(true)` - /// when bytes were added, `Ok(false)` at clean EOF (sets `handle.done`). + /// read deadline, the absolute total, and cooperative stream cancellation + /// (close/expire while reading). Returns `Ok(true)` when bytes were added, + /// `Ok(false)` at clean EOF (sets `handle.done`). async fn stream_pull( &self, handle: &mut HttpStreamHandle, budget: &Arc, + cancel: &StreamCancel, ) -> Result { use futures_util::StreamExt; if handle.done { return Ok(false); } + if cancel.is_cancelled() { + return Err(HttpClientError::Closed); + } let max_response_bytes = budget.limits().max_response_bytes; - // Per-read idle timeout, bounded by the remaining time to the stream's - // absolute total deadline so a single read can never wait past the total - // (and a trickling upstream cannot outlive the total cap). let idle_timeout = Duration::from_secs(self.config.timeout_seconds.max(1)); let configured_timeout = match handle.total_deadline { Some(deadline) => { let remaining = deadline.saturating_duration_since(Instant::now()); if remaining.is_zero() { - // Already past the absolute total lifetime. return Err(HttpClientError::Timeout { seconds: self.config.outbound_stream_max_seconds, }); @@ -2239,10 +2329,23 @@ impl IoClient { } None => idle_timeout, }; - let next = Self::run_http_with_budget(Arc::clone(budget), configured_timeout, async { + + // Race the network read against stream cancellation so close-during- + // active-read aborts the upstream promptly. + let op = Self::run_http_with_budget(Arc::clone(budget), configured_timeout, async { Ok::>>, HttpClientError>(handle.stream.next().await) - }) - .await?; + }); + tokio::pin!(op); + let next = tokio::select! { + result = &mut op => result?, + _ = cancel.notify.notified() => { + return Err(HttpClientError::Closed); + } + }; + // Re-check after select (notify may have raced with a false wake). + if cancel.is_cancelled() { + return Err(HttpClientError::Closed); + } match next { Some(Ok(bytes)) => { @@ -2292,37 +2395,30 @@ impl IoClient { handle_id: &str, budget: Arc, ) -> Result>, HttpClientError> { - let mut handle = self.take_stream(handle_id).await?; + let TakenStream { mut handle, cancel } = self.take_stream(handle_id)?; - // Enforce the absolute stream lifetime before serving ANY bytes — even - // ones already buffered by a prior read — so `outbound_stream_max_seconds` - // is a true absolute lifetime, not merely a per-network-read bound. On - // expiry the slot is finished, so the upstream request is dropped. if let Err(e) = self.check_stream_deadline(&handle) { let _ = self.finish_stream_slot(handle_id).await; return Err(e); } - // Any bytes buffered by a prior `next line` are served first. if !handle.buffer.is_empty() { let chunk = std::mem::take(&mut handle.buffer); - self.put_stream(handle_id, handle).await?; + self.put_stream(handle_id, handle, &cancel)?; return Ok(Some(chunk)); } - match self.stream_pull(&mut handle, &budget).await { + match self.stream_pull(&mut handle, &budget, &cancel).await { Ok(true) => { let chunk = std::mem::take(&mut handle.buffer); - self.put_stream(handle_id, handle).await?; + self.put_stream(handle_id, handle, &cancel)?; Ok(Some(chunk)) } Ok(false) => { - // clean EOF: drop the slot + abort reaper let _ = self.finish_stream_slot(handle_id).await; Ok(None) } Err(e) => { - // error/timeout: drop the slot (cancels upstream + aborts reaper) let _ = self.finish_stream_slot(handle_id).await; Err(e) } @@ -2337,12 +2433,9 @@ impl IoClient { handle_id: &str, budget: Arc, ) -> Result, HttpClientError> { - let mut handle = self.take_stream(handle_id).await?; + let TakenStream { mut handle, cancel } = self.take_stream(handle_id)?; loop { - // Enforce the absolute stream lifetime before serving a buffered line - // (a prior read may have buffered several lines); on expiry the handle - // is dropped, cancelling the upstream. See `next_chunk`. if let Err(e) = self.check_stream_deadline(&handle) { let _ = self.finish_stream_slot(handle_id).await; return Err(e); @@ -2354,39 +2447,35 @@ impl IoClient { if line.last() == Some(&b'\r') { line.pop(); // drop paired '\r' (CRLF) } - self.put_stream(handle_id, handle).await?; + self.put_stream(handle_id, handle, &cancel)?; return Ok(Some(String::from_utf8_lossy(&line).into_owned())); } if handle.done { - // No newline left. Emit any final unterminated line, then EOF. + // Final unterminated line (if any), then fully finish — do NOT + // leave a done slot + reaper parked for a subsequent read. if handle.buffer.is_empty() { let _ = self.finish_stream_slot(handle_id).await; - return Ok(None); // drop the exhausted handle + return Ok(None); } let mut line = std::mem::take(&mut handle.buffer); if line.last() == Some(&b'\r') { line.pop(); } - // Re-insert the now-drained (done, empty) handle so the *next* - // read cleanly returns `nothing` instead of erroring on a - // missing handle. - self.put_stream(handle_id, handle).await?; + let _ = self.finish_stream_slot(handle_id).await; return Ok(Some(String::from_utf8_lossy(&line).into_owned())); } - // Need more bytes to find a newline. - if let Err(e) = self.stream_pull(&mut handle, &budget).await { + if let Err(e) = self.stream_pull(&mut handle, &budget, &cancel).await { let _ = self.finish_stream_slot(handle_id).await; return Err(e); } } } - /// Close a streaming response handle if present. Dropping the handle - /// cancels the in-flight upstream request and aborts its reaper timer. - /// Returns whether a slot was found. Idempotent: closing an - /// unknown/already-closed handle is a no-op. + /// Close a streaming response handle if present. Signals cancel (aborting + /// any active read), drops the handle, and aborts the reaper timer. + /// Idempotent. async fn close_stream(&self, handle_id: &str) -> bool { self.finish_stream_slot(handle_id).await } @@ -4170,6 +4259,9 @@ impl Interpreter { column, ErrorKind::Timeout, ), + HttpClientError::Closed => { + RuntimeError::new("Stream was closed while reading".to_string(), line, column) + } HttpClientError::Disconnected => RuntimeError::with_kind( "Client disconnected; upstream read cancelled".to_string(), line, @@ -4295,13 +4387,19 @@ impl Interpreter { if ids.is_empty() { return; } - if let Ok(mut map) = self.io_client.stream_handles.try_lock() { - for id in ids { - if let Some(mut slot) = map.remove(id) - && let Some(abort) = slot.reaper_abort.take() - { + // std mutex: guaranteed cleanup from Drop (no silent try_lock abandon). + let mut map = self + .io_client + .stream_handles + .lock() + .unwrap_or_else(|e| e.into_inner()); + for id in ids { + if let Some(mut slot) = map.remove(id) { + slot.cancel.cancel(); + if let Some(abort) = slot.reaper_abort.take() { abort.abort(); } + drop(slot.handle.take()); } } } @@ -4466,6 +4564,60 @@ impl Interpreter { /// owns the request. That path is `ErrorKind::Cancelled`. A missing entry when /// the handler no longer owns the id (duplicate respond after a successful /// one, or a forged request id) remains a general error. + /// Check that this handler still owns `request_id` and a pending entry is + /// present (or was pruned as disconnected → Cancelled), WITHOUT taking the + /// sender. Used before evaluating respond/stream-head expressions so the + /// parked sender remains a disconnect signal for upstream work. + async fn ensure_pending_response_owned( + &self, + request_id: &str, + line: usize, + column: usize, + ) -> Result<(), RuntimeError> { + let was_owned = self + .open_pending_requests + .borrow() + .iter() + .any(|id| id == request_id); + if !was_owned { + return Err(RuntimeError::new( + "Request ID not found - response may have already been sent".to_string(), + line, + column, + )); + } + let present = self.pending_responses.borrow().contains_key(request_id); + if !present { + // Sibling prune of a closed sender while we still own the request. + return Err(RuntimeError::with_kind( + "Client disconnected before the response was sent".to_string(), + line, + column, + ErrorKind::Cancelled, + )); + } + // Optionally check is_closed early for a clearer cancel before heavy eval. + let closed = { + let pending = self.pending_responses.borrow(); + match pending.get(request_id) { + Some(p) => match p.sender.try_lock() { + Ok(g) => g.as_ref().is_none_or(|s| s.is_closed()), + Err(_) => false, + }, + None => true, + } + }; + if closed { + return Err(RuntimeError::with_kind( + "Client disconnected before the response was sent".to_string(), + line, + column, + ErrorKind::Cancelled, + )); + } + Ok(()) + } + async fn take_pending_response_completion( &self, request_id: &str, @@ -4630,18 +4782,17 @@ impl Interpreter { // Expected / request-local outcomes must NEVER feed the structural // consecutive-failure breaker: // - `Cancelled`: client disconnect (cooperative cancellation) - // - `Timeout`: finite `wait for request ... with timeout` expiry - // (healthy idle server, not a hot-spin) + // - finite `wait for request ... with timeout` expiry ONLY + // (not every ErrorKind::Timeout — e.g. pre-request structural + // timeouts still feed the breaker) // - any error from a handler that already accepted a request - // (upstream/network/response failure is local to that request) // - // Structural pre-request failures only (channel closed, missing - // server, deterministic bad expression before `wait`, …) can - // hot-spin the loop if not backstopped — those feed the breaker. + // Structural pre-request failures feed the breaker. Some((Ok(Err(err)), accepted)) => { - let non_structural = accepted - || err.kind == ErrorKind::Cancelled - || err.kind == ErrorKind::Timeout; + let is_request_wait_timeout = err.kind == ErrorKind::Timeout + && err.message.starts_with("Timeout waiting for request"); + let non_structural = + accepted || err.kind == ErrorKind::Cancelled || is_request_wait_timeout; if non_structural { log::debug!( "concurrent main loop: non-structural handler outcome \ @@ -9273,9 +9424,22 @@ impl Interpreter { .evaluate_expression(timeout_expr, Rc::clone(&env)) .await?; match timeout_val { - Value::Number(ms) if ms > 0.0 => { + // Reject fractional values that would truncate to 0 ms + // (e.g. 0.5) and hot-spin forever with zero-duration + // timeouts. Require at least 1 millisecond. + Value::Number(ms) if ms >= 1.0 => { Some(std::time::Duration::from_millis(ms as u64)) } + Value::Number(ms) if ms > 0.0 => { + return Err(RuntimeError::new( + format!( + "Timeout must be at least 1 millisecond (got {ms} ms); \ + fractional values below 1 would truncate to zero and spin" + ), + *line, + *column, + )); + } _ => { return Err(RuntimeError::new( "Timeout must be a positive number (milliseconds)".to_string(), @@ -9493,21 +9657,14 @@ impl Interpreter { } }; - // Take the response sender out of the pending map (and out of its - // mutex) up front, into an RAII completion guard, *before* any - // fallible response construction below (content/status/type/header - // evaluation, byte-cap checks). On an early error the guard's Drop - // answers 500, so the request is always resolved instead of - // hanging until its timeout; a successful respond disarms it via - // `take_sender`. - // - // Ownership is checked BEFORE removing the id from - // `open_pending_requests`: a sibling `wait for request` may have - // globally pruned a closed (disconnected) sender, leaving the - // entry missing while this handler still owns the request. That - // is cooperative cancellation, not a duplicate-response fault. - let mut completion = self - .take_pending_response_completion(&request_id, *line, *column) + // Keep the pending request parked until content/status/header + // expressions are evaluated so a browser disconnect still cancels + // any upstream open/read performed during that evaluation + // (`any_pending_request_disconnected` watches the parked sender). + // Only after evaluation do we take the sender into the completion + // guard. Early eval errors leave the id in open_pending so the + // handler-exit 500 path still resolves the client. + self.ensure_pending_response_owned(&request_id, *line, *column) .await?; // Evaluate response content. Binary values are carried through @@ -9664,8 +9821,10 @@ impl Interpreter { headers: custom_headers, }; - // Deliver the response and disarm the guard's 500 fallback. The - // sender was taken up front, so this is the sole delivery path. + // Now commit: take the sender (disconnect signal ends) and deliver. + let mut completion = self + .take_pending_response_completion(&request_id, *line, *column) + .await?; match completion.take_sender() { Some(sender) => { if sender.send(HandlerReply::Buffered(response)).is_err() { @@ -9725,13 +9884,10 @@ impl Interpreter { } }; - // Take the oneshot into an RAII guard up front: an early error - // while evaluating status/content type/headers still resolves - // the client with 500 instead of hanging. Same ownership rule as - // `respond`: missing while still owned => Cancelled (sibling - // prune of a disconnected client), not a structural fault. - let mut completion = self - .take_pending_response_completion(&request_id, *line, *column) + // Keep pending parked through status/content-type/header + // evaluation so disconnect still cancels any upstream work those + // expressions perform. Handler-exit 500 covers early eval errors. + self.ensure_pending_response_owned(&request_id, *line, *column) .await?; let status_code = match status { @@ -9828,9 +9984,11 @@ impl Interpreter { } } - // Hand the streaming head (and body receiver) to the transport, - // disarming the guard's 500 fallback. + // Commit: take the sender and hand the streaming head to the transport. let (tx, rx) = mpsc::channel::>(RESPONSE_STREAM_BUFFER); + let mut completion = self + .take_pending_response_completion(&request_id, *line, *column) + .await?; match completion.take_sender() { Some(sender) => { if sender @@ -10046,29 +10204,32 @@ impl Interpreter { match outcome { Ok(()) => Ok((Value::Null, ControlFlow::None)), Err(stalled) => { - // Disconnect or stall: drop the handle so the body - // ends, untrack it so a handler that catches this - // keeps no stale id, and surface a `Cancelled` error. - // Both are client-caused cooperative cancellations, - // not handler faults — `Cancelled` keeps the - // concurrent breaker from counting them (256 stalled - // or hung-up clients must not tear the server down). + // Disconnect → Cancelled (cooperative). Stall + // (client still connected, not reading) → Timeout + // (not Cancelled). Post-accept either way so the + // concurrent breaker does not tear the server down. self.server_response_streams.borrow_mut().remove(&handle_id); self.open_response_streams .borrow_mut() .retain(|s| s != &handle_id); - let message = if stalled { - "Cannot write to response stream: the client stopped reading \ - (write timed out)" + if stalled { + Err(RuntimeError::with_kind( + "Cannot write to response stream: the client stopped reading \ + (write timed out)" + .to_string(), + *line, + *column, + ErrorKind::Timeout, + )) } else { - "Cannot write to response stream: the client has disconnected" - }; - Err(RuntimeError::with_kind( - message.to_string(), - *line, - *column, - ErrorKind::Cancelled, - )) + Err(RuntimeError::with_kind( + "Cannot write to response stream: the client has disconnected" + .to_string(), + *line, + *column, + ErrorKind::Cancelled, + )) + } } } } @@ -10086,57 +10247,39 @@ impl Interpreter { column, } => { // Backward compatibility: before `flush` was a streaming command, - // a bare `flush cache` was an ordinary expression statement that - // evaluated the full merged name. Preserve the COMPLETE old - // expression-statement fallback when the full name resolves - // (issue #642) — not only the zero-argument action happy path: - // - zero-arg Function / Overloaded → call it - // - any other bound value (number, text, non-zero-arg overload, …) - // → evaluate-and-discard (old expression-statement no-op success) - // Only when the full name is unbound fall through to stream flush. - // Only the bare-identifier form carries a fallback (see - // `parse_flush_stream`). - if let Some(name) = action_fallback { - let lookup = env.borrow().get(name); - match lookup { - Some(Value::Function(func)) => { - if func.params.is_empty() { - return self - .call_function(&func, vec![], *line, *column) - .await - .map(|value| (value, ControlFlow::None)); - } - // Function exists but is not zero-argument: old - // expression-statement evaluation of the function - // value was a no-op success (did not auto-invoke with - // missing args), not a stream error. - return Ok((Value::Null, ControlFlow::None)); - } - Some(Value::Overloaded(overloaded)) => { - if let Some(func) = overloaded - .overloads - .iter() - .find(|func| func.params.is_empty()) - { - let func = Rc::clone(func); - return self - .call_function(&func, vec![], *line, *column) - .await - .map(|value| (value, ControlFlow::None)); - } - // Overloaded with no zero-argument overload: old - // expression-statement evaluation of the overloaded - // value was a no-op success, not a stream error. - return Ok((Value::Null, ControlFlow::None)); - } - Some(_other) => { - // Non-callable full-name binding (e.g. `store flush - // cache as 1` then `flush cache`): evaluate-and- - // discard, matching the pre-streaming expression - // statement. - return Ok((Value::Null, ControlFlow::None)); - } - None => {} + // the full merged form (including postfix) was an expression + // statement. When the root binding of the legacy AST exists, + // evaluate it with the same ExpressionStatement semantics + // (zero-arg auto-call; parameterized bare call → arity error). + if let Some(fallback_expr) = action_fallback { + let root_name = match fallback_expr { + Expression::Variable(n, ..) => Some(n.as_str()), + Expression::IndexAccess { collection, .. } + | Expression::PropertyAccess { + object: collection, .. + } + | Expression::MethodCall { + object: collection, .. + } => match collection.as_ref() { + Expression::Variable(n, ..) => Some(n.as_str()), + _ => None, + }, + _ => None, + }; + let root_bound = root_name.is_some_and(|n| env.borrow().get(n).is_some()); + if root_bound { + // Reuse ExpressionStatement semantics by dispatching a + // synthetic statement. + return self + .execute_statement( + &Statement::ExpressionStatement { + expression: fallback_expr.clone(), + line: *line, + column: *column, + }, + Rc::clone(&env), + ) + .await; } } let handle_id = self diff --git a/src/parser/ast.rs b/src/parser/ast.rs index 4f5ee462..da53472d 100644 --- a/src/parser/ast.rs +++ b/src/parser/ast.rs @@ -624,14 +624,13 @@ pub enum Statement { /// queued bytes to the transport. FlushStreamStatement { target: Expression, - /// The full merged command phrase (`"flush cache"`) when this parsed from - /// the merged `flush ` form. Backward compatibility: before `flush` - /// existed, `flush cache` was an expression statement that auto-invoked a - /// zero-argument action named `flush cache`. If such an action is defined, - /// the interpreter calls it instead of treating `cache` as a stream, so a - /// pre-existing program is never hijacked. `None` when no action shadowing - /// is possible (e.g. a postfix operand like `flush obj.out`). - action_fallback: Option, + /// Complete old expression-statement AST for the merged `flush …` form + /// (e.g. `Variable("flush cache")`, or `IndexAccess`/`PropertyAccess` over + /// that full name). Before streaming, `flush cache[0]` was an ordinary + /// expression statement; when the root binding exists the interpreter + /// evaluates this fallback instead of a stream flush. `None` when the + /// form cannot collide with a legacy expression (bare non-merged target). + action_fallback: Option, line: usize, column: usize, }, diff --git a/src/parser/expr/binary.rs b/src/parser/expr/binary.rs index 4dfbd0f5..3945b05d 100644 --- a/src/parser/expr/binary.rs +++ b/src/parser/expr/binary.rs @@ -35,6 +35,16 @@ pub(crate) trait BinaryExprParser<'a> { precedence: u8, ) -> Result; + /// Like [`parse_binary_continuation`], but stops before streaming-response + /// clause connectives (`and headers`, `and content type`, `as out`, …) so a + /// `content type` / `headers` operand does not swallow the next clause as a + /// Boolean-AND / `with` continuation. + fn parse_binary_continuation_stopping_at_clause( + &mut self, + left: Expression, + precedence: u8, + ) -> Result; + /// Parses a function/action call expression. /// /// # Parameters @@ -65,16 +75,12 @@ pub(crate) trait BinaryExprParser<'a> { fn parse_of_call_arg_term(&mut self) -> Result; } -impl<'a> BinaryExprParser<'a> for Parser<'a> { - fn parse_binary_expression(&mut self, precedence: u8) -> Result { - let left = self.parse_primary_expression()?; - self.parse_binary_continuation(left, precedence) - } - - fn parse_binary_continuation( +impl<'a> Parser<'a> { + pub(crate) fn parse_binary_continuation_inner( &mut self, mut left: Expression, precedence: u8, + stop_at_clause: bool, ) -> Result { while let Some(token_pos) = self.cursor.peek() { let token = &token_pos.token; @@ -85,6 +91,34 @@ impl<'a> BinaryExprParser<'a> for Parser<'a> { if matches!(token, Token::Eol) || Parser::is_statement_starter(token) { break; } + // Streaming-response clause connectives must not be absorbed as + // Boolean AND / `with` concatenation inside a clause operand. + if stop_at_clause { + if matches!(token, Token::KeywordAs) { + break; + } + if matches!(token, Token::KeywordAnd | Token::KeywordWith) { + let next = self.cursor.peek_n(1).map(|t| &t.token); + let is_clause = match next { + Some(Token::KeywordAs) + | Some(Token::KeywordContent) + | Some(Token::KeywordStatus) => true, + Some(Token::Identifier(id)) => { + id == "headers" + || id.starts_with("headers ") + || id == "content_type" + || id.starts_with("content_type ") + || id.starts_with("content type") + || id == "type" + || id.starts_with("type ") + } + _ => false, + }; + if is_clause { + break; + } + } + } // Precedence ladder (higher binds tighter): // 0: and, or @@ -754,6 +788,29 @@ impl<'a> BinaryExprParser<'a> for Parser<'a> { Ok(left) } +} + +impl<'a> BinaryExprParser<'a> for Parser<'a> { + fn parse_binary_expression(&mut self, precedence: u8) -> Result { + let left = self.parse_primary_expression()?; + self.parse_binary_continuation(left, precedence) + } + + fn parse_binary_continuation( + &mut self, + left: Expression, + precedence: u8, + ) -> Result { + self.parse_binary_continuation_inner(left, precedence, false) + } + + fn parse_binary_continuation_stopping_at_clause( + &mut self, + left: Expression, + precedence: u8, + ) -> Result { + self.parse_binary_continuation_inner(left, precedence, true) + } fn parse_call_expression( &mut self, diff --git a/src/parser/stmt/io.rs b/src/parser/stmt/io.rs index cdb54a47..f016e4f1 100644 --- a/src/parser/stmt/io.rs +++ b/src/parser/stmt/io.rs @@ -78,6 +78,81 @@ impl<'a> Parser<'a> { self.parse_binary_continuation(lead, 0) } + /// Like [`parse_merged_operand_from_lead`], but binary continuation stops + /// before clause connectives (`and`/`with`/`as`) so + /// `content type mime_type of path and headers h` does not swallow `headers` + /// as a Boolean-AND operand (issue #642 re-review). + pub(crate) fn parse_clause_operand_from_lead( + &mut self, + lead: Expression, + ) -> Result { + let lead = self.parse_trailing_postfix(lead)?; + let lead = if matches!(self.cursor.peek().map(|t| &t.token), Some(Token::KeywordOf)) { + let (of_line, of_column) = self + .bump_sync() + .map(|t| (t.line, t.column)) + .expect("peeked `of` immediately above"); + let mut arguments = vec![crate::parser::ast::Argument { + name: None, + value: self.parse_of_call_argument()?, + }]; + while let Some(sep) = self.cursor.peek() { + let is_separator = matches!( + &sep.token, + Token::KeywordAnd | Token::KeywordFrom | Token::KeywordBy + ) || matches!( + &sep.token, + Token::Identifier(id) if id.eq_ignore_ascii_case("length") + ); + // For multi-arg `of` calls, `and` between arguments is fine — + // only stop when we've finished the of-call and the next token + // would start a new clause (handled by binary continuation stop). + if !is_separator { + break; + } + // If `and` is followed by a clause keyword (headers/content/as), + // it is a clause connective, not an of-arg separator. + if matches!(&sep.token, Token::KeywordAnd) + && Self::is_streaming_clause_keyword(self.cursor.peek_n(1).map(|t| &t.token)) + { + break; + } + self.bump_sync(); + arguments.push(crate::parser::ast::Argument { + name: None, + value: self.parse_of_call_argument()?, + }); + } + Expression::FunctionCall { + function: Box::new(lead), + arguments, + line: of_line, + column: of_column, + } + } else { + lead + }; + self.parse_binary_continuation_stopping_at_clause(lead, 0) + } + + fn is_streaming_clause_keyword(tok: Option<&Token>) -> bool { + match tok { + Some(Token::KeywordAs) | Some(Token::KeywordContent) | Some(Token::KeywordStatus) => { + true + } + Some(Token::Identifier(id)) => { + id == "headers" + || id.starts_with("headers ") + || id == "content_type" + || id.starts_with("content_type ") + || id.starts_with("content type") + || id == "type" + || id.starts_with("type ") + } + _ => false, + } + } + /// Alias used by the write-statement parsers. fn parse_write_value_from_lead(&mut self, lead: Expression) -> Result { self.parse_merged_operand_from_lead(lead) diff --git a/src/parser/stmt/web.rs b/src/parser/stmt/web.rs index c1d98826..7f9ea7b5 100644 --- a/src/parser/stmt/web.rs +++ b/src/parser/stmt/web.rs @@ -452,7 +452,7 @@ impl<'a> WebParser<'a> for Parser<'a> { // operators), same as ordinary `` — e.g. // `content type mime_type of path` (issue #642). let lead = Expression::Variable(rest, l, c); - self.parse_merged_operand_from_lead(lead)? + self.parse_clause_operand_from_lead(lead)? } None => self.parse_primary_expression()?, }); @@ -479,7 +479,7 @@ impl<'a> WebParser<'a> for Parser<'a> { content_type = Some(self.parse_primary_expression()?); } else { let lead = Expression::Variable(rest.to_string(), id_line, id_column); - content_type = Some(self.parse_merged_operand_from_lead(lead)?); + content_type = Some(self.parse_clause_operand_from_lead(lead)?); } } // `headers ` (bare or merged `headers `). @@ -495,10 +495,10 @@ impl<'a> WebParser<'a> for Parser<'a> { if rest.is_empty() { headers = Some(self.parse_primary_expression()?); } else { - // Full expression continuation so direct forwarding like - // `headers upstream.headers` and operator/`of` forms bind. + // Clause operand: postfix/`of`/operators but stop before + // the next clause connective (`and content type`, `as`). let lead = Expression::Variable(rest.to_string(), id_line, id_column); - headers = Some(self.parse_merged_operand_from_lead(lead)?); + headers = Some(self.parse_clause_operand_from_lead(lead)?); } } // A connective directly before `as` just joins the clause list to @@ -550,24 +550,17 @@ impl<'a> WebParser<'a> for Parser<'a> { let (target, action_fallback) = if rest.is_empty() { (self.parse_primary_expression()?, None) } else { - // The lexer merged `flush` with the operand identifier, so any postfix - // accessors (`flush streams["a"]`, `flush obj.out`) are left as separate - // tokens. Compose them onto the split-off lead so the operand parses - // consistently with a normal expression instead of dangling. - let lead = Expression::Variable(rest.to_string(), line, column); - let composed = self.parse_trailing_postfix(lead)?; - // Backward compatibility: `flush ` used to auto-invoke a - // zero-argument action named "flush ". Carry the full phrase so - // the interpreter can prefer that action when it exists. Only a - // bare-identifier operand can collide with such an action name; a - // postfix operand (`flush obj.out`) cannot, so it carries no fallback - // (else a defined `flush obj` action would wrongly swallow `.out`). - let fallback = if matches!(composed, Expression::Variable(..)) { - Some(phrase.clone()) - } else { - None - }; - (composed, fallback) + // Stream reading: postfix on the split-off rest (`cache` from + // `flush cache`). Legacy expression: same postfix on the FULL phrase + // (`flush cache`) so `flush cache[0]` / `.property` / `.method()` / + // `at` keep their old expression-statement AST (issue #642 re-review). + let cp = self.cursor.checkpoint(); + let stream_lead = Expression::Variable(rest.to_string(), line, column); + let target = self.parse_trailing_postfix(stream_lead)?; + self.cursor.rewind(cp); + let legacy_lead = Expression::Variable(phrase.clone(), line, column); + let fallback = self.parse_trailing_postfix(legacy_lead)?; + (target, Some(fallback)) }; Ok(Statement::FlushStreamStatement { diff --git a/src/typechecker/mod.rs b/src/typechecker/mod.rs index 93f1ab34..6e7da941 100644 --- a/src/typechecker/mod.rs +++ b/src/typechecker/mod.rs @@ -1046,23 +1046,17 @@ impl TypeChecker { // `Map`) so `close out` is accepted without `close` also // type-checking an ordinary user map. // - // Always bind/refine in the *current* scope: analyzer loop - // scopes are discarded after body analysis, so `out` from - // `start streaming response ... as out` inside `main loop` - // would otherwise be missing here and remain Unknown — - // letting a gradual/file fallback mask an invalid stream - // payload (issue #642). - if let Some(symbol) = self.analyzer.get_symbol_mut(variable_name) { - symbol.symbol_type = Some(Type::Custom("ResponseStream".to_string())); - } else { - self.analyzer.define_or_replace_symbol(Symbol { - name: variable_name.clone(), - kind: SymbolKind::Variable { mutable: true }, - symbol_type: Some(Type::Custom("ResponseStream".to_string())), - line: *_line, - column: *_column, - }); - } + // Always bind in the *current* scope only (shadow, do not + // mutate an outer symbol of the same name via get_symbol_mut + // parent walk). Analyzer loop scopes are discarded after + // body analysis, so we re-create the binding here. + self.analyzer.define_or_replace_symbol(Symbol { + name: variable_name.clone(), + kind: SymbolKind::Variable { mutable: true }, + symbol_type: Some(Type::Custom("ResponseStream".to_string())), + line: *_line, + column: *_column, + }); } } Statement::StreamWriteStatement { @@ -1091,10 +1085,13 @@ impl TypeChecker { self.check_expression_names_defined(value, *line, *column); let value_type = self.infer_expression_type(value); self.check_streamable_payload(&value_type, *line, *column); - } else if matches!(target_type, Type::Text) && has_fallback { - // Concrete text path: the classic file-write reading is taken. - // Validate the fallback (including definedness), not the stream - // `value` the runtime never evaluates here. + } else if has_fallback + && (matches!(target_type, Type::Text) + || matches!(&target_type, Type::Custom(n) if n == "File")) + { + // Concrete text path OR open-file handle (`Custom("File")`): + // the classic file-write reading is taken. Validate the + // fallback (including definedness), not the stream `value`. if let Some(fallback) = fallback_content { self.check_expression_names_defined(fallback, *line, *column); let _ = self.infer_expression_type(fallback); @@ -1132,18 +1129,32 @@ impl TypeChecker { line, column, } => { - // A merged `flush ` may resolve at runtime to the full - // merged name as an ordinary expression statement (backward - // compatibility — see `parse_flush_stream` / issue #642). Stay - // lenient when the full name is bound as an action OR any other - // value; otherwise this is a stream flush and a concrete - // non-stream target is still a static error (`flush n` where - // `n` is a number and there is no binding named `flush n`). - let is_expression_fallback = action_fallback.as_ref().is_some_and(|name| { + // Legacy full-name expression (e.g. Variable("flush cache") or + // IndexAccess over it): when its root is bound, typecheck that + // expression. Otherwise this is a stream flush. + let legacy_root = action_fallback.as_ref().and_then(|e| match e { + Expression::Variable(n, ..) => Some(n.as_str()), + Expression::IndexAccess { collection, .. } + | Expression::PropertyAccess { + object: collection, .. + } + | Expression::MethodCall { + object: collection, .. + } => match &**collection { + Expression::Variable(n, ..) => Some(n.as_str()), + _ => None, + }, + _ => None, + }); + let is_expression_fallback = legacy_root.is_some_and(|name| { self.action_signatures(name).is_some() || self.analyzer.get_symbol(name).is_some() }); - if !is_expression_fallback { + if is_expression_fallback { + if let Some(fb) = action_fallback { + let _ = self.infer_expression_type(fb); + } + } else { let target_type = self.infer_expression_type(target); if !self.is_response_stream_target_type(&target_type) { self.type_error( @@ -4920,15 +4931,9 @@ impl TypeChecker { /// may have stayed silent on a one-sided undefined lead because the other /// reading was defined (issue #642). fn name_is_defined_for_write(&self, name: &str) -> bool { - if self.analyzer.get_symbol(name).is_some() - || self.analyzer.get_action_parameters().contains(name) - || Analyzer::is_builtin_function(name) - || name == "count" - || name == "loopcounter" - { - return true; - } - false + // Match analyzer `name_is_defined` so container properties and inherited + // bindings are not false-rejected on the selected write branch. + self.analyzer.name_is_defined_for_write(name) } /// Walk an expression and report every undefined bare name. Used for the diff --git a/tests/flush_action_backcompat_test.rs b/tests/flush_action_backcompat_test.rs index 35486e3d..94ec650a 100644 --- a/tests/flush_action_backcompat_test.rs +++ b/tests/flush_action_backcompat_test.rs @@ -91,11 +91,37 @@ fn flush_non_callable_full_name_binding_is_expression_statement() { } #[test] -fn flush_overloaded_action_without_zero_arg_is_expression_statement() { - // An overloaded `flush cache` with only a one-parameter overload used to be - // an ordinary bare-expression evaluation (no-op success). Must not become a - // stream error (issue #642). +fn flush_parameterized_single_action_still_arity_errors() { + // A single parameterized `flush cache` action: old expression-statement + // auto-call with zero args produced an arity error — not a silent success + // and not a stream flush (issue #642 re-review). let src = "define action called flush cache with parameters x:\n\ + \x20\x20\x20\x20display x\n\ + end action\n\ + \n\ + flush cache\n"; + let (out, code) = run_src(src); + assert_ne!( + code, + Some(0), + "parameterized flush cache with no args must arity-error; output:\n{out}" + ); + assert!( + out.to_lowercase().contains("argument") + || out.to_lowercase().contains("expected") + || out.contains("1"), + "expected an arity error, got:\n{out}" + ); +} + +#[test] +fn flush_overloaded_without_zero_arg_is_expression_not_stream() { + // True overload set with no zero-arg member: bare name evaluates as an + // overloaded value (expression statement), not a stream flush. + let src = "define action called flush cache with parameters x:\n\ + \x20\x20\x20\x20display x\n\ + end action\n\ + define action called flush cache with parameters x and y:\n\ \x20\x20\x20\x20display x\n\ end action\n\ \n\ @@ -112,3 +138,19 @@ fn flush_overloaded_action_without_zero_arg_is_expression_statement() { "program should reach the trailing display; output:\n{out}" ); } + +#[test] +fn flush_with_postfix_uses_legacy_expression_when_bound() { + // `flush cache[0]` must evaluate IndexAccess on Variable("flush cache"), not + // try to flush stream `cache[0]`. + let src = "store flush cache as [\"a\" and \"b\"]\n\ + flush cache[0]\n\ + display \"OK\"\n"; + let (out, code) = run_src(src); + assert_eq!( + code, + Some(0), + "flush cache[0] with a bound list must be an expression statement; output:\n{out}" + ); + assert!(out.contains("OK"), "expected OK; output:\n{out}"); +} diff --git a/tests/outbound_stream_close_during_read_test.rs b/tests/outbound_stream_close_during_read_test.rs new file mode 100644 index 00000000..6df5cdab --- /dev/null +++ b/tests/outbound_stream_close_during_read_test.rs @@ -0,0 +1,113 @@ +//! Close-during-active-read must cancel the upstream promptly (issue #642 re-review). +//! +//! `take_stream` removes the handle for the await; a concurrent `close` must trip +//! shared cancellation so the active read aborts and the upstream is dropped — +//! not leave the connection open until idle timeout while `put_stream` treats a +//! missing slot as success. + +use std::sync::Arc; +use std::time::{Duration, Instant}; +use tokio::io::{AsyncReadExt, AsyncWriteExt}; +use wfl::Interpreter; +use wfl::config::WflConfig; +use wfl::lexer::lex_wfl_with_positions; +use wfl::parser::Parser; + +async fn spawn_stall_upstream() -> (u16, tokio::sync::oneshot::Receiver<()>) { + let listener = tokio::net::TcpListener::bind("127.0.0.1:0") + .await + .expect("bind"); + let port = listener.local_addr().unwrap().port(); + let (tx, rx) = tokio::sync::oneshot::channel(); + tokio::spawn(async move { + if let Ok((mut sock, _)) = listener.accept().await { + let mut buf = [0u8; 1024]; + let _ = sock.read(&mut buf).await; + let head = "HTTP/1.1 200 OK\r\n\ + Content-Type: text/plain\r\n\ + Transfer-Encoding: chunked\r\n\r\n"; + let _ = sock.write_all(head.as_bytes()).await; + let _ = sock.flush().await; + loop { + match sock.read(&mut buf).await { + Ok(0) | Err(_) => { + let _ = tx.send(()); + return; + } + Ok(_) => {} + } + } + } + }); + (port, rx) +} + +#[tokio::test] +async fn close_during_active_read_drops_upstream_promptly() { + let (port, mut upstream_closed) = spawn_stall_upstream().await; + + // Open stream, start a body read that will park on the stalled upstream, then + // close from another path via a short wait then close — implemented as: + // open, spawn wait for next chunk (parks), wait 200ms, close, then the read + // must fail and upstream drop well before the idle timeout (30s). + let code = format!( + r#" + open url at "http://127.0.0.1:{port}/" and stream response as s + wait for 200 milliseconds + close s + wait for next chunk from s as c + "# + ); + + let (result_tx, result_rx) = std::sync::mpsc::channel::>(); + let client = std::thread::spawn(move || { + let rt = tokio::runtime::Runtime::new().expect("runtime"); + rt.block_on(async { + let tokens = lex_wfl_with_positions(&code); + let program = Parser::new(&tokens).parse().expect("parse"); + let config = WflConfig { + timeout_seconds: 30, + outbound_stream_max_seconds: 60, + ..WflConfig::default() + }; + let mut interp = Interpreter::with_config(Arc::new(config)); + let result = interp.interpret(&program).await; + let summary = match result { + Ok(_) => Ok(()), + Err(errs) => Err(format!("{errs:?}")), + }; + let _ = result_tx.send(summary); + }); + }); + + let start = Instant::now(); + tokio::time::timeout(Duration::from_secs(3), &mut upstream_closed) + .await + .expect("upstream should drop promptly after close, not at 30s idle timeout") + .expect("upstream close sender dropped"); + let elapsed = start.elapsed(); + assert!( + elapsed < Duration::from_secs(2), + "close-during-read should drop upstream promptly; took {elapsed:?}" + ); + + let summary = result_rx + .recv_timeout(Duration::from_secs(5)) + .expect("interpreter should finish"); + // After close, wait for next chunk should fail (closed stream). + assert!( + summary.is_err(), + "read after close must error, got {summary:?}" + ); + let msg = summary.unwrap_err().to_lowercase(); + assert!( + msg.contains("closed") || msg.contains("unknown") || msg.contains("stream"), + "expected a closed-stream error, got: {msg}" + ); + + match tokio::task::spawn_blocking(move || client.join()).await { + Ok(Ok(())) => {} + Ok(Err(panic)) => std::panic::resume_unwind(panic), + Err(e) => panic!("join failed: {e}"), + } +} diff --git a/tests/write_web_postfix_test.rs b/tests/write_web_postfix_test.rs index f2230996..eae001a9 100644 --- a/tests/write_web_postfix_test.rs +++ b/tests/write_web_postfix_test.rs @@ -161,6 +161,45 @@ fn streaming_response_content_type_of_call_parses() { } } +#[test] +fn streaming_response_content_type_then_headers_both_orders() { + // Clause connectives must not be swallowed as Boolean AND (both orders). + let a = parse( + "start streaming response to req with status 200 and content type ct and headers h as out\n", + ); + match &a.statements[0] { + Statement::StartStreamingResponseStatement { + content_type, + headers, + .. + } => { + assert!(content_type.is_some(), "content type must bind"); + assert!( + headers.is_some(), + "headers must bind (not swallowed by content type)" + ); + } + other => panic!("expected StartStreamingResponseStatement, got {other:#?}"), + } + let b = parse( + "start streaming response to req with status 200 and headers h and content type ct as out\n", + ); + match &b.statements[0] { + Statement::StartStreamingResponseStatement { + content_type, + headers, + .. + } => { + assert!(headers.is_some(), "headers must bind"); + assert!( + content_type.is_some(), + "content type must bind (not swallowed by headers)" + ); + } + other => panic!("expected StartStreamingResponseStatement, got {other:#?}"), + } +} + #[test] fn write_line_of_call_argument_absorbs_arithmetic() { // `double of n minus 1` must parse as `double of (n minus 1)` — the same From 8e8be0fcde944d0d7b357b94d5951497af5ff0b7 Mon Sep 17 00:00:00 2001 From: Brad Byrd Date: Fri, 24 Jul 2026 22:20:12 -0500 Subject: [PATCH 091/132] fix: finalize issue #642 pass to harden server stability and streaming This completion pass makes the web server more resilient by ensuring that individual client disconnects or timeouts no longer risk stopping the entire server. It fixes several bugs in how data streams are managed, ensuring they clean up after themselves and provide accurate error messages instead of generic failures. The update also improves the code parser so that web commands like `write`, `headers`, and `flush` now support complex calculations and list lookups just like ordinary code. Additionally, it repairs several platform-specific issues on Windows and adds a comprehensive suite of tests to prove the system can handle messy network conditions and high traffic. Fixes #642 --- Dev diary/2026-07-24-issue-642-completion.md | 199 +++++ .../2026-07-24-issue-642-p1-followups.md | 4 + .../2026-07-24-issue-642-rereview-fixes.md | 5 + Docs/reference/configuration-reference.md | 8 +- TestPrograms/subprocess_blocking_helper.wfl | 4 + TestPrograms/subprocess_comprehensive.wfl | 51 +- crates/wflpkg/src/archive.rs | 4 +- scripts/run_integration_tests.ps1 | 53 +- scripts/run_web_tests.ps1 | 49 ++ src/analyzer/mod.rs | 26 +- src/analyzer/static_analyzer.rs | 51 +- src/interpreter/mod.rs | 603 +++++++++++--- src/parser/ast.rs | 6 + src/parser/expr/binary.rs | 234 +++++- src/parser/expr/primary.rs | 183 ++++- src/parser/stmt/io.rs | 111 ++- src/parser/stmt/patterns.rs | 21 +- src/parser/stmt/web.rs | 20 +- src/typechecker/mod.rs | 166 ++-- .../ambiguous_write_branch_typecheck_test.rs | 218 ++++- .../concurrent_disconnect_paths_burst_test.rs | 761 +++++++++++++++--- .../dropped_interpret_server_cleanup_test.rs | 367 ++++++--- tests/execute_file_test.rs | 4 +- tests/file_io_performance_test.rs | 24 +- tests/flush_action_backcompat_test.rs | 152 ++++ tests/http_stream_test.rs | 10 + tests/outbound_stream_open_expiry_test.rs | 37 + tests/outbound_stream_reaper_race_test.rs | 40 +- tests/response_stream_backpressure_test.rs | 162 +++- tests/stream_handle_type_test.rs | 10 +- tests/write_web_postfix_test.rs | 392 +++++++++ 31 files changed, 3290 insertions(+), 685 deletions(-) create mode 100644 Dev diary/2026-07-24-issue-642-completion.md create mode 100644 TestPrograms/subprocess_blocking_helper.wfl diff --git a/Dev diary/2026-07-24-issue-642-completion.md b/Dev diary/2026-07-24-issue-642-completion.md new file mode 100644 index 00000000..1498a12c --- /dev/null +++ b/Dev diary/2026-07-24-issue-642-completion.md @@ -0,0 +1,199 @@ +# Dev Diary - 2026-07-24: issue #642 completion pass + +Issue [#642](https://github.com/WebFirstLanguage/wfl/issues/642) re-reviewed +PR #641 at `b25aed57` and identified five P1 groups plus missing R3 +lifecycle evidence. The completion pass fixes the remaining behavior, hardens +the tests so they prove the promised boundaries, and records unrelated +platform defects discovered by the full presubmit. + +## Risk and compatibility + +- **Risk class:** R3. +- **Issue triggers:** concurrency, cancellation, HTTP lifecycle, streaming, + resource ownership, arbitrary duration configuration, and backward + compatibility of existing WFL expressions. +- **Additional gate trigger:** untrusted archive paths and path containment. + This was a pre-existing Windows portability defect found by the full gate, + not a sixth issue #642 requirement. +- **Public contract:** no existing WFL syntax is intentionally removed or + changed. The fixes restore classic `write` and `flush` expression behavior + and preserve typed timeout/cancellation outcomes. +- **External state:** none. Rollback is a source revert; there is no data + migration or persistent-state recovery step. + +## Selected design + +The completion pass keeps the existing architecture and hardens each boundary: + +1. **Concurrent server:** handler state records sticky request acceptance. + A centralized classifier separates request-local outcomes from structural + pre-request failures. Pending-response ownership is checked before + expression evaluation and consumed atomically only at response commit. +2. **Outbound streams:** each handle keeps a stable slot and a first-wins + `watch` terminal reason. Expiry records a `Timeout` tombstone, wakes an + active reader, drops a parked upstream body, and refuses reinsertion. EOF, + error, and close abort the per-stream reaper. +3. **Ambiguous writes:** type checking follows the branch selected by a + concrete target and checks every viable branch for a gradual target. + Typechecker-local and inherited container context participates in + definedness and property typing. +4. **Merged operands:** one seeded-expression continuation parser is shared by + write operands, response content type, response headers, and legacy flush + candidates. Response-clause boundaries propagate through recursive primary + wrappers and explicit-call argument lists. +5. **Flush compatibility:** the parser records the original merged binding as + `legacy_binding` metadata before split/find/replace rewrites. Analyzer, + static unused-variable analysis, typechecker, and runtime consult the same + metadata, then dispatch the full fallback AST through ordinary + expression-statement semantics. + +This removes the reported races and restores grammar parity without changing +public `ErrorKind` variants or adding a second request-lease subsystem. + +## Acceptance criteria to regression coverage + +| Issue requirement | Regression evidence | +|---|---| +| Request-local failures cannot stop the server | `concurrent_disconnect_paths_burst_test`: two serialized 256-client waves per disconnect path (512 disconnects total), exact checkpoint/ack synchronization, handler-start ordinal 768 (the initial 256 plus one replacement for every consumed disconnect result), then `/ping`; every socket operation and join is bounded | +| Owned missing pending entry is cancellation; duplicate remains an error | `concurrent_handler_classification_tests::missing_pending_entry_is_cancelled_only_while_the_handler_owns_it` | +| Repeated finite request waits survive | 257 direct classifier observations plus a real server using 1 ms waits until handler-start ordinal 512 | +| Expiry/read ownership is atomic and preserves `Timeout` | active-read real-socket test, unread-expiry test, and deterministic `take -> expire -> ready result -> put` unit ordering | +| Stream EOF is stable | final unterminated line, then exactly one EOF `nothing`, then the documented closed-handle result | +| Reaper resources remain bounded | retained-runtime live-reaper counter across rapid close, clean EOF, truncated-body error, and opened-but-unread expiry | +| Extreme duration cannot panic or disable the cap | deadline unit test proves `u64::MAX` becomes a finite one-year cap and diagnostics use that effective duration | +| Concrete and gradual write branches are sound | one-sided leads in both directions, property roots, handler/action scopes, direct and inherited container properties, wrapped file/stream/list payloads, gradual definedness, and gradual payload tests | +| Merged operands match ordinary expression grammar | 3 x 7 parser matrix plus clause boundaries in concatenation, `at` indexes, nested `of` calls, builtins, unary operands, explicit `call ... with ...` arguments, and `file exists at ` | +| Full flush fallback is preserved | non-callable, overload, nested postfix, binary continuation, `of` calls, split/find/replace rewrites, invalid container-property types, and unused-variable accounting for `legacy_binding` and fallback operands | +| Backpressure test proves the intended path | client confirms the 200 head, stays connected without reading, then asserts the exact typed stall timeout and lower bound | +| Dropped pending request sends the explicit 500 | exact dequeue/drop/release synchronization followed by exact status, content type, body, prompt completion, and clean interpreter/server joins | + +## Red evidence observed + +The following focused regressions failed before their matching implementation +changes: + +- GitHub Actions run `30106107011` and the local focused run: + `http_stream_test::test_next_line_returns_final_unterminated_line` returned + `Unknown or already-closed stream handle 'httpstream1'` instead of EOF + `nothing`. +- Active hard-expiry cancellation produced `ErrorKind::General` with a closed + stream message instead of `ErrorKind::Timeout`. +- Reading an unread expired stream produced an unknown-handle General error + instead of a typed timeout. +- Container method `write line value to "C:/tmp/out"` falsely reported + `Variable 'line value' is not defined`. +- Wrapped write operands skipped undefined names in concrete-file, + concrete-stream, and gradual-target branches. +- Unmerged response content type `"text/" with subtype` stopped parsing at + `with`. +- Response clauses were swallowed by concatenation operands, `at` indexes, and + recursively nested `of`/builtin/unary expressions. +- `call render with value and headers h` swallowed `headers h` as a second + explicit-call argument. +- `file exists at paths at kind and headers h` absorbed the headers clause + into the nested index/Boolean-And expression. +- `flush cache plus 1` left `plus` dangling, and `flush cache[0][0]` selected + the short `cache` root at runtime. +- Split/find/replace flush rewrites discarded the original legacy binding, and + an invalid live container property was not rejected by the typechecker. +- Static unused-variable analysis reported both `flush cache` and `dead`; only + `dead` should have been unused. +- The original lifecycle tests could finish without proving the post- + disconnect checkpoint, exact drop point, bounded joins, or admission beyond + the breaker threshold. Deterministic synchronization was added before the + production behavior was accepted as Green. + +The original server-breaker defect is anchored to issue head `b25aed57`. Only +GitHub Actions run `30106107011` is retained policy-compliant CI Red evidence. +The other Reds were observed locally but are not preserved as Red commits or +durable CI artifacts. + +A pre-existing zero-byte `.git/objects/maintenance.lock` (dated +2026-07-18 03:36 local time) prevented Git object writes; its owning process +was not established. Consequently there is no committed Red-to-Green ancestry +for the local regressions, including the existing archive reproduction test. +This is a testing-policy handoff limitation and must not be overstated as +formal Red evidence. + +## Unrelated full-gate defects repaired + +These changes are not issue #642 acceptance items, but each blocked or weakened +the repository's required verification: + +- `file_io_performance_test::test_directory_listing_performance` recursively + scanned the repository and `target` (about 65,000 files), exceeded its + 10-second bound at 11.28 seconds, and left 30 fixtures. It now uses an + auto-cleaned temporary directory; focused Green was 1/1 in 0.02 seconds. +- On Windows, `Path::is_absolute()` did not classify the portable archive entry + `/etc/shadow` as absolute. The existing containment guard still rejected it + as escaping the destination, but with the wrong classification and message. + `Path::has_root()` now performs portable rooted-path rejection; the + `wflpkg` security suite is 31/31. +- `execute_file_test` reserved port 58123. It now uses `free_tcp_port()` to + avoid unrelated local collisions. +- Windows PowerShell 5.1 rejected inherited environments containing identical + `Path` and `PATH` keys before `Start-Process` could run. Both official + scripts now canonicalize only identical duplicates and fail closed on + conflicting values. The integration runner also retains the child process + handle before a timed wait so a real exit code is available. +- `subprocess_comprehensive.wfl` treated shell-only `echo` as a Windows + executable. It now uses `cargo --version`, an existing runner prerequisite, + explicitly waits after output capture so shutdown is orphan-free, and uses a + repo-owned blocking WFL helper to prove the child is running before kill and + absent afterward. The helper carries the first-line `CI-SKIP` directive used + by both platform runners so it is never treated as a standalone test. + +## Green evidence + +Focused final results: + +- Language-focused combined run: 68/68. +- `write_web_postfix_test`: 21/21. +- Static-analyzer focused units: 15/15. +- Parser units: 110/110. +- Strengthened disconnect binary: 4/4 in about 11.5 seconds; the handler-entry + barrier case also passed focused 1/1. +- Directory-performance fixture: 1/1 in 0.02 seconds. +- `cargo test -p wflpkg --test security_tests --verbose`: 31/31. +- Final subprocess fixture: exit 0 with a live-child kill assertion and no + orphan warning. + +Final-tree gates: + +| Command | Result | +|---|---| +| `cargo fmt --all -- --check` | pass | +| `git diff --check` | pass | +| `cargo clippy --all-targets --all-features -- -D warnings` | pass | +| `cargo build --release` | pass | +| `cargo test --all --verbose --jobs 2` | pass; core 627 passed / 6 ignored, all workspace integration packages passed, WFL doctests 28 passed / 11 ignored | +| `scripts/run_integration_tests.ps1 -TestOnly` | pass; Rust integration binaries passed and TestPrograms finished 110 passed / 0 failed / 24 explicit skips | +| `scripts/run_web_tests.ps1` | pass; 2/2 HTTP tests, TLS script case explicitly skipped because OpenSSL was unavailable | +| Git Bash syntax + first-line skip probe | pass; the Unix runner parses and recognizes the helper's `CI-SKIP` directive | +| `python scripts/validate_docs_examples.py --ci --force` | pass; 18/18 examples across validation layers | + +The first unbounded `cargo test --all --verbose` attempt hit a pre-test Windows +linker fan-out failure, `LNK1104: cannot open msvcrt.lib`. The library was +present and readable, and the exact failed target linked immediately +afterward. `--jobs 2` preserved the complete test selection while bounding +concurrent linkers. + +The Cargo cache also reported a read-only last-use database in this sandbox. +That warning did not affect dependency resolution, compilation, or test +selection. + +## Residual risk and recovery + +- Each disconnect path covers 512 clients in two 256-client waves, with at + most 256 simultaneous handlers. This proves every disconnected result is + consumed before the post-check while staying within the configured admission + bound. +- The repeated finite-timeout proof is separate and uses handler-start ordinal + 512. +- `0` continues to disable the outbound absolute cap. Positive values above + one year use the documented one-year effective cap. +- The script-level TLS case was not run because OpenSSL was unavailable, but + the Rust TLS integration suite passed 8/8 in the workspace test gate. +- No deployment or external state changed. Reverting this source/test set is + the rollback; forward repair is preferred if a platform timing or socket- + limit issue appears. diff --git a/Dev diary/2026-07-24-issue-642-p1-followups.md b/Dev diary/2026-07-24-issue-642-p1-followups.md index c12cc61a..484895c8 100644 --- a/Dev diary/2026-07-24-issue-642-p1-followups.md +++ b/Dev diary/2026-07-24-issue-642-p1-followups.md @@ -1,5 +1,9 @@ # Dev Diary — 2026-07-24: issue #642 PR #641 follow-up P1s +> **Superseded:** this records the first implementation pass, not the final +> verified state. See `2026-07-24-issue-642-completion.md` for the subsequent +> correctness fixes and strengthened R3 evidence. + Follow-up to the exact-head re-review of #641 (`b25aed57`). CI was green but five P1 lifecycle/compatibility blockers remained. Risk class **R3** (concurrency, cancellation, lifecycle, streaming, compatibility). diff --git a/Dev diary/2026-07-24-issue-642-rereview-fixes.md b/Dev diary/2026-07-24-issue-642-rereview-fixes.md index 0f81f39a..8197b03c 100644 --- a/Dev diary/2026-07-24-issue-642-rereview-fixes.md +++ b/Dev diary/2026-07-24-issue-642-rereview-fixes.md @@ -1,5 +1,10 @@ # Dev Diary — 2026-07-24: issue #642 re-review fixes +> **Superseded:** this records an intermediate branch state. The completion +> audit found remaining typed-timeout, EOF, parser, container-property, and test +> evidence gaps. See `2026-07-24-issue-642-completion.md` for the corrected +> design and final evidence. + Follow-up to the maintainer checklist on the #642 round. ## CI diff --git a/Docs/reference/configuration-reference.md b/Docs/reference/configuration-reference.md index 2311ebb7..8c782d72 100644 --- a/Docs/reference/configuration-reference.md +++ b/Docs/reference/configuration-reference.md @@ -214,7 +214,7 @@ All keys currently loaded from config files, with defaults. | `web_server_max_response_size` | integer ≥ 1 | `67108864` (64 MiB) | Max handler or outbound HTTP response body size (bytes) | | `web_server_request_queue_bound` | integer ≥ 1 | `256` | Max queued HTTP requests before shedding with 503 | | `web_server_response_timeout_seconds` | integer ≥ 0 | `300` | Seconds to await a handler before shedding with 504; `0` disables | -| `outbound_stream_max_seconds` | integer ≥ 0 | `300` | Absolute total lifetime (seconds) of one outbound streaming response, distinct from the per-read idle timeout; `0` disables | +| `outbound_stream_max_seconds` | integer ≥ 0 | `300` | Absolute total lifetime (seconds) of one outbound streaming response, distinct from the per-read idle timeout; `0` disables; values above one year use the one-year safety cap | | `web_socket_queue_bound` | integer ≥ 1 | `1024` | Max queued frames/events per WebSocket channel before shedding | | `web_socket_max_connections` | integer ≥ 1 | `1024` | Max simultaneous live WebSocket connections | | `web_socket_max_message_size` | integer ≥ 1 | `1048576` (1 MiB) | Max size of a single WebSocket text message (bytes); larger frames are dropped | @@ -575,7 +575,11 @@ Absolute total lifetime, in seconds, of a single **outbound** streaming response - **Default:** `300` - **Example:** `outbound_stream_max_seconds = 60` -A value of `0` disables the absolute cap (the idle timeout still applies per read). +A value of `0` disables the absolute cap (the idle timeout still applies per +read). Positive values above 31,536,000 seconds (one year) are safely clamped to +one year when the runtime creates the deadline. This keeps extreme configuration +values finite and prevents platform `Instant` overflow; timeout diagnostics +report the effective clamped duration. #### `web_socket_queue_bound` diff --git a/TestPrograms/subprocess_blocking_helper.wfl b/TestPrograms/subprocess_blocking_helper.wfl new file mode 100644 index 00000000..e83f9297 --- /dev/null +++ b/TestPrograms/subprocess_blocking_helper.wfl @@ -0,0 +1,4 @@ +// CI-SKIP: helper process for subprocess_comprehensive.wfl +// Helper for subprocess_comprehensive.wfl. The integration runner skips this +// standalone file; its parent starts it and proves live-process termination. +wait for 10 seconds diff --git a/TestPrograms/subprocess_comprehensive.wfl b/TestPrograms/subprocess_comprehensive.wfl index 4d6fbd77..1d0ed6d1 100644 --- a/TestPrograms/subprocess_comprehensive.wfl +++ b/TestPrograms/subprocess_comprehensive.wfl @@ -6,19 +6,19 @@ display "" // Test 1: Simple Command Execution display "Test 1: Execute Command" -wait for execute command "echo Hello from subprocess" as cmd_result +wait for execute command "cargo --version" as cmd_result display " Command executed successfully" display "" // Test 2: Execute Without Storing Result display "Test 2: Execute Without Variable" -wait for execute command "echo No variable needed" +wait for execute command "cargo --version" display " Execution completed" display "" // Test 3: Background Process Spawn and Wait display "Test 3: Spawn and Wait for Process" -wait for spawn command "echo Background process" as bg_proc +wait for spawn command "cargo --version" as bg_proc display " Process spawned" wait for process bg_proc to complete as exit_status display " Process completed" @@ -26,7 +26,7 @@ display "" // Test 4: Process Status Check display "Test 4: Check Process Status" -wait for spawn command "echo Quick task" as status_proc +wait for spawn command "cargo --version" as status_proc store proc_status as process status_proc is running check if proc_status: display " Process was running (or completed too fast)" @@ -37,17 +37,48 @@ display "" // Test 5: Process Termination display "Test 5: Kill Process" -wait for spawn command "echo Terminated" as term_proc -wait for 100 milliseconds -kill process term_proc +// Use the repo-owned WFL binary and a blocking fixture, then prove the child is +// live before termination and absent afterward. +check if file exists at "target/release/wfl.exe": + wait for spawn command "target/release/wfl.exe" with arguments ["TestPrograms/subprocess_blocking_helper.wfl"] as windows_term_proc + wait for 100 milliseconds + store windows_term_running as process windows_term_proc is running + check if windows_term_running: + kill process windows_term_proc + otherwise: + store windows_live_kill_failure as 1 divided by 0 + display windows_live_kill_failure + end check + store windows_term_after_kill as process windows_term_proc is running + check if windows_term_after_kill: + store windows_post_kill_failure as 1 divided by 0 + display windows_post_kill_failure + end check +otherwise: + wait for spawn command "target/release/wfl" with arguments ["TestPrograms/subprocess_blocking_helper.wfl"] as unix_term_proc + wait for 100 milliseconds + store unix_term_running as process unix_term_proc is running + check if unix_term_running: + kill process unix_term_proc + otherwise: + store unix_live_kill_failure as 1 divided by 0 + display unix_live_kill_failure + end check + store unix_term_after_kill as process unix_term_proc is running + check if unix_term_after_kill: + store unix_post_kill_failure as 1 divided by 0 + display unix_post_kill_failure + end check +end check display " Process terminated" display "" // Test 6: Read Process Output display "Test 6: Capture Process Output" -wait for spawn command "echo Output captured" as out_proc +wait for spawn command "cargo --version" as out_proc wait for 200 milliseconds wait for read output from process out_proc as captured_data +wait for process out_proc to complete display " Output captured successfully" display "" @@ -63,8 +94,8 @@ display "" // Test 8: Multiple Processes display "Test 8: Multiple Concurrent Processes" -wait for spawn command "echo Process 1" as p1 -wait for spawn command "echo Process 2" as p2 +wait for spawn command "cargo --version" as p1 +wait for spawn command "cargo --version" as p2 display " Two processes spawned" wait for process p1 to complete wait for process p2 to complete diff --git a/crates/wflpkg/src/archive.rs b/crates/wflpkg/src/archive.rs index b244ab31..77fcf5fe 100644 --- a/crates/wflpkg/src/archive.rs +++ b/crates/wflpkg/src/archive.rs @@ -267,7 +267,9 @@ pub fn extract_archive(archive_path: &Path, dest_dir: &Path) -> Result<(), Packa .into_owned(); // Reject absolute paths - if entry_path.is_absolute() { + // `Path::is_absolute` requires a drive prefix on Windows, but archive + // paths are portable and a leading slash is still rooted there. + if entry_path.has_root() { return Err(PackageError::General(format!( "Archive contains absolute path: {}", entry_path.display() diff --git a/scripts/run_integration_tests.ps1 b/scripts/run_integration_tests.ps1 index 2a7dd289..0bc49a27 100644 --- a/scripts/run_integration_tests.ps1 +++ b/scripts/run_integration_tests.ps1 @@ -22,6 +22,52 @@ if ($Help) { exit 0 } +# Windows treats environment variable names case-insensitively, but a process +# launched from a cross-platform host can still inherit both Path and PATH. +# Windows PowerShell 5.1's Start-Process rejects that environment block. Keep a +# single canonical key only when the duplicate values are identical; never +# merge conflicting executable search paths. +if ([System.Environment]::OSVersion.Platform -eq [System.PlatformID]::Win32NT) { + $processEnvironment = [System.Environment]::GetEnvironmentVariables( + [System.EnvironmentVariableTarget]::Process + ) + $pathKeys = @( + $processEnvironment.Keys | Where-Object { + [string]::Equals( + [string]$_, + "Path", + [System.StringComparison]::OrdinalIgnoreCase + ) + } + ) + + if ($pathKeys.Count -gt 1) { + $pathValue = [string]$processEnvironment[$pathKeys[0]] + foreach ($pathKey in $pathKeys) { + if (-not [string]::Equals( + $pathValue, + [string]$processEnvironment[$pathKey], + [System.StringComparison]::Ordinal + )) { + throw "Conflicting case-variant PATH values in the process environment." + } + } + + foreach ($pathKey in $pathKeys) { + [System.Environment]::SetEnvironmentVariable( + [string]$pathKey, + $null, + [System.EnvironmentVariableTarget]::Process + ) + } + [System.Environment]::SetEnvironmentVariable( + "Path", + $pathValue, + [System.EnvironmentVariableTarget]::Process + ) + } +} + Write-Host "[INFO] WFL Integration Test Runner" -ForegroundColor Blue Write-Host "[INFO] ==========================" -ForegroundColor Blue @@ -112,7 +158,8 @@ $SkipTests = @( "websocket_test.wfl", # WebSocket - needs WS client "web_route_params_test.wfl", # Web server - tested via run_web_tests.ps1 "module_helper.wfl", # Helper module, not a standalone program - "module_bare_zero_arg_helper.wfl" # Helper module for #592 fixture, not standalone + "module_bare_zero_arg_helper.wfl", # Helper module for #592 fixture, not standalone + "subprocess_blocking_helper.wfl" # Helper process for subprocess_comprehensive.wfl ) # Tests that intentionally end with an error; they pass when wfl exits nonzero @@ -172,6 +219,10 @@ if (-not (Test-Path "TestPrograms")) { $outFile = New-TemporaryFile $errFile = New-TemporaryFile $process = Start-Process -FilePath ".\$BinaryPath" -ArgumentList $wflArgs -NoNewWindow -PassThru -RedirectStandardOutput $outFile.FullName -RedirectStandardError $errFile.FullName + # Windows PowerShell 5.1 can discard the process handle before a + # timed WaitForExit, leaving ExitCode null. Materialize it while the + # child is live so timeout and exit-code assertions remain valid. + $null = $process.Handle $completed = $process.WaitForExit($TestTimeout * 1000) $isExpectedFail = $ExpectedFailTests -contains $wflFile.Name diff --git a/scripts/run_web_tests.ps1 b/scripts/run_web_tests.ps1 index 8375d7ac..c4b549a7 100644 --- a/scripts/run_web_tests.ps1 +++ b/scripts/run_web_tests.ps1 @@ -22,6 +22,52 @@ if ($Help) { exit 0 } +# Windows treats environment variable names case-insensitively, but a process +# launched from a cross-platform host can still inherit both Path and PATH. +# Windows PowerShell 5.1's Start-Process rejects that environment block. Keep a +# single canonical key only when the duplicate values are identical; never +# merge conflicting executable search paths. +if ([System.Environment]::OSVersion.Platform -eq [System.PlatformID]::Win32NT) { + $processEnvironment = [System.Environment]::GetEnvironmentVariables( + [System.EnvironmentVariableTarget]::Process + ) + $pathKeys = @( + $processEnvironment.Keys | Where-Object { + [string]::Equals( + [string]$_, + "Path", + [System.StringComparison]::OrdinalIgnoreCase + ) + } + ) + + if ($pathKeys.Count -gt 1) { + $pathValue = [string]$processEnvironment[$pathKeys[0]] + foreach ($pathKey in $pathKeys) { + if (-not [string]::Equals( + $pathValue, + [string]$processEnvironment[$pathKey], + [System.StringComparison]::Ordinal + )) { + throw "Conflicting case-variant PATH values in the process environment." + } + } + + foreach ($pathKey in $pathKeys) { + [System.Environment]::SetEnvironmentVariable( + [string]$pathKey, + $null, + [System.EnvironmentVariableTarget]::Process + ) + } + [System.Environment]::SetEnvironmentVariable( + "Path", + $pathValue, + [System.EnvironmentVariableTarget]::Process + ) + } +} + Write-Host "[INFO] WFL Web Server Test Runner" -ForegroundColor Blue Write-Host "[INFO] ============================" -ForegroundColor Blue @@ -150,6 +196,7 @@ function Test-WflWebServer { $outLog = Join-Path ([System.IO.Path]::GetTempPath()) "wfl_web_$Port.out.log" $errLog = Join-Path ([System.IO.Path]::GetTempPath()) "wfl_web_$Port.err.log" $serverProcess = Start-Process -FilePath ".\$BinaryPath" -ArgumentList $TestFile -NoNewWindow -PassThru -RedirectStandardOutput $outLog -RedirectStandardError $errLog + $null = $serverProcess.Handle try { # Wait for the server to start, bounded by a real wall-clock deadline. @@ -234,6 +281,7 @@ if (Test-Path "TestPrograms\web_route_params_test.wfl") { $routeOutLog = Join-Path ([System.IO.Path]::GetTempPath()) "wfl_web_route.out.log" $routeErrLog = Join-Path ([System.IO.Path]::GetTempPath()) "wfl_web_route.err.log" $routeProcess = Start-Process -FilePath ".\$BinaryPath" -ArgumentList "TestPrograms\web_route_params_test.wfl" -NoNewWindow -PassThru -RedirectStandardOutput $routeOutLog -RedirectStandardError $routeErrLog + $null = $routeProcess.Handle try { # Wall-clock deadline (see Test-WflWebServer) instead of a retry count. @@ -331,6 +379,7 @@ if (Test-Path "TestPrograms\web_server_tls.wfl") { # Distinct redirect targets under the (auto-cleaned) temp dir; the same # path for both streams errors on PowerShell 7. $tlsProcess = Start-Process -FilePath $absBinary -ArgumentList $absTest -WorkingDirectory $tlsDir -NoNewWindow -PassThru -RedirectStandardOutput (Join-Path $tlsDir "server.out.log") -RedirectStandardError (Join-Path $tlsDir "server.err.log") + $null = $tlsProcess.Handle try { # Probe readiness via the redirect port: it answers natively and does diff --git a/src/analyzer/mod.rs b/src/analyzer/mod.rs index 7c0b6021..20ff45d0 100644 --- a/src/analyzer/mod.rs +++ b/src/analyzer/mod.rs @@ -1691,15 +1691,16 @@ impl Analyzer { Statement::FlushStreamStatement { target, + legacy_binding, action_fallback, .. } => { // When the legacy full-name expression's root is defined, analyze // that expression (expression-statement path). Otherwise analyze // the stream target. - let legacy_root_defined = action_fallback.as_ref().is_some_and(|expr| { - Self::expression_root_name(expr).is_some_and(|n| self.name_is_defined(n)) - }); + let legacy_root_defined = legacy_binding + .as_deref() + .is_some_and(|name| self.name_is_defined(name)); if legacy_root_defined { if let Some(fb) = action_fallback { self.analyze_expression(fb); @@ -3828,25 +3829,6 @@ impl Analyzer { false } - /// Root variable name of an expression used as a flush legacy fallback - /// (`flush cache[0]` → `"flush cache"`). - fn expression_root_name(expr: &Expression) -> Option<&str> { - match expr { - Expression::Variable(name, ..) => Some(name.as_str()), - Expression::IndexAccess { collection, .. } - | Expression::PropertyAccess { - object: collection, .. - } - | Expression::MemberAccess { - object: collection, .. - } - | Expression::MethodCall { - object: collection, .. - } => Self::expression_root_name(collection), - _ => None, - } - } - fn analyze_expression(&mut self, expression: &Expression) { // Recursive front-end checkpoint for expressions. `analyze_statement` // polls per statement, but one statement can hold an arbitrarily large diff --git a/src/analyzer/static_analyzer.rs b/src/analyzer/static_analyzer.rs index 30d64cbd..747faf23 100644 --- a/src/analyzer/static_analyzer.rs +++ b/src/analyzer/static_analyzer.rs @@ -1355,8 +1355,26 @@ impl Analyzer { self.mark_used_in_expression(fallback, usages); } } - Statement::FlushStreamStatement { target, .. } => { + Statement::FlushStreamStatement { + target, + legacy_binding, + action_fallback, + .. + } => { + // The runtime chooses between the stream target and the legacy + // expression interpretation. Mark both conservatively, matching + // ambiguous stream writes, and mark the preserved merged binding + // separately because split/find/replace rewrites may legitimately + // discard that seed from the fallback AST. self.mark_used_in_expression(target, usages); + if let Some(name) = legacy_binding + && let Some(usage) = usages.get_mut(name) + { + usage.used = true; + } + if let Some(fallback) = action_fallback { + self.mark_used_in_expression(fallback, usages); + } } Statement::HttpStreamStatement { url, @@ -2248,6 +2266,37 @@ display ln"; ); } + #[test] + fn test_legacy_flush_binding_and_fallback_operands_are_not_reported_unused() { + // `replace ... in ...` rewrites the expression AST and discards its + // seeded `flush cache` leaf. The explicit legacy-binding metadata must + // therefore count that declaration as used independently of the + // rewritten target/fallback expression. + let input = "create pattern letter_a:\n\ + \x20\x20\x20\x20\"a\"\n\ + end pattern\n\ + store flush cache as 1\n\ + store replacement_value as \"z\"\n\ + store text_value as \"abc\"\n\ + flush cache replace letter_a with replacement_value in text_value\n\ + store dead as \"never read\""; + let tokens = crate::lexer::lex_wfl_with_positions(input); + let program = crate::parser::Parser::new(&tokens).parse().unwrap(); + + let analyzer = Analyzer::new(); + let diagnostics = analyzer.check_unused_variables(&program, 0); + + assert_eq!( + diagnostics.len(), + 1, + "expected only `dead` unused, got: {diagnostics:?}" + ); + assert!( + diagnostics[0].message.contains("dead"), + "the legacy binding and fallback operands must count as used; got: {diagnostics:?}" + ); + } + #[test] fn test_respond_headers_expression_marks_variable_used() { // Regression: a variable referenced only in the diff --git a/src/interpreter/mod.rs b/src/interpreter/mod.rs index e1bcf1e2..9df6da87 100644 --- a/src/interpreter/mod.rs +++ b/src/interpreter/mod.rs @@ -104,6 +104,27 @@ const RESPONSE_STREAM_BUFFER: usize = 64; /// beyond this run as handlers free up (and the transport sheds 503 if its queue /// also fills). const CONCURRENT_HANDLER_LIMIT: usize = 256; +const MAX_CONSECUTIVE_HANDLER_FAILURES: u32 = 256; +const REQUEST_WAIT_TIMEOUT_PREFIX: &str = "Timeout waiting for request"; + +#[derive(Debug, Clone, Copy, PartialEq, Eq)] +enum ConcurrentHandlerDisposition { + RequestLocal, + Structural, +} + +fn classify_concurrent_handler_error( + error: &RuntimeError, + accepted_request: bool, +) -> ConcurrentHandlerDisposition { + let request_wait_timeout = + error.kind == ErrorKind::Timeout && error.message.starts_with(REQUEST_WAIT_TIMEOUT_PREFIX); + if accepted_request || error.kind == ErrorKind::Cancelled || request_wait_timeout { + ConcurrentHandlerDisposition::RequestLocal + } else { + ConcurrentHandlerDisposition::Structural + } +} // Web server data structures #[derive(Debug)] @@ -155,13 +176,12 @@ pub enum HandlerReply { }, } -/// Ensures an HTTP `respond` always resolves its request. The response sender is -/// taken out of `pending_responses` (and out of its mutex) up front and held -/// here; if a fallible step in `respond` returns early before a response is -/// built, `Drop` answers 500 so the client is resolved deterministically instead -/// of hanging until the request timeout. A successful `respond` calls -/// [`ResponseCompletion::take_sender`] to disarm the fallback and deliver the -/// real response. +/// Ensures an HTTP `respond` always resolves its request after the commit point. +/// Fallible response expressions are evaluated while the pending sender remains +/// available as a disconnect signal. At commit, the sender is atomically removed +/// and held here; if delivery then exits early, `Drop` answers 500 rather than +/// leaving the client hanging. Successful delivery calls +/// [`ResponseCompletion::take_sender`] to disarm that fallback. struct ResponseCompletion { sender: Option>, } @@ -944,7 +964,7 @@ impl Drop for OutboundStreamCleanup { .unwrap_or_else(|e| e.into_inner()); for id in &http_ids { if let Some(mut slot) = map.remove(id) { - slot.cancel.cancel(); + slot.cancel.terminate(StreamTerminal::Closed); if let Some(abort) = slot.reaper_abort.take() { abort.abort(); } @@ -1700,9 +1720,36 @@ pub struct IoClient { /// short (no `.await` while held). stream_handles: Arc>>, next_stream_id: Mutex, + /// Test-only live-task accounting. The production build carries no + /// instrumentation; unit tests retain a runtime and assert that closing a + /// stream actually drops its sleeping reaper instead of relying on runtime + /// shutdown to hide leaked timers. + #[cfg(test)] + active_stream_reapers: Arc, config: Arc, } +#[cfg(test)] +struct ActiveStreamReaperGuard { + active: Arc, +} + +#[cfg(test)] +impl ActiveStreamReaperGuard { + fn new(active: Arc) -> Self { + active.fetch_add(1, std::sync::atomic::Ordering::SeqCst); + Self { active } + } +} + +#[cfg(test)] +impl Drop for ActiveStreamReaperGuard { + fn drop(&mut self) { + self.active + .fetch_sub(1, std::sync::atomic::Ordering::SeqCst); + } +} + /// Hard ceiling on `outbound_stream_max_seconds` when converting to an /// `Instant` deadline. Extreme `u64` values must not panic /// `Instant::now() + Duration::from_secs(secs)` (which can overflow). @@ -1710,46 +1757,56 @@ const MAX_OUTBOUND_STREAM_DEADLINE_SECS: u64 = 365 * 24 * 60 * 60; // 1 year /// Compute the absolute stream deadline from a configured second cap. /// `0` is the documented sentinel for "no absolute total cap". Values above -/// [`MAX_OUTBOUND_STREAM_DEADLINE_SECS`] are clamped; overflow uses -/// `checked_add` and falls back to "no cap" rather than panicking. +/// [`MAX_OUTBOUND_STREAM_DEADLINE_SECS`] are clamped before `Instant` +/// arithmetic so even an extreme configuration remains finite and cannot +/// panic. fn outbound_stream_deadline(secs: u64) -> Option { - if secs == 0 { - return None; - } - let capped = secs.min(MAX_OUTBOUND_STREAM_DEADLINE_SECS); - Instant::now().checked_add(Duration::from_secs(capped)) + let effective = outbound_stream_effective_seconds(secs)?; + Instant::now().checked_add(Duration::from_secs(effective)) +} + +fn outbound_stream_effective_seconds(secs: u64) -> Option { + (secs != 0).then(|| secs.min(MAX_OUTBOUND_STREAM_DEADLINE_SECS)) } /// Shared per-stream cancellation: close, expire, and EOF all trip this so an /// active body read can select against it and drop the upstream promptly /// (rather than only noticing when `put_stream` finds a missing slot). struct StreamCancel { - cancelled: std::sync::atomic::AtomicBool, - notify: tokio::sync::Notify, + terminal: tokio::sync::watch::Sender>, } impl StreamCancel { fn new() -> Arc { - Arc::new(Self { - cancelled: std::sync::atomic::AtomicBool::new(false), - notify: tokio::sync::Notify::new(), - }) + let (terminal, _receiver) = tokio::sync::watch::channel(None); + Arc::new(Self { terminal }) } - fn is_cancelled(&self) -> bool { - self.cancelled.load(std::sync::atomic::Ordering::SeqCst) + fn terminal(&self) -> Option { + *self.terminal.borrow() } - fn cancel(&self) { - self.cancelled - .store(true, std::sync::atomic::Ordering::SeqCst); - self.notify.notify_waiters(); + fn terminate(&self, reason: StreamTerminal) -> StreamTerminal { + self.terminal.send_if_modified(|terminal| { + if terminal.is_none() { + *terminal = Some(reason); + true + } else { + false + } + }); + self.terminal() + .expect("stream terminal reason must be set after terminate") + } + + fn subscribe(&self) -> tokio::sync::watch::Receiver> { + self.terminal.subscribe() } } -/// Why a stream slot was terminated. Never left as a long-lived tombstone in the -/// map — the slot is removed when finished; readers that still hold a cancel -/// watch observe the flag. +/// Why a stream slot was terminated. Timeout remains in the stable slot until +/// the next read consumes it, so an unread expired handle keeps its typed +/// terminal reason instead of degrading to an unknown-handle error. #[derive(Debug, Clone, Copy, PartialEq, Eq)] enum StreamTerminal { Timeout, @@ -1759,9 +1816,10 @@ enum StreamTerminal { /// Per-handle shared lifecycle for an outbound stream. /// /// Reads take the inner [`HttpStreamHandle`] out for the duration of the await -/// (so the global map lock is not held across the network). The slot stays only -/// while the stream is live; finish/close/expire remove it entirely after -/// signalling [`StreamCancel`] so mid-read work aborts. +/// (so the global map lock is not held across the network). Explicit +/// finish/close removes the slot after signalling [`StreamCancel`]; expiry +/// records a timeout tombstone and drops the parked body so mid-read work aborts +/// without losing the terminal reason. struct StreamSlot { /// The live body handle. `None` while a body read owns it. handle: Option, @@ -1923,6 +1981,8 @@ impl IoClient { next_db_id: Mutex::new(1), stream_handles: Arc::new(std::sync::Mutex::new(HashMap::new())), next_stream_id: Mutex::new(1), + #[cfg(test)] + active_stream_reapers: Arc::new(std::sync::atomic::AtomicUsize::new(0)), config, } } @@ -2061,9 +2121,7 @@ impl IoClient { Some(deadline) => { let remaining = deadline.saturating_duration_since(Instant::now()); if remaining.is_zero() { - return Err(HttpClientError::Timeout { - seconds: self.config.outbound_stream_max_seconds, - }); + return Err(self.outbound_stream_timeout_error()); } idle_timeout.min(remaining) } @@ -2145,15 +2203,21 @@ impl IoClient { let handles = Arc::clone(&self.stream_handles); let reap_id = handle_id.clone(); let cancel_reap = Arc::clone(&cancel); + #[cfg(test)] + let reaper_guard = + ActiveStreamReaperGuard::new(Arc::clone(&self.active_stream_reapers)); let join = tokio::spawn(async move { + #[cfg(test)] + let _reaper_guard = reaper_guard; let remaining = deadline.saturating_duration_since(Instant::now()); tokio::time::sleep(remaining).await; - // Absolute-lifetime reaper: signal cancel (wakes any active - // read), abort is a no-op for ourselves, drop the handle, and - // REMOVE the slot (no tombstone accumulation). - cancel_reap.cancel(); + // Preserve a Timeout tombstone in the stable slot. This + // wakes an active read with the typed reason and lets a + // later read of an unread expired handle report Timeout + // instead of "unknown handle". let mut map = handles.lock().unwrap_or_else(|e| e.into_inner()); - if let Some(mut slot) = map.remove(&reap_id) { + if let Some(slot) = map.get_mut(&reap_id) { + cancel_reap.terminate(StreamTerminal::Timeout); slot.reaper_abort = None; // we are the reaper drop(slot.handle.take()); } @@ -2173,13 +2237,13 @@ impl IoClient { /// Signal cancel, abort the reaper, drop any parked handle, and remove the /// slot. Guaranteed (std mutex) — usable from Drop. Returns whether a slot /// was present. - fn finish_stream_slot_sync(&self, handle_id: &str, _terminal: StreamTerminal) -> bool { + fn finish_stream_slot_sync(&self, handle_id: &str, terminal: StreamTerminal) -> bool { let mut map = self .stream_handles .lock() .unwrap_or_else(|e| e.into_inner()); if let Some(mut slot) = map.remove(handle_id) { - slot.cancel.cancel(); + slot.cancel.terminate(terminal); if let Some(abort) = slot.reaper_abort.take() { abort.abort(); } @@ -2207,30 +2271,29 @@ impl IoClient { "Unknown or already-closed stream handle '{handle_id}'" ))); }; - if slot.cancel.is_cancelled() { - // Fully finish and remove. + if let Some(terminal) = slot.cancel.terminal() { + // Consume the stable terminal tombstone. if let Some(mut slot) = map.remove(handle_id) { if let Some(abort) = slot.reaper_abort.take() { abort.abort(); } drop(slot.handle.take()); } - return Err(HttpClientError::Closed); + return Err(self.stream_terminal_error(terminal)); } let past_deadline = slot .deadline .is_some_and(|d| d.saturating_duration_since(Instant::now()).is_zero()); if past_deadline { if let Some(mut slot) = map.remove(handle_id) { - slot.cancel.cancel(); + let terminal = slot.cancel.terminate(StreamTerminal::Timeout); if let Some(abort) = slot.reaper_abort.take() { abort.abort(); } drop(slot.handle.take()); + return Err(self.stream_terminal_error(terminal)); } - return Err(HttpClientError::Timeout { - seconds: self.config.outbound_stream_max_seconds, - }); + return Err(self.outbound_stream_timeout_error()); } let cancel = Arc::clone(&slot.cancel); match slot.handle.take() { @@ -2249,12 +2312,11 @@ impl IoClient { handle: HttpStreamHandle, cancel: &StreamCancel, ) -> Result<(), HttpClientError> { - if cancel.is_cancelled() { + if let Some(terminal) = cancel.terminal() { drop(handle); // Ensure the slot is gone (reaper/close may already have removed it). - let _ = self.finish_stream_slot_sync(handle_id, StreamTerminal::Closed); - // Prefer Timeout if the absolute deadline has elapsed. - return Err(HttpClientError::Closed); + let _ = self.finish_stream_slot_sync(handle_id, terminal); + return Err(self.stream_terminal_error(terminal)); } let mut map = self .stream_handles @@ -2262,41 +2324,57 @@ impl IoClient { .unwrap_or_else(|e| e.into_inner()); let Some(slot) = map.get_mut(handle_id) else { drop(handle); - return Err(HttpClientError::Closed); + return Err(cancel + .terminal() + .map(|terminal| self.stream_terminal_error(terminal)) + .unwrap_or(HttpClientError::Closed)); }; - if slot.cancel.is_cancelled() - || slot - .deadline - .is_some_and(|d| d.saturating_duration_since(Instant::now()).is_zero()) - { - let terminal = if slot + let terminal = slot.cancel.terminal().or_else(|| { + if slot .deadline .is_some_and(|d| d.saturating_duration_since(Instant::now()).is_zero()) { - StreamTerminal::Timeout + Some(slot.cancel.terminate(StreamTerminal::Timeout)) } else { - StreamTerminal::Closed - }; + None + } + }); + if let Some(terminal) = terminal { drop(handle); - if let Some(mut slot) = map.remove(handle_id) { - slot.cancel.cancel(); - if let Some(abort) = slot.reaper_abort.take() { - abort.abort(); - } + if let Some(mut slot) = map.remove(handle_id) + && let Some(abort) = slot.reaper_abort.take() + { + abort.abort(); } - return Err(match terminal { - StreamTerminal::Timeout => HttpClientError::Timeout { - seconds: self.config.outbound_stream_max_seconds, - }, - StreamTerminal::Closed => HttpClientError::Closed, - }); + return Err(self.stream_terminal_error(terminal)); + } + // A final unterminated line needs one subsequent read to produce + // `nothing`, matching the established WFL stream contract. Retain the + // exhausted handle for that one read, but abort its timer immediately + // so EOF never leaves a sleeping reaper task. + if handle.done + && let Some(abort) = slot.reaper_abort.take() + { + abort.abort(); } - // Done streams must not leave a parked reaper: finish fully on EOF - // after the final unterminated line is served (caller uses finish). slot.handle = Some(handle); Ok(()) } + fn stream_terminal_error(&self, terminal: StreamTerminal) -> HttpClientError { + match terminal { + StreamTerminal::Timeout => self.outbound_stream_timeout_error(), + StreamTerminal::Closed => HttpClientError::Closed, + } + } + + fn outbound_stream_timeout_error(&self) -> HttpClientError { + HttpClientError::Timeout { + seconds: outbound_stream_effective_seconds(self.config.outbound_stream_max_seconds) + .unwrap_or(self.config.timeout_seconds.max(1)), + } + } + /// Pull one network chunk into `handle.buffer`, bounded by the per-chunk /// read deadline, the absolute total, and cooperative stream cancellation /// (close/expire while reading). Returns `Ok(true)` when bytes were added, @@ -2312,8 +2390,9 @@ impl IoClient { if handle.done { return Ok(false); } - if cancel.is_cancelled() { - return Err(HttpClientError::Closed); + let mut terminal_rx = cancel.subscribe(); + if let Some(terminal) = *terminal_rx.borrow() { + return Err(self.stream_terminal_error(terminal)); } let max_response_bytes = budget.limits().max_response_bytes; let idle_timeout = Duration::from_secs(self.config.timeout_seconds.max(1)); @@ -2321,9 +2400,7 @@ impl IoClient { Some(deadline) => { let remaining = deadline.saturating_duration_since(Instant::now()); if remaining.is_zero() { - return Err(HttpClientError::Timeout { - seconds: self.config.outbound_stream_max_seconds, - }); + return Err(self.outbound_stream_timeout_error()); } idle_timeout.min(remaining) } @@ -2338,13 +2415,16 @@ impl IoClient { tokio::pin!(op); let next = tokio::select! { result = &mut op => result?, - _ = cancel.notify.notified() => { - return Err(HttpClientError::Closed); + changed = terminal_rx.changed() => { + let _ = changed; + let terminal = (*terminal_rx.borrow()).unwrap_or(StreamTerminal::Closed); + return Err(self.stream_terminal_error(terminal)); } }; - // Re-check after select (notify may have raced with a false wake). - if cancel.is_cancelled() { - return Err(HttpClientError::Closed); + // Re-check after select so a simultaneously-ready body chunk cannot win + // over an already-recorded hard deadline and be reinserted. + if let Some(terminal) = cancel.terminal() { + return Err(self.stream_terminal_error(terminal)); } match next { @@ -2379,9 +2459,7 @@ impl IoClient { if let Some(deadline) = handle.total_deadline && deadline.saturating_duration_since(Instant::now()).is_zero() { - return Err(HttpClientError::Timeout { - seconds: self.config.outbound_stream_max_seconds, - }); + return Err(self.outbound_stream_timeout_error()); } Ok(()) } @@ -2462,7 +2540,9 @@ impl IoClient { if line.last() == Some(&b'\r') { line.pop(); } - let _ = self.finish_stream_slot(handle_id).await; + // Preserve one exhausted read so the next wait binds `nothing`. + // `put_stream` aborts the reaper before parking a done handle. + self.put_stream(handle_id, handle, &cancel)?; return Ok(Some(String::from_utf8_lossy(&line).into_owned())); } @@ -4395,7 +4475,7 @@ impl Interpreter { .unwrap_or_else(|e| e.into_inner()); for id in ids { if let Some(mut slot) = map.remove(id) { - slot.cancel.cancel(); + slot.cancel.terminate(StreamTerminal::Closed); if let Some(abort) = slot.reaper_abort.take() { abort.abort(); } @@ -4740,7 +4820,6 @@ impl Interpreter { // iteration in between: back off between them, and terminate the loop once // it's clearly a structural failure rather than incidental handler errors. let mut consecutive_failures: u32 = 0; - const MAX_CONSECUTIVE_FAILURES: u32 = 256; loop { self.check_time()?; @@ -4789,26 +4868,25 @@ impl Interpreter { // // Structural pre-request failures feed the breaker. Some((Ok(Err(err)), accepted)) => { - let is_request_wait_timeout = err.kind == ErrorKind::Timeout - && err.message.starts_with("Timeout waiting for request"); - let non_structural = - accepted || err.kind == ErrorKind::Cancelled || is_request_wait_timeout; - if non_structural { - log::debug!( - "concurrent main loop: non-structural handler outcome \ - (kind={:?}, accepted_request={accepted}): {err}", - err.kind - ); - } else { - log::warn!("concurrent main loop: structural handler error: {err}"); - if self - .backoff_or_break_concurrent( - &mut consecutive_failures, - MAX_CONSECUTIVE_FAILURES, - ) - .await - { - break; + match classify_concurrent_handler_error(&err, accepted) { + ConcurrentHandlerDisposition::RequestLocal => { + log::debug!( + "concurrent main loop: non-structural handler outcome \ + (kind={:?}, accepted_request={accepted}): {err}", + err.kind + ); + } + ConcurrentHandlerDisposition::Structural => { + log::warn!("concurrent main loop: structural handler error: {err}"); + if self + .backoff_or_break_concurrent( + &mut consecutive_failures, + MAX_CONSECUTIVE_HANDLER_FAILURES, + ) + .await + { + break; + } } } } @@ -4829,7 +4907,7 @@ impl Interpreter { if self .backoff_or_break_concurrent( &mut consecutive_failures, - MAX_CONSECUTIVE_FAILURES, + MAX_CONSECUTIVE_HANDLER_FAILURES, ) .await { @@ -9477,7 +9555,7 @@ impl Interpreter { // down after enough empty poll intervals. return Err(RuntimeError::with_kind( format!( - "Timeout waiting for request ({} ms)", + "{REQUEST_WAIT_TIMEOUT_PREFIX} ({} ms)", duration.as_millis() ), *line, @@ -10242,6 +10320,7 @@ impl Interpreter { } Statement::FlushStreamStatement { target, + legacy_binding, action_fallback, line, column, @@ -10252,21 +10331,9 @@ impl Interpreter { // evaluate it with the same ExpressionStatement semantics // (zero-arg auto-call; parameterized bare call → arity error). if let Some(fallback_expr) = action_fallback { - let root_name = match fallback_expr { - Expression::Variable(n, ..) => Some(n.as_str()), - Expression::IndexAccess { collection, .. } - | Expression::PropertyAccess { - object: collection, .. - } - | Expression::MethodCall { - object: collection, .. - } => match collection.as_ref() { - Expression::Variable(n, ..) => Some(n.as_str()), - _ => None, - }, - _ => None, - }; - let root_bound = root_name.is_some_and(|n| env.borrow().get(n).is_some()); + let root_bound = legacy_binding + .as_deref() + .is_some_and(|name| env.borrow().get(name).is_some()); if root_bound { // Reuse ExpressionStatement semantics by dispatching a // synthetic statement. @@ -14228,14 +14295,156 @@ impl Interpreter { } } +#[cfg(test)] +mod concurrent_handler_classification_tests { + use super::*; + + fn error(kind: ErrorKind, message: &str) -> RuntimeError { + RuntimeError::with_kind(message.to_string(), 1, 1, kind) + } + + #[test] + fn more_than_the_breaker_threshold_of_request_wait_timeouts_stays_request_local() { + let timeout = error( + ErrorKind::Timeout, + &format!("{REQUEST_WAIT_TIMEOUT_PREFIX} (1 ms)"), + ); + for observed in 0..=MAX_CONSECUTIVE_HANDLER_FAILURES { + assert_eq!( + classify_concurrent_handler_error(&timeout, false), + ConcurrentHandlerDisposition::RequestLocal, + "finite request-wait timeout #{observed} must not feed the structural breaker" + ); + } + } + + #[test] + fn only_the_expected_timeout_origin_is_exempt_before_request_acceptance() { + let structural_timeout = error( + ErrorKind::Timeout, + "unrelated pre-request operation timed out", + ); + assert_eq!( + classify_concurrent_handler_error(&structural_timeout, false), + ConcurrentHandlerDisposition::Structural + ); + assert_eq!( + classify_concurrent_handler_error(&error(ErrorKind::General, "request failed"), true), + ConcurrentHandlerDisposition::RequestLocal, + "any failure after request acceptance is isolated to that request" + ); + assert_eq!( + classify_concurrent_handler_error( + &error(ErrorKind::Cancelled, "client disconnected"), + false, + ), + ConcurrentHandlerDisposition::RequestLocal + ); + } + + #[tokio::test] + async fn missing_pending_entry_is_cancelled_only_while_the_handler_owns_it() { + let interpreter = Interpreter::new(); + interpreter + .open_pending_requests + .borrow_mut() + .push("request-1".to_string()); + + let disconnected = interpreter + .ensure_pending_response_owned("request-1", 1, 1) + .await + .expect_err("owned-but-pruned pending entry must be cancellation"); + assert_eq!(disconnected.kind, ErrorKind::Cancelled); + + interpreter.open_pending_requests.borrow_mut().clear(); + let duplicate = interpreter + .ensure_pending_response_owned("request-1", 1, 1) + .await + .expect_err("non-owned missing entry must remain a duplicate-response error"); + assert_eq!(duplicate.kind, ErrorKind::General); + assert!( + duplicate.message.contains("already been sent"), + "duplicate response diagnostic changed unexpectedly: {duplicate}" + ); + } +} + #[cfg(test)] mod outbound_stream_deadline_tests { use super::*; + use tokio::io::{AsyncReadExt, AsyncWriteExt}; + + async fn spawn_stream_cleanup_upstream(expected_requests: usize) -> u16 { + let listener = tokio::net::TcpListener::bind("127.0.0.1:0") + .await + .expect("bind stream cleanup upstream"); + let port = listener.local_addr().expect("upstream address").port(); + tokio::spawn(async move { + for _ in 0..expected_requests { + let (mut socket, _) = listener.accept().await.expect("accept cleanup request"); + tokio::spawn(async move { + let mut request = Vec::new(); + let mut buf = [0u8; 512]; + while !request.windows(4).any(|window| window == b"\r\n\r\n") { + let read = socket.read(&mut buf).await.expect("read request head"); + if read == 0 { + return; + } + request.extend_from_slice(&buf[..read]); + } + let request = String::from_utf8_lossy(&request); + let truncated = request.starts_with("GET /truncated "); + let response = if truncated { + "HTTP/1.1 200 OK\r\nContent-Length: 10\r\nConnection: close\r\n\r\nx" + } else { + "HTTP/1.1 200 OK\r\nContent-Length: 0\r\nConnection: close\r\n\r\n" + }; + socket + .write_all(response.as_bytes()) + .await + .expect("write response"); + socket.flush().await.expect("flush response"); + }); + } + }); + port + } + + async fn assert_reapers_drained(client: &IoClient) { + tokio::time::timeout(Duration::from_secs(2), async { + loop { + if client + .active_stream_reapers + .load(std::sync::atomic::Ordering::SeqCst) + == 0 + { + break; + } + tokio::task::yield_now().await; + } + }) + .await + .expect("finished streams left hard-lifetime reaper tasks sleeping"); + } #[test] fn extreme_outbound_stream_max_seconds_does_not_panic() { - // u64::MAX must not panic Instant arithmetic (clamped / checked_add). - let _ = outbound_stream_deadline(u64::MAX); + // u64::MAX must remain a finite cap rather than panicking or silently + // disabling the hard lifetime. + let before = Instant::now(); + let extreme = outbound_stream_deadline(u64::MAX) + .expect("an extreme positive cap must still produce a finite deadline"); + let effective = extreme.saturating_duration_since(before); + assert!( + effective <= Duration::from_secs(MAX_OUTBOUND_STREAM_DEADLINE_SECS + 1), + "extreme values must be clamped to the documented implementation ceiling; \ + got {effective:?}" + ); + assert_eq!( + outbound_stream_effective_seconds(u64::MAX), + Some(MAX_OUTBOUND_STREAM_DEADLINE_SECS), + "timeout diagnostics must report the effective clamp, not u64::MAX" + ); assert!( outbound_stream_deadline(0).is_none(), "0 is the documented sentinel for no absolute total cap" @@ -14249,6 +14458,136 @@ mod outbound_stream_deadline_tests { "the clamp ceiling itself must still produce a deadline" ); } + + #[tokio::test] + async fn eof_error_and_rapid_close_cancel_reaper_tasks_on_a_retained_runtime() { + const RAPID_CLOSES: usize = 40; + let port = spawn_stream_cleanup_upstream(RAPID_CLOSES + 2).await; + let config = Arc::new(WflConfig { + outbound_stream_max_seconds: 60 * 60, + timeout_seconds: 10, + ..WflConfig::default() + }); + let client = IoClient::new(Arc::clone(&config)); + let budget = Arc::new(ExecutionBudget::from_config(&config)); + + for sequence in 0..RAPID_CLOSES { + let (_, _, handle) = client + .open_http_stream( + "GET", + &format!("http://127.0.0.1:{port}/close/{sequence}"), + &[], + None, + Arc::clone(&budget), + ) + .await + .expect("open stream for explicit close"); + assert!( + client.finish_stream_slot(&handle).await, + "explicit close should remove its live stream" + ); + } + + let (_, _, eof_handle) = client + .open_http_stream( + "GET", + &format!("http://127.0.0.1:{port}/empty"), + &[], + None, + Arc::clone(&budget), + ) + .await + .expect("open empty stream"); + assert_eq!( + client + .next_chunk(&eof_handle, Arc::clone(&budget)) + .await + .expect("clean EOF"), + None + ); + + let (_, _, error_handle) = client + .open_http_stream( + "GET", + &format!("http://127.0.0.1:{port}/truncated"), + &[], + None, + Arc::clone(&budget), + ) + .await + .expect("open truncated stream"); + let first = client.next_chunk(&error_handle, Arc::clone(&budget)).await; + let terminal = match first { + Ok(Some(_)) => client.next_chunk(&error_handle, budget).await, + other => other, + }; + assert!( + matches!(terminal, Err(HttpClientError::Request(_))), + "a truncated body should end as a network read error, got {terminal:?}" + ); + + // Keep this Tokio runtime alive while observing the task count. Runtime + // shutdown would cancel leaked timers and make this regression false-green. + assert_reapers_drained(&client).await; + assert!( + client + .stream_handles + .lock() + .unwrap_or_else(|error| error.into_inner()) + .is_empty(), + "EOF, error, and explicit close must remove every stream slot" + ); + } + + #[test] + fn a_ready_read_result_cannot_reinsert_after_expiry_claims_the_slot() { + let client = IoClient::new(Arc::new(WflConfig { + outbound_stream_max_seconds: 1, + ..WflConfig::default() + })); + let handle_id = "httpstream-race"; + let cancel = StreamCancel::new(); + client + .stream_handles + .lock() + .unwrap_or_else(|error| error.into_inner()) + .insert( + handle_id.to_string(), + StreamSlot { + handle: Some(HttpStreamHandle { + // Model a network chunk that became ready while the read + // owned the handle. + stream: Box::pin(futures_util::stream::iter([Ok(vec![1u8])])), + buffer: vec![1], + done: false, + bytes_read: 1, + total_deadline: outbound_stream_deadline(1), + }), + deadline: outbound_stream_deadline(1), + cancel: Arc::clone(&cancel), + reaper_abort: None, + }, + ); + + let TakenStream { handle, cancel } = client + .take_stream(handle_id) + .expect("active read takes body"); + cancel.terminate(StreamTerminal::Timeout); + let result = client.put_stream(handle_id, handle, &cancel); + + assert!( + matches!(result, Err(HttpClientError::Timeout { .. })), + "expiry must win over a ready body result, got {result:?}" + ); + assert!( + !client + .stream_handles + .lock() + .unwrap_or_else(|error| error.into_inner()) + .contains_key(handle_id), + "an expired handle must never be reinserted after its active read" + ); + } } #[cfg(test)] diff --git a/src/parser/ast.rs b/src/parser/ast.rs index da53472d..112d1da0 100644 --- a/src/parser/ast.rs +++ b/src/parser/ast.rs @@ -624,6 +624,12 @@ pub enum Statement { /// queued bytes to the transport. FlushStreamStatement { target: Expression, + /// Original lexer-merged binding (`flush cache`) used to choose the + /// pre-streaming expression-statement interpretation. This metadata is + /// retained separately because ordinary expression rewrites such as + /// explicit `find ... in`, `replace ... in`, and `split ... by` can + /// legitimately discard the seeded left operand from `action_fallback`. + legacy_binding: Option, /// Complete old expression-statement AST for the merged `flush …` form /// (e.g. `Variable("flush cache")`, or `IndexAccess`/`PropertyAccess` over /// that full name). Before streaming, `flush cache[0]` was an ordinary diff --git a/src/parser/expr/binary.rs b/src/parser/expr/binary.rs index 3945b05d..1786d2e2 100644 --- a/src/parser/expr/binary.rs +++ b/src/parser/expr/binary.rs @@ -4,10 +4,25 @@ //! comparison, pattern matching, and custom language constructs. use super::super::{Argument, Expression, Operator, ParseError, Parser}; -use super::{ExprParser, PrimaryExprParser}; +use super::PrimaryExprParser; use crate::diagnostics::Span; use crate::lexer::token::Token; +#[derive(Clone, Copy)] +enum BinaryExpressionTerminator { + With, + In, +} + +impl BinaryExpressionTerminator { + fn matches(self, token: &Token) -> bool { + matches!( + (self, token), + (Self::With, Token::KeywordWith) | (Self::In, Token::KeywordIn) + ) + } +} + /// Trait for parsing binary expressions with operator precedence pub(crate) trait BinaryExprParser<'a> { /// Parses a binary expression with operator precedence. @@ -21,6 +36,13 @@ pub(crate) trait BinaryExprParser<'a> { /// Returns an `Expression` representing the parsed binary expression, or a `ParseError` if the syntax is invalid. fn parse_binary_expression(&mut self, precedence: u8) -> Result; + /// Parse a fresh binary expression while retaining a surrounding + /// streaming-response clause boundary through recursive operands. + fn parse_binary_expression_stopping_at_clause( + &mut self, + precedence: u8, + ) -> Result; + /// Continue a binary expression from an already-parsed left-hand side. /// /// `parse_binary_expression` parses a fresh primary and then runs the @@ -54,10 +76,22 @@ pub(crate) trait BinaryExprParser<'a> { &mut self, call_line: usize, call_column: usize, + stop_at_clause: bool, ) -> Result; /// Parses a comma-separated or 'and'-separated argument list for action calls. - fn parse_argument_list(&mut self) -> Result, ParseError>; + fn parse_argument_list(&mut self) -> Result, ParseError> { + self.parse_argument_list_with_clause_boundary(false) + } + + fn parse_argument_list_stopping_at_clause(&mut self) -> Result, ParseError> { + self.parse_argument_list_with_clause_boundary(true) + } + + fn parse_argument_list_with_clause_boundary( + &mut self, + stop_at_clause: bool, + ) -> Result, ParseError>; /// Parses a single argument of an `of`-call (e.g. `fibonacci of n minus 1`). /// @@ -68,19 +102,56 @@ pub(crate) trait BinaryExprParser<'a> { /// separator), `with` (concatenation), `from`/`by`/`length` (stdlib call /// separators), comparisons, and pattern keywords, leaving those for the /// caller so multi-argument and postfix forms keep working. - fn parse_of_call_argument(&mut self) -> Result; + fn parse_of_call_argument(&mut self) -> Result { + self.parse_of_call_argument_with_clause_boundary(false) + } + + fn parse_of_call_argument_stopping_at_clause(&mut self) -> Result { + self.parse_of_call_argument_with_clause_boundary(true) + } + + fn parse_of_call_argument_with_clause_boundary( + &mut self, + stop_at_clause: bool, + ) -> Result; /// Multiplicative level of an `of`-call argument: times / divided by / `/` /// / `%` / modulo (precedence 3). - fn parse_of_call_arg_term(&mut self) -> Result; + fn parse_of_call_arg_term_with_clause_boundary( + &mut self, + stop_at_clause: bool, + ) -> Result; } impl<'a> Parser<'a> { - pub(crate) fn parse_binary_continuation_inner( + fn parse_binary_expression_for_context( + &mut self, + precedence: u8, + stop_at_clause: bool, + terminator: Option, + ) -> Result { + let left = if stop_at_clause { + self.parse_primary_expression_stopping_at_clause()? + } else { + self.parse_primary_expression()? + }; + self.parse_binary_continuation_inner(left, precedence, stop_at_clause, terminator) + } + + fn parse_binary_expression_for_clause_context( + &mut self, + precedence: u8, + stop_at_clause: bool, + ) -> Result { + self.parse_binary_expression_for_context(precedence, stop_at_clause, None) + } + + fn parse_binary_continuation_inner( &mut self, mut left: Expression, precedence: u8, stop_at_clause: bool, + terminator: Option, ) -> Result { while let Some(token_pos) = self.cursor.peek() { let token = &token_pos.token; @@ -91,6 +162,9 @@ impl<'a> Parser<'a> { if matches!(token, Token::Eol) || Parser::is_statement_starter(token) { break; } + if terminator.is_some_and(|terminator| terminator.matches(token)) { + break; + } // Streaming-response clause connectives must not be absorbed as // Boolean AND / `with` concatenation inside a clause operand. if stop_at_clause { @@ -168,12 +242,20 @@ impl<'a> Parser<'a> { // desugars to `X >= A and X <= B`. Token::KeywordBetween => { self.bump_sync(); // Consume "between" - let lower = self.parse_binary_expression(2)?; + let lower = self.parse_binary_expression_for_context( + 2, + stop_at_clause, + terminator, + )?; self.expect_token( Token::KeywordAnd, "Expected 'and' between the bounds of 'is between'", )?; - let upper = self.parse_binary_expression(2)?; + let upper = self.parse_binary_expression_for_context( + 2, + stop_at_clause, + terminator, + )?; let lower_bound = Expression::BinaryOperation { left: Box::new(left.clone()), @@ -386,7 +468,11 @@ impl<'a> Parser<'a> { if name != "count" && crate::builtins::is_builtin_function(name) { // Builtin function - keep legacy syntax self.bump_sync(); // Consume "with" - let arguments = self.parse_argument_list()?; + let arguments = if stop_at_clause { + self.parse_argument_list_stopping_at_clause()? + } else { + self.parse_argument_list()? + }; left = Expression::ActionCall { name: name.clone(), @@ -401,7 +487,8 @@ impl<'a> Parser<'a> { // For all other cases (including user-defined actions), // treat 'with' as concatenation self.bump_sync(); // Consume "with" - let right = self.parse_expression()?; + let right = + self.parse_binary_expression_for_context(0, stop_at_clause, terminator)?; left = Expression::Concatenation { left: Box::new(left), right: Box::new(right), @@ -477,7 +564,11 @@ impl<'a> Parser<'a> { self.bump_sync(); // Consume "pattern" } - let pattern_expr = self.parse_binary_expression(precedence + 1)?; + let pattern_expr = self.parse_binary_expression_for_context( + precedence + 1, + stop_at_clause, + terminator, + )?; left = Expression::PatternMatch { text: Box::new(left), @@ -497,14 +588,22 @@ impl<'a> Parser<'a> { self.bump_sync(); // Consume "pattern" } - let pattern_expr = self.parse_binary_expression(precedence + 1)?; + let pattern_expr = self.parse_binary_expression_for_context( + precedence + 1, + stop_at_clause, + Some(BinaryExpressionTerminator::In), + )?; if let Some(in_token) = self.cursor.peek() && matches!(&in_token.token, Token::KeywordIn) { self.bump_sync(); // Consume "in" - let text_expr = self.parse_binary_expression(precedence + 1)?; + let text_expr = self.parse_binary_expression_for_context( + precedence + 1, + stop_at_clause, + terminator, + )?; left = Expression::PatternFind { text: Box::new(text_expr), @@ -533,21 +632,33 @@ impl<'a> Parser<'a> { self.bump_sync(); // Consume "pattern" } - let pattern_expr = self.parse_binary_expression(precedence + 1)?; + let pattern_expr = self.parse_binary_expression_for_context( + precedence + 1, + stop_at_clause, + Some(BinaryExpressionTerminator::With), + )?; if let Some(with_token) = self.cursor.peek() && matches!(&with_token.token, Token::KeywordWith) { self.bump_sync(); // Consume "with" - let replacement_expr = self.parse_binary_expression(precedence + 1)?; + let replacement_expr = self.parse_binary_expression_for_context( + precedence + 1, + stop_at_clause, + Some(BinaryExpressionTerminator::In), + )?; if let Some(in_token) = self.cursor.peek() && matches!(&in_token.token, Token::KeywordIn) { self.bump_sync(); // Consume "in" - let text_expr = self.parse_binary_expression(precedence + 1)?; + let text_expr = self.parse_binary_expression_for_context( + precedence + 1, + stop_at_clause, + terminator, + )?; left = Expression::PatternReplace { text: Box::new(text_expr), @@ -584,7 +695,11 @@ impl<'a> Parser<'a> { self.consume_optional_of(); // Parse the text expression to split - let text_expr = self.parse_binary_expression(precedence + 1)?; + let text_expr = self.parse_binary_expression_for_context( + precedence + 1, + stop_at_clause, + terminator, + )?; // Check for "by" (string split) or "on" (pattern split) if let Some(next_token) = self.cursor.peek() { @@ -592,8 +707,11 @@ impl<'a> Parser<'a> { Token::KeywordBy => { // Handle "split text by delimiter" syntax self.bump_sync(); // Consume "by" - let delimiter_expr = - self.parse_binary_expression(precedence + 1)?; + let delimiter_expr = self.parse_binary_expression_for_context( + precedence + 1, + stop_at_clause, + terminator, + )?; left = Expression::StringSplit { text: Box::new(text_expr), @@ -614,7 +732,11 @@ impl<'a> Parser<'a> { self.bump_sync(); // Consume "pattern" } - let pattern_expr = self.parse_binary_expression(precedence + 1)?; + let pattern_expr = self.parse_binary_expression_for_context( + precedence + 1, + stop_at_clause, + terminator, + )?; left = Expression::PatternSplit { text: Box::new(text_expr), @@ -667,7 +789,8 @@ impl<'a> Parser<'a> { self.bump_sync(); // Consume "with" // RHS binds at precedence 2 (tighter than comparison), matching // how `contains`/`is` parse their right-hand side. - let right = self.parse_binary_expression(2)?; + let right = + self.parse_binary_expression_for_context(2, stop_at_clause, terminator)?; let fn_name = if is_starts { "starts_with" } else { @@ -702,7 +825,11 @@ impl<'a> Parser<'a> { { self.bump_sync(); // Consume "pattern" - let pattern_expr = self.parse_binary_expression(precedence + 1)?; + let pattern_expr = self.parse_binary_expression_for_context( + precedence + 1, + stop_at_clause, + terminator, + )?; left = Expression::PatternMatch { text: Box::new(left), @@ -772,7 +899,11 @@ impl<'a> Parser<'a> { } } - let right = self.parse_binary_expression(op_precedence + 1)?; + let right = self.parse_binary_expression_for_context( + op_precedence + 1, + stop_at_clause, + terminator, + )?; left = Expression::BinaryOperation { left: Box::new(left), @@ -796,12 +927,20 @@ impl<'a> BinaryExprParser<'a> for Parser<'a> { self.parse_binary_continuation(left, precedence) } + fn parse_binary_expression_stopping_at_clause( + &mut self, + precedence: u8, + ) -> Result { + let left = self.parse_primary_expression_stopping_at_clause()?; + self.parse_binary_continuation_inner(left, precedence, true, None) + } + fn parse_binary_continuation( &mut self, left: Expression, precedence: u8, ) -> Result { - self.parse_binary_continuation_inner(left, precedence, false) + self.parse_binary_continuation_inner(left, precedence, false, None) } fn parse_binary_continuation_stopping_at_clause( @@ -809,13 +948,14 @@ impl<'a> BinaryExprParser<'a> for Parser<'a> { left: Expression, precedence: u8, ) -> Result { - self.parse_binary_continuation_inner(left, precedence, true) + self.parse_binary_continuation_inner(left, precedence, true, None) } fn parse_call_expression( &mut self, call_line: usize, call_column: usize, + stop_at_clause: bool, ) -> Result { // We've already consumed Token::KeywordCall in the caller @@ -869,7 +1009,11 @@ impl<'a> BinaryExprParser<'a> for Parser<'a> { } // Parse argument list - let arguments = self.parse_argument_list()?; + let arguments = if stop_at_clause { + self.parse_argument_list_stopping_at_clause()? + } else { + self.parse_argument_list()? + }; Ok(Expression::ActionCall { name, @@ -879,9 +1023,12 @@ impl<'a> BinaryExprParser<'a> for Parser<'a> { }) } - fn parse_of_call_argument(&mut self) -> Result { + fn parse_of_call_argument_with_clause_boundary( + &mut self, + stop_at_clause: bool, + ) -> Result { // Additive level: plus / minus (precedence 2). - let mut left = self.parse_of_call_arg_term()?; + let mut left = self.parse_of_call_arg_term_with_clause_boundary(stop_at_clause)?; while let Some(token_pos) = self.cursor.peek() { let (operator, line, column) = match &token_pos.token { @@ -894,7 +1041,7 @@ impl<'a> BinaryExprParser<'a> for Parser<'a> { _ => break, }; self.bump_sync(); // Consume the additive operator - let right = self.parse_of_call_arg_term()?; + let right = self.parse_of_call_arg_term_with_clause_boundary(stop_at_clause)?; left = Expression::BinaryOperation { left: Box::new(left), operator, @@ -907,8 +1054,15 @@ impl<'a> BinaryExprParser<'a> for Parser<'a> { Ok(left) } - fn parse_of_call_arg_term(&mut self) -> Result { - let mut left = self.parse_primary_expression()?; + fn parse_of_call_arg_term_with_clause_boundary( + &mut self, + stop_at_clause: bool, + ) -> Result { + let mut left = if stop_at_clause { + self.parse_primary_expression_stopping_at_clause()? + } else { + self.parse_primary_expression()? + }; while let Some(token_pos) = self.cursor.peek() { let line = token_pos.line; @@ -937,7 +1091,11 @@ impl<'a> BinaryExprParser<'a> for Parser<'a> { } } - let right = self.parse_primary_expression()?; + let right = if stop_at_clause { + self.parse_primary_expression_stopping_at_clause()? + } else { + self.parse_primary_expression()? + }; left = Expression::BinaryOperation { left: Box::new(left), operator, @@ -950,7 +1108,10 @@ impl<'a> BinaryExprParser<'a> for Parser<'a> { Ok(left) } - fn parse_argument_list(&mut self) -> Result, ParseError> { + fn parse_argument_list_with_clause_boundary( + &mut self, + stop_at_clause: bool, + ) -> Result, ParseError> { let mut arguments = Vec::with_capacity(4); let start_pos = self.cursor.pos(); @@ -981,7 +1142,7 @@ impl<'a> BinaryExprParser<'a> for Parser<'a> { // FIX: Parse expressions with precedence >= 1 (arithmetic operators) // This stops at 'and' (precedence 0), which is then used as argument separator - let arg_value = self.parse_binary_expression(1)?; + let arg_value = self.parse_binary_expression_for_clause_context(1, stop_at_clause)?; arguments.push(Argument { name: arg_name, @@ -990,6 +1151,13 @@ impl<'a> BinaryExprParser<'a> for Parser<'a> { if let Some(token) = self.cursor.peek() { if matches!(&token.token, Token::KeywordAnd) { + if stop_at_clause + && Parser::is_streaming_clause_keyword( + self.cursor.peek_n(1).map(|t| &t.token), + ) + { + break; + } self.bump_sync(); // Consume "and" continue; // Continue parsing next argument } else { diff --git a/src/parser/expr/primary.rs b/src/parser/expr/primary.rs index f9716b34..1e78c437 100644 --- a/src/parser/expr/primary.rs +++ b/src/parser/expr/primary.rs @@ -12,17 +12,33 @@ use std::sync::Arc; /// Trait for parsing primary (atomic) expressions pub(crate) trait PrimaryExprParser<'a> { /// Parses a primary expression (atomic expression like literals, variables, etc.) - fn parse_primary_expression(&mut self) -> Result; + fn parse_primary_expression(&mut self) -> Result { + self.parse_primary_expression_with_clause_boundary(false) + } + + /// Parses a primary while preserving a surrounding streaming-response + /// clause boundary through un-delimited postfix forms such as `values at 0`. + fn parse_primary_expression_stopping_at_clause(&mut self) -> Result { + self.parse_primary_expression_with_clause_boundary(true) + } + + fn parse_primary_expression_with_clause_boundary( + &mut self, + stop_at_clause: bool, + ) -> Result; /// Parses a single list element without parsing binary operators fn parse_list_element(&mut self) -> Result; } impl<'a> PrimaryExprParser<'a> for Parser<'a> { - fn parse_primary_expression(&mut self) -> Result { + fn parse_primary_expression_with_clause_boundary( + &mut self, + stop_at_clause: bool, + ) -> Result { #[cfg(debug_assertions)] let leading = self.cursor.peek().cloned(); - let result = self.parse_primary_expression_dispatch(); + let result = self.parse_primary_expression_dispatch(stop_at_clause); // Runtime coupling check between `can_start_primary_expression` (the // predicate `display`'s multi-value fold is built on, in @@ -89,6 +105,19 @@ impl<'a> PrimaryExprParser<'a> for Parser<'a> { } impl<'a> Parser<'a> { + /// Parse an un-delimited recursive expression while retaining the + /// streaming-response clause boundary inherited from its outer operand. + fn parse_expression_with_clause_boundary( + &mut self, + stop_at_clause: bool, + ) -> Result { + if stop_at_clause { + self.parse_binary_expression_stopping_at_clause(0) + } else { + self.parse_expression() + } + } + /// Consume postfix accessors — bracket index (`["key"]`, `[0]`) and dotted /// property access (`.field`) — that chain off a completed lead expression, so /// they bind to the lead instead of splitting into bogus separate statements. @@ -100,8 +129,26 @@ impl<'a> Parser<'a> { /// merged-command operands (`flush streams["a"]`, `flush obj.out`). Handles /// arbitrary chains (`grid.rows[0][1]`, `obj.a.b["k"]`). pub(crate) fn parse_trailing_postfix( + &mut self, + expr: Expression, + ) -> Result { + self.parse_trailing_postfix_with_clause_boundary(expr, false) + } + + /// Clause-aware counterpart to [`Self::parse_trailing_postfix`]. Natural + /// `at` indexes are not delimited, so their recursive expression parser must + /// inherit the response-clause boundary from the surrounding operand. + pub(crate) fn parse_trailing_postfix_stopping_at_clause( + &mut self, + expr: Expression, + ) -> Result { + self.parse_trailing_postfix_with_clause_boundary(expr, true) + } + + fn parse_trailing_postfix_with_clause_boundary( &mut self, mut expr: Expression, + stop_at_clause: bool, ) -> Result { while let Some(tok) = self.cursor.peek() { let (line, column) = (tok.line, tok.column); @@ -267,7 +314,11 @@ impl<'a> Parser<'a> { // Natural-language indexing (`values at 0`) — same as primary. Token::KeywordAt => { self.bump_sync(); - let index = self.parse_expression()?; + let index = if stop_at_clause { + self.parse_binary_expression_stopping_at_clause(0)? + } else { + self.parse_expression()? + }; expr = Expression::IndexAccess { collection: Box::new(expr), index: Box::new(index), @@ -286,7 +337,10 @@ impl<'a> Parser<'a> { /// with a debug-only check that keeps `can_start_primary_expression` from /// silently drifting away from what this dispatch really accepts, and /// recursive calls from within the arms below go through that wrapper too. - fn parse_primary_expression_dispatch(&mut self) -> Result { + fn parse_primary_expression_dispatch( + &mut self, + stop_at_clause: bool, + ) -> Result { // Strided run-budget checkpoint. Every operand (list element, operator- // chain term, call argument) routes through here, so this bounds a single // huge expression that the statement-boundary checkpoint would miss. @@ -409,7 +463,7 @@ impl<'a> Parser<'a> { let call_line = token.line; let call_column = token.column; self.bump_sync(); // Consume 'call' - return self.parse_call_expression(call_line, call_column); + return self.parse_call_expression(call_line, call_column, stop_at_clause); } Token::Identifier(name) => { self.bump_sync(); @@ -480,7 +534,10 @@ impl<'a> Parser<'a> { line: token_line, column: token_column, }; - return self.parse_trailing_postfix(call); + return self.parse_trailing_postfix_with_clause_boundary( + call, + stop_at_clause, + ); } // Property access without method call. @@ -500,7 +557,10 @@ impl<'a> Parser<'a> { line: token_line, column: token_column, }; - return self.parse_trailing_postfix(access); + return self.parse_trailing_postfix_with_clause_boundary( + access, + stop_at_clause, + ); } else { return Err(ParseError::from_token( "Expected property name after '.'".to_string(), @@ -518,7 +578,11 @@ impl<'a> Parser<'a> { { self.bump_sync(); // Consume "with" - let arguments = self.parse_argument_list()?; + let arguments = if stop_at_clause { + self.parse_argument_list_stopping_at_clause()? + } else { + self.parse_argument_list()? + }; return Ok(Expression::ActionCall { name: name.clone(), @@ -548,7 +612,8 @@ impl<'a> Parser<'a> { } Token::KeywordNot => { self.bump_sync(); // Consume "not" - let expr = self.parse_primary_expression()?; + let expr = + self.parse_primary_expression_with_clause_boundary(stop_at_clause)?; let token_line = token.line; let token_column = token.column; Ok(Expression::UnaryOperation { @@ -560,7 +625,8 @@ impl<'a> Parser<'a> { } Token::Minus => { self.bump_sync(); // Consume "-" - let expr = self.parse_primary_expression()?; + let expr = + self.parse_primary_expression_with_clause_boundary(stop_at_clause)?; let token_line = token.line; let token_column = token.column; Ok(Expression::UnaryOperation { @@ -572,7 +638,7 @@ impl<'a> Parser<'a> { } Token::KeywordWith => { self.bump_sync(); // Consume "with" - let expr = self.parse_expression()?; + let expr = self.parse_expression_with_clause_boundary(stop_at_clause)?; Ok(expr) } Token::KeywordCount => { @@ -702,7 +768,8 @@ impl<'a> Parser<'a> { { self.bump_sync(); // Consume "size" self.expect_token(Token::KeywordOf, "Expected 'of' after 'file size'")?; - let file_handle = self.parse_primary_expression()?; + let file_handle = + self.parse_primary_expression_with_clause_boundary(stop_at_clause)?; return Ok(Expression::FileSizeOf { file_handle: Box::new(file_handle), line: token_line, @@ -716,7 +783,8 @@ impl<'a> Parser<'a> { { self.bump_sync(); // Consume "exists" self.expect_token(Token::KeywordAt, "Expected 'at' after 'file exists'")?; - let path = self.parse_primary_expression()?; + let path = + self.parse_primary_expression_with_clause_boundary(stop_at_clause)?; return Ok(Expression::FileExists { path: Box::new(path), line: token_line, @@ -745,7 +813,8 @@ impl<'a> Parser<'a> { Token::KeywordAt, "Expected 'at' after 'directory exists'", )?; - let path = self.parse_primary_expression()?; + let path = + self.parse_primary_expression_with_clause_boundary(stop_at_clause)?; return Ok(Expression::DirectoryExists { path: Box::new(path), line: token_line, @@ -766,7 +835,8 @@ impl<'a> Parser<'a> { let token_column = token.column; // Parse process ID expression - let process_id = self.parse_primary_expression()?; + let process_id = + self.parse_primary_expression_with_clause_boundary(stop_at_clause)?; // Check if followed by "is running" if let Some(next_token) = self.cursor.peek() @@ -815,7 +885,8 @@ impl<'a> Parser<'a> { self.expect_token(Token::KeywordOf, "Expected 'of' after header name")?; // Parse request expression - let request = self.parse_primary_expression()?; + let request = + self.parse_primary_expression_with_clause_boundary(stop_at_clause)?; Ok(Expression::HeaderAccess { header_name, @@ -918,7 +989,8 @@ impl<'a> Parser<'a> { Token::KeywordIn, "Expected 'in' after 'list files [recursively]'", )?; - let path = self.parse_primary_expression()?; + let path = + self.parse_primary_expression_with_clause_boundary(stop_at_clause)?; // Handle recursive listing (if not already handled) if is_recursive { @@ -927,7 +999,7 @@ impl<'a> Parser<'a> { && with_token.token == Token::KeywordWith { self.bump_sync(); // Consume "with" - let extensions = self.parse_extension_filter()?; + let extensions = self.parse_extension_filter(stop_at_clause)?; return Ok(Expression::ListFilesRecursive { path: Box::new(path), extensions: Some(extensions), @@ -956,7 +1028,8 @@ impl<'a> Parser<'a> { && with_token.token == Token::KeywordWith { self.bump_sync(); // Consume "with" - let extensions = self.parse_extension_filter()?; + let extensions = + self.parse_extension_filter(stop_at_clause)?; return Ok(Expression::ListFilesRecursive { path: Box::new(path), extensions: Some(extensions), @@ -975,7 +1048,7 @@ impl<'a> Parser<'a> { } Token::KeywordWith => { self.bump_sync(); // Consume "with" - let extensions = self.parse_extension_filter()?; + let extensions = self.parse_extension_filter(stop_at_clause)?; return Ok(Expression::ListFilesFiltered { path: Box::new(path), extensions, @@ -1015,7 +1088,8 @@ impl<'a> Parser<'a> { Token::KeywordFrom, "Expected 'from' after 'read content'", )?; - let file_handle = self.parse_primary_expression()?; + let file_handle = + self.parse_primary_expression_with_clause_boundary(stop_at_clause)?; return Ok(Expression::ReadContent { file_handle: Box::new(file_handle), line: token_line, @@ -1030,7 +1104,8 @@ impl<'a> Parser<'a> { Token::KeywordFrom, "Expected 'from' after 'read binary'", )?; - let file_handle = self.parse_primary_expression()?; + let file_handle = + self.parse_primary_expression_with_clause_boundary(stop_at_clause)?; return Ok(Expression::ReadBinaryContent { file_handle: Box::new(file_handle), line: token_line, @@ -1045,7 +1120,8 @@ impl<'a> Parser<'a> { ) { // Speculatively parse count expression, then check for "bytes" let saved_pos = self.cursor.checkpoint(); - if let Ok(count_expr) = self.parse_primary_expression() + if let Ok(count_expr) = + self.parse_primary_expression_with_clause_boundary(stop_at_clause) && let Some(bytes_tok) = self.cursor.peek() && bytes_tok.token == Token::KeywordBytes { @@ -1054,7 +1130,10 @@ impl<'a> Parser<'a> { Token::KeywordFrom, "Expected 'from' after 'read N bytes'", )?; - let file_handle = self.parse_primary_expression()?; + let file_handle = self + .parse_primary_expression_with_clause_boundary( + stop_at_clause, + )?; return Ok(Expression::ReadBinaryN { file_handle: Box::new(file_handle), count: Box::new(count_expr), @@ -1076,12 +1155,13 @@ impl<'a> Parser<'a> { } Token::KeywordFind => { self.bump_sync(); // Consume "find" - let pattern_expr = self.parse_expression()?; + let pattern_expr = + self.parse_expression_with_clause_boundary(stop_at_clause)?; self.expect_token( Token::KeywordIn, "Expected 'in' after pattern in find expression", )?; - let text_expr = self.parse_expression()?; + let text_expr = self.parse_expression_with_clause_boundary(stop_at_clause)?; Ok(Expression::PatternFind { pattern: Box::new(pattern_expr), text: Box::new(text_expr), @@ -1091,17 +1171,19 @@ impl<'a> Parser<'a> { } Token::KeywordReplace => { self.bump_sync(); // Consume "replace" - let pattern_expr = self.parse_primary_expression()?; + let pattern_expr = + self.parse_primary_expression_with_clause_boundary(stop_at_clause)?; self.expect_token( Token::KeywordWith, "Expected 'with' after pattern in replace expression", )?; - let replacement_expr = self.parse_expression()?; + let replacement_expr = + self.parse_expression_with_clause_boundary(stop_at_clause)?; self.expect_token( Token::KeywordIn, "Expected 'in' after replacement in replace expression", )?; - let text_expr = self.parse_expression()?; + let text_expr = self.parse_expression_with_clause_boundary(stop_at_clause)?; Ok(Expression::PatternReplace { pattern: Box::new(pattern_expr), replacement: Box::new(replacement_expr), @@ -1117,7 +1199,7 @@ impl<'a> Parser<'a> { // optionally consuming "of" (equivalent to `split X by DELIM`). self.consume_optional_of(); - let text_expr = self.parse_expression()?; + let text_expr = self.parse_expression_with_clause_boundary(stop_at_clause)?; // Check for "by" (string split) or "on" (pattern split) if let Some(next_token) = self.cursor.peek() { @@ -1125,7 +1207,8 @@ impl<'a> Parser<'a> { Token::KeywordBy => { // Handle "split text by delimiter" syntax self.bump_sync(); // Consume "by" - let delimiter_expr = self.parse_expression()?; + let delimiter_expr = + self.parse_expression_with_clause_boundary(stop_at_clause)?; Ok(Expression::StringSplit { text: Box::new(text_expr), delimiter: Box::new(delimiter_expr), @@ -1140,7 +1223,8 @@ impl<'a> Parser<'a> { Token::KeywordPattern, "Expected 'pattern' after 'on' in split expression", )?; - let pattern_expr = self.parse_expression()?; + let pattern_expr = + self.parse_expression_with_clause_boundary(stop_at_clause)?; Ok(Expression::PatternSplit { text: Box::new(text_expr), pattern: Box::new(pattern_expr), @@ -1219,7 +1303,8 @@ impl<'a> Parser<'a> { } else { // Try to parse as "contains X in Y" // Parse the needle expression - let needle = self.parse_primary_expression()?; + let needle = + self.parse_primary_expression_with_clause_boundary(stop_at_clause)?; // Check if next token is "in" if let Some(in_token) = self.cursor.peek() @@ -1228,7 +1313,9 @@ impl<'a> Parser<'a> { self.bump_sync(); // Consume "in" // Parse the haystack expression - let haystack = self.parse_primary_expression()?; + let haystack = self.parse_primary_expression_with_clause_boundary( + stop_at_clause, + )?; // Create a function call expression for contains Ok(Expression::FunctionCall { @@ -1407,7 +1494,11 @@ impl<'a> Parser<'a> { // but stops at `and`, `with`, `from`/`by`, comparisons, // and pattern keywords so multi-argument and postfix // forms keep working. - let first_arg = self.parse_of_call_argument()?; + let first_arg = if stop_at_clause { + self.parse_of_call_argument_stopping_at_clause()? + } else { + self.parse_of_call_argument()? + }; let is_function_call = matches!( expr, @@ -1437,11 +1528,23 @@ impl<'a> Parser<'a> { ); if is_separator { + if stop_at_clause + && matches!(&sep_token.token, Token::KeywordAnd) + && Self::is_streaming_clause_keyword( + self.cursor.peek_n(1).map(|t| &t.token), + ) + { + break; + } self.bump_sync(); // Consume the separator // Each argument absorbs arithmetic while // `and`/`with`/`from`/`by` stay separators. - let arg_value = self.parse_of_call_argument()?; + let arg_value = if stop_at_clause { + self.parse_of_call_argument_stopping_at_clause()? + } else { + self.parse_of_call_argument()? + }; arguments.push(Argument { name: None, @@ -1469,7 +1572,11 @@ impl<'a> Parser<'a> { Token::KeywordAt => { self.bump_sync(); // Consume "at" - let index = self.parse_expression()?; + let index = if stop_at_clause { + self.parse_binary_expression_stopping_at_clause(0)? + } else { + self.parse_expression()? + }; expr = Expression::IndexAccess { collection: Box::new(expr), diff --git a/src/parser/stmt/io.rs b/src/parser/stmt/io.rs index f016e4f1..4130e491 100644 --- a/src/parser/stmt/io.rs +++ b/src/parser/stmt/io.rs @@ -7,14 +7,14 @@ use crate::parser::expr::{BinaryExprParser, ExprParser, PrimaryExprParser}; use std::sync::Arc; impl<'a> Parser<'a> { - /// Parse a merged-command operand from an already-chosen leading identifier: + /// Continue an operand from an already-parsed leading expression: /// trailing postfix (`[]`, `.field`, `.method()`, `at`, direct-integer index), /// an optional ` of ` call, then any `with`/operator /// continuation — exactly as a normal expression value would parse. /// - /// Shared by `write line|chunk` values and by merged `content type` / `headers` - /// clause operands so they all support the same postfix/call/operator grammar - /// (issue #642). + /// Shared by ordinary, merged, and unmerged `write line|chunk`, `content + /// type`, `headers`, and `flush` operands so they all support the same + /// postfix/call/operator grammar (issue #642). /// /// The ambiguous merged `write line|chunk ...` form has two readings /// (stream: split-off ``; classic file write: whole `line `) @@ -24,15 +24,20 @@ impl<'a> Parser<'a> { /// an `ActionCall`, `is between` duplicates the left, `starts/ends with` and /// the pattern operators build calls), so deriving one AST from the other by /// leaf-swapping silently corrupted the classic reading. - pub(crate) fn parse_merged_operand_from_lead( + pub(crate) fn parse_seeded_expression_continuation( &mut self, lead: Expression, + stop_at_clause: bool, ) -> Result { // The lexer merges the command word with the operand identifier and leaves // any bracket-index / dotted-property / `at` / integer-index accessors as // following tokens, so compose them onto the lead instead of leaving them // to dangle after the statement. - let lead = self.parse_trailing_postfix(lead)?; + let lead = if stop_at_clause { + self.parse_trailing_postfix_stopping_at_clause(lead)? + } else { + self.parse_trailing_postfix(lead)? + }; let lead = if matches!(self.cursor.peek().map(|t| &t.token), Some(Token::KeywordOf)) { // Anchor the ` of ` call to the `of` keyword itself, // matching how the rest of the parser positions FunctionCall nodes so @@ -47,7 +52,11 @@ impl<'a> Parser<'a> { // `and`/`from`/`by`/`length` join multiple arguments. let mut arguments = vec![crate::parser::ast::Argument { name: None, - value: self.parse_of_call_argument()?, + value: if stop_at_clause { + self.parse_of_call_argument_stopping_at_clause()? + } else { + self.parse_of_call_argument()? + }, }]; while let Some(sep) = self.cursor.peek() { let is_separator = matches!( @@ -60,10 +69,20 @@ impl<'a> Parser<'a> { if !is_separator { break; } + if stop_at_clause + && matches!(&sep.token, Token::KeywordAnd) + && Self::is_streaming_clause_keyword(self.cursor.peek_n(1).map(|t| &t.token)) + { + break; + } self.bump_sync(); // Consume the separator arguments.push(crate::parser::ast::Argument { name: None, - value: self.parse_of_call_argument()?, + value: if stop_at_clause { + self.parse_of_call_argument_stopping_at_clause()? + } else { + self.parse_of_call_argument()? + }, }); } Expression::FunctionCall { @@ -75,67 +94,37 @@ impl<'a> Parser<'a> { } else { lead }; - self.parse_binary_continuation(lead, 0) + if stop_at_clause { + self.parse_binary_continuation_stopping_at_clause(lead, 0) + } else { + self.parse_binary_continuation(lead, 0) + } } - /// Like [`parse_merged_operand_from_lead`], but binary continuation stops - /// before clause connectives (`and`/`with`/`as`) so - /// `content type mime_type of path and headers h` does not swallow `headers` - /// as a Boolean-AND operand (issue #642 re-review). + /// Parse a lexer-merged response-clause operand, stopping before the next + /// response clause connective. pub(crate) fn parse_clause_operand_from_lead( &mut self, lead: Expression, ) -> Result { - let lead = self.parse_trailing_postfix(lead)?; - let lead = if matches!(self.cursor.peek().map(|t| &t.token), Some(Token::KeywordOf)) { - let (of_line, of_column) = self - .bump_sync() - .map(|t| (t.line, t.column)) - .expect("peeked `of` immediately above"); - let mut arguments = vec![crate::parser::ast::Argument { - name: None, - value: self.parse_of_call_argument()?, - }]; - while let Some(sep) = self.cursor.peek() { - let is_separator = matches!( - &sep.token, - Token::KeywordAnd | Token::KeywordFrom | Token::KeywordBy - ) || matches!( - &sep.token, - Token::Identifier(id) if id.eq_ignore_ascii_case("length") - ); - // For multi-arg `of` calls, `and` between arguments is fine — - // only stop when we've finished the of-call and the next token - // would start a new clause (handled by binary continuation stop). - if !is_separator { - break; - } - // If `and` is followed by a clause keyword (headers/content/as), - // it is a clause connective, not an of-arg separator. - if matches!(&sep.token, Token::KeywordAnd) - && Self::is_streaming_clause_keyword(self.cursor.peek_n(1).map(|t| &t.token)) - { - break; - } - self.bump_sync(); - arguments.push(crate::parser::ast::Argument { - name: None, - value: self.parse_of_call_argument()?, - }); - } - Expression::FunctionCall { - function: Box::new(lead), - arguments, - line: of_line, - column: of_column, - } + self.parse_seeded_expression_continuation(lead, true) + } + + /// Parse a complete ordinary/unmerged operand through the same seeded + /// continuation used for lexer-merged operands. + pub(crate) fn parse_unmerged_operand( + &mut self, + stop_at_clause: bool, + ) -> Result { + let lead = if stop_at_clause { + self.parse_primary_expression_stopping_at_clause()? } else { - lead + self.parse_primary_expression()? }; - self.parse_binary_continuation_stopping_at_clause(lead, 0) + self.parse_seeded_expression_continuation(lead, stop_at_clause) } - fn is_streaming_clause_keyword(tok: Option<&Token>) -> bool { + pub(crate) fn is_streaming_clause_keyword(tok: Option<&Token>) -> bool { match tok { Some(Token::KeywordAs) | Some(Token::KeywordContent) | Some(Token::KeywordStatus) => { true @@ -155,7 +144,7 @@ impl<'a> Parser<'a> { /// Alias used by the write-statement parsers. fn parse_write_value_from_lead(&mut self, lead: Expression) -> Result { - self.parse_merged_operand_from_lead(lead) + self.parse_seeded_expression_continuation(lead, false) } } @@ -995,7 +984,7 @@ impl<'a> IoParser<'a> for Parser<'a> { // whole expression — including `with` concatenation — parses // cleanly from here. This form was never a valid classic file // write (`write line "x" to f` did not parse), so no fallback. - (self.parse_expression()?, None) + (self.parse_unmerged_operand(false)?, None) } else { // Ambiguous merged form: `` alone (stream) vs the full // merged `line ` (classic file write of that variable). diff --git a/src/parser/stmt/patterns.rs b/src/parser/stmt/patterns.rs index 33c15ab2..b070eb6d 100644 --- a/src/parser/stmt/patterns.rs +++ b/src/parser/stmt/patterns.rs @@ -80,7 +80,10 @@ pub(crate) trait PatternParser<'a>: ExprParser<'a> { i: &mut usize, base_pattern: PatternExpression, ) -> Result; - fn parse_extension_filter(&mut self) -> Result, ParseError>; + fn parse_extension_filter( + &mut self, + stop_at_clause: bool, + ) -> Result, ParseError>; } impl<'a> PatternParser<'a> for Parser<'a> { @@ -189,14 +192,17 @@ impl<'a> PatternParser<'a> for Parser<'a> { }) } - fn parse_extension_filter(&mut self) -> Result, ParseError> { + fn parse_extension_filter( + &mut self, + stop_at_clause: bool, + ) -> Result, ParseError> { // Expect "extension", "extensions", or "pattern" if let Some(token) = self.cursor.peek() { match &token.token { Token::KeywordExtension => { self.bump_sync(); // Consume "extension" // Parse single extension - let ext = self.parse_primary_expression()?; + let ext = self.parse_primary_expression_with_clause_boundary(stop_at_clause)?; Ok(vec![ext]) } Token::KeywordExtensions => { @@ -210,7 +216,8 @@ impl<'a> PatternParser<'a> for Parser<'a> { if has_bracket { // Parse list literal - let list_expr = self.parse_primary_expression()?; + let list_expr = + self.parse_primary_expression_with_clause_boundary(stop_at_clause)?; if let Expression::Literal(Literal::List(items), _, _) = list_expr { Ok(items) } else { @@ -223,14 +230,16 @@ impl<'a> PatternParser<'a> for Parser<'a> { } } else { // Allow a variable containing the extensions list - let expr = self.parse_primary_expression()?; + let expr = + self.parse_primary_expression_with_clause_boundary(stop_at_clause)?; Ok(vec![expr]) } } Token::KeywordPattern => { self.bump_sync(); // Consume "pattern" // Parse pattern expression (e.g., "*.wfl") - let expr = self.parse_primary_expression()?; + let expr = + self.parse_primary_expression_with_clause_boundary(stop_at_clause)?; Ok(vec![expr]) } _ => { diff --git a/src/parser/stmt/web.rs b/src/parser/stmt/web.rs index 7f9ea7b5..941b0393 100644 --- a/src/parser/stmt/web.rs +++ b/src/parser/stmt/web.rs @@ -454,7 +454,7 @@ impl<'a> WebParser<'a> for Parser<'a> { let lead = Expression::Variable(rest, l, c); self.parse_clause_operand_from_lead(lead)? } - None => self.parse_primary_expression()?, + None => self.parse_unmerged_operand(true)?, }); } // Merged `content_type ` / `content type ` form. @@ -476,7 +476,7 @@ impl<'a> WebParser<'a> for Parser<'a> { .unwrap_or("") }); if rest.is_empty() { - content_type = Some(self.parse_primary_expression()?); + content_type = Some(self.parse_unmerged_operand(true)?); } else { let lead = Expression::Variable(rest.to_string(), id_line, id_column); content_type = Some(self.parse_clause_operand_from_lead(lead)?); @@ -493,7 +493,7 @@ impl<'a> WebParser<'a> for Parser<'a> { .map(str::trim_start) .unwrap_or(""); if rest.is_empty() { - headers = Some(self.parse_primary_expression()?); + headers = Some(self.parse_unmerged_operand(true)?); } else { // Clause operand: postfix/`of`/operators but stop before // the next clause connective (`and content type`, `as`). @@ -547,8 +547,8 @@ impl<'a> WebParser<'a> for Parser<'a> { .strip_prefix("flush") .map(str::trim_start) .unwrap_or(""); - let (target, action_fallback) = if rest.is_empty() { - (self.parse_primary_expression()?, None) + let (target, legacy_binding, action_fallback) = if rest.is_empty() { + (self.parse_primary_expression()?, None, None) } else { // Stream reading: postfix on the split-off rest (`cache` from // `flush cache`). Legacy expression: same postfix on the FULL phrase @@ -556,15 +556,17 @@ impl<'a> WebParser<'a> for Parser<'a> { // `at` keep their old expression-statement AST (issue #642 re-review). let cp = self.cursor.checkpoint(); let stream_lead = Expression::Variable(rest.to_string(), line, column); - let target = self.parse_trailing_postfix(stream_lead)?; + let target = self.parse_seeded_expression_continuation(stream_lead, false)?; self.cursor.rewind(cp); - let legacy_lead = Expression::Variable(phrase.clone(), line, column); - let fallback = self.parse_trailing_postfix(legacy_lead)?; - (target, Some(fallback)) + let legacy_binding = phrase.clone(); + let legacy_lead = Expression::Variable(legacy_binding.clone(), line, column); + let fallback = self.parse_seeded_expression_continuation(legacy_lead, false)?; + (target, Some(legacy_binding), Some(fallback)) }; Ok(Statement::FlushStreamStatement { target, + legacy_binding, action_fallback, line, column, diff --git a/src/typechecker/mod.rs b/src/typechecker/mod.rs index 6e7da941..4a6b1176 100644 --- a/src/typechecker/mod.rs +++ b/src/typechecker/mod.rs @@ -1082,7 +1082,7 @@ impl TypeChecker { // Concrete response stream: the stream reading is taken. // Report undefined names on this branch (analyzer may have // stayed silent because the classic lead alone was defined). - self.check_expression_names_defined(value, *line, *column); + self.check_expression_names_defined(value); let value_type = self.infer_expression_type(value); self.check_streamable_payload(&value_type, *line, *column); } else if has_fallback @@ -1093,7 +1093,7 @@ impl TypeChecker { // the classic file-write reading is taken. Validate the // fallback (including definedness), not the stream `value`. if let Some(fallback) = fallback_content { - self.check_expression_names_defined(fallback, *line, *column); + self.check_expression_names_defined(fallback); let _ = self.infer_expression_type(fallback); } } else if self.is_gradual_type(&target_type) { @@ -1102,11 +1102,11 @@ impl TypeChecker { // validate EVERY viable branch (not "accept if either is ok"), // so a valid file fallback cannot mask an invalid stream // payload or an undefined stream lead (issue #642). - self.check_expression_names_defined(value, *line, *column); + self.check_expression_names_defined(value); let value_type = self.infer_expression_type(value); self.check_streamable_payload(&value_type, *line, *column); if let Some(fallback) = fallback_content { - self.check_expression_names_defined(fallback, *line, *column); + self.check_expression_names_defined(fallback); let _ = self.infer_expression_type(fallback); } } else { @@ -1125,6 +1125,7 @@ impl TypeChecker { } Statement::FlushStreamStatement { target, + legacy_binding, action_fallback, line, column, @@ -1132,24 +1133,9 @@ impl TypeChecker { // Legacy full-name expression (e.g. Variable("flush cache") or // IndexAccess over it): when its root is bound, typecheck that // expression. Otherwise this is a stream flush. - let legacy_root = action_fallback.as_ref().and_then(|e| match e { - Expression::Variable(n, ..) => Some(n.as_str()), - Expression::IndexAccess { collection, .. } - | Expression::PropertyAccess { - object: collection, .. - } - | Expression::MethodCall { - object: collection, .. - } => match &**collection { - Expression::Variable(n, ..) => Some(n.as_str()), - _ => None, - }, - _ => None, - }); - let is_expression_fallback = legacy_root.is_some_and(|name| { - self.action_signatures(name).is_some() - || self.analyzer.get_symbol(name).is_some() - }); + let is_expression_fallback = legacy_binding + .as_deref() + .is_some_and(|name| self.name_is_defined_for_write(name)); if is_expression_fallback { if let Some(fb) = action_fallback { let _ = self.infer_expression_type(fb); @@ -3131,6 +3117,8 @@ impl TypeChecker { // where a concrete type is required at runtime. Type::Unknown } + } else if let Some(property_type) = self.current_container_property_type(name) { + property_type } else { // Check if this is an action parameter, builtin function, or special function name before reporting it as undefined if self.analyzer.get_action_parameters().contains(name) @@ -4931,22 +4919,55 @@ impl TypeChecker { /// may have stayed silent on a one-sided undefined lead because the other /// reading was defined (issue #642). fn name_is_defined_for_write(&self, name: &str) -> bool { - // Match analyzer `name_is_defined` so container properties and inherited - // bindings are not false-rejected on the selected write branch. - self.analyzer.name_is_defined_for_write(name) + if self.analyzer.name_is_defined_for_write(name) { + return true; + } + + self.current_container_property_type(name).is_some() + } + + /// Resolve a direct or inherited property against the typechecker's live + /// container context. The analyzer has restored its own container context + /// by the time method bodies are typechecked, so both definedness and type + /// inference must use this view. + fn current_container_property_type(&self, name: &str) -> Option { + // Analyzer has already completed its container walk and restored its + // own `current_container` by the time TypeChecker revisits method + // bodies. Use TypeChecker's live container context here so direct and + // inherited properties remain defined on the selected write branch. + let mut container_name = self.current_container.as_deref(); + while let Some(container_key) = container_name { + let Some(container) = self.analyzer.get_container(container_key) else { + break; + }; + if let Some(property) = container + .properties + .get(name) + .or_else(|| container.static_properties.get(name)) + { + return Some(property.property_type.clone()); + } + container_name = container.extends.as_deref(); + } + + None } /// Walk an expression and report every undefined bare name. Used for the /// selected (or every viable gradual) `write line|chunk` branch so a missing /// classic `line ` lead is not accepted just because the stream lead /// alone exists (and vice versa). - fn check_expression_names_defined( - &mut self, - expression: &Expression, - line: usize, - column: usize, - ) { + fn check_expression_names_defined(&mut self, expression: &Expression) { match expression { + Expression::Literal(Literal::List(items), ..) => { + for item in items { + self.check_expression_names_defined(item); + } + } + Expression::Literal(_, _, _) + | Expression::StaticMemberAccess { .. } + | Expression::CurrentTimeMilliseconds { .. } + | Expression::CurrentTimeFormatted { .. } => {} Expression::Variable(name, l, c) => { if !self.name_is_defined_for_write(name) { self.type_error( @@ -4980,32 +5001,47 @@ impl TypeChecker { delimiter: right, .. } => { - self.check_expression_names_defined(left, line, column); - self.check_expression_names_defined(right, line, column); + self.check_expression_names_defined(left); + self.check_expression_names_defined(right); } Expression::UnaryOperation { expression: inner, .. } | Expression::AwaitExpression { expression: inner, .. + } + | Expression::FileExists { path: inner, .. } + | Expression::DirectoryExists { path: inner, .. } + | Expression::ListFiles { path: inner, .. } + | Expression::ReadContent { + file_handle: inner, .. + } + | Expression::ReadBinaryContent { + file_handle: inner, .. + } + | Expression::FileSizeOf { + file_handle: inner, .. + } + | Expression::ProcessRunning { + process_id: inner, .. } => { - self.check_expression_names_defined(inner, line, column); + self.check_expression_names_defined(inner); } Expression::IndexAccess { collection, index, .. } => { - self.check_expression_names_defined(collection, line, column); - self.check_expression_names_defined(index, line, column); + self.check_expression_names_defined(collection); + self.check_expression_names_defined(index); } Expression::PropertyAccess { object, .. } | Expression::MemberAccess { object, .. } => { - self.check_expression_names_defined(object, line, column); + self.check_expression_names_defined(object); } Expression::MethodCall { object, arguments, .. } => { - self.check_expression_names_defined(object, line, column); + self.check_expression_names_defined(object); for arg in arguments { - self.check_expression_names_defined(&arg.value, line, column); + self.check_expression_names_defined(&arg.value); } } Expression::FunctionCall { @@ -5013,14 +5049,14 @@ impl TypeChecker { arguments, .. } => { - self.check_expression_names_defined(function, line, column); + self.check_expression_names_defined(function); for arg in arguments { - self.check_expression_names_defined(&arg.value, line, column); + self.check_expression_names_defined(&arg.value); } } Expression::ActionCall { arguments, .. } => { for arg in arguments { - self.check_expression_names_defined(&arg.value, line, column); + self.check_expression_names_defined(&arg.value); } } Expression::PatternReplace { @@ -5029,16 +5065,48 @@ impl TypeChecker { replacement, .. } => { - self.check_expression_names_defined(text, line, column); - self.check_expression_names_defined(pattern, line, column); - self.check_expression_names_defined(replacement, line, column); + self.check_expression_names_defined(text); + self.check_expression_names_defined(pattern); + self.check_expression_names_defined(replacement); } Expression::HeaderAccess { request, .. } => { - self.check_expression_names_defined(request, line, column); + self.check_expression_names_defined(request); } - // Literals and other leaves need no definedness walk. - _ => { - let _ = (line, column); + Expression::ReadBinaryN { + file_handle, count, .. + } => { + self.check_expression_names_defined(file_handle); + self.check_expression_names_defined(count); + } + Expression::ListFilesRecursive { + path, extensions, .. + } => { + self.check_expression_names_defined(path); + if let Some(extensions) = extensions { + for extension in extensions { + self.check_expression_names_defined(extension); + } + } + } + Expression::ListFilesFiltered { + path, extensions, .. + } => { + self.check_expression_names_defined(path); + for extension in extensions { + self.check_expression_names_defined(extension); + } + } + Expression::DatabaseQuery { + db, + sql, + parameters, + .. + } => { + self.check_expression_names_defined(db); + self.check_expression_names_defined(sql); + if let Some(parameters) = parameters { + self.check_expression_names_defined(parameters); + } } } } diff --git a/tests/ambiguous_write_branch_typecheck_test.rs b/tests/ambiguous_write_branch_typecheck_test.rs index 6fe70717..6303f06f 100644 --- a/tests/ambiguous_write_branch_typecheck_test.rs +++ b/tests/ambiguous_write_branch_typecheck_test.rs @@ -96,9 +96,10 @@ fn text_target_one_sided_undefined_classic_lead_is_caught() { // (issue #642: previously analysis passed because only the stream lead existed). let code = "store value as \"x\"\n\ write line value to \"/tmp/wfl_onesided_out\""; + let errors = typecheck(code).expect_err("undefined classic lead must be rejected"); assert!( - typecheck(code).is_err(), - "undefined classic lead on a concrete text target must be a static error" + errors.contains("Variable 'line value' is not defined"), + "expected the selected classic-lead diagnostic, got: {errors}" ); } @@ -114,10 +115,10 @@ fn main_loop_stream_binding_rejects_list_payload() { \x20\x20\x20\x20store line items as \"legacy\"\n\ \x20\x20\x20\x20write line items to out\n\ end loop"; + let errors = typecheck(code).expect_err("list payload must be rejected"); assert!( - typecheck(code).is_err(), - "list payload to a main-loop response stream must be a static error, got: {:?}", - typecheck(code).err() + errors.contains("can only send text, binary, a number, or a boolean"), + "expected the concrete stream-payload diagnostic, got: {errors}" ); } @@ -127,8 +128,211 @@ fn property_access_undefined_object_on_text_target_is_caught() { // classic is `line upstream.status`. Neither object is defined; PropertyAccess // must not evade definedness on the concrete text-target branch (issue #642). let code = "write line upstream.status to \"/tmp/wfl_prop_out\""; + let errors = typecheck(code).expect_err("undefined property root must be rejected"); assert!( - typecheck(code).is_err(), - "undefined property object on a text target must be a static error" + errors.contains("Variable 'line upstream' is not defined"), + "expected the selected classic property-root diagnostic, got: {errors}" + ); +} + +#[test] +fn property_access_checks_the_one_sided_root_for_each_concrete_target() { + let file_errors = typecheck( + "store upstream as \"stream-only\"\n\ + write line upstream.status to \"/tmp/wfl_prop_out\"", + ) + .expect_err("the selected classic property root is undefined"); + assert!( + file_errors.contains("Variable 'line upstream' is not defined"), + "text target must diagnose the classic property root, got: {file_errors}" + ); + + let stream_errors = typecheck( + "store line upstream as \"classic-only\"\n\ + listen on port 8080 as s\n\ + wait for request comes in on s as req\n\ + start streaming response to req with status 200 as out\n\ + write line upstream.status to out", + ) + .expect_err("the selected stream property root is undefined"); + assert!( + stream_errors.contains("Variable 'upstream' is not defined"), + "response-stream target must diagnose the stream property root, got: {stream_errors}" + ); +} + +#[test] +fn container_property_is_defined_on_the_concrete_file_branch() { + // The analyzer recognizes `line value` as a property while it is visiting + // `Writer`, then restores its own container context before the typechecker + // revisits the method. Branch-specific definedness must use the + // typechecker's live container context rather than falsely rejecting the + // valid classic file-write reading. + let code = "create container Writer:\n\ + \x20\x20\x20\x20property line value: Text\n\ + \x20\x20\x20\x20action dump:\n\ + \x20\x20\x20\x20\x20\x20\x20\x20write line value to \"C:/tmp/out\"\n\ + \x20\x20\x20\x20end\n\ + end"; + assert!( + typecheck(code).is_ok(), + "a container property used by the selected file-write branch must remain defined: {:?}", + typecheck(code).err() + ); +} + +#[test] +fn response_stream_branch_rejects_an_undefined_stream_lead() { + // Only the classic merged lead (`line value`) exists. Because `out` is a + // concrete ResponseStream, runtime selects the stream reading (`value`), + // which must still be rejected as undefined. + let code = "store line value as \"legacy file payload\"\n\ + listen on port 8080 as s\n\ + wait for request comes in on s as req\n\ + start streaming response to req with status 200 as out\n\ + write line value to out"; + let errors = typecheck(code).expect_err("undefined stream lead must be rejected"); + assert!( + errors.contains("Variable 'value' is not defined"), + "expected the selected stream-lead diagnostic, got: {errors}" + ); +} + +#[test] +fn action_body_stream_binding_remains_concrete_for_payload_checking() { + let errors = typecheck( + "define action called handle with parameters request_value:\n\ + start streaming response to request_value with status 200 as out\n\ + store items as [1 and 2]\n\ + store line items as \"classic\"\n\ + write line items to out\n\ + end action", + ) + .expect_err("an action-local ResponseStream must reject a List payload"); + assert!( + errors.contains("can only send text, binary, a number, or a boolean"), + "expected the action-body stream-payload diagnostic, got: {errors}" + ); +} + +#[test] +fn action_local_is_defined_on_the_concrete_file_branch() { + let code = "define action called dump:\n\ + \x20\x20\x20\x20store line value as \"action-local\"\n\ + \x20\x20\x20\x20write line value to \"C:/tmp/out\"\n\ + end action"; + assert!( + typecheck(code).is_ok(), + "an action-local merged lead must remain visible during branch checking: {:?}", + typecheck(code).err() + ); +} + +#[test] +fn main_loop_local_is_defined_on_the_concrete_file_branch() { + let code = "main loop:\n\ + \x20\x20\x20\x20store line value as \"loop-local\"\n\ + \x20\x20\x20\x20write line value to \"C:/tmp/out\"\n\ + end loop"; + assert!( + typecheck(code).is_ok(), + "a main-loop-local merged lead must remain visible during branch checking: {:?}", + typecheck(code).err() + ); +} + +#[test] +fn wrapped_missing_name_is_rejected_on_the_concrete_file_branch() { + let errors = typecheck( + "store line value as \"prefix\"\n\ + write line value with file exists at missing_path to \"C:/tmp/wfl_wrapped_out\"", + ) + .expect_err("the selected classic branch must validate names inside FileExists"); + assert!( + errors.contains("Variable 'missing_path' is not defined"), + "expected the wrapped path diagnostic, got: {errors}" + ); +} + +#[test] +fn wrapped_missing_name_is_rejected_on_the_concrete_stream_branch() { + let errors = typecheck( + "store value as \"prefix\"\n\ + listen on port 8080 as s\n\ + wait for request comes in on s as req\n\ + start streaming response to req with status 200 as out\n\ + write line value with file exists at missing_path to out", + ) + .expect_err("the selected stream branch must validate names inside FileExists"); + assert!( + errors.contains("Variable 'missing_path' is not defined"), + "expected the wrapped path diagnostic, got: {errors}" + ); +} + +#[test] +fn wrapped_missing_name_is_rejected_on_every_gradual_branch() { + let errors = typecheck( + "define action called send with parameters destination:\n\ + store value as \"stream\"\n\ + store line value as \"classic\"\n\ + write line value with file exists at missing_path to destination\n\ + end action", + ) + .expect_err("a gradual target must validate wrapped names on every viable branch"); + assert!( + errors.contains("Variable 'missing_path' is not defined"), + "expected the wrapped path diagnostic, got: {errors}" + ); +} + +#[test] +fn gradual_target_requires_both_candidate_leads_to_be_defined() { + let undefined_stream_lead = "define action called send with parameters target:\n\ + \x20\x20\x20\x20store line value as \"classic\"\n\ + \x20\x20\x20\x20write line value to target\n\ + end action"; + let stream_errors = + typecheck(undefined_stream_lead).expect_err("the viable stream lead must be defined"); + assert!( + stream_errors.contains("Variable 'value' is not defined"), + "expected the gradual stream-lead diagnostic, got: {stream_errors}" + ); + + let undefined_classic_lead = "define action called send with parameters target:\n\ + \x20\x20\x20\x20store value as \"stream\"\n\ + \x20\x20\x20\x20write line value to target\n\ + end action"; + let classic_errors = + typecheck(undefined_classic_lead).expect_err("the viable classic lead must be defined"); + assert!( + classic_errors.contains("Variable 'line value' is not defined"), + "expected the gradual classic-lead diagnostic, got: {classic_errors}" + ); +} + +#[test] +fn gradual_target_validates_every_viable_payload_branch() { + let invalid_stream_payload = "define action called send with parameters target:\n\ + \x20\x20\x20\x20store value as [1 and 2]\n\ + \x20\x20\x20\x20store line value as \"classic\"\n\ + \x20\x20\x20\x20write line value to target\n\ + end action"; + let payload_errors = + typecheck(invalid_stream_payload).expect_err("the viable stream payload must be valid"); + assert!( + payload_errors.contains("can only send text, binary, a number, or a boolean"), + "expected the gradual stream-payload diagnostic, got: {payload_errors}" + ); + + let both_valid = "define action called send with parameters target:\n\ + \x20\x20\x20\x20store value as \"stream\"\n\ + \x20\x20\x20\x20store line value as \"classic\"\n\ + \x20\x20\x20\x20write line value to target\n\ + end action"; + assert!( + typecheck(both_valid).is_ok(), + "both viable gradual branches are valid: {:?}", + typecheck(both_valid).err() ); } diff --git a/tests/concurrent_disconnect_paths_burst_test.rs b/tests/concurrent_disconnect_paths_burst_test.rs index c68c1540..75ac6cc2 100644 --- a/tests/concurrent_disconnect_paths_burst_test.rs +++ b/tests/concurrent_disconnect_paths_burst_test.rs @@ -12,41 +12,426 @@ //! //! Also: every client that is intended to exercise a path must actually connect and //! reach that lifecycle point (no silent early-return that leaves the burst under the -//! breaker threshold), and the test waits long enough for every handler result to be -//! consumed before probing `/ping` (so a General-classified disconnect cannot race -//! past a premature success that resets the counter). +//! breaker threshold), and an explicit handler-start barrier proves every intended +//! result was consumed before probing `/ping` (so a General-classified disconnect +//! cannot race past a premature success that resets the counter). +use std::collections::HashSet; use std::sync::Arc; use std::sync::atomic::{AtomicUsize, Ordering}; use std::time::Duration; use tokio::io::{AsyncReadExt, AsyncWriteExt}; -use tokio::sync::Semaphore; +use tokio::sync::{mpsc, watch}; use wfl::Interpreter; use wfl::lexer::lex_wfl_with_positions; use wfl::parser::Parser; mod common; -/// More than the breaker threshold (256), with NO successful request in between. -const DISCONNECT_BURST: usize = 270; -const CLIENT_CONCURRENCY: usize = 40; -/// Upper bound on how long the General-failure backoff would take to consume 256 -/// failures (~11.5s) plus handler work. Waiting past that guarantees the counter -/// would have tripped if any of the disconnects were misclassified as structural. -const DRAIN_AFTER_BURST: Duration = Duration::from_secs(15); +/// Exactly fill the concurrent-handler cap. A second full wave cannot reach its +/// lifecycle checkpoint until every result from the first wave has been consumed. +const DISCONNECT_WAVE: usize = 256; +/// Both waves disconnect before `/ping`, so each path exercises 512 clients (>256). +const DISCONNECT_TOTAL: usize = DISCONNECT_WAVE * 2; +const _: () = assert!(DISCONNECT_TOTAL > 256); +/// The initial 256 handlers plus one replacement handler per consumed disconnect +/// across both waves. Observing this ordinal proves every one of the 512 intended +/// disconnect outcomes left `FuturesUnordered` before `/ping` is allowed to run. +const POST_DISCONNECT_BARRIER: usize = DISCONNECT_WAVE + DISCONNECT_TOTAL; +const WAVE_DEADLINE: Duration = Duration::from_secs(30); +/// The WFL handler waits this long after its checkpoint is released. That gives the +/// test time to close every downstream socket before `respond` / response-head send. +const POST_CHECKPOINT_DELAY_MS: u64 = 1_000; +const ITERATION_PROOF_DEADLINE: Duration = Duration::from_secs(20); -fn start_proxy_server(code: String) -> std::thread::JoinHandle<()> { - std::thread::spawn(move || { +/// These cases deliberately fill the 256-handler cap. Rust's test harness otherwise +/// runs all four cases in this binary in parallel, multiplying the live socket/task +/// peak. Serializing the heavyweight cases keeps the test-host resource bound at one +/// full handler wave while preserving the real >256-request breaker proof. +static HEAVY_CASE_LOCK: tokio::sync::Mutex<()> = tokio::sync::Mutex::const_new(()); + +struct CountingGate { + port: u16, + arrivals: mpsc::UnboundedReceiver, + acknowledgements: mpsc::UnboundedReceiver, + errors: mpsc::UnboundedReceiver, + release_wave: watch::Sender, +} + +/// An HTTP checkpoint shared by the two request waves. Each handler opens the +/// checkpoint before its response operation. The mock reports all arrivals, then +/// withholds the HTTP head until the test releases that wave. This proves all 256 +/// handlers reached the intended path without relying on sleeps or successful TCP +/// writes alone. +async fn spawn_counting_gate() -> CountingGate { + let listener = tokio::net::TcpListener::bind("127.0.0.1:0") + .await + .expect("bind counting gate"); + let port = listener.local_addr().expect("counting gate address").port(); + let next_checkpoint_ordinal = Arc::new(AtomicUsize::new(0)); + let next_ack_ordinal = Arc::new(AtomicUsize::new(0)); + let (arrival_tx, arrivals) = mpsc::unbounded_channel(); + let (ack_tx, acknowledgements) = mpsc::unbounded_channel(); + let (error_tx, errors) = mpsc::unbounded_channel(); + let (release_wave, release_guard) = watch::channel(0usize); + + tokio::spawn(async move { + // Keep one receiver alive between waves so `release_wave.send()` cannot + // fail in the brief interval after the prior wave's connections close. + let release_guard = release_guard; + loop { + let Ok((mut sock, _)) = listener.accept().await else { + return; + }; + let next_checkpoint_ordinal = Arc::clone(&next_checkpoint_ordinal); + let next_ack_ordinal = Arc::clone(&next_ack_ordinal); + let arrival_tx = arrival_tx.clone(); + let ack_tx = ack_tx.clone(); + let error_tx = error_tx.clone(); + let mut release_rx = release_guard.clone(); + tokio::spawn(async move { + let result = async { + let head = read_http_head(&mut sock).await?; + let request = String::from_utf8_lossy(&head); + let path = request + .lines() + .next() + .and_then(|line| line.split_whitespace().nth(1)) + .ok_or_else(|| { + "counting gate received a malformed request line".to_string() + })?; + match path { + "/checkpoint" => { + let ordinal = + next_checkpoint_ordinal.fetch_add(1, Ordering::Relaxed) + 1; + let wave = ((ordinal - 1) / DISCONNECT_WAVE) + 1; + arrival_tx + .send(ordinal) + .map_err(|_| "checkpoint arrival receiver dropped".to_string())?; + release_rx + .wait_for(|released_wave| *released_wave >= wave) + .await + .map_err(|_| "counting-gate release sender dropped".to_string())?; + sock.write_all( + b"HTTP/1.1 200 OK\r\n\ + Content-Length: 0\r\n\ + Connection: close\r\n\r\n", + ) + .await + .map_err(|error| format!("write checkpoint response: {error}"))?; + sock.flush() + .await + .map_err(|error| format!("flush checkpoint response: {error}")) + } + "/ack" => { + let ordinal = next_ack_ordinal.fetch_add(1, Ordering::Relaxed) + 1; + // Deliberately leave this chunked response unfinished. The + // handler reads the marker and explicitly closes its + // outbound stream; observing EOF below is an exact + // handler-side acknowledgement that the earlier + // `/checkpoint` open returned and the local post-checkpoint + // wait is now the next operation. + sock.write_all( + b"HTTP/1.1 200 OK\r\n\ + Content-Type: text/plain\r\n\ + Transfer-Encoding: chunked\r\n\ + Connection: keep-alive\r\n\r\n\ + 1\r\nA\r\n", + ) + .await + .map_err(|error| format!("write acknowledgement marker: {error}"))?; + sock.flush().await.map_err(|error| { + format!("flush acknowledgement marker: {error}") + })?; + let deadline = tokio::time::Instant::now() + WAVE_DEADLINE; + let mut byte = [0u8; 1]; + loop { + let read = tokio::time::timeout_at(deadline, sock.read(&mut byte)) + .await + .map_err(|_| { + format!( + "handler did not close acknowledgement stream {ordinal}" + ) + })? + .map_err(|error| { + format!("read acknowledgement close {ordinal}: {error}") + })?; + if read == 0 { + break; + } + } + ack_tx + .send(ordinal) + .map_err(|_| "handler acknowledgement receiver dropped".to_string()) + } + other => Err(format!("unexpected counting-gate path {other:?}")), + } + } + .await; + if let Err(error) = result { + let _ = error_tx.send(error); + } + }); + } + }); + + CountingGate { + port, + arrivals, + acknowledgements, + errors, + release_wave, + } +} + +async fn read_http_head(sock: &mut tokio::net::TcpStream) -> Result, String> { + let deadline = tokio::time::Instant::now() + Duration::from_secs(5); + let mut head = Vec::new(); + let mut buf = [0u8; 512]; + loop { + let n = tokio::time::timeout_at(deadline, sock.read(&mut buf)) + .await + .map_err(|_| "timed out waiting for HTTP head".to_string())? + .map_err(|error| format!("read HTTP head: {error}"))?; + if n == 0 { + return Err("connection closed before the complete HTTP head".to_string()); + } + head.extend_from_slice(&buf[..n]); + if head.windows(4).any(|window| window == b"\r\n\r\n") { + return Ok(head); + } + if head.len() > 16 * 1024 { + return Err("HTTP head exceeded 16 KiB".to_string()); + } + } +} + +struct IterationCounter { + port: u16, + arrivals: mpsc::UnboundedReceiver, + errors: mpsc::UnboundedReceiver, +} + +/// Count a handler-start request and return an empty response immediately. A +/// concurrent handler calls this before waiting for a request. Since the handler +/// then remains parked in `wait for request`, each later start is observable proof +/// that the outer loop consumed one prior handler result and refilled its slot. +async fn spawn_iteration_counter() -> IterationCounter { + let listener = tokio::net::TcpListener::bind("127.0.0.1:0") + .await + .expect("bind iteration counter"); + let port = listener + .local_addr() + .expect("iteration counter address") + .port(); + let ordinal = Arc::new(AtomicUsize::new(0)); + let (arrival_tx, arrivals) = mpsc::unbounded_channel(); + let (error_tx, errors) = mpsc::unbounded_channel(); + + tokio::spawn(async move { + loop { + let Ok((mut sock, _)) = listener.accept().await else { + return; + }; + let ordinal = Arc::clone(&ordinal); + let arrival_tx = arrival_tx.clone(); + let error_tx = error_tx.clone(); + tokio::spawn(async move { + let result = async { + let head = read_http_head(&mut sock).await?; + let request = String::from_utf8_lossy(&head); + if !request.starts_with("GET /tick ") { + return Err(format!( + "unexpected iteration-counter request: {:?}", + request.lines().next() + )); + } + let current = ordinal.fetch_add(1, Ordering::Relaxed) + 1; + arrival_tx + .send(current) + .map_err(|_| "iteration arrival receiver dropped".to_string())?; + sock.write_all( + b"HTTP/1.1 200 OK\r\n\ + Content-Length: 0\r\n\ + Connection: close\r\n\r\n", + ) + .await + .map_err(|error| format!("write iteration response: {error}"))?; + sock.flush() + .await + .map_err(|error| format!("flush iteration response: {error}")) + } + .await; + if let Err(error) = result { + let _ = error_tx.send(error); + } + }); + } + }); + + IterationCounter { + port, + arrivals, + errors, + } +} + +async fn wait_for_proven_handler_iterations( + counter: &mut IterationCounter, + expected: usize, + context: &str, +) { + let deadline = tokio::time::Instant::now() + ITERATION_PROOF_DEADLINE; + let mut seen = HashSet::with_capacity(expected); + while seen.len() < expected { + let ordinal = tokio::time::timeout_at(deadline, async { + tokio::select! { + ordinal = counter.arrivals.recv() => { + ordinal.expect("iteration arrival channel closed") + } + error = counter.errors.recv() => { + panic!( + "iteration counter failed before proving {expected} {context} \ + handler starts: {}", + error.expect("iteration counter error channel closed") + ) + } + } + }) + .await + .unwrap_or_else(|_| { + panic!( + "observed only {} of {expected} {context} handler starts; the \ + {expected}th start is required to prove the loop consumed every \ + intended result before the liveness probe", + seen.len() + ) + }); + assert!( + seen.insert(ordinal), + "iteration counter duplicated handler-start ordinal {ordinal}" + ); + } +} + +async fn wait_for_gate_arrivals( + arrivals: &mut mpsc::UnboundedReceiver, + errors: &mut mpsc::UnboundedReceiver, + wave: usize, + context: &str, +) { + let first = (wave - 1) * DISCONNECT_WAVE + 1; + let last = wave * DISCONNECT_WAVE; + let deadline = tokio::time::Instant::now() + WAVE_DEADLINE; + let mut seen = HashSet::with_capacity(DISCONNECT_WAVE); + while seen.len() < DISCONNECT_WAVE { + let ordinal = tokio::time::timeout_at(deadline, async { + tokio::select! { + ordinal = arrivals.recv() => { + ordinal.expect("counting-gate arrival channel closed") + } + error = errors.recv() => { + panic!( + "{context} counting gate failed while waiting for wave {wave}: {}", + error.expect("counting-gate error channel closed") + ) + } + } + }) + .await + .unwrap_or_else(|_| { + panic!( + "only {} of {DISCONNECT_WAVE} {context} handlers reached the \ + counting checkpoint in wave {wave}", + seen.len() + ) + }); + assert!( + (first..=last).contains(&ordinal), + "unexpected counting-gate ordinal {ordinal} while waiting for wave {wave} \ + ({first}..={last})" + ); + assert!( + seen.insert(ordinal), + "duplicate counting-gate arrival ordinal {ordinal}" + ); + } +} + +async fn wait_for_handler_acknowledgements( + acknowledgements: &mut mpsc::UnboundedReceiver, + errors: &mut mpsc::UnboundedReceiver, + wave: usize, + context: &str, +) { + let first = (wave - 1) * DISCONNECT_WAVE + 1; + let last = wave * DISCONNECT_WAVE; + let deadline = tokio::time::Instant::now() + WAVE_DEADLINE; + let mut seen = HashSet::with_capacity(DISCONNECT_WAVE); + while seen.len() < DISCONNECT_WAVE { + let ordinal = tokio::time::timeout_at(deadline, async { + tokio::select! { + ordinal = acknowledgements.recv() => { + ordinal.expect("handler acknowledgement channel closed") + } + error = errors.recv() => { + panic!( + "{context} counting gate failed while waiting for acknowledgements \ + in wave {wave}: {}", + error.expect("counting-gate error channel closed") + ) + } + } + }) + .await + .unwrap_or_else(|_| { + panic!( + "only {} of {DISCONNECT_WAVE} {context} handlers acknowledged the \ + completed checkpoint in wave {wave}", + seen.len() + ) + }); + assert!( + (first..=last).contains(&ordinal), + "unexpected handler acknowledgement ordinal {ordinal} while waiting for wave {wave}" + ); + assert!( + seen.insert(ordinal), + "duplicate handler acknowledgement ordinal {ordinal}" + ); + } +} + +struct ProxyServer { + thread: std::thread::JoinHandle<()>, + abort: Option>, +} + +fn start_proxy_server(code: String) -> ProxyServer { + let (abort, abort_rx) = tokio::sync::oneshot::channel(); + let thread = std::thread::spawn(move || { let rt = tokio::runtime::Runtime::new().expect("runtime"); rt.block_on(async { let tokens = lex_wfl_with_positions(&code); let ast = Parser::new(&tokens).parse().expect("parse"); let mut interp = Interpreter::new(); - if let Err(errors) = interp.interpret(&ast).await { - panic!("server interpreter failed: {errors:?}"); + tokio::select! { + result = interp.interpret(&ast) => { + if let Err(errors) = result { + panic!("server interpreter failed: {errors:?}"); + } + } + _ = abort_rx => { + // Test cleanup: dropping the interpret future closes listeners, + // pending responses, and response streams. + } } }); - }) + }); + ProxyServer { + thread, + abort: Some(abort), + } } async fn wait_for_server(port: u16) { @@ -60,85 +445,149 @@ async fn wait_for_server(port: u16) { panic!("server on {addr} did not become ready"); } -/// Connect, send the request so the server enqueues and dequeues it, briefly hold so -/// the handler is inside its pre-reply work, then disconnect. -/// -/// - `read_head == false`: disconnect after a short hold so the disconnect lands -/// before `respond` / before the streaming head is sent. -/// - `read_head == true`: wait for the streaming response head first, so the -/// disconnect lands after the head, at the write path. -/// -/// Returns whether the client successfully connected and sent the request (so the -/// burst can assert every intended disconnect actually reached the server). -async fn fire_disconnect(port: u16, path: &str, read_head: bool) -> bool { - let Ok(mut sock) = tokio::net::TcpStream::connect(("127.0.0.1", port)).await else { - return false; - }; +async fn hold_client_until_disconnect( + port: u16, + path: &'static str, + mut disconnect: watch::Receiver, +) -> Result<(), String> { + let mut sock = tokio::time::timeout( + Duration::from_secs(5), + tokio::net::TcpStream::connect(("127.0.0.1", port)), + ) + .await + .map_err(|_| format!("timed out connecting to {path}"))? + .map_err(|error| format!("connect {path}: {error}"))?; let req = format!("GET {path} HTTP/1.1\r\nHost: 127.0.0.1\r\n\r\n"); - if sock.write_all(req.as_bytes()).await.is_err() { - return false; - } - if sock.flush().await.is_err() { - return false; - } - if read_head { - let mut acc = Vec::new(); - let mut tmp = [0u8; 256]; - let mut saw_head = false; - loop { - match tokio::time::timeout(Duration::from_secs(5), sock.read(&mut tmp)).await { - Ok(Ok(0)) | Err(_) | Ok(Err(_)) => break, - Ok(Ok(n)) => { - acc.extend_from_slice(&tmp[..n]); - if acc.windows(4).any(|w| w == b"\r\n\r\n") { - saw_head = true; - break; - } - } - } - } - if !saw_head { - return false; - } - } else { - // Give the server time to enqueue + dequeue the request and enter the - // handler's pre-reply wait, so the disconnect lands before `respond` / - // before the streaming head. - tokio::time::sleep(Duration::from_millis(200)).await; - } - // Drop `sock` -> disconnect. - true + sock.write_all(req.as_bytes()) + .await + .map_err(|error| format!("send {path} request: {error}"))?; + sock.flush() + .await + .map_err(|error| format!("flush {path} request: {error}"))?; + disconnect + .wait_for(|should_disconnect| *should_disconnect) + .await + .map_err(|_| format!("{path} disconnect signal sender dropped"))?; + drop(sock); + Ok(()) } -async fn fire_burst(port: u16, path: &'static str, read_head: bool) -> usize { - let sem = Arc::new(Semaphore::new(CLIENT_CONCURRENCY)); - let connected = Arc::new(AtomicUsize::new(0)); - let mut tasks = Vec::with_capacity(DISCONNECT_BURST); - for _ in 0..DISCONNECT_BURST { - let sem = Arc::clone(&sem); - let connected = Arc::clone(&connected); +async fn disconnect_after_stream_head(port: u16, path: &'static str) -> Result<(), String> { + let mut sock = tokio::time::timeout( + Duration::from_secs(5), + tokio::net::TcpStream::connect(("127.0.0.1", port)), + ) + .await + .map_err(|_| format!("timed out connecting to {path}"))? + .map_err(|error| format!("connect {path}: {error}"))?; + let req = format!("GET {path} HTTP/1.1\r\nHost: 127.0.0.1\r\n\r\n"); + sock.write_all(req.as_bytes()) + .await + .map_err(|error| format!("send {path} request: {error}"))?; + sock.flush() + .await + .map_err(|error| format!("flush {path} request: {error}"))?; + let head = read_http_head(&mut sock).await?; + let head = String::from_utf8_lossy(&head); + assert!( + head.starts_with("HTTP/1.1 200"), + "{path} must reach a successful streaming response head before disconnect; \ + got {head:?}" + ); + drop(sock); + Ok(()) +} + +fn spawn_gated_client_wave( + port: u16, + path: &'static str, +) -> ( + watch::Sender, + Vec>>, +) { + let (disconnect, disconnect_rx) = watch::channel(false); + let mut tasks = Vec::with_capacity(DISCONNECT_WAVE); + for _ in 0..DISCONNECT_WAVE { + let disconnect_rx = disconnect_rx.clone(); tasks.push(tokio::spawn(async move { - let _permit = sem.acquire().await.expect("semaphore"); - if fire_disconnect(port, path, read_head).await { - connected.fetch_add(1, Ordering::Relaxed); - } + hold_client_until_disconnect(port, path, disconnect_rx).await })); } - for t in tasks { - let _ = t.await; - } - let n = connected.load(Ordering::Relaxed); - assert!( - n > 256, - "expected more than 256 clients to actually connect and reach the intended \ - lifecycle point (so the burst exceeds the structural breaker threshold); \ - only {n} of {DISCONNECT_BURST} succeeded (path={path}, read_head={read_head})" + (disconnect, tasks) +} + +fn spawn_stream_client_wave( + port: u16, + path: &'static str, +) -> Vec>> { + (0..DISCONNECT_WAVE) + .map(|_| tokio::spawn(disconnect_after_stream_head(port, path))) + .collect() +} + +async fn join_client_wave( + tasks: Vec>>, + wave: usize, + context: &str, +) { + assert_eq!( + tasks.len(), + DISCONNECT_WAVE, + "each {context} wave must contain exactly {DISCONNECT_WAVE} clients" ); - // Drain past the General-failure backoff window so every intended handler - // result is consumed before `/ping`. If any disconnect were still classified - // as structural General, the breaker would trip during this wait. - tokio::time::sleep(DRAIN_AFTER_BURST).await; - n + tokio::time::timeout(WAVE_DEADLINE, async move { + for (index, task) in tasks.into_iter().enumerate() { + let result = task + .await + .unwrap_or_else(|error| panic!("{context} client task {index} panicked: {error}")); + result.unwrap_or_else(|error| { + panic!("{context} client task {index} failed in wave {wave}: {error}") + }); + } + }) + .await + .unwrap_or_else(|_| panic!("{context} client joins timed out in wave {wave}")); +} + +/// Drive two full checkpointed waves. All 256 first-wave handlers are held inside +/// the checkpoint simultaneously. The second wave cannot put all 256 handlers into +/// that checkpoint until the loop has consumed every first-wave result. After this +/// returns, the separate handler-start barrier proves the second-wave results were +/// consumed too, before `/ping` is sent. +async fn drive_two_gated_disconnect_waves( + port: u16, + path: &'static str, + gate: &mut CountingGate, + context: &str, +) { + for wave in 1..=2 { + let (disconnect, clients) = spawn_gated_client_wave(port, path); + wait_for_gate_arrivals(&mut gate.arrivals, &mut gate.errors, wave, context).await; + gate.release_wave + .send(wave) + .expect("counting-gate release receiver stays alive"); + wait_for_handler_acknowledgements( + &mut gate.acknowledgements, + &mut gate.errors, + wave, + context, + ) + .await; + disconnect + .send(true) + .expect("all gated clients remain alive until explicitly disconnected"); + join_client_wave(clients, wave, context).await; + } +} + +/// The streaming-head variant needs no auxiliary checkpoint: receiving a valid +/// response head is itself the lifecycle proof. As above, all 256 second-wave heads +/// can only arrive after every first-wave disconnect result was consumed. +async fn drive_two_stream_disconnect_waves(port: u16, path: &'static str, context: &str) { + for wave in 1..=2 { + let clients = spawn_stream_client_wave(port, path); + join_client_wave(clients, wave, context).await; + } } async fn assert_ping_survives(port: u16, context: &str) { @@ -159,28 +608,57 @@ async fn assert_ping_survives(port: u16, context: &str) { assert_eq!(ping.text().await.unwrap(), "pong"); } -async fn shutdown(port: u16, server: std::thread::JoinHandle<()>) { - let _ = reqwest::Client::new() - .get(format!("http://127.0.0.1:{port}/shutdown")) - .send() - .await; - match tokio::task::spawn_blocking(move || server.join()).await { - Ok(Ok(())) => {} - Ok(Err(panic)) => std::panic::resume_unwind(panic), - Err(e) => panic!("server join task failed: {e}"), +async fn shutdown(port: u16, mut server: ProxyServer) { + let _ = tokio::time::timeout( + Duration::from_secs(10), + reqwest::Client::new() + .get(format!("http://127.0.0.1:{port}/shutdown")) + .send(), + ) + .await; + let graceful = tokio::time::timeout(Duration::from_secs(10), async { + while !server.thread.is_finished() { + tokio::time::sleep(Duration::from_millis(10)).await; + } + }) + .await + .is_ok(); + if !graceful { + let _ = server + .abort + .take() + .expect("proxy abort signal is sent at most once") + .send(()); + tokio::time::timeout(Duration::from_secs(5), async { + while !server.thread.is_finished() { + tokio::time::sleep(Duration::from_millis(10)).await; + } + }) + .await + .expect("proxy server ignored both graceful shutdown and forced test cleanup"); + } + match server.thread.join() { + Ok(()) => {} + Err(panic) => std::panic::resume_unwind(panic), } } #[tokio::test] async fn test_disconnect_before_buffered_respond_does_not_kill_the_loop() { + let _heavy_case = HEAVY_CASE_LOCK.lock().await; + let mut gate = spawn_counting_gate().await; + let mut iterations = spawn_iteration_counter().await; let port = common::free_tcp_port(); - // `/slow` waits, then responds — the client disconnects during the wait, so the - // buffered `respond` send fails (or the pending entry is sibling-pruned). That - // must be a cancellation, not a structural failure. + // `/slow` reaches the counting checkpoint, waits after its release, then + // responds. The test disconnects every client during that wait, so the buffered + // `respond` send fails (or the pending entry is sibling-pruned). That must be a + // cancellation, not a structural failure. let code = format!( r#" listen on port {port} as srv main loop concurrently: + open url at "http://127.0.0.1:{counter_port}/tick" and stream response as iteration + close iteration wait for request comes in on srv as req with timeout 30000 store p as req["path"] check if p is equal to "/ping": @@ -191,22 +669,37 @@ async fn test_disconnect_before_buffered_respond_does_not_kill_the_loop() { close server srv break otherwise: - wait for 500 milliseconds + open url at "http://127.0.0.1:{gate_port}/checkpoint" and stream response as checkpoint + close checkpoint + open url at "http://127.0.0.1:{gate_port}/ack" and stream response as acknowledgement + wait for next chunk from acknowledgement as acknowledged + close acknowledgement + wait for {POST_CHECKPOINT_DELAY_MS} milliseconds respond to req with "late" end check end check end loop - "# + "#, + gate_port = gate.port, + counter_port = iterations.port, ); let server = start_proxy_server(code); wait_for_server(port).await; - let _ = fire_burst(port, "/slow", false).await; + drive_two_gated_disconnect_waves(port, "/slow", &mut gate, "buffered-respond disconnect").await; + wait_for_proven_handler_iterations( + &mut iterations, + POST_DISCONNECT_BARRIER, + "buffered-disconnect", + ) + .await; assert_ping_survives(port, "buffered-respond disconnect").await; shutdown(port, server).await; } #[tokio::test] async fn test_disconnect_before_stream_write_does_not_kill_the_loop() { + let _heavy_case = HEAVY_CASE_LOCK.lock().await; + let mut iterations = spawn_iteration_counter().await; let port = common::free_tcp_port(); // `/stream` sends the head, waits (the client reads the head then disconnects), // then writes — the write send fails. That must be a cancellation, not a failure. @@ -214,6 +707,8 @@ async fn test_disconnect_before_stream_write_does_not_kill_the_loop() { r#" listen on port {port} as srv main loop concurrently: + open url at "http://127.0.0.1:{counter_port}/tick" and stream response as iteration + close iteration wait for request comes in on srv as req with timeout 30000 store p as req["path"] check if p is equal to "/ping": @@ -237,26 +732,38 @@ async fn test_disconnect_before_stream_write_does_not_kill_the_loop() { end check end check end loop - "# + "#, + counter_port = iterations.port, ); let server = start_proxy_server(code); wait_for_server(port).await; - let _ = fire_burst(port, "/stream", true).await; + drive_two_stream_disconnect_waves(port, "/stream", "stream-write disconnect").await; + wait_for_proven_handler_iterations( + &mut iterations, + POST_DISCONNECT_BARRIER, + "stream-write-disconnect", + ) + .await; assert_ping_survives(port, "stream-write disconnect").await; shutdown(port, server).await; } #[tokio::test] async fn test_disconnect_before_streaming_head_does_not_kill_the_loop() { + let _heavy_case = HEAVY_CASE_LOCK.lock().await; + let mut gate = spawn_counting_gate().await; + let mut iterations = spawn_iteration_counter().await; let port = common::free_tcp_port(); // Client disconnects *before* the streaming head is sent (no head read). The - // handler parks, then reaches `start streaming response` with a missing/closed - // pending entry — must be Cancelled, not a structural General that trips the - // breaker after >256 instances. + // handler reaches the counting checkpoint, parks after its release, then reaches + // `start streaming response` with a missing/closed pending entry — must be + // Cancelled, not a structural General that trips the breaker after >256 instances. let code = format!( r#" listen on port {port} as srv main loop concurrently: + open url at "http://127.0.0.1:{counter_port}/tick" and stream response as iteration + close iteration wait for request comes in on srv as req with timeout 30000 store p as req["path"] check if p is equal to "/ping": @@ -267,24 +774,40 @@ async fn test_disconnect_before_streaming_head_does_not_kill_the_loop() { close server srv break otherwise: - wait for 500 milliseconds + open url at "http://127.0.0.1:{gate_port}/checkpoint" and stream response as checkpoint + close checkpoint + open url at "http://127.0.0.1:{gate_port}/ack" and stream response as acknowledgement + wait for next chunk from acknowledgement as acknowledged + close acknowledgement + wait for {POST_CHECKPOINT_DELAY_MS} milliseconds start streaming response to req with status 200 and content type "text/plain" as out write line "late" to out close out end check end check end loop - "# + "#, + gate_port = gate.port, + counter_port = iterations.port, ); let server = start_proxy_server(code); wait_for_server(port).await; - let _ = fire_burst(port, "/prehead", false).await; + drive_two_gated_disconnect_waves(port, "/prehead", &mut gate, "pre-streaming-head disconnect") + .await; + wait_for_proven_handler_iterations( + &mut iterations, + POST_DISCONNECT_BARRIER, + "pre-streaming-head-disconnect", + ) + .await; assert_ping_survives(port, "pre-streaming-head disconnect").await; shutdown(port, server).await; } #[tokio::test] async fn test_repeated_wait_timeouts_do_not_kill_the_loop() { + let _heavy_case = HEAVY_CASE_LOCK.lock().await; + let mut counter = spawn_iteration_counter().await; let port = common::free_tcp_port(); // Finite `wait for request ... with timeout` that repeatedly expires with no // client traffic must not trip the structural breaker. After many idle @@ -293,7 +816,9 @@ async fn test_repeated_wait_timeouts_do_not_kill_the_loop() { r#" listen on port {port} as srv main loop concurrently: - wait for request comes in on srv as req with timeout 50 + open url at "http://127.0.0.1:{counter_port}/tick" and stream response as tick + close tick + wait for request comes in on srv as req with timeout 1 store p as req["path"] check if p is equal to "/ping": respond to req with "pong" @@ -307,14 +832,16 @@ async fn test_repeated_wait_timeouts_do_not_kill_the_loop() { end check end check end loop - "# + "#, + counter_port = counter.port, ); let server = start_proxy_server(code); wait_for_server(port).await; - // Idle long enough for well over 256 consecutive wait timeouts (50ms each; - // concurrency multiplies the rate). 8s >> 256 * structural backoff would - // also have completed if they were misclassified. - tokio::time::sleep(Duration::from_secs(8)).await; + // The loop initially starts 256 handlers. It can start handler 512 only after + // consuming 256 finite request-wait expiries and refilling once more. A buggy + // structural classifier breaks while consuming expiry 256 and tops out at 511, + // so this is an observable threshold proof rather than an elapsed-time proxy. + wait_for_proven_handler_iterations(&mut counter, DISCONNECT_TOTAL, "finite-timeout").await; assert_ping_survives(port, "repeated wait timeouts").await; shutdown(port, server).await; } diff --git a/tests/dropped_interpret_server_cleanup_test.rs b/tests/dropped_interpret_server_cleanup_test.rs index 7106e60e..34b77276 100644 --- a/tests/dropped_interpret_server_cleanup_test.rs +++ b/tests/dropped_interpret_server_cleanup_test.rs @@ -1,17 +1,18 @@ //! Real-socket regression (maintainer re-review, P1): when the `interpret()` future -//! is DROPPED after a handler started a streaming response, the interpret-scoped +//! is DROPPED after a handler starts a streaming response, the interpret-scoped //! cleanup guard must close the server response stream (ending the client's body) //! and 500 any unanswered request — even though the reusable `Interpreter` itself //! stays alive. //! -//! Previously the drop guard covered only outbound streams, so a dropped run left -//! `server_response_streams` alive on the still-alive interpreter and the client -//! hung. To prove it is the GUARD (not the interpreter's eventual drop) that closes -//! the body, the interpreter is held alive for well after the future is dropped: the -//! client body must end shortly after the drop, not when the interpreter drops. +//! The tests use explicit drop/release channels. They first observe the exact +//! lifecycle checkpoint, drop only the `interpret()` future, assert the client +//! outcome while the interpreter remains alive, and release the interpreter +//! immediately afterward. No fixed multi-second sleep stands in for dequeue or +//! cleanup completion. use std::sync::Arc; -use std::time::{Duration, Instant}; +use std::time::Duration; +use tokio::io::AsyncReadExt; use wfl::Interpreter; use wfl::config::WflConfig; use wfl::lexer::lex_wfl_with_positions; @@ -30,30 +31,20 @@ async fn wait_for_server(port: u16) { panic!("server on {addr} did not become ready"); } -#[tokio::test] -async fn test_dropped_run_closes_server_stream_while_interpreter_stays_alive() { - let port = common::free_tcp_port(); - - // Handler: start streaming, send a chunk, flush, then park for a long time. The - // run is dropped while it is parked here, before it ever closes the stream. - let code = format!( - r#" - listen on port {port} as srv - main loop: - wait for request comes in on srv as req with timeout 60000 - start streaming response to req with status 200 and content type "text/plain" as out - write chunk "hello" to out - flush out - wait for 60000 milliseconds - end loop - "# - ); +struct ControlledServer { + thread: std::thread::JoinHandle<()>, + drop_run: Option>, + dropped: tokio::sync::oneshot::Receiver<()>, + release_interpreter: Option>, +} - // The server runs in its own thread. It runs `interpret()` under a 3s timeout — - // when that elapses the FUTURE is dropped (moved into `timeout`) — then holds - // the interpreter ALIVE for 10 more seconds. So during [3s, 13s] the future is - // gone but the interpreter lives: only the drop guard can end the client body. - let server = std::thread::spawn(move || { +/// Run `interpret()` until the test explicitly drops that future, then keep the +/// same interpreter alive until the client-side assertions finish. +fn start_controlled_server(code: String) -> ControlledServer { + let (drop_run, drop_rx) = tokio::sync::oneshot::channel(); + let (dropped_tx, dropped) = tokio::sync::oneshot::channel(); + let (release_interpreter, release_rx) = tokio::sync::oneshot::channel(); + let thread = std::thread::spawn(move || { let rt = tokio::runtime::Builder::new_current_thread() .enable_all() .build() @@ -67,136 +58,270 @@ async fn test_dropped_run_closes_server_stream_while_interpreter_stays_alive() { }; let mut interp = Interpreter::with_config(Arc::new(config)); { - let fut = interp.interpret(&program); - let _ = tokio::time::timeout(Duration::from_secs(3), fut).await; - // `fut` is dropped here (timeout took ownership and elapsed). + let future = interp.interpret(&program); + tokio::pin!(future); + tokio::select! { + result = &mut future => { + panic!( + "interpret() returned before the test requested its drop: {result:?}" + ); + } + signal = drop_rx => { + signal.expect("drop-run controller disappeared"); + } + } } - // Interpreter deliberately kept alive well past the drop. - tokio::time::sleep(Duration::from_secs(10)).await; + // The pinned future and its cleanup guard have now been dropped, while + // the reusable interpreter remains alive below. + let _ = dropped_tx.send(()); + // Dropping the sender during a test panic releases this immediately; + // the timeout is only a final leak backstop. + let _ = tokio::time::timeout(Duration::from_secs(10), release_rx).await; drop(interp); }); }); - wait_for_server(port).await; + ControlledServer { + thread, + drop_run: Some(drop_run), + dropped, + release_interpreter: Some(release_interpreter), + } +} - // Client: read the streaming body. It must deliver "hello" and then END shortly - // after the 3s drop (the guard closing the stream), NOT hang until the 13s - // interpreter drop or the 60s handler park. - let mut resp = reqwest::Client::new() - .get(format!("http://127.0.0.1:{port}/")) - .send() +async fn request_run_drop(server: &mut ControlledServer) { + server + .drop_run + .take() + .expect("drop signal is sent exactly once") + .send(()) + .expect("controlled server still waits for the drop signal"); + tokio::time::timeout(Duration::from_secs(3), &mut server.dropped) .await - .expect("request send"); - assert_eq!(resp.status().as_u16(), 200); + .expect("interpret() future was not dropped promptly") + .expect("controlled server ended before reporting the drop"); +} + +async fn finish_controlled_server(mut server: ControlledServer) { + let _ = server + .release_interpreter + .take() + .expect("interpreter release is sent exactly once") + .send(()); + tokio::time::timeout(Duration::from_secs(12), async { + while !server.thread.is_finished() { + tokio::time::sleep(Duration::from_millis(10)).await; + } + }) + .await + .expect("controlled server did not stop within 12 seconds"); + if let Err(panic) = server.thread.join() { + std::panic::resume_unwind(panic); + } +} + +/// A handler calls this endpoint only after it dequeues the real client request. +/// The mock reports that exact checkpoint and withholds its response, keeping the +/// handler's pending response parked until the test drops `interpret()`. +async fn spawn_dequeue_checkpoint() -> (u16, tokio::sync::oneshot::Receiver>) { + let listener = tokio::net::TcpListener::bind("127.0.0.1:0") + .await + .expect("bind dequeue checkpoint"); + let port = listener + .local_addr() + .expect("dequeue checkpoint address") + .port(); + let (ready_tx, ready_rx) = tokio::sync::oneshot::channel(); + tokio::spawn(async move { + let setup = async { + let (mut socket, _) = tokio::time::timeout(Duration::from_secs(5), listener.accept()) + .await + .map_err(|_| "timed out waiting for dequeue checkpoint".to_string())? + .map_err(|error| format!("accept dequeue checkpoint: {error}"))?; + let deadline = tokio::time::Instant::now() + Duration::from_secs(5); + let mut head = Vec::new(); + let mut buffer = [0u8; 512]; + loop { + let count = tokio::time::timeout_at(deadline, socket.read(&mut buffer)) + .await + .map_err(|_| "timed out reading dequeue checkpoint".to_string())? + .map_err(|error| format!("read dequeue checkpoint: {error}"))?; + if count == 0 { + return Err("handler closed before sending the dequeue checkpoint".to_string()); + } + head.extend_from_slice(&buffer[..count]); + if head.windows(4).any(|window| window == b"\r\n\r\n") { + break; + } + if head.len() > 16 * 1024 { + return Err("dequeue checkpoint head exceeded 16 KiB".to_string()); + } + } + let request = String::from_utf8_lossy(&head); + if !request.starts_with("GET /dequeued ") { + return Err(format!( + "unexpected dequeue checkpoint request: {:?}", + request.lines().next() + )); + } + Ok(socket) + } + .await; + + match setup { + Ok(mut socket) => { + let _ = ready_tx.send(Ok(())); + // Keep the outbound request parked. Dropping `interpret()` cancels + // it and closes this socket; no HTTP response is intentionally sent. + let _ = tokio::time::timeout(Duration::from_secs(10), async { + let mut byte = [0u8; 1]; + loop { + match socket.read(&mut byte).await { + Ok(0) | Err(_) => break, + Ok(_) => {} + } + } + }) + .await; + } + Err(error) => { + let _ = ready_tx.send(Err(error)); + } + } + }); + (port, ready_rx) +} + +#[tokio::test] +async fn test_dropped_run_closes_server_stream_while_interpreter_stays_alive() { + let port = common::free_tcp_port(); + + // Handler: start streaming, send a chunk, flush, then park. Receiving `hello` + // is the exact checkpoint that the server stream exists before the run is + // explicitly dropped. + let code = format!( + r#" + listen on port {port} as srv + main loop: + wait for request comes in on srv as req with timeout 60000 + start streaming response to req with status 200 and content type "text/plain" as out + write chunk "hello" to out + flush out + wait for 60000 milliseconds + end loop + "# + ); + + let mut server = start_controlled_server(code); + wait_for_server(port).await; + + let mut response = tokio::time::timeout( + Duration::from_secs(5), + reqwest::Client::new() + .get(format!("http://127.0.0.1:{port}/")) + .send(), + ) + .await + .expect("timed out waiting for streaming response") + .expect("streaming request failed"); + assert_eq!(response.status().as_u16(), 200); - let start = Instant::now(); let mut body = Vec::new(); - let mut ended = false; - loop { - match tokio::time::timeout(Duration::from_secs(8), resp.chunk()).await { - Ok(Ok(Some(bytes))) => body.extend_from_slice(&bytes), - // Clean EOF or a transport end both mean the body finished. - Ok(Ok(None)) | Ok(Err(_)) => { - ended = true; - break; + tokio::time::timeout(Duration::from_secs(5), async { + while !String::from_utf8_lossy(&body).contains("hello") { + match response.chunk().await { + Ok(Some(bytes)) => body.extend_from_slice(&bytes), + Ok(None) => panic!("stream ended before the pre-drop chunk arrived"), + Err(error) => panic!("stream failed before the run was dropped: {error}"), } - Err(_) => break, // outer timeout: body never ended } - } - let elapsed = start.elapsed(); + }) + .await + .expect("timed out waiting for the pre-drop streamed chunk"); + + request_run_drop(&mut server).await; + // The interpreter is deliberately still alive here. Only the run's cleanup + // guard can close the body, and it must be a clean end rather than a transport + // failure accepted as equivalent. + tokio::time::timeout(Duration::from_secs(3), async { + loop { + match response.chunk().await { + Ok(Some(bytes)) => body.extend_from_slice(&bytes), + Ok(None) => break, + Err(error) => { + panic!("drop-guard stream cleanup must end with clean EOF: {error}") + } + } + } + }) + .await + .expect("client body did not end while the interpreter was kept alive"); assert!( String::from_utf8_lossy(&body).contains("hello"), - "the streamed chunk should have been delivered before the drop; body: {:?}", + "the pre-drop streamed chunk disappeared: {:?}", String::from_utf8_lossy(&body) ); - assert!( - ended, - "the client body must END after the run was dropped (the guard closing the \ - server response stream), not hang" - ); - assert!( - elapsed < Duration::from_secs(7), - "the body should end shortly after the ~3s drop (the guard), not at the ~13s \ - interpreter drop; took {elapsed:?}" - ); - match tokio::task::spawn_blocking(move || server.join()).await { - Ok(Ok(())) => {} - Ok(Err(panic)) => std::panic::resume_unwind(panic), - Err(e) => panic!("server join task failed: {e}"), - } + finish_controlled_server(server).await; } #[tokio::test] async fn test_dropped_run_answers_pending_request_with_500() { - // Exercise the dropped-run pending-request 500 branch: handler dequeues a - // request and parks WITHOUT ever responding or starting a streaming response, - // so the pending oneshot is still in `pending_responses` when interpret() is - // dropped. The cleanup guard must answer 500 promptly (issue #642 R3). + let (checkpoint_port, checkpoint_ready) = spawn_dequeue_checkpoint().await; let port = common::free_tcp_port(); let code = format!( r#" listen on port {port} as srv main loop: wait for request comes in on srv as req with timeout 60000 + open url at "http://127.0.0.1:{checkpoint_port}/dequeued" and stream response as checkpoint wait for 60000 milliseconds end loop "# ); - let server = std::thread::spawn(move || { - let rt = tokio::runtime::Builder::new_current_thread() - .enable_all() - .build() - .expect("server runtime"); - rt.block_on(async { - let tokens = lex_wfl_with_positions(&code); - let program = Parser::new(&tokens).parse().expect("parse"); - let config = WflConfig { - timeout_seconds: 60, - ..WflConfig::default() - }; - let mut interp = Interpreter::with_config(Arc::new(config)); - { - let fut = interp.interpret(&program); - // Drop after the handler has had time to dequeue and park. - let _ = tokio::time::timeout(Duration::from_secs(2), fut).await; - } - // Keep the interpreter alive so only the drop guard can 500 the request. - tokio::time::sleep(Duration::from_secs(8)).await; - drop(interp); - }); - }); - + let mut server = start_controlled_server(code); wait_for_server(port).await; - let start = Instant::now(); - let resp = tokio::time::timeout( - Duration::from_secs(6), - reqwest::Client::new() - .get(format!("http://127.0.0.1:{port}/")) - .send(), - ) - .await - .expect("client should not hang waiting for a pending request after interpret() drop") - .expect("request failed"); - let elapsed = start.elapsed(); + let url = format!("http://127.0.0.1:{port}/"); + let request = tokio::spawn(async move { reqwest::Client::new().get(url).send().await }); + tokio::time::timeout(Duration::from_secs(5), checkpoint_ready) + .await + .expect("handler did not reach the post-dequeue checkpoint") + .expect("dequeue checkpoint task ended without a result") + .expect("dequeue checkpoint failed"); + + request_run_drop(&mut server).await; + let response = tokio::time::timeout(Duration::from_secs(5), request) + .await + .expect("client hung waiting for the dropped-run 500") + .expect("pending request task panicked") + .expect("pending request failed"); assert_eq!( - resp.status().as_u16(), + response.status().as_u16(), 500, "dropped run must answer the still-pending request with 500, got {}", - resp.status() + response.status() ); - assert!( - elapsed < Duration::from_secs(5), - "500 should arrive shortly after the ~2s drop, not at the 60s request timeout; took {elapsed:?}" + assert_eq!( + response + .headers() + .get(reqwest::header::CONTENT_TYPE) + .and_then(|value| value.to_str().ok()), + Some("text/plain; charset=utf-8"), + "dropped-run cleanup must use the explicit plain-text 500 response" + ); + let body = tokio::time::timeout(Duration::from_secs(3), response.bytes()) + .await + .expect("timed out reading dropped-run cleanup response body") + .expect("read dropped-run cleanup response body"); + assert_eq!( + body.as_ref(), + b"Internal Server Error\n", + "dropped-run cleanup must resolve the pending request with its exact 500 body" ); - match tokio::task::spawn_blocking(move || server.join()).await { - Ok(Ok(())) => {} - Ok(Err(panic)) => std::panic::resume_unwind(panic), - Err(e) => panic!("server join task failed: {e}"), - } + finish_controlled_server(server).await; } diff --git a/tests/execute_file_test.rs b/tests/execute_file_test.rs index 20cb1afb..4e279628 100644 --- a/tests/execute_file_test.rs +++ b/tests/execute_file_test.rs @@ -4,6 +4,8 @@ // Executes another WFL file in-process with a nested interpreter, optionally // passing HTTP request context and capturing the child's display/print output. +mod common; + use std::collections::HashMap; use std::fs; use std::sync::Arc; @@ -497,7 +499,7 @@ async fn test_web_server_serves_executed_wfl_page() { ) .expect("Failed to write page file"); - let port: u16 = 58123; + let port = common::free_tcp_port(); let server_file = temp_dir.path().join("server.wfl"); let server_code = format!( concat!( diff --git a/tests/file_io_performance_test.rs b/tests/file_io_performance_test.rs index 57db989b..6b7289e4 100644 --- a/tests/file_io_performance_test.rs +++ b/tests/file_io_performance_test.rs @@ -200,27 +200,31 @@ mod file_io_performance_tests { #[tokio::test] async fn test_directory_listing_performance() { - let test_files: Vec = (0..30).map(|i| format!("dir_perf_{}.txt", i)).collect(); - let test_file_refs: Vec<&str> = test_files.iter().map(|s| s.as_str()).collect(); - cleanup_test_files(&test_file_refs); + let test_dir = tempfile::tempdir().expect("Failed to create directory-listing fixture"); + let test_files: Vec<_> = (0..30) + .map(|i| test_dir.path().join(format!("dir_perf_{}.txt", i))) + .collect(); // Create multiple files for directory listing for (i, file) in test_files.iter().enumerate() { fs::write(file, format!("Content for file {}", i)).expect("Failed to create test file"); } - let code = r#" + let fixture_path = test_dir.path().to_string_lossy().replace('\\', "/"); + let code = format!( + r#" // Test directory listing performance - wait for store all_files as list files in "." - wait for store txt_files as list files in "." with pattern "dir_perf_*.txt" - wait for store recursive_files as list files recursively in "." + wait for store all_files as list files in "{fixture_path}" + wait for store txt_files as list files in "{fixture_path}" with pattern "dir_perf_*.txt" + wait for store recursive_files as list files recursively in "{fixture_path}" display "Listed all files: " with length of all_files display "Listed TXT files: " with length of txt_files display "Listed recursive files: " with length of recursive_files - "#; + "# + ); - let result = execute_wfl_code_with_timing(code).await; + let result = execute_wfl_code_with_timing(&code).await; assert!( result.is_ok(), "Directory listing performance test failed: {:?}", @@ -234,8 +238,6 @@ mod file_io_performance_tests { "Directory listing took too long: {:?}", elapsed ); - - cleanup_test_files(&test_file_refs); } #[tokio::test] diff --git a/tests/flush_action_backcompat_test.rs b/tests/flush_action_backcompat_test.rs index 94ec650a..221499dc 100644 --- a/tests/flush_action_backcompat_test.rs +++ b/tests/flush_action_backcompat_test.rs @@ -10,6 +10,9 @@ use std::fs; use std::process::Command; use tempfile::TempDir; +use wfl::lexer::lex_wfl_with_positions; +use wfl::parser::Parser; +use wfl::typechecker::TypeChecker; fn run_src(src: &str) -> (String, Option) { let dir = TempDir::new().expect("tempdir"); @@ -154,3 +157,152 @@ fn flush_with_postfix_uses_legacy_expression_when_bound() { ); assert!(out.contains("OK"), "expected OK; output:\n{out}"); } + +fn typecheck_src(src: &str) -> Result<(), String> { + let tokens = lex_wfl_with_positions(src); + let program = Parser::new(&tokens).parse().expect("parse"); + TypeChecker::new() + .check_types(&program) + .map_err(|errors| format!("{errors:?}")) +} + +#[test] +fn flush_with_nested_postfix_uses_the_recursive_legacy_root() { + let src = "store flush cache as [[\"a\"]]\n\ + flush cache[0][0]\n\ + display \"OK\"\n"; + let (out, code) = run_src(src); + assert_eq!( + code, + Some(0), + "nested legacy postfix must resolve the full-name root; output:\n{out}" + ); + assert!(out.contains("OK"), "expected OK; output:\n{out}"); +} + +#[test] +fn flush_with_binary_expression_uses_the_legacy_binding() { + let src = "store flush cache as 1\n\ + flush cache plus 1\n\ + display \"OK\"\n"; + let (out, code) = run_src(src); + assert_eq!( + code, + Some(0), + "binary legacy expression must resolve the full-name root; output:\n{out}" + ); + assert!(out.contains("OK"), "expected OK; output:\n{out}"); +} + +#[test] +fn flush_with_of_call_uses_the_full_legacy_action_name() { + let src = "define action called flush cache with parameters value:\n\ + \x20\x20\x20\x20display value\n\ + end action\n\ + flush cache of 7\n"; + let (out, code) = run_src(src); + assert_eq!( + code, + Some(0), + "an `of` continuation must call the full legacy action name; output:\n{out}" + ); + assert!( + out.contains('7'), + "the full-name action should receive its argument; output:\n{out}" + ); +} + +#[test] +fn flush_split_rewrite_keeps_the_original_legacy_binding() { + let src = "store flush cache as 1\n\ + flush cache split \"a,b\" by \",\"\n\ + display \"OK\"\n"; + let (out, code) = run_src(src); + assert_eq!( + code, + Some(0), + "split rewrite must still select the bound legacy expression; output:\n{out}" + ); + assert!(out.contains("OK"), "expected OK; output:\n{out}"); +} + +#[test] +fn flush_explicit_find_in_rewrite_keeps_the_original_legacy_binding() { + let src = "create pattern letter_a:\n\ + \x20\x20\x20\x20\"a\"\n\ + end pattern\n\ + store flush cache as 1\n\ + flush cache find letter_a in \"abc\"\n\ + display \"OK\"\n"; + let (out, code) = run_src(src); + assert_eq!( + code, + Some(0), + "explicit find-in rewrite must select the bound legacy expression; output:\n{out}" + ); + assert!(out.contains("OK"), "expected OK; output:\n{out}"); +} + +#[test] +fn flush_explicit_replace_in_rewrite_keeps_the_original_legacy_binding() { + let src = "create pattern letter_a:\n\ + \x20\x20\x20\x20\"a\"\n\ + end pattern\n\ + store flush cache as 1\n\ + flush cache replace letter_a with \"z\" in \"abc\"\n\ + display \"OK\"\n"; + let (out, code) = run_src(src); + assert_eq!( + code, + Some(0), + "explicit replace-in rewrite must select the bound legacy expression; output:\n{out}" + ); + assert!(out.contains("OK"), "expected OK; output:\n{out}"); +} + +#[test] +fn flush_direct_container_property_uses_the_legacy_expression_branch() { + let src = "create container Cache:\n\ + \x20\x20\x20\x20property flush cache: Number\n\ + \x20\x20\x20\x20action inspect:\n\ + \x20\x20\x20\x20\x20\x20\x20\x20flush cache plus 1\n\ + \x20\x20\x20\x20end\n\ + end"; + assert!( + typecheck_src(src).is_ok(), + "a direct container property must select the legacy branch: {:?}", + typecheck_src(src).err() + ); +} + +#[test] +fn flush_inherited_container_property_uses_the_legacy_expression_branch() { + let src = "create container Base:\n\ + \x20\x20\x20\x20property flush cache: Number\n\ + end\n\ + create container Child extends Base:\n\ + \x20\x20\x20\x20action inspect:\n\ + \x20\x20\x20\x20\x20\x20\x20\x20flush cache plus 1\n\ + \x20\x20\x20\x20end\n\ + end"; + assert!( + typecheck_src(src).is_ok(), + "an inherited container property must select the legacy branch: {:?}", + typecheck_src(src).err() + ); +} + +#[test] +fn flush_invalid_container_property_expression_is_statically_rejected() { + let src = "create container Cache:\n\ + \x20\x20\x20\x20property flush cache: Text\n\ + \x20\x20\x20\x20action inspect:\n\ + \x20\x20\x20\x20\x20\x20\x20\x20flush cache minus 1\n\ + \x20\x20\x20\x20end\n\ + end"; + let errors = typecheck_src(src).expect_err("Text minus Number must be rejected"); + assert!( + !errors.contains("`flush` requires a response-stream handle"), + "the bound property must be checked as the legacy expression, got: {errors}" + ); +} diff --git a/tests/http_stream_test.rs b/tests/http_stream_test.rs index 6e67d486..cf3e2f11 100644 --- a/tests/http_stream_test.rs +++ b/tests/http_stream_test.rs @@ -219,6 +219,12 @@ async fn test_next_line_returns_final_unterminated_line() { wait for next line from up as a wait for next line from up as b wait for next line from up as c + store closed_after_eof as no + try: + wait for next line from up as d + catch: + store closed_after_eof as yes + end try "# ); let interpreter = run_wfl(&code).await; @@ -228,6 +234,10 @@ async fn test_next_line_returns_final_unterminated_line() { Value::Null => {} other => panic!("Expected nothing at EOF, got {other:?}"), } + assert!( + matches!(get_var(&interpreter, "closed_after_eof"), Value::Bool(true)), + "the one EOF result must consume the exhausted handle; a later read is catchably closed" + ); } #[tokio::test] diff --git a/tests/outbound_stream_open_expiry_test.rs b/tests/outbound_stream_open_expiry_test.rs index 99266ead..eb787b2a 100644 --- a/tests/outbound_stream_open_expiry_test.rs +++ b/tests/outbound_stream_open_expiry_test.rs @@ -120,3 +120,40 @@ wait for 6000 milliseconds"# Err(e) => panic!("client join task failed: {e}"), } } + +#[tokio::test] +async fn test_read_after_unread_stream_expiry_reports_typed_timeout() { + let (port, upstream_closed) = spawn_head_then_stall_upstream().await; + let code = format!( + r#"open url at "http://127.0.0.1:{port}/" and stream response as s +wait for 1200 milliseconds +wait for next chunk from s as chunk"# + ); + + let tokens = lex_wfl_with_positions(&code); + let program = Parser::new(&tokens).parse().expect("parse"); + let config = WflConfig { + timeout_seconds: 30, + outbound_stream_max_seconds: 1, + ..WflConfig::default() + }; + let mut interp = Interpreter::with_config(Arc::new(config)); + let errors = interp + .interpret(&program) + .await + .expect_err("reading a stream after its hard deadline must fail"); + let message = format!("{errors:?}"); + + assert!( + message.contains("kind: Timeout"), + "expired stream must preserve ErrorKind::Timeout, got: {message}" + ); + assert!( + !message.to_lowercase().contains("unknown or already-closed"), + "expired stream must not degrade to an unknown-handle error: {message}" + ); + tokio::time::timeout(Duration::from_secs(2), upstream_closed) + .await + .expect("expired unread stream should drop its upstream") + .expect("upstream close sender dropped"); +} diff --git a/tests/outbound_stream_reaper_race_test.rs b/tests/outbound_stream_reaper_race_test.rs index 99c04cd1..10498c8a 100644 --- a/tests/outbound_stream_reaper_race_test.rs +++ b/tests/outbound_stream_reaper_race_test.rs @@ -89,6 +89,10 @@ async fn test_active_read_near_deadline_surfaces_timeout_and_drops_upstream() { .expect("upstream was not dropped near the absolute cap during an active read") .expect("upstream close sender dropped"); let elapsed = start.elapsed(); + assert!( + elapsed >= Duration::from_millis(700), + "the 1s hard lifetime must not fire as an unrelated immediate error; took {elapsed:?}" + ); assert!( elapsed < Duration::from_secs(3), "upstream should drop near the 1s absolute cap, not the 30s idle timeout; took {elapsed:?}" @@ -105,22 +109,21 @@ async fn test_active_read_near_deadline_surfaces_timeout_and_drops_upstream() { let msg = result.expect_err("active read past absolute cap must fail (Timeout), not succeed"); assert!( - msg.to_lowercase().contains("timeout") - || msg.contains("Timeout") - || msg.contains("outbound"), - "expected a typed Timeout-class error, got: {msg}" + msg.contains("kind: Timeout"), + "absolute-cap cancellation must preserve ErrorKind::Timeout, got: {msg}" ); assert!( - !msg.to_lowercase().contains("unknown or already-closed"), - "expired slot must surface Timeout, not 'unknown/already-closed'; got: {msg}" + !msg.to_lowercase().contains("closed"), + "expired slot must surface Timeout, not a closed-stream error; got: {msg}" ); } #[tokio::test] -async fn test_rapid_open_close_does_not_leak_reaper_tasks() { +async fn test_rapid_open_close_completes_against_stalled_upstreams() { // Open and immediately close many outbound streams against a real (stalling) - // upstream. Each close must abort its reaper timer so resource usage stays - // bounded (not request-rate × cap sleeping tasks). + // upstream. This real-socket smoke test proves close does not wait for the hard + // lifetime. Sleeping-task accounting itself is asserted by the retained-runtime + // unit test in `interpreter::tests`, where runtime shutdown cannot hide a leak. let listener = tokio::net::TcpListener::bind("127.0.0.1:0") .await .expect("bind"); @@ -149,8 +152,8 @@ async fn test_rapid_open_close_does_not_leak_reaper_tasks() { } }); - // Cap is large (60s) so a leaked reaper would still be parked after the program - // ends if timers were not aborted on close. We open/close 40 streams quickly. + // Cap is large (60s); explicit close must still complete promptly for all 40 + // sequential real upstreams. let mut lines = String::new(); for i in 0..40 { lines.push_str(&format!( @@ -178,15 +181,20 @@ async fn test_rapid_open_close_does_not_leak_reaper_tasks() { .expect("rapid open/close must succeed"); }); }); - match tokio::task::spawn_blocking(move || client.join()).await { - Ok(Ok(())) => {} - Ok(Err(panic)) => std::panic::resume_unwind(panic), - Err(e) => panic!("client join failed: {e}"), + tokio::time::timeout(Duration::from_secs(20), async { + while !client.is_finished() { + tokio::time::sleep(Duration::from_millis(10)).await; + } + }) + .await + .expect("rapid open/close client did not finish within 20 seconds"); + if let Err(panic) = client.join() { + std::panic::resume_unwind(panic); } let elapsed = start.elapsed(); // Should finish in well under the 60s cap (and under a few seconds of network). assert!( elapsed < Duration::from_secs(20), - "rapid open/close should finish promptly with reapers aborted; took {elapsed:?}" + "rapid open/close should finish without waiting for the 60s hard cap; took {elapsed:?}" ); } diff --git a/tests/response_stream_backpressure_test.rs b/tests/response_stream_backpressure_test.rs index 38ad525b..08cb7302 100644 --- a/tests/response_stream_backpressure_test.rs +++ b/tests/response_stream_backpressure_test.rs @@ -10,9 +10,10 @@ use std::sync::Arc; use std::time::{Duration, Instant}; -use tokio::io::AsyncWriteExt; +use tokio::io::{AsyncReadExt, AsyncWriteExt}; use wfl::Interpreter; use wfl::config::WflConfig; +use wfl::interpreter::error::ErrorKind; use wfl::lexer::lex_wfl_with_positions; use wfl::parser::Parser; @@ -29,6 +30,40 @@ async fn wait_for_server(port: u16) { panic!("server on {addr} did not become ready"); } +async fn read_response_head(sock: &mut tokio::net::TcpStream) -> Vec { + let deadline = tokio::time::Instant::now() + Duration::from_secs(5); + let mut head = Vec::new(); + let mut buf = [0u8; 512]; + loop { + let n = tokio::time::timeout_at(deadline, sock.read(&mut buf)) + .await + .expect("timed out waiting for streaming response head") + .expect("failed to read streaming response head"); + assert_ne!(n, 0, "connection closed before the complete response head"); + head.extend_from_slice(&buf[..n]); + if head.windows(4).any(|window| window == b"\r\n\r\n") { + return head; + } + assert!( + head.len() <= 16 * 1024, + "streaming response head exceeded 16 KiB" + ); + } +} + +async fn join_server(server: std::thread::JoinHandle<()>) { + tokio::time::timeout(Duration::from_secs(10), async { + while !server.is_finished() { + tokio::time::sleep(Duration::from_millis(10)).await; + } + }) + .await + .expect("server thread did not stop within 10 seconds"); + if let Err(panic) = server.join() { + std::panic::resume_unwind(panic); + } +} + #[tokio::test] async fn test_backpressured_write_to_a_non_reading_client_is_bounded() { let port = common::free_tcp_port(); @@ -60,8 +95,11 @@ async fn test_backpressured_write_to_a_non_reading_client_is_bounded() { "# ); - // Summary is Send (string), unlike Value/RuntimeError. - let (done_tx, done_rx) = tokio::sync::oneshot::channel::<(Duration, Result<(), String>)>(); + // Reduce the result to Send error fields before crossing the server-thread + // boundary. Keeping the typed kind separate prevents a broad text assertion + // from accepting an unrelated cancellation or pre-head failure. + let (done_tx, done_rx) = + tokio::sync::oneshot::channel::<(Duration, Result<(), Vec<(ErrorKind, String)>>)>(); let server = std::thread::spawn(move || { let rt = tokio::runtime::Builder::new_current_thread() .enable_all() @@ -81,7 +119,10 @@ async fn test_backpressured_write_to_a_non_reading_client_is_bounded() { let result = interp.interpret(&program).await; let summary = match result { Ok(_) => Ok(()), - Err(errs) => Err(format!("{errs:?}")), + Err(errs) => Err(errs + .into_iter() + .map(|error| (error.kind, error.message)) + .collect()), }; let _ = done_tx.send((start.elapsed(), summary)); }); @@ -89,40 +130,61 @@ async fn test_backpressured_write_to_a_non_reading_client_is_bounded() { wait_for_server(port).await; - // Connect, send the request, then NEVER read. Hold the socket open so the write - // stalls on backpressure rather than a disconnect. + // Connect and confirm the 200 streaming head first. Only then stop reading and + // hold the socket open. This excludes a pending-request 504, a pre-head + // cancellation, or any other early exit from false-greening the stall test. let mut sock = tokio::net::TcpStream::connect(("127.0.0.1", port)) .await .expect("connect"); sock.write_all(b"GET / HTTP/1.1\r\nHost: 127.0.0.1\r\n\r\n") .await .expect("send request"); - sock.flush().await.ok(); - + sock.flush().await.expect("flush request"); + let head = read_response_head(&mut sock).await; + let head_text = String::from_utf8_lossy(&head); + assert!( + head_text.starts_with("HTTP/1.1 200"), + "expected a successful streaming response head, got {head_text:?}" + ); + assert!( + head_text + .to_ascii_lowercase() + .contains("content-type: text/plain"), + "expected the streaming content type in the response head, got {head_text:?}" + ); // `interpret()` must return once the stalled write times out (~2s). If the write // were unbounded it would pin the handler and this never fires. let (elapsed, summary) = tokio::time::timeout(Duration::from_secs(12), done_rx) .await .expect("interpret() never returned — the backpressured write pinned the handler forever") .expect("done sender dropped"); - // Must fail with a write-timeout / cancelled class error — not succeed, and not - // exit for an unrelated reason (issue #642 R3: previously discarded interpret()). - let err = summary.expect_err( + // Exact typed contract: only the bounded backpressure branch is acceptable. + // A Cancelled disconnect, a generic write error, or any other Timeout is not. + let errors = summary.expect_err( "backpressured write to a non-reading client must error (write timeout), not succeed", ); - let err_l = err.to_lowercase(); - assert!( - err_l.contains("timeout") - || err_l.contains("stopped reading") - || err_l.contains("cancelled") - || err_l.contains("write"), - "expected a write-timeout/stall error, got: {err}" + assert_eq!( + errors.len(), + 1, + "expected exactly one backpressure error, got {errors:?}" + ); + assert_eq!( + errors[0].0, + ErrorKind::Timeout, + "backpressured connected client must produce ErrorKind::Timeout, got {errors:?}" ); - // Lower bound: the 2s response timeout must actually be waited out (not an - // immediate unrelated exit that would false-green the test). + assert_eq!( + errors[0].1, + "Cannot write to response stream: the client stopped reading (write timed out)", + "backpressure must report the exact stalled-client diagnostic" + ); + // The exact kind/message above identifies the backpressure timeout branch. + // Measure its lower bound from interpreter start: measuring from when the test + // thread happens to observe the head can false-fail if the server had already + // started the write timer before that observation. assert!( elapsed >= Duration::from_millis(1500), - "stall should wait for ~2s web_server_response_timeout_seconds; took only {elapsed:?}" + "the configured 2s write timeout fired implausibly early; took only {elapsed:?}" ); assert!( elapsed < Duration::from_secs(9), @@ -131,11 +193,7 @@ async fn test_backpressured_write_to_a_non_reading_client_is_bounded() { ); drop(sock); // keep the client connected until the assertion above - match tokio::task::spawn_blocking(move || server.join()).await { - Ok(Ok(())) => {} - Ok(Err(panic)) => std::panic::resume_unwind(panic), - Err(e) => panic!("server join task failed: {e}"), - } + join_server(server).await; } #[tokio::test] @@ -161,13 +219,22 @@ async fn test_early_chunk_is_visible_before_the_body_completes() { "# ); + let (done_tx, done_rx) = + tokio::sync::oneshot::channel::>>(); let server = std::thread::spawn(move || { let rt = tokio::runtime::Runtime::new().expect("server runtime"); rt.block_on(async { let tokens = lex_wfl_with_positions(&code); let program = Parser::new(&tokens).parse().expect("parse"); let mut interp = Interpreter::new(); - let _ = interp.interpret(&program).await; + let result = match interp.interpret(&program).await { + Ok(_) => Ok(()), + Err(errors) => Err(errors + .into_iter() + .map(|error| (error.kind, error.message)) + .collect()), + }; + let _ = done_tx.send(result); }); }); @@ -184,21 +251,27 @@ async fn test_early_chunk_is_visible_before_the_body_completes() { let mut early_at = None; let mut late_at = None; let mut acc = String::new(); - loop { - match tokio::time::timeout(Duration::from_secs(6), resp.chunk()).await { - Ok(Ok(Some(bytes))) => { - acc.push_str(&String::from_utf8_lossy(&bytes)); - if early_at.is_none() && acc.contains("EARLY") { - early_at = Some(start.elapsed()); + tokio::time::timeout(Duration::from_secs(6), async { + loop { + match resp.chunk().await { + Ok(Some(bytes)) => { + acc.push_str(&String::from_utf8_lossy(&bytes)); + if early_at.is_none() && acc.contains("EARLY") { + early_at = Some(start.elapsed()); + } + if late_at.is_none() && acc.contains("LATE") { + late_at = Some(start.elapsed()); + } } - if late_at.is_none() && acc.contains("LATE") { - late_at = Some(start.elapsed()); + Ok(None) => break, + Err(error) => { + panic!("stream transport failed instead of ending cleanly: {error}") } } - Ok(Ok(None)) | Ok(Err(_)) => break, - Err(_) => panic!("streaming body stalled"), } - } + }) + .await + .expect("streaming body stalled"); let early = early_at.expect("the EARLY chunk was never received"); let late = late_at.expect("the LATE chunk was never received"); @@ -214,9 +287,12 @@ async fn test_early_chunk_is_visible_before_the_body_completes() { a small gap means the body was buffered to completion instead of streamed" ); - match tokio::task::spawn_blocking(move || server.join()).await { - Ok(Ok(())) => {} - Ok(Err(panic)) => std::panic::resume_unwind(panic), - Err(e) => panic!("server join task failed: {e}"), - } + let interpreter_result = tokio::time::timeout(Duration::from_secs(3), done_rx) + .await + .expect("interpreter did not finish after the streaming body closed") + .expect("interpreter result sender dropped"); + interpreter_result.unwrap_or_else(|errors| { + panic!("streaming visibility program must finish successfully, got {errors:?}") + }); + join_server(server).await; } diff --git a/tests/stream_handle_type_test.rs b/tests/stream_handle_type_test.rs index c896a133..7420060c 100644 --- a/tests/stream_handle_type_test.rs +++ b/tests/stream_handle_type_test.rs @@ -194,14 +194,16 @@ fn test_write_and_flush_response_stream_is_ok() { } #[test] -fn test_write_line_variable_to_file_path_is_ok() { +fn test_write_line_variable_to_file_path_is_ok_when_classic_lead_is_defined() { // The AMBIGUOUS merged form (`write line ... to `) carries a - // classic file-write fallback, so a text file-path target must NOT be rejected. - let code = "store payload as \"data\"\n\ + // classic file-write fallback whose legacy variable is the full merged name + // (`line payload`). A concrete text path selects that branch, so its actual + // lead must be defined; the speculative stream lead (`payload`) need not be. + let code = "store line payload as \"data\"\n\ write line payload to \"/tmp/out.txt\""; assert!( typecheck(code).is_ok(), - "an ambiguous `write line to ` must accept a text file target: {:?}", + "an ambiguous file write must accept its defined classic lead: {:?}", typecheck(code).err() ); } diff --git a/tests/write_web_postfix_test.rs b/tests/write_web_postfix_test.rs index eae001a9..c83de31e 100644 --- a/tests/write_web_postfix_test.rs +++ b/tests/write_web_postfix_test.rs @@ -27,6 +27,100 @@ fn stream_write_value(stmt: &Statement) -> &Expression { } } +fn stream_write_fallback(stmt: &Statement) -> &Expression { + match stmt { + Statement::StreamWriteStatement { + fallback_content: Some(fallback), + .. + } => fallback, + other => panic!("expected an ambiguous StreamWriteStatement fallback, got {other:#?}"), + } +} + +fn expression_shape(expr: &Expression) -> &'static str { + match expr { + Expression::IndexAccess { .. } => "index", + Expression::PropertyAccess { .. } => "property", + Expression::MethodCall { .. } => "method", + Expression::FunctionCall { .. } => "of-call", + Expression::BinaryOperation { .. } => "operator", + other => panic!("unexpected operand shape in parity matrix: {other:#?}"), + } +} + +fn streaming_clause_operand<'a>(stmt: &'a Statement, clause: &str) -> &'a Expression { + match stmt { + Statement::StartStreamingResponseStatement { + content_type, + headers, + .. + } => match clause { + "content type" => content_type.as_ref().expect("content type operand"), + "headers" => headers.as_ref().expect("headers operand"), + other => panic!("unknown test clause {other}"), + }, + other => panic!("expected StartStreamingResponseStatement, got {other:#?}"), + } +} + +#[test] +fn write_and_streaming_clauses_share_the_full_expression_suffix_grammar() { + let cases = [ + ("values[0]", "index"), + ("object.field", "property"), + ("object.method()", "method"), + ("values at 0", "index"), + ("values 0", "index"), + ("convert of values", "of-call"), + ("values plus 1", "operator"), + ]; + + for (operand, expected) in cases { + let write = parse(&format!("write line {operand} to out\n")); + assert_eq!( + write.statements.len(), + 1, + "write operand `{operand}` split into extra statements: {:#?}", + write.statements + ); + assert_eq!( + expression_shape(stream_write_value(&write.statements[0])), + expected, + "write operand `{operand}`" + ); + + for clause in ["content type", "headers"] { + let program = parse(&format!( + "start streaming response to req with status 200 and {clause} {operand} as out\n" + )); + assert_eq!( + program.statements.len(), + 1, + "{clause} operand `{operand}` split into extra statements: {:#?}", + program.statements + ); + assert_eq!( + expression_shape(streaming_clause_operand(&program.statements[0], clause)), + expected, + "{clause} operand `{operand}`" + ); + } + } +} + +#[test] +fn unmerged_content_type_literal_keeps_operator_continuation() { + let program = parse( + "start streaming response to req with status 200 and content type \"text/\" with subtype as out\n", + ); + assert_eq!(program.statements.len(), 1, "got {:#?}", program.statements); + let content_type = streaming_clause_operand(&program.statements[0], "content type"); + assert!( + matches!(content_type, Expression::Concatenation { .. }), + "unmerged literal operand must retain `with subtype`, got {content_type:#?}" + ); +} + #[test] fn write_line_indexed_operand_composes_into_one_index_access() { let program = parse("write line chunks[0] to out\n"); @@ -125,6 +219,16 @@ fn write_line_at_indexing_parses() { "write value must be IndexAccess for `at` indexing, got {:#?}", stream_write_value(write) ); + assert!( + matches!( + stream_write_fallback(write), + Expression::IndexAccess { collection, index, .. } + if matches!(collection.as_ref(), Expression::Variable(name, ..) if name == "line values") + && matches!(index.as_ref(), Expression::Literal(wfl::parser::ast::Literal::Integer(0), ..)) + ), + "classic fallback must index the full `line values` binding, got {:#?}", + stream_write_fallback(write) + ); } #[test] @@ -140,6 +244,16 @@ fn write_line_direct_integer_indexing_parses() { "write value must be IndexAccess for direct integer indexing, got {:#?}", stream_write_value(&program.statements[0]) ); + assert!( + matches!( + stream_write_fallback(&program.statements[0]), + Expression::IndexAccess { collection, index, .. } + if matches!(collection.as_ref(), Expression::Variable(name, ..) if name == "line values") + && matches!(index.as_ref(), Expression::Literal(wfl::parser::ast::Literal::Integer(0), ..)) + ), + "classic fallback must retain the full merged binding for direct indexing, got {:#?}", + stream_write_fallback(&program.statements[0]) + ); } #[test] @@ -200,6 +314,211 @@ fn streaming_response_content_type_then_headers_both_orders() { } } +#[test] +fn streaming_clause_boundary_survives_nested_concatenation_rhs_in_both_orders() { + let cases = [ + ( + "start streaming response to req with content type \"text/\" with subtype and headers h as out\n", + "content type", + ), + ( + "start streaming response to req with headers base_headers with extra and content type ct as out\n", + "headers", + ), + ]; + + for (source, nested_clause) in cases { + let program = parse(source); + assert_eq!( + program.statements.len(), + 1, + "`{source}` must remain one statement; got {:#?}", + program.statements + ); + let statement = &program.statements[0]; + assert!( + matches!( + streaming_clause_operand(statement, nested_clause), + Expression::Concatenation { .. } + ), + "{nested_clause} must retain its complete concatenation operand; got {statement:#?}" + ); + assert!( + matches!( + statement, + Statement::StartStreamingResponseStatement { + content_type: Some(_), + headers: Some(_), + .. + } + ), + "the following response clause must not be swallowed into the concatenation RHS; \ + got {statement:#?}" + ); + } +} + +#[test] +fn streaming_clause_boundary_survives_at_index_expression_in_both_orders() { + let cases = [ + ( + "start streaming response to req with content type media_types at kind and headers h as out\n", + "content type", + ), + ( + "start streaming response to req with headers header_sets at kind and content type ct as out\n", + "headers", + ), + ]; + + for (source, indexed_clause) in cases { + let program = parse(source); + assert_eq!( + program.statements.len(), + 1, + "`{source}` must remain one statement; got {:#?}", + program.statements + ); + let statement = &program.statements[0]; + assert!( + matches!( + streaming_clause_operand(statement, indexed_clause), + Expression::IndexAccess { .. } + ), + "{indexed_clause} must retain its complete `at` index operand; got {statement:#?}" + ); + assert!( + matches!( + statement, + Statement::StartStreamingResponseStatement { + content_type: Some(_), + headers: Some(_), + .. + } + ), + "the following response clause must not be swallowed into the `at` index; \ + got {statement:#?}" + ); + } +} + +#[test] +fn streaming_clause_boundary_propagates_through_recursive_operand_forms() { + let cases = [ + ( + "start streaming response to req with content type lookup of media_types at kind and headers h as out\n", + "of-call", + ), + ( + "start streaming response to req with content type touppercase with media_types at kind and headers h as out\n", + "builtin-call", + ), + ( + "start streaming response to req with content type not media_types at kind and headers h as out\n", + "unary", + ), + ]; + + for (source, operand_kind) in cases { + let program = parse(source); + assert_eq!( + program.statements.len(), + 1, + "`{source}` must remain one statement; got {:#?}", + program.statements + ); + let statement = &program.statements[0]; + let content_type = streaming_clause_operand(statement, "content type"); + let has_expected_shape = match operand_kind { + "of-call" => matches!(content_type, Expression::FunctionCall { .. }), + "builtin-call" => matches!(content_type, Expression::ActionCall { .. }), + "unary" => matches!(content_type, Expression::UnaryOperation { .. }), + _ => unreachable!("unknown recursive operand kind"), + }; + assert!( + has_expected_shape, + "{operand_kind} operand must retain its complete AST; got {content_type:#?}" + ); + assert!( + matches!( + statement, + Statement::StartStreamingResponseStatement { + content_type: Some(_), + headers: Some(_), + .. + } + ), + "the following headers clause must survive {operand_kind} recursion; got {statement:#?}" + ); + } +} + +#[test] +fn streaming_clause_boundary_propagates_through_explicit_call_arguments() { + let program = parse( + "start streaming response to req with content type call render with value and headers h as out\n", + ); + assert_eq!(program.statements.len(), 1, "got {:#?}", program.statements); + + let statement = &program.statements[0]; + let content_type = streaming_clause_operand(statement, "content type"); + assert!( + matches!( + content_type, + Expression::ActionCall { arguments, .. } + if arguments.len() == 1 + && matches!(&arguments[0].value, Expression::Variable(name, ..) if name == "value") + ), + "the following headers clause must not become another explicit-call argument; \ + got {content_type:#?}" + ); + assert!( + matches!( + statement, + Statement::StartStreamingResponseStatement { + headers: Some(Expression::Variable(name, ..)), + .. + } if name == "h" + ), + "the headers clause must remain outside the explicit call; got {statement:#?}" + ); +} + +#[test] +fn streaming_clause_boundary_propagates_through_at_index_under_file_exists() { + let program = parse( + "start streaming response to req with content type file exists at paths at kind and headers h as out\n", + ); + assert_eq!(program.statements.len(), 1, "got {:#?}", program.statements); + + let statement = &program.statements[0]; + let content_type = streaming_clause_operand(statement, "content type"); + assert!( + matches!( + content_type, + Expression::FileExists { path, .. } + if matches!( + path.as_ref(), + Expression::IndexAccess { collection, index, .. } + if matches!(collection.as_ref(), Expression::Variable(name, ..) if name == "paths") + && matches!(index.as_ref(), Expression::Variable(name, ..) if name == "kind") + ) + ), + "the wrapper's nested `at` index must stop before the next response clause; \ + got {content_type:#?}" + ); + assert!( + matches!( + statement, + Statement::StartStreamingResponseStatement { + headers: Some(Expression::Variable(name, ..)), + .. + } if name == "h" + ), + "the headers clause must survive recursion through `file exists at`; got {statement:#?}" + ); +} + #[test] fn write_line_of_call_argument_absorbs_arithmetic() { // `double of n minus 1` must parse as `double of (n minus 1)` — the same @@ -254,6 +573,79 @@ fn flush_method_call_operand_composes() { } } +fn leftmost_variable(expr: &Expression) -> Option<&str> { + match expr { + Expression::Variable(name, ..) => Some(name), + Expression::IndexAccess { collection, .. } => leftmost_variable(collection), + Expression::PropertyAccess { object, .. } | Expression::MethodCall { object, .. } => { + leftmost_variable(object) + } + Expression::BinaryOperation { left, .. } => leftmost_variable(left), + Expression::FunctionCall { function, .. } => leftmost_variable(function), + _ => None, + } +} + +#[test] +fn flush_preserves_binary_continuation_for_both_interpretations() { + let program = parse("flush cache plus 1\n"); + assert_eq!(program.statements.len(), 1, "got {:#?}", program.statements); + match &program.statements[0] { + Statement::FlushStreamStatement { + target, + action_fallback, + .. + } => { + assert!( + matches!(target, Expression::BinaryOperation { .. }), + "stream target must keep `plus 1`, got {target:#?}" + ); + assert_eq!(leftmost_variable(target), Some("cache")); + let fallback = action_fallback.as_ref().expect("legacy fallback"); + assert!( + matches!(fallback, Expression::BinaryOperation { .. }), + "legacy expression must keep `plus 1`, got {fallback:#?}" + ); + assert_eq!(leftmost_variable(fallback), Some("flush cache")); + } + other => panic!("expected FlushStreamStatement, got {other:#?}"), + } +} + +#[test] +fn flush_preserves_arbitrarily_nested_postfix_for_both_interpretations() { + let program = parse("flush cache[0][0]\n"); + assert_eq!(program.statements.len(), 1, "got {:#?}", program.statements); + match &program.statements[0] { + Statement::FlushStreamStatement { + target, + action_fallback, + .. + } => { + assert!( + matches!( + target, + Expression::IndexAccess { collection, .. } + if matches!(collection.as_ref(), Expression::IndexAccess { .. }) + ), + "stream target must retain both indexes, got {target:#?}" + ); + assert_eq!(leftmost_variable(target), Some("cache")); + let fallback = action_fallback.as_ref().expect("legacy fallback"); + assert!( + matches!( + fallback, + Expression::IndexAccess { collection, .. } + if matches!(collection.as_ref(), Expression::IndexAccess { .. }) + ), + "legacy expression must retain both indexes, got {fallback:#?}" + ); + assert_eq!(leftmost_variable(fallback), Some("flush cache")); + } + other => panic!("expected FlushStreamStatement, got {other:#?}"), + } +} + #[test] fn classic_indexed_file_write_still_works_at_runtime() { // The ambiguous merged form's classic file-write reading must keep working with From 09115f88b0ba1bcf8ecbdba3ca81ab62eaa07e40 Mon Sep 17 00:00:00 2001 From: Brad Byrd Date: Fri, 24 Jul 2026 23:53:31 -0500 Subject: [PATCH 092/132] test: expose full streaming status operands --- tests/write_web_postfix_test.rs | 55 +++++++++++++++++++++++++++++++++ 1 file changed, 55 insertions(+) diff --git a/tests/write_web_postfix_test.rs b/tests/write_web_postfix_test.rs index c83de31e..a77579e0 100644 --- a/tests/write_web_postfix_test.rs +++ b/tests/write_web_postfix_test.rs @@ -63,6 +63,61 @@ fn streaming_clause_operand<'a>(stmt: &'a Statement, clause: &str) -> &'a Expres } } +fn streaming_status_and_headers(stmt: &Statement) -> (&Expression, &Expression) { + match stmt { + Statement::StartStreamingResponseStatement { + status: Some(status), + headers: Some(headers), + .. + } => (status, headers), + other => panic!( + "expected a streaming response with status and headers operands, got {other:#?}" + ), + } +} + +#[test] +fn streaming_status_clause_accepts_full_expressions_without_swallowing_headers() { + let cases = [ + ( + "start streaming response to req with status base plus 1 and headers h as out\n", + "operator", + ), + ( + "start streaming response to req with headers h and status codes at i as out\n", + "index", + ), + ( + "start streaming response to req with status response.code and headers h as out\n", + "property", + ), + ( + "start streaming response to req with status choose of req and headers h as out\n", + "of-call", + ), + ]; + + for (source, expected_shape) in cases { + let program = parse(source); + assert_eq!( + program.statements.len(), + 1, + "`{source}` must remain one statement; got {:#?}", + program.statements + ); + let (status, headers) = streaming_status_and_headers(&program.statements[0]); + assert_eq!( + expression_shape(status), + expected_shape, + "status operand in `{source}`" + ); + assert!( + matches!(headers, Expression::Variable(name, ..) if name == "h"), + "headers must remain a separate clause in `{source}`; got {headers:#?}" + ); + } +} + #[test] fn write_and_streaming_clauses_share_the_full_expression_suffix_grammar() { let cases = [ From 99353201518917b350009822554c7d41f6662582 Mon Sep 17 00:00:00 2001 From: Brad Byrd Date: Fri, 24 Jul 2026 23:54:16 -0500 Subject: [PATCH 093/132] fix: parse complete streaming status operands --- src/parser/stmt/web.rs | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/src/parser/stmt/web.rs b/src/parser/stmt/web.rs index 941b0393..89cbcfab 100644 --- a/src/parser/stmt/web.rs +++ b/src/parser/stmt/web.rs @@ -425,7 +425,7 @@ impl<'a> WebParser<'a> for Parser<'a> { Token::KeywordStatus => { self.bump_sync(); // with/and self.bump_sync(); // status - status = Some(self.parse_primary_expression()?); + status = Some(self.parse_unmerged_operand(true)?); } // `content type ` — `content` keyword then optional `type`. // When `` is a bare identifier the lexer merges it into the From f23fb6bc0c3b2b77cf1f9eeab567b38032710f9c Mon Sep 17 00:00:00 2001 From: Brad Byrd Date: Fri, 24 Jul 2026 23:55:02 -0500 Subject: [PATCH 094/132] test: use valid streaming status fixture --- tests/write_web_postfix_test.rs | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/tests/write_web_postfix_test.rs b/tests/write_web_postfix_test.rs index a77579e0..944cfb3b 100644 --- a/tests/write_web_postfix_test.rs +++ b/tests/write_web_postfix_test.rs @@ -88,7 +88,7 @@ fn streaming_status_clause_accepts_full_expressions_without_swallowing_headers() "index", ), ( - "start streaming response to req with status response.code and headers h as out\n", + "start streaming response to req with status reply.code and headers h as out\n", "property", ), ( From d97f15b6d9a05be7f35d54b1bbf3d627472ea7d6 Mon Sep 17 00:00:00 2001 From: Brad Byrd Date: Fri, 24 Jul 2026 23:57:00 -0500 Subject: [PATCH 095/132] test: expose postfix loss after of calls --- tests/write_web_postfix_test.rs | 97 +++++++++++++++++++++++++++++++++ 1 file changed, 97 insertions(+) diff --git a/tests/write_web_postfix_test.rs b/tests/write_web_postfix_test.rs index 944cfb3b..893dac99 100644 --- a/tests/write_web_postfix_test.rs +++ b/tests/write_web_postfix_test.rs @@ -574,6 +574,103 @@ fn streaming_clause_boundary_propagates_through_at_index_under_file_exists() { ); } +fn assert_post_of_index(expr: &Expression, expected_function: &str, expected_argument: &str) { + match expr { + Expression::IndexAccess { + collection, index, .. + } => { + assert!( + matches!( + index.as_ref(), + Expression::Literal(wfl::parser::ast::Literal::Integer(0), ..) + ), + "post-call index must be integer zero, got {index:#?}" + ); + match collection.as_ref() { + Expression::FunctionCall { + function, + arguments, + .. + } => { + assert_eq!( + leftmost_variable(function), + Some(expected_function), + "unexpected function root in {expr:#?}" + ); + assert!( + matches!( + arguments.as_slice(), + [wfl::parser::ast::Argument { + value: Expression::Variable(name, ..), + .. + }] if name == expected_argument + ), + "the parenthesized argument must stay inside the call, got {arguments:#?}" + ); + } + other => panic!("index must wrap an `of` FunctionCall, got {other:#?}"), + } + } + other => panic!("expected postfix index after an `of` call, got {other:#?}"), + } +} + +#[test] +fn seeded_operands_resume_postfix_parsing_after_of_calls() { + let write = parse("write line choose of (chunks)[0] to out\n"); + assert_eq!(write.statements.len(), 1, "got {:#?}", write.statements); + assert_post_of_index( + stream_write_value(&write.statements[0]), + "choose", + "chunks", + ); + assert_post_of_index( + stream_write_fallback(&write.statements[0]), + "line choose", + "chunks", + ); + + let streaming = parse( + "start streaming response to req with content type choose of (types)[0] and headers h as out\n", + ); + assert_eq!( + streaming.statements.len(), + 1, + "got {:#?}", + streaming.statements + ); + assert_post_of_index( + streaming_clause_operand(&streaming.statements[0], "content type"), + "choose", + "types", + ); + assert!( + matches!( + &streaming.statements[0], + Statement::StartStreamingResponseStatement { + headers: Some(Expression::Variable(name, ..)), + .. + } if name == "h" + ), + "headers must remain outside the indexed call; got {:#?}", + streaming.statements[0] + ); + + let flush = parse("flush cache of (items)[0]\n"); + assert_eq!(flush.statements.len(), 1, "got {:#?}", flush.statements); + match &flush.statements[0] { + Statement::FlushStreamStatement { + target, + action_fallback: Some(fallback), + .. + } => { + assert_post_of_index(target, "cache", "items"); + assert_post_of_index(fallback, "flush cache", "items"); + } + other => panic!("expected ambiguous FlushStreamStatement, got {other:#?}"), + } +} + #[test] fn write_line_of_call_argument_absorbs_arithmetic() { // `double of n minus 1` must parse as `double of (n minus 1)` — the same From 764685c081f62123a56cf2bbe11aa2b4617d2711 Mon Sep 17 00:00:00 2001 From: Brad Byrd Date: Fri, 24 Jul 2026 23:57:42 -0500 Subject: [PATCH 096/132] fix: resume postfix parsing after of calls --- src/parser/stmt/io.rs | 37 ++++++++++++++++++++++--------------- 1 file changed, 22 insertions(+), 15 deletions(-) diff --git a/src/parser/stmt/io.rs b/src/parser/stmt/io.rs index 4130e491..92a2f229 100644 --- a/src/parser/stmt/io.rs +++ b/src/parser/stmt/io.rs @@ -29,16 +29,25 @@ impl<'a> Parser<'a> { lead: Expression, stop_at_clause: bool, ) -> Result { - // The lexer merges the command word with the operand identifier and leaves - // any bracket-index / dotted-property / `at` / integer-index accessors as - // following tokens, so compose them onto the lead instead of leaving them - // to dangle after the statement. - let lead = if stop_at_clause { - self.parse_trailing_postfix_stopping_at_clause(lead)? - } else { - self.parse_trailing_postfix(lead)? - }; - let lead = if matches!(self.cursor.peek().map(|t| &t.token), Some(Token::KeywordOf)) { + let mut lead = lead; + loop { + // The lexer merges the command word with the operand identifier and + // leaves postfix accessors as following tokens. Compose them before + // checking for `of`, and repeat after an `of` call so + // `choose of (values)[0]` indexes the call result just like an + // ordinary expression. + lead = if stop_at_clause { + self.parse_trailing_postfix_stopping_at_clause(lead)? + } else { + self.parse_trailing_postfix(lead)? + }; + if !matches!( + self.cursor.peek().map(|t| &t.token), + Some(Token::KeywordOf) + ) { + break; + } + // Anchor the ` of ` call to the `of` keyword itself, // matching how the rest of the parser positions FunctionCall nodes so // error spans point at the operator, not the lead (review feedback). @@ -85,15 +94,13 @@ impl<'a> Parser<'a> { }, }); } - Expression::FunctionCall { + lead = Expression::FunctionCall { function: Box::new(lead), arguments, line: of_line, column: of_column, - } - } else { - lead - }; + }; + } if stop_at_clause { self.parse_binary_continuation_stopping_at_clause(lead, 0) } else { From c8cfa08c0352555bd4d302fd4ded21b827e5ceca Mon Sep 17 00:00:00 2001 From: Brad Byrd Date: Fri, 24 Jul 2026 23:58:25 -0500 Subject: [PATCH 097/132] test: expose false type response boundary --- tests/write_web_postfix_test.rs | 28 ++++++++++++++++++++++++++++ 1 file changed, 28 insertions(+) diff --git a/tests/write_web_postfix_test.rs b/tests/write_web_postfix_test.rs index 893dac99..d3c25f98 100644 --- a/tests/write_web_postfix_test.rs +++ b/tests/write_web_postfix_test.rs @@ -413,6 +413,34 @@ fn streaming_clause_boundary_survives_nested_concatenation_rhs_in_both_orders() } } +#[test] +fn type_prefixed_identifier_is_content_not_a_response_clause() { + let source = "start streaming response to req with content type \"application/\" with type suffix and headers h as out\n"; + let program = parse(source); + assert_eq!( + program.statements.len(), + 1, + "`{source}` must remain one statement; got {:#?}", + program.statements + ); + match &program.statements[0] { + Statement::StartStreamingResponseStatement { + content_type: Some(Expression::Concatenation { right, .. }), + headers: Some(Expression::Variable(headers, ..)), + .. + } => { + assert!( + matches!(right.as_ref(), Expression::Variable(name, ..) if name == "type suffix"), + "`type suffix` must remain the concatenation RHS, got {right:#?}" + ); + assert_eq!(headers, "h", "the following headers clause must remain separate"); + } + other => panic!( + "expected concatenated content type plus a separate headers clause, got {other:#?}" + ), + } +} + #[test] fn streaming_clause_boundary_survives_at_index_expression_in_both_orders() { let cases = [ From 485bc34b4daad1354b838a74e59938b5041c0db5 Mon Sep 17 00:00:00 2001 From: Brad Byrd Date: Fri, 24 Jul 2026 23:59:14 -0500 Subject: [PATCH 098/132] fix: remove false type response boundary --- src/parser/expr/binary.rs | 17 +---------------- src/parser/stmt/io.rs | 2 -- 2 files changed, 1 insertion(+), 18 deletions(-) diff --git a/src/parser/expr/binary.rs b/src/parser/expr/binary.rs index 1786d2e2..ac9a9d6a 100644 --- a/src/parser/expr/binary.rs +++ b/src/parser/expr/binary.rs @@ -173,22 +173,7 @@ impl<'a> Parser<'a> { } if matches!(token, Token::KeywordAnd | Token::KeywordWith) { let next = self.cursor.peek_n(1).map(|t| &t.token); - let is_clause = match next { - Some(Token::KeywordAs) - | Some(Token::KeywordContent) - | Some(Token::KeywordStatus) => true, - Some(Token::Identifier(id)) => { - id == "headers" - || id.starts_with("headers ") - || id == "content_type" - || id.starts_with("content_type ") - || id.starts_with("content type") - || id == "type" - || id.starts_with("type ") - } - _ => false, - }; - if is_clause { + if Self::is_streaming_clause_keyword(next) { break; } } diff --git a/src/parser/stmt/io.rs b/src/parser/stmt/io.rs index 92a2f229..c339460d 100644 --- a/src/parser/stmt/io.rs +++ b/src/parser/stmt/io.rs @@ -142,8 +142,6 @@ impl<'a> Parser<'a> { || id == "content_type" || id.starts_with("content_type ") || id.starts_with("content type") - || id == "type" - || id.starts_with("type ") } _ => false, } From 55f3d507c741f44576afce24affbf643ee7d258e Mon Sep 17 00:00:00 2001 From: Brad Byrd Date: Sat, 25 Jul 2026 00:00:08 -0500 Subject: [PATCH 099/132] test: expose unmerged flush operands --- tests/flush_action_backcompat_test.rs | 23 +++++++++++++++++++ tests/http_server_streaming_test.rs | 33 ++++++++++++++++++++++++++- 2 files changed, 55 insertions(+), 1 deletion(-) diff --git a/tests/flush_action_backcompat_test.rs b/tests/flush_action_backcompat_test.rs index 221499dc..8facc05a 100644 --- a/tests/flush_action_backcompat_test.rs +++ b/tests/flush_action_backcompat_test.rs @@ -55,6 +55,29 @@ fn flush_calls_a_matching_zero_arg_action_instead_of_flushing_a_stream() { ); } +#[test] +fn truly_bare_flush_still_calls_the_legacy_zero_argument_action() { + let src = "define action called flush:\n\ + \x20\x20\x20\x20display \"CALLED\"\n\ + end action\n\ + \n\ + flush\n"; + let (out, code) = run_src(src); + assert_eq!( + code, + Some(0), + "the exact bare `flush` action must remain callable; output was:\n{out}" + ); + assert!( + out.contains("CALLED"), + "the exact bare `flush` statement must auto-call its legacy action; output was:\n{out}" + ); + assert!( + !out.to_lowercase().contains("stream"), + "the exact bare `flush` statement must not become a stream operation; output was:\n{out}" + ); +} + #[test] fn flush_without_a_matching_action_still_errors_as_a_stream_flush() { // With no action `flush cache` and no stream `cache`, `flush cache` falls diff --git a/tests/http_server_streaming_test.rs b/tests/http_server_streaming_test.rs index 9321e49b..6b1bdef1 100644 --- a/tests/http_server_streaming_test.rs +++ b/tests/http_server_streaming_test.rs @@ -11,7 +11,7 @@ use std::time::Duration; use wfl::Interpreter; use wfl::lexer::lex_wfl_with_positions; use wfl::parser::Parser; -use wfl::parser::ast::Statement; +use wfl::parser::ast::{Expression, Statement}; mod common; @@ -158,6 +158,37 @@ fn test_flush_with_property_operand_parses() { } } +#[test] +fn test_unmerged_flush_targets_parse_as_single_flush_statements() { + let cases = [ + ("flush (out)", "variable"), + ("flush call acquire stream", "call"), + ("flush (streams)[0]", "index"), + ("flush (holder).stream", "property"), + ("flush (holder).method()", "method"), + ]; + + for (source, expected_shape) in cases { + let statement = parse_single_statement(source); + let target = match &statement { + Statement::FlushStreamStatement { target, .. } => target, + other => panic!("expected FlushStreamStatement for `{source}`, got {other:#?}"), + }; + let actual_shape = match target { + Expression::Variable(..) => "variable", + Expression::ActionCall { .. } => "call", + Expression::IndexAccess { .. } => "index", + Expression::PropertyAccess { .. } => "property", + Expression::MethodCall { .. } => "method", + other => panic!("unexpected flush target for `{source}`: {other:#?}"), + }; + assert_eq!( + actual_shape, expected_shape, + "wrong target shape for `{source}`: {target:#?}" + ); + } +} + #[test] fn test_write_bare_line_variable_to_file_still_parses() { // Backward compat: `write to ` with a variable literally named From 4a838459bf985611338e69f81253b2a6eee0e269 Mon Sep 17 00:00:00 2001 From: Brad Byrd Date: Sat, 25 Jul 2026 00:00:55 -0500 Subject: [PATCH 100/132] fix: dispatch unmerged flush operands --- src/parser/mod.rs | 13 +++++++++++++ src/parser/stmt/web.rs | 2 +- 2 files changed, 14 insertions(+), 1 deletion(-) diff --git a/src/parser/mod.rs b/src/parser/mod.rs index 7c96e952..62a2d0fb 100644 --- a/src/parser/mod.rs +++ b/src/parser/mod.rs @@ -633,6 +633,19 @@ impl<'a> StmtParser<'a> for Parser<'a> { // operand follows, so a bare `flush` used as an action/variable // name still parses as an expression statement. Token::Identifier(id) if id.starts_with("flush ") => self.parse_flush_stream(), + // Parenthesized and explicit-call targets do not merge into the + // leading `flush` token. These two starters are unambiguous; + // broader primary-expression dispatch would steal legacy + // expressions such as `flush with suffix` and `flush at 0`. + Token::Identifier(id) + if id == "flush" + && matches!( + self.cursor.peek_next().map(|t| &t.token), + Some(Token::LeftParen | Token::KeywordCall) + ) => + { + self.parse_flush_stream() + } Token::Identifier(id) if id.starts_with("send websocket message") => { self.parse_send_websocket_message() } diff --git a/src/parser/stmt/web.rs b/src/parser/stmt/web.rs index 89cbcfab..384466fe 100644 --- a/src/parser/stmt/web.rs +++ b/src/parser/stmt/web.rs @@ -548,7 +548,7 @@ impl<'a> WebParser<'a> for Parser<'a> { .map(str::trim_start) .unwrap_or(""); let (target, legacy_binding, action_fallback) = if rest.is_empty() { - (self.parse_primary_expression()?, None, None) + (self.parse_unmerged_operand(false)?, None, None) } else { // Stream reading: postfix on the split-off rest (`cache` from // `flush cache`). Legacy expression: same postfix on the FULL phrase From 8b10f8bff36fba1df4e6bae0eda07e9f05c16721 Mon Sep 17 00:00:00 2001 From: Brad Byrd Date: Sat, 25 Jul 2026 00:02:52 -0500 Subject: [PATCH 101/132] test: expose response stream scope leaks --- .../typechecker_response_stream_scope_test.rs | 61 +++++++++++++++++++ 1 file changed, 61 insertions(+) create mode 100644 tests/typechecker_response_stream_scope_test.rs diff --git a/tests/typechecker_response_stream_scope_test.rs b/tests/typechecker_response_stream_scope_test.rs new file mode 100644 index 00000000..942ce771 --- /dev/null +++ b/tests/typechecker_response_stream_scope_test.rs @@ -0,0 +1,61 @@ +//! Regression coverage for type-checker scopes that must mirror runtime child +//! environments when response-stream bindings shadow outer file handles. + +use wfl::lexer::lex_wfl_with_positions; +use wfl::parser::Parser; +use wfl::typechecker::TypeChecker; + +fn typecheck(code: &str) -> Result<(), String> { + let tokens = lex_wfl_with_positions(code); + let program = Parser::new(&tokens).parse().expect("parse"); + TypeChecker::new() + .check_types(&program) + .map_err(|errors| format!("{errors:?}")) +} + +#[test] +fn response_stream_bindings_do_not_escape_runtime_child_scopes() { + let scoped_blocks = [ + "repeat while false:\n\ + \x20\x20\x20\x20start streaming response to \"request\" with status 200 as out\n\ + end repeat\n", + "try:\n\ + \x20\x20\x20\x20start streaming response to \"request\" with status 200 as out\n\ + end try\n", + "count from 1 to 1:\n\ + \x20\x20\x20\x20start streaming response to \"request\" with status 200 as out\n\ + end count\n", + ]; + + for scoped_block in scoped_blocks { + let source = format!( + "open file at \"unused.txt\" for writing as out\n\ + {scoped_block}\ + store value as \"wrong stream type\"\n\ + store line value as 10\n\ + store n as 1\n\ + write line value minus n to out\n" + ); + assert!( + typecheck(&source).is_ok(), + "a response stream created in a runtime child scope must not replace \ + the outer File type; source:\n{source}\nerrors: {:?}", + typecheck(&source).err() + ); + } +} + +#[test] +fn default_count_binding_does_not_retype_an_outer_count_variable() { + let source = "store count as \"outside\"\n\ + count from 1 to 1:\n\ + \x20\x20\x20\x20display count\n\ + end count\n\ + store invalid as count minus 1\n"; + let errors = + typecheck(source).expect_err("the outer Text `count minus 1` must remain a type error"); + assert!( + errors.contains("Cannot perform Minus operation"), + "expected the outer Text/Number subtraction error, got: {errors}" + ); +} From a1bdd9d75bd3c8134cb0fb49dc601ff209d6c26f Mon Sep 17 00:00:00 2001 From: Brad Byrd Date: Sat, 25 Jul 2026 00:04:01 -0500 Subject: [PATCH 102/132] fix: mirror runtime child scopes in type checker --- src/typechecker/mod.rs | 26 ++++++++++++++++++++++---- 1 file changed, 22 insertions(+), 4 deletions(-) diff --git a/src/typechecker/mod.rs b/src/typechecker/mod.rs index 4a6b1176..152b89d6 100644 --- a/src/typechecker/mod.rs +++ b/src/typechecker/mod.rs @@ -648,9 +648,11 @@ impl TypeChecker { )); } + self.analyzer.push_scope(); for stmt in body { self.check_statement_types(stmt); } + self.analyzer.pop_scope(); } Statement::ExitStatement { line: _, column: _ } => {} Statement::WaitForStatement { @@ -688,6 +690,9 @@ impl TypeChecker { line: _line, column: _column, } => { + // Runtime evaluates the try body, handlers, otherwise, and + // finally block inside one shared child environment. + self.analyzer.push_scope(); for stmt in body { self.check_statement_types(stmt); } @@ -736,6 +741,7 @@ impl TypeChecker { self.check_statement_types(stmt); } } + self.analyzer.pop_scope(); } Statement::HttpGetStatement { url, @@ -1518,6 +1524,7 @@ impl TypeChecker { body, line: _line, column: _column, + variable_name, .. } => { let start_type = self.infer_expression_type(start); @@ -1562,14 +1569,25 @@ impl TypeChecker { } } - // Register the "count" variable with type Number - if let Some(symbol) = self.analyzer.get_symbol_mut("count") { - symbol.symbol_type = Some(Type::Number); - } + // Runtime creates the loop variable in a child environment, + // shadowing rather than retyping an outer `count` or custom + // loop-variable binding. + self.analyzer.push_scope(); + self.analyzer.define_or_replace_symbol(Symbol { + name: variable_name + .as_deref() + .unwrap_or("count") + .to_string(), + kind: SymbolKind::Variable { mutable: true }, + symbol_type: Some(Type::Number), + line: *_line, + column: *_column, + }); for stmt in body { self.check_statement_types(stmt); } + self.analyzer.pop_scope(); } Statement::WhileLoop { condition, From 7bafc6da8682de19886bb3c47cc14e67c5d2b9e2 Mon Sep 17 00:00:00 2001 From: Brad Byrd Date: Sat, 25 Jul 2026 00:04:39 -0500 Subject: [PATCH 103/132] test: use valid try scope fixture --- tests/typechecker_response_stream_scope_test.rs | 2 ++ 1 file changed, 2 insertions(+) diff --git a/tests/typechecker_response_stream_scope_test.rs b/tests/typechecker_response_stream_scope_test.rs index 942ce771..b98d78b1 100644 --- a/tests/typechecker_response_stream_scope_test.rs +++ b/tests/typechecker_response_stream_scope_test.rs @@ -21,6 +21,8 @@ fn response_stream_bindings_do_not_escape_runtime_child_scopes() { end repeat\n", "try:\n\ \x20\x20\x20\x20start streaming response to \"request\" with status 200 as out\n\ + when error:\n\ + \x20\x20\x20\x20display \"ignored\"\n\ end try\n", "count from 1 to 1:\n\ \x20\x20\x20\x20start streaming response to \"request\" with status 200 as out\n\ From 24f57d63dcd7018a1ea31d1f14c63c2e4069a982 Mon Sep 17 00:00:00 2001 From: Brad Byrd Date: Sat, 25 Jul 2026 00:06:16 -0500 Subject: [PATCH 104/132] test: expose missing response stream type joins --- .../typechecker_response_stream_join_test.rs | 132 ++++++++++++++++++ 1 file changed, 132 insertions(+) create mode 100644 tests/typechecker_response_stream_join_test.rs diff --git a/tests/typechecker_response_stream_join_test.rs b/tests/typechecker_response_stream_join_test.rs new file mode 100644 index 00000000..1490971a --- /dev/null +++ b/tests/typechecker_response_stream_join_test.rs @@ -0,0 +1,132 @@ +//! Regression coverage for conservative type-state joins across conditional +//! control flow involving response-stream and file-handle bindings. + +use std::sync::Arc; +use wfl::lexer::lex_wfl_with_positions; +use wfl::parser::Parser; +use wfl::parser::ast::{Expression, FileOpenMode, Literal, Program, Statement}; +use wfl::typechecker::TypeChecker; + +fn parse(source: &str) -> Program { + Parser::new(&lex_wfl_with_positions(source)) + .parse() + .expect("parse") +} + +fn typecheck(program: &Program) -> Result<(), String> { + TypeChecker::new() + .check_types(program) + .map_err(|errors| format!("{errors:?}")) +} + +fn bool_literal(value: bool) -> Expression { + Expression::Literal(Literal::Boolean(value), 2, 1) +} + +fn text_literal(value: &str) -> Expression { + Expression::Literal(Literal::String(Arc::from(value)), 2, 1) +} + +fn stream_binding() -> Statement { + Statement::StartStreamingResponseStatement { + request: text_literal("request"), + status: Some(Expression::Literal(Literal::Integer(200), 2, 1)), + content_type: None, + headers: None, + variable_name: "out".to_string(), + line: 2, + column: 1, + } +} + +fn ambiguous_file_write_program(control: Statement) -> Program { + let mut program = parse( + "open file at \"unused.txt\" for writing as out\n\ + store value as 10\n\ + store line value as \"text\"\n\ + store n as 1\n\ + write line value minus n to out\n", + ); + program.statements.insert(1, control); + program +} + +#[test] +fn maybe_skipped_stream_bindings_require_both_write_readings_to_be_valid() { + let controls = [ + ( + "if", + Statement::IfStatement { + condition: bool_literal(false), + then_block: vec![stream_binding()], + else_block: None, + line: 2, + column: 1, + }, + ), + ( + "single-line if", + Statement::SingleLineIf { + condition: bool_literal(false), + then_stmt: Box::new(stream_binding()), + else_stmt: None, + line: 2, + column: 1, + }, + ), + ( + "while", + Statement::WhileLoop { + condition: bool_literal(false), + body: vec![stream_binding()], + line: 2, + column: 1, + }, + ), + ]; + + for (label, control) in controls { + let errors = typecheck(&ambiguous_file_write_program(control)) + .expect_err("File or ResponseStream must conservatively validate both write readings"); + assert!( + errors.contains("Cannot perform Minus operation"), + "{label} must retain the possible outer File path and reject the \ + Text/Number classic fallback; got: {errors}" + ); + } +} + +#[test] +fn two_concrete_branch_types_join_instead_of_taking_the_last_checked_branch() { + let program = Program { + statements: vec![ + Statement::IfStatement { + condition: bool_literal(true), + then_block: vec![stream_binding()], + else_block: Some(vec![Statement::OpenFileStatement { + path: text_literal("unused.txt"), + variable_name: "out".to_string(), + mode: FileOpenMode::Write, + line: 3, + column: 1, + }]), + line: 1, + column: 1, + }, + Statement::FlushStreamStatement { + target: Expression::Variable("out".to_string(), 5, 1), + legacy_binding: None, + action_fallback: None, + line: 5, + column: 1, + }, + ], + }; + + assert!( + typecheck(&program).is_ok(), + "ResponseStream or File must join to a gradual type instead of treating \ + the last checked File branch as definite; errors: {:?}", + typecheck(&program).err() + ); +} From 046b012e8fd9d79a34ee2032fd2ae36da405816d Mon Sep 17 00:00:00 2001 From: Brad Byrd Date: Sat, 25 Jul 2026 00:08:02 -0500 Subject: [PATCH 105/132] fix: join conditional type states --- src/typechecker/mod.rs | 81 +++++++++++++++++++++++++++++++++++++++--- 1 file changed, 77 insertions(+), 4 deletions(-) diff --git a/src/typechecker/mod.rs b/src/typechecker/mod.rs index 152b89d6..e38404be 100644 --- a/src/typechecker/mod.rs +++ b/src/typechecker/mod.rs @@ -197,6 +197,59 @@ impl TypeChecker { self.analyzer.get_action_parameters() } + fn join_type_snapshots( + states: &[Vec>>], + ) -> Vec>> { + let Some(first) = states.first() else { + return Vec::new(); + }; + let mut joined = first.clone(); + + // Analyzer pass 1 normally pre-registers branch-visible symbols, but + // include names from every state so direct AST users receive the same + // conservative result. + for state in states.iter().skip(1) { + if joined.len() < state.len() { + joined.resize_with(state.len(), HashMap::new); + } + for (layer_index, layer) in state.iter().enumerate() { + for name in layer.keys() { + joined[layer_index].entry(name.clone()).or_insert(None); + } + } + } + + for layer_index in 0..joined.len() { + let names: Vec = joined[layer_index].keys().cloned().collect(); + for name in names { + let values: Vec> = states + .iter() + .map(|state| { + state + .get(layer_index) + .and_then(|layer| layer.get(&name)) + .cloned() + .unwrap_or(None) + }) + .collect(); + let first_value = values.first().cloned().unwrap_or(None); + let merged = if values.iter().all(|value| value == &first_value) { + first_value + } else if values.iter().any(|value| { + value.is_none() || matches!(value.as_ref(), Some(Type::Unknown)) + }) + { + Some(Type::Unknown) + } else { + Some(Type::Any) + }; + joined[layer_index].insert(name, merged); + } + } + + joined + } + /// Get the return type for builtin functions fn get_builtin_function_type(&self, name: &str, _arg_count: usize) -> Type { match name { @@ -1428,15 +1481,23 @@ impl TypeChecker { ); } + let entry_types = self.analyzer.snapshot_symbol_types(); for stmt in then_block { self.check_statement_types(stmt); } + let then_types = self.analyzer.snapshot_symbol_types(); + self.analyzer.restore_symbol_types(entry_types.clone()); - if let Some(else_stmts) = else_block { + let else_types = if let Some(else_stmts) = else_block { for stmt in else_stmts { self.check_statement_types(stmt); } - } + self.analyzer.snapshot_symbol_types() + } else { + entry_types + }; + let joined = Self::join_type_snapshots(&[then_types, else_types]); + self.analyzer.restore_symbol_types(joined); } Statement::SingleLineIf { condition, @@ -1459,11 +1520,19 @@ impl TypeChecker { ); } + let entry_types = self.analyzer.snapshot_symbol_types(); self.check_statement_types(then_stmt); + let then_types = self.analyzer.snapshot_symbol_types(); + self.analyzer.restore_symbol_types(entry_types.clone()); - if let Some(else_stmt) = else_stmt { + let else_types = if let Some(else_stmt) = else_stmt { self.check_statement_types(else_stmt); - } + self.analyzer.snapshot_symbol_types() + } else { + entry_types + }; + let joined = Self::join_type_snapshots(&[then_types, else_types]); + self.analyzer.restore_symbol_types(joined); } Statement::ForEachLoop { item_name, @@ -1609,9 +1678,13 @@ impl TypeChecker { ); } + let entry_types = self.analyzer.snapshot_symbol_types(); for stmt in body { self.check_statement_types(stmt); } + let body_types = self.analyzer.snapshot_symbol_types(); + let joined = Self::join_type_snapshots(&[body_types, entry_types]); + self.analyzer.restore_symbol_types(joined); } Statement::RepeatUntilLoop { condition, From a30fe4f50f8beff3d3b3af67aa234723f6d858fb Mon Sep 17 00:00:00 2001 From: Brad Byrd Date: Sat, 25 Jul 2026 00:09:13 -0500 Subject: [PATCH 106/132] test: expose missing local file handle types --- tests/open_file_local_type_test.rs | 77 ++++++++++++++++++++++++++++++ 1 file changed, 77 insertions(+) create mode 100644 tests/open_file_local_type_test.rs diff --git a/tests/open_file_local_type_test.rs b/tests/open_file_local_type_test.rs new file mode 100644 index 00000000..080ffc89 --- /dev/null +++ b/tests/open_file_local_type_test.rs @@ -0,0 +1,77 @@ +//! Regression coverage for local `open file ... as ...` bindings that analyzer +//! body scopes do not retain for the type-checker pass. + +use wfl::lexer::lex_wfl_with_positions; +use wfl::parser::Parser; +use wfl::typechecker::TypeChecker; + +fn typecheck(source: &str) -> Result<(), String> { + let program = Parser::new(&lex_wfl_with_positions(source)) + .parse() + .expect("parse"); + TypeChecker::new() + .check_types(&program) + .map_err(|errors| format!("{errors:?}")) +} + +#[test] +fn fresh_local_file_handles_are_concrete_in_action_loop_and_method_scopes() { + let sources = [ + ( + "action", + "define action called dump:\n\ + \x20\x20\x20\x20open file at \"unused.txt\" for writing as out\n\ + \x20\x20\x20\x20store line value as \"classic\"\n\ + \x20\x20\x20\x20write line value to out\n\ + \x20\x20\x20\x20close out\n\ + end action\n", + ), + ( + "main loop", + "main loop:\n\ + \x20\x20\x20\x20open file at \"unused.txt\" for writing as out\n\ + \x20\x20\x20\x20store line value as \"classic\"\n\ + \x20\x20\x20\x20write line value to out\n\ + \x20\x20\x20\x20close out\n\ + \x20\x20\x20\x20break\n\ + end loop\n", + ), + ( + "container method", + "create container Writer:\n\ + \x20\x20\x20\x20action dump:\n\ + \x20\x20\x20\x20\x20\x20\x20\x20open file at \"unused.txt\" for writing as out\n\ + \x20\x20\x20\x20\x20\x20\x20\x20store line value as \"classic\"\n\ + \x20\x20\x20\x20\x20\x20\x20\x20write line value to out\n\ + \x20\x20\x20\x20\x20\x20\x20\x20close out\n\ + \x20\x20\x20\x20end\n\ + end\n", + ), + ]; + + for (scope, source) in sources { + assert!( + typecheck(source).is_ok(), + "a fresh File handle in {scope} must select only the classic write \ + reading; errors: {:?}\nsource:\n{source}", + typecheck(source).err() + ); + } +} + +#[test] +fn opening_a_local_file_shadows_instead_of_retyping_an_outer_binding() { + let source = "store out as \"outer.txt\"\n\ + main loop:\n\ + \x20\x20\x20\x20open file at \"inner.txt\" for writing as out\n\ + \x20\x20\x20\x20close out\n\ + \x20\x20\x20\x20break\n\ + end loop\n\ + store contents as read content from out\n"; + let errors = + typecheck(source).expect_err("the outer Text binding must remain Text after the loop"); + assert!( + errors.contains("Expected a File object"), + "expected the outer Text/File diagnostic, got: {errors}" + ); +} From 370073e4431af2e3cbad7273bace3ee0ff307e9d Mon Sep 17 00:00:00 2001 From: Brad Byrd Date: Sat, 25 Jul 2026 00:09:50 -0500 Subject: [PATCH 107/132] fix: recreate local file handle types --- src/typechecker/mod.rs | 14 +++++++++++--- 1 file changed, 11 insertions(+), 3 deletions(-) diff --git a/src/typechecker/mod.rs b/src/typechecker/mod.rs index e38404be..2557756e 100644 --- a/src/typechecker/mod.rs +++ b/src/typechecker/mod.rs @@ -1763,9 +1763,17 @@ impl TypeChecker { ); } - if let Some(symbol) = self.analyzer.get_symbol_mut(variable_name) { - symbol.symbol_type = Some(Type::Custom("File".to_string())); - } + // Runtime binds the opened handle in the current environment. + // Analyzer body scopes are discarded before this pass, so + // recreate the local symbol here and shadow (rather than + // parent-walk/retype) any outer binding with the same name. + self.analyzer.define_or_replace_symbol(Symbol { + name: variable_name.clone(), + kind: SymbolKind::Variable { mutable: true }, + symbol_type: Some(Type::Custom("File".to_string())), + line: *_line, + column: *_column, + }); } Statement::ReadFileStatement { path, From f0dc05db5b2c6722a3a400e629544295a3b07609 Mon Sep 17 00:00:00 2001 From: Brad Byrd Date: Sat, 25 Jul 2026 00:10:40 -0500 Subject: [PATCH 108/132] test: exercise asynchronous file read type --- tests/open_file_local_type_test.rs | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/tests/open_file_local_type_test.rs b/tests/open_file_local_type_test.rs index 080ffc89..7a036880 100644 --- a/tests/open_file_local_type_test.rs +++ b/tests/open_file_local_type_test.rs @@ -67,7 +67,7 @@ fn opening_a_local_file_shadows_instead_of_retyping_an_outer_binding() { \x20\x20\x20\x20close out\n\ \x20\x20\x20\x20break\n\ end loop\n\ - store contents as read content from out\n"; + wait for store contents as read content from out\n"; let errors = typecheck(source).expect_err("the outer Text binding must remain Text after the loop"); assert!( From 85768c2384b2e414fe66c26751b28f12f2614890 Mon Sep 17 00:00:00 2001 From: Brad Byrd Date: Sat, 25 Jul 2026 00:11:25 -0500 Subject: [PATCH 109/132] test: use unambiguous handle collision guard --- tests/open_file_local_type_test.rs | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/tests/open_file_local_type_test.rs b/tests/open_file_local_type_test.rs index 7a036880..a3071cd4 100644 --- a/tests/open_file_local_type_test.rs +++ b/tests/open_file_local_type_test.rs @@ -67,11 +67,11 @@ fn opening_a_local_file_shadows_instead_of_retyping_an_outer_binding() { \x20\x20\x20\x20close out\n\ \x20\x20\x20\x20break\n\ end loop\n\ - wait for store contents as read content from out\n"; + close out\n"; let errors = typecheck(source).expect_err("the outer Text binding must remain Text after the loop"); assert!( - errors.contains("Expected a File object"), - "expected the outer Text/File diagnostic, got: {errors}" + errors.contains("file or stream handle") || errors.contains("File"), + "expected the outer Text/handle diagnostic, got: {errors}" ); } From 0e98fe35415abe1e067293edfaa47a4509446303 Mon Sep 17 00:00:00 2001 From: Brad Byrd Date: Sat, 25 Jul 2026 00:12:13 -0500 Subject: [PATCH 110/132] test: expose real file handle write fallback --- tests/open_file_local_type_test.rs | 40 ++++++++++++++++++++++++++++++ 1 file changed, 40 insertions(+) diff --git a/tests/open_file_local_type_test.rs b/tests/open_file_local_type_test.rs index a3071cd4..822ec5e8 100644 --- a/tests/open_file_local_type_test.rs +++ b/tests/open_file_local_type_test.rs @@ -1,6 +1,9 @@ //! Regression coverage for local `open file ... as ...` bindings that analyzer //! body scopes do not retain for the type-checker pass. +use std::fs; +use std::process::Command; +use tempfile::TempDir; use wfl::lexer::lex_wfl_with_positions; use wfl::parser::Parser; use wfl::typechecker::TypeChecker; @@ -75,3 +78,40 @@ fn opening_a_local_file_shadows_instead_of_retyping_an_outer_binding() { "expected the outer Text/handle diagnostic, got: {errors}" ); } + +#[test] +fn ambiguous_write_uses_the_classic_branch_for_a_real_open_file_handle() { + let temp = TempDir::new().expect("tempdir"); + let program_path = temp.path().join("program.wfl"); + let output_path = temp.path().join("actual-output.txt"); + let wfl_output_path = output_path.to_string_lossy().replace('\\', "/"); + let source = format!( + "define action called dump:\n\ + \x20\x20\x20\x20open file at \"{wfl_output_path}\" for writing as out\n\ + \x20\x20\x20\x20store line value as \"classic\"\n\ + \x20\x20\x20\x20write line value to out\n\ + \x20\x20\x20\x20close out\n\ + end action\n\ + call dump\n" + ); + fs::write(&program_path, source).expect("write WFL program"); + + let output = Command::new(env!("CARGO_BIN_EXE_wfl")) + .arg(&program_path) + .output() + .expect("run WFL program"); + let combined = format!( + "{}{}", + String::from_utf8_lossy(&output.stdout), + String::from_utf8_lossy(&output.stderr) + ); + assert!( + output.status.success(), + "a real opened File handle must select the classic write branch; output:\n{combined}" + ); + assert_eq!( + fs::read_to_string(&output_path).expect("read output file"), + "classic", + "the classic fallback content must be written to the opened handle" + ); +} From 5bef23578d0c315dd12b5613e63f6c9192d4e79a Mon Sep 17 00:00:00 2001 From: Brad Byrd Date: Sat, 25 Jul 2026 00:13:21 -0500 Subject: [PATCH 111/132] test: expose stale deadline after clean eof --- src/interpreter/mod.rs | 60 ++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 60 insertions(+) diff --git a/src/interpreter/mod.rs b/src/interpreter/mod.rs index 9df6da87..0ddc3c05 100644 --- a/src/interpreter/mod.rs +++ b/src/interpreter/mod.rs @@ -14394,8 +14394,11 @@ mod outbound_stream_deadline_tests { } let request = String::from_utf8_lossy(&request); let truncated = request.starts_with("GET /truncated "); + let unterminated = request.starts_with("GET /unterminated "); let response = if truncated { "HTTP/1.1 200 OK\r\nContent-Length: 10\r\nConnection: close\r\n\r\nx" + } else if unterminated { + "HTTP/1.1 200 OK\r\nContent-Length: 3\r\nConnection: close\r\n\r\nabc" } else { "HTTP/1.1 200 OK\r\nContent-Length: 0\r\nConnection: close\r\n\r\n" }; @@ -14459,6 +14462,63 @@ mod outbound_stream_deadline_tests { ); } + #[tokio::test] + async fn final_unterminated_line_survives_deadline_after_clean_eof() { + let port = spawn_stream_cleanup_upstream(1).await; + let config = Arc::new(WflConfig { + outbound_stream_max_seconds: 1, + timeout_seconds: 10, + ..WflConfig::default() + }); + let client = IoClient::new(Arc::clone(&config)); + let budget = Arc::new(ExecutionBudget::from_config(&config)); + let (_, _, handle) = tokio::time::timeout( + Duration::from_secs(3), + client.open_http_stream( + "GET", + &format!("http://127.0.0.1:{port}/unterminated"), + &[], + None, + Arc::clone(&budget), + ), + ) + .await + .expect("open stream hung") + .expect("open stream"); + + let first = tokio::time::timeout( + Duration::from_secs(3), + client.next_line(&handle, Arc::clone(&budget)), + ) + .await + .expect("first line read hung") + .expect("first line read"); + assert_eq!( + first.as_deref(), + Some("abc"), + "the final unterminated line is returned only after clean EOF was observed" + ); + + tokio::time::sleep(Duration::from_millis(1_100)).await; + let eof = tokio::time::timeout( + Duration::from_secs(2), + client.next_line(&handle, Arc::clone(&budget)), + ) + .await + .expect("clean EOF read hung") + .expect("clean EOF observed before the cap must not become Timeout"); + assert_eq!(eof, None); + + let later = client + .next_line(&handle, budget) + .await + .expect_err("the single clean-EOF result must consume the handle"); + assert!( + matches!(&later, HttpClientError::Request(message) if message.contains("already-closed")), + "a later read must retain the established closed-handle error, got {later:?}" + ); + } + #[tokio::test] async fn eof_error_and_rapid_close_cancel_reaper_tasks_on_a_retained_runtime() { const RAPID_CLOSES: usize = 40; From af800a7dfe44a6188b591aebfd4f8211d51719e8 Mon Sep 17 00:00:00 2001 From: Brad Byrd Date: Sat, 25 Jul 2026 00:14:28 -0500 Subject: [PATCH 112/132] fix: preserve clean eof after stream deadline --- src/interpreter/mod.rs | 15 ++++++++++----- 1 file changed, 10 insertions(+), 5 deletions(-) diff --git a/src/interpreter/mod.rs b/src/interpreter/mod.rs index 0ddc3c05..c59f10a2 100644 --- a/src/interpreter/mod.rs +++ b/src/interpreter/mod.rs @@ -2309,7 +2309,7 @@ impl IoClient { fn put_stream( &self, handle_id: &str, - handle: HttpStreamHandle, + mut handle: HttpStreamHandle, cancel: &StreamCancel, ) -> Result<(), HttpClientError> { if let Some(terminal) = cancel.terminal() { @@ -2352,10 +2352,15 @@ impl IoClient { // `nothing`, matching the established WFL stream contract. Retain the // exhausted handle for that one read, but abort its timer immediately // so EOF never leaves a sleeping reaper task. - if handle.done - && let Some(abort) = slot.reaper_abort.take() - { - abort.abort(); + if handle.done { + // Clean EOF already won before the absolute deadline. Preserve the + // one established follow-up `nothing` read without letting the old + // wall-clock cap retroactively turn that EOF into Timeout. + handle.total_deadline = None; + slot.deadline = None; + if let Some(abort) = slot.reaper_abort.take() { + abort.abort(); + } } slot.handle = Some(handle); Ok(()) From 5d8fa3d6775145f8f63a4684f365f5f2e95c55c4 Mon Sep 17 00:00:00 2001 From: Brad Byrd Date: Sat, 25 Jul 2026 00:20:14 -0500 Subject: [PATCH 113/132] test: expose unbounded expired stream metadata --- src/interpreter/mod.rs | 122 +++++++++++++++++++++++++++++++++++++++++ 1 file changed, 122 insertions(+) diff --git a/src/interpreter/mod.rs b/src/interpreter/mod.rs index c59f10a2..a509b7b4 100644 --- a/src/interpreter/mod.rs +++ b/src/interpreter/mod.rs @@ -14418,6 +14418,53 @@ mod outbound_stream_deadline_tests { port } + async fn spawn_stalled_streams( + expected_requests: usize, + ) -> (u16, tokio::sync::mpsc::UnboundedReceiver<()>) { + let listener = tokio::net::TcpListener::bind("127.0.0.1:0") + .await + .expect("bind terminal-retention upstream"); + let port = listener.local_addr().expect("upstream address").port(); + let (peer_closed_tx, peer_closed_rx) = tokio::sync::mpsc::unbounded_channel(); + tokio::spawn(async move { + for _ in 0..expected_requests { + let (mut socket, _) = listener + .accept() + .await + .expect("accept terminal-retention request"); + let peer_closed_tx = peer_closed_tx.clone(); + tokio::spawn(async move { + let mut request = Vec::new(); + let mut buffer = [0u8; 512]; + while !request.windows(4).any(|window| window == b"\r\n\r\n") { + let read = socket.read(&mut buffer).await.expect("read request head"); + if read == 0 { + return; + } + request.extend_from_slice(&buffer[..read]); + } + socket + .write_all( + b"HTTP/1.1 200 OK\r\nContent-Length: 1048576\r\n\ + Connection: close\r\n\r\n", + ) + .await + .expect("write stalled response head"); + socket.flush().await.expect("flush stalled response head"); + + loop { + match socket.read(&mut buffer).await { + Ok(0) | Err(_) => break, + Ok(_) => {} + } + } + let _ = peer_closed_tx.send(()); + }); + } + }); + (port, peer_closed_rx) + } + async fn assert_reapers_drained(client: &IoClient) { tokio::time::timeout(Duration::from_secs(2), async { loop { @@ -14524,6 +14571,81 @@ mod outbound_stream_deadline_tests { ); } + #[tokio::test] + async fn unread_expired_stream_metadata_and_ownership_are_bounded() { + const EXPECTED_RECENT_TIMEOUT_CAPACITY: usize = 64; + const STREAM_COUNT: usize = EXPECTED_RECENT_TIMEOUT_CAPACITY + 8; + + let (port, mut peer_closed) = spawn_stalled_streams(STREAM_COUNT).await; + let config = Arc::new(WflConfig { + outbound_stream_max_seconds: 2, + timeout_seconds: 10, + ..WflConfig::default() + }); + let interpreter = Interpreter::with_config(Arc::clone(&config)); + let budget = Arc::new(ExecutionBudget::from_config(&config)); + let mut handles = Vec::with_capacity(STREAM_COUNT); + + for sequence in 0..STREAM_COUNT { + let (_, _, handle) = interpreter + .io_client + .open_http_stream( + "GET", + &format!("http://127.0.0.1:{port}/stall/{sequence}"), + &[], + None, + Arc::clone(&budget), + ) + .await + .expect("open stalled stream"); + interpreter + .open_http_streams + .borrow_mut() + .push(handle.clone()); + handles.push(handle); + } + + tokio::time::timeout(Duration::from_secs(5), async { + for _ in 0..STREAM_COUNT { + peer_closed + .recv() + .await + .expect("upstream close notification"); + } + }) + .await + .expect("expired stream bodies were not dropped promptly"); + assert_reapers_drained(&interpreter.io_client).await; + + let retained_slots = interpreter + .io_client + .stream_handles + .lock() + .unwrap_or_else(|error| error.into_inner()) + .len(); + let retained_ownership = interpreter.open_http_streams.borrow().len(); + assert!( + retained_slots <= EXPECTED_RECENT_TIMEOUT_CAPACITY, + "expired terminal slots must have a hard ceiling of \ + {EXPECTED_RECENT_TIMEOUT_CAPACITY}, got {retained_slots}" + ); + assert_eq!( + retained_ownership, 0, + "the reaper must remove expired ids from handler ownership immediately" + ); + + let newest = handles.last().expect("newest stream"); + let recent = interpreter + .io_client + .next_chunk(newest, budget) + .await + .expect_err("a recent expired stream must retain its typed terminal"); + assert!( + matches!(recent, HttpClientError::Timeout { seconds: 2 }), + "a recent read-after-expiry must report Timeout, got {recent:?}" + ); + } + #[tokio::test] async fn eof_error_and_rapid_close_cancel_reaper_tasks_on_a_retained_runtime() { const RAPID_CLOSES: usize = 40; From c7f57b9594a7286d57692efa066502fd6c08c16e Mon Sep 17 00:00:00 2001 From: Brad Byrd Date: Sat, 25 Jul 2026 00:27:04 -0500 Subject: [PATCH 114/132] fix: bound expired outbound stream state --- src/interpreter/mod.rs | 347 +++++++++++++++++++++++++++++------------ 1 file changed, 249 insertions(+), 98 deletions(-) diff --git a/src/interpreter/mod.rs b/src/interpreter/mod.rs index a509b7b4..eafb33c0 100644 --- a/src/interpreter/mod.rs +++ b/src/interpreter/mod.rs @@ -53,7 +53,7 @@ use crate::parser::ast::{ use crate::pattern::CompiledPattern; use crate::stdlib; use std::cell::{Cell, RefCell}; -use std::collections::HashMap; +use std::collections::{HashMap, HashSet, VecDeque}; use std::io::{self, Write}; use std::net::IpAddr; use std::path::PathBuf; @@ -841,7 +841,7 @@ struct RunState { /// exit) these are dropped from `IoClient.stream_handles`, which cancels the /// in-flight upstream request — so an abandoned proxy read never leaks an /// upstream connection or handle past the handler's lifetime. - open_http_streams: Vec, + open_http_streams: StreamOwner, /// Sticky: this handler successfully dequeued at least one request via /// `wait for request`. Used by the concurrent main loop to distinguish /// structural pre-request failures (feed the consecutive-failure breaker) @@ -944,7 +944,7 @@ impl<'a, T> Drop for IsolatedHandler<'a, T> { /// `Drop`). struct OutboundStreamCleanup { io_client: Rc, - open_http_streams: Rc>>, + open_http_streams: StreamOwner, open_response_streams: Rc>>, server_response_streams: Rc>>, open_pending_requests: Rc>>, @@ -955,23 +955,7 @@ impl Drop for OutboundStreamCleanup { fn drop(&mut self) { // Outbound upstream streams: removing a handle drops its reqwest stream, // cancelling the in-flight upstream request. - let http_ids = std::mem::take(&mut *self.open_http_streams.borrow_mut()); - if !http_ids.is_empty() { - let mut map = self - .io_client - .stream_handles - .lock() - .unwrap_or_else(|e| e.into_inner()); - for id in &http_ids { - if let Some(mut slot) = map.remove(id) { - slot.cancel.terminate(StreamTerminal::Closed); - if let Some(abort) = slot.reaper_abort.take() { - abort.abort(); - } - drop(slot.handle.take()); - } - } - } + self.io_client.close_stream_owner(&self.open_http_streams); // Server response streams: dropping the sender ends the client's body. let stream_ids = std::mem::take(&mut *self.open_response_streams.borrow_mut()); @@ -1409,7 +1393,7 @@ pub struct Interpreter { /// the `interpret()` future can share the list and close these handles if the /// future is dropped/cancelled before its normal exit sites run (see /// `OutboundStreamCleanup`). - open_http_streams: Rc>>, + open_http_streams: Rc>, /// Sticky per-handler flag: at least one request was dequeued and parked. /// Part of `RunState` (swapped per poll); see `RunState::accepted_request`. accepted_request: Cell, @@ -1718,7 +1702,7 @@ pub struct IoClient { /// async mutex only offers `try_lock` from sync Drop, which previously /// abandoned handles when the map was briefly held). Critical sections are /// short (no `.await` while held). - stream_handles: Arc>>, + stream_handles: Arc>, next_stream_id: Mutex, /// Test-only live-task accounting. The production build carries no /// instrumentation; unit tests retain a runtime and assert that closing a @@ -1813,6 +1797,63 @@ enum StreamTerminal { Closed, } +type StreamOwner = Arc>>; + +/// Keep a short, bounded window of typed terminal outcomes after a reaper has +/// removed the live body. This lets the next read report `Timeout` without +/// retaining the request body, cancel channel, owner, or sleeping task. +const MAX_RECENT_STREAM_TERMINALS: usize = 64; +const RECENT_STREAM_TERMINAL_TTL: Duration = Duration::from_secs(60); + +#[derive(Default)] +struct StreamRegistry { + live: HashMap, + recent: VecDeque, +} + +struct RecentStreamTerminal { + id: String, + reason: StreamTerminal, + expires_at: Instant, +} + +impl StreamRegistry { + fn prune_recent(&mut self, now: Instant) { + while self + .recent + .front() + .is_some_and(|entry| entry.expires_at <= now) + { + self.recent.pop_front(); + } + } + + fn remember_recent(&mut self, id: String, reason: StreamTerminal, now: Instant) { + self.prune_recent(now); + self.recent.retain(|entry| entry.id != id); + while self.recent.len() >= MAX_RECENT_STREAM_TERMINALS { + self.recent.pop_front(); + } + self.recent.push_back(RecentStreamTerminal { + id, + reason, + expires_at: now + RECENT_STREAM_TERMINAL_TTL, + }); + } + + fn take_recent(&mut self, id: &str, now: Instant) -> Option { + self.prune_recent(now); + let index = self.recent.iter().position(|entry| entry.id == id)?; + self.recent.remove(index).map(|entry| entry.reason) + } + + fn forget_recent(&mut self, id: &str) -> bool { + let before = self.recent.len(); + self.recent.retain(|entry| entry.id != id); + self.recent.len() != before + } +} + /// Per-handle shared lifecycle for an outbound stream. /// /// Reads take the inner [`HttpStreamHandle`] out for the duration of the await @@ -1830,6 +1871,18 @@ struct StreamSlot { /// Abort handle for the reaper timer. Cancelled on EOF, error, or explicit /// close so rapid open/close cycles do not accumulate sleeping tasks. reaper_abort: Option, + /// Handler ownership is stored with the live slot so the reaper can remove + /// the id immediately. Recent terminal records never retain an owner. + owner: Option, +} + +fn remove_stream_owner(slot: &mut StreamSlot, handle_id: &str) { + if let Some(owner) = slot.owner.take() { + owner + .lock() + .unwrap_or_else(|error| error.into_inner()) + .remove(handle_id); + } } /// A live, parked outbound streaming response body. @@ -1979,7 +2032,7 @@ impl IoClient { next_process_id: Mutex::new(1), db_handles: Mutex::new(HashMap::new()), next_db_id: Mutex::new(1), - stream_handles: Arc::new(std::sync::Mutex::new(HashMap::new())), + stream_handles: Arc::new(std::sync::Mutex::new(StreamRegistry::default())), next_stream_id: Mutex::new(1), #[cfg(test)] active_stream_reapers: Arc::new(std::sync::atomic::AtomicUsize::new(0)), @@ -2186,17 +2239,19 @@ impl IoClient { // races with open still sees a consistent cancel handle). let cancel = StreamCancel::new(); { - let mut map = self + let mut registry = self .stream_handles .lock() .unwrap_or_else(|e| e.into_inner()); - map.insert( + registry.prune_recent(Instant::now()); + registry.live.insert( handle_id.clone(), StreamSlot { handle: Some(handle), deadline: total_deadline, cancel: Arc::clone(&cancel), reaper_abort: None, + owner: None, }, ); if let Some(deadline) = total_deadline { @@ -2211,18 +2266,20 @@ impl IoClient { let _reaper_guard = reaper_guard; let remaining = deadline.saturating_duration_since(Instant::now()); tokio::time::sleep(remaining).await; - // Preserve a Timeout tombstone in the stable slot. This - // wakes an active read with the typed reason and lets a - // later read of an unread expired handle report Timeout - // instead of "unknown handle". - let mut map = handles.lock().unwrap_or_else(|e| e.into_inner()); - if let Some(slot) = map.get_mut(&reap_id) { - cancel_reap.terminate(StreamTerminal::Timeout); + // Remove the heavy live slot and preserve only a bounded, + // lightweight typed terminal record for one later read. + let now = Instant::now(); + let mut registry = handles.lock().unwrap_or_else(|e| e.into_inner()); + registry.prune_recent(now); + if let Some(mut slot) = registry.live.remove(&reap_id) { + let terminal = cancel_reap.terminate(StreamTerminal::Timeout); slot.reaper_abort = None; // we are the reaper drop(slot.handle.take()); + remove_stream_owner(&mut slot, &reap_id); + registry.remember_recent(reap_id, terminal, now); } }); - if let Some(slot) = map.get_mut(&handle_id) { + if let Some(slot) = registry.live.get_mut(&handle_id) { slot.reaper_abort = Some(join.abort_handle()); } else { // Already finished before we armed — cancel the timer. @@ -2234,23 +2291,83 @@ impl IoClient { Ok((status, response_headers, handle_id)) } + /// Atomically attach a handler owner to a freshly opened stream. If expiry + /// won the race, return its typed outcome without creating stale ownership. + fn claim_stream_owner( + &self, + handle_id: &str, + owner: &StreamOwner, + ) -> Result<(), HttpClientError> { + let now = Instant::now(); + let mut registry = self + .stream_handles + .lock() + .unwrap_or_else(|error| error.into_inner()); + registry.prune_recent(now); + if let Some(slot) = registry.live.get_mut(handle_id) { + owner + .lock() + .unwrap_or_else(|error| error.into_inner()) + .insert(handle_id.to_string()); + slot.owner = Some(Arc::clone(owner)); + return Ok(()); + } + if let Some(terminal) = registry.take_recent(handle_id, now) { + return Err(self.stream_terminal_error(terminal)); + } + Err(HttpClientError::Closed) + } + + /// Close every live stream owned by one handler. The owner lock is released + /// before the registry lock is acquired, preserving the registry->owner + /// nesting order used by the reaper. + fn close_stream_owner(&self, owner: &StreamOwner) { + let ids: Vec = { + let mut owned = owner.lock().unwrap_or_else(|error| error.into_inner()); + owned.drain().collect() + }; + if ids.is_empty() { + return; + } + + let mut registry = self + .stream_handles + .lock() + .unwrap_or_else(|error| error.into_inner()); + registry.prune_recent(Instant::now()); + for id in ids { + if let Some(mut slot) = registry.live.remove(&id) { + slot.cancel.terminate(StreamTerminal::Closed); + if let Some(abort) = slot.reaper_abort.take() { + abort.abort(); + } + drop(slot.handle.take()); + remove_stream_owner(&mut slot, &id); + } + registry.forget_recent(&id); + } + } + /// Signal cancel, abort the reaper, drop any parked handle, and remove the /// slot. Guaranteed (std mutex) — usable from Drop. Returns whether a slot /// was present. fn finish_stream_slot_sync(&self, handle_id: &str, terminal: StreamTerminal) -> bool { - let mut map = self + let mut registry = self .stream_handles .lock() .unwrap_or_else(|e| e.into_inner()); - if let Some(mut slot) = map.remove(handle_id) { + registry.prune_recent(Instant::now()); + let had_recent = registry.forget_recent(handle_id); + if let Some(mut slot) = registry.live.remove(handle_id) { slot.cancel.terminate(terminal); if let Some(abort) = slot.reaper_abort.take() { abort.abort(); } drop(slot.handle.take()); + remove_stream_owner(&mut slot, handle_id); true } else { - false + had_recent } } @@ -2262,39 +2379,56 @@ impl IoClient { /// holding the global handle lock. The cancel watch stays alive so close/ /// expire aborts the read. Errors if unknown, closed, or past deadline. fn take_stream(&self, handle_id: &str) -> Result { - let mut map = self + let now = Instant::now(); + let mut registry = self .stream_handles .lock() .unwrap_or_else(|e| e.into_inner()); - let Some(slot) = map.get_mut(handle_id) else { + registry.prune_recent(now); + if !registry.live.contains_key(handle_id) { + if let Some(terminal) = registry.take_recent(handle_id, now) { + return Err(self.stream_terminal_error(terminal)); + } return Err(HttpClientError::Request(format!( "Unknown or already-closed stream handle '{handle_id}'" ))); - }; - if let Some(terminal) = slot.cancel.terminal() { - // Consume the stable terminal tombstone. - if let Some(mut slot) = map.remove(handle_id) { + } + if let Some(terminal) = registry + .live + .get(handle_id) + .and_then(|slot| slot.cancel.terminal()) + { + if let Some(mut slot) = registry.live.remove(handle_id) { if let Some(abort) = slot.reaper_abort.take() { abort.abort(); } drop(slot.handle.take()); + remove_stream_owner(&mut slot, handle_id); } return Err(self.stream_terminal_error(terminal)); } - let past_deadline = slot + let past_deadline = registry + .live + .get(handle_id) + .expect("live stream checked above") .deadline - .is_some_and(|d| d.saturating_duration_since(Instant::now()).is_zero()); + .is_some_and(|deadline| deadline <= now); if past_deadline { - if let Some(mut slot) = map.remove(handle_id) { + if let Some(mut slot) = registry.live.remove(handle_id) { let terminal = slot.cancel.terminate(StreamTerminal::Timeout); if let Some(abort) = slot.reaper_abort.take() { abort.abort(); } drop(slot.handle.take()); + remove_stream_owner(&mut slot, handle_id); return Err(self.stream_terminal_error(terminal)); } return Err(self.outbound_stream_timeout_error()); } + let slot = registry + .live + .get_mut(handle_id) + .expect("live stream checked above"); let cancel = Arc::clone(&slot.cancel); match slot.handle.take() { Some(handle) => Ok(TakenStream { handle, cancel }), @@ -2318,36 +2452,45 @@ impl IoClient { let _ = self.finish_stream_slot_sync(handle_id, terminal); return Err(self.stream_terminal_error(terminal)); } - let mut map = self + let now = Instant::now(); + let mut registry = self .stream_handles .lock() .unwrap_or_else(|e| e.into_inner()); - let Some(slot) = map.get_mut(handle_id) else { + registry.prune_recent(now); + if !registry.live.contains_key(handle_id) { drop(handle); - return Err(cancel + let terminal = cancel .terminal() - .map(|terminal| self.stream_terminal_error(terminal)) + .or_else(|| registry.take_recent(handle_id, now)); + return Err(terminal + .map(|reason| self.stream_terminal_error(reason)) .unwrap_or(HttpClientError::Closed)); - }; - let terminal = slot.cancel.terminal().or_else(|| { - if slot - .deadline - .is_some_and(|d| d.saturating_duration_since(Instant::now()).is_zero()) - { - Some(slot.cancel.terminate(StreamTerminal::Timeout)) - } else { - None - } + } + let terminal = registry.live.get(handle_id).and_then(|slot| { + slot.cancel.terminal().or_else(|| { + if slot.deadline.is_some_and(|deadline| deadline <= now) { + Some(slot.cancel.terminate(StreamTerminal::Timeout)) + } else { + None + } + }) }); if let Some(terminal) = terminal { drop(handle); - if let Some(mut slot) = map.remove(handle_id) - && let Some(abort) = slot.reaper_abort.take() - { - abort.abort(); + if let Some(mut slot) = registry.live.remove(handle_id) { + if let Some(abort) = slot.reaper_abort.take() { + abort.abort(); + } + drop(slot.handle.take()); + remove_stream_owner(&mut slot, handle_id); } return Err(self.stream_terminal_error(terminal)); } + let slot = registry + .live + .get_mut(handle_id) + .expect("live stream checked above"); // A final unterminated line needs one subsequent read to produce // `nothing`, matching the established WFL stream contract. Retain the // exhausted handle for that one read, but abort its timer immediately @@ -3889,7 +4032,9 @@ impl Interpreter { server_response_streams: Rc::new(RefCell::new(HashMap::new())), open_response_streams: Rc::new(RefCell::new(Vec::new())), open_pending_requests: Rc::new(RefCell::new(Vec::new())), - open_http_streams: Rc::new(RefCell::new(Vec::new())), + open_http_streams: Rc::new(RefCell::new(Arc::new(std::sync::Mutex::new( + HashSet::new(), + )))), accepted_request: Cell::new(false), next_response_stream_id: std::cell::Cell::new(1), config, @@ -4468,25 +4613,8 @@ impl Interpreter { /// synchronous (usable from `Drop`): if the async lock is momentarily held, /// the handles remain and are reclaimed at interpreter teardown. Idempotent — /// an id already removed by EOF/error/explicit `close` is a no-op. - fn close_http_streams(&self, ids: &[String]) { - if ids.is_empty() { - return; - } - // std mutex: guaranteed cleanup from Drop (no silent try_lock abandon). - let mut map = self - .io_client - .stream_handles - .lock() - .unwrap_or_else(|e| e.into_inner()); - for id in ids { - if let Some(mut slot) = map.remove(id) { - slot.cancel.terminate(StreamTerminal::Closed); - if let Some(abort) = slot.reaper_abort.take() { - abort.abort(); - } - drop(slot.handle.take()); - } - } + fn close_http_streams(&self, owner: &StreamOwner) { + self.io_client.close_stream_owner(owner); } /// Build an RAII guard that closes any outbound stream handles still tracked @@ -4498,7 +4626,7 @@ impl Interpreter { fn outbound_stream_cleanup_guard(&self) -> OutboundStreamCleanup { OutboundStreamCleanup { io_client: Rc::clone(&self.io_client), - open_http_streams: Rc::clone(&self.open_http_streams), + open_http_streams: Arc::clone(&self.open_http_streams.borrow()), open_response_streams: Rc::clone(&self.open_response_streams), server_response_streams: Rc::clone(&self.server_response_streams), open_pending_requests: Rc::clone(&self.open_pending_requests), @@ -4510,8 +4638,8 @@ impl Interpreter { /// open. Called at the end of each serial `main loop` iteration and at program /// exit, mirroring the concurrent path's per-handler `Drop`. fn close_open_http_streams(&self) { - let ids = std::mem::take(&mut *self.open_http_streams.borrow_mut()); - self.close_http_streams(&ids); + let owner = Arc::clone(&self.open_http_streams.borrow()); + self.close_http_streams(&owner); } /// Stop tracking an outbound stream id as handler-owned — it has already left @@ -4520,7 +4648,9 @@ impl Interpreter { fn untrack_http_stream(&self, handle_id: &str) { self.open_http_streams .borrow_mut() - .retain(|id| id != handle_id); + .lock() + .unwrap_or_else(|error| error.into_inner()) + .remove(handle_id); } /// Clone the sender of every downstream response stream this handler owns. A @@ -7915,7 +8045,10 @@ impl Interpreter { // Track the outbound handle as handler-owned so it is // dropped (cancelling the upstream) if the handler ends // without closing/exhausting it. - self.open_http_streams.borrow_mut().push(handle_id.clone()); + let owner = Arc::clone(&self.open_http_streams.borrow()); + self.io_client + .claim_stream_owner(&handle_id, &owner) + .map_err(|error| self.http_client_error(error, *line, *column))?; let mut headers_map = HashMap::new(); for (name, value) in response_headers { headers_map.insert(name, Value::Text(value.into())); @@ -14599,9 +14732,12 @@ mod outbound_stream_deadline_tests { .await .expect("open stalled stream"); interpreter - .open_http_streams - .borrow_mut() - .push(handle.clone()); + .io_client + .claim_stream_owner( + &handle, + &Arc::clone(&interpreter.open_http_streams.borrow()), + ) + .expect("claim stalled stream ownership"); handles.push(handle); } @@ -14617,17 +14753,28 @@ mod outbound_stream_deadline_tests { .expect("expired stream bodies were not dropped promptly"); assert_reapers_drained(&interpreter.io_client).await; - let retained_slots = interpreter - .io_client - .stream_handles + let (live_slots, retained_terminals) = { + let registry = interpreter + .io_client + .stream_handles + .lock() + .unwrap_or_else(|error| error.into_inner()); + (registry.live.len(), registry.recent.len()) + }; + let retained_ownership = interpreter + .open_http_streams + .borrow() .lock() .unwrap_or_else(|error| error.into_inner()) .len(); - let retained_ownership = interpreter.open_http_streams.borrow().len(); + assert_eq!( + live_slots, 0, + "expired stream bodies must leave no live registry slots" + ); assert!( - retained_slots <= EXPECTED_RECENT_TIMEOUT_CAPACITY, - "expired terminal slots must have a hard ceiling of \ - {EXPECTED_RECENT_TIMEOUT_CAPACITY}, got {retained_slots}" + retained_terminals <= EXPECTED_RECENT_TIMEOUT_CAPACITY, + "recent terminal records must have a hard ceiling of \ + {EXPECTED_RECENT_TIMEOUT_CAPACITY}, got {retained_terminals}" ); assert_eq!( retained_ownership, 0, @@ -14721,6 +14868,7 @@ mod outbound_stream_deadline_tests { .stream_handles .lock() .unwrap_or_else(|error| error.into_inner()) + .live .is_empty(), "EOF, error, and explicit close must remove every stream slot" ); @@ -14738,6 +14886,7 @@ mod outbound_stream_deadline_tests { .stream_handles .lock() .unwrap_or_else(|error| error.into_inner()) + .live .insert( handle_id.to_string(), StreamSlot { @@ -14753,6 +14902,7 @@ mod outbound_stream_deadline_tests { deadline: outbound_stream_deadline(1), cancel: Arc::clone(&cancel), reaper_abort: None, + owner: None, }, ); @@ -14771,6 +14921,7 @@ mod outbound_stream_deadline_tests { .stream_handles .lock() .unwrap_or_else(|error| error.into_inner()) + .live .contains_key(handle_id), "an expired handle must never be reinserted after its active read" ); From 00a2a3fa5f60bf414ac211b2c76d546f784b0d49 Mon Sep 17 00:00:00 2001 From: Brad Byrd Date: Sat, 25 Jul 2026 00:28:02 -0500 Subject: [PATCH 115/132] refactor: satisfy formatter and lint gates --- src/parser/stmt/io.rs | 5 +---- src/typechecker/mod.rs | 17 +++++++---------- tests/write_web_postfix_test.rs | 17 ++++++++--------- 3 files changed, 16 insertions(+), 23 deletions(-) diff --git a/src/parser/stmt/io.rs b/src/parser/stmt/io.rs index c339460d..f54049e6 100644 --- a/src/parser/stmt/io.rs +++ b/src/parser/stmt/io.rs @@ -41,10 +41,7 @@ impl<'a> Parser<'a> { } else { self.parse_trailing_postfix(lead)? }; - if !matches!( - self.cursor.peek().map(|t| &t.token), - Some(Token::KeywordOf) - ) { + if !matches!(self.cursor.peek().map(|t| &t.token), Some(Token::KeywordOf)) { break; } diff --git a/src/typechecker/mod.rs b/src/typechecker/mod.rs index 2557756e..8b5ed0ad 100644 --- a/src/typechecker/mod.rs +++ b/src/typechecker/mod.rs @@ -219,8 +219,8 @@ impl TypeChecker { } } - for layer_index in 0..joined.len() { - let names: Vec = joined[layer_index].keys().cloned().collect(); + for (layer_index, joined_layer) in joined.iter_mut().enumerate() { + let names: Vec = joined_layer.keys().cloned().collect(); for name in names { let values: Vec> = states .iter() @@ -235,15 +235,15 @@ impl TypeChecker { let first_value = values.first().cloned().unwrap_or(None); let merged = if values.iter().all(|value| value == &first_value) { first_value - } else if values.iter().any(|value| { - value.is_none() || matches!(value.as_ref(), Some(Type::Unknown)) - }) + } else if values + .iter() + .any(|value| value.is_none() || matches!(value.as_ref(), Some(Type::Unknown))) { Some(Type::Unknown) } else { Some(Type::Any) }; - joined[layer_index].insert(name, merged); + joined_layer.insert(name, merged); } } @@ -1643,10 +1643,7 @@ impl TypeChecker { // loop-variable binding. self.analyzer.push_scope(); self.analyzer.define_or_replace_symbol(Symbol { - name: variable_name - .as_deref() - .unwrap_or("count") - .to_string(), + name: variable_name.as_deref().unwrap_or("count").to_string(), kind: SymbolKind::Variable { mutable: true }, symbol_type: Some(Type::Number), line: *_line, diff --git a/tests/write_web_postfix_test.rs b/tests/write_web_postfix_test.rs index d3c25f98..0c01b465 100644 --- a/tests/write_web_postfix_test.rs +++ b/tests/write_web_postfix_test.rs @@ -70,9 +70,9 @@ fn streaming_status_and_headers(stmt: &Statement) -> (&Expression, &Expression) headers: Some(headers), .. } => (status, headers), - other => panic!( - "expected a streaming response with status and headers operands, got {other:#?}" - ), + other => { + panic!("expected a streaming response with status and headers operands, got {other:#?}") + } } } @@ -433,7 +433,10 @@ fn type_prefixed_identifier_is_content_not_a_response_clause() { matches!(right.as_ref(), Expression::Variable(name, ..) if name == "type suffix"), "`type suffix` must remain the concatenation RHS, got {right:#?}" ); - assert_eq!(headers, "h", "the following headers clause must remain separate"); + assert_eq!( + headers, "h", + "the following headers clause must remain separate" + ); } other => panic!( "expected concatenated content type plus a separate headers clause, got {other:#?}" @@ -647,11 +650,7 @@ fn assert_post_of_index(expr: &Expression, expected_function: &str, expected_arg fn seeded_operands_resume_postfix_parsing_after_of_calls() { let write = parse("write line choose of (chunks)[0] to out\n"); assert_eq!(write.statements.len(), 1, "got {:#?}", write.statements); - assert_post_of_index( - stream_write_value(&write.statements[0]), - "choose", - "chunks", - ); + assert_post_of_index(stream_write_value(&write.statements[0]), "choose", "chunks"); assert_post_of_index( stream_write_fallback(&write.statements[0]), "line choose", From 4c45f1617097cbd39f92183bbe9dcbd986cea41d Mon Sep 17 00:00:00 2001 From: Brad Byrd Date: Sat, 25 Jul 2026 00:31:56 -0500 Subject: [PATCH 116/132] test: expose response expression disconnect stalls --- src/interpreter/mod.rs | 262 +++++++++++++++++++++++++++++++++++++++++ 1 file changed, 262 insertions(+) diff --git a/src/interpreter/mod.rs b/src/interpreter/mod.rs index eafb33c0..ad1c12d1 100644 --- a/src/interpreter/mod.rs +++ b/src/interpreter/mod.rs @@ -14507,6 +14507,268 @@ mod concurrent_handler_classification_tests { } } +#[cfg(test)] +mod response_expression_disconnect_tests { + use super::*; + use crate::lexer::lex_wfl_with_positions; + use crate::parser::Parser; + use tokio::io::{AsyncReadExt, AsyncWriteExt}; + + struct StalledUpstream { + port: u16, + head_sent: oneshot::Receiver<()>, + peer_closed: oneshot::Receiver<()>, + release: Option>, + } + + async fn spawn_stalled_upstream() -> StalledUpstream { + let listener = tokio::net::TcpListener::bind("127.0.0.1:0") + .await + .expect("bind stalled upstream"); + let port = listener.local_addr().expect("upstream address").port(); + let (head_sent_tx, head_sent) = oneshot::channel(); + let (peer_closed_tx, peer_closed) = oneshot::channel(); + let (release_tx, release_rx) = oneshot::channel(); + + tokio::spawn(async move { + let (mut socket, _) = listener.accept().await.expect("accept upstream request"); + let mut request = Vec::new(); + let mut chunk = [0u8; 1024]; + while !request.windows(4).any(|window| window == b"\r\n\r\n") { + let read = socket + .read(&mut chunk) + .await + .expect("read upstream request"); + assert!(read > 0, "client closed before sending request head"); + request.extend_from_slice(&chunk[..read]); + } + socket + .write_all( + b"HTTP/1.1 200 OK\r\n\ + Transfer-Encoding: chunked\r\n\ + Connection: close\r\n\r\n", + ) + .await + .expect("send stalled response head"); + let _ = head_sent_tx.send(()); + + let mut probe = [0u8; 1]; + tokio::select! { + result = socket.read(&mut probe) => { + assert!( + matches!(result, Ok(0) | Err(_)), + "client unexpectedly sent bytes while response body was stalled: {result:?}" + ); + let _ = peer_closed_tx.send(()); + } + _ = release_rx => { + let _ = socket.shutdown().await; + } + } + }); + + StalledUpstream { + port, + head_sent, + peer_closed, + release: Some(release_tx), + } + } + + fn parse_statements(source: &str) -> Vec { + let tokens = lex_wfl_with_positions(source); + let mut parser = Parser::new(&tokens); + parser + .parse() + .unwrap_or_else(|errors| { + panic!("response disconnect fixture did not parse: {errors:?}") + }) + .statements + } + + fn install_pending_request( + interpreter: &Interpreter, + env: &Rc>, + request_id: &str, + ) -> oneshot::Receiver { + let (sender, receiver) = oneshot::channel(); + interpreter.pending_responses.borrow_mut().insert( + request_id.to_string(), + PendingResponse { + sender: Arc::new(tokio::sync::Mutex::new(Some(sender))), + }, + ); + interpreter + .open_pending_requests + .borrow_mut() + .push(request_id.to_string()); + + let mut request = HashMap::new(); + request.insert( + "_response_sender".to_string(), + Value::Text(Arc::from(request_id)), + ); + env.borrow_mut() + .define_or_replace("req", Value::Object(Rc::new(RefCell::new(request)))); + receiver + } + + fn response_eval_is_stalled(interpreter: &Interpreter) -> bool { + let owned_streams = { + let owner = Arc::clone(&interpreter.open_http_streams.borrow()); + owner + .lock() + .unwrap_or_else(|error| error.into_inner()) + .len() + }; + owned_streams == 1 + && interpreter.get_call_stack().len() == 1 + && *interpreter.current_count.borrow() == Some(1.0) + && *interpreter.in_count_loop.borrow() + } + + async fn assert_precommit_disconnect_cancels(response_statement: &str, action_return: &str) { + let mut upstream = spawn_stalled_upstream().await; + let config = Arc::new(WflConfig { + timeout_seconds: 120, + outbound_stream_max_seconds: 120, + ..WflConfig::default() + }); + let interpreter = Interpreter::with_config(config); + let source = format!( + "define action called stalled_value:\n\ + \x20\x20\x20\x20count from 1 to 1:\n\ + \x20\x20\x20\x20\x20\x20\x20\x20open url at \"http://127.0.0.1:{}/stall\" and stream response as upstream\n\ + \x20\x20\x20\x20\x20\x20\x20\x20wait for 60000 milliseconds\n\ + \x20\x20\x20\x20end count\n\ + \x20\x20\x20\x20return {action_return}\n\ + end action\n\ + {response_statement}\n", + upstream.port + ); + let statements = parse_statements(&source); + assert_eq!( + statements.len(), + 2, + "unexpected fixture AST: {statements:#?}" + ); + let env = Rc::clone(interpreter.global_env()); + interpreter + .execute_statement(&statements[0], Rc::clone(&env)) + .await + .expect("define stalled action"); + + // Simulate an already-active outer count loop. Dropping the response + // evaluation must restore this exact run-state snapshot. + *interpreter.current_count.borrow_mut() = Some(41.0); + *interpreter.in_count_loop.borrow_mut() = true; + let receiver = install_pending_request(&interpreter, &env, "request-1"); + + let mut response = Box::pin(interpreter.execute_statement(&statements[1], env)); + let mut stalled = Box::pin(async { + loop { + if response_eval_is_stalled(&interpreter) { + return; + } + tokio::task::yield_now().await; + } + }); + tokio::time::timeout(Duration::from_secs(3), async { + tokio::select! { + result = response.as_mut() => { + panic!("response evaluation finished before the disconnect latch: {result:?}") + } + _ = stalled.as_mut() => {} + } + }) + .await + .expect("response expression never reached its stalled action"); + upstream + .head_sent + .await + .expect("stalled upstream did not send its response head"); + + // Causal trigger: only after the action owns a live upstream stream and + // is sleeping inside a count loop do we drop the client receiver. + drop(receiver); + let result = match tokio::time::timeout(Duration::from_secs(2), response.as_mut()).await { + Ok(result) => result, + Err(_) => { + drop(response); + interpreter.close_open_http_streams(); + if let Some(release) = upstream.release.take() { + let _ = release.send(()); + } + panic!( + "response expression evaluation did not cancel after its request disconnected" + ); + } + }; + drop(response); + + let error = result.expect_err("disconnect must cancel response precommit evaluation"); + assert_eq!(error.kind, ErrorKind::Cancelled, "wrong error: {error:?}"); + assert!( + interpreter.get_call_stack().is_empty(), + "dropped response evaluation leaked call frames" + ); + assert_eq!(interpreter.call_depth.get(), 0, "call depth leaked"); + assert_eq!( + *interpreter.current_count.borrow(), + Some(41.0), + "outer count binding was not restored" + ); + assert!( + *interpreter.in_count_loop.borrow(), + "outer count-loop state was not restored" + ); + assert!( + !interpreter + .pending_responses + .borrow() + .contains_key("request-1"), + "cancelled request remained pending" + ); + assert!( + !interpreter + .open_pending_requests + .borrow() + .iter() + .any(|id| id == "request-1"), + "cancelled request remained handler-owned" + ); + + tokio::time::timeout(Duration::from_secs(2), &mut upstream.peer_closed) + .await + .expect("cancelled response evaluation did not drop its upstream socket") + .expect("upstream close observation task ended early"); + assert!( + interpreter + .open_http_streams + .borrow() + .lock() + .unwrap_or_else(|error| error.into_inner()) + .is_empty(), + "cancelled response evaluation retained upstream ownership" + ); + } + + #[tokio::test] + async fn buffered_content_evaluation_cancels_on_request_disconnect() { + assert_precommit_disconnect_cancels("respond to req with call stalled_value", "\"late\"") + .await; + } + + #[tokio::test] + async fn streaming_head_evaluation_cancels_on_request_disconnect() { + assert_precommit_disconnect_cancels( + "start streaming response to req with status call stalled_value and content type \"text/plain\" as out", + "201", + ) + .await; + } +} + #[cfg(test)] mod outbound_stream_deadline_tests { use super::*; From 0d4b26b23bcd356bd62fc4de6abf89e062e0279c Mon Sep 17 00:00:00 2001 From: Brad Byrd Date: Sat, 25 Jul 2026 00:36:32 -0500 Subject: [PATCH 117/132] fix: cancel disconnected response evaluations --- src/interpreter/mod.rs | 239 +++++++++++++++++++++++++++++++++++++---- 1 file changed, 218 insertions(+), 21 deletions(-) diff --git a/src/interpreter/mod.rs b/src/interpreter/mod.rs index ad1c12d1..942ac347 100644 --- a/src/interpreter/mod.rs +++ b/src/interpreter/mod.rs @@ -186,6 +186,21 @@ struct ResponseCompletion { sender: Option>, } +/// Interpreter state that must survive cancellation of fallible response +/// expressions. The evaluation future is explicitly dropped before this state +/// is restored, so partially-entered actions and loops cannot leak into a +/// handler that catches `Cancelled` and continues. +struct ResponsePrecommitSnapshot { + call_stack: Vec, + call_depth: usize, + current_count: Option, + in_count_loop: bool, + http_owner: StreamOwner, + http_streams: HashSet, + response_streams: HashSet, + pending_requests: HashSet, +} + impl ResponseCompletion { fn take_sender(&mut self) -> Option> { self.sender.take() @@ -2326,6 +2341,12 @@ impl IoClient { let mut owned = owner.lock().unwrap_or_else(|error| error.into_inner()); owned.drain().collect() }; + self.close_stream_ids(&ids); + } + + /// Close a selected set of live streams without disturbing other handles + /// owned by the same handler. + fn close_stream_ids(&self, ids: &[String]) { if ids.is_empty() { return; } @@ -2336,15 +2357,15 @@ impl IoClient { .unwrap_or_else(|error| error.into_inner()); registry.prune_recent(Instant::now()); for id in ids { - if let Some(mut slot) = registry.live.remove(&id) { + if let Some(mut slot) = registry.live.remove(id) { slot.cancel.terminate(StreamTerminal::Closed); if let Some(abort) = slot.reaper_abort.take() { abort.abort(); } drop(slot.handle.take()); - remove_stream_owner(&mut slot, &id); + remove_stream_owner(&mut slot, id); } - registry.forget_recent(&id); + registry.forget_recent(id); } } @@ -4770,6 +4791,166 @@ impl Interpreter { self.close_response_streams(&ids); } + fn pending_response_disconnected_now(&self, request_id: &str) -> bool { + let owned = self + .open_pending_requests + .borrow() + .iter() + .any(|id| id == request_id); + if !owned { + return false; + } + + let pending = self.pending_responses.borrow(); + match pending.get(request_id) { + Some(entry) => match entry.sender.try_lock() { + Ok(sender) => sender.as_ref().is_none_or(|sender| sender.is_closed()), + Err(_) => false, + }, + None => true, + } + } + + /// Wait for one specific still-owned request to disconnect. Losing ownership + /// is deliberately not treated as cancellation: duplicate/forged responses + /// must retain their established general-error classification. + async fn pending_response_disconnected(&self, request_id: &str) { + loop { + if self.pending_response_disconnected_now(request_id) { + return; + } + if !self + .open_pending_requests + .borrow() + .iter() + .any(|id| id == request_id) + { + std::future::pending::<()>().await; + return; + } + tokio::time::sleep(REQUEST_DISCONNECT_POLL_INTERVAL).await; + } + } + + fn response_precommit_snapshot(&self) -> ResponsePrecommitSnapshot { + let http_owner = Arc::clone(&self.open_http_streams.borrow()); + let http_streams = http_owner + .lock() + .unwrap_or_else(|error| error.into_inner()) + .clone(); + ResponsePrecommitSnapshot { + call_stack: self.call_stack.borrow().clone(), + call_depth: self.call_depth.get(), + current_count: *self.current_count.borrow(), + in_count_loop: *self.in_count_loop.borrow(), + http_owner, + http_streams, + response_streams: self + .open_response_streams + .borrow() + .iter() + .cloned() + .collect(), + pending_requests: self + .open_pending_requests + .borrow() + .iter() + .cloned() + .collect(), + } + } + + /// Restore poll-local state and release only resources opened by the + /// cancelled evaluation. Existing handler resources remain owned. + fn cancel_response_precommit(&self, request_id: &str, snapshot: ResponsePrecommitSnapshot) { + *self.call_stack.borrow_mut() = snapshot.call_stack; + self.call_depth.set(snapshot.call_depth); + *self.current_count.borrow_mut() = snapshot.current_count; + *self.in_count_loop.borrow_mut() = snapshot.in_count_loop; + + let new_http_streams: Vec = snapshot + .http_owner + .lock() + .unwrap_or_else(|error| error.into_inner()) + .iter() + .filter(|id| !snapshot.http_streams.contains(*id)) + .cloned() + .collect(); + self.io_client.close_stream_ids(&new_http_streams); + + let new_response_streams: Vec = self + .open_response_streams + .borrow() + .iter() + .filter(|id| !snapshot.response_streams.contains(*id)) + .cloned() + .collect(); + self.close_response_streams(&new_response_streams); + self.open_response_streams + .borrow_mut() + .retain(|id| snapshot.response_streams.contains(id)); + + let new_pending_requests: Vec = self + .open_pending_requests + .borrow() + .iter() + .filter(|id| !snapshot.pending_requests.contains(*id)) + .cloned() + .collect(); + self.fail_unanswered_requests(&new_pending_requests); + self.open_pending_requests + .borrow_mut() + .retain(|id| id != request_id && snapshot.pending_requests.contains(id)); + self.pending_responses.borrow_mut().remove(request_id); + } + + /// Evaluate every fallible response field while racing the target request's + /// disconnect signal. Disconnect wins ties, and the evaluation future is + /// dropped before partially-entered interpreter state is restored. + async fn evaluate_response_precommit( + &self, + request_id: &str, + line: usize, + column: usize, + disconnect_message: &'static str, + evaluation: F, + ) -> Result + where + F: std::future::Future>, + { + self.ensure_pending_response_owned(request_id, line, column) + .await?; + let snapshot = self.response_precommit_snapshot(); + let mut evaluation = Box::pin(evaluation); + let mut disconnected = Box::pin(self.pending_response_disconnected(request_id)); + let outcome = tokio::select! { + biased; + _ = disconnected.as_mut() => None, + result = evaluation.as_mut() => Some(result), + }; + let disconnected_now = self.pending_response_disconnected_now(request_id); + + if let Some(result) = outcome + && !disconnected_now + { + drop(evaluation); + drop(disconnected); + return result; + } + + // This ordering is intentional: dropping the future runs RAII guards + // before we overwrite any manually-restored run-state fields. + drop(evaluation); + drop(disconnected); + self.cancel_response_precommit(request_id, snapshot); + Err(RuntimeError::with_kind( + disconnect_message.to_string(), + line, + column, + ErrorKind::Cancelled, + )) + } + /// Take a pending response sender into an RAII completion guard for /// `respond` / `start streaming response`. /// @@ -9880,13 +10061,18 @@ impl Interpreter { // Only after evaluation do we take the sender into the completion // guard. Early eval errors leave the id in open_pending so the // handler-exit 500 path still resolves the client. - self.ensure_pending_response_owned(&request_id, *line, *column) - .await?; - - // Evaluate response content. Binary values are carried through - // as raw bytes so fonts/images/etc. serve losslessly; text and - // scalar values keep their existing UTF-8 rendering. - let content_val = self.evaluate_expression(content, Rc::clone(&env)).await?; + let response = self + .evaluate_response_precommit( + &request_id, + *line, + *column, + "Client disconnected before the response was sent", + async { + // Evaluate response content. Binary values are carried through + // as raw bytes so fonts/images/etc. serve losslessly; text and + // scalar values keep their existing UTF-8 rendering. + let content_val = + self.evaluate_expression(content, Rc::clone(&env)).await?; let is_binary = matches!(content_val, Value::Binary(_)); // Enforce the response-body ceiling on the *borrowed* length @@ -10029,13 +10215,15 @@ impl Interpreter { } } - // Create response - let response = WflHttpResponse { - content: content_bytes, - status: status_code, - content_type: content_type_str, - headers: custom_headers, - }; + Ok(WflHttpResponse { + content: content_bytes, + status: status_code, + content_type: content_type_str, + headers: custom_headers, + }) + }, + ) + .await?; // Now commit: take the sender (disconnect signal ends) and deliver. let mut completion = self @@ -10103,10 +10291,14 @@ impl Interpreter { // Keep pending parked through status/content-type/header // evaluation so disconnect still cancels any upstream work those // expressions perform. Handler-exit 500 covers early eval errors. - self.ensure_pending_response_owned(&request_id, *line, *column) - .await?; - - let status_code = match status { + let (status_code, content_type_str, custom_headers) = self + .evaluate_response_precommit( + &request_id, + *line, + *column, + "Client disconnected before the streaming response started", + async { + let status_code = match status { Some(expr) => { let v = self.evaluate_expression(expr, Rc::clone(&env)).await?; match &v { @@ -10200,6 +10392,11 @@ impl Interpreter { } } + Ok((status_code, content_type_str, custom_headers)) + }, + ) + .await?; + // Commit: take the sender and hand the streaming head to the transport. let (tx, rx) = mpsc::channel::>(RESPONSE_STREAM_BUFFER); let mut completion = self From 90a225d4de36fc74a0b17e4312889c2fa3511c93 Mon Sep 17 00:00:00 2001 From: Brad Byrd Date: Sat, 25 Jul 2026 00:39:19 -0500 Subject: [PATCH 118/132] refactor: make expiry tie handling deterministic --- src/interpreter/mod.rs | 4 ++++ 1 file changed, 4 insertions(+) diff --git a/src/interpreter/mod.rs b/src/interpreter/mod.rs index 942ac347..3feb9e2b 100644 --- a/src/interpreter/mod.rs +++ b/src/interpreter/mod.rs @@ -2583,6 +2583,10 @@ impl IoClient { }); tokio::pin!(op); let next = tokio::select! { + // Prefer a simultaneously-ready body result so the terminal + // re-check below is the single deterministic arbiter: expiry still + // wins, and a ready chunk can never be reinserted after the reaper. + biased; result = &mut op => result?, changed = terminal_rx.changed() => { let _ = changed; From edb8ce89c3d693015f655daac012c73cbc12d293 Mon Sep 17 00:00:00 2001 From: Brad Byrd Date: Sat, 25 Jul 2026 00:43:47 -0500 Subject: [PATCH 119/132] test: replace false-positive streaming lifecycle coverage --- src/interpreter/mod.rs | 559 ++++++++++++++++++ .../outbound_stream_close_during_read_test.rs | 113 ---- ...onse_expression_disconnect_runtime_test.rs | 455 ++++++++++++++ 3 files changed, 1014 insertions(+), 113 deletions(-) delete mode 100644 tests/outbound_stream_close_during_read_test.rs create mode 100644 tests/response_expression_disconnect_runtime_test.rs diff --git a/src/interpreter/mod.rs b/src/interpreter/mod.rs index 3feb9e2b..6c909c15 100644 --- a/src/interpreter/mod.rs +++ b/src/interpreter/mod.rs @@ -14970,9 +14970,225 @@ mod response_expression_disconnect_tests { } } +#[cfg(test)] +mod response_disconnect_result_tests { + use super::*; + use crate::lexer::lex_wfl_with_positions; + use crate::parser::Parser; + use std::future::Future; + use std::task::Poll; + + fn parse_statement(source: &str) -> Statement { + let tokens = lex_wfl_with_positions(source); + let mut parser = Parser::new(&tokens); + let program = parser + .parse() + .unwrap_or_else(|errors| panic!("disconnect fixture did not parse: {errors:?}")); + assert_eq!(program.statements.len(), 1); + program.statements.into_iter().next().expect("statement") + } + + fn request_value(request_id: &str) -> Value { + let mut request = HashMap::new(); + request.insert( + "_response_sender".to_string(), + Value::Text(Arc::from(request_id)), + ); + Value::Object(Rc::new(RefCell::new(request))) + } + + async fn assert_response_commit_disconnect_is_cancelled(statement_source: &str) { + let interpreter = Interpreter::new(); + let env = Rc::clone(interpreter.global_env()); + env.borrow_mut() + .define_or_replace("req", request_value("request-commit")); + let (sender, receiver) = oneshot::channel(); + let shared_sender = Arc::new(tokio::sync::Mutex::new(Some(sender))); + interpreter.pending_responses.borrow_mut().insert( + "request-commit".to_string(), + PendingResponse { + sender: Arc::clone(&shared_sender), + }, + ); + interpreter + .open_pending_requests + .borrow_mut() + .push("request-commit".to_string()); + + // Hold the sender lock so the statement can finish evaluation and reach + // the exact commit await without taking the sender yet. + let guard = shared_sender.lock().await; + let statement = parse_statement(statement_source); + let mut execution = Box::pin(interpreter.execute_statement(&statement, env)); + tokio::time::timeout(Duration::from_secs(2), async { + loop { + if !interpreter + .pending_responses + .borrow() + .contains_key("request-commit") + { + break; + } + tokio::select! { + result = execution.as_mut() => { + panic!("response finished before reaching the commit latch: {result:?}") + } + _ = tokio::task::yield_now() => {} + } + } + }) + .await + .expect("response did not reach its commit latch"); + + drop(receiver); + drop(guard); + let error = tokio::time::timeout(Duration::from_secs(2), execution.as_mut()) + .await + .expect("commit did not observe the closed receiver") + .expect_err("closed receiver must cancel the response commit"); + assert_eq!(error.kind, ErrorKind::Cancelled, "wrong error: {error:?}"); + } + + #[tokio::test] + async fn buffered_commit_disconnect_is_exactly_cancelled() { + assert_response_commit_disconnect_is_cancelled("respond to req with \"ok\"").await; + } + + #[tokio::test] + async fn streaming_head_commit_disconnect_is_exactly_cancelled() { + assert_response_commit_disconnect_is_cancelled( + "start streaming response to req with status 200 as out", + ) + .await; + } + + #[tokio::test] + async fn backpressured_stream_write_disconnect_is_exactly_cancelled() { + let config = Arc::new(WflConfig { + web_server_response_timeout_seconds: 0, + ..WflConfig::default() + }); + let interpreter = Interpreter::with_config(config); + let env = Rc::clone(interpreter.global_env()); + let handle_id = "respstream-test"; + let (sender, receiver) = mpsc::channel(RESPONSE_STREAM_BUFFER); + for _ in 0..RESPONSE_STREAM_BUFFER { + sender + .try_send(vec![0]) + .expect("fill response stream buffer"); + } + interpreter + .server_response_streams + .borrow_mut() + .insert(handle_id.to_string(), (sender, 0)); + interpreter + .open_response_streams + .borrow_mut() + .push(handle_id.to_string()); + let mut stream = HashMap::new(); + stream.insert( + "_server_stream".to_string(), + Value::Text(Arc::from(handle_id)), + ); + env.borrow_mut() + .define_or_replace("out", Value::Object(Rc::new(RefCell::new(stream)))); + + let statement = parse_statement("write chunk \"next\" to out"); + let mut execution = Box::pin(interpreter.execute_statement(&statement, env)); + futures_util::future::poll_fn(|cx| match execution.as_mut().poll(cx) { + Poll::Pending => Poll::Ready(()), + Poll::Ready(result) => { + panic!("full response stream write did not backpressure: {result:?}") + } + }) + .await; + drop(receiver); + + let error = tokio::time::timeout(Duration::from_secs(2), execution.as_mut()) + .await + .expect("write did not wake after receiver disconnect") + .expect_err("closed stream receiver must cancel the write"); + assert_eq!(error.kind, ErrorKind::Cancelled, "wrong error: {error:?}"); + assert!( + !interpreter + .server_response_streams + .borrow() + .contains_key(handle_id), + "cancelled write retained its response stream" + ); + } +} + +#[cfg(test)] +mod request_wait_timeout_tests { + use super::*; + use crate::lexer::lex_wfl_with_positions; + use crate::parser::Parser; + + fn parse_statement(source: &str) -> Statement { + let tokens = lex_wfl_with_positions(source); + let mut parser = Parser::new(&tokens); + parser + .parse() + .unwrap_or_else(|errors| panic!("timeout fixture did not parse: {errors:?}")) + .statements + .into_iter() + .next() + .expect("statement") + } + + async fn timeout_error(value: &str) -> RuntimeError { + let interpreter = Interpreter::new(); + let env = Rc::clone(interpreter.global_env()); + let (request_sender, request_receiver) = mpsc::channel(1); + interpreter.web_servers.borrow_mut().insert( + "srv".to_string(), + WflWebServer { + request_receiver: Arc::new(tokio::sync::Mutex::new(request_receiver)), + request_sender, + server_handle: None, + }, + ); + env.borrow_mut() + .define_or_replace("srv", Value::Text(Arc::from("WebServer::127.0.0.1:1"))); + let statement = parse_statement(&format!( + "wait for request comes in on srv as req with timeout {value}" + )); + interpreter + .execute_statement(&statement, env) + .await + .expect_err("invalid sub-millisecond timeout must be rejected") + } + + #[tokio::test] + async fn zero_request_timeout_has_the_established_positive_number_error() { + let error = timeout_error("0").await; + assert_eq!(error.kind, ErrorKind::General); + assert_eq!( + error.message, + "Timeout must be a positive number (milliseconds)" + ); + } + + #[tokio::test] + async fn fractional_request_timeout_below_one_millisecond_is_rejected() { + let error = timeout_error("0.5").await; + assert_eq!(error.kind, ErrorKind::General); + assert_eq!( + error.message, + "Timeout must be at least 1 millisecond (got 0.5 ms); fractional values below 1 would truncate to zero and spin" + ); + } +} + #[cfg(test)] mod outbound_stream_deadline_tests { use super::*; + use futures_util::task::AtomicWaker; + use std::future::Future; + use std::pin::Pin; + use std::sync::atomic::{AtomicBool, Ordering}; + use std::task::{Context, Poll}; use tokio::io::{AsyncReadExt, AsyncWriteExt}; async fn spawn_stream_cleanup_upstream(expected_requests: usize) -> u16 { @@ -15078,6 +15294,349 @@ mod outbound_stream_deadline_tests { .expect("finished streams left hard-lifetime reaper tasks sleeping"); } + struct DelayedHeadUpstream { + port: u16, + request_received: oneshot::Receiver<()>, + release_head: Option>, + peer_closed: oneshot::Receiver<()>, + } + + async fn spawn_delayed_head_upstream() -> DelayedHeadUpstream { + let listener = tokio::net::TcpListener::bind("127.0.0.1:0") + .await + .expect("bind delayed-head upstream"); + let port = listener.local_addr().expect("upstream address").port(); + let (request_tx, request_received) = oneshot::channel(); + let (release_head, release_rx) = oneshot::channel(); + let (peer_closed_tx, peer_closed) = oneshot::channel(); + tokio::spawn(async move { + let (mut socket, _) = listener.accept().await.expect("accept delayed head"); + let mut request = Vec::new(); + let mut buffer = [0u8; 512]; + while !request.windows(4).any(|window| window == b"\r\n\r\n") { + let read = socket.read(&mut buffer).await.expect("read request"); + assert!(read > 0, "client closed before request head"); + request.extend_from_slice(&buffer[..read]); + } + let _ = request_tx.send(()); + release_rx.await.expect("release delayed response head"); + socket + .write_all( + b"HTTP/1.1 200 OK\r\nContent-Length: 1048576\r\n\ + Connection: close\r\n\r\n", + ) + .await + .expect("write delayed response head"); + socket.flush().await.expect("flush delayed response head"); + loop { + match socket.read(&mut buffer).await { + Ok(0) | Err(_) => break, + Ok(_) => {} + } + } + let _ = peer_closed_tx.send(()); + }); + DelayedHeadUpstream { + port, + request_received, + release_head: Some(release_head), + peer_closed, + } + } + + struct GatedChunkState { + polled: AtomicBool, + ready: AtomicBool, + dropped: AtomicBool, + waker: AtomicWaker, + } + + impl GatedChunkState { + fn new() -> Arc { + Arc::new(Self { + polled: AtomicBool::new(false), + ready: AtomicBool::new(false), + dropped: AtomicBool::new(false), + waker: AtomicWaker::new(), + }) + } + + fn make_ready(&self) { + self.ready.store(true, Ordering::SeqCst); + self.waker.wake(); + } + } + + struct GatedChunkStream { + state: Arc, + yielded: bool, + } + + impl futures_util::Stream for GatedChunkStream { + type Item = reqwest::Result>; + + fn poll_next(mut self: Pin<&mut Self>, cx: &mut Context<'_>) -> Poll> { + if self.yielded { + return Poll::Ready(None); + } + self.state.polled.store(true, Ordering::SeqCst); + self.state.waker.register(cx.waker()); + if self.state.ready.load(Ordering::SeqCst) { + self.yielded = true; + Poll::Ready(Some(Ok(vec![7]))) + } else { + Poll::Pending + } + } + } + + impl Drop for GatedChunkStream { + fn drop(&mut self) { + self.state.dropped.store(true, Ordering::SeqCst); + } + } + + #[tokio::test] + async fn close_during_active_read_returns_closed_and_drops_upstream() { + let (port, mut peer_closed) = spawn_stalled_streams(1).await; + let config = Arc::new(WflConfig { + outbound_stream_max_seconds: 60, + timeout_seconds: 30, + ..WflConfig::default() + }); + let client = IoClient::new(Arc::clone(&config)); + let budget = Arc::new(ExecutionBudget::from_config(&config)); + let (_, _, handle_id) = client + .open_http_stream( + "GET", + &format!("http://127.0.0.1:{port}/active-close"), + &[], + None, + Arc::clone(&budget), + ) + .await + .expect("open stalled stream"); + + let mut read = Box::pin(client.next_chunk(&handle_id, budget)); + futures_util::future::poll_fn(|cx| match read.as_mut().poll(cx) { + Poll::Pending => Poll::Ready(()), + Poll::Ready(result) => panic!("stalled read unexpectedly completed: {result:?}"), + }) + .await; + assert!( + client + .stream_handles + .lock() + .unwrap_or_else(|error| error.into_inner()) + .live + .get(&handle_id) + .is_some_and(|slot| slot.handle.is_none()), + "the close latch must observe a body read actively owning the handle" + ); + + assert!( + client.finish_stream_slot_sync(&handle_id, StreamTerminal::Closed), + "close must claim the active stream slot" + ); + let error = tokio::time::timeout(Duration::from_secs(2), read.as_mut()) + .await + .expect("active read did not wake after close") + .expect_err("active read must return Closed"); + assert!( + matches!(error, HttpClientError::Closed), + "active close returned the wrong error: {error:?}" + ); + tokio::time::timeout(Duration::from_secs(2), peer_closed.recv()) + .await + .expect("active close did not drop the upstream socket") + .expect("upstream close notifier ended early"); + assert_reapers_drained(&client).await; + assert!( + !client + .stream_handles + .lock() + .unwrap_or_else(|error| error.into_inner()) + .live + .contains_key(&handle_id), + "active close retained its stream slot" + ); + } + + #[tokio::test] + async fn delayed_head_keeps_the_request_start_as_the_total_deadline_origin() { + let mut upstream = spawn_delayed_head_upstream().await; + let config = Arc::new(WflConfig { + outbound_stream_max_seconds: 2, + timeout_seconds: 30, + ..WflConfig::default() + }); + let client = IoClient::new(Arc::clone(&config)); + let budget = Arc::new(ExecutionBudget::from_config(&config)); + let started = Instant::now(); + let url = format!("http://127.0.0.1:{}/delayed-head", upstream.port); + let mut opening = Box::pin(client.open_http_stream( + "GET", + &url, + &[], + None, + Arc::clone(&budget), + )); + tokio::select! { + result = opening.as_mut() => { + panic!("stream opened before the response-head latch: {result:?}") + } + received = &mut upstream.request_received => { + received.expect("upstream did not receive request"); + } + } + tokio::time::sleep(Duration::from_millis(1_200)).await; + upstream + .release_head + .take() + .expect("head release") + .send(()) + .expect("release response head"); + let (_, _, handle_id) = tokio::time::timeout(Duration::from_secs(2), opening.as_mut()) + .await + .expect("stream did not open after response-head release") + .expect("delayed-head stream open"); + + let deadline = client + .stream_handles + .lock() + .unwrap_or_else(|error| error.into_inner()) + .live + .get(&handle_id) + .and_then(|slot| slot.deadline) + .expect("positive cap must register a slot deadline"); + assert!( + deadline.saturating_duration_since(started) <= Duration::from_millis(2_100), + "the total deadline was restarted after the delayed head" + ); + assert!( + deadline.saturating_duration_since(Instant::now()) < Duration::from_secs(1), + "the delayed head must leave less than one second of the original cap" + ); + + tokio::time::timeout(Duration::from_secs(2), &mut upstream.peer_closed) + .await + .expect("spawned reaper did not drop delayed-head upstream") + .expect("upstream close notifier ended early"); + let error = client + .next_chunk(&handle_id, budget) + .await + .expect_err("read after delayed-head expiry must fail"); + assert!( + matches!(error, HttpClientError::Timeout { seconds: 2 }), + "delayed-head expiry lost its typed timeout: {error:?}" + ); + assert_reapers_drained(&client).await; + } + + #[tokio::test] + async fn spawned_reaper_wins_over_a_simultaneously_ready_chunk() { + let (port, mut peer_closed) = spawn_stalled_streams(1).await; + let config = Arc::new(WflConfig { + outbound_stream_max_seconds: 1, + timeout_seconds: 30, + ..WflConfig::default() + }); + let client = IoClient::new(Arc::clone(&config)); + let budget = Arc::new(ExecutionBudget::from_config(&config)); + let (_, _, handle_id) = client + .open_http_stream( + "GET", + &format!("http://127.0.0.1:{port}/ready-expiry-race"), + &[], + None, + Arc::clone(&budget), + ) + .await + .expect("open stalled stream"); + let state = GatedChunkState::new(); + let cancel = { + let mut registry = client + .stream_handles + .lock() + .unwrap_or_else(|error| error.into_inner()); + let slot = registry.live.get_mut(&handle_id).expect("live stream slot"); + let handle = slot.handle.as_mut().expect("parked stream body"); + handle.stream = Box::pin(GatedChunkStream { + state: Arc::clone(&state), + yielded: false, + }); + // Leave the slot deadline and real spawned reaper intact, but keep + // the inner read timeout from independently deciding this race. + handle.total_deadline = None; + Arc::clone(&slot.cancel) + }; + tokio::time::timeout(Duration::from_secs(2), peer_closed.recv()) + .await + .expect("replacing the real body did not drop its upstream socket") + .expect("upstream close notifier ended early"); + + let mut read = Box::pin(client.next_chunk(&handle_id, budget)); + futures_util::future::poll_fn(|cx| match read.as_mut().poll(cx) { + Poll::Pending => Poll::Ready(()), + Poll::Ready(result) => panic!("gated read unexpectedly completed: {result:?}"), + }) + .await; + assert!( + state.polled.load(Ordering::SeqCst), + "gated body was not polled" + ); + assert!( + client + .stream_handles + .lock() + .unwrap_or_else(|error| error.into_inner()) + .live + .get(&handle_id) + .is_some_and(|slot| slot.handle.is_none()), + "active read did not take ownership before expiry" + ); + + tokio::time::timeout(Duration::from_secs(2), async { + loop { + if cancel.terminal() == Some(StreamTerminal::Timeout) + && client.active_stream_reapers.load(Ordering::SeqCst) == 0 + { + break; + } + tokio::task::yield_now().await; + } + }) + .await + .expect("the production reaper did not establish Timeout"); + state.make_ready(); + + let error = tokio::time::timeout(Duration::from_secs(2), read.as_mut()) + .await + .expect("simultaneously-ready race did not resolve") + .expect_err("expiry must win over the ready body chunk"); + assert!( + matches!(error, HttpClientError::Timeout { seconds: 1 }), + "ready chunk beat the spawned reaper: {error:?}" + ); + assert!( + state.dropped.load(Ordering::SeqCst), + "expired active body was not dropped" + ); + let registry = client + .stream_handles + .lock() + .unwrap_or_else(|error| error.into_inner()); + assert!( + !registry.live.contains_key(&handle_id), + "expired body was reinserted after the reaper" + ); + assert_eq!( + client.active_stream_reapers.load(Ordering::SeqCst), + 0, + "spawned reaper survived the race" + ); + } + #[test] fn extreme_outbound_stream_max_seconds_does_not_panic() { // u64::MAX must remain a finite cap rather than panicking or silently diff --git a/tests/outbound_stream_close_during_read_test.rs b/tests/outbound_stream_close_during_read_test.rs deleted file mode 100644 index 6df5cdab..00000000 --- a/tests/outbound_stream_close_during_read_test.rs +++ /dev/null @@ -1,113 +0,0 @@ -//! Close-during-active-read must cancel the upstream promptly (issue #642 re-review). -//! -//! `take_stream` removes the handle for the await; a concurrent `close` must trip -//! shared cancellation so the active read aborts and the upstream is dropped — -//! not leave the connection open until idle timeout while `put_stream` treats a -//! missing slot as success. - -use std::sync::Arc; -use std::time::{Duration, Instant}; -use tokio::io::{AsyncReadExt, AsyncWriteExt}; -use wfl::Interpreter; -use wfl::config::WflConfig; -use wfl::lexer::lex_wfl_with_positions; -use wfl::parser::Parser; - -async fn spawn_stall_upstream() -> (u16, tokio::sync::oneshot::Receiver<()>) { - let listener = tokio::net::TcpListener::bind("127.0.0.1:0") - .await - .expect("bind"); - let port = listener.local_addr().unwrap().port(); - let (tx, rx) = tokio::sync::oneshot::channel(); - tokio::spawn(async move { - if let Ok((mut sock, _)) = listener.accept().await { - let mut buf = [0u8; 1024]; - let _ = sock.read(&mut buf).await; - let head = "HTTP/1.1 200 OK\r\n\ - Content-Type: text/plain\r\n\ - Transfer-Encoding: chunked\r\n\r\n"; - let _ = sock.write_all(head.as_bytes()).await; - let _ = sock.flush().await; - loop { - match sock.read(&mut buf).await { - Ok(0) | Err(_) => { - let _ = tx.send(()); - return; - } - Ok(_) => {} - } - } - } - }); - (port, rx) -} - -#[tokio::test] -async fn close_during_active_read_drops_upstream_promptly() { - let (port, mut upstream_closed) = spawn_stall_upstream().await; - - // Open stream, start a body read that will park on the stalled upstream, then - // close from another path via a short wait then close — implemented as: - // open, spawn wait for next chunk (parks), wait 200ms, close, then the read - // must fail and upstream drop well before the idle timeout (30s). - let code = format!( - r#" - open url at "http://127.0.0.1:{port}/" and stream response as s - wait for 200 milliseconds - close s - wait for next chunk from s as c - "# - ); - - let (result_tx, result_rx) = std::sync::mpsc::channel::>(); - let client = std::thread::spawn(move || { - let rt = tokio::runtime::Runtime::new().expect("runtime"); - rt.block_on(async { - let tokens = lex_wfl_with_positions(&code); - let program = Parser::new(&tokens).parse().expect("parse"); - let config = WflConfig { - timeout_seconds: 30, - outbound_stream_max_seconds: 60, - ..WflConfig::default() - }; - let mut interp = Interpreter::with_config(Arc::new(config)); - let result = interp.interpret(&program).await; - let summary = match result { - Ok(_) => Ok(()), - Err(errs) => Err(format!("{errs:?}")), - }; - let _ = result_tx.send(summary); - }); - }); - - let start = Instant::now(); - tokio::time::timeout(Duration::from_secs(3), &mut upstream_closed) - .await - .expect("upstream should drop promptly after close, not at 30s idle timeout") - .expect("upstream close sender dropped"); - let elapsed = start.elapsed(); - assert!( - elapsed < Duration::from_secs(2), - "close-during-read should drop upstream promptly; took {elapsed:?}" - ); - - let summary = result_rx - .recv_timeout(Duration::from_secs(5)) - .expect("interpreter should finish"); - // After close, wait for next chunk should fail (closed stream). - assert!( - summary.is_err(), - "read after close must error, got {summary:?}" - ); - let msg = summary.unwrap_err().to_lowercase(); - assert!( - msg.contains("closed") || msg.contains("unknown") || msg.contains("stream"), - "expected a closed-stream error, got: {msg}" - ); - - match tokio::task::spawn_blocking(move || client.join()).await { - Ok(Ok(())) => {} - Ok(Err(panic)) => std::panic::resume_unwind(panic), - Err(e) => panic!("join failed: {e}"), - } -} diff --git a/tests/response_expression_disconnect_runtime_test.rs b/tests/response_expression_disconnect_runtime_test.rs new file mode 100644 index 00000000..29b09775 --- /dev/null +++ b/tests/response_expression_disconnect_runtime_test.rs @@ -0,0 +1,455 @@ +//! Real-socket coverage for response-expression cancellation (issue #642). +//! +//! A buffered response body and each fallible streaming-head operand (status, +//! content type, and headers) are evaluated before the response commits. If the +//! browser disconnects during one of those evaluations, the handler must cancel +//! the evaluation, release its real outbound stream, and leave the concurrent +//! server able to serve unrelated work. +//! +//! Each case uses two real upstream sockets. The data upstream sends one chunk +//! and then withholds the rest of its body. After consuming that first chunk, the +//! WFL action opens a checkpoint stream, consumes its marker, explicitly closes +//! it, and only then blocks on the data upstream again. The test waits for the +//! checkpoint socket to close before dropping the browser socket, so the +//! synchronization is event-controlled rather than based on a handler sleep. + +use std::time::Duration; + +use tokio::io::{AsyncReadExt, AsyncWriteExt}; +use tokio::sync::oneshot; +use wfl::Interpreter; +use wfl::lexer::lex_wfl_with_positions; +use wfl::parser::Parser; + +mod common; + +const PROBE_DEADLINE: Duration = Duration::from_secs(10); + +struct EvaluationProbe { + data_port: u16, + checkpoint_port: u16, + checkpoint_passed: oneshot::Receiver>, + data_closed: oneshot::Receiver>, +} + +async fn read_http_head(socket: &mut tokio::net::TcpStream) -> Result, String> { + let mut request = Vec::new(); + let mut byte = [0u8; 1]; + while !request.ends_with(b"\r\n\r\n") { + let count = socket + .read(&mut byte) + .await + .map_err(|error| format!("read request head: {error}"))?; + if count == 0 { + return Err("peer closed before sending a complete request head".to_string()); + } + request.push(byte[0]); + if request.len() > 16 * 1024 { + return Err("request head exceeded 16 KiB".to_string()); + } + } + Ok(request) +} + +async fn wait_for_peer_close(socket: &mut tokio::net::TcpStream) { + let mut byte = [0u8; 1]; + loop { + match socket.read(&mut byte).await { + Ok(0) | Err(_) => return, + Ok(_) => {} + } + } +} + +async fn spawn_evaluation_probe(label: &'static str) -> EvaluationProbe { + let data_listener = tokio::net::TcpListener::bind("127.0.0.1:0") + .await + .unwrap_or_else(|error| panic!("{label}: bind data upstream: {error}")); + let data_port = data_listener + .local_addr() + .unwrap_or_else(|error| panic!("{label}: inspect data upstream address: {error}")) + .port(); + let (data_closed_tx, data_closed) = oneshot::channel(); + tokio::spawn(async move { + let result = async { + let (mut socket, _) = data_listener + .accept() + .await + .map_err(|error| format!("{label}: accept data upstream: {error}"))?; + read_http_head(&mut socket).await?; + socket + .write_all( + b"HTTP/1.1 200 OK\r\n\ + Content-Type: application/octet-stream\r\n\ + Transfer-Encoding: chunked\r\n\ + Connection: keep-alive\r\n\r\n\ + 5\r\nready\r\n", + ) + .await + .map_err(|error| format!("{label}: write data marker: {error}"))?; + socket + .flush() + .await + .map_err(|error| format!("{label}: flush data marker: {error}"))?; + wait_for_peer_close(&mut socket).await; + Ok(()) + } + .await; + let _ = data_closed_tx.send(result); + }); + + let checkpoint_listener = tokio::net::TcpListener::bind("127.0.0.1:0") + .await + .unwrap_or_else(|error| panic!("{label}: bind checkpoint upstream: {error}")); + let checkpoint_port = checkpoint_listener + .local_addr() + .unwrap_or_else(|error| panic!("{label}: inspect checkpoint address: {error}")) + .port(); + let (checkpoint_passed_tx, checkpoint_passed) = oneshot::channel(); + tokio::spawn(async move { + let result = async { + let (mut socket, _) = checkpoint_listener + .accept() + .await + .map_err(|error| format!("{label}: accept checkpoint: {error}"))?; + read_http_head(&mut socket).await?; + // Deliberately omit the terminating zero-length chunk. The WFL + // action reads this marker and then closes the still-live stream. + // Observing that close proves the action passed the checkpoint. + socket + .write_all( + b"HTTP/1.1 200 OK\r\n\ + Content-Type: application/octet-stream\r\n\ + Transfer-Encoding: chunked\r\n\ + Connection: keep-alive\r\n\r\n\ + 1\r\nA\r\n", + ) + .await + .map_err(|error| format!("{label}: write checkpoint marker: {error}"))?; + socket + .flush() + .await + .map_err(|error| format!("{label}: flush checkpoint marker: {error}"))?; + wait_for_peer_close(&mut socket).await; + Ok(()) + } + .await; + let _ = checkpoint_passed_tx.send(result); + }); + + EvaluationProbe { + data_port, + checkpoint_port, + checkpoint_passed, + data_closed, + } +} + +struct ProxyServer { + thread: Option>, + abort: Option>, +} + +impl Drop for ProxyServer { + fn drop(&mut self) { + if let Some(abort) = self.abort.take() { + let _ = abort.send(()); + } + } +} + +fn start_proxy_server(code: String) -> ProxyServer { + let (abort, abort_rx) = oneshot::channel(); + let thread = std::thread::Builder::new() + .name("response-expression-disconnect-proxy".to_string()) + .stack_size(wfl::INTERPRETER_STACK_SIZE) + .spawn(move || { + let runtime = tokio::runtime::Runtime::new().expect("create proxy runtime"); + runtime.block_on(async { + let tokens = lex_wfl_with_positions(&code); + let ast = Parser::new(&tokens) + .parse() + .unwrap_or_else(|errors| panic!("parse proxy program: {errors:?}")); + let mut interpreter = Interpreter::new(); + tokio::select! { + result = interpreter.interpret(&ast) => { + if let Err(errors) = result { + panic!("proxy interpreter failed: {errors:?}"); + } + } + _ = abort_rx => { + // Test cleanup drops the interpreter and all live listeners. + } + } + }); + }) + .expect("spawn proxy interpreter thread"); + ProxyServer { + thread: Some(thread), + abort: Some(abort), + } +} + +async fn wait_for_server(port: u16) { + let address = format!("127.0.0.1:{port}"); + for _ in 0..300 { + if tokio::net::TcpStream::connect(&address).await.is_ok() { + return; + } + tokio::time::sleep(Duration::from_millis(20)).await; + } + panic!("proxy server on {address} did not become ready"); +} + +async fn assert_ping_survives(port: u16, context: &str) { + let response = tokio::time::timeout( + PROBE_DEADLINE, + reqwest::Client::new() + .get(format!("http://127.0.0.1:{port}/ping")) + .send(), + ) + .await + .unwrap_or_else(|_| panic!("{context}: /ping timed out after expression cancellation")) + .unwrap_or_else(|error| panic!("{context}: /ping failed: {error}")); + assert_eq!( + response.status().as_u16(), + 200, + "{context}: server returned a non-success /ping status" + ); + assert_eq!( + response + .text() + .await + .unwrap_or_else(|error| panic!("{context}: read /ping body: {error}")), + "pong", + "{context}: server returned the wrong /ping body" + ); +} + +async fn disconnect_at_checkpoint( + proxy_port: u16, + path: &'static str, + label: &'static str, + mut probe: EvaluationProbe, +) { + let mut browser = tokio::net::TcpStream::connect(("127.0.0.1", proxy_port)) + .await + .unwrap_or_else(|error| panic!("{label}: connect browser socket: {error}")); + browser + .write_all( + format!( + "GET {path} HTTP/1.1\r\n\ + Host: 127.0.0.1\r\n\ + Connection: close\r\n\r\n" + ) + .as_bytes(), + ) + .await + .unwrap_or_else(|error| panic!("{label}: send browser request: {error}")); + browser + .flush() + .await + .unwrap_or_else(|error| panic!("{label}: flush browser request: {error}")); + + tokio::time::timeout(PROBE_DEADLINE, &mut probe.checkpoint_passed) + .await + .unwrap_or_else(|_| { + panic!("{label}: response expression never passed its upstream checkpoint") + }) + .unwrap_or_else(|_| panic!("{label}: checkpoint task ended without a result")) + .unwrap_or_else(|error| panic!("{error}")); + + assert!( + matches!( + probe.data_closed.try_recv(), + Err(oneshot::error::TryRecvError::Empty) + ), + "{label}: data upstream closed before the browser disconnected" + ); + + // The action has consumed its first real upstream chunk, passed and closed + // its marker stream, and is now blocked reading the unfinished data body. + // Dropping this socket is the only release signal for that evaluation. + drop(browser); + + tokio::time::timeout(PROBE_DEADLINE, &mut probe.data_closed) + .await + .unwrap_or_else(|_| { + panic!("{label}: stalled upstream remained open after browser disconnect") + }) + .unwrap_or_else(|_| panic!("{label}: data upstream task ended without a result")) + .unwrap_or_else(|error| panic!("{error}")); + + assert_ping_survives(proxy_port, label).await; +} + +async fn shutdown_proxy(port: u16, mut server: ProxyServer) { + let _ = tokio::time::timeout( + PROBE_DEADLINE, + reqwest::Client::new() + .get(format!("http://127.0.0.1:{port}/shutdown")) + .send(), + ) + .await; + + let graceful = tokio::time::timeout(PROBE_DEADLINE, async { + while !server + .thread + .as_ref() + .expect("proxy thread exists until shutdown") + .is_finished() + { + tokio::task::yield_now().await; + } + }) + .await + .is_ok(); + if !graceful { + let _ = server + .abort + .take() + .expect("proxy abort signal is sent at most once") + .send(()); + } + + let thread = server + .thread + .take() + .expect("proxy thread is joined at most once"); + match tokio::task::spawn_blocking(move || thread.join()).await { + Ok(Ok(())) => {} + Ok(Err(panic)) => std::panic::resume_unwind(panic), + Err(error) => panic!("proxy join task failed: {error}"), + } +} + +fn stalled_action( + name: &str, + prefix: &str, + probe: &EvaluationProbe, + return_statements: &str, +) -> String { + format!( + r#" +define action called {name}: + open url at "http://127.0.0.1:{data_port}/data" and stream response as {prefix}_source + wait for next chunk from {prefix}_source as {prefix}_ready + open url at "http://127.0.0.1:{checkpoint_port}/checkpoint" and stream response as {prefix}_checkpoint + wait for next chunk from {prefix}_checkpoint as {prefix}_acknowledged + close {prefix}_checkpoint + wait for next chunk from {prefix}_source as {prefix}_blocked +{return_statements} +end action +"#, + data_port = probe.data_port, + checkpoint_port = probe.checkpoint_port, + ) +} + +#[tokio::test] +async fn response_expression_disconnects_cancel_upstreams_and_preserve_server_liveness() { + let buffered = spawn_evaluation_probe("buffered response content").await; + let stream_status = spawn_evaluation_probe("streaming response status").await; + let stream_content_type = spawn_evaluation_probe("streaming response content type").await; + let stream_headers = spawn_evaluation_probe("streaming response headers").await; + let proxy_port = common::free_tcp_port(); + + let actions = [ + stalled_action( + "stalled_buffered_content", + "buffered_value", + &buffered, + " return \"late body\"", + ), + stalled_action( + "stalled_stream_status", + "stream_status", + &stream_status, + " return 201", + ), + stalled_action( + "stalled_stream_content_type", + "stream_content_type", + &stream_content_type, + " return \"text/plain\"", + ), + stalled_action( + "stalled_stream_headers", + "stream_headers", + &stream_headers, + " create map delayed_headers:\n \"X-Probe\" is \"late\"\n end map\n return delayed_headers", + ), + ] + .join("\n"); + + let program = format!( + r#" +{actions} +listen on port {proxy_port} as srv +main loop concurrently: + wait for request comes in on srv as req with timeout 30000 + store request_path as req["path"] + check if request_path is equal to "/ping": + respond to req with "pong" + otherwise: + check if request_path is equal to "/shutdown": + respond to req with "bye" + close server srv + break + otherwise: + check if request_path is equal to "/buffered": + respond to req with call stalled_buffered_content + otherwise: + check if request_path is equal to "/stream-status": + start streaming response to req with status call stalled_stream_status as out + close out + otherwise: + check if request_path is equal to "/stream-content-type": + start streaming response to req with status 200 and content type call stalled_stream_content_type as out + close out + otherwise: + start streaming response to req with status 200 and content type "text/plain" and headers call stalled_stream_headers as out + close out + end check + end check + end check + end check + end check +end loop +"# + ); + + let server = start_proxy_server(program); + wait_for_server(proxy_port).await; + + disconnect_at_checkpoint( + proxy_port, + "/buffered", + "buffered response content", + buffered, + ) + .await; + disconnect_at_checkpoint( + proxy_port, + "/stream-status", + "streaming response status", + stream_status, + ) + .await; + disconnect_at_checkpoint( + proxy_port, + "/stream-content-type", + "streaming response content type", + stream_content_type, + ) + .await; + disconnect_at_checkpoint( + proxy_port, + "/stream-headers", + "streaming response headers", + stream_headers, + ) + .await; + + shutdown_proxy(proxy_port, server).await; +} From 3bc38c668a91229e213a10d8eaebdba3789556a9 Mon Sep 17 00:00:00 2001 From: Brad Byrd Date: Sat, 25 Jul 2026 00:47:54 -0500 Subject: [PATCH 120/132] test: expose residual response cancellation leaks --- src/interpreter/mod.rs | 227 +++++++++++++++++++++++++++++++++++++++-- 1 file changed, 220 insertions(+), 7 deletions(-) diff --git a/src/interpreter/mod.rs b/src/interpreter/mod.rs index 6c909c15..53f1f3b3 100644 --- a/src/interpreter/mod.rs +++ b/src/interpreter/mod.rs @@ -14968,6 +14968,21 @@ mod response_expression_disconnect_tests { ) .await; } + + #[tokio::test] + async fn buffered_request_operand_cancels_on_request_disconnect() { + assert_precommit_disconnect_cancels("respond to (call stalled_value) with \"ok\"", "req") + .await; + } + + #[tokio::test] + async fn streaming_request_operand_cancels_on_request_disconnect() { + assert_precommit_disconnect_cancels( + "start streaming response to (call stalled_value) with status 200 as out", + "req", + ) + .await; + } } #[cfg(test)] @@ -14977,6 +14992,7 @@ mod response_disconnect_result_tests { use crate::parser::Parser; use std::future::Future; use std::task::Poll; + use tokio::io::{AsyncReadExt, AsyncWriteExt}; fn parse_statement(source: &str) -> Statement { let tokens = lex_wfl_with_positions(source); @@ -14988,6 +15004,15 @@ mod response_disconnect_result_tests { program.statements.into_iter().next().expect("statement") } + fn parse_statements(source: &str) -> Vec { + let tokens = lex_wfl_with_positions(source); + let mut parser = Parser::new(&tokens); + parser + .parse() + .unwrap_or_else(|errors| panic!("disconnect fixture did not parse: {errors:?}")) + .statements + } + fn request_value(request_id: &str) -> Value { let mut request = HashMap::new(); request.insert( @@ -15049,6 +15074,141 @@ mod response_disconnect_result_tests { assert_eq!(error.kind, ErrorKind::Cancelled, "wrong error: {error:?}"); } + async fn spawn_commit_upstream() -> (u16, oneshot::Receiver<()>, oneshot::Receiver<()>) { + let listener = tokio::net::TcpListener::bind("127.0.0.1:0") + .await + .expect("bind commit upstream"); + let port = listener + .local_addr() + .expect("commit upstream address") + .port(); + let (head_tx, head_sent) = oneshot::channel(); + let (closed_tx, peer_closed) = oneshot::channel(); + tokio::spawn(async move { + let (mut socket, _) = listener.accept().await.expect("accept commit upstream"); + let mut request = Vec::new(); + let mut buffer = [0u8; 512]; + while !request.windows(4).any(|window| window == b"\r\n\r\n") { + let read = socket.read(&mut buffer).await.expect("read request head"); + assert!(read > 0, "client closed before commit request"); + request.extend_from_slice(&buffer[..read]); + } + socket + .write_all( + b"HTTP/1.1 200 OK\r\nContent-Length: 1048576\r\n\ + Connection: close\r\n\r\n", + ) + .await + .expect("write commit response head"); + socket.flush().await.expect("flush commit response head"); + let _ = head_tx.send(()); + loop { + match socket.read(&mut buffer).await { + Ok(0) | Err(_) => break, + Ok(_) => {} + } + } + let _ = closed_tx.send(()); + }); + (port, head_sent, peer_closed) + } + + async fn assert_commit_disconnect_closes_evaluation_stream(response_statement: &str) { + let (port, head_sent, mut peer_closed) = spawn_commit_upstream().await; + let config = Arc::new(WflConfig { + outbound_stream_max_seconds: 120, + timeout_seconds: 120, + ..WflConfig::default() + }); + let interpreter = Interpreter::with_config(config); + let source = format!( + "define action called open_then_return:\n\ + \x20\x20\x20\x20open url at \"http://127.0.0.1:{port}/commit\" and stream response as upstream\n\ + \x20\x20\x20\x20return 201\n\ + end action\n\ + {response_statement}\n" + ); + let statements = parse_statements(&source); + assert_eq!(statements.len(), 2); + let env = Rc::clone(interpreter.global_env()); + interpreter + .execute_statement(&statements[0], Rc::clone(&env)) + .await + .expect("define commit action"); + + env.borrow_mut() + .define_or_replace("req", request_value("request-resource")); + let (sender, receiver) = oneshot::channel(); + let shared_sender = Arc::new(tokio::sync::Mutex::new(Some(sender))); + interpreter.pending_responses.borrow_mut().insert( + "request-resource".to_string(), + PendingResponse { + sender: Arc::clone(&shared_sender), + }, + ); + interpreter + .open_pending_requests + .borrow_mut() + .push("request-resource".to_string()); + let guard = shared_sender.lock().await; + + let mut execution = + Box::pin(interpreter.execute_statement(&statements[1], Rc::clone(&env))); + tokio::time::timeout(Duration::from_secs(3), async { + loop { + let at_commit = !interpreter + .pending_responses + .borrow() + .contains_key("request-resource"); + let owns_stream = interpreter + .open_http_streams + .borrow() + .lock() + .unwrap_or_else(|error| error.into_inner()) + .len() + == 1; + if at_commit && owns_stream { + break; + } + tokio::select! { + result = execution.as_mut() => { + panic!("response finished before resource commit latch: {result:?}") + } + _ = tokio::task::yield_now() => {} + } + } + }) + .await + .expect("response did not reach resource commit latch"); + head_sent.await.expect("commit upstream did not send head"); + + drop(receiver); + drop(guard); + let error = tokio::time::timeout(Duration::from_secs(2), execution.as_mut()) + .await + .expect("commit did not observe receiver disconnect") + .expect_err("commit disconnect must cancel"); + assert_eq!(error.kind, ErrorKind::Cancelled, "wrong error: {error:?}"); + + if tokio::time::timeout(Duration::from_secs(1), &mut peer_closed) + .await + .is_err() + { + interpreter.close_open_http_streams(); + let _ = tokio::time::timeout(Duration::from_secs(2), &mut peer_closed).await; + panic!("commit-time cancellation retained a stream opened during evaluation"); + } + assert!( + interpreter + .open_http_streams + .borrow() + .lock() + .unwrap_or_else(|error| error.into_inner()) + .is_empty(), + "commit cancellation retained stream ownership" + ); + } + #[tokio::test] async fn buffered_commit_disconnect_is_exactly_cancelled() { assert_response_commit_disconnect_is_cancelled("respond to req with \"ok\"").await; @@ -15062,6 +15222,64 @@ mod response_disconnect_result_tests { .await; } + #[tokio::test] + async fn already_disconnected_precheck_removes_stale_pending_state() { + let interpreter = Interpreter::new(); + let env = Rc::clone(interpreter.global_env()); + env.borrow_mut() + .define_or_replace("req", request_value("request-closed")); + let (sender, receiver) = oneshot::channel(); + interpreter.pending_responses.borrow_mut().insert( + "request-closed".to_string(), + PendingResponse { + sender: Arc::new(tokio::sync::Mutex::new(Some(sender))), + }, + ); + interpreter + .open_pending_requests + .borrow_mut() + .push("request-closed".to_string()); + drop(receiver); + + let statement = parse_statement("respond to req with \"late\""); + let error = interpreter + .execute_statement(&statement, env) + .await + .expect_err("closed request must cancel before evaluation"); + assert_eq!(error.kind, ErrorKind::Cancelled); + assert!( + !interpreter + .pending_responses + .borrow() + .contains_key("request-closed"), + "early cancellation retained the pending sender" + ); + assert!( + !interpreter + .open_pending_requests + .borrow() + .iter() + .any(|id| id == "request-closed"), + "early cancellation retained handler ownership" + ); + } + + #[tokio::test] + async fn buffered_commit_disconnect_closes_evaluation_streams() { + assert_commit_disconnect_closes_evaluation_stream( + "respond to req with call open_then_return", + ) + .await; + } + + #[tokio::test] + async fn streaming_commit_disconnect_closes_evaluation_streams() { + assert_commit_disconnect_closes_evaluation_stream( + "start streaming response to req with status call open_then_return as out", + ) + .await; + } + #[tokio::test] async fn backpressured_stream_write_disconnect_is_exactly_cancelled() { let config = Arc::new(WflConfig { @@ -15474,13 +15692,8 @@ mod outbound_stream_deadline_tests { let budget = Arc::new(ExecutionBudget::from_config(&config)); let started = Instant::now(); let url = format!("http://127.0.0.1:{}/delayed-head", upstream.port); - let mut opening = Box::pin(client.open_http_stream( - "GET", - &url, - &[], - None, - Arc::clone(&budget), - )); + let mut opening = + Box::pin(client.open_http_stream("GET", &url, &[], None, Arc::clone(&budget))); tokio::select! { result = opening.as_mut() => { panic!("stream opened before the response-head latch: {result:?}") From c73260ff61a32694c5ecfe72ab8749810033de0d Mon Sep 17 00:00:00 2001 From: Brad Byrd Date: Sat, 25 Jul 2026 00:52:30 -0500 Subject: [PATCH 121/132] fix: close residual response cancellation gaps --- src/interpreter/mod.rs | 156 +++++++++++++++++++++++++++++++++++++---- 1 file changed, 141 insertions(+), 15 deletions(-) diff --git a/src/interpreter/mod.rs b/src/interpreter/mod.rs index 53f1f3b3..3f01b80c 100644 --- a/src/interpreter/mod.rs +++ b/src/interpreter/mod.rs @@ -4836,6 +4836,35 @@ impl Interpreter { } } + fn first_pending_response_disconnected_now(&self, request_ids: &[String]) -> Option { + request_ids + .iter() + .find(|request_id| self.pending_response_disconnected_now(request_id)) + .cloned() + } + + /// A response request operand can itself be asynchronous, so its target id + /// is not available until evaluation finishes. Watch only the pending + /// requests owned when evaluation starts; requests accepted by nested work + /// are cleaned as newly-created resources if this evaluation is cancelled. + async fn first_pending_response_disconnected(&self, request_ids: &[String]) -> String { + loop { + if let Some(request_id) = self.first_pending_response_disconnected_now(request_ids) { + return request_id; + } + let any_still_owned = { + let owned = self.open_pending_requests.borrow(); + request_ids + .iter() + .any(|request_id| owned.iter().any(|id| id == request_id)) + }; + if !any_still_owned { + std::future::pending::<()>().await; + } + tokio::time::sleep(REQUEST_DISCONNECT_POLL_INTERVAL).await; + } + } + fn response_precommit_snapshot(&self) -> ResponsePrecommitSnapshot { let http_owner = Arc::clone(&self.open_http_streams.borrow()); let http_streams = http_owner @@ -4908,6 +4937,57 @@ impl Interpreter { self.pending_responses.borrow_mut().remove(request_id); } + /// Evaluate the response request operand while racing every request this + /// handler already owns. The target request id is not known until the + /// operand resolves, so this establishes the response-attempt snapshot that + /// is carried through field evaluation and commit. + async fn evaluate_response_request( + &self, + line: usize, + column: usize, + disconnect_message: &'static str, + evaluation: F, + ) -> Result<(T, ResponsePrecommitSnapshot), RuntimeError> + where + F: std::future::Future>, + { + let snapshot = self.response_precommit_snapshot(); + let mut watched_requests: Vec = snapshot.pending_requests.iter().cloned().collect(); + watched_requests.sort(); + let mut evaluation = Box::pin(evaluation); + let mut disconnected = + Box::pin(self.first_pending_response_disconnected(&watched_requests)); + let outcome = tokio::select! { + biased; + request_id = disconnected.as_mut() => (None, Some(request_id)), + result = evaluation.as_mut() => (Some(result), None), + }; + let disconnected_request = outcome + .1 + .or_else(|| self.first_pending_response_disconnected_now(&watched_requests)); + + if let Some(result) = outcome.0 + && disconnected_request.is_none() + { + drop(evaluation); + drop(disconnected); + return result.map(|value| (value, snapshot)); + } + + // Drop first so action/loop RAII runs before restoring the baseline. + drop(evaluation); + drop(disconnected); + let request_id = + disconnected_request.expect("disconnect watcher completed without a request id"); + self.cancel_response_precommit(&request_id, snapshot); + Err(RuntimeError::with_kind( + disconnect_message.to_string(), + line, + column, + ErrorKind::Cancelled, + )) + } + /// Evaluate every fallible response field while racing the target request's /// disconnect signal. Disconnect wins ties, and the evaluation future is /// dropped before partially-entered interpreter state is restored. @@ -4917,14 +4997,21 @@ impl Interpreter { line: usize, column: usize, disconnect_message: &'static str, + snapshot: ResponsePrecommitSnapshot, evaluation: F, - ) -> Result + ) -> Result<(T, ResponsePrecommitSnapshot), RuntimeError> where F: std::future::Future>, { - self.ensure_pending_response_owned(request_id, line, column) - .await?; - let snapshot = self.response_precommit_snapshot(); + if let Err(error) = self + .ensure_pending_response_owned(request_id, line, column) + .await + { + if error.kind == ErrorKind::Cancelled { + self.cancel_response_precommit(request_id, snapshot); + } + return Err(error); + } let mut evaluation = Box::pin(evaluation); let mut disconnected = Box::pin(self.pending_response_disconnected(request_id)); let outcome = tokio::select! { @@ -4939,7 +5026,7 @@ impl Interpreter { { drop(evaluation); drop(disconnected); - return result; + return result.map(|value| (value, snapshot)); } // This ordering is intentional: dropping the future runs RAII guards @@ -10033,8 +10120,17 @@ impl Interpreter { line, column, } => { - // Get the request object - let request_val = self.evaluate_expression(request, Rc::clone(&env)).await?; + // The request operand can call actions or wait asynchronously. + // Establish the response-attempt baseline before evaluating it + // so a client disconnect cancels that work too. + let (request_val, response_snapshot) = self + .evaluate_response_request( + *line, + *column, + "Client disconnected before the response request was resolved", + self.evaluate_expression(request, Rc::clone(&env)), + ) + .await?; let request_id = match &request_val { Value::Object(obj) => { let obj_ref = obj.borrow(); @@ -10065,12 +10161,13 @@ impl Interpreter { // Only after evaluation do we take the sender into the completion // guard. Early eval errors leave the id in open_pending so the // handler-exit 500 path still resolves the client. - let response = self + let (response, response_snapshot) = self .evaluate_response_precommit( &request_id, *line, *column, "Client disconnected before the response was sent", + response_snapshot, async { // Evaluate response content. Binary values are carried through // as raw bytes so fonts/images/etc. serve losslessly; text and @@ -10230,9 +10327,18 @@ impl Interpreter { .await?; // Now commit: take the sender (disconnect signal ends) and deliver. - let mut completion = self + let mut completion = match self .take_pending_response_completion(&request_id, *line, *column) - .await?; + .await + { + Ok(completion) => completion, + Err(error) => { + if error.kind == ErrorKind::Cancelled { + self.cancel_response_precommit(&request_id, response_snapshot); + } + return Err(error); + } + }; match completion.take_sender() { Some(sender) => { if sender.send(HandlerReply::Buffered(response)).is_err() { @@ -10242,6 +10348,7 @@ impl Interpreter { // so the concurrent loop's structural-failure breaker // skips it (a burst of post-dequeue disconnects must not // tear the server down). + self.cancel_response_precommit(&request_id, response_snapshot); return Err(RuntimeError::with_kind( "Client disconnected before the response was sent".to_string(), *line, @@ -10270,8 +10377,16 @@ impl Interpreter { line, column, } => { - // Resolve the request id, mirroring `respond`. - let request_val = self.evaluate_expression(request, Rc::clone(&env)).await?; + // Resolve the request id under the same cancellation baseline as + // the streaming-head fields and final transport commit. + let (request_val, response_snapshot) = self + .evaluate_response_request( + *line, + *column, + "Client disconnected before the streaming response request was resolved", + self.evaluate_expression(request, Rc::clone(&env)), + ) + .await?; let request_id = match &request_val { Value::Object(obj) => match obj.borrow().get("_response_sender") { Some(Value::Text(id)) => id.as_ref().to_string(), @@ -10295,12 +10410,13 @@ impl Interpreter { // Keep pending parked through status/content-type/header // evaluation so disconnect still cancels any upstream work those // expressions perform. Handler-exit 500 covers early eval errors. - let (status_code, content_type_str, custom_headers) = self + let ((status_code, content_type_str, custom_headers), response_snapshot) = self .evaluate_response_precommit( &request_id, *line, *column, "Client disconnected before the streaming response started", + response_snapshot, async { let status_code = match status { Some(expr) => { @@ -10403,9 +10519,18 @@ impl Interpreter { // Commit: take the sender and hand the streaming head to the transport. let (tx, rx) = mpsc::channel::>(RESPONSE_STREAM_BUFFER); - let mut completion = self + let mut completion = match self .take_pending_response_completion(&request_id, *line, *column) - .await?; + .await + { + Ok(completion) => completion, + Err(error) => { + if error.kind == ErrorKind::Cancelled { + self.cancel_response_precommit(&request_id, response_snapshot); + } + return Err(error); + } + }; match completion.take_sender() { Some(sender) => { if sender @@ -10421,6 +10546,7 @@ impl Interpreter { // cooperative cancellation, not a fault (see the buffered // `respond` path). `Cancelled` keeps the concurrent // breaker from counting a disconnect as a failure. + self.cancel_response_precommit(&request_id, response_snapshot); return Err(RuntimeError::with_kind( "Client disconnected before the streaming response started" .to_string(), From 7554ed1271325c373a651ab71b3afe6cf947b345 Mon Sep 17 00:00:00 2001 From: Brad Byrd Date: Sat, 25 Jul 2026 01:00:05 -0500 Subject: [PATCH 122/132] test: complete issue 642 evidence coverage --- Dev diary/2026-07-24-issue-642-completion.md | 362 ++++++++---------- Docs/development/response-streaming-design.md | 22 ++ .../2026-07-24-pr-641-red-chronology.md | 248 ++++++++++++ src/interpreter/mod.rs | 106 ++++- .../concurrent_disconnect_paths_burst_test.rs | 174 +++++++-- tests/flush_action_backcompat_test.rs | 45 +++ tests/http_server_streaming_test.rs | 50 +++ tests/open_file_local_type_test.rs | 7 +- .../typechecker_response_stream_scope_test.rs | 50 ++- tests/write_web_postfix_test.rs | 127 ++++++ 10 files changed, 947 insertions(+), 244 deletions(-) create mode 100644 Docs/development/testing-policy-exceptions/2026-07-24-pr-641-red-chronology.md diff --git a/Dev diary/2026-07-24-issue-642-completion.md b/Dev diary/2026-07-24-issue-642-completion.md index 1498a12c..19ca3705 100644 --- a/Dev diary/2026-07-24-issue-642-completion.md +++ b/Dev diary/2026-07-24-issue-642-completion.md @@ -1,199 +1,177 @@ -# Dev Diary - 2026-07-24: issue #642 completion pass +# Dev Diary — 2026-07-24: PR #641 / issue #642 re-review repair -Issue [#642](https://github.com/WebFirstLanguage/wfl/issues/642) re-reviewed -PR #641 at `b25aed57` and identified five P1 groups plus missing R3 -lifecycle evidence. The completion pass fixes the remaining behavior, hardens -the tests so they prove the promised boundaries, and records unrelated -platform defects discovered by the full presubmit. +Issue [#642](https://github.com/WebFirstLanguage/wfl/issues/642) requested a +fresh review of PR #641 from reviewed head +`8e8be0fcde944d0d7b357b94d5951497af5ff0b7`. This pass repaired every newly +confirmed product defect in auditable test-only Red → later Green commits, +replaced false-positive R3 tests with causal tests, and recorded the older +evidence gap without rewriting history. -## Risk and compatibility +No merge or PR comment is part of this work. The previously preserved green +Actions run is `30142079511`; it is not final evidence for the repaired +candidate. Final local gates and a new complete Linux/Windows Actions matrix are +required after the documentation and characterization commit. + +## Risk, compatibility, and gate status - **Risk class:** R3. -- **Issue triggers:** concurrency, cancellation, HTTP lifecycle, streaming, - resource ownership, arbitrary duration configuration, and backward - compatibility of existing WFL expressions. -- **Additional gate trigger:** untrusted archive paths and path containment. - This was a pre-existing Windows portability defect found by the full gate, - not a sixth issue #642 requirement. -- **Public contract:** no existing WFL syntax is intentionally removed or - changed. The fixes restore classic `write` and `flush` expression behavior - and preserve typed timeout/cancellation outcomes. -- **External state:** none. Rollback is a source revert; there is no data - migration or persistent-state recovery step. - -## Selected design - -The completion pass keeps the existing architecture and hardens each boundary: - -1. **Concurrent server:** handler state records sticky request acceptance. - A centralized classifier separates request-local outcomes from structural - pre-request failures. Pending-response ownership is checked before - expression evaluation and consumed atomically only at response commit. -2. **Outbound streams:** each handle keeps a stable slot and a first-wins - `watch` terminal reason. Expiry records a `Timeout` tombstone, wakes an - active reader, drops a parked upstream body, and refuses reinsertion. EOF, - error, and close abort the per-stream reaper. -3. **Ambiguous writes:** type checking follows the branch selected by a - concrete target and checks every viable branch for a gradual target. - Typechecker-local and inherited container context participates in - definedness and property typing. -4. **Merged operands:** one seeded-expression continuation parser is shared by - write operands, response content type, response headers, and legacy flush - candidates. Response-clause boundaries propagate through recursive primary - wrappers and explicit-call argument lists. -5. **Flush compatibility:** the parser records the original merged binding as - `legacy_binding` metadata before split/find/replace rewrites. Analyzer, - static unused-variable analysis, typechecker, and runtime consult the same - metadata, then dispatch the full fallback AST through ordinary - expression-statement semantics. - -This removes the reported races and restores grammar parity without changing -public `ErrorKind` variants or adding a second request-lease subsystem. - -## Acceptance criteria to regression coverage - -| Issue requirement | Regression evidence | -|---|---| -| Request-local failures cannot stop the server | `concurrent_disconnect_paths_burst_test`: two serialized 256-client waves per disconnect path (512 disconnects total), exact checkpoint/ack synchronization, handler-start ordinal 768 (the initial 256 plus one replacement for every consumed disconnect result), then `/ping`; every socket operation and join is bounded | -| Owned missing pending entry is cancellation; duplicate remains an error | `concurrent_handler_classification_tests::missing_pending_entry_is_cancelled_only_while_the_handler_owns_it` | -| Repeated finite request waits survive | 257 direct classifier observations plus a real server using 1 ms waits until handler-start ordinal 512 | -| Expiry/read ownership is atomic and preserves `Timeout` | active-read real-socket test, unread-expiry test, and deterministic `take -> expire -> ready result -> put` unit ordering | -| Stream EOF is stable | final unterminated line, then exactly one EOF `nothing`, then the documented closed-handle result | -| Reaper resources remain bounded | retained-runtime live-reaper counter across rapid close, clean EOF, truncated-body error, and opened-but-unread expiry | -| Extreme duration cannot panic or disable the cap | deadline unit test proves `u64::MAX` becomes a finite one-year cap and diagnostics use that effective duration | -| Concrete and gradual write branches are sound | one-sided leads in both directions, property roots, handler/action scopes, direct and inherited container properties, wrapped file/stream/list payloads, gradual definedness, and gradual payload tests | -| Merged operands match ordinary expression grammar | 3 x 7 parser matrix plus clause boundaries in concatenation, `at` indexes, nested `of` calls, builtins, unary operands, explicit `call ... with ...` arguments, and `file exists at ` | -| Full flush fallback is preserved | non-callable, overload, nested postfix, binary continuation, `of` calls, split/find/replace rewrites, invalid container-property types, and unused-variable accounting for `legacy_binding` and fallback operands | -| Backpressure test proves the intended path | client confirms the 200 head, stays connected without reading, then asserts the exact typed stall timeout and lower bound | -| Dropped pending request sends the explicit 500 | exact dequeue/drop/release synchronization followed by exact status, content type, body, prompt completion, and clean interpreter/server joins | - -## Red evidence observed - -The following focused regressions failed before their matching implementation -changes: - -- GitHub Actions run `30106107011` and the local focused run: - `http_stream_test::test_next_line_returns_final_unterminated_line` returned - `Unknown or already-closed stream handle 'httpstream1'` instead of EOF - `nothing`. -- Active hard-expiry cancellation produced `ErrorKind::General` with a closed - stream message instead of `ErrorKind::Timeout`. -- Reading an unread expired stream produced an unknown-handle General error - instead of a typed timeout. -- Container method `write line value to "C:/tmp/out"` falsely reported - `Variable 'line value' is not defined`. -- Wrapped write operands skipped undefined names in concrete-file, - concrete-stream, and gradual-target branches. -- Unmerged response content type `"text/" with subtype` stopped parsing at - `with`. -- Response clauses were swallowed by concatenation operands, `at` indexes, and - recursively nested `of`/builtin/unary expressions. -- `call render with value and headers h` swallowed `headers h` as a second - explicit-call argument. -- `file exists at paths at kind and headers h` absorbed the headers clause - into the nested index/Boolean-And expression. -- `flush cache plus 1` left `plus` dangling, and `flush cache[0][0]` selected - the short `cache` root at runtime. -- Split/find/replace flush rewrites discarded the original legacy binding, and - an invalid live container property was not rejected by the typechecker. -- Static unused-variable analysis reported both `flush cache` and `dead`; only - `dead` should have been unused. -- The original lifecycle tests could finish without proving the post- - disconnect checkpoint, exact drop point, bounded joins, or admission beyond - the breaker threshold. Deterministic synchronization was added before the - production behavior was accepted as Green. - -The original server-breaker defect is anchored to issue head `b25aed57`. Only -GitHub Actions run `30106107011` is retained policy-compliant CI Red evidence. -The other Reds were observed locally but are not preserved as Red commits or -durable CI artifacts. - -A pre-existing zero-byte `.git/objects/maintenance.lock` (dated -2026-07-18 03:36 local time) prevented Git object writes; its owning process -was not established. Consequently there is no committed Red-to-Green ancestry -for the local regressions, including the existing archive reproduction test. -This is a testing-policy handoff limitation and must not be overstated as -formal Red evidence. - -## Unrelated full-gate defects repaired - -These changes are not issue #642 acceptance items, but each blocked or weakened -the repository's required verification: - -- `file_io_performance_test::test_directory_listing_performance` recursively - scanned the repository and `target` (about 65,000 files), exceeded its - 10-second bound at 11.28 seconds, and left 30 fixtures. It now uses an - auto-cleaned temporary directory; focused Green was 1/1 in 0.02 seconds. -- On Windows, `Path::is_absolute()` did not classify the portable archive entry - `/etc/shadow` as absolute. The existing containment guard still rejected it - as escaping the destination, but with the wrong classification and message. - `Path::has_root()` now performs portable rooted-path rejection; the - `wflpkg` security suite is 31/31. -- `execute_file_test` reserved port 58123. It now uses `free_tcp_port()` to - avoid unrelated local collisions. -- Windows PowerShell 5.1 rejected inherited environments containing identical - `Path` and `PATH` keys before `Start-Process` could run. Both official - scripts now canonicalize only identical duplicates and fail closed on - conflicting values. The integration runner also retains the child process - handle before a timed wait so a real exit code is available. -- `subprocess_comprehensive.wfl` treated shell-only `echo` as a Windows - executable. It now uses `cargo --version`, an existing runner prerequisite, - explicitly waits after output capture so shutdown is orphan-free, and uses a - repo-owned blocking WFL helper to prove the child is running before kill and - absent afterward. The helper carries the first-line `CI-SKIP` directive used - by both platform runners so it is never treated as a standalone test. - -## Green evidence - -Focused final results: - -- Language-focused combined run: 68/68. -- `write_web_postfix_test`: 21/21. -- Static-analyzer focused units: 15/15. -- Parser units: 110/110. -- Strengthened disconnect binary: 4/4 in about 11.5 seconds; the handler-entry - barrier case also passed focused 1/1. -- Directory-performance fixture: 1/1 in 0.02 seconds. -- `cargo test -p wflpkg --test security_tests --verbose`: 31/31. -- Final subprocess fixture: exit 0 with a live-child kill assertion and no - orphan warning. - -Final-tree gates: - -| Command | Result | -|---|---| -| `cargo fmt --all -- --check` | pass | -| `git diff --check` | pass | -| `cargo clippy --all-targets --all-features -- -D warnings` | pass | -| `cargo build --release` | pass | -| `cargo test --all --verbose --jobs 2` | pass; core 627 passed / 6 ignored, all workspace integration packages passed, WFL doctests 28 passed / 11 ignored | -| `scripts/run_integration_tests.ps1 -TestOnly` | pass; Rust integration binaries passed and TestPrograms finished 110 passed / 0 failed / 24 explicit skips | -| `scripts/run_web_tests.ps1` | pass; 2/2 HTTP tests, TLS script case explicitly skipped because OpenSSL was unavailable | -| Git Bash syntax + first-line skip probe | pass; the Unix runner parses and recognizes the helper's `CI-SKIP` directive | -| `python scripts/validate_docs_examples.py --ci --force` | pass; 18/18 examples across validation layers | - -The first unbounded `cargo test --all --verbose` attempt hit a pre-test Windows -linker fan-out failure, `LNK1104: cannot open msvcrt.lib`. The library was -present and readable, and the exact failed target linked immediately -afterward. `--jobs 2` preserved the complete test selection while bounding -concurrent linkers. - -The Cargo cache also reported a read-only last-use database in this sandbox. -That warning did not affect dependency resolution, compilation, or test -selection. +- **Triggers:** concurrency, cancellation, HTTP lifecycle, streaming, resource + ownership, bounded retention, async control flow, and backward-compatible WFL + grammar/typechecking. +- **Compatibility:** no WFL syntax or public `ErrorKind` variant was removed. + The parser fixes restore ordinary-expression parity and legacy `flush` + behavior; typechecker fixes accept every runtime-viable classic/streaming + branch without changing runtime binding rules. +- **External state:** none. Rollback is a source revert; no data migration is + involved. +- **Policy gate:** unresolved pending maintainer approval of the Section 17 + exception for pre-existing work that lacks retained Red chronology. New + defects found during this re-review do have valid Red ancestry. + +## Implemented behavior + +1. Streaming response `status` parses the full clause-aware expression grammar + without consuming `headers`, `content type`, or `as`. +2. Seeded write/response/flush operands resume postfix composition after `of` + calls, matching ordinary expressions. +3. Bare `type` is no longer treated as a nonexistent response clause boundary. +4. Same-line unmerged `flush` operands reach stream parsing while genuinely + bare legacy bindings/actions keep their old meaning. +5. Repeat, try, and count bodies receive checker child scopes; conditional and + possibly-zero-iteration control flow conservatively joins all runtime-viable + binding types. +6. Locally opened files are recreated as `Custom("File")` when analyzer scope + reconstruction leaves no current checker symbol. +7. A final unterminated outbound line is followed by clean EOF even after the + former absolute deadline. +8. Expired unread streams release the live body, reaper, and handler ownership. + Typed terminal results use at most 64 lightweight records with a 60-second + TTL and are consumed by the next read. +9. The complete buffered/streaming response precommit phase—including the + request operand, actions it calls, all response fields, ownership precheck, + sender take, and transport commit—observes disconnects as + `ErrorKind::Cancelled`. +10. Cancellation drops the active future before restoring action/loop state and + closes only resources opened by that response attempt. Ordinary expression + failures and duplicate/forged response errors retain their prior behavior. + +## Auditable Red → Green ledger + +Every Red below is a test-only ancestor of its Green implementation. Fixture +corrections and Green-first characterization commits are listed separately and +are not represented as Red evidence. + +| Behavior | Affected base | Test-only Red | Green implementation | Focused command | +|---|---|---|---|---| +| Full streaming status operands | `8e8be0fcde944d0d7b357b94d5951497af5ff0b7` | `09115f88b0ba1bcf8ecbdba3ca81ab62eaa07e40` | `99353201518917b350009822554c7d41f6662582` | `cargo test --test write_web_postfix_test -- --nocapture --test-threads=1` | +| Post-`of` postfix continuation | `f23fb6bc0c3b2b77cf1f9eeab567b38032710f9c` | `d97f15b6d9a05be7f35d54b1bbf3d627472ea7d6` | `764685c081f62123a56cf2bbe11aa2b4617d2711` | `cargo test --test write_web_postfix_test -- --nocapture --test-threads=1` | +| Remove false bare-`type` boundary | `764685c081f62123a56cf2bbe11aa2b4617d2711` | `c8cfa08c0352555bd4d302fd4ded21b827e5ceca` | `485bc34b4daad1354b838a74e59938b5041c0db5` | `cargo test --test write_web_postfix_test -- --nocapture --test-threads=1` | +| Reach unmerged flush targets | `485bc34b4daad1354b838a74e59938b5041c0db5` | `55f3d507c741f44576afce24affbf643ee7d258e` | `4a838459bf985611338e69f81253b2a6eee0e269` | `cargo test --test flush_action_backcompat_test -- --nocapture --test-threads=1` and `cargo test --test http_server_streaming_test -- --nocapture --test-threads=1` | +| Checker child scopes | `4a838459bf985611338e69f81253b2a6eee0e269` | `8b10f8bff36fba1df4e6bae0eda07e9f05c16721` | `a1bdd9d75bd3c8134cb0fb49dc601ff209d6c26f` | `cargo test --test typechecker_response_stream_scope_test -- --nocapture --test-threads=1` | +| Conditional/loop type joins | `7bafc6da8682de19886bb3c47cc14e67c5d2b9e2` | `24f57d63dcd7018a1ea31d1f14c63c2e4069a982` | `046b012e8fd9d79a34ee2032fd2ae36da405816d` | `cargo test --test typechecker_response_stream_join_test -- --nocapture --test-threads=1` | +| Recreate local File symbols | `046b012e8fd9d79a34ee2032fd2ae36da405816d` | `a30fe4f50f8beff3d3b3af67aa234723f6d858fb` | `370073e4431af2e3cbad7273bace3ee0ff307e9d` | `cargo test --test open_file_local_type_test -- --nocapture --test-threads=1` | +| Stable clean EOF after final line | `0e98fe35415abe1e067293edfaa47a4509446303` | `5bef23578d0c315dd12b5613e63f6c9192d4e79a` | `af800a7dfe44a6188b591aebfd4f8211d51719e8` | `cargo test --lib interpreter::outbound_stream_deadline_tests::final_unterminated_line_survives_deadline_after_clean_eof -- --nocapture --test-threads=1` | +| Bounded expired-stream state | `af800a7dfe44a6188b591aebfd4f8211d51719e8` | `5d8fa3d6775145f8f63a4684f365f5f2e95c55c4` | `c7f57b9594a7286d57692efa066502fd6c08c16e` | `cargo test --lib interpreter::outbound_stream_deadline_tests::unread_expired_stream_metadata_and_ownership_are_bounded -- --nocapture --test-threads=1` | +| Cancel buffered content and streaming-head evaluation | `00a2a3fa5f60bf414ac211b2c76d546f784b0d49` | `4c45f1617097cbd39f92183bbe9dcbd986cea41d` | `0d4b26b23bcd356bd62fc4de6abf89e062e0279c` | `cargo test --lib interpreter::response_expression_disconnect_tests -- --nocapture --test-threads=1` | +| Cancel request operands; clean precheck and commit races | `edb8ce89c3d693015f655daac012c73cbc12d293` | `3bc38c668a91229e213a10d8eaebdba3789556a9` | `c73260ff61a32694c5ecfe72ab8749810033de0d` | `cargo test --lib interpreter::response_expression_disconnect_tests -- --nocapture --test-threads=1` and `cargo test --lib interpreter::response_disconnect_result_tests -- --nocapture --test-threads=1` | + +The intended Red failures included incomplete/misbounded ASTs, unreachable flush +forms, leaked checker types, unknown local File types, stale-deadline Timeout, +unbounded live stream/owner populations, response evaluation that remained +pending after its client disconnected, stale pending-response ownership, and +upstream streams retained after commit-time cancellation. + +Post-Green fixture corrections were +`f23fb6bc0c3b2b77cf1f9eeab567b38032710f9c`, +`7bafc6da8682de19886bb3c47cc14e67c5d2b9e2`, +`f0dc05db5b2c6722a3a400e629544295a3b07609`, and +`85768c2384b2e414fe66c26751b28f12f2614890`. They correct or broaden +test fixtures; none is claimed as a new Red. + +## R3 characterization and preservation evidence + +- `0e98fe35415abe1e067293edfaa47a4509446303` proves the classic write fallback + with a real opened File handle. +- `90a225d4de36fc74a0b17e4312889c2fa3511c93` makes simultaneous body/expiry + arbitration deterministic. +- `edb8ce89c3d693015f655daac012c73cbc12d293` replaces timing-only lifecycle + coverage with active-read close, spawned-reaper, exact disconnect + classification, zero/fractional timeout, backpressure, and real client + disconnect tests. The real TCP test covers buffered content plus streaming + status/content-type/headers, asserts upstream EOF, and proves `/ping` + remains serviceable. +- `tests/concurrent_disconnect_paths_burst_test.rs` uses causal release markers + and iteration barriers. Each 256-client wave is fully consumed before the + next wave or `/ping`; fixed handler sleeps are not used as proof. +- Green-first breadth checks cover builtin status operands, ordinary/seeded AST + parity and runtime behavior after `of`, a genuinely bare non-callable + `flush` binding, outer Text/File scope reconstruction, and ordinary + expression/error preservation. + +Focused preservation commands run on the repaired tree include: + +```text +cargo test --lib interpreter::response_expression_disconnect_tests -- --nocapture --test-threads=1 +cargo test --lib interpreter::response_disconnect_result_tests -- --nocapture --test-threads=1 +cargo test --lib interpreter::request_wait_timeout_tests -- --nocapture --test-threads=1 +cargo test --lib interpreter::outbound_stream_deadline_tests -- --nocapture --test-threads=1 +cargo test --test response_expression_disconnect_runtime_test -- --nocapture --test-threads=1 +cargo test --test concurrent_disconnect_paths_burst_test -- --nocapture --test-threads=1 +cargo clippy --lib -- -D warnings +``` + +All completed focused commands passed without retry, skip, quarantine, +weakened assertions, or replacement with timing-only assertions. + +## Historical evidence gap + +The original PR work before reviewed head `8e8be0fc` does not have retained +test-only Red ancestors for every behavioral change. Actions run `30106107011` +is the only located durable pre-Green Red artifact for that earlier work. +Writing passing tests now, reverting finished code, or rewriting commit history +would not establish the missing chronology. + +The repository therefore contains a narrowly scoped Section 17 exception draft +under `Docs/development/testing-policy-exceptions/`. It records the exact +missing rule/scope, reason, compensating verification, residual risk, +containment, rollback, owner, repair deadline, and seven-day R3 expiry. It is +explicitly **PENDING MAINTAINER APPROVAL**. Until approved, the testing-policy +merge/release gate remains unresolved; the exception does not turn missing +evidence into a pass. + +## Required final verification + +The final candidate must run these exact commands after all code, tests, and +documentation are committed: + +```text +cargo fmt --all -- --check +git diff --check +cargo clippy --all-targets --all-features -- -D warnings +cargo build --release +cargo test --all --verbose --jobs 2 +scripts/run_integration_tests.sh +python3 scripts/validate_docs_examples.py --ci --force +scripts/run_web_tests.sh +``` + +After push, the new GitHub Actions run must finish successfully across Linux +and Windows, including the integration gate, TestPrograms, docs validation, web +tests, TLS, PostgreSQL, MariaDB, and fuzz-target compilation. Those results +belong in the final handoff rather than being preclaimed here. ## Residual risk and recovery -- Each disconnect path covers 512 clients in two 256-client waves, with at - most 256 simultaneous handlers. This proves every disconnected result is - consumed before the post-check while staying within the configured admission - bound. -- The repeated finite-timeout proof is separate and uses handler-start ordinal - 512. -- `0` continues to disable the outbound absolute cap. Positive values above - one year use the documented one-year effective cap. -- The script-level TLS case was not run because OpenSSL was unavailable, but - the Rust TLS integration suite passed 8/8 in the workspace test gate. -- No deployment or external state changed. Reverting this source/test set is - the rollback; forward repair is preferred if a platform timing or socket- - limit issue appears. +- Recent typed stream terminals are deliberately bounded to 64 records and 60 + seconds. A much later read, or a read after capacity eviction, receives the + documented unknown/closed-handle result rather than retaining metadata + indefinitely. +- A response request operand is raced against the stable set of requests owned + when it begins; normal handlers own one request. Newly accepted resources are + treated as work created by the attempt and are cleaned if it is cancelled. +- The Section 17 approval is an explicit unresolved governance risk, not a + product-test failure. +- No deployment or persistent state changed. A rollback returns the complete + affected change set to its recorded base and reruns the gate; it must not + claim the reverted behavior remains repaired. Forward repair is preferred + for any later race or platform defect. diff --git a/Docs/development/response-streaming-design.md b/Docs/development/response-streaming-design.md index afa4039d..882949e0 100644 --- a/Docs/development/response-streaming-design.md +++ b/Docs/development/response-streaming-design.md @@ -148,6 +148,19 @@ close out (pre-`start streaming response` — the head phase — polled via `is_closed()`). So a browser disconnect cancels the handler whether it is blocked opening the upstream head or reading its body, dropping the upstream. +- Whole response evaluation is also cancellation-aware. The runtime snapshots + handler state and owned resources before evaluating the `respond` or + `start streaming response` request operand, races that operand and every + fallible response field against the pending request's disconnect signal, and + carries the same snapshot through the transport commit. Cancellation drops + the active evaluation future first, then restores action/loop state and closes + only outbound streams, response streams, and pending requests created by that + response attempt. This covers actions that sleep or perform buffered work, not + only outbound streaming operations that already observe disconnects. +- A disconnect discovered by the ownership precheck, sender take, or final + oneshot send follows the same cleanup path and returns + `ErrorKind::Cancelled`. Ordinary expression failures and duplicate/forged + response errors retain their existing classifications. - Disconnect is a normal cancellation, not a handler failure: it unwinds with `ErrorKind::Cancelled`, which the concurrent `main loop` treats as an expected outcome (it does NOT feed the structural consecutive-failure breaker), so a @@ -155,6 +168,15 @@ close out - Absolute lifetime (`outbound_stream_max_seconds`) is enforced before EVERY read return — including reads served from locally-buffered bytes — not only on a network read, so a buffered drain cannot outlive the stream's absolute cap. +- Expiry removes the heavy live stream slot, aborts the body, and immediately + removes handler ownership. To preserve the next read's typed `Timeout` + without retaining an unbounded tombstone table, the registry keeps at most 64 + lightweight terminal records for at most 60 seconds; reading a record consumes + it. Active readers share a first-wins terminal signal, and the post-`select!` + recheck makes expiry win over a simultaneously ready body chunk. +- Clean EOF is terminal independently of the old absolute deadline. A final + unterminated line is returned once, the following read returns `nothing`, and + only a later read reports the documented closed-handle error. - Outbound close-on-exit (shipped): outbound `httpstream*` handles are also handler-owned — tracked in `RunState.open_http_streams` (swapped per poll) and dropped from `IoClient.stream_handles` when the handler ends on any path, diff --git a/Docs/development/testing-policy-exceptions/2026-07-24-pr-641-red-chronology.md b/Docs/development/testing-policy-exceptions/2026-07-24-pr-641-red-chronology.md new file mode 100644 index 00000000..95f08036 --- /dev/null +++ b/Docs/development/testing-policy-exceptions/2026-07-24-pr-641-red-chronology.md @@ -0,0 +1,248 @@ +# WFL testing-policy exception draft: PR #641 Red chronology + +> **STATUS: DRAFT — PENDING MAINTAINER APPROVAL** +> +> This exception is not active. It does not turn missing Red evidence into a +> pass. PR #641's merge and release gates remain unresolved until every approval +> condition and signature below is complete. Silence, a green test run, or merge +> authority alone is not approval. + +## Exception record + +| Field | Value | +|---|---| +| Exception ID | `WFL-TEST-EXC-2026-07-24-PR641` | +| Repository | `WebFirstLanguage/wfl` | +| Change record | [PR #641](https://github.com/WebFirstLanguage/wfl/pull/641) | +| Repair ticket | [Issue #642](https://github.com/WebFirstLanguage/wfl/issues/642), item 11 | +| Risk class | **R3** — concurrency, cancellation, lifecycle, streaming, backward compatibility, untrusted archive input, and release-test controls | +| Requested start | `2026-07-24T00:00:00-05:00` (`America/Chicago`) | +| Expiration | `2026-07-31T00:00:00-05:00` (`America/Chicago`), exactly seven days after the requested start | +| Maximum affected releases | **One** WFL release: the first release containing the approved candidate, and no later release | +| Affected base | `b25aed57ea50697c596796446d1f47466668773d` | +| Commits containing the earlier mixed Green work | `5e01e446ab9250d72a0f255bc81a27a79c5b5d63`, `fce5d86fe923666885e40ec484d902cfd18c4c85`, and `8e8be0fcde944d0d7b357b94d5951497af5ff0b7` | +| Exact executable candidate for this draft | `c73260ff61a32694c5ecfe72ab8749810033de0d` | +| Requested project/reliability owner approval | Brad, Maintainer, Logbie LLC — **PENDING** | +| Requested security-owner approval | Brad, Maintainer, Logbie LLC — **PENDING**, required for the archive-path item | + +The exact executable candidate above is the latest production-code commit +covered by this draft. Evidence-only test or documentation descendants do not +expand the affected production scope. Any later executable change, force-push, +or different candidate invalidates this draft until the SHA, scope, evidence, +and approvals are updated and reviewed again. + +If approval occurs after the requested start, the exception becomes active only +at the recorded approval time and still expires at the fixed expiration above. +It never applies retroactively to authorize an earlier merge or release. + +## Exact rule and affected scope + +This draft requests a temporary exception only from the retained chronology +requirements in root `testing.md`: + +- Section 6.1, which requires the Red step before the Green implementation; +- Section 6.2, which requires a test-only Red ancestor or independently + timestamped pre-Green artifact tied to the affected base and requires the + base, Red, and Green identifiers; and +- Section 15, only the requirement to attach retained Red evidence for each + behavior or defect fix. + +It does **not** waive the R3 classification, any required test layer, Section +11.3 concurrency/lifecycle coverage, independent review, a required CI job, a +known product failure, or any non-waivable condition in Section 14. A required +test may not be skipped, retried to manufacture Green, quarantined, muted, +weakened, or relabeled under this record. + +The missing chronology is limited to the following behavior that remains in the +candidate from the earlier mixed implementation commits: + +| Area | Exact behavior lacking retained pre-Green Red chronology | Present regression/verification surface | +|---|---|---| +| Concurrent request containment | Once a handler has accepted a request, request-local `Cancelled`, finite request-wait timeout, response-send failure, and other post-accept failures do not feed the global structural-failure breaker; an owned pending request removed by sibling pruning is cancellation, while an unowned duplicate response remains an ordinary error; unrelated `/ping` work stays serviceable after disconnect bursts. | `concurrent_disconnect_paths_burst_test`, concurrent-handler classifier units, and finite request-timeout units | +| Pending-response and dropped-run cleanup | Dropping a handler/interpreter future closes its owned pending request and server response streams, emits the documented dropped-request response where applicable, and does not leave work attached to a reused interpreter. | `dropped_interpret_server_cleanup_test` and interpreter cleanup units | +| Response-stream backpressure | A connected client that stops reading cannot park a response-stream write indefinitely; timeout and disconnect remain distinct typed outcomes, and unrelated handlers continue. | `response_stream_backpressure_test` and response-disconnect classifier units | +| Outbound-stream deadline and close lifecycle | The configured absolute lifetime includes response-head time, bounds active and unread streams, closes the upstream socket, wakes an active reader with the typed terminal result, gives expiry priority on a ready/expiry tie, prevents reinsertion after terminal state, aborts reapers on terminal paths, and safely clamps extreme positive durations. | `outbound_stream_deadline_test`, `outbound_stream_open_expiry_test`, `outbound_stream_reaper_race_test`, and deterministic active-read close/expiry units | +| Ambiguous classic/streaming writes | Type checking selects the runtime-viable classic file or response-stream branch for concrete targets and checks both branches for gradual targets; container/property context and the analyzer's shared continuations do not hide undefined names or reject the inactive reading. | `ambiguous_write_branch_typecheck_test`, `ambiguous_write_analyzer_test`, `stream_handle_type_test`, and focused analyzer/typechecker units | +| Merged write and response operands | Existing write, response content-type, and response-header operands retain ordinary expression composition for property/index/`at` access, concatenation, nested calls, builtins, unary forms, explicit call arguments, and clause boundaries. This row excludes the later status-operand and post-`of` fixes listed below. | `write_web_postfix_test` and parser AST units | +| Legacy merged `flush` compatibility | A previously valid action, overload, non-callable binding, split/find/replace binding, postfix target, or full fallback expression beginning with merged `flush` retains expression-statement behavior; analyzer, unused-variable analysis, typechecker, and runtime use the same preserved legacy binding metadata. This row excludes the later unmerged-target dispatch and post-`of` fixes listed below. | `flush_action_backcompat_test`, `write_web_postfix_test`, and static-analyzer units | +| Fractional request waits | A positive request timeout below one millisecond is rejected deterministically instead of being rounded into the distinct zero/unlimited behavior. | request-wait timeout units | +| Portable archive containment | A rooted archive entry such as `/etc/shadow` is rejected as rooted on Windows as well as Unix before extraction and cannot escape the destination. | `wflpkg` security tests, including archive traversal/rooted-path cases | +| Gate and fixture correctness | The official Windows integration runner handles equivalent `Path`/`PATH` entries without hiding conflicting values and retains the child process exit status; the Windows web runner fails on cleanup failures; subprocess, free-port, and directory-performance fixtures exercise repository-owned bounded resources instead of shell-only commands, fixed ports, or the repository tree. | Official Linux/Windows integration and web scripts, `execute_file_test`, `file_io_performance_test`, and `subprocess_comprehensive.wfl` | + +The following repairs are explicitly **outside** this exception because the +repair branch contains genuine test-only Red ancestors followed by Green +implementation commits: + +- complete status-clause operands; +- postfix continuation after `of`; +- removal of the nonexistent bare `type` response boundary; +- unmerged streaming `flush` dispatch; +- ResponseStream child scopes and conservative branch/loop joins; +- local opened-File symbol recreation; +- clean EOF after a final unterminated line; +- bounded expired-stream terminal metadata; +- response-expression disconnect cancellation, including request operands, + early prechecks, and commit-time cleanup. + +The final-unterminated-line defect also has retained pre-Green CI evidence in +[Actions run 30106107011](https://github.com/WebFirstLanguage/wfl/actions/runs/30106107011). +Neither that behavior nor any later genuine Red-to-Green repair depends on this +exception. + +## Why normal compliance cannot now be supplied + +The earlier implementation combined regression tests and production changes in +the same commits. The focused failures described in the completion diary were +observed locally, but no test-only ancestor commit and no independently +timestamped pre-Green artifact was retained for the affected behaviors above. +A local `.git/objects/maintenance.lock` blocked the intended Git object writes +during that pass. + +The missing historical ordering cannot be created after the implementation +date. Reverting or disabling completed code now would only demonstrate test +sensitivity; under Section 6.2 it would not prove the original TDD chronology. +Rewriting timestamps or presenting later characterization as earlier Red +evidence would manufacture evidence and is prohibited. This request is +therefore for temporarily unavailable historical evidence, not for schedule +pressure, test duration, inconvenience, a small-change claim, or permission to +ignore a current failure. + +## Current Green evidence + +- The reviewed Green head + `8e8be0fcde944d0d7b357b94d5951497af5ff0b7` completed + [Actions run 30142079511](https://github.com/WebFirstLanguage/wfl/actions/runs/30142079511), + including Linux and Windows integration, TestPrograms, documentation + validation, web/TLS, PostgreSQL, MariaDB, and fuzz-target compilation. +- The completion diary maps the affected contracts to focused Rust, + real-socket, parser/typechecker/analyzer, WFL end-to-end, and security tests. +- Later issue #642 repairs use retained test-only Red ancestors and Green + commits; those repairs strengthen the candidate but do not retroactively + supply the chronology missing from the earlier mixed commits. + +Actions run 30142079511 is evidence for the reviewed Green head, not automatic +evidence for the exact candidate in this draft. Before approval, the approval +record must link one complete, successful, unretried Actions run for the exact +final candidate (or its evidence-only descendant) and the final local gate +record. Until those fields are complete, current Green evidence is incomplete +for merge. + +## Compensating verification and containment + +Approval is conditional on all of the following: + +1. Run, once and without changing test selection: + `cargo fmt --all -- --check`, `git diff --check`, + `cargo clippy --all-targets --all-features -- -D warnings`, + `cargo build --release`, `cargo test --all --verbose --jobs 2`, + `scripts/run_integration_tests.sh`, + `python3 scripts/validate_docs_examples.py --ci --force`, and + `scripts/run_web_tests.sh`. +2. Preserve the exact commands, exit conclusions, candidate SHA, and complete + logs in the PR evidence record. +3. Require one complete GitHub Actions matrix on the exact candidate, covering + Linux and Windows integration, TestPrograms, documentation validation, web + tests, TLS, PostgreSQL, MariaDB, and fuzz-target compilation. Every required + job must pass. +4. Obtain an independent R3 review of the implementation, regression + assertions, real-boundary coverage, cleanup paths, and this exception's + exact scope. +5. Confirm in the approval record that no required test was skipped, retried, + quarantined, muted, weakened, or converted into a timing-only success + assertion. +6. Freeze the executable candidate after approval. Any executable change + requires a new full gate, scope review, exact SHA, and approval decision. +7. Do not release more than the single affected release, and do not merge or + release after expiration. Expiration fails closed. + +These controls establish present behavior and contain the exposure. They do not +replace or reconstruct the missing historical chronology. + +## Residual risk + +- Because the tests and earlier implementation were committed together, the + record cannot prove that each test was specified independently of the chosen + implementation. A test could encode the implementation while missing a + different contract-preserving failure mode. +- Concurrency and socket lifecycle tests cover deterministic checkpoints and + supported CI platforms, but do not exhaust every OS scheduler, socket-buffer + size, cancellation ordering, or long-duration accumulation pattern. +- Parser/typechecker compatibility matrices cover the reported operand and + scope shapes but cannot prove compatibility for every existing WFL program. +- Windows and Unix archive containment tests cover known rooted and traversal + forms but do not constitute a proof over every filesystem namespace or future + archive format. +- The older local Red observations are narrative only. They must not be cited + as policy-compliant Red evidence. + +There is no accepted known product-test failure in this draft. Discovery of a +reproducible product failure, authorization bypass, data loss/corruption, +exposed secret, unresolved Critical vulnerability, or another Section 14 +non-waivable condition immediately invalidates this exception and blocks merge +or release. + +## Rollback and recovery + +No deployment, schema, persistent data, or external service state is changed by +PR #641. + +- **Before merge:** stop the PR and rebuild the candidate from affected base + `b25aed57ea50697c596796446d1f47466668773d`, preserving genuine test-only Red + commits before each production repair. Do not force-push or rewrite evidence + without an explicit maintainer decision and a retained mapping from the old + candidate to the replacement. +- **After merge, before release:** revert the PR's merge/squash commit (or the + exact affected commits if merged unsquashed), then run the complete gate on + the revert candidate. Prefer forward repair when a broad revert would remove + compatibility or lifecycle fixes that other changes now depend on. +- **After the one permitted release:** publish a normal tested forward repair + or a revert release under the ordinary release gate. This exception cannot be + reused for that release. + +If a concurrency or cleanup regression appears, first disable release of the +candidate, preserve the failing boundary evidence, and repair it with a genuine +Red commit. If archive containment regresses, stop distribution of the affected +package artifacts and route the finding through `SECURITY.md`; do not disclose +new vulnerability details in a public issue. + +## Repair ticket, owner, and deadline + +- **Ticket:** [WebFirstLanguage/wfl issue #642](https://github.com/WebFirstLanguage/wfl/issues/642), + testing-policy evidence gap. +- **Owner:** Brad, Maintainer and WFL test/reliability owner, Logbie LLC. +- **Deadline:** `2026-07-31T00:00:00-05:00`, before this exception expires and + before merge or release. +- **Required resolution:** either (a) locate and retain admissible pre-Green + artifacts for every row above, reducing or eliminating this scope; (b) + replace the mixed implementation stack from the recorded affected base with + genuine Red-to-Green ancestry and rerun the full gate; or (c) complete and + approve this narrowly scoped record for the exact candidate and archive it + with the release evidence. A later characterization run alone does not + satisfy options (a) or (b). + +Issue closure does not itself approve this exception. If the deadline passes +without one of these resolutions, the exception expires and the affected merge +or release remains blocked. + +## Approval record — must be completed before activation + +| Approval field | Required entry | +|---|---| +| Exact final executable candidate SHA | **PENDING** | +| Final evidence-only descendant SHA, if any | **PENDING / N/A** | +| Final local gate record | **PENDING** — commands, date, environment, and conclusions | +| Final GitHub Actions run | **PENDING** — URL and every required job conclusion | +| Independent R3 reviewer | **PENDING** — identity, date, and scope reviewed | +| Requester | **PENDING** — identity and date | +| Project/reliability owner decision | **PENDING** — Brad must record `APPROVE` or `REJECT`, rationale, date, and signature | +| Security-owner decision for archive-path scope | **PENDING** — Brad must record `APPROVE` or `REJECT`, rationale, date, and signature | +| No skip/retry/quarantine/muting/weakening/timing-only conversion attestation | **PENDING** | +| Maximum-release and expiration acknowledgment | **PENDING** | + +The requester must not be the sole approver. If Brad is also the requester, a +separate authorized project/domain approver must approve; an independent review +that has no approval authority is not a substitute. The completed record must +remain attached to the PR and archived with release evidence for the retention +period required by Section 15. + +**PENDING MAINTAINER APPROVAL — PR #641 MERGE GATE UNRESOLVED.** diff --git a/src/interpreter/mod.rs b/src/interpreter/mod.rs index 3f01b80c..772fac9d 100644 --- a/src/interpreter/mod.rs +++ b/src/interpreter/mod.rs @@ -1803,9 +1803,9 @@ impl StreamCancel { } } -/// Why a stream slot was terminated. Timeout remains in the stable slot until -/// the next read consumes it, so an unread expired handle keeps its typed -/// terminal reason instead of degrading to an unknown-handle error. +/// Why a stream slot was terminated. Active readers observe the shared +/// first-wins signal; unread expiry is retained briefly in the bounded recent +/// terminal queue so the next read can still report the typed reason. #[derive(Debug, Clone, Copy, PartialEq, Eq)] enum StreamTerminal { Timeout, @@ -1874,8 +1874,8 @@ impl StreamRegistry { /// Reads take the inner [`HttpStreamHandle`] out for the duration of the await /// (so the global map lock is not held across the network). Explicit /// finish/close removes the slot after signalling [`StreamCancel`]; expiry -/// records a timeout tombstone and drops the parked body so mid-read work aborts -/// without losing the terminal reason. +/// drops the parked body, removes the live slot, and records a bounded recent +/// terminal so mid-read work aborts without losing the reason. struct StreamSlot { /// The live body handle. `None` while a body read owns it. handle: Option, @@ -15390,6 +15390,102 @@ mod response_disconnect_result_tests { ); } + #[tokio::test] + async fn ordinary_response_expression_errors_remain_general_and_pending() { + let interpreter = Interpreter::new(); + let env = Rc::clone(interpreter.global_env()); + env.borrow_mut() + .define_or_replace("req", request_value("request-error")); + let (sender, _receiver) = oneshot::channel(); + interpreter.pending_responses.borrow_mut().insert( + "request-error".to_string(), + PendingResponse { + sender: Arc::new(tokio::sync::Mutex::new(Some(sender))), + }, + ); + interpreter + .open_pending_requests + .borrow_mut() + .push("request-error".to_string()); + + let statement = parse_statement("respond to req with missing_value"); + let error = interpreter + .execute_statement(&statement, env) + .await + .expect_err("undefined content must remain an ordinary expression error"); + assert_eq!(error.kind, ErrorKind::General, "wrong error: {error:?}"); + assert!( + error.message.contains("missing_value"), + "ordinary expression diagnostic changed: {error}" + ); + assert!( + interpreter + .pending_responses + .borrow() + .contains_key("request-error"), + "ordinary expression error consumed the pending response" + ); + assert!( + interpreter + .open_pending_requests + .borrow() + .iter() + .any(|id| id == "request-error"), + "ordinary expression error removed handler ownership" + ); + } + + #[tokio::test] + async fn successful_buffered_response_behavior_is_preserved() { + let interpreter = Interpreter::new(); + let env = Rc::clone(interpreter.global_env()); + env.borrow_mut() + .define_or_replace("req", request_value("request-success")); + let (sender, receiver) = oneshot::channel(); + interpreter.pending_responses.borrow_mut().insert( + "request-success".to_string(), + PendingResponse { + sender: Arc::new(tokio::sync::Mutex::new(Some(sender))), + }, + ); + interpreter + .open_pending_requests + .borrow_mut() + .push("request-success".to_string()); + + let statement = parse_statement( + "respond to req with \"ok\" and content_type \"text/plain\" and status 201", + ); + interpreter + .execute_statement(&statement, env) + .await + .expect("connected buffered response must still succeed"); + let reply = receiver.await.expect("buffered response was not delivered"); + match reply { + HandlerReply::Buffered(response) => { + assert_eq!(response.status, 201); + assert_eq!(response.content_type, "text/plain"); + assert_eq!(response.content, b"ok"); + } + HandlerReply::Streaming { .. } => panic!("buffered respond produced streaming reply"), + } + assert!( + !interpreter + .pending_responses + .borrow() + .contains_key("request-success"), + "successful response remained pending" + ); + assert!( + !interpreter + .open_pending_requests + .borrow() + .iter() + .any(|id| id == "request-success"), + "successful response retained handler ownership" + ); + } + #[tokio::test] async fn buffered_commit_disconnect_closes_evaluation_streams() { assert_commit_disconnect_closes_evaluation_stream( diff --git a/tests/concurrent_disconnect_paths_burst_test.rs b/tests/concurrent_disconnect_paths_burst_test.rs index 75ac6cc2..d4f9dba4 100644 --- a/tests/concurrent_disconnect_paths_burst_test.rs +++ b/tests/concurrent_disconnect_paths_burst_test.rs @@ -15,11 +15,15 @@ //! breaker threshold), and an explicit handler-start barrier proves every intended //! result was consumed before probing `/ping` (so a General-classified disconnect //! cannot race past a premature success that resets the counter). +//! Exact `ErrorKind::Cancelled` assertions remain at the interpreter unit layer; these +//! real-boundary bursts prove the externally observable breaker and liveness contract. use std::collections::HashSet; +use std::path::PathBuf; use std::sync::Arc; use std::sync::atomic::{AtomicUsize, Ordering}; use std::time::Duration; +use tempfile::TempDir; use tokio::io::{AsyncReadExt, AsyncWriteExt}; use tokio::sync::{mpsc, watch}; use wfl::Interpreter; @@ -34,14 +38,7 @@ const DISCONNECT_WAVE: usize = 256; /// Both waves disconnect before `/ping`, so each path exercises 512 clients (>256). const DISCONNECT_TOTAL: usize = DISCONNECT_WAVE * 2; const _: () = assert!(DISCONNECT_TOTAL > 256); -/// The initial 256 handlers plus one replacement handler per consumed disconnect -/// across both waves. Observing this ordinal proves every one of the 512 intended -/// disconnect outcomes left `FuturesUnordered` before `/ping` is allowed to run. -const POST_DISCONNECT_BARRIER: usize = DISCONNECT_WAVE + DISCONNECT_TOTAL; const WAVE_DEADLINE: Duration = Duration::from_secs(30); -/// The WFL handler waits this long after its checkpoint is released. That gives the -/// test time to close every downstream socket before `respond` / response-head send. -const POST_CHECKPOINT_DELAY_MS: u64 = 1_000; const ITERATION_PROOF_DEADLINE: Duration = Duration::from_secs(20); /// These cases deliberately fill the 256-handler cap. Rust's test harness otherwise @@ -50,6 +47,53 @@ const ITERATION_PROOF_DEADLINE: Duration = Duration::from_secs(20); /// full handler wave while preserving the real >256-request breaker proof. static HEAVY_CASE_LOCK: tokio::sync::Mutex<()> = tokio::sync::Mutex::const_new(()); +/// A causal release latch visible to WFL through `file exists at`. +/// +/// The marker is created before a wave begins. Handlers poll its existence after +/// reaching the precise response lifecycle point under test. The test removes it +/// only after every browser task has returned, which is positive proof that all +/// intended client sockets were dropped. The short wait inside the WFL loop is only +/// a cooperative polling yield; marker removal, not elapsed time, releases handlers. +struct WaveLatch { + _directory: TempDir, + marker: PathBuf, + wfl_path: String, +} + +impl WaveLatch { + fn new(context: &str) -> Self { + let directory = tempfile::Builder::new() + .prefix("wfl-disconnect-release-") + .tempdir() + .unwrap_or_else(|error| panic!("{context}: create release-latch directory: {error}")); + let marker = directory.path().join("hold-wave"); + let wfl_path = marker + .to_string_lossy() + .replace('\\', "/") + .replace('"', "\\\""); + Self { + _directory: directory, + marker, + wfl_path, + } + } + + fn hold(&self, wave: usize, context: &str) { + assert!( + !self.marker.exists(), + "{context}: release marker unexpectedly existed before wave {wave}" + ); + std::fs::write(&self.marker, format!("hold {context} wave {wave}\n")).unwrap_or_else( + |error| panic!("{context}: create release marker for wave {wave}: {error}"), + ); + } + + fn release(&self, wave: usize, context: &str) { + std::fs::remove_file(&self.marker) + .unwrap_or_else(|error| panic!("{context}: release wave {wave}: {error}")); + } +} + struct CountingGate { port: u16, arrivals: mpsc::UnboundedReceiver, @@ -210,6 +254,7 @@ struct IterationCounter { port: u16, arrivals: mpsc::UnboundedReceiver, errors: mpsc::UnboundedReceiver, + observed: HashSet, } /// Count a handler-start request and return an empty response immediately. A @@ -273,6 +318,7 @@ async fn spawn_iteration_counter() -> IterationCounter { port, arrivals, errors, + observed: HashSet::new(), } } @@ -282,8 +328,7 @@ async fn wait_for_proven_handler_iterations( context: &str, ) { let deadline = tokio::time::Instant::now() + ITERATION_PROOF_DEADLINE; - let mut seen = HashSet::with_capacity(expected); - while seen.len() < expected { + while !(1..=expected).all(|ordinal| counter.observed.contains(&ordinal)) { let ordinal = tokio::time::timeout_at(deadline, async { tokio::select! { ordinal = counter.arrivals.recv() => { @@ -300,20 +345,30 @@ async fn wait_for_proven_handler_iterations( }) .await .unwrap_or_else(|_| { + let proven = (1..=expected) + .filter(|ordinal| counter.observed.contains(ordinal)) + .count(); panic!( "observed only {} of {expected} {context} handler starts; the \ {expected}th start is required to prove the loop consumed every \ intended result before the liveness probe", - seen.len() + proven ) }); assert!( - seen.insert(ordinal), + counter.observed.insert(ordinal), "iteration counter duplicated handler-start ordinal {ordinal}" ); } } +fn post_wave_iteration_target(wave: usize) -> usize { + // The loop initially fills all 256 slots. Each completed disconnect wave must + // then yield another 256 starts. Reaching this exact prefix proves every result + // from this wave left FuturesUnordered before the next wave or `/ping`. + DISCONNECT_WAVE * (wave + 1) +} + async fn wait_for_gate_arrivals( arrivals: &mut mpsc::UnboundedReceiver, errors: &mut mpsc::UnboundedReceiver, @@ -551,16 +606,20 @@ async fn join_client_wave( /// Drive two full checkpointed waves. All 256 first-wave handlers are held inside /// the checkpoint simultaneously. The second wave cannot put all 256 handlers into -/// that checkpoint until the loop has consumed every first-wave result. After this -/// returns, the separate handler-start barrier proves the second-wave results were -/// consumed too, before `/ping` is sent. +/// that checkpoint until the loop has consumed every first-wave result. The marker +/// latch remains held until all 256 browser tasks confirm their sockets are dropped; +/// after release, an exact handler-start prefix proves this wave's results were +/// consumed before the next wave or `/ping`. async fn drive_two_gated_disconnect_waves( port: u16, path: &'static str, gate: &mut CountingGate, + latch: &WaveLatch, + iterations: &mut IterationCounter, context: &str, ) { for wave in 1..=2 { + latch.hold(wave, context); let (disconnect, clients) = spawn_gated_client_wave(port, path); wait_for_gate_arrivals(&mut gate.arrivals, &mut gate.errors, wave, context).await; gate.release_wave @@ -577,16 +636,30 @@ async fn drive_two_gated_disconnect_waves( .send(true) .expect("all gated clients remain alive until explicitly disconnected"); join_client_wave(clients, wave, context).await; + latch.release(wave, context); + wait_for_proven_handler_iterations(iterations, post_wave_iteration_target(wave), context) + .await; } } -/// The streaming-head variant needs no auxiliary checkpoint: receiving a valid -/// response head is itself the lifecycle proof. As above, all 256 second-wave heads -/// can only arrive after every first-wave disconnect result was consumed. -async fn drive_two_stream_disconnect_waves(port: u16, path: &'static str, context: &str) { +/// Receiving a valid streaming head is the lifecycle proof for the write path. +/// Every browser task drops its socket while the marker remains held. Only after +/// all 256 tasks return does the test release the handlers to write, then the +/// iteration prefix proves all results were consumed. +async fn drive_two_stream_disconnect_waves( + port: u16, + path: &'static str, + latch: &WaveLatch, + iterations: &mut IterationCounter, + context: &str, +) { for wave in 1..=2 { + latch.hold(wave, context); let clients = spawn_stream_client_wave(port, path); join_client_wave(clients, wave, context).await; + latch.release(wave, context); + wait_for_proven_handler_iterations(iterations, post_wave_iteration_target(wave), context) + .await; } } @@ -648,11 +721,12 @@ async fn test_disconnect_before_buffered_respond_does_not_kill_the_loop() { let _heavy_case = HEAVY_CASE_LOCK.lock().await; let mut gate = spawn_counting_gate().await; let mut iterations = spawn_iteration_counter().await; + let latch = WaveLatch::new("buffered-respond disconnect"); let port = common::free_tcp_port(); - // `/slow` reaches the counting checkpoint, waits after its release, then - // responds. The test disconnects every client during that wait, so the buffered - // `respond` send fails (or the pending entry is sibling-pruned). That must be a - // cancellation, not a structural failure. + // `/slow` reaches the counting checkpoint and then parks on a filesystem + // marker. The test removes that marker only after every client task confirms + // its socket was dropped, so the subsequent buffered response deterministically + // sees a disconnected receiver. let code = format!( r#" listen on port {port} as srv @@ -674,7 +748,9 @@ async fn test_disconnect_before_buffered_respond_does_not_kill_the_loop() { open url at "http://127.0.0.1:{gate_port}/ack" and stream response as acknowledgement wait for next chunk from acknowledgement as acknowledged close acknowledgement - wait for {POST_CHECKPOINT_DELAY_MS} milliseconds + repeat while file exists at "{release_path}": + wait for 1 milliseconds + end repeat respond to req with "late" end check end check @@ -682,14 +758,17 @@ async fn test_disconnect_before_buffered_respond_does_not_kill_the_loop() { "#, gate_port = gate.port, counter_port = iterations.port, + release_path = latch.wfl_path.as_str(), ); let server = start_proxy_server(code); wait_for_server(port).await; - drive_two_gated_disconnect_waves(port, "/slow", &mut gate, "buffered-respond disconnect").await; - wait_for_proven_handler_iterations( + drive_two_gated_disconnect_waves( + port, + "/slow", + &mut gate, + &latch, &mut iterations, - POST_DISCONNECT_BARRIER, - "buffered-disconnect", + "buffered-respond disconnect", ) .await; assert_ping_survives(port, "buffered-respond disconnect").await; @@ -700,9 +779,11 @@ async fn test_disconnect_before_buffered_respond_does_not_kill_the_loop() { async fn test_disconnect_before_stream_write_does_not_kill_the_loop() { let _heavy_case = HEAVY_CASE_LOCK.lock().await; let mut iterations = spawn_iteration_counter().await; + let latch = WaveLatch::new("stream-write disconnect"); let port = common::free_tcp_port(); - // `/stream` sends the head, waits (the client reads the head then disconnects), - // then writes — the write send fails. That must be a cancellation, not a failure. + // `/stream` sends the head and parks on a filesystem marker. Each client reads + // that head and drops its socket; only after all client tasks return does the + // test remove the marker and let the handler attempt its writes. let code = format!( r#" listen on port {port} as srv @@ -720,7 +801,9 @@ async fn test_disconnect_before_stream_write_does_not_kill_the_loop() { break otherwise: start streaming response to req with status 200 and content type "text/plain" as out - wait for 300 milliseconds + repeat while file exists at "{release_path}": + wait for 1 milliseconds + end repeat store payload as "0123456789" count from 1 to 9: store payload as payload with payload @@ -734,13 +817,15 @@ async fn test_disconnect_before_stream_write_does_not_kill_the_loop() { end loop "#, counter_port = iterations.port, + release_path = latch.wfl_path.as_str(), ); let server = start_proxy_server(code); wait_for_server(port).await; - drive_two_stream_disconnect_waves(port, "/stream", "stream-write disconnect").await; - wait_for_proven_handler_iterations( + drive_two_stream_disconnect_waves( + port, + "/stream", + &latch, &mut iterations, - POST_DISCONNECT_BARRIER, "stream-write-disconnect", ) .await; @@ -753,11 +838,12 @@ async fn test_disconnect_before_streaming_head_does_not_kill_the_loop() { let _heavy_case = HEAVY_CASE_LOCK.lock().await; let mut gate = spawn_counting_gate().await; let mut iterations = spawn_iteration_counter().await; + let latch = WaveLatch::new("pre-streaming-head disconnect"); let port = common::free_tcp_port(); // Client disconnects *before* the streaming head is sent (no head read). The - // handler reaches the counting checkpoint, parks after its release, then reaches - // `start streaming response` with a missing/closed pending entry — must be - // Cancelled, not a structural General that trips the breaker after >256 instances. + // handler reaches the counting checkpoint and then parks on a marker. The test + // releases it only after every client socket is confirmed dropped, so + // `start streaming response` deterministically sees the disconnected request. let code = format!( r#" listen on port {port} as srv @@ -779,7 +865,9 @@ async fn test_disconnect_before_streaming_head_does_not_kill_the_loop() { open url at "http://127.0.0.1:{gate_port}/ack" and stream response as acknowledgement wait for next chunk from acknowledgement as acknowledged close acknowledgement - wait for {POST_CHECKPOINT_DELAY_MS} milliseconds + repeat while file exists at "{release_path}": + wait for 1 milliseconds + end repeat start streaming response to req with status 200 and content type "text/plain" as out write line "late" to out close out @@ -789,15 +877,17 @@ async fn test_disconnect_before_streaming_head_does_not_kill_the_loop() { "#, gate_port = gate.port, counter_port = iterations.port, + release_path = latch.wfl_path.as_str(), ); let server = start_proxy_server(code); wait_for_server(port).await; - drive_two_gated_disconnect_waves(port, "/prehead", &mut gate, "pre-streaming-head disconnect") - .await; - wait_for_proven_handler_iterations( + drive_two_gated_disconnect_waves( + port, + "/prehead", + &mut gate, + &latch, &mut iterations, - POST_DISCONNECT_BARRIER, - "pre-streaming-head-disconnect", + "pre-streaming-head disconnect", ) .await; assert_ping_survives(port, "pre-streaming-head disconnect").await; diff --git a/tests/flush_action_backcompat_test.rs b/tests/flush_action_backcompat_test.rs index 8facc05a..404ef5bc 100644 --- a/tests/flush_action_backcompat_test.rs +++ b/tests/flush_action_backcompat_test.rs @@ -78,6 +78,27 @@ fn truly_bare_flush_still_calls_the_legacy_zero_argument_action() { ); } +#[test] +fn truly_bare_flush_still_evaluates_a_non_callable_legacy_variable() { + let src = "store flush as 1\n\ + flush\n\ + display flush\n"; + let (out, code) = run_src(src); + assert_eq!( + code, + Some(0), + "the exact bare `flush` variable must remain a valid expression statement; output:\n{out}" + ); + assert!( + out.contains('1'), + "the bare legacy variable must remain readable after evaluation; output:\n{out}" + ); + assert!( + !out.to_lowercase().contains("stream"), + "the exact bare `flush` variable must not become a stream operation; output:\n{out}" + ); +} + #[test] fn flush_without_a_matching_action_still_errors_as_a_stream_flush() { // With no action `flush cache` and no stream `cache`, `flush cache` falls @@ -235,6 +256,30 @@ fn flush_with_of_call_uses_the_full_legacy_action_name() { ); } +#[test] +fn flush_with_post_of_index_preserves_the_legacy_action_result() { + let src = "define action called flush cache with parameters values:\n\ + \x20\x20\x20\x20return values\n\ + end action\n\ + store items as [7]\n\ + flush cache of (items)[0]\n\ + display \"OK\"\n"; + let (out, code) = run_src(src); + assert_eq!( + code, + Some(0), + "the indexed result of the full legacy action must remain an expression statement; output:\n{out}" + ); + assert!( + out.contains("OK"), + "the legacy post-`of` expression must complete; output:\n{out}" + ); + assert!( + !out.to_lowercase().contains("stream"), + "the legacy action result must not be reinterpreted as a stream target; output:\n{out}" + ); +} + #[test] fn flush_split_rewrite_keeps_the_original_legacy_binding() { let src = "store flush cache as 1\n\ diff --git a/tests/http_server_streaming_test.rs b/tests/http_server_streaming_test.rs index 6b1bdef1..e0fb8bcb 100644 --- a/tests/http_server_streaming_test.rs +++ b/tests/http_server_streaming_test.rs @@ -287,6 +287,56 @@ async fn test_streamed_response_lines_and_headers() { join_server(server_handle); } +#[tokio::test] +async fn test_post_of_index_operands_execute_for_content_write_and_flush() { + let port = common::free_tcp_port(); + let server_code = format!( + r#" + define action called choose with parameters values: + return values + end action + define action called cache with parameters values: + return values + end action + store types as ["text/plain"] + store chunks as ["post-of body"] + listen on port {port} as s + wait for request comes in on s as req with timeout 10000 + start streaming response to req with status 200 and content type choose of (types)[0] as out + store streams as [out] + write line choose of (chunks)[0] to out + flush cache of (streams)[0] + close out + close server s + "# + ); + + let server_handle = start_server_thread(server_code); + wait_for_server(port).await; + + let response = reqwest::Client::new() + .get(format!("http://127.0.0.1:{port}/post-of")) + .send() + .await + .expect("request failed"); + assert_eq!(response.status().as_u16(), 200); + assert_eq!( + response + .headers() + .get("content-type") + .and_then(|value| value.to_str().ok()), + Some("text/plain"), + "the post-`of` index must select the content type returned by `choose`" + ); + assert_eq!( + response.text().await.expect("read response body"), + "post-of body\n", + "the indexed return value must reach the streamed write" + ); + + join_server(server_handle); +} + #[tokio::test] async fn test_write_after_close_does_not_reach_client() { // Writing after `close out` is a catchable error and does NOT reach the diff --git a/tests/open_file_local_type_test.rs b/tests/open_file_local_type_test.rs index 822ec5e8..0fad03b0 100644 --- a/tests/open_file_local_type_test.rs +++ b/tests/open_file_local_type_test.rs @@ -1,5 +1,8 @@ //! Regression coverage for local `open file ... as ...` bindings that analyzer //! body scopes do not retain for the type-checker pass. +//! +//! These type-checker tests do not claim that runtime file bindings can shadow +//! an outer variable: `Environment::define` rejects parent-scope collisions. use std::fs; use std::process::Command; @@ -63,7 +66,9 @@ fn fresh_local_file_handles_are_concrete_in_action_loop_and_method_scopes() { } #[test] -fn opening_a_local_file_shadows_instead_of_retyping_an_outer_binding() { +fn reconstructed_local_file_type_does_not_retype_an_outer_visible_binding() { + // The type checker must reconstruct `out` as File while checking the loop, + // then expose the original outer Text binding after leaving that scope. let source = "store out as \"outer.txt\"\n\ main loop:\n\ \x20\x20\x20\x20open file at \"inner.txt\" for writing as out\n\ diff --git a/tests/typechecker_response_stream_scope_test.rs b/tests/typechecker_response_stream_scope_test.rs index b98d78b1..553f6835 100644 --- a/tests/typechecker_response_stream_scope_test.rs +++ b/tests/typechecker_response_stream_scope_test.rs @@ -1,5 +1,8 @@ -//! Regression coverage for type-checker scopes that must mirror runtime child -//! environments when response-stream bindings shadow outer file handles. +//! Regression coverage for response-stream symbols reconstructed in type-checker +//! child scopes after analyzer body scopes have been discarded. +//! +//! These tests cover type visibility during checking, not runtime shadowing: +//! `Environment::define` rejects parent-scope name collisions. use wfl::lexer::lex_wfl_with_positions; use wfl::parser::Parser; @@ -14,7 +17,7 @@ fn typecheck(code: &str) -> Result<(), String> { } #[test] -fn response_stream_bindings_do_not_escape_runtime_child_scopes() { +fn response_stream_bindings_do_not_escape_typechecker_child_scopes() { let scoped_blocks = [ "repeat while false:\n\ \x20\x20\x20\x20start streaming response to \"request\" with status 200 as out\n\ @@ -40,13 +43,52 @@ fn response_stream_bindings_do_not_escape_runtime_child_scopes() { ); assert!( typecheck(&source).is_ok(), - "a response stream created in a runtime child scope must not replace \ + "a response stream reconstructed in a type-checker child scope must not replace \ the outer File type; source:\n{source}\nerrors: {:?}", typecheck(&source).err() ); } } +#[test] +fn response_stream_bindings_are_local_while_outer_text_remains_visible_afterward() { + let scoped_blocks = [ + "repeat while false:\n\ + \x20\x20\x20\x20start streaming response to \"request\" with status 200 as out\n\ + \x20\x20\x20\x20flush out\n\ + end repeat\n", + "try:\n\ + \x20\x20\x20\x20start streaming response to \"request\" with status 200 as out\n\ + \x20\x20\x20\x20flush out\n\ + when error:\n\ + \x20\x20\x20\x20display \"ignored\"\n\ + end try\n", + "count from 1 to 1:\n\ + \x20\x20\x20\x20start streaming response to \"request\" with status 200 as out\n\ + \x20\x20\x20\x20flush out\n\ + end count\n", + ]; + + for scoped_block in scoped_blocks { + let source = format!( + "store out as \"outer text\"\n\ + {scoped_block}\ + store invalid as out minus 1\n" + ); + let errors = typecheck(&source) + .expect_err("the outer Text binding must remain Text after the child scope"); + assert!( + errors.contains("Cannot perform Minus operation"), + "expected the restored outer Text subtraction error; source:\n{source}\nerrors: {errors}" + ); + assert!( + !errors.contains("`flush` requires a response-stream handle"), + "the local binding must be visible as ResponseStream while checking its child scope; \ + source:\n{source}\nerrors: {errors}" + ); + } +} + #[test] fn default_count_binding_does_not_retype_an_outer_count_variable() { let source = "store count as \"outside\"\n\ diff --git a/tests/write_web_postfix_test.rs b/tests/write_web_postfix_test.rs index 0c01b465..969a1d9b 100644 --- a/tests/write_web_postfix_test.rs +++ b/tests/write_web_postfix_test.rs @@ -118,6 +118,38 @@ fn streaming_status_clause_accepts_full_expressions_without_swallowing_headers() } } +#[test] +fn streaming_status_clause_accepts_a_builtin_call_without_swallowing_headers() { + let program = parse( + "start streaming response to req with status abs of requested_status and headers h as out\n", + ); + assert_eq!(program.statements.len(), 1, "got {:#?}", program.statements); + + let (status, headers) = streaming_status_and_headers(&program.statements[0]); + assert!( + matches!( + status, + Expression::FunctionCall { + function, + arguments, + .. + } if matches!(function.as_ref(), Expression::Variable(name, ..) if name == "abs") + && matches!( + arguments.as_slice(), + [wfl::parser::ast::Argument { + value: Expression::Variable(name, ..), + .. + }] if name == "requested_status" + ) + ), + "the builtin status operand must remain `abs of requested_status`; got {status:#?}" + ); + assert!( + matches!(headers, Expression::Variable(name, ..) if name == "h"), + "headers must remain a separate response clause; got {headers:#?}" + ); +} + #[test] fn write_and_streaming_clauses_share_the_full_expression_suffix_grammar() { let cases = [ @@ -646,6 +678,50 @@ fn assert_post_of_index(expr: &Expression, expected_function: &str, expected_arg } } +fn post_of_index_shape(expr: &Expression) -> (String, String, i64) { + match expr { + Expression::IndexAccess { + collection, index, .. + } => { + let index = match index.as_ref() { + Expression::Literal(wfl::parser::ast::Literal::Integer(index), ..) => *index, + other => panic!("expected an integer post-call index, got {other:#?}"), + }; + match collection.as_ref() { + Expression::FunctionCall { + function, + arguments, + .. + } => { + let function = leftmost_variable(function).unwrap_or_else(|| { + panic!("expected a variable call root, got {function:#?}") + }); + let argument = match arguments.as_slice() { + [ + wfl::parser::ast::Argument { + value: Expression::Variable(name, ..), + .. + }, + ] => name, + other => panic!("expected one variable call argument, got {other:#?}"), + }; + (function.to_string(), argument.to_string(), index) + } + other => panic!("expected the index to wrap an `of` call, got {other:#?}"), + } + } + other => panic!("expected a post-`of` index expression, got {other:#?}"), + } +} + +fn initializer_expression(source: &str) -> Expression { + let program = parse(&format!("store parity result as {source}\n")); + match &program.statements[0] { + Statement::VariableDeclaration { value, .. } => value.clone(), + other => panic!("expected a variable initializer, got {other:#?}"), + } +} + #[test] fn seeded_operands_resume_postfix_parsing_after_of_calls() { let write = parse("write line choose of (chunks)[0] to out\n"); @@ -698,6 +774,57 @@ fn seeded_operands_resume_postfix_parsing_after_of_calls() { } } +#[test] +fn post_of_operands_match_the_ordinary_expression_ast_shape() { + let write = parse("write line choose of (chunks)[0] to out\n"); + let ordinary_write = initializer_expression("choose of (chunks)[0]"); + assert_eq!( + post_of_index_shape(stream_write_value(&write.statements[0])), + post_of_index_shape(&ordinary_write), + "the seeded write operand must compose exactly like an ordinary expression" + ); + let ordinary_classic_write = initializer_expression("line choose of (chunks)[0]"); + assert_eq!( + post_of_index_shape(stream_write_fallback(&write.statements[0])), + post_of_index_shape(&ordinary_classic_write), + "the classic write fallback must preserve the ordinary post-`of` AST" + ); + + let streaming = + parse("start streaming response to req with content type choose of (types)[0] as out\n"); + let ordinary_content_type = initializer_expression("choose of (types)[0]"); + assert_eq!( + post_of_index_shape(streaming_clause_operand( + &streaming.statements[0], + "content type", + )), + post_of_index_shape(&ordinary_content_type), + "the content-type operand must compose exactly like an ordinary expression" + ); + + let flush = parse("flush cache of (items)[0]\n"); + let (target, fallback) = match &flush.statements[0] { + Statement::FlushStreamStatement { + target, + action_fallback: Some(fallback), + .. + } => (target, fallback), + other => panic!("expected an ambiguous FlushStreamStatement, got {other:#?}"), + }; + let ordinary_flush_target = initializer_expression("cache of (items)[0]"); + assert_eq!( + post_of_index_shape(target), + post_of_index_shape(&ordinary_flush_target), + "the streaming flush target must preserve the ordinary post-`of` AST" + ); + let ordinary_legacy_flush = initializer_expression("flush cache of (items)[0]"); + assert_eq!( + post_of_index_shape(fallback), + post_of_index_shape(&ordinary_legacy_flush), + "the legacy flush fallback must preserve the ordinary post-`of` AST" + ); +} + #[test] fn write_line_of_call_argument_absorbs_arithmetic() { // `double of n minus 1` must parse as `double of (n minus 1)` — the same From c9c748ce850b7d106ffef90e299e4b7221411517 Mon Sep 17 00:00:00 2001 From: Brad Byrd Date: Sat, 25 Jul 2026 01:13:25 -0500 Subject: [PATCH 123/132] docs:correct-response-stream-scope-characterization --- tests/typechecker_response_stream_scope_test.rs | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/tests/typechecker_response_stream_scope_test.rs b/tests/typechecker_response_stream_scope_test.rs index 553f6835..ea993e36 100644 --- a/tests/typechecker_response_stream_scope_test.rs +++ b/tests/typechecker_response_stream_scope_test.rs @@ -1,8 +1,8 @@ //! Regression coverage for response-stream symbols reconstructed in type-checker //! child scopes after analyzer body scopes have been discarded. //! -//! These tests cover type visibility during checking, not runtime shadowing: -//! `Environment::define` rejects parent-scope name collisions. +//! These tests cover type visibility and restoration during checking. Runtime +//! shadows response-stream names inside the corresponding child environments. use wfl::lexer::lex_wfl_with_positions; use wfl::parser::Parser; From 96d53052388f75bd809c2af42f12445944e8fc69 Mon Sep 17 00:00:00 2001 From: Brad Byrd Date: Sat, 25 Jul 2026 02:23:25 -0500 Subject: [PATCH 124/132] test: expose retained clean eof stream state --- src/interpreter/mod.rs | 122 +++++++++++++++++++++++++++++++++++++++-- 1 file changed, 117 insertions(+), 5 deletions(-) diff --git a/src/interpreter/mod.rs b/src/interpreter/mod.rs index 772fac9d..b20a442e 100644 --- a/src/interpreter/mod.rs +++ b/src/interpreter/mod.rs @@ -16112,11 +16112,11 @@ mod outbound_stream_deadline_tests { timeout_seconds: 10, ..WflConfig::default() }); - let client = IoClient::new(Arc::clone(&config)); + let interpreter = Interpreter::with_config(Arc::clone(&config)); let budget = Arc::new(ExecutionBudget::from_config(&config)); let (_, _, handle) = tokio::time::timeout( Duration::from_secs(3), - client.open_http_stream( + interpreter.io_client.open_http_stream( "GET", &format!("http://127.0.0.1:{port}/unterminated"), &[], @@ -16127,10 +16127,19 @@ mod outbound_stream_deadline_tests { .await .expect("open stream hung") .expect("open stream"); + interpreter + .io_client + .claim_stream_owner( + &handle, + &Arc::clone(&interpreter.open_http_streams.borrow()), + ) + .expect("claim unterminated stream ownership"); let first = tokio::time::timeout( Duration::from_secs(3), - client.next_line(&handle, Arc::clone(&budget)), + interpreter + .io_client + .next_line(&handle, Arc::clone(&budget)), ) .await .expect("first line read hung") @@ -16141,17 +16150,47 @@ mod outbound_stream_deadline_tests { "the final unterminated line is returned only after clean EOF was observed" ); + let (live_slots, retained_terminal_records) = { + let registry = interpreter + .io_client + .stream_handles + .lock() + .unwrap_or_else(|error| error.into_inner()); + (registry.live.len(), registry.recent.len()) + }; + assert_eq!( + live_slots, 0, + "returning the final unterminated line must remove its live stream slot" + ); + assert_eq!( + retained_terminal_records, 1, + "the final line must leave exactly one lightweight clean-EOF record" + ); + assert_eq!( + interpreter + .open_http_streams + .borrow() + .lock() + .unwrap_or_else(|error| error.into_inner()) + .len(), + 0, + "returning the final line must remove handler ownership immediately" + ); + tokio::time::sleep(Duration::from_millis(1_100)).await; let eof = tokio::time::timeout( Duration::from_secs(2), - client.next_line(&handle, Arc::clone(&budget)), + interpreter + .io_client + .next_line(&handle, Arc::clone(&budget)), ) .await .expect("clean EOF read hung") .expect("clean EOF observed before the cap must not become Timeout"); assert_eq!(eof, None); - let later = client + let later = interpreter + .io_client .next_line(&handle, budget) .await .expect_err("the single clean-EOF result must consume the handle"); @@ -16161,6 +16200,79 @@ mod outbound_stream_deadline_tests { ); } + #[tokio::test] + async fn unconsumed_clean_eof_records_are_bounded() { + const STREAM_COUNT: usize = MAX_RECENT_STREAM_TERMINALS + 8; + + let port = spawn_stream_cleanup_upstream(STREAM_COUNT).await; + let config = Arc::new(WflConfig { + outbound_stream_max_seconds: 60 * 60, + timeout_seconds: 10, + ..WflConfig::default() + }); + let interpreter = Interpreter::with_config(Arc::clone(&config)); + let budget = Arc::new(ExecutionBudget::from_config(&config)); + + for sequence in 0..STREAM_COUNT { + let (_, _, handle) = interpreter + .io_client + .open_http_stream( + "GET", + &format!("http://127.0.0.1:{port}/unterminated"), + &[], + None, + Arc::clone(&budget), + ) + .await + .expect("open unterminated stream"); + interpreter + .io_client + .claim_stream_owner( + &handle, + &Arc::clone(&interpreter.open_http_streams.borrow()), + ) + .expect("claim unterminated stream ownership"); + + let final_line = interpreter + .io_client + .next_line(&handle, Arc::clone(&budget)) + .await + .expect("read final unterminated line"); + assert_eq!( + final_line.as_deref(), + Some("abc"), + "stream {sequence} did not yield its final unterminated line" + ); + + let (live_slots, recent_records) = { + let registry = interpreter + .io_client + .stream_handles + .lock() + .unwrap_or_else(|error| error.into_inner()); + (registry.live.len(), registry.recent.len()) + }; + assert_eq!( + live_slots, 0, + "stream {sequence} retained a live slot after its final line" + ); + assert_eq!( + recent_records, + (sequence + 1).min(MAX_RECENT_STREAM_TERMINALS), + "clean-EOF records must fill only the bounded recent queue" + ); + assert!( + interpreter + .open_http_streams + .borrow() + .lock() + .unwrap_or_else(|error| error.into_inner()) + .is_empty(), + "stream {sequence} retained handler ownership after its final line" + ); + } + } + #[tokio::test] async fn unread_expired_stream_metadata_and_ownership_are_bounded() { const EXPECTED_RECENT_TIMEOUT_CAPACITY: usize = 64; From 68569b31b9fd969cb5adc3b8c0832ec604bb98e2 Mon Sep 17 00:00:00 2001 From: Brad Byrd Date: Sat, 25 Jul 2026 02:25:08 -0500 Subject: [PATCH 125/132] test: expose checker backedge and handler state gaps --- .../typechecker_response_stream_join_test.rs | 81 +++++++++- .../typechecker_response_stream_scope_test.rs | 97 ++++++++++++ tests/typechecker_try_finally_join_test.rs | 144 ++++++++++++++++++ 3 files changed, 321 insertions(+), 1 deletion(-) create mode 100644 tests/typechecker_try_finally_join_test.rs diff --git a/tests/typechecker_response_stream_join_test.rs b/tests/typechecker_response_stream_join_test.rs index 1490971a..dadd9330 100644 --- a/tests/typechecker_response_stream_join_test.rs +++ b/tests/typechecker_response_stream_join_test.rs @@ -4,7 +4,7 @@ use std::sync::Arc; use wfl::lexer::lex_wfl_with_positions; use wfl::parser::Parser; -use wfl::parser::ast::{Expression, FileOpenMode, Literal, Program, Statement}; +use wfl::parser::ast::{Expression, FileOpenMode, Literal, Operator, Program, Statement}; use wfl::typechecker::TypeChecker; fn parse(source: &str) -> Program { @@ -39,6 +39,33 @@ fn stream_binding() -> Statement { } } +fn invalid_stream_lead_with_valid_file_fallback() -> Statement { + Statement::StreamWriteStatement { + value: Expression::BinaryOperation { + left: Box::new(Expression::Literal(Literal::Integer(10), 2, 1)), + operator: Operator::Minus, + right: Box::new(text_literal("not a number")), + line: 2, + column: 1, + }, + target: Expression::Variable("out".to_string(), 2, 1), + is_line: true, + fallback_content: Some(Box::new(text_literal("valid file text"))), + line: 2, + column: 1, + } +} + +fn open_out_file() -> Statement { + Statement::OpenFileStatement { + path: text_literal("unused.txt"), + variable_name: "out".to_string(), + mode: FileOpenMode::Write, + line: 1, + column: 1, + } +} + fn ambiguous_file_write_program(control: Statement) -> Program { let mut program = parse( "open file at \"unused.txt\" for writing as out\n\ @@ -96,6 +123,58 @@ fn maybe_skipped_stream_bindings_require_both_write_readings_to_be_valid() { } } +#[test] +fn while_loop_rechecks_stream_lead_after_tail_response_stream_rebind() { + let program = Program { + statements: vec![ + open_out_file(), + Statement::WhileLoop { + condition: bool_literal(true), + body: vec![ + invalid_stream_lead_with_valid_file_fallback(), + stream_binding(), + ], + line: 2, + column: 1, + }, + ], + }; + + let errors = typecheck(&program) + .expect_err("the loop backedge must recheck the body under ResponseStream or File"); + assert!( + errors.contains("Cannot perform Minus operation"), + "the first iteration has a valid File fallback, but a later iteration must reject \ + the Number/Text stream lead after the tail ResponseStream rebind; got: {errors}" + ); +} + +#[test] +fn repeat_while_loop_rechecks_stream_lead_after_tail_response_stream_rebind() { + let program = Program { + statements: vec![ + open_out_file(), + Statement::RepeatWhileLoop { + condition: bool_literal(true), + body: vec![ + invalid_stream_lead_with_valid_file_fallback(), + stream_binding(), + ], + line: 2, + column: 1, + }, + ], + }; + + let errors = typecheck(&program) + .expect_err("the repeat-loop backedge must recheck the body under ResponseStream or File"); + assert!( + errors.contains("Cannot perform Minus operation"), + "the first iteration has a valid File fallback, but a later iteration must reject \ + the Number/Text stream lead after the tail ResponseStream rebind; got: {errors}" + ); +} + #[test] fn two_concrete_branch_types_join_instead_of_taking_the_last_checked_branch() { let program = Program { diff --git a/tests/typechecker_response_stream_scope_test.rs b/tests/typechecker_response_stream_scope_test.rs index ea993e36..62a4587c 100644 --- a/tests/typechecker_response_stream_scope_test.rs +++ b/tests/typechecker_response_stream_scope_test.rs @@ -4,8 +4,10 @@ //! These tests cover type visibility and restoration during checking. Runtime //! shadows response-stream names inside the corresponding child environments. +use std::sync::Arc; use wfl::lexer::lex_wfl_with_positions; use wfl::parser::Parser; +use wfl::parser::ast::{Expression, Literal, Operator, Program, Statement, WsHandlerEvent}; use wfl::typechecker::TypeChecker; fn typecheck(code: &str) -> Result<(), String> { @@ -16,6 +18,52 @@ fn typecheck(code: &str) -> Result<(), String> { .map_err(|errors| format!("{errors:?}")) } +fn typecheck_program(program: &Program) -> Result<(), String> { + TypeChecker::new() + .check_types(program) + .map_err(|errors| format!("{errors:?}")) +} + +fn text_literal(value: &str) -> Expression { + Expression::Literal(Literal::String(Arc::from(value)), 1, 1) +} + +fn stream_binding() -> Statement { + Statement::StartStreamingResponseStatement { + request: text_literal("request"), + status: Some(Expression::Literal(Literal::Integer(200), 2, 1)), + content_type: None, + headers: None, + variable_name: "out".to_string(), + line: 2, + column: 1, + } +} + +fn outer_number_binding() -> Statement { + Statement::VariableDeclaration { + name: "out".to_string(), + value: Expression::Literal(Literal::Integer(10), 1, 1), + is_constant: false, + line: 1, + column: 1, + } +} + +fn subtract_from_outer_out() -> Statement { + Statement::DisplayStatement { + value: Expression::BinaryOperation { + left: Box::new(Expression::Variable("out".to_string(), 4, 1)), + operator: Operator::Minus, + right: Box::new(Expression::Literal(Literal::Integer(1), 4, 1)), + line: 4, + column: 1, + }, + line: 4, + column: 1, + } +} + #[test] fn response_stream_bindings_do_not_escape_typechecker_child_scopes() { let scoped_blocks = [ @@ -103,3 +151,52 @@ fn default_count_binding_does_not_retype_an_outer_count_variable() { "expected the outer Text/Number subtraction error, got: {errors}" ); } + +#[test] +fn event_handler_body_types_do_not_leak_after_registration() { + let program = Program { + statements: vec![ + outer_number_binding(), + Statement::EventHandler { + event_source: text_literal("source"), + event_name: "changed".to_string(), + handler_body: vec![stream_binding()], + line: 2, + column: 1, + }, + subtract_from_outer_out(), + ], + }; + + assert!( + typecheck_program(&program).is_ok(), + "a deferred event body runs in a fresh runtime child and must not retype outer `out`; \ + errors: {:?}", + typecheck_program(&program).err() + ); +} + +#[test] +fn websocket_handler_body_types_do_not_leak_after_registration() { + let program = Program { + statements: vec![ + outer_number_binding(), + Statement::WebSocketHandlerStatement { + event: WsHandlerEvent::Connect, + server: text_literal("WebSocketServer::127.0.0.1:0"), + binding: "conn".to_string(), + body: vec![stream_binding()], + line: 2, + column: 1, + }, + subtract_from_outer_out(), + ], + }; + + assert!( + typecheck_program(&program).is_ok(), + "a deferred WebSocket body runs in a fresh runtime child and must not retype outer \ + `out`; errors: {:?}", + typecheck_program(&program).err() + ); +} diff --git a/tests/typechecker_try_finally_join_test.rs b/tests/typechecker_try_finally_join_test.rs new file mode 100644 index 00000000..88c16add --- /dev/null +++ b/tests/typechecker_try_finally_join_test.rs @@ -0,0 +1,144 @@ +//! Regression coverage for type-state joins from `try` success/error endpoints +//! into `finally`, while keeping `when` error aliases clause-local. + +use std::sync::Arc; +use wfl::analyzer::{Analyzer, Symbol, SymbolKind}; +use wfl::parser::ast::{ + ErrorType, Expression, FileOpenMode, Literal, Operator, Program, Statement, Type, WhenClause, +}; +use wfl::typechecker::TypeChecker; + +fn text_literal(value: &str) -> Expression { + Expression::Literal(Literal::String(Arc::from(value)), 1, 1) +} + +fn stream_binding() -> Statement { + Statement::StartStreamingResponseStatement { + request: text_literal("request"), + status: Some(Expression::Literal(Literal::Integer(200), 3, 1)), + content_type: None, + headers: None, + variable_name: "out".to_string(), + line: 3, + column: 1, + } +} + +fn display_text(value: &str, line: usize) -> Statement { + Statement::DisplayStatement { + value: text_literal(value), + line, + column: 1, + } +} + +fn flush_out() -> Statement { + Statement::FlushStreamStatement { + target: Expression::Variable("out".to_string(), 5, 1), + legacy_binding: None, + action_fallback: None, + line: 5, + column: 1, + } +} + +fn subtract_one(name: &str, line: usize) -> Statement { + Statement::DisplayStatement { + value: Expression::BinaryOperation { + left: Box::new(Expression::Variable(name.to_string(), line, 1)), + operator: Operator::Minus, + right: Box::new(Expression::Literal(Literal::Integer(1), line, 1)), + line, + column: 1, + }, + line, + column: 1, + } +} + +#[test] +fn handler_response_stream_state_is_joined_before_finally() { + let program = Program { + statements: vec![ + Statement::OpenFileStatement { + path: text_literal("unused.txt"), + variable_name: "out".to_string(), + mode: FileOpenMode::Write, + line: 1, + column: 1, + }, + Statement::TryStatement { + body: vec![display_text("success", 2)], + when_clauses: vec![WhenClause { + error_type: ErrorType::General, + error_name: "caught".to_string(), + body: vec![stream_binding()], + }], + otherwise_block: None, + finally_block: Some(vec![flush_out()]), + line: 2, + column: 1, + }, + ], + }; + + assert!( + TypeChecker::new().check_types(&program).is_ok(), + "finally must see the gradual join of the successful File path and the handler's \ + ResponseStream path; errors: {:?}", + TypeChecker::new().check_types(&program).err() + ); +} + +#[test] +fn handler_error_aliases_remain_clause_local_before_finally() { + let mut analyzer = Analyzer::new(); + for name in ["caught", "error_message"] { + analyzer + .define_symbol(Symbol { + name: name.to_string(), + kind: SymbolKind::Variable { mutable: true }, + symbol_type: Some(Type::Number), + line: 1, + column: 1, + }) + .expect("define outer Number binding"); + } + + let program = Program { + statements: vec![Statement::TryStatement { + body: vec![display_text("success", 2)], + when_clauses: vec![WhenClause { + error_type: ErrorType::General, + error_name: "caught".to_string(), + body: vec![ + Statement::DisplayStatement { + value: Expression::Variable("caught".to_string(), 3, 1), + line: 3, + column: 1, + }, + Statement::DisplayStatement { + value: Expression::Variable("error_message".to_string(), 3, 1), + line: 3, + column: 1, + }, + ], + }], + otherwise_block: None, + finally_block: Some(vec![ + subtract_one("caught", 5), + subtract_one("error_message", 6), + ]), + line: 2, + column: 1, + }], + }; + + let result = TypeChecker::with_analyzer(analyzer).check_types(&program); + assert!( + result.is_ok(), + "finally must resolve the outer Number bindings, not clause-local Text aliases; \ + got: {:?}", + result.err() + ); +} From b32ff55fa76fd03b07e2ade7159d3719f2ac0642 Mon Sep 17 00:00:00 2001 From: Brad Byrd Date: Sat, 25 Jul 2026 02:29:55 -0500 Subject: [PATCH 126/132] fix: terminalize clean eof streams immediately --- src/interpreter/mod.rs | 104 +++++++++++++++++++++++++++-------------- 1 file changed, 68 insertions(+), 36 deletions(-) diff --git a/src/interpreter/mod.rs b/src/interpreter/mod.rs index b20a442e..48987ae7 100644 --- a/src/interpreter/mod.rs +++ b/src/interpreter/mod.rs @@ -1804,19 +1804,20 @@ impl StreamCancel { } /// Why a stream slot was terminated. Active readers observe the shared -/// first-wins signal; unread expiry is retained briefly in the bounded recent -/// terminal queue so the next read can still report the typed reason. +/// first-wins signal; clean EOF and unread expiry are retained briefly in the +/// bounded recent queue so the next read can consume the terminal outcome. #[derive(Debug, Clone, Copy, PartialEq, Eq)] enum StreamTerminal { + CleanEof, Timeout, Closed, } type StreamOwner = Arc>>; -/// Keep a short, bounded window of typed terminal outcomes after a reaper has -/// removed the live body. This lets the next read report `Timeout` without -/// retaining the request body, cancel channel, owner, or sleeping task. +/// Keep a short, bounded window of terminal outcomes after removing the live +/// body. This preserves one clean-EOF read or a typed timeout without retaining +/// the request body, cancel channel, owner, or sleeping task. const MAX_RECENT_STREAM_TERMINALS: usize = 64; const RECENT_STREAM_TERMINAL_TTL: Duration = Duration::from_secs(60); @@ -1875,7 +1876,8 @@ impl StreamRegistry { /// (so the global map lock is not held across the network). Explicit /// finish/close removes the slot after signalling [`StreamCancel`]; expiry /// drops the parked body, removes the live slot, and records a bounded recent -/// terminal so mid-read work aborts without losing the reason. +/// terminal so mid-read work aborts without losing the reason. Clean EOF uses +/// the same queue for its one follow-up read. struct StreamSlot { /// The live body handle. `None` while a body read owns it. handle: Option, @@ -2398,8 +2400,9 @@ impl IoClient { /// Remove a stream handle from its slot so a body read can await without /// holding the global handle lock. The cancel watch stays alive so close/ - /// expire aborts the read. Errors if unknown, closed, or past deadline. - fn take_stream(&self, handle_id: &str) -> Result { + /// expire aborts the read. A recent clean EOF yields `Ok(None)`; unknown, + /// closed, and past-deadline handles remain errors. + fn take_stream(&self, handle_id: &str) -> Result, HttpClientError> { let now = Instant::now(); let mut registry = self .stream_handles @@ -2408,7 +2411,10 @@ impl IoClient { registry.prune_recent(now); if !registry.live.contains_key(handle_id) { if let Some(terminal) = registry.take_recent(handle_id, now) { - return Err(self.stream_terminal_error(terminal)); + return match terminal { + StreamTerminal::CleanEof => Ok(None), + terminal => Err(self.stream_terminal_error(terminal)), + }; } return Err(HttpClientError::Request(format!( "Unknown or already-closed stream handle '{handle_id}'" @@ -2452,7 +2458,7 @@ impl IoClient { .expect("live stream checked above"); let cancel = Arc::clone(&slot.cancel); match slot.handle.take() { - Some(handle) => Ok(TakenStream { handle, cancel }), + Some(handle) => Ok(Some(TakenStream { handle, cancel })), None => Err(HttpClientError::Request(format!( "Unknown or already-closed stream handle '{handle_id}'" ))), @@ -2464,7 +2470,7 @@ impl IoClient { fn put_stream( &self, handle_id: &str, - mut handle: HttpStreamHandle, + handle: HttpStreamHandle, cancel: &StreamCancel, ) -> Result<(), HttpClientError> { if let Some(terminal) = cancel.terminal() { @@ -2508,30 +2514,31 @@ impl IoClient { } return Err(self.stream_terminal_error(terminal)); } + if handle.done { + drop(handle); + if let Some(mut slot) = registry.live.remove(handle_id) { + let terminal = slot.cancel.terminate(StreamTerminal::CleanEof); + if let Some(abort) = slot.reaper_abort.take() { + abort.abort(); + } + drop(slot.handle.take()); + remove_stream_owner(&mut slot, handle_id); + registry.remember_recent(handle_id.to_string(), terminal, now); + } + return Ok(()); + } + let slot = registry .live .get_mut(handle_id) .expect("live stream checked above"); - // A final unterminated line needs one subsequent read to produce - // `nothing`, matching the established WFL stream contract. Retain the - // exhausted handle for that one read, but abort its timer immediately - // so EOF never leaves a sleeping reaper task. - if handle.done { - // Clean EOF already won before the absolute deadline. Preserve the - // one established follow-up `nothing` read without letting the old - // wall-clock cap retroactively turn that EOF into Timeout. - handle.total_deadline = None; - slot.deadline = None; - if let Some(abort) = slot.reaper_abort.take() { - abort.abort(); - } - } slot.handle = Some(handle); Ok(()) } fn stream_terminal_error(&self, terminal: StreamTerminal) -> HttpClientError { match terminal { + StreamTerminal::CleanEof => HttpClientError::Closed, StreamTerminal::Timeout => self.outbound_stream_timeout_error(), StreamTerminal::Closed => HttpClientError::Closed, } @@ -2646,7 +2653,9 @@ impl IoClient { handle_id: &str, budget: Arc, ) -> Result>, HttpClientError> { - let TakenStream { mut handle, cancel } = self.take_stream(handle_id)?; + let Some(TakenStream { mut handle, cancel }) = self.take_stream(handle_id)? else { + return Ok(None); + }; if let Err(e) = self.check_stream_deadline(&handle) { let _ = self.finish_stream_slot(handle_id).await; @@ -2684,7 +2693,9 @@ impl IoClient { handle_id: &str, budget: Arc, ) -> Result, HttpClientError> { - let TakenStream { mut handle, cancel } = self.take_stream(handle_id)?; + let Some(TakenStream { mut handle, cancel }) = self.take_stream(handle_id)? else { + return Ok(None); + }; loop { if let Err(e) = self.check_stream_deadline(&handle) { @@ -2713,8 +2724,8 @@ impl IoClient { if line.last() == Some(&b'\r') { line.pop(); } - // Preserve one exhausted read so the next wait binds `nothing`. - // `put_stream` aborts the reaper before parking a done handle. + // Preserve one lightweight clean-EOF result so the next wait + // binds `nothing`; `put_stream` removes all live stream state. self.put_stream(handle_id, handle, &cancel)?; return Ok(Some(String::from_utf8_lossy(&line).into_owned())); } @@ -16150,20 +16161,27 @@ mod outbound_stream_deadline_tests { "the final unterminated line is returned only after clean EOF was observed" ); - let (live_slots, retained_terminal_records) = { + let (live_slots, clean_eof_records) = { let registry = interpreter .io_client .stream_handles .lock() .unwrap_or_else(|error| error.into_inner()); - (registry.live.len(), registry.recent.len()) + ( + registry.live.len(), + registry + .recent + .iter() + .filter(|entry| entry.reason == StreamTerminal::CleanEof) + .count(), + ) }; assert_eq!( live_slots, 0, "returning the final unterminated line must remove its live stream slot" ); assert_eq!( - retained_terminal_records, 1, + clean_eof_records, 1, "the final line must leave exactly one lightweight clean-EOF record" ); assert_eq!( @@ -16244,13 +16262,20 @@ mod outbound_stream_deadline_tests { "stream {sequence} did not yield its final unterminated line" ); - let (live_slots, recent_records) = { + let (live_slots, recent_records, all_clean_eof) = { let registry = interpreter .io_client .stream_handles .lock() .unwrap_or_else(|error| error.into_inner()); - (registry.live.len(), registry.recent.len()) + ( + registry.live.len(), + registry.recent.len(), + registry + .recent + .iter() + .all(|entry| entry.reason == StreamTerminal::CleanEof), + ) }; assert_eq!( live_slots, 0, @@ -16261,6 +16286,10 @@ mod outbound_stream_deadline_tests { (sequence + 1).min(MAX_RECENT_STREAM_TERMINALS), "clean-EOF records must fill only the bounded recent queue" ); + assert!( + all_clean_eof, + "the no-follow-up wave retained a non-clean-EOF terminal" + ); assert!( interpreter .open_http_streams @@ -16475,9 +16504,12 @@ mod outbound_stream_deadline_tests { }, ); - let TakenStream { handle, cancel } = client + let Some(TakenStream { handle, cancel }) = client .take_stream(handle_id) - .expect("active read takes body"); + .expect("active read takes body") + else { + panic!("live stream unexpectedly reported clean EOF"); + }; cancel.terminate(StreamTerminal::Timeout); let result = client.put_stream(handle_id, handle, &cancel); From 527b8fb184245e7df35fe5229b23e2a969c74520 Mon Sep 17 00:00:00 2001 From: Brad Byrd Date: Sat, 25 Jul 2026 02:34:36 -0500 Subject: [PATCH 127/132] fix: stabilize checker control-flow state --- src/analyzer/mod.rs | 28 +++++++++ src/typechecker/mod.rs | 135 ++++++++++++++++++++++++++++++++++++----- 2 files changed, 149 insertions(+), 14 deletions(-) diff --git a/src/analyzer/mod.rs b/src/analyzer/mod.rs index 20ff45d0..69694535 100644 --- a/src/analyzer/mod.rs +++ b/src/analyzer/mod.rs @@ -2753,6 +2753,21 @@ impl Analyzer { layers } + /// Snapshot the symbols owned by the current scope. + /// + /// Type-state joins normally only need [`snapshot_symbol_types`], but + /// independently checked control-flow branches can also introduce new + /// bindings. Restoring this map before each branch prevents a binding from + /// one branch shadowing names while another branch is checked. + pub fn snapshot_current_scope_symbols(&self) -> HashMap { + self.current_scope.symbols.clone() + } + + /// Restore the exact set of symbols owned by the current scope. + pub fn restore_current_scope_symbols(&mut self, symbols: HashMap) { + self.current_scope.symbols = symbols; + } + /// Restore `symbol_type` values previously captured by /// [`snapshot_symbol_types`]. Only updates symbols that still exist; does /// not remove symbols defined after the snapshot. @@ -2875,6 +2890,19 @@ impl Analyzer { } } + /// Pop the current scope while promoting every binding except the listed + /// temporary aliases into its parent. + pub fn pop_scope_promoting_except(&mut self, excluded: &[String]) { + if let Some(mut parent) = self.current_scope.parent.take() { + for (name, symbol) in std::mem::take(&mut self.current_scope.symbols) { + if !excluded.iter().any(|excluded_name| excluded_name == &name) { + parent.define_or_replace(symbol); + } + } + self.current_scope = *parent; + } + } + /// Validates a call against every registered signature of `name`: /// filters candidates by argument count, then (when several share the /// count) by static argument types. A single surviving candidate gets the diff --git a/src/typechecker/mod.rs b/src/typechecker/mod.rs index 8b5ed0ad..847a6a0f 100644 --- a/src/typechecker/mod.rs +++ b/src/typechecker/mod.rs @@ -250,6 +250,39 @@ impl TypeChecker { joined } + /// Check a loop body under the conservative type state seen at the top of + /// every iteration. Exploratory passes contribute only their backedge + /// state; diagnostics are emitted once after the header stabilizes. + fn check_loop_body_fixed_point(&mut self, body: &[Statement]) { + let entry = self.analyzer.snapshot_symbol_types(); + let mut header = entry.clone(); + + loop { + self.analyzer.restore_symbol_types(header.clone()); + let error_count = self.errors.len(); + for statement in body { + self.check_statement_types(statement); + } + if self.budget_error.is_some() { + return; + } + self.errors.truncate(error_count); + + let backedge = self.analyzer.snapshot_symbol_types(); + let next = Self::join_type_snapshots(&[entry.clone(), header.clone(), backedge]); + if next == header { + break; + } + header = next; + } + + self.analyzer.restore_symbol_types(header.clone()); + for statement in body { + self.check_statement_types(statement); + } + self.analyzer.restore_symbol_types(header); + } + /// Get the return type for builtin functions fn get_builtin_function_type(&self, name: &str, _arg_count: usize) -> Type { match name { @@ -688,6 +721,16 @@ impl TypeChecker { line: _line, column: _column, } => { + // Runtime keeps one child environment alive for every + // iteration, so bindings from a backedge are visible at the + // next header but remain local after the loop. + self.analyzer.push_scope(); + self.check_loop_body_fixed_point(body); + if self.budget_error.is_some() { + self.analyzer.pop_scope(); + return; + } + let condition_type = self.infer_expression_type(condition); if condition_type != Type::Boolean && condition_type != Type::Unknown { self.errors.push(TypeError::new( @@ -700,11 +743,6 @@ impl TypeChecker { *_column, )); } - - self.analyzer.push_scope(); - for stmt in body { - self.check_statement_types(stmt); - } self.analyzer.pop_scope(); } Statement::ExitStatement { line: _, column: _ } => {} @@ -746,9 +784,27 @@ impl TypeChecker { // Runtime evaluates the try body, handlers, otherwise, and // finally block inside one shared child environment. self.analyzer.push_scope(); + let entry_types = self.analyzer.snapshot_symbol_types(); for stmt in body { self.check_statement_types(stmt); } + if self.budget_error.is_some() { + self.analyzer.pop_scope(); + return; + } + let success_endpoint = self.analyzer.snapshot_symbol_types(); + + // An error can leave the body from any statement, so handlers + // start from the conservative entry/success join. Keep the + // success scope's symbol set as the structural baseline: + // success-only bindings remain resolvable as gradual types, + // while exact restoration prevents one handler's new symbols + // from contaminating the next handler. + let handler_entry = + Self::join_type_snapshots(&[entry_types, success_endpoint.clone()]); + let handler_scope_symbols = self.analyzer.snapshot_current_scope_symbols(); + let mut joined_scope_symbols = handler_scope_symbols.clone(); + let mut endpoints = vec![success_endpoint]; // Type check each when clause in its own scope so the bound // error name cannot clobber an outer variable of the same @@ -757,6 +813,9 @@ impl TypeChecker { // the binding lives only in the child scope (runtime does the // same via Environment::define_or_replace). for when_clause in when_clauses { + self.analyzer + .restore_current_scope_symbols(handler_scope_symbols.clone()); + self.analyzer.restore_symbol_types(handler_entry.clone()); self.analyzer.push_scope(); self.analyzer.define_or_replace_symbol(Symbol { name: when_clause.error_name.clone(), @@ -780,14 +839,57 @@ impl TypeChecker { for stmt in &when_clause.body { self.check_statement_types(stmt); } - self.analyzer.pop_scope(); + let mut excluded_aliases = vec![when_clause.error_name.clone()]; + if when_clause.error_name != "error_message" { + excluded_aliases.push("error_message".to_string()); + } + self.analyzer.pop_scope_promoting_except(&excluded_aliases); + + if self.budget_error.is_some() { + self.analyzer.pop_scope(); + return; + } + + endpoints.push(self.analyzer.snapshot_symbol_types()); + for (name, symbol) in self.analyzer.snapshot_current_scope_symbols() { + joined_scope_symbols.entry(name).or_insert(symbol); + } } if let Some(otherwise_stmts) = otherwise_block { + self.analyzer + .restore_current_scope_symbols(handler_scope_symbols.clone()); + self.analyzer.restore_symbol_types(handler_entry.clone()); for stmt in otherwise_stmts { self.check_statement_types(stmt); } + if self.budget_error.is_some() { + self.analyzer.pop_scope(); + return; + } + + endpoints.push(self.analyzer.snapshot_symbol_types()); + for (name, symbol) in self.analyzer.snapshot_current_scope_symbols() { + joined_scope_symbols.entry(name).or_insert(symbol); + } + } else if !when_clauses.iter().any(|when_clause| { + matches!( + &when_clause.error_type, + crate::parser::ast::ErrorType::General + ) + }) { + // A non-matching error reaches finally without running a + // handler when there is no catch-all or otherwise block. + endpoints.push(handler_entry.clone()); + } + + self.analyzer + .restore_current_scope_symbols(handler_scope_symbols); + for symbol in joined_scope_symbols.into_values() { + self.analyzer.define_or_replace_symbol(symbol); } + let joined_endpoint = Self::join_type_snapshots(&endpoints); + self.analyzer.restore_symbol_types(joined_endpoint); if let Some(finally_stmts) = finally_block { for stmt in finally_stmts { @@ -1661,6 +1763,11 @@ impl TypeChecker { line: _line, column: _column, } => { + self.check_loop_body_fixed_point(body); + if self.budget_error.is_some() { + return; + } + let condition_type = self.infer_expression_type(condition); if condition_type != Type::Boolean && condition_type != Type::Unknown @@ -1674,14 +1781,6 @@ impl TypeChecker { *_column, ); } - - let entry_types = self.analyzer.snapshot_symbol_types(); - for stmt in body { - self.check_statement_types(stmt); - } - let body_types = self.analyzer.snapshot_symbol_types(); - let joined = Self::join_type_snapshots(&[body_types, entry_types]); - self.analyzer.restore_symbol_types(joined); } Statement::RepeatUntilLoop { condition, @@ -2576,9 +2675,13 @@ impl TypeChecker { line: _line, column: _column, } => { + self.analyzer.push_scope(); + let outer_type_snapshot = self.analyzer.snapshot_symbol_types(); for stmt in handler_body { self.check_statement_types(stmt); } + self.analyzer.restore_symbol_types(outer_type_snapshot); + self.analyzer.pop_scope(); } Statement::ParentMethodCall { method_name: _method_name, @@ -2842,9 +2945,13 @@ impl TypeChecker { // bound variable resolves as an object at runtime (gradual typing // keeps member access like `body of msg` permissive). self.check_server_expression_type(server, *line, *column); + self.analyzer.push_scope(); + let outer_type_snapshot = self.analyzer.snapshot_symbol_types(); for stmt in body { self.check_statement_types(stmt); } + self.analyzer.restore_symbol_types(outer_type_snapshot); + self.analyzer.pop_scope(); } Statement::SendWebSocketMessageStatement { message, target, .. From 03966f06e78aec7c3bcdbd40feabc2bdff37a16d Mon Sep 17 00:00:00 2001 From: Brad Byrd Date: Sat, 25 Jul 2026 02:52:40 -0500 Subject: [PATCH 128/132] test: expose eof latch and try scope races --- src/interpreter/mod.rs | 102 +++++++++++++++++++++ tests/typechecker_try_finally_join_test.rs | 98 ++++++++++++++++++++ 2 files changed, 200 insertions(+) diff --git a/src/interpreter/mod.rs b/src/interpreter/mod.rs index 48987ae7..25ab6c64 100644 --- a/src/interpreter/mod.rs +++ b/src/interpreter/mod.rs @@ -16083,6 +16083,108 @@ mod outbound_stream_deadline_tests { ); } + #[tokio::test] + async fn observed_clean_eof_wins_over_a_later_deadline_claim() { + let config = Arc::new(WflConfig { + outbound_stream_max_seconds: 0, + timeout_seconds: 10, + ..WflConfig::default() + }); + let client = IoClient::new(Arc::clone(&config)); + let budget = Arc::new(ExecutionBudget::from_config(&config)); + let handle_id = "observed-clean-eof".to_string(); + let cancel = StreamCancel::new(); + let owner: StreamOwner = + Arc::new(std::sync::Mutex::new(HashSet::from([handle_id.clone()]))); + + client + .stream_handles + .lock() + .unwrap_or_else(|error| error.into_inner()) + .live + .insert( + handle_id.clone(), + StreamSlot { + handle: Some(HttpStreamHandle { + stream: Box::pin(futures_util::stream::empty::>>()), + buffer: b"abc".to_vec(), + done: false, + bytes_read: 3, + total_deadline: None, + }), + deadline: None, + cancel, + reaper_abort: None, + owner: Some(Arc::clone(&owner)), + }, + ); + + let Some(TakenStream { mut handle, cancel }) = client + .take_stream(&handle_id) + .expect("take synthetic stream") + else { + panic!("synthetic stream unexpectedly resolved as clean EOF"); + }; + + assert!( + !client + .stream_pull(&mut handle, &budget, &cancel) + .await + .expect("observe clean EOF"), + "the empty body must report clean EOF" + ); + assert!(handle.done, "observing EOF must mark the body complete"); + + let winner = cancel.terminate(StreamTerminal::Timeout); + assert_eq!( + winner, + StreamTerminal::CleanEof, + "a deadline claim after the upstream yielded EOF must not overwrite clean EOF" + ); + + let final_line = std::mem::take(&mut handle.buffer); + client + .put_stream(&handle_id, handle, &cancel) + .expect("terminalize the EOF-latched body"); + assert_eq!(final_line, b"abc"); + assert!( + owner + .lock() + .unwrap_or_else(|error| error.into_inner()) + .is_empty(), + "terminalization must remove the stream owner" + ); + + let registry = client + .stream_handles + .lock() + .unwrap_or_else(|error| error.into_inner()); + assert!( + !registry.live.contains_key(&handle_id), + "terminalization must remove the live stream slot" + ); + assert_eq!( + registry + .recent + .iter() + .filter(|entry| { + entry.id == handle_id && entry.reason == StreamTerminal::CleanEof + }) + .count(), + 1, + "terminalization must retain exactly one one-shot clean-EOF record" + ); + drop(registry); + + assert_eq!( + client + .next_line(&handle_id, budget) + .await + .expect("consume one-shot clean EOF"), + None + ); + } + #[test] fn extreme_outbound_stream_max_seconds_does_not_panic() { // u64::MAX must remain a finite cap rather than panicking or silently diff --git a/tests/typechecker_try_finally_join_test.rs b/tests/typechecker_try_finally_join_test.rs index 88c16add..7602f211 100644 --- a/tests/typechecker_try_finally_join_test.rs +++ b/tests/typechecker_try_finally_join_test.rs @@ -56,6 +56,24 @@ fn subtract_one(name: &str, line: usize) -> Statement { } } +fn store_text(name: &str, value: &str, line: usize) -> Statement { + Statement::VariableDeclaration { + name: name.to_string(), + value: text_literal(value), + is_constant: false, + line, + column: 1, + } +} + +fn display_variable(name: &str, line: usize) -> Statement { + Statement::DisplayStatement { + value: Expression::Variable(name.to_string(), line, 1), + line, + column: 1, + } +} + #[test] fn handler_response_stream_state_is_joined_before_finally() { let program = Program { @@ -142,3 +160,83 @@ fn handler_error_aliases_remain_clause_local_before_finally() { result.err() ); } + +#[test] +fn handler_created_binding_is_semantically_visible_in_finally() { + let program = Program { + statements: vec![Statement::TryStatement { + body: vec![display_text("success", 2)], + when_clauses: vec![WhenClause { + error_type: ErrorType::FileNotFound, + error_name: "caught".to_string(), + body: vec![store_text("cleanup_message", "handled", 3)], + }], + otherwise_block: None, + finally_block: Some(vec![display_variable("cleanup_message", 5)]), + line: 1, + column: 1, + }], + }; + + assert!( + TypeChecker::new().check_types(&program).is_ok(), + "the analyzer and checker must preserve an ordinary handler binding in the shared \ + runtime try scope until finally; errors: {:?}", + TypeChecker::new().check_types(&program).err() + ); +} + +#[test] +fn otherwise_created_binding_is_semantically_visible_in_finally() { + let program = Program { + statements: vec![Statement::TryStatement { + body: vec![display_text("success", 2)], + when_clauses: vec![], + otherwise_block: Some(vec![store_text("cleanup_message", "otherwise", 3)]), + finally_block: Some(vec![display_variable("cleanup_message", 5)]), + line: 1, + column: 1, + }], + }; + + assert!( + TypeChecker::new().check_types(&program).is_ok(), + "the analyzer and checker must preserve an ordinary otherwise binding in the shared \ + runtime try scope until finally; errors: {:?}", + TypeChecker::new().check_types(&program).err() + ); +} + +#[test] +fn full_pipeline_error_alias_is_clause_local() { + let program = Program { + statements: vec![ + Statement::VariableDeclaration { + name: "caught".to_string(), + value: Expression::Literal(Literal::Integer(10), 1, 1), + is_constant: false, + line: 1, + column: 1, + }, + Statement::TryStatement { + body: vec![display_text("success", 2)], + when_clauses: vec![WhenClause { + error_type: ErrorType::General, + error_name: "caught".to_string(), + body: vec![display_variable("caught", 3)], + }], + otherwise_block: None, + finally_block: Some(vec![subtract_one("caught", 5)]), + line: 2, + column: 1, + }, + ], + }; + + assert!( + TypeChecker::new().check_types(&program).is_ok(), + "the implicit Text error alias must shadow only inside its clause, and finally must \ + resolve the outer Number; errors: {:?}", + TypeChecker::new().check_types(&program).err() + ); +} From de34e32e513d7d73634b0d7308c681930953a4db Mon Sep 17 00:00:00 2001 From: Brad Byrd Date: Sat, 25 Jul 2026 03:00:57 -0500 Subject: [PATCH 129/132] fix: linearize eof and analyzer try state --- src/analyzer/mod.rs | 77 +++++++++------ src/interpreter/mod.rs | 106 +++++++++++++-------- tests/typechecker_try_finally_join_test.rs | 21 +++- 3 files changed, 131 insertions(+), 73 deletions(-) diff --git a/src/analyzer/mod.rs b/src/analyzer/mod.rs index 69694535..c2519f4b 100644 --- a/src/analyzer/mod.rs +++ b/src/analyzer/mod.rs @@ -1361,15 +1361,19 @@ impl Analyzer { let flow_handler_entry = self.flow_entry(); let mut flow_paths: Vec = vec![flow_try]; - let try_scope = std::mem::take(&mut self.current_scope); - if let Some(parent) = try_scope.parent { - self.current_scope = *parent; - } + // Runtime keeps this try child environment alive through the + // selected handler/otherwise clause and finally. Snapshot the + // post-body structure so every statically possible clause is + // analyzed independently, then union its ordinary bindings + // back into the shared try scope for finally. + let clause_entry_scope = self.current_scope.clone(); + let mut joined_scope_symbols = clause_entry_scope.symbols.clone(); // Analyze each when clause for when_clause in when_clauses { - let outer_scope = std::mem::take(&mut self.current_scope); - self.current_scope = Scope::with_parent(outer_scope); + self.current_scope = clause_entry_scope.clone(); + self.restore_flow(&flow_handler_entry); + self.push_scope(); let error_symbol = Symbol { name: when_clause.error_name.clone(), @@ -1379,9 +1383,7 @@ impl Analyzer { column: 0, }; - if let Err(error) = self.current_scope.define(error_symbol) { - self.errors.push(error); - } + self.define_or_replace_symbol(error_symbol); // `error_message` is always available in error-handling // clauses as an alias for the caught error's message. @@ -1393,53 +1395,66 @@ impl Analyzer { line: 0, column: 0, }; - let _ = self.current_scope.define(error_message_symbol); + self.define_or_replace_symbol(error_message_symbol); } - self.restore_flow(&flow_handler_entry); for stmt in &when_clause.body { self.analyze_statement(stmt); } - flow_paths.push(self.take_flow_branch(&flow_handler_entry)); + let mut excluded_aliases = vec![when_clause.error_name.clone()]; + if when_clause.error_name != "error_message" { + excluded_aliases.push("error_message".to_string()); + } + self.pop_scope_promoting_except(&excluded_aliases); - let when_scope = std::mem::take(&mut self.current_scope); - if let Some(parent) = when_scope.parent { - self.current_scope = *parent; + flow_paths.push(self.take_flow_branch(&flow_handler_entry)); + for (name, symbol) in &self.current_scope.symbols { + joined_scope_symbols + .entry(name.clone()) + .or_insert_with(|| symbol.clone()); } } if let Some(otherwise_stmts) = otherwise_block { - let outer_scope = std::mem::take(&mut self.current_scope); - self.current_scope = Scope::with_parent(outer_scope); - + self.current_scope = clause_entry_scope.clone(); self.restore_flow(&flow_handler_entry); for stmt in otherwise_stmts { self.analyze_statement(stmt); } flow_paths.push(self.take_flow_branch(&flow_handler_entry)); - - let otherwise_scope = std::mem::take(&mut self.current_scope); - if let Some(parent) = otherwise_scope.parent { - self.current_scope = *parent; + for (name, symbol) in &self.current_scope.symbols { + joined_scope_symbols + .entry(name.clone()) + .or_insert_with(|| symbol.clone()); } + } else if !when_clauses.iter().any(|when_clause| { + matches!( + &when_clause.error_type, + crate::parser::ast::ErrorType::General + ) + }) { + // Without a catch-all or otherwise block, a non-matching + // error reaches finally directly from the handler entry. + flow_paths.push(flow_handler_entry.clone()); + } + + self.current_scope = clause_entry_scope; + for symbol in joined_scope_symbols.into_values() { + self.define_or_replace_symbol(symbol); } - // After the construct, any of the recorded paths may have run. + // Finally can be reached from success, any selected error + // clause, or an unmatched error. Join those flow endpoints + // before checking it in the shared runtime try scope. self.join_flow_branches(&flow_paths); if let Some(finally_stmts) = finally_block { - let outer_scope = std::mem::take(&mut self.current_scope); - self.current_scope = Scope::with_parent(outer_scope); - for stmt in finally_stmts { self.analyze_statement(stmt); } - - let finally_scope = std::mem::take(&mut self.current_scope); - if let Some(parent) = finally_scope.parent { - self.current_scope = *parent; - } } + + self.pop_scope(); } Statement::ReadFileStatement { variable_name, .. } => { let symbol = Symbol { diff --git a/src/interpreter/mod.rs b/src/interpreter/mod.rs index 25ab6c64..d1a51613 100644 --- a/src/interpreter/mod.rs +++ b/src/interpreter/mod.rs @@ -2473,6 +2473,37 @@ impl IoClient { handle: HttpStreamHandle, cancel: &StreamCancel, ) -> Result<(), HttpClientError> { + // Observing upstream EOF is the linearization point for clean + // completion. Finalize that already-latched result before consulting + // the wall clock or generic terminal rejection below: a reaper/close + // may have removed the slot after EOF won, but it must not erase the + // one follow-up `nothing` read. + if handle.done { + let terminal = cancel.terminate(StreamTerminal::CleanEof); + if terminal == StreamTerminal::CleanEof { + drop(handle); + let now = Instant::now(); + let mut registry = self + .stream_handles + .lock() + .unwrap_or_else(|e| e.into_inner()); + registry.prune_recent(now); + if let Some(mut slot) = registry.live.remove(handle_id) { + slot.cancel.terminate(StreamTerminal::CleanEof); + if let Some(abort) = slot.reaper_abort.take() { + abort.abort(); + } + drop(slot.handle.take()); + remove_stream_owner(&mut slot, handle_id); + } + // `remember_recent` replaces an existing record for this ID, + // so terminalization leaves exactly one bounded one-shot + // result even when the racing reaper already recorded EOF. + registry.remember_recent(handle_id.to_string(), StreamTerminal::CleanEof, now); + return Ok(()); + } + } + if let Some(terminal) = cancel.terminal() { drop(handle); // Ensure the slot is gone (reaper/close may already have removed it). @@ -2514,20 +2545,6 @@ impl IoClient { } return Err(self.stream_terminal_error(terminal)); } - if handle.done { - drop(handle); - if let Some(mut slot) = registry.live.remove(handle_id) { - let terminal = slot.cancel.terminate(StreamTerminal::CleanEof); - if let Some(abort) = slot.reaper_abort.take() { - abort.abort(); - } - drop(slot.handle.take()); - remove_stream_owner(&mut slot, handle_id); - registry.remember_recent(handle_id.to_string(), terminal, now); - } - return Ok(()); - } - let slot = registry .live .get_mut(handle_id) @@ -2564,7 +2581,12 @@ impl IoClient { use futures_util::StreamExt; if handle.done { - return Ok(false); + let terminal = cancel.terminate(StreamTerminal::CleanEof); + return if terminal == StreamTerminal::CleanEof { + Ok(false) + } else { + Err(self.stream_terminal_error(terminal)) + }; } let mut terminal_rx = cancel.subscribe(); if let Some(terminal) = *terminal_rx.borrow() { @@ -2624,8 +2646,13 @@ impl IoClient { "Failed to read response chunk: {e}" ))), None => { - handle.done = true; - Ok(false) + let terminal = cancel.terminate(StreamTerminal::CleanEof); + if terminal == StreamTerminal::CleanEof { + handle.done = true; + Ok(false) + } else { + Err(self.stream_terminal_error(terminal)) + } } } } @@ -2698,7 +2725,9 @@ impl IoClient { }; loop { - if let Err(e) = self.check_stream_deadline(&handle) { + let clean_eof_latched = + handle.done && cancel.terminal() == Some(StreamTerminal::CleanEof); + if !clean_eof_latched && let Err(e) = self.check_stream_deadline(&handle) { let _ = self.finish_stream_slot(handle_id).await; return Err(e); } @@ -16155,26 +16184,27 @@ mod outbound_stream_deadline_tests { "terminalization must remove the stream owner" ); - let registry = client - .stream_handles - .lock() - .unwrap_or_else(|error| error.into_inner()); - assert!( - !registry.live.contains_key(&handle_id), - "terminalization must remove the live stream slot" - ); - assert_eq!( - registry - .recent - .iter() - .filter(|entry| { - entry.id == handle_id && entry.reason == StreamTerminal::CleanEof - }) - .count(), - 1, - "terminalization must retain exactly one one-shot clean-EOF record" - ); - drop(registry); + { + let registry = client + .stream_handles + .lock() + .unwrap_or_else(|error| error.into_inner()); + assert!( + !registry.live.contains_key(&handle_id), + "terminalization must remove the live stream slot" + ); + assert_eq!( + registry + .recent + .iter() + .filter(|entry| { + entry.id == handle_id && entry.reason == StreamTerminal::CleanEof + }) + .count(), + 1, + "terminalization must retain exactly one one-shot clean-EOF record" + ); + } assert_eq!( client diff --git a/tests/typechecker_try_finally_join_test.rs b/tests/typechecker_try_finally_join_test.rs index 7602f211..fee6fefe 100644 --- a/tests/typechecker_try_finally_join_test.rs +++ b/tests/typechecker_try_finally_join_test.rs @@ -218,15 +218,28 @@ fn full_pipeline_error_alias_is_clause_local() { line: 1, column: 1, }, + Statement::VariableDeclaration { + name: "error_message".to_string(), + value: Expression::Literal(Literal::Integer(20), 1, 1), + is_constant: false, + line: 1, + column: 1, + }, Statement::TryStatement { body: vec![display_text("success", 2)], when_clauses: vec![WhenClause { error_type: ErrorType::General, error_name: "caught".to_string(), - body: vec![display_variable("caught", 3)], + body: vec![ + display_variable("caught", 3), + display_variable("error_message", 3), + ], }], otherwise_block: None, - finally_block: Some(vec![subtract_one("caught", 5)]), + finally_block: Some(vec![ + subtract_one("caught", 5), + subtract_one("error_message", 6), + ]), line: 2, column: 1, }, @@ -235,8 +248,8 @@ fn full_pipeline_error_alias_is_clause_local() { assert!( TypeChecker::new().check_types(&program).is_ok(), - "the implicit Text error alias must shadow only inside its clause, and finally must \ - resolve the outer Number; errors: {:?}", + "the implicit Text error aliases must shadow only inside their clause, and finally \ + must resolve the outer Numbers; errors: {:?}", TypeChecker::new().check_types(&program).err() ); } From 81b25745e757671538a833ddf7bc837e19ad83c7 Mon Sep 17 00:00:00 2001 From: Brad Byrd Date: Sat, 25 Jul 2026 03:15:08 -0500 Subject: [PATCH 130/132] test: cover clean eof missing-slot races --- src/interpreter/mod.rs | 185 +++++++++++++++++++++++++++++++++++++++++ 1 file changed, 185 insertions(+) diff --git a/src/interpreter/mod.rs b/src/interpreter/mod.rs index d1a51613..cdc24c68 100644 --- a/src/interpreter/mod.rs +++ b/src/interpreter/mod.rs @@ -15774,6 +15774,115 @@ mod outbound_stream_deadline_tests { .expect("finished streams left hard-lifetime reaper tasks sleeping"); } + fn install_owned_empty_stream(client: &IoClient, handle_id: &str) -> StreamOwner { + let owner: StreamOwner = + Arc::new(std::sync::Mutex::new(HashSet::from( + [handle_id.to_string()], + ))); + client + .stream_handles + .lock() + .unwrap_or_else(|error| error.into_inner()) + .live + .insert( + handle_id.to_string(), + StreamSlot { + handle: Some(HttpStreamHandle { + stream: Box::pin(futures_util::stream::empty::>>()), + buffer: Vec::new(), + done: false, + bytes_read: 0, + total_deadline: None, + }), + deadline: None, + cancel: StreamCancel::new(), + reaper_abort: None, + owner: Some(Arc::clone(&owner)), + }, + ); + owner + } + + async fn take_and_observe_clean_eof( + client: &IoClient, + handle_id: &str, + budget: &Arc, + ) -> TakenStream { + let Some(mut taken) = client + .take_stream(handle_id) + .expect("take synthetic empty stream") + else { + panic!("live synthetic stream unexpectedly resolved as clean EOF"); + }; + assert!( + !client + .stream_pull(&mut taken.handle, budget, &taken.cancel) + .await + .expect("observe upstream clean EOF"), + "an empty upstream must return clean EOF" + ); + assert!(taken.handle.done, "observing EOF must mark the body done"); + assert_eq!( + taken.cancel.terminal(), + Some(StreamTerminal::CleanEof), + "observing upstream None must latch CleanEof before slot removal" + ); + taken + } + + async fn assert_one_shot_clean_eof_after_put( + client: &IoClient, + handle_id: &str, + owner: &StreamOwner, + budget: Arc, + ) { + { + let registry = client + .stream_handles + .lock() + .unwrap_or_else(|error| error.into_inner()); + assert!( + !registry.live.contains_key(handle_id), + "put_stream must not recreate a heavy live slot after clean EOF" + ); + assert_eq!( + registry + .recent + .iter() + .filter(|entry| { + entry.id == handle_id && entry.reason == StreamTerminal::CleanEof + }) + .count(), + 1, + "put_stream must leave exactly one CleanEof tombstone" + ); + } + assert!( + owner + .lock() + .unwrap_or_else(|error| error.into_inner()) + .is_empty(), + "slot removal before put_stream must clear handler ownership" + ); + assert_eq!( + client + .next_line(handle_id, Arc::clone(&budget)) + .await + .expect("consume restored CleanEof tombstone"), + None, + "the first read after terminalization must observe clean EOF" + ); + let later = client + .next_line(handle_id, budget) + .await + .expect_err("the CleanEof tombstone must be one-shot"); + assert!( + matches!(&later, HttpClientError::Request(message) + if message.contains("Unknown or already-closed")), + "the read after consuming CleanEof must report a closed/unknown handle, got {later:?}" + ); + } + struct DelayedHeadUpstream { port: u16, request_received: oneshot::Receiver<()>, @@ -16215,6 +16324,82 @@ mod outbound_stream_deadline_tests { ); } + #[tokio::test] + async fn put_stream_restores_clean_eof_after_close_removed_the_live_slot() { + let config = Arc::new(WflConfig { + outbound_stream_max_seconds: 0, + timeout_seconds: 10, + ..WflConfig::default() + }); + let client = IoClient::new(Arc::clone(&config)); + let budget = Arc::new(ExecutionBudget::from_config(&config)); + let handle_id = "clean-eof-after-close"; + let owner = install_owned_empty_stream(&client, handle_id); + let TakenStream { handle, cancel } = + take_and_observe_clean_eof(&client, handle_id, &budget).await; + + assert!( + client.finish_stream_slot_sync(handle_id, StreamTerminal::Closed), + "close must remove the live slot while the EOF-observing read owns its body" + ); + assert_eq!( + cancel.terminal(), + Some(StreamTerminal::CleanEof), + "the later close claim must not overwrite the observed CleanEof" + ); + + client + .put_stream(handle_id, handle, &cancel) + .expect("put_stream must restore CleanEof after close removed the slot"); + assert_one_shot_clean_eof_after_put(&client, handle_id, &owner, budget).await; + } + + #[tokio::test] + async fn put_stream_deduplicates_clean_eof_after_reaper_removed_the_live_slot() { + let config = Arc::new(WflConfig { + outbound_stream_max_seconds: 0, + timeout_seconds: 10, + ..WflConfig::default() + }); + let client = IoClient::new(Arc::clone(&config)); + let budget = Arc::new(ExecutionBudget::from_config(&config)); + let handle_id = "clean-eof-after-reaper"; + let owner = install_owned_empty_stream(&client, handle_id); + let TakenStream { handle, cancel } = + take_and_observe_clean_eof(&client, handle_id, &budget).await; + + // Mirror the production reaper's critical section without a wall-clock + // sleep: remove the slot, attempt Timeout, clear ownership, and retain + // the first-wins terminal outcome before the active read calls put. + { + let now = Instant::now(); + let mut registry = client + .stream_handles + .lock() + .unwrap_or_else(|error| error.into_inner()); + registry.prune_recent(now); + let mut slot = registry + .live + .remove(handle_id) + .expect("reaper-style terminalization must claim the live slot"); + let terminal = slot.cancel.terminate(StreamTerminal::Timeout); + assert_eq!( + terminal, + StreamTerminal::CleanEof, + "a reaper Timeout after observed EOF must retain CleanEof" + ); + slot.reaper_abort = None; + drop(slot.handle.take()); + remove_stream_owner(&mut slot, handle_id); + registry.remember_recent(handle_id.to_string(), terminal, now); + } + + client + .put_stream(handle_id, handle, &cancel) + .expect("put_stream must preserve CleanEof after the reaper removed the slot"); + assert_one_shot_clean_eof_after_put(&client, handle_id, &owner, budget).await; + } + #[test] fn extreme_outbound_stream_max_seconds_does_not_panic() { // u64::MAX must remain a finite cap rather than panicking or silently From a543650fdfa202beef71da2248d2e5b025e2f661 Mon Sep 17 00:00:00 2001 From: Brad Byrd Date: Sat, 25 Jul 2026 03:52:24 -0500 Subject: [PATCH 131/132] docs: record stream state review fixes --- ...5-stream-terminal-typeflow-review-fixes.md | 204 ++++++++ Docs/development/response-streaming-design.md | 21 +- .../2026-07-24-pr-641-red-chronology.md | 69 ++- ...26-07-25-stream-terminal-typeflow-fixes.md | 479 ++++++++++++++++++ 4 files changed, 744 insertions(+), 29 deletions(-) create mode 100644 Dev diary/2026-07-25-stream-terminal-typeflow-review-fixes.md create mode 100644 Docs/superpowers/plans/2026-07-25-stream-terminal-typeflow-fixes.md diff --git a/Dev diary/2026-07-25-stream-terminal-typeflow-review-fixes.md b/Dev diary/2026-07-25-stream-terminal-typeflow-review-fixes.md new file mode 100644 index 00000000..16483404 --- /dev/null +++ b/Dev diary/2026-07-25-stream-terminal-typeflow-review-fixes.md @@ -0,0 +1,204 @@ +# Dev Diary — 2026-07-25: Stream terminal and type-flow review fixes + +This pass resolves four R3 findings found while reviewing the PR #641 / issue +#642 candidate: clean EOF retained a live outbound stream indefinitely, +`try` handler state did not reach `finally` in the checker, loop backedges were +not rechecked under later-iteration types, and deferred event/WebSocket handler +types leaked into the registration scope. + +The affected repair base is +`c9c748ce850b7d106ffef90e299e4b7221411517`. The final executable candidate is +`de34e32e513d7d73634b0d7308c681930953a4db`. The latest executable-test +descendant is `81b25745e757671538a833ddf7bc837e19ad83c7`; it adds deterministic +test-only coverage and does not change production code. The documentation-only +commit containing this entry does not change either executable identity. + +## Risk, contracts, and compatibility + +- **Risk class:** R3. +- **Triggers:** streaming lifecycle, cancellation and reaper races, bounded + resource retention, control-flow joins, loop iteration, deferred callbacks, + and backward-compatible typechecking. +- **Lifecycle contract:** observing upstream EOF immediately removes the live + slot, body, owner, and reaper. Only one lightweight `CleanEof` result remains, + sharing the existing recent-terminal bound of 64 records and 60 seconds. +- **Type-flow contract:** `try` success, handler, `otherwise`, and unmatched + endpoints are conservatively joined before `finally`; error aliases remain + clause-local. Loop bodies are checked at a conservative header fixed point. + Deferred event and WebSocket bodies are checked in child scopes. +- **Compatibility:** no syntax, public error variant, or runtime binding rule + was removed. Runtime-viable gradual joins stay permissive, while concrete + invalid later-iteration branches now produce diagnostics. +- **External state:** none. There is no deployment, schema, data migration, + secret, or external-service mutation. + +## Acceptance criteria and exact regressions + +| Acceptance criterion | Exact regression tests | +|---|---| +| A final unterminated line releases all heavy stream state immediately, retains one bounded EOF result, and remains first-wins against later timeout/close cleanup | `final_unterminated_line_survives_deadline_after_clean_eof`; `unconsumed_clean_eof_records_are_bounded`; `observed_clean_eof_wins_over_a_later_deadline_claim`; `put_stream_restores_clean_eof_after_close_removed_the_live_slot`; `put_stream_deduplicates_clean_eof_after_reaper_removed_the_live_slot` | +| Handler and `otherwise` endpoint types reach `finally`, while temporary error aliases do not | `handler_response_stream_state_is_joined_before_finally`; `handler_error_aliases_remain_clause_local_before_finally`; `handler_created_binding_is_semantically_visible_in_finally`; `otherwise_created_binding_is_semantically_visible_in_finally`; `full_pipeline_error_alias_is_clause_local` | +| Later loop iterations are checked after a body changes a target from File/Text-compatible state to `ResponseStream` | `while_loop_rechecks_stream_lead_after_tail_response_stream_rebind`; `repeat_while_loop_rechecks_stream_lead_after_tail_response_stream_rebind` | +| Merely registering a deferred handler cannot overwrite the enclosing checker binding | `event_handler_body_types_do_not_leak_after_registration`; `websocket_handler_body_types_do_not_leak_after_registration` | + +The regressions live in `src/interpreter/mod.rs`'s unit-test module, +`tests/typechecker_response_stream_join_test.rs`, +`tests/typechecker_response_stream_scope_test.rs`, and +`tests/typechecker_try_finally_join_test.rs`. + +## Auditable Red → Green ledger + +Each Red commit below contains tests only and is an ancestor of its Green +implementation. + +| Repair | Red parent / affected base | Test-only Red | Green implementation | Intended Red observation | +|---|---|---|---|---| +| Immediate bounded clean-EOF terminalization | `c9c748ce850b7d106ffef90e299e4b7221411517` | `96d53052388f75bd809c2af42f12445944e8fc69` | `b32ff55fa76fd03b07e2ade7159d3719f2ac0642` | After the final line, the live-slot assertion observed one retained slot instead of zero. | +| Loop-header fixed points, joined `try` endpoints, and deferred-handler isolation | `96d53052388f75bd809c2af42f12445944e8fc69` | `68569b31b9fd969cb5adc3b8c0832ec604bb98e2` | `527b8fb184245e7df35fe5229b23e2a969c74520` | Later-iteration invalid branches were missed, valid `finally` cleanup was rejected, and deferred handler registration changed outer types. | +| First-wins EOF and analyzer/runtime `try`-scope parity | `527b8fb184245e7df35fe5229b23e2a969c74520` | `03966f06e78aec7c3bcdbd40feabc2bdff37a16d` | `de34e32e513d7d73634b0d7308c681930953a4db` | A later timeout displaced observed EOF; handler/`otherwise` bindings were absent in analyzer `finally`; aliases collided in the full pipeline. | + +Commit `81b25745e757671538a833ddf7bc837e19ad83c7` broadens the Green +evidence with deterministic close/reaper missing-slot tests. Those tests use +the real close helper and the reaper's critical-section behavior, then assert +zero live slots and owners, exactly one `CleanEof`, one `nothing` result, and a +subsequent closed/unknown-handle error. + +## Implementation + +`StreamTerminal::CleanEof` now uses the same bounded, expiring, one-shot recent +terminal queue as timeout records. Upstream EOF is the first-wins +linearization point. Returning the final buffered line no longer parks the +completed handle or clears its deadline; `put_stream` drops heavy state, +removes ownership, aborts the reaper, and deduplicates/restores the one +lightweight EOF record even if close or reaper cleanup already removed the +slot. + +The analyzer and typechecker now model the runtime's shared `try` child +environment. Ordinary endpoint bindings are promoted and joined before +`finally`, while the named error and `error_message` aliases are discarded +with their clause scope. `while` and `repeat while` widen entry and backedge +snapshots to a stable conservative header before their diagnostic pass. +Event and WebSocket callback bodies are checked under isolated child scopes, +matching runtime dispatch. + +## Focused and boundary verification + +The following completed without product-test retry, quarantine, assertion +weakening, or timing-only substitution: + +```text +cargo test --lib put_stream_ --jobs 1 -- --nocapture --test-threads=1 +cargo test --lib observed_clean_eof_wins_over_a_later_deadline_claim --jobs 1 -- --nocapture --test-threads=1 +cargo test --lib clean_eof --jobs 1 -- --nocapture --test-threads=1 +cargo test --lib interpreter::outbound_stream_deadline_tests --jobs 1 -- --nocapture --test-threads=1 +cargo test --test typechecker_try_finally_join_test --jobs 1 -- --nocapture +cargo test --test typechecker_response_stream_join_test --jobs 1 -- --nocapture +cargo test --test typechecker_response_stream_scope_test --jobs 1 -- --nocapture +cargo test --test nothing_reassign_widen_test --jobs 1 -- --nocapture +cargo test --test overload_alias_resolution_test --jobs 1 -- --nocapture +cargo test --test http_stream_test --jobs 1 -- --nocapture --test-threads=1 +cargo test --test stream_backpressure_test --jobs 1 -- --nocapture --test-threads=1 +cargo test --test open_file_local_type_test --jobs 1 -- --nocapture +``` + +The respective focused results were 2, 1, 3, 10, 5, 4, 5, 7, 1, 12, 2, +and 3 tests passed with zero failures. + +The official Windows integration runner completed all Rust integration targets +and then reported **110 WFL programs passed, 0 failed, 24 documented skips**. +The official web runner reported **2/2 passed**; its separate certificate-file +journey was explicitly skipped because OpenSSL is unavailable on this host. +The Rust integration suite's eight TLS server tests passed. Forced docs-example +validation reported **18 passed, 0 failed** across validation layers 1–5. +The validator also emitted existing manifest-schema-key warnings; they did not +represent failed examples. + +## Infrastructure interruption and test integrity + +The first `cargo test --all` invocation terminated during rustc compilation +while memory-mapping an rlib with Windows error 1455: the paging file was too +small. No test binary produced a product-test result. The complete unchanged +suite was then invoked with `cargo test --all --jobs 1`, altering only Cargo's +compiler parallelism to bound peak memory. This is an infrastructure rerun +under `testing.md` §8.2, not a product-test retry; no product failure was +retried. + +The initial script invocations inside the restricted process sandbox did not +run product tests: Windows execution policy blocked one integration-script +launch, and a later sandbox process launch could not access Cargo's artifact +database. The same repository scripts were then run once successfully through +PowerShell with `-ExecutionPolicy Bypass` outside that boundary. Existing +ignored Rust tests and the integration runner's documented WFL skips were not +introduced or changed by this repair. + +## Final local gate record + +The final local gate ran on 2026-07-25 against executable candidate +`de34e32e513d7d73634b0d7308c681930953a4db`, test-only descendant +`81b25745e757671538a833ddf7bc837e19ad83c7`, and the final documentation +working tree. The host reported Windows NT `10.0.26200.0`, rustc/cargo +`1.97.0`, and PowerShell `7.6.4`. The official `.ps1` runners executed under +Windows PowerShell `5.1.26100.8875`. + +| Gate | Result | +|---|---| +| `cargo fmt --all -- --check` | Passed. | +| `cargo clippy --all-targets --all-features --jobs 1 -- -D warnings` | Passed. | +| `cargo test --all --jobs 1` | Passed across the complete workspace, integration binaries, LSP, `wflpkg`, and doctests. Existing ignored tests remained reported. Pre-existing unused-code warnings were emitted only by `wfl-lsp` test fixtures; the strict Clippy gate passed. | +| `cargo build --release --jobs 1` | Passed. | +| `git diff --check` | Passed; Git emitted only the repository's Windows LF-to-CRLF working-copy notices for two Markdown files. | +| `$env:CARGO_BUILD_JOBS='1'; ... run_integration_tests.ps1 -TestOnly` | Passed all Rust integration targets and 110 WFL programs; 0 failed and 24 documented programs were skipped by the runner. | +| `... run_web_tests.ps1` | Passed 2/2 runnable journeys; the OpenSSL-dependent certificate-file journey was visibly skipped on this host. | +| `python scripts/validate_docs_examples.py --ci --force` | Passed 18/18 examples with 0 failures. | + +No changed-code warning, product-test failure, retry, quarantine, mute, or +weakened assertion was used to obtain this result. Exact-candidate CI still +must exercise the repository's required platform and service matrix, including +the TLS journey that the local web script could not generate without OpenSSL. + +## Coverage, review, and residual risk + +Coverage was not instrumented, so there is no numeric result. Root +`testing.md` records the absent automated coverage gate as a tracked +conformance gap; no percentage or threshold pass is claimed. This change adds +behavioral, real-boundary, negative, race, and higher-layer regression +coverage. + +An independent Codex review inspected `c9c748ce..de34e32e` for lifecycle and +concurrency correctness, type-flow soundness, compatibility, and test +integrity. Its three Important findings were deterministic close/reaper +missing-slot coverage, explicit 64-record/60-second clean-EOF documentation, +and current candidate chronology. Commit `81b25745` and the accompanying +documentation resolve them. Follow-up review reported no remaining Critical or +Important issue. This is review evidence, not maintainer or security-owner +approval. + +Recent `CleanEof` results intentionally expire after 60 seconds and share a +64-record cap with other terminal results. After expiry, eviction, or +consumption, another read reports an unknown/already-closed handle. Type joins +conservatively widen missing or path-disagreeing bindings to the project's +gradual `Any`/`Unknown` behavior. These are documented design choices. + +PR merge and release remain blocked until the exception record has the required +project/reliability and security-owner approvals, requester identity/date, +maximum-release and expiration acknowledgment, and a successful GitHub Actions +matrix on the final integrated PR head. That head must include the +`81b25745` evidence tests and this documentation; a run on production identity +`de34e32e` alone is insufficient. The older Actions run `30142079511` is Green +evidence for an earlier head, not the final candidate. + +## Rollback + +No external recovery is needed. Revert the complete post-base range beginning +after `c9c748ce850b7d106ffef90e299e4b7221411517`, including its test-only +commits, so intentionally failing Red tests are not left active. Preserve the +original Red and Green commits in repository history as evidence, then run the +complete gate on the rollback candidate. Prefer a forward repair if later work +depends on these compatibility or lifecycle corrections. + +Related durable records are +`Dev diary/2026-07-24-issue-642-completion.md`, +`Docs/development/response-streaming-design.md`, +`Docs/development/testing-policy-exceptions/2026-07-24-pr-641-red-chronology.md`, +the implementation plan under `Docs/superpowers/plans/`, and root +`testing.md`. diff --git a/Docs/development/response-streaming-design.md b/Docs/development/response-streaming-design.md index 882949e0..d0c5bfa3 100644 --- a/Docs/development/response-streaming-design.md +++ b/Docs/development/response-streaming-design.md @@ -169,14 +169,19 @@ close out read return — including reads served from locally-buffered bytes — not only on a network read, so a buffered drain cannot outlive the stream's absolute cap. - Expiry removes the heavy live stream slot, aborts the body, and immediately - removes handler ownership. To preserve the next read's typed `Timeout` - without retaining an unbounded tombstone table, the registry keeps at most 64 - lightweight terminal records for at most 60 seconds; reading a record consumes - it. Active readers share a first-wins terminal signal, and the post-`select!` - recheck makes expiry win over a simultaneously ready body chunk. -- Clean EOF is terminal independently of the old absolute deadline. A final - unterminated line is returned once, the following read returns `nothing`, and - only a later read reports the documented closed-handle error. + removes handler ownership. To preserve one follow-up typed terminal result + without an unbounded tombstone table, the registry keeps at most 64 recent + lightweight terminal records (clean EOF and timeout records combined), each + for at most 60 seconds; reading a record consumes it. Active readers share a + first-wins terminal signal, and the post-`select!` recheck makes expiry win + over a simultaneously ready body chunk. +- Clean EOF terminalizes the stream as soon as the runtime observes upstream + EOF: it removes the live slot and handler ownership, aborts the reaper, and + retains only a one-shot `CleanEof` record in that bounded recent-terminal + queue. A final unterminated line is returned once. The next read returns + `nothing` only if its `CleanEof` record is still retained; if the record has + expired after 60 seconds, was evicted by the 64-record bound, or was already + consumed, the read reports `Unknown or already-closed stream handle`. - Outbound close-on-exit (shipped): outbound `httpstream*` handles are also handler-owned — tracked in `RunState.open_http_streams` (swapped per poll) and dropped from `IoClient.stream_handles` when the handler ends on any path, diff --git a/Docs/development/testing-policy-exceptions/2026-07-24-pr-641-red-chronology.md b/Docs/development/testing-policy-exceptions/2026-07-24-pr-641-red-chronology.md index 95f08036..e4ab6e9a 100644 --- a/Docs/development/testing-policy-exceptions/2026-07-24-pr-641-red-chronology.md +++ b/Docs/development/testing-policy-exceptions/2026-07-24-pr-641-red-chronology.md @@ -21,7 +21,8 @@ | Maximum affected releases | **One** WFL release: the first release containing the approved candidate, and no later release | | Affected base | `b25aed57ea50697c596796446d1f47466668773d` | | Commits containing the earlier mixed Green work | `5e01e446ab9250d72a0f255bc81a27a79c5b5d63`, `fce5d86fe923666885e40ec484d902cfd18c4c85`, and `8e8be0fcde944d0d7b357b94d5951497af5ff0b7` | -| Exact executable candidate for this draft | `c73260ff61a32694c5ecfe72ab8749810033de0d` | +| Exact executable candidate for this draft | `de34e32e513d7d73634b0d7308c681930953a4db` | +| Latest executable-test evidence descendant | `81b25745e757671538a833ddf7bc837e19ad83c7` (test-only; no production code) | | Requested project/reliability owner approval | Brad, Maintainer, Logbie LLC — **PENDING** | | Requested security-owner approval | Brad, Maintainer, Logbie LLC — **PENDING**, required for the archive-path item | @@ -84,6 +85,19 @@ implementation commits: - response-expression disconnect cancellation, including request operands, early prechecks, and commit-time cleanup. +The final candidate adds these three genuine Red-to-Green chains after the +previous draft candidate: + +| Repair | Test-only Red | Green implementation | Evidence broadening | +|---|---|---|---| +| Immediate bounded clean-EOF terminalization | `96d53052388f75bd809c2af42f12445944e8fc69` | `b32ff55fa76fd03b07e2ade7159d3719f2ac0642` | `81b25745e757671538a833ddf7bc837e19ad83c7` deterministically covers close/reaper missing-slot races | +| Loop-header fixed points, joined `try` endpoints, and deferred handler type isolation | `68569b31b9fd969cb5adc3b8c0832ec604bb98e2` | `527b8fb184245e7df35fe5229b23e2a969c74520` | Focused integration suites retained in the Green ancestry | +| First-wins EOF observation and analyzer `try`-scope parity | `03966f06e78aec7c3bcdbd40feabc2bdff37a16d` | `de34e32e513d7d73634b0d7308c681930953a4db` | `81b25745e757671538a833ddf7bc837e19ad83c7` broadens terminal-race coverage without executable changes | + +These chains do not repair the older mixed-commit chronology rows in this +exception. They do establish ordinary policy-compliant chronology for every +behavior changed after `c73260ff61a32694c5ecfe72ab8749810033de0d`. + The final-unterminated-line defect also has retained pre-Green CI evidence in [Actions run 30106107011](https://github.com/WebFirstLanguage/wfl/actions/runs/30106107011). Neither that behavior nor any later genuine Red-to-Green repair depends on this @@ -119,31 +133,44 @@ ignore a current failure. - Later issue #642 repairs use retained test-only Red ancestors and Green commits; those repairs strengthen the candidate but do not retroactively supply the chronology missing from the earlier mixed commits. +- The latest executable candidate is + `de34e32e513d7d73634b0d7308c681930953a4db`; the test-only descendant + `81b25745e757671538a833ddf7bc837e19ad83c7` adds deterministic coverage for + the clean-EOF close/reaper missing-slot paths without changing production + behavior. Actions run 30142079511 is evidence for the reviewed Green head, not automatic evidence for the exact candidate in this draft. Before approval, the approval -record must link one complete, successful, unretried Actions run for the exact -final candidate (or its evidence-only descendant) and the final local gate -record. Until those fields are complete, current Green evidence is incomplete -for merge. +record must link one complete, successful, unretried Actions run for the final +integrated PR head: a documentation descendant of +`81b25745e757671538a833ddf7bc837e19ad83c7` that contains the deterministic +evidence tests and this completed record. A run on executable identity +`de34e32e513d7d73634b0d7308c681930953a4db` alone is insufficient because it +omits those later tests and documents. Until that run and the final local gate +record are complete, current Green evidence is incomplete for merge. ## Compensating verification and containment Approval is conditional on all of the following: -1. Run, once and without changing test selection: +1. Run, once and without changing product-test selection, the host-appropriate + complete local gate. The recorded Windows gate is: `cargo fmt --all -- --check`, `git diff --check`, - `cargo clippy --all-targets --all-features -- -D warnings`, - `cargo build --release`, `cargo test --all --verbose --jobs 2`, - `scripts/run_integration_tests.sh`, - `python3 scripts/validate_docs_examples.py --ci --force`, and - `scripts/run_web_tests.sh`. + `cargo clippy --all-targets --all-features --jobs 1 -- -D warnings`, + `cargo build --release --jobs 1`, `cargo test --all --jobs 1`, + `run_integration_tests.ps1 -TestOnly`, `run_web_tests.ps1`, and + `python scripts/validate_docs_examples.py --ci --force`. The single Cargo + job bounds compiler memory after an unbounded rustc invocation ended before + any test result with Windows pagefile error 1455; it does not alter test + selection. Exact Windows PowerShell invocation details are retained in the + Dev Diary. The required final Actions matrix separately runs the repository's + supported Linux and Windows commands. 2. Preserve the exact commands, exit conclusions, candidate SHA, and complete logs in the PR evidence record. -3. Require one complete GitHub Actions matrix on the exact candidate, covering - Linux and Windows integration, TestPrograms, documentation validation, web - tests, TLS, PostgreSQL, MariaDB, and fuzz-target compilation. Every required - job must pass. +3. Require one complete GitHub Actions matrix on the final integrated PR head + described above, covering Linux and Windows integration, TestPrograms, + documentation validation, web tests, TLS, PostgreSQL, MariaDB, and + fuzz-target compilation. Every required job must pass. 4. Obtain an independent R3 review of the implementation, regression assertions, real-boundary coverage, cleanup paths, and this exception's exact scope. @@ -228,15 +255,15 @@ or release remains blocked. | Approval field | Required entry | |---|---| -| Exact final executable candidate SHA | **PENDING** | -| Final evidence-only descendant SHA, if any | **PENDING / N/A** | -| Final local gate record | **PENDING** — commands, date, environment, and conclusions | -| Final GitHub Actions run | **PENDING** — URL and every required job conclusion | -| Independent R3 reviewer | **PENDING** — identity, date, and scope reviewed | +| Exact final executable candidate SHA | `de34e32e513d7d73634b0d7308c681930953a4db` | +| Final evidence-only descendant SHA, if any | `81b25745e757671538a833ddf7bc837e19ad83c7` (latest code/test descendant; the commit containing this documentation record is documentation-only) | +| Final local gate record | **PASSED 2026-07-25** — Windows NT `10.0.26200.0`, rustc/cargo `1.97.0`, PowerShell `7.6.4`; the official `.ps1` runners executed under Windows PowerShell `5.1.26100.8875`. `cargo fmt --all -- --check`, `cargo clippy --all-targets --all-features --jobs 1 -- -D warnings`, `cargo test --all --jobs 1`, `cargo build --release --jobs 1`, and `git diff --check` passed. The official Windows integration runner passed all Rust targets plus 110 WFL programs (0 failed, 24 documented skips); web passed 2/2 runnable journeys with its OpenSSL-dependent certificate journey visibly skipped; forced docs validation passed 18/18. The earlier unbounded `cargo test --all` stopped in rustc before a test result with Windows pagefile error 1455; the unchanged suite passed with one compiler job. Full command context is in `Dev diary/2026-07-25-stream-terminal-typeflow-review-fixes.md`. | +| Final GitHub Actions run | **PENDING** — URL, final integrated PR-head SHA (a documentation descendant of `81b25745`), and every required job conclusion | +| Independent R3 reviewer | Independent Codex review task `/root/final_independent_review`, `2026-07-25`; reviewed `c9c748ce..de34e32e` for lifecycle/concurrency correctness, type-flow soundness, compatibility, and test integrity. Its three Important evidence/documentation findings are addressed by `81b25745` and the documentation-only descendant containing this record; this is review evidence, not approval authority. | | Requester | **PENDING** — identity and date | | Project/reliability owner decision | **PENDING** — Brad must record `APPROVE` or `REJECT`, rationale, date, and signature | | Security-owner decision for archive-path scope | **PENDING** — Brad must record `APPROVE` or `REJECT`, rationale, date, and signature | -| No skip/retry/quarantine/muting/weakening/timing-only conversion attestation | **PENDING** | +| No skip/retry/quarantine/muting/weakening/timing-only conversion attestation | **RECORDED 2026-07-25** — no changed-behavior test was skipped, retried, quarantined, muted, weakened, or converted to timing-only proof. The error-1455 compiler interruption produced no product-test result and was rerun only with bounded compiler parallelism. Existing Rust ignores, 24 documented WFL program skips, and the host's OpenSSL-dependent web skip remained visible; exact-candidate CI coverage is still required. | | Maximum-release and expiration acknowledgment | **PENDING** | The requester must not be the sole approver. If Brad is also the requester, a diff --git a/Docs/superpowers/plans/2026-07-25-stream-terminal-typeflow-fixes.md b/Docs/superpowers/plans/2026-07-25-stream-terminal-typeflow-fixes.md new file mode 100644 index 00000000..6faabe22 --- /dev/null +++ b/Docs/superpowers/plans/2026-07-25-stream-terminal-typeflow-fixes.md @@ -0,0 +1,479 @@ +# Stream Terminal and Type-Flow Fixes Implementation Plan + +> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking. + +**Goal:** Eliminate completed outbound-stream retention and make typechecking match runtime state across `try`, loop backedges, and deferred handler bodies. + +**Architecture:** Clean EOF becomes a lightweight `StreamTerminal::CleanEof` entry in the existing bounded recent-terminal queue; the live slot, owner ID, body, cancel channel, and reaper are removed as soon as the final unterminated line is returned. The typechecker will use symbol-type snapshots as a small control-flow lattice: branch endpoints are joined before `finally`, loop headers are widened to a fixed point and checked once at that fixed point, and deferred event/WebSocket bodies are checked in isolated child scopes whose refinements are restored afterward. + +**Tech Stack:** Rust 2024, Tokio, reqwest streaming, WFL analyzer/typechecker, Rust unit and integration tests. + +## Global Constraints + +- Risk class is **R3** because this changes streaming lifecycle and language backward-compatibility behavior. +- Every behavioral fix requires a test-only Red commit that is an ancestor of its Green commit. +- Existing WFL programs remain compatible; gradual `Unknown`/`Any` joins must remain permissive while concrete invalid branches remain diagnostics. +- Error aliases introduced by `when` clauses remain clause-local. +- Clean EOF retention uses the existing hard limits: at most **64** lightweight records for at most **60 seconds**, and the record is consumed by one follow-up read. +- No required test may be retried, skipped, quarantined, muted, or weakened. +- Required final gates are `cargo fmt --all -- --check`, `cargo clippy --all-targets --all-features --jobs 1 -- -D warnings`, `cargo test --all --jobs 1`, a release build, Windows integration/web scripts, and forced docs example validation. Cargo jobs are bounded to one on this Windows host because the unbounded compiler process hit pagefile error 1455 before any test binary ran. + +--- + +### Task 1: Test-only Red for clean EOF terminalization + +**Files:** +- Modify: `src/interpreter/mod.rs` (unit-test module only) + +**Interfaces:** +- Consumes: `IoClient::open_http_stream`, `IoClient::claim_stream_owner`, `IoClient::next_line`, `StreamRegistry::{live,recent}`, and `MAX_RECENT_STREAM_TERMINALS`. +- Produces: a regression proving live state is gone before the one-shot EOF read. + +- [x] **Step 1: Strengthen the final-unterminated-line test** + +After the first `Some("abc")`, inspect the registry and owner before any follow-up read: + +```rust +let (live_slots, clean_eof_records) = { + let registry = interpreter + .io_client + .stream_handles + .lock() + .unwrap_or_else(|error| error.into_inner()); + ( + registry.live.len(), + registry + .recent + .iter() + .filter(|entry| entry.reason == StreamTerminal::CleanEof) + .count(), + ) +}; +assert_eq!(live_slots, 0); +assert_eq!(clean_eof_records, 1); +assert_eq!( + interpreter + .open_http_streams + .borrow() + .lock() + .unwrap_or_else(|error| error.into_inner()) + .len(), + 0 +); +``` + +Then retain the existing assertions that one delayed read returns `None` and a later read reports an already-closed handle. + +- [x] **Step 2: Add a bounded no-follow-up-read wave** + +Open and claim more than `MAX_RECENT_STREAM_TERMINALS` `/unterminated` streams under one `Interpreter`, read only each final line, and assert: + +```rust +assert_eq!(registry.live.len(), 0); +assert!(registry.recent.len() <= MAX_RECENT_STREAM_TERMINALS); +assert!(registry + .recent + .iter() + .all(|entry| entry.reason == StreamTerminal::CleanEof)); +assert!(interpreter + .open_http_streams + .borrow() + .lock() + .unwrap_or_else(|error| error.into_inner()) + .is_empty()); +``` + +- [x] **Step 3: Run the focused tests and verify Red** + +Run: + +```powershell +cargo test --lib interpreter::outbound_stream_deadline_tests::final_unterminated_line_survives_deadline_after_clean_eof -- --nocapture --test-threads=1 +cargo test --lib interpreter::outbound_stream_deadline_tests::unconsumed_clean_eof_records_are_bounded -- --nocapture --test-threads=1 +``` + +Expected and observed: the lifecycle assertions fail because the completed +handle remains in `registry.live`; Red is a behavioral failure, not a +compile-failure placeholder. + +- [x] **Step 4: Commit Red evidence** + +```powershell +git add src/interpreter/mod.rs +git commit -m "test: expose retained clean eof stream state" +``` + +### Task 2: Bounded one-shot `CleanEof` + +**Files:** +- Modify: `src/interpreter/mod.rs` + +**Interfaces:** +- Consumes: `StreamRegistry::remember_recent` and `take_recent`. +- Produces: `StreamTerminal::CleanEof`; `take_stream` returns `Ok(None)` for that one-shot result. + +- [x] **Step 1: Add the terminal reason and optional take result** + +```rust +enum StreamTerminal { + CleanEof, + Timeout, + Closed, +} + +fn take_stream( + &self, + handle_id: &str, +) -> Result, HttpClientError> +``` + +When `take_recent` yields `CleanEof`, return `Ok(None)`; typed failures continue through `stream_terminal_error`. + +- [x] **Step 2: Terminalize in `put_stream`** + +Make the upstream `None` observation the linearization point: +`stream_pull` first-wins latches `CleanEof` on the shared cancel state before +returning control to `next_line`/`next_chunk`. In `put_stream`, handle the +latched `handle.done` case before generic cancellation or wall-deadline +rejection: + +```rust +if handle.done { + let terminal = cancel.terminate(StreamTerminal::CleanEof); + if terminal == StreamTerminal::CleanEof { + drop(handle); + let now = Instant::now(); + let mut registry = self + .stream_handles + .lock() + .unwrap_or_else(|error| error.into_inner()); + registry.prune_recent(now); + if let Some(mut slot) = registry.live.remove(handle_id) { + slot.cancel.terminate(StreamTerminal::CleanEof); + if let Some(abort) = slot.reaper_abort.take() { + abort.abort(); + } + drop(slot.handle.take()); + remove_stream_owner(&mut slot, handle_id); + } + registry.remember_recent( + handle_id.to_string(), + StreamTerminal::CleanEof, + now, + ); + return Ok(()); + } +} +``` + +`remember_recent` replaces any record for the same ID, so this restores exactly +one one-shot result even when close or the reaper already removed the live +slot. Do not clear either deadline and do not park a completed handle. If +`Timeout` or `Closed` won before upstream EOF was observed, preserve that +earlier typed terminal instead. + +- [x] **Step 3: Consume clean EOF in both read APIs** + +```rust +let Some(TakenStream { mut handle, cancel }) = self.take_stream(handle_id)? else { + return Ok(None); +}; +``` + +Use the same form in `next_chunk` and `next_line`. Lifecycle-only call sites that cannot legitimately observe clean EOF map it to closed without retaining heavy state. + +- [x] **Step 4: Verify Green and adjacent lifecycle behavior** + +Run: + +```powershell +cargo test --lib interpreter::outbound_stream_deadline_tests -- --nocapture --test-threads=1 +cargo test --test http_stream_test -- --nocapture --test-threads=1 +``` + +Expected: all focused lifecycle and real-boundary streaming tests pass. + +- [x] **Step 5: Commit Green** + +```powershell +git add src/interpreter/mod.rs +git commit -m "fix: terminalize clean eof streams immediately" +``` + +### Task 3: Test-only Red for checker control-flow state + +**Files:** +- Modify: `tests/typechecker_response_stream_join_test.rs` +- Modify: `tests/typechecker_response_stream_scope_test.rs` +- Create: `tests/typechecker_try_finally_join_test.rs` + +**Interfaces:** +- Consumes: direct `Program`/`Statement` AST construction and the public `TypeChecker::check_types`. +- Produces: later-iteration, `finally`, alias-isolation, event-handler, and WebSocket-handler regressions. + +- [x] **Step 1: Add later-iteration loop tests** + +Construct `WhileLoop` and `RepeatWhileLoop` bodies in this order: + +```rust +Statement::StreamWriteStatement { + value: Expression::BinaryOperation { + left: Box::new(Expression::Literal(Literal::Integer(10), 2, 1)), + operator: Operator::Minus, + right: Box::new(text_literal("not a number")), + line: 2, + column: 1, + }, + target: Expression::Variable("out".to_string(), 2, 1), + is_line: true, + fallback_content: Some(Box::new(text_literal("valid file text"))), + line: 2, + column: 1, +}, +stream_binding(), +``` + +Precede the loop with an `OpenFileStatement` binding `out`. Each test must expect `Cannot perform Minus operation`: the first iteration takes the valid File fallback, while the backedge can make the next iteration take the invalid ResponseStream reading. + +- [x] **Step 2: Add `try` endpoint/finally tests** + +Build a `TryStatement` whose handler binds `out` as a response stream and whose `finally` flushes `out`, with an outer concrete File binding. Assert the checker accepts the joined gradual state rather than resolving only the outer File. + +Add a control where outer Number bindings reuse the handler error name and `error_message`; subtraction in `finally` must remain valid, proving both aliases stay clause-local. + +- [x] **Step 3: Add deferred-handler isolation tests** + +For both `EventHandler` and `WebSocketHandlerStatement`, start with outer `out: Number`, put `stream_binding()` in the registered body, and subtract one from outer `out` after registration. Assert typechecking succeeds. + +- [x] **Step 4: Run focused tests and verify Red** + +Run: + +```powershell +cargo test --test typechecker_response_stream_join_test -- --nocapture +cargo test --test typechecker_try_finally_join_test -- --nocapture +cargo test --test typechecker_response_stream_scope_test -- --nocapture +``` + +Expected: the new later-iteration tests miss the invalid stream branch, the `finally` test rejects a File flush, and the event/WebSocket tests leak `ResponseStream` into outer `out`. + +- [x] **Step 5: Commit Red evidence** + +```powershell +git add tests/typechecker_response_stream_join_test.rs tests/typechecker_response_stream_scope_test.rs tests/typechecker_try_finally_join_test.rs +git commit -m "test: expose checker backedge and handler state gaps" +``` + +### Task 4: Checker joins, fixed points, and deferred scopes + +**Files:** +- Modify: `src/analyzer/mod.rs` +- Modify: `src/typechecker/mod.rs` + +**Interfaces:** +- Consumes: `Analyzer::{push_scope,pop_scope,snapshot_symbol_types,restore_symbol_types}` and `TypeChecker::join_type_snapshots`. +- Produces: `Analyzer::pop_scope_promoting_except` and `TypeChecker::check_loop_body_fixed_point`. + +- [x] **Step 1: Add selective clause-scope promotion** + +```rust +pub fn pop_scope_promoting_except(&mut self, excluded: &[String]) { + if let Some(mut parent) = self.current_scope.parent.take() { + for (name, symbol) in std::mem::take(&mut self.current_scope.symbols) { + if !excluded.iter().any(|excluded_name| excluded_name == &name) { + parent.define_or_replace(symbol); + } + } + self.current_scope = *parent; + } +} +``` + +This models the runtime’s shared try child while dropping only the temporary error aliases. + +- [x] **Step 2: Join `try` endpoints before `finally`** + +Within the shared try checker scope: + +1. Snapshot entry. +2. Check the body and capture the success endpoint. +3. Form a conservative handler entry from entry plus body endpoint. +4. Restore that entry before every handler. +5. Check a handler in an alias child scope, promote all non-alias bindings, and capture its endpoint. +6. Restore handler entry before `otherwise` and capture its endpoint. +7. Join success, handler, otherwise, and possible unmatched-error endpoints. +8. Restore the join, then check `finally` once. + +- [x] **Step 3: Compute a widening loop-header fixed point** + +```rust +fn check_loop_body_fixed_point(&mut self, body: &[Statement]) { + let entry = self.analyzer.snapshot_symbol_types(); + let mut header = entry.clone(); + loop { + self.analyzer.restore_symbol_types(header.clone()); + let error_count = self.errors.len(); + for statement in body { + self.check_statement_types(statement); + } + if self.budget_error.is_some() { + return; + } + self.errors.truncate(error_count); + let backedge = self.analyzer.snapshot_symbol_types(); + let next = + Self::join_type_snapshots(&[entry.clone(), header.clone(), backedge]); + if next == header { + break; + } + header = next; + } + self.analyzer.restore_symbol_types(header.clone()); + for statement in body { + self.check_statement_types(statement); + } + self.analyzer.restore_symbol_types(header); +} +``` + +Use it inside the persistent child scope for `RepeatWhileLoop` and in the current scope for `WhileLoop`. Validate each condition once under the stable header. + +- [x] **Step 4: Isolate deferred callback bodies** + +For `EventHandler` and `WebSocketHandlerStatement`, push a checker child scope, snapshot types, check the body, restore the snapshot, and pop. The WebSocket server operand remains checked outside the child. + +- [x] **Step 5: Verify Green and affected checker suites** + +Run: + +```powershell +cargo test --test typechecker_response_stream_join_test -- --nocapture +cargo test --test typechecker_try_finally_join_test -- --nocapture +cargo test --test typechecker_response_stream_scope_test -- --nocapture +cargo test --test nothing_reassign_widen_test -- --nocapture +cargo test --test open_file_local_type_test -- --nocapture +``` + +Expected: all pass with no duplicated diagnostics. + +- [x] **Step 6: Commit Green** + +```powershell +git add src/analyzer/mod.rs src/typechecker/mod.rs +git commit -m "fix: stabilize checker control-flow state" +``` + +### Task 4.25: Linearize EOF and align analyzer `try` scopes + +The first Green pass exposed two adjacent gaps during independent review. +They were repaired with another genuine Red-to-Green pair: + +- [x] **Step 1: Commit deterministic Red regressions** + +Commit `03966f06e78aec7c3bcdbd40feabc2bdff37a16d` adds a deterministic +EOF-observation-versus-later-timeout regression, analyzer-created +handler/`otherwise` binding regressions, and full-pipeline alias-shadowing +coverage. + +- [x] **Step 2: Commit the Green implementation** + +Commit `de34e32e513d7d73634b0d7308c681930953a4db` makes terminal signals +first-wins at upstream EOF and gives analyzer clauses/finally the same shared +runtime child environment while retaining clause-local error aliases. + +### Task 4.5: Resolve independent-review evidence gaps + +**Files:** +- Modify: `src/interpreter/mod.rs` (unit-test module only) +- Modify: `Docs/development/response-streaming-design.md` +- Modify: `Docs/development/testing-policy-exceptions/2026-07-24-pr-641-red-chronology.md` + +- [x] **Step 1: Cover close/reaper missing-slot paths after observed EOF** + +Add deterministic tests that latch `CleanEof`, remove the live slot through +the real close helper and a reaper-equivalent critical section, then prove +`put_stream` leaves exactly one one-shot `CleanEof`, no owner, no live slot, +one `nothing` read, and then the closed-handle error. + +- [x] **Step 2: Document bounded clean-EOF retention** + +Document that the one-shot result shares the 64-record recent-terminal queue, +expires after 60 seconds, and can be evicted or consumed. + +- [x] **Step 3: Refresh the candidate chronology** + +Record the exact executable candidate and all three new Red-to-Green chains +without treating independent review as maintainer approval. + +- [x] **Step 4: Obtain follow-up review** + +Have the independent reviewer verify the new deterministic tests and both +documentation repairs, and resolve any remaining Critical or Important issue. + +### Task 5: Evidence, documentation, and full verification + +**Files:** +- Create: `Dev diary/2026-07-25-stream-terminal-typeflow-review-fixes.md` +- Modify: `Docs/superpowers/plans/2026-07-25-stream-terminal-typeflow-fixes.md` +- Modify: `Docs/development/response-streaming-design.md` +- Modify: `Docs/development/testing-policy-exceptions/2026-07-24-pr-641-red-chronology.md` + +**Interfaces:** +- Consumes: Red/Green commit IDs and exact command output. +- Produces: durable R3 acceptance-criteria mapping and residual-risk record. + +- [x] **Step 1: Record change evidence** + +The Dev Diary entry must include: + +```markdown +- Risk class: R3 +- Acceptance criteria -> exact test names +- Base, Red, and Green commit IDs for all three Red/Green pairs +- Focused, unit, integration, web, docs, format, and clippy commands +- Windows platform result and any explicitly non-applicable layers +- Rollback: revert the complete post-base repair range, including test-only + commits, so intentionally failing Red tests are not left active; preserve + the original Red/Green commits in history as evidence +- Residual risk: recent CleanEof records are intentionally capped at 64/60s +``` + +- [x] **Step 2: Run static and complete Rust gates** + +```powershell +cargo fmt --all -- --check +cargo clippy --all-targets --all-features --jobs 1 -- -D warnings +cargo test --all --jobs 1 +cargo build --release --jobs 1 +``` + +Expected: every command exits zero without warnings from changed code. + +- [x] **Step 3: Run real-boundary and documentation gates** + +```powershell +$env:CARGO_BUILD_JOBS='1' +& 'C:\Windows\System32\WindowsPowerShell\v1.0\powershell.exe' -NoProfile -ExecutionPolicy Bypass -File '.\scripts\run_integration_tests.ps1' -TestOnly +& 'C:\Windows\System32\WindowsPowerShell\v1.0\powershell.exe' -NoProfile -ExecutionPolicy Bypass -File '.\scripts\run_web_tests.ps1' +python scripts/validate_docs_examples.py --ci --force +``` + +Expected: all required programs, web journeys, and docs examples pass without retry. + +- [x] **Step 4: Obtain independent review** + +Review the complete diff from `c9c748ce` through the final code commit for specification compliance, concurrency/lifecycle correctness, type-lattice soundness, compatibility, and test integrity. Resolve every Critical or Important finding and rerun its covering tests. + +- [x] **Step 5: Commit evidence** + +```powershell +git add "Dev diary/2026-07-25-stream-terminal-typeflow-review-fixes.md" Docs/development/response-streaming-design.md Docs/development/testing-policy-exceptions/2026-07-24-pr-641-red-chronology.md Docs/superpowers/plans/2026-07-25-stream-terminal-typeflow-fixes.md +git commit -m "docs: record stream state review fixes" +``` + +## Self-Review + +- Spec coverage: all four review findings map to Tasks 1–4; R3 evidence and full gates map to Task 5. +- Placeholder scan: no TBD/TODO/later placeholders remain. +- Type consistency: `CleanEof`, `take_stream -> Result, _>`, `pop_scope_promoting_except`, and `check_loop_body_fixed_point` are named consistently in every task. +- Execution choice: this request already asks for the fixes in the current branch, so execute inline in this session without pausing between tasks. From 690be0af6eb2c2bf3ed411c47e8f3333c7ca9aaf Mon Sep 17 00:00:00 2001 From: Brad Byrd Date: Sat, 25 Jul 2026 05:45:25 -0500 Subject: [PATCH 132/132] fix: restore backward compatibility for classic syntax regressions Ensures existing programs using file writes, custom flush actions, and display folds keep their original behavior. This prevents the new web streaming syntax from incorrectly taking over older code patterns during parsing and JavaScript transpilation. --- ...-25-parser-streaming-compat-regressions.md | 99 +++++++++++++ .../2026-07-25-parser-compat-regressions.md | 106 ++++++++++++++ ...arser_streaming_compat_regression.test.wfl | 50 +++++++ src/parser/expr/primary.rs | 55 ++++++- src/parser/stmt/io.rs | 73 ++++++++-- src/parser/stmt/web.rs | 12 +- src/transpiler/javascript.rs | 26 +++- tests/flush_action_backcompat_test.rs | 40 ++++++ tests/transpiler_test.rs | 18 +++ tests/write_web_postfix_test.rs | 134 ++++++++++++++++++ 10 files changed, 591 insertions(+), 22 deletions(-) create mode 100644 Dev diary/2026-07-25-parser-streaming-compat-regressions.md create mode 100644 Docs/superpowers/plans/2026-07-25-parser-compat-regressions.md create mode 100644 TestPrograms/parser_streaming_compat_regression.test.wfl diff --git a/Dev diary/2026-07-25-parser-streaming-compat-regressions.md b/Dev diary/2026-07-25-parser-streaming-compat-regressions.md new file mode 100644 index 00000000..f87ac18b --- /dev/null +++ b/Dev diary/2026-07-25-parser-streaming-compat-regressions.md @@ -0,0 +1,99 @@ +# Dev Diary — 2026-07-25: Parser streaming compatibility regressions + +This change repairs four backward-compatibility regressions introduced by the +response-streaming grammar: classic `write line`/`write chunk` expressions with +continuations, exact zero-argument actions named `flush`, display folds after +property access, and JavaScript transpilation of ambiguous classic file writes. + +## Risk and compatibility contract + +- **Risk class:** R3. +- **Triggers:** backward-compatible parsing, streaming dispatch, file writes, + action dispatch, and silent output changes. +- **Compatibility contract:** existing classic programs keep their pre-streaming + interpretation while unambiguous response-stream operations remain available. +- **External state:** none beyond temporary files created and removed by tests. + +## Acceptance criteria and coverage + +| Acceptance criterion | Regression coverage | +|---|---| +| A variable literally named `line` or `chunk` remains a classic file-write expression when followed by an operator, `at`, property access, or indexing | `tests/write_web_postfix_test.rs`; `TestPrograms/parser_streaming_compat_regression.test.wfl` | +| Direct integer stream targets such as `write line 0` remain streaming syntax | `tests/write_web_postfix_test.rs` and existing streaming parser tests | +| `flush (expr)` and same-line `flush call ...` invoke an exact zero-argument action named `flush` when one exists | `tests/flush_action_backcompat_test.rs`; `TestPrograms/parser_streaming_compat_regression.test.wfl` | +| Spaced values after property access remain display folds, while adjacent postfix indexing remains indexing | `tests/write_web_postfix_test.rs`; existing `tests/property_index_access_test.rs` | +| JavaScript transpilation uses the valid classic fallback for ambiguous `write line ... to ...`, but still rejects an unambiguous response-stream write | `tests/transpiler_test.rs` | + +## Red → Green record + +The tests were added and run before the corresponding production changes. +The observed Red failures were: + +- `cargo test --test flush_action_backcompat_test`: both new tests failed + because the parser/typechecker treated the target as a response stream + (`Expected a server response stream`). +- `cargo test --test write_web_postfix_test`: six new regressions failed, + reproducing the reported stream-target error, undefined `at`, bracket + misparse, display parse error, and runtime text-indexing error. +- `cargo test --test transpiler_test ambiguous_write_line_uses_classic_file_fallback`: + the transpiler rejected the statement as unsupported streaming. +- The standalone WFL E2E program failed on the same write and flush dispatch + errors. +- Additional direct-binary tests were observed Red for `line + ...` and direct + integer-target discrimination before their parser changes. + +The final Green focused command was: + +```text +cargo test --test write_web_postfix_test --test flush_action_backcompat_test --test transpiler_test --test property_index_access_test --test http_server_streaming_test +``` + +It passed all 123 focused tests. The standalone E2E command +`target\debug\wfl.exe --test TestPrograms\parser_streaming_compat_regression.test.wfl` +passed 4/4 tests. + +This working-tree session records the Red observations but does not claim a +test-only Red commit ancestor; preserving commit-level Red ancestry remains a +maintainer integration responsibility if these changes are committed. + +## Implementation + +The write parser now classifies exact bare `line`/`chunk` markers followed by a +classic expression continuation as classic file writes. For the still +ambiguous direct-integer form, it keeps a span-matched classic fallback while +preserving the response-stream interpretation. + +Exact `flush` forms with a parenthesized target or explicit same-line call now +carry the same exact-binding action fallback as the merged-identifier form. +The existing analyzer/runtime binding check decides between that legacy action +and response-stream flushing. + +Property-origin postfix parsing now requires source adjacency for brackets and +does not consume a spaced integer as an index. Direct and chained adjacent +property indexes continue to work. The JavaScript transpiler emits the classic +file-write fallback when the parser supplied one and continues rejecting +unambiguous streaming writes. + +## Verification + +Completed successfully: + +- `cargo fmt --all -- --check` +- `cargo clippy --all-targets --all-features -- -D warnings` +- `cargo test --all --jobs 2` (complete workspace and doctests) +- `cargo build --release` +- Focused 123-test parser, streaming, property, and transpiler command +- Standalone WFL E2E: 4 passed, 0 failed + +The official integration wrapper's split layer passed 11/11. Its unconstrained +parallel `cargo test --test '*'` then failed during compilation with Windows +error 1455 (paging file too small); no product test failed. The equivalent +complete workspace suite passed with compiler parallelism bounded to two jobs. +The wrapper also emits an existing read-only Cargo cache warning on this host. + +## Residual risk + +The discrimination is deliberately narrow: only exact marker/action bindings +receive compatibility fallbacks, and adjacent postfix syntax remains available. +CI should rerun the official integration and platform matrix on the final +committed candidate. No coverage percentage is claimed. diff --git a/Docs/superpowers/plans/2026-07-25-parser-compat-regressions.md b/Docs/superpowers/plans/2026-07-25-parser-compat-regressions.md new file mode 100644 index 00000000..9476c815 --- /dev/null +++ b/Docs/superpowers/plans/2026-07-25-parser-compat-regressions.md @@ -0,0 +1,106 @@ +# Parser Compatibility Regressions Implementation Plan + +> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking. + +**Goal:** Restore legacy file-write, zero-argument `flush` action, display-fold, and JavaScript transpilation behavior without weakening the new HTTP-stream syntax. + +**Architecture:** Keep ambiguous syntax represented explicitly in the AST and defer branch selection to the existing analyzer/typechecker/runtime or transpiler boundary. Restrict trailing postfix parsing to postfix forms that were historically owned by the preceding expression, using source adjacency where whitespace distinguishes a display fold from an index. + +**Tech Stack:** Rust 2024, WFL lexer/parser/analyzer/interpreter, JavaScript transpiler, Cargo integration tests, WFL end-to-end programs. + +## Global Constraints + +- Risk class is R3 because this changes language backward compatibility and streaming syntax. +- Preserve every previously valid WFL program; do not broaden streaming dispatch over legacy expression syntax. +- Follow Red → Green → Refactor → Broaden → Record with observed failing tests before production edits. +- Add coverage at the parser/transpiler layer and through the real `wfl` binary. + +--- + +### Task 1: Record compatibility regressions as failing tests + +**Files:** +- Modify: `tests/write_web_postfix_test.rs` +- Modify: `tests/transpiler_test.rs` +- Create: `TestPrograms/parser_streaming_compat_regression.wfl` + +**Interfaces:** +- Consumes: public lexer/parser, `wfl::transpiler::JavaScriptTranspiler`, and the built `wfl` binary. +- Produces: regression tests for classic `write line|chunk` continuations, exact `flush` action fallback, display folding, and transpiler fallback. + +- [ ] Add parser/runtime tests for `write line with "!"`, `write line at 0`, and `write line[0]` targeting a file. +- [ ] Add runtime tests proving zero-argument action `flush` still runs for `flush (expr)` and same-line `flush call ...`. +- [ ] Add parser/runtime tests proving `display alice.name [1, 2]` and `display alice.name 5` remain display folds. +- [ ] Add a transpiler test proving `write line note to "f.txt"` uses its classic file-write fallback. +- [ ] Add a real WFL end-to-end program that asserts the compatible runtime results. +- [ ] Run the focused tests and retain their expected failures as Red evidence. + +### Task 2: Restore classic write ownership for bare marker continuations + +**Files:** +- Modify: `src/parser/stmt/io.rs` +- Test: `tests/write_web_postfix_test.rs` + +**Interfaces:** +- Consumes: the token immediately following an exact `line`/`chunk` contextual marker. +- Produces: `WriteToStatement` for legacy continuations and `StreamWriteStatement` for genuine stream operands. + +- [ ] Extend the bare-marker guard to recognize legacy continuation starters (`with`, `at`, `[`, and `.`) rather than only `to`. +- [ ] Run the focused write parser/runtime tests and confirm Green. + +### Task 3: Preserve exact `flush` action fallback + +**Files:** +- Modify: `src/parser/stmt/web.rs` +- Test: `tests/write_web_postfix_test.rs` + +**Interfaces:** +- Consumes: exact `flush` dispatch followed by a parenthesized or explicit-call stream target. +- Produces: `FlushStreamStatement.action_fallback` containing the legacy zero-argument `flush` action expression. + +- [ ] Build the exact-token legacy fallback from the `flush` binding while independently parsing the stream target. +- [ ] Run the focused flush parser/runtime tests and confirm Green. + +### Task 4: Stop postfix parsing from stealing display folds + +**Files:** +- Modify: `src/parser/expr/primary.rs` +- Test: `tests/write_web_postfix_test.rs` + +**Interfaces:** +- Consumes: a property/method expression followed by possible postfix tokens. +- Produces: adjacent `property[index]` chaining while leaving whitespace-separated list and scalar expressions to the display fold. + +- [ ] Require bracket adjacency after a property/method expression before treating `[` as its postfix. +- [ ] Do not reinterpret a trailing bare integer as direct indexing after property/method access. +- [ ] Run the focused display and existing property-index tests and confirm Green. + +### Task 5: Transpile ambiguous classic writes through their fallback + +**Files:** +- Modify: `src/transpiler/javascript.rs` +- Test: `tests/transpiler_test.rs` + +**Interfaces:** +- Consumes: `StreamWriteStatement` with `fallback_content: Some`. +- Produces: the same `WFL.file.write(...)` JavaScript emitted for the legacy file-write reading; unambiguous streaming statements remain unsupported. + +- [ ] Split the transpiler match arm so ambiguous writes use `fallback_content` and `target`. +- [ ] Keep unambiguous HTTP stream writes as clear transpilation errors. +- [ ] Run transpiler tests and confirm Green. + +### Task 6: Broaden verification and record evidence + +**Files:** +- Modify: `Dev diary/2026-07-25-parser-streaming-compat-regressions.md` + +**Interfaces:** +- Consumes: final implementation and test output. +- Produces: durable R3 acceptance criteria, Red/Green commands, and residual-risk record. + +- [ ] Run `cargo fmt --all -- --check`. +- [ ] Run focused parser, transpiler, analyzer/typechecker, and property-index suites. +- [ ] Run `cargo test --all`. +- [ ] Run `cargo build --release` followed by `scripts/run_integration_tests.ps1`. +- [ ] Run `cargo clippy --all-targets --all-features -- -D warnings`. +- [ ] Record exact evidence and any residual risk in the Dev Diary. diff --git a/TestPrograms/parser_streaming_compat_regression.test.wfl b/TestPrograms/parser_streaming_compat_regression.test.wfl new file mode 100644 index 00000000..4b18947d --- /dev/null +++ b/TestPrograms/parser_streaming_compat_regression.test.wfl @@ -0,0 +1,50 @@ +// End-to-end compatibility coverage for syntax made ambiguous by HTTP streaming. + +define action called flush: + display "FLUSH_ACTION_CALLED" +end action + +define action called acquire stream: + return 1 +end action + +describe "Parser streaming compatibility": + + test "classic write keeps a with continuation on a variable named line": + store line as ["hello"] + write line with "!" to "parser_streaming_compat_with.txt" + open file at "parser_streaming_compat_with.txt" for reading as with_reader + wait for store with_result as read content from with_reader + close with_reader + expect with_result to contain "!" + delete file at "parser_streaming_compat_with.txt" + end test + + test "classic write keeps an at continuation on a variable named line": + store line as ["first"] + write line at 0 to "parser_streaming_compat_at.txt" + open file at "parser_streaming_compat_at.txt" for reading as at_reader + wait for store at_result as read content from at_reader + close at_reader + expect at_result to equal "first" + delete file at "parser_streaming_compat_at.txt" + end test + + test "classic write keeps a bracket continuation on a variable named line": + store line as ["first"] + write line[0] to "parser_streaming_compat_bracket.txt" + open file at "parser_streaming_compat_bracket.txt" for reading as bracket_reader + wait for store bracket_result as read content from bracket_reader + close bracket_reader + expect bracket_result to equal "first" + delete file at "parser_streaming_compat_bracket.txt" + end test + + test "exact flush forms keep the zero argument action": + store ignored as 1 + flush (ignored) + flush call acquire stream + expect ignored to equal 1 + end test + +end describe diff --git a/src/parser/expr/primary.rs b/src/parser/expr/primary.rs index 1e78c437..ba00e123 100644 --- a/src/parser/expr/primary.rs +++ b/src/parser/expr/primary.rs @@ -132,7 +132,7 @@ impl<'a> Parser<'a> { &mut self, expr: Expression, ) -> Result { - self.parse_trailing_postfix_with_clause_boundary(expr, false) + self.parse_trailing_postfix_with_clause_boundary(expr, false, None) } /// Clause-aware counterpart to [`Self::parse_trailing_postfix`]. Natural @@ -142,18 +142,36 @@ impl<'a> Parser<'a> { &mut self, expr: Expression, ) -> Result { - self.parse_trailing_postfix_with_clause_boundary(expr, true) + self.parse_trailing_postfix_with_clause_boundary(expr, true, None) + } + + fn parse_trailing_postfix_after_member( + &mut self, + expr: Expression, + stop_at_clause: bool, + member_end: usize, + ) -> Result { + self.parse_trailing_postfix_with_clause_boundary(expr, stop_at_clause, Some(member_end)) } fn parse_trailing_postfix_with_clause_boundary( &mut self, mut expr: Expression, stop_at_clause: bool, + mut adjacent_member_end: Option, ) -> Result { while let Some(tok) = self.cursor.peek() { let (line, column) = (tok.line, tok.column); match &tok.token { Token::LeftBracket => { + // A bracket separated from `.property` / `.method()` by + // whitespace starts a fresh display value (`display + // alice.name [1, 2]`). Only an adjacent bracket belongs to + // the completed member expression. Seeded streaming operands + // pass `None` and retain their historical permissive spacing. + if adjacent_member_end.is_some_and(|member_end| member_end != tok.byte_start) { + break; + } // Anchor a "missing `]`" span to the `[` token itself, not the // start of the file. let (bracket_start, bracket_end) = (tok.byte_start, tok.byte_end); @@ -163,6 +181,7 @@ impl<'a> Parser<'a> { match self.cursor.peek() { Some(closing) if closing.token == Token::RightBracket => { + adjacent_member_end = Some(closing.byte_end); self.bump_sync(); // Consume ']' } Some(closing) => { @@ -196,13 +215,13 @@ impl<'a> Parser<'a> { // of the file. let (dot_start, dot_end) = (tok.byte_start, tok.byte_end); self.bump_sync(); // Consume '.' - let property = match self.cursor.peek() { + let (property, property_end) = match self.cursor.peek() { // Keywords that double as common property names (e.g. // `response.status`) are accepted, matching the primary // dispatch's property-access handling. Some(prop) => match &prop.token { - Token::Identifier(name) => name.clone(), - Token::KeywordStatus => "status".to_string(), + Token::Identifier(name) => (name.clone(), prop.byte_end), + Token::KeywordStatus => ("status".to_string(), prop.byte_end), _ => { return Err(ParseError::from_token( "Expected a property name after '.'".to_string(), @@ -224,6 +243,7 @@ impl<'a> Parser<'a> { } }; self.bump_sync(); // Consume the property name + adjacent_member_end = Some(property_end); // `.method(args)` — a method call, not a bare property access. // Mirrors the primary dispatch so merged-command operands like // `write line obj.method() to out` / `flush obj.method()` compose @@ -247,10 +267,16 @@ impl<'a> Parser<'a> { }); } } + let method_end = self + .cursor + .peek() + .filter(|token| token.token == Token::RightParen) + .map(|token| token.byte_end); self.expect_token( Token::RightParen, "Expected ')' after method arguments", )?; + adjacent_member_end = method_end; expr = Expression::MethodCall { object: Box::new(expr), method: property, @@ -271,6 +297,12 @@ impl<'a> Parser<'a> { // primary postfix loop. Required so classic // `write line values 0 to "/tmp/out"` still parses (issue #642). Token::IntLiteral(index) => { + // Direct integer indexing after a property/method was not + // part of the legacy primary grammar. Leaving it unconsumed + // lets `display alice.name 5` fold two display values. + if adjacent_member_end.is_some() { + break; + } if matches!( expr, Expression::Variable(_, _, _) @@ -476,6 +508,7 @@ impl<'a> Parser<'a> { self.bump_sync(); // Consume '.' if let Some(property_token) = self.cursor.peek() { + let property_end = property_token.byte_end; // Keywords that are also common property names // (e.g. `response.status`) are accepted here let parsed_property = match &property_token.token { @@ -518,6 +551,12 @@ impl<'a> Parser<'a> { } } + let method_end = self + .cursor + .peek() + .filter(|token| token.token == Token::RightParen) + .map(|token| token.byte_end) + .unwrap_or(property_end); self.expect_token( Token::RightParen, "Expected ')' after method arguments", @@ -534,9 +573,10 @@ impl<'a> Parser<'a> { line: token_line, column: token_column, }; - return self.parse_trailing_postfix_with_clause_boundary( + return self.parse_trailing_postfix_after_member( call, stop_at_clause, + method_end, ); } @@ -557,9 +597,10 @@ impl<'a> Parser<'a> { line: token_line, column: token_column, }; - return self.parse_trailing_postfix_with_clause_boundary( + return self.parse_trailing_postfix_after_member( access, stop_at_clause, + property_end, ); } else { return Err(ParseError::from_token( diff --git a/src/parser/stmt/io.rs b/src/parser/stmt/io.rs index f54049e6..08c2b2a6 100644 --- a/src/parser/stmt/io.rs +++ b/src/parser/stmt/io.rs @@ -943,20 +943,46 @@ impl<'a> IoParser<'a> for Parser<'a> { // the same token (`line payload` -> Identifier("line payload")), so // split the value off the marker, mirroring the websocket-message form. // - // Do NOT intercept a bare `line`/`chunk` that is immediately followed by - // `to`: that is the classic `write to ` form using a - // variable literally named `line`/`chunk` (common in line-by-line file - // processing). The streaming form always has a value between the marker - // and `to`, so a bare marker directly before `to` is not a stream write. - let bare_marker_before_to = matches!( + // Do NOT intercept a bare `line`/`chunk` followed by an expression + // continuation. In that shape the marker is itself the classic file-write + // value (`write line with "!" to file`, `write line[0] to file`, ...), + // not a streaming marker with a missing operand. This is common in + // line-by-line file code and predates response streaming. + let bare_marker_before_classic_continuation = matches!( self.cursor.peek(), Some(t) if matches!(&t.token, Token::Identifier(id) if id == "line" || id == "chunk") ) && matches!( self.cursor.peek_kind_n(1), - Some(Token::KeywordTo) + Some( + Token::KeywordTo + | Token::KeywordWith + | Token::KeywordAt + | Token::Dot + | Token::LeftBracket + | Token::KeywordOf + | Token::Plus + | Token::KeywordPlus + | Token::Minus + | Token::KeywordMinus + | Token::KeywordTimes + | Token::KeywordDividedBy + | Token::KeywordDivided + | Token::Slash + | Token::Percent + | Token::KeywordModulo + | Token::Equals + | Token::KeywordIs + | Token::KeywordAnd + | Token::KeywordOr + | Token::KeywordMatches + | Token::KeywordContains + | Token::KeywordFind + | Token::KeywordReplace + | Token::KeywordSplit + ) ); - if !bare_marker_before_to + if !bare_marker_before_classic_continuation && let Some(next_token) = self.cursor.peek() && let Token::Identifier(id) = &next_token.token && (id == "line" @@ -982,11 +1008,32 @@ impl<'a> IoParser<'a> for Parser<'a> { // space-separated names), so we carry the file-write interpretation // and let the interpreter pick based on whether `target` is a stream. let (value, fallback_content) = if rest.is_empty() { - // Value begins with a non-identifier (string/number), so the - // whole expression — including `with` concatenation — parses - // cleanly from here. This form was never a valid classic file - // write (`write line "x" to f` did not parse), so no fallback. - (self.parse_unmerged_operand(false)?, None) + // Non-identifier values normally have only the stream reading. + // Direct integers are the exception because legacy WFL also + // supports `line 0` as direct indexing. + let direct_index = matches!( + self.cursor.peek().map(|token| &token.token), + Some(Token::IntLiteral(_)) + ); + if direct_index { + // A direct integer is ambiguous: stream value `0` vs legacy + // direct indexing on a variable literally named `line`. + let value_start = self.cursor.checkpoint(); + let value = self.parse_unmerged_operand(false)?; + let after_stream = self.cursor.checkpoint(); + + self.cursor.rewind(value_start); + let file_left = + Expression::Variable(marker.to_string(), marker_line, marker_column); + let fallback = self.parse_write_value_from_lead(file_left).ok(); + let fallback_end = self.cursor.checkpoint(); + let fallback = fallback.filter(|_| fallback_end == after_stream); + self.cursor.rewind(after_stream); + + (value, fallback.map(Box::new)) + } else { + (self.parse_unmerged_operand(false)?, None) + } } else { // Ambiguous merged form: `` alone (stream) vs the full // merged `line ` (classic file write of that variable). diff --git a/src/parser/stmt/web.rs b/src/parser/stmt/web.rs index 384466fe..6fab753f 100644 --- a/src/parser/stmt/web.rs +++ b/src/parser/stmt/web.rs @@ -548,7 +548,17 @@ impl<'a> WebParser<'a> for Parser<'a> { .map(str::trim_start) .unwrap_or(""); let (target, legacy_binding, action_fallback) = if rest.is_empty() { - (self.parse_unmerged_operand(false)?, None, None) + // Exact `flush` followed by an unmerged target is dispatched only for + // the unambiguous streaming starters `(` and `call`. Before streaming, + // however, a defined zero-argument action named exactly `flush` still + // auto-ran and the remaining same-line expression did not turn that + // action into a stream operation. Preserve that binding as the same + // action fallback used by the merged form. + ( + self.parse_unmerged_operand(false)?, + Some(phrase.clone()), + Some(Expression::Variable(phrase.clone(), line, column)), + ) } else { // Stream reading: postfix on the split-off rest (`cache` from // `flush cache`). Legacy expression: same postfix on the FULL phrase diff --git a/src/transpiler/javascript.rs b/src/transpiler/javascript.rs index 65e9fdf0..b2077389 100644 --- a/src/transpiler/javascript.rs +++ b/src/transpiler/javascript.rs @@ -635,11 +635,35 @@ impl JavaScriptTranspiler { }) } + Statement::StreamWriteStatement { + target, + fallback_content: Some(fallback_content), + .. + } => { + // The parser preserves both readings of ambiguous classic syntax + // such as `write line note to "f.txt"`. JavaScript has no response + // stream implementation, but it can still emit the pre-streaming + // file-write reading exactly as it did before. + let content_expr = self.transpile_expression(fallback_content)?; + let file_expr = self.transpile_expression(target)?; + Ok(format!( + "{}WFL.file.write({}.path, {});\n", + self.indent(), + file_expr, + content_expr + )) + } + Statement::HttpStreamStatement { line, column, .. } | Statement::WaitForNextChunkStatement { line, column, .. } | Statement::WaitForNextLineStatement { line, column, .. } | Statement::StartStreamingResponseStatement { line, column, .. } - | Statement::StreamWriteStatement { line, column, .. } + | Statement::StreamWriteStatement { + line, + column, + fallback_content: None, + .. + } | Statement::FlushStreamStatement { line, column, .. } => { // Streaming HTTP relies on the interpreter's parked-stream // handles; emitting broken JS would be worse than a clear error. diff --git a/tests/flush_action_backcompat_test.rs b/tests/flush_action_backcompat_test.rs index 404ef5bc..8319a0b8 100644 --- a/tests/flush_action_backcompat_test.rs +++ b/tests/flush_action_backcompat_test.rs @@ -78,6 +78,46 @@ fn truly_bare_flush_still_calls_the_legacy_zero_argument_action() { ); } +#[test] +fn parenthesized_flush_target_keeps_the_exact_zero_argument_action_fallback() { + let src = "define action called flush:\n\ + \x20\x20\x20\x20display \"PAREN_CALLED\"\n\ + end action\n\ + store ignored as 1\n\ + flush (ignored)\n"; + let (out, code) = run_src(src); + assert_eq!( + code, + Some(0), + "`flush (expr)` must keep the exact `flush` action fallback; output:\n{out}" + ); + assert!( + out.contains("PAREN_CALLED"), + "the exact zero-argument action must run; output:\n{out}" + ); +} + +#[test] +fn explicit_call_flush_target_keeps_the_exact_zero_argument_action_fallback() { + let src = "define action called flush:\n\ + \x20\x20\x20\x20display \"CALL_CALLED\"\n\ + end action\n\ + define action called acquire stream:\n\ + \x20\x20\x20\x20return 1\n\ + end action\n\ + flush call acquire stream\n"; + let (out, code) = run_src(src); + assert_eq!( + code, + Some(0), + "`flush call ...` must keep the exact `flush` action fallback; output:\n{out}" + ); + assert!( + out.contains("CALL_CALLED"), + "the exact zero-argument action must run; output:\n{out}" + ); +} + #[test] fn truly_bare_flush_still_evaluates_a_non_callable_legacy_variable() { let src = "store flush as 1\n\ diff --git a/tests/transpiler_test.rs b/tests/transpiler_test.rs index fc5e0d72..bedcb188 100644 --- a/tests/transpiler_test.rs +++ b/tests/transpiler_test.rs @@ -67,6 +67,24 @@ fn test_display_statement() { assert_contains(&js, r#"WFL.display("Hello, World!");"#); } +#[test] +fn test_ambiguous_write_line_uses_classic_file_fallback() { + let source = "store line note as \"hello\"\nwrite line note to \"f.txt\""; + let js = transpile_wfl(source) + .expect("an ambiguous write with a classic file fallback must transpile"); + assert_contains(&js, "WFL.file.write(\"f.txt\".path, line_note);"); +} + +#[test] +fn test_unambiguous_stream_write_still_fails_to_transpile() { + let error = transpile_wfl("write line \"hello\" to out") + .expect_err("a genuine response-stream write has no JavaScript translation"); + assert!( + error.contains("Streaming HTTP statements are not supported"), + "expected the streaming-specific transpiler error, got: {error}" + ); +} + #[test] fn test_if_statement() { let source = r#" diff --git a/tests/write_web_postfix_test.rs b/tests/write_web_postfix_test.rs index 969a1d9b..6b776a88 100644 --- a/tests/write_web_postfix_test.rs +++ b/tests/write_web_postfix_test.rs @@ -983,3 +983,137 @@ fn classic_indexed_file_write_still_works_at_runtime() { "the indexed element must be written, not the whole list" ); } + +fn run_file_write_with_line_binding(statement: &str, declaration: &str) -> String { + let dir = TempDir::new().expect("tempdir"); + let out = dir.path().join("out.txt"); + let out_str = out.to_string_lossy().replace('\\', "/"); + let src = format!("{declaration}\n{statement} to \"{out_str}\"\n"); + let program_file = dir.path().join("main.wfl"); + fs::write(&program_file, &src).expect("write program"); + let output = Command::new(env!("CARGO_BIN_EXE_wfl")) + .arg(&program_file) + .output() + .expect("run wfl"); + let combined = format!( + "{}{}", + String::from_utf8_lossy(&output.stdout), + String::from_utf8_lossy(&output.stderr) + ); + assert!( + output.status.success(), + "`{statement}` must remain a classic file write; output:\n{combined}" + ); + fs::read_to_string(out).expect("classic write must create output file") +} + +#[test] +fn bare_line_binding_keeps_with_continuation_in_classic_file_write() { + let written = + run_file_write_with_line_binding("write line with \"!\"", "store line as \"hello\""); + assert_eq!(written, "hello!"); +} + +#[test] +fn bare_line_binding_keeps_natural_index_continuation_in_classic_file_write() { + let written = run_file_write_with_line_binding("write line at 0", "store line as [\"first\"]"); + assert_eq!(written, "first"); +} + +#[test] +fn bare_line_binding_keeps_bracket_index_continuation_in_classic_file_write() { + let written = run_file_write_with_line_binding("write line[0]", "store line as [\"first\"]"); + assert_eq!(written, "first"); +} + +#[test] +fn bare_line_binding_keeps_binary_continuation_in_classic_file_write() { + let written = run_file_write_with_line_binding("write line plus 1", "store line as 4"); + assert_eq!(written, "5"); +} + +#[test] +fn bare_line_binding_keeps_direct_integer_index_as_classic_fallback() { + let written = run_file_write_with_line_binding("write line 0", "store line as [\"first\"]"); + assert_eq!(written, "first"); +} + +#[test] +fn display_property_followed_by_spaced_list_keeps_legacy_statement_split() { + let program = parse("display alice.name [1, 2]\n"); + assert_eq!(program.statements.len(), 2, "got {:#?}", program.statements); + assert!( + matches!( + &program.statements[0], + Statement::DisplayStatement { + value: Expression::PropertyAccess { .. }, + .. + } + ), + "the property itself must remain the displayed value; got {:#?}", + program.statements[0] + ); + assert!( + matches!( + &program.statements[1], + Statement::ExpressionStatement { + expression: Expression::Literal(wfl::parser::ast::Literal::List(_), ..), + .. + } + ), + "the spaced list must remain the separate legacy expression, not an index; got {:#?}", + program.statements[1] + ); +} + +#[test] +fn display_property_followed_by_integer_remains_a_display_fold() { + let program = parse("display alice.name 5\n"); + assert_eq!(program.statements.len(), 1, "got {:#?}", program.statements); + assert!( + matches!( + &program.statements[0], + Statement::DisplayStatement { + value: Expression::Concatenation { left, right, .. }, + .. + } if matches!(left.as_ref(), Expression::PropertyAccess { .. }) + && matches!( + right.as_ref(), + Expression::Literal(wfl::parser::ast::Literal::Integer(5), ..) + ) + ), + "the integer must be a second display value, not direct indexing; got {:#?}", + program.statements[0] + ); +} + +#[test] +fn display_property_compatibility_executes_through_the_real_binary() { + let dir = TempDir::new().expect("tempdir"); + let program_file = dir.path().join("main.wfl"); + fs::write( + &program_file, + "create map alice:\n\ + \x20\x20\x20\x20\"name\" is \"Alice\"\n\ + end map\n\ + display alice.name 5\n", + ) + .expect("write program"); + let output = Command::new(env!("CARGO_BIN_EXE_wfl")) + .arg(&program_file) + .output() + .expect("run wfl"); + let combined = format!( + "{}{}", + String::from_utf8_lossy(&output.stdout), + String::from_utf8_lossy(&output.stderr) + ); + assert!( + output.status.success(), + "display fold must not index `Alice` by 5; output:\n{combined}" + ); + assert!( + combined.contains("Alice5"), + "legacy display fold must print both values; output:\n{combined}" + ); +}