Add streaming response support: client, server, and concurrent handlers

.github/workflows/ci.yml

@@ -66,6 +66,20 @@ jobs:
         with:
           components: rustfmt, clippy
 
+      # Reclaim runner disk before building. This job compiles the workspace
+      # several times (debug + two release builds) plus the whole test suite, and
+      # the release profile keeps full debuginfo (`debug = true`), so the target
+      # tree is large; a full GitHub-hosted runner can otherwise exhaust its disk
+      # mid-link (a linker `Bus error`/SIGBUS). Removing preinstalled SDKs we do
+      # not use frees ~20 GB with no third-party action.
+      - name: Free disk space (Linux)
+        if: runner.os == 'Linux'
+        run: |
+          sudo rm -rf /usr/share/dotnet /usr/local/lib/android /opt/ghc \
+            /opt/hostedtoolcache/CodeQL /usr/local/share/boost /usr/local/graalvm || true
+          sudo docker image prune --all --force > /dev/null 2>&1 || true
+          df -h /
+
       # Cache Cargo registry and target directory for faster builds
       - name: Cache Cargo registry and target directory
         uses: Swatinem/rust-cache@v2
@@ -114,9 +128,11 @@ jobs:
       - name: Build LSP
         run: cargo build -p wfl-lsp --verbose
 
-      # Run Clippy for code quality
+      # Run Clippy for code quality. `--all-features` matches the binding gate in
+      # testing.md (the only features are opt-in dhat profiling, so this just
+      # compiles the feature-gated code for linting — it never runs it).
       - name: Run Clippy
-        run: cargo clippy --all-targets -- -D warnings
+        run: cargo clippy --all-targets --all-features -- -D warnings
 
   # Cross-platform integration test verification
   integration-tests:
@@ -135,6 +151,19 @@ jobs:
       - uses: actions/checkout@v4
       - uses: dtolnay/rust-toolchain@stable
 
+      # Reclaim runner disk before building: this job builds the (debuginfo-heavy)
+      # release tree AND every integration test binary (`cargo test --test '*'`),
+      # which together can exhaust a full runner's disk mid-link (linker
+      # `Bus error`/SIGBUS). Freeing unused preinstalled SDKs gives ~20 GB of
+      # headroom. Linux only — the Windows runner is not affected.
+      - name: Free disk space (Linux)
+        if: runner.os == 'Linux'
+        run: |
+          sudo rm -rf /usr/share/dotnet /usr/local/lib/android /opt/ghc \
+            /opt/hostedtoolcache/CodeQL /usr/local/share/boost /usr/local/graalvm || true
+          sudo docker image prune --all --force > /dev/null 2>&1 || true
+          df -h /
+
       # Cache Cargo registry and target directory for faster builds
       - name: Cache Cargo registry and target directory
         uses: Swatinem/rust-cache@v2
@@ -164,13 +193,47 @@ jobs:
           }
           Write-Host "✓ Release binary found: target/release/wfl.exe"
 
-      # Run integration tests specifically
-      - name: Run Integration Tests
-        run: cargo test --test split_functionality --verbose
+      # Run the DOCUMENTED WFL integration gate (testing.md): the same script a
+      # contributor runs locally. It executes the integration test binaries
+      # (`cargo test --test '*'`) AND the TestPrograms end-to-end programs —
+      # crucially including the intentional-error programs, which it asserts exit
+      # nonzero (previously those assertions lived only in this script and the
+      # script was never invoked in CI, so they never ran). Uses the release
+      # binary built above. Run on BOTH OSes so the declared Windows integration
+      # command is actually exercised, not merely documented.
+      - name: Run Integration Gate (Unix)
+        if: runner.os != 'Windows'
+        run: ./scripts/run_integration_tests.sh
+
+      - name: Run Integration Gate (Windows)
+        if: runner.os == 'Windows'
+        shell: pwsh
+        run: ./scripts/run_integration_tests.ps1
+
+      # Validate that documentation examples still parse/analyze/lint against the
+      # current release binary (testing.md requires docs validation in CI).
+      # `--force` ignores the committed cache so CI always re-validates rather
+      # than trusting a stale cached result.
+      - name: Validate Docs Examples (Unix)
+        if: runner.os != 'Windows'
+        run: python3 scripts/validate_docs_examples.py --ci --force
+
+      - name: Validate Docs Examples (Windows)
+        if: runner.os == 'Windows'
+        run: python scripts/validate_docs_examples.py --ci --force
+
+      # Web-server integration tests: start real WFL servers and exercise them
+      # over HTTP (testing.md requires web tests in CI for the R3 web/streaming
+      # surface). Uses the release binary built above. Both OSes are covered so
+      # Windows web-server behavior does not go unvalidated.
+      - name: Run Web Server Tests (Unix)
+        if: runner.os != 'Windows'
+        run: ./scripts/run_web_tests.sh
 
-      # Run all integration tests to ensure comprehensive coverage
-      - name: Run All Integration Tests
-        run: cargo test --test '*' --verbose
+      - name: Run Web Server Tests (Windows)
+        if: runner.os == 'Windows'
+        shell: pwsh
+        run: ./scripts/run_web_tests.ps1
 
   # Database integration tests against live PostgreSQL and MariaDB servers.
   # SQLite database tests need no services and already run everywhere via

AGENTS.md

@@ -12,12 +12,13 @@ Binding community and contribution policy lives at the **repo root** (not only u
 | `AI_POLICY.md` | **AI-assisted work is welcome** — WFL was built with AI; do not discriminate against AI use; human author remains accountable |
 | `CONTRIBUTING.md` | How to contribute; **Contributor application** process (Discussion or email) |
 | `SECURITY.md` | Private vulnerability reporting only — never file security bugs as public issues |
+| `testing.md` | **Binding Logbie Testing Policy + WFL testing profile** — Red→Green TDD evidence, required test layers, risk classes, and merge/release gates (see **Testing Guidelines** below) |
 
 **Agent implications (already in force via governance):**
 
 - **AI is first-class** — use coding agents freely; same quality bar as hand-written work (tests, docs, compatibility, reviewability).
 - **Backward compatibility is sacred** — never break existing WFL programs without the documented deprecation path.
-- **TDD mandatory** — failing tests first (`tests/`, `TestPrograms/`).
+- **TDD mandatory** — failing tests first (`tests/`, `TestPrograms/`), governed by the binding **Logbie Testing Policy** in root `testing.md`: auditable **Red→Green** evidence for every behavioral change, coverage at the lowest useful layer plus every affected higher layer.
 - **Docs ship with the feature** — same change; validate examples; Dev Diary for non-trivial work.
 - **Quality gates** — `cargo fmt`, `clippy -D warnings`, `cargo test`; conventional commits.
 - **Do not invent maintainer identity or process** — Contributor status is by application; Maintainers own merges and releases unless those responsibilities are **explicitly delegated**. Prefer first name **Brad** only if referring to the primary maintainer in docs (no last name).
@@ -121,12 +122,40 @@ Source Code → Lexer → Parser → Analyzer → Type Checker → Interpreter
   - Constants: `SCREAMING_SNAKE_CASE`
 
 ## Testing Guidelines
-- **TDD is mandatory**: Write failing tests FIRST for any feature or bug fix.
+
+**Binding policy:** root `testing.md` holds the **Logbie Testing Policy** and the
+WFL testing profile. It governs every behavioral change. Non-negotiables an agent
+MUST follow:
+
+- **Red → Green → Refactor → Broaden → Record** — write the smallest useful test
+  FIRST, run it, confirm it **fails for the intended reason**, then make it pass;
+  a defect fix reproduces the defect. Keep auditable Red evidence (a Red commit
+  that is an ancestor of Green, or a timestamped CI artifact). A test first
+  observed after the code already passed is **not** a valid Red step. (§3, §6)
+- **Classify risk first (R0–R3)** — concurrency, cancellation, lifecycle,
+  streaming, untrusted input, crypto/secrets, and backward compatibility are
+  **R3** and require negative/failure-path plus §11 risk-triggered tests. Risk is
+  never lowered to dodge a gate. (§5, §11)
+- **Real boundaries, real assertions** — don't mock the boundary under test;
+  assert outcomes + side effects (not "didn't crash"); use negative assertions
+  for cancellation, writes-after-close, denial. (§7, §8.3)
+- **No manufactured green** — never retry/skip/quarantine a required test to go
+  green; a flaky required test is failing. Non-executable docs programs use the
+  runner's `// CI-SKIP:` first-line directive and stay statically validated. (§8.2)
+- **Concurrency/streaming/lifecycle (§11.3)** — for this repo's async/web/
+  streaming work, prove races/ordering, cancellation, timeouts, disconnects,
+  bounded queues/backpressure, resource limits, clean shutdown, writes-after-
+  close, and that one slow/failed handler doesn't block unrelated work.
+- **PR evidence (§15)** — record risk class, acceptance criteria → tests, Red
+  evidence, layers run, and residual risk (template in `testing.md`).
+
+### Mechanics
 - **Locations**:
   - Rust Unit/Integration: `tests/`
   - WFL End-to-End: `TestPrograms/` (must pass with release build)
   - WFL Test Framework: Use `describe`/`test` blocks, run with `wfl --test <file>`
 - **Conventions**: feature‑oriented names (`*_test.rs`, `*.test.wfl`), keep perf benches under `benches/`.
+- **Commands & profile**: one command per layer + the "run all presubmit" block are in root `testing.md`.
 - **Testing Guide**: See `Docs/guides/testing-guide.md` for WFL testing framework documentation.
 
 ## Commit & Pull Request Guidelines

CLAUDE.md

@@ -14,12 +14,13 @@ Binding community and contribution policy lives at the **repo root** (not only u
 | `AI_POLICY.md` | **AI-assisted work is welcome** — WFL was built with AI; do not discriminate against AI use; human author remains accountable |
 | `CONTRIBUTING.md` | How to contribute; **Contributor application** process (Discussion or email) |
 | `SECURITY.md` | Private vulnerability reporting only — never file security bugs as public issues |
+| `testing.md` | **Binding Logbie Testing Policy + WFL testing profile** — Red→Green TDD evidence, required test layers, risk classes, and merge/release gates (see **Testing Policy** below) |
 
 **Agent implications (already in force via governance):**
 
 - **AI is first-class** — use coding agents freely; same quality bar as hand-written work (tests, docs, compatibility, reviewability).
 - **Backward compatibility is sacred** — never break existing WFL programs without the documented deprecation path.
-- **TDD mandatory** — failing tests first (`tests/`, `TestPrograms/`).
+- **TDD mandatory** — failing tests first (`tests/`, `TestPrograms/`). Governed by the binding **Logbie Testing Policy** in root `testing.md` (see **Testing Policy** below): every behavioral change needs auditable **Red→Green** evidence and coverage at the lowest useful layer plus every affected higher layer.
 - **Docs ship with the feature** — same change; validate examples; Dev Diary for non-trivial work.
 - **Quality gates** — `cargo fmt`, `clippy -D warnings`, `cargo test`; conventional commits.
 - **Do not invent maintainer identity or process** — Contributor status is by application; Maintainers own merges and releases unless those responsibilities are **explicitly delegated**. Prefer first name **Brad** only if referring to the primary maintainer in docs (no last name).
@@ -159,13 +160,49 @@ Source Code → Lexer → Parser → Analyzer → Type Checker → Interpreter
   - Types/Traits: `CamelCase`
   - Constants: `SCREAMING_SNAKE_CASE`
 
-## Testing Guidelines
-- **TDD is mandatory**: Write failing tests FIRST for any feature or bug fix.
+## Testing Policy (binding — root `testing.md`)
+
+WFL adopts the **Logbie Testing Policy** (full text + the WFL testing profile in
+root `testing.md`). It is binding for every behavioral change; the highlights an
+agent MUST follow:
+
+- **Red → Green → Refactor → Broaden → Record.** Write the smallest useful test
+  FIRST and run it to confirm it **fails for the intended reason**, then make it
+  pass. A defect fix MUST reproduce the defect. Keep auditable evidence (a Red
+  test-only commit that is an ancestor of the Green commit, or a timestamped CI
+  artifact) — a test first observed *after* the code already passed does **not**
+  establish Red. (§3, §6)
+- **Risk class first.** Classify R0–R3 before implementing; when ambiguous, the
+  higher class applies, and it MUST NOT be lowered to dodge a gate. Anything
+  touching **concurrency, cancellation, lifecycle, streaming, untrusted input,
+  crypto/secrets, or backward compatibility is R3** and needs negative/
+  failure-path + the §11.3/§11.1 risk-triggered tests. (§5, §11)
+- **Real boundaries.** A test MUST NOT mock the boundary it claims to verify;
+  "end-to-end" means the real binary/socket/file. Assert outcomes and side
+  effects, not "did not crash." Use negative assertions where absence matters
+  (cancellation, writes-after-close, denial). (§7, §8.3)
+- **No manufactured green.** Required tests are never made green via retries,
+  skips, ignores, quarantine, or relaxed assertions; a flaky required test is a
+  failing test. Non-executable docs examples use the runner's `// CI-SKIP:`
+  first-line directive and are still validated statically. (§8.2)
+- **Concurrency/streaming/lifecycle (§11.3) — always required for this repo's
+  async/web/streaming work:** prove races/ordering, cancellation, timeouts,
+  disconnects, bounded queues/backpressure, resource limits, clean shutdown, and
+  writes-after-close, and that one slow/failed handler does not block unrelated
+  work.
+- **PR evidence (§15).** Every behavioral PR records risk class, acceptance
+  criteria → tests, Red evidence, the layers run, and residual risk (template in
+  `testing.md`).
+- **Same bar for AI work.** AI-authored code/tests get the same verification —
+  "the model said it works" is not evidence.
+
+### Testing mechanics
 - **Locations**:
   - Rust Unit/Integration: `tests/`
   - WFL End-to-End: `TestPrograms/` (must pass with release build)
   - WFL Test Framework: Use `describe`/`test` blocks, run with `wfl --test <file>`
 - **Conventions**: feature‑oriented names (`*_test.rs`, `*.test.wfl`), keep perf benches under `benches/`.
+- **Commands & profile**: one command per layer + the "run all presubmit" block are in root `testing.md`.
 - **Testing Guide**: See `Docs/guides/testing-guide.md` for WFL testing framework documentation.
 
 ## Commit & Pull Request Guidelines

Dev diary/2026-07-22-concurrent-handler-run-state-isolation.md

@@ -0,0 +1,72 @@
+# Dev Diary — 2026-07-22 — Per-handler run-state isolation for `main loop concurrently:`
+
+## Context
+
+PR #641 shipped `main loop concurrently:` — opt-in cooperative concurrency for
+HTTP request handlers, driven by a `FuturesUnordered` on the single interpreter
+thread. Review (maintainer P1 #1, echoed by Copilot) flagged a soundness gap:
+the concurrent loop isolated each handler's **environment** (variables), but the
+interpreter's **run-state** — the count-loop variable and its flag
+(`current_count` / `in_count_loop`), the live recursion depth (`call_depth`),
+the diagnostic call stack (`call_stack`), and the current block's overload-dup
+set — still lived on the shared `Interpreter` behind `RefCell`/`Cell`.
+
+Under serial execution that state is never contended. Under
+`main loop concurrently:` several handler futures interleave on one thread, so at
+every `await` one handler's run-state was visible to — and overwritable by —
+whichever sibling was polled next. A handler that yielded *inside a `count` loop*
+would resume and read a `count` set by another handler.
+
+## The bug, concretely
+
+`count` does not resolve through the environment while a count loop is active;
+`try_evaluate_variable_sync` short-circuits on `in_count_loop` and reads
+`self.current_count` directly. Both fields are global, so two concurrent count
+loops share one `current_count`. A handler counting `1..5` that yields mid-loop
+could come back reading `100..104` from a sibling.
+
+## Fix — a poll-swap wrapper (no `Rc`→`Arc`, no threads)
+
+The interpreter core stays single-threaded and `Rc`-based (a hard constraint).
+Rather than thread a per-handler execution context through every `&self` method,
+each handler owns a `RunState` snapshot and an `IsolatedHandler` future wraps the
+handler:
+
+- On **each `poll`**, `swap_run_state` swaps the handler's `RunState` into the
+  interpreter's live fields (a field-by-field `mem::swap`, its own inverse).
+- The inner handler future is polled.
+- The instant `poll` returns — `Ready` **or** `Pending` — the state is swapped
+  back out into the handler's `RunState`.
+
+So the interpreter's run-state fields become effectively poll-local: exactly one
+handler's state is installed at a time, and a suspended handler's state is parked
+in its own `RunState` where no sibling can touch it. Each handler starts from
+`RunState::fresh(base_call_depth)`. The inner future is already wrapped in
+`catch_unwind`, so a panic surfaces as `Ready` and the swap-back still runs,
+leaving the scratch fields clean for the next sibling.
+
+Serial execution is completely untouched — `IsolatedHandler` is used only by
+`execute_concurrent_main_loop`.
+
+## Testing (Red → Green)
+
+`tests/concurrent_main_loop_test.rs::test_concurrent_handlers_do_not_share_count_loop_state`:
+two concurrent handlers each run a `count` loop over a **disjoint** range
+(`1..5` vs `100..104`), yielding via `wait for` mid-iteration and then reading
+`count`. With isolation each handler observes only its own range.
+
+- **Red** (isolation bypassed — plain handler pushed to `FuturesUnordered`):
+  `/a` returned `100-101-102-103-104-`, i.e. it observed the *other* handler's
+  entire count range. `assertion left == right failed`.
+- **Green** (isolation restored): `/a` → `1-2-3-4-5-`, `/b` →
+  `100-101-102-103-104-`.
+
+Risk class **R3** (concurrency + lifecycle). The test asserts a concrete wrong
+outcome under sharing, not merely "did not crash".
+
+## Follow-ups still open from the review
+
+Larger P1 items remain and are tracked in
+`Docs/development/concurrency-phase-plan.md`: immediate-500 on pre-respond
+failure, browser-disconnect/504 cancellation threaded into `wait for` and
+upstream reads, and an absolute total-stream deadline.