getworkloop · bft-codebot · Apr 29, 2026 · Apr 29, 2026
diff --git a/reviews/2026-04-28-browser-introspection-tools-AAR.md b/reviews/2026-04-28-browser-introspection-tools-AAR.md
@@ -0,0 +1,109 @@
+# Browser Runtime Introspection Tools AAR
+
+## Context
+
+Workloop needed more browser read tools after the pointer-targeting pass:
+waiting, page state, text/region queries, point inspection, viewport controls,
+highlighting, link extraction, and recent console/network diagnostics. The
+Workloop tool boundary should remain narrow and must not expose arbitrary
+browser eval.
+
+Because Workloop routes browser commands through the shared Gambit browser
+runtime, the live-session command contract needed to support those capabilities.
+
+## Intent
+
+- Purpose: add structured browser live-session introspection commands that
+  Workloop can expose as narrow coworker tools.
+- End State: live sessions support read-only page state, text, region, element,
+  viewport, point, link, console, and network-failure inspection, plus bounded
+  wait and highlight commands.
+- Constraints / Tradeoffs: keep arbitrary eval out of the Workloop-facing
+  surface; keep the runtime command payloads explicit and typed; preserve
+  existing query/ref/mouse commands.
+- Phase (if applicable): Workloop browser-use Phase 2 support with Phase 6
+  screenshot/debug evidence support.
+
+## What Happened
+
+Extended `BrowserLiveSessionCommand` with wait, page-state, text query, region
+query, element description, stable-layout wait, viewport measurement/resize,
+point inspection, highlight, link extraction, console read, and network failure
+read commands.
+
+The live daemon now keeps bounded in-memory rings for console messages and
+failed network requests. It can derive visible text blocks, element boxes,
+visible links with `href`, and the element stack at a coordinate. Highlighting
+adds a visible overlay that can be captured by the existing screenshot command.
+
+The new DOM-inspection helpers live in `liveSessionInspection.ts`, keeping
+`liveSessionDaemon.ts` below the repository file-length hard limit.
+
+## Delta Analysis
+
+The earlier runtime supported enough structure to click a known target, but it
+did not provide enough state for robust multi-step page diagnosis. Adding these
+commands to the runtime keeps Workloop's tool wrapper simple and lets future
+CLI/operator surfaces reuse the same underlying behavior if needed.
+
+The implementation intentionally does not remove the runtime's existing low
+level eval command because it predates Workloop and is useful for developer
+verification. Workloop still does not expose it as an assistant tool.
+
+## Initiative Assessment
+
+Disciplined initiative: the command contract was extended in one place and the
+Workloop wrapper simply forwards typed commands.
+
+Disciplined initiative: link extraction now includes `href`, which avoids
+forcing callers to infer navigation targets from text and coordinates.
+
+Disciplined initiative: verification exercised a real browser session and
+captured console/network failure events rather than relying only on typechecks.
+
+## Weaknesses In Intent
+
+The runtime still has duplicated accessible-name logic between query and element
+description. That is acceptable for this pass, but a follow-up should extract a
+shared helper if more DOM description behavior is added.
+
+## What We Will Sustain
+
+- Keep Workloop-facing commands typed and narrow.
+- Preserve mouse/ref based actuation as the default action model.
+- Keep diagnostic event buffers bounded so sessions do not grow unbounded in
+  long tasks.
+
+## What We Will Improve
+
+- Add stale-ref generation semantics if real traces show refs reused after
+  navigation.
+- Consider exposing these commands in the developer CLI only if operator
+  workflows need them directly.
+
+## Ownership And Follow-Up
+
+- Owner: Gambit browser runtime maintainers.
+- Action: monitor Workloop browser-use traces for missed controls, stale refs,
+  and whether console/network diagnostics explain page failures.
+- Target date: next browser-runtime polish pass.
+
+## Verification Evidence
+
+- `deno fmt apps/workloop/sidecar/chief_runtime_browser_tools.ts apps/workloop/sidecar/chief_runtime_browser_tools_test.ts apps/workloop/sidecar/chief_runtime_workloop_tools_test.ts packages/browser-runtime/src/liveControl.ts packages/browser-runtime/src/liveSessionDaemon.ts packages/browser-runtime/src/liveSessionInspection.ts`
+  passed as part of the final formatting run with `Checked 10 files`.
+- `deno check --config packages/browser-runtime/deno.json packages/browser-runtime/src/liveSessionDaemon.ts packages/browser-runtime/src/liveSessionInspection.ts`
+  passed.
+- `deno test -A --config packages/browser-runtime/deno.json packages/browser-runtime/src/liveControl.test.ts packages/browser-runtime/src/liveSessionDaemon.test.ts`
+  passed: 8 tests.
+- Live smoke: a headless session named `workloop-js-tools-smoke` validated all
+  new runtime commands on a local `data:` page. The run extracted a visible link
+  with `href` `https://example.com/docs`, captured console event
+  `smoke-console-error`, captured failed request
+  `http://127.0.0.1:9/missing-smoke.png`, and wrote screenshot evidence to
+- Live smoke after the file split repeated the same command sequence and wrote
+  screenshot evidence to
+  `/Users/randallb/code/bolt-foundry/codebot-workspaces/shared/bft-e2e/browser-live-workloop-js-tools-smoke/__latest__/screenshots/2026-04-28T22-23-36-724Z_browser-js-tools-smoke-refactor.png`.
+- `direnv exec . bft precommit` passed after splitting Gambit and Workloop
+  commits: codegen produced no tracked changes, format/lint/typecheck passed,
+  and the full test run reported `1484 passed`, `0 failed`, and `3 ignored`.
diff --git a/reviews/2026-04-28-browser-pointer-targeting-AAR.md b/reviews/2026-04-28-browser-pointer-targeting-AAR.md
@@ -0,0 +1,94 @@
+# Browser Runtime Pointer Targeting AAR
+
+## Context
+
+The shared browser runtime had live-session click support for selectors and raw
+coordinates. Workloop needed a safer higher-level tool surface where agents
+could inspect visible controls, choose a target, move the browser mouse there,
+and click without exposing arbitrary page evaluation.
+
+Because Workloop browser tools depend on the Gambit browser runtime, the runtime
+needed to provide the underlying query/ref/mouse contract before the Workloop
+tool wrapper could expose it.
+
+## Intent
+
+- Purpose: support reliable pointer-based browser targeting from the shared
+  browser runtime.
+- End State: live sessions can query visible interactive elements, return
+  short-lived refs, move the mouse to refs/selectors/coordinates, click via the
+  mouse, and include the tracked cursor in screenshots.
+- Constraints / Tradeoffs: keep arbitrary eval out of the Workloop agent tool
+  surface; use DOM inspection only inside the runtime to derive visible
+  coordinates; preserve existing selector and coordinate callers.
+- Phase (if applicable): Workloop browser-use Phase 2 support with Phase 6
+  screenshot evidence support.
+
+## What Happened
+
+Added a `query` live-session command and ref fields for `mouse-move` and
+`click`. Query uses constrained runtime-internal DOM inspection to find visible
+interactive elements and returns refs such as `e1` for follow-up commands.
+
+Changed live-session click handling to resolve refs and selectors to center
+points, move the Playwright mouse there, and click with `page.mouse.click`.
+Screenshots now temporarily render the tracked cursor position before capture.
+
+Updated the browser CLI so operator workflows can exercise `live query`,
+`live mouse move --ref`, and `live click --ref`.
+
+## Delta Analysis
+
+The previous selector/coordinate API was too low-level for agents that need to
+decide among visually similar controls. Returning short-lived refs gives the
+agent enough structure to choose a target while keeping actuation mouse-based.
+
+The query command intentionally implements a pragmatic accessible-name subset
+rather than a full accessibility tree. That keeps the change small and suitable
+for the current failure mode.
+
+## Initiative Assessment
+
+Disciplined initiative: the runtime preserved existing selector and coordinate
+contracts while adding refs as an optional, safer target handoff.
+
+Disciplined initiative: screenshot cursor rendering was kept as a temporary
+artifact-time overlay rather than a durable page mutation.
+
+## Weaknesses In Intent
+
+No material weaknesses identified for the runtime slice. Future intent should
+say whether a full accessibility-tree source is required.
+
+## What We Will Sustain
+
+- Keep browser actuation mouse-based.
+- Keep query/read support narrow and structured.
+- Keep runtime CLI support aligned with programmatic live-session commands.
+
+## What We Will Improve
+
+- Add stronger stale-ref semantics if query refs are reused after navigation in
+  real traces.
+- Replace the accessible-name subset with a browser accessibility-tree source if
+  query misses become common.
+
+## Ownership And Follow-Up
+
+- Owner: Gambit browser runtime maintainers.
+- Action: monitor Workloop browser traces for query quality and stale-ref
+  frequency.
+- Target date: next browser-runtime polish pass.
+
+## Verification Evidence
+
+- `deno fmt` on the touched browser runtime and mirrored browser files passed.
+- `deno check --config packages/browser-runtime/deno.json packages/browser-runtime/src/liveSessionDaemon.ts packages/browser-runtime/src/browserCli.ts`
+  passed.
+- `deno test -A --config packages/browser-runtime/deno.json packages/browser-runtime/src/liveControl.test.ts packages/browser-runtime/src/liveSessionDaemon.test.ts`
+  passed: 8 tests.
+- Live smoke: a headless session queried a checkbox by role/name, returned ref
+  `e1`, moved to it, captured a screenshot with the visible cursor overlay,
+  clicked it with the mouse path, and verified the checkbox became checked.
+- `direnv exec . bft precommit` passed the full repo gate: codegen no tracked
+  changes, format, lint, typecheck, and 1479 tests passed with 3 ignored.
diff --git a/src/codex_preflight.ts b/src/codex_preflight.ts
@@ -4,6 +4,12 @@ import {
   summarizeCodexAuthBundle,
 } from "./codex_auth.ts";
 import { logCodexAppServerDebug } from "./codex_app_server_debug.ts";
+import {
+  callRuntimeHostService,
+  CODEX_REFRESH_HOST_SERVICE_METHOD,
+  type CodexRefreshHostServiceResult,
+  RUNTIME_HOST_SERVICE_SOCKET_ENV,
+} from "./runtime_host_service.ts";
 
 const CODEX_BIN_ENV = "GAMBIT_CODEX_BIN";
 export const MINIMUM_SUPPORTED_CODEX_CLI_VERSION = "0.121.0";
@@ -82,15 +88,37 @@ async function appServerPreflightRequestResult(input: {
   };
 }> {
   if (input.method === "account/chatgptAuthTokens/refresh") {
-    const refreshed = await refreshCodexChatgptAuthTokens({
-      bundle: input.bundle,
-      previousAccountId: typeof input.params.previousAccountId === "string"
-        ? input.params.previousAccountId
-        : null,
-      reason: typeof input.params.reason === "string" && input.params.reason
+    const previousAccountId = typeof input.params.previousAccountId === "string"
+      ? input.params.previousAccountId
+      : null;
+    const reason =
+      typeof input.params.reason === "string" && input.params.reason
         ? input.params.reason
-        : "account/chatgptAuthTokens/refresh",
-    });
+        : "account/chatgptAuthTokens/refresh";
+    const hostServiceSocket = Deno.env.get(RUNTIME_HOST_SERVICE_SOCKET_ENV)
+      ?.trim();
+    const hostRefreshed = hostServiceSocket
+      ? await callRuntimeHostService({
+        method: CODEX_REFRESH_HOST_SERVICE_METHOD,
+        params: {
+          previousAccountId,
+          reason,
+        },
+      })
+      : null;
+    const refreshed = hostRefreshed
+      ? {
+        ...input.bundle,
+        accessToken: hostRefreshed.accessToken,
+        chatgptAccountId: hostRefreshed.chatgptAccountId,
+        chatgptPlanType: hostRefreshed.chatgptPlanType,
+        lastRefresh: new Date().toISOString(),
+      }
+      : await refreshCodexChatgptAuthTokens({
+        bundle: input.bundle,
+        previousAccountId,
+        reason,
+      });
     return {
       bundle: refreshed,
       result: {
@@ -111,6 +139,17 @@ async function appServerPreflightRequestResult(input: {
   };
 }
 
+async function refreshCodexPreflightViaHost(input: {
+  previousAccountId?: string | null;
+  reason: string;
+}): Promise<CodexRefreshHostServiceResult | null> {
+  if (!Deno.env.get(RUNTIME_HOST_SERVICE_SOCKET_ENV)?.trim()) return null;
+  return await callRuntimeHostService({
+    method: CODEX_REFRESH_HOST_SERVICE_METHOD,
+    params: input,
+  });
+}
+
 async function readLegacyCodexLoginStatus(): Promise<CodexLoginStatus> {
   const codexBin = Deno.env.get(CODEX_BIN_ENV)?.trim() || "codex";
   const codexVersion = await readCodexCliVersion();
@@ -337,26 +376,49 @@ export async function readCodexLoginStatus(): Promise<CodexLoginStatus> {
         capabilities: { experimentalApi: true },
       });
       await writeMessage({ method: "initialized", params: {} });
-      await request("account/login/start", {
-        accessToken: bundle.accessToken,
-        chatgptAccountId: bundle.chatgptAccountId,
-        chatgptPlanType: bundle.chatgptPlanType,
-        type: "chatgptAuthTokens",
-      });
-      const result = await request("account/read", {
-        type: "chatgptAuthTokens",
-      }) as Record<
-        string,
-        unknown
-      >;
+      let loginBundle = bundle;
+      const loginAndRead = async () => {
+        await request("account/login/start", {
+          accessToken: loginBundle.accessToken,
+          chatgptAccountId: loginBundle.chatgptAccountId,
+          chatgptPlanType: loginBundle.chatgptPlanType,
+          type: "chatgptAuthTokens",
+        });
+        return await request("account/read", {
+          type: "chatgptAuthTokens",
+        }) as Record<string, unknown>;
+      };
+      let result = await loginAndRead();
       const account = asRecord(result.account);
-      const requiresOpenaiAuth = result.requiresOpenaiAuth === true;
-      const confirmedAccountId = typeof account.id === "string"
+      let requiresOpenaiAuth = result.requiresOpenaiAuth === true;
+      let confirmedAccountId = typeof account.id === "string"
         ? account.id.trim()
         : "";
-      const planType = typeof account.planType === "string"
-        ? account.planType
-        : bundle.chatgptPlanType;
+      if (requiresOpenaiAuth || !confirmedAccountId) {
+        const refreshed = await refreshCodexPreflightViaHost({
+          previousAccountId: confirmedAccountId || bundle.chatgptAccountId,
+          reason: "codex-preflight-account-read-stale",
+        });
+        if (refreshed) {
+          loginBundle = {
+            ...loginBundle,
+            accessToken: refreshed.accessToken,
+            chatgptAccountId: refreshed.chatgptAccountId,
+            chatgptPlanType: refreshed.chatgptPlanType,
+            lastRefresh: new Date().toISOString(),
+          };
+          result = await loginAndRead();
+          const retryAccount = asRecord(result.account);
+          requiresOpenaiAuth = result.requiresOpenaiAuth === true;
+          confirmedAccountId = typeof retryAccount.id === "string"
+            ? retryAccount.id.trim()
+            : "";
+        }
+      }
+      const finalAccount = asRecord(result.account);
+      const planType = typeof finalAccount.planType === "string"
+        ? finalAccount.planType
+        : loginBundle.chatgptPlanType;
       const hasConfirmedAccountId = confirmedAccountId.length > 0;
       return {
         codexLoggedIn: hasConfirmedAccountId,