Skip to content

feat: add mm network-log, a passive request and console log with service-worker traffic - #106

Closed
cryptotavares wants to merge 9 commits into
mainfrom
cryptotavares/network-log
Closed

cryptotavares wants to merge 9 commits into
mainfrom
cryptotavares/network-log

Conversation

@cryptotavares

Copy link
Copy Markdown
Collaborator

Stacked on #93. The base is cryptotavares/mock-network-request-bodies, so this diff shows only the network log. It reuses #93's URL filter, secret field list and service-worker check. After #93 merges, this PR will be retargeted to main.

Why

Extension release validation needs, per flow: the slowest endpoints, HTTP and network errors, JSON-RPC methods and console errors. Mobile already gets this from a fetch interceptor. The Extension has nothing like it:

  • Most confirmation traffic (gas, Blockaid security alerts, bridge, intents, Relay, Hyperliquid, RPC) is sent by the MV3 service worker. mm cdp and page-level patches do not reach it, and LavaMoat blocks global patches in production builds.
  • mock-network only sees hosts that a rule routes, and it records no status or timing.

Playwright 1.57+ reports service-worker requests and console messages on the browser context. So a recorder that only listens can see both the pages and the worker.

What

  • New read-only tool network_log and mm network-log. It only listens (request, response, requestfinished, requestfailed, console, weberror). It never routes, blocks or changes a request.
  • Starts by itself after mm launch (browser only), so agents cannot forget to start it. It keeps recording across navigation, tab switches and service-worker restarts.
  • Request record: start time, method, redacted URL, resource type, status, duration (ms), failure text, source (page or service-worker, with page/frame URL) and the JSON-RPC method(s) from the request body (single and batch).
  • Console record: level error only by default, uncaught page errors included. Other levels can be added with configure --console-levels. Text is redacted and cut at 2000 characters, with source and url:line:column.
  • Buffers and cursors:
    • The log keeps the newest 2000 requests and 1000 console messages, with dropped counts.
    • One sequence number is shared by requests and console messages. mark returns a cursor, and --cursor returns only newer records, oldest first, with hasMore and gap.
    • Reads also take --since, --url (same glob/substring rules as feat: record request bodies in mock-network #93), --errors, --limit and --clear.
  • Redaction:
    • Removed: credentials, key-like host labels and path segments (for example Infura ids), and values of sensitive query parameters (feat: record request bodies in mock-network #93's list plus key, token, sig, ...).
    • Kept readable: addresses, hashes and UUIDs.
  • Opt-in JSON-RPC errors: configure --rpc-errors reads JSON-RPC response bodies (up to 64 KiB, 8 at a time, 2 s timeout) and records rpcError. It is off by default.
  • BREAKING: Playwright ^1.57.0. The peer and dev dependency ranges change from ^1.49.0. 1.57 is the first version with service-worker network and console events without an environment flag.

CLI

mm network-log mark                          # {"cursor":0,"summary":{"startedAt":"…",…}}
# …do the flow…
mm network-log requests --cursor 0           # every request after the mark
mm network-log requests --cursor 0 --errors  # failures, HTTP >= 400, JSON-RPC errors
mm network-log console --cursor 0            # console errors after the mark
mm network-log configure --rpc-errors        # also record JSON-RPC error objects

Sample output from the smoke test. The extension's service worker sends a 200, a 404, a JSON-RPC call that reverts, a batch, an aborted request and console.error. --rpc-errors is on.

{"seq":3,"method":"GET","url":"http://127.0.0.1:47123/ok?from=page","type":"fetch","status":200,"ms":146,"source":"page","pageUrl":"chrome-extension://…/trigger.html?tag=flow1"}
{"seq":4,"method":"GET","url":"http://127.0.0.1:47123/ok?tag=flow1&apiKey=[REDACTED]","type":"fetch","status":200,"ms":122,"source":"service-worker"}
{"seq":5,"method":"GET","url":"http://127.0.0.1:47123/missing?tag=flow1","type":"fetch","status":404,"ms":0,"source":"service-worker"}
{"seq":6,"method":"POST","url":"http://127.0.0.1:47123/v3/[REDACTED]","type":"fetch","status":200,"ms":42,"rpcMethod":"eth_call","rpcError":{"code":-32000,"message":"execution reverted"},"source":"service-worker"}
{"seq":9,"method":"GET","url":"http://127.0.0.1:47123/slow","type":"fetch","ms":57,"failure":"net::ERR_ABORTED","source":"service-worker"}
{"seq":10,"method":"POST","url":"http://127.0.0.1:47123/v3/[REDACTED]","type":"fetch","status":200,"ms":42,"rpcMethod":["eth_chainId","eth_blockNumber"],"rpcBatchSize":2,"source":"service-worker"}
{"seq":1,"level":"error","text":"page error for flow1","source":"page"}
{"seq":2,"level":"error","text":"Uncaught TypeError: uncaught in page flow1","source":"page","uncaught":true}
{"seq":7,"level":"error","text":"bg flow flow1 failed: https://mainnet.infura.io/v3/[REDACTED]","source":"service-worker"}

(t and some URL fields are left out here. The worker's console.warn and console.log are not recorded with the default level.)

Cursors start at 0 again after a relaunch. mark and every read return summary.startedAt. When it changes, the caller reads from cursor 0 and records a capture gap.

Testing

  • Unit tests: 55 tests in network-log.test.ts with mocked browser-context events. They cover timing, sources, failures, JSON-RPC single/batch/error capture and its limits, console levels, redaction, buffer limits, cursors, since, clear, and startedAt across a relaunch. There are also tests for the CLI, schemas, registry, platform gating and launch.

  • Checks: yarn lint, yarn build and CI=1 yarn test pass (60 files, 2227 tests, coverage floors met; network-log.ts has 100% lines).

  • Real smoke test through the daemon and mm CLI built from this branch, with a tiny MV3 extension and a local HTTP server. The output above comes from it.

  • Playwright version matrix (same extension, service-worker requests / console messages recorded):

    Playwright SW requests SW console
    1.56.1 0 0
    1.56.1 + PW_EXPERIMENTAL_SERVICE_WORKER_NETWORK_EVENTS=1 5 0
    1.57.0 5 2
    1.58.1 (this repo) 5 2
    1.59.1 (metamask-extension) 5 2

Known limits

  • Service-worker restart is only unit-tested. I could not force a real restart in the harness. ServiceWorker.stopWorker, Target.closeTarget and 90 s idle did not stop the worker, and chrome.runtime.reload() disables an unpacked extension. The listeners are on the browser context, so a new worker should report to the same recorder.
  • Traffic during mm launch is not recorded (for example self-onboarding). The log starts after launch returns.
  • No request or response bodies are stored. Use mm mock-network with record (feat: record request bodies in mock-network #93) for bodies.

An agent that validates an Extension PR needs to check which analytics
events the Extension sent to Segment. mock-network only recorded
method, URL and status, so the agent could not see the batch payload.

- A rule can set `record: { body?, headers?, maxBodyBytes? }`. It is off
  by default and applies only to requests that the rule matches.
- Bodies: JSON is parsed. Other UTF-8 text is kept as text. Binary
  bodies record only their type and size. Bodies longer than
  maxBodyBytes (default 64 KiB, max 512 KiB) are cut on a UTF-8
  boundary, kept as text and marked `truncated`.
- Redaction: secret body fields (writeKey, password, apiKey,
  accessToken, privateKey, mnemonic, ...) and sensitive headers
  (authorization, cookie, and names with token/secret/key/auth/
  session/password) are replaced with [REDACTED].
- Each record has `serviceWorker: true` when a service worker sent the
  request.
- `mm mock-network requests` gains `--bodies` and
  `--url <glob-or-text>`. The default output is unchanged.
network_log reuses the body-field secret list and the service-worker
check from mock-network instead of copying them.
Release validation needs per-flow network and console data from the
Extension, and most confirmation traffic (gas, security alerts, bridge,
RPC) is sent by the MV3 service worker, which a page-level fetch patch
cannot see. Playwright 1.57+ reports service-worker requests and
console messages on the browser context, so a listener-only recorder
covers both the pages and the worker.

- The recorder listens to request/response/requestfinished/
  requestfailed/console/weberror on the browser context. It never
  routes a request. launch starts it for browser sessions, so the log
  covers every flow without a start call; attachNetworkLog is exported
  so a session manager can start it earlier.
- Requests: start time, method, redacted URL, resource type, status,
  duration (Playwright timing), failure text, source (page or
  service-worker, with page/frame URL) and JSON-RPC method(s) from the
  request body (single and batch). URL redaction removes credentials,
  key-like host labels and path segments (Infura ids) and sensitive
  query values; addresses, hashes and UUIDs stay readable.
- Console: error and warning by default (configurable), text redacted
  and cut at 2000 characters, source, location; uncaught page errors
  are recorded as errors.
- Bounded ring buffers (2000 requests, 1000 console messages) with
  dropped counts. Records share one sequence number, so a cursor from
  mark (or the last read) returns only newer records, oldest first,
  with hasMore and a gap flag when records after the cursor were
  dropped. Reads also take since, url, errors, limit and clear.
- configure rpcErrors (off by default) reads JSON-RPC response bodies
  up to 64 KiB, at most 8 at a time with a 2 s limit, and records the
  error object.
- Browser only, readonly category (no observation).
mm network-log requests|console [--cursor <n>] [--since <iso|epoch-ms>]
[--limit <n>] [--url <glob-or-text>] [--errors] [--clear],
mm network-log mark|clear and
mm network-log configure [--console-levels <list>] [--rpc-errors|--no-rpc-errors].
A flag without a valid value fails with a usage error before any
request is sent.
Playwright 1.57 is the first version that reports the network and
console events of an MV3 extension service worker without an
environment flag, and network-log depends on them. The playwright and
@playwright/test peer and dev dependency ranges are now ^1.57.0. The
lockfile still resolves 1.58.1.
Warnings from the Extension are noisy and not useful in a release
report. The default console level is now error (uncaught page errors
included); configure --console-levels still adds warning or others.
A test checks that mark and every read return summary.startedAt, which
the release skill uses to detect a relaunch.
Base automatically changed from cryptotavares/mock-network-request-bodies to main November 18, 2026 09:55
@cryptotavares

Copy link
Copy Markdown
Collaborator Author

Closing in favour of the mm CLI v2 plan (P2-15 events buffer, P2-16 mm logs, P2-17 mm network), built on the persistent CDP sessions from #90 instead of Playwright's context events. That gives one capture path and one record format for Extension and Mobile, and avoids the Playwright 1.57 minimum. The buffer/cursor design, URL and console redaction, JSON-RPC parsing, CLI read options and the MV3 smoke-test setup from this PR will be reused there. Branch cryptotavares/network-log is kept for reference.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant