Skip to content
Closed
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
12 changes: 12 additions & 0 deletions TESTING_COVERAGE.md
Original file line number Diff line number Diff line change
Expand Up @@ -2284,3 +2284,15 @@ deployed app's base64 JSON format with an `atomic:agent:` subject through the
same `Agent.fromSecret` parser used by the local welcome form. It verifies
the identity and public key. Browser sign-in and data recovery are separate
flows.

## Hidden-tab liveness and presence after reconnect (#1800)

`browser/lib/src/liveness.test.ts` replays the WS liveness timer at a hidden
tab's once-a-minute cadence and checks that a socket whose probes are answered
is never closed, while a dead one still closes one tick after its probe times
out. It does not drive a real browser's throttling.
`server/tests/it/drive_presence.rs`
(`presence_sent_right_after_subscribe_is_delivered`) sends a presence update
right behind `PRESENCE_SUBSCRIBE`, as every reconnect does, and checks it
reaches the other subscriber without a retry, and that an update held for a
refused subscribe is dropped.
101 changes: 90 additions & 11 deletions browser/lib/src/liveness.test.ts
Original file line number Diff line number Diff line change
@@ -1,8 +1,9 @@
import { describe, it } from 'vitest';
import {
livenessAction,
LIVENESS_DEADLINE_MS,
LIVENESS_CHECK_MS,
LIVENESS_IDLE_MS,
LIVENESS_PROBE_TIMEOUT_MS,
} from './liveness.js';

/**
Expand All @@ -12,25 +13,103 @@ import {
*/
describe('livenessAction', () => {
it('does nothing while frames keep arriving', ({ expect }) => {
expect(livenessAction(0, false)).toBe('none');
expect(livenessAction(LIVENESS_IDLE_MS - 1, false)).toBe('none');
expect(livenessAction(0, undefined)).toBe('none');
expect(livenessAction(LIVENESS_IDLE_MS - 1, undefined)).toBe('none');
});

it('probes once past the idle threshold', ({ expect }) => {
expect(livenessAction(LIVENESS_IDLE_MS, false)).toBe('probe');
expect(livenessAction(LIVENESS_IDLE_MS, undefined)).toBe('probe');
});

it('does not probe twice while one is outstanding', ({ expect }) => {
expect(livenessAction(LIVENESS_IDLE_MS, true)).toBe('none');
expect(livenessAction(LIVENESS_DEADLINE_MS - 1, true)).toBe('none');
expect(livenessAction(LIVENESS_IDLE_MS, 0)).toBe('none');
expect(
livenessAction(
LIVENESS_IDLE_MS + LIVENESS_PROBE_TIMEOUT_MS - 1,
LIVENESS_PROBE_TIMEOUT_MS - 1,
),
).toBe('none');
});

it('closes once the deadline passes, probe or not', ({ expect }) => {
expect(livenessAction(LIVENESS_DEADLINE_MS, true)).toBe('close');
expect(livenessAction(LIVENESS_DEADLINE_MS, false)).toBe('close');
it('closes once a probe has gone unanswered past its timeout', ({
expect,
}) => {
expect(
livenessAction(
LIVENESS_IDLE_MS + LIVENESS_PROBE_TIMEOUT_MS,
LIVENESS_PROBE_TIMEOUT_MS,
),
).toBe('close');
});

it('keeps the deadline after the idle threshold', ({ expect }) => {
expect(LIVENESS_DEADLINE_MS).toBeGreaterThan(LIVENESS_IDLE_MS);
it('never closes on silence alone, however long', ({ expect }) => {
// A hidden tab's timer can first run minutes after the last frame.
// Without a probe out, the server has not been asked yet.
expect(livenessAction(10 * 60_000, undefined)).toBe('probe');
});
});

/**
* #1800: in a hidden tab Chrome batches timers to once a minute, so the 5 s
* liveness check ran every 60 s. Its first look at a quiet socket saw 60 s
* of silence, past the old 45 s deadline, and closed a healthy connection
* without ever sending the probe ("no frame from the server for 45000ms
* (probe unanswered)"). The app then showed "Working offline".
*
* Replays the WSClient timer loop against a server that answers every probe
* within a round trip, at the throttled cadence and at the normal one.
*/
describe('liveness under timer throttling', () => {
function simulate(tickMs: number, durationMs: number, rttMs: number) {
let now = 0;
let lastFrameAt = 0;
let probeSentAt: number | undefined;
let echoAt: number | undefined;
const actions: string[] = [];

for (now = tickMs; now <= durationMs; now += tickMs) {
// Socket events are not throttled: an echo lands when it arrives.
if (echoAt !== undefined && echoAt <= now) {
lastFrameAt = echoAt;
probeSentAt = undefined;
echoAt = undefined;
}

const action = livenessAction(
now - lastFrameAt,
probeSentAt === undefined ? undefined : now - probeSentAt,
);
actions.push(action);

if (action === 'probe') {
probeSentAt = now;
echoAt = now + rttMs;
} else if (action === 'close') {
break;
}
}

return actions;
}

it('keeps a healthy socket open when timers fire once a minute', ({
expect,
}) => {
const actions = simulate(60_000, 30 * 60_000, 50);
expect(actions).not.toContain('close');
expect(actions).toContain('probe');
});

it('keeps a healthy socket open at the normal cadence', ({ expect }) => {
const actions = simulate(LIVENESS_CHECK_MS, 30 * 60_000, 50);
expect(actions).not.toContain('close');
});

it('still closes a dead socket, one tick after the probe times out', ({
expect,
}) => {
// The echo never comes.
const actions = simulate(60_000, 30 * 60_000, Infinity);
expect(actions).toEqual(['probe', 'close']);
});
});
31 changes: 21 additions & 10 deletions browser/lib/src/liveness.ts
Original file line number Diff line number Diff line change
Expand Up @@ -4,31 +4,42 @@
* alone whether to probe (`KEEPALIVE`) or give the socket up. The decision
* is a pure function of two numbers, kept apart from the socket so it can
* be tested without one.
*
* Silence alone never closes the socket; only an unanswered probe does.
* A hidden tab's timers are throttled (Chrome batches them to once a minute),
* so the check can first run long after the socket went quiet. Closing on
* silence then dropped a healthy connection without ever asking the server,
* every minute the tab stayed hidden (#1800).
*/

/** After this long without any inbound frame, send a `KEEPALIVE` probe. */
export const LIVENESS_IDLE_MS = 20_000;
/** After this long without any inbound frame, the socket is presumed dead
* and closed so the reconnect loop takes over. Comfortably more than the
* idle threshold plus a round trip, so one slow echo is not a disconnect. */
export const LIVENESS_DEADLINE_MS = 45_000;
/** How long a probe may go unanswered before the socket is presumed dead
* and closed so the reconnect loop takes over. Measured from when the probe
* was sent, not from the last frame, so a late timer cannot skip the probe.
* Comfortably more than a round trip, so one slow echo is not a disconnect. */
export const LIVENESS_PROBE_TIMEOUT_MS = 25_000;
/** How often the liveness timer looks. */
export const LIVENESS_CHECK_MS = 5_000;

export type LivenessAction = 'none' | 'probe' | 'close';

/**
* What the liveness timer should do given how long the socket has been
* silent and whether a probe is already outstanding. Pure, so it is testable
* without a socket: `probe` once past the idle threshold, `close` once past
* the deadline, otherwise nothing.
* silent and how long the outstanding probe has gone unanswered (`undefined`
* when none is outstanding). Pure, so it is testable without a socket:
* `probe` once past the idle threshold, `close` once a probe has waited past
* its timeout, otherwise nothing.
*/
export function livenessAction(
idleMs: number,
probeSent: boolean,
probeAgeMs: number | undefined,
): LivenessAction {
if (idleMs >= LIVENESS_DEADLINE_MS) return 'close';
if (idleMs >= LIVENESS_IDLE_MS && !probeSent) return 'probe';
if (probeAgeMs !== undefined) {
return probeAgeMs >= LIVENESS_PROBE_TIMEOUT_MS ? 'close' : 'none';
}

if (idleMs >= LIVENESS_IDLE_MS) return 'probe';

return 'none';
}
24 changes: 13 additions & 11 deletions browser/lib/src/websockets.ts
Original file line number Diff line number Diff line change
Expand Up @@ -72,7 +72,7 @@ import { hexToBytes } from './value.js';
import {
livenessAction,
LIVENESS_CHECK_MS,
LIVENESS_DEADLINE_MS,
LIVENESS_PROBE_TIMEOUT_MS,
} from './liveness.js';
import { perfMark, perfSpan } from './perf-trace.js';

Expand Down Expand Up @@ -248,13 +248,14 @@ export class WSClient {
private _rbsrItemsQueue: Array<(items: Item[]) => void> = [];
/** What the server said it speaks, from its `AUTH_OK` payload. */
private _serverCaps: string[] = [];
/** Liveness: when the last inbound frame arrived, whether a `KEEPALIVE`
* probe is outstanding, and the timer that checks both. A browser cannot
/** Liveness: when the last inbound frame arrived, when the outstanding
* `KEEPALIVE` probe was sent (`undefined` when none is), and the timer
* that checks both. A browser cannot
* observe the server's protocol-level pings, so without this a socket
* the network silently dropped stays "connected" until the next write
* fails — every subscription push in between is lost. */
private _lastFrameAt = 0;
private _probeSent = false;
private _probeSentAt: number | undefined;
private _livenessTimer: ReturnType<typeof setInterval> | undefined;

/** When true, all WS frames are logged to the console in human-readable form. */
Expand Down Expand Up @@ -1105,7 +1106,7 @@ export class WSClient {
private handleMessage(ev: MessageEvent) {
// Any inbound frame proves the socket is alive.
this._lastFrameAt = Date.now();
this._probeSent = false;
this._probeSentAt = undefined;

if (ev.data instanceof ArrayBuffer) {
this.handleBinary(new Uint8Array(ev.data));
Expand Down Expand Up @@ -1675,28 +1676,29 @@ export class WSClient {
* Start the liveness timer for the current socket. Only probes a server
* that advertised `keepalive` (older servers do not echo the frame, and a
* probe that is never answered would make an idle socket look dead every
* `LIVENESS_DEADLINE_MS`). Anonymous sessions never authenticate, so they
* `LIVENESS_PROBE_TIMEOUT_MS`). Anonymous sessions never authenticate, so they
* learn no capabilities and keep the pre-2026-09 reactive behaviour.
*/
private startLiveness() {
this.stopLiveness();
this._lastFrameAt = Date.now();
this._probeSent = false;
this._probeSentAt = undefined;
this._livenessTimer = setInterval(() => {
if (this.readyState !== WebSocket.OPEN) return;
if (!this._serverCaps.includes('keepalive')) return;

const now = Date.now();
const action = livenessAction(
Date.now() - this._lastFrameAt,
this._probeSent,
now - this._lastFrameAt,
this._probeSentAt === undefined ? undefined : now - this._probeSentAt,
);

if (action === 'probe') {
this._probeSent = true;
this._probeSentAt = now;
this.sendBinary(encodeKeepalive());
} else if (action === 'close') {
console.warn(
`[WS] no frame from the server for ${LIVENESS_DEADLINE_MS}ms (probe unanswered); closing so the reconnect loop takes over`,
`[WS] keepalive probe unanswered for ${LIVENESS_PROBE_TIMEOUT_MS}ms; closing so the reconnect loop takes over`,
);
this.stopLiveness();
// Closing fires the `close` handler, which reports disconnection,
Expand Down
8 changes: 5 additions & 3 deletions docs/src/websockets.md
Original file line number Diff line number Diff line change
Expand Up @@ -818,9 +818,11 @@ saturated renderer can stall pong delivery for seconds.

A browser cannot observe protocol-level pings, so it has its own probe. When
the server advertised `keepalive`, the client checks every 5 s: after 20 s
with no inbound frame it sends one `KEEPALIVE (0x41)`; if nothing has arrived
after 45 s it closes the socket and lets the reconnect loop take over. Any
inbound frame resets both. The server **echoes** `KEEPALIVE` verbatim, which
with no inbound frame it sends one `KEEPALIVE (0x41)`; if that probe is still
unanswered 25 s after it was sent, it closes the socket and lets the reconnect
loop take over. Any inbound frame resets both. Silence alone never closes the
socket: a hidden tab's timers can fire a minute apart, and the first check
after such a gap must ask the server rather than give up on it. The server **echoes** `KEEPALIVE` verbatim, which
is the whole point of it. Against a server that did not advertise
`keepalive` the client does not probe, since an unanswered probe would make
every idle socket look dead.
Expand Down
Loading