Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 4 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -11,6 +11,10 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0

- feat: `mm` sees and acts inside iframes on the browser, including cross-origin ones such as the candlestick chart. A page snapshot (`describe_screen`, `accessibility_snapshot` without `rootSelector`, observations) lists the content of up to 5 visible iframes after the page, with refs namespaced by frame (`f1e1`, `f2e1`, ...) and a `path` ending in `iframe:<title>`; these refs act inside their iframe. Each iframe element also gets a ref, usable as `frame`. `describe_screen`, `click`, `type`, `hover`, `wait_for`, `get_text`, `inspect` and `screenshot` take `frame` (`--frame <iframe>`) to work inside one iframe, the way Playwright's `frameLocator` does. `mm screenshot` adds `--selector` ([#99](https://github.com/MetaMask/client-mcp-core/pull/99))

### Fixed

- fix: `mm click` on a visually hidden checkbox or radio (such as the input of a toggle) clicks its label instead of waiting out the whole timeout

## [0.14.0]

### Added
Expand Down
2 changes: 1 addition & 1 deletion README.md
Original file line number Diff line number Diff line change
Expand Up @@ -399,7 +399,7 @@ The daemon routes `POST /tool/:name` requests through the registry, applies Zod
| `launch` | Launches a new browser session with the configured extension. Supports state modes (`default`, `onboarding`, `custom`), fixture presets, tag metadata, and optional contract seeding on start. |
| `cleanup` | Tears down the active browser session and cleans up all resources (browser, services, fixtures). |
| **Interaction** | |
| `click` | Clicks an element identified by a11y ref, test ID, or CSS selector. Waits for the element to be visible before clicking. Supports `within` to scope the target inside a parent element, and `frame` (browser) to target inside an iframe. |
| `click` | Clicks an element identified by a11y ref, test ID, or CSS selector. Waits for the element to be visible before clicking. Supports `within` to scope the target inside a parent element, and `frame` (browser) to target inside an iframe. Browser: a visually hidden (at most 1×1 pixel) checkbox or radio with one label is toggled by clicking its label, and the result has `clickedLabel: true`. |
| `type` | Types text into an input element identified by a11y ref, test ID, or CSS selector. Clears the field first, then sets the new value (uses Playwright's `fill()`). Supports `within` and `frame` scoping. On the browser, a read-only or disabled element fails at once with `MM_ELEMENT_NOT_EDITABLE` instead of waiting for the timeout. |
| `wait_for` | Waits for an element to become visible on the page within a configurable timeout. Supports `within` to scope the target inside a parent element, and `frame` (browser) to target inside an iframe. |
| `scroll_to_element` | Scrolls until an element (a11y ref, test ID, or selector) is visible. Browser: waits for the element (`timeoutMs`, default 15 000), then scrolls it into view, also inside a scrollable container; supports `within`; ignores `direction` and `maxAttempts`. Mobile: swipes until the target is inside the viewport, not only in the UI tree, and returns `swipes`, `inViewport` and `bounds`; `direction` is the content direction (`down` reveals content below), default `auto` picks it from the element's position; `maxAttempts` = maximum swipes (default 10); fails with `MM_TARGET_NOT_VISIBLE` (found, not in view) or `MM_TARGET_NOT_FOUND`; `within` is rejected. |
Expand Down
2 changes: 2 additions & 0 deletions SKILL.md
Original file line number Diff line number Diff line change
Expand Up @@ -333,6 +333,8 @@ Use `--within` to scope the target inside a parent element. Values use the forma

If the page closes after clicking (e.g., confirmation popup), the response includes `pageClosedAfterClick: true` — this is normal, not an error.

**Browser hidden checkbox or radio:** toggles often keep the real `<input type="checkbox">` (and its test id) at 1×1 pixel under decorative elements, for example `mm click --testid basic-functionality-toggle`. When the target is such an input and it has one label (a wrapping `<label>` or `<label for="<id>">`), the label is clicked instead, which toggles the input, and the response includes `clickedLabel: true`. A hidden input without a label is clicked as before.

**Mobile:** before tapping, iOS hit-tests the tap point. When another element is on top of the target (the keyboard, a sheet, an overlay), nothing is tapped and the click fails with `MM_TARGET_OBSCURED`; the message names the covering element. A target outside the screen fails with `MM_TARGET_NOT_VISIBLE` (run `mm scroll-to-element` first). `--dismiss-keyboard` closes the keyboard when it is the cover and taps once more (the result has `keyboardDismissed: true`). With an observation (not `--observe none`) the result has `screenChanged`: `false` means the screen looks the same as before the click; that is normal for some taps (the open tab) but a hint for a button that should navigate.

```
Expand Down
84 changes: 84 additions & 0 deletions src/platform/playwright-driver.test.ts
Original file line number Diff line number Diff line change
Expand Up @@ -4,6 +4,7 @@ import { PlaywrightPlatformDriver } from './playwright-driver.js';
import type { IPlatformDriver } from './types.js';
import { createMockSessionManager } from '../tools/test-utils/mock-factories.js';
import * as discoveryModule from '../tools/utils/discovery.js';
import * as hiddenControlModule from '../tools/utils/hidden-control.js';
import * as inspectModule from '../tools/utils/inspect.js';

function createMockLocator() {
Expand All @@ -15,6 +16,9 @@ function createMockLocator() {
textContent: vi.fn().mockResolvedValue('Hello World'),
isEditable: vi.fn().mockResolvedValue(true),
isDisabled: vi.fn().mockResolvedValue(false),
boundingBox: vi
.fn()
.mockResolvedValue({ x: 0, y: 0, width: 100, height: 40 }),
scrollIntoViewIfNeeded: vi.fn().mockResolvedValue(undefined),
};
}
Expand Down Expand Up @@ -193,6 +197,86 @@ describe('PlaywrightPlatformDriver', () => {
).rejects.toThrowError('visibility wait consumed entire budget');
expect(locator.click).not.toHaveBeenCalled();
});

it('clicks the label of a visually hidden checkbox', async () => {
const locator = createMockLocator();
const label = createMockLocator();
vi.spyOn(discoveryModule, 'waitForTarget').mockResolvedValue(
locator as any,
);
const findSpy = vi
.spyOn(hiddenControlModule, 'findLabelForHiddenControl')
.mockResolvedValue(label as any);

const driver = createDriver();
const result = await driver.click(
'testId',
'basic-functionality-toggle',
new Map(),
5000,
);

expect(findSpy).toHaveBeenCalledWith(locator, expect.any(Number));
expect(label.click).toHaveBeenCalledOnce();
expect(locator.click).not.toHaveBeenCalled();
expect(result).toStrictEqual({
clicked: true,
target: 'testId:basic-functionality-toggle',
clickedLabel: true,
});
});

it('reports the label click when the page closes during it', async () => {
const label = createMockLocator();
label.click.mockRejectedValue(
new Error('Target page, context or browser has been closed'),
);
vi.spyOn(discoveryModule, 'waitForTarget').mockResolvedValue(
createMockLocator() as any,
);
vi.spyOn(
hiddenControlModule,
'findLabelForHiddenControl',
).mockResolvedValue(label as any);

const driver = createDriver();
const result = await driver.click('testId', 'toggle', new Map(), 5000);

expect(result).toStrictEqual({
clicked: true,
target: 'testId:toggle',
pageClosedAfterClick: true,
clickedLabel: true,
});
});

it('does not click when the signal aborts while finding the label', async () => {
const locator = createMockLocator();
const controller = new AbortController();
vi.spyOn(discoveryModule, 'waitForTarget').mockResolvedValue(
locator as any,
);
vi.spyOn(
hiddenControlModule,
'findLabelForHiddenControl',
).mockImplementation(async () => {
controller.abort(new Error('Tool execution timed out'));
return undefined;
});

const driver = createDriver();
await expect(
driver.click(
'testId',
'toggle',
new Map(),
5000,
undefined,
controller.signal,
),
).rejects.toThrowError('Tool execution timed out');
expect(locator.click).not.toHaveBeenCalled();
});
});

describe('abort during the target wait', () => {
Expand Down
8 changes: 7 additions & 1 deletion src/platform/playwright-driver.ts
Original file line number Diff line number Diff line change
Expand Up @@ -38,6 +38,7 @@ import {
collectTrimmedA11ySnapshot,
waitForTarget,
} from '../tools/utils/discovery.js';
import { findLabelForHiddenControl } from '../tools/utils/hidden-control.js';
import { inspectLocator } from '../tools/utils/inspect.js';

/**
Expand Down Expand Up @@ -116,16 +117,20 @@ export class PlaywrightPlatformDriver implements IPlatformDriver {
);
}

const label = await findLabelForHiddenControl(locator, remaining);
const labelResult = label ? { clickedLabel: true } : {};

// The request may have timed out while we waited for the target.
signal?.throwIfAborted();
try {
await locator.click({ timeout: remaining });
await (label ?? locator).click({ timeout: remaining });
} catch (error) {
if (isPageClosedError(error)) {
return {
clicked: true,
target: `${targetType}:${targetValue}`,
pageClosedAfterClick: true,
...labelResult,
};
}
throw error;
Expand All @@ -134,6 +139,7 @@ export class PlaywrightPlatformDriver implements IPlatformDriver {
return {
clicked: true,
target: `${targetType}:${targetValue}`,
...labelResult,
};
}

Expand Down
5 changes: 5 additions & 0 deletions src/platform/types.ts
Original file line number Diff line number Diff line change
Expand Up @@ -28,6 +28,11 @@ export type ClickActionResult = {
pageClosedAfterClick?: boolean;
/** Mobile: the keyboard covered the target and was dismissed first. */
keyboardDismissed?: boolean;
/**
* Browser: the target was a visually hidden checkbox or radio, so its label
* was clicked instead.
*/
clickedLabel?: boolean;
};

export type ClickOptions = {
Expand Down
3 changes: 3 additions & 0 deletions src/tools/interaction.test.ts
Original file line number Diff line number Diff line change
Expand Up @@ -34,6 +34,9 @@ function createMockLocator() {
textContent: vi.fn().mockResolvedValue('Hello World'),
isEditable: vi.fn().mockResolvedValue(true),
isDisabled: vi.fn().mockResolvedValue(false),
boundingBox: vi
.fn()
.mockResolvedValue({ x: 0, y: 0, width: 100, height: 40 }),
};
}

Expand Down
5 changes: 5 additions & 0 deletions src/tools/types/tool-outputs.ts
Original file line number Diff line number Diff line change
Expand Up @@ -91,6 +91,11 @@ export type ClickResult = {
pageClosedAfterClick?: boolean;
/** Mobile: the keyboard covered the target and was dismissed first. */
keyboardDismissed?: boolean;
/**
* Browser: the target was a visually hidden checkbox or radio, so its label
* was clicked instead.
*/
clickedLabel?: boolean;
/**
* Mobile, with an observation: whether the screen after the click differs
* from the last screen the agent received. `false` is a hint (many taps
Expand Down
Loading
Loading