Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
23 commits
Select commit Hold shift + click to select a range
2386ca3
fix: make agent-device replays fail safely
BartekObudzinski Jul 31, 2026
912eb89
Merge remote-tracking branch 'upstream/main' into fix/agent-device-he…
BartekObudzinski Aug 7, 2026
bcdd816
fix: resolve agent-device tab selectors on iOS
BartekObudzinski Aug 7, 2026
1027fd5
Revert "fix: resolve agent-device tab selectors on iOS"
BartekObudzinski Aug 7, 2026
ac9f893
chore: drop flow validation script in favour of authoring rules
BartekObudzinski Aug 10, 2026
3e87b81
docs: record the replay timeout hang through agent-device 0.20.6
BartekObudzinski Aug 10, 2026
2381579
docs: record that --settle does not replay in .ad flows
BartekObudzinski Aug 10, 2026
e31f36f
Revert "docs: record that --settle does not replay in .ad flows"
BartekObudzinski Aug 10, 2026
fe1ac9f
refactor: move Sentry span flows into measure-telemetry-span skill
BartekObudzinski Aug 20, 2026
a204a37
Merge remote-tracking branch 'origin/main' into move-sentry-scenarios…
BartekObudzinski Aug 20, 2026
e783cfb
Merge branch 'main' of https://github.com/Expensify/App into fix/agen…
BartekObudzinski Aug 20, 2026
6e212f1
fix: verify React Native overlay dismissal from the snapshot
BartekObudzinski Aug 20, 2026
a10f6eb
fix: fail measurement runs that emit no span
BartekObudzinski Aug 20, 2026
9718055
Merge branch 'main' of https://github.com/Expensify/App into fix/agen…
BartekObudzinski Aug 20, 2026
656a480
chore: drop unrelated fork changes from this branch
BartekObudzinski Aug 20, 2026
50aeee2
fix: silence shellcheck on the literal interpolation marker
BartekObudzinski Aug 20, 2026
3d43202
docs: record that the tab flows are Android only
BartekObudzinski Aug 20, 2026
6d69c12
fix: split expense span ownership between submit-expense and create-e…
BartekObudzinski Aug 21, 2026
f1e50c2
docs: remove cross-skill references and stale version note from flows…
BartekObudzinski Aug 21, 2026
e151466
docs: remove sentry references from agent-device skill
BartekObudzinski Aug 26, 2026
1638f72
Merge remote-tracking branch 'upstream/main' into fix/agent-device-he…
BartekObudzinski Aug 27, 2026
57534ba
docs: decouple agent-device flow docs from measurement skill
BartekObudzinski Sep 2, 2026
6b7ab73
fix: restore durable selectors and reset macros in measurement flows
BartekObudzinski Sep 4, 2026
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 4 additions & 2 deletions .claude/skills/agent-device/SKILL.md
Original file line number Diff line number Diff line change
Expand Up @@ -10,7 +10,7 @@ allowed-tools: Bash(agent-device *) Bash(npm root *) Bash(scripts/is-hybrid-app.

These checks evaluate at skill load. If any line shows `FAIL`, stop and surface the fix before running any device command.

`agent-device` version: !`R=0.18.0; V=$(agent-device --version 2>/dev/null); [ -n "$V" ] && [ "$(printf '%s\n%s\n' "$R" "$V" | sort -V | head -1)" = "$R" ] && echo "OK ($V)" || echo "FAIL (need v$R+, got: ${V:-not installed}). Fix: npm install -g agent-device@latest"`
`agent-device` version: !`R=0.20.0; V=$(agent-device --version 2>/dev/null); [ -n "$V" ] && [ "$(printf '%s\n%s\n' "$R" "$V" | sort -V | head -1)" = "$R" ] && echo "OK ($V)" || echo "FAIL (need v$R+, got: ${V:-not installed}). Fix: npm install -g agent-device@latest"`

Bundled CLI skills dir: !`D="$(npm root -g)/agent-device/skills/agent-device"; test -s "$D/SKILL.md" && echo "OK ($D)" || echo "FAIL (missing $D/SKILL.md). Fix: npm install -g agent-device@latest"`

Expand Down Expand Up @@ -121,4 +121,6 @@ Read these files directly for device automation guidance (bootstrap, exploration

## Flows

Repeatable steps (sign-in, onboarding, etc.) are captured as composable `.ad` snippets under [`flows/`](flows/README.md). For interactive usage, propose and run only `flows/macros/` helpers. `flows/tests/` belongs to a separate QA workflow and must not be proposed by this skill; QA/perf runs execute them via `agent-device test <path>`.
This skill owns interactive automation only: reusable setup and navigation macros under [`flows/macros/`](flows/README.md), with platform overrides under `flows/macros/<platform>/`. Propose and run macros through the [Agent decision loop](flows/README.md). `flows/README.md` is also the reference for the `.ad` metadata spec, selector rules, and recording workflow, so read it before authoring any `.ad` file in this repository.

Measurement flows live in the [`measure-telemetry-span`](../measure-telemetry-span/SKILL.md) skill, which owns its own `flows/` and runner. Do not add measurement flows here.
58 changes: 31 additions & 27 deletions .claude/skills/agent-device/flows/README.md
Original file line number Diff line number Diff line change
@@ -1,12 +1,13 @@
# Flows

This file is the reference for every `.ad` file in the repository: the metadata spec, selector rules, recording, and maintenance.

## Directory layout

- `macros/` - reusable helpers for common setup/navigation actions that stop in a navigable state for further interactive work.
- `macros/<platform>/` - platform-specific overrides of a `macros/` flow, for flows whose selectors differ per platform. See [Platform scoping](#platform-scoping).
- `tests/` - critical-scenario scripts for QA/perf verification that assert explicit outcomes (for example Sentry spans) and then stop.
1. `macros/` contains reusable setup and navigation helpers for interactive work. These are the only flows an interactive agent may propose.
2. `macros/<platform>/` contains platform-specific overrides of a `macros/` flow, for flows whose selectors differ per platform. See [Platform scoping](#platform-scoping).

Composable `.ad` snippets - bounded units of work. A flow may span one or multiple screens as long as it represents a coherent, reusable action with clear start (`@pre`) and completion (`@post`) checkpoints. Each flow advertises machine-matchable metadata (`@pre`, `@post`, `@tag`, `@param`) via `# @`-prefixed comment headers, while flow type is derived from location (`flows/macros/` or `flows/tests/`).
`macros/` contains composable `.ad` snippets. The caller owns application startup, authentication, platform selection, session state, and cleanup. A flow may span one or multiple screens as long as it represents a coherent action with clear start (`@pre`) and completion (`@post`) checkpoints.

## Platform scoping

Expand Down Expand Up @@ -34,29 +35,24 @@ Before manually navigating, use this human-in-the-loop loop:
7. `agent-device replay <path> -e KEY=VALUE ...`.
8. If the flow declares `@post`, verify each `@post` with `is exists`. On success, re-enter the loop only if the user's stated goal is not complete; otherwise stop and report completion. On failure, propose peer flow/manual fallback options and ask before continuing. If no `@post` is declared (utility flow), rely on explicit user confirmation or the next snapshot before continuing.

## QA workflow (separate)
## Replay lifecycle

`flows/tests/` is reserved for dedicated QA automation and should not be offered to users as part of the interactive helper loop above. Run these flows with the dedicated test runner:

```bash
agent-device test .claude/skills/agent-device/flows/tests/<name>.ad -e KEY=VALUE ...
```
No flow in this repository owns application lifecycle, so none of them is a self-contained `agent-device test` input. `agent-device test` creates an isolated session per attempt; use it only for scripts that own `context`, `open`, and cleanup. Repository flows expect the caller to open the app and prepare a named session first.

## Metadata header spec

Each flow starts with `# @key value` comment lines. The `.ad` parser treats `#` lines as no-ops, so headers cost nothing at replay time.

| Field | Cardinality | Value |
| -------- | ----------- | ------------------------------------------------------------------------------------------------ |
| `@desc` | 1 | One-line human summary. |
| `@pre` | 1..N | Selector that must resolve in the current snapshot. Multiple lines are ANDed. |
| `@post` | 0..N | Selector expected after replay. Multiple lines are ANDed. Used for chaining + success. |
| `@tag` | 0..N | Free-form category (`auth`, `onboarding`, ...) or scoped (`sentry-<spanName>`). |
| `@param` | 0..N | Runtime input contract: `@param KEY description.` Use with `${KEY}` in flow body. |
| Field | Cardinality | Value |
| -------- | ----------- | -------------------------------------------------------------------------------------- |
| `@desc` | 1 | One-line human summary. |
| `@pre` | 1..N | Selector that must resolve in the current snapshot. Multiple lines are ANDed. |
| `@post` | 0..N | Selector expected after replay. Multiple lines are ANDed. The flow body enforces them. |
| `@param` | 0..N | Runtime input contract: `@param KEY description.` Use with `${KEY}` in flow body. |

Selector syntax matches the body: `id="..."`, `role="..." label="..."`, `text="..."`, `||` for fallbacks.

## Parametrization (`agent-device` v0.13.0+)
## Parametrization

Declare runtime inputs via metadata (`@param`) and reference them in the body with `${VAR}` interpolation. Values are supplied by caller arguments (`-e`) or shell imports (`AD_VAR_*`) - never by in-file `env` directives.

Expand All @@ -78,11 +74,16 @@ agent-device replay <flow>.ad -e EMAIL=other@example.com
## Authoring rules

- **No `open`, no `close`, no `context` header.** Caller owns lifecycle.
- **No fixed `wait` calls.** `fill`/`press` resolve selectors with retry. Only add `wait <selector>` for real post-action blocks.
- **No fixed-duration `wait` calls.** Guard every asynchronous transition with `wait "<selector>" <timeoutMs>`, which polls until the selector resolves. A bare `wait <ms>` only burns wall clock.
- **No command flags in a flow body.** The `.ad` parser only reads flags for `press`, `fill`, `click`, and a few capture commands; for `is`, `wait`, and `find` it treats `--first` and friends as positionals. Disambiguate with a tighter selector instead.
- **Durable selectors.** Prefer `id=...` first, then `role=... label=...`, with `||` fallbacks. Avoid `@eN` refs.
- **A `@pre` must distinguish the start screen from the destination.** A bottom-tab label such as `text="Home"` is rendered on every tab, so asserting it proves nothing. Anchor on a screen-scoped `id=` instead - `id="HomePage"` for Home, `id="BaseSidebarScreen"` for Inbox - otherwise a flow that starts on the wrong screen reports success while the action it measures never happens.
- **Confirm every selector on the platform it is written for.** `snapshot -i` prints display tags, which are not selector values - a node printed as `[text-field]` may only match `role="textbox"`. Check the exit code of `agent-device is visible "<selector>"` before committing a selector, and never carry one across platforms unchecked.
- **Every flow declares `@desc` and `@pre`.** Add `@post` for outcome-bearing flows; utility flows (for example `go-back`) may omit it. Add `@tag` when applicable.
- **Choose directory intentionally.** Put reusable setup/navigation steps in `flows/macros/`; put outcome verification scenarios in `flows/tests/`.
- **Every flow declares `@desc` and `@pre`.** Outcome-bearing flows also declare at least one `@post`. Utility macros (for example `go-back`) may omit `@post`.
- **Every declared `@pre` and `@post` runs as a body assertion.** Use `is exists` or `wait`. Metadata documents the contract but does not execute it.
- **Assertion order decides whether a pass means anything.** Enforce every `@pre` before the first mutation and every `@post` after the last one. An assertion on the wrong side of a mutation lets a flow report success from the screen it started on.
- **No `find <selector> "click"`.** It resolves at runtime and hides which element was hit. Press an exact selector instead.
- **A bare `label=` alternative must carry `hittable=true`.** Without it the match can land on an off-screen or non-interactive node and the press silently does nothing.
- **Keep scope coherent, not artificially tiny.** Flows can span multiple screens when that sequence is the reusable intent (for example "create and submit manual expense").
- **Peers share `@pre` and differ on `@post`.** One flow per narrow outcome is better than a mega-flow with conditional branches.
- **Use `@param` for substituted values.** If a literal is interpolated into the body, declare `# @param KEY description.` and reference it as `${KEY}`.
Expand All @@ -94,21 +95,24 @@ agent-device replay <flow>.ad -e EMAIL=other@example.com
1. Drive the target screen manually.
2. Start a session with `--save-script`:
```bash
agent-device open <app> --save-script .claude/skills/agent-device/flows/<kind>/<name>.ad
agent-device open <app> --save-script .claude/skills/agent-device/flows/macros/<name>.ad
```
3. Perform the steps.
4. `agent-device close` - flushes the `.ad`.
5. Edit the generated file:
- Delete the `context` line, leading `open ... --relaunch`, trailing `close`, and eyeballing `wait`s.
- Move file to `flows/macros/` or `flows/tests/`, then add `@desc`, `@pre`, optional `@post`, optional `@tag`, and any needed `@param` headers.
6. Verify: pre-check from a matching state, replay, post-check.
- Move the file to `flows/macros/`, then add `@desc`, `@pre`, optional `@post`, and any needed `@param` headers.
6. Add executable checks for every declared `@pre` and `@post`.
7. Verify: pre-check from a matching state, replay, post-check.

## Maintenance

Heal selector drift in place:
Heal selector drift by hand. Replay the flow and read the divergence report:

```bash
agent-device replay -u .claude/skills/agent-device/flows/<kind>/<name>.ad
agent-device replay <path-to-flow>.ad --session <name>
```

Re-verify `@pre`/`@post` still hold, then commit. Note: `replay -u` can rewrite interpolated lines to concrete selectors/values; review diffs and restore `${KEY}` placeholders where needed. Keep runtime inputs in `@param` + `-e`/`AD_VAR_*`; do not reintroduce in-file `env` directives.
Every divergence carries ranked selector suggestions. Apply them yourself: `replay --update`/`-u` no longer rewrites the `.ad` file. Editing by hand also keeps `${KEY}` placeholders intact. Re-verify `@pre`/`@post` still hold, then commit. Keep runtime inputs in `@param` + `-e`/`AD_VAR_*`; do not reintroduce in-file `env` directives.

After editing a flow, re-read the authoring rules above and replay it from a matching start state. A green replay plus enforced `@pre`/`@post` is the check.
Original file line number Diff line number Diff line change
Expand Up @@ -3,7 +3,6 @@ context platform=android
# @pre text="What’s your work email?"
# @post text="Home"
# @post role="button" label="Search"
# @tag onboarding
# @param FIRST_NAME First name to enter on onboarding profile step.
# @param LAST_NAME Last name to enter on onboarding profile step.

Expand Down
Original file line number Diff line number Diff line change
@@ -1,8 +1,7 @@
context platform=android
# @desc Send a chat message from inside an already-open chat. Reusable helper for setting up state in other flows. Does not navigate or open a chat - assumes the composer is visible. For the QA scenario that exercises the ManualSendMessageVisible Sentry span, see flows/tests/send-message.ad.
# @desc Send a chat message from inside an already-open chat. Reusable helper for setting up state in other flows. Does not navigate or open a chat - assumes the composer is visible.
# @pre label="Write something..." editable=true
# @post label="Write something..." editable=true
# @tag chat
# @param MESSAGE Message text to send in the currently open chat.

is exists "label=\"Write something...\" editable=true"
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -3,7 +3,6 @@ context platform=android
# @pre id="username"
# @pre role="button" label="Continue"
# @post role="button" label="Join" || role="button" label="Search"
# @tag auth
# @param EMAIL Login email. Use randomized alias format `agent-device-testing+<9digits>@gmail.com` to avoid account flagging.

fill "id=\"username\" || label=\"Phone or email\" editable=true" "${EMAIL}"
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -2,7 +2,6 @@
# @pre text="What’s your work email?"
# @post text="Home"
# @post role="button" label="Search"
# @tag onboarding
# @param FIRST_NAME First name to enter on onboarding profile step.
# @param LAST_NAME Last name to enter on onboarding profile step.

Expand Down
1 change: 0 additions & 1 deletion .claude/skills/agent-device/flows/macros/go-back.ad
Original file line number Diff line number Diff line change
@@ -1,4 +1,3 @@
# @desc Generic back navigation via the HeaderWithBackButton chevron. Works on any screen that mounts HeaderWithBackButton (Create Expense, Search, Report details, settings pages, etc.).
# @pre role="button" label="Back"
# @tag navigation
press "role=\"button\" label=\"Back\""
8 changes: 8 additions & 0 deletions .claude/skills/agent-device/flows/macros/return-to-inbox.ad
Original file line number Diff line number Diff line change
@@ -0,0 +1,8 @@
# @desc Return from a child screen to the Inbox and wait for its sidebar anchor.
# @pre role="button" label="Back"
# @post id="BaseSidebarScreen"
is exists "role=\"button\" label=\"Back\""
press "role=\"button\" label=\"Back\""
wait "role=\"tab\" label=\"Inbox\" || role=\"tab\" label=\"Inbox. Your review is required\" || label=\"Inbox\" hittable=true || label=\"Inbox. Your review is required\" hittable=true" 10000
press "role=\"tab\" label=\"Inbox\" || role=\"tab\" label=\"Inbox. Your review is required\" || label=\"Inbox\" hittable=true || label=\"Inbox. Your review is required\" hittable=true"
wait "id=\"BaseSidebarScreen\"" 10000
3 changes: 1 addition & 2 deletions .claude/skills/agent-device/flows/macros/send-message.ad
Original file line number Diff line number Diff line change
@@ -1,7 +1,6 @@
# @desc Send a chat message from inside an already-open chat. Reusable helper for setting up state in other flows. Does not navigate or open a chat - assumes the composer is visible. For the QA scenario that exercises the ManualSendMessageVisible Sentry span, see flows/tests/send-message.ad.
# @desc Send a chat message from inside an already-open chat. Reusable helper for setting up state in other flows. Does not navigate or open a chat - assumes the composer is visible.
# @pre label="Write something..." editable=true
# @post label="Write something..." editable=true
# @tag chat
# @param MESSAGE Message text to send in the currently open chat.

is exists "label=\"Write something...\" editable=true"
Expand Down
6 changes: 6 additions & 0 deletions .claude/skills/agent-device/flows/macros/switch-to-home.ad
Original file line number Diff line number Diff line change
@@ -0,0 +1,6 @@
# @desc Switch to the Home tab and wait for the Home screen anchor.
# @pre role="tab" label="Home" || label="Home" hittable=true
# @post id="HomePage"
is exists "role=\"tab\" label=\"Home\" || label=\"Home\" hittable=true"
press "role=\"tab\" label=\"Home\" || label=\"Home\" hittable=true"
wait "id=\"HomePage\"" 10000
Original file line number Diff line number Diff line change
Expand Up @@ -3,7 +3,6 @@ context platform=web
# @pre text="What’s your work email?"
# @post text="Home"
# @post role="button" label="Search"
# @tag onboarding
# @param FIRST_NAME First name to enter on onboarding profile step.
# @param LAST_NAME Last name to enter on onboarding profile step.

Expand Down
Original file line number Diff line number Diff line change
@@ -1,8 +1,7 @@
context platform=web
# @desc Send a chat message from inside an already-open chat. Reusable helper for setting up state in other flows. Does not navigate or open a chat - assumes the composer is visible. For the QA scenario that exercises the ManualSendMessageVisible Sentry span, see flows/tests/send-message.ad.
# @desc Send a chat message from inside an already-open chat. Reusable helper for setting up state in other flows. Does not navigate or open a chat - assumes the composer is visible.
# @pre role="textbox" label="Write something..."
# @post role="textbox" label="Write something..."
# @tag chat
# @param MESSAGE Message text to send in the currently open chat.

is exists "role=\"textbox\" label=\"Write something...\""
Expand Down
1 change: 0 additions & 1 deletion .claude/skills/agent-device/flows/macros/web/sign-in.ad
Original file line number Diff line number Diff line change
Expand Up @@ -4,7 +4,6 @@ context platform=web
# @pre role="button" label="Continue"
# @post text="Welcome!" || text="Home"
# @post role="button" label="Join" || role="button" label="Search"
# @tag auth
# @param EMAIL Login email. Use randomized alias format `agent-device-testing+<9digits>@gmail.com` to avoid account flagging.

fill "role=\"textbox\" label=\"Phone or email\" || label=\"Phone or email\"" "${EMAIL}"
Expand Down

This file was deleted.

11 changes: 0 additions & 11 deletions .claude/skills/agent-device/flows/tests/open-create-expense.ad

This file was deleted.

Loading
Loading