From 6f273b50445ac76743e419c0e9bd1ec3f0007c40 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Kacper=20Miko=C5=82ajczak?= Date: Fri, 10 Apr 2026 19:25:01 +0200 Subject: [PATCH 01/14] Install agent-device skill and add App-specific mobile testing wrapper - Install callstackincubator/agent-device skill (+ bundled dogfood skill) - Add agent-device-app-testing wrapper skill with Expensify-specific context: package name, sign-in flow, usage guidance, and proactive triggers --- .agents/skills/agent-device/SKILL.md | 72 ++++ .../references/bootstrap-install.md | 209 +++++++++++ .../references/coordinate-system.md | 28 ++ .../agent-device/references/debugging.md | 136 +++++++ .../agent-device/references/exploration.md | 334 ++++++++++++++++++ .../agent-device/references/macos-desktop.md | 87 +++++ .../agent-device/references/remote-tenancy.md | 99 ++++++ .../agent-device/references/verification.md | 111 ++++++ .agents/skills/dogfood/SKILL.md | 183 ++++++++++ .../dogfood/references/issue-taxonomy.md | 83 +++++ .../templates/dogfood-report-template.md | 52 +++ .claude/skills/agent-device | 1 + .../skills/agent-device-app-testing/SKILL.md | 114 ++++++ .claude/skills/dogfood | 1 + skills-lock.json | 15 + 15 files changed, 1525 insertions(+) create mode 100644 .agents/skills/agent-device/SKILL.md create mode 100644 .agents/skills/agent-device/references/bootstrap-install.md create mode 100644 .agents/skills/agent-device/references/coordinate-system.md create mode 100644 .agents/skills/agent-device/references/debugging.md create mode 100644 .agents/skills/agent-device/references/exploration.md create mode 100644 .agents/skills/agent-device/references/macos-desktop.md create mode 100644 .agents/skills/agent-device/references/remote-tenancy.md create mode 100644 .agents/skills/agent-device/references/verification.md create mode 100644 .agents/skills/dogfood/SKILL.md create mode 100644 .agents/skills/dogfood/references/issue-taxonomy.md create mode 100644 .agents/skills/dogfood/templates/dogfood-report-template.md create mode 120000 .claude/skills/agent-device create mode 100644 .claude/skills/agent-device-app-testing/SKILL.md create mode 120000 .claude/skills/dogfood create mode 100644 skills-lock.json diff --git a/.agents/skills/agent-device/SKILL.md b/.agents/skills/agent-device/SKILL.md new file mode 100644 index 000000000000..91a5068e5b1d --- /dev/null +++ b/.agents/skills/agent-device/SKILL.md @@ -0,0 +1,72 @@ +--- +name: agent-device +description: Automates interactions for Apple-platform apps (iOS, tvOS, macOS) and Android devices. Use when navigating apps, taking snapshots/screenshots, tapping, typing, scrolling, or extracting UI info across mobile, TV, and desktop targets. +--- + +# agent-device + +Use this skill as a router with mandatory defaults. Read this file first. For normal device tasks, always load `references/bootstrap-install.md` and `references/exploration.md` before acting. Use bootstrap to confirm or establish deterministic setup. Use exploration for UI inspection, interaction, and verification once the app session is open. + +## Default operating rules + +- Start conservative. Prefer read-only inspection before mutating the UI. +- Start deterministic. If the app name, package, device, or session is uncertain, load bootstrap and discover them before interacting. +- Use plain `snapshot` when the task is to verify what text or structure is currently visible on screen. +- Use `snapshot -i` only when you need interactive refs such as `@e3` for a requested action or targeted query. On iOS and Android, default snapshot output uses the same visible-first model: off-screen interactive content is exposed as discovery hints, not tappable refs. +- Prefer `diff snapshot` after a nearby mutation when you only need to know what changed. +- Avoid speculative mutations. You may take the smallest reversible UI action needed to unblock inspection or complete the requested task, such as dismissing a popup, closing an alert, or clearing an unintended surface. +- In React Native dev or debug builds, check early for visible warning or error overlays, tooltips, and toasts that can steal focus or intercept taps. If they are not part of the requested behavior, dismiss them and continue. If you saw them, report them in the final summary. +- Do not browse the web or use external sources unless the user explicitly asks. +- Re-snapshot after meaningful UI changes instead of reusing stale refs. +- Treat refs in default snapshot output as actionable-now, not durable identities. If a target is off-screen, use `scrollintoview` or scroll and re-snapshot. +- Prefer `@ref` or selector targeting over raw coordinates. +- Ensure the correct target is pinned and an app session is open before interacting. +- Keep the loop short: `open` -> inspect/act -> verify if needed -> `close`. + +## Default flow + +1. Load [references/bootstrap-install.md](references/bootstrap-install.md) and [references/exploration.md](references/exploration.md) before acting on a normal device task. +2. Use bootstrap first to confirm or establish the correct target, app install, and open app session. +3. Once the app session is open and stable, use exploration for inspection, interaction, and verification. +4. Start with plain `snapshot` if the goal is to read or verify what is visible. +5. Escalate to `snapshot -i` only if you need refs for interactive exploration or a requested action. +6. Use `get`, `is`, or `find` before mutating the UI when a read-only command can answer the question. +7. End by capturing proof if needed, then `close`. + +## QA modes + +- Open-ended bug hunt with reporting: use [../dogfood/SKILL.md](../dogfood/SKILL.md). +- Pass/fail QA from acceptance criteria: stay in this skill, start with [references/bootstrap-install.md](references/bootstrap-install.md), then use the QA loop in [references/exploration.md](references/exploration.md). + +## Required references + +- For every normal device task, after reading this file, load [references/bootstrap-install.md](references/bootstrap-install.md) first, then [references/exploration.md](references/exploration.md), before acting. +- Use bootstrap to confirm or establish deterministic setup, especially in sandbox or cloud environments. +- Use exploration once the app session is open and stable. +- Load additional references only when their scope is needed. + +## Decision rules + +- Use plain `snapshot` when you need to verify whether text is visible. +- Use `snapshot -i` mainly for interactive exploration and choosing refs. +- Use `diff snapshot` for compact post-action verification; use `snapshot --diff` when that alias is easier to discover from snapshot help. +- Use `get`, `is`, or `find` when they can answer the question without changing UI state. +- Use `fill` to replace text. +- Use `type` to append text. +- Do not write `type @eN "text"`. Use `fill @eN "text"` to target a field directly, or `press @eN` then `type "text"` when the field already has focus and you want append semantics. +- If the on-screen keyboard blocks the next step, prefer `keyboard dismiss` over navigation. On iOS, keep an app session open first; `keyboard status|get` remains Android-only. +- When a task asks to "go back", use plain `back` for predictable app-owned navigation and reserve `back --system` for platform back gestures or button semantics. +- Use `type --delay-ms` or `fill --delay-ms` for debounced search fields that drop characters when typed too quickly. +- If there is no simulator, no app install, or no open app session yet, switch to `bootstrap-install.md` instead of improvising setup steps. +- Use the smallest unblock action first when transient UI blocks inspection, but do not navigate, search, or enter new text just to make the UI reveal data unless the user asked for that interaction. +- In React Native dev or debug apps, treat visible warning or error overlays as transient blockers unless the user is explicitly asking you to diagnose them. Dismiss them when safe, then continue the requested flow. +- Do not use external lookups to compensate for missing on-screen data unless the user asked for them. +- If the needed information is not exposed on screen, say that plainly instead of compensating with extra navigation, text entry, or web search. +- Prefer `@ref` or selector targeting over raw coordinates. + +## Additional references + +- Need logs, network, alerts, permissions, or failure triage: [references/debugging.md](references/debugging.md) +- Need screenshots, diff, recording, replay maintenance, or perf data: [references/verification.md](references/verification.md) +- Need desktop surfaces, menu bar behavior, or macOS-specific interaction rules: [references/macos-desktop.md](references/macos-desktop.md) +- Need remote HTTP transport, `--remote-config` launches, or tenant leases on a remote macOS host: [references/remote-tenancy.md](references/remote-tenancy.md) diff --git a/.agents/skills/agent-device/references/bootstrap-install.md b/.agents/skills/agent-device/references/bootstrap-install.md new file mode 100644 index 000000000000..2e876fced195 --- /dev/null +++ b/.agents/skills/agent-device/references/bootstrap-install.md @@ -0,0 +1,209 @@ +# Bootstrap and Install + +## When to open this file + +Open this file when you still need to choose the right target, start the right session, install or relaunch the app, or pin automation to one device before interacting. This is the deterministic setup layer for sandbox, cloud, or other environments where install paths, device state, or app readiness may be uncertain. + +## Open-first path + +- `devices` +- `apps` +- `ensure-simulator` +- `open` +- `session list` + +Use this exact order when you are not sure about the installed app identifier. On Android dev builds in particular, `apps` is cheaper than guessing package suffixes and retrying failed `open` calls. + +## Install path + +- `install` or `reinstall` + +## Most common mistake to avoid + +Do not start acting before you have pinned the correct target and opened an `app` session. In mixed-device environments, always pass `--device`, `--udid`, or `--serial`. + +## Deterministic setup rule + +If there is no simulator, no app install, no open app session, or any uncertainty about where the app should come from, stay in this file and use deterministic setup commands or bootstrap scripts first. Do not improvise install paths or app-launch flows while exploring. + +After setup is confirmed or completed, move to `exploration.md` before doing UI inspection or interaction. + +## Open-first rule + +- If the user asks to test an app and does not provide an install artifact or explicit install instruction, try `open ` first. +- If `open ` fails, run `agent-device apps` and retry with a discovered app name before considering install steps. +- Do not install or reinstall on the first attempt unless the user explicitly asks for installation or provides a concrete artifact path or URL. +- When installation is required from a known location, prefer a checked-in shell script or other deterministic bootstrap command over ad hoc path guessing. + +- If `open ` fails, or you are not sure which app name is available on the target, run `agent-device apps` first and choose from the discovered app list instead of guessing. +- Use `apps --platform ` together with `--device`, `--udid`, or `--serial` when target selection matters. +- Once you have the correct app name, retry `open` with that exact discovered value. + +## Common starting points + +These are examples, not required exact sequences. Use the smallest setup flow that matches the task. + +### Boot a simulator and open an app + +```bash +agent-device ensure-simulator --platform ios --device "iPhone 17 Pro" --boot +agent-device open MyApp --platform ios --device "iPhone 17 Pro" --relaunch +``` + +### Install an app artifact + +```bash +agent-device install com.example.app ./build/app.apk --platform android --serial emulator-5554 +``` + +```bash +agent-device install com.example.app ./build/MyApp.app --platform ios --device "iPhone 17 Pro" +``` + +## Install guidance + +- Use `install ` when the app may already be installed and you do not need a fresh-state reset. +- Use `reinstall ` when you explicitly need uninstall plus install as one deterministic step. +- Keep install and open as separate phases. Do not turn them into one default command flow. +- Supported binary formats: + - Android: `.apk` and `.aab` + - iOS: `.app` and `.ipa` +- For iOS `.ipa` files, `` is used as the bundle id or bundle name hint when the archive contains multiple app bundles. +- After install or reinstall, later use `open ` with the exact discovered or known package/bundle identifier, not the artifact path. + +## Choose the right starting point + +- iOS local QA: prefer simulators unless the task explicitly requires physical hardware. +- iOS in mixed simulator and device environments: run `ensure-simulator` first, then keep using `--device` or `--udid`. +- TV targets: use `--target tv` together with `--platform` when the task is for tvOS or Android TV rather than phone or tablet surfaces. +- Android binary flow: use `install` or `reinstall` for `.apk` or `.aab`, then open by installed package name. +- macOS desktop app flow: use `open --platform macos`. Only load [macos-desktop.md](macos-desktop.md) if a desktop surface or macOS-specific behavior matters. + +TV example: + +```bash +agent-device open MyTvApp --platform ios --target tv +agent-device open com.example.androidtv --platform android --target tv +``` + +## Session rules + +- Use `--session ` when you need a named session: + +```bash +agent-device --session auth open Settings --platform ios +agent-device --session auth snapshot -i +``` + +- Use `open ` before interactions. +- Use `close` when done. Add `--shutdown` when you want simulators or emulators torn down with the session. +- Use semantic session names when you need multiple concurrent runs. +- Use `--save-script=` on `close` when you want to keep a replay script. +- For dev loops where state can linger, prefer `open --relaunch`. +- In iOS sessions, use `open ` for the app itself. Use `open ` for deep links, and `open ` when you need to launch the app and deep link in one step. +- On iOS, `appstate` is session-scoped and requires the matching active session on the target device. + +## After a session is established + +Once you have opened the correct session on the correct target, default to the conservative rule: keep the session binding on follow-up commands, and stop repeating device-routing flags unless you are intentionally retargeting. + +- Prefer `--session ` on follow-up commands, or use sandboxed `AGENT_DEVICE_SESSION`. +- Do not keep repeating `--platform`, `--target`, `--device`, `--udid`, `--serial`, or similar target-selection flags on normal follow-up commands. +- Only omit follow-up session flags when the environment explicitly guarantees isolation. + +Good shared-host pattern: + +```bash +agent-device --session auth open Settings --platform ios --device "iPhone 17 Pro" +agent-device --session auth snapshot -i +agent-device --session auth press @e3 +agent-device --session auth close +``` + +Bad shared-host pattern: + +```bash +agent-device --session auth open Settings --platform ios --device "iPhone 17 Pro" +agent-device --session auth snapshot -i --platform ios --device "iPhone 17 Pro" +``` + +Use target-selection flags again only when you are choosing the target before opening a session, or when you explicitly mean to retarget. + +## Session-bound automation + +Use this when an orchestrator must keep plain CLI calls on one session and device. + +```bash +export AGENT_DEVICE_SESSION=qa-ios +export AGENT_DEVICE_PLATFORM=ios +export AGENT_DEVICE_SESSION_LOCK=strip + +agent-device open MyApp --relaunch +``` + +- `AGENT_DEVICE_SESSION` plus `AGENT_DEVICE_PLATFORM` provides the default binding. +- `--session-lock reject|strip` controls whether conflicting per-call routing flags fail or are ignored. +- Conflicts include explicit retargeting flags such as `--platform`, `--target`, `--device`, `--udid`, `--serial`, `--ios-simulator-device-set`, and `--android-device-allowlist`. +- Lock policy applies to nested `batch` steps too. +- Compatibility aliases remain supported: `--session-locked`, `--session-lock-conflicts`, `AGENT_DEVICE_SESSION_LOCKED`, and `AGENT_DEVICE_SESSION_LOCK_CONFLICTS`. + +Android emulator variant: + +```bash +export AGENT_DEVICE_SESSION=qa-android +export AGENT_DEVICE_PLATFORM=android + +agent-device --session-lock reject open com.example.myapp --relaunch +``` + +## Scoped discovery + +Use scoped discovery when one run must not see host-global device lists. + +```bash +agent-device devices --platform ios --ios-simulator-device-set /tmp/tenant-a/simulators +agent-device devices --platform android --android-device-allowlist emulator-5554,device-1234 +``` + +- Scope is applied before `--device`, `--udid`, and `--serial`. +- Out-of-scope selectors fail with `DEVICE_NOT_FOUND`. +- With iOS simulator-set scope enabled, iOS physical devices are not enumerated. +- If the scoped iOS simulator set is empty, the error should point at the set path and suggest creating a simulator in that set. +- Environment equivalents: + - `AGENT_DEVICE_IOS_SIMULATOR_DEVICE_SET` + - `AGENT_DEVICE_ANDROID_DEVICE_ALLOWLIST` + +## Session inspection and replay + +```bash +agent-device session list +agent-device replay ./session.ad --session auth +agent-device replay -u ./session.ad --session auth +``` + +- iOS session entries include `device_udid` and `ios_simulator_device_set`. Use them to confirm routing in concurrent runs. +- Prefer selector-based actions and assertions in saved replay scripts. +- Tenant isolation namespaces sessions as `:` during tenant-scoped runs. + +## When to leave this file + +- Once the correct target and session are pinned, move to [exploration.md](exploration.md). +- If opening, startup, permissions, or logs become the blocker, switch to [debugging.md](debugging.md). + +## Install examples + +```bash +agent-device reinstall MyApp /path/to/app-debug.apk --platform android --serial emulator-5554 +``` + +```bash +agent-device install com.example.app ./build/MyApp.ipa --platform ios --device "iPhone 17 Pro" +``` + +Do not use `open --relaunch` on Android. + +## Security and trust notes + +- Treat signing, provisioning, and daemon auth values as host secrets. Do not paste them into shared logs or commit them to source control. +- Prefer Xcode Automatic Signing over manual overrides when a physical iOS device is involved. +- Keep persistent host-specific defaults in environment variables rather than checked-in project config. diff --git a/.agents/skills/agent-device/references/coordinate-system.md b/.agents/skills/agent-device/references/coordinate-system.md new file mode 100644 index 000000000000..03b8f2ef0aac --- /dev/null +++ b/.agents/skills/agent-device/references/coordinate-system.md @@ -0,0 +1,28 @@ +# Coordinate System + +## When to open this file + +Open this file only when you must use raw coordinates instead of selectors or `@ref` targeting. + +## Main commands to reach for first + +- `screenshot` +- coordinate-based `click` or `swipe` + +## Most common mistake to avoid + +Do not assume coordinates mean the same thing across platforms or runs. Prefer selectors and refs first. + +## Canonical loop + +```bash +agent-device screenshot /tmp/current-screen.png +agent-device click 120 240 +``` + +## Rules + +- Origin is the top-left of the device screen. +- iOS uses device points. +- Android uses pixels. +- Use screenshots to reason about coordinates before acting. diff --git a/.agents/skills/agent-device/references/debugging.md b/.agents/skills/agent-device/references/debugging.md new file mode 100644 index 000000000000..885fdf2b92ef --- /dev/null +++ b/.agents/skills/agent-device/references/debugging.md @@ -0,0 +1,136 @@ +# Debugging + +## When to open this file + +Open this file when the task turns into failure triage, logs, network inspection, permission prompts, setup trouble, or unstable session behavior. + +## Main commands to reach for first + +- `logs clear --restart` +- `network dump` +- `logs path` +- `logs doctor` +- `alert wait` +- `alert accept` or `alert dismiss` + +## Most common mistake to avoid + +Do not leave logging on for normal flows or dump full log files into context. Keep debug windows short and inspect logs with `grep` or `tail`. + +In React Native dev or debug builds, do not dismiss visible warning or error overlays without remembering to report them later. If you close one to keep the flow moving, keep at least a screenshot or a short marked log window so the summary can name it. + +## Canonical loop + +```bash +agent-device open MyApp --platform ios +agent-device logs clear --restart +agent-device network dump 25 +agent-device logs path +agent-device close +``` + +## Log and network flow + +Logging is off by default. Enable it only when you need a debugging window. + +- Default app logs live under `~/.agent-device/sessions//app.log`. +- `logs clear --restart` is the fastest clean repro loop. +- `network dump [limit] [summary|headers|body|all]` parses recent HTTP(s) entries from the same session app log. +- On macOS, `network dump` is app-scoped and only sees Unified Logging associated with the active session app. +- On iOS simulators, `network dump` can recover recent app log history with `simctl log show` when the live session stream is sparse, so check the returned notes before assuming the repro window was empty. +- On iOS, `network dump` is still limited to what Unified Logging exposes for the app process. If the app does not emit request metadata there, `network dump` can legitimately return no HTTP entries even during a real repro. +- Summary output already shows timestamp, status, and duration when the log backend exposes them. +- Prefer the explicit flag form `network dump 25 --include headers|body|all` when you need more than the default summary view. +- If iOS simulator notes say app logs were recovered but none looked like HTTP traffic, treat that as an app instrumentation gap rather than a missing repro and inspect `logs path` for the non-network diagnostics that were captured. +- `logs doctor` checks backend and runtime readiness for the current session and device. +- `logs mark "before tap"` inserts a timestamped marker into the app log. +- Android `network dump` surfaces timestamps from logcat-style prefixes and can backfill status and request/response duration from adjacent GIBSDK packet lines, so check it before dumping raw log windows. +- Android app-log streaming rebinds to the current app PID after relaunches, so rerun the repro window before assuming the last log slice is stale. +- Marker lines are emitted with the `[agent-device][mark][...]` prefix. When you grep later, prefer a narrow pattern such as `grep -n -E "agent-device.*mark|before tap" `. +- Session app logs can contain runtime data, headers, or payload fragments. Review them before sharing. +- `logs start` requires an active app session and appends to `app.log`. +- `logs stop` stops streaming. `close` also stops logging. +- `logs clear` truncates `app.log` and removes rotated `app.log.N` files, and requires logging to be stopped first. +- `logs path` returns the log path plus metadata about the active backend and file state. +- `network log` is an alias for `network dump`. + +Operational limits: + +- `app.log` rotates to `app.log.1` after 5 MB by default. +- `network dump` scans the last 4000 app-log lines, returns up to 200 entries, and truncates header or payload fields at 2048 characters. +- Retention knobs: + - `AGENT_DEVICE_APP_LOG_MAX_BYTES` + - `AGENT_DEVICE_APP_LOG_MAX_FILES` +- Redaction hook: + - `AGENT_DEVICE_APP_LOG_REDACT_PATTERNS` + +Useful shell follow-up after `logs path`: + +```bash +grep -n -E "Error|Exception|Fatal|crash" +grep -n -E "agent-device.*mark|before tap" +tail -50 +``` + +If the app showed a visible warning or error overlay during the flow: + +- Prefer a narrow grep window around your `logs mark` lines instead of loading the whole file. +- Mention the surfaced warning or error in the final summary even if it did not block completion. +- If the overlay kept returning, call that out as a stability issue instead of treating it as operator noise. + +## Alerts and permissions + +Use `alert` for iOS simulator permission dialogs and macOS desktop alerts instead of tapping coordinates. + +```bash +agent-device alert wait 5000 +agent-device alert accept +``` + +- `alert` is supported on iOS simulators and macOS desktop targets. +- `alert accept` and `alert dismiss` retry internally for a short window, so you usually do not need manual sleeps. +- If a permission sheet or modal is visible in `snapshot` or `screenshot` but `alert accept` says no alert was found, treat it as normal tappable UI for that run: take a scoped `snapshot -i -s ""` and `press @ref` instead of looping on `alert`. +- iOS 16+ "Allow Paste" prompts are suppressed under XCUITest. Use `xcrun simctl pbcopy booted` when you need to seed simulator clipboard content directly. + +## Setup problems worth recognizing early + +- iOS snapshots do not require macOS Accessibility permissions. +- iOS physical-device XCTest setup does require valid signing and provisioning. +- If physical-device runner setup fails, prefer Xcode Automatic Signing first. +- Optional overrides are: + - `AGENT_DEVICE_IOS_TEAM_ID` + - `AGENT_DEVICE_IOS_SIGNING_IDENTITY` + - `AGENT_DEVICE_IOS_PROVISIONING_PROFILE` + - `AGENT_DEVICE_IOS_BUNDLE_ID` +- If daemon startup is timing out during setup, increase `AGENT_DEVICE_DAEMON_TIMEOUT_MS`. +- If daemon startup fails with stale metadata hints, clean `~/.agent-device/daemon.json` and `~/.agent-device/daemon.lock`, then retry. +- Free Apple Developer personal-team accounts may reject generic bundle IDs. Use a unique reverse-DNS value for `AGENT_DEVICE_IOS_BUNDLE_ID` when that happens. + +## Common failure patterns + +- `snapshot` returns 0 nodes: the app may no longer be foregrounded or the UI is not stable yet. Re-open the app or retry when state settles. +- Logs are empty: confirm you opened an app session before `logs clear --restart`. +- Android logs look stale after relaunch: retry the repro window after the process rebinds. +- Android accessibility snapshots can lag behind visible screen transitions. The next snapshot now retries briefly after navigation-sensitive actions, but if the tree still looks stale, use `screenshot` as visual truth, wait briefly, then re-run `snapshot -i`. +- React Native dev warnings or errors keep reappearing: treat them as part of the app state, not as disposable chrome. Capture one clean repro and include them in the summary. +- Permission prompts block the flow: wait for the alert and handle it explicitly. +- If snapshots keep returning 0 nodes on an iOS simulator, restart Simulator and re-open the app. +- If a macOS snapshot looks incomplete, compare with `snapshot --raw --platform macos` to separate collector filtering from missing AX content. + +## Crash triage fast path + +Always start from the session app log, then branch by platform. + +```bash +agent-device logs path +grep -n -E "SIGABRT|SIGSEGV|EXC_|fatal|exception|terminated|killed|jetsam|memorystatus|FATAL EXCEPTION|Abort message" +``` + +- iOS: if the log suggests `ReportCrash`, `SIGABRT`, or `EXC_*`, inspect `~/Library/Logs/DiagnosticReports`. +- Android: if the app log is not enough, use `adb logcat` for `FATAL EXCEPTION`, `Abort message`, or `signal` lines around process death. +- If no crash signature appears in app logs, stop collecting broad logs and switch to the platform-native crash source. + +## When to leave this file + +- Return to [exploration.md](exploration.md) once the app is stable again. +- Load [verification.md](verification.md) if you need evidence artifacts after reproducing the issue. diff --git a/.agents/skills/agent-device/references/exploration.md b/.agents/skills/agent-device/references/exploration.md new file mode 100644 index 000000000000..2ff799f703ec --- /dev/null +++ b/.agents/skills/agent-device/references/exploration.md @@ -0,0 +1,334 @@ +# Exploration + +## When to open this file + +Open this file when the app or screen is already running and you need to discover the UI, choose targets, read state, wait for conditions, or perform normal interactions. + +## Read-only first + +- If the question is what text, labels, or structure is visible on screen, start with plain `snapshot`. +- Escalate to `snapshot -i` only when you need refs such as `@e3` for interactive exploration or a requested action. +- If you intend to `press`, `fill`, or otherwise interact, start with `snapshot -i` and fall back to plain `snapshot` only if interactive refs are unavailable. +- Prefer `get`, `is`, or `find` before mutating the UI when a read-only command can answer the question. +- You may take the smallest reversible UI action needed to unblock inspection, such as dismissing a popup, closing an alert, or backing out of an unintended surface. +- Do not type or fill text just to make hidden information easier to access unless the user asked for that interaction. +- Do not use external sources to infer missing UI state unless the user explicitly asked. +- If the answer is not visible or exposed in the UI, report that gap instead of compensating with search, navigation, or text entry. + +## Decision shortcut + +- User asks what is visible on screen: `snapshot` +- User asks for exact text from a known target: `get text` +- User asks you to tap, type, or choose an element: `snapshot -i`, then act +- React Native dev or debug build shows warning/error UI: capture enough evidence to identify it, dismiss it if it is not the requested behavior, then continue the flow and report it in the summary +- The on-screen keyboard is blocking the next step: `keyboard dismiss`; on iOS do this only while an app session is active, and use `keyboard status|get` only on Android +- UI does not expose the answer: say so plainly; do not browse or force the app into a new state unless asked + +## Read-only commands + +- `snapshot` +- `get` +- `is` +- `find` +- `keyboard status|get` on Android when keyboard visibility or input type matters + +## Interaction commands + +- `snapshot -i` +- `press` +- `fill` +- `type` +- `scrollintoview` +- `wait` +- `keyboard dismiss` when the keyboard obscures the next target + +## Common mistakes to avoid + +**Stale refs.** Do not treat `@ref` values as durable after navigation or dynamic updates. Re-snapshot after the UI changes, and switch to selectors when the flow must stay stable. + +**Android AX tree lag.** After submits, route changes, or composer transitions, the accessibility tree can lag behind the visible UI. If `snapshot -i` and `screenshot` disagree: + +1. Trust the screenshot as visual truth. +2. Take one fresh `snapshot -i`. Android retries briefly after navigation-sensitive actions. +3. If the tree still disagrees with the screenshot, wait briefly, then take one more fresh snapshot. Do not loop snapshots immediately. + +**React Native dev overlays.** In dev or debug builds, warning or error overlays can block taps, change focus, or hide the real UI. Check for them near app open and after major transitions. + +- Not blocking the task: dismiss and continue. +- Blocking or recurring: switch to [debugging.md](debugging.md) and collect evidence. +- Seen at any point: mention in the final summary even if dismissed. + +## Common example loops + +These are examples, not required exact sequences. Adapt them to the app, state, and task at hand. + +### Interactive exploration loop + +```bash +agent-device open Settings --platform ios +agent-device snapshot -i +agent-device press @e3 +agent-device wait visible 'label="Privacy & Security"' 3000 +agent-device get text 'label="Privacy & Security"' +agent-device close +``` + +### Screen verification loop + +```bash +agent-device open MyApp --platform ios +# perform the necessary actions to reach the state you need to verify +agent-device snapshot +# verify whether the expected element or text is present +agent-device close +``` + +## Snapshot choices + +- Use plain `snapshot` when you only need to verify whether visible text or structure is on screen. +- Use `snapshot -i` when you need refs such as `@e3` for interactive exploration or for an intended interaction. +- On iOS and Android, default snapshot output is visible-first. Off-screen interactive content is surfaced as discovery hints (including inline scroll/list hidden-content hints when known), not shown as directly tappable refs. +- Treat large text-surface lines in `snapshot -i` as discovery output. If a node shows preview or truncation metadata, use `get text @ref` only after you have already decided that `snapshot -i` is needed for that surface. +- Use `snapshot -i -s "Camera"` or `snapshot -i -s @e3` when you want a smaller, scoped result. +- If `snapshot -i -s ""` returns 0 nodes, the scope did not match the current screen. Widen the query or re-check the screen state instead of assuming the command silently fell back to the full tree. +- If `snapshot -i` returns 0 nodes but the screen is visibly populated, treat `screenshot` as visual truth, wait briefly, then re-run `snapshot -i` once before escalating. +- If `snapshot -i -d ` says the interactive output is empty at that depth, retry without `-d` instead of taking more shallow snapshots. + +Example: + +```bash +agent-device snapshot -i +``` + +Sample output: + +```text +Page: com.apple.Preferences +App: com.apple.Preferences + +@e1 [ioscontentgroup] + @e2 [button] "Camera" + @e3 [button] "Privacy & Security" +[off-screen below] 2 interactive items: "Location Services", "Battery" +``` + +## Refs vs selectors + +- Use refs for discovery, debugging, and short local loops. +- Use `scrollintoview @ref` when the target is already known from the current snapshot and you want the command to re-snapshot after each swipe until the element reaches the viewport safe band. +- If `scrollintoview @ref` succeeds, prefer the returned `currentRef` for the next action. +- Visible-first off-screen summaries are intentionally compact. If you need the full off-screen tree instead of a short summary, retry with `snapshot --raw`. +- Cap long searches with `--max-scrolls ` when the list may be unbounded or the target may not exist. +- Use selectors for deterministic scripts, assertions, and replay-friendly actions. +- Prefer selector or `@ref` targeting over raw coordinates. +- For tap interactions, `press` is canonical and `click` is an equivalent alias. + +Examples: + +```bash +agent-device press @e2 +agent-device fill @e5 "test" +agent-device press 'id="camera_row" || label="Camera" role=button' +agent-device is visible 'id="camera_settings_anchor"' +``` + +## Interaction fallbacks + +When `press @ref` fails: + +1. If the error says the ref is off-screen, run `scrollintoview @ref` and reuse the returned `currentRef` or take one fresh snapshot. +2. Re-snapshot if the UI may have changed. +3. Retry `press @ref` or a selector-based `press`. +4. If `screenshot --overlay-refs --json` returned a reliable `overlayRefs[].center`, use `agent-device press `. +5. Use an external vision-based tap tool only after semantic and coordinate targeting fail. + +- Prefer `@ref` over coordinates. +- Do not guess coordinates from the image when structured `center` is available. +- `agent-device` does not provide a built-in vision-tap flag. + +## Text entry rules + +- Use `fill` to replace text in an editable field. +- Use `type` to append text to the current insertion point. +- Use `fill @ref "text"` when you need to target a field directly by ref. +- Use `press @ref`, then `type "text"` when the field is already focused and you need append semantics. +- Do not write `type @ref "text"`; `type` only accepts text and will not target that ref for you. +- If the keyboard blocks the next control after text entry, prefer `keyboard dismiss` instead of backing out of the screen. +- On iOS, `keyboard dismiss` depends on the active app session to keep the target app foregrounded, so do not rely on selector-only dismiss calls after closing or without `open`. +- Do not use `fill` or `type` just to make the app reveal information that is not currently visible unless the user asked for that interaction. + +## React Native dev or debug overlays + +Use this loop for React Native dev clients, Metro-backed builds, and local debug sessions where warnings or errors may appear as tooltips, banners, toasts, or modal overlays. + +1. After `open`, inspect the visible UI for warning or error surfaces before relying on the next tap. +2. If a warning or error is visible, capture enough evidence to identify it: + - preferred: `screenshot` + - optional: `logs mark "warning visible"` or `logs mark "error visible"` if you are already in a debug window +3. If the overlay is not the thing the user asked you to investigate, dismiss or close it with the smallest reversible action. +4. Re-check the intended screen before continuing the task. +5. Report any visible warnings or errors in the final summary, even if the flow succeeded after dismissal. + +Use this rule of thumb: + +- Warning overlay that does not block the task: dismiss and keep going. +- Error overlay that does not block the task: dismiss, keep going, and report it. +- Error overlay that blocks the task or keeps returning: stop treating it as noise and switch to [debugging.md](debugging.md). + +## Query and sync rules + +- Use `get` to read text, attrs, or state from a known target. +- Use `is` for assertions. +- Use `wait` when the UI needs time to settle after a mutation. +- Use `find "" click --json` when you need search-driven targeting plus matched-target metadata. +- Use `find "" click --first` or `--last` when ambiguous matches are expected and you want the first or last occurrence without falling back to raw coordinates. +- If you are forced onto raw coordinates, open [coordinate-system.md](coordinate-system.md) first. + +Example: + +```bash +agent-device find "Increment" click --json +``` + +Returned metadata comes from the matched snapshot node and can be used for observability or replay maintenance. + +## QA from acceptance criteria + +Use this loop when the task starts from acceptance criteria and you need to turn them into concrete checks. + +Preferred mapping: + +- visibility claim for what is on-screen now: `is visible` or plain `snapshot` +- presence claim regardless of viewport visibility: `is exists` +- exact text, label, or value claim: `get text` +- post-action state change: act, then `wait`, then `is` or `get` +- nearby structural UI change: `diff snapshot` +- proof artifact for the final result: `screenshot` or `record` + +Notes: + +- `wait text` is useful for synchronizing on text presence, but it is not the same as `is visible`. +- After a nearby navigation or submit on Android, prefer `screenshot`, then `wait 500` or `wait 1000`, then one fresh `snapshot -i` if the accessibility tree seems stale. + +Anti-hallucination rules: + +- Do not invent app names, device ids, session names, refs, selectors, or package names. +- Discover them first with `devices`, `open`, `snapshot -i`, `find`, or `session list`. +- If refs drift after navigation, re-snapshot or switch to selectors instead of guessing. + +Avoid this escalation path for visible-text questions: + +- Do not jump from `snapshot -i` to `get text @ref`, then to web search, then to typing into a search box just to force the app to reveal the answer. +- Start with `snapshot`. If the text is not visible or exposed, report that directly. +- After Android submit or navigation-heavy actions when the UI looks wrong: `screenshot` first, then `snapshot -i`. + +Canonical QA loop: + +```bash +agent-device open MyApp --platform ios +agent-device snapshot -i +agent-device press @e3 +agent-device wait visible 'label="Success"' 3000 +agent-device is visible 'label="Success"' +agent-device screenshot /tmp/qa-proof.png +agent-device close +``` + +## Accessibility audit + +Use this pattern when you need to find UI that is visible to a user but missing from the accessibility tree. + +Audit loop: + +1. Capture a `screenshot` to see what is visually rendered. +2. Capture a `snapshot` or `snapshot -i` to see what the accessibility tree exposes. +3. Compare the two: + - visible in screenshot and present in snapshot: exposed to accessibility + - visible in screenshot and missing from snapshot: likely accessibility gap +4. If you suspect the node exists in AX but is filtered from interactive output, retry with `snapshot --raw`. + +Example: + +```bash +agent-device screenshot /tmp/accessibility-screen.png +agent-device snapshot -i +``` + +Use `screenshot` as the visual source of truth and `snapshot` as the accessibility source of truth for this audit. + +## Batch only when the sequence is already known + +Use `batch` when a short command sequence is already planned and belongs to one logical screen flow. + +```bash +agent-device batch --session sim --platform ios --steps-file /tmp/batch-steps.json --json +``` + +- Keep batch size moderate, roughly 5 to 20 steps. +- Add `wait` or `is exists` guards after mutating steps. +- Do not use `batch` for highly dynamic flows that need replanning after each step. + +Example: known chat-send flow + +```json +[ + { "command": "open", "positionals": ["ChatApp"], "flags": { "platform": "android" } }, + { "command": "click", "positionals": ["label=\"Travel chat\""], "flags": {} }, + { "command": "wait", "positionals": ["label=\"Message\"", "3000"], "flags": {} }, + { "command": "fill", "positionals": ["label=\"Message\"", "Filed the expense"], "flags": {} }, + { "command": "press", "positionals": ["label=\"Send\""], "flags": {} } +] +``` + +Step payload contract: + +```json +[ + { "command": "open", "positionals": ["Settings"], "flags": { "platform": "ios" } }, + { "command": "wait", "positionals": ["label=\"Privacy & Security\"", "3000"], "flags": {} }, + { "command": "click", "positionals": ["label=\"Privacy & Security\""], "flags": {} }, + { "command": "get", "positionals": ["text", "label=\"Tracking\""], "flags": {} } +] +``` + +- `positionals` is optional and defaults to `[]`. +- `flags` is optional and defaults to `{}`. +- Only `command`, `positionals`, `flags`, and `runtime` are accepted as top-level step keys. +- Nested `batch` and `replay` are rejected. +- Supported error mode is stop-on-first-error. + +Response handling: + +- Success returns fields such as `total`, `executed`, `totalDurationMs`, and `results[]`. +- Human-mode `batch` runs also print a short per-step success summary. +- Failed runs include `details.step`, `details.command`, `details.executed`, and `details.partialResults`. +- Replan from the first failing step instead of rerunning the whole flow blindly. + +Canonical batch recipe: open app -> open action menu -> choose option -> verify + +```json +[ + { "command": "open", "positionals": ["com.example.app"], "flags": { "platform": "android" } }, + { "command": "wait", "positionals": ["text", "Home", "3000"], "flags": {} }, + { "command": "press", "positionals": ["label=\"More actions\" role=button"], "flags": {} }, + { "command": "wait", "positionals": ["text", "Camera scan", "2000"], "flags": {} }, + { "command": "press", "positionals": ["label=\"Camera scan\""], "flags": {} }, + { "command": "wait", "positionals": ["text", "Expense created", "15000"], "flags": {} }, + { "command": "is", "positionals": ["visible", "label=\"Expense created\""], "flags": {} } +] +``` + +Common batch error categories: + +- `INVALID_ARGS`: fix the payload shape and retry. +- `SESSION_NOT_FOUND`: open or select the correct session, then retry. +- `UNSUPPORTED_OPERATION`: switch to a supported command or surface. +- `AMBIGUOUS_MATCH`: refine the selector or locator, then retry the failed step. +- `COMMAND_FAILED`: add sync guards and retry from the failing step. + +## Stop conditions + +- If refs drift after transitions, switch to selectors. +- If a desktop surface or context menu is involved on macOS, load [macos-desktop.md](macos-desktop.md). +- If logs, network, alerts, or setup failures become the blocker, switch to [debugging.md](debugging.md). +- If the flow is stable and you need proof or replay maintenance, switch to [verification.md](verification.md). diff --git a/.agents/skills/agent-device/references/macos-desktop.md b/.agents/skills/agent-device/references/macos-desktop.md new file mode 100644 index 000000000000..4c63c6405b94 --- /dev/null +++ b/.agents/skills/agent-device/references/macos-desktop.md @@ -0,0 +1,87 @@ +# macOS Desktop + +## When to open this file + +Open this file only when `--platform macos` is involved or the task needs `frontmost-app`, `desktop`, or `menubar` surfaces. + +## Main commands to reach for first + +- `open --platform macos` +- `open --platform macos --surface frontmost-app|desktop|menubar` +- `snapshot -i` +- `get` +- `is` +- `click --button secondary` + +## Most common mistake to avoid + +Do not treat every macOS surface the same. Use the normal `app` surface when you want to act inside one app. Use `frontmost-app`, `desktop`, or `menubar` mainly to inspect what is visible before switching back to `app` for most interactions. + +## Canonical loop + +```bash +agent-device open TextEdit --platform macos +agent-device snapshot +agent-device close +``` + +## Surface rules + +- `app`: default surface and the normal choice for `click`, `fill`, `press`, `scroll`, `screenshot`, and `record`. +- `frontmost-app`: inspect the currently focused app without naming it first. +- `desktop`: inspect visible desktop windows across apps. +- `menubar`: inspect the active app menu bar and system menu extras. Use `open --platform macos --surface menubar` when you need one menu bar app's extras, such as a status-item app. +- Menu bar apps can expose a sparse or empty default `app` tree. Prefer the `menubar` surface first when the app lives entirely in the top bar. + +Use inspect-first surfaces to understand desktop-global UI, then switch back to `app` when you need to act in one app. + +## Snapshot expectations + +- `snapshot -i` should describe UI visible to a human. +- `desktop` snapshots can include multiple windows from multiple apps. +- `menubar` snapshots can include both app-menu items and system menu extras. +- Finder-style rows, sidebar items, toolbar controls, search fields, and opened context menus should appear when visible. +- Finder and other native apps may expose duplicate-looking row, cell, and child text nodes. Treat them as distinct AX nodes unless you have a stronger selector anchor. + +## Context menus + +Context menus are not ambient UI. Open them explicitly, then re-snapshot. + +```bash +agent-device click @e66 --button secondary --platform macos +agent-device snapshot -i +``` + +Expected loop: + +1. Snapshot visible content. +2. Secondary-click the target item. +3. Snapshot again. +4. Interact with the new `menu-item` nodes. + +## Targeting rules + +- Prefer selectors or `@ref` values over raw coordinates. +- On macOS, window position can vary across runs, so coordinate-only flows are fragile. +- If the task only needs shared exploration rules, return to [exploration.md](exploration.md). + +Selector guidance: + +- Good selectors usually anchor on stable labels or app-owned identifiers such as `label="Downloads"` or `role=menu-item label="Rename"`. +- Avoid relying on framework-generated `_NS:*` identifiers as stable selectors. + +Use `snapshot --raw --platform macos` only when debugging AX structure or collector filtering. Do not make raw snapshots the default agent loop. + +Things not to rely on: + +- Mobile-only helpers such as `install`, `reinstall`, or `push`. +- Desktop-global click or fill parity from `desktop` or `menubar` sessions. +- Raw coordinate assumptions across runs. + +Troubleshooting: + +- If visible content is missing from `snapshot -i`, re-snapshot after the UI settles. +- If `desktop` is too broad, retry with `frontmost-app`. +- If `menubar` is missing the expected menu, retry with `open --platform macos --surface menubar` for menu bar apps, or make the app frontmost first and retry the generic menubar surface. +- If the wrong menu opened, retry secondary-clicking the row or cell wrapper rather than the nested text node. +- If the app has multiple windows, make the correct window frontmost before relying on refs. diff --git a/.agents/skills/agent-device/references/remote-tenancy.md b/.agents/skills/agent-device/references/remote-tenancy.md new file mode 100644 index 000000000000..6ca54a8547e2 --- /dev/null +++ b/.agents/skills/agent-device/references/remote-tenancy.md @@ -0,0 +1,99 @@ +# Remote Tenancy + +## When to open this file + +Open this file for remote daemon HTTP flows, including `--remote-config` launches, that let an agent running in a Linux sandbox talk to another `agent-device` instance on a remote macOS host in order to control devices that are not available locally. This file covers daemon URL setup, authentication, lease allocation, and tenant-scoped command admission. + +## Main commands to reach for first + +- `agent-device open --remote-config --relaunch` +- `AGENT_DEVICE_DAEMON_BASE_URL=...` +- `AGENT_DEVICE_DAEMON_AUTH_TOKEN=...` +- `agent-device --tenant ... --session-isolation tenant --run-id ... --lease-id ...` + +## Most common mistake to avoid + +Do not run a tenant-isolated command without matching `tenant`, `run`, and `lease` scope. Admission checks require all three to line up. + +## Preferred remote launch path + +Use this when the agent needs the simplest remote control flow: a Linux sandbox agent talks over HTTP to `agent-device` on a remote macOS host and launches the target app through a checked-in `--remote-config` profile. + +```bash +agent-device open com.example.myapp --remote-config ./agent-device.remote.json --relaunch +``` + +- This is the preferred remote launch path for sandbox or cloud agents. +- For Android React Native relaunch flows, install or reinstall the APK first, then relaunch by installed package name. +- Do not use `open --relaunch`; remote runtime hints are applied through the installed app sandbox. + +## Lease flow example + +```bash +export AGENT_DEVICE_DAEMON_BASE_URL= +export AGENT_DEVICE_DAEMON_AUTH_TOKEN= + +agent-device \ + --tenant acme \ + --session-isolation tenant \ + --run-id run-123 \ + --lease-id \ + session list --json +``` + +Low-level lease operations exist for host-side automation, but do not point them at arbitrary hosts. The remote daemon executes device-control commands, so only use a trusted daemon base URL and an auth token managed by the same operator boundary. + +Lease lifecycle methods exposed by the daemon: + +- `agent_device.lease.allocate` +- `agent_device.lease.heartbeat` +- `agent_device.lease.release` +- `agent_device.command` + +## Transport prerequisites + +- Start the daemon in HTTP mode with `AGENT_DEVICE_DAEMON_SERVER_MODE=http|dual`. +- Point the client at the remote host with `AGENT_DEVICE_DAEMON_BASE_URL=http(s)://host:port[/base-path]`. +- For non-loopback remote hosts, set `AGENT_DEVICE_DAEMON_AUTH_TOKEN` or `--daemon-auth-token`. The client rejects non-loopback remote daemon URLs without auth. +- Direct JSON-RPC callers can authenticate with request params, `Authorization: Bearer `, or `x-agent-device-token`. +- Prefer an auth hook such as `AGENT_DEVICE_HTTP_AUTH_HOOK` when the host needs caller validation or tenant injection. + +## Lease lifecycle + +Use JSON-RPC methods on `POST /rpc`: + +- `agent_device.lease.allocate` +- `agent_device.lease.heartbeat` +- `agent_device.lease.release` + +Keep the lease alive for the duration of the run and release it when the tenant-scoped work is complete. + +Host-level lease knobs: + +- `AGENT_DEVICE_MAX_SIMULATOR_LEASES` +- `AGENT_DEVICE_LEASE_TTL_MS` +- `AGENT_DEVICE_LEASE_MIN_TTL_MS` +- `AGENT_DEVICE_LEASE_MAX_TTL_MS` + +## Command admission contract + +For tenant-isolated command execution, pass all four CLI flags together: + +```bash +agent-device \ + --tenant acme \ + --session-isolation tenant \ + --run-id run-123 \ + --lease-id \ + session list --json +``` + +The CLI sends `AGENT_DEVICE_DAEMON_AUTH_TOKEN` in both the JSON-RPC request token field and HTTP auth headers so existing daemon auth paths continue to work. + +## Failure semantics and trust notes + +- Missing tenant, run, or lease fields in tenant-isolation mode should fail as `INVALID_ARGS`. +- Inactive or scope-mismatched leases should fail as `UNAUTHORIZED`. +- Inspect logs on the remote host during remote debugging. Client-side `--debug` does not tail a local daemon log once `AGENT_DEVICE_DAEMON_BASE_URL` is set. +- Do not point `AGENT_DEVICE_DAEMON_BASE_URL` at untrusted hosts. Remote daemon requests can launch apps and execute interaction commands. +- Treat daemon auth tokens and lease identifiers as sensitive operational data. diff --git a/.agents/skills/agent-device/references/verification.md b/.agents/skills/agent-device/references/verification.md new file mode 100644 index 000000000000..8a38ad4d2c3c --- /dev/null +++ b/.agents/skills/agent-device/references/verification.md @@ -0,0 +1,111 @@ +# Verification + +## When to open this file + +Open this file when the task needs evidence, regression checks, replay maintenance, or session performance measurements after the main interaction flow is already working. + +## Main commands to reach for first + +- `screenshot` +- `diff snapshot` +- `record` +- `replay -u` +- `perf` + +## Most common mistake to avoid + +Do not use verification tools as the first exploration step. First get the app into the correct state with the normal interaction flow, then capture proof or maintain replay assets. + +## Canonical loop + +```bash +agent-device open Settings --platform ios +# after using exploration to reach the state you want to verify +agent-device snapshot +agent-device screenshot /tmp/settings-proof.png --overlay-refs +agent-device close +``` + +## Structural verification with diff snapshot + +Use `diff snapshot` when you need a compact view of how the UI changed between nearby states. + +```bash +agent-device snapshot -i +agent-device press @e5 +agent-device diff snapshot -i +``` + +- Initialize the baseline at a stable point. +- Perform the mutation. +- Run `diff snapshot` to confirm the expected structural change. +- Re-run full `snapshot` only when you need fresh refs. + +## Visual artifacts + +Use `screenshot` when the proof needs a rendered image instead of a structural tree. + +- Add `--overlay-refs` when you want the saved PNG to show fresh `@eN` refs burned into the screenshot. + +## Session recording + +Use `record` for debugging, documentation, or shareable verification artifacts. + +```bash +agent-device record start ./recordings/ios.mov +agent-device open App +agent-device snapshot -i +agent-device press @e3 +agent-device close +agent-device record stop +``` + +- `record` supports iOS simulators, iOS devices, and Android. +- On iOS, recording is a wrapper around `simctl` for simulators and the corresponding device capture path for physical devices. +- On Android, recording is a wrapper around `adb`. +- Recording writes a video artifact and a gesture-telemetry sidecar JSON. +- On macOS hosts, touch overlay burn-in is available for supported recordings. +- On non-macOS hosts, recording still succeeds but the video stays raw and `record stop` can return an `overlayWarning`. +- If the agent already knows the interaction sequence and wants a more lifelike, uninterrupted recording, drive the flow with `batch` while recording instead of replanning between each step. + +Example: + +```bash +agent-device record start ./recordings/smoke.mov +agent-device batch --session sim --platform ios --steps-file /tmp/smoke-steps.json --json +agent-device record stop +``` + +- Use this only after exploration has stabilized the flow. +- Keep the batch short and add `wait` or `is exists` guards after mutating steps so the recorded flow still tracks realistic UI timing. + +## Replay maintenance + +Use replay updates when selectors drift but the recorded scenario is still correct. + +```bash +agent-device replay -u ./session.ad +agent-device test ./smoke --platform android +``` + +- Prefer selector-based actions in recorded `.ad` replays. +- Use `test` when you already have multiple `.ad` flows and need a quick regression pass after updating or recording them. +- Keep the skill-level rule simple: use `replay -u` to maintain one script, use `test` to verify a folder or matcher of scripts. +- Treat `test` as a human and CI-facing suite runner that an agent can invoke for verification, not as the main source of product documentation. +- Failed runs keep suite artifacts under `.agent-device/test-artifacts` by default, which is usually enough for debugging without extra agent-side processing. +- Use update mode for maintenance, not as a substitute for fixing a broken interaction strategy. + +## Performance checks + +Use `perf --json` or `metrics --json` when you need session performance data for the active session. + +```bash +agent-device open Settings --platform ios +agent-device perf --json +``` + +- `startup` is command round-trip timing around `open`. +- It is not true first-frame or first-interactive telemetry. +- Android app sessions also expose `memory` (`dumpsys meminfo`) and `cpu` (`dumpsys cpuinfo`) snapshots when the session has an app package context. +- Apple app sessions on macOS and iOS simulators also expose `memory` and `cpu` process snapshots when the session has an app bundle ID. +- `fps` is still unavailable, and physical iOS devices still leave `memory` and `cpu` unavailable in this release. diff --git a/.agents/skills/dogfood/SKILL.md b/.agents/skills/dogfood/SKILL.md new file mode 100644 index 000000000000..e5d99f4f48bd --- /dev/null +++ b/.agents/skills/dogfood/SKILL.md @@ -0,0 +1,183 @@ +--- +name: dogfood +description: 'Systematically explore and test a mobile app on iOS/Android with agent-device to find bugs, UX issues, and other problems. Use when asked to "dogfood", "QA", "exploratory test", "find issues", "bug hunt", or "test this app" on mobile. Produces a structured report with reproducible evidence: screenshots, optional repro videos, and detailed steps for every issue.' +allowed-tools: Bash(agent-device:*), Bash(npx agent-device:*) +--- + +# Dogfood (agent-device) + +Systematically explore a mobile app, find issues, and produce a report with full reproduction evidence for every finding. + +## Setup + +Only the **Target app** is required. Everything else has sensible defaults. + +| Parameter | Default | Example override | +| -------------------- | ----------------------------------------------------------- | -------------------------------------------- | +| **Target app** | _(required)_ | `Settings`, `com.example.app`, deep link URL | +| **Platform** | Infer from user context; otherwise ask (`ios` or `android`) | `--platform ios` | +| **Session name** | Slugified app/platform (for example `settings-ios`) | `--session my-session` | +| **Output directory** | `./dogfood-output/` | `Output directory: /tmp/mobile-qa` | +| **Scope** | Full app | `Focus on onboarding and profile` | +| **Authentication** | None | `Sign in to user@example.com` | + +If the user gives enough context to start, begin immediately with defaults. Ask follow-up only when a required detail is missing (for example platform or credentials). + +Prefer direct `agent-device` binary when available. + +## Workflow + +``` +1. Initialize Set up session, output dirs, report file +2. Launch/Auth Open app and sign in if needed +3. Orient Capture initial snapshot and map navigation +4. Explore Systematically test flows and states +5. Document Record reproducible evidence per issue +6. Wrap up Reconcile summary, close session +``` + +### 1. Initialize + +```bash +mkdir -p {OUTPUT_DIR}/screenshots {OUTPUT_DIR}/videos +cp {SKILL_DIR}/templates/dogfood-report-template.md {OUTPUT_DIR}/report.md +``` + +### 2. Launch/Auth + +Start a named session and launch target app: + +```bash +agent-device --session {SESSION} open {TARGET_APP} --platform {PLATFORM} +agent-device --session {SESSION} snapshot -i +``` + +If login is required: + +```bash +agent-device --session {SESSION} snapshot -i +agent-device --session {SESSION} fill @e1 "{EMAIL}" +agent-device --session {SESSION} fill @e2 "{PASSWORD}" +agent-device --session {SESSION} press @e3 +agent-device --session {SESSION} wait 1000 +agent-device --session {SESSION} snapshot -i +``` + +For OTP/email codes: ask the user, wait for input, then continue. + +### 3. Orient + +Capture initial evidence and navigation anchors: + +```bash +agent-device --session {SESSION} screenshot {OUTPUT_DIR}/screenshots/initial.png +agent-device --session {SESSION} snapshot -i +``` + +Map top-level navigation, tabs, and key workflows before deep testing. + +### 4. Explore + +Read [references/issue-taxonomy.md](references/issue-taxonomy.md) for severity/category calibration. + +Strategy: + +- Move through each major app area (tabs, drawers, settings pages). +- Test core journeys end-to-end (create, edit, delete, submit, recover). +- Validate edge states (empty/error/loading/offline/permissions denied). +- Use `diff snapshot -i` after UI transitions to avoid stale refs. +- Periodically capture `logs path` and inspect the app log when behavior looks suspicious. + +Useful commands per screen: + +```bash +agent-device --session {SESSION} snapshot -i +agent-device --session {SESSION} screenshot {OUTPUT_DIR}/screenshots/{screen-name}.png +agent-device --session {SESSION} appstate +agent-device --session {SESSION} logs path +``` + +### 5. Document Issues (Repro-First) + +Explore and document in one pass. When you find an issue, stop and fully capture evidence before continuing. + +#### Interactive/behavioral issues + +Use video + step screenshots: + +1. Start recording: + +```bash +agent-device --session {SESSION} record start {OUTPUT_DIR}/videos/issue-{NNN}-repro.mp4 +``` + +2. Reproduce with visible pacing. Capture each step: + +```bash +agent-device --session {SESSION} screenshot {OUTPUT_DIR}/screenshots/issue-{NNN}-step-1.png +sleep 1 +# perform action +sleep 1 +agent-device --session {SESSION} screenshot {OUTPUT_DIR}/screenshots/issue-{NNN}-step-2.png +``` + +3. Capture final broken state: + +```bash +sleep 2 +agent-device --session {SESSION} screenshot {OUTPUT_DIR}/screenshots/issue-{NNN}-result.png +``` + +4. Stop recording: + +```bash +agent-device --session {SESSION} record stop +``` + +5. Append issue immediately to report with numbered steps and screenshot references. + +#### Static/on-load issues + +Single screenshot is sufficient; no video required: + +```bash +agent-device --session {SESSION} screenshot {OUTPUT_DIR}/screenshots/issue-{NNN}.png +``` + +Set **Repro Video** to `N/A` in the report. + +### 6. Wrap Up + +Target 5-10 well-evidenced issues, then finish: + +1. Reconcile summary severity counts in `report.md`. +2. Close session: + +```bash +agent-device --session {SESSION} close +``` + +3. Report total issues, severity breakdown, and highest-risk findings. + +## Guidance + +- Repro quality matters more than issue count. +- Use refs (`@eN`) for fast exploration, selectors for deterministic replay assertions when needed. +- Re-snapshot after any mutation (navigation, modal, list update, form submit). +- Use `fill` for clear-then-type semantics; use `type` for incremental typing behavior checks. +- Keep logs optional and targeted: enable/read app logs only when useful for diagnosis. +- Never read source code of the app under test; findings must come from observed runtime behavior. +- Write each issue immediately to avoid losing evidence. +- Never delete screenshots/videos/report artifacts during a session. + +## References + +| Reference | When to Read | +| ------------------------------------------------------------ | ----------------------------------------------- | +| [references/issue-taxonomy.md](references/issue-taxonomy.md) | Start of session; severity/categories/checklist | + +## Templates + +| Template | Purpose | +| ---------------------------------------------------------------------------- | --------------------------------------------- | +| [templates/dogfood-report-template.md](templates/dogfood-report-template.md) | Copy into output directory as the report file | diff --git a/.agents/skills/dogfood/references/issue-taxonomy.md b/.agents/skills/dogfood/references/issue-taxonomy.md new file mode 100644 index 000000000000..27a678d6163d --- /dev/null +++ b/.agents/skills/dogfood/references/issue-taxonomy.md @@ -0,0 +1,83 @@ +# Issue Taxonomy (Mobile) + +Reference for categorizing issues found during mobile dogfooding. + +## Severity Levels + +| Severity | Definition | +| ------------ | ------------------------------------------------------------------------- | +| **critical** | Blocks a core workflow, causes data loss, or crashes/freeze loops the app | +| **high** | Major feature broken or unusable, no practical workaround | +| **medium** | Feature works with notable friction or partial failure; workaround exists | +| **low** | Minor cosmetic or polish issue | + +## Categories + +### Visual / UI + +- Layout broken, clipped, overlapped, or unreadable text +- Safe-area/notch overlap issues +- Incorrect dark/light appearance rendering +- Missing assets/icons +- Animation glitches or flicker + +### Functional + +- Buttons/controls do nothing or trigger wrong action +- Flows fail (create/edit/delete/submit) +- Navigation dead-ends or wrong destination +- State loss after background/foreground transitions +- Deep link opens wrong screen or fails + +### UX + +- Confusing hierarchy or navigation labels +- Missing loading/progress feedback +- Unclear error handling or no recovery affordance +- Excessive steps for common tasks +- Inconsistent behavior between similar screens + +### Content + +- Typos, incorrect copy, placeholder text +- Wrong labels/help text +- Truncated text with no affordance +- Inconsistent terminology across screens + +### Performance + +- Slow startup or route transitions +- Input lag or gesture jank +- Scroll hitches/frame drops +- Notable battery/thermal symptoms during basic usage + +### Diagnostics / Logs + +- Native crashes or repeated fatal exceptions +- Repeated warnings correlated with broken behavior +- Unhandled runtime errors visible during repro + +### Permissions / Platform + +- Permission prompt flow broken or loops forever +- Denied permissions not handled gracefully +- Platform-specific regressions (iOS-only or Android-only) +- Background/foreground lifecycle regressions + +### Accessibility + +- Missing labels or incorrect accessibility names +- Focus order/navigation issues for assistive tech +- Low contrast or unreadable text scaling +- Touch targets too small for reliable interaction + +## Exploration Checklist + +1. Visual scan: capture screenshot; verify layout/safe areas/text/icon rendering. +2. Interactions: press controls, open menus/modals, validate expected response. +3. Forms/input: test valid/invalid/empty/boundary input. +4. Navigation: traverse all top-level sections and return paths. +5. App states: loading/empty/error/offline/permission-denied/background-resume. +6. Logs/diagnostics: inspect app logs when behavior is suspicious. +7. Platform parity: verify critical flows on each requested platform. +8. Accessibility basics: labels, touch target sizes, readability/contrast. diff --git a/.agents/skills/dogfood/templates/dogfood-report-template.md b/.agents/skills/dogfood/templates/dogfood-report-template.md new file mode 100644 index 000000000000..fd11566a6d6a --- /dev/null +++ b/.agents/skills/dogfood/templates/dogfood-report-template.md @@ -0,0 +1,52 @@ +# Dogfood Report: {APP_NAME} + +| Field | Value | +| -------------- | -------------- | +| **Date** | {DATE} | +| **Platform** | {PLATFORM} | +| **Target App** | {TARGET_APP} | +| **Session** | {SESSION_NAME} | +| **Scope** | {SCOPE} | + +## Summary + +| Severity | Count | +| --------- | ----- | +| Critical | 0 | +| High | 0 | +| Medium | 0 | +| Low | 0 | +| **Total** | **0** | + +## Issues + + + +### ISSUE-001: {Short title} + +| Field | Value | +| ------------------ | -------------------------------------------------------------------------------------------- | +| **Severity** | critical / high / medium / low | +| **Category** | visual / functional / ux / content / performance / diagnostics / permissions / accessibility | +| **Screen / Route** | {screen where issue was found} | +| **Repro Video** | {path to video, or N/A for static issues} | + +**Description** + +{What is wrong, what was expected, and what actually happened.} + +**Repro Steps** + +1. Open {screen/entry point} + ![Step 1](screenshots/issue-001-step-1.png) + +2. {Action} + ![Step 2](screenshots/issue-001-step-2.png) + +3. {Action} + ![Step 3](screenshots/issue-001-step-3.png) + +4. **Observe:** {broken behavior} + ![Result](screenshots/issue-001-result.png) + +--- diff --git a/.claude/skills/agent-device b/.claude/skills/agent-device new file mode 120000 index 000000000000..2d6563c39f63 --- /dev/null +++ b/.claude/skills/agent-device @@ -0,0 +1 @@ +../../.agents/skills/agent-device \ No newline at end of file diff --git a/.claude/skills/agent-device-app-testing/SKILL.md b/.claude/skills/agent-device-app-testing/SKILL.md new file mode 100644 index 000000000000..54fcb428ef0a --- /dev/null +++ b/.claude/skills/agent-device-app-testing/SKILL.md @@ -0,0 +1,114 @@ +--- +name: agent-device-app-testing +description: Test the Expensify App on Android/iOS emulators using agent-device. Use after mobile/React Native code changes, for debugging mobile UI issues, or when user requests mobile testing. +alwaysApply: false +--- + +# Mobile App Testing with agent-device + +## When to Use This Skill + +Use mobile device testing when: +- User requests testing the App on Android or iOS emulator/simulator +- Verifying fixes or improvements to mobile/React Native UI code +- Debugging mobile-specific issues (navigation, gestures, keyboard, native modules) +- Reproducing bugs that only appear on mobile + +**Proactively use after making mobile or React Native code changes** to verify your work functions correctly on device. + +## When NOT to Use This Skill + +Skip mobile device testing for: +- Web-only changes (use playwright-app-testing instead) +- Unit tests (use `npm run test`) +- Type checking (use `npm run typecheck`) +- Backend/API-only changes + +## App Details + +- **Package name**: `com.expensify.chat` +- **Screenshot directory**: `./agent-device-output/` + +## Dev Environment Sign-In + +When signing in on the emulator/simulator: +- **Email**: Generate a random Gmail address (e.g., `testuser+@gmail.com`) +- **Magic code**: Always `000000` +- **Onboarding**: Check `SKIP_ONBOARDING` in `.env` - set to `true` unless specifically testing onboarding flows + +## Workflow + +The base `agent-device` skill handles all device interaction commands. This skill provides App-specific context only. + +### 1. Boot and install + +```bash +# Android +agent-device boot --platform android +APK_PATH=$(find .rock/cache/android -name "*.apk" -print -quit) +agent-device reinstall com.expensify.chat "$APK_PATH" --platform android + +# iOS +agent-device boot --platform ios +IPA_PATH=$(find .rock/cache/ios -name "*.app" -print -quit) +agent-device reinstall com.expensify.chat "$IPA_PATH" --platform ios +``` + +### 2. Open app and start session + +```bash +agent-device open com.expensify.chat --platform android --session test +``` + +### 3. Sign in + +```bash +# Snapshot to see login screen +agent-device snapshot -i --session test + +# Fill email and continue +agent-device fill @ "testuser+@gmail.com" --session test +agent-device press @ --session test + +# Enter magic code +agent-device snapshot -i --session test +agent-device fill @ "000000" --session test +``` + +### 4. Navigate, interact, verify + +Use the standard agent-device snapshot/interact loop: +- `snapshot` to read the screen +- `snapshot -i` when you need interactive refs +- `press`, `fill`, `type`, `scroll` to interact +- Re-snapshot after every meaningful UI change + +### 5. Capture evidence and shutdown + +```bash +mkdir -p ./agent-device-output +agent-device screenshot ./agent-device-output/-.png --session test +agent-device close --shutdown --session test +``` + +## App-Specific Notes + +- **React Native dev overlays**: After opening, check for warning/error overlays. Dismiss if not blocking; report if recurring. +- **Keyboard**: Use `keyboard dismiss` if the on-screen keyboard blocks controls you need to interact with. +- **Navigation**: The app uses React Navigation - screen transitions may cause brief accessibility tree lag. If `snapshot` doesn't match expectations, take a `screenshot` for visual truth, then re-snapshot. + +## Example Usage + +``` +Scenario 1: User requests mobile testing +User: "Test the expense creation flow on Android" +-> Boot emulator, install app, sign in, test the flow + +Scenario 2: After making RN changes +You: "I've updated the expense input component" +-> Proactively test the change on emulator to verify + +Scenario 3: Mobile-specific bug +User: "The keyboard covers the submit button on iOS" +-> Boot iOS simulator, reproduce and verify the issue +``` diff --git a/.claude/skills/dogfood b/.claude/skills/dogfood new file mode 120000 index 000000000000..313d8dedc0e3 --- /dev/null +++ b/.claude/skills/dogfood @@ -0,0 +1 @@ +../../.agents/skills/dogfood \ No newline at end of file diff --git a/skills-lock.json b/skills-lock.json new file mode 100644 index 000000000000..4b02e3033f7d --- /dev/null +++ b/skills-lock.json @@ -0,0 +1,15 @@ +{ + "version": 1, + "skills": { + "agent-device": { + "source": "callstackincubator/agent-device", + "sourceType": "github", + "computedHash": "8c43b5cd5db26331396dfe93089bd70e259f4eebdff0aa2fbb6bf863c14b99c0" + }, + "dogfood": { + "source": "callstackincubator/agent-device", + "sourceType": "github", + "computedHash": "47e0ea18e07bbe14fc7547b3f79f9f4f1bd895e6d7cbb514055076d78c6ba9d0" + } + } +} From 4bfce77d66425b67d80c030a57cf8b4e7bfa219b Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Kacper=20Miko=C5=82ajczak?= Date: Tue, 14 Apr 2026 12:33:58 +0200 Subject: [PATCH 02/14] Address PR review findings for agent-device skills - Flatten .agents/skills/ into .claude/skills/ (remove symlink indirection and skills-lock.json created by `npx skills add`) - Add CLI prerequisites section to wrapper skill - Replace .rock/cache/ CI paths with local build as primary flow - Add agent-device-output/ to .gitignore - Fix email pattern and dev/release package names - Tighten trigger scope to explicit user requests only - Reduce verbosity per reviewer feedback --- .claude/skills/agent-device | 1 - .../skills/agent-device-app-testing/SKILL.md | 100 +++++++++--------- .../skills/agent-device/SKILL.md | 0 .../references/bootstrap-install.md | 0 .../references/coordinate-system.md | 0 .../agent-device/references/debugging.md | 0 .../agent-device/references/exploration.md | 0 .../agent-device/references/macos-desktop.md | 0 .../agent-device/references/remote-tenancy.md | 0 .../agent-device/references/verification.md | 0 .claude/skills/dogfood | 1 - {.agents => .claude}/skills/dogfood/SKILL.md | 0 .../dogfood/references/issue-taxonomy.md | 0 .../templates/dogfood-report-template.md | 0 .gitignore | 3 + skills-lock.json | 15 --- 16 files changed, 54 insertions(+), 66 deletions(-) delete mode 120000 .claude/skills/agent-device rename {.agents => .claude}/skills/agent-device/SKILL.md (100%) rename {.agents => .claude}/skills/agent-device/references/bootstrap-install.md (100%) rename {.agents => .claude}/skills/agent-device/references/coordinate-system.md (100%) rename {.agents => .claude}/skills/agent-device/references/debugging.md (100%) rename {.agents => .claude}/skills/agent-device/references/exploration.md (100%) rename {.agents => .claude}/skills/agent-device/references/macos-desktop.md (100%) rename {.agents => .claude}/skills/agent-device/references/remote-tenancy.md (100%) rename {.agents => .claude}/skills/agent-device/references/verification.md (100%) delete mode 120000 .claude/skills/dogfood rename {.agents => .claude}/skills/dogfood/SKILL.md (100%) rename {.agents => .claude}/skills/dogfood/references/issue-taxonomy.md (100%) rename {.agents => .claude}/skills/dogfood/templates/dogfood-report-template.md (100%) delete mode 100644 skills-lock.json diff --git a/.claude/skills/agent-device b/.claude/skills/agent-device deleted file mode 120000 index 2d6563c39f63..000000000000 --- a/.claude/skills/agent-device +++ /dev/null @@ -1 +0,0 @@ -../../.agents/skills/agent-device \ No newline at end of file diff --git a/.claude/skills/agent-device-app-testing/SKILL.md b/.claude/skills/agent-device-app-testing/SKILL.md index 54fcb428ef0a..5288c0908c81 100644 --- a/.claude/skills/agent-device-app-testing/SKILL.md +++ b/.claude/skills/agent-device-app-testing/SKILL.md @@ -1,73 +1,91 @@ --- name: agent-device-app-testing -description: Test the Expensify App on Android/iOS emulators using agent-device. Use after mobile/React Native code changes, for debugging mobile UI issues, or when user requests mobile testing. +description: Test the Expensify App on Android/iOS emulators using agent-device. Use when user explicitly requests mobile device testing, or when debugging a mobile-only issue that cannot be verified any other way. alwaysApply: false --- # Mobile App Testing with agent-device -## When to Use This Skill +## Prerequisites -Use mobile device testing when: -- User requests testing the App on Android or iOS emulator/simulator -- Verifying fixes or improvements to mobile/React Native UI code -- Debugging mobile-specific issues (navigation, gestures, keyboard, native modules) -- Reproducing bugs that only appear on mobile +The `agent-device` CLI must be installed on the host machine. It is **not** bundled with this repo. -**Proactively use after making mobile or React Native code changes** to verify your work functions correctly on device. +```bash +# Install (requires Node 20+) +npm install -g agent-device + +# Verify +agent-device --version +``` -## When NOT to Use This Skill +For Android testing, the Android SDK and an AVD (emulator image) must be available. For iOS, Xcode and a simulator runtime are required. -Skip mobile device testing for: -- Web-only changes (use playwright-app-testing instead) -- Unit tests (use `npm run test`) -- Type checking (use `npm run typecheck`) -- Backend/API-only changes +## When to Use + +- User explicitly requests testing the App on an Android emulator or iOS simulator +- Debugging a mobile-only issue (gestures, keyboard, native modules, navigation) +- Reproducing a bug that only appears on device + +Do **not** use for web-only changes (use `playwright-app-testing`), unit tests (`npm run test`), or type checking (`npm run typecheck`). ## App Details -- **Package name**: `com.expensify.chat` -- **Screenshot directory**: `./agent-device-output/` +| Key | Value | +|---|---| +| Android package | `com.expensify.chat.dev` (dev) / `com.expensify.chat` (release) | +| iOS bundle ID | `com.expensify.chat.dev` (dev) / `com.expensify.chat` (release) | +| Screenshot dir | `./agent-device-output/` | ## Dev Environment Sign-In -When signing in on the emulator/simulator: -- **Email**: Generate a random Gmail address (e.g., `testuser+@gmail.com`) -- **Magic code**: Always `000000` -- **Onboarding**: Check `SKIP_ONBOARDING` in `.env` - set to `true` unless specifically testing onboarding flows +- **Email**: Use `@expensify.com` or any address you control on the dev server +- **Magic code**: `000000` (always works on dev) +- **Onboarding**: Set `SKIP_ONBOARDING=true` in `.env` unless specifically testing onboarding ## Workflow -The base `agent-device` skill handles all device interaction commands. This skill provides App-specific context only. +The base `agent-device` skill handles all device interaction commands. This skill adds Expensify App context only. + +### 1. Boot emulator and build/install -### 1. Boot and install +**Local build (primary)** - builds and installs in one step: ```bash # Android agent-device boot --platform android -APK_PATH=$(find .rock/cache/android -name "*.apk" -print -quit) -agent-device reinstall com.expensify.chat "$APK_PATH" --platform android +npx react-native run-android --mode developmentDebug # iOS agent-device boot --platform ios -IPA_PATH=$(find .rock/cache/ios -name "*.app" -print -quit) -agent-device reinstall com.expensify.chat "$IPA_PATH" --platform ios +npx react-native run-ios --mode Development-Debug --simulator "iPhone 16 Pro" +``` + +**Pre-built artifact (optional)** - if you have an APK/IPA from CI or Rock cache: + +```bash +# Android +agent-device boot --platform android +agent-device reinstall com.expensify.chat.dev ./path/to/app.apk --platform android + +# iOS +agent-device boot --platform ios +agent-device reinstall com.expensify.chat.dev ./path/to/App.app --platform ios ``` ### 2. Open app and start session ```bash -agent-device open com.expensify.chat --platform android --session test +agent-device open com.expensify.chat.dev --platform android --session test ``` ### 3. Sign in ```bash -# Snapshot to see login screen +# Read login screen agent-device snapshot -i --session test # Fill email and continue -agent-device fill @ "testuser+@gmail.com" --session test +agent-device fill @ "testuser@expensify.com" --session test agent-device press @ --session test # Enter magic code @@ -75,7 +93,7 @@ agent-device snapshot -i --session test agent-device fill @ "000000" --session test ``` -### 4. Navigate, interact, verify +### 4. Navigate and interact Use the standard agent-device snapshot/interact loop: - `snapshot` to read the screen @@ -93,22 +111,6 @@ agent-device close --shutdown --session test ## App-Specific Notes -- **React Native dev overlays**: After opening, check for warning/error overlays. Dismiss if not blocking; report if recurring. -- **Keyboard**: Use `keyboard dismiss` if the on-screen keyboard blocks controls you need to interact with. -- **Navigation**: The app uses React Navigation - screen transitions may cause brief accessibility tree lag. If `snapshot` doesn't match expectations, take a `screenshot` for visual truth, then re-snapshot. - -## Example Usage - -``` -Scenario 1: User requests mobile testing -User: "Test the expense creation flow on Android" --> Boot emulator, install app, sign in, test the flow - -Scenario 2: After making RN changes -You: "I've updated the expense input component" --> Proactively test the change on emulator to verify - -Scenario 3: Mobile-specific bug -User: "The keyboard covers the submit button on iOS" --> Boot iOS simulator, reproduce and verify the issue -``` +- **RN dev overlays**: Dismiss warning/error overlays if not blocking; report if recurring. +- **Keyboard**: Use `keyboard dismiss` if the on-screen keyboard blocks controls. +- **Navigation**: React Navigation transitions may cause brief accessibility tree lag. If `snapshot` doesn't match, take a `screenshot` for visual truth, then re-snapshot. diff --git a/.agents/skills/agent-device/SKILL.md b/.claude/skills/agent-device/SKILL.md similarity index 100% rename from .agents/skills/agent-device/SKILL.md rename to .claude/skills/agent-device/SKILL.md diff --git a/.agents/skills/agent-device/references/bootstrap-install.md b/.claude/skills/agent-device/references/bootstrap-install.md similarity index 100% rename from .agents/skills/agent-device/references/bootstrap-install.md rename to .claude/skills/agent-device/references/bootstrap-install.md diff --git a/.agents/skills/agent-device/references/coordinate-system.md b/.claude/skills/agent-device/references/coordinate-system.md similarity index 100% rename from .agents/skills/agent-device/references/coordinate-system.md rename to .claude/skills/agent-device/references/coordinate-system.md diff --git a/.agents/skills/agent-device/references/debugging.md b/.claude/skills/agent-device/references/debugging.md similarity index 100% rename from .agents/skills/agent-device/references/debugging.md rename to .claude/skills/agent-device/references/debugging.md diff --git a/.agents/skills/agent-device/references/exploration.md b/.claude/skills/agent-device/references/exploration.md similarity index 100% rename from .agents/skills/agent-device/references/exploration.md rename to .claude/skills/agent-device/references/exploration.md diff --git a/.agents/skills/agent-device/references/macos-desktop.md b/.claude/skills/agent-device/references/macos-desktop.md similarity index 100% rename from .agents/skills/agent-device/references/macos-desktop.md rename to .claude/skills/agent-device/references/macos-desktop.md diff --git a/.agents/skills/agent-device/references/remote-tenancy.md b/.claude/skills/agent-device/references/remote-tenancy.md similarity index 100% rename from .agents/skills/agent-device/references/remote-tenancy.md rename to .claude/skills/agent-device/references/remote-tenancy.md diff --git a/.agents/skills/agent-device/references/verification.md b/.claude/skills/agent-device/references/verification.md similarity index 100% rename from .agents/skills/agent-device/references/verification.md rename to .claude/skills/agent-device/references/verification.md diff --git a/.claude/skills/dogfood b/.claude/skills/dogfood deleted file mode 120000 index 313d8dedc0e3..000000000000 --- a/.claude/skills/dogfood +++ /dev/null @@ -1 +0,0 @@ -../../.agents/skills/dogfood \ No newline at end of file diff --git a/.agents/skills/dogfood/SKILL.md b/.claude/skills/dogfood/SKILL.md similarity index 100% rename from .agents/skills/dogfood/SKILL.md rename to .claude/skills/dogfood/SKILL.md diff --git a/.agents/skills/dogfood/references/issue-taxonomy.md b/.claude/skills/dogfood/references/issue-taxonomy.md similarity index 100% rename from .agents/skills/dogfood/references/issue-taxonomy.md rename to .claude/skills/dogfood/references/issue-taxonomy.md diff --git a/.agents/skills/dogfood/templates/dogfood-report-template.md b/.claude/skills/dogfood/templates/dogfood-report-template.md similarity index 100% rename from .agents/skills/dogfood/templates/dogfood-report-template.md rename to .claude/skills/dogfood/templates/dogfood-report-template.md diff --git a/.gitignore b/.gitignore index 79175c075a16..6775d4e4fe81 100644 --- a/.gitignore +++ b/.gitignore @@ -162,5 +162,8 @@ modules/*/lib/ .playwright-output/ .playwright-mcp/ +# agent-device +agent-device-output/ + # cspell cache .cspellcache diff --git a/skills-lock.json b/skills-lock.json deleted file mode 100644 index 4b02e3033f7d..000000000000 --- a/skills-lock.json +++ /dev/null @@ -1,15 +0,0 @@ -{ - "version": 1, - "skills": { - "agent-device": { - "source": "callstackincubator/agent-device", - "sourceType": "github", - "computedHash": "8c43b5cd5db26331396dfe93089bd70e259f4eebdff0aa2fbb6bf863c14b99c0" - }, - "dogfood": { - "source": "callstackincubator/agent-device", - "sourceType": "github", - "computedHash": "47e0ea18e07bbe14fc7547b3f79f9f4f1bd895e6d7cbb514055076d78c6ba9d0" - } - } -} From c0d5618e1f98b63064a72e70b668fa8304b4c727 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Kacper=20Miko=C5=82ajczak?= Date: Tue, 14 Apr 2026 12:45:05 +0200 Subject: [PATCH 03/14] Trim wrapper skill to user-directed context layer Per Jules's comment: local testing is directed by user, not prescribed by the skill. Remove step-by-step workflow - the base agent-device skill handles interaction. Keep only the App-specific facts that avoid repetitive lookups (package names, build commands, sign-in creds, RN gotchas). --- .../skills/agent-device-app-testing/SKILL.md | 88 +++---------------- 1 file changed, 13 insertions(+), 75 deletions(-) diff --git a/.claude/skills/agent-device-app-testing/SKILL.md b/.claude/skills/agent-device-app-testing/SKILL.md index 5288c0908c81..9863e778a8b5 100644 --- a/.claude/skills/agent-device-app-testing/SKILL.md +++ b/.claude/skills/agent-device-app-testing/SKILL.md @@ -1,32 +1,23 @@ --- name: agent-device-app-testing -description: Test the Expensify App on Android/iOS emulators using agent-device. Use when user explicitly requests mobile device testing, or when debugging a mobile-only issue that cannot be verified any other way. +description: Test the Expensify App on Android/iOS emulators using agent-device. Use only when user explicitly requests mobile device testing. alwaysApply: false --- # Mobile App Testing with agent-device +Expensify App context for the base `agent-device` skill. The user directs what to test - this skill provides App-specific details so you don't have to look them up each time. + ## Prerequisites The `agent-device` CLI must be installed on the host machine. It is **not** bundled with this repo. ```bash -# Install (requires Node 20+) npm install -g agent-device - -# Verify agent-device --version ``` -For Android testing, the Android SDK and an AVD (emulator image) must be available. For iOS, Xcode and a simulator runtime are required. - -## When to Use - -- User explicitly requests testing the App on an Android emulator or iOS simulator -- Debugging a mobile-only issue (gestures, keyboard, native modules, navigation) -- Reproducing a bug that only appears on device - -Do **not** use for web-only changes (use `playwright-app-testing`), unit tests (`npm run test`), or type checking (`npm run typecheck`). +Android testing requires the Android SDK + an AVD. iOS requires Xcode + a simulator runtime. ## App Details @@ -36,81 +27,28 @@ Do **not** use for web-only changes (use `playwright-app-testing`), unit tests ( | iOS bundle ID | `com.expensify.chat.dev` (dev) / `com.expensify.chat` (release) | | Screenshot dir | `./agent-device-output/` | -## Dev Environment Sign-In - -- **Email**: Use `@expensify.com` or any address you control on the dev server -- **Magic code**: `000000` (always works on dev) -- **Onboarding**: Set `SKIP_ONBOARDING=true` in `.env` unless specifically testing onboarding - -## Workflow +## Building and Installing -The base `agent-device` skill handles all device interaction commands. This skill adds Expensify App context only. - -### 1. Boot emulator and build/install - -**Local build (primary)** - builds and installs in one step: +**Local build** (builds and installs in one step): ```bash # Android -agent-device boot --platform android npx react-native run-android --mode developmentDebug # iOS -agent-device boot --platform ios npx react-native run-ios --mode Development-Debug --simulator "iPhone 16 Pro" ``` -**Pre-built artifact (optional)** - if you have an APK/IPA from CI or Rock cache: - -```bash -# Android -agent-device boot --platform android -agent-device reinstall com.expensify.chat.dev ./path/to/app.apk --platform android - -# iOS -agent-device boot --platform ios -agent-device reinstall com.expensify.chat.dev ./path/to/App.app --platform ios -``` +If you have a pre-built APK/IPA (from CI, Rock cache, etc.), use `agent-device reinstall` with the package name above. -### 2. Open app and start session - -```bash -agent-device open com.expensify.chat.dev --platform android --session test -``` - -### 3. Sign in - -```bash -# Read login screen -agent-device snapshot -i --session test - -# Fill email and continue -agent-device fill @ "testuser@expensify.com" --session test -agent-device press @ --session test - -# Enter magic code -agent-device snapshot -i --session test -agent-device fill @ "000000" --session test -``` - -### 4. Navigate and interact - -Use the standard agent-device snapshot/interact loop: -- `snapshot` to read the screen -- `snapshot -i` when you need interactive refs -- `press`, `fill`, `type`, `scroll` to interact -- Re-snapshot after every meaningful UI change - -### 5. Capture evidence and shutdown +## Dev Environment Sign-In -```bash -mkdir -p ./agent-device-output -agent-device screenshot ./agent-device-output/-.png --session test -agent-device close --shutdown --session test -``` +- **Email**: Any address works on dev (e.g. `testuser@expensify.com`) +- **Magic code**: `000000` +- **Onboarding**: Set `SKIP_ONBOARDING=true` in `.env` to bypass -## App-Specific Notes +## Notes - **RN dev overlays**: Dismiss warning/error overlays if not blocking; report if recurring. - **Keyboard**: Use `keyboard dismiss` if the on-screen keyboard blocks controls. -- **Navigation**: React Navigation transitions may cause brief accessibility tree lag. If `snapshot` doesn't match, take a `screenshot` for visual truth, then re-snapshot. +- **Navigation**: React Navigation transitions may cause brief accessibility tree lag. If `snapshot` is stale, take a `screenshot` for visual truth, then re-snapshot. From a63542bba3881dbfcedc0244edded0f181a1116d Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Kacper=20Miko=C5=82ajczak?= Date: Tue, 14 Apr 2026 13:54:21 +0200 Subject: [PATCH 04/14] Update build instructions to Rock workflow, HybridApp only --- .claude/skills/agent-device-app-testing/SKILL.md | 13 +++++-------- 1 file changed, 5 insertions(+), 8 deletions(-) diff --git a/.claude/skills/agent-device-app-testing/SKILL.md b/.claude/skills/agent-device-app-testing/SKILL.md index 9863e778a8b5..067e399b72b8 100644 --- a/.claude/skills/agent-device-app-testing/SKILL.md +++ b/.claude/skills/agent-device-app-testing/SKILL.md @@ -1,7 +1,6 @@ --- name: agent-device-app-testing description: Test the Expensify App on Android/iOS emulators using agent-device. Use only when user explicitly requests mobile device testing. -alwaysApply: false --- # Mobile App Testing with agent-device @@ -29,17 +28,15 @@ Android testing requires the Android SDK + an AVD. iOS requires Xcode + a simula ## Building and Installing -**Local build** (builds and installs in one step): +The project uses [Rock](https://rockjs.dev/) which downloads pre-built artifacts from S3 first, falling back to local compilation if no match. Metro must be running in a separate terminal. ```bash -# Android -npx react-native run-android --mode developmentDebug - -# iOS -npx react-native run-ios --mode Development-Debug --simulator "iPhone 16 Pro" +npm install +npm run start # metro - separate terminal +npm run android # or: npm run ios ``` -If you have a pre-built APK/IPA (from CI, Rock cache, etc.), use `agent-device reinstall` with the package name above. +Full setup details: [SETUP_ANDROID.md](contributingGuides/SETUP_ANDROID.md) | [SETUP_IOS.md](contributingGuides/SETUP_IOS.md) ## Dev Environment Sign-In From 7af0dc555b31dd087b56cf21cd598b58d3995ebe Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Kacper=20Miko=C5=82ajczak?= Date: Tue, 14 Apr 2026 13:55:45 +0200 Subject: [PATCH 05/14] Drop dogfood skill, macos-desktop and remote-tenancy references Reduces context overhead for the PoC. dogfood (autonomous QA) is better suited for Phase 2/Melvin. macOS desktop and remote tenancy references are not relevant for local mobile testing. --- .claude/skills/agent-device/SKILL.md | 3 - .../agent-device/references/macos-desktop.md | 87 --------- .../agent-device/references/remote-tenancy.md | 99 ---------- .claude/skills/dogfood/SKILL.md | 183 ------------------ .../dogfood/references/issue-taxonomy.md | 83 -------- .../templates/dogfood-report-template.md | 52 ----- 6 files changed, 507 deletions(-) delete mode 100644 .claude/skills/agent-device/references/macos-desktop.md delete mode 100644 .claude/skills/agent-device/references/remote-tenancy.md delete mode 100644 .claude/skills/dogfood/SKILL.md delete mode 100644 .claude/skills/dogfood/references/issue-taxonomy.md delete mode 100644 .claude/skills/dogfood/templates/dogfood-report-template.md diff --git a/.claude/skills/agent-device/SKILL.md b/.claude/skills/agent-device/SKILL.md index 91a5068e5b1d..05d2058c4477 100644 --- a/.claude/skills/agent-device/SKILL.md +++ b/.claude/skills/agent-device/SKILL.md @@ -35,7 +35,6 @@ Use this skill as a router with mandatory defaults. Read this file first. For no ## QA modes -- Open-ended bug hunt with reporting: use [../dogfood/SKILL.md](../dogfood/SKILL.md). - Pass/fail QA from acceptance criteria: stay in this skill, start with [references/bootstrap-install.md](references/bootstrap-install.md), then use the QA loop in [references/exploration.md](references/exploration.md). ## Required references @@ -68,5 +67,3 @@ Use this skill as a router with mandatory defaults. Read this file first. For no - Need logs, network, alerts, permissions, or failure triage: [references/debugging.md](references/debugging.md) - Need screenshots, diff, recording, replay maintenance, or perf data: [references/verification.md](references/verification.md) -- Need desktop surfaces, menu bar behavior, or macOS-specific interaction rules: [references/macos-desktop.md](references/macos-desktop.md) -- Need remote HTTP transport, `--remote-config` launches, or tenant leases on a remote macOS host: [references/remote-tenancy.md](references/remote-tenancy.md) diff --git a/.claude/skills/agent-device/references/macos-desktop.md b/.claude/skills/agent-device/references/macos-desktop.md deleted file mode 100644 index 4c63c6405b94..000000000000 --- a/.claude/skills/agent-device/references/macos-desktop.md +++ /dev/null @@ -1,87 +0,0 @@ -# macOS Desktop - -## When to open this file - -Open this file only when `--platform macos` is involved or the task needs `frontmost-app`, `desktop`, or `menubar` surfaces. - -## Main commands to reach for first - -- `open --platform macos` -- `open --platform macos --surface frontmost-app|desktop|menubar` -- `snapshot -i` -- `get` -- `is` -- `click --button secondary` - -## Most common mistake to avoid - -Do not treat every macOS surface the same. Use the normal `app` surface when you want to act inside one app. Use `frontmost-app`, `desktop`, or `menubar` mainly to inspect what is visible before switching back to `app` for most interactions. - -## Canonical loop - -```bash -agent-device open TextEdit --platform macos -agent-device snapshot -agent-device close -``` - -## Surface rules - -- `app`: default surface and the normal choice for `click`, `fill`, `press`, `scroll`, `screenshot`, and `record`. -- `frontmost-app`: inspect the currently focused app without naming it first. -- `desktop`: inspect visible desktop windows across apps. -- `menubar`: inspect the active app menu bar and system menu extras. Use `open --platform macos --surface menubar` when you need one menu bar app's extras, such as a status-item app. -- Menu bar apps can expose a sparse or empty default `app` tree. Prefer the `menubar` surface first when the app lives entirely in the top bar. - -Use inspect-first surfaces to understand desktop-global UI, then switch back to `app` when you need to act in one app. - -## Snapshot expectations - -- `snapshot -i` should describe UI visible to a human. -- `desktop` snapshots can include multiple windows from multiple apps. -- `menubar` snapshots can include both app-menu items and system menu extras. -- Finder-style rows, sidebar items, toolbar controls, search fields, and opened context menus should appear when visible. -- Finder and other native apps may expose duplicate-looking row, cell, and child text nodes. Treat them as distinct AX nodes unless you have a stronger selector anchor. - -## Context menus - -Context menus are not ambient UI. Open them explicitly, then re-snapshot. - -```bash -agent-device click @e66 --button secondary --platform macos -agent-device snapshot -i -``` - -Expected loop: - -1. Snapshot visible content. -2. Secondary-click the target item. -3. Snapshot again. -4. Interact with the new `menu-item` nodes. - -## Targeting rules - -- Prefer selectors or `@ref` values over raw coordinates. -- On macOS, window position can vary across runs, so coordinate-only flows are fragile. -- If the task only needs shared exploration rules, return to [exploration.md](exploration.md). - -Selector guidance: - -- Good selectors usually anchor on stable labels or app-owned identifiers such as `label="Downloads"` or `role=menu-item label="Rename"`. -- Avoid relying on framework-generated `_NS:*` identifiers as stable selectors. - -Use `snapshot --raw --platform macos` only when debugging AX structure or collector filtering. Do not make raw snapshots the default agent loop. - -Things not to rely on: - -- Mobile-only helpers such as `install`, `reinstall`, or `push`. -- Desktop-global click or fill parity from `desktop` or `menubar` sessions. -- Raw coordinate assumptions across runs. - -Troubleshooting: - -- If visible content is missing from `snapshot -i`, re-snapshot after the UI settles. -- If `desktop` is too broad, retry with `frontmost-app`. -- If `menubar` is missing the expected menu, retry with `open --platform macos --surface menubar` for menu bar apps, or make the app frontmost first and retry the generic menubar surface. -- If the wrong menu opened, retry secondary-clicking the row or cell wrapper rather than the nested text node. -- If the app has multiple windows, make the correct window frontmost before relying on refs. diff --git a/.claude/skills/agent-device/references/remote-tenancy.md b/.claude/skills/agent-device/references/remote-tenancy.md deleted file mode 100644 index 6ca54a8547e2..000000000000 --- a/.claude/skills/agent-device/references/remote-tenancy.md +++ /dev/null @@ -1,99 +0,0 @@ -# Remote Tenancy - -## When to open this file - -Open this file for remote daemon HTTP flows, including `--remote-config` launches, that let an agent running in a Linux sandbox talk to another `agent-device` instance on a remote macOS host in order to control devices that are not available locally. This file covers daemon URL setup, authentication, lease allocation, and tenant-scoped command admission. - -## Main commands to reach for first - -- `agent-device open --remote-config --relaunch` -- `AGENT_DEVICE_DAEMON_BASE_URL=...` -- `AGENT_DEVICE_DAEMON_AUTH_TOKEN=...` -- `agent-device --tenant ... --session-isolation tenant --run-id ... --lease-id ...` - -## Most common mistake to avoid - -Do not run a tenant-isolated command without matching `tenant`, `run`, and `lease` scope. Admission checks require all three to line up. - -## Preferred remote launch path - -Use this when the agent needs the simplest remote control flow: a Linux sandbox agent talks over HTTP to `agent-device` on a remote macOS host and launches the target app through a checked-in `--remote-config` profile. - -```bash -agent-device open com.example.myapp --remote-config ./agent-device.remote.json --relaunch -``` - -- This is the preferred remote launch path for sandbox or cloud agents. -- For Android React Native relaunch flows, install or reinstall the APK first, then relaunch by installed package name. -- Do not use `open --relaunch`; remote runtime hints are applied through the installed app sandbox. - -## Lease flow example - -```bash -export AGENT_DEVICE_DAEMON_BASE_URL= -export AGENT_DEVICE_DAEMON_AUTH_TOKEN= - -agent-device \ - --tenant acme \ - --session-isolation tenant \ - --run-id run-123 \ - --lease-id \ - session list --json -``` - -Low-level lease operations exist for host-side automation, but do not point them at arbitrary hosts. The remote daemon executes device-control commands, so only use a trusted daemon base URL and an auth token managed by the same operator boundary. - -Lease lifecycle methods exposed by the daemon: - -- `agent_device.lease.allocate` -- `agent_device.lease.heartbeat` -- `agent_device.lease.release` -- `agent_device.command` - -## Transport prerequisites - -- Start the daemon in HTTP mode with `AGENT_DEVICE_DAEMON_SERVER_MODE=http|dual`. -- Point the client at the remote host with `AGENT_DEVICE_DAEMON_BASE_URL=http(s)://host:port[/base-path]`. -- For non-loopback remote hosts, set `AGENT_DEVICE_DAEMON_AUTH_TOKEN` or `--daemon-auth-token`. The client rejects non-loopback remote daemon URLs without auth. -- Direct JSON-RPC callers can authenticate with request params, `Authorization: Bearer `, or `x-agent-device-token`. -- Prefer an auth hook such as `AGENT_DEVICE_HTTP_AUTH_HOOK` when the host needs caller validation or tenant injection. - -## Lease lifecycle - -Use JSON-RPC methods on `POST /rpc`: - -- `agent_device.lease.allocate` -- `agent_device.lease.heartbeat` -- `agent_device.lease.release` - -Keep the lease alive for the duration of the run and release it when the tenant-scoped work is complete. - -Host-level lease knobs: - -- `AGENT_DEVICE_MAX_SIMULATOR_LEASES` -- `AGENT_DEVICE_LEASE_TTL_MS` -- `AGENT_DEVICE_LEASE_MIN_TTL_MS` -- `AGENT_DEVICE_LEASE_MAX_TTL_MS` - -## Command admission contract - -For tenant-isolated command execution, pass all four CLI flags together: - -```bash -agent-device \ - --tenant acme \ - --session-isolation tenant \ - --run-id run-123 \ - --lease-id \ - session list --json -``` - -The CLI sends `AGENT_DEVICE_DAEMON_AUTH_TOKEN` in both the JSON-RPC request token field and HTTP auth headers so existing daemon auth paths continue to work. - -## Failure semantics and trust notes - -- Missing tenant, run, or lease fields in tenant-isolation mode should fail as `INVALID_ARGS`. -- Inactive or scope-mismatched leases should fail as `UNAUTHORIZED`. -- Inspect logs on the remote host during remote debugging. Client-side `--debug` does not tail a local daemon log once `AGENT_DEVICE_DAEMON_BASE_URL` is set. -- Do not point `AGENT_DEVICE_DAEMON_BASE_URL` at untrusted hosts. Remote daemon requests can launch apps and execute interaction commands. -- Treat daemon auth tokens and lease identifiers as sensitive operational data. diff --git a/.claude/skills/dogfood/SKILL.md b/.claude/skills/dogfood/SKILL.md deleted file mode 100644 index e5d99f4f48bd..000000000000 --- a/.claude/skills/dogfood/SKILL.md +++ /dev/null @@ -1,183 +0,0 @@ ---- -name: dogfood -description: 'Systematically explore and test a mobile app on iOS/Android with agent-device to find bugs, UX issues, and other problems. Use when asked to "dogfood", "QA", "exploratory test", "find issues", "bug hunt", or "test this app" on mobile. Produces a structured report with reproducible evidence: screenshots, optional repro videos, and detailed steps for every issue.' -allowed-tools: Bash(agent-device:*), Bash(npx agent-device:*) ---- - -# Dogfood (agent-device) - -Systematically explore a mobile app, find issues, and produce a report with full reproduction evidence for every finding. - -## Setup - -Only the **Target app** is required. Everything else has sensible defaults. - -| Parameter | Default | Example override | -| -------------------- | ----------------------------------------------------------- | -------------------------------------------- | -| **Target app** | _(required)_ | `Settings`, `com.example.app`, deep link URL | -| **Platform** | Infer from user context; otherwise ask (`ios` or `android`) | `--platform ios` | -| **Session name** | Slugified app/platform (for example `settings-ios`) | `--session my-session` | -| **Output directory** | `./dogfood-output/` | `Output directory: /tmp/mobile-qa` | -| **Scope** | Full app | `Focus on onboarding and profile` | -| **Authentication** | None | `Sign in to user@example.com` | - -If the user gives enough context to start, begin immediately with defaults. Ask follow-up only when a required detail is missing (for example platform or credentials). - -Prefer direct `agent-device` binary when available. - -## Workflow - -``` -1. Initialize Set up session, output dirs, report file -2. Launch/Auth Open app and sign in if needed -3. Orient Capture initial snapshot and map navigation -4. Explore Systematically test flows and states -5. Document Record reproducible evidence per issue -6. Wrap up Reconcile summary, close session -``` - -### 1. Initialize - -```bash -mkdir -p {OUTPUT_DIR}/screenshots {OUTPUT_DIR}/videos -cp {SKILL_DIR}/templates/dogfood-report-template.md {OUTPUT_DIR}/report.md -``` - -### 2. Launch/Auth - -Start a named session and launch target app: - -```bash -agent-device --session {SESSION} open {TARGET_APP} --platform {PLATFORM} -agent-device --session {SESSION} snapshot -i -``` - -If login is required: - -```bash -agent-device --session {SESSION} snapshot -i -agent-device --session {SESSION} fill @e1 "{EMAIL}" -agent-device --session {SESSION} fill @e2 "{PASSWORD}" -agent-device --session {SESSION} press @e3 -agent-device --session {SESSION} wait 1000 -agent-device --session {SESSION} snapshot -i -``` - -For OTP/email codes: ask the user, wait for input, then continue. - -### 3. Orient - -Capture initial evidence and navigation anchors: - -```bash -agent-device --session {SESSION} screenshot {OUTPUT_DIR}/screenshots/initial.png -agent-device --session {SESSION} snapshot -i -``` - -Map top-level navigation, tabs, and key workflows before deep testing. - -### 4. Explore - -Read [references/issue-taxonomy.md](references/issue-taxonomy.md) for severity/category calibration. - -Strategy: - -- Move through each major app area (tabs, drawers, settings pages). -- Test core journeys end-to-end (create, edit, delete, submit, recover). -- Validate edge states (empty/error/loading/offline/permissions denied). -- Use `diff snapshot -i` after UI transitions to avoid stale refs. -- Periodically capture `logs path` and inspect the app log when behavior looks suspicious. - -Useful commands per screen: - -```bash -agent-device --session {SESSION} snapshot -i -agent-device --session {SESSION} screenshot {OUTPUT_DIR}/screenshots/{screen-name}.png -agent-device --session {SESSION} appstate -agent-device --session {SESSION} logs path -``` - -### 5. Document Issues (Repro-First) - -Explore and document in one pass. When you find an issue, stop and fully capture evidence before continuing. - -#### Interactive/behavioral issues - -Use video + step screenshots: - -1. Start recording: - -```bash -agent-device --session {SESSION} record start {OUTPUT_DIR}/videos/issue-{NNN}-repro.mp4 -``` - -2. Reproduce with visible pacing. Capture each step: - -```bash -agent-device --session {SESSION} screenshot {OUTPUT_DIR}/screenshots/issue-{NNN}-step-1.png -sleep 1 -# perform action -sleep 1 -agent-device --session {SESSION} screenshot {OUTPUT_DIR}/screenshots/issue-{NNN}-step-2.png -``` - -3. Capture final broken state: - -```bash -sleep 2 -agent-device --session {SESSION} screenshot {OUTPUT_DIR}/screenshots/issue-{NNN}-result.png -``` - -4. Stop recording: - -```bash -agent-device --session {SESSION} record stop -``` - -5. Append issue immediately to report with numbered steps and screenshot references. - -#### Static/on-load issues - -Single screenshot is sufficient; no video required: - -```bash -agent-device --session {SESSION} screenshot {OUTPUT_DIR}/screenshots/issue-{NNN}.png -``` - -Set **Repro Video** to `N/A` in the report. - -### 6. Wrap Up - -Target 5-10 well-evidenced issues, then finish: - -1. Reconcile summary severity counts in `report.md`. -2. Close session: - -```bash -agent-device --session {SESSION} close -``` - -3. Report total issues, severity breakdown, and highest-risk findings. - -## Guidance - -- Repro quality matters more than issue count. -- Use refs (`@eN`) for fast exploration, selectors for deterministic replay assertions when needed. -- Re-snapshot after any mutation (navigation, modal, list update, form submit). -- Use `fill` for clear-then-type semantics; use `type` for incremental typing behavior checks. -- Keep logs optional and targeted: enable/read app logs only when useful for diagnosis. -- Never read source code of the app under test; findings must come from observed runtime behavior. -- Write each issue immediately to avoid losing evidence. -- Never delete screenshots/videos/report artifacts during a session. - -## References - -| Reference | When to Read | -| ------------------------------------------------------------ | ----------------------------------------------- | -| [references/issue-taxonomy.md](references/issue-taxonomy.md) | Start of session; severity/categories/checklist | - -## Templates - -| Template | Purpose | -| ---------------------------------------------------------------------------- | --------------------------------------------- | -| [templates/dogfood-report-template.md](templates/dogfood-report-template.md) | Copy into output directory as the report file | diff --git a/.claude/skills/dogfood/references/issue-taxonomy.md b/.claude/skills/dogfood/references/issue-taxonomy.md deleted file mode 100644 index 27a678d6163d..000000000000 --- a/.claude/skills/dogfood/references/issue-taxonomy.md +++ /dev/null @@ -1,83 +0,0 @@ -# Issue Taxonomy (Mobile) - -Reference for categorizing issues found during mobile dogfooding. - -## Severity Levels - -| Severity | Definition | -| ------------ | ------------------------------------------------------------------------- | -| **critical** | Blocks a core workflow, causes data loss, or crashes/freeze loops the app | -| **high** | Major feature broken or unusable, no practical workaround | -| **medium** | Feature works with notable friction or partial failure; workaround exists | -| **low** | Minor cosmetic or polish issue | - -## Categories - -### Visual / UI - -- Layout broken, clipped, overlapped, or unreadable text -- Safe-area/notch overlap issues -- Incorrect dark/light appearance rendering -- Missing assets/icons -- Animation glitches or flicker - -### Functional - -- Buttons/controls do nothing or trigger wrong action -- Flows fail (create/edit/delete/submit) -- Navigation dead-ends or wrong destination -- State loss after background/foreground transitions -- Deep link opens wrong screen or fails - -### UX - -- Confusing hierarchy or navigation labels -- Missing loading/progress feedback -- Unclear error handling or no recovery affordance -- Excessive steps for common tasks -- Inconsistent behavior between similar screens - -### Content - -- Typos, incorrect copy, placeholder text -- Wrong labels/help text -- Truncated text with no affordance -- Inconsistent terminology across screens - -### Performance - -- Slow startup or route transitions -- Input lag or gesture jank -- Scroll hitches/frame drops -- Notable battery/thermal symptoms during basic usage - -### Diagnostics / Logs - -- Native crashes or repeated fatal exceptions -- Repeated warnings correlated with broken behavior -- Unhandled runtime errors visible during repro - -### Permissions / Platform - -- Permission prompt flow broken or loops forever -- Denied permissions not handled gracefully -- Platform-specific regressions (iOS-only or Android-only) -- Background/foreground lifecycle regressions - -### Accessibility - -- Missing labels or incorrect accessibility names -- Focus order/navigation issues for assistive tech -- Low contrast or unreadable text scaling -- Touch targets too small for reliable interaction - -## Exploration Checklist - -1. Visual scan: capture screenshot; verify layout/safe areas/text/icon rendering. -2. Interactions: press controls, open menus/modals, validate expected response. -3. Forms/input: test valid/invalid/empty/boundary input. -4. Navigation: traverse all top-level sections and return paths. -5. App states: loading/empty/error/offline/permission-denied/background-resume. -6. Logs/diagnostics: inspect app logs when behavior is suspicious. -7. Platform parity: verify critical flows on each requested platform. -8. Accessibility basics: labels, touch target sizes, readability/contrast. diff --git a/.claude/skills/dogfood/templates/dogfood-report-template.md b/.claude/skills/dogfood/templates/dogfood-report-template.md deleted file mode 100644 index fd11566a6d6a..000000000000 --- a/.claude/skills/dogfood/templates/dogfood-report-template.md +++ /dev/null @@ -1,52 +0,0 @@ -# Dogfood Report: {APP_NAME} - -| Field | Value | -| -------------- | -------------- | -| **Date** | {DATE} | -| **Platform** | {PLATFORM} | -| **Target App** | {TARGET_APP} | -| **Session** | {SESSION_NAME} | -| **Scope** | {SCOPE} | - -## Summary - -| Severity | Count | -| --------- | ----- | -| Critical | 0 | -| High | 0 | -| Medium | 0 | -| Low | 0 | -| **Total** | **0** | - -## Issues - - - -### ISSUE-001: {Short title} - -| Field | Value | -| ------------------ | -------------------------------------------------------------------------------------------- | -| **Severity** | critical / high / medium / low | -| **Category** | visual / functional / ux / content / performance / diagnostics / permissions / accessibility | -| **Screen / Route** | {screen where issue was found} | -| **Repro Video** | {path to video, or N/A for static issues} | - -**Description** - -{What is wrong, what was expected, and what actually happened.} - -**Repro Steps** - -1. Open {screen/entry point} - ![Step 1](screenshots/issue-001-step-1.png) - -2. {Action} - ![Step 2](screenshots/issue-001-step-2.png) - -3. {Action} - ![Step 3](screenshots/issue-001-step-3.png) - -4. **Observe:** {broken behavior} - ![Result](screenshots/issue-001-result.png) - ---- From e756806406da939db7c09d27c8bc8dd7c430cf59 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Kacper=20Miko=C5=82ajczak?= Date: Tue, 14 Apr 2026 14:55:04 +0200 Subject: [PATCH 06/14] Sync agent-device skill with v0.12.x and improve app-testing startup flow - Replace removed scrollintoview command with scroll + re-snapshot pattern - Add shell loop example for off-screen element discovery - Add diff screenshot section to verification reference - Rework app-testing skill with gated startup flow (device, metro, dev app) - Remove release build references, enforce dev-only app policy --- .../skills/agent-device-app-testing/SKILL.md | 57 ++++++++++++++++--- .claude/skills/agent-device/SKILL.md | 2 +- .../agent-device/references/exploration.md | 24 ++++++-- .../agent-device/references/verification.md | 20 ++++++- 4 files changed, 88 insertions(+), 15 deletions(-) diff --git a/.claude/skills/agent-device-app-testing/SKILL.md b/.claude/skills/agent-device-app-testing/SKILL.md index 067e399b72b8..8245e49b328f 100644 --- a/.claude/skills/agent-device-app-testing/SKILL.md +++ b/.claude/skills/agent-device-app-testing/SKILL.md @@ -22,21 +22,62 @@ Android testing requires the Android SDK + an AVD. iOS requires Xcode + a simula | Key | Value | |---|---| -| Android package | `com.expensify.chat.dev` (dev) / `com.expensify.chat` (release) | -| iOS bundle ID | `com.expensify.chat.dev` (dev) / `com.expensify.chat` (release) | +| Android package | `com.expensify.chat.dev` | +| iOS bundle ID | `com.expensify.chat.dev` | | Screenshot dir | `./agent-device-output/` | -## Building and Installing +## Startup Flow -The project uses [Rock](https://rockjs.dev/) which downloads pre-built artifacts from S3 first, falling back to local compilation if no match. Metro must be running in a separate terminal. +Follow these steps in order before testing. Each step gates the next. + +### 1. Device / Emulator + +Check if a simulator (iOS) or emulator (Android) is already running: + +```bash +# iOS +xcrun simctl list devices booted +# Android +adb devices +``` + +- **Running**: reuse it. +- **Not running**: prompt the user which device to boot (list available devices so they can pick). + +### 2. Metro Dev Server + +Check if Metro is already running: ```bash -npm install -npm run start # metro - separate terminal -npm run android # or: npm run ios +lsof -i :8081 ``` -Full setup details: [SETUP_ANDROID.md](contributingGuides/SETUP_ANDROID.md) | [SETUP_IOS.md](contributingGuides/SETUP_IOS.md) +- **Running**: proceed. +- **Not running**: start it in a background terminal: + ```bash + npm run start # runs Metro bundler on :8081 + ``` + Wait until Metro reports "Ready" before continuing. + +### 3. Dev App + +Check if the **dev** app (bundle ID ending in `.dev`) is installed on the device: + +```bash +# iOS +xcrun simctl listapps booted | grep com.expensify.chat.dev +# Android +adb shell pm list packages | grep com.expensify.chat.dev +``` + +- **Installed**: open it via `agent-device open com.expensify.chat.dev`. +- **Not installed**: build and install with: + ```bash + npm run ios # or: npm run android + ``` + This handles everything (dependency download, compilation, installation, launch). + +> **Never fall back to a different app.** Only `com.expensify.chat.dev` is valid for testing. If the build fails (expired tokens, missing deps, signing errors, etc.), stop and report the error to the user - do not attempt to use any other installed app. ## Dev Environment Sign-In diff --git a/.claude/skills/agent-device/SKILL.md b/.claude/skills/agent-device/SKILL.md index 05d2058c4477..5a5370f6ff42 100644 --- a/.claude/skills/agent-device/SKILL.md +++ b/.claude/skills/agent-device/SKILL.md @@ -18,7 +18,7 @@ Use this skill as a router with mandatory defaults. Read this file first. For no - In React Native dev or debug builds, check early for visible warning or error overlays, tooltips, and toasts that can steal focus or intercept taps. If they are not part of the requested behavior, dismiss them and continue. If you saw them, report them in the final summary. - Do not browse the web or use external sources unless the user explicitly asks. - Re-snapshot after meaningful UI changes instead of reusing stale refs. -- Treat refs in default snapshot output as actionable-now, not durable identities. If a target is off-screen, use `scrollintoview` or scroll and re-snapshot. +- Treat refs in default snapshot output as actionable-now, not durable identities. If a target appears only in an off-screen summary, use `scroll ` and re-snapshot until the target is visible. - Prefer `@ref` or selector targeting over raw coordinates. - Ensure the correct target is pinned and an app session is open before interacting. - Keep the loop short: `open` -> inspect/act -> verify if needed -> `close`. diff --git a/.claude/skills/agent-device/references/exploration.md b/.claude/skills/agent-device/references/exploration.md index 2ff799f703ec..0290dc58abaf 100644 --- a/.claude/skills/agent-device/references/exploration.md +++ b/.claude/skills/agent-device/references/exploration.md @@ -38,7 +38,7 @@ Open this file when the app or screen is already running and you need to discove - `press` - `fill` - `type` -- `scrollintoview` +- `scroll` - `wait` - `keyboard dismiss` when the keyboard obscures the next target @@ -115,10 +115,10 @@ App: com.apple.Preferences ## Refs vs selectors - Use refs for discovery, debugging, and short local loops. -- Use `scrollintoview @ref` when the target is already known from the current snapshot and you want the command to re-snapshot after each swipe until the element reaches the viewport safe band. -- If `scrollintoview @ref` succeeds, prefer the returned `currentRef` for the next action. +- When a target appears only in a visible-first off-screen summary, such as `[off-screen below] ... "Battery"`, use `scroll down` and then `snapshot -i`. For `[off-screen above]`, use `scroll up` and then `snapshot -i`. +- For more than two repeated scroll checks, create a short shell loop instead of issuing each command by hand. Stop when the label appears or the snapshot stops changing. - Visible-first off-screen summaries are intentionally compact. If you need the full off-screen tree instead of a short summary, retry with `snapshot --raw`. -- Cap long searches with `--max-scrolls ` when the list may be unbounded or the target may not exist. +- Cap long searches in the loop when the list may be unbounded or the target may not exist. - Use selectors for deterministic scripts, assertions, and replay-friendly actions. - Prefer selector or `@ref` targeting over raw coordinates. - For tap interactions, `press` is canonical and `click` is an equivalent alias. @@ -132,11 +132,25 @@ agent-device press 'id="camera_row" || label="Camera" role=button' agent-device is visible 'id="camera_settings_anchor"' ``` +Example loop: + +```bash +previous='' +for _ in 1 2 3 4 5 6; do + current="$(agent-device snapshot -i)" + printf '%s\n' "$current" + printf '%s\n' "$current" | grep -q 'Battery' && break + [ "$current" = "$previous" ] && break + previous="$current" + agent-device scroll down 0.5 >/dev/null +done +``` + ## Interaction fallbacks When `press @ref` fails: -1. If the error says the ref is off-screen, run `scrollintoview @ref` and reuse the returned `currentRef` or take one fresh snapshot. +1. If the error says the ref is off-screen, use the off-screen summary direction to run `scroll `, then take a fresh `snapshot -i`. 2. Re-snapshot if the UI may have changed. 3. Retry `press @ref` or a selector-based `press`. 4. If `screenshot --overlay-refs --json` returned a reliable `overlayRefs[].center`, use `agent-device press `. diff --git a/.claude/skills/agent-device/references/verification.md b/.claude/skills/agent-device/references/verification.md index 8a38ad4d2c3c..40500119fda5 100644 --- a/.claude/skills/agent-device/references/verification.md +++ b/.claude/skills/agent-device/references/verification.md @@ -8,6 +8,7 @@ Open this file when the task needs evidence, regression checks, replay maintenan - `screenshot` - `diff snapshot` +- `diff screenshot` - `record` - `replay -u` - `perf` @@ -41,12 +42,29 @@ agent-device diff snapshot -i - Run `diff snapshot` to confirm the expected structural change. - Re-run full `snapshot` only when you need fresh refs. -## Visual artifacts +## Screenshot artifacts Use `screenshot` when the proof needs a rendered image instead of a structural tree. - Add `--overlay-refs` when you want the saved PNG to show fresh `@eN` refs burned into the screenshot. +## Visual regression with diff screenshot + +Use `diff screenshot` when comparing the current rendered screen against a saved visual baseline. + +```bash +agent-device diff screenshot --baseline ./baseline.png --out /tmp/diff.png +agent-device diff screenshot --baseline ./baseline.png ./current.png --out /tmp/diff.png +agent-device diff screenshot --baseline ./baseline.png --out /tmp/diff.png --overlay-refs +``` + +- Text output includes ranked changed regions with screen-space rectangles, shape, size, density, average color, and luminance. JSON also includes normalized bounds. +- The diff PNG uses a light grayscale current-screen context with changed pixels tinted red and changed regions outlined. +- When a current image path is provided, `diff screenshot` compares the two saved files instead of capturing from the live device or requiring an active session. +- Install `tesseract` when you want `diff screenshot` to add best-effort OCR text deltas, movement clusters, and bbox size-change hints. OCR improves the text/JSON descriptions only; it does not change the pixel comparison or the diff PNG. +- When OCR is available, `diff screenshot` also reports best-effort non-text visual deltas by masking OCR text boxes out of the pixel diff and clustering the remaining residuals. Treat these as hints for icons, controls, and separators, not semantic icon recognition. +- Add `--overlay-refs` to `diff screenshot` when you also want a separate current-screen overlay guide for a live capture. The raw screenshot is still used for pixel comparison; the overlay guide is only context for non-text controls, icons, and tappable regions. When overlay refs intersect changed regions, the output lists the best current-screen ref matches under the affected region. Saved-image comparisons do not have live accessibility refs, so omit `--overlay-refs` when passing a current image path. + ## Session recording Use `record` for debugging, documentation, or shareable verification artifacts. From 9b21ec130a92090bad6a86707e69365eed82ae03 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Kacper=20Miko=C5=82ajczak?= Date: Tue, 14 Apr 2026 15:02:49 +0200 Subject: [PATCH 07/14] Trim app-testing skill: defer device bootstrap to base agent-device --- .../skills/agent-device-app-testing/SKILL.md | 31 ++----------------- 1 file changed, 3 insertions(+), 28 deletions(-) diff --git a/.claude/skills/agent-device-app-testing/SKILL.md b/.claude/skills/agent-device-app-testing/SKILL.md index 8245e49b328f..840179bbfd00 100644 --- a/.claude/skills/agent-device-app-testing/SKILL.md +++ b/.claude/skills/agent-device-app-testing/SKILL.md @@ -7,17 +7,6 @@ description: Test the Expensify App on Android/iOS emulators using agent-device. Expensify App context for the base `agent-device` skill. The user directs what to test - this skill provides App-specific details so you don't have to look them up each time. -## Prerequisites - -The `agent-device` CLI must be installed on the host machine. It is **not** bundled with this repo. - -```bash -npm install -g agent-device -agent-device --version -``` - -Android testing requires the Android SDK + an AVD. iOS requires Xcode + a simulator runtime. - ## App Details | Key | Value | @@ -28,23 +17,9 @@ Android testing requires the Android SDK + an AVD. iOS requires Xcode + a simula ## Startup Flow -Follow these steps in order before testing. Each step gates the next. - -### 1. Device / Emulator - -Check if a simulator (iOS) or emulator (Android) is already running: - -```bash -# iOS -xcrun simctl list devices booted -# Android -adb devices -``` - -- **Running**: reuse it. -- **Not running**: prompt the user which device to boot (list available devices so they can pick). +Follow these steps after the base agent-device bootstrap (device selection and session). Each step gates the next. -### 2. Metro Dev Server +### 1. Metro Dev Server Check if Metro is already running: @@ -59,7 +34,7 @@ lsof -i :8081 ``` Wait until Metro reports "Ready" before continuing. -### 3. Dev App +### 2. Dev App Check if the **dev** app (bundle ID ending in `.dev`) is installed on the device: From 18a1a81e1a76b6495ed7742bfd86c6c8abd9a57f Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Kacper=20Miko=C5=82ajczak?= Date: Tue, 14 Apr 2026 23:36:29 +0200 Subject: [PATCH 08/14] Lean glue-code skill: drop inlined base skill, keep only pre-flight gate Remove all inlined agent-device skill files and references - the CLI's bundled skills are the canonical source. The repo skill is now a thin glue layer: pre-flight check, usage principles, and a pointer to read the bundled skills from the installed package. --- .../skills/agent-device-app-testing/SKILL.md | 67 ---- .claude/skills/agent-device/SKILL.md | 72 ++-- .../references/bootstrap-install.md | 209 ----------- .../references/coordinate-system.md | 28 -- .../agent-device/references/debugging.md | 136 ------- .../agent-device/references/exploration.md | 348 ------------------ .../agent-device/references/verification.md | 129 ------- 7 files changed, 21 insertions(+), 968 deletions(-) delete mode 100644 .claude/skills/agent-device-app-testing/SKILL.md delete mode 100644 .claude/skills/agent-device/references/bootstrap-install.md delete mode 100644 .claude/skills/agent-device/references/coordinate-system.md delete mode 100644 .claude/skills/agent-device/references/debugging.md delete mode 100644 .claude/skills/agent-device/references/exploration.md delete mode 100644 .claude/skills/agent-device/references/verification.md diff --git a/.claude/skills/agent-device-app-testing/SKILL.md b/.claude/skills/agent-device-app-testing/SKILL.md deleted file mode 100644 index 840179bbfd00..000000000000 --- a/.claude/skills/agent-device-app-testing/SKILL.md +++ /dev/null @@ -1,67 +0,0 @@ ---- -name: agent-device-app-testing -description: Test the Expensify App on Android/iOS emulators using agent-device. Use only when user explicitly requests mobile device testing. ---- - -# Mobile App Testing with agent-device - -Expensify App context for the base `agent-device` skill. The user directs what to test - this skill provides App-specific details so you don't have to look them up each time. - -## App Details - -| Key | Value | -|---|---| -| Android package | `com.expensify.chat.dev` | -| iOS bundle ID | `com.expensify.chat.dev` | -| Screenshot dir | `./agent-device-output/` | - -## Startup Flow - -Follow these steps after the base agent-device bootstrap (device selection and session). Each step gates the next. - -### 1. Metro Dev Server - -Check if Metro is already running: - -```bash -lsof -i :8081 -``` - -- **Running**: proceed. -- **Not running**: start it in a background terminal: - ```bash - npm run start # runs Metro bundler on :8081 - ``` - Wait until Metro reports "Ready" before continuing. - -### 2. Dev App - -Check if the **dev** app (bundle ID ending in `.dev`) is installed on the device: - -```bash -# iOS -xcrun simctl listapps booted | grep com.expensify.chat.dev -# Android -adb shell pm list packages | grep com.expensify.chat.dev -``` - -- **Installed**: open it via `agent-device open com.expensify.chat.dev`. -- **Not installed**: build and install with: - ```bash - npm run ios # or: npm run android - ``` - This handles everything (dependency download, compilation, installation, launch). - -> **Never fall back to a different app.** Only `com.expensify.chat.dev` is valid for testing. If the build fails (expired tokens, missing deps, signing errors, etc.), stop and report the error to the user - do not attempt to use any other installed app. - -## Dev Environment Sign-In - -- **Email**: Any address works on dev (e.g. `testuser@expensify.com`) -- **Magic code**: `000000` -- **Onboarding**: Set `SKIP_ONBOARDING=true` in `.env` to bypass - -## Notes - -- **RN dev overlays**: Dismiss warning/error overlays if not blocking; report if recurring. -- **Keyboard**: Use `keyboard dismiss` if the on-screen keyboard blocks controls. -- **Navigation**: React Navigation transitions may cause brief accessibility tree lag. If `snapshot` is stale, take a `screenshot` for visual truth, then re-snapshot. diff --git a/.claude/skills/agent-device/SKILL.md b/.claude/skills/agent-device/SKILL.md index 5a5370f6ff42..5bfe8e435768 100644 --- a/.claude/skills/agent-device/SKILL.md +++ b/.claude/skills/agent-device/SKILL.md @@ -5,65 +5,35 @@ description: Automates interactions for Apple-platform apps (iOS, tvOS, macOS) a # agent-device -Use this skill as a router with mandatory defaults. Read this file first. For normal device tasks, always load `references/bootstrap-install.md` and `references/exploration.md` before acting. Use bootstrap to confirm or establish deterministic setup. Use exploration for UI inspection, interaction, and verification once the app session is open. +## Pre-flight Check -## Default operating rules +Verify the `agent-device` CLI is installed and its skills are accessible: -- Start conservative. Prefer read-only inspection before mutating the UI. -- Start deterministic. If the app name, package, device, or session is uncertain, load bootstrap and discover them before interacting. -- Use plain `snapshot` when the task is to verify what text or structure is currently visible on screen. -- Use `snapshot -i` only when you need interactive refs such as `@e3` for a requested action or targeted query. On iOS and Android, default snapshot output uses the same visible-first model: off-screen interactive content is exposed as discovery hints, not tappable refs. -- Prefer `diff snapshot` after a nearby mutation when you only need to know what changed. -- Avoid speculative mutations. You may take the smallest reversible UI action needed to unblock inspection or complete the requested task, such as dismissing a popup, closing an alert, or clearing an unintended surface. -- In React Native dev or debug builds, check early for visible warning or error overlays, tooltips, and toasts that can steal focus or intercept taps. If they are not part of the requested behavior, dismiss them and continue. If you saw them, report them in the final summary. -- Do not browse the web or use external sources unless the user explicitly asks. -- Re-snapshot after meaningful UI changes instead of reusing stale refs. -- Treat refs in default snapshot output as actionable-now, not durable identities. If a target appears only in an off-screen summary, use `scroll ` and re-snapshot until the target is visible. -- Prefer `@ref` or selector targeting over raw coordinates. -- Ensure the correct target is pinned and an app session is open before interacting. -- Keep the loop short: `open` -> inspect/act -> verify if needed -> `close`. +```bash +agent-device --version +``` -## Default flow +If missing, **STOP** and instruct the user to install it: -1. Load [references/bootstrap-install.md](references/bootstrap-install.md) and [references/exploration.md](references/exploration.md) before acting on a normal device task. -2. Use bootstrap first to confirm or establish the correct target, app install, and open app session. -3. Once the app session is open and stable, use exploration for inspection, interaction, and verification. -4. Start with plain `snapshot` if the goal is to read or verify what is visible. -5. Escalate to `snapshot -i` only if you need refs for interactive exploration or a requested action. -6. Use `get`, `is`, or `find` before mutating the UI when a read-only command can answer the question. -7. End by capturing proof if needed, then `close`. +```bash +npm install -g agent-device +``` -## QA modes +The `agent-device` CLI ships with built-in skills under `skills/` in the installed package. These contain the canonical reference for device automation - bootstrap, exploration, verification, debugging, and more. Use `agent-device --help` to discover available commands and skill names. Read the skill files directly from the installed package path when you need detailed guidance: -- Pass/fail QA from acceptance criteria: stay in this skill, start with [references/bootstrap-install.md](references/bootstrap-install.md), then use the QA loop in [references/exploration.md](references/exploration.md). +```bash +# Find the package location +npm root -g +# Then read: /agent-device/skills/agent-device/SKILL.md +``` -## Required references +> **Do not proceed without `agent-device` installed.** All device interaction depends on it. -- For every normal device task, after reading this file, load [references/bootstrap-install.md](references/bootstrap-install.md) first, then [references/exploration.md](references/exploration.md), before acting. -- Use bootstrap to confirm or establish deterministic setup, especially in sandbox or cloud environments. -- Use exploration once the app session is open and stable. -- Load additional references only when their scope is needed. +## How This Skill Works -## Decision rules +This skill enhances the local development and testing workflow. The developer may provide a full test plan upfront, give step-by-step instructions interactively, or just point at a screen and ask questions. Follow their lead. -- Use plain `snapshot` when you need to verify whether text is visible. -- Use `snapshot -i` mainly for interactive exploration and choosing refs. -- Use `diff snapshot` for compact post-action verification; use `snapshot --diff` when that alias is easier to discover from snapshot help. -- Use `get`, `is`, or `find` when they can answer the question without changing UI state. -- Use `fill` to replace text. -- Use `type` to append text. -- Do not write `type @eN "text"`. Use `fill @eN "text"` to target a field directly, or `press @eN` then `type "text"` when the field already has focus and you want append semantics. -- If the on-screen keyboard blocks the next step, prefer `keyboard dismiss` over navigation. On iOS, keep an app session open first; `keyboard status|get` remains Android-only. -- When a task asks to "go back", use plain `back` for predictable app-owned navigation and reserve `back --system` for platform back gestures or button semantics. -- Use `type --delay-ms` or `fill --delay-ms` for debounced search fields that drop characters when typed too quickly. -- If there is no simulator, no app install, or no open app session yet, switch to `bootstrap-install.md` instead of improvising setup steps. -- Use the smallest unblock action first when transient UI blocks inspection, but do not navigate, search, or enter new text just to make the UI reveal data unless the user asked for that interaction. -- In React Native dev or debug apps, treat visible warning or error overlays as transient blockers unless the user is explicitly asking you to diagnose them. Dismiss them when safe, then continue the requested flow. -- Do not use external lookups to compensate for missing on-screen data unless the user asked for them. -- If the needed information is not exposed on screen, say that plainly instead of compensating with extra navigation, text entry, or web search. -- Prefer `@ref` or selector targeting over raw coordinates. +### Principles -## Additional references - -- Need logs, network, alerts, permissions, or failure triage: [references/debugging.md](references/debugging.md) -- Need screenshots, diff, recording, replay maintenance, or perf data: [references/verification.md](references/verification.md) +- **Fail fast.** If something deviates from expectations - wrong screen, unexpected error, missing element - stop, explain what happened and suggest how to get past it. Do not silently work around problems. +- **Deviations are signal.** If the app behaves differently than expected, report the delta clearly so the developer can decide what needs fixing. \ No newline at end of file diff --git a/.claude/skills/agent-device/references/bootstrap-install.md b/.claude/skills/agent-device/references/bootstrap-install.md deleted file mode 100644 index 2e876fced195..000000000000 --- a/.claude/skills/agent-device/references/bootstrap-install.md +++ /dev/null @@ -1,209 +0,0 @@ -# Bootstrap and Install - -## When to open this file - -Open this file when you still need to choose the right target, start the right session, install or relaunch the app, or pin automation to one device before interacting. This is the deterministic setup layer for sandbox, cloud, or other environments where install paths, device state, or app readiness may be uncertain. - -## Open-first path - -- `devices` -- `apps` -- `ensure-simulator` -- `open` -- `session list` - -Use this exact order when you are not sure about the installed app identifier. On Android dev builds in particular, `apps` is cheaper than guessing package suffixes and retrying failed `open` calls. - -## Install path - -- `install` or `reinstall` - -## Most common mistake to avoid - -Do not start acting before you have pinned the correct target and opened an `app` session. In mixed-device environments, always pass `--device`, `--udid`, or `--serial`. - -## Deterministic setup rule - -If there is no simulator, no app install, no open app session, or any uncertainty about where the app should come from, stay in this file and use deterministic setup commands or bootstrap scripts first. Do not improvise install paths or app-launch flows while exploring. - -After setup is confirmed or completed, move to `exploration.md` before doing UI inspection or interaction. - -## Open-first rule - -- If the user asks to test an app and does not provide an install artifact or explicit install instruction, try `open ` first. -- If `open ` fails, run `agent-device apps` and retry with a discovered app name before considering install steps. -- Do not install or reinstall on the first attempt unless the user explicitly asks for installation or provides a concrete artifact path or URL. -- When installation is required from a known location, prefer a checked-in shell script or other deterministic bootstrap command over ad hoc path guessing. - -- If `open ` fails, or you are not sure which app name is available on the target, run `agent-device apps` first and choose from the discovered app list instead of guessing. -- Use `apps --platform ` together with `--device`, `--udid`, or `--serial` when target selection matters. -- Once you have the correct app name, retry `open` with that exact discovered value. - -## Common starting points - -These are examples, not required exact sequences. Use the smallest setup flow that matches the task. - -### Boot a simulator and open an app - -```bash -agent-device ensure-simulator --platform ios --device "iPhone 17 Pro" --boot -agent-device open MyApp --platform ios --device "iPhone 17 Pro" --relaunch -``` - -### Install an app artifact - -```bash -agent-device install com.example.app ./build/app.apk --platform android --serial emulator-5554 -``` - -```bash -agent-device install com.example.app ./build/MyApp.app --platform ios --device "iPhone 17 Pro" -``` - -## Install guidance - -- Use `install ` when the app may already be installed and you do not need a fresh-state reset. -- Use `reinstall ` when you explicitly need uninstall plus install as one deterministic step. -- Keep install and open as separate phases. Do not turn them into one default command flow. -- Supported binary formats: - - Android: `.apk` and `.aab` - - iOS: `.app` and `.ipa` -- For iOS `.ipa` files, `` is used as the bundle id or bundle name hint when the archive contains multiple app bundles. -- After install or reinstall, later use `open ` with the exact discovered or known package/bundle identifier, not the artifact path. - -## Choose the right starting point - -- iOS local QA: prefer simulators unless the task explicitly requires physical hardware. -- iOS in mixed simulator and device environments: run `ensure-simulator` first, then keep using `--device` or `--udid`. -- TV targets: use `--target tv` together with `--platform` when the task is for tvOS or Android TV rather than phone or tablet surfaces. -- Android binary flow: use `install` or `reinstall` for `.apk` or `.aab`, then open by installed package name. -- macOS desktop app flow: use `open --platform macos`. Only load [macos-desktop.md](macos-desktop.md) if a desktop surface or macOS-specific behavior matters. - -TV example: - -```bash -agent-device open MyTvApp --platform ios --target tv -agent-device open com.example.androidtv --platform android --target tv -``` - -## Session rules - -- Use `--session ` when you need a named session: - -```bash -agent-device --session auth open Settings --platform ios -agent-device --session auth snapshot -i -``` - -- Use `open ` before interactions. -- Use `close` when done. Add `--shutdown` when you want simulators or emulators torn down with the session. -- Use semantic session names when you need multiple concurrent runs. -- Use `--save-script=` on `close` when you want to keep a replay script. -- For dev loops where state can linger, prefer `open --relaunch`. -- In iOS sessions, use `open ` for the app itself. Use `open ` for deep links, and `open ` when you need to launch the app and deep link in one step. -- On iOS, `appstate` is session-scoped and requires the matching active session on the target device. - -## After a session is established - -Once you have opened the correct session on the correct target, default to the conservative rule: keep the session binding on follow-up commands, and stop repeating device-routing flags unless you are intentionally retargeting. - -- Prefer `--session ` on follow-up commands, or use sandboxed `AGENT_DEVICE_SESSION`. -- Do not keep repeating `--platform`, `--target`, `--device`, `--udid`, `--serial`, or similar target-selection flags on normal follow-up commands. -- Only omit follow-up session flags when the environment explicitly guarantees isolation. - -Good shared-host pattern: - -```bash -agent-device --session auth open Settings --platform ios --device "iPhone 17 Pro" -agent-device --session auth snapshot -i -agent-device --session auth press @e3 -agent-device --session auth close -``` - -Bad shared-host pattern: - -```bash -agent-device --session auth open Settings --platform ios --device "iPhone 17 Pro" -agent-device --session auth snapshot -i --platform ios --device "iPhone 17 Pro" -``` - -Use target-selection flags again only when you are choosing the target before opening a session, or when you explicitly mean to retarget. - -## Session-bound automation - -Use this when an orchestrator must keep plain CLI calls on one session and device. - -```bash -export AGENT_DEVICE_SESSION=qa-ios -export AGENT_DEVICE_PLATFORM=ios -export AGENT_DEVICE_SESSION_LOCK=strip - -agent-device open MyApp --relaunch -``` - -- `AGENT_DEVICE_SESSION` plus `AGENT_DEVICE_PLATFORM` provides the default binding. -- `--session-lock reject|strip` controls whether conflicting per-call routing flags fail or are ignored. -- Conflicts include explicit retargeting flags such as `--platform`, `--target`, `--device`, `--udid`, `--serial`, `--ios-simulator-device-set`, and `--android-device-allowlist`. -- Lock policy applies to nested `batch` steps too. -- Compatibility aliases remain supported: `--session-locked`, `--session-lock-conflicts`, `AGENT_DEVICE_SESSION_LOCKED`, and `AGENT_DEVICE_SESSION_LOCK_CONFLICTS`. - -Android emulator variant: - -```bash -export AGENT_DEVICE_SESSION=qa-android -export AGENT_DEVICE_PLATFORM=android - -agent-device --session-lock reject open com.example.myapp --relaunch -``` - -## Scoped discovery - -Use scoped discovery when one run must not see host-global device lists. - -```bash -agent-device devices --platform ios --ios-simulator-device-set /tmp/tenant-a/simulators -agent-device devices --platform android --android-device-allowlist emulator-5554,device-1234 -``` - -- Scope is applied before `--device`, `--udid`, and `--serial`. -- Out-of-scope selectors fail with `DEVICE_NOT_FOUND`. -- With iOS simulator-set scope enabled, iOS physical devices are not enumerated. -- If the scoped iOS simulator set is empty, the error should point at the set path and suggest creating a simulator in that set. -- Environment equivalents: - - `AGENT_DEVICE_IOS_SIMULATOR_DEVICE_SET` - - `AGENT_DEVICE_ANDROID_DEVICE_ALLOWLIST` - -## Session inspection and replay - -```bash -agent-device session list -agent-device replay ./session.ad --session auth -agent-device replay -u ./session.ad --session auth -``` - -- iOS session entries include `device_udid` and `ios_simulator_device_set`. Use them to confirm routing in concurrent runs. -- Prefer selector-based actions and assertions in saved replay scripts. -- Tenant isolation namespaces sessions as `:` during tenant-scoped runs. - -## When to leave this file - -- Once the correct target and session are pinned, move to [exploration.md](exploration.md). -- If opening, startup, permissions, or logs become the blocker, switch to [debugging.md](debugging.md). - -## Install examples - -```bash -agent-device reinstall MyApp /path/to/app-debug.apk --platform android --serial emulator-5554 -``` - -```bash -agent-device install com.example.app ./build/MyApp.ipa --platform ios --device "iPhone 17 Pro" -``` - -Do not use `open --relaunch` on Android. - -## Security and trust notes - -- Treat signing, provisioning, and daemon auth values as host secrets. Do not paste them into shared logs or commit them to source control. -- Prefer Xcode Automatic Signing over manual overrides when a physical iOS device is involved. -- Keep persistent host-specific defaults in environment variables rather than checked-in project config. diff --git a/.claude/skills/agent-device/references/coordinate-system.md b/.claude/skills/agent-device/references/coordinate-system.md deleted file mode 100644 index 03b8f2ef0aac..000000000000 --- a/.claude/skills/agent-device/references/coordinate-system.md +++ /dev/null @@ -1,28 +0,0 @@ -# Coordinate System - -## When to open this file - -Open this file only when you must use raw coordinates instead of selectors or `@ref` targeting. - -## Main commands to reach for first - -- `screenshot` -- coordinate-based `click` or `swipe` - -## Most common mistake to avoid - -Do not assume coordinates mean the same thing across platforms or runs. Prefer selectors and refs first. - -## Canonical loop - -```bash -agent-device screenshot /tmp/current-screen.png -agent-device click 120 240 -``` - -## Rules - -- Origin is the top-left of the device screen. -- iOS uses device points. -- Android uses pixels. -- Use screenshots to reason about coordinates before acting. diff --git a/.claude/skills/agent-device/references/debugging.md b/.claude/skills/agent-device/references/debugging.md deleted file mode 100644 index 885fdf2b92ef..000000000000 --- a/.claude/skills/agent-device/references/debugging.md +++ /dev/null @@ -1,136 +0,0 @@ -# Debugging - -## When to open this file - -Open this file when the task turns into failure triage, logs, network inspection, permission prompts, setup trouble, or unstable session behavior. - -## Main commands to reach for first - -- `logs clear --restart` -- `network dump` -- `logs path` -- `logs doctor` -- `alert wait` -- `alert accept` or `alert dismiss` - -## Most common mistake to avoid - -Do not leave logging on for normal flows or dump full log files into context. Keep debug windows short and inspect logs with `grep` or `tail`. - -In React Native dev or debug builds, do not dismiss visible warning or error overlays without remembering to report them later. If you close one to keep the flow moving, keep at least a screenshot or a short marked log window so the summary can name it. - -## Canonical loop - -```bash -agent-device open MyApp --platform ios -agent-device logs clear --restart -agent-device network dump 25 -agent-device logs path -agent-device close -``` - -## Log and network flow - -Logging is off by default. Enable it only when you need a debugging window. - -- Default app logs live under `~/.agent-device/sessions//app.log`. -- `logs clear --restart` is the fastest clean repro loop. -- `network dump [limit] [summary|headers|body|all]` parses recent HTTP(s) entries from the same session app log. -- On macOS, `network dump` is app-scoped and only sees Unified Logging associated with the active session app. -- On iOS simulators, `network dump` can recover recent app log history with `simctl log show` when the live session stream is sparse, so check the returned notes before assuming the repro window was empty. -- On iOS, `network dump` is still limited to what Unified Logging exposes for the app process. If the app does not emit request metadata there, `network dump` can legitimately return no HTTP entries even during a real repro. -- Summary output already shows timestamp, status, and duration when the log backend exposes them. -- Prefer the explicit flag form `network dump 25 --include headers|body|all` when you need more than the default summary view. -- If iOS simulator notes say app logs were recovered but none looked like HTTP traffic, treat that as an app instrumentation gap rather than a missing repro and inspect `logs path` for the non-network diagnostics that were captured. -- `logs doctor` checks backend and runtime readiness for the current session and device. -- `logs mark "before tap"` inserts a timestamped marker into the app log. -- Android `network dump` surfaces timestamps from logcat-style prefixes and can backfill status and request/response duration from adjacent GIBSDK packet lines, so check it before dumping raw log windows. -- Android app-log streaming rebinds to the current app PID after relaunches, so rerun the repro window before assuming the last log slice is stale. -- Marker lines are emitted with the `[agent-device][mark][...]` prefix. When you grep later, prefer a narrow pattern such as `grep -n -E "agent-device.*mark|before tap" `. -- Session app logs can contain runtime data, headers, or payload fragments. Review them before sharing. -- `logs start` requires an active app session and appends to `app.log`. -- `logs stop` stops streaming. `close` also stops logging. -- `logs clear` truncates `app.log` and removes rotated `app.log.N` files, and requires logging to be stopped first. -- `logs path` returns the log path plus metadata about the active backend and file state. -- `network log` is an alias for `network dump`. - -Operational limits: - -- `app.log` rotates to `app.log.1` after 5 MB by default. -- `network dump` scans the last 4000 app-log lines, returns up to 200 entries, and truncates header or payload fields at 2048 characters. -- Retention knobs: - - `AGENT_DEVICE_APP_LOG_MAX_BYTES` - - `AGENT_DEVICE_APP_LOG_MAX_FILES` -- Redaction hook: - - `AGENT_DEVICE_APP_LOG_REDACT_PATTERNS` - -Useful shell follow-up after `logs path`: - -```bash -grep -n -E "Error|Exception|Fatal|crash" -grep -n -E "agent-device.*mark|before tap" -tail -50 -``` - -If the app showed a visible warning or error overlay during the flow: - -- Prefer a narrow grep window around your `logs mark` lines instead of loading the whole file. -- Mention the surfaced warning or error in the final summary even if it did not block completion. -- If the overlay kept returning, call that out as a stability issue instead of treating it as operator noise. - -## Alerts and permissions - -Use `alert` for iOS simulator permission dialogs and macOS desktop alerts instead of tapping coordinates. - -```bash -agent-device alert wait 5000 -agent-device alert accept -``` - -- `alert` is supported on iOS simulators and macOS desktop targets. -- `alert accept` and `alert dismiss` retry internally for a short window, so you usually do not need manual sleeps. -- If a permission sheet or modal is visible in `snapshot` or `screenshot` but `alert accept` says no alert was found, treat it as normal tappable UI for that run: take a scoped `snapshot -i -s ""` and `press @ref` instead of looping on `alert`. -- iOS 16+ "Allow Paste" prompts are suppressed under XCUITest. Use `xcrun simctl pbcopy booted` when you need to seed simulator clipboard content directly. - -## Setup problems worth recognizing early - -- iOS snapshots do not require macOS Accessibility permissions. -- iOS physical-device XCTest setup does require valid signing and provisioning. -- If physical-device runner setup fails, prefer Xcode Automatic Signing first. -- Optional overrides are: - - `AGENT_DEVICE_IOS_TEAM_ID` - - `AGENT_DEVICE_IOS_SIGNING_IDENTITY` - - `AGENT_DEVICE_IOS_PROVISIONING_PROFILE` - - `AGENT_DEVICE_IOS_BUNDLE_ID` -- If daemon startup is timing out during setup, increase `AGENT_DEVICE_DAEMON_TIMEOUT_MS`. -- If daemon startup fails with stale metadata hints, clean `~/.agent-device/daemon.json` and `~/.agent-device/daemon.lock`, then retry. -- Free Apple Developer personal-team accounts may reject generic bundle IDs. Use a unique reverse-DNS value for `AGENT_DEVICE_IOS_BUNDLE_ID` when that happens. - -## Common failure patterns - -- `snapshot` returns 0 nodes: the app may no longer be foregrounded or the UI is not stable yet. Re-open the app or retry when state settles. -- Logs are empty: confirm you opened an app session before `logs clear --restart`. -- Android logs look stale after relaunch: retry the repro window after the process rebinds. -- Android accessibility snapshots can lag behind visible screen transitions. The next snapshot now retries briefly after navigation-sensitive actions, but if the tree still looks stale, use `screenshot` as visual truth, wait briefly, then re-run `snapshot -i`. -- React Native dev warnings or errors keep reappearing: treat them as part of the app state, not as disposable chrome. Capture one clean repro and include them in the summary. -- Permission prompts block the flow: wait for the alert and handle it explicitly. -- If snapshots keep returning 0 nodes on an iOS simulator, restart Simulator and re-open the app. -- If a macOS snapshot looks incomplete, compare with `snapshot --raw --platform macos` to separate collector filtering from missing AX content. - -## Crash triage fast path - -Always start from the session app log, then branch by platform. - -```bash -agent-device logs path -grep -n -E "SIGABRT|SIGSEGV|EXC_|fatal|exception|terminated|killed|jetsam|memorystatus|FATAL EXCEPTION|Abort message" -``` - -- iOS: if the log suggests `ReportCrash`, `SIGABRT`, or `EXC_*`, inspect `~/Library/Logs/DiagnosticReports`. -- Android: if the app log is not enough, use `adb logcat` for `FATAL EXCEPTION`, `Abort message`, or `signal` lines around process death. -- If no crash signature appears in app logs, stop collecting broad logs and switch to the platform-native crash source. - -## When to leave this file - -- Return to [exploration.md](exploration.md) once the app is stable again. -- Load [verification.md](verification.md) if you need evidence artifacts after reproducing the issue. diff --git a/.claude/skills/agent-device/references/exploration.md b/.claude/skills/agent-device/references/exploration.md deleted file mode 100644 index 0290dc58abaf..000000000000 --- a/.claude/skills/agent-device/references/exploration.md +++ /dev/null @@ -1,348 +0,0 @@ -# Exploration - -## When to open this file - -Open this file when the app or screen is already running and you need to discover the UI, choose targets, read state, wait for conditions, or perform normal interactions. - -## Read-only first - -- If the question is what text, labels, or structure is visible on screen, start with plain `snapshot`. -- Escalate to `snapshot -i` only when you need refs such as `@e3` for interactive exploration or a requested action. -- If you intend to `press`, `fill`, or otherwise interact, start with `snapshot -i` and fall back to plain `snapshot` only if interactive refs are unavailable. -- Prefer `get`, `is`, or `find` before mutating the UI when a read-only command can answer the question. -- You may take the smallest reversible UI action needed to unblock inspection, such as dismissing a popup, closing an alert, or backing out of an unintended surface. -- Do not type or fill text just to make hidden information easier to access unless the user asked for that interaction. -- Do not use external sources to infer missing UI state unless the user explicitly asked. -- If the answer is not visible or exposed in the UI, report that gap instead of compensating with search, navigation, or text entry. - -## Decision shortcut - -- User asks what is visible on screen: `snapshot` -- User asks for exact text from a known target: `get text` -- User asks you to tap, type, or choose an element: `snapshot -i`, then act -- React Native dev or debug build shows warning/error UI: capture enough evidence to identify it, dismiss it if it is not the requested behavior, then continue the flow and report it in the summary -- The on-screen keyboard is blocking the next step: `keyboard dismiss`; on iOS do this only while an app session is active, and use `keyboard status|get` only on Android -- UI does not expose the answer: say so plainly; do not browse or force the app into a new state unless asked - -## Read-only commands - -- `snapshot` -- `get` -- `is` -- `find` -- `keyboard status|get` on Android when keyboard visibility or input type matters - -## Interaction commands - -- `snapshot -i` -- `press` -- `fill` -- `type` -- `scroll` -- `wait` -- `keyboard dismiss` when the keyboard obscures the next target - -## Common mistakes to avoid - -**Stale refs.** Do not treat `@ref` values as durable after navigation or dynamic updates. Re-snapshot after the UI changes, and switch to selectors when the flow must stay stable. - -**Android AX tree lag.** After submits, route changes, or composer transitions, the accessibility tree can lag behind the visible UI. If `snapshot -i` and `screenshot` disagree: - -1. Trust the screenshot as visual truth. -2. Take one fresh `snapshot -i`. Android retries briefly after navigation-sensitive actions. -3. If the tree still disagrees with the screenshot, wait briefly, then take one more fresh snapshot. Do not loop snapshots immediately. - -**React Native dev overlays.** In dev or debug builds, warning or error overlays can block taps, change focus, or hide the real UI. Check for them near app open and after major transitions. - -- Not blocking the task: dismiss and continue. -- Blocking or recurring: switch to [debugging.md](debugging.md) and collect evidence. -- Seen at any point: mention in the final summary even if dismissed. - -## Common example loops - -These are examples, not required exact sequences. Adapt them to the app, state, and task at hand. - -### Interactive exploration loop - -```bash -agent-device open Settings --platform ios -agent-device snapshot -i -agent-device press @e3 -agent-device wait visible 'label="Privacy & Security"' 3000 -agent-device get text 'label="Privacy & Security"' -agent-device close -``` - -### Screen verification loop - -```bash -agent-device open MyApp --platform ios -# perform the necessary actions to reach the state you need to verify -agent-device snapshot -# verify whether the expected element or text is present -agent-device close -``` - -## Snapshot choices - -- Use plain `snapshot` when you only need to verify whether visible text or structure is on screen. -- Use `snapshot -i` when you need refs such as `@e3` for interactive exploration or for an intended interaction. -- On iOS and Android, default snapshot output is visible-first. Off-screen interactive content is surfaced as discovery hints (including inline scroll/list hidden-content hints when known), not shown as directly tappable refs. -- Treat large text-surface lines in `snapshot -i` as discovery output. If a node shows preview or truncation metadata, use `get text @ref` only after you have already decided that `snapshot -i` is needed for that surface. -- Use `snapshot -i -s "Camera"` or `snapshot -i -s @e3` when you want a smaller, scoped result. -- If `snapshot -i -s ""` returns 0 nodes, the scope did not match the current screen. Widen the query or re-check the screen state instead of assuming the command silently fell back to the full tree. -- If `snapshot -i` returns 0 nodes but the screen is visibly populated, treat `screenshot` as visual truth, wait briefly, then re-run `snapshot -i` once before escalating. -- If `snapshot -i -d ` says the interactive output is empty at that depth, retry without `-d` instead of taking more shallow snapshots. - -Example: - -```bash -agent-device snapshot -i -``` - -Sample output: - -```text -Page: com.apple.Preferences -App: com.apple.Preferences - -@e1 [ioscontentgroup] - @e2 [button] "Camera" - @e3 [button] "Privacy & Security" -[off-screen below] 2 interactive items: "Location Services", "Battery" -``` - -## Refs vs selectors - -- Use refs for discovery, debugging, and short local loops. -- When a target appears only in a visible-first off-screen summary, such as `[off-screen below] ... "Battery"`, use `scroll down` and then `snapshot -i`. For `[off-screen above]`, use `scroll up` and then `snapshot -i`. -- For more than two repeated scroll checks, create a short shell loop instead of issuing each command by hand. Stop when the label appears or the snapshot stops changing. -- Visible-first off-screen summaries are intentionally compact. If you need the full off-screen tree instead of a short summary, retry with `snapshot --raw`. -- Cap long searches in the loop when the list may be unbounded or the target may not exist. -- Use selectors for deterministic scripts, assertions, and replay-friendly actions. -- Prefer selector or `@ref` targeting over raw coordinates. -- For tap interactions, `press` is canonical and `click` is an equivalent alias. - -Examples: - -```bash -agent-device press @e2 -agent-device fill @e5 "test" -agent-device press 'id="camera_row" || label="Camera" role=button' -agent-device is visible 'id="camera_settings_anchor"' -``` - -Example loop: - -```bash -previous='' -for _ in 1 2 3 4 5 6; do - current="$(agent-device snapshot -i)" - printf '%s\n' "$current" - printf '%s\n' "$current" | grep -q 'Battery' && break - [ "$current" = "$previous" ] && break - previous="$current" - agent-device scroll down 0.5 >/dev/null -done -``` - -## Interaction fallbacks - -When `press @ref` fails: - -1. If the error says the ref is off-screen, use the off-screen summary direction to run `scroll `, then take a fresh `snapshot -i`. -2. Re-snapshot if the UI may have changed. -3. Retry `press @ref` or a selector-based `press`. -4. If `screenshot --overlay-refs --json` returned a reliable `overlayRefs[].center`, use `agent-device press `. -5. Use an external vision-based tap tool only after semantic and coordinate targeting fail. - -- Prefer `@ref` over coordinates. -- Do not guess coordinates from the image when structured `center` is available. -- `agent-device` does not provide a built-in vision-tap flag. - -## Text entry rules - -- Use `fill` to replace text in an editable field. -- Use `type` to append text to the current insertion point. -- Use `fill @ref "text"` when you need to target a field directly by ref. -- Use `press @ref`, then `type "text"` when the field is already focused and you need append semantics. -- Do not write `type @ref "text"`; `type` only accepts text and will not target that ref for you. -- If the keyboard blocks the next control after text entry, prefer `keyboard dismiss` instead of backing out of the screen. -- On iOS, `keyboard dismiss` depends on the active app session to keep the target app foregrounded, so do not rely on selector-only dismiss calls after closing or without `open`. -- Do not use `fill` or `type` just to make the app reveal information that is not currently visible unless the user asked for that interaction. - -## React Native dev or debug overlays - -Use this loop for React Native dev clients, Metro-backed builds, and local debug sessions where warnings or errors may appear as tooltips, banners, toasts, or modal overlays. - -1. After `open`, inspect the visible UI for warning or error surfaces before relying on the next tap. -2. If a warning or error is visible, capture enough evidence to identify it: - - preferred: `screenshot` - - optional: `logs mark "warning visible"` or `logs mark "error visible"` if you are already in a debug window -3. If the overlay is not the thing the user asked you to investigate, dismiss or close it with the smallest reversible action. -4. Re-check the intended screen before continuing the task. -5. Report any visible warnings or errors in the final summary, even if the flow succeeded after dismissal. - -Use this rule of thumb: - -- Warning overlay that does not block the task: dismiss and keep going. -- Error overlay that does not block the task: dismiss, keep going, and report it. -- Error overlay that blocks the task or keeps returning: stop treating it as noise and switch to [debugging.md](debugging.md). - -## Query and sync rules - -- Use `get` to read text, attrs, or state from a known target. -- Use `is` for assertions. -- Use `wait` when the UI needs time to settle after a mutation. -- Use `find "" click --json` when you need search-driven targeting plus matched-target metadata. -- Use `find "" click --first` or `--last` when ambiguous matches are expected and you want the first or last occurrence without falling back to raw coordinates. -- If you are forced onto raw coordinates, open [coordinate-system.md](coordinate-system.md) first. - -Example: - -```bash -agent-device find "Increment" click --json -``` - -Returned metadata comes from the matched snapshot node and can be used for observability or replay maintenance. - -## QA from acceptance criteria - -Use this loop when the task starts from acceptance criteria and you need to turn them into concrete checks. - -Preferred mapping: - -- visibility claim for what is on-screen now: `is visible` or plain `snapshot` -- presence claim regardless of viewport visibility: `is exists` -- exact text, label, or value claim: `get text` -- post-action state change: act, then `wait`, then `is` or `get` -- nearby structural UI change: `diff snapshot` -- proof artifact for the final result: `screenshot` or `record` - -Notes: - -- `wait text` is useful for synchronizing on text presence, but it is not the same as `is visible`. -- After a nearby navigation or submit on Android, prefer `screenshot`, then `wait 500` or `wait 1000`, then one fresh `snapshot -i` if the accessibility tree seems stale. - -Anti-hallucination rules: - -- Do not invent app names, device ids, session names, refs, selectors, or package names. -- Discover them first with `devices`, `open`, `snapshot -i`, `find`, or `session list`. -- If refs drift after navigation, re-snapshot or switch to selectors instead of guessing. - -Avoid this escalation path for visible-text questions: - -- Do not jump from `snapshot -i` to `get text @ref`, then to web search, then to typing into a search box just to force the app to reveal the answer. -- Start with `snapshot`. If the text is not visible or exposed, report that directly. -- After Android submit or navigation-heavy actions when the UI looks wrong: `screenshot` first, then `snapshot -i`. - -Canonical QA loop: - -```bash -agent-device open MyApp --platform ios -agent-device snapshot -i -agent-device press @e3 -agent-device wait visible 'label="Success"' 3000 -agent-device is visible 'label="Success"' -agent-device screenshot /tmp/qa-proof.png -agent-device close -``` - -## Accessibility audit - -Use this pattern when you need to find UI that is visible to a user but missing from the accessibility tree. - -Audit loop: - -1. Capture a `screenshot` to see what is visually rendered. -2. Capture a `snapshot` or `snapshot -i` to see what the accessibility tree exposes. -3. Compare the two: - - visible in screenshot and present in snapshot: exposed to accessibility - - visible in screenshot and missing from snapshot: likely accessibility gap -4. If you suspect the node exists in AX but is filtered from interactive output, retry with `snapshot --raw`. - -Example: - -```bash -agent-device screenshot /tmp/accessibility-screen.png -agent-device snapshot -i -``` - -Use `screenshot` as the visual source of truth and `snapshot` as the accessibility source of truth for this audit. - -## Batch only when the sequence is already known - -Use `batch` when a short command sequence is already planned and belongs to one logical screen flow. - -```bash -agent-device batch --session sim --platform ios --steps-file /tmp/batch-steps.json --json -``` - -- Keep batch size moderate, roughly 5 to 20 steps. -- Add `wait` or `is exists` guards after mutating steps. -- Do not use `batch` for highly dynamic flows that need replanning after each step. - -Example: known chat-send flow - -```json -[ - { "command": "open", "positionals": ["ChatApp"], "flags": { "platform": "android" } }, - { "command": "click", "positionals": ["label=\"Travel chat\""], "flags": {} }, - { "command": "wait", "positionals": ["label=\"Message\"", "3000"], "flags": {} }, - { "command": "fill", "positionals": ["label=\"Message\"", "Filed the expense"], "flags": {} }, - { "command": "press", "positionals": ["label=\"Send\""], "flags": {} } -] -``` - -Step payload contract: - -```json -[ - { "command": "open", "positionals": ["Settings"], "flags": { "platform": "ios" } }, - { "command": "wait", "positionals": ["label=\"Privacy & Security\"", "3000"], "flags": {} }, - { "command": "click", "positionals": ["label=\"Privacy & Security\""], "flags": {} }, - { "command": "get", "positionals": ["text", "label=\"Tracking\""], "flags": {} } -] -``` - -- `positionals` is optional and defaults to `[]`. -- `flags` is optional and defaults to `{}`. -- Only `command`, `positionals`, `flags`, and `runtime` are accepted as top-level step keys. -- Nested `batch` and `replay` are rejected. -- Supported error mode is stop-on-first-error. - -Response handling: - -- Success returns fields such as `total`, `executed`, `totalDurationMs`, and `results[]`. -- Human-mode `batch` runs also print a short per-step success summary. -- Failed runs include `details.step`, `details.command`, `details.executed`, and `details.partialResults`. -- Replan from the first failing step instead of rerunning the whole flow blindly. - -Canonical batch recipe: open app -> open action menu -> choose option -> verify - -```json -[ - { "command": "open", "positionals": ["com.example.app"], "flags": { "platform": "android" } }, - { "command": "wait", "positionals": ["text", "Home", "3000"], "flags": {} }, - { "command": "press", "positionals": ["label=\"More actions\" role=button"], "flags": {} }, - { "command": "wait", "positionals": ["text", "Camera scan", "2000"], "flags": {} }, - { "command": "press", "positionals": ["label=\"Camera scan\""], "flags": {} }, - { "command": "wait", "positionals": ["text", "Expense created", "15000"], "flags": {} }, - { "command": "is", "positionals": ["visible", "label=\"Expense created\""], "flags": {} } -] -``` - -Common batch error categories: - -- `INVALID_ARGS`: fix the payload shape and retry. -- `SESSION_NOT_FOUND`: open or select the correct session, then retry. -- `UNSUPPORTED_OPERATION`: switch to a supported command or surface. -- `AMBIGUOUS_MATCH`: refine the selector or locator, then retry the failed step. -- `COMMAND_FAILED`: add sync guards and retry from the failing step. - -## Stop conditions - -- If refs drift after transitions, switch to selectors. -- If a desktop surface or context menu is involved on macOS, load [macos-desktop.md](macos-desktop.md). -- If logs, network, alerts, or setup failures become the blocker, switch to [debugging.md](debugging.md). -- If the flow is stable and you need proof or replay maintenance, switch to [verification.md](verification.md). diff --git a/.claude/skills/agent-device/references/verification.md b/.claude/skills/agent-device/references/verification.md deleted file mode 100644 index 40500119fda5..000000000000 --- a/.claude/skills/agent-device/references/verification.md +++ /dev/null @@ -1,129 +0,0 @@ -# Verification - -## When to open this file - -Open this file when the task needs evidence, regression checks, replay maintenance, or session performance measurements after the main interaction flow is already working. - -## Main commands to reach for first - -- `screenshot` -- `diff snapshot` -- `diff screenshot` -- `record` -- `replay -u` -- `perf` - -## Most common mistake to avoid - -Do not use verification tools as the first exploration step. First get the app into the correct state with the normal interaction flow, then capture proof or maintain replay assets. - -## Canonical loop - -```bash -agent-device open Settings --platform ios -# after using exploration to reach the state you want to verify -agent-device snapshot -agent-device screenshot /tmp/settings-proof.png --overlay-refs -agent-device close -``` - -## Structural verification with diff snapshot - -Use `diff snapshot` when you need a compact view of how the UI changed between nearby states. - -```bash -agent-device snapshot -i -agent-device press @e5 -agent-device diff snapshot -i -``` - -- Initialize the baseline at a stable point. -- Perform the mutation. -- Run `diff snapshot` to confirm the expected structural change. -- Re-run full `snapshot` only when you need fresh refs. - -## Screenshot artifacts - -Use `screenshot` when the proof needs a rendered image instead of a structural tree. - -- Add `--overlay-refs` when you want the saved PNG to show fresh `@eN` refs burned into the screenshot. - -## Visual regression with diff screenshot - -Use `diff screenshot` when comparing the current rendered screen against a saved visual baseline. - -```bash -agent-device diff screenshot --baseline ./baseline.png --out /tmp/diff.png -agent-device diff screenshot --baseline ./baseline.png ./current.png --out /tmp/diff.png -agent-device diff screenshot --baseline ./baseline.png --out /tmp/diff.png --overlay-refs -``` - -- Text output includes ranked changed regions with screen-space rectangles, shape, size, density, average color, and luminance. JSON also includes normalized bounds. -- The diff PNG uses a light grayscale current-screen context with changed pixels tinted red and changed regions outlined. -- When a current image path is provided, `diff screenshot` compares the two saved files instead of capturing from the live device or requiring an active session. -- Install `tesseract` when you want `diff screenshot` to add best-effort OCR text deltas, movement clusters, and bbox size-change hints. OCR improves the text/JSON descriptions only; it does not change the pixel comparison or the diff PNG. -- When OCR is available, `diff screenshot` also reports best-effort non-text visual deltas by masking OCR text boxes out of the pixel diff and clustering the remaining residuals. Treat these as hints for icons, controls, and separators, not semantic icon recognition. -- Add `--overlay-refs` to `diff screenshot` when you also want a separate current-screen overlay guide for a live capture. The raw screenshot is still used for pixel comparison; the overlay guide is only context for non-text controls, icons, and tappable regions. When overlay refs intersect changed regions, the output lists the best current-screen ref matches under the affected region. Saved-image comparisons do not have live accessibility refs, so omit `--overlay-refs` when passing a current image path. - -## Session recording - -Use `record` for debugging, documentation, or shareable verification artifacts. - -```bash -agent-device record start ./recordings/ios.mov -agent-device open App -agent-device snapshot -i -agent-device press @e3 -agent-device close -agent-device record stop -``` - -- `record` supports iOS simulators, iOS devices, and Android. -- On iOS, recording is a wrapper around `simctl` for simulators and the corresponding device capture path for physical devices. -- On Android, recording is a wrapper around `adb`. -- Recording writes a video artifact and a gesture-telemetry sidecar JSON. -- On macOS hosts, touch overlay burn-in is available for supported recordings. -- On non-macOS hosts, recording still succeeds but the video stays raw and `record stop` can return an `overlayWarning`. -- If the agent already knows the interaction sequence and wants a more lifelike, uninterrupted recording, drive the flow with `batch` while recording instead of replanning between each step. - -Example: - -```bash -agent-device record start ./recordings/smoke.mov -agent-device batch --session sim --platform ios --steps-file /tmp/smoke-steps.json --json -agent-device record stop -``` - -- Use this only after exploration has stabilized the flow. -- Keep the batch short and add `wait` or `is exists` guards after mutating steps so the recorded flow still tracks realistic UI timing. - -## Replay maintenance - -Use replay updates when selectors drift but the recorded scenario is still correct. - -```bash -agent-device replay -u ./session.ad -agent-device test ./smoke --platform android -``` - -- Prefer selector-based actions in recorded `.ad` replays. -- Use `test` when you already have multiple `.ad` flows and need a quick regression pass after updating or recording them. -- Keep the skill-level rule simple: use `replay -u` to maintain one script, use `test` to verify a folder or matcher of scripts. -- Treat `test` as a human and CI-facing suite runner that an agent can invoke for verification, not as the main source of product documentation. -- Failed runs keep suite artifacts under `.agent-device/test-artifacts` by default, which is usually enough for debugging without extra agent-side processing. -- Use update mode for maintenance, not as a substitute for fixing a broken interaction strategy. - -## Performance checks - -Use `perf --json` or `metrics --json` when you need session performance data for the active session. - -```bash -agent-device open Settings --platform ios -agent-device perf --json -``` - -- `startup` is command round-trip timing around `open`. -- It is not true first-frame or first-interactive telemetry. -- Android app sessions also expose `memory` (`dumpsys meminfo`) and `cpu` (`dumpsys cpuinfo`) snapshots when the session has an app package context. -- Apple app sessions on macOS and iOS simulators also expose `memory` and `cpu` process snapshots when the session has an app bundle ID. -- `fps` is still unavailable, and physical iOS devices still leave `memory` and `cpu` unavailable in this release. From 3537d953cdb15a82f417e8b012801a5afc190877 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Kacper=20Miko=C5=82ajczak?= Date: Wed, 15 Apr 2026 00:06:00 +0200 Subject: [PATCH 09/14] Broaden agent-device skill scope and add project config - Widen skill trigger to cover testing, debugging, perf, bug repro, feature verification - Add usage principles (fail fast, deviations are signal) - Add early-development footnote with Expensify Slack contact - Add agent-device.json with iOS mobile defaults --- .claude/skills/agent-device/SKILL.md | 8 ++++++-- agent-device.json | 4 ++++ 2 files changed, 10 insertions(+), 2 deletions(-) create mode 100644 agent-device.json diff --git a/.claude/skills/agent-device/SKILL.md b/.claude/skills/agent-device/SKILL.md index 5bfe8e435768..6ea381e1ef91 100644 --- a/.claude/skills/agent-device/SKILL.md +++ b/.claude/skills/agent-device/SKILL.md @@ -1,6 +1,6 @@ --- name: agent-device -description: Automates interactions for Apple-platform apps (iOS, tvOS, macOS) and Android devices. Use when navigating apps, taking snapshots/screenshots, tapping, typing, scrolling, or extracting UI info across mobile, TV, and desktop targets. +description: Drive iOS and Android devices for the Expensify App - testing, debugging, performance profiling, bug reproduction, and feature verification. Use when the developer needs to interact with the mobile app on a device. --- # agent-device @@ -36,4 +36,8 @@ This skill enhances the local development and testing workflow. The developer ma ### Principles - **Fail fast.** If something deviates from expectations - wrong screen, unexpected error, missing element - stop, explain what happened and suggest how to get past it. Do not silently work around problems. -- **Deviations are signal.** If the app behaves differently than expected, report the delta clearly so the developer can decide what needs fixing. \ No newline at end of file +- **Deviations are signal.** If the app behaves differently than expected, report the delta clearly so the developer can decide what needs fixing. + +## Footnote + +> **Early development.** This integration between agent-device and the Expensify App is still in early stages. If you run into issues, please reach out via Expensify Slack or file an issue at https://github.com/callstackincubator/agent-device/issues. diff --git a/agent-device.json b/agent-device.json new file mode 100644 index 000000000000..bbb2d02e9870 --- /dev/null +++ b/agent-device.json @@ -0,0 +1,4 @@ +{ + "platform": "ios", + "target": "mobile" +} From d4abc1b434720779787d82ff8d82b8ac35a7b01b Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Kacper=20Miko=C5=82ajczak?= Date: Wed, 15 Apr 2026 00:09:03 +0200 Subject: [PATCH 10/14] Remove agent-device.json from PR --- agent-device.json | 4 ---- 1 file changed, 4 deletions(-) delete mode 100644 agent-device.json diff --git a/agent-device.json b/agent-device.json deleted file mode 100644 index bbb2d02e9870..000000000000 --- a/agent-device.json +++ /dev/null @@ -1,4 +0,0 @@ -{ - "platform": "ios", - "target": "mobile" -} From 44772360f581cb270dad70ca9ea726f085c7ee19 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Kacper=20Miko=C5=82ajczak?= Date: Mon, 20 Apr 2026 11:39:32 +0200 Subject: [PATCH 11/14] Drop footnote and how-this-works section from agent-device SKILL.md Both sections were not guiding the agent and duplicated what the agent-device CLI's bundled skills already cover. SKILL.md is now strictly a pre-flight install gate. --- .claude/skills/agent-device/SKILL.md | 13 ------------- 1 file changed, 13 deletions(-) diff --git a/.claude/skills/agent-device/SKILL.md b/.claude/skills/agent-device/SKILL.md index 6ea381e1ef91..47b8bcbea115 100644 --- a/.claude/skills/agent-device/SKILL.md +++ b/.claude/skills/agent-device/SKILL.md @@ -28,16 +28,3 @@ npm root -g ``` > **Do not proceed without `agent-device` installed.** All device interaction depends on it. - -## How This Skill Works - -This skill enhances the local development and testing workflow. The developer may provide a full test plan upfront, give step-by-step instructions interactively, or just point at a screen and ask questions. Follow their lead. - -### Principles - -- **Fail fast.** If something deviates from expectations - wrong screen, unexpected error, missing element - stop, explain what happened and suggest how to get past it. Do not silently work around problems. -- **Deviations are signal.** If the app behaves differently than expected, report the delta clearly so the developer can decide what needs fixing. - -## Footnote - -> **Early development.** This integration between agent-device and the Expensify App is still in early stages. If you run into issues, please reach out via Expensify Slack or file an issue at https://github.com/callstackincubator/agent-device/issues. From 1307ae5b90780b848415239665bf0ff976513d37 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Kacper=20Miko=C5=82ajczak?= Date: Mon, 20 Apr 2026 20:49:11 +0200 Subject: [PATCH 12/14] Inject agent-device pre-flight context automatically Replace the manual "ask the agent to run agent-device --version and npm root -g" instructions with dynamic context injection using the !`cmd` syntax. Commands run at skill load time (preprocessing), so the resolved version and canonical skill path land in the skill content directly - no tool call required from the agent. Pre-approves Bash(agent-device *) via allowed-tools in the skill frontmatter and also via .claude/settings.json so fresh checkouts do not get a permission prompt during the preprocessing step. Addresses https://github.com/Expensify/App/pull/87662#discussion_r3094245073 --- .claude/settings.json | 3 ++- .claude/skills/agent-device/SKILL.md | 25 +++++-------------------- 2 files changed, 7 insertions(+), 21 deletions(-) diff --git a/.claude/settings.json b/.claude/settings.json index a63b4a04a775..ff4eabddabb5 100644 --- a/.claude/settings.json +++ b/.claude/settings.json @@ -52,7 +52,8 @@ "mcp__sentry__search_events", "mcp__sentry__search_issue_events", "mcp__sentry__search_issues", - "mcp__sentry__whoami" + "mcp__sentry__whoami", + "Bash(agent-device *)" ] }, "enabledPlugins": { diff --git a/.claude/skills/agent-device/SKILL.md b/.claude/skills/agent-device/SKILL.md index 47b8bcbea115..b542aae75be5 100644 --- a/.claude/skills/agent-device/SKILL.md +++ b/.claude/skills/agent-device/SKILL.md @@ -1,30 +1,15 @@ --- name: agent-device description: Drive iOS and Android devices for the Expensify App - testing, debugging, performance profiling, bug reproduction, and feature verification. Use when the developer needs to interact with the mobile app on a device. +allowed-tools: Bash(agent-device *) Bash(npm root *) --- # agent-device -## Pre-flight Check +## Pre-flight -Verify the `agent-device` CLI is installed and its skills are accessible: +`agent-device` CLI version: !`agent-device --version 2>&1 || echo "NOT_INSTALLED"` -```bash -agent-device --version -``` +Canonical skill reference path (read these files directly for device automation guidance - bootstrap, exploration, verification, debugging): !`echo "$(npm root -g)/agent-device/skills/agent-device"` -If missing, **STOP** and instruct the user to install it: - -```bash -npm install -g agent-device -``` - -The `agent-device` CLI ships with built-in skills under `skills/` in the installed package. These contain the canonical reference for device automation - bootstrap, exploration, verification, debugging, and more. Use `agent-device --help` to discover available commands and skill names. Read the skill files directly from the installed package path when you need detailed guidance: - -```bash -# Find the package location -npm root -g -# Then read: /agent-device/skills/agent-device/SKILL.md -``` - -> **Do not proceed without `agent-device` installed.** All device interaction depends on it. +> If the version line above shows `NOT_INSTALLED` or a command-not-found error, **STOP** and instruct the developer to install it: `npm install -g agent-device`. All device interaction depends on it. From a9934fb8c448de31d77ae2cc43ddc5d8397bc758 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Kacper=20Miko=C5=82ajczak?= Date: Mon, 20 Apr 2026 21:02:26 +0200 Subject: [PATCH 13/14] Surface agent-device skill in CLAUDE.md and README Add a Mobile Device Testing subsection parallel to Browser Testing in CLAUDE.md, and an optional AI-assisted testing callout in README after Platform-Specific Setup. Makes the agent-device skill discoverable for Claude Code users without claiming it's required setup. --- CLAUDE.md | 3 +++ README.md | 2 ++ 2 files changed, 5 insertions(+) diff --git a/CLAUDE.md b/CLAUDE.md index 56cbf058019e..750df2c0d848 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -277,6 +277,9 @@ npm run web ### Browser Testing Use the `/playwright-app-testing` skill to test and debug the App in a browser. Use this skill after making frontend changes to verify your work, or when the user requests testing. +### Mobile Device Testing +Use the `/agent-device` skill to drive the App on iOS and Android (simulators or real devices) for interactive testing, performance profiling, bug reproduction, and device-specific debugging. Requires `npm install -g agent-device` - the skill's pre-flight check will surface the install instruction if missing. + ## Architecture Decisions ### React Native New Architecture diff --git a/README.md b/README.md index 3bc69752ef74..a46e1b31cc5a 100644 --- a/README.md +++ b/README.md @@ -48,6 +48,8 @@ For detailed setup instructions for each platform, see the following guides: * **📱 iOS Development**: [iOS Setup Instructions](contributingGuides/SETUP_IOS.md) * **🤖 Android Development**: [Android Setup Instructions](contributingGuides/SETUP_ANDROID.md) +**Optional AI-assisted mobile testing:** If you use Claude Code, the [`/agent-device` skill](.claude/skills/agent-device/SKILL.md) drives iOS and Android simulators or devices for interactive testing, debugging, and performance profiling. Requires `npm install -g agent-device`. + ## General Troubleshooting 1. If you are having issues with **_Getting Started_**, please reference [React Native's Documentation](https://reactnative.dev/docs/environment-setup) 2. If you are running into CORS errors like (in the browser dev console) From 3fc5701e5c7429f7e911a57cccd3224347128c49 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Kacper=20Miko=C5=82ajczak?= Date: Tue, 21 Apr 2026 21:59:18 +0200 Subject: [PATCH 14/14] Allow agent-device skill path echo in permissions allowlist The skill bootstraps by reading files from the installed npm package, resolved via an echo bang-command. Without this allowlist entry, every skill load prompted for permission. --- .claude/settings.json | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/.claude/settings.json b/.claude/settings.json index ff4eabddabb5..3cd2d1b82934 100644 --- a/.claude/settings.json +++ b/.claude/settings.json @@ -53,7 +53,8 @@ "mcp__sentry__search_issue_events", "mcp__sentry__search_issues", "mcp__sentry__whoami", - "Bash(agent-device *)" + "Bash(agent-device *)", + "Bash(echo \"$(npm root -g)/agent-device/skills/agent-device\")" ] }, "enabledPlugins": {