You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Melvin ability to run web and mobile device testing #87030
We are using playwright within Melvin for simple bug verification and test fixes (limited to web).
Our plans to extend Melvin include improved testing and expanding testing functionality to mobile.
agent-devicelooks like a great potential tool for this. The demo shows fast UI testing of the Expensify ND app.
Solution
Lets create a POC using the tool. Initially we would like to enable Melvin to:
Verify test cases similar to playwright
Post screenshots to App PRs (building into Neil's work)
Hi @Julesssss! After chat with Adam, I will push this effort from CK side forwards.
Let's sync up on the exact success criteria for this exploration. Could you start with more in-depth description of how you'd like to see it being used and integrated? It should help us be on the same page before jumping into implementation phase. Thanks ❤️
Hey folks 👋 Here's a design doc on the PoC after our internal discussion with @Julesssss:
PoC Design: Agentic Mobile Testing for Melvin
Goal
Improve PR QA/review process by enabling agentic mobile testing via agent-device, integrated with the existing MelvinBot App workflow. Additionally, enable agent-device for local developer use.
Background
MelvinBot currently handles ~200 requests per 36 hours on Expensify/App - automated issue analysis, CI fixes, code review, implementation, and more. The workflow (claude-melvinbot-app.yml) includes Playwright browser testing infrastructure, but analysis of ~650 production runs shows it's invoked only ~0.3% of the time. Mobile testing is entirely absent.
agent-device is a CLI tool by Callstack that gives AI agents control over mobile app UIs using accessibility tree snapshots (not screenshots), making interactions token-efficient and deterministic. It supports Android emulators headlessly on Linux - a natural fit for the existing CI setup.
The workflow leverages Rock's remote build cache to skip native builds entirely:
+-----------------------------+
| remote-build-android.yml | Runs on every push,
| (existing App workflow) | caches Debug APK to S3
+-------------+---------------+
|
| fingerprint + APK artifact
v
+-----------------------------+
| S3: ad-hoc-expensify-cash | Public-read bucket
| rock-android-*-<hash>/ | x86_64 compatible
+-------------+---------------+
|
| npx rock remote-cache download
| (no AWS creds needed)
v
+-----------------------------+
| MelvinBot workflow | APK ready for emulator
| .rock/cache/android/ | install
+-----------------------------+
Primary (Option A): Agent runs npx rock remote-cache download -p android -t developmentDebug which resolves the fingerprint and downloads the matching cached APK from S3. This is the most robust approach - fingerprint matching ensures the correct build for the current native state.
Fallback (Option B): If a ROCK_ARTIFACT_URL exists from a prior Adhoc build, agent uses agent-device install-from-source <url> to download and install directly. Simpler but requires a prior build.
Timeline
Phase 1 Phase 2 Phase 3
|----- Local dev -------->|----- Melvin CI -------->|----- Expand ---------->
- agent-device docs - Testing strategy - iOS simulator support
- Example test flows - PoC validation - Broader QA automation
- SKILL.md in App repo - Workflow changes - Usage analysis
- Melvin skill
- Screenshot upload
- End-to-end test
Phase 1 and 2 can be taken in parallel.
Phase 1 - Local developer setup
Deliverable
Description
Developer docs
Instructions for running agent-device against Expensify/App locally (Android emulator/iOS simulator)
Example test flows
Scripted flows for common scenarios (onboarding, expense creation, navigation)
Claude Code skill
SKILL.md in App repo so developers can use agent-device through Claude Code locally
Phase 2 - Melvin CI integration
Deliverable
Description
Testing strategy
Define what Melvin should verify on Android, where testing steps come from (PR Mobile QA section), and test account / auth approach
PoC validation
Verify the end-to-end emulator flow works on CI runners (boot, APK install, app interaction, shutdown) and iterate on any issues
Workflow changes
Add Android emulator + agent-device setup steps to claude-melvinbot-app.yml
Melvin skill
Claude Code skill teaching Melvin when/how to use agent-device for mobile testing
Screenshot integration
Reuse existing R2 upload infra for agent-device screenshots
End-to-end test
Trigger Melvin with a mobile-testable request, verify full flow
Phase 3 - Expand
iOS simulator support (agent-device already supports iOS; requires macOS runner)
Broader QA triggers and automation
Analyze usage and value to reviewers
Workflow changes (detailed)
New steps added to claude-melvinbot-app.yml before Claude Code runs:
Runner: blacksmith-4vcpu-ubuntu-2404 (upgrade to 8 vCPU if insufficient)
Timeout: 30 min (adjust if needed)
Playwright setup: untouched, remains available
Screenshot upload: reuses existing R2 infra (melvinbot-assets.exops.io)
Design decisions
Decision
Choice
Rationale
Platform
Android only (PoC)
Avoids macOS runner cost; Linux + KVM is sufficient for validation
APK source
Rock remote-cache (primary)
Fingerprint matching ensures correct build; public-read S3, no creds needed
SDK setup
Raw sdkmanager
Runner already has Android SDK installed; no third-party action needed
Caching
None for PoC
System image cache saves ~60s; not worth the complexity yet
Node version
Dual setup-node (22 + restore 20)
App engines field locks to 20.19.5; agent-device requires 22
Trigger
Skill-based, on-demand
Emulator only boots when agent decides to test; no overhead on every run
Emulator lifecycle
agent-device manages
Handles boot, install, interaction, and shutdown autonomously
Perception model
Accessibility tree snapshots
~3-4x more token-efficient than screenshots; deterministic element refs
Action items
Write developer docs and Claude Code skill for local agent-device usage (Phase 1)
Define testing strategy - what to verify, where testing steps come from (PR Mobile QA section), auth approach
Add Android emulator + agent-device setup steps to claude-melvinbot-app.yml
Write Melvin skill for mobile testing with agent-device
Integrate screenshot upload with existing R2 infra
End-to-end validation - trigger Melvin with a mobile-testable request, verify full flow
Open questions
Where should testing steps come from? The PR template already includes # Mobile QA and # Web QA sections. Proposal: Melvin writes Mobile QA steps when creating the PR, then uses those same steps to drive agent-device verification. This mirrors the human QA process and requires no extra input. When mobile testing is triggered manually (e.g. @MelvinBot test this on Android), it can also pull from the existing PR testing steps rather than requiring the user to restate them. Currently the Playwright app-web-tester receives freeform TASK/VERIFY from melvinbot with no defined source - using the PR testing steps would be an improvement for both web and mobile testing.
Runner capacity. Running Android emulator + Claude Code on 4 vCPU may be tight. Will test during PoC validation and upgrade to 8 vCPU if needed.
PoC implementations
Although still under construction, here are the draft PRs if you'd like to peek a bit more than just high level overview:
The plan sounds good, I am mostly excited for phase 1 currently. Lets see how this goes before moving onto 2/3.
I think it's fine. You can see that we have many possible flows in our current melvin flow. There are plans to extract these out, but for now we need to make sure the flow isn't confused. For example, web screenshot and upload might be tightly coupled with playwright-app-testing.
Caching | None for PoC
Just thinking ahead, this would be cache of the rock build, and only helpful on the second+ test run?
Open questions
Where should testing steps come from?
Testing:
Local: Directed by user
PRs authored by Melvin:
We recently attempted to make Melvin verify and list it's changes seperately from engineering verifications
To begin with, a few standard app smoke tests would be good, following that we could have run tests related to changes
Runner capacity. Running Android emulator + Claude Code on 4 vCPU may be tight. Will test during PoC validation and upgrade to 8 vCPU if needed.
The app testing workflow runs on the larger Github Action runner currently, so I don't think this is a problem.
changed the title [-][Due for payment 2026-08-25] Melvin ability to run web and mobile device testing[/-][+]Melvin ability to run web and mobile device testing[/+]on Aug 18, 2026
With Kacpers recent fixes, we want to run as many Melvin test runs as possible. Lets check in daily with the runs and compare success/costs. Starting tomorrow❗
Today @rlinoz added a simple check that adds the Web/Android testing labels to it's own PRs. It's working here. Although the Android run failed.
@kacper-mikolajczak I think for now this simpler method should be fine for getting Melvin to run testing on it's own PRs, what do you think? Please keep your draft PR for when we want to improve the kick off process.
Then if you can continue monitoring these runs and making improvements, we should be able to share with the team soon! It would also be great if you could prioritise your draft for this issue, just a link back to the workflow so we can more easily debug the failures.
P.S. Is everyone reading this sure this is really a near-term priority? Be brave: if you disagree, go ahead and close it out. If someone disagrees, they'll reopen it, and if they don't: one less thing to do!
Melvin is testing App PRs consistently. next step would be to give it ability to solve some specific test cases that it struggles with, which we are discussing here:
Activity
adhorodyski commentedon Apr 3, 2026
@Julesssss looks like I'm missing access to the main repo with Melvin's setup, can you help with that?
Julesssss commentedon Apr 3, 2026
@justinpersaud would you mind inviting @adhorodyski to E/E please.
kacper-mikolajczak commentedon Apr 8, 2026
Hi @Julesssss! After chat with Adam, I will push this effort from CK side forwards.
Let's sync up on the exact success criteria for this exploration. Could you start with more in-depth description of how you'd like to see it being used and integrated? It should help us be on the same page before jumping into implementation phase. Thanks ❤️
Julesssss commentedon Apr 9, 2026
Hey @kacper-mikolajczak, sure. So this is ideally a quick POC to see if it will be possible without too much compromise.
Plan:
We have two Melvin workflows:
justinpersaud commentedon Apr 9, 2026
Added @adhorodyski to
Callstack AI developmentteam which should grant accesskacper-mikolajczak commentedon Apr 10, 2026
Hey folks 👋 Here's a design doc on the PoC after our internal discussion with @Julesssss:
PoC Design: Agentic Mobile Testing for Melvin
Goal
Improve PR QA/review process by enabling agentic mobile testing via agent-device, integrated with the existing MelvinBot App workflow. Additionally, enable agent-device for local developer use.
Background
MelvinBot currently handles ~200 requests per 36 hours on Expensify/App - automated issue analysis, CI fixes, code review, implementation, and more. The workflow (
claude-melvinbot-app.yml) includes Playwright browser testing infrastructure, but analysis of ~650 production runs shows it's invoked only ~0.3% of the time. Mobile testing is entirely absent.agent-device is a CLI tool by Callstack that gives AI agents control over mobile app UIs using accessibility tree snapshots (not screenshots), making interactions token-efficient and deterministic. It supports Android emulators headlessly on Linux - a natural fit for the existing CI setup.
Architecture
Mobile testing flow (when triggered)
APK delivery
The workflow leverages Rock's remote build cache to skip native builds entirely:
Primary (Option A): Agent runs
npx rock remote-cache download -p android -t developmentDebugwhich resolves the fingerprint and downloads the matching cached APK from S3. This is the most robust approach - fingerprint matching ensures the correct build for the current native state.Fallback (Option B): If a
ROCK_ARTIFACT_URLexists from a prior Adhoc build, agent usesagent-device install-from-source <url>to download and install directly. Simpler but requires a prior build.Timeline
Phase 1 - Local developer setup
Phase 2 - Melvin CI integration
claude-melvinbot-app.ymlPhase 3 - Expand
Workflow changes (detailed)
New steps added to
claude-melvinbot-app.ymlbefore Claude Code runs:What stays the same:
blacksmith-4vcpu-ubuntu-2404(upgrade to 8 vCPU if insufficient)melvinbot-assets.exops.io)Design decisions
sdkmanagerenginesfield locks to 20.19.5; agent-device requires 22Action items
claude-melvinbot-app.ymlOpen questions
Where should testing steps come from? The PR template already includes
# Mobile QAand# Web QAsections. Proposal: Melvin writes Mobile QA steps when creating the PR, then uses those same steps to drive agent-device verification. This mirrors the human QA process and requires no extra input. When mobile testing is triggered manually (e.g.@MelvinBot test this on Android), it can also pull from the existing PR testing steps rather than requiring the user to restate them. Currently the Playwrightapp-web-testerreceives freeform TASK/VERIFY from melvinbot with no defined source - using the PR testing steps would be an improvement for both web and mobile testing.Runner capacity. Running Android emulator + Claude Code on 4 vCPU may be tight. Will test during PoC validation and upgrade to 8 vCPU if needed.
PoC implementations
Although still under construction, here are the draft PRs if you'd like to peek a bit more than just high level overview:
I am looking forward for your thoughts and improvements, thanks a lot!
CC @adhorodyski @Julesssss @AndrewGable @justinpersaud
kacper-mikolajczak commentedon Apr 13, 2026
Hi folks! Let me know what you think and how would you like to see it moving forwards. Thanks!
Julesssss commentedon Apr 13, 2026
The plan sounds good, I am mostly excited for phase 1 currently. Lets see how this goes before moving onto 2/3.
I think it's fine. You can see that we have many possible flows in our current melvin flow. There are plans to extract these out, but for now we need to make sure the flow isn't confused. For example, web screenshot and upload might be tightly coupled with
playwright-app-testing.Just thinking ahead, this would be cache of the rock build, and only helpful on the second+ test run?
Open questions
Testing:
The app testing workflow runs on the larger Github Action runner currently, so I don't think this is a problem.
138 remaining items
melvin-bot commentedon Aug 18, 2026
[-][Due for payment 2026-08-25] Melvin ability to run web and mobile device testing[/-][+]Melvin ability to run web and mobile device testing[/+]Julesssss commentedon Aug 18, 2026
With Kacpers recent fixes, we want to run as many Melvin test runs as possible. Lets check in daily with the runs and compare success/costs. Starting tomorrow❗
Today @rlinoz added a simple check that adds the Web/Android testing labels to it's own PRs. It's working here. Although the Android run failed.
@kacper-mikolajczak I think for now this simpler method should be fine for getting Melvin to run testing on it's own PRs, what do you think? Please keep your draft PR for when we want to improve the kick off process.
Then if you can continue monitoring these runs and making improvements, we should be able to share with the team soon! It would also be great if you could prioritise your draft for this issue, just a link back to the workflow so we can more easily debug the failures.
kacper-mikolajczak commentedon Aug 18, 2026
Exciting, let's do that! Let me pivot to the mentioned issue tomorrow and monitor upcoming runs 🚀
kacper-mikolajczak commentedon Aug 18, 2026
@Julesssss checked the android logs and it looks like failure came from the infra - it was stopped during emulator step with such error:
Julesssss commentedon Aug 25, 2026
We reviewed latest results today here. Mean cost was $3 Android / $5 web which isn't bad as we can reduce this in the future.
The last step before we share this testing tool with the team is to solve common problems the agent gets stuck at.
melvin-bot commentedon Sep 18, 2026
This issue has not been updated in over 15 days. @rlinoz, @Julesssss, @kacper-mikolajczak eroding to Monthly issue.
P.S. Is everyone reading this sure this is really a near-term priority? Be brave: if you disagree, go ahead and close it out. If someone disagrees, they'll reopen it, and if they don't: one less thing to do!
rlinoz commentedon Sep 28, 2026
Still improving on this.
Julesssss commentedon Sep 29, 2026
Melvin is testing App PRs consistently. next step would be to give it ability to solve some specific test cases that it struggles with, which we are discussing here:
Closing this out!