Skip to content

Melvin ability to run web and mobile device testing #87030

Closed
17 of 18 issues completed
Closed

Description

@Julesssss

Problem

We are using playwright within Melvin for simple bug verification and test fixes (limited to web).
Our plans to extend Melvin include improved testing and expanding testing functionality to mobile.

agent-device looks like a great potential tool for this. The demo shows fast UI testing of the Expensify ND app.

Solution

Lets create a POC using the tool. Initially we would like to enable Melvin to:

  1. Verify test cases similar to playwright
  2. Post screenshots to App PRs (building into Neil's work)

@AndrewGable @justinpersaud


  • Here's our existing Playwright skill
  • Here's documentation on Melvin
  • Here's how Melvin uploads images and attatches them to Github comments
  • Here's a collection of test runs

Initial Questions:

  • Limitations?
  • How to integrate with Melvin, which is designed for one-shot requests

Activity

  1. adhorodyski commented on Apr 3, 2026

    @adhorodyski
    Contributor

    @Julesssss looks like I'm missing access to the main repo with Melvin's setup, can you help with that?

  2. Julesssss commented on Apr 3, 2026

    @Julesssss
    ContributorAuthor

    @justinpersaud would you mind inviting @adhorodyski to E/E please.

  3. kacper-mikolajczak commented on Apr 8, 2026

    @kacper-mikolajczak
    Contributor

    Hi @Julesssss! After chat with Adam, I will push this effort from CK side forwards.

    Let's sync up on the exact success criteria for this exploration. Could you start with more in-depth description of how you'd like to see it being used and integrated? It should help us be on the same page before jumping into implementation phase. Thanks ❤️

  4. Julesssss commented on Apr 9, 2026

    @Julesssss
    ContributorAuthor

    Hey @kacper-mikolajczak, sure. So this is ideally a quick POC to see if it will be possible without too much compromise.

    Plan:

    • Give Melvin the ability to test HybridApp builds
    • Give Melvin ability to screenshot during these tests
      • We have existing logic for storing the screenshots so we can attach them to Melvin's GitHub comments

    We have two Melvin workflows:

    • Melvin Anywhere: Has full dev environment
    • Melvin App: Has access to App, can make web builds pointing to prod
      • We should aim for this flow only, keep it simple
  5. justinpersaud commented on Apr 9, 2026

    @justinpersaud
    Contributor

    Added @adhorodyski to Callstack AI development team which should grant access

  6. kacper-mikolajczak commented on Apr 10, 2026

    @kacper-mikolajczak
    Contributor

    Hey folks 👋 Here's a design doc on the PoC after our internal discussion with @Julesssss:

    PoC Design: Agentic Mobile Testing for Melvin

    Goal

    Improve PR QA/review process by enabling agentic mobile testing via agent-device, integrated with the existing MelvinBot App workflow. Additionally, enable agent-device for local developer use.

    Background

    MelvinBot currently handles ~200 requests per 36 hours on Expensify/App - automated issue analysis, CI fixes, code review, implementation, and more. The workflow (claude-melvinbot-app.yml) includes Playwright browser testing infrastructure, but analysis of ~650 production runs shows it's invoked only ~0.3% of the time. Mobile testing is entirely absent.

    agent-device is a CLI tool by Callstack that gives AI agents control over mobile app UIs using accessibility tree snapshots (not screenshots), making interactions token-efficient and deterministic. It supports Android emulators headlessly on Linux - a natural fit for the existing CI setup.

    Architecture

                            MelvinBot App Workflow
                         (claude-melvinbot-app.yml)
                                   |
                +------------------+------------------+
                |                                     |
         Existing steps                        New steps (PoC)
         (unchanged)                           (added to workflow)
                |                                     |
      +---------+---------+              +------------+------------+
      | Checkout repos    |              | Enable KVM              |
      | Setup Node        |              | Install Android SDK     |
      | Install App deps  |              | Install agent-device    |
      | Configure Git     |              | Install skill           |
      | Playwright setup  |              +------------+------------+
      +---------+---------+                           |
                |                                     |
                +------------------+------------------+
                                   |
                        Claude Code Action runs
                                   |
                     Agent decides what to do:
                                   |
                  +----------------+----------------+
                  |                |                |
            Code analysis    Web testing      Mobile testing
            (default)        (Playwright)     (agent-device)
                                                   |
                                        +----------+----------+
                                        | Fetch APK from Rock |
                                        | Boot emulator       |
                                        | Install + open app  |
                                        | Interact + verify   |
                                        | Upload screenshots  |
                                        | Shutdown emulator   |
                                        +---------------------+
    

    Mobile testing flow (when triggered)

    +-------------------+     +-------------------+     +--------------------+
    | 1. Fetch APK      |---->| 2. Boot emulator  |---->| 3. Install app     |
    | rock remote-cache |     | agent-device boot |     | agent-device       |
    | download          |     | --headless        |     | install <app> <apk>|
    +-------------------+     +-------------------+     +--------------------+
                                                                 |
    +-------------------+     +-------------------+     +--------v-----------+
    | 6. Shutdown       |<----| 5. Screenshot +   |<----| 4. Interact +      |
    | agent-device      |     | upload to R2      |     | verify via         |
    | close --shutdown  |     | (existing infra)  |     | accessibility tree |
    +-------------------+     +-------------------+     +--------------------+
    

    APK delivery

    The workflow leverages Rock's remote build cache to skip native builds entirely:

    +-----------------------------+
    | remote-build-android.yml    |  Runs on every push,
    | (existing App workflow)     |  caches Debug APK to S3
    +-------------+---------------+
                  |
                  | fingerprint + APK artifact
                  v
    +-----------------------------+
    | S3: ad-hoc-expensify-cash   |  Public-read bucket
    | rock-android-*-<hash>/      |  x86_64 compatible
    +-------------+---------------+
                  |
                  | npx rock remote-cache download
                  | (no AWS creds needed)
                  v
    +-----------------------------+
    | MelvinBot workflow          |  APK ready for emulator
    | .rock/cache/android/        |  install
    +-----------------------------+
    

    Primary (Option A): Agent runs npx rock remote-cache download -p android -t developmentDebug which resolves the fingerprint and downloads the matching cached APK from S3. This is the most robust approach - fingerprint matching ensures the correct build for the current native state.

    Fallback (Option B): If a ROCK_ARTIFACT_URL exists from a prior Adhoc build, agent uses agent-device install-from-source <url> to download and install directly. Simpler but requires a prior build.

    Timeline

    Phase 1                   Phase 2                   Phase 3
    |----- Local dev -------->|----- Melvin CI -------->|----- Expand ---------->
    
     - agent-device docs       - Testing strategy        - iOS simulator support
     - Example test flows      - PoC validation          - Broader QA automation
     - SKILL.md in App repo    - Workflow changes         - Usage analysis
                               - Melvin skill
                               - Screenshot upload
                               - End-to-end test
    

    Phase 1 and 2 can be taken in parallel.

    Phase 1 - Local developer setup

    Deliverable Description
    Developer docs Instructions for running agent-device against Expensify/App locally (Android emulator/iOS simulator)
    Example test flows Scripted flows for common scenarios (onboarding, expense creation, navigation)
    Claude Code skill SKILL.md in App repo so developers can use agent-device through Claude Code locally

    Phase 2 - Melvin CI integration

    Deliverable Description
    Testing strategy Define what Melvin should verify on Android, where testing steps come from (PR Mobile QA section), and test account / auth approach
    PoC validation Verify the end-to-end emulator flow works on CI runners (boot, APK install, app interaction, shutdown) and iterate on any issues
    Workflow changes Add Android emulator + agent-device setup steps to claude-melvinbot-app.yml
    Melvin skill Claude Code skill teaching Melvin when/how to use agent-device for mobile testing
    Screenshot integration Reuse existing R2 upload infra for agent-device screenshots
    End-to-end test Trigger Melvin with a mobile-testable request, verify full flow

    Phase 3 - Expand

    • iOS simulator support (agent-device already supports iOS; requires macOS runner)
    • Broader QA triggers and automation
    • Analyze usage and value to reviewers

    Workflow changes (detailed)

    New steps added to claude-melvinbot-app.yml before Claude Code runs:

    # 1. Hardware-accelerated Android emulation
    - name: Enable KVM
      run: |
        echo 'KERNEL=="kvm", GROUP="kvm", MODE="0666", OPTIONS+="static_node=kvm"' \
          | sudo tee /etc/udev/rules.d/99-kvm4all.rules
        sudo udevadm control --reload-rules
        sudo udevadm trigger --name-match=kvm
    
    # 2. Android SDK components (runner already has base SDK)
    - name: Install Android emulator and system image
      run: |
        yes | sdkmanager --install "emulator"
        yes | sdkmanager --install "system-images;android-34;google_atd;x86_64"
    
    # 3. agent-device requires Node 22; App requires Node 20
    - name: Setup Node 22 for agent-device
      uses: actions/setup-node@v4
      with:
        node-version: '22'
    
    - name: Install agent-device
      run: npm install -g agent-device
    
    - name: Restore App Node version
      uses: actions/setup-node@v4
      with:
        node-version-file: App/.nvmrc
    
    # 4. Install agent-device Claude Code skill
    - name: Install agent-device skill
      run: cd App && npx skills add callstackincubator/agent-device

    What stays the same:

    • Runner: blacksmith-4vcpu-ubuntu-2404 (upgrade to 8 vCPU if insufficient)
    • Timeout: 30 min (adjust if needed)
    • Playwright setup: untouched, remains available
    • Screenshot upload: reuses existing R2 infra (melvinbot-assets.exops.io)

    Design decisions

    Decision Choice Rationale
    Platform Android only (PoC) Avoids macOS runner cost; Linux + KVM is sufficient for validation
    APK source Rock remote-cache (primary) Fingerprint matching ensures correct build; public-read S3, no creds needed
    SDK setup Raw sdkmanager Runner already has Android SDK installed; no third-party action needed
    Caching None for PoC System image cache saves ~60s; not worth the complexity yet
    Node version Dual setup-node (22 + restore 20) App engines field locks to 20.19.5; agent-device requires 22
    Trigger Skill-based, on-demand Emulator only boots when agent decides to test; no overhead on every run
    Emulator lifecycle agent-device manages Handles boot, install, interaction, and shutdown autonomously
    Perception model Accessibility tree snapshots ~3-4x more token-efficient than screenshots; deterministic element refs

    Action items

    1. Write developer docs and Claude Code skill for local agent-device usage (Phase 1)
    2. Define testing strategy - what to verify, where testing steps come from (PR Mobile QA section), auth approach
    3. Add Android emulator + agent-device setup steps to claude-melvinbot-app.yml
    4. Write Melvin skill for mobile testing with agent-device
    5. Integrate screenshot upload with existing R2 infra
    6. End-to-end validation - trigger Melvin with a mobile-testable request, verify full flow

    Open questions

    1. Where should testing steps come from? The PR template already includes # Mobile QA and # Web QA sections. Proposal: Melvin writes Mobile QA steps when creating the PR, then uses those same steps to drive agent-device verification. This mirrors the human QA process and requires no extra input. When mobile testing is triggered manually (e.g. @MelvinBot test this on Android), it can also pull from the existing PR testing steps rather than requiring the user to restate them. Currently the Playwright app-web-tester receives freeform TASK/VERIFY from melvinbot with no defined source - using the PR testing steps would be an improvement for both web and mobile testing.

    2. Runner capacity. Running Android emulator + Claude Code on 4 vCPU may be tight. Will test during PoC validation and upgrade to 8 vCPU if needed.

    PoC implementations

    Although still under construction, here are the draft PRs if you'd like to peek a bit more than just high level overview:


    I am looking forward for your thoughts and improvements, thanks a lot!

    CC @adhorodyski @Julesssss @AndrewGable @justinpersaud

  7. kacper-mikolajczak commented on Apr 13, 2026

    @kacper-mikolajczak
    Contributor

    Hi folks! Let me know what you think and how would you like to see it moving forwards. Thanks!

  8. Julesssss commented on Apr 13, 2026

    @Julesssss
    ContributorAuthor

    The plan sounds good, I am mostly excited for phase 1 currently. Lets see how this goes before moving onto 2/3.

    Image

    I think it's fine. You can see that we have many possible flows in our current melvin flow. There are plans to extract these out, but for now we need to make sure the flow isn't confused. For example, web screenshot and upload might be tightly coupled with playwright-app-testing.

    Caching | None for PoC

    Just thinking ahead, this would be cache of the rock build, and only helpful on the second+ test run?

    Open questions

    Where should testing steps come from?

    Testing:

    • Local: Directed by user
    • PRs authored by Melvin:
      • We recently attempted to make Melvin verify and list it's changes seperately from engineering verifications
      • To begin with, a few standard app smoke tests would be good, following that we could have run tests related to changes

    Runner capacity. Running Android emulator + Claude Code on 4 vCPU may be tight. Will test during PoC validation and upgrade to 8 vCPU if needed.

    The app testing workflow runs on the larger Github Action runner currently, so I don't think this is a problem.

  9. 138 remaining items

  10. melvin-bot commented on Aug 18, 2026

    @melvin-bot
  11. changed the title [-][Due for payment 2026-08-25] Melvin ability to run web and mobile device testing[/-] [+]Melvin ability to run web and mobile device testing[/+] on Aug 18, 2026
  12. Julesssss commented on Aug 18, 2026

    @Julesssss
    ContributorAuthor

    With Kacpers recent fixes, we want to run as many Melvin test runs as possible. Lets check in daily with the runs and compare success/costs. Starting tomorrow❗

    Today @rlinoz added a simple check that adds the Web/Android testing labels to it's own PRs. It's working here. Although the Android run failed.

    @kacper-mikolajczak I think for now this simpler method should be fine for getting Melvin to run testing on it's own PRs, what do you think? Please keep your draft PR for when we want to improve the kick off process.

    Then if you can continue monitoring these runs and making improvements, we should be able to share with the team soon! It would also be great if you could prioritise your draft for this issue, just a link back to the workflow so we can more easily debug the failures.

  13. kacper-mikolajczak commented on Aug 18, 2026

    @kacper-mikolajczak
    Contributor

    Exciting, let's do that! Let me pivot to the mentioned issue tomorrow and monitor upcoming runs 🚀

  14. kacper-mikolajczak commented on Aug 18, 2026

    @kacper-mikolajczak
    Contributor

    @Julesssss checked the android logs and it looks like failure came from the infra - it was stopped during emulator step with such error:

    The self-hosted runner lost communication with the server. Verify the machine is running and has a healthy network connection

  15. Julesssss commented on Aug 25, 2026

    @Julesssss
    ContributorAuthor

    We reviewed latest results today here. Mean cost was $3 Android / $5 web which isn't bad as we can reduce this in the future.

    The last step before we share this testing tool with the team is to solve common problems the agent gets stuck at.

  16. melvin-bot commented on Sep 18, 2026

    @melvin-bot

    This issue has not been updated in over 15 days. @rlinoz, @Julesssss, @kacper-mikolajczak eroding to Monthly issue.

    P.S. Is everyone reading this sure this is really a near-term priority? Be brave: if you disagree, go ahead and close it out. If someone disagrees, they'll reopen it, and if they don't: one less thing to do!

  17. rlinoz commented on Sep 28, 2026

    @rlinoz
    Contributor

    Still improving on this.

  18. Julesssss commented on Sep 29, 2026

    @Julesssss
    ContributorAuthor

    Melvin is testing App PRs consistently. next step would be to give it ability to solve some specific test cases that it struggles with, which we are discussing here:

    • Test requires 2 accounts
    • Test requires account to be validated
    • Test requires private domain

    Closing this out!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Labels

EngineeringInternalRequires API changes or must be handled by Expensify staffMonthlyKSv2ReviewingHas a PR in review

Type

No type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions