简体中文 | English
Failure-Pruned Skill Learning: Replay-Verified Self-Improvement for GUI Agents
A self-iterating harness that helps GUI agents learn from real execution failures, repair their operating experience, and promote candidate knowledge into verified skills through real replay.
Failures prune exploration. Successes anchor experience. Replay verifies skill.
MagicGUI-Agent-Harness focuses on a simple but important question:
When a GUI agent cannot solve a task, or cannot solve it reliably, can it use its own real executions to improve round by round?
MagicGUI treats every real GUI rollout as learning evidence. A failure is not just a 0 score: it helps locate bottlenecks, prune unproductive routes, and guide the next attempt. A success is not copied as a fixed trajectory either; it is distilled into a reusable skill, then replayed and stabilized in the real environment.
GUI Task
↓
Real Rollout
↓
Success / Failure Evidence
↓
Generate or Patch Skill
↓
Run Again With Direction
↓
First Success
↓
Stabilization
↓
Verified Skill
The model proposes experience; real execution verifies it.
Real GUI execution is expensive, but each run leaves behind useful evidence:
- which route failed;
- where the agent actually got stuck;
- which operation or assumption was invalidated;
- which part of the route already worked;
- what local operating knowledge may be reusable later.
MagicGUI turns those rollouts from plain logs into reusable experience. The core of the harness is not simply adding more memory; it is a closed learning loop:
execute → analyze → learn → execute again → verify → stabilize
MagicGUI first learns inside a concrete task. Each task has its own boundary for failure analysis, skill generation, skill patching, replay verification, and stabilization.
Here, task-local is mainly the boundary for learning and verification: every piece of experience has clear provenance and real execution evidence. But the goal of a skill is not to memorize one benchmark task.
During skill distillation, the system tries to avoid writing rollout-discovered answers, transient screen states, one-off intermediate values, and task-only execution details into runtime skills. Necessary task literals, UI labels, and goal constraints are preserved as anchors.
At runtime, the agent recalls similar experience from a shared skill store using the current task goal and current subgoal.
Concrete Task Execution
↓
Task-Local Learning
↓
Experience Abstraction / De-Specification
↓
Real Replay Verification
↓
Shared Skill Store
↓
Task Goal / Subgoal Semantic Recall
So:
A skill is learned from a concrete task, but it does not have to serve only that task.
A concrete example is the Mastodon task family. Different tasks may ask for very different final outcomes: manage filters, import muted users, export follows, adjust notifications, or edit preferences. Yet many of them share a similar intermediate step: reliably entering the web More / Preferences / Settings area before navigating to a specific feature page. MagicGUI should not memorize the answer to one task; it can distill subgoal-level operating knowledge such as how to find the settings entry, how to verify that the current page is the target settings surface, and how to avoid confusing a timeline overflow menu with account-level settings. When a later task contains a similar subgoal, that experience can be recalled.
MagicGUI does not treat failure as “just try again.” Each failure is analyzed to answer:
- where the agent really got stuck;
- which paths real execution has already ruled out;
- what the next attempt should avoid repeating;
- which directions are still worth exploring.
Failure
↓
Locate Bottleneck
↓
Record Failed Routes
↓
Shrink Next Search Space
↓
Patch Current Skill
↓
Replay
Multi-round execution is therefore not random retry. It is a process of using historical evidence to constrain the next exploration.
Failure itself becomes a constraint for the next attempt.
MagicGUI does not compress the whole task experience into a single memory blob. The planner decomposes a long task into subgoals, and experience is organized around those nodes:
Task
├─ Strategy / Subgoal Sequence
│
└─ Subgoal
└─ Executor Notes
There are two layers of knowledge.
Task-level strategy describes the route: what to do first, what to do next, which states must be reached, and how completion should be checked.
Executor-level notes describe how to perform a specific subgoal reliably: how to locate a control, whether to wait or scroll, which action is fragile, and what visible result should be checked.
This lets experience be reused at different granularities. The strategy guides the whole route, while executor notes are recalled around the current subgoal. Two tasks may have different final goals but still contain similar subgoals, allowing verified local operating knowledge to transfer naturally.
MagicGUI's planner-executor execution framework is conceptually inspired by AutoDevice / android_world, especially its separation between high-level task planning and low-level GUI operation. This project is not a direct reproduction of its code or prompt stack; instead, it reorganizes execution granularity and feedback loops around issues observed in real-device evaluation, including long-horizon tasks, page stalls, inter-stage information gaps, and reusable skill distillation.
There are two layers of responsibility here. The Planner / Executor in this section are the internal execution structure of the GUI agent being driven by the harness; they are responsible for completing one concrete task run. The outer harness is responsible for collecting rollouts, analyzing failures, generating or patching skills, triggering replay, and running stabilization. In other words, the Planner does not perform offline skill generation. During a real run, it decomposes the task, reads Executor reports, and makes stage-level decisions; skill generation happens afterward as a harness learning step over the collected evidence.
Typical issues include:
- Overly fine-grained atomic actions. If a
subgoalis merely “tap the search box / type text / press search,” the executor becomes a button-pushing loop. It loses the broader task intent and cannot judge whether the current stage is actually complete. - Page stalls and micro-loops. Real phones introduce loading, dynamic content, ads, scrolling lists, and hidden entries. A single agent can repeatedly scroll the same page, enter and exit the same detail view, or spend the whole budget searching for a missing target.
- Information gaps between stages. If the next stage does not know what the previous stage did, saw, and why it stopped, the agent may backtrack unnecessarily, repeat searches, or redo already completed work.
- Long-horizon memory drift. Multi-item browsing, comparison, recording, and final decision tasks can overload one context with low-value history, causing missed items, duplicate notes, or distorted target selection.
MagicGUI therefore uses coarser planner subgoals as the basic execution nodes, separating global judgment from local UI operation:
Planner
Understands the task, decomposes the route, tracks progress, reads Executor reports,
and decides whether to continue, backtrack, or change strategy
↓
Subgoal
A small independently executable objective, usually several GUI actions plus an observable completion condition
↓
Executor
Acts on the current UI, observes results, handles local details,
and reports completion, failure, or stalls
↓
Planner Review
Uses reports and historical evidence to choose the next step,
instead of letting the Executor loop locally forever
The key is not adding another agent label. The key is placing control at the right level. The Executor focuses on reliable local action and evidence collection. The Planner receives structured stage reports and makes global decisions using the task goal, failure history, and remaining budget. When the Executor stalls on a page, falls into a micro-loop, or reports that an entry cannot be found, the Planner decides whether to continue searching, switch entry points, relax the strategy, request clarification, or hand that route to the outer harness as failure_pruning.
This split directly shapes the skill design:
strategy_subgoal_sequencestores task logic: stages, route order, and completion checks.executor_notesstores operation logic: what tends to fail under a subgoal, what to wait for, and how to confirm the UI state.failure_pruningstores exploration boundaries: routes that real rollouts have shown should not be repeated.- runtime recall does not need to guess whether two tasks are identical; it checks whether the current
task goaland currentsubgoalmatch prior verified experience.
In other words, a MagicGUI skill is not a large memory paragraph. It is a set of experience records aggregated around subgoal nodes: the planner defines the nodes and performs stage-level review, the executor acts and reports inside each node, and the skill records the verified strategy, cautions, and pruning evidence for those nodes.
After a failure, MagicGUI does not throw away the current candidate and start from scratch. It patches the current candidate skill using fresh failure evidence, historical failed attempts, and any verified success path.
Current Skill
↓
Real Replay
↓
Current Failure
+ Historical Failures
+ Successful Route Evidence (only available during stabilization)
↓
Targeted Patch
↓
Next Replay
The goal is to preserve what already works and only repair the part that real execution exposed as broken.
Keep the useful part; patch the failing part.
LLMs can analyze failures and propose skills, but those skills are only candidates at first. MagicGUI does not promote a skill just because it reads well.
Candidate Skill
↓
Real GUI Replay
↙ ↘
Failure Success
↓ ↓
Patch Again Promote
The final judge of a skill is not model self-evaluation; it is real execution.
GUI execution is naturally unstable: model outputs vary, clicks and swipes can drift, and page loading or UI state may change. One success may simply be a lucky route.
So MagicGUI continues with stabilization replay after the first success:
First Success
↓
Repeated Replay
↓
Find Instability
↓
Patch Again
↓
Verified Skill
Exploration finds a route that can work. Stabilization makes that route more likely to work again.
The loop produces two kinds of assets.
Explicit experience includes task strategy, app/UI operation patterns, executor guidance, failure avoidance, and completion checks. This experience can be used directly by later agent runs.
High-value trajectories include difficult successful routes, failure-to-success repair paths, success/failure contrast traces, and stabilized trajectories. These trajectories can also become training data for later model improvement.
MagicGUI is therefore more than a memory module. It aims to be:
An experience learning layer connecting agent execution, skill distillation, and model learning.
flowchart TD
A["GUI Task"] --> B["Real Rollout"]
B --> C{"Success?"}
C -->|Failure| D["Analyze Failure"]
D --> E["Update Failure Prune"]
E --> F["Patch Skill"]
F --> B
C -->|First Success| G["Keep Success Anchor"]
G --> H["Stabilization Replay"]
H -->|Unstable| I["Analyze New Failure"]
I --> J["Stabilization Patch"]
J --> H
H -->|Stable| K["Verified Skill"]
K --> L["Shared Skill Store"]
L --> M["Task / Subgoal Semantic Recall"]
MobileWorld is the public benchmark and evaluation environment used here for studying mobile GUI agents and validating MagicGUI's self-iteration workflow. The final submitted MagicGUI-Agent setting reached 66.5 Overall on the MobileWorld leaderboard, with 66.7 on GUI-Only and 65.9 on User-Instruction.
Benchmark: MobileWorld
Model setting: GPT-5.4 + Seed-1.8
Agent: MagicGUI-Agent
Final public scores: Overall 66.5; GUI-Only 66.7; User-Instruction 65.9
The staged checkpoints show where the gain comes from:
| Setting | MobileWorld pass@1 success rate | Δ |
|---|---|---|
| No-skill baseline | 54.7% (88/161) |
- |
| Pass@1 after exploration-learned skills | 64.0% (103/161) |
+9.3 pp |
| Pass@1 after stabilized skills | 66.5% (107/161) |
+2.5 pp |
All numbers in the table are MobileWorld pass@1 evaluation results: after each learning stage finishes, the agent is evaluated with one attempt per task on the same 161-task benchmark set across GUI-Only and User-Instruction categories. They are not pass@10 scores and do not count the exploration attempts themselves as benchmark successes.
The learning phase and the evaluation phase are separated. During exploration, each task can be iterated up to 10 times so the harness can collect successful routes, failure patterns, and reusable operating experience from real rollouts. After that learned experience is written into the skill store, the benchmark checkpoint is a fresh pass@1 run. Stabilization is the refinement phase: once a verified route exists, the harness allows at most two additional stabilization patch rounds to improve the reproducibility and success rate of the learned skill. Overall, the staged process moves pass@1 from 54.7% to 66.5%, a +11.8 pp absolute improvement over the no-skill baseline.
This repository provides the reusable learning workflow: task-local self-iteration, failure-pruned exploration, strategy/executor skill generation, semantic skill recall, real-environment replay verification, stabilization, provenance tracking, scripts and sanitized examples. The learning loop is benchmark-agnostic in design; this release keeps the HAD request/response service adapter and MobileWorld-style artifact utilities, but the external MobileWorld runtime and evaluator must be supplied separately.
The current system validates one central idea:
A GUI agent can use its own real execution experience to gradually learn difficult tasks that it could not solve reliably at first.
The same experience-learning mechanism can be connected to other agent workflows.
Collect Failed Attempts
↓
Learn Skill
↓
Collect Again With Direction
↓
Obtain Successful Trajectory
↓
Training Data
Explicit experience can improve the success rate of the next collection round; the final successful trajectories can then be used as training data.
MagicGUI can also act as an experience layer outside model parameters during online RL rollout collection:
Current Policy + Current Skill
↓
Rollout
↓
Reward
↙ ↘
Skill Update Policy Update
↘ ↙
Next Sampling Round
The harness does not need to participate in gradient updates. It can use explicit experience from the previous round to guide the next sampling round, reduce repeated exploration of known-bad routes, and increase the chance of collecting successful trajectories and useful reward signals. Online RL can then gradually internalize successful behaviors into model parameters.
Explicit experience helps the agent find the path faster; parameter learning helps the model learn to walk it by itself.
This is a future direction, not a current result claim.
examples/skill_generation_demo/
Sanitized real-failure skill-generation demo.
scripts/
Skill generation, feedback extraction, and stabilization utilities.
scripts/visual_agent_loop/
Bounded visual evidence processing loop.
magicgui_agent/
GUI agent runtime, planner/executor integration, skill recall, and model adapters.
For prompt contracts, schemas, validators, skill-store runtime, and replay implementation details, see the code.
Create an environment and, if needed, copy .env.example for an OpenAI-compatible model endpoint:
python3 -m venv .venv
source .venv/bin/activate
pip install -r requirements.txt
cp .env.example .envMinimal model variables:
OPENAI_API_KEY=...
OPENAI_BASE_URL=https://api.openai.com/v1
OPENAI_MODEL=...
Run the sanitized skill-generation sample:
python scripts/task_skill_loop.py \
--task-name AdjustBrightnessMaximumTask \
--single-failure-record-path examples/skill_generation_demo/AdjustBrightnessMaximumTask/failed_rollout.json \
--output-root runs/demo_skill_generation \
--skill-store-dir runs/demo_skill_store \
--iterations 1 \
--dry-run \
--learning-signal-source benchmark \
--no-persistThe command prints a compact JSON object with skill_summary; open that Markdown file for the demo output. For a real LLM call, replace --dry-run with --use-llm --backend <model-or-alias>. Custom aliases can be configured with MAGICGUI_DEFAULT_MODEL, MAGICGUI_GPT54_MODEL, MAGICGUI_GPT55_MODEL, or MAGICGUI_SKILL_GENERATION_BACKEND.
After connecting a real GUI benchmark adapter and keeping its workers running, Stage1 exploration is one runner command:
bash scripts/stage1_explore.sh \
--task-file path/to/tasks.txt \
--run-id my_stage1_run \
--replace-task-list \
--max-iterations 20 \
--parallelism 3 \
--wsl-queue-dir runs/wsl_mobileworld_queue_my_stage1_runStage2 stabilization continues from successful Stage1 skills through one watchdog command:
bash scripts/stage2_watchdog.sh \
--stage1-run-id my_stage1_run \
--stage2-run-id my_stage2_run \
--tasks-file path/to/tasks.txt \
--max-tasks 3 \
--parallelism 3 \
--target-mode rescued_unstable \
--auto-patch \
--patch-backend <model-or-alias>The watchdog records finished replay batches, generates and applies stabilization patches when a candidate reaches the 2-of-3 failure gate, and then plans the next replay candidate.
- Learn from task-local execution, recall semantically from a shared skill store.
- Distill experience instead of memorizing trajectories.
- Failure is also a learning signal.
- Subgoals connect task logic with operating experience.
- Keep the useful part; patch the failing part.
- Real replay is more trustworthy than model self-judgment.
- First success is not the same as robust learning.
- Every stored experience should remain traceable to evidence.
MobileWorld is the external benchmark and evaluation setting used for the reported results. The planner-executor split is related to AutoDevice / android_world at the design level; MagicGUI reorganizes the implementation around task-local learning, replay verification, skill recall, and stabilization.
- MobileWorld: project and leaderboard, code, paper.
- AutoDevice / android_world: code.
This project is released under the Apache License 2.0. See LICENSE.