[senamakel]

Summary

Make Jev desktop goals tolerate app startup and delayed UI transitions while preserving exact target and window scope. Verify visible text even when it has no actionable ref, ignore ephemeral snapshot refs in progress checks, and add an ancestor-scoped name_contains success predicate for results with dynamic timestamps. A delivered but unverified consequential action stops as uncertain rather than being replayed.

Related issue

None.

API or behavior changes

Additive VisiblePredicate::NameContains { fragment, within }; TinyDesktop bus contract 1.5. RunGoal retries transient app/window readiness errors for up to three seconds and gives an unverified action up to two seconds to expose success. Permission failures still return immediately. A premature Jev DONE gets one corrective decision and cannot override the requested predicates.

Validation

Commands actually run, with their outcome:

• cargo fmt --all -- --check
• cargo clippy --all-targets --all-features -- -D warnings
• cargo build --all-targets --all-features
• cargo test --all-features

A direct, attested local module run completed a four-click Calculator goal in one call with Done and verified=true. An opt-in native chat run verified an active conversation and one exact outgoing message. Its first combined goal stopped at low confidence after composing; the scoped send-draft goal finished and an independent read-only observation confirmed the outgoing message and cleared composer. No private chat data or credentials are in this PR.

Tests

Added regression coverage for ref-less visible result text, ephemeral snapshot refs, contained message evidence, startup retries versus permission failures, delayed success after unverified delivery, and the contract wire shape. Added safe direct Calculator and filtered accessibility probe examples. The private live chat diagnostic remains local.

Documentation

Updated the bus and native Jev loop READMEs for the new predicate and retry behavior.

Checklist

• The change is focused on goal execution and verification
• No new #[allow(...)], #[ignore], or relaxed lints
• No secrets, tokens, or .env contents in the diff or the description

Summary by CodeRabbit

• New Features

  • Added a visible-text matching option that can find a text fragment within a specifically named container, even when nearby timestamps or delivery statuses vary.
  • Added example tools for inspecting an app’s accessibility information and trying a guided Calculator task.

• Improvements

  • Task verification can now use visible text that isn’t directly actionable.
  • The system waits briefly for apps and windows to become available and checks for success after an action’s result is uncertain, without repeating the action.

[tinysweeper]

Tiny Sweeper review

Tiny Sweeper reviewed this change across 6 lane(s) and found 11 active actionable finding(s). Detailed lane evidence and any incomplete work are listed below.

State: Changes requested
Priority: high
Reviewed head: 47d16a9d0f2f
Updated: 1790314437 (Unix time)

Review snapshot

Change surface	Files	Review signal	Count
Production	7	Active findings	11
Tests	5	Noted findings	0
Documentation	2	Resolved findings	0
Configuration	0	Pending checks/questions	0

Completeness: Complete
Test assessment: No supported feature-to-test mapping was available; this does not mean tests are absent or passed.

What changed

The review could not produce a supported behavioral summary; inspect the cited changed surface and lane details below.

Features

None identified with supported citations.

Tests

No supported feature-to-test mapping was produced. Test execution is not inferred.

Findings

• high · critique · Stop after an unverified destructive action — This branch is reached only inside the `success`-predicate block, so a request with no success predicates skips the uncertainty handling entirely. Such requests are valid when conf (crates/tinydesktop/src/agentic/task\.rs:457)
• medium · critique · Reject an invalid mode before launching Calculator — The mode validation occurs only after the module has been loaded, Calculator has been launched, and the snapshot/focus polling has completed. For an input such as `live_goal typo m (crates/tinydesktop\-examples/src/bin/live\_goal\.rs:106)
• medium · critique · Report the matched element's actual name — When a candidate named `Start Playlist` matches `NameContains { fragment: "Playlist", .. }`, this returns `observed_name: Some("Playlist")` rather than the observed element name. ` (crates/tinydesktop/src/agentic/verify\.rs:51)
• medium · critique · Do not silently omit matches from deep trees — A matching node at depth 33 or deeper is discarded, and its entire subtree is skipped, but the caller still prints `matching_nodes` and exits successfully. Accessibility trees can (crates/tinydesktop\-examples/src/bin/live\_probe\.rs:89)
• medium · critique · Include actionability and structure in the fingerprint — The fingerprint now ignores `available_actions` and `children_count` (as well as other structural fields). For example, a node can retain the same role, label, path, states, and va (crates/tinydesktop/src/agentic/screen\.rs:295)
• medium · critique · Require a tree before reporting a successful probe — A `DesktopResponse` with `ok == true` but missing `data` or `data.tree` is accepted, `print_matching` returns zero, and the example exits successfully. This hides an incomplete or (crates/tinydesktop\-examples/src/bin/live\_probe\.rs:75)
• high · security · Stop when a destructive action is delivered unverified — This handling is nested inside the `!self.request.success.is_empty()` branch. When a destructive action returns `delivered_unverified` and the request has no success predicates, ex (crates/tinydesktop/src/agentic/task\.rs:457)
• medium · security · Bound the data included in fingerprints — `observed` contains up to `MAX_VISITED_NODES` nodes and is no longer limited by the `candidates.truncate(254)` bound. The subsequent fingerprint formats each node's full path, stat (crates/tinydesktop/src/agentic/screen\.rs:283)
• medium · tests · Add a test for an invalid NameContains predicate — The `is_predicate_invalid` function now rejects `NameContains` with an empty fragment or `within`. This validation path is exercised by no test. Add a test that calls `run_goal` wi (crates/tinydesktop/src/agentic/task\.rs:108)
• medium · tests · Test the destructive-action timeout path for delivered_unverified — When the settling window expires and the action is destructive, the loop stops with `ActionUncertain`. This path is not covered by `goal_waits_for_delayed_success_without_replaying (crates/tinydesktop/src/agentic/task\.rs:457)
• high · description · Fix ancestor path matching in NameContains verification — The `ancestor` string is built with a leading space (`format!(" {within:?}")`), but path parts are constructed as `"{} {name:?}"` (role, space, then the debug representation of the (\(pull request description\))

Before merge

• Address Stop after an unverified destructive action (crates/tinydesktop/src/agentic/task\.rs).
• Address Stop when a destructive action is delivered unverified (crates/tinydesktop/src/agentic/task\.rs).
• Address Fix ancestor path matching in NameContains verification (\(pull request description\)).

How this fits together

flowchart LR
  n0["default"]:::impacted
  n1["run_goal_with"]:::impacted
  n2["runtime"]:::impacted
  n3["backend"]:::impacted
  n4["...ions_and_post_action_observation_failures"]:::impacted
  n5["...hosen_window_id_through_every_observation"]:::impacted
  n4 -->|calls| n0
  n4 -->|tests| n0
  n4 -->|calls| n1
  n4 -->|tests| n1
  n4 -->|calls| n2
  n4 -->|tests| n2
  n4 -->|calls| n3
  n4 -->|tests| n3
  n5 -->|calls| n0
  n5 -->|tests| n0
  n5 -->|calls| n1
  n5 -->|tests| n1
  n5 -->|calls| n2
  n5 -->|tests| n2
  n5 -->|calls| n3
  n5 -->|tests| n3
  classDef changed fill:#0d4429,stroke:#238636,color:#e6edf3
  classDef impacted fill:#161b22,stroke:#6e7681,color:#c9d1d9
  classDef flagged fill:#5a1e02,stroke:#d93f0b,color:#ffffff
  classDef blocking fill:#67060c,stroke:#f85149,color:#ffffff

Loading

Agent review details

critique

• Conclusion: Failure
• Scope reviewed: all assigned evidence
• Lane summary: Reviewed 14 files; 7 findings. (1 observation(s) grouped into shared inline comments) _The code index is behind this pull request (indexed at `40723470cbb8`), so retrieved context may be out of date._ _1 memory call(s) failed (model: cortex: v1/answer: timed out after 20s), so this review saw part of what the engine holds._
• Evidence: crates/tinydesktop/src/agentic/task\.rs — Stop after an unverified destructive action
• Evidence: crates/tinydesktop\-examples/src/bin/live\_goal\.rs — Reject an invalid mode before launching Calculator
• Evidence: crates/tinydesktop/src/agentic/verify\.rs — Report the matched element's actual name
• Evidence: crates/tinydesktop\-examples/src/bin/live\_probe\.rs — Do not silently omit matches from deep trees
• Evidence: crates/tinydesktop/src/agentic/screen\.rs — Include actionability and structure in the fingerprint
• Evidence: crates/tinydesktop\-examples/src/bin/live\_probe\.rs — Require a tree before reporting a successful probe

security

• Conclusion: Failure
• Scope reviewed: all assigned evidence
• Lane summary: Reviewed 12 files; 2 findings. 2 files were not security-reviewed: crates/tinydesktop-bus/README.md (prose or tabular data), crates/tinydesktop/src/agentic/README.md (prose or tabular data). _The code index is behind this pull request (indexed at `40723470cbb8`), so retrieved context may be out of date._ _1 memory call(s) failed (model: cortex: v1/answer: timed out after 20s), so this review saw part of what the engine holds._
• Evidence: crates/tinydesktop/src/agentic/task\.rs — Stop when a destructive action is delivered unverified
• Evidence: crates/tinydesktop/src/agentic/screen\.rs — Bound the data included in fingerprints

tests

• Conclusion: Success
• Scope reviewed: all assigned evidence
• Lane summary: This PR adds the `NameContains` visible predicate, retry logic for transient app/window errors, settling-window handling for unverified deliveries, and a corrective note when Jev prematurely claims `DONE`. The tests cover the new predicate's wire shape, the retry behavior, the unverified-click success path, verification of ref-less nodes, fingerprint changes, and the corrective note. Two untested branches remain: invalid `NameContains` predicates (empty fragment/within) and the destructive-action timeout fallback for `delivered_unverified`. (1 observation(s) grouped into shared inline comments) _The code index is behind this pull request (indexed at `40723470cbb8`), so retrieved context may be out of date._ _1 memory call(s) failed (model: cortex: v1/answer: timed out after 20s), so this review saw part of what the engine holds._
• Evidence: crates/tinydesktop/src/agentic/task\.rs — Add a test for an invalid NameContains predicate
• Evidence: crates/tinydesktop/src/agentic/task\.rs — Test the destructive-action timeout path for delivered_unverified

commits

• Conclusion: Neutral
• Scope reviewed: all assigned evidence
• Lane summary: Nothing sensitive found in what this pull request commits.

description

• Conclusion: Failure
• Scope reviewed: all assigned evidence
• Lane summary: This pull request adds a `NameContains` visible predicate, retry logic for transient app/window errors, and handling of unverified deliveries. However, the ancestor path matching in `NameContains` uses an incorrect `ends_with` check that will never match, making the new predicate non-functional. The test that exercises this path would also fail with the current code. Everything else looks sound, but this bug should be fixed before merging. _The code index is behind this pull request (indexed at `40723470cbb8`), so retrieved context may be out of date._ _1 memory call(s) failed (model: cortex: v1/answer: timed out after 20s), so this review saw part of what the engine holds._
• Evidence: \(pull request description\) — Fix ancestor path matching in NameContains verification

e2e

• Conclusion: Neutral
• Scope reviewed: all assigned evidence
• Lane summary: No end-to-end harness in this repository: no e2e test files and no e2e workflow.

Evidence and run details

• Models: ladder/vectors, gpt-5.6-luna, deepseek/deepseek-v4-flash, deepseek-v4-flash
• Spend: $0.035349
• Tokens: 1042292 input · 54813 output · 81828 cached · 1175 embedding

Head	State	Pass summary
47d16a9d0f2f	changes requested	11 active finding(s), 0 resolved finding(s) (at 1790314437)

[coderabbitai]

Navigate logical layers of code changes, visualize relationships, and explore their blast radius.

Note

Currently processing new changes in this PR. This may take a few minutes, please wait...

⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Advanced

Run ID: 38439390-325a-46fc-ac5d-00a54c008c24

📥 Commits

Reviewing files that changed from the base of the PR and between 799e242 and 47d16a9.

📒 Files selected for processing (14)

• crates/tinydesktop-bus/README.md
• crates/tinydesktop-bus/src/agentic/test.rs
• crates/tinydesktop-bus/src/agentic/types.rs
• crates/tinydesktop-bus/src/version/mod.rs
• crates/tinydesktop-bus/src/version/test.rs
• crates/tinydesktop-examples/src/bin/live_goal.rs
• crates/tinydesktop-examples/src/bin/live_probe.rs
• crates/tinydesktop/src/agentic/README.md
• crates/tinydesktop/src/agentic/screen.rs
• crates/tinydesktop/src/agentic/task.rs
• crates/tinydesktop/src/agentic/test.rs
• crates/tinydesktop/src/agentic/verify.rs
• crates/tinydesktop/src/agentic/verify/test.rs
• crates/tinydesktop/tests/public_api.rs

 _______________________________________________
< Crouching Tiger, Hidden NullPointerException. >
 -----------------------------------------------
  \
   \   \
        \ /\
        ( )
      .( o ).

---

_{Comment @coderabbitai help to get the list of available commands.}

[tinysweeper]

Requesting changes: 3 lane(s) blocking, worst finding is high.

Fix or reply to the findings below and push. The next review clears this automatically once they are gone — you should not need to dismiss anything by hand.

             $0.0353 · 1,042,292 in / 54,813 out · 81,828 cached (8%) · ladder/vectors, gpt-5.6-luna, deepseek/deepseek-v4-flash, deepseek-v4-flash · 1,175 embedded
critique:    $0.0182 · 554,486 in   / 24,075 out · 30,760 cached (6%) · gpt-5.6-luna, deepseek/deepseek-v4-flash
security:    $0.0120 · 425,499 in   / 14,585 out · 34,172 cached (8%) · gpt-5.6-luna
tests:       $0.0034 · 27,850 in    / 4,797 out  · 0 cached (0%)      · deepseek/deepseek-v4-flash
description: $0.0005 · 18,335 in    / 6,504 out  · 1,024 cached (6%)  · deepseek-v4-flash

[senamakel]

The actionable late review findings are addressed in follow-up #17, with replies on each inline thread. The hub-only ancestor-path finding is a false positive: screen.rs builds each path component as role + " " + Debug(name), and verify.rs matches the suffix " " + Debug(within). The contained-name unit test passes for the intended chat and fails for a different chat; a direct native goal also returned verified=true using this predicate. I kept the matcher unchanged.