Skip to content

bug(verify): The webview AI source menu draws but ax-settings pick cannot see it #797

Closed

Description

@omesser

What happens

Open Settings as a webview (AI_BUDDY_SETTINGS_WEBVIEW=1) and click the AI source dropdown. The two choices, Model API and Harness, cannot be found anywhere on screen by the accessibility tools the verify scripts drive. Nothing can pick one, so the Harness half of Settings is unreachable from an automated run.

The native Settings window does not have this problem. The same pick succeeds there.

What is measured

From the step 7 run on 7d296120 (evidence comment):

  • The dropdown itself is present and correct: AXPopUpButton|AI source|Model API||true|false.
  • AXPress on it returns success.
  • After the press, a search of the popup's own subtree finds no menu.
  • After the press, a search of the entire application element finds only the system menu bar and ai-buddy's own tray menu. No source-options menu anywhere.

So the options are unreachable everywhere, not merely attached somewhere unexpected.

What is not known

Whether the menu draws on screen and is invisible to the accessibility tree, or never opens at all. AXPress reported success, so the script never fell back to a synthesized click, and no one has watched this happen with their own eyes.

Those two possibilities have different severities:

  • Menu draws, accessibility cannot see it. A verification problem. Users are fine, automated runs are not.
  • Menu never opens. A user-facing bug. Nobody can switch AI source in the webview Settings at all.

Settling that is the first job here, before any fix.

Why this matters now

Step 7 of #706 cannot close without it. The AI-source pick is what drives the Settings window through both of its states, so a run that cannot pick reaches only Model API. The step 7 evidence run failed five assertions and this is one of the three root causes.

This is also the shape of #634, whose root cause is still unknown, and #706's own plan lists <select> behaviour in WKWebView above the overlay as artefact 5 and names #634 explicitly.

Suggested first move

Watch it happen. Open the webview Settings by hand with the flag on, click the dropdown, and say whether a menu appears. That one observation splits the two cases above and decides whether this is a product bug or a verification gap.

If the menu does draw, the next question is why AXPress leaves nothing in the tree, and whether a synthesized click at the element's frame behaves differently.

Related

— Claude Code agent, on @omesser's behalf.

Activity

  1. added
    bugSomething isn't working
    macosmacos / OSX specific
    on Sep 17, 2026
  2. added this to the V1 milestone on Sep 17, 2026
  3. omesser commented on Sep 17, 2026

    @omesser
    OwnerAuthor

    The menu draws. This is a verification gap, not a product bug.

    Pressing the AI source <select> in the webview Settings opens a normal macOS popup menu with all nine options on it. Nobody is blocked from switching AI source. The accessibility tree cannot see that menu, and scripts/ax-settings.swift pick is the only thing that breaks.

    Everything below was measured on db52e2df, in a worktree, on one target/debug/ai-buddy build. The screen was unlocked and the native path was run first in the same session as a control, because AXWindows on this Mac has degraded mid-session before.

    How it was settled

    The repo's rule is to script GUI checks and assert on an AX dump. That rule cannot answer this one question, because the AX dump is the instrument under suspicion. So the answer comes from a screenshot and from CGWindowListCopyWindowInfo, which reads the window server directly and needs no Accessibility grant. Everything after this went back to scripted assertions.

    A probe listed every on-screen window owned by the app, pressed the popup, and listed them again.

    POPUP value=Model API frame=(678.0, 732.0, 547.0, 32.0)
    POPUP actions=["AXPress", "AXShowMenu", "AXScrollToVisible"]
    WINDOWS-BEFORE count=3
      layer=25 num=2590 600x520+660+140 name=Settings
      layer=3  num=2581 1920x1080+0+0  name=ai-buddy
      layer=3  num=2582 1728x1117+1920+0 name=ai-buddy
    MENUS-BEFORE 16
    ACTION AXPress -> 0 in 0ms
    WINDOWS-AFTER count=4
      layer=101 num=2609 547x226+665+726 name=
      layer=25  num=2590 600x520+660+140 name=Settings
      layer=3   num=2581 1920x1080+0+0  name=ai-buddy
      layer=3   num=2582 1728x1117+1920+0 name=ai-buddy
    NEW-WINDOWS 1
      layer=101 num=2609 547x226+665+726 name=
    MENUS-AFTER 16
    RESULT NEW-WINDOW
    

    A window appears at layer 101, the menu layer, 547 points wide. That is the select's own width to the point, from the POPUP frame line above it. MENUS is the count of AXMenu elements anywhere under the application element, and it does not move.

    The still taken 1.5s after the press shows the menu open over the desktop below the Settings window, with all nine rows drawn and Model API checked and highlighted: Model API, Harness · claude, Harness · codex, Harness · cursor-agent, Harness · grok, Harness · hermes, Harness · opencode, Harness · pi, Harness · Custom. That still is local to this Mac and is not attached here.

    The same probe against the native window, run first as the control:

    ACTION AXPress -> 0 in 0ms
    NEW-WINDOWS 1
      layer=101 num=2558 467x226+126+6 name=
    MENUS-BEFORE 16
    MENUS-AFTER 17
    RESULT NEW-WINDOW
    

    Both draw a menu. Only the native one publishes an AXMenu for it.

    Why the tree cannot see it

    The menu is absent from every route, not merely filed somewhere unexpected.

    • Not under the popup. popup children while open = [].
    • Not under the application. APP-CHILD-ROLES ["AXWindow", "AXWindow", "AXWindow", "AXMenuBar", "AXMenuBar"].
    • Not by hit test. AXUIElementCopyElementAtPosition at three points inside the menu's own window returns kAXErrorFailure from both the application element and the system-wide element.

    This is not the app failing to answer. While the menu is up, reading the popup's AXValue returns Model API in 0ms. The process is responsive and the menu is not in its tree. In WKWebView, WebKit's own popup proxy presents a <select> popup rather than an NSPopUpButton, and that path publishes nothing to accessibility. The native renderer's NSPopUpButton publishes its AXMenu under itself, which is the only reason pick has ever worked.

    What breaks

    pick presses the popup, searches the popup's subtree for the option, finds nothing, and dies.

    --- pick AI source -> Harness · claude ---
    ax-settings: the AI source popup has no option Harness · claude
    pick exit=2
    

    The same command on the native window, same build, same session:

    --- pick AI source -> Harness · claude ---
    pick exit=0
    dump reads: AXPopUpButton||Harness · claude||true|false
    

    One consequence is worth recording, because it explains why one failed pick costs five assertions rather than one. pick dies with the menu still tracking, and while a menu tracks the webview publishes no subtree at all. The next dump is a bare window shell:

    AXWindow:AXStandardWindow|Settings||||
    AXGroup||||true|false
    AXButton:AXCloseButton||||true|false
    

    Every later assertion in the run is reading that.

    Where the fix goes

    In scripts/ax-settings.swift, not in the product. Two things are wrong with pick and both are the instrument's.

    1. It knows only one kind of popup menu. macOS menus select by typed prefix, and typing the option's title into the open menu picks it by name, which is the same handle pick already uses everywhere else. That works when measured. The popup reads Harness · claude afterwards.
    2. Its closing check reads a stale element. The webview rebuilds its tree on the source change, and the AXUIElement captured before the pick then reports nil while a freshly resolved one reports Harness · claude. The check has to re-resolve the popup by its label.

    The second point may also be all that #634 ever was. #634 is a native pick Harness Off that exited 0 while a fresh dump still read claude, which is what reading a stale element after a rebuild looks like. That is a hypothesis from the shape, not something reproduced here. #634 needs an attached Harness and is closed.

    Nothing here argues for lowering the overlay or for hand-building a dropdown. The menu draws above the overlay already. Artefact 5 of #706 asked whether the <select> popup survives the overlay panel, and on this evidence it does.

    scripts/verify-settings-macos.sh is untouched and stays that way. Its row addressing is a separate question and yours.

    — Claude Code agent, on @omesser's behalf.

  4. changed the title [-]bug(settings): The AI source dropdown opens no reachable menu in the webview Settings[/-] [+]bug(verify): The webview AI source menu draws but ax-settings pick cannot see it[/+] on Sep 17, 2026
  5. self-assigned this
    on Sep 17, 2026
  6. moved this from 👷 Todo to 🏗 In Progress in Fidgeton Sep 17, 2026
  7. moved this from 🏗 In Progress to 🏁 Done in Fidgeton Sep 17, 2026
  8. omesser commented on Sep 17, 2026

    @omesser
    OwnerAuthor

    Shipped on main via #801

    #801 merged (b08980ba). On main now, in scripts/ax-settings.swift only:

    • pick races the native AXMenuItem search against a new window on the menu layer
    • When WebKit draws a menu with nothing in the AX tree, it type-selects the option title once that window is up
    • The closing check re-resolves the popup by label after the webview rebuilds its tree
    • Failure text is now the <label> popup drew no menu when neither path appears

    Measured on the PR (rebased on #800): native verify still baseline green; webview pick to Harness · claude and back to Model API both exit 0.

    Acceptance met. This issue stays closed.

    — Cursor agent (Coder), on @omesser's behalf.

  9. moved this from 🏁 Done to 🏗 In Progress in Fidgeton Sep 17, 2026
  10. omesser commented on Sep 17, 2026

    @omesser
    OwnerAuthor

    Fix landed via #801

    The verification gap (menu draws; AX cannot see WebKit’s <select> menu; pick type-selects once a menu-layer window appears) is on main via #801 (b08980ba).

    #802 was a parallel branch for the same change; closed as duplicate after #801 merged (conflicted, weaker settle path). Closing this issue.

    — Cursor agent (Coder), on @omesser's behalf.

  11. moved this from 🏗 In Progress to 🏁 Done in Fidgeton Sep 17, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

bugSomething isn't workingmacosmacos / OSX specific

Projects

  • Status
    Done

Milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions