Skip to content

fix(dash): run exactly the tool call the approval card shows - #980

Merged
rominf merged 1 commit into
mainfrom
fix/dash-approval-runs-shown-call
Nov 17, 2026
Merged

rominf merged 1 commit into
mainfrom
fix/dash-approval-runs-shown-call

Conversation

@rominf

@rominf rominf commented Nov 9, 2026 •

Copy link
Copy Markdown
Collaborator

Problem

In rocm dash and interactive rocm chat, a mutating assistant tool call (launch_server, install_sdk, install_engine, stop_server, watcher_enable or watcher_disable) pauses on an approval card built from the call's argument object. Once approved, the call was replayed through run_internal_mcp_call, which swapped in a nested arguments object whenever one was present. A tool call carrying such a field therefore showed one command on the card and ran a different one after the user approved it: the card read rocm serve shown/model --managed while approval ran rocm serve other/model --managed --port 8123.

Fix

  • The replay runs what it is given. run_internal_mcp_call (in chat/mcp_bridge.rs) validates and runs the argument object it receives, unchanged, through internal_mcp_tool_call, and never unwraps it.
  • Calls with a nested arguments field are refused. validate_chat_tool_call in tool_policy.rs refuses them. No ROCm tool declares a parameter of that name (pinned by no_rocm_tool_declares_an_arguments_parameter against the CLI's tool catalog, with a dash-side copy over the dash's tool definitions), so such a call holds two payloads, and it is refused rather than resolved to either one. The check runs on every path that validates a tool call: the dash and interactive chat refuse before a card is shown, and one-shot chat and rocm mcp-call refuse before anything runs, read-only tools included.
  • rocm mcp-call keeps a narrow envelope unwrap at its own boundary. The hidden command still accepts a pasted MCP tools/call params object and unwraps it in mcp_call_cli_arguments, but only a pure envelope: name, arguments and the protocol's optional _meta, which is dropped. It refuses an envelope whose name is not the tool on the command line. An object mixing tool fields with an arguments key is passed through whole and refused by validation.
  • Docs. docs/llm-tool-use.md states the rule; docs/architecture.md (the chat/mcp_bridge.rs and tool_policy.rs rows) and docs/testing.md (a new "Assistant tool-call arguments" block beside the direct MCP helper smoke) match it.

User-visible changes:

  • A model's tool call with a nested arguments field is refused. In dash and interactive chat, the model is told why and no approval prompt appears.
  • In one-shot rocm chat "<prompt>", such a call ends the turn with that error, as any invalid tool call already does there.
  • rocm mcp-call no longer unwraps an arguments object mixed with other fields; it refuses the call.
  • rocm mcp-call refuses --arguments-json values that are not a JSON object (null, [], a string), which it used to run as {}. Omitting the flag still means {}.

Tests

Scenarios (tests/e2e-cucumber/features/chat.feature, mock lane, no GPU):

  • chat-17 (@id:chat-ambiguous-tool-call-refused-before-approval, @requires-os:linux): a mock model answers "please start a model server" with a launch_server call carrying two sets of arguments. It asserts that the tool message the CLI sends back says the call was refused for its nested arguments field and was not surfaced for approval, and that no approval prompt is on screen. The mock server gains a scripted tool-call reply keyed on the user's words, with its own harness test.
  • chat-18 (@id:chat-mcp-call-runs-a-whole-tool-call-request): a whole tools/call params object (with _meta) handed to rocm mcp-call path_exists runs the tool it names and reports the requested directory.
  • chat-19 (@id:chat-mcp-call-refuses-two-sets-of-arguments): rocm mcp-call path_exists with an existing outer path and a missing nested one is refused for its nested field and prints no tool result.

chat-18 and chat-19 carry no OS tag, so they run on every OS the mock lane runs on; chat-17 drives the TUI through a pseudo-terminal and is Linux only.

Unit tests (apps/rocm unless noted): validate_chat_tool_call_refuses_a_nested_arguments_field_for_every_tool, seam_refuses_nested_arguments_before_approval_and_on_replay, seam_refuses_nested_arguments_on_read_only_tools, seam_approval_card_matches_the_command_approval_runs, seam_approval_card_matches_the_command_for_plain_calls (guards against refusing valid calls), mcp_call_cli_unwraps_only_a_pure_params_envelope, mcp_call_refuses_arguments_that_are_not_an_object, mcp_call_refuses_tool_fields_mixed_with_a_nested_arguments_object, no_rocm_tool_declares_an_arguments_parameter (here and in rocm-dash-tui), and an_approval_dialog_shows_its_review_title_and_approve_button in rocm-dash-tui, which pins the two dialog strings chat-17 checks are absent.

Each new test was seen to fail with its target broken:

Broken on purpose Goes red
Refusal removed from validate_chat_tool_call the validator test, both seam_refuses_* tests, the mixed-object mcp_call test, and chat-17
Refusal removed and internal_mcp_tool_call unwrapping the nested object additionally seam_approval_card_matches_the_command_approval_runs
An unwrap added in run_internal_mcp_call, refusal kept seam_refuses_nested_arguments_before_approval_and_on_replay, mcp_call_refuses_tool_fields_mixed_with_a_nested_arguments_object
The mcp_call_cli_arguments call removed from the rocm mcp-call dispatch chat-18
_meta dropped from the envelope keys mcp_call_cli_unwraps_only_a_pure_params_envelope
The mock's user-role guard, or its case folding of the user's words, removed a_scripted_tool_call_answers_only_the_triggering_user_turn

Verified locally (Linux)

  • cargo fmt --all --check and cargo clippy --workspace --all-targets -- -D warnings are clean; cargo xtask check-architecture-doc and cargo xtask check-module-size pass.
  • cargo xtask e2e -- -n "^(chat|dash)-": 69 of 73 scenarios pass, chat-17, chat-18 and chat-19 among them. The other four need a real served model with network downloads (chat-06, chat-07, dash-52) or a WSL host (dash-24).
  • cargo test --workspace --all-targets --no-fail-fast fails only in tests this change does not touch: two build_script tests that run git commit in a scratch repo, and service_control::tests::stop_of_a_container_engine_is_unconfirmed_when_the_engine_stop_fails, which fails the same way on main on this host.

Not verified

  • cargo xtask smoke stops at rocmd sandbox examine snapshot, because bubblewrap cannot configure loopback in the local sandbox; every step before it passes.
  • chat-17 runs on Linux only. On Windows the dash/chat refusal is covered by the unit tests, and the rocm mcp-call path by chat-18 and chat-19.
  • The dash-side no_rocm_tool_declares_an_arguments_parameter lists the dash's tool definitions by hand, tied only by count to its tool-name lists, so a dash tool added later is not covered automatically. The CLI validator refuses such a call either way.
  • The validator test lists the 21 tool names by hand, so a tool added later is not covered by that test automatically. The refusal itself runs before the per-tool match, so it applies to any tool name.

In `rocm dash` and interactive chat, the approval card for a mutating
assistant tool call (launch_server, install_sdk, install_engine,
stop_server, watcher_enable, watcher_disable) was rendered from the
call's argument object, but the approved replay went through
run_internal_mcp_call, which swapped in a nested `arguments` object
when one was present. A model-supplied call carrying such a field
showed one command on the card and ran a different one once the user
approved it.

The replay now validates and runs the object it is given, unchanged.
validate_chat_tool_call refuses any `arguments` field outright, so the
dash, chat and `rocm mcp-call` paths all refuse such a call before it
runs, and the dash and chat refuse it before a card is shown: no ROCm
tool declares a parameter of that name, so a call carrying one holds
two payloads and is refused rather than resolved to either.

The hidden `rocm mcp-call` command still accepts a pasted MCP
tools/call params envelope and unwraps it at its own boundary, but only
a pure envelope: `name`, `arguments` and the protocol's optional
`_meta`, which is dropped. It refuses an envelope naming a different
tool, and passes an object mixing tool fields with `arguments` through
whole, so validation refuses it. Arguments that are not a JSON object
are now refused rather than run as `{}`.

Tested by unit tests that the validator refuses a nested `arguments`
field for every ROCm tool name, that the dash card and the approved
replay agree, that the replay refuses a nested payload, that the
mcp-call boundary unwraps only a pure envelope, and that no tool in
either catalog declares an `arguments` parameter; and end to end by
chat-17, where a mock model answers with a two-payload launch_server
call and the model is told the call was refused and no approval prompt
is shown, chat-18, where a whole tools/call request handed to
`rocm mcp-call` runs the tool it names, and chat-19, where a request
with tool fields beside a nested `arguments` object is refused and
prints no tool result. The e2e mock server gains a scripted tool-call
reply keyed on the user's words, with its own harness test.

Signed-off-by: Roman Inflianskas <Roman.Inflianskas@amd.com>
@rominf
rominf force-pushed the fix/dash-approval-runs-shown-call branch from a6a4b2f to cde2775 Compare November 17, 2026 11:29
@github-actions

github-actions Bot commented Nov 17, 2026 •

Copy link
Copy Markdown
Contributor
Security scan found no critical or high findings on lines changed by this pull request

View the security scan run for complete reports.

“New” means the finding is located in a changed file on an added or modified line; it is not a base-versus-head findings delta.

@rominf
rominf marked this pull request as ready for review November 17, 2026 13:36
@rominf
rominf requested a review from a team as a code owner November 17, 2026 13:36

@juhovainio juhovainio left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good to me. The fix sits at the right layer: run_internal_mcp_call no longer unwraps a nested arguments object, and validate_chat_tool_call refuses it on every path, so the dash approval card and the command that runs can no longer diverge. The regression tests plus chat-17/18/19 cover the behavior, and the docs are updated. CI is green apart from the MI300X lane, which has no runners.

Two small non-blocking nits are inline. Also minor: the dash-side tool list and the validator's tool-name list are hand-maintained (the PR body already says so; safety doesn't depend on them since the refusal runs first), and one comment in chat.rs wraps wider than its neighbours.

This is based on reading the diff and CI results, not on running it locally.

// A call that carries one holds a second payload next to the first, and
// the approval card and the command it approves must never be able to
// read different ones.
if call.arguments.get("arguments").is_some() {

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nit: .get("arguments").is_some() also refuses "arguments": null. Worth a Null case in validate_chat_tool_call_refuses_a_nested_arguments_field_for_every_tool so that behavior is pinned rather than incidental.

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Right: .get("arguments").is_some() is true for "arguments": null too, so that call is refused with the same nested-arguments error. That is the intended outcome (no tool takes an arguments parameter, so a null one is no more meaningful than an object), but you're correct that no test pins it: the every-tool test only sends {"arguments": {}}. I'm leaving it for a follow-up that adds a Null case to validate_chat_tool_call_refuses_a_nested_arguments_field_for_every_tool, rather than moving the approved head for a test-only change.

.is_some_and(serde_json::Value::is_object)
&& object
.keys()
.all(|key| matches!(key.as_str(), "name" | "arguments" | "_meta"));

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nit: an MCP params object carrying any other protocol key (e.g. task) counts as "mixed" and gets the nested-arguments refusal. Fine for a hidden command, but the refusal could say which keys make a pure envelope (name, arguments, _meta).

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Agreed on both counts. Any key outside name, arguments and _meta makes is_envelope false, so the object passes through whole and validate_chat_tool_call refuses it with "must not contain a nested arguments field", which doesn't tell the caller the envelope itself was the problem. It fails closed, so I've kept it out of this PR; a follow-up can have mcp_call_cli_arguments name the keys a pure envelope may hold when it sees an arguments object next to other keys.

@juhovainio juhovainio left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Independent review

Traced the fix end to end against the code (not just the PR's own framing):

  • Every path that can build an approval card or execute a tool now calls validate_chat_tool_call first, and that validator unconditionally refuses a nested arguments key: the dash seam (BinToolExecutor::execute/execute_approved in dash_seam.rs), the interactive/one-shot chat turn loop (two call sites in chat/turn.rs, both before the read-only dispatch and before chat_tool_approval_request builds the card), and run_internal_mcp_call/internal_mcp_tool_call in mcp_bridge.rs. The approval card (tool_policy.rs::chat_tool_approval_request/rocm_chat_tool_requested_args) and the replayed command both read fields from the same, already-validated call.arguments object, so they cannot disagree.
  • mcp_call_cli_arguments's envelope unwrap is correctly scoped to the rocm mcp-call CLI boundary only, runs before validation, and only unwraps a "pure" envelope (keys limited to name/arguments/_meta, with a name mismatch refused); anything else (mixed objects) is passed through whole and still caught by the downstream nested-arguments refusal. Checked the branch-by-branch logic (absent/non-string/mismatched name) against the test cases and it holds.
  • The install_sdk_dry_run arm's switch away from the old unwrap helper is safe: validate_chat_tool_call is still called upstream of it in the turn loop, so non-object/nested-argument input never reaches it.
  • Test coverage is non-vacuous: the dash-side no_rocm_tool_declares_an_arguments_parameter test's defs array matches ROCM_READ_TOOL_NAMES/ROCM_MUTATING_TOOL_NAMES 1:1 (14+6=20, verified by reading both lists), so the length-equality assertion isn't trivially satisfied by omitting a tool from both sides. The new an_approval_dialog_shows_its_review_title_and_approve_button test exercises the real draw_approval/draw_approval_inner/buttons_line rendering path, the same one the e2e absence-check depends on. The e2e scenarios (chat-17/18/19) target the actual refusal/approval boundary and their trigger string doesn't collide with other fixtures (MANAGED_MODEL_PROMPT).
  • Docs (architecture.md, llm-tool-use.md, testing.md) accurately describe the new behavior and boundary, and the listed cargo test/cargo xtask e2e filters match real test/scenario names.
  • No AGENTS.md rule violations found, no related prior-fix history for this bug (confirmed via git log -S on the removed unwrap helper — this was a long-standing latent bug, not a reintroduction or incomplete prior fix), and the PR description's claims match the diff exactly (968/+, 47/-, 15 files).
  • CI is green; the only non-success check is E2E tests (MI300X, current) queued with no available runner, not a failure.

I independently arrived at the same two non-blocking observations already on this PR (the "arguments": null edge case being refused too, and the mixed-envelope refusal message not naming the allowed keys) — agree both are nitpicks, not blockers.

No blocking findings.

🤖 Automated by an AI agent

@rominf

rominf commented Nov 17, 2026

Copy link
Copy Markdown
Collaborator Author

Thanks for both reviews. On the points in the review bodies that have no inline thread:

  • The hand-maintained lists (the dash-side tool definitions in no_rocm_tool_declares_an_arguments_parameter and the validator test's 21 tool names) stay as they are in this PR. As you say, the refusal runs before the per-tool match, so a tool missing from either list is still refused. Deriving both from their registries is follow-up material.
  • The chat.rs re-export comment that runs wider than its neighbours: I'll rewrap it in the follow-up with the two inline nits, so this approved head doesn't move for a comment-only change.

@rominf
rominf added this pull request to the merge queue Nov 17, 2026
Merged via the queue into main with commit c9d50ed Nov 17, 2026
56 checks passed
@rominf
rominf deleted the fix/dash-approval-runs-shown-call branch November 17, 2026 14:54
r0x0r added a commit that referenced this pull request Nov 17, 2026
One conflict, in `apps/rocm/src/chat/mcp_bridge.rs`, and it is the
code-was-removed shape rather than two additions colliding.

Main's #980 deletes `internal_mcp_arguments` outright and replaces it with
`internal_mcp_tool_call` plus `mcp_call_cli_arguments`, so that a pasted MCP
envelope is unwrapped in one named step instead of silently inside the
dispatcher -- the point of that change being that the approval card and the
executed call cannot disagree. This branch still carried the old function.

Resolved by taking main's design whole and keeping only this branch's own
addition from the other side, the `engines` MCP tool-result helper. Keeping
this branch's `internal_mcp_arguments` would have reinstated the unwrapping
main had just removed, which is the opposite of what #980 is for; grep
confirms no caller of it survives.

Signed-off-by: Roman Sirokov <roman.sirokov@amd.com>
r0x0r added a commit that referenced this pull request Nov 17, 2026
One conflict, in `apps/rocm/src/chat/mcp_bridge.rs`, and it is the
code-was-removed shape rather than two additions colliding.

Main's #980 deletes `internal_mcp_arguments` outright and replaces it with
`internal_mcp_tool_call` plus `mcp_call_cli_arguments`, so that a pasted MCP
envelope is unwrapped in one named step instead of silently inside the
dispatcher -- the point of that change being that the approval card and the
executed call cannot disagree. This branch still carried the old function.

Resolved by taking main's design whole and keeping only this branch's own
addition from the other side, the `engines` MCP tool-result helper. Keeping
this branch's `internal_mcp_arguments` would have reinstated the unwrapping
main had just removed, which is the opposite of what #980 is for; grep
confirms no caller of it survives.

Signed-off-by: Roman Sirokov <roman.sirokov@amd.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants