Repository navigation
fix(flows): builder chat hygiene — run-tool honesty (Bld #4) + reply hygiene - #5007
Conversation
The workflow_builder prompt unconditionally told the agent it can run_flow
('Testing a saved flow: run_flow'), but run_flow/resume_flow_run/cancel_flow_run
are hidden on the flows_build (copilot) path via FLOWS_BUILD_HIDDEN_TOOLS. Result:
the agent offers to run the flow, the user says yes, and it replies it doesn't
have access — a confusing offer-then-refuse.
Make the run/resume/cancel guidance capability-conditional: the agent must check
whether the tool is on its belt first, and when it isn't, point the user to the
Workflows UI Run control instead of offering. Honest on both paths (the chat
delegate path still has run_flow and can run). Prompt-only + test — no capability
or safety change (the tool stays hidden on flows_build). Adds a builder_prompt
regression assertion.
|
Warning Review limit reachedYou’ve reached a temporary PR review limit under our Fair Usage Limits Policy. Next review available in: 18 minutes Enable usage-based reviews in Billing to review now. Otherwise, wait until the next included review is available. How can I continue?After more reviews become available, a review can be triggered using the To avoid repeated limits, reduce automatic review volume by pausing incremental auto-reviews earlier, using label-based review opt-in, excluding WIP or generated PR titles, or requesting reviews manually when the PR is ready. If your team needs uninterrupted high-volume reviews, an organization admin can enable usage-based reviews. How do review limits work?CodeRabbit enforces per-developer PR review limits for each organization. Most developers receive the normal plan review availability. For paid Pro and Pro+ PR reviews, CodeRabbit uses adaptive limits for sustained high-volume activity. When a developer's recent PR review activity reaches the 95th percentile or higher among CodeRabbit users, additional reviews become available more gradually as earlier reviews age out of the rolling window. Please refer docs for additional details. Review details⚙️ Run configurationConfiguration used: Organization UI Review profile: CHILL Plan: Pro Run ID: 📒 Files selected for processing (2)
📝 WalkthroughWalkthroughChangesWorkflow run-control guidance
Estimated code review effort: 1 (Trivial) | ~5 minutes Suggested labels: Suggested reviewers: Poem
🚥 Pre-merge checks | ✅ 5✅ Passed checks (5 passed)
Comment |
coderabbitai
Bot
added
agent
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: f468a94fa1
ℹ️ About Codex in GitHub
Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".
| **First check whether `run_flow` is in your available tools — on some surfaces it | ||
| is not.** If you do **not** have a `run_flow` tool, never offer to run the flow | ||
| yourself and never say you'll run it: instead tell the user they can run it | ||
| themselves from the **Run** control on the flow in the Workflows UI (or by | ||
| triggering it however it's configured). The one thing to avoid is offering to run |
There was a problem hiding this comment.
On the Workflows copilot path, flows_build still hides run_flow/resume/cancel from the builder belt, but the server-rendered turn prompt is appended after this standing prompt and still tells revise turns with a flow_id that “if I ask you to run/test it, you may run_flow that id” (builder_prompt.rs lines 135-139). When a user asks the copilot to test an existing flow, that later per-turn instruction contradicts this new capability check and can still drive the model to offer/call an unavailable tool, preserving the reported offer-then-refuse behavior. The per-turn directives need the same “only if on your belt, otherwise point to the UI” wording.
Useful? React with 👍 / 👎.
There was a problem hiding this comment.
Fixed in 179f3ce. You're right — the per-turn revise directive (DIRECTIVE_REVISE in builder_prompt.rs) and its flow_id note unconditionally asserted "you may run_flow", contradicting the new standing-prompt capability check since the per-turn brief is appended after it. Both now say "follow the run_flow capability rule ... only if that tool is on your belt ... otherwise point me to the Run control in the Workflows UI" instead of asserting availability. Added a dedicated regression test (revise_run_guidance_is_capability_conditional) that asserts the capability-deferral wording and rejects the stale unconditional phrasing.
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
🧹 Nitpick comments (1)
src/openhuman/flows/agents/workflow_builder/builder_prompt.rs (1)
362-376: 📐 Maintainability & Code Quality | 🔵 Trivial | ⚡ Quick winMake the regression assertion cover the full capability-gating contract.
These independent substring checks could still pass if an unconditional
run_flowoffer returns, or if the newresume_flow_run/cancel_flow_runfallback inprompt.mdlines 184-187 regresses. Bind the assertion to the specificrun_flowwording, reject the stale offer-then-refuse phrasing, and add positive checks for the resume/cancel availability fallback.🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@src/openhuman/flows/agents/workflow_builder/builder_prompt.rs` around lines 362 - 376, Strengthen the regression assertion around STANDING_PROMPT to verify the specific run_flow capability-gated wording, reject any stale unconditional offer-then-refuse phrasing, and add positive checks covering the resume_flow_run and cancel_flow_run availability fallback. Keep the assertion focused on the complete capability-gating contract rather than independent generic substrings.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Nitpick comments:
In `@src/openhuman/flows/agents/workflow_builder/builder_prompt.rs`:
- Around line 362-376: Strengthen the regression assertion around
STANDING_PROMPT to verify the specific run_flow capability-gated wording, reject
any stale unconditional offer-then-refuse phrasing, and add positive checks
covering the resume_flow_run and cancel_flow_run availability fallback. Keep the
assertion focused on the complete capability-gating contract rather than
independent generic substrings.
ℹ️ Review info
⚙️ Run configuration
Configuration used: Organization UI
Review profile: CHILL
Plan: Pro
Run ID: 52aaedb0-f9ba-40b5-9bf0-b84ca04339cb
📒 Files selected for processing (2)
src/openhuman/flows/agents/workflow_builder/builder_prompt.rssrc/openhuman/flows/agents/workflow_builder/prompt.md
…-then-restate
The workflow_builder final reply leaked the model's stream-of-consciousness
deliberation ('let me think… actually wait… let me reconsider…') and asked its
clarifying questions twice (draft then 'more concisely' restate). Confirmed the
harness already keeps real reasoning blocks out of the visible text
(ContentBlock::as_text() returns None for Thinking), so this is the model
narrating in its output — a prompt rule is the correct fix.
Add a Style 'Reply hygiene' rule: every message is the finished reply, not a
scratchpad — no deliberation narration, no draft-then-restate, lead with
substance. Adds a builder_prompt regression test. Prompt-only, architect-level
(every turn), platform-agnostic.
graycyrus
changed the title
…ertion (addresses @chatgpt-codex-connector on prompt.md:80) The per-turn revise directive (and its flow_id note) unconditionally told the agent "you may run_flow" — rendered *after* the standing prompt's new capability check (Bld #4), so it could still override that check and reproduce the offer-then-refuse bug on the flows_build path where run_flow/resume_flow_run/cancel_flow_run are hidden. Both now defer to the capability rule instead of asserting the tool is available. Also strengthens the builder_prompt.rs regression assertion (CodeRabbit nitpick) to bind to the specific run_flow capability-gated wording, reject the stale unconditional "ask first!" / ungated resume-cancel phrasing, and positively assert the resume_flow_run/cancel_flow_run availability fallback — not just independent generic substrings.
|
Addressed CodeRabbit's nitpick on
Also fixed a related, more substantive finding from @chatgpt-codex-connector on All four |
|
To use Codex here, create a Codex account and connect to github. |
…) + reply hygiene (tinyhumansai#5007)\n
Two builder-prompt discipline fixes to the same agent (
workflow_builder), both about the copilot saying things it shouldn't:1. Run-tool honesty (Bld #4). The prompt unconditionally advertised
run_flow, but it's hidden on theflows_buildpath (FLOWS_BUILD_HIDDEN_TOOLS) — so the agent offered to run a flow, the user said yes, and it replied it couldn't. Made the run/resume/cancel guidance capability-conditional: check whether the tool is on the belt, and when it isn't, point the user to the Workflows UI Run control instead of offering. No capability/safety change.2. Reply hygiene. The final reply leaked the model's deliberation ("let me think… actually wait… let me reconsider…") and asked its questions twice (draft then restate). Confirmed via pipeline trace that the harness already keeps real reasoning blocks out of visible text — so this is the model narrating in its output. Added a Style 'Reply hygiene' rule: the message is the finished reply, not a scratchpad — no deliberation narration, no draft-then-restate, lead with substance.
Both are prompt-only, architect-level (every turn), platform-agnostic, each locked with a
builder_prompt.rsregression test.