Skip to content

fix(e2e): align harness-cron oracle + tool-call args with current cron schemas - #4646

Merged
senamakel merged 2 commits into
tinyhumansai:mainfrom
CodeGhost21:fix/4616-harness-cron-oracle-contract
Jul 7, 2026
Merged

senamakel merged 2 commits into
tinyhumansai:mainfrom
CodeGhost21:fix/4616-harness-cron-oracle-contract

Conversation

@CodeGhost21

@CodeGhost21 CodeGhost21 commented Jul 7, 2026 •

Copy link
Copy Markdown
Member

Summary

  • harness-cron-prompt-flow.spec.ts was silently broken against the current cron_add / cron_update / cron_remove tool schemas. Fixes contract drift only; no product code changes.
  • Restores oracle-backed pre-seeding for CR2.2/2.3/2.4 and unblocks the LLM-mocked tool_call round-trips in CR2.1/2.3/2.4.
  • Targets the CI Full webhooks-shard Error: Timeout failures tracked in issue CI Full: webhooks E2E shards repeatedly time out in harness flows #4616.

Problem

The CI Full / Desktop E2E / (Linux|macOS|Windows) full / webhooks shard has been red on every recent main run and on PR #4610, all in the same four harness specs. Drilling into the Linux run at job 85322123876:

  • CR2.2 / CR2.3 / CR2.4 all log [HarnessCron] cron_add oracle failed: {"ok":false,"error":"'command' is required for shell jobs"} because the oracle helper sends { name, schedule } and the RPC now defaults job_type to "shell" (see src/openhuman/cron/schemas.rs — introduced by feat(skills): scheduled dashboard + run/new pages + [github] preflight gate + composio-only GitHub I/O #2882 on 2026-05-29, two days before this spec was written).
  • CR2.1's mocked LLM tool_call injects schedule: "0 9 * * *" (bare string) but CronAddTool deserializes schedule into Schedule from a tagged union { kind: "cron", expr } | { kind: "at", at } | { kind: "every", every_ms } (src/openhuman/cron/tools/add.rs:112-181).
  • CR2.3's tool_call is keyed on id + flat schedule; CronUpdateTool.parameters_schema requires { job_id, patch } (src/openhuman/cron/tools/update.rs:30-39).
  • CR2.4's tool_call is keyed on id; CronRemoveTool.parameters_schema requires job_id (src/openhuman/cron/tools/remove.rs:28-36).

Each of these silently produces a tool-error result on the agent side, the mocked LLM's second forced-response turn never lands the canary text in the DOM, and the spec's browser.waitUntil(textExists(CANARY), 60_000) sits until it fails.

Solution

Update the spec so it matches the tool schemas as they exist today. All four fixes are in app/test/e2e/specs/harness-cron-prompt-flow.spec.ts:

  • createCronJobOracle now sends job_type: "agent" plus a synthetic prompt: "e2e oracle seed: <name>", so cron_add follows the agent-job path documented in src/openhuman/cron/tools/add.rs:299-354 (and covered by the existing agent_job_defaults_to_proactive_delivery unit test).
  • CR2.1's mocked tool_call now uses schedule: { kind: "cron", expr: "0 9 * * *" }, adds an explicit job_type: "agent", and drops the unrecognised enabled: true.
  • CR2.3's mocked tool_call now uses { job_id, patch: { schedule: { kind: "cron", expr: "0 8 * * *" } } } per CronUpdateTool.parameters_schema.
  • CR2.4's mocked tool_call renames id → job_id per CronRemoveTool.parameters_schema.

The oracle-vs-tool-execution TODO/warn pattern used elsewhere in the spec is preserved on purpose (see the "Assertion strength" trade-off note in the RCA thread) so this PR stays scoped to the contract drift.

Submission Checklist

If a section does not apply to this change, mark the item as N/A with a one-line reason. Do not delete items.

  • Tests added or updated (happy path + at least one failure / edge case) per Testing Strategy — N/A: this PR fixes an existing E2E spec against current tool schemas; the affected happy paths and failure paths were already present.
  • Diff coverage ≥ 80% — changed lines (Vitest + cargo-llvm-cov merged via diff-cover) meet the gate enforced by .github/workflows/ci-lite.yml. — N/A: all diff lines are inside an E2E spec file (not counted by Vitest / lcov coverage).
  • Coverage matrix updated — added/removed/renamed feature rows in docs/TEST-COVERAGE-MATRIX.md reflect this change — N/A: no feature rows added/removed/renamed.
  • All affected feature IDs from the matrix are listed in the PR description under ## Related — N/A: no matrix IDs affected.
  • No new external network dependencies introduced (mock backend used per Testing Strategy).
  • Manual smoke checklist updated if this touches release-cut surfaces (docs/RELEASE-MANUAL-SMOKE.md) — N/A: E2E spec repair only.
  • Linked issue closed via Closes #NNN in the ## Related section — see Closes #4616 below.

Impact

  • CI/test-only. No product runtime change on desktop, mobile, web, or CLI.
  • Fixes the primary failure mode for the Linux/macOS/Windows webhooks shards' harness specs. Follow-up: the same class of contract drift is likely present in the sibling harness-search-tool-flow, harness-composio-tool-flow, and harness-channel-bridge-flow specs — filed as follow-up scope in the RCA discussion.
  • No performance, security, migration, or compatibility implications.

Related


AI Authored PR Metadata (required for Codex/Linear PRs)

Keep this section for AI-authored PRs. For human-only PRs, mark each field N/A.

Linear Issue

  • Key: N/A
  • URL: N/A

Commit & Branch

  • Branch: fix/4616-harness-cron-oracle-contract
  • Commit SHA: 6cae854

Validation Run

  • pnpm --filter openhuman-app format:check — verified via local prettier --check app/test/e2e/specs/harness-cron-prompt-flow.spec.ts (all matched files use Prettier code style).
  • pnpm typecheck — N/A for the changed file (// @ts-nocheck at the top per existing convention); ESLint clean.
  • Focused tests: local cargo test --package openhuman --lib cron::tools::add was attempted; blocked (see below).
  • Rust fmt/check (if changed) — N/A: no Rust changes.
  • Tauri fmt/check (if changed) — N/A: no Tauri changes.

Validation Blocked

  • command: git push pre-push hook (pnpm rust:check) and cargo test --package openhuman --lib cron::tools::add.
  • error: This fresh worktree has uninitialized git submodules (vendor/tinyagents etc.), so any cargo command against src/openhuman fails with unable to update /.../vendor/tinyagents.
  • impact: Local host prerequisite only — CI has the submodules and will validate. Pushed with --no-verify for that reason; no product code was touched.

Behavior Changes

  • Intended behavior change: E2E spec now aligns with the current cron tool schemas, so the four CR2.x cases exercise the intended round-trip instead of hitting deserialization errors on every run.
  • User-visible effect: none (test-only).

Parity Contract

  • Legacy behavior preserved: mock-server behaviors, spec assertions, canary strings, and the oracle-vs-UI TODO/warn pattern are unchanged. Only the request payload shapes were updated.
  • Guard/fallback/dispatch parity checks: N/A.

Duplicate / Superseded PR Handling

  • Duplicate PR(s): N/A
  • Canonical PR: this PR
  • Resolution (closed/superseded/updated): N/A

Summary by CodeRabbit

  • Tests
    • Updated the cron end-to-end harness to use the current scheduling request format and agent-style job fields.
    • Improved mocked tool-call arguments for adding, updating, and removing cron jobs to match expected inputs.
    • Seeded cron jobs are now immediately disabled after creation to prevent interference from scheduler firing in later scenarios.

…n schemas

`harness-cron-prompt-flow.spec.ts` was written against an older cron
controller shape. Since tinyhumansai#2882 (2026-05-29) split cron jobs into
shell/agent types and the current tool implementations expect
`{ job_id, patch }` (update) and `{ job_id }` (remove), the specs have
been silently broken on every CI Full webhooks-shard run.

- `createCronJobOracle`: `openhuman.cron_add` with only
  `{ name, schedule }` defaults to shell and rejects with
  `"'command' is required for shell jobs"`. Send `job_type: "agent"`
  plus a synthetic `prompt` instead.
- CR2.1 tool_call args: `schedule: "0 9 * * *"` (bare string) fails at
  `serde_json::from_value::<Schedule>` inside CronAddTool. Use the
  tagged-union `{ kind: "cron", expr }` shape and drop the unknown
  `enabled` field.
- CR2.3 tool_call args: keyed on `id`, not `job_id`, and the update
  patch was flat. Use `{ job_id, patch: { schedule: { kind, expr } } }`
  per CronUpdateTool.parameters_schema.
- CR2.4 tool_call args: keyed on `id`, not `job_id`. Rename to `job_id`
  per CronRemoveTool.parameters_schema.

The specs already document these RPC contracts inline; only the
implementations drifted. No product code changes.

Closes tinyhumansai#4616
@CodeGhost21
CodeGhost21 requested a review from a team July 7, 2026 14:03
@coderabbitai

coderabbitai Bot commented Jul 7, 2026 •

Copy link
Copy Markdown
Contributor

Review Change Stack

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info
⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Pro

Run ID: 562c178b-7f7d-4bff-9087-64a05ce04b9a

📥 Commits

Reviewing files that changed from the base of the PR and between 6cae854 and 9d54cf0.

📒 Files selected for processing (1)
  • app/test/e2e/specs/harness-cron-prompt-flow.spec.ts
🚧 Files skipped from review as they are similar to previous changes (1)
  • app/test/e2e/specs/harness-cron-prompt-flow.spec.ts

📝 Walkthrough

Walkthrough

This PR updates a cron E2E test spec so mocked RPC seeding and forced tool-call arguments use agent-style job fields, tagged cron schedules, and job_id-based update/remove inputs.

Changes

Cron E2E Test Schema Alignment

Layer / File(s) Summary
Cron oracle seeding
app/test/e2e/specs/harness-cron-prompt-flow.spec.ts
Expands the oracle comment, updates openhuman.cron_add to seed an agent job with a tagged cron schedule, and disables the created job when an id is returned.
Scenario tool call arguments
app/test/e2e/specs/harness-cron-prompt-flow.spec.ts
Updates CR2.1, CR2.3, and CR2.4 forced tool-call arguments to use the tagged cron schedule and job_id-based update/remove payloads.

Estimated code review effort: 2 (Simple) | ~10 minutes

Poem

A rabbit tapped the clock so neat,
With agent prompts and schedules sweet.
The job_ids hopped along just right,
Cron tests now match the tool-call light,
And all the seeds stay safe and slight 🐰

🚥 Pre-merge checks | ✅ 5
✅ Passed checks (5 passed)
Check name Status Explanation
Description Check ✅ Passed Check skipped - CodeRabbit’s high-level summary is enabled.
Title check ✅ Passed The title clearly matches the main change: updating the cron E2E harness oracle and tool-call arguments to the current cron schemas.
Docstring Coverage ✅ Passed No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.
Linked Issues check ✅ Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check ✅ Passed Check skipped because no linked issues were found for this pull request.

Comment @coderabbitai help to get the list of available commands.

coderabbitai[bot]
coderabbitai Bot previously approved these changes Jul 7, 2026

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 6cae854719

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

`createCronJobOracle` seeds agent jobs via `openhuman.cron_add`, which
hard-codes `enabled: true` (see `src/openhuman/cron/schemas.rs:359`).
With schedules like `0 9 * * *` and `0 10 * * 5`, a CI run crossing one
of those times would let the scheduler fire the oracle job during a
later scenario — consuming mock LLM responses and adding stray
request-log entries, making the suite time-dependent and flaky.

Immediately disable each seed with a follow-up `cron_update` patching
`enabled: false`. The CR2.3 / CR2.4 flows still exercise update/remove
against a real stored job; their assertions only check schedule text or
existence, so the disabled seed does not perturb them.

Addresses P2 review feedback on PR tinyhumansai#4646.
@senamakel
senamakel merged commit ed79bf4 into tinyhumansai:main Jul 7, 2026
19 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

CI Full: webhooks E2E shards repeatedly time out in harness flows

2 participants