Skip to content

Stop filing Issue Monster Assigner workflow-failure issues - #56866

Merged
nagilson merged 1 commit into
dotnet:mainfrom
nagilson:nagilson-silence-issue-monster-safe-outs
Nov 2, 2026
Merged

nagilson merged 1 commit into
dotnet:mainfrom
nagilson:nagilson-silence-issue-monster-safe-outs

Conversation

@nagilson

@nagilson nagilson commented Oct 30, 2026 •

Copy link
Copy Markdown
Member

Summary

Workflow failures and the explicit Copilot-assignment verification remain enabled. They were only failing when the PAT expired. We don't need a new issue every day for the expired PAT.

Validation

  • Recompiled and validated Issue Monster Assigner with gh-aw v0.89.21 on Linux (WSL).
  • git diff --check passes. The compiler reports the existing missing concurrency.job-discriminator warning.

Stop filing automated failure issues while preserving workflow failures and assignment verification. Regenerate the lock with gh-aw v0.89.21.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 5274162c-4594-4a2b-adfc-95a229988a96
@azure-pipelines

Copy link
Copy Markdown
Azure Pipelines:
Successfully started running 1 pipeline(s).
2 pipeline(s) were filtered out due to trigger conditions.
There may be pipelines that require an authorized user to comment /azp run to run.

@nagilson

Copy link
Copy Markdown
Member Author

Investigation: recurring failures are authentication failures, not prompt-injection flags

The latest 50 main runs all failed (October 14-30). I inspected five representative October runs, including the run behind #56847; each reached the same failure:

GET /repos/dotnet/sdk - 401
Failed to load handler for assign_to_agent:
  Failed to fetch pull request repository for dotnet/sdk: Bad credentials

Evidence and failure chain

  • In run 40983517668, agent_output.json contains the valid assign_to_agent item for issue 56540. The tool call succeeds and the agent exits 0. detection_result.json reports prompt_injection: false, secret_leak: false, and malicious_patch: false.
  • The safe_outputs job uses the dedicated assignment client, but repository lookup returns HTTP 401. Its assignment handler never initializes, so the valid output is not applied. The configured credential is ISSUE_MONSTER_ASSIGNMENT_TOKEN, not an inference-pool PAT.
  • This also occurs in runs 40983528702, 40797927521, 39883140174, and 37415244170, covering October 6-30. Their detectors report prompt_injection: false.
  • The conclusion job correctly fails Verify Copilot assignment, but that is a pre-step: its failure skips Download agent output artifact, Setup agent output environment variable, and Download detection artifact. The always-running failure reporter then has no agent-output path and incorrectly categorizes this as missing_safe_outputs.
  • gh-aw also only exports assignment-specific errors if the handler loaded, so the initialization failure is not exported as an assignment error. The missing-output check then produces the misleading title.
  • The failure-issue deduplication query considers only open issues created within the previous 24 hours. That allows the persistent failure to generate a new issue on later days, rather than keep using one old issue.

Long-term repair

  1. Replace/rotate ISSUE_MONSTER_ASSIGNMENT_TOKEN with a valid, appropriately approved fine-grained PAT scoped to dotnet/sdk: metadata read; actions, contents, issues, and pull requests write, as documented in the workflow. A previous October 1 run successfully assigned an issue; by October 6 the same workflow was receiving 401. Expiry/revocation is plausible, but logs cannot identify the precise reason the credential became invalid. Secret metadata is readable, not its value or expiry; I have not changed credentials.
  2. Add an authenticated preflight for that exact credential in the trusted issue-monster environment, before spending agent/detector credits. Fail explicitly on an authentication or repository-access error; do not substitute the read-only inference PATs or disable threat detection. Add credential-expiry/rotation monitoring.
  3. Preserve diagnostics on assignment-verification failure, and propagate handler-initialization errors as assignment failures upstream. Keep the final assignment-verification gate so failed assignments cannot look successful.

An older September 3 sample had a different failure: threat detection requested an unavailable model and returned HTTP 400. The September 8 model change addressed that; it is not evidence that the present October runs are repeatedly flagging a skill or instruction.

This draft deliberately remains the requested two-file reporting change. Restoring assignments requires credential replacement by a token owner/admin; reporting suppression alone does not restore them.

@nagilson
nagilson requested a review from marcpopMSFT October 30, 2026 22:34
@nagilson
nagilson marked this pull request as ready for review October 30, 2026 22:34
Copilot AI balanced review requested due to automatic review settings October 30, 2026 22:34
@nagilson

Copy link
Copy Markdown
Member Author

Investigation: recurring failures are authentication failures, not prompt-injection flags

TLDR: The PAT token expired and that made safe outputs fail most of the time, so we just need to remember to refresh the token more often. Some of the suggestions the AI made are incorrect or not feasible.

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟢 Approval recommended

No active review issues are reported for the change disabling automated workflow-failure issue creation.

No open issues 🎉

What changed in this PR

Stops Issue Monster Assigner from opening automated workflow-failure issues such as #56847, while preserving workflow failure reporting and explicit Copilot-assignment verification.

Changes:

  • Disables safe-outputs.report-failure-as-issue in the workflow source.
  • Updates the generated lock file’s reporting flags and removes activation-job issue-write permission, retaining existing compiler and runtime pins.
File Description
.github/​workflows/​issue-monster-assigner.md Disables automated failure-issue creation.
.github/​workflows/​issue-monster-assigner.lock.yml Applies the setting to the generated workflow without changing dependency pins.

Review effort: Balanced

@nagilson
nagilson merged commit 78c3739 into dotnet:main Nov 2, 2026
18 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants