Skip to content

test(desktop): use a wordlist-safe separator in passphrase word-count test - #6356

Merged
wesbillman merged 1 commit into
mainfrom
fizz/fix-6249-flaky-passphrase-test
Aug 20, 2026
Merged

test(desktop): use a wordlist-safe separator in passphrase word-count test#6356
wesbillman merged 1 commit into
mainfrom
fizz/fix-6249-flaky-passphrase-test

Conversation

@ngthuydiem

Copy link
Copy Markdown
Contributor

Summary

Fixes #6249.

key_backup::tests::generated_passphrase_respects_word_count_and_separator joined words with - and asserted the phrase splits back into exactly count parts. The EFF short wordlist contains exactly one hyphenated entry (yo-yo, line 1281 of 1296), so drawing it into either hyphen-joined arm yields one extra part — a ~1-in-186 flake per full suite run (1 - (1 - 1/1296)^7 ≈ 0.539%).

This switches the two hyphen arms to |, which cannot appear in the wordlist — the exact guard the sibling test generated_passphrase_clamps_word_count already documents and uses. The space, dot, and empty-separator arms are untouched (no wordlist entry contains a space or a dot), so the test still covers word count, wordlist membership, and minimum length.

generate_passphrase itself is unchanged — a hyphenated word in a hyphen-joined passphrase is not a product defect, only an ambiguity the test's parsing could not handle.

Verification

At main (196d62f), compiled the desktop test binary once and looped it 2000× per state:

State Failures / 2000 runs Expected
old code 15 ~10.8 (P ≈ 0.539%)
fixed 0 0

Full desktop Tauri suite (cargo test --workspace in desktop/src-tauri): 2693 passed, 0 failed.

… test

generated_passphrase_respects_word_count_and_separator joined words with
"-" and asserted the phrase splits back into exactly count parts. The EFF
short wordlist contains exactly one hyphenated entry (yo-yo, line 1281),
so a draw of yo-yo into either hyphen-joined arm yields one extra part —
a ~1-in-186 flake per full suite run (1 - (1 - 1/1296)^7).

Switch the two hyphen arms to "|", which cannot appear in the wordlist,
matching the guard the sibling test generated_passphrase_clamps_word_count
already documents and uses. The space, dot, and empty-separator arms are
unaffected (no wordlist entry contains a space or dot).

Verified by looping the compiled test binary 2000x at both states:
old code 15/2000 failures (~10.8 expected), fixed code 0/2000.

Fixes #6249

Signed-off-by: Fizz <1f3b09af3c417274e5516bf95fadd3c118f35a31ae922257d550c69159ba931c@buzz.block.builderlab.xyz>
@ngthuydiem
ngthuydiem requested a review from a team as a code owner August 19, 2026 21:09
@ngthuydiem ngthuydiem added the triage-ready Appropriate for agentic review label Aug 19, 2026
@ngthuydiem

Copy link
Copy Markdown
Contributor Author

The full EFF wordlist includes 4 words with "-" in them: https://www.eff.org/files/2016/07/18/eff_large_wordlist.txt

@ngthuydiem

ngthuydiem commented Aug 20, 2026

Copy link
Copy Markdown
Contributor Author

Loop to run this test n times given its flakiness:

RUST_BACKTRACE=1 cargo test --manifest-path desktop/src-tauri/Cargo.toml --lib --no-run
for i in $(seq 1000); do
    printf '\riteration %d' "$i"
    desktop/src-tauri/target/debug/deps/buzz_lib-ac080c7e34975d7e \
      key_backup::tests::generated_passphrase >/dev/null 2>&1 \
      || { echo " FAILED on iteration $i"; break; }
  done

@ngthuydiem

Copy link
Copy Markdown
Contributor Author

Verification:

  • Before on main:
image
  • After on fix branch:
image

@wesbillman wesbillman left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Carl, an automated reviewer, commenting via Wes’s GitHub account.

Reviewed exact head f2828d5c2ec1b58ed19dac1c4cadc001b1eb385f against base f96f1883c5ad94825d43a856e5d25e307a0540cb. No material findings.

The failure mechanism and repair match: the bundled 1,296-entry wordlist contains yo-yo, while it contains no pipe, space, or dot characters. Replacing the two hyphen-delimited cases with | therefore makes split-based word-count and membership assertions unambiguous without changing production passphrase generation. The sibling clamp test already uses the same invariant, and the remaining cases still exercise alternate and empty separators.

Validation: exact-head wordlist invariants and git diff --check passed; all applicable GitHub checks on this head are complete and green. I did not rerun CI-equivalent suites locally. I am leaving a comment rather than approving because Wes requested review, not approval.

@wesbillman
wesbillman merged commit 886cef7 into main Aug 20, 2026
42 of 46 checks passed
@wesbillman
wesbillman deleted the fizz/fix-6249-flaky-passphrase-test branch August 20, 2026 17:53
wpfleger96 pushed a commit that referenced this pull request Aug 20, 2026
…arer-auth

* origin/main:
  test(desktop): use a wordlist-safe separator in passphrase word-count test (#6356)
  fix(models): curate Databricks alias-aware labels for 5 missing endpoints (#6360)
  fix(acp): guard against unrequested public relay skills (#6394)
  feat(desktop): refine context-aware Projects collaboration (#6396)
  fix(desktop): distinguish duplicate agent devices (#6337)

Signed-off-by: Duncan <dcfd242e557282d7a1e2cf2e6877522682f1e5c6156dc92ca7d90eaedd3b0f95@buzz.block.builderlab.xyz>
wpfleger96 added a commit that referenced this pull request Aug 20, 2026
…-scoping

* origin/main:
  perf(desktop): resolve references without directory scans (#6328)
  feat(llm): stamp thinking effort on call-completed log line (#6424)
  Fix cross-owner relay agent mentions in owner-only builds (#6338)
  feat(cli): accept Buzz message links for thread reads (#6359)
  feat(workflows): add workflow editor (#6248)
  fix(desktop): preserve huddle speech boundaries (#6397)
  test(desktop): use a wordlist-safe separator in passphrase word-count test (#6356)
  fix(models): curate Databricks alias-aware labels for 5 missing endpoints (#6360)
  fix(acp): guard against unrequested public relay skills (#6394)

Signed-off-by: Duncan <dcfd242e557282d7a1e2cf2e6877522682f1e5c6156dc92ca7d90eaedd3b0f95@buzz.block.builderlab.xyz>
Co-authored-by: Will Pfleger <pfleger.will@gmail.com>
Signed-off-by: Will Pfleger <pfleger.will@gmail.com>
atishpatel added a commit that referenced this pull request Aug 20, 2026
…ounting-and-solo

* origin/main: (48 commits)
  fix(hooks): scope pre-push lanes to branch merge-base diff (#6423)
  Enforce a three-day dependency cooldown (#6426)
  perf(desktop): resolve references without directory scans (#6328)
  feat(llm): stamp thinking effort on call-completed log line (#6424)
  Fix cross-owner relay agent mentions in owner-only builds (#6338)
  feat(cli): accept Buzz message links for thread reads (#6359)
  feat(workflows): add workflow editor (#6248)
  fix(desktop): preserve huddle speech boundaries (#6397)
  test(desktop): use a wordlist-safe separator in passphrase word-count test (#6356)
  fix(models): curate Databricks alias-aware labels for 5 missing endpoints (#6360)
  fix(acp): guard against unrequested public relay skills (#6394)
  feat(desktop): refine context-aware Projects collaboration (#6396)
  fix(desktop): distinguish duplicate agent devices (#6337)
  feat(desktop): close Buzz window with Cmd+W (#6314)
  refactor(prompt): simplify Buzz agent guidance (#6340)
  feat(desktop): make Projects workspaces selectable (#6368)
  Add Buzz-native collaboration benchmarks (#6264)
  Polish mobile timeline and emoji interactions (#6297)
  feat(desktop): make the Projects overview follow the selected section (#6335)
  refactor(desktop): coordinate TTS playback (#6341)
  ...

Signed-off-by: Atish Patel <atishpatel2012@gmail.com>
brow added a commit that referenced this pull request Aug 20, 2026
…ifications-pr

* origin/main: (33 commits)
  fix(hooks): scope pre-push lanes to branch merge-base diff (#6423)
  Enforce a three-day dependency cooldown (#6426)
  perf(desktop): resolve references without directory scans (#6328)
  feat(llm): stamp thinking effort on call-completed log line (#6424)
  Fix cross-owner relay agent mentions in owner-only builds (#6338)
  feat(cli): accept Buzz message links for thread reads (#6359)
  feat(workflows): add workflow editor (#6248)
  fix(desktop): preserve huddle speech boundaries (#6397)
  test(desktop): use a wordlist-safe separator in passphrase word-count test (#6356)
  fix(models): curate Databricks alias-aware labels for 5 missing endpoints (#6360)
  fix(acp): guard against unrequested public relay skills (#6394)
  feat(desktop): refine context-aware Projects collaboration (#6396)
  fix(desktop): distinguish duplicate agent devices (#6337)
  feat(desktop): close Buzz window with Cmd+W (#6314)
  refactor(prompt): simplify Buzz agent guidance (#6340)
  feat(desktop): make Projects workspaces selectable (#6368)
  Add Buzz-native collaboration benchmarks (#6264)
  Polish mobile timeline and emoji interactions (#6297)
  feat(desktop): make the Projects overview follow the selected section (#6335)
  refactor(desktop): coordinate TTS playback (#6341)
  ...

Signed-off-by: Tom Brow <tomb@block.xyz>
TheSentinel454 added a commit that referenced this pull request Aug 21, 2026
* origin/main: (64 commits)
  chore(deps): pin earshot below 1.2.0 pending a VAD threshold re-pick (#6392)
  polish(desktop): finish Projects navigation and context chrome (#6429)
  fix(desktop): clarify add agents channel action (#6374)
  Repair stale large channel roster snapshots (#6251)
  feat(desktop-messages): show compact Buzz link metadata (#6252)
  feat(workflows): reply in-thread from send_message action (#6178)
  perf(desktop): split discover_acp_providers into cheap and forced paths (#6330)
  fix(desktop): restore recent channel sorting (#6402)
  fix(desktop): isolate main timeline stacking context from focus drawer (#6398)
  fix(desktop): make reconnect repair lossless (#6415)
  fix(hooks): scope pre-push lanes to branch merge-base diff (#6423)
  Enforce a three-day dependency cooldown (#6426)
  perf(desktop): resolve references without directory scans (#6328)
  feat(llm): stamp thinking effort on call-completed log line (#6424)
  Fix cross-owner relay agent mentions in owner-only builds (#6338)
  feat(cli): accept Buzz message links for thread reads (#6359)
  feat(workflows): add workflow editor (#6248)
  fix(desktop): preserve huddle speech boundaries (#6397)
  test(desktop): use a wordlist-safe separator in passphrase word-count test (#6356)
  fix(models): curate Databricks alias-aware labels for 5 missing endpoints (#6360)
  ...

Signed-off-by: Luke Tornquist <tornquist@squareup.com>
TheSentinel454 added a commit that referenced this pull request Aug 21, 2026
…kflow-management

* origin/main: (27 commits)
  chore(deps): pin earshot below 1.2.0 pending a VAD threshold re-pick (#6392)
  polish(desktop): finish Projects navigation and context chrome (#6429)
  fix(desktop): clarify add agents channel action (#6374)
  Repair stale large channel roster snapshots (#6251)
  feat(desktop-messages): show compact Buzz link metadata (#6252)
  feat(workflows): reply in-thread from send_message action (#6178)
  perf(desktop): split discover_acp_providers into cheap and forced paths (#6330)
  fix(desktop): restore recent channel sorting (#6402)
  fix(desktop): isolate main timeline stacking context from focus drawer (#6398)
  fix(desktop): make reconnect repair lossless (#6415)
  fix(hooks): scope pre-push lanes to branch merge-base diff (#6423)
  Enforce a three-day dependency cooldown (#6426)
  perf(desktop): resolve references without directory scans (#6328)
  feat(llm): stamp thinking effort on call-completed log line (#6424)
  Fix cross-owner relay agent mentions in owner-only builds (#6338)
  feat(cli): accept Buzz message links for thread reads (#6359)
  feat(workflows): add workflow editor (#6248)
  fix(desktop): preserve huddle speech boundaries (#6397)
  test(desktop): use a wordlist-safe separator in passphrase word-count test (#6356)
  fix(models): curate Databricks alias-aware labels for 5 missing endpoints (#6360)
  ...

Signed-off-by: Luke Tornquist <tornquist@squareup.com>

# Conflicts:
#	desktop/src/features/workflows/ui/WorkflowDeleteDialog.tsx
#	desktop/src/features/workflows/ui/WorkflowsView.tsx
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

triage-ready Appropriate for agentic review

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Flaky test: generated_passphrase_respects_word_count_and_separator fails ~1 in 186 runs when 'yo-yo' is drawn into a hyphen-joined passphrase

2 participants