sdk/similarity has no batch sweep path: full-tracker duplicate detection costs N x tracker-scan and the exported scoring helpers cannot avoid it without reimplementing the issue_code signal #709

The new sdk/similarity module (2026.7.25) is excellent for the create-time case it was built for, but there is no way to use it for a full-tracker duplicate sweep without paying N x tracker-scan, and the exported scoring helpers cannot be used to avoid that without reimplementing one of the CLI's own match signals.

Context for why a sweep is needed at all: create-time advisory only sees the branch it runs on. When two agents each file a new item for the same problem on separate branches, the field-aware merge driver merges both cleanly — correctly, that is the design — and neither advisory ever saw the other. main ends up with two items for one problem and nothing flags it. Detecting that requires sweeping the merged tracker after the fact.

Measured cost

Against a real 1,934-item tracker (pm-cli's own .agents/pm), calling findSimilarItems once per candidate:

source was metadata_fallback throughout (no vector store configured), so each of the 1,934 calls re-reads tracker metadata.

Why the exported scoring helpers do not solve it

The obvious fix is to read the tracker once and score pairs in memory with the exported scoreItemSimilarity. Measured on the same 1,934 items — 1,869,211 unordered pairs:

pairs 1869211  above_0.6 516  elapsed_ms 29105  rss_mb 169

29.1s — barely better, because scoreItemSimilarity(leftTitle, rightTitle) takes raw strings and re-tokenizes both on every call, so a full sweep tokenizes ~3.7M times for 1,934 distinct titles.

The next obvious fix — precompute tokenizeSimilarityText once per item, pre-filter pairs with jaccardSimilarity on the cached token arrays, and only call scoreItemSimilarity for survivors — is not sound, and this is the crux. scoreItemSimilarity returns the strongest of three signals, and issue_code can score far above the token-overlap signal for the same pair. Real examples from that tracker at threshold 0.6:

pm-02c4 | pm-cg7l   score 0.99  issue_code
  "Track GitHub Dependabot alert #27 for undici (GHSA-4992-7rv2-5pvq)"
  "Track GitHub Dependabot alert #21 for undici (GHSA-4992-7rv2-5pvq)"

pm-0c0j | pm-1lll   score 0.99  issue_code
  "pm list --no-truncate/--all: explicit override of any --limit + surface total vs returned count (GH-154)"
  "pm list output formats: csv/table for human export (GH-154 remainder)"

The second pair in particular has low token overlap — a jaccard pre-filter at any useful threshold discards it, so the sweep would silently lose exactly the duplicates that matter most. The only way to keep it is to extract issue codes the same way the CLI does, and the issue-code extractor is not exported. Reimplementing it locally would break the exact agreement with create-time advisory that a sweep needs in order to be trustworthy — if pm create warns about a pair the sweep does not report (or vice versa) the feature is worse than useless.

So consumers are stuck with N x tracker-scan.

Requested

Any one of these resolves it:

1. A batch entry point, which would be the most useful: something like
   findDuplicateClusters({ pmRoot, threshold, limit, since?, statuses? }) returning collapsed unordered pairs/clusters. The CLI already has everything needed to do this with a single tracker read, and every consumer wanting a sweep needs the same pair-collapsing logic (AB and BA are one finding).
2. A pre-tokenized scoring variant, e.g. scoreItemSimilarityTokens(left: {title, tokens}, right: {title, tokens}), so callers can tokenize once per item. Cheap to add and keeps scoring canonical.
3. Export the issue-code extractor (e.g. extractIssueCodes(title): string[]), which would let consumers build a correct cheap pre-filter — an exact-title map plus an issue-code inverted index plus jaccard — without duplicating CLI logic.

A pm duplicates command over the same primitive would also be a natural fit alongside pm events, given the merge-driver scenario above is squarely a first-class multi-agent concern.

Aside

FindSimilarItemsOptions.limit is documented as "from zero through twenty". A sweep wants all matches above the threshold for a candidate, and 20 is a low ceiling when auditing a large tracker; the cap is not obviously right for the batch case even if it is for the interactive one.