Conversation
github-actions
Bot
added
the
contribution-gate
The keyless Bing and DuckDuckGo scrape backends ignored the configured search locale: run_bing_search sent only `q` and duckduckgo_search_url likewise, so locale-scoped queries silently returned wrong-region results while Firecrawl, Serply, and SearXNG honored the same knob. Bing now receives `mkt=<locale>` plus a `setlang` derived from the BCP 47 tag: an explicit script subtag wins, otherwise the script is picked from the region (zh-TW/zh-HK/zh-MO map to zh-Hant, everything else to zh-Hans) because a bare `zh` is invalid for Bing. Both scrapes send an Accept-Language header matching the resolved market, without duplicating the generic `en` range for English markets. DuckDuckGo receives `kl` only for the verified region pairs (cn-zh, us-en, jp-jp, kr-kr, tw-tzh, hk-tzh, including the script+region Chinese forms); anything else sends no `kl` at all rather than an off-list guess, and the receipt reports the missing region signal as ignored instead of claiming the knob was honored. When the model omits `locale`, a query containing Han ideographs falls back to the zh-CN market - a locale-less Chinese query previously got an English Accept-Language and unrelated Japanese results - unless the query also carries kana or hangul, which would push Japanese or Korean queries onto the Chinese market. A model-supplied locale is shape-checked before it reaches URLs or headers: malformed values keep the historical request shape and are receipted as ignored. On the DuckDuckGo-to-Bing fallback the stale DDG-leg ignored flag is re-derived from Bing's own rule so the receipt describes the backend that actually produced the results, and both scrape backends now declare locale support in their query capabilities. The model-visible web and web_search schemas state the BCP 47 format and which backend honors what. Signed-off-by: asto <asto18089@126.com>
asto18089
force-pushed
the
upstream/scrape-locale
branch
from
October 5, 2026 14:20
904e116 to
fff8829
Compare
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
The keyless Bing and DuckDuckGo scrapes ignored the configured search locale — Bing sent only
q, DDG the same — while Firecrawl/Serply/SearXNG honored the knob, so locale-scoped queries silently returned wrong-region results.mkt=<market>plussetlangderived from BCP 47 (explicit script subtag wins; otherwise zh-TW/HK/MO →zh-Hant, elsezh-Hans, since barezhis invalid for Bing) and a matchingAccept-Language.klonly for verified region pairs (cn-zh, us-en, jp-jp, kr-kr, tw-tzh, hk-tzh, including script+region Chinese variants); anything else sends noklrather than an off-list guess, and is receipted as ignored. SharedAccept-Languagethreading through the existing fetch helper.localeis omitted, a Han-script query falls back to zh-CN unless kana/hangul veto it; model-supplied locales are shape-checked, and malformed values keep the historical request shape.Tests: per-locale param derivation, the verified
kltable (including script+region), the Han fallback and its kana/hangul veto, shape checks, fallback re-derivation, receipt honesty, and an exhaustive per-provider capability table.Testing
cargo test -p codewhale-tui --lib web_search web::backendcargo clippy -p codewhale-tui --all-targets --all-features --lockedcargo fmt --all -- --checkDisclosures: the in-function flag-push/prune wiring and the emitted URL params are not covered by a networked test (the existing wiremock suites run with locale unset) — the pure helpers, the receipt contract, and the capability table are. With no resolvable market on the DDG leg,
Accept-Languagechanges fromen-US,en;q=0.5to the unifieden-US,en;q=0.9(in-code comment).Adapted from the Pinvou fork's surface-alignment audit (Pinvou/CodeWhale
6f780290f, the scrape-locale slice), integrated with the current backend chain (Serply, #6746 fallback).Checklist
CHANGELOG.mdchanges