Repository navigation
Conversation
add js-audit preset, streamline trufflehog HTTP ingestion
Update trufflehog to 3.96.0
Bumps [pre-commit](https://github.com/pre-commit/pre-commit) from 4.6.0 to 4.6.1. - [Release notes](https://github.com/pre-commit/pre-commit/releases) - [Changelog](https://github.com/pre-commit/pre-commit/blob/main/CHANGELOG.md) - [Commits](pre-commit/pre-commit@v4.6.0...v4.6.1) --- updated-dependencies: - dependency-name: pre-commit dependency-version: 4.6.1 dependency-type: direct:development update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <support@github.com>
Bumps the github-actions group with 1 update: [actions/setup-python](https://github.com/actions/setup-python). Updates `actions/setup-python` from 6 to 7 - [Release notes](https://github.com/actions/setup-python/releases) - [Commits](actions/setup-python@v6...v7) --- updated-dependencies: - dependency-name: actions/setup-python dependency-version: '7' dependency-type: direct:production update-type: version-update:semver-major dependency-group: github-actions ... Signed-off-by: dependabot[bot] <support@github.com>
Bumps [fastapi](https://github.com/fastapi/fastapi) from 0.139.2 to 0.140.0. - [Release notes](https://github.com/fastapi/fastapi/releases) - [Commits](fastapi/fastapi@0.139.2...0.140.0) --- updated-dependencies: - dependency-name: fastapi dependency-version: 0.140.0 dependency-type: direct:development update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <support@github.com>
Bumps [pytest-env](https://github.com/pytest-dev/pytest-env) from 1.6.0 to 1.7.0. - [Release notes](https://github.com/pytest-dev/pytest-env/releases) - [Commits](pytest-dev/pytest-env@1.6.0...1.7.0) --- updated-dependencies: - dependency-name: pytest-env dependency-version: 1.7.0 dependency-type: direct:development update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <support@github.com>
Bumps [mmh3](https://github.com/hajimes/mmh3) from 5.2.0 to 5.2.1. - [Release notes](https://github.com/hajimes/mmh3/releases) - [Changelog](https://github.com/hajimes/mmh3/blob/master/CHANGELOG.md) - [Commits](hajimes/mmh3@v5.2.0...v5.2.1) --- updated-dependencies: - dependency-name: mmh3 dependency-version: 5.2.1 dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <support@github.com>
Bumps [cachetools](https://github.com/tkem/cachetools) from 7.1.4 to 7.1.6. - [Changelog](https://github.com/tkem/cachetools/blob/master/CHANGELOG.rst) - [Commits](tkem/cachetools@v7.1.4...v7.1.6) --- updated-dependencies: - dependency-name: cachetools dependency-version: 7.1.6 dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <support@github.com>
Bumps [orjson](https://github.com/ijl/orjson) from 3.11.7 to 3.11.9. - [Release notes](https://github.com/ijl/orjson/releases) - [Changelog](https://github.com/ijl/orjson/blob/master/CHANGELOG.md) - [Commits](ijl/orjson@3.11.7...3.11.9) --- updated-dependencies: - dependency-name: orjson dependency-version: 3.11.9 dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <support@github.com>
Bumps [regex](https://github.com/mrabarnett/mrab-regex) from 2026.7.10 to 2026.7.19. - [Changelog](https://github.com/mrabarnett/mrab-regex/blob/hg/changelog.txt) - [Commits](mrabarnett/mrab-regex@2026.7.10...2026.7.19) --- updated-dependencies: - dependency-name: regex dependency-version: 2026.7.19 dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <support@github.com>
…regex-2026.7.19 Bump regex from 2026.7.10 to 2026.7.19
…mmh3-5.2.1 Bump mmh3 from 5.2.0 to 5.2.1
Automated Docs Update
…pre-commit-4.6.1 Bump pre-commit from 4.6.0 to 4.6.1
…cachetools-7.1.6 Bump cachetools from 7.1.4 to 7.1.6
…fastapi-0.140.0 Bump fastapi from 0.139.2 to 0.140.0
…pytest-env-1.7.0 Bump pytest-env from 1.6.0 to 1.7.0
…orjson-3.11.9 Bump orjson from 3.11.7 to 3.11.9
Bumps [uvicorn](https://github.com/Kludex/uvicorn) from 0.39.0 to 0.51.0. - [Release notes](https://github.com/Kludex/uvicorn/releases) - [Changelog](https://github.com/Kludex/uvicorn/blob/main/docs/release-notes.md) - [Commits](Kludex/uvicorn@0.39.0...0.51.0) --- updated-dependencies: - dependency-name: uvicorn dependency-version: 0.51.0 dependency-type: direct:development update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <support@github.com>
…actions/dev/github-actions-696b649f44 Bump actions/setup-python from 6 to 7 in the github-actions group
…uvicorn-0.51.0 Bump uvicorn from 0.39.0 to 0.51.0
…esponse Fix docker_pull crash on failed registry request
Merge stable into dev to resolve release conflicts
Contributor
Performance Benchmark Report
|
- Ask before loading a custom module folder (-y answers yes) - Refuse folders and module files that are world-writable or owned by another user - Forbid custom modules from reusing an existing module's name - Remove config.module_dirs, which only validation acted on
Bump mkdocstrings to 1.0.8
…l fixes (#3404) * Bound compare_body input to prevent quadratic DeepDiff stall HttpCompare.compare_body ran DeepDiff with ignore_order=True on line-lists. When two bodies are highly similar but differ on many lines (the exact shape wildcard detection produces on large catch-all hosts), DeepDiff's pairing is quadratic in the differing-line count and holds the GIL, stalling entire scans for tens of minutes. Short-circuit to 'different' when the multiset difference of two line-lists exceeds web.http_compare_max_differing_lines (default 500) before invoking DeepDiff. Caps worst-case comparison at a couple seconds. Dict inputs (XML/JSON) keep the full diff path unchanged. Fixes #3339 * Register http_compare_max_differing_lines in WebConfig schema defaults.yml added the key; the strict pydantic WebConfig model must accept it or test_defaults_yml_validates_against_schema fails. * Broaden compare_body tests: dict path, threshold boundary, config knob - dict inputs still take the DeepDiff path (guard must not short-circuit them) - reordered identical line-lists stay equal (multiset is order-insensitive) - at-threshold vs over-threshold boundary around max_differing_lines - http_compare_max_differing_lines config value reaches the helper * test(diff): pin compare_body regressions the threshold guard introduced Four focused tests, all failing against the current guard: filtered_lines_not_counted asserts a subject differing only in lines the baseline already marked dynamic still compares equal. The guard counts those lines before DeepDiff excludes them, so it returns a false "changed" on any page with many nonces. bounds_dict_bodies asserts an XML/JSON body settles in bounded time. xmltodict.parse succeeds on ordinary HTML, so those bodies are dicts and skip the list-only guard into the unbounded quadratic path. threshold_boundary makes the at-threshold case one DeepDiff calls a match, so the assertion can distinguish the guard firing from the lists simply differing. null_threshold_config covers an explicit null, which resolves to None and raises TypeError on the comparison. Replaces the previous boundary assertions, which expected False on both sides and passed with the feature deleted. * fix(diff): count leaves outside ddiff_filters, and bound dict bodies The guard counted raw list items, which broke two ways. It ignored ddiff_filters, the paths the baseline established as dynamic. Those lines are excluded from the DeepDiff that follows, but they were still counted toward the threshold, so a page with enough per-request nonces reported its body as changed on every probe. That generates findings rather than suppressing them. It also required both sides to be lists, but xmltodict.parse succeeds on ordinary HTML, so those bodies arrive as dicts and dropped into the unbounded quadratic DeepDiff the guard exists to prevent. _leaf_counts walks either shape, skips any subtree whose path is already filtered, and counts leaves, so the threshold sees the same content the diff will. * fix(diff): coerce a null differing-lines threshold to the default http_compare_max_differing_lines is Optional[int], so an explicit null is valid config. dict.get returns None for present-but-null, not the default, and the threshold comparison then raised TypeError on every body compare. * perf(ci): cut test suite wall time via teardown, poll, and dep-install fixes The suite spent most of its wall time waiting rather than working. Each fix targets a specific source of idle time: test harness: - FastShutdownHTTPServer overrides thread_target to poll every 5ms instead of socketserver's 0.5s default, so every httpserver fixture teardown no longer pays half a second - stop_server busy-wait drops from 100ms to 5ms per check - remove an unconditional 0.5s sleep in test_manager_scope_accuracy that waited on module init the scan already guarantees scanner: - main scan loop polls at 2ms while events are flowing and backs off exponentially to 100ms when idle, instead of a flat 100ms sleep that throttled every event batch helpers: - cache the CloudCheck instance and the wordninja LanguageModel, both of which were rebuilt per scan despite identical inputs - search_format_dict short-circuits when no placeholder is present and uses a single compiled regex pass instead of one str.replace per kwarg deps installer: - batch every module's pip deps into one resolver pass before the per module loop, so pip resolves once instead of once per module. Modules with custom pip_constraints are excluded since they must resolve against their own set. * style(test): move FastShutdownHTTPServer below the import block ruff format rejected the class sitting between two import groups. It belongs after the imports regardless, so move it there rather than padding the original spot with a blank line. * perf(ci): run distro tests in parallel distro_tests.yml invoked pytest with no -n, so all six distro containers ran the entire suite serially at ~32 minutes each, while tests.yml ran the same suite in parallel at ~15. The distro matrix, not the test matrix, was setting the wall time for the whole PR. Use the same -n/--dist loadgroup flags tests.yml already uses. worker_count.py bounds the count by cores and available RAM, so the container gets a worker count it can actually feed. * perf(ci): scope distro tests to install and runtime surface The distro matrix ran the full 151-module suite on all six containers. Module tests exercise module logic, which does not vary by distro and is already covered five times over by the Python matrix in tests.yml. Six redundant full-suite runs were setting the wall time for every PR. The question the distro matrix exists to answer is narrower: does bbot install, resolve deps, and run here. Scope it to the files that actually probe that surface (e2e, cli, depsinstaller, command, files, python_api, config, dns) and let tests.yml own module coverage. * perf(test): split test_cli_args into four independent tests test_cli_args was a single 292s test wrapping roughly forty independent cli._main() invocations. xdist cannot split one test, so it set a hard floor on wall time no matter how many workers were available. Each section already set argv, called, and asserted in isolation, so the split is mechanical. --install-all-deps gets its own test since it dominates the runtime and now occupies a worker without blocking the other three sections. * perf(test): shard test_module_loading and perdomainonly across workers Both tests walked all 116 scan modules in a single body, so xdist could not split them and they sat at 259s and 246s respectively, setting a floor on wall time alongside test_cli_args. Parametrize both over a module shard so each case loads its own subset. The union of the shards is the full module list, verified for both the scan and output sets, so no module loses coverage. Four shards measured better than eight: each shard pays scanner init once, so past a point the fixed cost outweighs the split. * perf(ci): oversubscribe cores when sizing xdist workers worker_count capped at core count, so GitHub's 4-core runners ran 4 workers while every worker sat waiting on subprocesses, local HTTP servers, and DNS. The suite is I/O-bound, so cores are the wrong ceiling. Double the cpu ceiling and leave the memory bound untouched, since memory is what actually OOM-kills a run. A 4-core runner now gets 8 workers; the 16-core/16GB case the memory guard was written for still resolves to 16. * test(ci): update worker_count guards for core oversubscription The core-cap guards pinned the old ceiling and failed once workers were oversubscribed. Assert against OVERSUBSCRIBE rather than a literal so the guards track the constant instead of restating it. The memory-cap and never-zero guards are untouched: memory is still the bound that prevents an OOM-killed run. * Revert "test(ci): update worker_count guards for core oversubscription" This reverts commit df3cdb8cdfb2193a6ec1f1cb8bea5fa9edfb7c41. * Revert "perf(ci): oversubscribe cores when sizing xdist workers" This reverts commit eb3c58d7bf75870f004ce7f780f019e9326ff435. * perf(ci): shard the test matrix across runners xdist only parallelises within a single runner, so the suite was pinned to one 4-core machine per Python version and roughly 3800s of work could not go faster than about 950s of wall time. Add BBOT_TEST_SHARDS/BBOT_TEST_SHARD, sharding on the sorted nodeid at collection time, and fan the matrix out to four shards per version. Every test lands in exactly one shard, verified against a full collection: 869 tests, union of the four shards, no overlap and nothing dropped. Deselected tests are reported through pytest_deselected so the counts in each job stay honest about what ran. * fix(test): install neo4j before the test that imports it setup_before_prep monkeypatches neo4j.AsyncGraphDatabase, which imports neo4j, but the pip_install lived in setup_after_prep and therefore ran later. The test only passed because something earlier in the session had already pulled the package in, an ordering dependency that sharding exposed as ModuleNotFoundError. Move the install ahead of the patch so the test stands on its own. Verified against an environment with neo4j uninstalled. * fix(ci): restore required test contexts after shard fanout Sharding added a second matrix dimension, renaming every job from "test (3.10)" to "test (3.10, 0)". The protecc ruleset on dev requires the un-sharded context names, so those four contexts were never reported and PRs hung on Expected forever. Add a test_gate job matrixed on python-version only, named to emit the exact required contexts, gated on the sharded matrix result. * Revert "fix(ci): restore required test contexts after shard fanout" This reverts commit 4326dcd2c541a90173de6648ce86774c836ced7f. * Revert "perf(ci): shard the test matrix across runners" This reverts commit 65a3ccc902879ab7dc7691625355db02ab9d92a1. * Revert "perf(test): shard test_module_loading and perdomainonly across workers" This reverts commit b2ae7b6262692a1c39a150ccf8f4bd34f737e7a5. * Revert "perf(test): split test_cli_args into four independent tests" This reverts commit 3f31acef3b72c4de39efeca192f3656c76844e2c. * perf(test): install module deps once into a shared dir Every xdist worker got its own BBOT home, and dependency installs are keyed on that home, so all 126 modules' deps were installed once per worker instead of once per run. That was the single largest fixed cost in the suite and it scaled with worker count. Split the home into two halves: - cache, tools and lib move to BBOT_SHARED_DEPS_DIR when set. These hold install state, downloaded binaries and libs, all of which are keyed on content and safe to share between concurrent scans. - scans, temp and the rest stay per worker, so scan output and the sessionfinish cleanup remain isolated exactly as before. The module hash now keys on the deps directory rather than bbot_home, otherwise a shared install would still be invalidated per worker. Concurrent installs into the shared dir are serialized with an flock on the deps dir, and setup status is re-read after acquiring it, so a worker that waited picks up what the holder just installed instead of redoing it. * fix(test): point core deps dirs at the shared dir too The shared deps dir was only wired into ConfigAwareHelper, but BBOTCore exposes its own cache_dir/tools_dir/lib_dir off home, and those are what set BBOT_TOOLS for the ansible playbooks. Installs therefore targeted a tools dir under the per-worker home that nothing had created, and every ansible download task failed with: dest '/root/.bbot/tools' must be an existing dir Move the redirect to BBOTCore.deps_home so core and helper resolve the same paths, and have the helper and the module hash read it from there instead of recomputing it. temp and scans stay on home. Writing the module preload cache is now atomic (temp file plus rename) since the cache dir is shared between concurrent scans and a reader must never see a half-written pickle. * test: assert BBOT_TOOLS against the resolved tools dir Both tests hardcoded the tools dir as home/tools, which only held while deps lived under the scan home. With the install-once dirs redirected, BBOT_TOOLS is the shared path and both failed: assert '/tmp/.bbot_test_shared/tools' == '/tmp/.bbot_python_api_test_gw1/tools' assert '/tmp/.bbot_test_gw3/tools' in [...] Assert against helpers.tools_dir so the check follows wherever tools resolve. test_python_api keeps its home coverage by asserting scans_dir still lands under the configured home, which is the half of the split these tests were really pinning. * test(nuclei): point template paths at the shared tools dir The nuclei module tests built template paths from the per-worker home, but nuclei downloads its templates into the tools dir, which now resolves to the shared location. The module updated templates under the shared dir and was then handed a path under the worker dir: Could not find template '/tmp/.bbot_test_gw3/tools/nuclei-state/templates/...' Could not run nuclei: no templates provided for scan Export BBOT_TEST_TOOLS_DIR alongside the shared dir and build the three template paths from it, so the tests reference the same tools dir the module installs into. The XDG env paths stay on the per-worker home, since those intentionally point at throwaway dirs. * fix(test): check every loaded module in test_module_loading Two problems, one root cause: the test called _prep() when it only ever reads class attributes off the module instances. The async-method check was dead. not_async was reset at the top of each iteration of the module loop while the assert sat outside it, so only the last module's methods were ever examined. Hoist the list out of the loop and report the offending qualnames. _prep() also runs setup() on every module. The ones that talk to a service spend their whole connect-retry budget failing against a host that isn't running: rabbitmq alone takes 29s of a hardcoded 30-attempt loop, with postgres and mysql behind it. Worse, _prep() then drops the failures from scan.modules, so the modules most likely to regress were deleted before the loop could inspect them. load_modules() instantiates every module without dialing anything, which is all the assertions need. The test now covers 151 modules instead of 110 and takes 1.9s instead of 47.6s. * fix(test): exercise the dead per_domain_only branch in perdomainonly The per_domain_only assertions never ran. The loop only probed modules watching URL, but all four per_domain_only modules (azure_tenant, emailformat, skymem, viewdns) watch DNS_NAME, so the branch was unreachable and its assertions had rotted since the per_host_only rework in 0111db706: _per_host_tracker has not been written to since that commit; dedup state moved to _incoming_dup_tracker. The reason string also changed from "per_domain_only enabled and already seen domain" to "module has already seen it (per_domain_only=True)". Probe DNS_NAME modules too so the branch is reached, and assert against the state and reason string the implementation actually produces. Verified negatively: the old _per_host_tracker assertion fails when the branch is genuinely executed. Separately, _prep() already calls setup_modules() internally, so the explicit setup_modules() call after it re-ran setup() on every module. The test only reads dedup state off each module, so load_modules() is enough. Modules that dial an absent service (rabbitmq, postgres, mysql) were each burning their full connect-retry budget, twice. 128 modules loaded vs 99 after setup_modules() pops the failures, so coverage rises: 75 modules probed, 4 per_domain_only exercised where previously 0. Test body drops from ~12.3s to 0.9s. * perf(deps): skip the install lock when every dep is already satisfied install() took an exclusive flock before deciding whether there was any work to do. Under xdist that serializes every worker behind whichever one is actually installing: a warm scan needing zero installs still blocked for the full duration of an unrelated worker's install. Visible in CI as a convoy. On py3.13 run 32818237342, test_manager_scope_accuracy_correct was credited 241.8s while its body runs in ~7s. The log shows gw2 stalled 06:48:53 to 06:52:54, releasing only after gw1 finished the dep-install tests, with three workers reporting PASSED within 20ms of each other. Check first, outside the lock, whether the core deps cache is present and every module is already marked installed with its pip deps satisfied. If so, return the same (succeeded, failed) the locked path would have. The check only reads. Anything unresolved, an unknown module, a missing cache entry, force_install or retry_failed, still falls through to the lock, so the install path itself is unchanged. Readers now read setup_status.json without the lock, so write it via a temp file plus os.replace to avoid a torn read. Hoisted the module hash into _module_hash() so both paths compute it identically. Measured on test_depsinstaller.py + test_modules_basic.py, -n 4 --dist loadgroup: 184s -> 8s, identical pass/fail set. * perf(deps): key the dep build scratch dir off deps_home, not home BBOT_TEMP is substituted into module ansible playbooks by find_and_replace(), so it lands inside preloaded["deps"], and _module_hash() hashes that dict. Pointing it at home meant the hash carried the per-worker home path, so the three modules whose playbooks reference BBOT_TEMP (medusa, legba, retirejs) hashed differently on every xdist worker. Each worker missed the shared setup_status entry and rebuilt them from source, medusa via autoreconf + configure + make. Measured in CI run 32821811073 (py3.13): medusa built 3 separate times for 315.3s total, avg 105.1s per build, plus retirejs 4x/54.8s and legba 5x/19.3s. Because install() holds the lock while building, the other workers block on it: three of the four released within 300ms of each other at 07:36:32 after 103s of silence, and again at 07:38:39. That is the same convoy b9bdb4e25 fixed for the lock, arriving here through a stale hash instead. BBOT_TEMP is build scratch keyed on content, exactly like cache/tools/ lib, so it belongs with the other install-once dirs. Scan-scoped temp files are unaffected: those live under Scanner.temp_dir (home/scans/<name>/temp), which is unchanged and still per-worker. Production is a provable no-op. deps_home returns home unless BBOT_SHARED_DEPS_DIR is set, and only the test suite sets it. Verified: - medusa/legba/retirejs hash identically across two worker homes now; distinct=2 before, distinct=1 after. nuclei was already 1 (no BBOT_TEMP in its playbook), confirming the placeholder is the cause. - End to end: seed setup_status from gw0's home, then call install() from gw1's home against the same shared deps dir. Fast path hits in 0.000s, succeeded=[legba, medusa, retirejs]. Negative control on the pre-fix tree returns None from _all_deps_satisfied, i.e. it would have rebuilt medusa. - With no BBOT_SHARED_DEPS_DIR, temp_dir is still home/temp. - test_modules_basic + test_helpers + test_presets + test_bloom_filter: 59 passed. * perf(medusa): copy the build tree with remote_src, not through the controller The "Copy medusa repo" ansible task copied a 161-file directory without remote_src, so every file round-tripped through the ansible controller individually. Measured locally at 114s for that single task; the whole rest of the build (git clone, autoreconf, configure, make, make install) is only about 20s. medusa is the only module that builds from source, and --install-all-deps in test_cli_args runs it while holding the global install lock. In CI (py3.13 run 32825691146) that produced a 108s window, 08:18:54 to 08:20:42, with zero log lines from any of the 4 xdist workers. Every worker was blocked on one directory copy. Setting remote_src keeps the copy on the target instead of streaming each file through the controller. Same 161 files, same working binary. Full playbook end to end, package task excluded: before 143.4s, after 30.5s (4.7x), binary present and valid in both. test_module_medusa passes (1 passed, 17.9s). * perf(nuclei): parse budget templates with the libyaml loader NucleiBudget parses every file under nuclei-templates twice (once to build path frequencies, once to find collapsible templates). That is 13391 yaml files, and parse_yaml used yaml.safe_load, which is the pure python SafeLoader. Measured 58.9s of straight parsing during module setup, all of it on the critical path before the first scan event. pyyaml is already built with libyaml here, so CSafeLoader is available and produces identical output. Import it as YamlLoader with a SafeLoader fallback for builds without the C extension. Verified: - Deep-equality over all 13391 real templates: 0 diffs, and both loaders raise YAMLError on the same 0 files, so the except branch is unchanged. - Type resolution parity checked on anchors, merge keys, unicode, block scalars, implicit dates, nulls, octal/hex/underscore ints, inf/nan. - NucleiBudget end to end: 58.91s -> 8.54s (6.9x), peak RSS 537MB -> 436MB, with byte-identical collapsible_templates, severity_stats and budget_paths. - NEGATIVE CONTROL on the stashed pre-fix tree, same script: 58.91s. - Fallback branch exercised by deleting yaml.CSafeLoader before import; module resolves SafeLoader and still imports clean. - test_modules_basic 7 passed, test_helpers + test_presets + test_bloom_filter 52 passed. ruff check and format clean. Found via the debug-artifact method: bucketing pytest_debug.log by second showed a 47s window (09:22:43 -> 09:23:30) with zero output from any worker, terminating exactly at nuclei's "Loaded [1309] templates" line. Largest remaining dead zone in the run. * perf(nuclei): serialize concurrent template updates on a lock Every nuclei module setup() ran -update-templates unconditionally against the shared tools dir. Under xdist all four workers hit the same 13k-file template tree at once, each extracting over the others' work. Measured against the real binary, four concurrent cold updates into one shared dir: 73.4s wall. A single uncontended update: 14.5s. The extra 59s is pure contention, and it showed up in CI as a 23.2s dead zone where no worker logged anything, terminating exactly on "Successfully updated nuclei templates". Take an exclusive flock beside the templates dir. The winner updates. A waiter that had to block wakes up, sees the tree already populated, and skips its redundant update rather than re-running it. The lock is taken off the event loop via run_in_executor_io so blocking on it cannot stall the worker's loop. Cold 4-way: 73.4s -> 15.9s, identical 13619 templates installed. Warm 4-way stays at 0.95s because the fast path never runs the subprocess. The corruption-repair branch in setup() is unaffected: an empty templates dir still reads as not-installed and still triggers the wipe and retry. * fix(test): make docker container startup fail fast and self-heal test (3.10) failed on TestKafka after burning 532s of wall time. Three linked defects, each of which turns a transient docker error into a long dead wait plus a hard failure on every retry. 1. Container leak on a failed attempt. kafka's check() called consumer.start() OUTSIDE its try block, so when the consumer could not reach a broker the finally block never ran and bbot-test-zookeeper stayed up holding port 2181. Moved consumer.start() inside the try so teardown always runs. 2. docker run errors were silently discarded. Every container was started with a fire-and-forget create_subprocess_exec whose returncode was never checked (mysql and postgres checked it only AFTER the port wait, which is too late to matter). With the leaked container still holding the name, attempt 2 got "Conflict. The container name /bbot-test-zookeeper is already in use" and attempt 3 got "failed to bind host port for 0.0.0.0:9092: address already in use". Both were thrown away, so the test proceeded as if the broker had started. 3. wait_for_port_open looped forever. Having ignored the start failure, the test then waited on a port nothing would ever bind. The log shows 1051 consecutive "Waiting for port 9092" lines at 0.5s each, roughly 532s of a 14m11s job spent spinning on a container that was never running. Adds start_container/stop_container on ModuleTestBase. start_container does docker rm -f first so a leaked container from a prior attempt cannot poison the retry, and raises with docker's stderr when the run fails. Bounds wait_for_port_open by the existing CONTAINER_READY_TIMEOUT (180s) and reuses the OOM hint. stop_container uses rm -f rather than stop, so teardown removes the name even when the container is already dead. Also drops the mysql/postgres teardown exceptions that raised before the sibling container was stopped, which was another way to leak a container. Applied the helpers to all seven docker module tests: kafka, mongo, mysql, nats, postgres, rabbitmq, elastic. No assertions changed. Verified against real docker: - start over a live container of the same name recovers in 0.6s - bad image raises in 1.3s with docker's stderr instead of hanging - wait_for_port_open(timeout=3) raises at 3.0s - stop_container removes the container; a no-op on an absent name - NEGATIVE CONTROL: faithful pre-fix reimplementation with a leaked container still spinning at the 12s cap, unbounded in CI. This is the proof the change is load-bearing. - mysql/postgres/social fail identically on the stashed clean tree, local ports 3306/5432 are held by unrelated containers. Not regressions. - ruff check + ruff format --check clean. * perf(deps): skip the community.general install when ansible-core suffices install_core_deps() unconditionally shelled out to `ansible-galaxy collection install community.general` on every cold config dir. On the CI runners that is 15.6s of download for a collection nothing in the run ever loads. Measured on py3.10 run 32839809554 (job 97776672473), pytest_debug.log: 10:58:12,758 Installing Ansible Community General Collection 10:58:28,373 Successfully installed The scan that paid it had "No scan modules to load" and every module reporting "No dependency work to do". Pure dead wall time. Root cause: the collection is only load-bearing because ansible's `package` action resolves to a per-manager module. ansible-core bundles apt, dnf and dnf5; pacman, apk, zypper, portage and friends live in community.general. CI runs on ubuntu-latest, which is apt, so the dispatch target was always already present. Every ansible module bbot actually names in its task definitions (command, copy, file, get_url, git, package, shell, unarchive, uri) is bundled in ansible-core. Fix: probe which package manager binaries exist via ansible's own PKG_MGRS table and skip the galaxy install only when every one of them is bundled. Hosts that need it still install it, so non-Debian users are unaffected. Verified: - dispatch, with the collection unavailable: apt and dnf resolve, pacman and zypper fail with 'Could not find a matching action for the "pacman" package manager'. Confirms both the necessity and the exemption. - platform matrix: ubuntu/apt True, fedora/dnf True, arch/pacman False, alpine/apk False, suse/zypper False, macos/brew False, no-pkg-mgr False, and ubuntu with a stray pacman False (any unbundled manager vetoes). - this Arch box correctly returns False and keeps installing. - fails safe: ansible internals unimportable returns False. - cost 69ms cold, 0.1ms warm, against the 15.6s it removes. - ruff check + ruff format --check clean; test_modules_basic, test_presets, test_python_api pass. test_depsinstaller fails identically on the stashed clean tree (known local sudo/ansible env issue), not a regression. * perf(test): stop forcing a full dep reinstall in the gowitness tests TestGowitness and TestGowitness_MultiPort set deps.behavior=force_install, which makes every scan they run reinstall gowitness from scratch: the 11-task shared chromium playbook plus the gowitness binary download. Three of the five tests in the file inherit that setting, so CI paid it four separate times. The flag dates back to 2023 (36acee410), when tools_dir was derived from bbot_home. The tests set a private home and rmtree it at import, so back then the wipe really did remove the installed tools and force_install was load-bearing. It is not anymore. tools_dir, cache_dir and temp_dir now come from BBOTCore.deps_home, which the test suite points at a shared dir via BBOT_SHARED_DEPS_DIR; only scans_dir still follows home. So the rmtree clears scan output and leaves the tools completely untouched, and force_install now reinstalls tools that were never removed. It was also actively harmful beyond its own runtime. force_install bypasses the already-installed check for shared deps (installer.py:322), so these tests rewrote the shared chromium setup_status entry on every run. Removing it costs no coverage: TestGoWitnessWithBlob and TestGoWitnessLongFilename already run without force_install and pass in CI, exercising the same module against the same shared tools dir. Measured on py3.10 run 32839809554, gowitness ran the chromium playbook 4x plus 4 binary downloads, 26.6s of the 107.7s total ansible time. In-CI test durations show the split cleanly: the three force_install tests took 9.9s, 14.3s and 14.8s while the two without it took 6.2s and 4.6s. Verified: - Structurally: tools_dir/cache_dir resolve under deps_home, not under the test's home, so rmtree(home_dir) cannot remove them. - Cold runner: with an empty shared deps dir and no setup_status.json, the fast path returns None and still falls through to a real install. - Warm runner: returns (['gowitness'], []) and skips the playbook. - Guard intact: a missing pip dep still declines the fast path. - gowitness is the only deps_common=chromium consumer, so no sibling module depended on the forced refresh. - Local setup time 8.79s -> 0.53s per test, same 5 pre-existing chromium errors before and after (chromium needs sudo to install on this box). - ruff check and ruff format clean. test_depsinstaller fails identically on the stashed clean tree, confirmed by stash and rerun. * perf(deps): cache ansible facts across playbook runs Every ansible_run() re-ran the "Gathering Facts" step from scratch. That step costs ~1.6s and runs before any task, so a 1-task playbook that does almost nothing still took ~2.5s. A CI job makes ~16 ansible_run() calls, so the suite paid the fact-gathering toll ~16 times for facts that never change within a run. The facts are needed: shared_deps.py and installer.py gate 30+ tasks on ansible_facts os_family/distribution/system/architecture, so gather_facts cannot simply be turned off. Caching them is the correct fix. ansible_runner forces ANSIBLE_CACHE_PLUGIN_CONNECTION to <artifact_dir>/<ident>/fact_cache, and bbot rmtree'd its data dir on every call, so the cache could never survive. Point artifact_dir at a stable shared dir and use the settings fact_cache key (which is joined onto artifact_dir) to hop up one level, so the cache lands beside the per-run idents rather than inside one. ident stays a fresh uuid4 so each run's events remain isolated; sharing it replays stale events and would corrupt failure detection. The per-run artifact dir is removed after res.events is consumed, since events are read lazily off disk. A corrupt cache entry would otherwise make every subsequent playbook fail until the 24h timeout expired, so unreadable entries are discarded at construction. Measured, real ansible_run() on conditional tasks: 2.5s -> 0.9s warm, cold path unchanged. Verified skip/ok conditionals resolve identically before and after, failing playbooks still report status and the same error string, 4 concurrent cold writers all succeed (jsonfile cache writes are atomic via mkstemp+rename), expired entries are re-gathered, and a missing or corrupt cache falls back to a full gather. * fix(test): wait for the container service, not the published port test (3.11) failed TestKafka with "Unable to bootstrap from [('localhost', 9092)]" and test (3.14) errored TestMySQL with "Lost connection to MySQL server during query" after 11 attempts. Both are the same defect, exposed by 89245a182 rather than caused by it. ROOT CAUSE: docker binds a published host port at container CREATE time, before the containerized service is listening. Connections that arrive in that window are accepted by the docker proxy and immediately EOF'd. is_port_open() did a bare connect and returned True on success, so wait_for_port_open() was satisfied by the proxy alone and returned while the broker/server was still booting. Before 89245a182 the `docker run` subprocess was not awaited, so the port did not exist yet on the first probe and the accidental 1.5s of sleep(0.5) loop plus the trailing sleep(1) usually covered zookeeper's boot. Awaiting `docker run` removed that accidental delay and the race went from usually-hidden to usually-lost. Measured on this box: PRE-FIX (docker run NOT awaited): ready at 1.51s, loops=1 POST-FIX (docker run awaited) : ready at 1.38s, loops=0 Zero loop iterations means the port check never actually waited for anything. This also explains why "Waiting for port" appears 0 times in every job log, passing and failing alike, and why the kafka retries in the 3.11 log took only 1.3s and 3.1s: far too fast for Kafka to boot. FIX: treat an instant EOF as not-yet-listening. Only a connection that stays open past a short settle window, or one that sends a banner, counts as ready. Callers and the CONTAINER_READY_TIMEOUT bound are unchanged, so a service that never comes up still fails fast with the same error. Verified against real docker: - Time-to-ready now tracks real service boot instead of port binding: zookeeper 1.0s -> 2.5s, nats 1.0s -> 2.0s, rabbitmq 1.0s -> 9.6s, mysql 1.0s -> 16.1s. - DECISIVE NEGATIVE CONTROL on mysql, handshake attempted at the instant each helper declares ready: OLD: ready at 1.01s -> read returns b'' (server not up, module fails) NEW: ready at 18.10s -> real MySQL greeting - Guards: dead port raises at the timeout bound (3.02s of 3s); a published-but-silent container correctly raises instead of being declared ready (the old bug); a real listener that sends no banner is still detected at 1.50s, so no false negative. - kafka 1 passed in 79.7s, mysql 1 passed in 21.0s, both previously failing in CI. kafka/nats/rabbitmq/mysql/elastic: 5 passed. - mongo errors locally on "port is already allocated", 27017 is held by an unrelated bbot-backend container. Fails identically on the stashed clean tree. Not a regression. - ruff check + ruff format --check clean. No assertions changed, no tests split. * perf(diff): skip DeepDiff pair-matching for flat string bodies HttpCompare._baseline() diffs two baseline responses with ignore_order=True. When the body is not XML, both sides are text.split("\n"), so a page with many dynamic lines hands DeepDiff a large set of unmatched string items. DeepDiff then runs _get_most_in_common_pairs_in_iterables, which constructs a nested DeepDiff for every added/removed combination to compute a rough distance. For 600 differing lines that is 360000 nested diffs: 28.8s in one call, holding the GIL. This is the same quadratic pairing stall that dd91685f7 bounded in compare_body for issue #3339, but _baseline() was never given the same protection and runs before any threshold can apply. Pair-matching only exists to align items that can be diffed deeper. When every item is a string there is no substructure to align, and DeepDiff's TreeResult.mutual_add_removes_to_become_value_changes already folds equal-index add/remove pairs back into values_changed. So for flat string sequences the pairing pass is pure cost with no effect on the result. _ordered_diff() centralizes the three ignore_order call sites and sets cutoff_intersection_for_pairs=0 only when both sides are flat all-string lists. Dict bodies (xmltodict output) and any mixed-type list keep the full pairing path, since there the pass does change path granularity. Verified: - 2650 randomized and exhaustive flat-string cases compared on the full payload (report key, path, t1, t2, report_type): 0 mismatches. 8045 cases compared on paths and key counts: 0 mismatches. - Confirmed the pass is genuinely skipped by spying on _get_most_in_common_pairs_in_iterables: 0 calls for all-string input, still 1 call for dict input and for mixed-type lists. - Guard checked against lists, tuples, dicts, bytes, None, nested and unicode. Only flat all-string lists take the fast path. - The 600-line baseline diff goes 24.4s -> 0.06s (380x) with byte-identical tree output. Pre-fix timing on the same input is the negative control. - test_web.py: 52.35s -> 26.67s, with an identical 3 failed / 18 passed set before and after. Those 3 fail on the clean tree too (stale local Rust extension, no decode_error attribute). - test_helpers/test_presets/test_bloom_filter/test_scan: 65 passed. - url_manipulation, bypass403, paramminer_headers: identical 9 failed / 3 passed before and after, all pre-existing locally. This is product code, so real scans against large dynamic pages benefit, not just CI. * perf(ci): cut distro matrix down to the install surface The distro jobs still ran eight test files on all six containers. Only three of them probe anything that varies by distro: test_e2e (builds a venv and runs the real binary as a subprocess), test_depsinstaller (ansible and the package-manager path), and test_command (subprocess and sudo handling). The other five were pure library logic. test_dns, test_cli, test_config, test_files and test_python_api exercise Python that does not change between Debian and Fedora, and tests.yml already runs the entire suite on five Python versions. Running them six more times per PR bought nothing. Those five were also the entire cost. Per distro they accounted for roughly 5.5 of the 7-8.5 minutes of test time, with test_dns and test_cli alone over 300s on every container. Dropping them also removes the fedora hang from this workflow. test_dns_helpers reaches the live nameservers list on raw.githubusercontent.com through resolver_file(), unmocked. When that fetch stalls it burns two 300s download attempts, which matches the 668s gap seen on back-to-back fedora runs where four workers blocked and released together. That download still needs mocking for tests.yml; this just stops paying for it six times over. Collection drops from 48 tests to 9. * perf(lightfuzz): stop delay probing once blind sqli is confirmed The time-based blind SQLi loop iterated all 8 DELAY_PROBE_TEMPLATES even after a template passed both confirmation stages. Every additional template that also fires costs delay_stage1_reps*delay_low + delay_stage2_reps*delay_high (3*3s + 3*8s = 33s) and appends a second result for the same injection point. Several templates differ only by quote context, so a vulnerable parameter routinely matches more than one. Test_Lightfuzz_sqli_delay_or_rowindependent matched both the quoted and unquoted "OR SLEEP(n) IS NOT NULL" variants and emitted two findings whose payloads differed by a single leading quote. Measured with a standard_probe tracer over the sqli tests: or_rowindependent 66.5s probe wall, 60 probes, 2 findings -> 33.3s probe wall, 45 probes, 1 finding sqli_delay 33.3s, 57 probes -> 33.3s, 42 probes, 1 finding Non-firing tests are unchanged at 0 findings with identical probe counts, including the jitter false-positive guard (54 probes, 7.6s, 0 findings). Full file: 114 passed, 388.82s -> 360.12s. url_manipulation, bypass403 and test_web: 26 passed. * perf(lightfuzz): stop padding oracle probing once the verdict is fixed padding_oracle_execute always sent all 254 probes even when the outcome was already decided. The return value is a function of differ_count and block_size only: once differ_count exceeds block_size the result is pinned to None (possible_first_byte) or False, and no further increment can move it. Break there instead of finishing the sweep. Verified exhaustively over every (block_size, possible_first_byte, break-point, reachable-final-count) combination: 0 mismatches, so the early exit is provably equivalent to the full sweep. Measured on the crypto cluster, per-block-size verdicts byte-identical before and after: CBCBitflipDetection_NoPaddingOracle 1020 -> 56 probes, 8.37s -> 2.41s envelope_isolation_cbc_bitflip_no_po 1020 -> 56 probes, 7.90s -> 2.43s PaddingOracleDetection_Noisy 767 -> 492 probes, 5.35s -> 4.26s Confirming cases are untouched: they return True at 254 probes, exactly as before, so detection sensitivity is unchanged. Also fixed a duplicate-finding defect in padding_oracle: it kept testing remaining block sizes after appending a result, so one vulnerable parameter could emit multiple Padding Oracle findings differing only by block size. cbc_bitflip already returns after its first match; this makes padding_oracle consistent with it. Affects real scans, not just tests. Full file: 114 passed, 355.33s -> 333.84s. * perf(test): build the e2e venv once instead of once per worker test_e2e.py's bbot_venv fixture is module-scoped, but xdist scopes fixtures per worker, not per session. With --dist loadgroup the four TestE2E tests were free to land on four different workers, so the fixture was constructed once on each of them. Each construction runs python -m venv plus pip install -e on the local checkout, measured at 40s for the pip step alone. Confirmed in CI (py3.13 run 32864709722, job 97857382582): test_install_and_help and test_scan_dns ran on gw2, test_scan_web and test_clean_shutdown on gw3. test_scan_web was credited 88.7s while the scan it performs takes 8.25s when run directly against the same two-page local server, and test_install_and_help was credited 29.1s for a bbot -h. The difference on both is a venv rebuild. Pinning the class to one xdist group keeps the module together so the venv is built once. Proved by counting the fixture's tmp dirs after a local -n 4 --dist loadgroup run: four bbot_e2e_venv0 dirs (popen-gw0 through gw3) before, one (popen-gw0) after. This is fixture locality, not sharding. All four tests still run, on the same worker, with every assertion intact: 4 passed both before and after. * fix(ci): run benchmarks once and fail loudly The benchmark job checked out the base branch, ran the full suite, then checked out the head branch and ran it again. On a fork PR the second checkout could never work: actions/checkout only fetches refs from origin, so the head branch does not exist locally and git exits with "pathspec did not match any file(s) known to git". Every run today burned 14 minutes benchmarking the base branch, crashed before the head branch, and reported success anyway because the step carried continue-on-error: true. The PR comment step is gated on the head repo matching, so nothing surfaced. Twelve consecutive green runs produced no report. Drop the branch comparison. The script now benchmarks the checked-out tree once and reports it. No checkouts, so no git state to clean up or restore, and no way to fail on a ref that was never fetched. Make failure visible. run_benchmarks returns False on a non-zero pytest exit instead of accepting whatever pytest-benchmark managed to write, main returns an exit code, and continue-on-error is gone. Verified locally: 31 passed, report written, working tree and branch untouched. A seeded failure run exits non-zero instead of reporting success. * perf(wayback): consume archive cache entries as they are fetched finish() built its fetch list from _archive_cache but never removed the entries it processed. The scan's finish phase is re-entrant by design: base.py re-invokes finish() on every FINISHED event, and finish() itself emits HTTP_RESPONSE events, so it re-triggers itself. Every re-entry refetched the full archive set from scratch. The second pass is pure waste. Its URLs were already emitted, so the bloom filter rejects them as duplicates and nothing new reaches the scan. But the fetch still happens first, and against 429 or error responses it pays the whole retry budget again: three per-request retries plus a 30s batch retry backoff, for URLs already fetched successfully. TestWaybackArchive429Retry made this visible because it exercises the retry path. The test drops the 429 delay to 0.01s, but the batch retry backoff is a hardcoded 30 * retry_num, so the redundant pass burned 30s that no test knob could reach. Delete each entry as it is consumed. Unpaired entries (a string, not yet matched to a parent event) are left in place, since a later parent event can still pair them. Verified: finish() archive passes across the file drop from 12 to 6, one per archive-enabled test. The "Archive loading complete" lines lose exactly the failing duplicates (5x "0/1" and 1x "0/2") and keep every successful pass. 429 handling, retry, and emission each still run once. TestWaybackArchive429Retry 41.26s -> 2.93s, whole file 78.63s -> 38.45s. wayback + unarchive + excavate: 73 passed. * fix(scanner): hard-fail modules whose setup() raises A module whose setup() raised was reported as "Setup succeeded" and left in the scan in a permanently errored state. This hung CI: the py3.11 job at 3aaf24d69 ran 37 minutes and was only stopped by auto-cancel, and py3.13 at 9bd71773a sat over two hours. Both stalled on TestNucleiRetries. _setup() initializes status=False, then overwrites it per function in the funcs loop. When setup() raises, the except branch sets the message but never resets status, so it keeps the value left by the preceding setup_deps() call, which is True. setup_modules() then reads status is True, logs "Setup succeeded for excavate (dictionary changed size during iteration)", and skips the removal at scanner.py:635. The errored module stays registered, and since set_error_state() makes it refuse new events, the scan never reaches modules_finished. The debug artifact shows excavate(3:0:0) held 3 events with "Modules errored: 1 (excavate)" repeating every 15s until the runner died. Nothing interrupts this: the suite runs with timeout_func_only=true, so the 1200s timeout covers only the test function, not fixture setup where the scan is driven from. A fixture hang runs forever instead of failing. Set status = False in the except branch so a raising setup() is treated as the hard failure it is. The trigger was a second defect in excavate.setup(), fixed here too. It iterated self.scan.modules directly, twice. setup_modules() runs every module's _setup() concurrently under as_completed and pops failed modules from that same dict, so excavate could be walking it as an entry was removed, raising "dictionary changed size during iteration". Snapshot the values once up front. Verified: - Reproduced the status bug in isolation: raising setup() returned (True, "dictionary changed size during iteration"), matching the CI log line exactly. Now returns False with errored=True. - Negative control on the stashed pre-fix tree returns True, confirming the one-line change is load-bearing. - module.name is provably identical to the dict key it replaces: 15 modules in a real scan, zero mismatches, identical WEB_PARAMETER list both ways. - test_modules_basic 7 passed, test_module_excavate 52 passed, paramminer getparams/headers/cookies 17 passed, test_scan 13 passed. - ruff check and ruff format --check clean on both files. * perf(deps): let retry_failed use the lock-free precheck The install fast path added in b9bdb4e25 was dead for the entire test suite. bbot/test/test.conf sets deps.behavior: retry_failed, and _all_deps_satisfied() bailed out immediately for that mode, so every scan fell through to the exclusive flock on install.lock. Under xdist that serializes all four workers behind whichever one is installing. test_cli_args runs --install-all-deps and holds the lock for ~150s. Any worker calling _prep() in that window blocked for the remainder, and the resulting stall was charged to whichever innocent test happened to be running. That is why the number two slot moved between jobs of the same run: test_dns_graph_structure 107.9s on 3.12, test_event_discovery_context 98.8s on 3.10 and 100.4s on 3.11, test_event_web_spider_distance 123.0s on 3.14. All four overlap test_cli_args at 97 to 100 percent, and all run in about 1s alone. The GitHub log timestamps make this look like idle workers, but the pytest_debug.log artifact shows the victims logging "Executing scan._prep()" and then nothing until the CLI install writes "Successfully installed dependencies for 142 modules", after which they resume within the same second. retry_failed differs from the default only in that it revisits modules recorded as False. The precheck loop already declines on any status that is not True, so it cannot skip a retry. force_install still declines unconditionally. Verified: with the lock held, _prep() under retry_failed goes 15.56s to 0.61s. A module marked False still declines the fast path and gets retried, force_install still declines, and the fast path agrees with _install() on all 149 modules, 0 disagreements. test_depsinstaller fails locally on a clean tree (ansible/sudo), and is unaffected by this change. * perf(webbrute): memoize the fuzz baseline per shape `baseline_fuzz()` built a fresh `HttpCompare` and probed the target on every call, including for a shape it had already characterized. A baseline describes the response to a nonexistent path of a given shape, so it is a pure function of (url, prefix, suffix, ext). Re-probing an identical shape costs two HTTP requests plus the 0.5s inter-sample sleep in `HttpCompare._baseline()` and returns the same filter. webbrute_shortnames fans out hard on this. Each URL_HINT triggers a run for the plain hint, then again for the detected delimiter, then again for the detected subword, then once more per common prefix in `finish()` -- each against the same host with the same extension set. Profiling test_module_webbrute_shortnames showed 42 `baseline_fuzz()` calls collapsing to only 16 distinct shapes, so 50 baselines were established where 24 requests' worth of work was needed. Baseline time dominated the test: 28.3s of a 32s run. Shapes are now memoized per module instance behind a per-key lock, so concurrent handlers racing on a cold shape collapse onto one probe instead of each running their own. Only successful baselines are cached; an abort verdict (CONNECTIVITY_ISSUES, WAF_BLOCK_PAGE, RECEIVED_429) is returned but never stored, so a transient failure cannot poison later events, and host-level aborts still short-circuit the remaining extensions exactly as before. The cache is an LRU bounded at 1000 entries because each entry pins an HttpCompare and its baseline snapshot, and a real scan spans many hosts. test_module_webbrute_shortnames: 33s -> 27s, baselines 50 -> 24. Verified: - Cache invariants asserted directly: identical shape reuses the same HttpCompare object, four distinct shapes take four probes, 10 concurrent callers on a cold shape yield exactly 1 probe, LRU holds at the bound across 50 hosts, and both abort paths stay uncached while still propagating host_abort to every remaining ext. - NEGATIVE CONTROL on the stashed pre-fix tree: the reuse assertion fails with 2 probes for one shape. Load-bearing. - webbrute 11, webbrute_shortnames 1, iis_shortnames 2, test_web, paramminer getparams/headers: 50 passed, 0 failed. Every false-positive defense test (canary, mid-scan drift, hit cap, WAF, redirect, wildcard skip) still passes. - ruff check + ruff format --check clean. * perf(trufflehog): stop the binary re-execing itself on every invocation Every trufflehog invocation forked a second copy of itself and paid the Go package init twice. The re-exec comes from the overseer self-update library: main() hands control to overseer.RunErr(), which starts the real program as a child process with OVERSEER_IS_SLAVE=1 in its environment. Both parent and child run the full init chain, and that chain is not cheap: 1616 packages, dominated by go-re2 at 488ms compiling the detector regex set. Measured on the pinned 3.97.0 binary: a stdin scan of a single line takes 4.6s wall, of which the scan itself is 1.8ms per trufflehog's own "finished scanning" log. GODEBUG=inittrace=1 shows 3232 init lines for a real run against 1615 for an invalid flag, exactly 1616 packages initialized twice, and ps shows the second pid appearing 2.5s in. trufflehog exposes --local-dev for precisely this, it skips overseer and runs the program directly. --no-update already told it not to fetch updates, so the overseer wrapper was pure overhead in every bbot code path. bbot spawns one process per event, so the cost is paid per CODE_REPOSITORY, FILESYSTEM, HTTP_RESPONSE and RAW_TEXT event, not once per scan. Verified: - Output equivalence across every source mode bbot uses (stdin, filesystem, git, plus aws/slack/http secret fixtures): DetectorName, DecoderName, Raw, RawV2, Verified and SourceMetadata identical, same returncode, same result counts. - The one field that differs under git, repository_local_path, embeds the pid of the scanning process and already differs between two consecutive plain runs. Nothing in bbot reads it. - Binary is not modified and no update is attempted, checked with and without --no-update; mtime and size unchanged, zero update-related stderr. - Per invocation 4.6s -> 2.4s, a 1.95x speedup measured over 6 source modes. - In the test suite, subprocess time 103.2s -> 59.7s over 20 invocations. - NEGATIVE CONTROL, same file with the change stashed: 63.0s vs 42.7s, 5 passed both ways. This is the proof the change is load-bearing. - test_module_trufflehog, test_module_badsecrets, test_module_web_report, test_module_github_codesearch, test_modules_basic all pass. - ruff check + ruff format --check clean. This is product code, so real scans get the same 2x on every trufflehog call, not just CI. * perf(test): stop test_async_helpers idling for 25 seconds as_completed defaults to max_concurrent=20, so the wall time of this test was exactly sum(sleeps)/20. With 1000 sleeps drawn from random.random() that sum is ~496s, giving a measured 25.3s of pure idling. Nothing was being exercised during it: the coroutines only slept. Divide the sleeps by 100. The scheduling behavior under test is unchanged, only the constant shrinks. Verified across divisors 1, 20, 50, 100 and 200 that completion stays out of submission order and all 1000 results still come back distinct, so scaling does not collapse the test into a trivially ordered drain. While here, cover the branches this test was walking past. It only ever checked the happy path at the default limit, leaving three untested: - max_concurrent is now asserted to be saturated, not merely respected. Peak in-flight count is exactly the limit, confirmed stable at 1, 5, 20 and 50 over 25 runs each. - max_concurrent=None (the unlimited path) was never entered. - a raising coroutine is yielded rather than swallowed, and does not abort the remaining tasks. as_completed has explicit exception handling that nothing exercised. Also assert completion order differs from submission order, which is the actual contract and was previously unchecked because results went straight into a set. Both new assertion groups verified load-bearing by mutation: forcing the limit to 3 fails on peak == limit, and swallowing raised tasks fails on len(yielded) == 4. test_async_helpers 25.33s -> 1.82s. Full file 20 passed, ~45s -> 21.4s. * fix(test): stop websocket test wedging the py3.13 job for 6 hours The py3.13 job on PR #3404 (job 98941202149) ran 6h0m18s and was killed by the workflow limit. The scan itself finished in under a second: the debug artifact shows "Finished websocket module test" and "No unfinished tasks detected", then "Cancelling 2 orphaned tasks after websocket" and nothing further. Three of four xdist workers went idle and the run never completed. The test served on a task and cancelled it in check(). Server.close() does not close synchronously, it spawns an internal _close() task which is what eventually resolves closed_waiter. The fixture teardown in base.py cancels every remaining task and gathers them, so it cancelled that internal task mid-flight. wait_closed() awaits closed_waiter under asyncio.shield, so once its resolver is cancelled the shield swallows the cancellation and the await can never complete. The gather then blocks forever. Reproduced deterministically outside pytest, 5/5 runs wedged with the server task cancelled while a connection was open, matching the 2 orphaned tasks the CI artifact reports (Server._close and the server coroutine). Fixed by awaiting serve() directly instead of wrapping it in a task, and shutting the server down through close() plus wait_closed(). That drains the internal task before teardown runs, so no orphan is left to cancel. Verified zero orphaned tasks remain after the test, where previously there were two. Shutdown runs in _execute_scan rather than check() because the fixture and the test body run on different event loops, and awaiting the server from check() raises "attached to a different loop". All assertions unchanged. * fix(test): bound the orphan-task cancel in module teardown The websocket hang (2f4032862) was one instance of a general defect. Fixture teardown cancels every leftover task and then does an unbounded `asyncio.gather` on them. A task that absorbs its cancellation, an await shielded from it whose resolver was itself cancelled first, never completes, so that gather blocks forever. Nothing caps it. The workflow passes `--timeout 1200` together with `-o timeout_func_only=true`, and per pytest-timeout that setting evaluates the timeout against the test function body only, ignoring fixture time. Teardown is therefore uncovered, which is how a 20 minute cap silently became the 6 hour job limit on job 98941202149. Wait with a timeout instead, and warn with the surviving tasks when they outlast it. Abandoning a stuck task is strictly better than wedging the worker: the process is torn down at session end regardless. Verified with a module test that leaves a deliberately wedged orphan. On the current tree it passes in 15.7s; with this change stashed the same test returns no result at all and is killed at 240s despite `--timeout 150`, confirming both the hang and that the func-only timeout cannot catch it. No assertions changed, nothing split. Regression: 57 passed under `-n 2`. * revert(ci): restore benchmark.yml to unblock the push The benchmark.yml change from 3aa43a7f8 cannot be pushed: the available OAuth token lacks the `workflow` scope, so the remote rejects any push whose net diff touches a workflow file. That single mid-stack commit was gating 11 unpushed commits, two of which cure a 6 hour py3.13 hang. GitHub evaluates the net diff of the push rather than each commit, so restoring the file forward puts the workflow back to its origin state and lets the rest of the stack land. No rebase, no history rewrite. Reapply once the token has `workflow` scope: gh auth refresh -h github.com -s workflow git revert 37161606a * perf(test): drop zookeeper from the kafka test, use a KRaft broker test_module_kafka was the #2 slot in the suite at 101.8s, but the local warm run is 13.2s. The gap is image pull: CI has no pre-pull step, so docker run pulls implicitly on every run, and the test started two containers totalling ~785MB (wurstmeister/kafka 468MB plus zookeeper:3.9 317MB). Kafka has not needed zookeeper since KRaft went GA. One apache/kafka-native:4.1.2 broker at ~150MB replaces both, so the cold pull drops by ~635MB and the second container bring-up disappears entirely. Readiness was also wrong. wait_for_port_open plus a fixed sleep(1) only proves docker bound the port, which it does at container create time while the broker is still booting; the fixed sleep was papering over that race. Now it probes an actual produce round-trip via the existing wait_for_container helper, on a separate bbot_readiness topic so the topic under assertion is untouched. Measured, same machine, images removed first so both paths pull cold: before 207.7s -> after 57.5s (3.6x) Warm: 13.2s -> 4.3s. check() and its assertion are byte identical. Verified the failure path stays bounded: against a dead broker it raises in 10.1s rather than hanging. * fix(test): stop kafka readiness probe importing an uninstalled dep 38a82bfea replaced the port wait with an aiokafka produce round-trip to close a real readiness race. That probe lives in setup_before_prep, which runs BEFORE scan._prep(), and _prep() is what pip-installs a module's deps_pip. On CI aiokafka is therefore not importable yet, so every one of the 5 python jobs errored with ModuleNotFoundError at collection of TestKafka. It passed locally only because the dep was already in the venv. Go back to wait_for_port_open(9092). That is sufficient here: is_port_open only reports ready once a connection survives its settle window, and the KRaft broker opens its listener as the last step of boot, so the docker proxy race that motivated the produce probe is already covered. Measured against the real broker, probing at the instant each method reports ready and immediately attempting a real produce: port_open 1.39s, api_versions 0.87s, metadata 0.94s, all three followed by PRODUCE OK. Held 0/5 failures with the port probe under 2x-cpu load. The consumer in check() still imports aiokafka, which is correct, that runs after _prep(). Verified with a scoped control that blocks aiokafka for exactly the setup_before_prep phase, reproducing the CI condition: pre-fix tree fails with the identical ModuleNotFoundError, post-fix tree succeeds in 2.41s. Test passes 3/3 consecutively at ~5.7s. No assertions touched. * fix(ci): let pytest-timeout cover fixture setup and teardown timeout_func_only=true scopes the 1200s timeout to the test body only, so time spent in fixture setup and teardown is never counted. That is exactly where this suite's worst wedges have happened: the websocket hang (2f4032862) burned 6h0m18s and was killed by the workflow limit, not by pytest, because it wedged in fixture teardown cancelling orphaned tasks. The orphan-cancel bound (52e41fa65) fixed that specific hang, but the blind spot itself is still open, and every module test does its real work in the module_test fixture rather than in the body. Measured the semantics directly rather than trusting the flag name. With timeout_func_only=true and --timeout 3: an 8s sleep in fixture setup passes, an 8s sleep in teardown passes, only a body sleep is caught. With timeout_func_only=false all three are caught, setup and teardown as ERROR. The timeout also becomes cumulative across phases, so the budget has to clear the slowest whole test, not just its body. Confirmed with a 2s setup plus 2s body against --timeout 3: passes under the old setting, trips under the new one. Worst real test across all five green py jobs is test_cli_args at 150.9s to 189.6s wall including setup, against a 1200s budget, so the margin is 6.3x. Nothing is near the line. Verified: bbot/test/test_step_1/ under the new setting, 298 passed, zero timeouts. badsecrets plus webbrute_shortnames plus modules_basic, 11 passed. ruff check and ruff format clean. No test logic touched, no assertions changed, nothing split. * perf(scanner): stop compiling throwaway regexes for the YARA dns rules dns_regexes_yara compiled a Python regex object per target and then only ever read .pattern back off it. The compiled objects were never matched against anything: their single consumer, dns_yara_rules_uncompiled, splices the source string into YARA rule text, and YARA does its own compilation. So every target paid a full regex compile whose only product was the string that went into it. That is invisible on a normal scan and brutal on a big one. Profiled test_huge_target_list (10,005 targets) under cProfile: scan._prep spent 16.3s in excavate setup, 16.1s of that in _generate_dns_regexes, and 12.4s of that inside regex._compile for 10,032 patterns that were immediately discarded. Split the generation in two. _generate_dns_regex_patterns builds the source strings, _generate_dns_regexes compiles them for dns_regexes (which really does match, via oauth.py and helpers.re.findall_multi), and the YARA property now takes the uncompiled strings. Equivalence is exact, not approximate: the old path emitted re.compile(p, re.I).pattern and the new one emits p. Verified those are byte-identical across 10,013 targets including punycode, underscores, dashes, mixed case, IPs and multi-label public suffixes. Rule text for the single, multi and huge target shapes is unchanged, and dns_regexes still yields compiled objects (10,005 of them, all with .finditer). Note the rule-dict hash is …
Harden preset and custom module loading
Automated Docs Update
This was referenced Oct 30, 2026
Add 3.1 breaking changes doc
…boundaries Stop extracting hostnames from partway through a hyphenated label
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Release 3.1.0
Depends on the stack #3457 (blastdns 2.0) → #3474 (Python range) → #3583 (version bump) landing in dev first. These notes cover everything since 3.0.2, including that stack.
Blockers
Python 3.15 support (#3474) is waiting on two upstream dependencies:
cachebox>=5.2,<6. cachebox 5.x is built on PyO3 0.27, which caps at Python 3.14, and has no 3.15 wheels, so the source build fails (this is the currenttest (3.15)failure on Drop Python 3.10, add Python 3.15 #3474). cachebox 6.2.8 ships 3.15 wheels; deepdiff needs a release that allows it.AttributeError: module 'dataclasses' has no attribute '_is_type'. 2.22.0rc1 declares 3.15 and works (ansible localhost -m pingsucceeds), so this clears with the 2.22.0 final. 2.22 requires Python >=3.13, so 3.11 and 3.12 stay on older ansible-core.Breaking changes
requires-pythonis now>=3.11,<3.16; CI tests 3.11 through 3.15. 3.10 reaches EOL this month.portfiltermodule removed (Filter CDN/WAF ports by default and remove the portfilter module #3534). CDN/WAF port filtering now happens in core and is on by default; opt out with--allow-cdn-ports.dns.nameserversnow aborts the scan (Replace massdns with blastdns 2.0 #3457). The DNS helper is built during scan prep, so a typo fails fast instead of producing a "completed" scan with zero results.Highlights
DNS (#3457)
--custom-nameserversflag, replacing the system resolvers for normal resolution.dns.threads.is_wildcard(Fix nested wildcard detection in is_wildcard #3536).baddns 2.5.0 (#3457)
DELEGATIONsubmodule, enabled by default and in thebaddns/baddns-heavypresets. Checks_acme-challenge,_dmarcand 15 common DKIM selectors for dangling delegations, with wildcard and generic-finding filtering to keep volume down.NSgained a lame-delegation finding (ceiling raised to HIGH/CONFIRMED).WordMatcheris passed through, matching all signatures in a single pass.blastdnscapped at<2.1to match baddns's own ceiling.New modules
favicon: technology detection from favicon hashes (Add favicon module for technology detection #3532).js_unpacker: detects and unpacks obfuscated JavaScript (Add js_unpacker module for detecting and unpacking obfuscated JS #3335).nowafpls: Cloudflare WAF bypass detector (Add nowafpls Cloudflare WAF bypass detector #3336).otilabs: subdomain enumeration (Add otilabs subdomain module #3496).Modules
/path, which crt.sh requires (crt: request an explicit / path (blasthttp sends an empty path, which crt.sh rejects with 400) #3428).excavate
<base href>(Resolve excavated relative URLs against <base href> #3522).Core
api_requestrate limiting, plus config deprecation support viaField(deprecated=True)(Overhaul api_request rate limiting and add config deprecation support #3461).affiliate(Don't tag in-scope events as affiliate #3524).HTTP_RESPONSEevents carryurl_extension(Set url_extension on HTTP_RESPONSE events #3406).ClosestHostEventkeeps its inherited port, and the port is restored from JSON (Keep inherited port on ClosestHostEvent and restore port from JSON #3462).download()respectscache_hrsin its freshness check (Respectcache_hrsindownload()freshness check #3563).Target.add()(fix TypeError when a hostless target is sorted against a /32 #3391).Performance
compute_simhash, with corrected executor placements (Speed up compute_simhash and correct executor placements #3331).response_to_jsonremoved in favor ofresponse_to_event_dict(Delete duplicate response_to_json, unify on response_to_event_dict #3473).Dependencies
Maintenance
Refs #3555