[liquidsec]

Summary

Adds a WAF-bypass technique based on prepending a large junk padding to POST bodies —
inspired by the nowafpls burp extension.
Cloudflare (and several other inline WAFs) has a bounded inspection buffer; once
the request body grows past it, the malicious payload lands after the inspection
window and reaches the app unfiltered. This branch delivers:

• bbot/core/helpers/nowafpls.py — shared helper. is_bypassable(event) probes
  the host once (baseline vs unpadded-malicious vs padded-malicious via
  http_compare), memoized per host. pad_form_body(event, body) and
  pad_json(event, data) are drop-ins for adversarial POST bodies.
• bbot/modules/nowafpls.py — dedicated detector that emits a FINDING when a
  WAF-tagged host is bypassable, with the padding size in the description.
• Lightfuzz integration — tri-state avoid_wafs config (always / never /
  try_bypasses, default try_bypasses). Booleans are still accepted (true ->
  always, false -> never), so configs written against the previous boolean
  option keep working. Submodule prepare_request transparently
  pads POSTPARAM / BODYJSON probes when the host's WAF is bypassable, and flips
  used_nowafpls=True so lightfuzz tags the finding used-nowafpls.
• ajaxpro + generic_ssrf — POST paths now call the padding helpers so their
  fuzz traffic uses the same bypass on WAF-tagged hosts.
• lightfuzz-xss preset — enables try_get_as_post so GET params get re-fuzzed
  as padded POSTs, catching XSS reachable from an attacker-controlled form when the
  WAF blocks the raw payload.

Shared-framework change: _is_http_wildcard_host

BaseModule._is_http_wildcard_host no longer returns True as soon as the host is
flagged a wildcard responder. It now probes the specific URL against the wildcard
baseline and reports a wildcard only when the response matches it, so a URL that
diverges from the baseline is treated as a real endpoint and is no longer skipped.

For WEB_PARAMETER GETPARAMs the probe bakes the parameter into the URL, since a bare
path can be a catchall while ?name=value is a real endpoint.

This is shared framework behavior and affects every module that relies on wildcard-host
filtering, not just lightfuzz. Covered by
test_base_module_is_http_wildcard_host_per_url in bbot/test/test_step_1/test_web.py.

[github-actions]

📊 Performance Benchmark Report

Comparing dev (baseline) vs nowafpls (current)

📈 Detailed Results (All Benchmarks)

📋 Complete results for all benchmarks - includes both significant and insignificant changes

🧪 Test Name	📏 Base	📏 Current	📈 Change	🎯 Status
Bloom Filter Dns Mutation Tracking Performance	2.91ms	2.94ms	+0.7% ⚪	✅
Bloom Filter Large Scale Dns Brute Force	11.65ms	11.91ms	+2.2% ⚪	✅
Large Closest Match Lookup	255.00ms	272.89ms	+7.0% ⚪	✅
Realistic Closest Match Workload	140.76ms	141.61ms	+0.6% ⚪	✅
Event Memory Medium Scan	1402 B/event	1403 B/event	+0.1% ⚪	✅
Event Memory Large Scan	1527 B/event	1527 B/event	+0.0% ⚪	✅
Event Validation Full Scan Startup Small Batch	309.08ms	309.00ms	-0.0% ⚪	✅
Event Validation Full Scan Startup Large Batch	410.74ms	395.34ms	-3.7% ⚪	✅
Make Event Autodetection Small	16.08ms	16.27ms	+1.2% ⚪	✅
Make Event Autodetection Large	162.75ms	162.82ms	+0.0% ⚪	✅
Make Event Explicit Types	6.68ms	6.63ms	-0.8% ⚪	✅
Excavate Single Thread Small	3.003s	2.984s	-0.6% ⚪	✅
Excavate Single Thread Large	7.349s	7.320s	-0.4% ⚪	✅
Excavate Parallel Tasks Small	3.129s	3.093s	-1.1% ⚪	✅
Excavate Parallel Tasks Large	4.775s	4.666s	-2.3% ⚪	✅
Intercept Throughput Small	761.13ms	763.54ms	+0.3% ⚪	✅
Intercept Throughput Medium	750.98ms	752.35ms	+0.2% ⚪	✅
Dns Throughput Quiet	1.922s	1.912s	-0.6% ⚪	✅
Dns Throughput Loaded	1.312s	1.319s	+0.5% ⚪	✅
Dns Throughput Inherited	1.641s	1.765s	+7.6% ⚪	✅
Is Ip Performance	1.77ms	1.77ms	-0.3% ⚪	✅
Make Ip Type Performance	172.66µs	164.29µs	-4.8% ⚪	✅
Mixed Ip Operations	1.86ms	1.85ms	-0.5% ⚪	✅
Memory Use Web Crawl	1.0 MB	1.2 MB	+15.8% 🟡🟡	⚠️
Memory Use Subdomain Enum	28.3 MB	28.3 MB	+0.0% ⚪	✅
Memory Use Deep Chain	5.0 MB	5.0 MB	+0.2% ⚪	✅
Memory Use Parallel Chains	11.4 MB	12.1 MB	+5.9% ⚪	✅
Scan Throughput 100	2.049s	2.143s	+4.6% ⚪	✅
Scan Throughput 1000	13.585s	13.624s	+0.3% ⚪	✅
Typical Queue Shuffle	4.23µs	4.20µs	-0.6% ⚪	✅
Priority Queue Shuffle	21.07µs	20.72µs	-1.6% ⚪	✅

🎯 Performance Summary

! 1 regression ⚠️
  30 unchanged ✅

🔍 Significant Changes (>10%)

• Memory Use Web Crawl: 15.8% 🐌 more memory

---

🐍 Python Version 3.11.16

[codecov]

Codecov Report

❌ Patch coverage is 95.35865% with 33 lines in your changes missing coverage. Please review.
✅ Project coverage is 90%. Comparing base (76d63cb) to head (a9fcb24).
⚠️ Report is 4 commits behind head on dev.

Files with missing lines	Patch %	Lines
bbot/core/helpers/nowafpls.py	91%	10 Missing ⚠️
...t/test_step_2/module_tests/test_module_nowafpls.py	96%	8 Missing ⚠️
bbot/modules/lightfuzz/lightfuzz.py	90%	4 Missing ⚠️
...st_step_2/module_tests/test_module_generic_ssrf.py	94%	4 Missing ⚠️
bbot/core/helpers/web/web.py	50%	2 Missing ⚠️
...st/test_step_2/module_tests/test_module_ajaxpro.py	97%	2 Missing ⚠️
.../test_step_2/module_tests/test_module_lightfuzz.py	99%	2 Missing ⚠️
bbot/modules/nowafpls.py	96%	1 Missing ⚠️

Additional details and impacted files

@@           Coverage Diff           @@
##             dev   #3336     +/-   ##
=======================================
+ Coverage     90%     90%     +1%     
=======================================
  Files        454     457      +3     
  Lines      47093   48115   +1022     
=======================================
+ Hits       42313   43275    +962     
- Misses      4780    4840     +60     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[singlerider]

Tests pass (nowafpls/ajaxpro/generic_ssrf, test_web, and the full lightfuzz suite). The prepare_request/compare_baseline sync-to-async conversion touches every submodule; verified every call site is awaited, no un-awaited coroutines.

Two things to flag before merge:

• Breaking config change: avoid_wafs went from bool to Literal[always,never,try_bypasses]. An existing avoid_wafs: true/false in a user config is now rejected at load time. In-repo presets are migrated but external configs will break. Recommend a coercing validator (True->always, False->never) or a breaking-change changelog note.

• _is_http_wildcard_host in base.py was rewritten to probe the specific URL (baking in GETPARAM values) against the wildcard baseline instead of returning True. It is well tested via the new test_web cases, but it is a shared-framework behavior change with no mention in the PR body. Please call it out.

Nit: is_bypassable memoizes per-host but ignores padding_size/payload on cache hits, so if the standalone module and lightfuzz probe the same host, the first caller's params win.

[liquidsec]

Both addressed.

avoid_wafs now accepts the boolean form (true -> always, false -> never) via a before-validator, so existing configs load unchanged. Worth noting schema validation doesn't rewrite the runtime config, so the module applies the same coercion where it reads the option; the validator alone would have let true load and then behave as never. Both halves are tested in test_validate_preset.py. webbrute.avoid_wafs stays a bool on purpose, we don't want to brute force dirs through a WAF just because nowafpls exists.

Called out the _is_http_wildcard_host change in the PR body. Also merged dev.

[liquidsec]

Three defects were causing WAF-tagged hosts to be skipped entirely, one of which
could also produce a false positive. Fixed in 0d98eb6.

1. no_interference was treated the same as blocked

lightfuzz.filter_event gated on not result.bypassed under
avoid_wafs: try_bypasses. BypassResult.bypassed is only true for
STATUS_BYPASSED, so that single check collapsed three distinct verdicts into
one rejection:

• no_interference - the payload reached the application untouched
• blocked - the gate held and padding did not help
• error - the probe never produced a verdict

Only the last two are reasons to skip. no_interference means there is nothing
to work around, so those events should be fuzzed normally. Since the helper
memoizes per host, the first probe's verdict applied to every subsequent event
on that host, so a host fronted by a CDN that isn't actively blocking received
no fuzzing at all.

This matters because the waf tag is assigned from the CDN/WAF provider's
identity, not from observed blocking. A host can carry the tag while gating
nothing. nowafpls exists to measure what the host actually does, so the
measurement should win over the tag.

2. _post_capable was gating the probe rather than the bypass

The same branch rejected any event that could not fire a POST probe before
consulting nowafpls at all:

if not self._post_capable(event):
    return False, "WAF-tagged event has no POST-style probe to pad"
result = await self.helpers.nowafpls.is_bypassable(event)

_post_capable is false for every URL event, and since try_get_as_post
defaults to false, for every GETPARAM as well. On a tagged host the majority of
discovered parameters were therefore dropped before any measurement happened.
Padding is body-only, so that gate is only meaningful once the verdict is
bypassed; when nothing is gating the payload there is no bypass to apply and
no reason to restrict which event types get fuzzed.

3. HttpCompare.compare() reported a dead request as a baseline match

if subject_response is None:
    return (True, "403", reflection, subject_response)

A timeout or connection reset produced match=True. For header and cookie
fuzzing that is a reasonable conservative default, but two other consumers read
it as a positive verdict:

• nowafpls - a killed unpadded request became "matches baseline", so the
  probe reported no_interference and returned before ever testing padding. A
  WAF that drops the connection instead of serving a block page was invisible.
  The mirror case is worse: a killed padded request became bypassed,
  emitting a CONFIRMED finding on no evidence.
• HTTP wildcard detection (web.py, base.py) - a dead probe became "root
  matches the random-path baseline", so the host was declared a wildcard
  responder. lightfuzz.filter_event checks this before any WAF logic, so a
  single timeout could black out a host completely.

The reasons value on that path was also a bare string while every other path
returns a list, so consumers doing ",".join(reasons) rendered it as 4,0,3.

Changes

• compare() takes none_is_match (default True, preserving existing
  behavior) and returns ["request_failed"] as a list. nowafpls and both
  wildcard probes pass False; the wildcard paths now map a dead probe to
  "unknown" rather than to a verdict.
• filter_event branches on the explicit status and applies _post_capable
  only when the verdict is bypassed.
• Probe verdicts are logged through a shared BypassResult.summary carrying the
  error and diff reasons, reported at INFO. Previously the only report of a
  verdict was a debug line, and nothing was logged at all when lightfuzz
  consulted the helper without the nowafpls module enabled.

Tests

Each new test was confirmed to fail against the pre-fix code:

• Test_Lightfuzz_filter_event_try_bypasses - accept/reject across all four
  statuses for URL, GETPARAM and POSTPARAM events.
• Test_Nowafpls_try_bypasses_no_interference - a host with no interference is
  fuzzed, and unpadded.
• TestNowafplsUnpaddedConnectionKilled - a killed unpadded request with a
  succeeding padded one is a bypass, not silence.
• TestNowafplsPaddedConnectionKilled - a killed padded request is not a bypass.
• test_base_module_is_http_wildcard_host_per_url gains a dead-probe case. Its
  FakeCompare stub previously returned None for the response on every call,
  which only happens for real when the request fails; it now returns a response
  object and models the failure explicitly.

180 tests pass across lightfuzz, nowafpls, test_web, paramminer
(headers/getparams/cookies), bypass403, url_manipulation and webbrute.

[liquidsec]

@singlerider looks like this may have inadvertently also fixed a bug that was in stable currently. So its a high priority now. @shart123456

[singlerider]

Re-reviewed at cb8db4f. Both flagged items resolved, and 0d98eb6 is a real fix on top. Suites pass on the branch: lightfuzz 125, nowafpls + validate_preset 36, test_web + bypass403 + paramminer_headers 29.

🟢 avoid_wafs backward compatibility

RESOLVED ✅. Your point about the validator not rewriting the runtime config is the part I would have missed: a before-validator alone would have let true load and then behave as never, since lightfuzz.setup() reads the raw dict. Coercing in both places is correct, and test_lightfuzz_avoid_wafs_bool_coerced_at_runtime pins the half that actually decides behavior. Confirmed "bogus" and 5 still fail at load with the expected-one-of message.

🟢 _is_http_wildcard_host documented

RESOLVED ✅. The PR body now has its own section calling it out as shared framework behavior with the GETPARAM rationale and the covering test named.

🟢 none_is_match on a dead probe

Not something I caught on the first pass and it is the strongest commit in the branch. compare() returning (True, "403", ...) meant a killed request read as a baseline match, so nowafpls scored a dropped unpadded request as no-interference and a dropped padded one as a bypass, which is exactly backwards for the thing this PR exists to detect. Defaulting to True keeps every other call site unchanged, and the three that care opt out explicitly. Turning reasons into a list also fixes the "4,0,3" render in paramminer's ",".join(reasons).

🟢 _post_capable ordering

Applying it before the probe meant a WAF-tagged host dropped every URL and GETPARAM event regardless of the verdict. Gating it on STATUS_BYPASSED is right, since the padding is what needs a body in the first place.

🔴 Blocking, STATUS_ERROR now silently skips the event

filter_event lumps STATUS_ERROR in with STATUS_BLOCKED and returns a bare False, so a probe that failed for an infrastructure reason (a baseline that could not be established, a transient timeout) is indistinguishable from a WAF that held. The old code at least had one meaning. Either return the string form so the skip is reported, or treat an error as unknown and fall through to fuzzing, but a bare False on an error path loses the event with no record outside debug logging.

[liquidsec]

Addressed in 58bb592.

STATUS_ERROR no longer skips silently

filter_event branches on the two statuses separately now, and both carry a reason:

• blocked -> "WAF blocked the payload and body padding did not help (status=blocked provider=... diff_reasons=...)"
• error -> "WAF bypass probe was inconclusive (status=error provider=... error=could not baseline ...)"

The hand-rolled parsed_url / self.debug block is gone, since _event_postcheck_inner already appends the returned reason to the rejection message.

Two things I decided against, with the reasoning:

An error still skips rather than falling through to fuzzing. STATUS_ERROR can only come from HttpCompareError, which is raised only by HttpCompare._baseline, so the verdict means the host's baseline could not be established. Every lightfuzz submodule baselines through that same code, so falling through would re-fail per submodule per event without producing anything.

Error verdicts stay memoized. Dropping them from the per-host cache would re-probe on every event, and _baseline is two requests at retries=2 / timeout=10 plus a 0.5s sleep, so one unreachable host could stall filter_event for close to a minute per event. The reported reason is what makes the transient case visible instead of silent.

Verdict logging

is_bypassable logs the verdict once per host at verbose, keyed off whether the call created the probe task. Previously a lightfuzz-only run (nowafpls module not enabled) had no record of a verdict above debug, which is the other half of the "no record" problem.

Tests

Test_Lightfuzz_filter_event_try_bypasses now asserts that every rejection carries a reason, that blocked and error produce different reasons, and that the verdict is included in the string. Confirmed failing against the pre-fix code with AssertionError: URL rejection on status=blocked must carry a reason, got False.

Locally: lightfuzz 125 passed, test_web + nowafpls + test_presets 53 passed. Branch CI is still running as of this comment.

[singlerider]

Third pass at e898ab4. CI green.

🟢 STATUS_ERROR no longer skips silently

RESOLVED ✅. Checked your claim rather than taking it: compare() awaits _baseline() at diff.py:280, and HttpCompareError is raised only at diff.py:160 and 165, both inside _baseline. A failed subject request returns none_is_match, it does not raise. So the memoize-and-skip reasoning holds.

Two blockers inline: the padded probe cannot tell a 413 from a WAF block, and the pad helpers ignore the configured padding size.

[singlerider]

🔴 none_is_match=False means any non-match reads as "no bypass", so a server rejecting the oversized body is indistinguishable from a WAF that held:

padded 413  -> status=blocked diff_reasons=code
padded dies -> status=blocked diff_reasons=body

DEFAULT_PADDING_SIZE is 1048576, exactly nginx's default client_max_body_size of 1m. On any nginx-fronted host this draws a 413, concludes blocked, and memoizes it for the scan, so lightfuzz skips every POST event on that host. Needs its own status, or a backoff and re-probe.

[singlerider]

🔴 Hardcodes DEFAULT_PADDING_SIZE, discarding both the padding_size config and the result.padding_size of the probe awaited two lines up. Set padding_size: 2097152 and the probe runs at 2 MiB while ajaxpro and generic_ssrf still pad at 1 MiB. This is the exact knob an operator reaches for after hitting the 413 above.

[singlerider]

🔴 Same hardcode as pad_form_body. The verdict and the exploit disagree whenever padding_size is not the default.

[liquidsec]

Fourth pass pushed. Both inline blockers addressed, plus a fourth hardcode site the review didn't reach.

413 vs WAF block

padding_size is now a descending list (web.nowafpls_padding_sizes, default [1048576, 131072]) instead of a scalar. The probe walks it largest-first and steps down only on an explicit 413. Anything else is the final verdict, because a WAF that held at one size will only hold harder at a smaller one, so there is nothing to gain by continuing. That bounds the extra traffic to one rung in the common nginx case.

If every rung is rejected, the result is STATUS_TOO_LARGE, distinct from STATUS_BLOCKED, and lightfuzz reports it with its own skip reason instead of filing it as a WAF verdict.

A dropped connection deliberately does not advance the ladder. It is ambiguous (the WAF may have killed it), and IIS reports maxAllowedContentLength as a plain 404, so neither can be told from interference without guessing. 413 is the only signal acted on. TestNowafplsPaddedConnectionKilled now asserts the probe stops at the first rung, so that stays pinned.

Hardcoded padding size

Root cause was that padding_size and payload were per-call arguments to a per-host memoized probe, while the config lived on the nowafpls module. ajaxpro, generic_ssrf and lightfuzz reach helpers.nowafpls without access to that config, so they hardcoded. Note modules.nowafpls.padding_size was also unreachable for them by construction: the helper is used whether or not the module is enabled.

Both now live on the helper, sourced once from web.*, and is_bypassable() takes no tuning arguments at all. BypassResult.padding_size reports the size that actually produced the verdict, and pad_form_body / pad_json pad with that. Verdict and exploit can no longer disagree.

This also dissolves the pass-1 nit about first-caller-wins memoization: there are no per-call params left to race over.

Fourth site

lightfuzz/submodules/base.py imported PADDING_FIELD_NAME and DEFAULT_PADDING_SIZE and rebuilt the pad inline rather than calling the helpers. It now calls pad_json, which covers POSTPARAM and BODYJSON both, since prepare_request builds a dict either way.

Tests

• TestNowafplsSizeLadderFallsBack: 413 on the first rung, bypass found on the second, finding reports the size that worked
• TestNowafplsAllSizesTooLarge: every rung 413s, no finding, status is too_large not blocked
• TestNowafplsPaddedConnectionKilled: asserts the ladder does not advance on a dead connection
• ajaxpro and lightfuzz tests now pin the configured size to the pad length actually sent, which is what blockers 2 and 3 describe and what nothing covered before

Suites on the branch: lightfuzz 125, nowafpls + ajaxpro + generic_ssrf + test_web + validate_preset 62, bypass403 + paramminer_headers 11.

[liquidsec]

CI fix: two pre-existing dev failures, unrelated to nowafpls

Chasing the red CI on this branch surfaced two bugs that live on dev itself, not in the nowafpls changes. Both are fixed here so this PR goes green; they'll also fix dev once this merges.

1. telerik tests hardcoded the httpserver port (352114f7)
Three telerik test classes used targets = ["http://127.0.0.1:8888"]. Under pytest -n, each xdist worker binds its httpserver on a per-worker port (8888 + worker_index*100), so on any worker but gw0 the probes hit a dead port, the scan finishes with no findings, and the assertions fail. Which subtests fail depends on worker assignment, so it looked flaky (dev fails 1, this branch failed 3) and never reproduced serially. Switched them to HTTPSERVER_URL, matching every other httpserver test.

2. sql output models rejected naive datetimes under sqlmodel 0.0.46 (a9fcb242)
sqlite/mysql/postgres all failed inserting the scan row with ValueError: Datetime values must have timezone information from sqlmodel/sql/sqltypes.py. sqlmodel 0.0.46's default column type now rejects naive datetimes on bind. Scan.finished_at already declared an explicit naive-tolerant sa_column, but Scan.started_at did not, so it got the aware-enforcing default. Gave started_at the same explicit column. This is why the first CI run here was green and later runs weren't: the sqlmodel bump landed in between.

[singlerider]

Fourth pass at a9fcb24. Pulled the branch, ran the suites locally, CI green across the fleet. Both inline blockers from pass three are resolved, and I checked the fixes against the code rather than the writeup.

🟢 413 no longer scores as a WAF block

RESOLVED ✅. padding_size is a descending ladder now (web.nowafpls_padding_sizes, default [1048576, 131072]), walked largest-first, and it steps down only on an explicit 413 (SIZE_REJECTION_CODES). Any other outcome is the final verdict, which is right: a gate that held at one size holds harder at a smaller one. A fully-rejected ladder returns STATUS_TOO_LARGE, distinct from STATUS_BLOCKED, and lightfuzz.filter_event reports it with its own skip reason. A dropped connection deliberately does not advance the ladder, since it is ambiguous. TestNowafplsSizeLadderFallsBack, TestNowafplsAllSizesTooLarge, and TestNowafplsPaddedConnectionKilled pin all three paths, and I confirmed the padded compare guards padded_response is not None before reading status_code, so a dead probe can't throw there.

🟢 Pad helpers no longer ignore the configured size

RESOLVED ✅. padding_size and payload moved onto the helper, sourced once from web.*, and is_bypassable() takes no tuning args. BypassResult.padding_size reports the size that actually produced the verdict, and pad_form_body / pad_json pad with exactly that. The fourth site in lightfuzz/submodules/base.py that rebuilt the pad inline now calls pad_json too. Verdict and exploit can no longer disagree, and the old first-caller-wins memoization nit dissolves with the per-call params gone. test_lightfuzz_avoid_wafs_bool_coerced_at_runtime and the ajaxpro size assertions cover it.

🟢 prepare_request / compare_baseline sync to async

The two methods went async so the pad probe can be awaited from inside them. I grepped every call site under bbot/modules/lightfuzz/; all are awaited, no stray coroutine. cmdi, crypto, path, serial, sqli, and the three base-class probe helpers all updated.

🟢 is_http_wildcard_host contract change

The helper now returns an HttpCompare on a wildcard hit instead of True, and base.py probes this specific URL (with the GETPARAM baked in) against that baseline before skipping. Only one caller consumes the helper directly and it handles the object, the scalar backward-compat path, and the dead-probe None. Both the standalone test_is_http_wildcard_host and the new test_base_module_is_http_wildcard_host_per_url cover it.

Local validation

• nowafpls + test_web wildcard cases + validate_preset: 40 passed
• lightfuzz + ajaxpro + generic_ssrf: 131 passed
• ruff format --check and ruff check clean on every touched file

The two CI-fix commits (telerik per-worker HTTPSERVER_URL, sql naive started_at column) are unrelated dev breakage and are fine to ride along.

Approving.