Skip to content

Add long redirection response detection to excavate - #3481

Merged
liquidsec merged 5 commits into
devfrom
long-redirection-response
Oct 7, 2026
Merged

liquidsec merged 5 commits into
devfrom
long-redirection-response

Conversation

@liquidsec

@liquidsec liquidsec commented Oct 1, 2026 •

Copy link
Copy Markdown
Contributor

Closes #3416

Detects a redirect whose body keeps going after the HTML document ends. Browsers discard the body of a 3xx, so content delivered there is never displayed. A server that writes a redirect document and then keeps writing has run the handler the redirect was meant to skip (CWE-698, Execution After Redirect).

A 3xx with a Location header is reported at LOW severity, HIGH confidence, when at least long_redirect_threshold bytes (default 2048, 0 disables) follow </html>.

Why not body size

Size was the original signal and it doesn't hold up. Measured against 30,894 responses from in-scope assets of public bug bounty programs, 14,874 of them 3xx with a Location, across 7,654 hosts:

gate hosts flagged genuine
body >= 2048 66 8
trailing content >= 2048 6 6

What size catches is inlined RUM agents, SPA bundles served on canonical 307/308s, and platform error pages. All large, none of them withheld resources. Trailing content separates cleanly: where a handler kept running, the appended page is 23 KB to 352 KB, and ordinary redirect bodies trail 0 to 61 bytes of whitespace. Nothing lands in between.

Trailing content does miss two of the eight. Both are single-document cases, where the redirect is set as a header and the page renders normally, leaving nothing after </html> to measure. That is a different defect shape, better served by its own check than by loosening this one, which would cost 58 false positives to recover 2 findings.

The check measures the decoded body. content_length reports the compressed size and was missing or unusable on two thirds of the candidates.

Also

produced_events gains FINDING, which excavate rules have always emitted via report() but which was never declared.

@codecov

codecov Bot commented Oct 1, 2026 •

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 91%. Comparing base (7aeee20) to head (fa16e79).

Additional details and impacted files
@@          Coverage Diff          @@
##             dev   #3481   +/-   ##
=====================================
+ Coverage     91%     91%   +1%     
=====================================
  Files        461     461           
  Lines      49520   49580   +60     
=====================================
+ Hits       44655   44710   +55     
- Misses      4865    4870    +5     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@github-actions

github-actions Bot commented Oct 1, 2026 •

Copy link
Copy Markdown
Contributor

📊 Performance Benchmark Report

Comparing dev (baseline) vs long-redirection-response (current)

📈 Detailed Results (All Benchmarks)

📋 Complete results for all benchmarks - includes both significant and insignificant changes

🧪 Test Name 📏 Base 📏 Current 📈 Change 🎯 Status
Bloom Filter Dns Mutation Tracking Performance 2.16ms 2.14ms -0.7% ⚪ ✅
Bloom Filter Large Scale Dns Brute Force 8.14ms 8.18ms +0.5% ⚪ ✅
Large Closest Match Lookup 179.83ms 190.81ms +6.1% ⚪ ✅
Realistic Closest Match Workload 100.65ms 101.22ms +0.6% ⚪ ✅
Event Memory Medium Scan 1402 B/event 1405 B/event +0.3% ⚪ ✅
Event Memory Large Scan 1527 B/event 1527 B/event +0.0% ⚪ ✅
Event Validation Full Scan Startup Small Batch 220.72ms 220.75ms +0.0% ⚪ ✅
Event Validation Full Scan Startup Large Batch 284.63ms 285.17ms +0.2% ⚪ ✅
Make Event Autodetection Small 12.02ms 12.35ms +2.8% ⚪ ✅
Make Event Autodetection Large 117.50ms 122.75ms +4.5% ⚪ ✅
Make Event Explicit Types 5.09ms 5.23ms +2.8% ⚪ ✅
Excavate Single Thread Small 2.466s 2.385s -3.3% ⚪ ✅
Excavate Single Thread Large 5.617s 5.491s -2.2% ⚪ ✅
Excavate Parallel Tasks Small 2.488s 2.395s -3.8% ⚪ ✅
Excavate Parallel Tasks Large 3.705s 3.757s +1.4% ⚪ ✅
Intercept Throughput Small 650.99ms 651.48ms +0.1% ⚪ ✅
Intercept Throughput Medium 645.98ms 643.65ms -0.4% ⚪ ✅
Dns Throughput Quiet 1.405s 1.364s -2.9% ⚪ ✅
Dns Throughput Loaded 915.62ms 930.50ms +1.6% ⚪ ✅
Dns Throughput Inherited 1.263s 1.229s -2.7% ⚪ ✅
Is Ip Performance 1.29ms 1.27ms -0.9% ⚪ ✅
Make Ip Type Performance 108.82µs 114.08µs +4.8% ⚪ ✅
Mixed Ip Operations 1.33ms 1.34ms +0.5% ⚪ ✅
Memory Use Web Crawl 1.2 MB 1.2 MB +0.0% ⚪ ✅
Memory Use Subdomain Enum 28.3 MB 28.3 MB +0.0% ⚪ ✅
Memory Use Deep Chain 5.0 MB 5.0 MB +0.0% ⚪ ✅
Memory Use Parallel Chains 12.6 MB 11.1 MB -11.7% 🟢🟢 🚀
Scan Throughput 100 1.565s 1.612s +3.0% ⚪ ✅
Scan Throughput 1000 9.835s 10.051s +2.2% ⚪ ✅
Typical Queue Shuffle 2.94µs 2.92µs -0.7% ⚪ ✅
Priority Queue Shuffle 14.38µs 14.53µs +1.1% ⚪ ✅

🎯 Performance Summary

+ 1 improvement 🚀
  30 unchanged ✅

🔍 Significant Changes (>10%)

  • Memory Use Parallel Chains: 11.7% 🚀 less memory

🐍 Python Version 3.11.16

Collapse the guard chain onto event.http_status, which already handles a
missing or non-integer status code.

Require the body to be navigable or interactive: more than 10 distinct
links, or more than 2 inputs. Across 30,894 real responses (14,874 3xx with
a Location, 7,654 hosts), size alone flags 66 hosts of which 8 are genuine.
The shape gate flags 9, keeping all 8.

Report the link and input counts in the description so triage can see which
arm fired.
Report a 3xx whose body continues for at least the threshold past the end of
the HTML document. That is the server running the handler the redirect was
meant to skip, rather than any body that happens to be large.

Body size was the wrong signal. Across 14,874 real 3xx responses it flagged
66 hosts, of which 8 carried anything withheld; the rest were inlined
analytics agents, framework payloads and error pages. Trailing content flags
6 hosts with no false positives.

Raise severity to LOW now that the check identifies a specific defect.
@liquidsec
liquidsec requested a review from singlerider October 6, 2026 22:01
@liquidsec
liquidsec marked this pull request as ready for review October 6, 2026 22:01
@liquidsec liquidsec linked an issue Oct 7, 2026 that may be closed by this pull request
@liquidsec
liquidsec merged commit 3be1b99 into dev Oct 7, 2026
19 checks passed
liquidsec added a commit that referenced this pull request Oct 7, 2026
Two conflicts, both from each side appending tests at EOF. Kept both sides.

bbot/test/test_step_1/test_depsinstaller.py: dev added
test_sql_modules_request_sqlalchemy_asyncio_extra, this branch added the five
installer review-fix tests.

bbot/test/test_step_2/module_tests/test_module_excavate.py: dev added
TestExcavateLongRedirect and TestExcavateLongRedirectDisabled (#3481), this
branch added test_excavate_find_subclasses_does_not_touch_descriptors.
@liquidsec liquidsec mentioned this pull request Oct 26, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

New Module: "Long" Redirect Detection

2 participants