[liquidsec]

Excavate resolved relative references against the response URL and ignored <base href>. On a page with <base href="/"> and a link like app/page, every copy of the page at a nested path produced a deeper URL (/app/sub/app/page). On servers that return the same page for any path, this became an unbounded chain of nested URLs, each one crawled, screenshotted and scanned.

Excavate now resolves relative URLs against the page's <base href> when present, for both parameter extraction and URL extraction. Forms with no action still submit to the page URL, per the HTML spec. Location headers and cookies are unchanged.

[github-actions]

📊 Performance Benchmark Report

Comparing dev (baseline) vs excavate-base-href (current)

📈 Detailed Results (All Benchmarks)

📋 Complete results for all benchmarks - includes both significant and insignificant changes

🧪 Test Name	📏 Base	📏 Current	📈 Change	🎯 Status
Bloom Filter Dns Mutation Tracking Performance	3.51ms	3.16ms	-9.8% ⚪	✅
Bloom Filter Large Scale Dns Brute Force	15.56ms	14.07ms	-9.6% ⚪	✅
Large Closest Match Lookup	324.03ms	338.87ms	+4.6% ⚪	✅
Realistic Closest Match Workload	179.31ms	180.95ms	+0.9% ⚪	✅
Event Memory Medium Scan	1402 B/event	1402 B/event	+0.0% ⚪	✅
Event Memory Large Scan	1527 B/event	1527 B/event	+0.0% ⚪	✅
Event Validation Full Scan Startup Small Batch	396.49ms	394.07ms	-0.6% ⚪	✅
Event Validation Full Scan Startup Large Batch	522.25ms	514.95ms	-1.4% ⚪	✅
Make Event Autodetection Small	22.48ms	22.05ms	-1.9% ⚪	✅
Make Event Autodetection Large	226.94ms	226.71ms	-0.1% ⚪	✅
Make Event Explicit Types	9.68ms	9.60ms	-0.8% ⚪	✅
Excavate Single Thread Small	3.932s	3.801s	-3.3% ⚪	✅
Excavate Single Thread Large	10.033s	10.239s	+2.1% ⚪	✅
Excavate Parallel Tasks Small	4.021s	3.869s	-3.8% ⚪	✅
Excavate Parallel Tasks Large	6.471s	6.529s	+0.9% ⚪	✅
Intercept Throughput Small	871.64ms	869.65ms	-0.2% ⚪	✅
Intercept Throughput Medium	860.10ms	918.04ms	+6.7% ⚪	✅
Dns Throughput Quiet	2.364s	2.354s	-0.4% ⚪	✅
Dns Throughput Loaded	1.623s	1.718s	+5.9% ⚪	✅
Dns Throughput Inherited	2.144s	2.176s	+1.5% ⚪	✅
Is Ip Performance	2.06ms	2.07ms	+0.4% ⚪	✅
Make Ip Type Performance	183.61µs	185.19µs	+0.9% ⚪	✅
Mixed Ip Operations	2.13ms	2.12ms	-0.3% ⚪	✅
Memory Use Web Crawl	1.2 MB	1.2 MB	+0.9% ⚪	✅
Memory Use Subdomain Enum	28.3 MB	28.4 MB	+0.1% ⚪	✅
Memory Use Deep Chain	5.0 MB	5.0 MB	+0.0% ⚪	✅
Memory Use Parallel Chains	13.4 MB	13.5 MB	+0.4% ⚪	✅
Scan Throughput 100	2.564s	2.543s	-0.8% ⚪	✅
Scan Throughput 1000	17.595s	17.554s	-0.2% ⚪	✅
Typical Queue Shuffle	4.97µs	5.08µs	+2.2% ⚪	✅
Priority Queue Shuffle	24.04µs	23.88µs	-0.7% ⚪	✅

🎯 Performance Summary

✅ No significant performance changes detected (all changes <10%)

---

🐍 Python Version 3.11.17

[codecov]

Codecov Report

❌ Patch coverage is 98.61111% with 1 line in your changes missing coverage. Please review.
✅ Project coverage is 91%. Comparing base (ce6decf) to head (7c43c55).
⚠️ Report is 3 commits behind head on dev.

Files with missing lines	Patch %	Lines
bbot/modules/internal/excavate.py	96%	1 Missing ⚠️

Additional details and impacted files

@@          Coverage Diff          @@
##             dev   #3522   +/-   ##
=====================================
+ Coverage     91%     91%   +1%     
=====================================
  Files        461     461           
  Lines      49625   49723   +98     
=====================================
+ Hits       44769   44862   +93     
- Misses      4856    4861    +5     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[en0f]

ParameterExtractor.process crashes on RAW_TEXT events (bbot/modules/internal/excavate.py, around lines 955–959)

What changed: excavate watches both HTTP_RESPONSE and RAW_TEXT. The PR moved the URL setup to the top of process(), so it now runs every time. That setup reads event.parsed_url (through _event_base_url and event.parsed_url.query), but RAW_TEXT events don’t have that attribute.
What happens: any RAW_TEXT event that matches a parameter rule now throws an AttributeError before a single parameter is handled. Before this PR, that code only ran when the form action was missing or relative, so parameters with full URLs in extracted text were still emitted.
Damage: the error escapes search() and handle_event, so excavate stops processing that whole event. The parameters are lost, and so is anything the later rules would have found in the same text (URLs, emails, DNS names). Text extracted from archived HTML or JS files would hit this.
Why CI missed it: no test sends a RAW_TEXT event containing form or link HTML.
Fix: guard it the way URLExtractor already does with hasattr(event, "parsed_url"), or build document_url/base_url only when it’s needed and the event has a URL.

[liquidsec]

Fixed in 7c43c55. The page and base URLs are now worked out only when a relative endpoint needs them; on events without a URL (RAW_TEXT), relative endpoints are skipped and full-URL parameters are emitted as before. Added a RAW_TEXT test.