[liquidsec]

speculate emits a URL_UNVERIFIED for an HTTP_RESPONSE whose URL isn't already in its history. For lightfuzz baseline responses, that URL came from a link on another page, but the new event started at spider distance 0 with no spider-danger tag, so it was visited regardless of web.spider_distance. Since lightfuzz re-mines its own baselines, URLs could chain well past the configured limits.

speculate now tags these spider-danger, so they cost a hop like any excavated link. The URL is still emitted and still visited when it's within the spider limits.

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 91%. Comparing base (ce6decf) to head (f32193b).

Additional details and impacted files

@@          Coverage Diff          @@
##             dev   #3523   +/-   ##
=====================================
- Coverage     91%     91%   -0%     
=====================================
  Files        461     461           
  Lines      49625   49644   +19     
=====================================
+ Hits       44769   44779   +10     
- Misses      4856    4865    +9     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[github-actions]

📊 Performance Benchmark Report

Comparing dev (baseline) vs lightfuzz-baseline-spider-distance (current)

📈 Detailed Results (All Benchmarks)

📋 Complete results for all benchmarks - includes both significant and insignificant changes

🧪 Test Name	📏 Base	📏 Current	📈 Change	🎯 Status
Bloom Filter Dns Mutation Tracking Performance	3.83ms	3.88ms	+1.4% ⚪	✅
Bloom Filter Large Scale Dns Brute Force	15.52ms	15.79ms	+1.7% ⚪	✅
Large Closest Match Lookup	355.47ms	380.93ms	+7.2% ⚪	✅
Realistic Closest Match Workload	196.22ms	205.31ms	+4.6% ⚪	✅
Event Memory Medium Scan	1402 B/event	1402 B/event	+0.0% ⚪	✅
Event Memory Large Scan	1527 B/event	1527 B/event	+0.0% ⚪	✅
Event Validation Full Scan Startup Small Batch	427.29ms	418.21ms	-2.1% ⚪	✅
Event Validation Full Scan Startup Large Batch	566.03ms	560.37ms	-1.0% ⚪	✅
Make Event Autodetection Small	27.23ms	27.68ms	+1.6% ⚪	✅
Make Event Autodetection Large	276.79ms	278.11ms	+0.5% ⚪	✅
Make Event Explicit Types	11.97ms	12.14ms	+1.4% ⚪	✅
Excavate Single Thread Small	4.763s	4.249s	-10.8% 🟢🟢	🚀
Excavate Single Thread Large	10.291s	9.984s	-3.0% ⚪	✅
Excavate Parallel Tasks Small	6.985s	4.587s	-34.3% 🟢🟢🟢	🚀
Excavate Parallel Tasks Large	6.988s	7.062s	+1.1% ⚪	✅
Intercept Throughput Small	894.31ms	910.34ms	+1.8% ⚪	✅
Intercept Throughput Medium	984.64ms	896.31ms	-9.0% ⚪	✅
Dns Throughput Quiet	2.869s	2.936s	+2.3% ⚪	✅
Dns Throughput Loaded	2.010s	1.963s	-2.3% ⚪	✅
Dns Throughput Inherited	2.475s	2.477s	+0.1% ⚪	✅
Is Ip Performance	2.33ms	2.40ms	+2.9% ⚪	✅
Make Ip Type Performance	250.73µs	241.10µs	-3.8% ⚪	✅
Mixed Ip Operations	2.43ms	2.51ms	+3.5% ⚪	✅
Memory Use Web Crawl	1.2 MB	1.2 MB	+0.0% ⚪	✅
Memory Use Subdomain Enum	28.3 MB	28.3 MB	+0.0% ⚪	✅
Memory Use Deep Chain	5.0 MB	5.0 MB	+0.0% ⚪	✅
Memory Use Parallel Chains	11.7 MB	12.0 MB	+2.3% ⚪	✅
Scan Throughput 100	2.869s	3.105s	+8.2% ⚪	✅
Scan Throughput 1000	21.156s	21.159s	+0.0% ⚪	✅
Typical Queue Shuffle	5.40µs	5.55µs	+2.7% ⚪	✅
Priority Queue Shuffle	26.18µs	26.14µs	-0.1% ⚪	✅

🎯 Performance Summary

+ 2 improvements 🚀
  29 unchanged ✅

🔍 Significant Changes (>10%)

• Excavate Single Thread Small: 10.8% 🚀 faster
• Excavate Parallel Tasks Small: 34.3% 🚀 faster

---

🐍 Python Version 3.11.17