[liquidsec]

Fixes #3441

The dnsdumpster module has been silently returning nothing. It scrapes a JWT out of the homepage's hx-headers attribute, and that attribute is now served empty:

hx-headers='{"Authorization": ""}'

An empty string is falsy, so the module falls through to verbose("Error obtaining JWT token") and sets errorState. At verbose level this is invisible in a default scan; it only turned up when a module-attribution comparison showed the module going from ~57 discovered names to 0.

This is not transient and not scraping detection. Repeated fetches, a cookie jar, a plain user agent and a full browser header set (including Sec-Fetch-* and sec-ch-ua) all return an empty token, no cookies are set, and no site JavaScript populates it. The endpoint itself is alive: POST api.dnsdumpster.com/htmld/ returns HTTP 200 with Error: Invalid or missing token, reload page and try again, identically for an empty token or no header at all.

dnsdumpster has moved to a membership model, and there is now a documented API:

GET https://api.dnsdumpster.com/domain/<domain>[?page=N]
Header: X-API-Key: <key>

• Rate limit of one request per two seconds, 429 on exceed
• Response carries a / cname / mx / ns arrays of {host, ips[...]}, a txt array of strings, and total_a_recs
• MX host values include the priority ("0 mail.example.com"), so hostnames are extracted from the whole response rather than read from host
• Free keys get a single page; asking for ?page=2 returns 401 {"error":"pagination requires plus or max membership"}

Changes

• Converted to subdomain_enum_apikey with a sensitive, mandatory api_key. There is no keyless path any more: the public page's token is always empty.
• Auth moved to the X-API-Key header
• Pages are requested until the A records seen reach total_a_recs, up to a configurable max_pages (default 10), stopping early on a 401/403, an empty page, a missing total, or a parse failure
• Rate limiting is left to api_request's 429 handling
• No ping_url, deliberately: a health check would spend one of only 50 daily requests on a free key

Validation

Validated against the live API with a free key: authentication, the response shape above, the 401 on page 2, and a module run that emitted every in-scope A record in one request. Multi-page paging needs a paid key and is covered by mocked tests only.

Note

The free tier is 50 records per domain per request with no pagination, and 50 requests per day. That is thin enough for subdomain enumeration that this module is realistically only useful with a paid key.

[singlerider]

Catching this from a module-attribution drop of ~57 names to 0, then ruling out scraping detection with a cookie jar and full browser headers, and confirming the endpoint is alive because POST /htmld/ returns the same error for an empty token and no token, is the right way to work a silent failure.

🔴 Not validated against the live API

Your own blocker.

[singlerider]

A mandatory api_key means subdomain_enum_apikey.setup ends in require_api_key() and the module disables itself for anyone without a key, while still carrying the subdomain-enum flag. dnsdumpster runs for everybody today. You note the free tier is not realistically useful, so this may be correct, but it is a product decision and it is not argued anywhere in the PR.

[singlerider]

max_pages is arbitrary and it is the exposed option. page_size below is the value that actually varies per account (50 free, 200 paid) and it is hardcoded. That is backwards.

Also, the 10 default is duplicated in setup. anubisdb does the same thing so this one is precedent, not your invention.

[singlerider]

record_count sums a/cname/mx/ns and compares the total against a per-page host cap you have not observed. If the cap is per record type, this truncates silently.

A free key also only terminates here by accident, because 50 happens to be less than the hardcoded 200.

[liquidsec]

Addressed the clear-cut items in f7fda55:

• Dropped meta["auth_required"]; it's derived from the mandatory field.
• Dropped request_interval and the manual sleep. 429s are handled by api_request, which paces across handlers (and more so after Overhaul api_request rate limiting and add config deprecation support #3461).
• Every early exit now logs.
• JSON is parsed once per page.
• Hostnames come from extract_in_scope_hostnames over the whole response, as the previous version did, so names inside MX/NS/TXT values are kept. Added a TXT case to the tests.
• Renamed page_size to full_page_record_count and record_keys to host_record_keys.

Further action on this PR is blocked until we obtain a paid API key. The pagination rework, the mandatory-key decision, and validation against the live API all depend on observing real responses.

[codecov]

Codecov Report

❌ Patch coverage is 91.46341% with 7 lines in your changes missing coverage. Please review.
✅ Project coverage is 91%. Comparing base (7aeee20) to head (8a21863).
⚠️ Report is 3 commits behind head on dev.

Files with missing lines	Patch %	Lines
bbot/modules/dnsdumpster.py	84%	7 Missing ⚠️

Additional details and impacted files

@@          Coverage Diff          @@
##             dev   #3458   +/-   ##
=====================================
+ Coverage     91%     91%   +1%     
=====================================
  Files        461     461           
  Lines      49520   49565   +45     
=====================================
+ Hits       44655   44696   +41     
- Misses      4865    4869    +4     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[github-actions]

📊 Performance Benchmark Report

Comparing dev (baseline) vs dnsdumpster-api-key (current)

📈 Detailed Results (All Benchmarks)

📋 Complete results for all benchmarks - includes both significant and insignificant changes

🧪 Test Name	📏 Base	📏 Current	📈 Change	🎯 Status
Bloom Filter Dns Mutation Tracking Performance	2.27ms	2.14ms	-5.7% ⚪	✅
Bloom Filter Large Scale Dns Brute Force	8.72ms	8.19ms	-6.1% ⚪	✅
Large Closest Match Lookup	193.97ms	193.11ms	-0.4% ⚪	✅
Realistic Closest Match Workload	107.29ms	104.10ms	-3.0% ⚪	✅
Event Memory Medium Scan	1402 B/event	1402 B/event	+0.0% ⚪	✅
Event Memory Large Scan	1527 B/event	1527 B/event	+0.0% ⚪	✅
Event Validation Full Scan Startup Small Batch	241.90ms	233.14ms	-3.6% ⚪	✅
Event Validation Full Scan Startup Large Batch	316.11ms	291.85ms	-7.7% ⚪	✅
Make Event Autodetection Small	12.99ms	12.08ms	-7.0% ⚪	✅
Make Event Autodetection Large	130.01ms	120.42ms	-7.4% ⚪	✅
Make Event Explicit Types	5.57ms	5.08ms	-8.9% ⚪	✅
Excavate Single Thread Small	2.543s	2.384s	-6.2% ⚪	✅
Excavate Single Thread Large	6.001s	5.624s	-6.3% ⚪	✅
Excavate Parallel Tasks Small	2.591s	2.415s	-6.8% ⚪	✅
Excavate Parallel Tasks Large	4.021s	3.774s	-6.1% ⚪	✅
Intercept Throughput Small	683.12ms	665.69ms	-2.6% ⚪	✅
Intercept Throughput Medium	739.39ms	666.44ms	-9.9% ⚪	✅
Dns Throughput Quiet	1.409s	1.430s	+1.5% ⚪	✅
Dns Throughput Loaded	1.147s	963.28ms	-16.0% 🟢🟢	🚀
Dns Throughput Inherited	1.173s	1.263s	+7.7% ⚪	✅
Is Ip Performance	1.36ms	1.27ms	-7.2% ⚪	✅
Make Ip Type Performance	115.35µs	110.49µs	-4.2% ⚪	✅
Mixed Ip Operations	1.56ms	1.36ms	-12.6% 🟢🟢	🚀
Memory Use Web Crawl	1.2 MB	1.2 MB	+0.0% ⚪	✅
Memory Use Subdomain Enum	28.3 MB	28.3 MB	+0.0% ⚪	✅
Memory Use Deep Chain	5.0 MB	5.0 MB	+0.0% ⚪	✅
Memory Use Parallel Chains	11.1 MB	11.2 MB	+0.8% ⚪	✅
Scan Throughput 100	1.582s	1.654s	+4.6% ⚪	✅
Scan Throughput 1000	10.006s	10.533s	+5.3% ⚪	✅
Typical Queue Shuffle	3.03µs	2.91µs	-3.9% ⚪	✅
Priority Queue Shuffle	15.12µs	14.51µs	-4.0% ⚪	✅

🎯 Performance Summary

+ 2 improvements 🚀
  29 unchanged ✅

🔍 Significant Changes (>10%)

• Dns Throughput Loaded: 16.0% 🚀 faster
• Mixed Ip Operations: 12.6% 🚀 faster

---

🐍 Python Version 3.11.16

[liquidsec]

Correction to my earlier comment: validated against the live API with a free key, so this is no longer blocked. The response includes total_a_recs, so paging now follows that instead of a guessed page size (77387c6). Page 2 on a free key returns 401 pagination requires plus or max membership, which the module already handles. Multi-page paging on a paid key is the only path left untested live.

[liquidsec]

On the mandatory key: the unauthenticated scraping path is dead -- dnsdumpster removed their public HTML interface. There's nothing to fall back to without a key. This is also established precedent: shodan_dns, c99, bufferoverrun, fullhunt, bevigil, subdomainradar, and trickest are all subdomain-enum + mandatory=True inheriting from subdomain_enum_apikey.

On the template bypass: overriding query() is the established pattern for APIs that don't fit api_page_iter. dehashed, hunterio, github_codesearch, and censys_dns all do the same thing. dnsdumpster's pagination doesn't fit the template for three reasons: page 1 has no ?page= param while page 2+ does, the stop condition is a running total (a_records >= total_a_recs) rather than an empty page, and the free tier returns 401/403 on page 2 which is normal behavior, not an error. Forcing it through api_page_iter would require a custom iter_key, cross-page state tracking in parse_results, and _json=False (because we use extract_in_scope_hostnames on raw text) -- more code than the hand-rolled loop and harder to follow. The template still earns its keep for dedup, scope filtering, wildcard rejection, API key validation, and event emission.