[OsirisTechnicalInstitute]

Adds otilabs, a passive subdomain-enum module that queries the OTI Labs Domain Intelligence API. Disclosure: I run this API.

• Inherits subdomain_enum_apikey and follows the same shape as bufferoverrun (RapidAPI headers) and securitytrails.
• One GET per queried domain: /domain/{domain}/subdomains. Every name in the response's subdomains list that ends with .{domain} is emitted. The API builds that list from certificate transparency logs, passive DNS and a DNS brute force, with names that currently resolve listed first.
• Needs a RapidAPI key in modules.otilabs.api_key. The free plan is 1,000 requests a month.
• No ping: the API has no free health endpoint through RapidAPI, so a ping would spend one of the user's requests.
• Uses the API's default fast response, which returns well inside the 10 s http_timeout. The API's ?wait=1 mode can take around 20 s, so it isn't used.

Tests: test_module_otilabs.py mocks the API response and checks that both a live and a historical name are emitted and that a host outside the queried domain is not. ruff check and ruff format --check pass, and the existing bufferoverrun test still passes locally.

AI Use Disclosure

Collaborative: the module and test were written with Claude (Anthropic, Claude Opus 5.5) following AGENTS.md and the existing bufferoverrun and securitytrails modules, then reviewed and run locally before submitting.

[github-actions]

All contributors have signed the CLA ✍️ ✅
_{Posted by the CLA Assistant Lite bot.}

[OsirisTechnicalInstitute]

I have read the CLA Document and I hereby sign the CLA

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 91%. Comparing base (da1397a) to head (11d862d).
⚠️ Report is 35 commits behind head on dev.

Additional details and impacted files

@@          Coverage Diff           @@
##             dev   #3496    +/-   ##
======================================
+ Coverage     91%     91%    +1%     
======================================
  Files        460     463     +3     
  Lines      48998   49443   +445     
======================================
+ Hits       44129   44580   +451     
+ Misses      4869    4863     -6     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[liquidsec]

Hi, thanks for the PR. Overall, the module looks syntactically correct and follows all of our guidelines for a module of this type. However, a few concerns:

The default (fast) response does not contain the API's full result set. It returns whatever sources have finished so far, reports the rest as still running in sources_used, then continues enriching in the background.

Querying the same domain twice, back to back:

1st call (cold)   2nd call (warm)
latency               3.77s             0.40s
names returned        31                32
live_count            null              14
"live" key present    no                yes
crt.sh                enriching         completed
subfinder             enriching         completed
liveness              enriching         completed

The module inherits subdomain_enum_apikey, so _incoming_dedup_hash dedups by highest parent and BBOT issues exactly one request per domain. There is no second call. That means BBOT systematically receives the cold, partial answer and never the warm one.

Is there a mode that blocks only until enrichment completes, without the full ~20 s worst case?

My thoughts are maybe its worth waiting the time? we can override the default 10 second http_timeout if needed.

The other thing is the test. The mock response doesn't match the default (fast) response. So if we are only going to ever get the "cold" response, the test should mock that, but obviously based on the first part of my reply there's some debate to whether that's what we want.

Can you also tell me what we would expect to see once the user exceeds the 1000 limit for a free key?

[OsirisTechnicalInstitute]

Thanks for testing it so thoroughly. Your numbers match what I'd expect from the fast mode.

Complete results: yes, ?wait=1. It returns as soon as every source has finished rather than waiting a fixed time, and a domain that's already cached comes back immediately. Cold lookups I ran today:

domain	names	time
projectdiscovery.io	89	15.4 s
bosch.com	1,105	15.2 s
unilever.com	4,186	17.1 s

There isn't a faster way to get the complete set for a domain the API hasn't seen: most of those ~15 s is subfinder's per-source timeout. Results are cached for 24 hours, so scanning the same domain again that day is immediate.

It stops waiting at 22 s. If a slow source is still running then, it returns what has finished and marks that source as still enriching in sources_used. The one thing wait=1 doesn't wait for is the live/historical split (live, live_count), because that means resolving every name. The module only reads subdomains, and BBOT resolves the names itself, so it doesn't need it.

I've pushed a commit that requests ?wait=1 and sets timeout=self.http_timeout_infrastructure + 20 (30 s by default), the same pattern as crt.

Test: the mock now has the shape of a real cold ?wait=1 response: no live key, live_count: null, and liveness: enriching in sources_used.

Over the free limit: the free plan has a hard cut-off at 1,000 requests a month, enforced by RapidAPI's gateway, so the request never reaches the API. The gateway answers HTTP 429 with a JSON body along the lines of {"message": "You have exceeded the MONTHLY quota for Requests on your current plan, BASIC. Upgrade your plan at https://rapidapi.com/..."}.

The free plan also has a limit of 10 requests a minute, which returns 429 as well, and BBOT's usual sleep-and-retry handles that fine. Retrying can't help with the monthly cut-off, though: BBOT will retry (sleeping between attempts), count the failures and eventually put the module in an error state. If you'd prefer, I can make the module recognise the quota message, log one clear warning and skip its remaining queries for the scan.

Paid plans don't stop at their quota; extra requests are billed as overage. From 80% of the free quota, responses also carry a _notice field and X-Quota-* headers, which the module ignores.

[liquidsec]

Understood, yeah I think for the way BBOT works the trade off of a longer wait for the answer vs more complete is absolutely worth it.

As for the rate limit questions, that all sounds normal and I believe our existing machinery for erroring out a module should apply fine (i will test that), just wanted to make sure there wasn't anything abnormal there that would require an override of our standard api handling.

[liquidsec]

I tested the quota case on current dev with the monthly-quota 429 response. Our existing handling covers it: each 429 counts as a rate limit, and after 10 in a row the module goes into error state and stops querying. Because the response has no subdomainskey, it just parses to an empty result, so nothing breaks.

The one downside is that the 429 has no Retry-After header. Without it, BBOT waits its default 30 seconds between attempts, so a scan spends about 5 minutes retrying a quota that won't reset before it gives up. The error message is also generic ("ratelimited 10 times in a row"), so the user isn't told directly that their monthly quota is used up. If the API could sendRetry-After on these responses, that would shorten the wait. Otherwise, the existing failure handling is fine as it is.

[liquidsec]

Thanks again for the PR, we are merging this into dev now, it will make it into the 3.1 release.