[yashikam19]

Related discussion: #2245

Summary

Adds two constructor parameters to BFSDeepCrawlStrategy and
BestFirstCrawlingStrategy for finer concurrency control during deep crawls:

1. batch_size (BestFirstCrawlingStrategy only) — previously a hardcoded
   module constant (BATCH_SIZE = 10) governing how many URLs are pulled from
   the priority queue per round before being handed to arun_many(). Now a
   constructor parameter, defaulting to the same value (10), so nothing changes
   for existing callers.
2. dispatcher (both strategies) — previously neither strategy forwarded a
   dispatcher to its internal arun_many() calls, so callers had no way to
   supply their own (e.g. SemaphoreDispatcher, or a MemoryAdaptiveDispatcher
   with a custom RateLimiter/CrawlerMonitor) - only CrawlerRunConfig.semaphore_count
   was reachable, which controls concurrency count but not dispatcher behavior.
   Now optional and forwarded when set.

Both changes are additive and opt-in — dispatcher is only passed to
arun_many() when explicitly provided, so the call shape (and any existing
test doubles built against it) is unchanged for the default case.

List of files changed and why

• crawl4ai/deep_crawling/bfs_strategy.py - added dispatcher param, forwarded
  in both _arun_batch and _arun_stream.
• crawl4ai/deep_crawling/bff_strategy.py - added batch_size and dispatcher
  params; batch_size replaces the module-level BATCH_SIZE constant at its
  three usage sites; dispatcher forwarded in _arun_best_first.
• tests/deep_crawling/test_deep_crawl_dispatcher_batch_size.py (new) — covers
  batch_size defaults/override/actual round-size behavior, and dispatcher
  default-omitted vs explicitly-forwarded behavior for both strategies.

How Has This Been Tested?

• pytest tests/deep_crawling/ tests/general/test_deep_crawl*.py: 80 passed,
  0 regressions
• New test file (10 tests) specifically covering the two new parameters.

Checklist:

• My code follows the style guidelines of this project
• I have performed a self-review of my own code
• I have commented my code, particularly in hard-to-understand areas
• I have made corresponding changes to the documentation
• I have added/updated unit tests that prove my fix is effective or that my feature works
• New and existing unit tests pass locally with my changes

[yashikam19]

Hi @ntohidi, a gentle ping on this one. It adds two opt-in deep-crawl options (batch_size and dispatcher) for per-site rate limiting, with no change to default behaviour, and still merges cleanly into develop. Happy to adjust the API or add docs if needed (context in #2245). Thanks!

[yashikam19]

Update:

• Rebased on latest develop; merges cleanly.
• Added a fix for [Bug]: BestFirstCrawlingStrategy fetches a full batch past max_pages #2381: the best-first batch loop now respects the remaining max_pages budget, with a regression test.

[ntohidi]

@yashikam19
Hi! Thank you for the PR and your contribution! I apologize for missing the PR! I will check it by tomorrow.

[ntohidi]

@yashikam19
Thanks, this looks good and the max_pages fix is right. One thing: DFSDeepCrawlStrategy inherits the new dispatcher param but its own _arun_batch/_arun_stream still call arun_many without it, so it's silently ignored for DFS. Could you forward it there too and add a test? Also worth a short docs note, since the dispatcher (and its monitor) gets restarted on every batch.

[yashikam19]

Thanks for the review @ntohidi! Both points are addressed:

• DFS: DFSDeepCrawlStrategy now forwards dispatcher in _arun_batch and _arun_stream, with tests for both, plus tests that the default (no dispatcher) call shape is unchanged.
• Docs: added dispatcher / batch_size to the strategy parameter lists and a new section 8.3 Using batch_size and dispatcher in deep-crawling.md, with a rate-limiting example and a note that the dispatcher is started/stopped per batch.