Repository navigation
[Bug]: After a streamed deep crawl is stopped early, the next deep crawl returns only the start page #2380
Description
crawl4ai version
0.9.5 (same on 0.9.0)
Expected Behavior
Every crawler.arun(url, config) with a deep_crawl_strategy and stream=True runs a deep crawl, including after a previous streamed deep crawl was stopped early.
Current Behavior
If the caller stops iterating a streamed deep crawl (break), the next deep crawl from the same task returns only the start page.
DeepCrawlDecorator sets the deep_crawl_active ContextVar to True and resets it in the finally of the stream wrapper. When the caller leaves the stream without closing it, that finally runs later, when the generator is garbage-collected, in another context. The flag stays True in the caller's context, so the next arun() is treated as a nested call and falls through to a single-page crawl. The same happens if the stream is consumed from another task (for example asyncio.create_task(anext(stream))): the reset only reaches that task's copy of the context.
Is this reproducible?
Yes
Code snippets
import asyncio
from aiohttp import web
from crawl4ai import AsyncWebCrawler, CrawlerRunConfig
from crawl4ai.deep_crawling import BFSDeepCrawlStrategy
PAGES = 5
async def handle(request):
links = "".join(f'<a href="/p{i}">p{i}</a>' for i in range(PAGES)) if request.path == "/" else ""
return web.Response(text=f"<html><body>{links}<p>{'text ' * 200}</p></body></html>", content_type="text/html")
async def deep_crawl(crawler, url, stop_after=None):
config = CrawlerRunConfig(deep_crawl_strategy=BFSDeepCrawlStrategy(max_depth=1), stream=True)
pages = []
async for result in await crawler.arun(url, config=config):
pages.append(result.url)
if stop_after and len(pages) == stop_after:
break # e.g. the caller restarts the crawl with another browser config
return pages
async def main():
app = web.Application()
app.router.add_get("/{tail:.*}", handle)
runner = web.AppRunner(app)
await runner.setup()
await web.TCPSite(runner, "127.0.0.1", 8765).start()
url = "http://127.0.0.1:8765/"
async with AsyncWebCrawler() as crawler:
print("full crawl:", len(await deep_crawl(crawler, url))) # 6
print("stopped early:", len(await deep_crawl(crawler, url, stop_after=2))) # 2
print("next crawl:", len(await deep_crawl(crawler, url))) # 1, expected 6
await runner.cleanup()
asyncio.run(main())Suggested fix
Reset the flag as soon as deep_crawl_strategy.arun() has returned the stream, with the token from set(), instead of in the stream's finally. The strategies' own batches already run with deep_crawl_strategy=None, so the guard is not needed while the stream is consumed. Callers can work around it by closing the stream in their own task (async with contextlib.aclosing(stream)).
Activity
PiUnknown commentedon Oct 15, 2026
Taking this one. The cause is that DeepCrawlDecorator sets deep_crawl_active as a ContextVar and resets it in the generator's finally block, which runs late in a different context when the stream is stopped early - so the flag stays set and the next crawl exits after the start page. Plan: reset the flag using the token from set() (guarded so a late finally can't clobber a new crawl's state), plus a regression test built on the repro above (aiohttp server, break after the first result). PR coming shortly.