Skip to content

fix: reset deep_crawl_active around stream yields so an early break doesn't poison the next crawl (fixes #2380) - #2386

Open
PiUnknown wants to merge 1 commit into
unclecode:mainfrom
PiUnknown:fix/deep-crawl-reset
Open

PiUnknown wants to merge 1 commit into
unclecode:mainfrom
PiUnknown:fix/deep-crawl-reset

Conversation

@PiUnknown

Copy link
Copy Markdown

Fixes #2380

Problem

When a streamed deep crawl is abandoned early (caller breaks out of async for without closing the generator), deep_crawl_active stays True in the caller's context. The flag was only reset in the generator's finally, which runs on close/GC - too late, and possibly in a different context. The next arun() with a deep-crawl config then saw the flag still set, treated itself as a nested call, and fell through to a single-page crawl.

Fix

Manage the flag around each yield of the stream wrapper instead of relying on generator cleanup:

  • the flag is True only while the wrapper is actively advancing the inner generator (when nested arun calls actually happen)
  • it is reset while suspended at the yield, so an early break leaves the caller's context clean and the next crawl starts fresh
  • a guarded reset(token) (falls back to set(False)) handles consumption from a different context/task
  • the wrapper's finally also aclose()s the inner generator so strategy cleanup isn't left to GC
  • non-stream path now uses a proper reset(token) and holds the flag across the strategy call itself

Testing

  • New end-to-end regression test: local aiohttp server, full crawl (6 pages) -> stream stopped early at 2 without closing -> next crawl returns all 6 (previously returned 1).
  • Issue repro before fix: full 6 / stopped 2 / next 1. After fix: full 6 / stopped 2 / next 6.
  • pytest tests/deep_crawling/: 87 passed, 1 pre-existing failure in test_deep_crawl_perf_2242.py (wall-clock timing flake on Windows, unrelated).

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Bug]: After a streamed deep crawl is stopped early, the next deep crawl returns only the start page

1 participant