[Bug]: CrawlerRunConfig.clone() drops fallback_fetch_function, so deep crawls never use it #2379

crawl4ai version

0.9.5 (same on 0.9.0)

Expected Behavior

CrawlerRunConfig.clone() returns a copy with every field, fallback_fetch_function included. BFS, DFS and BestFirst deep crawls clone the run config for every batch (config.clone(deep_crawl_strategy=None, stream=...)), so a configured fallback should also apply to deep-crawled pages.

Current Behavior

clone() rebuilds the config from to_dict(), and to_dict() leaves fallback_fetch_function out, so the clone has fallback_fetch_function=None. In a deep crawl, a page whose fetch raises (or is blocked) never reaches the fallback, while the same page fetched with arun() / arun_many() does.

Is this reproducible?

Yes

Code snippets

from crawl4ai import CrawlerRunConfig


async def fallback(url: str) -> str:
    return "<html>...</html>"


config = CrawlerRunConfig(fallback_fetch_function=fallback)
print(config.fallback_fetch_function)  # <function fallback ...>
print(config.clone(stream=True).fallback_fetch_function)  # None

Suggested fix

Carry callables over in clone(), for example by copying fallback_fetch_function (and any other non-serializable field) from self onto the clone unless it is passed in kwargs. A subclass of CrawlerRunConfig whose clone() copies the field over works as a workaround.