Repository navigation
[Bug]: PruningContentFilter strips out <a> and <strong> tags completely #582
Description
crawl4ai version
Crawl4AI 0.4.247
Expected Behavior
PruningContentFilter works perfect for my use case, just keeps the meat / pure content.
BUT it removes a and strong tags completely.
Is this reproducible?
Yes
Inputs Causing the Bug
Steps to Reproduce
Code snippets
### INPUT:
###<p>If you want to examine this <a href="https://www.domain.com">Raspberry Pi project</a> in greater detail and see how it's constructed, visit <strong>GitHub</strong> where you can also follow on X for updates.</p>
### OUTPUT ("Raspberry Pi project" and "GitHub" are missing):
### If you want to examine this in greater detail and see how it's constructed, visit where you can also follow on X for updates.
def get_md_conf():
prune_filter = PruningContentFilter(
threshold_type="dynamic",
min_word_threshold=20,
)
md_generator = DefaultMarkdownGenerator(
content_filter=prune_filter,
)
return md_generator
def get_crawl_conf():
crawl_config = CrawlerRunConfig(
cache_mode=CacheMode.BYPASS,
markdown_generator=get_md_conf(),
excluded_tags=["form", "header", "footer", "nav"],
word_count_threshold=20,
)
return crawl_config
async def process_url(
url: str,
crawler: AsyncWebCrawler,
):
crawl_config = get_crawl_conf()
result = await crawler.arun(
url=url,
config=crawl_config,
)
if result.success:
print(result.markdown_v2.fit_markdown)OS
Windows
Python version
3.13
Browser
No response
Browser version
No response
Error logs & Screenshots (if applicable)
No response
Activity
aravindkarnam commentedon Jan 31, 2025
@qoder1337 Thanks for reporting this. I'll investigate this tomorrow.
aravindkarnam commentedon Jan 31, 2025
RCA
@qoder1337 This can't exactly be classified as a bug. If you set the
PruningContentFilteras follows then it will capture the<a>and<strong>tags also in fit_markdown (ie you'll see the semantic meaning conserved in the markdown)Issue with this setting, is it will allow all other useless tags also to move along causing your output to become very heavy downstream,
Now I think this(suppressing anchor and strong tags) is rather the feature, than a bug 🤓. In pruning algorithm there's a score that's being calculated based on various factors among which there's
tag_importance(for dynamic threshold) andtag_weights. If the score is above threshold the tags are preserved for markdown conversion, else they are decomposed. Here they areAs you can see both
<a>and<strong>are not even in the list. They get a very small score, hence discarded. (-1.0 actually, that's why I had to set negative threshold in earlier example)So if we are to say tags like
<a>and text styling tags like<strong>,<em>,<b>are to be preserved for their semantic meaning in markdown, that's a wider discussion to be had. Did you tryBM25ContentFilter? How's that working out in this scenario?@unclecode What's your view on this?
qoder1337 commentedon Feb 6, 2025
hi thanks for your answer and sorry for late reply. BM25ContentFilter doesnt work as intended for my use case here.
i went with the following setting, where markdown syntax gets shown in a and strong tags, but it seems to be a good compromise for my scenario to keep the other noise out:
prune_filter = PruningContentFilter(
threshold=0.6,
threshold_type="dynamic",
)
aravindkarnam commentedon Feb 7, 2025
@qoder1337 Cool then! Since changing the threshold was able to get you your desired output. I'm closing this issue. Also I'm moving this to forums, so that it may be of use for others as well.