fix(scraping): keep text after removed style/link/meta/noscript tags (#2416)

[zhongxiao-chang]

Summary

Fixes #2416.

LXMLWebScrapingStrategy._scrap removed <style>, <link>, <meta> and <noscript> elements with element.getparent().remove(element). In lxml the text that follows an element inside its parent lives in that element's .tail, and remove() throws the tail away with the element. A paragraph containing one of those tags therefore lost everything after the tag, while the scrape still reported success — silent data loss.

Wikipedia emits <link rel="mw:PageProp/Category"> inside paragraph text, so ordinary article prose was being truncated.

Before (develop c1a9c8d, offline, no browser):

<p>The port grew quickly, and as of 2020</p>
<p>The old warehouses stood on the quay</p>

After:

<p>The port grew quickly, and as of 2020, it handles more cargo than any other port.</p>
<p>The old warehouses stood on the quay until the great fire destroyed them.</p>

The fix is remove() -> drop_tree(), which moves the tail onto the previous sibling (or onto the parent when there is no previous sibling). The <script> loop a few lines below in the same method already re-attaches .tail by hand before removing, so keeping the following text is clearly the intent of the surrounding code; this loop was the one site that still dropped it.

List of files changed and why

• crawl4ai/content_scraping_strategy.py — switch the style/link/meta/noscript removal loop to drop_tree() so the element's tail (following text) survives; comment explains why.
• tests/unit/test_removed_tags_keep_tail.py — new regression tests: five parametrised shapes (link / meta / style / noscript inside a sentence, tag as first child of the paragraph) plus tag-between-two-sentences and tag-between-inline-siblings.

How Has This Been Tested?

Fully offline — LXMLWebScrapingStrategy.scrap() plus DefaultMarkdownGenerator.generate_markdown(), no browser, no network.

• New test file on this branch: 7 passed.
• Same file on develop c1a9c8d (source reverted, test kept): 6 failed, 1 passed — so the tests really do pin this bug.
• Whole tests/unit suite, before vs after the change: the failure set is byte-identical (24 failures, all in test_browser_tls_sandbox_flags.py, the pdf tests and the egress tests). Those are environment reds on this Windows box (playwright browser binaries and outbound network are unavailable here), not caused by this change — hence the "existing unit tests pass locally" box below is left unchecked with that caveat.
• The new test file is black --check clean. content_scraping_strategy.py as a whole is not black-clean on develop either (pre-existing trailing whitespace and quote-style deviations elsewhere in the file); black --diff does not touch any line in this change, and running the formatter over the whole file would bury the fix in unrelated churn, so I left it alone.

Related, for merge order: #2377 touches the same class of bug in the excluded-links branch (link.drop_tree(), around line 327) and is still open. This loop is a separate site; the two patches do not overlap, so whichever lands first the other still applies cleanly.

Checklist:

• My code follows the style guidelines of this project
• I have performed a self-review of my own code
• I have commented my code, particularly in hard-to-understand areas
• I have made corresponding changes to the documentation — no user-facing API or docs change; behaviour only becomes what the code already intended.
• I have added/updated unit tests that prove my fix is effective or that my feature works
• New and existing unit tests pass locally with my changes — new tests pass; 24 pre-existing tests/unit failures are environment-related (no browser binaries / no outbound network on this machine) and are identical before and after this change.