[Bug]: PDF URI links are silently discarded by PDFContentScrapingStrategy due to Links schema mismatch #2374

crawl4ai version

0.9.5 (also observed in 0.9.4).

Expected Behavior

URI link annotations extracted from a PDF should survive PDFContentScrapingStrategy.scrap() and be accessible in the returned ScrapingResult.links, with their source page numbers preserved or represented by an explicitly documented contract.

This report concerns preservation of the extracted link inventory, not reconstructing clickable text positions in HTML/Markdown or PDF-internal page destinations.

Current Behavior

The public processor extracts the URI correctly, but the public PDF scraping strategy returns success=True and empty internal / external link lists for the same file.

The apparent cause is a schema mismatch:

• PDFContentScrapingStrategy.scrap() builds links = {"urls": []} and appends {"url": link, "page": page.page_number}.
• It passes that dictionary to ScrapingResult.
• ScrapingResult.links is typed as Links, which defines only internal and external. Its Link items use href rather than url and have no explicit PDF page field.
• The unrecognized urls key is silently ignored during model construction, losing the collected links.

Sources: PDF scraper, models, and PDF processor.

The PDF parsing documentation also describes links["urls"], suggesting that preserving PDF links is intended, but the current result model cannot represent that structure.

Is this reproducible?

Yes.

Inputs Causing the Bug

A local, generated one-page PDF containing one URI annotation. No browser, network requests, external files, credentials, or application-specific adapter is needed. The page is intentionally blank apart from its annotation to isolate link preservation.

Steps to Reproduce

1. Use Python 3.12 and install pip install "crawl4ai[pdf]==0.9.5".
2. Save the following snippet as repro_pdf_links.py.
3. Run python repro_pdf_links.py.
4. Observe that the processor contains the URI but the scraping result has empty link lists.

Code snippets

from pathlib import Path
from tempfile import TemporaryDirectory

from pypdf import PdfWriter
from pypdf.annotations import Link

from crawl4ai.processors.pdf import PDFContentScrapingStrategy
from crawl4ai.processors.pdf.processor import NaivePDFProcessorStrategy

with TemporaryDirectory() as folder:
    path = Path(folder) / "linked.pdf"
    writer = PdfWriter()
    writer.add_blank_page(width=612, height=792)
    writer.add_annotation(
        0,
        Link(rect=(20, 20, 200, 40), url="https://example.org/reference"),
    )
    writer.write(path)

    processed = NaivePDFProcessorStrategy(extract_images=False).process(path)
    scraped = PDFContentScrapingStrategy(extract_images=False).scrap(str(path), "")

    print("processor links:", processed.pages[0].links)
    print("scraping success:", scraped.success)
    print("scraping links:", scraped.links.model_dump())

OS

macOS 27.0.1, Apple Silicon (arm64).

Python version

3.12.13; pypdf 6.20.1.

Browser

Not applicable; the reproduction uses a local PDF and public SDK methods directly.

Browser version

Not applicable.

Error logs & Screenshots (if applicable)

No exception is raised. Actual output on 0.9.5:

processor links: ['https://example.org/reference']
scraping success: True
scraping links: {'internal': [], 'external': []}

Proposed direction / contribution

I would like to contribute a focused fix targeting develop, with regression coverage using a generated PDF and updated documentation. Before changing the public link contract, could you confirm the intended representation?

One possible approach is to map PDF URI links into the existing internal / external structure using href, and add an optional 1-based page field to Link so PDF page attribution is not lost. For remote PDFs, classification could reuse the existing URL classification rules; for local PDFs without a source URL, treating URI links as external is a possible fallback, subject to maintainer preference.

This is a proposal rather than an assumed API decision. Reader reuse / parallel parsing optimization and reconstruction of inline clickable text would remain outside this fix.