Skip to content

fix: match standard metadata names case-insensitively - #2410

Open
linjiaye929 wants to merge 1 commit into
unclecode:developfrom
linjiaye929:bugfix/metadata-name-case
Open

linjiaye929 wants to merge 1 commit into
unclecode:developfrom
linjiaye929:bugfix/metadata-name-case

Conversation

@linjiaye929

@linjiaye929 linjiaye929 commented Oct 16, 2026 •

Copy link
Copy Markdown

Summary

The public LXML scraper drops standard metadata such as <meta name="Description" content="Page summary">. HTML standard metadata names are ASCII case-insensitive, but the three current selectors require lowercase names.

Match description, keywords, and author names with XPath ASCII translation. Preserve content casing, first content-bearing tag order, missing/empty content behavior, and all Open Graph/Twitter handling. The legacy BeautifulSoup helper is outside this change.

Files and verification

  • crawl4ai/utils.py: only the three standard-name predicates.
  • tests/unit/test_metadata_name_case.py: public scraper regressions and compatibility controls.
  • 118 focused/related tests pass, including 36 new cases (12 failed on unchanged develop).
  • Local HTML fixtures; no browsers, live websites, or models. New tests pass Black and git diff --check; production formatting debt is pre-existing on both base and candidate. Full repository suite was not run. The exact-head Discord notification workflow completed with action_required; it requires maintainer action and does not establish hosted code-test success.

AI assistance

Developed and reviewed with AI assistance; no human full-diff review is claimed.

Checklist

  • Based on and targets develop
  • Self-reviewed the bounded change
  • Added regressions and compatibility controls
  • Existing metadata values and selection order preserved
  • No new metadata fields or documentation contract introduced

@linjiaye929
linjiaye929 marked this pull request as ready for review October 16, 2026 11:04
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant