fix(html): extract <details>/<summary> content instead of discarding it

[r0h1tb]

Problem

<details> and <summary> were both mapped to RemovedBlock, so partition_html() discarded the whole subtree. On the FAQ page from the issue, every question and answer disappears and the caller gets a shorter element list rather than an error:

>>> partition_html(text="<h1>Support FAQ</h1>"
...   "<details><summary>Which Bluetooth profiles are supported?</summary>"
...   "<p>Your device supports A2DP and HFP profiles.</p></details>")
[<Title 'Support FAQ'>]          # both the question and the answer are gone

Fix

<details> is an ordinary block container, so it becomes Flow, like <div>.

<summary> becomes Heading. The spec calls it "a summary, caption, or legend for the rest of the contents", the issue asks for the questions as titles, and it is what makes the fix useful downstream — chunk_by_title() opens a new section at each Title, so a question stays attached to its own answer instead of blending into the next entry.

<summary> was also listed twice, under both phrasing and the removed-form group; there is now one mapping.

Scope

Only these two tags. <dl>, <figure> and the form/nav tags stay in RemovedBlock:

• <figure>/<figcaption> removal is asserted by DescribeRemovedBlock.it_is_skipped_during_parsing, so it is a deliberate decision, not an oversight.
• <dl>/<dt>/<dd> also drop their content, and ListBlock's docstring already says "maybe a <dl> element at some point". That looks worth doing, but reclassifying it changes the ancestor chain that _category_depth walks (there is a parametrised case at test_parser.py:298 relying on <dl><dd> nesting), so it belongs in its own PR rather than riding along here.

Tests

Four tests in test_unstructured/partition/html/test_partition.py. With the parser change reverted on this branch:

FAILED test_partition_html_extracts_details_and_summary_content        - assert [] == [(...5 elements)]
FAILED test_partition_html_makes_a_summary_start_its_own_chunk_section
FAILED test_partition_html_extracts_a_details_block_that_has_no_summary
FAILED test_partition_html_extracts_nested_details_blocks

Also checked by hand and behaving: unclosed <details>, <summary> not the first child, two <summary> siblings, empty <summary>, <summary> outside any <details>, attributes on <details>, inline markup inside <summary>, <details> inside a table cell and inside a list item, 20-deep nesting, and a JSON round-trip.

Suite before 146 failed / 2531 passed, after 146 failed / 2535 passed — the same 146 IDs, all pre-existing here (pandoc and the ML extras aren't installed locally). 14 modules that cannot be collected without those extras were excluded from both runs.

Note: this bumps to 0.26.4, as does #4449 — whichever lands second needs the version and changelog re-pointed.

[cubic-dev-ai]

No issues found across 4 files

<sub>Shadow auto-approve: would auto-approve. Bug fix in HTML parsing: previously dropped <details>/<summary> content is now extracted, with tests pinned; low-risk mapping change.

Re-trigger cubic</sub>

[cragwolfe]

High — a leading image or block wrapper makes a <summary> lose its Title boundary.

unstructured/partition/html/parser.py:1023 maps summary to Heading, but Heading supplies Title only to the initial text/phrasing run in Flow.iter_elements() (parser.py:365-376). Since img is a non-phrasing ImageBlock, <summary><img src="/chevron.svg" alt="">Second question?</summary> first emits the image, then processes Second question? through the generic tail path without Heading._ElementCls; it becomes Text, not Title. A <div>/<p>-wrapped question follows the same child-block path. As a result, chunk_by_title(..., combine_text_under_n_chars=0) never sees the second FAQ boundary and can merge that question/answer into the preceding section.

Please add summary-specific traversal that preserves the title role for the summary label across leading images and ordinary block wrappers, while keeping the details body and post-summary tail outside the title and avoiding a duplicate title for a nested heading. The regression should assert exact element types/order and two separate chunks; covering link/emphasis and page metadata on the label would also protect the annotations carried by this path.

(authored by codex)

[r0h1tb]

Good catch, thanks. Fixed in 51dabe8, and rebased onto main since the branch had gone stale.

<summary> now has its own element class that emits every run of label text as a Title, including text after a leading image like the chevron and text inside plain <p>/<div> wrappers (recursively). Images, a nested <h3> and other specialized blocks keep their own types, so a nested heading still produces exactly one Title. The summary's tail and the rest of the <details> body are still handled by the parent, so they stay outside the title.

New tests assert the exact element types and order plus two separate chunks for the icon case, and cover the <p>/<div> wrappers, a nested heading, text after the summary, and emphasis, link and page-number metadata on a wrapped label. The html, chunking, documents, md and text tests go from 1129 passed on main to 1139 on this branch, with no failures.

[cragwolfe]

Finding

• [P1] Preserve heading hierarchy for summary-generated Titles — unstructured/partition/html/parser.py:475-483. Summary._iter_label_elements() now emits every summary run as Title, but _ElementAccumulator._category_depth() maps a non-h1–h6 Title to depth 0. The shared set_element_hierarchy() stack then treats each summary as a top-level heading. I reproduced this on the exact head 51dabe8436708a28cb1713a90ad7ca00f78805b9: <h2>FAQ</h2><details><summary>Q1</summary><p>A1</p></details><h2>Shipping</h2><p>x</p> produces Q1 with parent_id=None instead of FAQ, and incorrectly assigns Shipping under Q1; nested Outer/Inner summaries are also siblings instead of parent/child. This silently regresses parent_id/section hierarchy for existing headings and affects downstream HTML/ontology/RAG consumers.

Please fix the hierarchy semantics for summary-derived Titles, preserving real heading levels and making nested details/summary relationships explicit, then add regression assertions for parent IDs and category depths (including a heading after a details block and nested details). The current extraction and focused text/type tests do not catch this.

Verification: uv run --locked reproduction against the exact PR head confirmed the incorrect parent IDs; all reported exact-head CI checks were green, but CI does not cover this hierarchy case.

(authored by codex)