fix(html): keep non-li children of lists in document order

[morten-lagabote]

Children of <ul>/<ol> other than <li> are invalid HTML, but common in CMS output. _handle_list only iterated the li/ul/ol children: <p>, <table> and other children were dropped, and a <ul>/<ol> child was added directly to the list group (the workaround referring to docling-core#357), which broke the numbering of the following items.

_handle_list now walks all the children in DOM order, following the convention of the DOCX backend for interleaved blocks (#3896):

• content before the first <li> is emitted before the list;
• a nested <ul>/<ol> stays a sub-list of the preceding list item;
• any other content closes the list group and is emitted at the parent level; the items that follow open a new list group whose numbering continues from the running counter. A child that adds nothing, like a <br>, does not close the list.

_handle_list returns every item it adds at the current level, so the content emitted around a split list stays in its container (e.g. a table cell).

In the Lovdata page of the issue, the 7 paragraphs that were dropped are now kept, each between the items it separates.

The loop of _walk moves to _walk_nodes, which walks a subset of the children of an element, so these children go through the same code path as any other content. The signature of _walk is unchanged, so subclasses that override it keep working.

Unchanged: an <li> without text creates no list item.

No change in the ground truth.

Issue resolved by this Pull Request:
Resolves #4424

Checklist:

• Documentation has been updated, if necessary.
• Examples have been added, if necessary.
• Tests have been added, if necessary.

[github-actions]

✅ DCO Check Passed

Thanks @morten-lagabote, all your commits are properly signed off. 🎉

[mergify]

Merge Protections

🟢 Merge protection satisfied — ready to merge.

Show 1 satisfied protection

🟢 Enforce conventional commit

Make sure that we follow https://www.conventionalcommits.org/en/v1.0.0/

• title ~= ^(fix|feat|docs|style|refactor|perf|test|build|ci|chore|revert)(?:\(.+\))?(!)?:

[ceberam]

Thanks @morten-lagabote for the careful investigation and for putting together a working fix. The bug is real and the content loss is a genuine problem worth solving.

After reviewing the PR carefully, I think the approach of nesting non-<li> children under the preceding list item is not the right choice for Docling, for the following reasons:

1. The spec-compliant DOM does not nest the <p> under the preceding <li>. The HTML is already invalid. When the browser's HTML parser is inside a <ul> element, it is in a state expecting only <li> (or script-supporting tags). When it encounters an unexpected start tag like <p>, the specification executes "foster parenting" and error recovery rules. It will likely force-close the <ul> early, render the paragraph outside of it, and then open a new hidden list for the remaining items, or place <p> as a sibling of it. So "browsers render it inside the list, below the preceding item" is not accurate for a spec-compliant parser.
2. In DoclingDocument's schema, a GroupList (list group) only accepts ListItem or nested GroupList children. Appending a TextItem as a child of a ListItem does not violate the schema per se, but it does mean the interleaved paragraph becomes a sub-item of the list item before it, which may be fine in the examples you posted (e.g., Lovdata.no pages) but can be a semantic distortion for others. The paragraph "About the first item" is not logically part of "First item"; it is independent content that happens to be misplaced in the source HTML.
3. The DOCX contract (issue #3896) already established a consistent convention: an interleaved block that is not a list item forces the current list to close, the block is emitted at the parent level, and a new list opens afterwards. This is the semantically correct interpretation, and it is exactly what spec-compliant HTML parsers (and the HTML5 tree construction algorithm) would produce anyway.

Instead, I would suggest implementing the following behavior in _handle_list:

1. Content before the first <li>: your existing takewhile/leading logic is correct and should be kept.
2. Non-<li>, non-sublist children between list items: close the current list group, emit the node via _walk_nodes at the enclosing parent level, then open a new list group for the items that follow. If the list is ordered, initialize the new group's start counter from the running list_item_counter so that numbering is preserved.
3. Nested <ul>/<ol> children: continue to handle these as proper sub-lists (the existing behavior here is fine).

The _walk_nodes helper you introduced is a useful building block and can remain as-is. The test you added is also a good skeleton, it would just need its assertions updated to match the new expected structure.

Thank you again for identifying the issue and for the clear reproduction case. We look forward to a revised version.

[morten-lagabote]

Thanks for the review, @ceberam. I pushed the changes you suggested (f3d3656):

• content between items closes the list group and is emitted at the parent level with _walk_nodes; the items that follow open a new list group whose numbering continues from the running counter (<ol start="5">: 5., interruption, 6.);
• nested <ul>/<ol> children remain sub-lists of the preceding item;
• the leading content and _walk_nodes are unchanged;
• the test now asserts the new structure (text, list, text, list, table, list at body level).

One note on point 1, for the record: in the HTML tree construction algorithm a <p> start tag inside <ul> is inserted as a child of the <ul> (foster parenting applies to tables only), so browsers keep the paragraph inside the list, and BeautifulSoup builds the same tree. That does not change the conclusion: the DOCX convention gives one consistent structure across backends, and that is what the PR does now.

[morten-lagabote]

Two follow-ups from my own review, in the last commit: _handle_list now returns every item it adds at the current level, so the content around a split list stays in its table cell (it used to be re-parented outside the cell by group_cell_elements), and a child that adds nothing, like a <br>, no longer closes the list. Both are covered by the tests.

[codecov]

Codecov Report

❌ Patch coverage is 96.22642% with 2 lines in your changes missing coverage. Please review.

Files with missing lines	Patch %	Lines
docling/backend/html_backend.py	96.22%	0 Missing and 2 partials ⚠️

📢 Thoughts on this report? Let us know!

[PeterStaar-IBM]

@ceberam is this addressing your request?

[ceberam]

Thanks @morten-lagabote for refactoring the PR according to the contract.
It fixes the issue and I don't have any objection with the current implementation.

It would however help having a visual example of this edge case, so readers can understand how the Docling tree should be and how a markdown export looks like. The example that you brought when describing the issue #4424 is simple yet very illustrative:

<ul>
  <li>First item</li>
  <p>A paragraph placed directly inside the list.</p>
  <li>Second item</li>
</ul>
<p>After.</p>

It could be good to add it to one of our ground-truth test files, with a proper disclaimer that this is not valid HTML. The file tests/data/html/sources/html_nested_block_in_list_item.html would be the perfect place to append it.