feat: recognize Division/Subdivision heading markers in hierarchy inference - #4473
aImErYbArRaUlT wants to merge 1 commit into
Conversation
|
✅ DCO Check Passed Thanks @aImErYbArRaUlT, all your commits are properly signed off. 🎉 |
Merge Protections🟢 Merge protection satisfied — ready to merge. Show 1 satisfied protection🟢 Enforce conventional commitMake sure that we follow https://www.conventionalcommits.org/en/v1.0.0/
|
Codecov Report✅ All modified and coverable lines are covered by tests. 📢 Thoughts on this report? Let us know! |
…erence Signed-off-by: Aimery Barrault <aimery@barratec.com>
92ce259 to
69afd9a
Compare
|
Heads-up on the red CI, in case it's useful: the only failing check here is That's in Happy to rebase once |
Resolves #4472
Summary
The heading-hierarchy numbering recognizer handled
Part/Chapter/Articlebut notDivision/Subdivision, so legal-statute structure (Part > Division > Subdivision > section) stayed flat even withHeadingHierarchyOptions(enabled=True). This teaches the parser the two keywords. BecauseDivision/Subdivisionare ordinary words ("Division of Powers", "Division du congé"), they count only when a real enumerator follows; multi-letter tokens are validated with the existing_is_roman, consistent with the file's own rule in_classify_letter.docs/usage/heading_levels.mdis updated with the new markers and default order.Changes
_DEFAULT_FAMILY_ORDER: adddivision/subdivisionbetweenpartand the section-level families._parse_marker: recognize the two keywords via_KW_DIVISION/_KW_SUBDIVISION, gated by a new_keyword_enumeratorhelper that requires an Arabic index, a single letter, or a valid Roman numeral (_is_roman). The enumerator must be followed by whitespace, end of string, or separator punctuation, so French elisions likeDivision d'appelare not treated as markers._LEADING_MARKER: add the keywords so bookmark/title matching strips them in sync.pipeline_options.py: document the families in thenumbering_schemesfield description.docs/usage/heading_levels.md: add the markers and the default order.Evidence
Parser (
_parse_marker):Corpus (20 Canadian federal + Alberta statutes): 372 headings begin with
Division/Subdivision; 341 classified structural, 31 left unclassified (plain-word titles), 0 false negatives.End-to-end (Alberta Corporate Tax Act,
enabled=True): all 59Divisionheadings move##->###, nested under the 20Partheadings; total heading count unchanged (359).Side effect: inserting two families above the section rank pushes the pre-existing
(N)-subsection headings one level deeper. In this document one reaches level 6, which docling-core renders as seven#(beyond CommonMark's six-level cap), so it displays as text rather than a heading. That is the existingmax_levelclamp interacting with docling-core's level+1 hash mapping, not new behavior; I am happy to file it in docling-core separately.Behavior change for custom
numbering_schemes: a list that omitsdivision/subdivisionpreviously left those headings unnumbered; they are now numbered at the lowest rank, below every listed family. The default order is unaffected. Covered bytest_omitted_scheme_family_ranks_lowest. Users who want them ranked higher can add the two families to their list.Known limitation (follow-up): compound enumerators such as
Division 1A/Subdivision 6AA(common in Australian statutes) are not yet recognized and fall through as unnumbered. They can be added later without changing the default behavior. The default order also placespart/titleabovedivision; documents that nest Division above Title can reorder vianumbering_schemes(documented inheading_levels.md).Testing
tests/test_heading_hierarchy.py- 45 passed.make validateclean.Checklist
docs/usage/heading_levels.md).docs/examples/heading_levels.pyalready covers usage.)