Repository navigation
Support DocBook XML sources for complete systemd documentation #116
Description
Context
This is a follow-up to #110. I investigated adding the complete systemd documentation as systemd/systemd using the official repository:
https://github.com/systemd/systemd
A Markdown-only registry definition builds successfully, but it omits most of systemd's authoritative documentation.
Findings
The upstream documentation is split into two parts:
docs/: 95 supported Markdown filesman/: 484 DocBook XML files, approximately 8.1 MB
The Markdown-only prototype produced:
- 80 indexed documents
- 523 sections
- 219,498 tokens
This covers architectural and integration guides, but not the complete reference. Important documentation such as systemctl, journalctl, systemd.service, unit directives, network configuration, and most APIs lives under man/*.xml.
Context currently discovers Markdown, MDX, HTML, AsciiDoc, and reStructuredText, but not XML. Registry sources support Git and ZIP without a preprocessing step.
The XML cannot simply be treated as plain text:
- 388 files use XInclude
- 43 files reference generated entities
- 483 files declare DocBook DTDs
- upstream renders HTML using Meson and
xsltproc --xinclude - generated HTML is not committed or provided as a release asset
The official rendered documentation is available at:
https://www.freedesktop.org/software/systemd/man/latest/
However, that site has no llms.txt, and the registry cannot currently build a package by following an HTML index.
Proposed solution
My preferred solution is safe native DocBook support:
- Recognize
.xmlfiles whose root document is a supported DocBook type such asrefentry. - Resolve XInclude/XPointer references only within the cloned documentation root.
- Resolve required local entities without allowing external network access or arbitrary filesystem access.
- Convert DocBook sections, lists, tables, examples, commands, filenames, variables, and program listings into the existing parsed-document representation.
- Add support for multiple documentation roots, for example
docsandman, rather than scanning the entire source repository. - Add security tests for external entities, path traversal, recursive includes, entity expansion, malformed XML, and oversized documents.
A possible alternative is a constrained HTML-index source that follows same-origin links from systemd's official rendered man-page index. This would be simpler for systemd but less reproducible and would make nightly builds depend on hundreds of individual website requests.
Activity
moshest commentedon Aug 25, 2026
Thank you for this — it's the most thorough issue this project has received. I verified your numbers rather than taking them on trust, and I'm proposing a different route than native DocBook. Reasoning below.
Your measurements
man/XML sizexi:includeman/docs/.md(+5.html)The gap you identified is real.
systemctl,journalctl,systemd.service,systemd.unitandsystemd.networkexist only asman/*.xml— none are reachable fromdocs/, which is 90 architecture notes. A Markdown-only systemd package would be close to useless, exactly as you said.Why I don't want to parse the XML
I tested
linkedom(our only XML-capable dependency) against the actual files. Two things decided it:XInclude isn't processed.
xi:includesurvives as an inert node contributing no text — and it appears 5,036 times across those 388 files. Every page would silently lose its options, specifiers and version tables. No error, no failed build, just quietly gutted documentation. In a pipeline where one failing definition takes down the nightly publish for all 140 packages, a failure mode that doesn't fail is worse than one that does — we'd be serving confidently incomplete man pages to agents, and finding out later means re-parsing everything, not just re-fetching.The entities can't be resolved from a checkout. All 21 are defined in
man/custom-entities.ent.in— a meson template. Without running systemd's build they stay literal, so those 57 occurrences index as garbage.One piece of good news from the test:
linkedomnever resolves external entities at all —<!ENTITY x SYSTEM "file:///etc/hostname">comes back as literal text. So XXE isn't the blocker you reasonably feared. XInclude is.What I'd rather do
freedesktop.org already publishes the rendered man pages, and they're version-pinnable:
The index is a real DocBook-XSL table of contents with 1,326 unique
.htmllinks,robots.txtpermitsman/, and.htmlis a formatparseDocumentalready handles. No XML parser, no XInclude, no entity resolution, no new dependency — and pinning/man/258/rather than/latest/keeps builds reproducible.What still has to be built — I want to be straight about this
Your alternative wasn't wrong to be cautious, and this isn't free:
SourceSchemaisgit | ziponly (definition.ts:49), and there's no rendered-docs tarball (systemd-man.tar.gz→ 404). Ingesting an HTML index means a new source type that walks a same-origin index — smaller than a DocBook parser, but still a real change with its own failure modes.docs_pathis a singlestringin both schemas, sodocs+manneedsstring | string[](touching four call sites inbuild.ts) or two separate registry entries.So: I'm accepting the goal and rejecting the mechanism. If you'd like to take the index-crawl source on I'd be glad to review it, and I'm happy to spec the schema shape first so you're not guessing at the interface. If you'd rather not, that's completely fine — this issue was valuable on its own, and correcting my assumption about the Podman rename earlier was too.
Generated by Claude Code
moshest commentedon Sep 21, 2026
#137 is open and implements the route agreed above: an
html-indexsource pinned to a version directory, plus two registry definitions —systemd/systemdfor the rendered manuals andsystemd/systemd-guidesfor the Markdown underdocs/from Git. That also answers your point 5, by splitting the two roots into separate packages rather than makingdocs_patha list. I've left this open to track with #137; the goal closes with it, but native DocBook XML parsing stays out of scope, so if you'd rather keep that as a separate request say so and I'll split it out.Generated by Claude Code
feat(registry): add versioned HTML index sources and systemd document…