Skip to content

XLSX: keep unmodelled parts on save (#330) - #333

Merged
h-a-n-a-s-h-i merged 3 commits into
storytold:mainfrom
vangel76:feat/xlsx-passthrough
Oct 11, 2026
Merged

h-a-n-a-s-h-i merged 3 commits into
storytold:mainfrom
vangel76:feat/xlsx-passthrough

Conversation

@vangel76

Copy link
Copy Markdown
Contributor

Part of #330 (slicers, timelines and the data model are not done, see below)
Fixes #324

Problem

Opening and saving an .xlsx dropped every part GridCraft doesn't model: custom XML, links to external workbooks, data connections, query tables, printer settings (#324), form controls, OLE objects… (docs/gaps.md §3). The reader only warned (or said nothing) and the writer regenerates the package from the model.

Design (crates/xlsx/src/passthrough.rs)

  • Model: gridcraft_model::Preserved (new preserved.rs): kept parts (name, content type, bytes, their own rels with ids unchanged), the owner's relationships to them (full type URI, target, external flag), raw owner XML fragments, and the sheet's VML drawing. Workbook.preserved holds package-root and workbook content, Sheet.preserved the sheet's. Both are #[serde(skip)] (not in the JSON format) and Arced, so undo snapshots stay cheap.
  • Read: for each owner, the relationship graph from an allow-listed root is followed transitively (content type from [Content_Types].xml, bytes read with limits). External targets are kept as text and never opened, followed or refreshed.
    • Package root: custom-properties, extensibility (customUI), webextensiontaskpanes.
    • Workbook: customXml, connections, and externalLink together with <externalReferences>. The links are kept all or nothing, because [1] in a formula is the position in that list. Formulas such as [1]Sheet1!$A$1 were already kept as text, so they still resolve in Excel.
    • Sheet: queryTable (relationship only), printerSettings + the <pageSetup r:id> (XLSX save drops printerSettings parts and the pageSetup r:id #324), and the raw <picture>, <oleObjects> and <controls> elements (also inside mc:AlternateContent). The worksheet root's namespace declarations are added to each fragment. The rels the fragments reference are kept too. Controls and OLE objects also keep the sheet's legacy VML drawing, minus its note shapes, plus the VML's own rels (preview images).
  • Write: kept parts keep their original name when it's in a folder our writer never uses (customXml/, xl/externalLinks/, xl/embeddings/, xl/ctrlProps/, xl/printerSettings/…). Otherwise (e.g. xl/media/) they get a fresh kept{N} name. Each gets a content-type override and its rels file with targets renamed (absolute targets). The owner relationships get new ids. The r:ids in the kept fragments are rewritten with quick-xml (any attribute in the relationships namespace). The fragments are emitted in schema order: externalReferences after <sheets>; pageSetup r:id; picture/oleObjects/controls after <legacyDrawing> and before <tableParts>.
  • VML: when the sheet still has notes, GridCraft's note shapes are merged into the kept VML. They go in a fresh shape block (o:idmap data="1,2", ids _x0000_s2049…) so they can't collide with the control/OLE shape ids that <control shapeId> / <oleObject shapeId> refer to. Without notes, the kept VML is written on its own. This is text-based because Excel's VML is often not well-formed XML (<br>).
  • Conservative choices (maintainer decisions welcome):
    • Content whose graph reaches a part GridCraft rewrites (worksheet, drawing, chart, table, pivot, comments, slicers, rich data, VBA…) is dropped with a warning.
    • A missing or damaged part, a damaged rels part or a size limit drops that root, never the read.
    • Sheet content goes away with its sheet. sheet.move with copy does not copy it (one line in engine/src/cmd/sheet.rs).
    • Cell references inside kept parts (a checkbox's linked cell, a query table's range) are kept as they were: they are not adjusted when rows or columns are inserted.
    • Strict Open XML files keep nothing (different namespaces from what we write), with a warning.
  • Bounds: 64 MB per kept part, 256 MB and 4,096 parts per workbook, 16 MB per rels/VML/content-types part, 10,000 rels per part, 16 relationship hops, 4 MB per XML fragment and nesting depth 256. Cycles are handled by a visited set. No unwrap/indexing on file data.
  • VBA is left to Preserve VBA macros + Excel-style macros UX (issue #6) #19 (vbaProject is in the rewritten/excluded list, so the two don't double-write). Feature/table slicers #193 models table slicers but doesn't write them to XLSX. Slicers aren't passed through here, so no double-write later.
  • Reader warnings: removed "embedded objects and form controls are not supported" and "links to external workbooks are not supported". Kept controls/OLE now say "not shown; they are kept for saving", kept links say "kept; they are not followed or refreshed", and dropped content says why.

Tests (built in code, no Excel files)

crates/xlsx/src/tests/passthrough_tests.rs:

  • unmodelled_parts_survive_a_round_trip: a package with every kept kind plus a note. After read → write, every kept part is byte-identical exactly once and the package is valid: every part has a content type, every internal target exists, no duplicate ids or overrides. The external path is unchanged and external. [1] formulas are unchanged. The pageSetup, oleObject, objectPr and control r:ids resolve to the right bytes. Element order is checked. The VML merge keeps the control shapes, replaces the old note and puts ours in block 2. Re-reading gives the same kept content.
  • edited_workbooks_keep_consistent_relationships: inserts a sheet before, renames the sheet, removes its note and edits a cell. The rels are still consistent and the VML is written alone. Deleting the sheet drops its content and keeps the workbook-level content.
  • printer_settings_and_page_setup_id_survive_save: the XLSX save drops printerSettings parts and the pageSetup r:id #324 repro (openpyxl-style package).
  • damaged_kept_content_is_dropped_without_failing_the_read: missing target, a cyclic custom-XML pair (kept), kept content pointing at a worksheet, a link without a part (drops all links), damaged rels, a missing printer-settings part, a control without VML / with an unknown r:id.
  • oversized_kept_parts_are_dropped: a part over the per-part cap.

Commands run (in a worktree, per crate):

cargo test -p gridcraft-xlsx                      # 84 passed
cargo clippy -p gridcraft-model -p gridcraft-xlsx -p gridcraft-engine --all-targets -- -D warnings
cargo fmt -p gridcraft-xlsx -p gridcraft-model -p gridcraft-engine -- --check
cargo check -p gridcraft-xlsx -p gridcraft-engine --target wasm32-unknown-unknown
cargo xtask layers

Not run: the full cargo xtask ci (CI covers it).

Not done

  • Slicers and slicer caches, timelines and their caches: they reference pivot caches by id, sheets by sheetId and drawing graphic frames, all of which GridCraft regenerates.
  • The data model (xl/model/item.data, Power Pivot) and Power Query output stored as table query tables (the queryTable hangs off the table part and needs tableType="queryTable" / queryTableFieldId on the table).
  • Unmodelled extLst entries in sheet and workbook XML, legacyDrawingHF (header/footer pictures), chart details.
  • External links are kept but not modelled: values aren't refreshed, and "Edit Links" doesn't exist.
  • Kept cell references aren't adjusted to structural edits (see above).

Unverified assumptions (needs confirmation in Excel / corpus)

  • Our output opens in Excel without a repair prompt for each kept kind. Only checked structurally (OPC consistency), not in Excel.
  • Excel accepts absolute relationship targets (/xl/...) for kept parts. openpyxl writes those and Excel opens them, but this wasn't checked here.
  • Excel tolerates VML shape-id blocks that are unique within a drawing but may repeat across sheets (GridCraft's note VML numbering was already per-sheet).
  • Content types of kept parts are copied from the source [Content_Types].xml as overrides. Names were not checked against Excel output.

🤖 Generated with Claude Code

vangel76 and others added 3 commits October 11, 2026 13:12
Pass-through of package content GridCraft does not model: on read the raw
parts are kept with their content types and relationship graph per owner
(package root, workbook, sheet); on write they get fresh part names where
needed and new relationship ids, and the r:ids in kept XML are rewritten.

Kept: custom document properties, ribbon customisation, add-in task panes,
custom XML, links to external workbooks with <externalReferences> (never
followed), connections, sheet query tables, printer settings with the
<pageSetup r:id> (storytold#324), form and ActiveX controls, OLE objects and sheet
backgrounds with their VML shapes (merged with GridCraft's note shapes).
Content whose graph reaches parts GridCraft rewrites is dropped with a
warning; sheet content goes with its sheet and is not copied by Copy Sheet.

Fixes storytold#324

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
# Conflicts:
#	ROADMAP.md
#	crates/model/src/lib.rs
#	crates/xlsx/src/sheet_read.rs
#	docs/file-format-parity.md
#	docs/gaps.md
…o the kept total

Content types (and extensions) of kept parts come from the input file and
were written into [Content_Types].xml unescaped, so a quote or angle bracket
injected XML; they are now attribute-escaped. Each sheet re-read and kept
its VML drawing (up to 16 MB) outside MAX_KEPT_TOTAL, so many sheets sharing
one highly compressed drawing amplified memory; the drawing is now read
within and charged to the remaining kept-bytes allowance.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@h-a-n-a-s-h-i
h-a-n-a-s-h-i merged commit fda7f5b into storytold:main Oct 11, 2026
@echelon

echelon commented Oct 11, 2026

Copy link
Copy Markdown
Member

Merged — thank you for the contribution! (Before merging I pushed two hardening fix-ups with tests: content types copied from the input file are now attribute-escaped in [Content_Types].xml, and kept VML drawings are charged to the MAX_KEPT_TOTAL budget.)

If you haven't already, come join us on the ArtCraft Discord: https://discord.gg/artcraft

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

XLSX save drops printerSettings parts and the pageSetup r:id

3 participants