Skip to content

Calc at Excel scale: topological recalc, lookup indexes, multi-threaded and background recalc, memory, shared formulas (#205) - #208

Open
arhammusheer wants to merge 11 commits into
storytold:mainfrom
arhammusheer:perf/calc-scale
Open

arhammusheer wants to merge 11 commits into
storytold:mainfrom
arhammusheer:perf/calc-scale

Conversation

@arhammusheer

@arhammusheer arhammusheer commented Oct 10, 2026 •

Copy link
Copy Markdown

Implements #205: recalculation that scales the way Excel's does. It brings lookups that no longer grow with the square of the data, multi-threaded recalculation, a desktop window that stays responsive during long recalculations, roughly half the memory, and fast opening of large XLSX files. It also fixes a wrong answer: on main, a long formula chain that reads against row order returns #CIRC!.

There are 11 commits: one per stage, one that fits the stages to what main has added since, and two that fix what review on real workbooks found (below). Each stage has its own numbers in its message, so it reads best commit by commit.

Fixes after review (2026-10-16)

A run over 247 real workbooks found wrong values (mostly a false #CIRC!) on 22 of them, results that changed with the thread count, and an iterative workbook running 3–4× slower than on main. Two commits fix these:

  • False #CIRC! (3a5ee32). When a formula reads cells that aren't calculated yet, the pass queues them and returns to the formula later. A read of a cell that was only queued next to the reader, not waiting on it, counted as a cycle. Take =SUM(A2:A4) where A3 reads A2 and A4 reads A3: A4 is evaluated first, reads A3 (queued but not waiting) and gets #CIRC!. This path is reached only by formulas whose inputs aren't known before the pass, such as names over ranges, INDIRECT/OFFSET, cells behind a real cycle and formulas a worker hands back. That is why it showed up on real workbooks and moved with the thread count. Now only formulas that are being evaluated or are waiting count toward a cycle.
  • Iterative calculation (3e3b238). Each iteration re-evaluated everything reachable from a loop, plus every INDIRECT/OFFSET formula in the workbook and everything those feed. Now only formulas that actually read a value from a loop are re-evaluated, whether directly or through a range, name, spill or lookup. On a synthetic workbook with 60,000 formulas, 40,000 of them OFFSET/INDIRECT, and a small loop, a recalculation went from ~740 ms to ~50 ms (main: ~140 ms), with bit-identical results.

Checked against main. I generated 34 random three-sheet workbooks of ~2,500 formulas each. They cover cross-sheet references, names over cells and ranges, INDIRECT, computed OFFSET, VLOOKUP/MATCH/INDEX/SUMIF/COUNTIF, IF branches, spills and # references. 16 are acyclic, 8 have real circular references and 8 have iterative loops. main (84acf62) built each one cell by cell, and this branch then recalculated it from scratch.

  • The branch matches main on every formula cell, at all cores and at one thread.
  • After 40 random edits, every cell value matches main.
  • The branch before these fixes got 37–66% of cells wrong on the same files, so the comparison catches this class of bug.

Results

Each figure is the median of 3 runs of crates/engine/examples/perf.rs, which this PR extends (every scenario checks its answers), unless noted. Re-measured on 2026-10-14 after the rebase: main at 84acf62 against this branch, both release builds, on an Apple M2 Pro (6 performance + 4 efficiency cores). Sizes are 50,000 rows unless noted.

Lookups and aggregates: algorithmic fixes

Scenario main This PR Speed-up
10,000 exact VLOOKUPs over a 10,000-row table 2,822 ms 30 ms 93×
10,000 XLOOKUPs 2,181 ms 33 ms 66×
10,000 MATCHes 2,165 ms 28 ms 78×
Edit the table that 30,000 lookups read 4,851 ms 14 ms 349×
100,000 VLOOKUPs over a 100,000-row table 359 s 0.49 s ~740×
100,000 XLOOKUPs 257 s 0.50 s ~510×
100,000 MATCHes 255 s 0.45 s ~560×
Edit the table that 300,000 lookups read 620 s 0.14 s ~4,500×
1,000 × SUM(A:A) 1,085 ms 21 ms 53×
Edit A1 under 1,000 SUM(A:A) 1,088 ms 19 ms 57×

The 100,000-row rows are one run for main (25 minutes in all) and the median of 3 for this PR.

Recalculation

Scenario main This PR Speed-up
20,000-cell chain, each cell reads the one below #CIRC! (wrong) 32 ms, correct —
20,000-cell chain, top to bottom 44 ms 26 ms 1.7×
10,000 formulas over a 1,000-cell array each 2,047 ms 309 ms 6.6× (all cores)
Edit A1 under those 10,000 heavy formulas 2,030 ms 284 ms 7.2×
Edit B1, everything below recalculates 57 ms 43 ms 1.3×
Desktop app: heavy edit (40,000 array formulas) window frozen for the whole recalc the edit returns in ~90 ms; the status bar shows "Calculating (10 threads): x%" —

The desktop-app row was observed before the rebase and not re-measured.

Editing at scale

Scenario main This PR Speed-up
Paste 1,000,000 values (200,000 rows) 1,271 ms 303 ms 4.2×
Paste over existing data (200,000 rows) 1,458 ms 390 ms 3.7×
Fill 200,000 formulas 714 ms 234 ms 3.1×
Undo / redo one edit over 200,000 formulas 127 / 117 ms 0.3 / 0.3 ms ~400×
Insert / delete a row above 200,000 formulas 648 / 646 ms 243 / 232 ms 2.7× / 2.8×
Sort 50,000 rows / undo it 161 / 43 ms 121 / 25 ms 1.3× / 1.7×

Large files: 1,000,000 rows × 8 columns, 2M formulas, 57 MB

A synthetic file (6 number columns and 2 formula columns, with cached values) opened and saved with gridcraft-cli. Each figure is the mean of 2 runs, with peak memory footprint from /usr/bin/time -l. The file has heavier formulas than the one measured before the rebase, so the times aren't comparable with earlier figures; the ratio to main is the same as before.

main This PR
Open (gridcraft-cli info) 15.8 s, 3.9 GB peak 6.8 s, 2.0 GB peak (2.3×)
Open and save (gridcraft-cli run --in … --out …) 22.6 s, 4.1 GB peak 11.3 s, 2.2 GB peak (2.0×)
Open and recalculate, 50,000 rows (perf) 553 ms 314 ms (1.8×)

Memory

main This PR
Peak, full benchmark at 50,000 rows 455 MB 271 MB
Peak, 1M values + 200,000 filled formulas with undo 1,251 MB 702 MB
One stored value 183 B 70 B
One filled formula cell ~590 B ~150 B

Peaks are memory footprints from /usr/bin/time -l. The per-cell sizes were worked out before the rebase and not re-measured.

Costs

  • One small scenario is slightly slower. The SUM, SUMIFS and XLOOKUP aggregate recalculation takes 12.7 ms instead of 10.2 ms, because building the shared ranges doesn't pay off at that size.
  • Saved files of 16 MB or more are about 8% larger (57.6 → 62.4 MB for the file above), because those parts are compressed at deflate level 3 to save faster.

Rebased onto main (84acf62, 2026-10-14)

main moved about 480 commits while this was open. The last commit, Fit #205 to main, holds the adaptations:

  • Iterative calculation (Calc: iterative calculation for circular references (Maximum Iterations, Maximum Change) #421) was written for the recursive evaluator this PR replaces. It's rebuilt on the non-recursive pass. A circular read gives the previous value. The formulas that read a loop, directly or through a range, name, spill or lookup, are then evaluated again in plan order (Gauss-Seidel) until Maximum Change or Maximum Iterations, with the same work cap. A recalculation with iteration on runs on one thread. All of upstream's iterative tests pass unchanged.
  • Lookups keep main's Excel-style bisection for approximate matches and XLOOKUP binary search. The per-recalc index now keeps only the exact-match hash.
  • The shifted printer used for shared formulas covers trim references (A1:.E10) and quoted sheet-qualified names.
  • Opening a file combines password decryption, row auto-fit and formula sharing with the cached-values plan.
  • Spill seeds are no longer capped at 65,536 cells (fix(calc): clearing a spill past 65536 cells leaves a later dependent stale #558).
  • In-place range folding (visit_range) walked the cell store for every SUM(A:A), bypassing the shared ranges. That made 1,000 of them 6× slower than before. A range read more than once now uses the shared array, and a range read once is still walked in place.

What changed

1. Topological recalculation without recursion (crates/calc/src/recalc.rs)

  • main evaluates the dirty set in row order and recurses into pending precedents. It treats a depth over 2,000 as a cycle, so a valid chain read against row order returns #CIRC!. With heavier formulas the recursion can overflow the stack, which catch_unwind can't catch.
  • Graph::levels now orders the dirty set with Kahn's algorithm, using a heap work stack instead of recursion.
  • A formula that turns out to read a cell that isn't done yet (through a name, a table or INDIRECT) is set aside and evaluated again once that cell is done.
  • Real cycles are still #CIRC!, and so are the cells that read them.

2. Shared ranges and lookup indexes (crates/calc, crates/functions/src/lookup.rs)

  • Within one recalculation, a range read more than once is built once and shared. It is never shared while it holds a pending cell, and a spill landing in it drops it.
  • VLOOKUP, HLOOKUP, LOOKUP, MATCH, XLOOKUP and XMATCH index a shared range on its second search. They use an exact-match hash, plus same-type positions for binary search.
  • Index keys match compare_numbers (15 digits, via the new core::round15) and fold case for text. Wildcard searches still scan.
  • Hosts reach the cache through Ctx::lookup_cache(), so layering is unchanged.

3. Multi-threaded recalculation (crates/calc)

  • Each level of 512 or more independent formulas runs on std::thread::scope workers, with no new dependency.
  • Formulas that read a cell still pending in their level, and those using RAND, RANDBETWEEN or RANDARRAY, are handed back to the main thread. Results are therefore identical for any thread count.
  • If a worker fails to start or panics, that level runs on one thread.
  • The settings mirror Excel's: CalcSettings.multi_threaded (on by default) and threads (0 means all processors).
    • They are saved as XLSX calcPr concurrentCalc / concurrentManualCount.
    • They are set with formulas.calculationOptions {multiThreaded, threads}.
    • wasm stays single-threaded.

4. Background recalculation in the desktop app (crates/engine/src/lib.rs, crates/ui-egui, apps/gridcraft)

  • When Session::set_background_calc(true) is on, an edit that leaves 512 or more formulas to calculate runs them on a thread over a copy of the workbook.
  • The command waits up to 80 ms, so ordinary edits still show results at once. Session::poll_calc applies the results as a patch.
  • Moving the selection and switching sheets work during a recalculation. Every other command waits for it first, so commands and undo snapshots always see final values.
  • Only the desktop app turns this on. The CLI, MCP, the offscreen snapshot and tests stay synchronous.

5. Memory and bulk edits (crates/calc, crates/model/src/store.rs, crates/engine/src/cmd/edit.rs)

  • An index of formulas showing #SPILL! replaces two scans that ran on every edit. Those scans were most of the cost of a large paste.
  • Calc::sync: undo and redo update the dependency graph only for the cells that changed. A sort or a structural edit rebuilds the graph instead, now in parallel.
  • The dependency index stores a cell's only dependent inline, and allocates a set only when several formulas read the cell.
  • Inserting or deleting rows and columns moves whole rows (CellStore::shift_rows / shift_cols). Formulas are rewritten in parallel, and only the ones that changed are printed again.
  • Rows are vectors sorted by column instead of B-tree maps. The CellStore API and its JSON shape are unchanged.
  • New helper: gridcraft_calc::par::filter_map. It keeps order, and runs serially on wasm and for small inputs.

6. Shared formulas (crates/formula, crates/model/src/cell.rs, crates/calc, fill, paste, sort, XLSX)

  • A Formula keeps its own text, but shares one parsed expression with the formulas copied from it, plus a (row, column) offset.
    • Copying is O(1) (Formula::moved, with text from print_shifted).
    • expr() still returns the expression as it reads at the cell.
  • The evaluator and precedents_shifted move the relative references by the offset. Names and INDIRECT text are not shifted, and a LAMBDA keeps the shift of the formula that wrote it.
  • Fill, Ctrl+Enter, paste, sort and XLSX t="shared" formulas all produce shared copies.
  • FormulaSharer re-shares equivalent formulas (keyed by R1C1 text) after a structural edit and when a file is opened.
  • The XLSX writer converts each shared expression to file form once.

7. Large XLSX files (crates/xlsx, crates/model/src/store.rs, crates/formula/src/printer.rs)

  • The streaming XML reader reuses each row's elements for the next row, and stores whole rows at once (CellStore::set_row).
  • A formula that is just the one above it moved down is recognised from its file text and shares that formula's parse.
  • The A1 printer writes straight into its output buffer, and whole numbers skip 15-digit formatting.
  • Recalculation results are written back in row order, copying a band only when a value changed.

8. Opening with cached values (crates/xlsx, Calc::load, crates/engine open_bytes_with_plan)

  • An opened XLSX keeps the values stored in the file, as Excel does. Only these are recalculated, plus whatever depends on them:
    • formulas stored without a value;
    • array and dynamic-array anchors;
    • volatile formulas.
  • A file with fullCalcOnLoad, or without calcPr, is recalculated in full.
  • Parts of 16 MB or more are compressed at deflate level 3.

Behaviour changes

  • After opening a file, cells show the values stored by the application that saved it, until something they depend on changes. This matches Excel.
  • In the desktop app, a repaint or screenshot right after a heavy edit can briefly show the previous values, with "Calculating" in the status bar. Every command, including the control channel's cell.get and sheet.read, waits for final values. This is documented in docs/control-protocol.md.

Testing

  • cargo xtask ci passes on the branch head: fmt, clippy -D warnings, tests, assets, layers, wasm and parity. Stages 1–8 were written against an older main; the last commit adapts them to what main has added since, so the stages in between don't all build on their own.
  • New tests cover:
    • long chains in both directions;
    • a 5,000-cell chain through a defined name on a 512 KiB stack;
    • cycles and the cells that read them;
    • 12 lookup edge cases, repeated so later copies go through the index;
    • identical results on 1 and 8 threads;
    • formulas queued together not being taken for a cycle;
    • a randomized comparison of a full recalculation at 1 and 8 threads with entering the cells one by one, covering names, ranges, INDIRECT, computed OFFSET and INDEX (GRIDCRAFT_ORDER_FUZZ_SEEDS=50 for a longer soak);
    • iterative calculation following every kind of read of a loop;
    • thread settings round-tripping through XLSX;
    • background and synchronous sessions agreeing across edits and undo;
    • spill blocking and unblocking;
    • undo and redo keeping the graph in step;
    • row and column shifts at the sheet edge;
    • shared copies calculating exactly like formulas typed at the same addresses (names, INDIRECT, LET/LAMBDA, OFFSET, #REF!) after edits, undo and inserts;
    • XLSX and JSON round trips of shared formulas;
    • cached values on open, including recalculating NOW(), spills and fullCalcOnLoad.
  • Each new test was confirmed to fail when the code it covers was deliberately broken.
  • I drove the desktop app over the control channel with 40,000 heavy formulas and took screenshots during and after the recalculation.

Reproduce the benchmarks with:

cargo run --release -p gridcraft-engine --example perf -- 50000 [--only basic,lookup,colsum,chain,heavy,scale] [--threads N]

Not in this PR

These follow-ups are listed under Performance in docs/target-app-parity.md:

  • typing doesn't interrupt a running recalculation; it waits for it;
  • opening a file still recalculates on the UI thread;
  • there's no UI for the thread settings yet;
  • the dependency graph isn't stored per filled block (about 650 B per formula, now the largest cost);
  • shared formulas aren't written to XLSX as t="shared";
  • there's no specialised sheetData scanner and no parallel compression;
  • structural edits still recalculate every formula.

Questions for maintainers

  • I used std::thread::scope rather than the workspace's unused rayon dependency, to avoid adding a dependency and keep wasm simple. I'm happy to switch.
  • The thread settings live in the workbook because XLSX keeps them in calcPr. Excel also has an application-wide default. Should GridCraft have one too?

🤖 Generated with Claude Code

@arhammusheer arhammusheer changed the title Calc at Excel scale: topological recalc, lookup indexes, multi-threaded and background recalculation (#205) Calc at Excel scale: topological recalc, lookup indexes, multi-threaded and background recalc, memory (#205) Oct 10, 2026
@arhammusheer arhammusheer changed the title Calc at Excel scale: topological recalc, lookup indexes, multi-threaded and background recalc, memory (#205) Calc at Excel scale: topological recalc, lookup indexes, multi-threaded and background recalc, memory, shared formulas (#205) Oct 10, 2026
@arhammusheer
arhammusheer force-pushed the perf/calc-scale branch 2 times, most recently from 539f651 to df5d4f8 Compare October 11, 2026 21:12
arhammusheer and others added 8 commits October 14, 2026 03:40
The dirty set was evaluated in row-major order, with recursion into
pending precedents as the fallback and recursion depth over 2,000
treated as a cycle. A valid chain read against row order (each cell
reading the one below) past 2,000 cells came out #CIRC!, and with
heavier formulas the recursion overflowed the stack, which
catch_unwind can't catch.

- Graph::order puts the dirty formulas in topological order with
  Kahn's algorithm over Graph::dependents, keeping the edges between
  dirty formulas in a flat list within a budget and querying again
  past it. Formulas on or behind a cycle come last.
- PassHost::settle replaces the recursive compute: a formula that
  reads a pending cell (through a name, a table, INDIRECT) is set
  aside on a heap stack, the cells it read go on top, and it is
  evaluated again once they are done. A cell read while it is set
  aside is a cycle. The depth limit is gone.
- Tests: a 20,000-cell reverse chain, a 5,000-cell chain through a
  defined name on a 512 KiB thread stack, cycles and what reads them,
  a range of formulas recalculated in the same pass.
- examples/perf.rs gains scenarios (basic, lookup, colsum, chain) that
  check every answer.

perf, 50,000 rows, Apple M2 Pro, 3 runs: reverse 20k chain #CIRC! ->
correct; edit with 100k formulas 49 -> 51 ms; sort 163 -> 170 ms;
open + full recalc 443 -> 452 ms.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Every range read built a fresh Vec<Value>, and every exact lookup then
copied its key column and scanned it, so n lookups over an n-row table
cost O(n²): 10,000 VLOOKUPs took 2.6 s, 100,000 several minutes.

- Host::range_array: within a pass, a range read a second time is
  built once and shared as one Arc<Array> (64+ cells, within a cell
  budget). It is shared only if building it read no pending cell and
  found no cycle, and a spill computed later in the pass that lands in
  it evicts it.
- LookupCache (functions crate, reached through Ctx::lookup_cache):
  the second search of the same line of a shared array builds an index
  of exact-match keys (text case-folded, numbers at 15 digits through
  the new core::round15, the same rule as compare_numbers) and of the
  positions of each type for binary search. VLOOKUP, HLOOKUP, LOOKUP,
  MATCH, XLOOKUP and XMATCH search through it; wildcard needles still
  scan. Hosts without a cache behave as before.
- Tests: twelve lookup cases (case-insensitive duplicates, first and
  last match, 15-digit equality, booleans, wildcards, approximate and
  binary modes), each three times so later copies use the index; a
  spill landing in a shared range mid-pass.

perf, Apple M2 Pro: 100,000 VLOOKUPs over 100,000 rows 0.72 s (was
minutes); 10,000 VLOOKUP / XLOOKUP / MATCH 2.6 / 3.2 / 2.1 s ->
46 / 49 / 44 ms; editing the table with 30,000 dependent lookups
5.8 s -> 20 ms; 1,000 SUM(A:A) over 50,000 rows 645 -> 89 ms.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Recalculation ran on one thread.

- Graph::levels splits the topological order into levels of formulas
  that don't read each other (statically).
- A level of 512+ formulas is evaluated on worker threads
  (std::thread::scope, no new dependency). Workers read the main
  thread's layer of the pass (results, pending, spills, shared ranges)
  and fill their own, merged after the level. A formula that turns out
  to read a cell still pending in its level (through a name, a table,
  INDIRECT) is handed back and finished on the main thread, as are
  formulas that draw random numbers, so results don't depend on the
  thread count. The main thread warms each level up (shared ranges and
  lookup indexes are built once; LookupCache::fork hands workers the
  indexes). A worker that can't start or panics makes the level run on
  one thread.
- Settings like Excel's: CalcSettings.multi_threaded (default on) and
  threads (0 = every processor), read and written as XLSX calcPr
  concurrentCalc / concurrentManualCount, set through
  formulas.calculationOptions {multiThreaded, threads}. One thread on
  wasm.
- Tests: a 3,000-row workbook mixing lookups, INDIRECT and a name over
  the same level, spills and their readers, running sums, text and a
  cycle gives the same values on 1 and 8 threads, before and after an
  edit; thread_count follows the settings; the XLSX round trip covers
  the new attributes.
- examples/perf.rs: --threads N and a `heavy` scenario.

perf, Apple M2 Pro (6P+4E): 10,000 formulas over a 1,000-cell array
each, recalc after an edit 1.89 s on 1 thread, 0.99 s on 2, 0.57 s on
4, 0.35 s on all cores; 1,000 SUM(A:A) over 50,000 rows 86 -> 20 ms.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A long recalculation froze the window until it was done.

- Calc splits into prepare / prepare_all (update the graph, return the
  formulas to recalculate) and run_dirty (evaluate them; reads the
  graph only), and reports progress through CalcProgress (per chunk
  within a level).
- Session::set_background_calc (off by default, never on wasm): a
  commit that leaves 512+ formulas defers them; when the outermost
  command is done they run on a thread over a copy of the workbook.
  The command waits 80 ms, so ordinary edits still show their results
  at once, then returns; Session::poll_calc brings the results in as a
  patch of values and spills. Commands that move around (selection.*,
  sheet.activate/next/previous) run meanwhile; every other command
  waits first, so it reads final values and every undo snapshot is
  calculated. A thread that doesn't start or panics falls back to a
  recalculation on the calling thread. F9 goes the same way.
- The desktop app turns it on, polls every frame and shows
  "Calculating (N threads): x%" in the status bar. CLI, MCP and the
  offscreen snapshot keep synchronous recalculation.
- Tests: background and synchronous sessions agree across edits, undo
  and later edits; a command right after a heavy edit reads final
  values without polling.

Desktop app, 40,000 formulas over 2,000-cell arrays, driven over the
control channel: the edit returns in ~90 ms instead of after the whole
recalculation; a selection change during it takes 18 ms; cell.get
afterwards waits and returns the final value.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
At 200,000 rows, pasting a million values took 1.3 s, undo 140 ms,
inserting a row 0.6 s, and a value cost ~183 bytes.

- Calc keeps an index of formulas showing #SPILL!: an edit seeds the
  blocked anchors near it from the index instead of scanning up to
  4,096 cells per edited cell (and every formula for a cleared cell);
  spill ranges are checked only where they meet the edited area.
- Calc::sync: undo and redo update the graph from the cells that
  differ (the cell store skips shared bands); when the sheets change or
  most cells do (a sort, a structural edit) it rebuilds instead.
- Rebuild works out precedents on every processor (new
  gridcraft_calc::par::filter_map, order-preserving, serial on wasm and
  for small inputs) and sizes its tables once.
- The dependency index keeps a referenced cell's one dependent inline;
  a set is allocated only for cells several formulas read. Node areas
  are boxed slices.
- Formula rewrite after a row/column insert or delete runs on every
  processor and prints a formula only when its expression changed.
- CellStore::shift_rows / shift_cols move whole rows (moved cells still
  win where they land); rows are vectors of cells sorted by column
  instead of B-tree maps (same API, same JSON shape).
- Tests: a far blocker unblocks a spill (also after a rebuild); undo
  and redo keep the graph in step across restored, cleared and redone
  formulas, a structural edit and a sheet rename; shifts with shared
  snapshots, collisions and the sheet edge; rows serialize as column
  maps; parallel filter_map keeps order.
- examples/perf.rs: a `scale` scenario.

perf, 200,000 rows, Apple M2 Pro: paste 1M values 1,307 -> 324 ms;
fill 200,000 formulas 688 -> 363 ms; undo / redo 141 / 128 -> 0.3 ms;
insert / delete a row 604 / 579 -> 235 / 216 ms; peak 1.26 -> 1.0 GB.
Memory probe (600,000 values + 200,000 formulas and graph): 390 -> 296
MB; a value 183 -> 70 bytes.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A formula cell cost ~590 bytes, most of it its own parsed expression,
even when 200,000 cells were filled from one formula.

- Formula keeps its own text, and a parsed expression shared with the
  formulas copied from it plus its (row, column) offset from where that
  was parsed. Formula::moved makes a copy in O(1) (its text printed
  with gridcraft_formula::print_shifted, the same text as
  shift_relative + print). expr()/expr_arc() still give the expression
  as it reads at the cell; parsed()/parsed_ref() give the shared one
  and the offset. Equality compares text (and array range).
- The evaluator moves the relative parts of the formula's own
  references by the offset as it resolves them (reference_areas);
  names and INDIRECT text are evaluated with no shift, and a LAMBDA
  keeps the shift of the formula that wrote it. precedents_shifted does
  the same for the graph. Hot paths borrow the shared expression
  instead of cloning it, so threads don't contend on its count.
- Copies share: fill, Ctrl+Enter, paste, sort, XLSX t="shared"
  formulas. FormulaSharer re-shares formulas that are the same relative
  to their cells (by R1C1 text): after a structural edit's rewrite, and
  when a file is opened (keys on every processor; JSON formulas get
  parsed once there instead of on every recalculation). The XLSX writer
  converts a shared expression to file form once.
- Tests: print_shifted matches shift_relative + print (7 formula shapes
  x 6 offsets, off-sheet included); shared copies calculate like typed
  formulas at the same addresses (relative and absolute refs, whole
  columns, names, INDIRECT, LET/LAMBDA, SUMIF, OFFSET, XLOOKUP, #REF!)
  after entry, edits, undo and a row insert, and share one expression;
  files reopened from XLSX and JSON share and calculate the same;
  shared copies write the same file text as before.

perf, Apple M2 Pro: a filled formula ~590 -> ~150 bytes; peak memory
of the scale scenario (200,000 rows) 1.0 -> 0.70 GB, of the full 50,000
row run 373 -> 298 MB; fill 200,000 formulas 363 -> ~305 ms. Trade-offs:
a row insert above 200,000 formulas 235 -> ~276 ms (re-sharing), opening
an XLSX ~5% slower (sharing pass).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Opening 1,000,000 rows x 8 columns with 2,000,000 formulas (47 MB,
313 MB of sheet XML) took 10.7 s and peaked at 3.05 GB; 7.0 s of it was
the reader, mostly allocation.

- xml::parse_streaming hands each split element (a row) to its callback
  by reference and reuses it and its children for the next one; names
  and attribute values are overwritten in place.
- The reader stores a row's cells at once (CellStore::set_row: one
  vector of exactly their number) instead of cell by cell.
- A formula that is the one above it in its column moved down (a filled
  column in a file that doesn't mark shared formulas) is recognised by
  its file text (fmla::copy_if_file_text) and shares that formula's
  parse: the reader no longer builds 2,000,000 expressions, and the text
  is printed once.
- The printer writes A1 references straight into its output (no
  temporary strings); whole numbers print without 15-digit formatting.
- Recalculation writes results back in row order and borrows a cell for
  writing (which copies its band if an undo step shares it) only when
  its value changed.
- Tests: column letters match core for A..XFD; whole numbers print as
  the 15-digit path did; set_row keeps the last of a column and skips
  blanks.

1M-row file, gridcraft-cli info, Apple M2 Pro: 10.7 s / 3.05 GB ->
5.5-6.4 s / 2.2 GB; reading 7.0 -> 3.3 s; full recalculation of 2M
formulas 2.15 -> 1.78 s. perf 50,000 rows: open xlsx 460 -> 300 ms, edit
recalculating everything below 51 -> 37 ms, peak 298 -> 274 MB.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ld#205, stage 8)

Every opened workbook was recalculated in full (1.8 s for 2,000,000
formulas), where Excel keeps the values the file stored.

- The XLSX reader lists the formulas whose stored values can't be used
  (ReadReport::recalc): formulas stored without a value, and array and
  dynamic-array anchors (the reader drops their other cells' values).
  A file with fullCalcOnLoad, or without calcPr, asks for a full
  recalculation (None).
- io::open_bytes_with_plan returns that plan; DocState::opened uses
  Calc::load, which builds the graph and recalculates only those, the
  volatile and dynamic-reference formulas, and what depends on them.
  file.open goes through it; other formats recalculate in full.
- Sharing formulas on open skips those already sharing a parse.
- Zip parts of 16 MB or more are deflated at level 3 instead of 6
  (about three times faster on a 327 MB sheet, ~12% more bytes).
- Test: a file whose stored value differs from GridCraft's result keeps
  it and stays live; NOW() is recalculated; a dynamic array spills
  again and a formula reading it is right; fullCalcOnLoad recalculates
  everything.

1M rows x 8 columns, 2M formulas, Apple M2 Pro: opening a calculated
file 6.0 -> ~4.6 s (10.7 s before stage 7); saving 6.6 -> ~4.6 s.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@arhammusheer

Copy link
Copy Markdown
Author

@echelon not to be a bother, but could you please take a look? This is a big PR but improves performance significantly

…ion, .xls reader

Rebasing onto main brought in code written against the old recalc host
and the old Formula fields:

- Iterative calculation (circular references with Maximum Iterations and
  Maximum Change) is rebuilt on the non-recursive pass. A read of a
  formula set aside waiting for its reader gives the previous value
  instead of #CIRC!, and the formulas set aside from it up are the loop.
  After the pass, the formulas the loops reach (statically or through a
  dynamic reference) are evaluated again in plan order, Gauss-Seidel,
  until no loop formula changes by Maximum Change, with the same work
  cap as before. Shared ranges and lookup indexes are dropped between
  iterations, and a recalculation with iteration on uses one thread.
  Range reads count circular reads themselves rather than the #CIRC!
  list, which stays empty while iterating.
- Spill seeds are no longer capped at 65,536 cells (storytold#558), as on main.
- Lookups keep main's Excel-style bisection for approximate matches and
  XLOOKUP binary search; the per-recalc index keeps only the exact-match
  hash, since the bisection doesn't need per-type position lists.
- The printer's shifted form covers trim references and quoted
  sheet-qualified names; a shared copy that moves a spill reference off
  the sheet is written to XLSX as #REF!, not ANCHORARRAY(#REF!).
- File open: password decryption, row auto-fit and formula sharing
  combine with the stale-formula plan (open_bytes_with_plan).
- Calculation Options reports multiThreaded and threads beside the
  iteration settings.
- PassHost::visit_range (SUM/COUNT/AVERAGE folded in place) reads new
  spills through the layered pass state (`all_spills`), as range_values
  does, and declines a block holding a formula still pending in the
  pass (or on a cycle); range_values takes it.
- SUM, COUNT and AVERAGE over a range that many formulas read
  (`SUM(A:A)` filled down) use the pass's shared array again instead
  of walking the cell store each time (`Host::share_range`): the first
  read walks in place, as on main, and later reads share. 1,000 x
  SUM(A:A) over 50,000 rows: 119 -> 20 ms (main: 1,085 ms). Test:
  aggregates over a shared range match the walked one, with text,
  logicals, blanks and errors.
- The .xls reader checks that a formula parsed with Formula::parsed_ref
  (the expression field is private since shared formulas).
- Tests for upstream features meeting this work: relative defined names
  in shared (filled) formulas, and XLOOKUP returning references over a
  shared, indexed table on one and four threads.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@Harrison-Booth

Copy link
Copy Markdown
Contributor

I ran this PR against 247 real-world workbooks, release builds of gridcraft-cli, this branch vs main at your merge base (84acf62). Each ran info and check on both builds, one at a time, 900 s timeout. Timings are a bit noisy since other jobs shared the CPU.

Speed and memory (239 workbooks that check on both): total open time 3,579 s to 522 s (p90 9.6 s to 4.0 s), peak memory p90 817 MB to 338 MB. The workbook that timed out on main now checks in 43 s, and the 4 that overflow the stack on main now complete.

Correctness is the blocker. Compared with Excel's saved values:

Threads: results change with thread count. On the 26 workbooks whose results changed, check with one thread vs all cores gives a different number of differing cells on 12. The regressions also show at one thread, so the new evaluation order seems to be the cause, not only threading.

Iterative calculation: one macro-enabled workbook (86k formulas, iteration on) opens in about 190 s on main and 650 to 840 s here. It may be related to the slow .xlsm opens in #891.

I can't share the files, but I can run a build or diagnostic on them, or try to reduce the #CIRC! case to a synthetic repro.

🤖 Generated with Claude Code. Reviewed by @Harrison-Booth before posting.

arhammusheer and others added 2 commits October 15, 2026 23:34
The stack-based evaluator treated every formula on its stack as part of a
cycle, including ones only queued beside the reader. With =SUM(A2:A4) where
A4 reads A3, all three were queued, A4 ran first, read the queued A3 and got
#CIRC!. Formulas whose inputs aren't known before calculation (defined names
over ranges, INDIRECT/OFFSET, formulas behind a real cycle, formulas handed
back by worker threads) took this path, so results also changed with the
thread count.

Only formulas being evaluated or waiting on what they read count toward a
cycle now; a formula queued twice is evaluated once.

Tests: two repros (INDIRECT, a defined name over a chain), and a randomized
test comparing a full recalc at 1 and 8 threads with entering the cells one
at a time (GRIDCRAFT_ORDER_FUZZ_SEEDS soaks it). Both fail on the previous
evaluator.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
After the first pass, iteration re-evaluated every formula statically
reachable from a loop plus every dynamic-reference formula (INDIRECT,
OFFSET) and everything they feed, on every pass. On a workbook with many
dynamic references that is the whole workbook each pass: a 60k-formula
benchmark with 40k OFFSET/INDIRECT formulas and a 2-cell loop recalculated
in ~740 ms against ~140 ms before storytold#205.

Each formula now records whether it read a previous value, directly or
through a formula, a spilled cell or a shared range that did (shared
ranges carry the flag, so lookups over them do too). Only those formulas
are evaluated again, as before storytold#205. The benchmark now takes ~50 ms with
bit-identical results.

Tests: a loop read through INDIRECT, a range read many times (shared, with
INDEX over it), a chain of readers and a spill. Starting unconverged, it
fails if the shared-range flag is dropped or only loop members iterate
(spill readers are also recalculated by the spill round, so those two
lines are defensive). The randomized order test now also covers names
over ranges, computed OFFSET and INDEX.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@RedBearAK

Copy link
Copy Markdown
Contributor

Hi @arhammusheer. I have a PR open, #1499, that touches some of the same code as this one. I don't think the two compete. They speed up recalculation in different ways, and I think they would work well together. I wanted to explain how they fit and point out the one spot that needs care if both are merged.

What each PR does

This PR makes recalculation do its work faster: it orders formulas properly, spreads them across processor cores, and adds lookup indexes.

#1499 makes recalculation do less work. On main, any formula that uses a defined name (a named range, or a name holding a LAMBDA) is treated as dynamic and recalculated after every edit anywhere in the workbook, just like INDIRECT or OFFSET. #1499 works out what each name actually refers to when the dependency graph is built, so those formulas become ordinary dependencies and are recalculated only when their inputs change. This PR still treats names as dynamic, so the two changes don't overlap in what they fix.

On a workbook I use with about a million cells and 49,000 formulas that call LAMBDA names, #1499 took opening from 447 seconds to 109 seconds, and an unrelated one-cell edit from about 10 seconds to about 1 second. With this PR alone, all 49,000 of those formulas would still be recalculated on every edit, just spread over more cores. With both, they wouldn't need recalculating at all for most edits, and the work that remains would be spread over cores.

The one spot that needs care

#1499 resolves each defined name once per graph build and remembers the answer in a NameCache, which it fills in as it goes. Without that, a workbook where names refer to other names many times over can take exponentially long to build the graph. Because the cache is written to while the graph is built, a single cache can't be shared between threads.

This PR builds the graph with par::filter_map, which splits the formulas into one chunk per thread. The simplest way to combine the two is to give each thread its own cache, created at the start of its chunk. Each thread then resolves every name it meets once, and the protection against exponential cases still holds. The cost is at most one copy of the name table per thread, which is small.

In code, that means a variant of filter_map that hands each thread its own piece of state:

/// Like `filter_map`, but each thread first creates its own `state` with `init`
/// and passes it to every call it makes.
pub fn filter_map_init<T, S, R, I, F>(items: &[T], init: I, f: F) -> Vec<R>
where
    T: Sync,
    R: Send,
    I: Fn() -> S + Sync,
    F: Fn(&mut S, &T) -> Option<R> + Sync,
{
    // The same body as `filter_map`, except that each chunk runs
    //     let mut state = init();
    //     part.iter().filter_map(|x| f(&mut state, x)).collect()
    // and the single-thread paths do the same with a single state.
}

and the graph build would call it like this:

let nodes = crate::par::filter_map_init(&formulas, NameCache::default, |names, &(k, cell)| {
    // as now, but finding precedents through the thread's own name cache:
    // precedents_shifted_with(wb, k.0, e, shift, names)
});

The places that find precedents on a single thread would use one NameCache for the whole loop, as #1499 does in rebuild and cells_changed.

Whichever PR is merged second has to make this change, if both are ultimately merged. It's about ten lines plus the plumbing, and I'm happy to do it on my side if this PR goes in first. I'm also putting together a local branch with both PRs combined, to check that this works and to measure the two together on large workbooks. I'll post the numbers here if that's useful.

@RedBearAK

Copy link
Copy Markdown
Contributor

@arhammusheer

Following up with the numbers from combining this PR with #1499. The short version: together they open my workbook about 30 times faster than main, and an edit afterwards that took about two minutes on main now takes a fraction of a second.

The combined branch is on my fork: fast_recalc_test_pr208_pr1499_pr2020_pr2033. It's current main with this PR merged in, then #1499, plus two small unrelated fixes (#2020 so cargo xtask ci passes on macOS, and #2033). Merging this PR into current main needed a few small conflict fixes (the new csvAsText option when opening files, range fill, and one upstream test that reads a field this PR made private). Merging #1499 on top needed the per-thread name cache I described above. That turned out as expected: a par::filter_map_init that gives each thread its own NameCache. cargo xtask ci passes on the combined branch (1,698 tests).

The workbook

A real file from work, not a generated one: 18 sheets, about 1.15 million cells, and 51,000 formulas, most of them in one sheet with 12,000 rows. The formulas call 14 LAMBDA names. The file asks for a full recalculation on open (fullCalcOnLoad="1"), so cached values don't help, and every formula is computed when it opens.

Timings

Measured with release builds of gridcraft-cli on an Apple M3 (4 performance and 4 efficiency cores), opening the file and then making one edit:

Build Open One-cell edit after opening
main 114 s about 115 s
#1499 alone 13 s almost nothing
this PR alone 25 s about 25 s
both about 4 s 0.17 s

The edit was a date in a lookup table that the LAMBDAs read, so it does change many results. An edit to an unrelated empty cell gave the same pattern.

Each PR on its own helps a lot, and together they do much better than either. #1499 means only the formulas whose inputs changed are recalculated. This PR means the formulas that do need calculating are spread across cores and ordered properly. With this PR alone, every formula that calls a LAMBDA name is still recalculated after every edit, which is why its edit time stays near its open time.

In the desktop app it feels the same way. The file opens in about 4 seconds (about 19 seconds with #1499 alone, and Excel takes about 5), a version of the same workbook with lighter formula usage opens in about 2 seconds, and changing the filter criteria on a sheet whose formulas filter the main table shows the new results instantly.

Correctness

All 51,014 formula results after opening are identical across the four builds. After the lookup-table edit, all one million cells in the combined build match this PR alone, which recalculates everything and so makes a good reference.

Thanks for the work on this; it's a big step up for large workbooks.

@Harrison-Booth

Copy link
Copy Markdown
Contributor

Thanks for the detailed write-up and for the follow-up fixes. I read the diff (eleven commits), the description and the thread. I did not build or run it. No workflow run exists for head 3e3b238, so cargo xtask ci has not run on it.

What it covers. Non-recursive topological recalculation, shared ranges with lookup indexes, per-level threading (levels of 512 or more formulas; multi_threaded / threads saved as concurrentCalc / concurrentManualCount), background recalculation in the desktop app (Session::set_background_calc, poll_calc), sorted-vector rows and shared formulas for memory, the XLSX reader reusing row elements, and opening with the cached values.

What looks right.

  • Workers hand back formulas that read a still-pending cell and formulas that use RAND, so results do not depend on the thread count. A worker that fails to start or panics sends the level back to one thread, and a failed background job falls back to a full recalculation on the calling thread.
  • The two commits after review (3a5ee32 false #CIRC!, 3e3b238 iterative calculation) address the failures reported in the 247-workbook run above, and the 34-workbook comparison against main is a good check for that class of bug.
  • Commands other than moving around wait for a background recalculation, so undo snapshots always see final values.

Not yet shown.

  • The 247-workbook comparison in this thread ran against 84acf62 and the commits before those two fixes. It needs a repeat on the fixed head once it is merged with current main, in particular the 22 workbooks that regressed and the one-thread versus all-cores check. I can run that.
  • .xlsm open time with iteration on (650–840 s there against about 190 s on main) was reported before 3e3b238; the description gives a synthetic figure only.

Points from reading.

  1. par::filter_map uses available_parallelism() (up to 64 threads) and does not look at calc.multi_threaded / calc.threads, so a workbook with multi-threaded calculation switched off still spawns threads for the graph build and formula rewrites. Passing the thread count in would fix that, and RedBearAK's filter_map_init suggestion for Recalc: defined names as static precedents, per-pass name cache, TEXT fast path #1499 touches the same function.
  2. Opening with cached values means anything GridCraft computes differently from the saving application stays as saved until a precedent changes. It matches Excel and is documented; it is worth a line in the release notes and a check that fullCalcOnLoad files and Ctrl+Alt+F9 behave as described.
  3. It does not change the 512 MiB part cap, so the workbook in Unable to open or work with large XLSX files (~100 MB) #175 (worksheet part about 612 MiB inflated) still fails to open; Gap #2: stream large XLSX reads and parallelize independent recalc #2122 raises that cap for sheets and shared strings.

Overlap and conflicts.

To land. Merge main, get the workflow approved, repeat the corpus comparison on the fixed head, and settle the scheduler overlap with #2122. Landing the stages in order (stages 1–2 first: topological order, which fixes the false #CIRC!, plus lookup indexes) would make review easier if the size is a problem.

@Harrison-Booth

Copy link
Copy Markdown
Contributor

Follow-up to my earlier review, which was from reading only. I have now checked the calculation claims that Excel can settle in real Excel (Microsoft 365, version 16.0 build 20430) through its automation interface, on new temporary workbooks. The formulas and values are taken from the tests in crates/calc/src/tests.rs in this PR. I did not build or run GridCraft, so "PR expectation" is what the test asserts, not a measured result.

Claim Excel result PR expectation
Iteration off: A1=B1+1, B1=A1+1, C1=A1*2, D1=5, E1=E1+1 (cycles_are_circular_and_so_is_what_reads_them) No error value. Excel reports the circular reference (E1) and leaves numbers: A1=1, B1=0, C1=0, E1=0, D1=5. ISERROR(A1) is FALSE, ISNUMBER(C1) is TRUE A1, B1, C1, E1 are #CIRC!, D1=5 differs (#CIRC! is how main already shows a cycle; the PR keeps it)
Break the cycle with B1=D1 A1=6, B1=5, C1=12 6, 5, 12 match
Iteration on (100 iterations, change 1e-9): K1=(K2+10)/2, K2=K1, readers INDIRECT("K"&1)*2, M1+1, INDEX(K1:K100,2), SUM(K1:K100)*row (iterative_calculation_follows_every_kind_of_read_of_a_loop) Loop settles at 10 (9.99999999996 after one F9 with the cells entered while calculation was manual, exactly 10 in automatic mode). Readers: M1=20, M2=21, M3=10, S3..S8 = 20*row K1=10, M1=20, M2=21, M3=10, S=20*row within 1e-6 match
Same, with a spill seeded from the loop (P1=SEQUENCE(2,1,K1), R1=P2*2, R2=SUM(P2:P3)*2), then K1=(K2+20)/2 R1=22, R2=22; after the edit K1=20, P2=21, R1=42, R2=42 22, 22, then 20, 21, 42, 42 match
Reads queued together are not a cycle: A2=D1, A3=A2+1, E1=INDIRECT("A2")+INDIRECT("A3") E1=3, no circular reference reported 3, circular empty match
Same through a name: Block=Sheet1!$A$2:$A$4, C1=SUM(Block), A2=1, A3=A2+1, A4=A3+1 C1=6, no circular reference 6 match
Long reverse chain, 20,000 cells (C1=C2+1 ... C20000=1) C1=20000, no circular reference; after C20000=10, C1=20009 n and n+9 match
Chain of 5,000 cells through a name (A(i)=A(i+1)+Step, Step=1) A1=5000 5000 match
SUM/COUNT/AVERAGE of A:A and SUM(A1:A200) over text, logicals and blanks (aggregates_over_a_shared_range_match_the_walked_one) 8080, 80, 101, 8080 in all 50 copies. With A7=1/0: SUM, AVERAGE, SUM(A1:A200) give #DIV/0! (COUNT stays 80) same values, #DIV/0! match
Indexed lookups (indexed_lookups_match_scanning_ones), the 12 cases 11 match. =MATCH(0.1+0.2,$A$1:$A$100,0) with A10 =0.3 gives #N/A expects 10 differs
Open with the cached values (the "matches Excel" statement in the PR) I saved a file in Excel, then changed a cached value in a copy (B1=A1*10, cached 999). Excel opens it showing 999 and C1 unchanged (21). F9 keeps 999; Ctrl+Alt+F9 gives 20. Editing A1 recalculates the dependents. With fullCalcOnLoad="1" Excel recalculates on open (B1=20). With the calcId attribute removed from calcPr, Excel also recalculates on open cached values are kept until a precedent changes; fullCalcOnLoad and Ctrl+Alt+F9 recalculate match

Two details from the lookup row, for whoever reads that test. (0.1+0.2)=0.3 is TRUE in Excel and COUNTIF(A1,0.1+0.2) and MATCH(ROUND(0.1+0.2,15),...) both find the cell, but MATCH, XMATCH and VLOOKUP with exact match compare the bits and return #N/A (I checked all three). The test's comment says all copies must agree with what a scan finds, so it follows main's scan, which is tolerant; Excel's lookups are not. And the last row shows that a file that has no calcId is fully recalculated by Excel on open, so whether GridCraft writes one decides what Excel does with the cached values GridCraft left in the file.

Not checked. Thread-count independence and the parallel scheduler, background recalculation, undo snapshots, the 11-commit timings and memory figures, the 34 random workbooks and the 247-workbook comparison (these need the corpus, and the false #CIRC! cases came from files I do not have), and par::filter_map's use of cores are not observable through Excel. For context, this Excel has multi-threaded calculation on, automatic thread mode and 20 threads (Application.MultiThreadedCalculation).

@Harrison-Booth

Copy link
Copy Markdown
Contributor

Comparison of the four open PRs that touch recalculation threading, for whoever decides how they fit together: #2319, #1499, #208 and #2122. I read the four diffs, descriptions and threads, checked conflicts against current main (89ac453) with git merge-tree, read the CI state from the forge, and checked the settings claims in real Excel. I did not build or run GridCraft. None of the four changed in the last day, so none was skipped. This is a comparison only.

#2319 #1499 #208 #2122
Author RedBearAK RedBearAK arhammusheer ppoinot
Size +4,007 / −182, 26 files, 4 commits +988 / −66, 12 files, 1 commit +3,626 / −567, 46 files, 11 commits +2,695 / −276, 30 files, 1 commit
Threads? Yes: levels of 512+ formulas on workers, merged in one thread's order No: names as static precedents, per-pass name cache, TEXT fast path Yes: levels of 512+ formulas on workers (par.rs), plus background recalculation in the desktop app Yes: "flat waves" of 64+ formulas with no dirty precedent, at most 8 threads
Setting multiThreaded, threads in the workbook, saved as concurrentCalc / concurrentManualCount none same two settings and attributes as #2319 maxWorkers 1–8 or null, not saved in the file
Other scope planner for dependency levels (also changes single-thread order and RAND sequence) the first commit of #2319 (#2319 is stacked on it) lookup indexes, shared ranges, sorted-vector rows, shared formulas, cached values on open, XLSX reader changes streaming XLSX reads, sparse whole-row/column Fill, bench_100k, desktop-theme change
Tests 2 and 8 threads compared with 1 on every case, rules removed in turn, two random-workbook fuzzers, cargo xtask ci runs calc and engine suites threaded unit tests at the depth and nesting bounds, hidden-name readers, TEXT sweep over 200,000 integers tests in calc/tests.rs, extended perf.rs example; the 34-workbook comparison with main is described but its generator is not among the changed files parallel vs serial dynamic results, worker-panic fallback, cap of 1, maxWorkers, spill SUM(A:P); the bench uses two formula shapes that take the scheduler's own small evaluator
CI on head CI and SonarCloud passed (9e083ee) CI and SonarCloud passed (b79f5da) CI is action_required; nothing has run on 3e3b238 CI is action_required; only CodeRabbit reported (7dfd66c)
Conflicts with main calc/eval.rs, calc/recalc.rs, engine/cmd/extra.rs, 3 docs calc/eval.rs, calc/recalc.rs, engine/cmd/extra.rs, 1 doc 32 files across calc, engine, formula, functions, model, xlsx, ui-egui, docs 19 files, including calc, xlsx reader, engine/cmd/edit.rs, ui-egui, xtask

Approach. #2319 and #208 both split the dirty set into dependency levels and run wide levels on workers, with the same 512 threshold and the same file attributes. #2319 keeps the result identical to one thread by construction (workers read only state that cannot change under them, and are re-run when an array changes shape) and says it left #208's lookup index, cell-store and shared-formula changes out on purpose. #208 carries those as well as the background recalculation, so it is the largest change. #2122 uses a different, narrower scheduler (formulas with no dirty precedent only) and a different setting. #1499 is not threading; it is in this set because #2319 builds on it.

Overlap. #208, #2319 and #2122 each add a scheduler in calc/recalc.rs and a thread setting, and the settings differ (multi_threaded + threads saved to the file, against maxWorkers not saved). #208 and #2319 write the same calcPr attributes in the same place in xlsx/write.rs. #208 and #2122 also both change the XLSX reader, store.rs and engine/cmd/edit.rs.

Excel checks (Microsoft 365, 16.0 build 20527, through its automation interface, new temporary workbooks, closed without saving). This Excel has multi-threaded calculation on, automatic mode, 20 threads; I restored that after each run.

Not checked. #2319's statement that Excel records what INDIRECT and OFFSET read and reorders its chain is not observable through Excel's automation interface, so I did not check it. Also not checked: thread-count independence of results, any timing or memory figure, the 247-workbook and 34-workbook comparisons, background recalculation, streaming limits, and the benchmarks. Earlier Excel checks on #208 and #2122 are in my two comments on each of those PRs.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants