local chDB checkpoints for Maple’s embedded chDB store

[robbiemu]

Summary

This PR adds the hardened local chDB checkpoint foundation for Maple’s embedded chDB store. It provides crash-conscious checkpoint creation, validated restore, and opt-in dirty-store recovery, and addresses the review feedback around detached --on-dirty-store forwarding, backup-configuration error classification, quarantine/operation identity collisions, and restart-safe recovery boundaries.

The goal is to give local-mode Maple a recoverable checkpoint path for dirty shutdowns without trying to copy a live chDB data directory. Instead, this uses chDB/ClickHouse-native BACKUP DATABASE ... TO Disk(...) and RESTORE DATABASE ... FROM Disk(...), then validates the backup in a sacrificial chDB instance before promoting it as restorable.

This PR includes:

• maple start --chdb-config-file <path>
• maple checkpoint
• maple restore --yes
• maple start --on-dirty-store=<wipe|fail|restore-checkpoint>
• checkpoint manifest validation for chDB version and schema fingerprint
• unit coverage for checkpoint config generation, checkpoint promotion, and manifest compatibility

Architecture

Maple still runs chDB embedded/in-process. This PR does not introduce a separate database server.

The checkpoint flow is:

1. maple start may be given a ClickHouse config file that enables backups:

   <clickhouse>
     <backups>
       <allowed_disk>default</allowed_disk>
       <allowed_path>backups</allowed_path>
     </backups>
   </clickhouse>

2. maple checkpoint asks the running local server to execute:

   BACKUP DATABASE default TO Disk('default', 'backups/building/backup')

3. The CLI opens a separate scratch chDB instance and restores that backup using a src disk pointed at the live data dir.

4. If restore validation succeeds, Maple writes a manifest and promotes:

   backups/building -> backups/current
   backups/current  -> backups/previous
   

5. maple restore --yes restores only from backups/current, into a sibling restore dir first. The dirty/original store is quarantined only after the restored store has validated.

Behavioral Cases

Normal checkpoint

If Maple is running with backup support enabled, maple checkpoint creates backups/current, validates it in scratch chDB, writes manifest.json, and keeps the prior checkpoint as backups/previous.

Missing backup config

If the server was not started with backup support, maple checkpoint now reports an actionable error instead of surfacing raw chDB Code: 318.

Manual restore

maple restore --yes:

• requires backups/current/backup
• reads backups/current/manifest.json
• refuses mismatched chdbVersion
• refuses mismatched schemaFingerprint
• restores into data.restore-building
• validates the restored store
• quarantines the old data dir
• moves the restored dir into place
• rewrites Maple’s local store marker

Dirty store on startup

The default policy is now fail, preventing silent data loss after an unclean shutdown:

maple start

Explicit alternatives are:

maple start --on-dirty-store=wipe
maple start --on-dirty-store=restore-checkpoint

The restore mode is opt-in. It attempts to restore from the last promoted checkpoint instead of wiping the dirty store.

Incompatible store

Compatibility checks run before dirty-store recovery. If the existing store was written by an incompatible chDB build, Maple refuses before attempting recovery.

Bad or old checkpoint

A checkpoint with an incompatible chDB version or schema fingerprint is rejected before restore.

Validation

Local validation performed:

• bunx oxfmt on touched files
• bunx oxlint on touched files
• bun --filter=@maple/cli typecheck
• bun test apps/cli/test
• native chDB smoke test:
  • maple start
  • maple checkpoint
  • maple restore --yes
  • restart from restored store

Key Adoption Questions

Should this be split?

This draft currently includes checkpoint creation and dirty-store restore behavior. We may want two PRs:

1. backup config + maple checkpoint
2. maple restore + dirty-store recovery

The second part changes startup recovery semantics, even though the new behavior is opt-in.

Should users manage the chDB config?

This PR exposes --chdb-config-file. That is useful and explicit, but probably not the final UX. Longer term, Maple may want to generate/manage the standard backup config automatically.

Disk and quota implications

A checkpoint is another copy of the local OLAP store, not a tiny journal. Keeping current and previous means operators should expect additional disk usage, potentially near 2x depending on compression and data shape.

This matters especially for containerized/local deployments with fixed volume quotas.

Restore-time expectations

Operators need to size their local store around acceptable recovery time.

If the checkpoint is 4 GB, 8 GB, 64 GB, etc., how long can startup recovery take? The answer should influence recommended volume size, checkpoint frequency, and whether automatic restore on startup is acceptable.

RAM pressure

This approach does not add a separate OLTP twin database, but checkpoint validation/restoration opens another chDB instance during the operation. We should confirm memory behavior at realistic store sizes, especially on 16 GB machines.

If we later choose an OTLP/OLTP mirror architecture, that would be a separate design with much larger steady-state disk and memory implications.

Promotion atomicity

The current building/current/previous promotion is pragmatic but not perfectly transactional across every crash point. A future hardening pass could use versioned checkpoint dirs plus an atomic pointer, or add startup reconciliation for stranded building/.

Should restore ever be automatic?

--on-dirty-store=restore-checkpoint is convenient, but it is still an implicit recovery action during startup. Maintainers may prefer manual maple restore --yes only, or may want restore mode gated behind an explicit operator flag.

Fixes #113

---

---

Publication update — 2026-07-09

This PR has now been updated from the original draft implementation to the hardened checkpoint foundation used by the dependent archive PR.

• Current PR local chDB checkpoints for Maple’s embedded chDB store #129 head: 3278b595aea5583b1fcba946b55c96a65a1fa0bd.
• The live PR branch had already merged newer main, so the final publication branch preserves that live base and cherry-picks the five approved Phase 1 checkpoint hardening commits on top.
• git range-diff 3fa344f3..178b8c49 bbe6be33..3278b595 showed the published range is patch-equivalent to the locally approved checkpoint head 178b8c49.
• Dependent draft archive PR Local telemetry archives for Maple's embedded store (depends on #129) #194 is now open: Local telemetry archives for Maple's embedded store (depends on #129) #194. It builds on this checkpoint foundation and should not merge before local chDB checkpoints for Maple’s embedded chDB store #129.

Additional hardening now included in this PR:

• strict, versioned checkpoint state selection instead of directory-name inference;
• immutable checkpoint snapshot directories;
• durable operation records and reconciliation for checkpoint create, restore, reset, retirement, and cleanup boundaries;
• reset/wipe behavior that preserves checkpoint registry, pins, operations, quarantine, and recovery evidence;
• fail-closed handling for malformed, legacy, ambiguous, or interrupted checkpoint state;
• symlink/path-escape and operation-identity protections; and
• expanded fault-boundary coverage for retirement, live restore, and destructive reset paths.

Final local validation for the approved Phase 1 range included formatting, lint, typecheck, 66/66 CLI tests, git diff --check, two consecutive expanded native checkpoint/recovery/reset smokes, and independent maintainer-style approval.

---

Updates

Update 2026-07-11

A follow-up to the dependent PR #194 review comment closed a missing lifecycle proof in the original checkpoint validation. The original restore path validated the checkpoint in a temporary chDB instance, but did not prove that a newly started Maple process could reopen the published store.

At the pre-fix #129 head (3278b595), I reproduced the failure independently: restore validation succeeded, but a later fresh process intermittently crashed in ClickHouse metadata loading (DB::AsyncLoader / RWLockImpl) or failed with ASYNC_LOAD_WAIT_FAILED. A never-restored control opened successfully 20/20 times. Limiting chDB metadata loading to serial pools passed 50/50 restored-store reopen attempts, which supports startup loader concurrency as the cause.

Follow-up commit 4fdcbb79 now:

• disables asynchronous database/system loading and uses single-entry loader pools;
• closes the restoring process before publishing the restored directory;
• re-executes Maple in a separate process to validate the restored store;
• fails closed if that fresh-process validation does not match the restoring process; and
• expands the native smoke test to repeat fresh reopens after restore, dirty recovery, checkpoint creation, and reset/restore.

The exact committed bundle passed the strengthened lifecycle, including all repeated fresh-process reopen cycles, 69/69 CLI tests, typecheck, lint, formatting, and Turbo test.

Restore reliability under repeated checkpoint validation

Extended testing of the checkpoint lifecycle reproduced an intermittent chDB failure inside scratch RESTORE DATABASE (recursive_mutex lock failed). The restore path uses a separate 16-thread worker pool, so restore work is now serialized with --restore_threads=1 in addition to persisted-table loading.
The corrected head passed 300/300 focused checkpoint restores, 10/10 complete native checkpoint lifecycles, all 69 relevant CLI tests, and CLI typecheck.

[robbiemu]

still looking at it btw. The changes requested happen to go well with a second change I want, at least for a fork if not for this project, so I am actually trying. What pullfrog wants is non-trivial in the details, but I think we've got it.

I have to say though, that fully satisfying the direction desired here probably veers away from the design principles I anticipate when I read "Maple" as the name of the project.

[devin-ai-integration]

✅ Devin Review: No Issues Found

Devin Review analyzed this PR and found no potential bugs to report.

View in Devin Review to see 2 additional findings.

[robbiemu]

Thanks for the review. The published update addresses the detached dirty-store forwarding bug, backup-config error classification, quarantine/operation identity collisions, and the broader crash-consistency/restart-safety concerns with immutable checkpoint state, durable operation records, and reconciliation coverage.

One caveat: the Pullfrog workflow SHA warning was noted as outside this PR’s feature scope. I would not claim that one is fixed by this PR.

[robbiemu]

The fresh-process concern raised on #194 was initially a hypothesis, so I reproduced it against the pre-fix #129 head before treating it as a defect.

The original restore validation opened the restored directory in a temporary chDB instance and then published it, but did not test a newly started Maple process. At the #129 head, restore itself succeeded, while a later fresh process intermittently crashed in ClickHouse metadata loading (DB::AsyncLoader / RWLockImpl) or failed with ASYNC_LOAD_WAIT_FAILED. The same never-restored store opened 20/20 times, and restored stores opened reliably when metadata loading was forced through serial pools.

The fix in 4fdcbb79 makes the lifecycle explicit: restore closes its connection, a separate Maple process reopens and validates the restored directory before publication, and chDB startup uses deterministic serial metadata loading. The native smoke now repeats this boundary after restore, dirty recovery, checkpoint creation, and reset/restore. The exact committed bundle passed all of those cycles, including 100/100 additional fresh opens against a patched restored store.

[robbiemu]

Extended validation reproduced the intermittent mutex failure directly on this branch. The first eight complete native checkpoint lifecycles happened to pass, but a larger run reproduced the failure twice within the next four lifecycles. Instrumentation located both failures inside the scratch database's RESTORE DATABASE statement, before validation or the new fresh-process reopen check.

We then tested whether this was simply a readiness delay between creating the scratch database and restoring into it. Delays reduced the observed frequency but did not eliminate it: zero delay failed 2/50 focused restores, while 1 ms, 5 ms, 10 ms, and even 100 ms each still failed under larger cohorts. A stronger create/insert/read/DROP ... SYNC readiness probe also completed successfully and was still followed by the same restore failure, so a fixed sleep or ordinary catalog readiness check was not a reliable fix.

Inspecting the exact chDB v26.1.0 source showed that RESTORE uses its own worker pool, defaulting to 16 threads. That pool is independent of the foreground/background table-loader pools addressed earlier in this PR. Commit d02fbbed therefore adds --restore_threads=1 to Maple's embedded chDB arguments and regression coverage for both configured and unconfigured startup.

Validation of the zero-delay production candidate:

• 300/300 focused checkpoint restores passed.
• 10/10 complete native checkpoint lifecycles passed, including explicit restores, dirty-store recovery, reset/restore, and repeated fresh-process reopen checks.
• 69/69 relevant CLI tests passed.
• CLI typecheck passed.

This validates checkpoint creation, restoration, recovery, reset, and repeatable fresh-process reopen behavior under substantially more repetition than the failing baseline. Because the original chDB failure was intermittent, these results are validation evidence rather than a claim that an upstream defect is mathematically impossible.

@Makisuo One caution: restore_threads=16 is chDB/ClickHouse’s normal default, not a value introduced here. The failure appears specific to concurrent restoration of this schema’s materialized-view dependency graph in the embedded chDB v26.1.0 workload. We observed that reducing the pool to one eliminated the failure throughout our stress validation, but we have not identified the exact upstream mutex or proven that all multithreaded restores are unsafe. This change therefore deliberately trades restore parallelism for reliability until the underlying chDB behavior can be isolated further.

[robbiemu]

woohoo!

[Makisuo]

very amazing work @robbiemu going to get your other PR merged tmrw, after doing some more testing