Skip to content

One key-wrapping scheme for the local DB key and vault keys - #1767

Closed
joepio wants to merge 4 commits into
claude/project-thread-mes3rjfrom
claude/project-thread-mes3rj-keywrap
Closed

joepio wants to merge 4 commits into
claude/project-thread-mes3rjfrom
claude/project-thread-mes3rj-keywrap

Conversation

@joepio

@joepio joepio commented Sep 24, 2026 •

Copy link
Copy Markdown
Member

Requested by Joep · project thread

Before: the agent secret protected keys in two unrelated ways. The local database key (localDbKey.ts) was AES-GCM under an HKDF key derived from the raw private key; drive vault keys (vaultWrapKey) were a Rust SecretEnvelope (XChaCha20-Poly1305, KEK = BLAKE3 over the agent's vault proof). The Rust envelope also carried recovery-code, password and WebAuthn-PRF wrappers that no production path ever wrote.

After: both keys use the same SecretEnvelope with an agent-secret wrapper. Existing local DB keys keep opening: the old record is read on sign-in and rewrapped into the new one. The Rust envelope only has the two wrappers that are used (agent-secret, node-key).

How: ensureDbKeyOnSignIn now gets the vault proof (already computed in storeSecret) alongside the private key, and loads the same wasm vaultWrapKey/vaultUnwrapKey that the vault uses. SecretEnvelope loses RecoveryCode, Password, WebauthnPrf, Unlock::Secret, the recovery-code generator and the Argon2 fields; a #[serde(other)] Unknown kind keeps old or foreign envelopes parseable.

What was checked first

  • RecoveryCode / Password / WebauthnPrf / Unlock::Secret / generate_recovery_code: only referenced in secret_envelope.rs tests. The wasm bindings only call create / unwrap_secret with AgentSecret; db.rs only uses NodeKey + Unlock::Kek. So no stored envelope in the wild was written with those kinds.
  • Account recovery is not this envelope. recovery.ts has its own "v2" format (AES-GCM, wrapper_type rows recovery-code / webauthn-prf) that atomic-saas stores and parses. That format is untouched by this PR, as is the Argon2id wasm export it uses (argon2idDeriveKey, still in vault/keys.rs).
  • atomic-saas stores vault key envelopes opaquely (it only counts wrappers). Wire names agent-secret / node-key are unchanged and pinned by a test.

Migration (local DB key, in IndexedDB)

record key status
v1 atomic.clientdb.wrapped-key.<fp> AES-GCM, legacy. Read and rewrapped; not deleted
v2 atomic.clientdb.wrapped-key-v2.<fp> {version: 2, envelope} SecretEnvelope, written from now on

On sign-in: v2 present → unwrap. Else v1 present → unwrap with WebCrypto, set the session key first, then write v2 (after checking the new envelope opens to the same key). Else session key only (pre-feature) or nothing → wrap / generate into v2.

Failure modes

  • Wasm does not load at sign-in (offline, timeout after 20 s): a v2 record is never discarded for this; sign-in's key step throws, the existing DB stays closed (as today when no key is available) and the next sign-in retries. With no v2 record, a v1 record still opens and stays v1. For a brand-new key, a v1 record is written as a fallback so the key is never left without a wrapped copy. This is the only case where v1 is still written.
  • Corrupt v2 record: discarded, then v1 is tried; if v1 opens, v2 is rewritten. If neither opens, a fresh key is generated. That was already the behaviour for a corrupt record, and the old cache is unreadable in that case either way.
  • Rollback to a build that only knows v1: agents that had a v1 record keep it (the migration never deletes it), so rollback still opens their cache. Agents whose first wrapped key was created after this change have only v2; on a rolled-back build, a sign-out followed by sign-in would generate a new key and lose that local cache. Server data is unaffected. Deleting the v1 records should wait for a later release.
  • Sign-in latency: the v2 path needs the wasm bundle on the main thread during sign-in. The ClientDb worker and recovery already load the same file, so it is usually cached.
  • Key reuse: the local DB envelope and drive vault envelopes now share one KEK (BLAKE3 over the vault proof). XChaCha's 192-bit random nonces make that safe, and both already required the same agent secret.
  • Rust: an envelope with a wrapper kind this build does not know still opens through the kinds it knows, but add_wrapper / remove_wrapper refuse it. Writing it back would store the placeholder in place of the real kind.

Tests

  • cargo test -p atomic_lib --features "db,db-sled,ws,iroh,config,rdf,discovery,ring" --lib -- vault:: db::test: 129 passed. That covers the 18 secret_envelope tests and the db.rs node-key tests. New tests: wire names are unchanged, and an envelope that carries removed wrapper kinds (with kdf/salt) still opens and rejects the wrong proof.
  • The full atomic_lib suite did not finish here. Two runs were cut short because the shared disk filled up. It needs a CI run. cargo clippy was also skipped for disk reasons.
  • cargo check -p atomic-wasm --target wasm32-unknown-unknown: ok.
  • browser/data-browser: vitest run gives 1306 passed; the only failure is the known scripts/integration-mcp.test.mjs. tsc --noEmit is clean, and pnpm run -r lint is clean.
  • New localDbKey.persistence.test.ts cases:
    • An old-format v1 record unlocks, is rewrapped as v2 and stays in place.
    • New writes are v2 only.
    • A different secret cannot get the key, from either v1 or v2.
    • A v2 record survives a failed wasm load and opens on retry.
    • Without wasm a v1 fallback is written and migrated later.
    • A corrupt v2 record is recovered from v1.
  • The e2e signout-signin-data.spec.ts now expects the v2 prefix. I did not run it here.
  • Recovery (recovery.ts, recovery-enrollment.test.ts) is unchanged and its tests pass.

The local ClientDb key now wraps into the same SecretEnvelope (agent-secret
wrapper, XChaCha20-Poly1305) as drive vault keys. Legacy AES-GCM v1 records
are read on sign-in and rewrapped as v2; they are kept for rollbacks and only
written when the wasm bundle cannot load.

Remove the unused recovery-code, password and webauthn-prf wrappers from the
Rust SecretEnvelope. Envelopes carrying unknown wrapper kinds still parse and
open through the kinds this build knows, and refuse to be rewritten.
@gitguardian

gitguardian Bot commented Sep 24, 2026 •

Copy link
Copy Markdown

⚠️ GitGuardian has uncovered 2 secrets following the scan of your pull request.

Please consider investigating the findings and remediating the incidents. Failure to do so may lead to compromising the associated services or software components.

🔎 Detected hardcoded secrets in your pull request
GitGuardian id GitGuardian status Secret Commit Filename
37585749 Triggered Generic Password 4f6fe33 lib/src/vault/secret_envelope.rs View secret
37585749 Triggered Generic Password 49ac330 lib/src/vault/secret_envelope.rs View secret
🛠 Guidelines to remediate hardcoded secrets
  1. Understand the implications of revoking this secret by investigating where it is used in your code.
  2. Replace and store your secrets safely. Learn here the best practices.
  3. Revoke and rotate these secrets.
  4. If possible, rewrite git history. Rewriting git history is not a trivial act. You might completely break other contributing developers' workflow and you risk accidentally deleting legitimate data.

To avoid such incidents in the future consider


🦉 GitGuardian detects secrets in your source code to help developers and security teams secure the modern development process. You are seeing this because you or someone else with access to this repository has authorized GitGuardian to scan your pull request.

@joepio joepio self-assigned this Sep 24, 2026

joepio commented Sep 24, 2026

Copy link
Copy Markdown
Member Author

The GitGuardian "Generic Password" hit on lib/src/vault/secret_envelope.rs is a false positive. It flagged a test fixture: a fake legacy wrapper of kind password with all-zero salt and nonce placeholders, not a secret. 49ac330 now builds those placeholders from zero bytes at runtime, so the fixture no longer looks like a credential literal.


Generated by Claude Code

joepio added a commit that referenced this pull request Sep 25, 2026
This reverts commit 1d80224, reversing
changes made to 94b3d07.
A v2 wrapped DbKey needs the wasm bundle, which ensureDbKeyOnSignIn now
loads on the main thread before it reads any record. When a caller sets the
agent before saving it (GettingStartedFlow does), the database opener finds
the wrapped record without a session key and polled for only 3s; on a cold
page the wasm load takes longer, so the opener gave up and the encrypted
database stayed closed until the next sign-in. The v1 unwrap this replaced
took milliseconds.

saveAgentToIDB now announces the sign-in (trackDbKeySignIn) before its
first await, and waitForSessionDbKey waits for an announced sign-in however
long it takes, keeping the 3s poll only when none is in flight.

vaultKeyOps also loads the versioned wasm pair like every other main-thread
load, and no longer caches a failed load for the page's lifetime.

joepio commented Sep 25, 2026

Copy link
Copy Markdown
Member Author

Traced the integration-workspace.spec.ts:254 bisect result. Short version: I could not find a way for this PR to put a different key on the local database across a reload, and the spec passes here on this head. I did find and fix a real regression in the same code, in 532cc3d.

Where the log line comes from. "cannot be decrypted with this agent's key … Discarding" is openClientDb in browser/lib/src/client-db-open.ts. It only deletes the file when the worker was started with discardUndecryptable, which initClientDb.startForIdentity sets to !hasWrappedDbKey(subject). So the discard needs two things at once when the page reloads: the session key is not the key the file was written with, and there is no wrapped record (neither v1 nor v2).

Why the devDrive flow can't produce that.

  • setAgent(newAgent) starts the worker. resolveDbKey finds no session key and no wrapped record, so getOrCreateSessionDbKey creates K1 in one IndexedDB readwrite transaction, and the file is created with K1.
  • saveAgentToIDB(secret) then runs ensureDbKeyOnSignIn. There is no v2 or v1 record, the session key K1 is there, so it wraps K1 into v2. If no session key existed yet, it would go through the same atomic getOrCreateSessionDbKey and get the same key.
  • Only two writes can overwrite an existing session key: the successful v2 and v1 unwrap branches. Both need a wrapped record from an earlier sign-in, and a fresh agent has none.
  • Reloads never call ensureDbKeyOnSignIn. They read the session key, which is still K1.
  • The Rust changes (removed wrappers, the Unknown kind) touch only SecretEnvelope. The ClientDb file encryption (db/encrypted_backend.rs) does not use it.

Checked by running it, on a Vite production build with the service worker and VITE_E2E, this head's wasm, and Chromium 141. The spec passed 3 of 3. An IndexedDB dump right after devDrive and again after a reload shows session-key.<fp> and wrapped-key-v2.<fp> both present, with the same session key. If Mancave shows the discard again, a dump of keyval-store right after before() would tell us which of the two conditions fails there.

What #1767 did break (fixed in 532cc3d). ensureDbKeyOnSignIn now loads the wasm bundle on the main thread before it reads any record. GettingStartedFlow.handleSignInWithSecret calls setAgent before saveAgentToIDB. When a returning user signs in after signing out, the opener finds the v2 record but no session key, and resolveDbKey only polled for 3s. On a cold page the wasm load takes longer than that, so the encrypted database stayed closed until the next sign-in. The v1 unwrap this replaced took milliseconds.

  • saveAgentToIDB now calls trackDbKeySignIn(subject, work) before its first await.
  • waitForSessionDbKey waits for an announced sign-in however long it takes, and keeps the 3s poll only when no sign-in is in flight.
  • vaultKeyOps now loads the versioned wasm pair (wasmJsUrl / wasmBinaryUrl), the same way recovery.ts already does. A failed load is no longer cached for the rest of the page's life.
  • Records are unchanged, so v1 still unlocks.

Tests.

  • New tests in localDbKey.persistence.test.ts and agentStorage.test.ts. Both fail with the old 3s poll and pass with the fix.
  • data-browser vitest: 1309 passed. The one failing file is the known scripts/integration-mcp.test.mjs.
  • tsc: clean apart from the known LocalThought errors. pnpm -r lint: clean.
  • e2e integration-workspace.spec.ts: 2 of 2 passed.
  • e2e signout-signin-data.spec.ts: failed only on the unexpected-console-errors guard, which caught atomic.place webrtc-signal proxy errors and RBSR warnings from a server binary built from a different branch. There were no DbKey warnings.

Generated by Claude Code

… spec

Dumps which keyval-store records exist (keys only; session keys as a 4-byte
SHA-256 prefix), the agent subject, its fingerprint and whether a wrapped key
record exists, right after devDrive and after the reload to the automation.
Logged to the CI output and attached to the test result, so a failure on CI
shows whether the key changed or the wrapped record was missing.
joepio added a commit that referenced this pull request Sep 25, 2026
joepio added a commit that referenced this pull request Sep 25, 2026
@joepio

joepio commented Sep 25, 2026

Copy link
Copy Markdown
Member Author

Landed on develop as part of the TrekMeester batch #1773 (squash commit 1d7f312), after a green Main pipeline run on the combined branch.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant