Repository navigation
feat(storage): adopt the uniform storage schema v2 (tinystoragedrivers v1.0.0) - #158
Merged
senamakel merged 99 commits intoOct 17, 2026
Merged
Conversation
Update the tinystoragedrivers-core and tinystoragedrivers-sqlite git dependencies from tag v0.11.0 to v1.0.0 across the adaptive, drivers, import, and module crates. Auto-committed-on: dragonfly
Update the tinystoragedrivers core and sqlite crates from v0.11.0 to v1.0.0, pulling in their new transitive dependencies such as zstd, ulid, and data-encoding. Auto-committed-on: dragonfly
Delete the checkpoint key and layout helper modules that are no longer referenced anywhere in the crate. Auto-committed-on: dragonfly
The checkpoint module in tinyflows-drivers is no longer referenced anywhere, so it has been deleted to keep the crate free of dead code. Auto-committed-on: dragonfly
Introduce a state store abstraction in the drivers crate to hold persistent state for driver runs. Auto-committed-on: dragonfly
Introduce a new crate that will host the driver implementations for tinyflows. It is added as a separate crate so drivers can be built and versioned independently of the core library. Auto-committed-on: dragonfly
Update the checkpoint and state store tests for the renamed namespace constructors and the v2 document layout, asserting that keys encode injectively and that stored rows carry the expected owner, namespace and thread fields. Auto-committed-on: dragonfly
Introduce a checkpoint layout module and a key-value import path for checkpoint state, along with tests covering both. This provides the storage primitives needed to persist and restore checkpoints. Auto-committed-on: dragonfly
Introduce a state store abstraction and a storage driver module so workflow state and blob storage can be backed by pluggable implementations. The new traits are re-exported from the crate root for downstream drivers to implement. Auto-committed-on: dragonfly
The catalog module was no longer referenced anywhere in the crate, so it has been dropped to keep the driver surface minimal. Auto-committed-on: dragonfly
Introduce a catalog of driver definitions so available drivers can be enumerated and described in one place. Auto-committed-on: dragonfly
Moved the revision listing and lookup helpers out of the catalog module into their own file so the catalog code stays focused on driver registration. No behaviour change. Auto-committed-on: dragonfly
Delete the drafts, runs, steps, and suggestions modules from the drivers catalog since they are no longer referenced. Auto-committed-on: dragonfly
Add a key-value store driver to the catalog, exposing get, set, and delete operations. This gives flows a simple persistent store for small pieces of state without pulling in a full database. Auto-committed-on: dragonfly
Moved the catalog migration code out of the main catalog module into a dedicated migrate module so the schema upgrade path can be tested and evolved independently. Added unit tests covering the migration steps. Auto-committed-on: dragonfly
Replace the shared test_support::scoped helper with a local implementation backed by MemoryStorage so the migration tests no longer depend on the catalog test support module. Auto-committed-on: dragonfly
Adds unit tests across the catalog drivers covering draft handling, key-value operations, revision gaps and replacement, and run replacement. These lock in existing behaviour for the catalog backends so regressions surface in CI. Auto-committed-on: dragonfly
flows_* become flows.* with bare snake_case bodies, one _ms per instant, ULID ids for new entities and v1 -> v2 steps; flows_kv and flows_state move to tsd.kv (owner flows) through import_legacy_kv; the checkpointer reads and writes the graph.* layout of tinyagents-graph under namespace flows. Co-authored-by: Medulla <medulla@tinyhumans.ai>
Reworked the schedule codec to streamline how entries are encoded and decoded, reducing duplication in the serialization paths. Behaviour is unchanged. Auto-committed-on: dragonfly
Reworked the schedule codec to streamline how schedules are encoded and decoded, reducing duplication in the serialization paths. Behaviour is unchanged. Auto-committed-on: dragonfly
Move the schedule driver implementation out of the crate root into a dedicated module so the driver's code is easier to find and extend. No behaviour changes. Auto-committed-on: dragonfly
…pty, and the stat is blank to Could you paste the diff? Once I have it, I'll write the Conventional Commits message. Auto-committed-on: dragonfly
The jobs module in the schedule driver was no longer referenced anywhere, so it has been deleted to keep the crate tidy. Auto-committed-on: dragonfly
Introduce a catalog module for registering and looking up drivers, along with id helpers used to identify catalog entries. The schedule module and crate root are wired up to expose the new pieces. Auto-committed-on: dragonfly
Add a driver for scheduling jobs, exposing the schedule jobs module through the drivers crate so callers can register and run scheduled work. Auto-committed-on: dragonfly
…pty, and the stat is blank to Could you paste the diff (or at least the stat and a summary of the change)? Once I have that, I'll write the Conventional Commits message. Auto-committed-on: dragonfly
The claims module in the schedule driver is no longer referenced anywhere, so it has been deleted to keep the crate free of dead code. Auto-committed-on: dragonfly
Introduce scheduling policy evaluation and job removal handling for the schedule driver, along with the runtime wiring needed to drive them. This lets scheduled jobs be governed by policy decisions and cleaned up when they are no longer needed. Auto-committed-on: dragonfly
Introduce the initial structure for the schedule migration driver so subsequent changes can build on it. Auto-committed-on: dragonfly
Adds an admission module that validates incoming schedule definitions before they are accepted, and an import module for loading schedules from external sources. Migration tests cover the new paths. Auto-committed-on: dragonfly
Added tests covering schedule import behaviour in the drivers crate to verify the import path works correctly. Auto-committed-on: dragonfly
Added a catalog entry for the kv driver and wired it into the catalog module. Also extended schedule admission and its test support to cover the new driver. Auto-committed-on: dragonfly
Introduce a module for working with schedule documents, along with tests covering its behaviour. Auto-committed-on: dragonfly
Document the crate's purpose and the available driver implementations so the package is discoverable and its role in the workspace is clear. Auto-committed-on: dragonfly
The catalog docs described a run's steps as a `steps_json` string, but the run document stores them as a `steps` JSON array. Updated the comments to match the actual field and clarify its relationship to the SQLite store's column. Auto-committed-on: dragonfly
…ema-v2 # Conflicts: # crates/tinyflows-drivers/README.md # crates/tinyflows-drivers/src/catalog/import.rs # crates/tinyflows-drivers/src/catalog/mod.rs # crates/tinyflows-drivers/src/catalog/revisions.rs # crates/tinyflows-drivers/src/catalog/runs.rs # crates/tinyflows-drivers/src/schedule/claims.rs # crates/tinyflows-module/Cargo.toml # crates/tinyflows-module/tests/module_artifact_tests.rs Resolved onto storage schema v2: claims and queued runs on tsd.leases, run events as a real array, catalog port on v2 collection names. Co-authored-by: Medulla <medulla@tinyhumans.ai>
The fake document store in the run artifact tests now matches the renamed collections (flows.runs, tsd.kv, graph.checkpoints) and reads the events array directly instead of parsing an events_json string, keeping the harness in sync with the current document shape. Auto-committed-on: dragonfly
Add eprintln diagnostics to the early-return branches of the cron claim loop so the reason a job is skipped can be observed while debugging. These are temporary and should be removed once the investigation is done. Auto-committed-on: dragonfly
The claim logic no longer writes debug output to stderr on each early return, and the recovery test now truncates its timestamp to millisecond precision so it matches the stored instant. Auto-committed-on: dragonfly
The migration now copies the queued placeholder run number from an embedded occurrence lease onto the job as queued_run, so the next claim can remove the placeholder instead of leaving it orphaned. The module docs were updated to note this exception to the rule that embedded leases are not carried over. Auto-committed-on: dragonfly
…igration Add migration tests asserting that v1 revision sequences, run event arrays, execution snapshots, and terminal event sequences survive migration and read back through the document stores. Also verify that a v1 claim's queued placeholder run is preserved on the job and cleared by the next claim. Auto-committed-on: dragonfly
Adds module documentation explaining that a claim which queued a placeholder run records its run number as the job's queued_run, and that completion, release, reacquisition and edits remove the placeholder along with that field. Also reformats a few expressions to satisfy rustfmt. Auto-committed-on: dragonfly
Member
Author
|
@coderabbitai review Head 39beaed merges main (#157) into the storage v2 epic. #157's queued-run claims now sit on |
|
Member
Author
|
@coderabbitai full review |
|
Acquisition now always rewrites the job with the lease epoch, and every claim operation verifies that epoch before committing, so a holder whose lease expired or whose job was edited can no longer commit against a stale read. Freeing a lease on edit is retried on conflict and treated as best effort, since the epoch check already guarantees correctness. Auto-committed-on: dragonfly
Completion now removes the claim epoch from the job body and surfaces a lease release failure instead of silently ignoring it. The run and schedule advance have already committed at that point, so the error is returned with context while the lease still expires on its own. Auto-committed-on: dragonfly
Extend the FaultDocuments test double with a Faults struct that can fail lease writes and a chosen cron.jobs read, and initialize it in the existing fixtures. This lets tests exercise storage failure paths that were previously unreachable. Auto-committed-on: dragonfly
Registers a new test module covering the safety of cron claim and lease handling, and reformats a warning call in the claim release path to satisfy line width. Auto-committed-on: dragonfly
Add a test asserting that a DocPutRequest serialized before the schema and expires_ms fields existed still decodes with both undeclared and re-encodes to the same shape, guarding backward compatibility. Auto-committed-on: dragonfly
The module documentation now explains that the job document carries the fence epoch of its current claim, written on every acquisition and dropped by an edit, and that every claim operation commits only against a job carrying its own epoch. This clarifies why a holder that lost its lease or its job to an edit cannot commit. Auto-committed-on: dragonfly
The test no longer installs a batch barrier, since the occurrence lease is acquired atomically and only the winning claim reaches the job write, so the barrier could never hold both claims anyway. A comment now records that reasoning. Auto-committed-on: dragonfly
…r pruning Once the run, receipt and schedule advance are durable, a failed lease release or history prune is liveness only: claim_epoch fences stale holders and the lease expires on its own. Return Ok(true) and log, so hosts still run delete_after_run cleanup and do not report a finished run as failed. Co-authored-by: Medulla <medulla@tinyhumans.ai>
Member
Author
|
@coderabbitai approve CodeRabbit can't review this head because it changes more than 100 files. An independent review checked the merged claims code instead. It found three claim-safety regressions, fixed in dc2d2eb, and one post-commit error-path issue, fixed in 7ceeabf. Each fix has a regression test that fails before it. Hosted CI is green on 7ceeabf. |
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
|
@senamakel, thanks for the review and fix details. The approval workflow will check current permissions and review readiness. 🧠 Learnings used✅ Action performedChanges approved. Review threads were not changed. |
![tinysweeper[bot]](https://avatars.githubusercontent.com/in/4526585?s=60&v=4){kind=small}
There was a problem hiding this comment.
tinysweeper found nothing blocking, but could not review everything, so this is not an approval: crates/tinyflows-adaptive/Cargo.toml, crates/tinyflows-adaptive/src/drivers/convert.rs, crates/tinyflows-adaptive/src/drivers/ledger/mod.rs, crates/tinyflows-adaptive/src/drivers/ledger/trait_impl.rs, crates/tinyflows-adaptive/src/drivers/ledger_tests.rs, crates/tinyflows-adaptive/src/drivers/migrate.rs, crates/tinyflows-adaptive/src/drivers/mod.rs, crates/tinyflows-adaptive/src/drivers/mod_tests.rs and 99 more.
$0.0095 · 1,675,047 in / 37,023 out · 108,992 cached (7%) · ladder/vectors-oai3, glm-5.3-flash, , deepseek-v4.1-flash · 788 embedded
critique: $0.0043 · 315,493 in / 20,430 out · 55,168 cached (17%) · glm-5.3-flash, , deepseek-v4.1-flash
security: $0.0048 · 803,259 in / 14,205 out · 49,600 cached (6%) · glm-5.3-flash, deepseek-v4.1-flash,
tests: $0.0001 · 184,106 in / 270 out · 2,176 cached (1%) · deepseek-v4.1-flash
description: $0.0001 · 170,912 in / 1,011 out · 2,048 cached (1%) · deepseek-v4.1-flash
tinysweeper
Bot
added
the
priority: p3
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Moves tinyflows persistence onto the uniform storage schema v2 (tinystoragedrivers v1.0.0): v2 collection names,
Key::ofids, snake_case_msbodies, andstorage_schema()with migration steps for the v1 shapes.Delivered in this repo:
mainmerged in (chore: merge upstream main into the storage schema v2 epic senamakel/tinyflows#30), no conflictsRolled-up fork PRs
senamakel#15, #16, #30
Sub-issues
tinyhumansai/openhuman#7982, #8058
Gates
cargo fmt --all -- --check,cargo clippy --workspace --all-targets --all-features -- -D warningsandcargo test --workspace --all-featurespass locally on the merged tree; hosted CI is green on senamakel#30.Part of tinyhumansai/openhuman#7973
Co-authored-by: Medulla medulla@tinyhumans.ai
Summary by CodeRabbit
New Features
Bug Fixes