Repository navigation
feat(drivers): flows, cron and key-value state on the storage schema v2 - #15
Merged
Merged
Conversation
Update the tinystoragedrivers-core and tinystoragedrivers-sqlite git dependencies from tag v0.11.0 to v1.0.0 across the adaptive, drivers, import, and module crates. Auto-committed-on: dragonfly
Update the tinystoragedrivers core and sqlite crates from v0.11.0 to v1.0.0, pulling in their new transitive dependencies such as zstd, ulid, and data-encoding. Auto-committed-on: dragonfly
Delete the checkpoint key and layout helper modules that are no longer referenced anywhere in the crate. Auto-committed-on: dragonfly
The checkpoint module in tinyflows-drivers is no longer referenced anywhere, so it has been deleted to keep the crate free of dead code. Auto-committed-on: dragonfly
Introduce a state store abstraction in the drivers crate to hold persistent state for driver runs. Auto-committed-on: dragonfly
Introduce a new crate that will host the driver implementations for tinyflows. It is added as a separate crate so drivers can be built and versioned independently of the core library. Auto-committed-on: dragonfly
Update the checkpoint and state store tests for the renamed namespace constructors and the v2 document layout, asserting that keys encode injectively and that stored rows carry the expected owner, namespace and thread fields. Auto-committed-on: dragonfly
Introduce a checkpoint layout module and a key-value import path for checkpoint state, along with tests covering both. This provides the storage primitives needed to persist and restore checkpoints. Auto-committed-on: dragonfly
Introduce a state store abstraction and a storage driver module so workflow state and blob storage can be backed by pluggable implementations. The new traits are re-exported from the crate root for downstream drivers to implement. Auto-committed-on: dragonfly
The catalog module was no longer referenced anywhere in the crate, so it has been dropped to keep the driver surface minimal. Auto-committed-on: dragonfly
Introduce a catalog of driver definitions so available drivers can be enumerated and described in one place. Auto-committed-on: dragonfly
Moved the revision listing and lookup helpers out of the catalog module into their own file so the catalog code stays focused on driver registration. No behaviour change. Auto-committed-on: dragonfly
Delete the drafts, runs, steps, and suggestions modules from the drivers catalog since they are no longer referenced. Auto-committed-on: dragonfly
Add a key-value store driver to the catalog, exposing get, set, and delete operations. This gives flows a simple persistent store for small pieces of state without pulling in a full database. Auto-committed-on: dragonfly
Moved the catalog migration code out of the main catalog module into a dedicated migrate module so the schema upgrade path can be tested and evolved independently. Added unit tests covering the migration steps. Auto-committed-on: dragonfly
Replace the shared test_support::scoped helper with a local implementation backed by MemoryStorage so the migration tests no longer depend on the catalog test support module. Auto-committed-on: dragonfly
Adds unit tests across the catalog drivers covering draft handling, key-value operations, revision gaps and replacement, and run replacement. These lock in existing behaviour for the catalog backends so regressions surface in CI. Auto-committed-on: dragonfly
flows_* become flows.* with bare snake_case bodies, one _ms per instant, ULID ids for new entities and v1 -> v2 steps; flows_kv and flows_state move to tsd.kv (owner flows) through import_legacy_kv; the checkpointer reads and writes the graph.* layout of tinyagents-graph under namespace flows. Co-authored-by: Medulla <medulla@tinyhumans.ai>
Reworked the schedule codec to streamline how entries are encoded and decoded, reducing duplication in the serialization paths. Behaviour is unchanged. Auto-committed-on: dragonfly
Reworked the schedule codec to streamline how schedules are encoded and decoded, reducing duplication in the serialization paths. Behaviour is unchanged. Auto-committed-on: dragonfly
Move the schedule driver implementation out of the crate root into a dedicated module so the driver's code is easier to find and extend. No behaviour changes. Auto-committed-on: dragonfly
…pty, and the stat is blank to Could you paste the diff? Once I have it, I'll write the Conventional Commits message. Auto-committed-on: dragonfly
The jobs module in the schedule driver was no longer referenced anywhere, so it has been deleted to keep the crate tidy. Auto-committed-on: dragonfly
Introduce a catalog module for registering and looking up drivers, along with id helpers used to identify catalog entries. The schedule module and crate root are wired up to expose the new pieces. Auto-committed-on: dragonfly
Add a driver for scheduling jobs, exposing the schedule jobs module through the drivers crate so callers can register and run scheduled work. Auto-committed-on: dragonfly
…pty, and the stat is blank to Could you paste the diff (or at least the stat and a summary of the change)? Once I have that, I'll write the Conventional Commits message. Auto-committed-on: dragonfly
The claims module in the schedule driver is no longer referenced anywhere, so it has been deleted to keep the crate free of dead code. Auto-committed-on: dragonfly
Introduce scheduling policy evaluation and job removal handling for the schedule driver, along with the runtime wiring needed to drive them. This lets scheduled jobs be governed by policy decisions and cleaned up when they are no longer needed. Auto-committed-on: dragonfly
Introduce the initial structure for the schedule migration driver so subsequent changes can build on it. Auto-committed-on: dragonfly
Adds an admission module that validates incoming schedule definitions before they are accepted, and an import module for loading schedules from external sources. Migration tests cover the new paths. Auto-committed-on: dragonfly
Cover the schedule driver's admission checks, codec round-trips, and run sweep behaviour with unit tests so regressions in these paths are caught without needing a full integration setup. Auto-committed-on: dragonfly
Adds tests for the schedule driver covering migration behaviour, the runs sweep, and run queries. These exercise the storage paths that previously had no direct coverage. Auto-committed-on: dragonfly
… runs Adds unit tests across the schedule driver covering admission control, claim handling, job queries, migrations and run bookkeeping. These lock in existing behaviour so future changes to the scheduling layer can be made with confidence. Auto-committed-on: dragonfly
cron_* become cron.jobs / cron.runs / cron.admission with object bodies and one _ms per instant, creation time as record metadata, run numbers on tsd.counters and occurrence claims on tsd.leases; v1 -> v2 steps and a counter importer. Co-authored-by: Medulla <medulla@tinyhumans.ai>
The module docs now state that the library-only workflow store and adaptive tables are not yet on v2 and will extend the same schema later, since an owner may only be declared once. The store feature no longer extends the schema, as that extension is deferred to a follow-up. Auto-committed-on: dragonfly
Drop three imports that are no longer referenced in the catalog drafts module and its tests, keeping the build warning-free. Auto-committed-on: dragonfly
Add a canonical helper that rewrites RFC 3339 strings to millisecond precision in UTC, and route the expected fixtures and serialized values through it. Storage keeps one epoch-millisecond field per instant, so comparisons with the legacy fixtures now happen at that precision. Auto-committed-on: dragonfly
… docs and import test precision Co-authored-by: Medulla <medulla@tinyhumans.ai>
Introduce an admission layer that validates schedule documents before they are accepted, so invalid or conflicting schedules are rejected early instead of failing later during execution. Auto-committed-on: dragonfly
The admission check compared the error kind against a bare `Unsupported` value, but the kind is now a variant carrying data, so the guard never matched. Matching the variant restores the fallback that files a job under the import time when the store cannot set a creation time. Auto-committed-on: dragonfly
The schedule document bridge now uses the dotted `cron.jobs` collection name, and the tests were updated to match. The unused `WriteOptions` import was dropped from the documents module. Auto-committed-on: dragonfly
Job creation time doubles as the identity for runs written against a job, so a job removed and registered again within the same millisecond could collide with its predecessor. Insertion now waits for a fresh millisecond before writing, and collection setup only declares the primitives the store actually uses. Auto-committed-on: dragonfly
The schedule feature now enables the optional tokio dependency so the scheduling driver can build with default features disabled. Auto-committed-on: dragonfly
Job creation times are now recorded from the timestamp the store assigns rather than the wall clock at insertion, so the uniqueness wait reflects the actual identity used for runs. The wait loop was simplified to spin until the clock passes the latest recorded creation millisecond. Auto-committed-on: dragonfly
The tests now read and write the cron.jobs collection instead of cron_jobs, matching the renamed collection, and drop the obsolete next_run_ns field from the stored document. Auto-committed-on: dragonfly
This was referenced Oct 16, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Reference: tinyhumansai/openhuman#7982 (part of #7973). Not a
Closes: the library-only workflow store and the adaptive collections are follow-ups (see below).What changed
v1.0.0in every crate (tinyflows-drivers,-adaptive,-import,-module).tinyflows_drivers::storage_schema()(ownerflows, Tenant) andcron_storage_schema()(ownercron, Tenant) declare the v2 collections and carry the v1 -> v2 steps.graph,steps,pending_approvals, drafts are real values), one_msper instant,*_nscopies gone, ULID ids for new flows/revisions/drafts (monotonic per process), hand-written CAS helpers nowRepo::update, writes declareschema: 2. Wire types (Flow,FlowRun, ...) are unchanged; times render back as RFC 3339 with millisecond precision. Caller-supplied times (an import keeps a flow'screated_at) stay body fields; the driver-clock record metadata cannot carry them.cron.jobs(objectschedule/delivery/origin/policy, a one-shot's instant isat_ms,next_run_msonly, creation time is the record'screated_msmetadata),cron.runs(started_ms/finished_ms),cron.admission(the shared admission ledger; members are a list). Run numbers usetsd.counters(Counters::next("cron","runs")); occurrence claims aretsd.leases(ownercron, keyKey::of([job_id, occurrence_ms]), random token as holder, lease epoch as fence). An edit, removal or re-creation of a job ends the live claim of its occurrence. The completion receipt (completed_occurrence) stays on the job document.flows_kvandflows_state->tsd.kvownerflows(DriverStateStorenamespacestate, catalog namespaceskv:<namespace>).graph.*documents oftinyagents-graphunder namespaceflows(idsKey::of([ns, thread, seq]), bodies{ns, thread, subgraph, seq, checkpoint_id, checkpoint},graph.threads,graph.writes,schema: 4).graph.*is not declared here (ownergraphdeclares it) and there is noflows_graph_*step (tinyagents-graph'scheckpoints_flows_v2etc. migrate it).checkpoint_layout_tests.rspins the layout with fixtures copied from tinyagents (feat(graph,tasks): one graph checkpointer and tasks on the v2 schema tinyagents#6). The engine'sCheckpointertrait has no lease methods, so notsd.leasesrecord is written; a host fencing a thread usesKey::of([ns, thread])under ownergraph. The payload insidecheckpointstays tinyflows' ownCheckpointshape.tinyflows-module: the schedule document port allowscron.*,tsd.counters,tsd.leases, and forwardsschema/expires_mson puts; a creation time is host-only, so a job imported through the module is filed at the time of the import.tinyflows-importtests compare at millisecond precision.v1 -> v2 mapping
flows_definitionsflows.definitions(graph_jsonparsed,created_at/updated_at/last_run_at->created_ms/updated_ms/last_run_ms,created_nsdropped)definitions_v2flows_revisionsflows.revisions(touched_ns->touched_ms)revisions_v2flows_runsflows.runs(steps_json/pending_approvals_json->steps/pending_approvals,started_ms/finished_ms)runs_v2flows_suggestionsflows.suggestions(steps_outline,suggested_connections,suggested_slugs,created_ms)suggestions_v2flows_draftsflows.drafts(draft_jsonunwrapped,created_ms/updated_ms)drafts_v2flows_kv,flows_statetsd.kvownerflowsimport_legacy_kv(storage)flows_graph_*graph.*nsflowscron_jobscron.jobsjobs_v2cron_runscron.runsruns_v2cron_countersadmissioncron.admission/ledgeradmission_v2cron_countersrunstsd.countersownercronnamerunsadmission_v2stages it ascron.admission/runs_counter;import_legacy_cron_counters(storage)moves itoccurrence_lease/completed_occurrencetsd.leases/ droppedA document step cannot write
tsd.*(ownertsd), hence the two importers; the host calls them once per scope afterRegistry::migrate. Both record completion intsd.migrations.Follow-ups (not in this PR)
store::DriverWorkflowStore(wf_*, featurestore): library-only, used by no host; left on v1 names.tinyflows-adaptiveadaptive_*->flows.adaptive_*(it would extendstorage_schema(), since an owner may be declared once) andadaptive_counters->tsd.counters.atomic_batch(the cron job + admission + run) carryschema: 0:WriteOphas no options.Gates
cargo fmt --all -- --check: cleancargo clippy -p tinyflows-drivers -p tinyflows-import -p tinyflows-module -p tinyflows-adaptive --all-targets --all-features -- -D warnings: cleancargo test --workspace --all-features: all passing (drivers 345 incl. new step, importer, claim and layout tests)