Skip to content

feat(drivers): flows, cron and key-value state on the storage schema v2 - #15

Merged
senamakel merged 45 commits into
epic/7973-storage-schema-v2from
epic-7973-7982-flows-v2
Oct 16, 2026
Merged

senamakel merged 45 commits into
epic/7973-storage-schema-v2from
epic-7973-7982-flows-v2

Conversation

@senamakel

Copy link
Copy Markdown
Owner

Reference: tinyhumansai/openhuman#7982 (part of #7973). Not a Closes: the library-only workflow store and the adaptive collections are follow-ups (see below).

What changed

  • tinystoragedrivers moves to v1.0.0 in every crate (tinyflows-drivers, -adaptive, -import, -module).
  • tinyflows_drivers::storage_schema() (owner flows, Tenant) and cron_storage_schema() (owner cron, Tenant) declare the v2 collections and carry the v1 -> v2 steps.
  • Catalog: bare snake_case bodies, no JSON-in-strings (graph, steps, pending_approvals, drafts are real values), one _ms per instant, *_ns copies gone, ULID ids for new flows/revisions/drafts (monotonic per process), hand-written CAS helpers now Repo::update, writes declare schema: 2. Wire types (Flow, FlowRun, ...) are unchanged; times render back as RFC 3339 with millisecond precision. Caller-supplied times (an import keeps a flow's created_at) stay body fields; the driver-clock record metadata cannot carry them.
  • Cron: cron.jobs (object schedule/delivery/origin/policy, a one-shot's instant is at_ms, next_run_ms only, creation time is the record's created_ms metadata), cron.runs (started_ms/finished_ms), cron.admission (the shared admission ledger; members are a list). Run numbers use tsd.counters (Counters::next("cron","runs")); occurrence claims are tsd.leases (owner cron, key Key::of([job_id, occurrence_ms]), random token as holder, lease epoch as fence). An edit, removal or re-creation of a job ends the live claim of its occurrence. The completion receipt (completed_occurrence) stays on the job document.
  • Key-value: flows_kv and flows_state -> tsd.kv owner flows (DriverStateStore namespace state, catalog namespaces kv:<namespace>).
  • Checkpointer: tinyflows does not depend on tinyagents, so it keeps its own checkpointer code but reads and writes exactly the graph.* documents of tinyagents-graph under namespace flows (ids Key::of([ns, thread, seq]), bodies {ns, thread, subgraph, seq, checkpoint_id, checkpoint}, graph.threads, graph.writes, schema: 4). graph.* is not declared here (owner graph declares it) and there is no flows_graph_* step (tinyagents-graph's checkpoints_flows_v2 etc. migrate it). checkpoint_layout_tests.rs pins the layout with fixtures copied from tinyagents (feat(graph,tasks): one graph checkpointer and tasks on the v2 schema tinyagents#6). The engine's Checkpointer trait has no lease methods, so no tsd.leases record is written; a host fencing a thread uses Key::of([ns, thread]) under owner graph. The payload inside checkpoint stays tinyflows' own Checkpoint shape.
  • tinyflows-module: the schedule document port allows cron.*, tsd.counters, tsd.leases, and forwards schema/expires_ms on puts; a creation time is host-only, so a job imported through the module is filed at the time of the import.
  • tinyflows-import tests compare at millisecond precision.

v1 -> v2 mapping

v1 v2 Step / helper
flows_definitions flows.definitions (graph_json parsed, created_at/updated_at/last_run_at -> created_ms/updated_ms/last_run_ms, created_ns dropped) definitions_v2
flows_revisions flows.revisions (touched_ns -> touched_ms) revisions_v2
flows_runs flows.runs (steps_json/pending_approvals_json -> steps/pending_approvals, started_ms/finished_ms) runs_v2
flows_suggestions flows.suggestions (steps_outline, suggested_connections, suggested_slugs, created_ms) suggestions_v2
flows_drafts flows.drafts (draft_json unwrapped, created_ms/updated_ms) drafts_v2
flows_kv, flows_state tsd.kv owner flows import_legacy_kv(storage)
flows_graph_* graph.* ns flows none here (tinyagents-graph)
cron_jobs cron.jobs jobs_v2
cron_runs cron.runs runs_v2
cron_counters admission cron.admission/ledger admission_v2
cron_counters runs tsd.counters owner cron name runs admission_v2 stages it as cron.admission/runs_counter; import_legacy_cron_counters(storage) moves it
embedded occurrence_lease / completed_occurrence tsd.leases / dropped not carried: a claim only means something to the worker holding it in memory; an occurrence in flight at upgrade may run once more

A document step cannot write tsd.* (owner tsd), hence the two importers; the host calls them once per scope after Registry::migrate. Both record completion in tsd.migrations.

Follow-ups (not in this PR)

  • store::DriverWorkflowStore (wf_*, feature store): library-only, used by no host; left on v1 names.
  • tinyflows-adaptive adaptive_* -> flows.adaptive_* (it would extend storage_schema(), since an owner may be declared once) and adaptive_counters -> tsd.counters.
  • Documents written inside an atomic_batch (the cron job + admission + run) carry schema: 0: WriteOp has no options.

Gates

  • cargo fmt --all -- --check: clean
  • cargo clippy -p tinyflows-drivers -p tinyflows-import -p tinyflows-module -p tinyflows-adaptive --all-targets --all-features -- -D warnings: clean
  • cargo test --workspace --all-features: all passing (drivers 345 incl. new step, importer, claim and layout tests)

senamakel and others added 30 commits October 16, 2026 22:46
Update the tinystoragedrivers-core and tinystoragedrivers-sqlite git
dependencies from tag v0.11.0 to v1.0.0 across the adaptive, drivers,
import, and module crates.

Auto-committed-on: dragonfly
Update the tinystoragedrivers core and sqlite crates from v0.11.0 to
v1.0.0, pulling in their new transitive dependencies such as zstd,
ulid, and data-encoding.

Auto-committed-on: dragonfly
Delete the checkpoint key and layout helper modules that are no longer
referenced anywhere in the crate.

Auto-committed-on: dragonfly
The checkpoint module in tinyflows-drivers is no longer referenced
anywhere, so it has been deleted to keep the crate free of dead code.

Auto-committed-on: dragonfly
Introduce a state store abstraction in the drivers crate to hold
persistent state for driver runs.

Auto-committed-on: dragonfly
Introduce a new crate that will host the driver implementations for
tinyflows. It is added as a separate crate so drivers can be built and
versioned independently of the core library.

Auto-committed-on: dragonfly
Update the checkpoint and state store tests for the renamed namespace
constructors and the v2 document layout, asserting that keys encode
injectively and that stored rows carry the expected owner, namespace and
thread fields.

Auto-committed-on: dragonfly
Introduce a checkpoint layout module and a key-value import path for
checkpoint state, along with tests covering both. This provides the
storage primitives needed to persist and restore checkpoints.

Auto-committed-on: dragonfly
Introduce a state store abstraction and a storage driver module so
workflow state and blob storage can be backed by pluggable
implementations. The new traits are re-exported from the crate root for
downstream drivers to implement.

Auto-committed-on: dragonfly
The catalog module was no longer referenced anywhere in the crate, so it
has been dropped to keep the driver surface minimal.

Auto-committed-on: dragonfly
Introduce a catalog of driver definitions so available drivers can be
enumerated and described in one place.

Auto-committed-on: dragonfly
Moved the revision listing and lookup helpers out of the catalog module into
their own file so the catalog code stays focused on driver registration. No
behaviour change.

Auto-committed-on: dragonfly
Delete the drafts, runs, steps, and suggestions modules from the
drivers catalog since they are no longer referenced.

Auto-committed-on: dragonfly
Add a key-value store driver to the catalog, exposing get, set, and
delete operations. This gives flows a simple persistent store for
small pieces of state without pulling in a full database.

Auto-committed-on: dragonfly
Moved the catalog migration code out of the main catalog module into a
dedicated migrate module so the schema upgrade path can be tested and
evolved independently. Added unit tests covering the migration steps.

Auto-committed-on: dragonfly
Replace the shared test_support::scoped helper with a local implementation
backed by MemoryStorage so the migration tests no longer depend on the
catalog test support module.

Auto-committed-on: dragonfly
Adds unit tests across the catalog drivers covering draft handling, key-value
operations, revision gaps and replacement, and run replacement. These lock in
existing behaviour for the catalog backends so regressions surface in CI.

Auto-committed-on: dragonfly
flows_* become flows.* with bare snake_case bodies, one _ms per instant,
ULID ids for new entities and v1 -> v2 steps; flows_kv and flows_state move
to tsd.kv (owner flows) through import_legacy_kv; the checkpointer reads and
writes the graph.* layout of tinyagents-graph under namespace flows.

Co-authored-by: Medulla <medulla@tinyhumans.ai>
Reworked the schedule codec to streamline how entries are encoded and
decoded, reducing duplication in the serialization paths. Behaviour is
unchanged.

Auto-committed-on: dragonfly
Reworked the schedule codec to streamline how schedules are encoded and
decoded, reducing duplication in the serialization paths. Behaviour is
unchanged.

Auto-committed-on: dragonfly
Move the schedule driver implementation out of the crate root into a
dedicated module so the driver's code is easier to find and extend. No
behaviour changes.

Auto-committed-on: dragonfly
…pty, and the stat is blank to

Could you paste the diff? Once I have it, I'll write the Conventional Commits message.

Auto-committed-on: dragonfly
The jobs module in the schedule driver was no longer referenced
anywhere, so it has been deleted to keep the crate tidy.

Auto-committed-on: dragonfly
Introduce a catalog module for registering and looking up drivers, along
with id helpers used to identify catalog entries. The schedule module and
crate root are wired up to expose the new pieces.

Auto-committed-on: dragonfly
Add a driver for scheduling jobs, exposing the schedule jobs module
through the drivers crate so callers can register and run scheduled
work.

Auto-committed-on: dragonfly
…pty, and the stat is blank to

Could you paste the diff (or at least the stat and a summary of the change)? Once I have that, I'll write the Conventional Commits message.

Auto-committed-on: dragonfly
The claims module in the schedule driver is no longer referenced
anywhere, so it has been deleted to keep the crate free of dead code.

Auto-committed-on: dragonfly
Introduce scheduling policy evaluation and job removal handling for the
schedule driver, along with the runtime wiring needed to drive them. This
lets scheduled jobs be governed by policy decisions and cleaned up when
they are no longer needed.

Auto-committed-on: dragonfly
Introduce the initial structure for the schedule migration driver so
subsequent changes can build on it.

Auto-committed-on: dragonfly
Adds an admission module that validates incoming schedule definitions before
they are accepted, and an import module for loading schedules from external
sources. Migration tests cover the new paths.

Auto-committed-on: dragonfly
senamakel and others added 15 commits October 16, 2026 23:06
Cover the schedule driver's admission checks, codec round-trips, and
run sweep behaviour with unit tests so regressions in these paths are
caught without needing a full integration setup.

Auto-committed-on: dragonfly
Adds tests for the schedule driver covering migration behaviour, the runs
sweep, and run queries. These exercise the storage paths that previously
had no direct coverage.

Auto-committed-on: dragonfly
… runs

Adds unit tests across the schedule driver covering admission control, claim handling, job queries, migrations and run bookkeeping. These lock in existing behaviour so future changes to the scheduling layer can be made with confidence.

Auto-committed-on: dragonfly
cron_* become cron.jobs / cron.runs / cron.admission with object bodies and
one _ms per instant, creation time as record metadata, run numbers on
tsd.counters and occurrence claims on tsd.leases; v1 -> v2 steps and a
counter importer.

Co-authored-by: Medulla <medulla@tinyhumans.ai>
The module docs now state that the library-only workflow store and
adaptive tables are not yet on v2 and will extend the same schema later,
since an owner may only be declared once. The store feature no longer
extends the schema, as that extension is deferred to a follow-up.

Auto-committed-on: dragonfly
Drop three imports that are no longer referenced in the catalog drafts
module and its tests, keeping the build warning-free.

Auto-committed-on: dragonfly
Add a canonical helper that rewrites RFC 3339 strings to millisecond
precision in UTC, and route the expected fixtures and serialized values
through it. Storage keeps one epoch-millisecond field per instant, so
comparisons with the legacy fixtures now happen at that precision.

Auto-committed-on: dragonfly
… docs and import test precision

Co-authored-by: Medulla <medulla@tinyhumans.ai>
Introduce an admission layer that validates schedule documents before they
are accepted, so invalid or conflicting schedules are rejected early
instead of failing later during execution.

Auto-committed-on: dragonfly
The admission check compared the error kind against a bare `Unsupported`
value, but the kind is now a variant carrying data, so the guard never
matched. Matching the variant restores the fallback that files a job
under the import time when the store cannot set a creation time.

Auto-committed-on: dragonfly
The schedule document bridge now uses the dotted `cron.jobs` collection
name, and the tests were updated to match. The unused `WriteOptions`
import was dropped from the documents module.

Auto-committed-on: dragonfly
Job creation time doubles as the identity for runs written against a
job, so a job removed and registered again within the same millisecond
could collide with its predecessor. Insertion now waits for a fresh
millisecond before writing, and collection setup only declares the
primitives the store actually uses.

Auto-committed-on: dragonfly
The schedule feature now enables the optional tokio dependency so the
scheduling driver can build with default features disabled.

Auto-committed-on: dragonfly
Job creation times are now recorded from the timestamp the store assigns
rather than the wall clock at insertion, so the uniqueness wait reflects
the actual identity used for runs. The wait loop was simplified to spin
until the clock passes the latest recorded creation millisecond.

Auto-committed-on: dragonfly
The tests now read and write the cron.jobs collection instead of cron_jobs, matching the renamed collection, and drop the obsolete next_run_ns field from the stored document.

Auto-committed-on: dragonfly
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant