Skip to content

Local-first resources - don't require a server when creating a resource #998

Closed

Description

@joepio

Problem

Creating a new resource requires a lot: an active internet connection, a working server, a resolvable Agent. That makes creating data kinda bothersome, which hinders adoption. It would be nice if Resources can be created and update local-first.

Additionally, it would be nice if resources could be resolved without a server. A P2P solution, perhaps something that uses content-addressing (e.g. something like ipfs / iroh) would be swell.

Solutions

Solution 1: use Nextgraph

See #1012

Solution 2: commit hashes and own schema

  • User can create a Commit that does not refer to an HTTP subject. It has no target. It's a RootCommit (not sure about the name).
  • The signature of that commit is the identifier. (or perhaps the iroh hash?)
  • This RootCommit has a date, a publicKey (linking
  • You can refer to the initial version of this resource using a did:atomic:RootHash. So we'll introduce a new did:atomic schema for this.
  • You can refer to a specific following version of this resource using using a did:atomic:CommitHash
  • We still want pretty URLs, in some usecases, of course! So how do we deal with that? How do we map these?
  • We can still resolve using HTTP: https://atomicdata.dev/cid/aeg8ae9gyg98ea9eahge98gha

API (@tomic/lib)

Not a lot would change for the user.

  • When creating a resource, the subject becomes optional, as well as a server. The resource that is returned will get a subject through its constructor, by taking the hash of that resource.
  • The

URL schema (resolve method)

If we have local-first resources, these should be able to resolve without HTTP. How would we go about this? If we use NextGraph, we just use their did schema. Easy.

What information should be in the URL?

  • the Schema, e.g. http or atomic: or did:atomic. This describes how the resource is to be resolved.
  • The hash / id of the first commit, I think. That's the cryptographic "root"!
  • optional: the HTTP server (domain), where the resource lives by default. This is mostly there for performance reasons. it allows the client to quickly get the resource from a server
  • optional: the path + query. This is the /my-site/hello part, that links to the actual content. Not sure if this should be a thing.

But if we don't, we have a couple of options:

  • did:atomic, a new schema that we should register. The URLs themselves

Questions

  • Should the identifier of the resource be based on the signature of the first resource? Or should it be the hash? For compatibility with something like ipfs, a hash would be nice. But Atomic uses signatures of the agent.
  • For authorship, the public key (or subject?) of the signing agent should be included in that initial resource.
  • What kind of merge algo should we use for branching and resolving commits? Some CRDT like Yjs or Automerge might be a good solution.

Relevant issues

Relevant reads

  • AnySync, the protocol that syncs AnyType. A CRDT that uses IPFS under the hood.
  • Willow, powered by iroh. Has rust and ts implementations.
  • Loro. CRDT with Rust and TS libraries. Seems to be fast and lightweight, compared to yjs / automerge.

Activity

  1. joepio commented on Jun 2, 2025

    @joepio
    MemberAuthor

    @Polleps suggested a different path:

    Locally Process Atomic Commits

    • Keep the requirement of having an accessible server / a place for resources to be persisted. This kills some usecases, but not a lot.
    • Store commits locally, in the browser, if the server is offline. When the client connects to the server, send all commits.
    • Use something like YJS for merging / branching to deal with conflicts (see Consider using yjs / y-crdt for document-related commits #720)
  2. joepio commented on Oct 1, 2025

    @joepio
    MemberAuthor

    Other options:

    Use Yjs for commits

    • Full CRDT! No need to do any of the hard work.
    • Yjs is bulky / heavy, way more complicated than Atomic Commits. There isn't even a Java implementation - wild, considering how popular it is.
    • Making edits requires Yjs state (contains history, if no garbage collection is turned on), so instead of sending plain Json objects to clients (who want to view the data) it might be smarter to send (minimal) Yjs state vectors, so they can edit.
    • Maybe we allow commits to be ephemeral (don't persist) because persisting all vectors is kinda mad if you use Yjs. The history is already inside the Yjs doc!
    • The server persists a Yjs doc per resource.
    • How do we do indexing? Currently, we use "add atoms" and "delete atoms". We could calculate these by comparing a before and after of a Yjs vector.
    • Do we want to allow updates using State Vectors? This means: can clients say "my state is X" and can the server create a tailored response "heres your update Y"? Would mean adding a new endpoint.
  3. Polleps commented on Oct 1, 2025

    @Polleps
    Collaborator

    I think there are 3 options for integrating Yjs into Atomic.

    Option 1:
    Go full CRDT by replacing Atomic Commits with Yjs state
    Described in Joeps comment above

    Questions: What do we send to the client, the client needs a Yjs state to do edits but in most cases (CMS usecase) It only wants to read so having to include Yjs just to decode the data seems like overkill.

    Option 2:
    Add a Yjs datatype so resources can have Yjs documents as prop values. We would then add a new property to commits alongside set, push, delete etc called something like change or update that contains a Yjs update bin.
    The propval would contain the full document state. We might also have to create a type of 'ephemeral commits' for data that is intended to change rapidly and doesn't really add value to the resource it self (things like user cursor position or online status). This would mean that replaying each commit would result in a different binary from the current state (although still with the same content if the client uses these commits responsibly).

    This would be a lot less work to implement but also would not make it possible to edit every type of property conflict free

    Option 3:
    Add Yjs as a datatype or maybe class-extender and persist the state/updates separate from the rest of the data (like how files are handled). This is more in line how Yjs is intended to be used (at least that's the idea I get when reading the readme).
    This would mean that a resources would reference some identifier that points to a Yjs doc stored somewhere else (can still be in sled just in a different tree). There would have to be an endpoint that handles syncing to the doc.
    With this approach there would be no history stored about the value of the doc in commits, only in the documents state.

  4. Polleps commented on Oct 1, 2025

    @Polleps
    Collaborator

    Option 4:
    Change our commit logic to be a CRDT (and use option 2 for yjs crdt specifically for text documents)
    Atomic Commits are already pretty close to being CRDTs. Currently they are not because they aren't deterministic, the server just applies the commits in the order it receives them. There is a timestamp component but that doesn't really guaranty anything. However we could use the lastCommit property together with some deterministic algorithm to resolve conflicts. For example if two commits reference the same lastCommit we could determine their order based on their hash.

    I'm not sure if this is true but I think you don't even need to replay every commit in order to apply an out of order commit. Atomic commits only allow set, delete and push. Push can be played back without issue, set and delete can't but it doesn't matter because when everything gets played back the set will eventually occur completely overriding anything that happened before it (a bit like multiplying by 0 at the end of a long calculation).

  5. AlexMikhalev commented on Jan 7, 2026

    @AlexMikhalev
    Contributor

    I think you can make option 4 commit logic to be a CRDT, expanding on @Polleps lastCommit property together with some deterministic algorithm to resolve conflicts. if you use vector clock as timestamps, quote from old couchdb API:

    CouchDB uses vector clocks as part of its conflict resolution mechanism in distributed systems.
    Vector clocks are used to determine the causal relationship between events across different nodes, helping identify whether updates are concurrent or causally related.
    Each node maintains a vector clock, which is an array of logical timestamps—one for each node in the system—where each entry is incremented upon internal events or upon receiving messages from other nodes.
    When a node receives an update, it updates its vector clock by taking the maximum value for each element from its own clock and the received clock.

    In CouchDB, vector clocks help detect conflicts when multiple updates occur simultaneously on different nodes without a clear ordering.
    However, vector clocks do not resolve conflicts by themselves; they only identify that a conflict exists.
    The system must then save all conflicting versions until an application or user resolves them. This approach allows for eventual consistency and supports use cases where multiple versions of data may be valid, such as in collaborative editing or distributed applications.

    So, the only adjustment for atomic commits to become CRDT is to order them by vector clock timestamps.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions