Libp2p CRDT Synchronization

Users and Developers js
1

I had need for distributed state over Libp2p but I could not find anything that handles this satisfactorily, so I created a a system for synchronizing CRDTs through Libp2p. This system is a little bit similar to OrbitDB but is simpler and runs on Libp2p without the need for IPFS.

The system is only designed for use in a private network where you can trust all the other nodes to follow the same protocol and trust all the nodes with read/write access. Let me know if you have any suggestions what can be done to provide a permission system so trust between the nodes is limited.

The libp2p-crdt-synchronizer module can synchronize any CRDT following the CRDT interface from the crdt-interfaces package. The CRDT interface defines a sync method which defines the synchronization protocol between instances. The interface also allows for additional methods like serialize (to allow saving state) and addBroadcaster (to allow faster synchronization over Libp2p pubsub).

Another thing to note is the same CRDT type can have different protocols for sync, currently they are required to be treated as separate CRDTs (in regards to synchronization) but perhaps there is a way to refactor the synchronization protocols so that multiple can be added to a single CRDT implementation.

I have created a bunch of basic CRDT implementations in the crdts package that will work with libp2p-crdt-synchronizer module, making it easy to get started with it. Most of these implementations use CBOR encoding so they work with a variety of data types, if you want to create a version of one of these CRDTs to use a specific data type you might want to create protobuf definitions for each type which would clearly specify how it’s protocol works, I have done this in crdt-protocols but have not implemented any of them due to the work needed to get each type working.

The following code should help you to get started quickly:

import { createCRDTSynchronizer } from "@organicdesign/libp2p-crdt-synchronizer";
import { createGCounter } from "@organicdesign/crdts";

const synchronizer = createCRDTSynchronizer()(libp2p);

synchronizer.start();

const counter = createGCounter({ id: libp2p.peerId.toBytes() });

synchronizer.setCRDT("my-counter", counter);

If you give this a try, please let me know how it goes, there is a lot of room for improvement in this system and I am hoping some of you would have suggestions on how it could be done better.

Here is a list of the npm modules that make up this system:

4 Likes
2

This looks like a very useful project. Thanks for sharing !
Dealing with the permission system, I think you can add to libp2p options a

  const connectionProtector = preSharedKey({
    psk: new Uint8Array(Buffer.from(swarmKey, "base64"))
  });

This makes quite a private p2p network as only nodes sharing the same swarmKey can communicate.

I would like to try your project, but I am currently using libp2p@0.42.1 and it looks like it is not compatible, as registrar and connectionManager have been removed from libp2pnode, as explained here.
Do you plan to upgrade to a new libp2p version ? I could help if this is relevant.

3

Thanks for your interest,

PNet/PSK is currently the only feasible and intended way to use this module at the moment but it would be nice if there was another way to use the module in a secure way without restricting the node to only a private network.

Thanks for letting me know about the changes in Libp2p@0.42, I have updated the module to work with the new version - update to @organicdesign/libp2p-crdt-synchronizer@0.3.0 and you should be good to go.

Please let me know if you encounter any other issues.