Libp2p CRDT Synchronization

Users and Developers js
1

I had need for distributed state over Libp2p but I could not find anything that handles this satisfactorily, so I created a a system for synchronizing CRDTs through Libp2p. This system is a little bit similar to OrbitDB but is simpler and runs on Libp2p without the need for IPFS.

The system is only designed for use in a private network where you can trust all the other nodes to follow the same protocol and trust all the nodes with read/write access. Let me know if you have any suggestions what can be done to provide a permission system so trust between the nodes is limited.

The libp2p-crdt-synchronizer module can synchronize any CRDT following the CRDT interface from the crdt-interfaces package. The CRDT interface defines provides a way to add synchronizer, serializer and broadcaster modules which allow you to choose what protocols to use.

I have created a bunch of basic CRDT implementations in the crdts package that will work with libp2p-crdt-synchronizer module, making it easy to get started with it. Most of these implementations use CBOR encoding so they work with a variety of data types, if you want to create a version of one of these CRDTs to use a specific data type you might want to create protobuf definitions for each type which would clearly specify how it’s protocol works, I have done this in crdt-protocols but have not implemented any of them due to the work needed to get each type working.

When implementing custom encoding and/or protocols for a type of CRDT you only need to create a synchronizer (for general synchronization over streams), serializer (for serializing to disk) or broadcaster (for synchronization over pubsub).

The following code should help you to get started quickly:

import { createCRDTSynchronizer } from "@organicdesign/libp2p-crdt-synchronizer";
import { createGCounter } from "@organicdesign/crdts";

const synchronizer = createCRDTSynchronizer()(libp2p);

synchronizer.start();

const counter = createGCounter({ id: libp2p.peerId.toBytes() });

synchronizer.set("my-counter", counter);

If you give this a try, please let me know how it goes, there is a lot of room for improvement in this system and I am hoping some of you would have suggestions on how it could be done better.

Here is a list of the npm modules that make up this system:

4 Likes
2

This looks like a very useful project. Thanks for sharing !
Dealing with the permission system, I think you can add to libp2p options a

  const connectionProtector = preSharedKey({
    psk: new Uint8Array(Buffer.from(swarmKey, "base64"))
  });

This makes quite a private p2p network as only nodes sharing the same swarmKey can communicate.

I would like to try your project, but I am currently using libp2p@0.42.1 and it looks like it is not compatible, as registrar and connectionManager have been removed from libp2pnode, as explained here.
Do you plan to upgrade to a new libp2p version ? I could help if this is relevant.

3

Thanks for your interest,

PNet/PSK is currently the only feasible and intended way to use this module at the moment but it would be nice if there was another way to use the module in a secure way without restricting the node to only a private network.

Thanks for letting me know about the changes in Libp2p@0.42, I have updated the module to work with the new version - update to @organicdesign/libp2p-crdt-synchronizer@0.3.0 and you should be good to go.

Please let me know if you encounter any other issues.

4

I have updated they CRDT system to separate protocols from CRDT implementations making it easy to swap them out and/or use them in conjunction with each other, this has made the CRDTs themselves a little more complex but once you understand how they work they should be easier to write and use. The system is similar to how Libp2p itself handles modules but the modules attach to the CRDTs themselves:

// Replace 'createCRDT' with a CRDT creation method.
const crdt = createCRDT({
  synchronizers: [mySyncronizer()] // Define the synchronizer modules.
});

// Need to construct the synchronizer directly  since Libp2p does not allow arbitrary modules yet...
const synchronizer = createCRDTSynchronizer()(libp2p);

// Add the CRDT to the synchronizer itself.
synchronizer.set("myCrdt", crdt);

This fixes the issue with CRDTs of the same type but slightly different protocols being completely incompatible with each other.

You can still use my basic implementations @organicdesign/crdts if they fit your use-case without having to worry about the changes and you will have the freedom to switch to more efficient protocols later on.

5

Cool! I built something similar with yjs: GitHub - MarcoPolo/y-libp2p: Yjs provider using libp2p

1 Like