A (blockchain governed) DHT on top of libp2p

Users and Developers
1

Hello,

I am trying to build a decentralized datastore with the following properties:

  • I have a family of objects (in my case type constrained, but that can be generalised) that users will write and share
  • Objects can be hashed
  • Anyone should be able to publish any object (sensorship resistance)
  • Anyone should be able to search for an object, given its hash (or a filter)
  • Objects can become invalid (consumed)
  • The validity of an object can be checked in a decentralized manner (in my case this requiers a smart contract view function call)
  • Events are available that notify if an object status as changed (invalidated)
  • Invalid objects should be removed from the datastore, valid objects should stay in the datastore
  • Bizantine fault tolerance, liveness …

Sharing the objects is not the difficult part, my vision is that a simple pubsub topic does the work. Storing, and garbage collecting is another matter.

I’ve started working on an (early) prototype that uses either a JS Map or a Mongo Database for local storage, and a pubsub topic for synchronisation between the nodes. In particular, see this DHT implementation

What remains to do:

  • Additional validity checks specific to my usecase (objects signature)
  • Search across the network (I fear if many nodes answer the same request, the requesting node would get overwelmed)
  • Better code
  • Modify the pubsub layer to only relay valid messages instead of relaying everything

I wonder if anyone tryied to build something similar, and what is available out there. I guess IPFS must do something simillar, in a much better way, but without the search by filter or the event based garbage collecting. My understanding is that:

  • The kad-dht is a peer indexing/discovery mechanism, and is not what I should use for data sharing/storing
  • Solutions like ThreadDB are not Bizantine fault tolerance since a bad actor could delete a valid object, and this delete woulf spread over the network.

Any opinion/comments/ideas ?

2

I think OrbitDb could be the way to go, with lients also having the consensus logic to tell what is a valid addition, and what and should things be deleted.
I’m not familiar with ThreadDb, but it seems to me that it could work too, if clients storing the information about to be deleted check that there is indeed a valid reason to do so, according to your app logic (blessed peers, owners or that records, outdated record, smart contract, etc). If the malicious node shoud “Delete this!” but no one listen (or even drop connection with it), it shouldn’t be a problem, should it?

3

The problem isn’t really to know if it is possible to fork a project like ThreadDB to add the features I need, but rather to now how hard it is, and how much knowlege about the project is required to make the change.
AFAIK, it’s not available out of the box, in the sens that I cannot just provide a few functions objects in the constructor to enable the validity check during the local deletes. If it is possible, it should be documented, but unfortunatelly project are often much more advanced in development then in documentation.

I’ll have a look ar OrbitDB. Hopefully there is some good documentation!