I am trying to build a decentralized datastore with the following properties:
- I have a family of objects (in my case type constrained, but that can be generalised) that users will write and share
- Objects can be hashed
- Anyone should be able to publish any object (sensorship resistance)
- Anyone should be able to search for an object, given its hash (or a filter)
- Objects can become invalid (consumed)
- The validity of an object can be checked in a decentralized manner (in my case this requiers a smart contract view function call)
- Events are available that notify if an object status as changed (invalidated)
- Invalid objects should be removed from the datastore, valid objects should stay in the datastore
- Bizantine fault tolerance, liveness …
Sharing the objects is not the difficult part, my vision is that a simple pubsub topic does the work. Storing, and garbage collecting is another matter.
I’ve started working on an (early) prototype that uses either a JS
Map or a Mongo Database for local storage, and a pubsub topic for synchronisation between the nodes. In particular, see this DHT implementation
What remains to do:
- Additional validity checks specific to my usecase (objects signature)
- Search across the network (I fear if many nodes answer the same request, the requesting node would get overwelmed)
- Better code
- Modify the pubsub layer to only relay valid messages instead of relaying everything
I wonder if anyone tryied to build something similar, and what is available out there. I guess IPFS must do something simillar, in a much better way, but without the search by filter or the event based garbage collecting. My understanding is that:
kad-dhtis a peer indexing/discovery mechanism, and is not what I should use for data sharing/storing
- Solutions like ThreadDB are not Bizantine fault tolerance since a bad actor could delete a valid object, and this delete woulf spread over the network.
Any opinion/comments/ideas ?