Hello all, I’m new to this community as a participant but I’ve been a long-time follower of the IPFS project (and libp2p indirectly through it).
I’ve been considering lately what work I could be doing personally to help the projects along, and it seems to me that the biggest hurdle for IPFS at the moment (and I assume libp2p by extension) is that of speed of data propagation in the network. Based on my research into the projects the gossip protocol would be the place to optimize in order to improve this aspect.
However, in order to do this effectively for the long-term, it would be good to have a way of evaluating gossip protocols without having to fully develop, ship, and observe them in the wild.
What I’m considering is the following:
An in-process, actor+message passing based, simulation framework. This is as-opposed to a one-to-one peer-to-process system, which would require an orchestration tool such as k8s/consul to manage all processes and their communication. Erlang would be an ideal choice for this, imo. We can spin up hundreds of thousands (millions?) of “peers”, each as an independent lightweight process, each communicating with the other solely via (a)synchronous message passing.
The framework would take care of artificially introducing latency and dropped messages between actors, as well as automatically churning the actors in the system over time. Other environmental conditions can also be implemented (such as random network partitions, bad actors, etc…). All conditions will be controlled by input variables, so that environments can be exactly re-created.
The framework would record various metrics over time, such as data propagation time, time taken to fulfill lookups, total messages sent, actor idle time, etc…
The protocols the actors use to communicate will implement a common interface which can be plugged into the framework, such that different protocols can be evaluated under identical circumstances.
The framework would not make any assumptions about communication between actors, including the shape of the network they create, how often and in what way the communicate, what state they hold, etc… Upon initialization each actor will only know of a common bootstrapping point to find some other actors, and they must take it from there.
I have other thoughts about how it would work/be used, but I don’t want this to get too long.
I’m looking for critiques of the general idea as well as advice. Does this seem like it would be a useful project? Is there some better avenue I could be focusing my time into? Are there existing projects that do this already (I looked at the testground, but that appears to be a much broader focus).
If there’s interest in this I would be happy to write up a more fleshed out proposal/spec. In any case, thanks for reading this far!