I am a developer on a distributed system. I am trying to add a peer to peer communication layer using libp2p. I specifically want this to be a completely independent network. I do not want to connect to the main IPFS network or anything like that. There will be one or more well-known bootstrap peers/seeds. The nodes are run by multiple parties using cloud infrastructure like AWS, so mDNS is not an option.
How do I bootstrap a network using a DHT for discovery and routing? Every example I can find either used direct connections, mDNS, or someone else’s servers. I specifically want to build a network without using any outside servers. I can build a network by having one or more seeds and gossiping peer addresses, but I want to be able to use NAT traversal and all the other powerful features that come with DHT routing/discovery. And I don’t want to reinvent the wheel.
I am open to using something other than DHT if there is a better alternative that works in my scenario
Here’s a very rough guideline one one possible implementation:
- Start bootstrap servers on a cloud provider. Make sure these have circuit relay v2 enabled.
- New libp2p nodes should have these bootstrap servers hard coded. And should connect to them on start.
- The DHT protocol will then query random keys to build up a routing table (a list of peers to connect to). This will also tell the peers about itself.
Note that nodes behind a NAT will need to be relayed over a relay server before they can do holepunching (in order to run dcutr).
I hope that’s enough to get you started. Out of curiosity, which implementation are you using?
1 Like
I’m using go-libp2p. Is there anything special I need to do to test this locally? Is it feasible to test locally, with the bootstrap servers running on my PC or LAN? For now I’d be happy to get peer discovery working at all (proper peer discovery, not my janky DIY version). Supporting nodes behind a NAT can come later.
The project is Accumulate. We’re using Tendermint for consensus and Tendermint runs its own p2p layer, which we don’t touch. But Accumulate consists of multiple partitions, each of which is a distinct Tendermint network, and those partitions need to communicate. Right now they’re doing that via direct HTTP connections. I’ve rebuilt the API on top of libp2p. I manually implemented peer discovery after I gave up on DHT, but its rather naive and not entirely reliable.
You should be able to follow the chat-with-rendezvous example. If you don’t want to use the libp2p bootstrap nodes you can run your own bootstrap node with this patch: withBootstrapMode.diff · GitHub (adds a bootstrap mode that just starts up and doesn’t try to connect to anyone else).
Then open 3 terminals:
- In the first run the bootstrap node:
go run . -listen /ip4/127.0.0.1/tcp/6666 -is_bootstrap=true
- In the next run one peer and connect to the above bootstrap node:
go run . -listen /ip4/127.0.0.1/tcp/6667 -peer "/ip4/127.0.0.1/tcp/6666/p2p/<bootstrapper's peer id from the "we are ..." log output>"
- In the last terminal, run another peer:
go run . -listen /ip4/127.0.0.1/tcp/6668 -peer "/ip4/127.0.0.1/tcp/6666/p2p/12D3KooWFazi9eaEZG1NortrGwprH3NVv2L7aC3n9GwyYjUZmmRd"
The peers from 2 & 3 will open a chat session in this example. This can scale up to many peers and many bootstrap nodes. Once peers learn about other peers in the DHT (while will happen automatically as they start looking for random peers in the key space) the bootstrap servers are not needed. The bootstrap servers are only needed for new nodes joining the network to bootstrap themselves.
Thanks, I got your example working with your patch. That helps me understand where I was going wrong before. Am I correct in understanding this example is essentially the same as what I should do when setting up the real network, plus circuit relay? Though I assume for the real network I’d set up one bootstrap server as in your example, and then the subsequent bootstrap servers would need to be pointed at that first one (or at least at one of the previously started servers).