I have looked for documentation, that describes the behaviour of the boostrap further, but I haven’t found any resource.
Let’s say I have service1, service2 and service3.
The bootstrap lists are: service1: [service2], service2: [service1], service3: [service1,service2].
Service3 starts first an can’t connect to the other nodes. From this I’m guessing, that if the node tries to bootstrap with a valid multiaddr and the peer doesn’t answer it won’t try again.
However I noticed if the multiaddr is invalid, the nodes tries again and again every x seconds, as defined in ‘interval’ config.
From a quick scan of the code the only thing I can think of is that the bootstrap is stopped after connections have been tried and not succeeded. I haven’t seen the code that is using the bootstrap, so I can’t tell that this is true.
Is there any way to keep retrying until the nodes connect?
Are you connected to other nodes on the network? If so, this could be due to the way auto dial works with peer discovery.
The bootstrap system is pretty crude at the moment. It just continues to emit the same list of peers as “discovered” on an interval and doesn’t stop.
The default minimum connection count for js-libp2p is 25. If you have at least that many connections libp2p won’t auto dial discovered peers, which will cause bootstrap peers to not be connected again.
Are these connections you want to maintain, or are the meant to just be entry points into the network?
No service 3 has no other connections. Service 1 and 2 are able to find each other.
I’m not at the minimum connection count.
Well those are connections I want to maintain, but also they are entry points. I’m using them as a private network (without restriction as in pnet) and somehow want a robust bootstrap configuration. But I’m wondering why Service 3 can’t connect to Service 1 and 2 if it starts first and 1 and 2 are not online when 3 starts.
It should eventually redial those when it starts. Can you check some logs on that node? Setting the DEBUG env variable will enable logs. DEBUG=libp2p*,-libp2p:mplex*,-libp2p:connection-manager*. This setting will turn most logs on aside from the noisier mplex and conn manager logs.
Initial guess is that it can’t connect to the other 2 for some reason. The logs should show whether that’s true or not.
today at 10:06 AM 2020-07-14T08:06:44.993Z libp2p:transports adding TCP
today at 10:06 AM 2020-07-14T08:06:44.999Z libp2p:transports adding Circuit
today at 10:06 AM 2020-07-14T08:06:45.029Z libp2p:transports creating listener for TCP on /ip4/0.0.0.0/tcp/6005
today at 10:06 AM 2020-07-14T08:06:45.045Z libp2p:tcp:listener Listening on { address: '0.0.0.0', family: 'IPv4', port: 6005 }
today at 10:06 AM 2020-07-14T08:06:45.055Z libp2p:gossipsub starting
today at 10:06 AM 2020-07-14T08:06:45.059Z libp2p:gossipsub started
today at 10:06 AM 2020-07-14T08:06:45.076Z libp2p:peer-store:address-book added provided multiaddrs for QmSmcB1AurmmjBTCUM1cfo1PpGdZW4pzqxqS868oFssTJ1
today at 10:06 AM 2020-07-14T08:06:45.087Z libp2p:dialer 1 tokens request, returning 1, 99 remaining
today at 10:06 AM 2020-07-14T08:06:45.195Z libp2p:peer-store:address-book added provided multiaddrs for QmQkKYHm1UcTVKhpLjgdmiiQJd7775ztryxeaKK1NDRGo3
today at 10:06 AM 2020-07-14T08:06:45.197Z libp2p:dialer 1 tokens request, returning 1, 98 remaining
today at 10:06 AM 2020-07-14T08:06:45.201Z libp2p:peer-store:address-book added provided multiaddrs for QmP1pUM5JAxo2KyUQBKgpz8qCXaPKGWy6q8YvwsNFeAuAQ
today at 10:06 AM 2020-07-14T08:06:45.209Z libp2p:dialer 1 tokens request, returning 1, 97 remaining
today at 10:06 AM 2020-07-14T08:06:45.216Z libp2p:peer-store:address-book added provided multiaddrs for QmQSXeEV5gFXGZ2jzWY83PztEu9aXnWro19EJNm6Dcgzyh
today at 10:06 AM 2020-07-14T08:06:45.222Z libp2p:dialer 1 tokens request, returning 1, 96 remaining
today at 10:06 AM 2020-07-14T08:06:45.228Z libp2p:tcp dialing {"family":"ipv6","host":"validator-b","transport":"tcp","port":6002}
today at 10:06 AM 2020-07-14T08:06:45.238Z libp2p:tcp dialing {"family":"ipv6","host":"validator-c","transport":"tcp","port":6003}
today at 10:06 AM 2020-07-14T08:06:45.247Z libp2p:tcp dialing {"family":"ipv6","host":"validator-a","transport":"tcp","port":6001}
today at 10:06 AM 2020-07-14T08:06:45.249Z libp2p:tcp dialing {"family":"ipv6","host":"validator-d","transport":"tcp","port":6004}
today at 10:06 AM 2020-07-14T08:06:45.258Z libp2p:gossipsub JOIN [ 'Transaction' ]
today at 10:06 AM 2020-07-14T08:06:45.267Z libp2p:gossipsub JOIN [ 'Consensus' ]
today at 10:06 AM 2020-07-14T08:06:45.269Z libp2p:gossipsub JOIN [ 'TimeSync' ]
today at 10:06 AM 2020-07-14T08:06:45.270Z libp2p:gossipsub JOIN [ 'SecretSharing' ]
today at 10:06 AM 2020-07-14T08:06:45.271Z libp2p:gossipsub JOIN [ 'Introduction' ]
today at 10:06 AM 2020-07-14T08:06:45.309Z libp2p:dialer token 0 released
today at 10:06 AM 2020-07-14T08:06:45.317Z libp2p:dialer:error AggregateError:
today at 10:06 AM 2020-07-14T08:06:45.320Z libp2p:error could not connect to discovered peer AggregateError:
today at 10:06 AM 2020-07-14T08:06:45.322Z libp2p:dialer token 1 released
today at 10:06 AM 2020-07-14T08:06:45.324Z libp2p:dialer:error AggregateError:
today at 10:06 AM 2020-07-14T08:06:45.325Z libp2p:error could not connect to discovered peer AggregateError:
today at 10:06 AM 2020-07-14T08:06:45.326Z libp2p:dialer token 2 released
today at 10:06 AM 2020-07-14T08:06:45.330Z libp2p:dialer:error AggregateError:
today at 10:06 AM 2020-07-14T08:06:45.330Z libp2p:error could not connect to discovered peer AggregateError:
today at 10:06 AM 2020-07-14T08:06:45.332Z libp2p:dialer token 3 released
today at 10:06 AM 2020-07-14T08:06:45.336Z libp2p:dialer:error AggregateError:
today at 10:06 AM 2020-07-14T08:06:45.338Z libp2p:error could not connect to discovered peer AggregateError:
today at 10:06 AM 2020-07-14T08:06:55.224Z libp2p:dht:random-walk:Qmf14ePA start
today at 10:06 AM 2020-07-14T08:06:55.225Z libp2p:dht:random-walk:Qmf14ePA running query 0
today at 10:06 AM 2020-07-14T08:06:55.227Z libp2p:dht:random-walk:Qmf14ePA query:QmfQwjiQBAz7hiQhT9K4sGQMN8BMi5tKKXbGZ8z6p5GxcM
today at 10:06 AM 2020-07-14T08:06:55.228Z libp2p:dht:Qmf14ePA findPeer QmfQwjiQBAz7hiQhT9K4sGQMN8BMi5tKKXbGZ8z6p5GxcM
today at 10:06 AM 2020-07-14T08:06:55.229Z libp2p:dht:Qmf14ePA findPeerLocal QmfQwjiQBAz7hiQhT9K4sGQMN8BMi5tKKXbGZ8z6p5GxcM
today at 10:06 AM 2020-07-14T08:06:55.230Z libp2p:dht:random-walk:Qmf14ePA:error query 0 finished with error Error: Peer lookup failed
today at 10:06 AM at Object.findPeer (/usr/src/app/node_modules/libp2p-kad-dht/src/peer-routing/index.js:132:23)
today at 10:06 AM at RandomWalk._query (/usr/src/app/node_modules/libp2p-kad-dht/src/random-walk.js:155:14)
today at 10:06 AM at /usr/src/app/node_modules/libp2p-kad-dht/src/random-walk.js:113:11
today at 10:06 AM 2020-07-14T08:06:55.232Z libp2p:dht:random-walk:Qmf14ePA finished queries
today at 10:06 AM 2020-07-14T08:06:55.232Z libp2p:dht:Qmf14ePA:error random-walk:error Error: Peer lookup failed
today at 10:06 AM at Object.findPeer (/usr/src/app/node_modules/libp2p-kad-dht/src/peer-routing/index.js:132:23)
today at 10:06 AM at RandomWalk._query (/usr/src/app/node_modules/libp2p-kad-dht/src/random-walk.js:155:14)
today at 10:06 AM at /usr/src/app/node_modules/libp2p-kad-dht/src/random-walk.js:113:11
today at 10:07 AM 2020-07-14T08:07:00.066Z libp2p:peer-store:address-book the addresses provided to store are already stored for QmSmcB1AurmmjBTCUM1cfo1PpGdZW4pzqxqS868oFssTJ1
today at 10:07 AM 2020-07-14T08:07:00.070Z libp2p:peer-store:address-book the addresses provided to store are already stored for QmQkKYHm1UcTVKhpLjgdmiiQJd7775ztryxeaKK1NDRGo3
today at 10:07 AM 2020-07-14T08:07:00.075Z libp2p:peer-store:address-book the addresses provided to store are already stored for QmP1pUM5JAxo2KyUQBKgpz8qCXaPKGWy6q8YvwsNFeAuAQ
today at 10:07 AM 2020-07-14T08:07:00.081Z libp2p:peer-store:address-book the addresses provided to store are already stored for QmQSXeEV5gFXGZ2jzWY83PztEu9aXnWro19EJNm6Dcgzyh
today at 10:07 AM 2020-07-14T08:07:15.067Z libp2p:peer-store:address-book the addresses provided to store are already stored for QmSmcB1AurmmjBTCUM1cfo1PpGdZW4pzqxqS868oFssTJ1
today at 10:07 AM 2020-07-14T08:07:15.069Z libp2p:peer-store:address-book the addresses provided to store are already stored for QmQkKYHm1UcTVKhpLjgdmiiQJd7775ztryxeaKK1NDRGo3
today at 10:07 AM 2020-07-14T08:07:15.071Z libp2p:peer-store:address-book the addresses provided to store are already stored for QmP1pUM5JAxo2KyUQBKgpz8qCXaPKGWy6q8YvwsNFeAuAQ
today at 10:07 AM 2020-07-14T08:07:15.078Z libp2p:peer-store:address-book the addresses provided to store are already stored for QmQSXeEV5gFXGZ2jzWY83PztEu9aXnWro19EJNm6Dcgzyh
today at 10:07 AM 2020-07-14T08:07:30.068Z libp2p:peer-store:address-book the addresses provided to store are already stored for QmSmcB1AurmmjBTCUM1cfo1PpGdZW4pzqxqS868oFssTJ1
today at 10:07 AM 2020-07-14T08:07:30.070Z libp2p:peer-store:address-book the addresses provided to store are already stored for QmQkKYHm1UcTVKhpLjgdmiiQJd7775ztryxeaKK1NDRGo3
today at 10:07 AM 2020-07-14T08:07:30.071Z libp2p:peer-store:address-book the addresses provided to store are already stored for QmP1pUM5JAxo2KyUQBKgpz8qCXaPKGWy6q8YvwsNFeAuAQ
today at 10:07 AM 2020-07-14T08:07:30.073Z libp2p:peer-store:address-book the addresses provided to store are already stored for QmQSXeEV5gFXGZ2jzWY83PztEu9aXnWro19EJNm6Dcgzyh
today at 10:07 AM 2020-07-14T08:07:45.068Z libp2p:peer-store:address-book the addresses provided to store are already stored for QmSmcB1AurmmjBTCUM1cfo1PpGdZW4pzqxqS868oFssTJ1
today at 10:07 AM 2020-07-14T08:07:45.070Z libp2p:peer-store:address-book the addresses provided to store are already stored for QmQkKYHm1UcTVKhpLjgdmiiQJd7775ztryxeaKK1NDRGo3
today at 10:07 AM 2020-07-14T08:07:45.071Z libp2p:peer-store:address-book the addresses provided to store are already stored for QmP1pUM5JAxo2KyUQBKgpz8qCXaPKGWy6q8YvwsNFeAuAQ
today at 10:07 AM 2020-07-14T08:07:45.073Z libp2p:peer-store:address-book the addresses provided to store are already stored for QmQSXeEV5gFXGZ2jzWY83PztEu9aXnWro19EJNm6Dcgzyh
today at 10:07 AM 2020-07-14T08:07:55.234Z libp2p:dht:random-walk:Qmf14ePA start
today at 10:07 AM 2020-07-14T08:07:55.235Z libp2p:dht:random-walk:Qmf14ePA running query 0
today at 10:07 AM 2020-07-14T08:07:55.235Z libp2p:dht:random-walk:Qmf14ePA query:QmQMzqRZ2RsU5JTuVjwiWGoz9e7LqY3PTNbgJ4yyiuJ8bY
today at 10:07 AM 2020-07-14T08:07:55.235Z libp2p:dht:Qmf14ePA findPeer QmQMzqRZ2RsU5JTuVjwiWGoz9e7LqY3PTNbgJ4yyiuJ8bY
today at 10:07 AM 2020-07-14T08:07:55.235Z libp2p:dht:Qmf14ePA findPeerLocal QmQMzqRZ2RsU5JTuVjwiWGoz9e7LqY3PTNbgJ4yyiuJ8bY
today at 10:07 AM 2020-07-14T08:07:55.235Z libp2p:dht:random-walk:Qmf14ePA:error query 0 finished with error Error: Peer lookup failed
today at 10:07 AM at Object.findPeer (/usr/src/app/node_modules/libp2p-kad-dht/src/peer-routing/index.js:132:23)
today at 10:07 AM at RandomWalk._query (/usr/src/app/node_modules/libp2p-kad-dht/src/random-walk.js:155:14)
today at 10:07 AM at /usr/src/app/node_modules/libp2p-kad-dht/src/random-walk.js:113:11
today at 10:07 AM 2020-07-14T08:07:55.236Z libp2p:dht:random-walk:Qmf14ePA finished queries
today at 10:07 AM 2020-07-14T08:07:55.236Z libp2p:dht:Qmf14ePA:error random-walk:error Error: Peer lookup failed
today at 10:07 AM at Object.findPeer (/usr/src/app/node_modules/libp2p-kad-dht/src/peer-routing/index.js:132:23)
today at 10:07 AM at RandomWalk._query (/usr/src/app/node_modules/libp2p-kad-dht/src/random-walk.js:155:14)
today at 10:07 AM at /usr/src/app/node_modules/libp2p-kad-dht/src/random-walk.js:113:11
today at 10:08 AM 2020-07-14T08:08:00.071Z libp2p:peer-store:address-book the addresses provided to store are already stored for QmSmcB1AurmmjBTCUM1cfo1PpGdZW4pzqxqS868oFssTJ1
today at 10:08 AM 2020-07-14T08:08:00.073Z libp2p:peer-store:address-book the addresses provided to store are already stored for QmQkKYHm1UcTVKhpLjgdmiiQJd7775ztryxeaKK1NDRGo3
today at 10:08 AM 2020-07-14T08:08:00.079Z libp2p:peer-store:address-book the addresses provided to store are already stored for QmP1pUM5JAxo2KyUQBKgpz8qCXaPKGWy6q8YvwsNFeAuAQ
today at 10:08 AM 2020-07-14T08:08:00.082Z libp2p:peer-store:address-book the addresses provided to store are already stored for QmQSXeEV5gFXGZ2jzWY83PztEu9aXnWro19EJNm6Dcgzyh
today at 10:08 AM 2020-07-14T08:08:15.075Z libp2p:peer-store:address-book the addresses provided to store are already stored for QmSmcB1AurmmjBTCUM1cfo1PpGdZW4pzqxqS868oFssTJ1
today at 10:08 AM 2020-07-14T08:08:15.076Z libp2p:peer-store:address-book the addresses provided to store are already stored for QmQkKYHm1UcTVKhpLjgdmiiQJd7775ztryxeaKK1NDRGo3
today at 10:08 AM 2020-07-14T08:08:15.078Z libp2p:peer-store:address-book the addresses provided to store are already stored for QmP1pUM5JAxo2KyUQBKgpz8qCXaPKGWy6q8YvwsNFeAuAQ
today at 10:08 AM 2020-07-14T08:08:15.079Z libp2p:peer-store:address-book the addresses provided to store are already stored for QmQSXeEV5gFXGZ2jzWY83PztEu9aXnWro19EJNm6Dcgzyh
today at 10:08 AM 2020-07-14T08:08:30.072Z libp2p:peer-store:address-book the addresses provided to store are already stored for QmSmcB1AurmmjBTCUM1cfo1PpGdZW4pzqxqS868oFssTJ1
today at 10:08 AM 2020-07-14T08:08:30.073Z libp2p:peer-store:address-book the addresses provided to store are already stored for QmQkKYHm1UcTVKhpLjgdmiiQJd7775ztryxeaKK1NDRGo3
today at 10:08 AM 2020-07-14T08:08:30.075Z libp2p:peer-store:address-book the addresses provided to store are already stored for QmP1pUM5JAxo2KyUQBKgpz8qCXaPKGWy6q8YvwsNFeAuAQ
today at 10:08 AM 2020-07-14T08:08:30.077Z libp2p:peer-store:address-book the addresses provided to store are already stored for QmQSXeEV5gFXGZ2jzWY83PztEu9aXnWro19EJNm6Dcgzyh
today at 10:08 AM 2020-07-14T08:08:45.076Z libp2p:peer-store:address-book the addresses provided to store are already stored for QmSmcB1AurmmjBTCUM1cfo1PpGdZW4pzqxqS868oFssTJ1
today at 10:08 AM 2020-07-14T08:08:45.081Z libp2p:peer-store:address-book the addresses provided to store are already stored for QmQkKYHm1UcTVKhpLjgdmiiQJd7775ztryxeaKK1NDRGo3
today at 10:08 AM 2020-07-14T08:08:45.082Z libp2p:peer-store:address-book the addresses provided to store are already stored for QmP1pUM5JAxo2KyUQBKgpz8qCXaPKGWy6q8YvwsNFeAuAQ
today at 10:08 AM 2020-07-14T08:08:45.084Z libp2p:peer-store:address-book the addresses provided to store are already stored for QmQSXeEV5gFXGZ2jzWY83PztEu9aXnWro19EJNm6Dcgzyh
today at 10:08 AM 2020-07-14T08:08:55.241Z libp2p:dht:random-walk:Qmf14ePA start
today at 10:08 AM 2020-07-14T08:08:55.242Z libp2p:dht:random-walk:Qmf14ePA running query 0
today at 10:08 AM 2020-07-14T08:08:55.243Z libp2p:dht:random-walk:Qmf14ePA query:QmPsvH9vTdhqevhMRp3VqF99GdBxj5zdM889x5FdozY3ra
today at 10:08 AM 2020-07-14T08:08:55.243Z libp2p:dht:Qmf14ePA findPeer QmPsvH9vTdhqevhMRp3VqF99GdBxj5zdM889x5FdozY3ra
today at 10:08 AM 2020-07-14T08:08:55.243Z libp2p:dht:Qmf14ePA findPeerLocal QmPsvH9vTdhqevhMRp3VqF99GdBxj5zdM889x5FdozY3ra
today at 10:08 AM 2020-07-14T08:08:55.243Z libp2p:dht:random-walk:Qmf14ePA:error query 0 finished with error Error: Peer lookup failed
today at 10:08 AM at Object.findPeer (/usr/src/app/node_modules/libp2p-kad-dht/src/peer-routing/index.js:132:23)
today at 10:08 AM at RandomWalk._query (/usr/src/app/node_modules/libp2p-kad-dht/src/random-walk.js:155:14)
today at 10:08 AM at /usr/src/app/node_modules/libp2p-kad-dht/src/random-walk.js:113:11
today at 10:08 AM 2020-07-14T08:08:55.244Z libp2p:dht:random-walk:Qmf14ePA finished queries
today at 10:08 AM 2020-07-14T08:08:55.245Z libp2p:dht:Qmf14ePA:error random-walk:error Error: Peer lookup failed
today at 10:08 AM at Object.findPeer (/usr/src/app/node_modules/libp2p-kad-dht/src/peer-routing/index.js:132:23)
today at 10:08 AM at RandomWalk._query (/usr/src/app/node_modules/libp2p-kad-dht/src/random-walk.js:155:14)
today at 10:08 AM at /usr/src/app/node_modules/libp2p-kad-dht/src/random-walk.js:113:11
today at 10:09 AM 2020-07-14T08:09:00.076Z libp2p:peer-store:address-book the addresses provided to store are already stored for QmSmcB1AurmmjBTCUM1cfo1PpGdZW4pzqxqS868oFssTJ1
today at 10:09 AM 2020-07-14T08:09:00.078Z libp2p:peer-store:address-book the addresses provided to store are already stored for QmQkKYHm1UcTVKhpLjgdmiiQJd7775ztryxeaKK1NDRGo3
today at 10:09 AM 2020-07-14T08:09:00.080Z libp2p:peer-store:address-book the addresses provided to store are already stored for QmP1pUM5JAxo2KyUQBKgpz8qCXaPKGWy6q8YvwsNFeAuAQ
today at 10:09 AM 2020-07-14T08:09:00.084Z libp2p:peer-store:address-book the addresses provided to store are already stored for QmQSXeEV5gFXGZ2jzWY83PztEu9aXnWro19EJNm6Dcgzyh
today at 10:09 AM 2020-07-14T08:09:15.078Z libp2p:peer-store:address-book the addresses provided to store are already stored for QmSmcB1AurmmjBTCUM1cfo1PpGdZW4pzqxqS868oFssTJ1
today at 10:09 AM 2020-07-14T08:09:15.082Z libp2p:peer-store:address-book the addresses provided to store are already stored for QmQkKYHm1UcTVKhpLjgdmiiQJd7775ztryxeaKK1NDRGo3
This are the logs.
Great thanks, yeah the dial retry is not happening. We’re working on a fix for this.
Great to hear! Let me know if I can be of any help.
This should be fixed in 0.28.7. The connection manager will now regularly check to try and establish the minimum connection watermark. This should also ensure if all connections are ever dropped (loss of internet connection), connections will eventually be reestablished to known peers when connectivity resumes.
It indeed does work now. Thanks for dealing with that issue!