LibP2P on the Polkadot Network

Users and Developers
1

I’m looking to build a DHT crawler for Polkadot.

A few month ago, I chatted with one of the devs, Pierre, and he mentioned that I might have to set the user-agent to a string that contains “substrate”. I’ve tried to connect to the network but i’m not getting a response. I’m using a boot node gotten from the Polkadot Source: https://github.com/paritytech/polkadot/search?q=Qm&unscoped_q=Qm

Has anyone got a connection working to the Polkadot network through LibP2P? Has anything changed, maybe some new network ID?

Thanks

2

I don’t know why it doesn’t show up in your GitHub search (maybe because the file is too large), but Polkadot’s bootnodes are listed here: https://raw.githubusercontent.com/paritytech/polkadot/master/service/res/alexander.json
(and for future reference, chain specs will be here)

Whatever happens, the connection initiation consists of secio and mplex/yamux, and that should be totally compatible with go-libp2p and js-libp2p.

Afterwards, however, you node will get disconnected if:

  • It doesn’t support ping or identify.
  • The response from identify doesn’t contain the string “substrate” (to prevent IPFS nodes from connecting).
1 Like
3

Thanks Pierre,

Do you know of any existing code that uses LibP2P (either NodeJs or GoLang) that is connecting to Substrate? I’m able to connect to IPFS but still unable to connect to the substrate network. It looks like that some of the libraries like (NodeJs multiaddr) might be built for IPFS only.

I was able to ping the IPFS bootnodes but when I just try to ping (using the standard setup provided by libp2p tutorial) the Substrate bootnodes, I get either one of two errors:

When pinging /ip4/40.117.153.33/tcp/30363/p2p/QmPiGU1jwL9UDw2FMyMQFr9FdpF9hURKxkfy6PWw6aLsur
The response is:

pinging remote peer at  /ip4/40.117.153.33/tcp/30363/ipfs/QmPiGU1jwL9UDw2FMyMQFr9FdpF9hURKxkfy6PWw6aLsur
error pinging:  { Error: No available transports to dial peer QmPiGU1jwL9UDw2FMyMQFr9FdpF9hURKxkfy6PWw6aLsur!

When Pinging: /ip4/104.211.48.247/tcp/30363/ipfs/QmYT3p4qGj1jwb7hDx1A6cDzAPtaHp3VR34vmw5BsXXB8D
Response:

pinging remote peer at  /ip4/104.211.48.247/tcp/30363/ipfs/QmYT3p4qGj1jwb7hDx1A6cDzAPtaHp3VR34vmw5BsXXB8D
error pinging:  Error: dialed to the wrong peer, Ids do not match

My config:

    transport: [
      TCP,
      WebSockets
    ],
    connEncryption: [
      SECIO
    ],
    streamMuxer: [
      Multiplex
    ]
  }

Looks like the multiaddr library is converting all p2p protocol into ipfs not sure if that’s intentional and if that’ll work for the substrate network.

If you can point me to some reference code (go/node) for connecting to Substrate using libP2P that would be very helpful.

Thanks

4

Commented over at https://github.com/multiformats/js-multiaddr/issues/93#issuecomment-514549901. Multiaddr shouldn’t be an issue. It looks like the BootNodes from the config aren’t available as TCP is getting an ECONNREFUSED error.

  libp2p:tcp:dial Connecting to 30363 40.117.153.33 +21s
  libp2p:switch:dialer:queue:error TCP:work { Error: connect ECONNREFUSED 40.117.153.33:30363
    at TCPConnectWrap.afterConnect [as oncomplete] (net.js:1106:14)
  errno: 'ECONNREFUSED',
  code: 'ECONNREFUSED',
  syscall: 'connect',
  address: '40.117.153.33',
  port: 30363 } +11s
5

It looks like the Alexander bootnodes are now down, indeed. We’re preparing to launch Kusama soon, so the devops team has probably stopped the old bootnodes.

In the meanwhile you should be able to try to connect to Flaming Fir, the Substrate testnet: https://raw.githubusercontent.com/paritytech/substrate/8c1ed555bfb2bcbe5abe4e901c3a6a1796b273ca/node/cli/res/flaming-fir.json
However, it is not unlikely that we kill it soon as well!

6

Thanks Pierre,

I’m trying the new bootnodes that you mentioned and I’m getting a lot of :

pinging remote peer at  /ip4/35.246.224.91/tcp/30334/ws/ipfs/QmSk5HQbn6LhUwDiNMseVUjuRYhEtYj4aUZ6WfWoGURpdV
error pinging:  Error: dialed to the wrong peer, Ids do not match

Jacob mentioned that this is most likely: node has connected but SECIO failed due to the actual ID is not matching the ID that was passed in. Any clue on why this would be the case?

Thanks

7

Oh, I suspect that what happens is that the public key’s format is secp256k1, which the Go and Js implementations don’t support.

We’re switching to ed25519, which Go and Js might support?
If you spawn your own Substrate (or Polkadot) node, its network key will be ed25519 by default.

8

Hey Pierre,

Just to follow up on this, do you know roughly when the GoLang or JS implementations will be able to support this? We are working on one of the grant projects and we have finished 2/3 milestones and we need to get the DHT crawler working for the final step to complete the project

Thanks.

9

I am not involved in the Go or JS implementations.
As for Polkadot, we should be deploying Kusama soon, which will use ED25519 keys.

10

@zthomas Is the project open. If it is, I would like to check it out.

11

Error: dialed to the wrong peer, Ids do not match

I faced similar issue when connecting Rust & JS via libp2p. The problem is that peer-id JS computes peer id from public key like this:

const computeDigest = (pubKey, cb) => {
  if (pubKey.bytes.length <= 42) {
    const digest = mh.encode(pubKey.bytes, 'identity') // This get's executed
    cb(null, digest)
  } else {
    pubKey.hash((err, digest) => {
      cb(err, digest)
    })
  }
}

Result is that in JS mh.encode(pubKey.bytes, 'identity') results in addresses of format 12D3KooWSwNXzEeGjgwEocRJBzbdoDqxbz3LdrwgSuKmKeGvbM4G, while in Rust PeerId for the same public key looks like QmTESkr2vWDCKqiHVsyvf4iRQCBgvNDqBJ6P3yTTDb6haw.

Error then happens in libp2p-secio/src/handshake/crypto.js in identify function in JS:

if (state.id.remote.toB58String() !== remoteId.toB58String()) { // <-- remoteId here is computed from public key in 12D3 format, while state.id.remote is in Qm format
    return callback(new Error('dialed to the wrong peer, Ids do not match'))
}

Currently, I don’t have a solution for that except to patch either JS or Rust.

12

I came to the same conclusion. They have following in rust-libp2p in peer_id.rs:62:

// Note: the correct behaviour, according to the libp2p specifications, is the
// commented-out code, which consists it transmitting small keys un-hashed. However, this
// version and all previous versions of rust-libp2p always hash the key. Starting from
// version 0.13, rust-libp2p accepts both hashed and non-hashed keys as input
// (see `from_bytes`). Starting from version 0.14, rust-libp2p will switch to not hashing
// the key (a.k.a. the correct behaviour).
// In other words, rust-libp2p 0.13 is compatible with all versions of rust-libp2p.
// Rust-libp2p 0.12 and below is **NOT** compatible with rust-libp2p 0.14 and above.
/*let hash_algorithm = if key_enc.len() <= MAX_INLINE_KEY_LENGTH {
    multihash::Hash::Identity
} else {
    multihash::Hash::SHA2256
};*/
let hash_algorithm = multihash::Hash::SHA2256;

If you uncomment correct behaviour (and remove let hash_algorithm = multihash::Hash::SHA2256;) it starts to produce correct peerId