About custom protocols and peer and stream managements

[giorgionetg]

Hi guys, what an awesome project libp2p is!! i’m new here, but I’m studying this project from 2018 almost… I’m current working on a network based on libp2p, but I have some questions over different fields.

Right now I’m using Gossipsub with different channels to simulate my custom protocol, but I guess I should build my own protocol. So, I made some easy implementation on Handle method. My misunderstanding concern about peer management and stream connections.

It’s hard to explain, but in a sample project I made a bootstrap with autodial true. With my custom protocol I have to set autodial to false and managing all peer and stream management by the protocol. Does it right? This was first question…

While I have two type of peers in my protocol, with the useful protobuffer message in my custom protocol have any kind of self routing or I have to manage like gossippub?

Thanks for the support!!

[lthibault]

Hi Giorgio,

It seems like we’re missing a lot of context here, which makes it difficult to answer your questions.

Right now I’m using Gossipsub with different channels to simulate my custom protocol, but I guess I should build my own protocol.

Could you elaborate on this a bit more? What do you mean by “simulate”? Are you ultimately planning to replace Gossipsub with a custom overlay topology / broadcast protocol? Assuming that is indeed the case, can you explain the significance of the different “channels” (do you mean “topics”?), and explain which part of your as-of-yet-unimplemented protocol they simulate?

This will help ensure our recommendations are in line with your actual goals … I’d hate for us to send you off in the wrong direction!

With my custom protocol I have to set autodial to false and managing all peer and stream management by the protocol.

What is your protocol designed to do? It seems like you wish to replace Gossipsub with another protocol that has (roughly) the same semantics as Gossipsub: namely, pubish-subscribe. In other words, it seems like you want to publish messages to topics, but you want these messages to be distributed to peers in a manner different than what Gossipsub does.

If this is indeed the case, your best bet would be to implement your own PubSubRouter and pass it to NewPubSub. This will give you a PubSub instance that uses your own routing protocol – and manages its own connections – underneath.

Regarding autodial, this rings a bell but I couldn’t find it in the docs … can you point me to it? I suspect it’s somewhat orthogonal to your query because libp2p’s pubsub system does it’s own connection management internally.

While I have two type of peers in my protocol, with the useful protobuffer message in my custom protocol have any kind of self routing or I have to manage like gossippub?

I’m not sure I understand the question. Could you try reformulating it?

I hope this helps! If not, I’m happy to try again!

[giorgionetg]

Thanks a lot for your response.

I would like to manage two kind of peers in the same network. Each kind of peer has a specific role and manage different tasks (in example: peerListener, peerWorker, where peerWorker make task and share in a topic/channel and the peerListener listen and save data to a database and make other task where peerWorker hasn’t access. Something like ACL).

To do this, I’m using GossipSub to send data between peers, but I guess and I would like to understand if it could be better make a custom protocol with protobuffer messages with a sets of rules.

I would like also to understand how could I implement something like authentication or how to discriminate some peer with a bad behaviour (probably with a scoreboard for each peers).

If you need more information I’m here, thanks in advance!!

ps: with “simulate” I mean that I use differents topics for each kind of peer, where peerWorker publish in a topic and peerListener has subscribed and publish other things in other topics.

pps: as I written above, my protocol should has a messages’ system between peers, but has some rules sets completely different from GossipSub (I would like to manage Authentication and Authorization). I studied for a while GossipSub and it is really low level library.

[lthibault]

To do this, I’m using GossipSub to send data between peers, but I guess and I would like to understand if it could be better make a custom protocol with protobuffer messages with a sets of rules.

In general, it’s preferable to build your protocols on top of the existing ones provided by libp2p. In this case, I would encourage you to publish protobuf messages on Gossipsub, and resist the urge to write a custom routing protocol. The latter is almost certain to be more complicated, less reliable, and of marginal advantage.

I would like also to understand how could I implement something like authentication or how to discriminate some peer with a bad behaviour (probably with a scoreboard for each peers).

There are two questions here: authentication (can a peer connect to another peer?) and enforcement of protocol (how can I ostracize a misbehaving peer?).

For the former, you should operate at the Host level, not the PubSub level. ConnectionGater is probably what you want.

For the latter, I would suggest passing a ValidatorEx function to RegisterTopicValidator. This will allow you to enforce valid behaviors at the swarm level.

[giorgionetg]

I got your first advice and I agree completely… I was thinking about and it become so fast so hard rewrite a custom protocol from scratch. What I have to understand how to write a custom protobuf message on gossipsub (but I guess it’s not a big trouble).

On question “can a peer connect to another peer”, I mean, in this my first configuration I start with a bootstrap node that works also as a relay node. I guess peer connection (authentication) should be free at first glance (you should know my bootstrap node to connect and start a network), then if the peer knows what it have to do, it will be authorize or not.

From my understanding of the network, I could have multiple networks (small and large) behind this library’ stack. I mean I’m using same library of ipfs but if I have my bootstrap and some kind of authorization rules my little network lives aside ipfs as in an isolate network.

Another last question: do you know some books to buy to understand peer to peer networks?

Thanks a lot to sharing your knoledge…