Gossipsub Unknown Messages

Implementers and Contributors
1

Most of the Gossipsub spec is pretty clear but after speaking with a member of Protocol Labs (Raúl) I clarified the notion of unknown messages, and I summarized some of the ambiguity around unknown messages below. For reference, the line in the spec referring to unkown messages is this: “On IHAVE(ids) it checks the seen set and requests unknown messages with an IWANT message.”

  1. What defines an unknown message and how does a router come to possess an unknown message?
  • We listen to IHAVE gossip, and if we spot a message that we don’t have (it’s not in our message cache) then that is an unknown message, so we send an IWANT, and then the peer returns the requested message(s) to us.
  1. If we get an IHAVE for a message that’s not in our message cache, then how do we determine if we want that message or if the message is old? Is it based on comparing the seqno of the messages that are in the cache to the message listed in the IHAVE?
  • No, if you receive an IHAVE for a message not in your message cache, you will create an IWANT message and request it. If you consume a message twice because you receive an IHAVE late for a message that slid off your cache, you will consume it twice by adding it to your cache again. Apps should make sure that messages have an idempotent effect

Hopefully, this helps others, and if any member of the Protocol Labs team wants to confirm (or modify, if something is incorrect) this that would be greatly appreciated

2 Likes
2

Thanks for jotting down the notes from our discussion! I’m sure they’ll be useful to other members.

Invoking @vyzo to cross-check the conclusions.

3

There is a secondary cache, the seen cache, that tracks message ids with a longer interval than the message cache and this determines if a message is known or not.
The current seen cache interval is 2min, but this can be increased programmatically (tradeoff is memory consumption).