Custom generic js-ipfs-repo implementation for libp2p projects

[HexaField]

I’m interested in using the functionality afforded by the js-ipfs-repo package, except for non-ipfs libp2p projects. The problem I currently see is that js-ipfs-repo has it’s IPFS specific datastores (blocks, keys, pins etc) hardcoded. It would make sense to start a new project, js-libp2p-repo which clones the functionality, but allows more configurability regarding custom datastores. I am also interested in dynamically opening and closing datastores at runtime.

Curious if the js devs have any thoughts on this. I will get to work on a prototype package for js-libp2p-repo based off js-ipfs-repo.

@vasco-santos @jacobheun Sorry for the ping.

Cheers,
Josh.

[HexaField]

Started work here https://github.com/hexafield/js-libp2p-repo

[vasco-santos]

Hello @HexaField, thanks for reaching out.

I think this will really depende on the functionalities/use cases you are looking for.

Libp2p currently uses datastores for the PeerStore persistence, DHT and keychain. These datastores follow the IPFS interface-datastore. At the moment, the PeerStore persistence and the DHT will rely in the same datastore, while the keychain will rely on its own. However, if a datastore is provided on the libp2p dht config, it will be used instead. So, you can from the application layer specify one datastore per libp2p component that leverages one.

The configuration is not user friendly at all, and we will probably rethink this on https://github.com/libp2p/js-libp2p/issues/576 .

With the above in mind, there are two possibilities for managing all the needed datastores. Libp2p can provide a “datastore manager” thing or this would be the responsibility of the application layer (perhaps with a module that everyone can use). At this point, this seems outside the libp2p scope.

Can you provide us a draft of how would you see this module API and interaction with both libp2p and applications? This would help figure out where this piece should exist and how it should operate.

[HexaField]

@vasco-santos thanks for the response and explanation.

I didn’t mean to say that this should be made a feature of libp2p, instead . I was more just wondering if this is something that would benefit the wider libp2p ecosystem.

I have finished a initial implementation linked above, with a few features needing work still. The architecture includes the repo.datastoreand repo.keys from js-ipfs-repo with their associated functionality. It also includes a loader taken from the js-ipfs package. I have added openDatastore and closeDatastore functions that do as they say, enabling datastores to be added at runtime (for use cases such as dynamically using multiple DHTs at runtime).

Is there a process to go through to submit to review to have it made part of the https://github.com/libp2p organisation? I’m more curious to just get involved with js-libp2p development seeing as it has become an integral part of my application.

Thanks!

[vasco-santos]

Sorry for taking longer to answer, but we are getting the last bits ready to ship libp2p@0.30

@HexaField

I didn’t mean to say that this should be made a feature of libp2p, instead . I was more just wondering if this is something that would benefit the wider libp2p ecosystem.

Yes, that is precisely why I was trying to understand if it should be a libp2p piece, or wider ecosystem

@HexaField

I have finished a initial implementation linked above, with a few features needing work still. The architecture includes the repo.datastore and repo.keys from js-ipfs-repo with their associated functionality. It also includes a loader taken from the js-ipfs package. I have added openDatastore and closeDatastore functions that do as they say, enabling datastores to be added at runtime (for use cases such as dynamically using multiple DHTs at runtime).

This seems nice to have. I will have a look next week on it.

@HexaField

Is there a process to go through to submit to review to have it made part of the https://github.com/libp2p organisation? I’m more curious to just get involved with js-libp2p development seeing as it has become an integral part of my application.

I do not thing we have a process. It is usually organic, but I can have a look and get back to you.
Also, one important aspect in maintenance. Repos in the libp2p org are expected to be maintained and I would leave this out of the core maintainers, at least for now. If we had a libp2p shipyard like https://github.com/ipfs-shipyard/ would probably be the best place.

Anyway, I can give you more details next week.

[vasco-santos]

Hey @HexaField

Sorry for the late answer, finally we shipped libp2p@0.30 and I am catching up on all the other things. I was looking into your repo, it would be worth having a README section where people can see how this would be integrated with a libp2p instance. This is important to understand the big picture on how people would use it.

Can you please add it?

[vasco-santos]

Regarding modules under a specific organization, we don’t have any specific guideline. I talked with the team, and as I mentioned above modules within the libp2p org are being actively maintained by libp2p contributors.
Taking into account that libp2p is a modular stack, it is possible for anyone to create a compatible module. Noise and Gossipsub are examples of recommended modules that we don’t house, but we also contribute to them because they are useful/necessary for the project and community.
With the above in mind, once you have this module in a state that other people/projects could use it, you can start involving yourself with the community. There are recurrent meetups and occasions where you could present this and its advantages. Organically, we could talk about moving it to an org if it makes sense. We also want to work on the awesome-libp2p repo, where I would like to expose these type of modules for people who are looking into them.