raul
April 17, 2019, 4:56pm
1
Hey friends!
The abstractions that make up go-libp2p (interfaces for transports, muxing, security, discovery, etc.) were split across ~15 repos, which made it hard to reason about the system, and had other detrimental effects across a number of axes.
There’s an effort underway to consolidate all core abstractions and types under go-libp2p-core. Check out the branch here:
Discussion is happening in:
opened 08:46PM - 16 Apr 19 UTC
closed 01:39PM - 27 May 19 UTC
## ... into [`go-libp2p-core`](https://github.com/libp2p/go-libp2p-core/tree/con… solidate-skeleton)! 🎉
We have positive consensus about consolidating the abstractions and core types of go-libp2p in a single repository. The rationale is clear:
* Reduces cognitive load when reasoning about the system.
* Makes go-libp2p significantly more approachable for newcomers and contributors.
* Decreases repo count by around 20%, thus alleviating maintenance burden and unnecessary complexity.
* Offers an opportunity to exterminate inconsistently named repos, and to fix package naming snafu.
* Enables building alternative go-libp2p backends easier by targeting a single collection of abstractions (e.g. lightweight proxy that binds to a remote/co-local libp2p daemon).
## Alternatives considered
There were two reasonable approaches, with their pros, cons, and notes:
1. Pull the abstractions and core types into `go-libp2p` under a dedicated package: `skel` (skeleton), `abstract`, or similar. Move out the constructor, host implementations and protocols.
* pro: putting the skeleton inside the entrypoint repo (go-libp2p) reduces indirection for beginners, when learning about the system by browsing code.
* con: stops being the development entrypoint; the constructor living elsewhere _adds_ an indirection when hacking together a libp2p app.
* con: landslide change.
* note: doing this progressively (absorbing abstractions, then moving out hosts and constructor) was impossible thanks to gomod's affinity for VCS; it was rather all-or-nothing, as gomod's approach does not play well with module cross-references.
* e.g. go-libp2p hosts reference go-libp2p-interface-connmgr which aliases back to go-libp2p.
* Locking in cross-references via commit hashes or tags is a catch-22.
2. Place abstractions in a new module, e.g. `go-libp2p-core`.
* pro: constructor stays in `go-libp2p`, thus that module continues serving as a true entrypoint for development.
* con: new users need to discover that the conceptual model of the system lives in another repo.
## Selected approach
I started with option 1, and after consultation with @Stebalien, decided to migrate to option 2. Hence, option 2 is what I'm proposing to adopt.
Here is a diagram of the proposed refactor, to ease digestion:
_WARNING: This diagram is now outdated, but it was accurate when the PR was filed._
## Proposed changes
All of these PRs are work in progress (despite the PRs not signalling so) and subject to change based on feedback.
This is a sizeable architectural change that will bring long-awaited improvements to our development workflow and experience, but we have to get it right. Consensus is the decision process I am invoking.
** recipients:**
go-libp2p-core: https://github.com/libp2p/go-libp2p-core/pull/1
go-libp2p-testing: https://github.com/libp2p/go-libp2p-testing/pull/1
**donors:**
https://github.com/libp2p/go-conn-security/pull/12
~https://github.com/libp2p/go-flow-metrics/pull/6~
https://github.com/libp2p/go-libp2p-discovery/pull/24
https://github.com/libp2p/go-libp2p-host/pull/29
https://github.com/libp2p/go-libp2p-interface-connmgr/pull/16
https://github.com/libp2p/go-libp2p-interface-pnet/pull/12
https://github.com/libp2p/go-libp2p-net/pull/44
https://github.com/libp2p/go-libp2p-peer/pull/48
https://github.com/libp2p/go-libp2p-peerstore/pull/69
https://github.com/libp2p/go-libp2p-pnet/pull/27
https://github.com/libp2p/go-libp2p-protocol/pull/8
https://github.com/libp2p/go-libp2p-routing/pull/43
https://github.com/libp2p/go-libp2p-transport/pull/49
https://github.com/libp2p/go-stream-muxer/pull/28
https://github.com/libp2p/go-libp2p-crypto/pull/60
https://github.com/libp2p/go-testutil/pull/24 (test helpers and suites)
**modules requiring backwards-compatible changes:**
https://github.com/libp2p/go-libp2p/pull/601
https://github.com/libp2p/go-libp2p-blankhost/pull/20
https://github.com/multiformats/go-multistream/pull/37
Summary of change in these modules: The `Host` interface was referencing the `go-multistream.MultistreamMuxer` struct directly. I extracted an interface `protocol.Switch` out of it, which in turn embeds `protocol.Router` (methods to attach handlers) and a `protocol.Negotiator` (methods to negotiate protocols).
`Host` now references the new interface, and backwards-compatible changes were required in those modules.
## Testing the changes for backwards compatibility
Unfortunately CI won't work because the go-libp2p-peerstore dependency is affected by this roadblock in the go toolchain: https://github.com/golang/go/issues/31491
To test building these changes locally:
```sh
$ git clone --recursive git@github.com:libp2p/workspace-go-libp2p.git
$ cd workspace-go-libp2p
$ # add the multiformats/go-multistream
$ git submodule add git@github.com:multiformats/go-multistream.git
$ # check out all consolidate-skeleton branches
$ git submodule foreach 'git checkout consolidate-skeleton || true'
$ # make modules cross-reference each other locally
$ ./workspace.sh local
$ # workaround to fix the import path of go-multistream in the replace directives
$ git submodule foreach 'go mod edit -replace=github.com/multiformats/go-multistream=../go-multistream -dropreplace=github.com/libp2p/go-multistream'
$ git submodule foreach 'go build ./...'
```
(TODO: run all tests)
## Notes
(will keep enhancing this section with notes that come off my mind)
- In deprecated donors, I have used `moved` as a package alias and `deprecated.go` as a filename. No strong opinion here, but I think it intuitively screams the signal that you should not use this.
- In order to test for backwards compatibility and non-breakage with high fidelity, I have intentionally left untouched everything in go-libp2p that was not strictly necessary to change.
- All implementations are therefore referencing deprecated types.
- Once we're confident we're fully backwards compatible, we should practice what we preach and switch to go-libp2p-core everywhere.
- I have not reviewed any docs during this process, I just copy pasted. We should take this opportunity to do so.
- I don't want to hold off on merging too long, as this has been a significant undertaking, a big code coordination effort, and I'm afraid branches will diverge rapidly, code will slide, and it'll be a mess.
## TODO
Open points:
- [X] what do we do with go-libp2p-crypto? (decision: absorb)
- [X] alias central types at root of go-libp2p-core.
- [X] what do we do with go-libp2p-loggables? (decision: nothing)
- [ ] what do we do with global variables? They cannot be aliased, as they are assigned by value. Keeping them in the original packages is misleading, and removing them introduces breakage.
- [ ] what do we do with helper functions?
- [ ] review all docs (@yusefnapora?).
- [ ] review open issues and PRs in donor repos, and close or move over to core.
Come by and let us know what you think!
4 Likes
bigs
April 17, 2019, 10:50pm
2
long awaited! thanks for taking this on, and thanks to go mod for making this less painful, ha!
1 Like