Error 'Both yamux endpoints are client'

I keep running into this error in my tests -
error opening initial stream for /ipfs/id/1.0.0: both yamux endpoints are client
I have a two nodes who connect to each other directly and exchange messages via streams. In addition, they both also subscribe to the same topic but never really use PubSub.
If I don’t let pubsub do a successful discovery.FindPeers, the test passes. I can come up with a working example demonstrating the error, but was wondering if someone has run into a similar issue.
thanks in advance.

Likely TCP will form a single connection and both sides will think they’re the initiator.

The best solution for now is to just connect from one side.

thank you for the reply. Appreciate it.
On peer can decide to talk to the other on its own accord, so not sure how I can make sure that connection is just initiated from one side. I have added a retry logic around NewStream and that seem to work.