I am experiencing some issues with node discovery after upgrading from substrate 2.0 to substrate 3.0
After upgrading our chain from using substrate 2.0 to substrate 3.0 (Upgrading the client & runtime with a forkless upgrade), the nodes fail to discover peers through Kademlia. mDNS is disabled (on purpose) and only bootnodes are being discovered.
If I the same build and create a new chainspec and chain starting from block Kademlia discovery is working as intended.
The version of libp2p used is 0.36.0.
I add –log sub-libp2p flag on the node and I can see in the logs that the substream is opened, the Kademlia discovery being invoked, but throws this error Failed to get record: NotFound
validator-1_1 | 2022-02-17 15:33:59.909 TRACE tokio-runtime-worker sub-libp2p: External API <= Message(12D3KooWEyoppNCUx8Yx66oV9fJnriXwCcXwDDUA2kj6vnc6iDEp, SetId(2))
validator-1_1 | 2022-02-17 15:33:59.910 TRACE tokio-runtime-worker sub-libp2p: External API => Notification(PeerId("12D3KooWEyoppNCUx8Yx66oV9fJnriXwCcXwDDUA2kj6vnc6iDEp"), "/paritytech/grandpa/1", 150 bytes)
validator-1_1 | 2022-02-17 15:33:59.910 TRACE tokio-runtime-worker sub-libp2p: Handler(PeerId("12D3KooWEyoppNCUx8Yx66oV9fJnriXwCcXwDDUA2kj6vnc6iDEp")) <= Sync notification
validator-1_1 | 2022-02-17 15:33:59.926 TRACE tokio-runtime-worker sub-libp2p: Handler(ConnectionId(TaskId(0))) => Notification(12D3KooWEyoppNCUx8Yx66oV9fJnriXwCcXwDDUA2kj6vnc6iDEp, SetId(2), 150 bytes)
...
validator-1_1 | 2022-02-18 08:20:58.033 TRACE tokio-runtime-worker sub-libp2p: Addresses of PeerId("12D3KooWEyoppNCUx8Yx66oV9fJnriXwCcXwDDUA2kj6vnc6iDEp"): ["/dns/validator-0/tcp/30333", "/dns/validator-0/tcp/30333", "/dns/validator-0/tcp/30333", "/dns/validator-0/tcp/30333/p2p/12D3KooWEyoppNCUx8Yx66oV9fJnriXwCcXwDDUA2kj6vnc6iDEp"]
...
2022-02-18 08:21:04.063 TRACE tokio-runtime-worker sub-libp2p: Libp2p => Failed to get record: NotFound { key: Key(b"h\xe9\xa0\xe3h\x17\x1e\xfc\xfds\xc0\x8c\x11\x8c\n\x9f\x03\xa6\xe5\xac\xa6\xd5{\x11\x83\x94\xeda\xc9\xa6P\xc9"), closest_peers: [PeerId("12D3KooWEyoppNCUx8Yx66oV9fJnriXwCcXwDDUA2kj6vnc6iDEp")] }
validator-1_1 | 2022-02-18 08:21:04.063 TRACE tokio-runtime-worker sub-libp2p: Libp2p
...
Are there any know issues in this area? Closest I could find was this, but the fix is already applied to our codebase. https://github.com/paritytech/substrate/pull/7985/files
I also tried to rotate the session keys in the validators, just in case there was an issue with old keys but nothing changed.
Any ideas/suggestions?
