0

I am experiencing some issues with node discovery after upgrading from substrate 2.0 to substrate 3.0

After upgrading our chain from using substrate 2.0 to substrate 3.0 (Upgrading the client & runtime with a forkless upgrade), the nodes fail to discover peers through Kademlia. mDNS is disabled (on purpose) and only bootnodes are being discovered.

If I the same build and create a new chainspec and chain starting from block Kademlia discovery is working as intended.

The version of libp2p used is 0.36.0. I add –log sub-libp2p flag on the node and I can see in the logs that the substream is opened, the Kademlia discovery being invoked, but throws this error Failed to get record: NotFound

validator-1_1   | 2022-02-17 15:33:59.909 TRACE tokio-runtime-worker sub-libp2p: External API <= Message(12D3KooWEyoppNCUx8Yx66oV9fJnriXwCcXwDDUA2kj6vnc6iDEp, SetId(2))    
validator-1_1   | 2022-02-17 15:33:59.910 TRACE tokio-runtime-worker sub-libp2p: External API => Notification(PeerId("12D3KooWEyoppNCUx8Yx66oV9fJnriXwCcXwDDUA2kj6vnc6iDEp"), "/paritytech/grandpa/1", 150 bytes)    
validator-1_1   | 2022-02-17 15:33:59.910 TRACE tokio-runtime-worker sub-libp2p: Handler(PeerId("12D3KooWEyoppNCUx8Yx66oV9fJnriXwCcXwDDUA2kj6vnc6iDEp")) <= Sync notification    
validator-1_1   | 2022-02-17 15:33:59.926 TRACE tokio-runtime-worker sub-libp2p: Handler(ConnectionId(TaskId(0))) => Notification(12D3KooWEyoppNCUx8Yx66oV9fJnriXwCcXwDDUA2kj6vnc6iDEp, SetId(2), 150 bytes)    
...
validator-1_1   | 2022-02-18 08:20:58.033 TRACE tokio-runtime-worker sub-libp2p: Addresses of PeerId("12D3KooWEyoppNCUx8Yx66oV9fJnriXwCcXwDDUA2kj6vnc6iDEp"): ["/dns/validator-0/tcp/30333", "/dns/validator-0/tcp/30333", "/dns/validator-0/tcp/30333", "/dns/validator-0/tcp/30333/p2p/12D3KooWEyoppNCUx8Yx66oV9fJnriXwCcXwDDUA2kj6vnc6iDEp"]    
...
2022-02-18 08:21:04.063 TRACE tokio-runtime-worker sub-libp2p: Libp2p => Failed to get record: NotFound { key: Key(b"h\xe9\xa0\xe3h\x17\x1e\xfc\xfds\xc0\x8c\x11\x8c\n\x9f\x03\xa6\xe5\xac\xa6\xd5{\x11\x83\x94\xeda\xc9\xa6P\xc9"), closest_peers: [PeerId("12D3KooWEyoppNCUx8Yx66oV9fJnriXwCcXwDDUA2kj6vnc6iDEp")] }    
validator-1_1   | 2022-02-18 08:21:04.063 TRACE tokio-runtime-worker sub-libp2p: Libp2p 
...

Are there any know issues in this area? Closest I could find was this, but the fix is already applied to our codebase. https://github.com/paritytech/substrate/pull/7985/files

I also tried to rotate the session keys in the validators, just in case there was an issue with old keys but nothing changed.

Any ideas/suggestions?

3
  • Which substrate 3 version? A monthly tag, master or something else?
    – Squirrel
    Feb 21, 2022 at 13:07
  • v3.0.0 tag from codebase
    – thduke
    Feb 23, 2022 at 8:03
  • I think 3.0.0 tag was a while ago. I would recommend choosing the latest monthly tag or the last tagged polkadot version or master.
    – Squirrel
    Mar 18, 2022 at 9:20

1 Answer 1

2

This sounds like some incompatibilities from the numerous libraries involved. Substrate 2.0 is very behind. I would rebuild your node starting from the latest node template to fix this.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.