Node doesn't peer with a network specifying bootNodes

Question

we have an existing parachain network consisting of one archive node and 4 rpc nodes that are peering and synced with the relay chain. They run inside docker containers in different virtual machines. We wanted to add a single collator node to this set, by specifying as bootNodes the four rpc nodes of the network. But for some reason, the collator node won't see any peers. I can confirm that the ProtocolId and the version of the nodes is the same (v0.9.24).

The RPC nodes are running with the following parameters (some were skipped):

--allow-private-ipv4 --unsafe-ws-external --rpc-cors all
--rpc-external --rpc-methods Unsafe
--enable-offchain-indexing=TRUE
--ws-port 8844 --port 30335 --rpc-port 9955
--chain chain_spec_raw.json --execution=wasm
--node-key-file node-key-file.txt
--
--port 30334 --chain kusama.json
--execution=wasm --unsafe-pruning
--pruning=256

When running our collator node, basically with similar parameters but indicating --collator, the node doesn't find any peer. I tried adding -lsync=debug to see what information could we get regarding not finding any peers (the bootNodes exist, and they are running), and what I found is a flood of messages like these:

2022-10-20 15:16:44.215 DEBUG tokio-runtime-worker sync: Too many full nodes, rejecting 12D3KooWB1ctQt4yhs5CTzKArW9TgWbVdY2z5BLw6SPbTSed45YY    
2022-10-20 15:16:44.215 DEBUG tokio-runtime-worker sync: 12D3KooWB1ctQt4yhs5CTzKArW9TgWbVdY2z5BLw6SPbTSed45YY disconnected    
2022-10-20 15:16:44.228 DEBUG tokio-runtime-worker sync: Too many full nodes, rejecting 12D3KooWQBrCAKiXydGW4Ab2J9X29Bvd6V54Awvzj8BWAi2xXQ22    
2022-10-20 15:16:44.228 DEBUG tokio-runtime-worker sync: 12D3KooWQBrCAKiXydGW4Ab2J9X29Bvd6V54Awvzj8BWAi2xXQ22 disconnected    
2022-10-20 15:16:44.238 DEBUG tokio-runtime-worker sync: Too many full nodes, rejecting 12D3KooWJzrRnu9So3JfPaBcfYUuzERrL6712aLoGfW4YQzAURmB    
2022-10-20 15:16:44.238 DEBUG tokio-runtime-worker sync: 12D3KooWJzrRnu9So3JfPaBcfYUuzERrL6712aLoGfW4YQzAURmB disconnected    
2022-10-20 15:16:44.238 DEBUG tokio-runtime-worker sync: Too many full nodes, rejecting 12D3KooWP51x2EsLiNssP61JVHJCgGDqLpCCkZMFBVFRE95xAfSy    
2022-10-20 15:16:44.238 DEBUG tokio-runtime-worker sync: 12D3KooWP51x2EsLiNssP61JVHJCgGDqLpCCkZMFBVFRE95xAfSy disconnected    
2022-10-20 15:16:44.254 DEBUG tokio-runtime-worker sync: Too many full nodes, rejecting 12D3KooWGFiLPZ7dE3Arx3TWAdweTCKwYwkxpkJ33REbpvrU5p6v    
2022-10-20 15:16:44.254 DEBUG tokio-runtime-worker sync: 12D3KooWGFiLPZ7dE3Arx3TWAdweTCKwYwkxpkJ33REbpvrU5p6v disconnected    
2022-10-20 15:16:44.259 DEBUG tokio-runtime-worker sync: Too many full nodes, rejecting 12D3KooWHRABR98BfKMbKp6J4dMuSRNCdMsCTGJ9sovQxA8XjbPg 

What does it mean that there are too many full nodes, while our node has 0 peers? What could be the reason that the network is rejecting the new node? Do you know about other levels of logging or filtering that we can use to tackle down the problem here?

EDIT: Adding more logs first 100 lines with debug level (https://logpaste.com/jd8tbxaB) and here maybe some relevant logs grepping by warning messages (https://logpaste.com/eaFlkjVJ)

Thanks in advance!

Answer 1

I solved the problem, and it was related to the multiaddress specification of the bootNodes in the chain spec.

I specified them as

/dns4/penrpc-kus-00.pendulumchain.tech/tcp/30335/p2p/12D3KooWPNRRYr8XZpPrjbugepo48PLF5URFoDh1epT5oRfNRQ8E

but apparently the correct form was to add the ws protocol also like this:

/dns4/penrpc-kus-00.pendulumchain.tech/tcp/30335/ws/p2p/12D3KooWPNRRYr8XZpPrjbugepo48PLF5URFoDh1epT5oRfNRQ8E

It's the first time I see this on a node, in general p2p is enough. I only found out by taking a look at the exposed address of one of the RPC nodes.

Can it be that the way they are constructed depends on parameters like --unsafe-rpc-external or --unsafe-ws-external?