I've been working on getting a test relay up and running in the cloud (Docker containers running on Debian virtual machines) for remote testing, and we've got everything running. The relay chain is able to sync, and sees the parachain. The collators see each other. We then noticed the following:
- The parachain is trying to collate, but the blocks are never picked up by the relay.
- We're getting a log that collation was not advertised to any validators.
- That seems to be because finality is dropping behind on the relay chain, to the point where the relay's finalized block was 200 behind the current best.
- 2 of our 3 validators are always offline, and it cycles almost perfectly. So the online validator will be A, then next block B, then next block C. That would cause the slow and stalled finality, and probably the lack of collation being advertised.
For reference, we also just registered a parathread on Rococo and put a request through for a temporary parachain slot, currently waiting for onboarding. My hope was to check whether that parachain worked appropriately.
I also tested this exact relay/para chain spec pair locally and it worked perfectly.
Has anyone seen something like this before? Thanks.
EDIT: Both the relay and the parachain are on polkadot-v0.9.37. We're running the parachain with
parachain-node --collator --force-authoring --chain raw-parachain-chainspec.json --base-path /tmp/parachain/alice --port 40333 --ws-port 9944 --rpc-port 9934 --telemetry-url "wss://telemetry.polkadot.io/submit/ 0" --rpc-methods Unsafe --name Fennel0 --node-key $(cat para-key.sh) --unsafe-ws-external --rpc-external --rpc-cors=all -- --execution wasm --chain /app/polkadotspec.json --port 30343 --ws-port 9977 --bootnodes /ip4/[relay_ip]/tcp/30333/p2p/12D3KooWLj8FgbwDtLQbA5ozWqqx6vM1n2rmjwQMczSiDveyWa5G --rpc-external --ws-external --prometheus-external --rpc-cors all
and the relay bootnode with
polkadot --validator --base-path /tmp/relay/alice --chain /app/chainspec.json --port 30333 --ws-port 9944 --rpc-port 9934 --telemetry-url "wss://telemetry.polkadot.io/submit/ 0" --name FnlDot0 --node-key $(cat node_key) --rpc-cors all --prometheus-external --rpc-methods Unsafe --rpc-external --ws-external
Update: After re-running, our Relay network is working again. All three nodes are in and finality has been running fine. Our parachain is, however, still not collating.
Here's a pastebin for the boot node's log: https://pastebin.com/raw/VAC1a50W
And one of the collator logs: https://pastebin.com/raw/EyCUCRKD
The Dockerfile for our relay nodes is https://github.com/fennelLabs/polkadot/blob/release-v0.9.37/Dockerfile
And the Dockerfile for our parachain nodes is https://github.com/fennelLabs/Fennel-Protocol/blob/terraform/Dockerfile
We are exposing more ports at runtime through
UPDATED LOGS WITH -lparachain::collator-protocol=trace
And just for more information and context, here's some sample scripts we're using to provision our VMs and run the nodes: