Local Relay Stops Finalizing

Question

I've been working on getting a test relay up and running in the cloud (Docker containers running on Debian virtual machines) for remote testing, and we've got everything running. The relay chain is able to sync, and sees the parachain. The collators see each other. We then noticed the following:

1. The parachain is trying to collate, but the blocks are never picked up by the relay.
2. We're getting a log that collation was not advertised to any validators.
3. That seems to be because finality is dropping behind on the relay chain, to the point where the relay's finalized block was 200 behind the current best.
4. 2 of our 3 validators are always offline, and it cycles almost perfectly. So the online validator will be A, then next block B, then next block C. That would cause the slow and stalled finality, and probably the lack of collation being advertised.

For reference, we also just registered a parathread on Rococo and put a request through for a temporary parachain slot, currently waiting for onboarding. My hope was to check whether that parachain worked appropriately.

I also tested this exact relay/para chain spec pair locally and it worked perfectly.

Has anyone seen something like this before? Thanks.

EDIT: Both the relay and the parachain are on polkadot-v0.9.37. We're running the parachain with

parachain-node --collator --force-authoring --chain raw-parachain-chainspec.json --base-path /tmp/parachain/alice --port 40333 --ws-port 9944 --rpc-port 9934 --telemetry-url "wss://telemetry.polkadot.io/submit/ 0" --rpc-methods Unsafe --name Fennel0 --node-key $(cat para-key.sh) --unsafe-ws-external --rpc-external --rpc-cors=all -- --execution wasm --chain /app/polkadotspec.json --port 30343 --ws-port 9977 --bootnodes /ip4/[relay_ip]/tcp/30333/p2p/12D3KooWLj8FgbwDtLQbA5ozWqqx6vM1n2rmjwQMczSiDveyWa5G --rpc-external --ws-external --prometheus-external --rpc-cors all

and the relay bootnode with

polkadot --validator --base-path /tmp/relay/alice --chain /app/chainspec.json --port 30333 --ws-port 9944 --rpc-port 9934 --telemetry-url "wss://telemetry.polkadot.io/submit/ 0" --name FnlDot0 --node-key $(cat node_key) --rpc-cors all --prometheus-external --rpc-methods Unsafe --rpc-external --ws-external

Update: After re-running, our Relay network is working again. All three nodes are in and finality has been running fine. Our parachain is, however, still not collating.

Here's a pastebin for the boot node's log: https://pastebin.com/raw/VAC1a50W

And one of the collator logs: https://pastebin.com/raw/EyCUCRKD

The Dockerfile for our relay nodes is https://github.com/fennelLabs/polkadot/blob/release-v0.9.37/Dockerfile

And the Dockerfile for our parachain nodes is https://github.com/fennelLabs/Fennel-Protocol/blob/terraform/Dockerfile

We are exposing more ports at runtime through docker run.

UPDATED LOGS WITH -lparachain::collator-protocol=trace

Bootnode: https://pastebin.com/raw/CnYe4M3r

Collator: https://pastebin.com/raw/EG2C7uY7

And just for more information and context, here's some sample scripts we're using to provision our VMs and run the nodes:

Bootnode: https://github.com/fennelLabs/Fennel-Protocol/blob/terraform/fennel-protocol-boot-terraform-start.sh

Collator: https://github.com/fennelLabs/Fennel-Protocol/blob/terraform/fennel-protocol-collator-1-terraform-start.sh

Answer 1

What eventually fixed this issue was an upgrade to the latest version of Polkadot. After the upgrade, and after the answer found in https://substrate.stackexchange.com/a/9036/4330, we're now collating properly.