The given validation code was rejected by the PVF pre-checking vote

Question

I'm running a local relay for testing, which is all working great. The issue comes in when joining a parachain to the network - I can get as far as submitting an upload for the genesis state and WASM. It sits for a minute, goes through an epoch or two, but then I get an event called "PvfCheckRejected".

Reading through some issues on the Polkadot repository, I can find that this seems to occur when 2/3 validators are unable to process the uploaded WASM in a reasonable amount of time, but the runtime used to upload fine and other than an upgrade to polkadot-v0.9.43 we haven't changed anything. I've rebuilt the chain spec a few times and regenerated the WASM and genesis state, but we're still getting the same response. I doubt it's a problem with the relay chain spec, since the relay's validators are working perfectly, and I did just double-check to make sure the validation code matches the validation code found in the chain spec.

I'm wondering if there's something I'm overlooking.

Our chain spec is https://github.com/fennelLabs/Fennel-Protocol/blob/terraform/plain-parachain-chainspec.json

If it's helpful, the WASM for the runtime on that chain is located next to the chain spec at https://github.com/fennelLabs/Fennel-Protocol/blob/terraform/para-1000-wasm

Bootnode runner script:https://github.com/fennelLabs/Fennel-Protocol/blob/terraform/boot.sh

Bootnode log with -ldebug: https://app.warp.dev/block/xQm8E0wWaMRIzXcVjff18V

Main collator runner script: https://github.com/fennelLabs/Fennel-Protocol/blob/terraform/collator-1.sh

Main collator log with -ldebug: https://app.warp.dev/block/cPt2XzChSTVfrEuN2QlJst

The parachain does onboard successfully (but doesn't begin collating, but that's a different problem) if I use the dev accounts (Alice, Bob, Charlie) for the validators and collators.

Answer 1

Thanks to the answer posted at https://substrate.stackexchange.com/a/5747/4330, I was able to determine that I was missing private keys that should have been inserted.

Following the documentation, I knew I needed to insert the babe and gran keys, but missed the other 5 keys (imon, para, asgn, audi, beef) listed in that table. They were already in my chain spec, but needed to be specifically inserted by name. Once I inserted them, it works perfectly.

From what I can tell, the PVF check runs before validation code is committed from a new parachain, and needs the para key inserted on every validator so they can sign their votes to accept or reject the parachain. That explains why validation worked perfectly but other pallets didn't.