Runtime upgrades: parachainSystem.enactAuthorizedUpgrade via referendum fails with `parachainSystem.ValidationDataNotAvailable`

Question

We can successfully upgrade both locally running and Rococo parachains with Sudo, but when I try upgrading a local parachain using the referenda system, it fails. I am submitting these extrinsics as Council external proposals. The authorizeUpgrade is successful, however, enactAuthorizedUpgrade fails with parachainSystem.ValidationDataNotAvailable.

I am seeing events democracy.PreimageUsed, followed by:

democracy.Executed
  refIndex: u32
    6
  result: Result<Null, SpRuntimeDispatchError>
    Err: DispatchError
    {
      Module: {
        index: 1
        error: 0x03000000
      }
    }
  type
    parachainSystem.ValidationDataNotAvailable

followed by:

scheduler.Dispatched
    Dispatched some task. 
    task: (u32, u32)
      u32
        284
      u32
        0
    id: Option<Bytes>
      0x64656d6f6372616306000000
    result: Result<Null, SpRuntimeDispatchError>
      Ok

When I look through the code it's not obvious how and where the validation data gets stored or is expected to be stored. When it's done via Sudo, I see events for parachainSystem.ValidationFunctionStored and parachainSystem.ValidationFunctionApplied that I never see during the referendum process. Why would this be different?

We're running substrate version 0.9.29 right now. I'm not sure what other information I should post, since the actual Democracy part itself is working fine. Our Democracy and Collectives configurations are nearly the same as defaults for Polkadot so we're not doing anything special. There are no runtime configurations for the cumulus dependencies. Please let me know what else is needed to help, and I'll update this question.

Answer 1

First, let's solve your problem. You don't need to call enactAuthorizedUpgrade from root. This call is actually callable by anyone, including unsigned transactions (if setup correctly). Only authorizeUpgrade is the function that requires root, as it sets up the expected hash and allows the upgrade. So, you only need to put authorizeUpgrade in your democracy proposal and after this is enacted, you call enactAuthorizedUpgrade from any account.

When I look through the code it's not obvious how and where the validation data gets stored or is expected to be stored.

The validation data is stored with the set_validation_data inherent. As the scheduler (that enacts the proposal) is running in on_initialize, it executes the proposal before the inherents are applied. Inherents are the first extrinsics being applied to the block and represent external data that the runtime can not access, like the timestamp or this validation data.

"Funnily" I left exactly one year before a comment about this on github.