Adding pallet-gilt to a running chain breaks the chain

Question

I am running a simple 2-validator substrate chain with my own keys (https://github.com/paritytech/substrate, branch = polkadot-v0.9.27). I have removed pallet-gilt from the runtime and what I am trying to do is a runtime upgrade (with the setCode extrinsic) and add pallet-gilt to the runtime. The upgrade is successful but the chain breaks just before the intake_duration of pallet-gilt. The error I am getting is the following:

2022-11-04 16:04:18 panicked at 'index out of bounds: the len is 0 but the index is 299', /Users/zycon/cherry/substrate/frame/gilt/src/lib.rs:600:28
2022-11-04 16:04:18 Proposing failed: Import failed: Error at calling runtime api: Execution failed: Execution aborted due to trap: wasm trap: wasm `unreachable` instruction executed
WASM backtrace:

    0: 0x4a57eb - <unknown>!rust_begin_unwind

The function that contains the line of code that breaks is the following: https://github.com/paritytech/substrate/blob/5bfdbab8ba28523ef10e365627fed64ef2b77cbf/frame/gilt/src/lib.rs#L585-L660

That only happens when I try to add pallet-gilt to an already running chain. If I add pallet-gilt to genesis and start the chain with the pallet, I have no errors.

Answer 1

The enlarge function is called by pursue_target, which is called periodically (every n % IntakePeriod blocks) by the on_initialize hook.

If you check line 600 (exactly where the code breaks), you'll see that enlarge is trying to read the qs array, where qs is the current state of the QueueTotals Storage Item (read up on Rust Closures to understand how mutate is working to modify QueueTotals).

The problem is that because the gilt pallet was introduced via Runtime Upgrade (and most likely, you didn't perform any Storage Migration), the QueueTotals Storage Item is empty. Therefore, the pallet's Rust code breaks when it tries to read the empty qs array.

'index out of bounds: the len is 0 but the index is 299'

If you check the pallet's GenesisBuild, you'll see that QueueTotals is being populated there. So when you add gilt: Default::default(), into GenesisConfig of your node/src/chain_spec.rs (which I'm assuming you did), the QueueTotals is properly bootstrapped at genesis and the pallet runs smoothly whenever the on_initialize hook is triggered.

The only solution to this problem is to implement a Storage Migration to be performed during your Runtime Upgrade. It shouldn't be too complex, all you have to do is reproduce whatever the pallet's GenesisBuild is already doing.