4

I'm facing some strange logs when doing a runtime upgrade a Polkadot rococo-local relay chain to 0.9.16 from 0.9.15. Hoping someone can help me debug them.

Using Polkadot's release-v0.9.15 and release-v0.9.16 branches to build the binary and wasm.

My machine is: Ubuntu 20.04 LTS.

My rustup version is:

1.57.0-x86_64-unknown-linux-gnu (default)
rustc 1.57.0 (f1edd0429 2021-11-29)

The logs:

Corrupted state at [6, 222, 61, 138, 84, 210, 126, 68, 169, 213, 206, 24, 150, 24, 242, 45, 180, 180, 157, 149, 50, 13, 144, 33, 153, 76, 133, 15, 37, 184, 227, 133]    
2022-02-21 11:33:20 Corrupted state at [6, 222, 61, 138, 84, 210, 126, 68, 169, 213, 206, 24, 150, 24, 242, 45, 180, 180, 157, 149, 50, 13, 144, 33, 153, 76, 133, 15, 37, 184, 227, 133]    
2022-02-21 11:33:20 Corrupted state at [6, 222, 61, 138, 84, 210, 126, 68, 169, 213, 206, 24, 150, 24, 242, 45, 180, 180, 157, 149, 50, 13, 144, 33, 153, 76, 133, 15, 37, 184, 227, 133]    
2022-02-21 11:33:20 Corrupted state at [6, 222, 61, 138, 84, 210, 126, 68, 169, 213, 206, 24, 150, 24, 242, 45, 180, 180, 157, 149, 50, 13, 144, 33, 153, 76, 133, 15, 37, 184, 227, 133]    
2022-02-21 11:33:20 Corrupted state at [6, 222, 61, 138, 84, 210, 126, 68, 169, 213, 206, 24, 150, 24, 242, 45, 180, 180, 157, 149, 50, 13, 144, 33, 153, 76, 133, 15, 37, 184, 227, 133]    
2022-02-21 11:33:20 Corrupted state at [6, 222, 61, 138, 84, 210, 126, 68, 169, 213, 206, 24, 150, 24, 242, 45, 180, 180, 157, 149, 50, 13, 144, 33, 153, 76, 133, 15, 37, 184, 227, 133]    
2022-02-21 11:33:20 Corrupted state at [6, 222, 61, 138, 84, 210, 126, 68, 169, 213, 206, 24, 150, 24, 242, 45, 180, 180, 157, 149, 50, 13, 144, 33, 153, 76, 133, 15, 37, 184, 227, 133]    
2022-02-21 11:33:20 Corrupted state at [6, 222, 61, 138, 84, 210, 126, 68, 169, 213, 206, 24, 150, 24, 242, 45, 180, 180, 157, 149, 50, 13, 144, 33, 153, 76, 133, 15, 37, 184, 227, 133]    
2022-02-21 11:33:20 Corrupted state at [6, 222, 61, 138, 84, 210, 126, 68, 169, 213, 206, 24, 150, 24, 242, 45, 180, 180, 157, 149, 50, 13, 144, 33, 153, 76, 133, 15, 37, 184, 227, 133]    
2022-02-21 11:33:20 Corrupted state at [6, 222, 61, 138, 84, 210, 126, 68, 169, 213, 206, 24, 150, 24, 242, 45, 180, 180, 157, 149, 50, 13, 144, 33, 153, 76, 133, 15, 37, 184, 227, 133]    
2022-02-21 11:33:20 Corrupted state at [6, 222, 61, 138, 84, 210, 126, 68, 169, 213, 206, 24, 150, 24, 242, 45, 180, 180, 157, 149, 50, 13, 144, 33, 153, 76, 133, 15, 37, 184, 227, 133]    
2022-02-21 11:33:20 Corrupted state at [38, 170, 57, 78, 234, 86, 48, 224, 124, 72, 174, 12, 149, 88, 206, 247, 128, 212, 30, 94, 22, 5, 103, 101, 188, 132, 97, 133, 16, 114, 201, 215]    
2022-02-21 11:33:20 Corrupted state at [38, 170, 57, 78, 234, 86, 48, 224, 124, 72, 174, 12, 149, 88, 206, 247, 128, 212, 30, 94, 22, 5, 103, 101, 188, 132, 97, 133, 16, 114, 201, 215]    
2022-02-21 11:33:21 πŸ’€ Idle (3 peers), best: #56 (0x2b75…08af), finalized #34 (0x6f38…7c18), ⬇ 177.6kiB/s ⬆ 0.9kiB/s    
2022-02-21 11:33:24 Corrupted state at [6, 222, 61, 138, 84, 210, 126, 68, 169, 213, 206, 24, 150, 24, 242, 45, 180, 180, 157, 149, 50, 13, 144, 33, 153, 76, 133, 15, 37, 184, 227, 133]    
2022-02-21 11:33:24 Running migration to v1 for session_historical with storage version StorageVersion(1)    
2022-02-21 11:33:24 Attempted to apply migration to v1 but failed because storage version is StorageVersion(1)    
2022-02-21 11:33:24 βœ… no migration for System    
2022-02-21 11:33:24 βœ… no migration for Babe    
2022-02-21 11:33:24 βœ… no migration for Timestamp    
2022-02-21 11:33:24 βœ… no migration for Indices    
2022-02-21 11:33:24 βœ… no migration for Balances    
2022-02-21 11:33:24 βœ… no migration for TransactionPayment    
2022-02-21 11:33:24 βœ… no migration for Authorship    
2022-02-21 11:33:24 βœ… no migration for Offences    
2022-02-21 11:33:24 βœ… no migration for Historical    
2022-02-21 11:33:24 βœ… no migration for Session    
2022-02-21 11:33:24 βœ… no migration for Grandpa    
2022-02-21 11:33:24 βœ… no migration for ImOnline    
2022-02-21 11:33:24 βœ… no migration for AuthorityDiscovery    
2022-02-21 11:33:24 βœ… no migration for ParachainsOrigin    
2022-02-21 11:33:24 ⚠️ Configuration declares internal migrations (which *might* execute). On-chain `StorageVersion(1)` vs current storage version `StorageVersion(2)`    
2022-02-21 11:33:24 βœ… no migration for ParasShared    
2022-02-21 11:33:24 βœ… no migration for ParaInclusion    
2022-02-21 11:33:24 βœ… no migration for ParaInherent    
2022-02-21 11:33:24 βœ… no migration for ParaScheduler    
2022-02-21 11:33:24 βœ… no migration for Paras    
2022-02-21 11:33:24 βœ… no migration for Initializer    
2022-02-21 11:33:24 βœ… no migration for Dmp    
2022-02-21 11:33:24 βœ… no migration for Ump    
2022-02-21 11:33:24 βœ… no migration for Hrmp    
2022-02-21 11:33:24 ⚠️ ParaSessionInfo declares internal migrations (which *might* execute). On-chain `StorageVersion(0)` vs current storage version `StorageVersion(1)`    
2022-02-21 11:33:24 βœ… no migration for ParasDisputes    
2022-02-21 11:33:24 βœ… no migration for Registrar    
2022-02-21 11:33:24 βœ… no migration for Auctions    
2022-02-21 11:33:24 βœ… no migration for Crowdloan    
2022-02-21 11:33:24 βœ… no migration for Slots    
2022-02-21 11:33:24 βœ… no migration for ParasSudoWrapper    
2022-02-21 11:33:24 βœ… no migration for AssignedSlots    
2022-02-21 11:33:24 βœ… no migration for Sudo    
2022-02-21 11:33:24 βœ… no migration for Mmr    
2022-02-21 11:33:24 βœ… no migration for Beefy    
2022-02-21 11:33:24 βœ… no migration for MmrLeaf    
2022-02-21 11:33:24 βœ… no migration for ValidatorManager    
2022-02-21 11:33:24 βœ… no migration for BridgeRococoGrandpa    
2022-02-21 11:33:24 βœ… no migration for BridgeWococoGrandpa    
2022-02-21 11:33:24 βœ… no migration for BridgeRococoMessages    
2022-02-21 11:33:24 βœ… no migration for BridgeWococoMessages    
2022-02-21 11:33:24 βœ… no migration for BridgeRococoMessagesDispatch    
2022-02-21 11:33:24 βœ… no migration for BridgeWococoMessagesDispatch    
2022-02-21 11:33:24 βœ… no migration for Collective    
2022-02-21 11:33:24 βœ… no migration for Membership    
2022-02-21 11:33:24 βœ… no migration for Utility    
2022-02-21 11:33:24 βœ… no migration for Proxy    
2022-02-21 11:33:24 βœ… no migration for Multisig    
2022-02-21 11:33:24 ⚠️ XcmPallet declares internal migrations (which *might* execute). On-chain `StorageVersion(0)` vs current storage version `StorageVersion(0)`    

My polkadot-launch setup is the following:

{
    "relaychain": {
        "bin": "polkadot915",
        "chain": "rococo-local",
        "nodes": [
            {
                "name": "alice",
                "wsPort": 9944,
                "port": 30444,
                "basePath": "validator1",
                "flags": [
                    "--rpc-cors=all",
                    "--execution=wasm",
                    "--wasm-execution=compiled"
                ]
            },
            {
                "name": "bob",
                "wsPort": 9955,
                "port": 30555,
                "basePath": "validator2",
                "flags": [
                    "--rpc-cors=all",
                    "--execution=wasm",
                    "--wasm-execution=compiled"
                ]
            },
            {
                "name": "charlie",
                "wsPort": 9966,
                "port": 30666,
                "basePath": "validator3",
                "flags": [
                    "--rpc-cors=all",
                    "--execution=wasm",
                    "--wasm-execution=compiled"
                ]
            }
        ],
        "genesis": {
            "runtime": {
                "runtime_genesis_config": {
                    "configuration": {
                        "config": {
                            "validation_upgrade_frequency": 10,
                            "validation_upgrade_delay": 10
                        }
                    }
                }
            }
        }
    },
    "parachains": [
        {
            "bin": "polkadot-collator915",
            "chain": "statemine-dev",
            "nodes": [
                {
                    "wsPort": 9801,
                    "port": 31201,
                    "name": "alice",
                    "flags": [
                        "--rpc-cors=all",
                        "--rpc-port=9971",
                        "--execution=wasm",
                        "-lcumulus-collator=trace,cumulus-network=trace,cumulus-consensus=trace",
                        "--wasm-execution=compiled"
                    ]
                }
            ]
        }
    ],
    "hrmpChannels": [
    ],
    "types": {},
    "finalization": false
}

First I do the client update from polkadot915 to polkadot916 without a problem. Then the runtime upgrade produces the logs, which bricks block production after several minutes. What could be the reason for such Corrupted state at... ?

This does not reproduce with westend-local, no problem there.

1 Answer 1

1

To be clear this corrupted state was only in the logs after the RTU was started? Please try with more verbose logs, something like:

RUST_LOG=debug RUST_BACKTRACE=1 ./target/release/polkadot --chain rococo-local -lruntime=debug

This may be an incorrect/missing migration, but not clear at first glance. I also think Rococo was restarted somewhere in the 15->16 timeframe. (Overall delta)

The migrations for Rococo runtime - is showing as your logs report for the expected migrations, but a breaking change in format for sign_transaction on bridges here and a change in enum here for the bridging may be the thing at fault. But unclear if that is it to me... Is this behavior you see reproducible?

Your Answer

By clicking β€œPost Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.