In the Substrate ecosystem, a bricked parachain – one that has halted block production – presents a unique set of challenges. This situation can occur in various environments, notably on Rococo, our test environment, and of course on Kusama and Polkadot, which are our production environments. Recovery actions on production parachains having Kusama and Polkadot as relay chains often require governance approval, unlike those on rococo where this can be done with sudo.
This post aims to compile and share knowledge on recovering bricked parachains across these different environments, drawing from past incidents and community experiences to offer a comprehensive recovery guide.
General Recovery Resources:
- How to Recover a Parachain - A general guide on parachain recovery.
Recovery in Rococo:
- Testnet Halt: Lessons Learned - Insights from a specific incident on the Rococo testnet.
Recovery in Kusama and Polkadot:
- Kintsugi Slot Duration Bug: Post Mortem - Analysis of a slot duration bug in Kintsugi.
- Snek Stall Post Mortem - A detailed post-mortem of the Snek stall incident.
- Referendum 162 on Kusama - Details of a governance action on Kusama.
- Ajuna Network Issue #342 - A discussion on a specific issue in the Ajuna network.
- Code Substitutes in Chain Spec - StackExchange discussion on code substitutes in chain specification.
- How to Recover a Parachain - StackExchange - Another useful thread on parachain recovery.
Request for Input
I welcome additional insights, experiences, or resources from the community to further enrich this guide.
Navigating the complexities of recovering a bricked parachain is challenging, yet with the appropriate resources and the collective wisdom of the community, it is a manageable endeavor. This guide represents an initial roadmap, designed to guide teams through the process of recovery.
Our ultimate goal is to evolve this into a comprehensive, step-by-step manual, empowering project teams not only to allocate resources effectively for issue resolution but also to have a well-documented, reliable procedure to follow. This collaborative effort can significantly enhance the resilience and robustness of the Substrate ecosystem.