Can a parachain recover from a "block taking too long" error?

Question

We have a parachain testnet running on a local relaychain based on rococo v2.

The Parachain is based on Cumulus (Polkadot and Substrate) v0.9.36 - so at the time of writing is the most up-to-date version;

We are attempting a large batch transaction making transfers from one user to a very large number (n=1000) of users - a simulation of a coin distribution after a crowdloan.

The transaction broke the parachain because the batch transaction apparently caused the block production to take took too long. The block time of 909ms mentioned seems absurdly short for it to break block production on a parachain that basically had a 12 second block time.

This also looks a lot like an attack vector, which would concern us greatly as we also have a production chain.

Is there any way to purge the extrinsics in the queue or recover the parachain some other way? It does not seem possible to restart the parachain once it has started producing blocks.

2023-01-13 18:11:18 [Parachain] ⌛️ Discarding proposal for slot 139469456; block production took too long
2023-01-13 18:12:54 [Parachain] 🎁 Prepared block for proposing at 563273 (909 ms) [hash: 0x2ad58e983e647f8f97dd955c0bfb0c0d7302465bc8ba1825184ce533f770073a; parent_hash: 0x790f…6483; extrinsics (3): [0xcd8e…bd1b, 0xb911…e88d, 0x85da…adb5]]

Answer 1

If this is just one transaction, you can restart the collators to get rid of the transaction. The transaction pool doesn't persist any transactions. But be aware that other nodes also may have the transaction in their pool and so nodes that restart may see it again. If your transaction is has a lifetime attached (not being immortal) it will also be removed from the pool at some point.

This also looks a lot like an attack vector, which would concern us greatly as we also have a production chain.

It is an attack vector. This issue shows that you are not having done proper benchmarking. Otherwise the block production should stop early while trying to apply the extrinsic as the weight is more than what fits in the block.