Substrate node not propagating transactions

Question

So we have a very strange bug that is happening when sending transaction to a substrate (crust node in this instance). We are sending signed transaction to a crust node via polkadot.js as such:

const api = await ApiPromise.create({
    provider,
    typesBundle: CrustTypesBundle,
    throwOnConnect: true,
  });
await api.isReady;
const signedTx = api.tx("");
  
await signedTx.send();

We do this for thousands of transaction which all get propagated and accepted on chain. But after a while the transactions get "stuck". Stuck means we send let say 200 transactions to the node. Node gives 0 errors. But only first 80 get actually accepted on chain.

Then if we send the 81. transaction again we get an error:

- 1013: Transaction Already Imported: Any

or

RPC-CORE: submitExtrinsic(extrinsic: Extrinsic): Hash:: 1012: Transaction is temporarily banned

And we can wait for any amount of time or send the transaction to the same node any out of times the transaction does not get propagated on chain.

BUT if we send the exact same transaction to a different RPC node then the transaction gets accepted normally to the chain, and the other transactions waiting on the previous node also get accepted and everything works normally from that point on for X amount of time until when this happens again.

Node that transactions have a lifetime era so they can be transmitted at any time.

EDIT:

The same thing has now happened with subsocial chain as well. We also contacted the RPC provider which checked that there are no errors on the node and no information in the logs that could explain this.

Answer 1

Your problem is basically what is described in this issue.

The TLDR is that the parachain is building multiple blocks at the same height. This means that it is forking. This leads to the transaction pool assuming that certain transactions already got included and thus, it tries to apply transactions that require a higher nonce. These transactions are failing and make the transaction pool ban them for some short time.