What is the best way to batch ~20 000 calls to a chain?

Question

What should I choose between these two options:

1. utility.batch()

I can batch multiple calls into one call, so it avoids making too many calls, but I'm afraid this will exceed the size of the block if I put all of my 20 000 transactions, and I would end up with something unwanted. Or does batch() can dispatch the calls on multiple blocks ?

2. the call

Or I could just signAndSend each transaction. It makes a lot of transaction, but I think it would be the best way as I'm (almost) sure everything will be sent across multiple blocks, but it will also be very long, and I'll need to make sure I don't have network/PC problem while the script is running

Answer 1

I can batch multiple calls into one call, so it avoid making too many calls, but i'm afraid this will exceed the size of the block if i put all of my 20 000 transactions, and i would end up with something unwanted... Or does batch() can dispatch the calls on multiple blocks ?

utility.batch() will be a single transaction and I guess with 20.000 inner transactions this will probably never be included in a block as it's too big.

• Are the transactions in any kind related to each other? Meaning, if tx_A fails then tx_B should not be submitted?

I ran into this problem once, and for me, the biggest issue was that the "default" transaction-pool-size is rather small. So transactions were being dropped as I tried submitting them. My rather amateurish solution was to simply restrict the number of concurrent transactions.

Not sure, if this is still working but you can find the code that manages the dispatching here: https://github.com/centrifuge/centrifuge-cli/tree/main/packages/libs/dispatcher/src

Answer 2

Alternative 3: smaller batches.

Figure out how much you are able to fit in a utiliy.batch() call, it depends on the weights of the inner transactions. When I did something similar on my chain I was able to fit 20-50 transactions into one batch.

1. Enable mortality, set lifetime to 16 blocks, so you know a batch which did not end up on chain does not suddenly come back alive.
2. Pace the submissions to not exhaust the transaction pool. Submit three to four blocks worth of extrinsics at once, to be able to fill all the block's capacity
3. wait until the first few batch extrinsics are completed, then submit a few more.

If some of the internal calls have dependencies it is possible put utility.batch(), utility.bactchAll() and utility.forceBatch() into each other to achieve failable atomic transaction without stopping the correct ones.