1

Context

  • OriginTrail Parachain Mainnet is pushing significant amount of transactions (175k/day) - with tendency of growth
  • Collators are filling blocks up at ~25% of block weight, with a large pending Tx pool
  • The pool is gradually growing, and after some time most of transactions are dropped.
  • Initial hardware upgrades didn’t contribute to any increase.

enter image description here

Goal

  • Increase throughput on OriginTrail Parachain Mainnet by using up more blockspace (target: 50%+ blockweight)

Code

Hardware

Servers have 16vCPU, 8 cores 3.5GHz, 32GB RAM, NVMe disks, NUMA balancing and SMT are disabled. We did not observe any high usage on CPU (~8%) or RAM (~12%).

Our hardware benchmark:

  • CPU score: 1.08 GiBs (Min: 1.00 GiBs)
  • Memory score: 13.74 GiBs (Min: 14.32 GiBs)
  • Disk score (seq. writes): 1.37 GiBs (Min: 450.00 MiBs)
  • Disk score (rand. writes): 663.99 MiBs (Min: 200.00 MiBs)

Tx pool

You can check current TX pool size here: https://polkadot.js.org/apps/?rpc=wss%3A%2F%2Fparachain-rpc.origin-trail.network#/explorer/node

Improve this question

1 Answer 1

Reset to default
0

Collators are filling blocks up at ~25% of block weight, with a large pending Tx pool

The worst case of the storage proof usage outputted by the benchmarking is too high currently, probably leading to the effects you are seeing. There is an issue that describe the problem better.

It also depends on how you calculated this 25% usage. I assume you looked only at polkadot-js which is currently ignoring the storage proof usage.

The pool is gradually growing, and after some time most of transactions are dropped.

All transactions are probably getting rejected at some point, when they are not valid anymore. Why are they pruned from the transaction pool?

Improve this answer
7
  • Thanks for the answer. Yes, we were looking only at Polkadot-js. Do you have a recommended way to measure block usage, that also takes in account the storage proof usage? How can I check why are they pruned from transaction pool?
    – NZT
    Nov 6 at 16:18
  • Polkadot js uses the weight for it, but ignores the storage_proof size. You would need to check this as well and write some tool on your own. (Maybe already exists somewhere, but should also not be that hard) github.com/paritytech/polkadot-sdk/blob/… looking at this log should help on finding out why the tx pool prunes. txpool=debug maybe being a good idea in general.
    – bkchr
    Nov 7 at 12:40
  • Hey @NZT, we've been using evm-tps to stress test networks (it can send EVM txns but also Substrate's extrinsics). We were able to track blocks' weights via pJS using this: github.com/paritytech/evm-tps/blob/main/scripts/…
    – Artur Gontijo
    Nov 8 at 12:40
  • Easiest was to track block weight usage is to subscribe to the storage item System::BlockWeight. It does show proof size as well.
    – Oliver Tale-Yazdi
    Nov 9 at 19:48
  • Thanks for the instructions guys. I wanted to return back with latest informations. I had some issues with evm-tps tool, but I created some custom tool that tested basic ERC20 and also heavier EVM transactions that uses 350k of gas on an internally hosted parachain+relay chain. Results are that this tool created enough load that block usage went up to 72% out of 75% which is maximum for regular transactions. I looks like a safe conclusion is that we don't have an issue with proof size. On the other side, we are still confused why we cannot reach this block usage in other environments.
    – NZT
    Nov 28 at 12:24

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.