Transaction pool extrinsic prioritization

Question

Currently, the transaction pool in Astar network is full with a lot of unsigned EVM transactions, which cause user transactions sent without a tip to be stuck.

We are currently researching a way to push user tx and prioritize it in the pool for fast inclusion. We see tipping mechanism to fit our needs. However, we are currently struggling with how to calculate an optimal tip.

Things I've already tried:

• Find min tip in the pending extrinsics (author_pendingExtrinsics) and add 1 or 10000 planks to it.

What is strange is that when adding 1 plank to the minimum tip, the node accepts transaction but it does not lately appear in transaction pool. However, with 10000 planks added, node immediately rejects transaction with the message "Cannot add extrinsic to the pool since it is full".

I've also checked How to calculate the optimal tip for Substrate where Gav shows that adding +1 increase the priority by the max number of these transactions per block. So in order to find an optimal tip, we need to check weights of all pending transactions to find out their priorities. However, doing so via payment_queryExtrinsicInfo will require 8092 requests to the node for each fee calculation. Maybe there is a way to do it more efficiently?

Answer 1

Substrate is generic over the notion of "Priority"

The first thing to understand is that the transaction pool itself does not determine whether a transaction is valid or what a transactions priority is. The transaction pool calls into the runtime (an off-chain call) through the TaggedTransactionQueue Runtime Api for that information.

If the runtime determines that the transaction is valid it returns a ValidTransaction struct which contains a priority.

Runtime engineers can implement the priority any way they see fit, and a few standards have emerged.

FRAME's pallet_transaction_payment

The defacto standard way to prioritize transactions in Substrate runtimes is to use Pallet Transaction Payment. In this pallet, fees are calculated based on resources consumed. The pallet provides a SignedExtension implementation which calculates priority from the fees as well as an optional flat tip amount that the user can add.

This pallet is used in every chain from the Substrate node template to Polkadot itself. Typically Substrate runtimes should use this default option unless they have a reason to do otherwise.

Ethereum's Gas metering model

In Ethereum and other smart contract blockchains resources are metered during execution, and the total resources consumed is not known until after execution. In such systems users typically specify a price per unit of gas and some kind of cap is imposed to bound the computation.

EVM compatible Substrate chains are at a crossroads

Chains like Moonbeam and Astar that are built on Substrate but support the EVM need to address the fact that while both of these systems are sound, they are not identical. Generally there are three paths you can go.

Use the FRAME model

This is not realistic because the gas metering is inherent to the EVM. Even if you hack the EVM to make it work, smart contract developers will need to target your modified EVM specifically which defeats the purpose of choosing the EVM to begin with (presuming you chose the evm to attract the existing dev ecosystem.

Use the Ethereum model

Substrate is generic enough that this could be practical especially if the EVM is the primary source of interest on your chain. You would need to remove pallet transaction payment, implement the TaggedTransactionQueue api appropriately, and make sure there is some reasonable way to charge for any out-of-evm transaction types your chain supports.

Use both, but smooth the rough edge

If your chain features the EVM prominently, but also has many non-evm transaction types, you could choose to use both systems together. Let's explore a potential approach below

Combining Substrate and Ethereum's native Approaches

When going this route there are two key challenges to overcome

1. Pallet transaction payment's tips are a single flat tip amount and are known before transaction execution. Ethereum's tips are per gas unit.

2. Substrate's notion of TransactionPriority is just a u64. So the two systems we are patching together may use different orders of magnitude for their priority.

The Moonbeam team chose to take this route, and I implemented the solution in https://github.com/PureStake/moonbeam/pull/581. The current code lives at https://github.com/PureStake/moonbeam/blob/81d69ac71da536bb41c98475ea0dcf0634d81198/runtime/moonbeam/src/lib.rs#L1200-L1262.

The algorithm is basically like this:

1. Let frame executive prioritize the transactions the normal way.
2. If it is a evm transaction, that's it; return. Otherwise, overwrite the priority:
3. Check the transaction's weight
4. Convert the weight to an effective gas using eg. Frontier's GasWeightMapping. This conversion itself is another place where FRAME and Ethereum differ, and additional care must be taken.
5. Check the transaction's fee from pallet_transaction_payment.
6. Divide to get an effective gas price.
7. This effective gas price is the new priority - same as it is done in the evm