Transactions lifecycle

Question

I was wondering what is the process of a transaction being removed or dropped from a transaction pool. When a transaction is submitted to the network, It is validated by a node and then gossiped to its peers and they will also validate the transaction and so on. Once the transaction makes it into a block, and this block is submitted to the network, do nodes check if any of their transactions within the tx pool is within the block? Does transaction within the tx pool get validated each time they are going to be included on a block? If so, I guess the transaction gets dropped as it will be invalidated by, for example, its nonce.

Answer 1

The transaction queue in Substrate has client side APIs that check for the validity of transactions and sorts them into two queues: the ready and future queues. Transactions in the ready queue are used to build the next block. And transactions end up in the future queue if they didn't satisfy some requirements to get added to the block but are still valid for the next block.

When a transaction is submitted to the network, it is validated by a node and then gossiped to its peers and they will also validate the transaction and so on.

Any transaction that's sent to a non-authoring node will just be gossiped to other nodes in the network and enter their transaction pool until it is received by an authoring node. Then, it will check if the transaction is valid and place it in order of transaction priority to include it in the block it broadcasts to the rest of the network. The other nodes receiving this block will then verify that it's properly constructed and once a majority reach consensus, the block is executed and state transitions are applied in the runtime.

What is the process of a transaction being removed or dropped from a transaction pool.

I'll take the opportunity here to provide a more detailed answer.

How transactions are validated

Rules defined in the runtime helps the transaction pool determine whether a transaction is valid or not (using the transaction_validity interface). Basically, the client then checks if a transaction is properly "tagged" with some validity characteristics, returning a Result. You can see what these look like from the Ok() case ValidTransaction. To help perform these checks, Substrate uses a concept called signed extensions. They attach additional data to a transaction in the runtime side (see the existing SignedExtensions in FRAME), which could really be thought of as a way to enforce the expected behaviour of a transaction. With them, the client can easily verify whatever these requirements are and accept or drop the transaction accordingly.

Transaction checks

Any given transaction is "tagged" with a bunch of information that a transaction requires (i.e. properties that must be satisfied before the transaction is added to a block, for example the nonce, the weight) and that it may provide (i.e. properties that will be satisfied once the transaction is added to a block, like a tip).

The transaction queue has a runtime API, called TaggedTransactionQueue that makes regular calls to these signed extensions to keep checking that a transaction is valid before it gets put in the ready queue. Valid just means "everything that the runtime says that this transaction requires to be valid is there".

The mechanism of making repeated calls into the runtime is a useful safeguard for verifying that transactions won't fail in a block and to add anti-spam and replay protection logic. With this, a transaction is either accepted (returning ValidTransaction) or rejected (returning TransactionValidityError).

The flow from when a user connects to an application and submits a transaction, to when the validate_transaction interface is hit looks like this:

Invalid transactions

A transaction can be invalidated or dropped at different stages in the queue (its not so black and white though). For instance:

• When a transaction is too large for a block and has exceeded the its required era (it will be dropped from the future queue).
• When a transaction doesn't contain a valid signature (it will be dropped immediately).
• When a transaction in the queue is stale (as it's already been included in a block) it will be dropped from the verifying queue.
• When a transaction is too large to fit in the current block, it will be put back in a queue for a new verification round.

Now that all has been checked, assuming the transaction is valid the ValidTransactionBuilder bundles the information previously verified and the transaction gets added to the ready queue, where it's sorted by priority among other transactions in the queue for block authors to build the next block. Transactions get put in the future queue if they haven't met all their requirements but still have valid longevity, so they can be revalidated.

Have a look at the API documentation for more details. Acknowledgements: huge thanks to Joyce for the helpful diagrams.

Answer 2

Every node validates every transaction that comes from the outside. From the outside here means either by a RPC call or from another node. If a transaction is valid it is stored in the ready or future queue. Transactions are stored in the future queue if they have requirement that can not be fulfilled by any of transactions in the ready queue. Every time a node imports a block, the transaction pool is notified. The transaction pool takes goes through all transactions in the block and removes them from their own pool.

Does transaction within the tx pool get validated each time they are going to be included on a block?

Yes, part of the execution is the validation of a transaction. This validation is the same as when the transaction entered the pool/was re-validated.

If so, I guess the transaction gets dropped as it will be invalidated by, for example, its nonce.

All transaction that fail to be applied while building a block are being recorded and being removed from the transaction pool afterwards.