Substrate supports "pluggable consensus" so a developer can choose from among several consensus algorithms. The main Substrate repository contains four algorithms:
Some of these (eg babe and grandpa) can even be used together in a single node. What are the differences between each consensus algorithm, and which ones can, or should, be used together?
For a blockchain to continue growing and adding new transactions in a meaningful way, two things must happen to solve the problem of distributed consensus. Typically, these jobs are performed by full nodes as is the case with the default Substrate node.
Block Authoring. Nodes create new blocks. Each new block contains a reference to a parent block.
Block Finalization. When forks appear in the chain, Nodes must choose which side of the fork to consider the real or "canonical" one. Once a block is Finalized, the canonical chain will always contain it.
In addition to the four consensus engines mentioned in the question, there are also some community-contributed consensus engines in the Substrate ecosystem. Let's look at each of the algorithms individually, and see how they accomplish these two tasks tasks.
Aura provides block authoring. In aura a known set of authorities are allowed to produce blocks. The authorities must be chosen before block production begins and all authorities must know the entire authority set. Time is divided up into "slots" of a fixed length. During each slot one block is produced, and the authorities take turns producing blocks in order forever.
In Aura, forks only happen when it takes longer than the slot duration for a block to traverse the network. Thus forks are uncommon in good network conditions.
Babe also provides block authoring. Like, Aura, it is a slot-based consensus algorithm with a known set of validators. In addition, each validator is assigned a weight which must be assigned before block production begins. Unlike Aura, the authorities don't take turns in order. Instead, during each round, each authority generates a pseudorandom number using a VRF. If the random number is lower than their weight, they are allowed to produce a block.
Because multiple validators may be able to produce a block during the same slot, forks are more common in Babe than they are in Aura, and are common even in good network conditions.
Substrate's implementation of Babe also has a fallback mechanism for when no authorities are chosen in a given slot.
Proof of Work also provides block authoring. Unlike Babe and Aura, it is not slot-based, and does not have a known authority set. In proof of Work, anyone can produce a block at any time, so long as they can solve a computationally challenging problem (typically a hash preimage search). The difficulty of this problem can be tuned to provide a statistical target block time.
Nimbus is a community-contributed block authoring framework for writing identity-based consensus engines rather than a specific individual engine. Both Aura and Babe can be expressed in the nimbus framework. Nimbus doubles down on Substrate's use of web assembly by moving even the core of the consensus logic into wasm. By doing this, Nimbus enables consensus hotswaps, and transforms the problem of writing a consensus engine into one of writing a pallet.
Manual Seal is not for production use, but rather for education use and integration tests. When using manual seal, any node may author a block at any time and it is considered valid. In production this would lead to unmanageable spam.
Each of the block authoring mechanisms we've discussed previously needs to know where on the chain is should build the next block. Methods such as the "longest chain rule" or the "heaviest observed subtree" can serve as a rule of thumb. When coupled with a PoW consensus algorithm, these methods provide a genuine probabilistic finality. That is, with each new block that is added to a chain, the probability that it will be reverted decreases, approaching zero. However, if true certainty that a block is final is desired, a more sophisticated game can be used.
Grandpa provides deterministic finalization. It has a known weighted authority set like Babe. However, Grandpa does not author blocks; it just listens to gossip about blocks that have been produced by some authoring engine like the ones discussed above. Each authority participates in two rounds of voting on blocks. The details of the voting are beyond the scope of this post. Once 2/3 of the grandpa authorities have voted for a particular block, it is considered finalized.
Aleph BFT is a community-contributed deterministic finality engine using a known authority set. Like Grandpa, it is not responsible for authoring blocks, and depends on an external block authoring mechanism. AlephBFT is similar to HashGraph but is unencumbered by patents, and provides finality over a traditional blockchain rather than replacing the blockchain with the graph entirely.
Manual Finality is the counterpart to manual seal. It is not for production use, but for learning and integration tests. In manual finality, any node can mark a block final at any time. In production, this would lead to nodes having inconsistent views of the chain.
In general a block authoring engine and a finality gadget can be used together in a single chain, as Babe and Grandpa are in the code linked in the question. When such a system is used, block authoring engines must be made aware of blocks that are finalized so that they don't waste time building on top of blocks that will never be in the canonical chain.
Note on weights: Babe, Grandpa, and many other algorithms that do not come bundled with Substrate rely on weights. Consensus algorithms themselves typically do not dictate how the weights are assigned, but rather assume that they are assigned somehow leaving the assignment to an external mechanism. In public networks, it is common to assign weights based on how many tokens are staked. In the default Substrate node, all weights are set to 1 because the phragmen algorithm keeps all validators close to equally staked.
