Performance tuning of Substrate node

Question

I was recently wondering what kind of performance tuning options can be specified while building a Substrate node. Motivation for doing this is based on the benchmark I ran on machines that utilize AMD Ryzen CPUs - performance of benchmarked CPU functions were 10-15% better when running a binary compiled specifically for Ryzen CPU (with -C target-cpu=znver3 flag).

This is the command I ran

polkadot benchmark machine --chain polkadot --verify-duration 60

So I started to wonder what other compiler options can be applied in order to tune node performance to max and this is where a lot of Substrate developers hangs out so it seems like a good place to ask. Basically I have two questions regarding this topic

1. How exactly node performance affects collecting points for activities in the active set (say on Kusama or Polkadot)? Especially when being a parachain validator
2. What can be done to get the maximum out of your node performance aside from having dedicated hardware* and good network latency?

*dedicated hardware obviously means bare metal machine with CPU having a good single-thread performance, NVMe drives and so on

Answer 1

I will try to respond with my findings and hopefully others will join with their tips & tricks as well

1. Compile for concrete CPU - you can find out type of CPU of your target machine via rustc --print target-cpus --target=x86_64-unknown-linux-gnu (assuming x86_64 standard CPU architecture) and then run compilation with RUSTFLAGS="-C target-cpu=skylake" cargo build --release (for Intel Skylake series of CPUs). This results in a significant peformance gain on AMD processors, not so much on Intel.
2. codegen-units = 1 - this tells Rust compiler to compile everything as a single crate (default is to compile crates in parallel to speed up the process). This allows for more optimizations to be done by compiler
3. lto = fat - this flag optimizes inter-binary linking process across all crates while increasing the compile time

Also it's worth noting that makes sense to build your node on the latest nightly releases of Rust to include all potential optimizations that can come with a new rustc version.