2

We are using a machine, similar to reference: https://github.com/paritytech/substrate/pull/5848, to benchmark our runtime. To ascertain that our machine is close enough to the reference, we ran substrate (branch: polkadot-v0.9.16, commit: 19162e43be45817b44c7d48e50d03f074f60fbf4) benchmarks as is for the runtime. The resulting weights we got were about 20% lower than what is specified in substrate (in the respective weights.rs files), indicating that our machine is 20% faster than the reference.

Is this expected? Should we get a slower machine? Downclock this one? Any help is greatly appreciated. Thank you.

2
  • 1
    cargo build --profile production turns on fat LTO and that's what we've been switching to using for all the benchmarking weight runs. That might explain some of the difference you're seeing. Bear in mind it's not an exact science: one would not expect to get it spot on - you can see from some of the statemint weight PRs that they move about a little from one release to the next.
    – Squirrel
    Mar 30, 2022 at 16:04
  • We updated the weights recently and compiler flags. I think that was after v0.9.16. Could please try again on master? Apr 15, 2022 at 17:26

1 Answer 1

1

Within 20% sounds fine. It's all about getting the order of magnitude right ( the big O complexity notation ). We have built in some machine benchmarks into substrate so that we can tell if our machine spec is roughly in line with the parity benchmark machines that the weights are generated on.

Now that this PR has landed you can run this:

cargo run --profile production benchmark machine --dev

These are the results of master at 1e0807fb477 on one of our benchmarking machines:

+----------+----------------+-------+------+
| Category | Function       | Score | Unit |
+----------+----------------+-------+------+
| CPU      | BLAKE2-256     | 1029  | MB/s |
+----------+----------------+-------+------+
| CPU      | SR25519 Verify | 665.3 | KB/s |
+----------+----------------+-------+------+
| Memory   | Copy           | 14697 | MB/s |
+----------+----------------+-------+------+
| Disk     | Seq Write      | 472   | MB/s |
+----------+----------------+-------+------+
| Disk     | Rnd Write      | 212   | MB/s |
+----------+----------------+-------+------+
1
  • Incidentally these stats don't care if it's a --release build or a --profile production build (the latter takes considerably longer to compile but should be used for running benchmarks)
    – Squirrel
    Apr 15, 2022 at 14:00

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.