2

I have written the benchmarking code for the pallet,

Used the below CLI command to run benchmarking -

./target/release/node-template benchmark pallet --pallet pallet_usd_rate --extrinsic "*" --steps=50 --repeat=20 --execution=wasm --wasm-execution=compiled --heap-pages=4098 --output=pallets/usd-rate/src/weights1.rs

But during execution I have seen one surprising thing that on each execution of benchmarking, the calculated weight are different. It is varying between 10_245_000 to 28_000_000.

I have tried many ways via changing the parameters, but the behaviour is same. Even I have tried to run the benchmarking of Pallet_timestamp.. But for pallet_timestamp also it is varying, it is not fixed.

I assume it must be fixed value on a particular machine....because there is no change in code.. It should not vary on each execution..

Can anyone explain me the reason for this behaviour.

3

2 Answers 2

1

I analyzed your benchmark and it looks good, quite consistent (minus one outlier) [µs]:

15.76
15.69
15.37
15.55
15.31
31.38
16.07
15.81
15.67
15.43
15.64
15.99
15.44

The benchmarks need to be run on the same hardware every time. This is since they measure time, and a faster PC scores lower (better) weights.
If you are just doing development you can obviously use any hardware. But before a release it is necessary to re-benchmark the weights on fixed hardware.
A good pick would be the Polkadot reference hardware.

1
  • Thank you for you confirmation @OliverTale-Yazdi I'll re-analyze the benchmarking before going for release on a vm.
    – Asmita
    Jul 28, 2022 at 7:58
2

As far as I understand, benchmarks have to be run in a stable environment. I've always been getting big variance when running any benchmarks on my PC, because there's a thousand other processes in the background, sharing RAM & CPU with the benchmarks. I think a dedicated (virtual) machine might be a good solution for production benchmarking.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.