How to benchmark functions with multiple logical paths? [duplicate]

Question

The purpose of a benchmarking functions is to find the resources consumed by a functions in the worst case scenario. Is there any tools/methodlogy available so that , we can find the most resource intensive path of a function?

Answer 1

Suppose, there is a function F, it has two paths. Path-A and path-B. Path-A is more resource intensive as compared to Path-B. Now the developer write a benchmarking function targeting path-B (instaed of path-A). I need a way/tool that tells SQA team that the developer has not targeted the most resource intensive path.

It is not possible for a computer to know which path is more complex as the complexity may also depend on the inputs to the function and any number of other factors.

If you have two paths, where it is not possible to know which one is more complex, you must benchmark both, and then construct a simple final weight formula which is the max of the two.

Let's take an abstract example, imagine there is Path-A and Path-B as you described. Let's imagine there is one input, and the final benchmarking formulas for these two paths are:

let path_a = |x: u32| -> Weight { 100 + x };
let path_b = |x: u32| -> Weight { 10 + 5x };

As you can see, you cannot say which one of these paths is more expensive, since when x == 1, Path-A is worse. But with x == 500, Path-B is worse.

So as mentioned, you need both formulas, so you need benchmarks for both cases, and your final weight formula should be:

let final_weight = path_a(x).max(path_b(x));

You will see concrete examples of this all throughout Substrate. For example:

https://github.com/paritytech/substrate/blob/master/frame/multisig/src/lib.rs#L373

#[pallet::weight({
    let s = other_signatories.len() as u32;
    let z = call.using_encoded(|d| d.len()) as u32;
    T::WeightInfo::as_multi_create(s, z)
        .max(T::WeightInfo::as_multi_approve(s, z))
        .max(T::WeightInfo::as_multi_complete(s, z))
        .saturating_add(*max_weight)
})]
pub fn as_multi( ... )

Where there are actually 3 benchmarks needed for just this one function:

1. as_multi_create
2. as_multi_approve
3. as_multi_complete

As found here: https://github.com/paritytech/substrate/blob/master/frame/multisig/src/benchmarking.rs

Suppose, we have a novice blockchain developer, who write a functions against the best case scenario (instead of worst case scenario). What will happen? The benchmarking functions will obviously not produced a good approximation of weights. Is there any methodology/ framework/ anything that may help SQA team the developer has written his/her benchmark against the worst case scenario and the benchmark functions must be updated to target the worst case scenario.

There is no purely automated way to catch or test for such an error. This would be equivalent to having some bug in your logic.

You would be able to notice this problem if you specifically did real world stress tests on your chain in a scenario which is worse than your benchmark, but then you would need to know what that scenario is.

If you have benchmarks which are under-weighing your functions, you risk users underpaying for transactions, for your blocks to take longer than normal to execute, and at worst, not being able to import these blocks into a relay chain like Polkadot.

As a best practice, you should try to keep your pallet logic linear, simple, and easy to understand so that problems like this do not occur.