# How to benchmark functions with multiple logical paths? [duplicate]

The purpose of a benchmarking functions is to find the resources consumed by a functions in the worst case scenario. Is there any tools/methodlogy available so that , we can find the most resource intensive path of a function?

– Nuke
Nov 28, 2022 at 19:39
• Question updated. Being a member of SQA team, I need to know, whether the benchmarking functions written by a developer is a good benchamarking function. From GOOD here I mean, a benchmarking functions that inovoke the most resource instensive path Nov 28, 2022 at 21:56
• have you reviewed docs.substrate.io/test/benchmark ?
– Nuke
Nov 28, 2022 at 23:00
• Yes I reviewed it. The benchmarking test basically is sort of unit testing. I am talking about SQA test. Let me give you an example. Suppose, there is a function F, it has two paths. Path-A and path-B. Path-A is more resource intensive as compared to Path-B. Now the developer write a benchmarking function targeting path-B (instaed of path-A). I need a way/tool that tells SQA team that the developer has not targeted the most resource intensive path. Nov 29, 2022 at 10:16
• Perhaps, and I am not sure here, you want a fuzzer to test if the benchmarks setting the worst case scenario are surpassed.
– Nuke
Nov 30, 2022 at 1:52

Suppose, there is a function F, it has two paths. Path-A and path-B. Path-A is more resource intensive as compared to Path-B. Now the developer write a benchmarking function targeting path-B (instaed of path-A). I need a way/tool that tells SQA team that the developer has not targeted the most resource intensive path.

It is not possible for a computer to know which path is more complex as the complexity may also depend on the inputs to the function and any number of other factors.

If you have two paths, where it is not possible to know which one is more complex, you must benchmark both, and then construct a simple final weight formula which is the max of the two.

Let's take an abstract example, imagine there is Path-A and Path-B as you described. Let's imagine there is one input, and the final benchmarking formulas for these two paths are:

``````let path_a = |x: u32| -> Weight { 100 + x };
let path_b = |x: u32| -> Weight { 10 + 5x };
``````

As you can see, you cannot say which one of these paths is more expensive, since when `x == 1`, Path-A is worse. But with `x == 500`, Path-B is worse.

So as mentioned, you need both formulas, so you need benchmarks for both cases, and your final weight formula should be:

``````let final_weight = path_a(x).max(path_b(x));
``````

You will see concrete examples of this all throughout Substrate. For example:

https://github.com/paritytech/substrate/blob/master/frame/multisig/src/lib.rs#L373

``````#[pallet::weight({
let s = other_signatories.len() as u32;
let z = call.using_encoded(|d| d.len()) as u32;
T::WeightInfo::as_multi_create(s, z)
.max(T::WeightInfo::as_multi_approve(s, z))
.max(T::WeightInfo::as_multi_complete(s, z))
})]
pub fn as_multi( ... )
``````

Where there are actually 3 benchmarks needed for just this one function:

1. `as_multi_create`
2. `as_multi_approve`
3. `as_multi_complete`

Suppose, we have a novice blockchain developer, who write a functions against the best case scenario (instead of worst case scenario). What will happen? The benchmarking functions will obviously not produced a good approximation of weights. Is there any methodology/ framework/ anything that may help SQA team the developer has written his/her benchmark against the worst case scenario and the benchmark functions must be updated to target the worst case scenario.

There is no purely automated way to catch or test for such an error. This would be equivalent to having some bug in your logic.

You would be able to notice this problem if you specifically did real world stress tests on your chain in a scenario which is worse than your benchmark, but then you would need to know what that scenario is.

If you have benchmarks which are under-weighing your functions, you risk users underpaying for transactions, for your blocks to take longer than normal to execute, and at worst, not being able to import these blocks into a relay chain like Polkadot.

As a best practice, you should try to keep your pallet logic linear, simple, and easy to understand so that problems like this do not occur.

• Thanks @Shawn Tabrizi for your detailed explanation, This really resolve my confusion. Dec 6, 2022 at 10:17