3

I'm running benchmark-cli storage tool to calculate average weights for DB read and writes and one thing that I've noticed is that there are significant difference between even 99th percentile and worst case scenarios (max). Here is an example ran on RocksDb.

    /// Stats nanoseconds:
    ///   Min, Max: 54_112, 124_655_974
    ///   Average:  133_227
    ///   Median:   131_560
    ///   Std-Dev:  213029.92
    ///
    /// Percentiles nanoseconds:
    ///   99th: 165_267
    ///   95th: 150_276
    ///   75th: 137_649
    write: 133_227 * constants::WEIGHT_PER_NANOS,

You can see that the max which is 124_655_974 is more than 700 times worst than 99th which is 165_267.

Does anyone know why this might happen and is that related to some flaw in benchmarking or some cache related side-effect?

Notes:

  • I've printed out these worst cases and they are different keys on every run but the worst case seems to still be around the same number.
2
  • 1
    Are you measuring RocksDB or ParityDB? In general when accessing data from a disk, you have a lot of different layers or properties which can affect the speed, from hardware all the way to software. This is obviously why we take an average value, and try to eliminate outliers. I am not sure what kind of answer you are expecting to get here.
    – Shawn Tabrizi
    Aug 20, 2022 at 1:53
  • Yes, this is measured using RocksDB. I was curious if you might have more insight into these numbers and if these numbers are real or some side effect of the way benchmarks are done.
    – Aramik M
    Aug 22, 2022 at 19:41

2 Answers 2

2

Rocks DB is a very complex database, which did have strange outlier behaviors we never fully understood.

I actually spoke about that to some extent here: https://youtu.be/Qa6sTyUqgek?t=4573

enter image description here

There were certain benchmarks where Rocks DB totally went crazy and gave huge outliers, and because of that, we had to remove Rocks DB from the main benchmarking pipeline (using an in-memory db), and instead isolate the database benchmarking to a separate process.

So unfortunately I do not have an answer "why exactly" these outliers exist, but it is something that is known about from our experience with the DB, and it is something reasonable to expect when using it. As Oliver mentions, your benchmark results show that the average time calculated is quite accurate, and you should not really get yourself hung up on the outliers. This is why we take averages and median values.

1
  • Thanks for explanation.
    – Aramik M
    Aug 24, 2022 at 21:34
2

Assuming that the math is correct; this shows you that there are less than one percent of very large outliers.
Polkadot shows similar behaviour with 310x worst case compared to the 99th percentile for RocksDB and 682x for ParityDB.
Are you using a NVMe SSD? With HDDs you will get worse results.
Your std deviation is also quite large, as comparison RocksDB Polkadot:

/// Time to write one storage item.
/// Calculated by multiplying the *Average* of all values with `1.1` and adding `0`.
///
/// Stats [NS]:
///   Min, Max: 16_368, 34_500_937
///   Average:  75_882
///   Median:   74_236
///   Std-Dev:  64706.41
///
/// Percentiles [NS]:
///   99th: 111_151
///   95th: 92_666
///   75th: 80_297
write: 83_471 * constants::WEIGHT_PER_NANOS,
1
  • I think this is done on HDD. Another thing that I've noticed is that the more keys are inside the storage the more std deviation grows.
    – Aramik M
    Aug 22, 2022 at 19:42

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.