Gotchas when looking at percentage improvements in a profiler

percentage-improvement.md

Gotchas with percentage improvements in a profiler

(Something similar to this post was originally written by Boris Zbarsky on his blog, but I can't find the post now.)

If you have a function which takes up 50% of our time, and you optimize that function so that it takes 30% of your time, by what percentage did you reduce overall time and by what percentage did you reduce time in that function?

The tricky bit here is that the 100% reference value is different, because it was also reduced.

    |<------------------------------- 100ms ------------------------------->|

    +----------------------------------+ +----------------------------------+
    |         your function            | |            the rest              |
    +----------------------------------+ +----------------------------------+
                   50%                                   50%
                   50ms                 |                50ms
                                        v

                       +---------------+ +----------------------------------+
                       | your function | |            the rest              |
                       +---------------+ +----------------------------------+
                              30%                        70%
                              ??ms                 still 50ms

Let's say the full workload takes 100ms in the beginning, of which your function takes 50ms.

How many milliseconds does your function take once you've reduced it to take 30% of the overall workload?

30ms is not the right answer: If your function now takes 30ms, this would mean that the full workload now takes 30ms + 50ms = 80ms, but 30ms is 37.5% of 80ms.

The right answer is 21.43ms. The rest of the workload still takes 50ms, so the new total is 21.43ms + 50ms = 71.43ms, and 21.43ms is 30% of 71.43ms.

So really, you reduced the time in your function from 50% of the "before total" to 21.43% of the "before total" (which equals 30% of the "after total").

And, looking at just your function, you reduced it to 42.86% of its previous time. You might have thought that you reduced it to 60% of its previous time (because 30% / 50% = 60%), but in reality you did better than that.

21.43% = 100% * compute_fraction_part_after_of_full_before(0.5, 0.3) = 100% * (1 - 0.5) / (1 - 0.3) * 0.3

42.86% = 100% * compute_part_relative_fraction(0.5, 0.3) = 100% * (1 - 0.5) / (1 - 0.3) * 0.3 / 0.5

The math

fraction_part_before = time_part_before / time_full_before
fraction_part_after = time_part_after / time_full_after
part_relative_fraction = time_part_after / time_part_before

reduction_time = time_part_before - time_part_after = time_full_before - time_full_after

fraction_part_before * time_full_before - fraction_part_after * time_full_after = time_full_before - time_full_after
(1 - fraction_part_after) * time_full_after = (1 - fraction_part_before) * time_full_before
time_full_after = (1 - fraction_part_before) / (1 - fraction_part_after) * time_full_before
fraction_part_after = time_part_after / time_full_after
time_part_after = time_full_after * fraction_part_after
time_part_after = (1 - fraction_part_before) / (1 - fraction_part_after) * fraction_part_after * time_full_before
part_relative_fraction = time_part_after / time_part_before
part_relative_fraction = (1 - fraction_part_before) / (1 - fraction_part_after) * fraction_part_after * time_full_before / (time_full_before * fraction_part_before)
part_relative_fraction = (1 - fraction_part_before) / (1 - fraction_part_after) * fraction_part_after / fraction_part_before

function compute_time_part_after(time_full_before, fraction_part_before, fraction_part_after) {
  return (1 - fraction_part_before) / (1 - fraction_part_after) * fraction_part_after * time_full_before
}

function compute_fraction_part_after_of_full_before(fraction_part_before, fraction_part_after) {
  return (1 - fraction_part_before) / (1 - fraction_part_after) * fraction_part_after
}

function compute_part_relative_fraction(fraction_part_before, fraction_part_after) {
  return (1 - fraction_part_before) / (1 - fraction_part_after) * fraction_part_after / fraction_part_before
}