Skip to content

Instantly share code, notes, and snippets.

@PJUllrich

PJUllrich/timing-attack-benchmark.livemd

Last active April 2, 2023 11:00
Show Gist options
  • Save PJUllrich/6544a8d2068329eb95a54e77bcdb1401 to your computer and use it in GitHub Desktop.
Save PJUllrich/6544a8d2068329eb95a54e77bcdb1401 to your computer and use it in GitHub Desktop.
Download ZIP
Raw
timing-attack-benchmark.livemd

The Code

Mix.install([{:benchee, "~> 1.0", only: :dev}, {:plug, "~> 1.14"}])

iterations = 10

# Compares the strings with "=="
# Returns 'nil' to minimize time spent on allocating and moving around memory
variable_compare = fn left, right -> 
  for _ <- 1..iterations, do: left == right
  nil
end

# Compares the strings using Plug.Crypto.secure_compare/2
# Also returns 'nil' to avoid memory overhead
secure_compare = fn left, right -> 
  for _ <- 1..iterations, do: Plug.Crypto.secure_compare(left, right)
  nil
end

Benchee.run(
  %{
    "Variable: 1st Byte" => fn ->  variable_compare.("B000000000000000000000000000000000000000", "F7A53E760828107A49D13A6D8C4B046674D5A902") end,
    "Variable: 2nd Byte" => fn ->  variable_compare.("FB00000000000000000000000000000000000000", "F7A53E760828107A49D13A6D8C4B046674D5A902") end,
    "Variable: 5th Byte" => fn ->  variable_compare.("F7A5B00000000000000000000000000000000000", "F7A53E760828107A49D13A6D8C4B046674D5A902") end,
    "Variable: 10th Byte" => fn -> variable_compare.("F7A53E760B000000000000000000000000000000", "F7A53E760828107A49D13A6D8C4B046674D5A902") end,
    "Variable: 20th Byte" => fn -> variable_compare.("F7A53E760828107A49DB00000000000000000000", "F7A53E760828107A49D13A6D8C4B046674D5A902") end,
    "Variable: 30th Byte" => fn -> variable_compare.("F7A53E760828107A49D13A6D8C4B0B0000000000", "F7A53E760828107A49D13A6D8C4B046674D5A902") end,
    "Variable: 40th Byte" => fn -> variable_compare.("F7A53E760828107A49D13A6D8C4B046674D5A90B", "F7A53E760828107A49D13A6D8C4B046674D5A902") end,
    "Constant: 1st Byte" => fn ->  secure_compare.("B000000000000000000000000000000000000000", "F7A53E760828107A49D13A6D8C4B046674D5A902") end,
    "Constant: 2nd Byte" => fn ->  secure_compare.("FB00000000000000000000000000000000000000", "F7A53E760828107A49D13A6D8C4B046674D5A902") end,
    "Constant: 5th Byte" => fn ->  secure_compare.("F7A5B00000000000000000000000000000000000", "F7A53E760828107A49D13A6D8C4B046674D5A902") end,
    "Constant: 10th Byte" => fn -> secure_compare.("F7A53E760B000000000000000000000000000000", "F7A53E760828107A49D13A6D8C4B046674D5A902") end,
    "Constant: 20th Byte" => fn -> secure_compare.("F7A53E760828107A49DB00000000000000000000", "F7A53E760828107A49D13A6D8C4B046674D5A902") end,
    "Constant: 30th Byte" => fn -> secure_compare.("F7A53E760828107A49D13A6D8C4B0B0000000000", "F7A53E760828107A49D13A6D8C4B046674D5A902") end,
    "Constant: 40th Byte" => fn -> secure_compare.("F7A53E760828107A49D13A6D8C4B046674D5A90B", "F7A53E760828107A49D13A6D8C4B046674D5A902") end
  },
  time: 15
)

The Results

I ran the above benchmark in two versions: First, every run compared the strings only once. That lead to very low iteration times of 2-5 microseconds. I was concerned about the measurement inaccuracy and added for-loops that run the comparison 10 times in every run. That's the code you see above.

Without for-loops

Name                          ips        average  deviation         median         99th %
Variable: 1st Byte       336.31 K        2.97 μs  ±1508.56%        2.06 μs        4.10 μs
Variable: 2nd Byte       305.19 K        3.28 μs  ±1474.47%        2.47 μs        4.91 μs
Variable: 5th Byte       307.29 K        3.25 μs  ±1479.03%        2.43 μs        4.95 μs
Variable: 10th Byte      318.17 K        3.14 μs  ±1523.79%        2.26 μs        4.41 μs
Variable: 20th Byte      331.79 K        3.01 μs  ±1488.85%        2.13 μs        4.25 μs
Variable: 30th Byte      306.18 K        3.27 μs  ±1480.08%        2.44 μs        4.97 μs
Variable: 40th Byte      307.73 K        3.25 μs  ±1473.67%        2.42 μs        4.84 μs

Constant: 1st Byte       192.97 K        5.18 μs   ±811.50%        4.13 μs       10.11 μs
Constant: 2nd Byte       197.36 K        5.07 μs   ±805.35%        4.06 μs       10.05 μs
Constant: 5th Byte       218.06 K        4.59 μs   ±798.72%        3.57 μs        7.41 μs
Constant: 10th Byte      196.04 K        5.10 μs   ±812.41%        4.06 μs       10.14 μs
Constant: 20th Byte      196.80 K        5.08 μs   ±808.10%        4.08 μs       10.10 μs
Constant: 30th Byte      196.46 K        5.09 μs   ±807.11%        4.07 μs       10.17 μs
Constant: 40th Byte      200.92 K        4.98 μs   ±775.65%        3.98 μs       10.01 μs

Comparison: 
Variable: 1st Byte       336.31 K
Variable: 20th Byte      331.79 K - 1.01x slower +0.0406 μs
Variable: 10th Byte      318.17 K - 1.06x slower +0.170 μs
Variable: 40th Byte      307.73 K - 1.09x slower +0.28 μs
Variable: 5th Byte       307.29 K - 1.09x slower +0.28 μs
Variable: 30th Byte      306.18 K - 1.10x slower +0.29 μs
Variable: 2nd Byte       305.19 K - 1.10x slower +0.30 μs

Constant: 5th Byte       218.06 K - 1.54x slower +1.61 μs
Constant: 40th Byte      200.92 K - 1.67x slower +2.00 μs
Constant: 2nd Byte       197.36 K - 1.70x slower +2.09 μs
Constant: 20th Byte      196.80 K - 1.71x slower +2.11 μs
Constant: 30th Byte      196.46 K - 1.71x slower +2.12 μs
Constant: 10th Byte      196.04 K - 1.72x slower +2.13 μs
Constant: 1st Byte       192.97 K - 1.74x slower +2.21 μs

With for-loops and 10 iterations per run

Name                          ips        average  deviation         median         99th %
Variable: 1st Byte        89.02 K       11.23 μs   ±201.22%        9.58 μs       36.17 μs
Variable: 2nd Byte        92.66 K       10.79 μs   ±194.45%        9.08 μs       35.60 μs
Variable: 5th Byte        90.73 K       11.02 μs   ±192.08%        9.20 μs       35.30 μs
Variable: 10th Byte       87.05 K       11.49 μs   ±194.71%        9.57 μs       39.80 μs
Variable: 20th Byte       90.96 K       10.99 μs   ±190.70%        9.19 μs       34.37 μs
Variable: 30th Byte       90.38 K       11.06 μs   ±192.00%        9.27 μs       35.95 μs
Variable: 40th Byte       87.84 K       11.38 μs   ±185.92%        9.34 μs       40.08 μs

Constant: 1st Byte        41.47 K       24.11 μs    ±26.11%       22.76 μs       45.22 μs
Constant: 2nd Byte        35.90 K       27.85 μs    ±48.86%       25.27 μs       74.90 μs
Constant: 5th Byte        40.41 K       24.75 μs    ±30.57%       22.86 μs       53.82 μs
Constant: 10th Byte       40.01 K       25.00 μs    ±31.96%       22.82 μs       48.49 μs
Constant: 20th Byte       41.67 K       24.00 μs    ±24.34%       22.82 μs       44.77 μs
Constant: 30th Byte       39.53 K       25.29 μs    ±37.48%       23.09 μs       56.92 μs
Constant: 40th Byte       39.79 K       25.13 μs   ±141.13%       23.15 μs       55.48 μs

Comparison: 
Variable: 2nd Byte        92.66 K
Variable: 20th Byte       90.96 K - 1.02x slower +0.20 μs
Variable: 5th Byte        90.73 K - 1.02x slower +0.23 μs
Variable: 30th Byte       90.38 K - 1.03x slower +0.27 μs
Variable: 1st Byte        89.02 K - 1.04x slower +0.44 μs
Variable: 40th Byte       87.84 K - 1.05x slower +0.59 μs
Variable: 10th Byte       87.05 K - 1.06x slower +0.70 μs

Constant: 20th Byte       41.67 K - 2.22x slower +13.20 μs
Constant: 1st Byte        41.47 K - 2.23x slower +13.32 μs
Constant: 5th Byte        40.41 K - 2.29x slower +13.95 μs
Constant: 10th Byte       40.01 K - 2.32x slower +14.20 μs
Constant: 40th Byte       39.79 K - 2.33x slower +14.34 μs
Constant: 30th Byte       39.53 K - 2.34x slower +14.50 μs
Constant: 2nd Byte        35.90 K - 2.58x slower +17.06 μs

Hardware Specs

Operating System: macOS
CPU Information: Intel(R) Core(TM) i7-8750H CPU @ 2.20GHz
Number of Available Cores: 12
Available memory: 32 GB
Elixir 1.14.2
Erlang 25.1.1

Benchmark suite executing with the following configuration:
warmup: 2 s
time: 15 s
memory time: 0 ns
reduction time: 0 ns
parallel: 1
inputs: none specified
Estimated total run time: 3.97 min
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment