Skip to content

Instantly share code, notes, and snippets.

@ahx
Last active June 24, 2024 09:18
Show Gist options
  • Save ahx/e6ffced58bd2e8d5baffb2f4d2c1f823 to your computer and use it in GitHub Desktop.
Save ahx/e6ffced58bd2e8d5baffb2f4d2c1f823 to your computer and use it in GitHub Desktop.
openapi_first benchmark comparison

Check out openapi_first: https://github.com/ahx/openapi_first

These are the results running these benchmarks (code).

openapi_first 2.0.0

ruby 3.3.0 (2023-12-25 revision 5124f9ac75) [arm64-darwin23]
Warming up --------------------------------------
 ./apps/committee.ru     1.000 i/100ms
./apps/committee_with_response_validation.ru
                         1.000 i/100ms
./apps/openapi_first.ru
                         2.000 i/100ms
./apps/openapi_first_with_response_validation.ru
                         1.000 i/100ms
Calculating -------------------------------------
 ./apps/committee.ru     14.010 (± 7.1%) i/s -     70.000 in   5.040169s
./apps/committee_with_response_validation.ru
                         11.906 (± 0.0%) i/s -     60.000 in   5.041775s
./apps/openapi_first.ru
                         21.491 (± 4.7%) i/s -    108.000 in   5.031617s
./apps/openapi_first_with_response_validation.ru
                         15.950 (± 6.3%) i/s -     80.000 in   5.021642s

Comparison:
./apps/openapi_first.ru:       21.5 i/s
./apps/openapi_first_with_response_validation.ru:       16.0 i/s - 1.35x  slower
 ./apps/committee.ru:       14.0 i/s - 1.53x  slower
./apps/committee_with_response_validation.ru:       11.9 i/s - 1.80x  slower

Calculating -------------------------------------
 ./apps/committee.ru    17.721M memsize (     8.201k retained)
                       275.428k objects (    56.000  retained)
                        50.000  strings (     6.000  retained)
./apps/committee_with_response_validation.ru
                        27.192M memsize (     8.201k retained)
                       449.249k objects (    56.000  retained)
                        50.000  strings (     6.000  retained)
./apps/openapi_first.ru
                         9.268M memsize (     3.760k retained)
                       116.859k objects (    37.000  retained)
                        48.000  strings (     6.000  retained)
./apps/openapi_first_with_response_validation.ru
                        13.524M memsize (     3.760k retained)
                       164.632k objects (    37.000  retained)
                        50.000  strings (     6.000  retained)

Comparison:
./apps/openapi_first.ru:    9268244 allocated
./apps/openapi_first_with_response_validation.ru:   13523980 allocated - 1.46x more
 ./apps/committee.ru:   17720793 allocated - 1.91x more
./apps/committee_with_response_validation.ru:   27191859 allocated - 2.93x more

openapi_first 1.4.3

This release extended example app with more routes for more realistic benchmarks.

ruby 3.3.0 (2023-12-25 revision 5124f9ac75) [arm64-darwin23]
Warming up --------------------------------------
 ./apps/committee.ru     1.000 i/100ms
./apps/committee_with_response_validation.ru
                         1.000 i/100ms
./apps/openapi_first.ru
                         1.000 i/100ms
./apps/openapi_first_with_response_validation.ru
                         1.000 i/100ms
Calculating -------------------------------------
 ./apps/committee.ru     15.623 (± 0.0%) i/s -     79.000 in   5.059764s
./apps/committee_with_response_validation.ru
                         11.877 (± 0.0%) i/s -     60.000 in   5.055495s
./apps/openapi_first.ru
                         15.890 (± 0.0%) i/s -     80.000 in   5.038707s
./apps/openapi_first_with_response_validation.ru
                         10.748 (± 0.0%) i/s -     54.000 in   5.025022s

Comparison:
./apps/openapi_first.ru:       15.9 i/s
 ./apps/committee.ru:       15.6 i/s - 1.02x  slower
./apps/committee_with_response_validation.ru:       11.9 i/s - 1.34x  slower
./apps/openapi_first_with_response_validation.ru:       10.7 i/s - 1.48x  slower

Calculating -------------------------------------
 ./apps/committee.ru    17.721M memsize (     8.201k retained)
                       275.428k objects (    56.000  retained)
                        50.000  strings (     6.000  retained)
./apps/committee_with_response_validation.ru
                        27.192M memsize (     8.201k retained)
                       449.249k objects (    56.000  retained)
                        50.000  strings (     6.000  retained)
./apps/openapi_first.ru
                        13.996M memsize (   200.000  retained)
                       166.955k objects (     5.000  retained)
                        50.000  strings (     3.000  retained)
./apps/openapi_first_with_response_validation.ru
                        23.394M memsize (   200.000  retained)
                       263.107k objects (     5.000  retained)
                        50.000  strings (     3.000  retained)

Comparison:
./apps/openapi_first.ru:   13996054 allocated
 ./apps/committee.ru:   17720793 allocated - 1.27x more
./apps/openapi_first_with_response_validation.ru:   23394003 allocated - 1.67x more
./apps/committee_with_response_validation.ru:   27191859 allocated - 1.94x more

openapi_first 1.3.2

ruby 3.3.0 (2023-12-25 revision 5124f9ac75) [arm64-darwin22]
Warming up --------------------------------------
 ./apps/committee.ru     1.000 i/100ms
./apps/committee_with_response_validation.ru
                         1.000 i/100ms
./apps/no-validation.ru
                         2.000 i/100ms
./apps/openapi_first.ru
                         2.000 i/100ms
./apps/openapi_first_with_response_validation.ru
                         1.000 i/100ms
Calculating -------------------------------------
 ./apps/committee.ru     19.097 (± 0.0%) i/s -     96.000 in   5.027948s
./apps/committee_with_response_validation.ru
                         15.910 (± 0.0%) i/s -     80.000 in   5.030133s
./apps/no-validation.ru
                         27.223 (± 0.0%) i/s -    138.000 in   5.069630s
./apps/openapi_first.ru
                         24.172 (± 4.1%) i/s -    122.000 in   5.050524s
./apps/openapi_first_with_response_validation.ru
                         14.822 (± 0.0%) i/s -     75.000 in   5.064667s

Comparison:
./apps/no-validation.ru:       27.2 i/s
./apps/openapi_first.ru:       24.2 i/s - 1.13x  slower
 ./apps/committee.ru:       19.1 i/s - 1.43x  slower
./apps/committee_with_response_validation.ru:       15.9 i/s - 1.71x  slower
./apps/openapi_first_with_response_validation.ru:       14.8 i/s - 1.84x  slower

Calculating -------------------------------------
 ./apps/committee.ru    11.773M memsize (     8.201k retained)
                       158.671k objects (    56.000  retained)
                        50.000  strings (     6.000  retained)
./apps/committee_with_response_validation.ru
                        15.296M memsize (     8.201k retained)
                       215.736k objects (    56.000  retained)
                        50.000  strings (     6.000  retained)
./apps/no-validation.ru
                         6.652M memsize (     1.816k retained)
                        90.294k objects (     7.000  retained)
                        50.000  strings (     3.000  retained)
./apps/openapi_first.ru
                         7.823M memsize (   240.000  retained)
                       101.808k objects (     6.000  retained)
                        41.000  strings (     3.000  retained)
./apps/openapi_first_with_response_validation.ru
                        15.778M memsize (   240.000  retained)
                       182.507k objects (     6.000  retained)
                        47.000  strings (     3.000  retained)

Comparison:
./apps/no-validation.ru:    6652264 allocated
./apps/openapi_first.ru:    7823460 allocated - 1.18x more
 ./apps/committee.ru:   11773065 allocated - 1.77x more
./apps/committee_with_response_validation.ru:   15296443 allocated - 2.30x more
./apps/openapi_first_with_response_validation.ru:   15778220 allocated - 2.37x more

openapi_first 1.1.0

ruby 3.3.0 (2023-12-25 revision 5124f9ac75) [arm64-darwin22]
Warming up --------------------------------------
 ./apps/committee.ru     1.000 i/100ms
./apps/committee_with_response_validation.ru
                         1.000 i/100ms
./apps/openapi_first.ru
                         1.000 i/100ms
./apps/openapi_first_with_response_validation.ru
                         1.000 i/100ms
Calculating -------------------------------------
 ./apps/committee.ru     10.064 (± 0.0%) i/s -     51.000 in   5.070207s
./apps/committee_with_response_validation.ru
                          8.642 (± 0.0%) i/s -     44.000 in   5.092091s
./apps/openapi_first.ru
                         12.869 (± 0.0%) i/s -     65.000 in   5.058367s
./apps/openapi_first_with_response_validation.ru
                          8.274 (± 0.0%) i/s -     42.000 in   5.086650s

Comparison:
./apps/openapi_first.ru:       12.9 i/s
 ./apps/committee.ru:       10.1 i/s - 1.28x  slower
./apps/committee_with_response_validation.ru:        8.6 i/s - 1.49x  slower
./apps/openapi_first_with_response_validation.ru:        8.3 i/s - 1.56x  slower

Calculating -------------------------------------
 ./apps/committee.ru    12.201M memsize (     8.201k retained)
                       162.408k objects (    56.000  retained)
                        50.000  strings (     6.000  retained)
./apps/committee_with_response_validation.ru
                        15.655M memsize (     8.201k retained)
                       217.756k objects (    56.000  retained)
                        50.000  strings (     6.000  retained)
./apps/openapi_first.ru
                         8.223M memsize (   200.000  retained)
                       105.343k objects (     5.000  retained)
                        41.000  strings (     3.000  retained)
./apps/openapi_first_with_response_validation.ru
                        15.959M memsize (   200.000  retained)
                       182.507k objects (     5.000  retained)
                        47.000  strings (     3.000  retained)

Comparison:
./apps/openapi_first.ru:    8222814 allocated
 ./apps/committee.ru:   12200699 allocated - 1.48x more
./apps/committee_with_response_validation.ru:   15655397 allocated - 1.90x more
./apps/openapi_first_with_response_validation.ru:   15959414 allocated - 1.94x more
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment