2019-02-11 – Janus/KrakenD Benchmark

janus-api.json

{
    "name" : "foo",
    "active" : true,
    "proxy" : {
        "listen_path" : "/foo",
        "upstreams" : {
            "balancing": "roundrobin",
            "targets": [
                {"target": "http://localhost:8080/bar"}
            ]
        },
        "methods" : ["GET"]
    }
}

janus-with-prometheus.toml

################################################################
# Global configuration
################################################################
port = 8888

[log]
    level = "error"

################################################################
# API configuration backend
################################################################
[web]
  port = 8081

  [web.credentials]
    secret = "secret"

    [web.credentials.basic]
    users = {admin = "admin"}

[database]
  dsn = "file:///tmp/"

[stats]
  exporter = "prometheus"

janus-with-statsd.toml

################################################################
# Global configuration
################################################################
port = 8888

[log]
    level = "error"

################################################################
# API configuration backend
################################################################
[web]
  port = 8081

  [web.credentials]
    secret = "secret"

    [web.credentials.basic]
    users = {admin = "admin"}

[database]
  dsn = "file:///tmp/"

[stats]
  dsn = "statsd://localhost:8125/janus.local/"

janus.toml

################################################################
# Global configuration
################################################################
port = 8888

[log]
    level = "error"

################################################################
# API configuration backend
################################################################
[web]
  port = 8081

  [web.credentials]
    secret = "secret"

    [web.credentials.basic]
    users = {admin = "admin"}

[database]
  dsn = "file:///tmp/"

krakend-with-prometheus.json

{
  "version": 2,
  "extra_config": {
    "github_com/devopsfaith/krakend-opencensus": {
      "sample_rate": 100,
      "reporting_period": 1,
      "exporters": {
        "prometheus": {
          "port": 9091
        }
      }
    }
  },
  "timeout": "3000ms",
  "cache_ttl": "300s",
  "output_encoding": "json",
  "name": "foo",
  "port": 8000,
  "endpoints": [
    {
      "endpoint": "/foo",
      "method": "GET",
      "extra_config": {},
      "output_encoding": "json",
      "concurrent_calls": 1,
      "backend": [
        {
          "url_pattern": "/bar",
          "encoding": "json",
          "extra_config": {},
          "sd": "static",
          "host": [
            "http://localhost:8080"
          ],
          "disable_host_sanitize": false
        }
      ]
    }
  ]
}

krakend.json

{
  "version": 2,
  "extra_config": {},
  "timeout": "3000ms",
  "cache_ttl": "300s",
  "output_encoding": "json",
  "name": "foo",
  "port": 8000,
  "endpoints": [
    {
      "endpoint": "/foo",
      "method": "GET",
      "extra_config": {},
      "output_encoding": "json",
      "concurrent_calls": 1,
      "backend": [
        {
          "url_pattern": "/bar",
          "encoding": "json",
          "extra_config": {},
          "sd": "static",
          "host": [
            "http://localhost:8080"
          ],
          "disable_host_sanitize": false
        }
      ]
    }
  ]
}

result.md

The following results are obtained by running the benchmark on:

• MacBook Pro (Retina, 13-inch, Early 2015)
• 3,1 GHz Intel Core i7

Notes

• Test API used: https://github.com/espebra/dummy-api
• Janus version: 3.8.7-rc8
• KrakenD version: 0.7.1

Summary

Setup	Requests per Second
No Gateway	21103.2602 rps
Janus	5451.2701 rps
Janus with statsd	5563.9192 rps
Janus with Prometheus	4781.4349 rps
KrakenD	7476.4710 rps
KrakenD with Prometheus	6326.9421 rps

Detailed

No Gateway

 $  hey -n 100000 http://localhost:8080/bar

Summary:
  Total:	4.7386 secs
  Slowest:	0.0447 secs
  Fastest:	0.0001 secs
  Average:	0.0023 secs
  Requests/sec:	21103.2602

  Total data:	10300000 bytes
  Size/request:	103 bytes

Response time histogram:
  0.000 [1]	|
  0.005 [93814]	|■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■
  0.009 [4314]	|■■
  0.013 [944]	|
  0.018 [382]	|
  0.022 [293]	|
  0.027 [151]	|
  0.031 [35]	|
  0.036 [21]	|
  0.040 [12]	|
  0.045 [33]	|


Latency distribution:
  10% in 0.0007 secs
  25% in 0.0012 secs
  50% in 0.0019 secs
  75% in 0.0027 secs
  90% in 0.0037 secs
  95% in 0.0051 secs
  99% in 0.0130 secs

Details (average, fastest, slowest):
  DNS+dialup:	0.0000 secs, 0.0001 secs, 0.0447 secs
  DNS-lookup:	0.0000 secs, 0.0000 secs, 0.0041 secs
  req write:	0.0000 secs, 0.0000 secs, 0.0098 secs
  resp wait:	0.0022 secs, 0.0001 secs, 0.0446 secs
  resp read:	0.0000 secs, 0.0000 secs, 0.0102 secs

Status code distribution:
  [200]	100000 responses

Janus

 $  hey -n 100000 http://localhost:8888/foo

Summary:
  Total:	18.3443 secs
  Slowest:	0.0691 secs
  Fastest:	0.0004 secs
  Average:	0.0091 secs
  Requests/sec:	5451.2701

  Total data:	10300000 bytes
  Size/request:	103 bytes

Response time histogram:
  0.000 [1]	|
  0.007 [41499]	|■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■
  0.014 [45218]	|■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■
  0.021 [9693]	|■■■■■■■■■
  0.028 [2874]	|■■■
  0.035 [517]	|
  0.042 [161]	|
  0.048 [29]	|
  0.055 [6]	|
  0.062 [1]	|
  0.069 [1]	|


Latency distribution:
  10% in 0.0035 secs
  25% in 0.0054 secs
  50% in 0.0082 secs
  75% in 0.0115 secs
  90% in 0.0155 secs
  95% in 0.0192 secs
  99% in 0.0266 secs

Details (average, fastest, slowest):
  DNS+dialup:	0.0000 secs, 0.0004 secs, 0.0691 secs
  DNS-lookup:	0.0000 secs, 0.0000 secs, 0.0077 secs
  req write:	0.0000 secs, 0.0000 secs, 0.0076 secs
  resp wait:	0.0090 secs, 0.0004 secs, 0.0690 secs
  resp read:	0.0001 secs, 0.0000 secs, 0.0185 secs

Status code distribution:
  [200]	100000 responses

Janus with statsd

 $  hey -n 100000 http://localhost:8888/foo

Summary:
  Total:	17.9729 secs
  Slowest:	0.0573 secs
  Fastest:	0.0005 secs
  Average:	0.0089 secs
  Requests/sec:	5563.9192

  Total data:	10300000 bytes
  Size/request:	103 bytes

Response time histogram:
  0.000 [1]	|
  0.006 [32463]	|■■■■■■■■■■■■■■■■■■■■■■■■■■■■■
  0.012 [45034]	|■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■
  0.018 [16282]	|■■■■■■■■■■■■■■
  0.023 [4189]	|■■■■
  0.029 [1427]	|■
  0.035 [420]	|
  0.040 [126]	|
  0.046 [40]	|
  0.052 [13]	|
  0.057 [5]	|


Latency distribution:
  10% in 0.0034 secs
  25% in 0.0053 secs
  50% in 0.0080 secs
  75% in 0.0114 secs
  90% in 0.0151 secs
  95% in 0.0188 secs
  99% in 0.0263 secs

Details (average, fastest, slowest):
  DNS+dialup:	0.0000 secs, 0.0005 secs, 0.0573 secs
  DNS-lookup:	0.0000 secs, 0.0000 secs, 0.0052 secs
  req write:	0.0000 secs, 0.0000 secs, 0.0053 secs
  resp wait:	0.0088 secs, 0.0004 secs, 0.0572 secs
  resp read:	0.0001 secs, 0.0000 secs, 0.0089 secs

Status code distribution:
  [200]	100000 responses

Janus with Prometheus

 $  hey -n 100000 http://localhost:8888/foo

Summary:
  Total:	20.9142 secs
  Slowest:	0.0779 secs
  Fastest:	0.0005 secs
  Average:	0.0104 secs
  Requests/sec:	4781.4349

  Total data:	10300000 bytes
  Size/request:	103 bytes

Response time histogram:
  0.001 [1]	|
  0.008 [43904]	|■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■
  0.016 [40971]	|■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■
  0.024 [10298]	|■■■■■■■■■
  0.031 [3631]	|■■■
  0.039 [894]	|■
  0.047 [230]	|
  0.055 [55]	|
  0.062 [14]	|
  0.070 [1]	|
  0.078 [1]	|


Latency distribution:
  10% in 0.0037 secs
  25% in 0.0057 secs
  50% in 0.0091 secs
  75% in 0.0133 secs
  90% in 0.0185 secs
  95% in 0.0235 secs
  99% in 0.0324 secs

Details (average, fastest, slowest):
  DNS+dialup:	0.0000 secs, 0.0005 secs, 0.0779 secs
  DNS-lookup:	0.0000 secs, 0.0000 secs, 0.0042 secs
  req write:	0.0000 secs, 0.0000 secs, 0.0092 secs
  resp wait:	0.0103 secs, 0.0004 secs, 0.0779 secs
  resp read:	0.0001 secs, 0.0000 secs, 0.0218 secs

Status code distribution:
  [200]	100000 responses

KrakenD

 $  hey -n 100000 http://localhost:8000/foo

Summary:
  Total:	13.3753 secs
  Slowest:	0.0877 secs
  Fastest:	0.0004 secs
  Average:	0.0066 secs
  Requests/sec:	7476.4710

  Total data:	10300000 bytes
  Size/request:	103 bytes

Response time histogram:
  0.000 [1]	|
  0.009 [82034]	|■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■
  0.018 [14582]	|■■■■■■■
  0.027 [2397]	|■
  0.035 [671]	|
  0.044 [189]	|
  0.053 [83]	|
  0.061 [33]	|
  0.070 [7]	|
  0.079 [1]	|
  0.088 [2]	|


Latency distribution:
  10% in 0.0027 secs
  25% in 0.0038 secs
  50% in 0.0054 secs
  75% in 0.0079 secs
  90% in 0.0115 secs
  95% in 0.0150 secs
  99% in 0.0264 secs

Details (average, fastest, slowest):
  DNS+dialup:	0.0000 secs, 0.0004 secs, 0.0877 secs
  DNS-lookup:	0.0000 secs, 0.0000 secs, 0.0039 secs
  req write:	0.0000 secs, 0.0000 secs, 0.0099 secs
  resp wait:	0.0065 secs, 0.0003 secs, 0.0876 secs
  resp read:	0.0001 secs, 0.0000 secs, 0.0294 secs

Status code distribution:
  [200]	100000 responses

KrakenD with Prometheus

 $  hey -n 100000 http://localhost:8000/foo

Summary:
  Total:	15.8054 secs
  Slowest:	0.0824 secs
  Fastest:	0.0004 secs
  Average:	0.0078 secs
  Requests/sec:	6326.9421

  Total data:	10300000 bytes
  Size/request:	103 bytes

Response time histogram:
  0.000 [1]	|
  0.009 [69132]	|■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■
  0.017 [25507]	|■■■■■■■■■■■■■■■
  0.025 [3842]	|■■
  0.033 [994]	|■
  0.041 [414]	|
  0.050 [78]	|
  0.058 [22]	|
  0.066 [3]	|
  0.074 [4]	|
  0.082 [3]	|


Latency distribution:
  10% in 0.0033 secs
  25% in 0.0046 secs
  50% in 0.0067 secs
  75% in 0.0095 secs
  90% in 0.0135 secs
  95% in 0.0172 secs
  99% in 0.0283 secs

Details (average, fastest, slowest):
  DNS+dialup:	0.0000 secs, 0.0004 secs, 0.0824 secs
  DNS-lookup:	0.0000 secs, 0.0000 secs, 0.0069 secs
  req write:	0.0000 secs, 0.0000 secs, 0.0091 secs
  resp wait:	0.0077 secs, 0.0004 secs, 0.0824 secs
  resp read:	0.0001 secs, 0.0000 secs, 0.0098 secs

Status code distribution:
  [200]	100000 responses

Many Thanks for sharing the Data !!!

can you also share config files you have used for statsd and prometheus client with Janus as how have you collected / scrape those metrics.
We will try to perform some tests at our end as well.