060-cache-benchmark.md

Kong Benchmark

Kong 0.6.0 introduces Clustering and a new in-memory core cache invalidation strategy which dramatically improves the performance of the system. The following benchmark measures the in-memory caching improvements of Kong 0.6.0 compared to the previous Kong 0.5.4 which still uses the old time-based expiration strategy.

The test involves consuming an API running on a separate machine, that returns a simple 200 OK JSON response with the following body:

{"message": "hello world"}

Note that we are not talking about HTTP caching, but the internal in-memory cache of Lua objects stored by Kong. Any improvement in the core cache will also improve the performance of Kong plugins.

Machines

Kong 0.5.4:

• 1 AWS m3.medium instance with Cassandra 2.2.4 (OpenJDK 7)
• 1 AWS m3.medium instance with Kong 0.5.4

Kong 0.6.0:

• 1 AWS m3.medium instance with Cassandra 2.2.4 (OpenJDK 7)
• 1 AWS m3.medium instance with Kong 0.6.0

Upstream API:

• 1 AWS c4.2xlarge instance running nginx with the following configuration:

user  nginx;
worker_processes  auto;
error_log /var/log/nginx/error.log;
pid /var/run/nginx.pid;

events {
  worker_connections 10240;
}

http {
  include /etc/nginx/mime.types;
  default_type application/json;

  server {
    listen 80;
    server_name localhost;
    root /usr/share/nginx/html;

    location / {
      return 200 "{\"message\":\"hello world\"}";
    }

  }
}

Benchmarking Client:

• 1 AWS c4.2xlarge instance running autobench.

Configuration

Each instance had the following configuration:

/etc/sysctl.conf

...

fs.file-max = 100000

/etc/security/limits.conf

...

* soft nofile 100000
* hard nofile 100000
* soft nproc 100000
* hard nproc 100000

Benchmarking Command

The benchmark was executed with Autobench, using the following command:

autobench --host1 .. --port1 8000 --uri1 /test \
          --host2 .. --port2 8000 --uri2 /test \
          --low_rate 20 --high_rate 200 --rate_step 20 --num_call 10 --num_conn 5000 --timeout 5 \
          --file results.tsv

Will benchmark the Kong machines with a series of tests starting at 20 connections per second (with 10 requests per connection), and increasing by 20 connections per second until 200 connections a second are being requested. Each test will comprise a total of 5000 connections, and any responses which take longer than 5 seconds to arrive will be counted as errors. The results will be saved in the file 'results.tsv'.

Results

(Y: req/s for reply rate, ms for full roundtrip response time, % for error rate - X: number of req/s)

As we can see in the charts, Kong 0.6.0 is able to handle more requests, with a smaller response time and error rate, than 0.5.4. Errors were expected since the test was meant to push Kong to the limit, so with an appropriate number of instances and the appropriate instance types, the error rate in production should aim at zero.

Another relevant conclusion is that Kong 0.6.0 is more predictable than 0.5.4, caused by the fact that it relies on the datastore only on the first request, and never communicates with the datastore again if the data doesn't change. On the other side we can notice that 0.5.4 is more unpredictable, and its capacity, response time and error rate goes up or down based on the performance of the datastore and the network (used to to communicate with the datastore). Thus 0.6.0, by relying on less variables, provides a better production uptime.