2017-07-26 Asset Manager Performance Testing

README.md

Creating files of n size

dd if=/dev/zero of=1MB.dat  bs=1m count=1

---

Upload file to Asset Manager

curl http://asset-manager.dev.gov.uk/assets --form "asset[file]=@1MB.dat"

---

Using Apache Bench to test performance

We request assets from asset-manager.dev.gov.uk in order to avoid the request going via assets-origin.dev.gov.uk and static.dev.gov.uk. The request does go through Nginx that handles the SENDFILE header sent from the Asset Manager Rails app.

The Asset Manager app is run within Webrick on the Development VM which we think means it's running single-threaded so we used the default concurrency of 1 with Apache Bench.

# Serving files from disk
$ ab -n1000 "http://asset-manager.dev.gov.uk/media/5978b0e8759b740ff9000000/1MB.dat"

# Serving files from S3
$ ab -n1000 "http://asset-manager.dev.gov.uk/media/5978b0e8759b740ff9000000/1MB.dat?stream_from_s3=true"

---

Proxying requests to S3 via nginx

https://distinctplace.com/2013/09/18/use-nginx-to-proxy-files-from-remote-location-using-x-accel-redirect/

results-1MB-file-serving-from-disk.txt

$ ab -n1000 "http://asset-manager.dev.gov.uk/media/5978b0e8759b740ff9000000/1MB.dat"
This is ApacheBench, Version 2.3 <$Revision: 1706008 $>
Copyright 1996 Adam Twiss, Zeus Technology Ltd, http://www.zeustech.net/
Licensed to The Apache Software Foundation, http://www.apache.org/

Benchmarking asset-manager.dev.gov.uk (be patient)
Completed 100 requests
Completed 200 requests
Completed 300 requests
Completed 400 requests
Completed 500 requests
Completed 600 requests
Completed 700 requests
Completed 800 requests
Completed 900 requests
Completed 1000 requests
Finished 1000 requests


Server Software:        nginx
Server Hostname:        asset-manager.dev.gov.uk
Server Port:            80

Document Path:          /media/5978b0e8759b740ff9000000/1MB.dat
Document Length:        1048576 bytes

Concurrency Level:      1
Time taken for tests:   24.920 seconds
Complete requests:      1000
Failed requests:        0
Total transferred:      1048920000 bytes
HTML transferred:       1048576000 bytes
Requests per second:    40.13 [#/sec] (mean)
Time per request:       24.920 [ms] (mean)
Time per request:       24.920 [ms] (mean, across all concurrent requests)
Transfer rate:          41105.02 [Kbytes/sec] received

Connection Times (ms)
              min  mean[+/-sd] median   max
Connect:        0    0   0.2      0       2
Processing:    17   24   9.5     21      96
Waiting:        9   14   7.8     11      67
Total:         17   25   9.5     21      96

Percentage of the requests served within a certain time (ms)
  50%     21
  66%     23
  75%     25
  80%     28
  90%     37
  95%     45
  98%     57
  99%     65
 100%     96 (longest request)
 

results-1MB-file-serving-from-s3-and-proxying-through-nginx-with-concurrency.txt

$ ab -c5 -n1000 "http://asset-manager.dev.gov.uk/media/5978bbe5759b7434d5000000/1MB.dat?stream_from_s3=true"
This is ApacheBench, Version 2.3 <$Revision: 1706008 $>
Copyright 1996 Adam Twiss, Zeus Technology Ltd, http://www.zeustech.net/
Licensed to The Apache Software Foundation, http://www.apache.org/

Benchmarking asset-manager.dev.gov.uk (be patient)
Completed 100 requests
Completed 200 requests
Completed 300 requests
Completed 400 requests
Completed 500 requests
Completed 600 requests
Completed 700 requests
Completed 800 requests
Completed 900 requests
Completed 1000 requests
Finished 1000 requests


Server Software:        nginx
Server Hostname:        asset-manager.dev.gov.uk
Server Port:            80

Document Path:          /media/5978bbe5759b7434d5000000/1MB.dat?stream_from_s3=true
Document Length:        1048576 bytes

Concurrency Level:      5
Time taken for tests:   998.892 seconds
Complete requests:      1000
Failed requests:        209
   (Connect: 0, Receive: 0, Length: 209, Exceptions: 0)
Non-2xx responses:      133
Total transferred:      869846310 bytes
HTML transferred:       869455196 bytes
Requests per second:    1.00 [#/sec] (mean)
Time per request:       4994.459 [ms] (mean)
Time per request:       998.892 [ms] (mean, across all concurrent requests)
Transfer rate:          850.40 [Kbytes/sec] received

Connection Times (ms)
              min  mean[+/-sd] median   max
Connect:        0    0   0.7      0      19
Processing:   172 4915 8295.2   1740   54705
Waiting:       44 1050 1831.6     95   15162
Total:        172 4915 8295.2   1741   54705

Percentage of the requests served within a certain time (ms)
  50%   1741
  66%   2709
  75%   4637
  80%   6285
  90%  10450
  95%  28406
  98%  36279
  99%  40528
 100%  54705 (longest request)
 

results-1MB-file-serving-from-s3-and-proxying-through-nginx.txt

$ ab -n1000 "http://asset-manager.dev.gov.uk/media/5978bbe5759b7434d5000000/1MB.dat?stream_from_s3=true"
This is ApacheBench, Version 2.3 <$Revision: 1706008 $>
Copyright 1996 Adam Twiss, Zeus Technology Ltd, http://www.zeustech.net/
Licensed to The Apache Software Foundation, http://www.apache.org/

Benchmarking asset-manager.dev.gov.uk (be patient)
Completed 100 requests
Completed 200 requests
Completed 300 requests
Completed 400 requests
Completed 500 requests
Completed 600 requests
Completed 700 requests
Completed 800 requests
Completed 900 requests
Completed 1000 requests
Finished 1000 requests


Server Software:        nginx
Server Hostname:        asset-manager.dev.gov.uk
Server Port:            80

Document Path:          /media/5978bbe5759b7434d5000000/1MB.dat?stream_from_s3=true
Document Length:        1048576 bytes

Concurrency Level:      1
Time taken for tests:   1521.197 seconds
Complete requests:      1000
Failed requests:        2
   (Connect: 0, Receive: 0, Length: 2, Exceptions: 0)
Non-2xx responses:      1
Total transferred:      1047178950 bytes
HTML transferred:       1046757182 bytes
Requests per second:    0.66 [#/sec] (mean)
Time per request:       1521.197 [ms] (mean)
Time per request:       1521.197 [ms] (mean, across all concurrent requests)
Transfer rate:          672.26 [Kbytes/sec] received

Connection Times (ms)
              min  mean[+/-sd] median   max
Connect:        0    0   0.2      0       2
Processing:   185 1521 1085.1   1187   25609
Waiting:       43  106 276.4     65    5078
Total:        185 1521 1085.1   1187   25610

Percentage of the requests served within a certain time (ms)
  50%   1187
  66%   1681
  75%   1926
  80%   2070
  90%   2555
  95%   2874
  98%   3460
  99%   3917
 100%  25610 (longest request)
 

results-1MB-file-serving-from-s3.txt

$ ab -n1000 "http://asset-manager.dev.gov.uk/media/5978b0e8759b740ff9000000/1MB.dat?stream_from_s3=true"
This is ApacheBench, Version 2.3 <$Revision: 1706008 $>
Copyright 1996 Adam Twiss, Zeus Technology Ltd, http://www.zeustech.net/
Licensed to The Apache Software Foundation, http://www.apache.org/

Benchmarking asset-manager.dev.gov.uk (be patient)
Completed 100 requests
Completed 200 requests
Completed 300 requests
Completed 400 requests
Completed 500 requests
Completed 600 requests
Completed 700 requests
Completed 800 requests
Completed 900 requests
Completed 1000 requests
Finished 1000 requests


Server Software:        nginx
Server Hostname:        asset-manager.dev.gov.uk
Server Port:            80

Document Path:          /media/5978b0e8759b740ff9000000/1MB.dat?stream_from_s3=true
Document Length:        1048576 bytes

Concurrency Level:      1
Time taken for tests:   1591.877 seconds
Complete requests:      1000
Failed requests:        0
Total transferred:      1049075000 bytes
HTML transferred:       1048576000 bytes
Requests per second:    0.63 [#/sec] (mean)
Time per request:       1591.877 [ms] (mean)
Time per request:       1591.877 [ms] (mean, across all concurrent requests)
Transfer rate:          643.57 [Kbytes/sec] received

Connection Times (ms)
              min  mean[+/-sd] median   max
Connect:        0    0   0.3      0       3
Processing:   280 1591 714.3   1391    4372
Waiting:      267 1579 714.5   1378    4361
Total:        280 1592 714.3   1391    4373

Percentage of the requests served within a certain time (ms)
  50%   1391
  66%   1806
  75%   1987
  80%   2118
  90%   2645
  95%   2995
  98%   3392
  99%   3800
 100%   4373 (longest request)
 

results-small-file-serving-from-disk.txt

$ ab -n1000 "http://asset-manager.dev.gov.uk/media/5978a7fc759b746b81000000/tmp.txt"
This is ApacheBench, Version 2.3 <$Revision: 1706008 $>
Copyright 1996 Adam Twiss, Zeus Technology Ltd, http://www.zeustech.net/
Licensed to The Apache Software Foundation, http://www.apache.org/

Benchmarking asset-manager.dev.gov.uk (be patient)
Completed 100 requests
Completed 200 requests
Completed 300 requests
Completed 400 requests
Completed 500 requests
Completed 600 requests
Completed 700 requests
Completed 800 requests
Completed 900 requests
Completed 1000 requests
Finished 1000 requests


Server Software:        nginx
Server Hostname:        asset-manager.dev.gov.uk
Server Port:            80

Document Path:          /media/5978a7fc759b746b81000000/tmp.txt
Document Length:        29 bytes

Concurrency Level:      1
Time taken for tests:   16.223 seconds
Complete requests:      1000
Failed requests:        0
Total transferred:      373000 bytes
HTML transferred:       29000 bytes
Requests per second:    61.64 [#/sec] (mean)
Time per request:       16.223 [ms] (mean)
Time per request:       16.223 [ms] (mean, across all concurrent requests)
Transfer rate:          22.45 [Kbytes/sec] received

Connection Times (ms)
              min  mean[+/-sd] median   max
Connect:        0    0   0.2      0       4
Processing:     9   16   8.8     12      90
Waiting:        9   16   8.8     12      90
Total:          9   16   8.9     12      91

Percentage of the requests served within a certain time (ms)
  50%     12
  66%     15
  75%     18
  80%     19
  90%     28
  95%     35
  98%     44
  99%     51
 100%     91 (longest request)
 

results-small-file-serving-from-s3.txt

$ ab -n1000 "http://asset-manager.dev.gov.uk/media/5978a7fc759b746b81000000/tmp.txt?stream_from_s3=true"
This is ApacheBench, Version 2.3 <$Revision: 1706008 $>
Copyright 1996 Adam Twiss, Zeus Technology Ltd, http://www.zeustech.net/
Licensed to The Apache Software Foundation, http://www.apache.org/

Benchmarking asset-manager.dev.gov.uk (be patient)
Completed 100 requests
Completed 200 requests
Completed 300 requests
Completed 400 requests
Completed 500 requests
Completed 600 requests
Completed 700 requests
Completed 800 requests
Completed 900 requests
Completed 1000 requests
Finished 1000 requests


Server Software:        nginx
Server Hostname:        asset-manager.dev.gov.uk
Server Port:            80

Document Path:          /media/5978a7fc759b746b81000000/tmp.txt?stream_from_s3=true
Document Length:        29 bytes

Concurrency Level:      1
Time taken for tests:   45.915 seconds
Complete requests:      1000
Failed requests:        0
Total transferred:      532000 bytes
HTML transferred:       29000 bytes
Requests per second:    21.78 [#/sec] (mean)
Time per request:       45.915 [ms] (mean)
Time per request:       45.915 [ms] (mean, across all concurrent requests)
Transfer rate:          11.32 [Kbytes/sec] received

Connection Times (ms)
              min  mean[+/-sd] median   max
Connect:        0    0   0.2      0       4
Processing:    32   46  24.7     38     417
Waiting:       32   45  24.7     38     416
Total:         32   46  24.7     39     417

Percentage of the requests served within a certain time (ms)
  50%     39
  66%     41
  75%     44
  80%     46
  90%     58
  95%     84
  98%    139
  99%    157
 100%    417 (longest request)