Server-side I/O Performance in Python

bottlebench.py

"""Server-side I/O Performance in Python

Based on the article and discussion at:

https://www.toptal.com/back-end/server-side-io-performance-node-php-java-go

The code was posted at:

https://peabody.io/post/server-env-benchmarks/

The winner was golang which is not too surprising given how the benchmark was
run. So I wondered, how would Python compare?

Here's the golang source code:

```go
package main

import (
    "io/ioutil"
    "net/http"
    "fmt"
    "crypto/sha256"
    "log"
)

func main() {

    http.HandleFunc("/test", func(w http.ResponseWriter, r *http.Request) {
        b, err := ioutil.ReadFile("/tmp/data")
        if err != nil { panic(err) }
        s := ""
        nstr := r.FormValue("n")
        var n int
        fmt.Sscanf(nstr, "%d", &n)
        for i := 0; i < n; i++ {
            h := sha256.New()
            h.Write(b)
            s = fmt.Sprintf("%x", h.Sum(nil))
        }
        w.Write([]byte(s))
    })

    log.Fatal(http.ListenAndServe("127.0.0.1:4000", nil))
}
```

To me, that source looks gross but it's pretty easy to translate to Python. I
chose the Bottle web framework because its easy and lightweight. Solution
below.

I benchmarked each of them as described in the original article:

    ab -n 2000 -c 300 "http://127.0.0.1:8000/test?n=1"

Here's the golang results:

```sh
$ ab -n 2000 -c 300 "http://127.0.0.1:4000/test?n=1"
This is ApacheBench, Version 2.3 <$Revision: 1807734 $>
Copyright 1996 Adam Twiss, Zeus Technology Ltd, http://www.zeustech.net/
Licensed to The Apache Software Foundation, http://www.apache.org/

Benchmarking 127.0.0.1 (be patient)
Completed 200 requests
Completed 400 requests
Completed 600 requests
Completed 800 requests
Completed 1000 requests
Completed 1200 requests
Completed 1400 requests
Completed 1600 requests
Completed 1800 requests
Completed 2000 requests
Finished 2000 requests


Server Software:        
Server Hostname:        127.0.0.1
Server Port:            4000

Document Path:          /test?n=1
Document Length:        64 bytes

Concurrency Level:      300
Time taken for tests:   0.292 seconds
Complete requests:      2000
Failed requests:        0
Total transferred:      362000 bytes
HTML transferred:       128000 bytes
Requests per second:    6856.69 [#/sec] (mean)
Time per request:       43.753 [ms] (mean)
Time per request:       0.146 [ms] (mean, across all concurrent requests)
Transfer rate:          1211.97 [Kbytes/sec] received

Connection Times (ms)
              min  mean[+/-sd] median   max
Connect:        0   16  24.6     10     212
Processing:     2   16   9.3     11      50
Waiting:        2   14   7.4     11      49
Total:          6   31  25.0     23     223

Percentage of the requests served within a certain time (ms)
  50%     23
  66%     29
  75%     33
  80%     34
  90%     47
  95%    119
  98%    122
  99%    122
 100%    223 (longest request)
```

And the Python results:

```sh
$ ab -n 2000 -c 300 "http://127.0.0.1:8000/test?n=1"
This is ApacheBench, Version 2.3 <$Revision: 1807734 $>
Copyright 1996 Adam Twiss, Zeus Technology Ltd, http://www.zeustech.net/
Licensed to The Apache Software Foundation, http://www.apache.org/

Benchmarking 127.0.0.1 (be patient)
Completed 200 requests
Completed 400 requests
Completed 600 requests
Completed 800 requests
Completed 1000 requests
Completed 1200 requests
Completed 1400 requests
Completed 1600 requests
Completed 1800 requests
Completed 2000 requests
Finished 2000 requests


Server Software:        gunicorn/19.7.1
Server Hostname:        127.0.0.1
Server Port:            8000

Document Path:          /test?n=1
Document Length:        64 bytes

Concurrency Level:      300
Time taken for tests:   0.374 seconds
Complete requests:      2000
Failed requests:        0
Total transferred:      448000 bytes
HTML transferred:       128000 bytes
Requests per second:    5347.55 [#/sec] (mean)
Time per request:       56.100 [ms] (mean)
Time per request:       0.187 [ms] (mean, across all concurrent requests)
Transfer rate:          1169.78 [Kbytes/sec] received

Connection Times (ms)
              min  mean[+/-sd] median   max
Connect:        0   15  36.8      5     306
Processing:     1   19   4.8     19      42
Waiting:        1   16   4.5     18      41
Total:          8   34  37.7     23     325

Percentage of the requests served within a certain time (ms)
  50%     23
  66%     25
  75%     28
  80%     32
  90%     39
  95%    128
  98%    134
  99%    230
 100%    325 (longest request)
```

The Python results are really good: 5,347 requests per second for Python vs
6,856 for Golang. I expected more from Golang. My guess is the benchmark is
CPU-bound so all the time is spent reading the file and hashing it. For Python,
that all happens in optimized C code.

"""

import hashlib
from bottle import get, request, run

@get('/test')
def test():
    n = int(request.query.n or 1)

    with open('/tmp/data', 'rb') as reader:
        content = reader.read()

    for count in range(n):
        h = hashlib.sha256(content)

    return h.hexdigest()

if __name__ == '__main__':
    run(host='127.0.0.1', port=8000, server='gunicorn', workers=8)

I also ran the benchmark using japronto which is on the bleeding-edge of Python webserver performance. Summary of results:

Configuration	Req/sec	% slower
Python japronto	7,753.77	baseline
Python bottle+gunicorn	5,347.55	31% slower
golang	6,856.69	12% slower

Here's the code:

import hashlib
import multiprocessing as mp
from japronto import Application

def test(request):
    n = int(request.query.get('n', 1))

    with open('/tmp/data', 'rb') as reader:
        content = reader.read()

    for count in range(n):
        h = hashlib.sha256(content)

    return request.Response(text=h.hexdigest())

app = Application()
app.router.add_route('/test', test)

if __name__ == '__main__':
    app.run('127.0.0.1', 8080, worker_num=mp.cpu_count())

And here's the results:

$ ab -n 2000 -c 300 "http://127.0.0.1:8080/test?n=1"
This is ApacheBench, Version 2.3 <$Revision: 1807734 $>
Copyright 1996 Adam Twiss, Zeus Technology Ltd, http://www.zeustech.net/
Licensed to The Apache Software Foundation, http://www.apache.org/

Benchmarking 127.0.0.1 (be patient)
Completed 200 requests
Completed 400 requests
Completed 600 requests
Completed 800 requests
Completed 1000 requests
Completed 1200 requests
Completed 1400 requests
Completed 1600 requests
Completed 1800 requests
Completed 2000 requests
Finished 2000 requests


Server Software:        
Server Hostname:        127.0.0.1
Server Port:            8080

Document Path:          /test?n=1
Document Length:        64 bytes

Concurrency Level:      300
Time taken for tests:   0.258 seconds
Complete requests:      2000
Failed requests:        0
Total transferred:      288000 bytes
HTML transferred:       128000 bytes
Requests per second:    7753.77 [#/sec] (mean)
Time per request:       38.691 [ms] (mean)
Time per request:       0.129 [ms] (mean, across all concurrent requests)
Transfer rate:          1090.37 [Kbytes/sec] received

Connection Times (ms)
              min  mean[+/-sd] median   max
Connect:        1   10  28.3      1     204
Processing:     1    5   7.9      2      38
Waiting:        1    4   7.3      2      33
Total:          2   14  29.2      3     207

Percentage of the requests served within a certain time (ms)
  50%      3
  66%      3
  75%      9
  80%     16
  90%     39
  95%    103
  98%    104
  99%    105
 100%    207 (longest request)

Nice benchmark, thank you.
Could you share hardware/architecture used to get these results ?

I don’t recall exactly. I think it was a 2015 MacBook Pro all running on localhost.

@grantjenks

ioutil.ReadFile has much more cost performance to read a file because it allocate dynamic buffer.

Try change your test with better implementation. Read this: https://www.openmymind.net/Your-Buffer-Is-Leaking/