Skip to content

Instantly share code, notes, and snippets.

Embed
What would you like to do?
Flask vs Falcon vs FastAPI benchmark
gunicorn run:app --workers=9
gunicorn run:app --workers=9 --worker-class=meinheld.gmeinheld.MeinheldWorker

Macbook Pro 2015 Python 3.7

Framework Server Req/s Max latency +/- Stdev
FastAPI uvicorn 15326.66 252.88ms 90.65%
hypercorn 8709.82 130.41ms 78.14%
gunicorn + uvicorn 9283.99 171.82ms 86.06%
FastAPI - async uvicorn 20473.65 142.84ms 85.13%
hypercorn 10436.90 61.90ms 71.20%
gunicorn + uvicorn 11485.82 122.48ms 77.98%
Falcon gunicorn 812.16 401.51ms 99.21%
gunicorn + meinheld 91454.62 134.42ms 91.90%
Falcon cython gunicorn 815.64 523.73ms 98.46%
gunicorn + meinheld 96540.80 128.47ms 90.44%
Flask gunicorn 823.77 314.30ms 99.21%
gunicorn + meinheld 24746.28 46.84ms 75.44%
gunicorn + gevent 6703.45 104.37ms 97.91%
import falcon
class ThingsResource(object):
def on_get(self, req, resp):
"""Handles GET requests"""
resp.status = falcon.HTTP_200 # This is the default status
resp.body = ('pong')
# falcon.API instances are callable WSGI apps
app = falcon.API()
# Resources are represented by long-lived class instances
things = ThingsResource()
# things will handle all requests to the '/things' URL path
app.add_route('/ping', things)
from flask import Flask
app = Flask(__name__)
@app.route("/ping")
def ping():
return "pong"
wrk --duration 20s --threads 1 --connections 100 http://localhost:8000/ping
--------------------------------
Falcon cython
--
Running 20s test @ http://localhost:8000/ping
1 threads and 100 connections
Thread Stats Avg Stdev Max +/- Stdev
Latency 7.14ms 3.68ms 51.96ms 73.50%
Req/Sec 9.33k 1.48k 12.44k 82.65%
185796 requests in 20.05s, 27.46MB read
Requests/sec: 9266.09
Transfer/sec: 1.37MB
--------------------------------
Falcon cython + meinheld
--
Running 20s test @ http://localhost:8000/ping
1 threads and 100 connections
Thread Stats Avg Stdev Max +/- Stdev
Latency 1.36ms 1.75ms 43.63ms 96.45%
Req/Sec 85.32k 10.13k 95.71k 82.00%
1697213 requests in 20.03s, 257.36MB read
Requests/sec: 84737.10
Transfer/sec: 12.85MB
--------------------------------
Falcon
--
Running 20s test @ http://localhost:8000/ping
1 threads and 100 connections
Thread Stats Avg Stdev Max +/- Stdev
Latency 6.97ms 3.36ms 48.88ms 72.74%
Req/Sec 9.51k 1.43k 12.42k 80.10%
189142 requests in 20.06s, 27.96MB read
Requests/sec: 9430.63
Transfer/sec: 1.39MB
--------------------------------
Falcon + meinheld
--
Running 20s test @ http://localhost:8000/ping
1 threads and 100 connections
Thread Stats Avg Stdev Max +/- Stdev
Latency 1.52ms 2.17ms 54.69ms 95.91%
Req/Sec 80.24k 13.45k 92.23k 81.50%
1596695 requests in 20.04s, 242.11MB read
Requests/sec: 79676.11
Transfer/sec: 12.08MB
--------------------------------
Flask + meinheld
--
Running 20s test @ http://localhost:8000/ping
1 threads and 100 connections
Thread Stats Avg Stdev Max +/- Stdev
Latency 3.57ms 2.29ms 52.72ms 87.86%
Req/Sec 29.47k 1.87k 30.87k 88.00%
586316 requests in 20.02s, 93.38MB read
Requests/sec: 29285.76
Transfer/sec: 4.66MB
--------------------------------
Flask
--
Running 20s test @ http://localhost:8000/ping
1 threads and 100 connections
Thread Stats Avg Stdev Max +/- Stdev
Latency 10.54ms 4.23ms 67.09ms 82.06%
Req/Sec 7.93k 1.03k 9.36k 70.05%
157485 requests in 20.01s, 24.48MB read
Requests/sec: 7872.26
Transfer/sec: 1.22MB
--------------------------------
Flask + pypy3.6-7.1.1
--
Running 20s test @ http://localhost:8000/ping
1 threads and 100 connections
Thread Stats Avg Stdev Max +/- Stdev
Latency 28.93ms 14.93ms 241.74ms 66.13%
Req/Sec 3.30k 1.49k 9.65k 83.92%
66034 requests in 20.02s, 10.26MB read
Requests/sec: 3298.62
Transfer/sec: 525.07KB
--------------------------------
Falcon + pypy3.6-7.1.1
--
Running 20s test @ http://localhost:8000/ping
1 threads and 100 connections
Thread Stats Avg Stdev Max +/- Stdev
Latency 9.11ms 10.42ms 146.96ms 83.78%
Req/Sec 9.21k 5.13k 23.23k 55.78%
184084 requests in 20.08s, 27.21MB read
Requests/sec: 9168.11
Transfer/sec: 1.36MB
wrk --duration 20s --threads 10 --connections 200 http://localhost:8000/ping
--------------------------------
Falcon cython
--
Running 20s test @ http://localhost:8000/ping
10 threads and 200 connections
Thread Stats Avg Stdev Max +/- Stdev
Latency 16.30ms 60.85ms 1.68s 98.87%
Req/Sec 1.14k 249.32 2.06k 71.67%
225481 requests in 20.03s, 33.33MB read
Socket errors: connect 0, read 0, write 0, timeout 14
Requests/sec: 11257.30
Transfer/sec: 1.66MB
--------------------------------
Falcon cython + meinheld
--
Running 20s test @ http://localhost:8000/ping
10 threads and 200 connections
Thread Stats Avg Stdev Max +/- Stdev
Latency 3.14ms 2.76ms 48.32ms 82.92%
Req/Sec 7.05k 3.73k 22.49k 83.59%
1407587 requests in 20.08s, 213.44MB read
Requests/sec: 70082.22
Transfer/sec: 10.63MB
--------------------------------
Falcon
--
Running 20s test @ http://localhost:8000/ping
10 threads and 200 connections
Thread Stats Avg Stdev Max +/- Stdev
Latency 16.00ms 50.71ms 1.67s 98.66%
Req/Sec 1.10k 275.07 1.86k 68.70%
218841 requests in 20.06s, 32.35MB read
Requests/sec: 10908.97
Transfer/sec: 1.61MB
--------------------------------
Falcon + meinheld
--
Running 20s test @ http://localhost:8000/ping
10 threads and 200 connections
Thread Stats Avg Stdev Max +/- Stdev
Latency 3.59ms 3.95ms 79.25ms 90.61%
Req/Sec 6.99k 3.49k 21.57k 76.60%
1391987 requests in 20.03s, 211.07MB read
Requests/sec: 69499.75
Transfer/sec: 10.54MB
--------------------------------
Flask + meinheld
--
Running 20s test @ http://localhost:8000/ping
10 threads and 200 connections
Thread Stats Avg Stdev Max +/- Stdev
Latency 8.55ms 7.00ms 87.67ms 74.76%
Req/Sec 2.63k 1.24k 7.20k 69.95%
523834 requests in 20.05s, 83.43MB read
Requests/sec: 26124.87
Transfer/sec: 4.16MB
--------------------------------
Flask
--
Running 20s test @ http://localhost:8000/ping
10 threads and 200 connections
Thread Stats Avg Stdev Max +/- Stdev
Latency 21.15ms 61.75ms 1.68s 98.64%
Req/Sec 823.05 209.44 1.34k 74.05%
163927 requests in 20.05s, 25.48MB read
Socket errors: connect 0, read 0, write 0, timeout 17
Requests/sec: 8174.99
Transfer/sec: 1.27MB
uvicorn run:app --workers 9
hypercorn run:app --workers 9
gunicorn run:app --workers=9 -k uvicorn.workers.UvicornWorker
gunicorn run:app --workers=9
gunicorn run:app --workers=9 --worker-class=meinheld.gmeinheld.MeinheldWorker
wrk --duration 20s --threads 10 --connections 200 http://localhost:8000/ping
------------------------------
FastAPI + uvicorn
--
Running 20s test @ http://localhost:8000/ping
10 threads and 200 connections
Thread Stats Avg Stdev Max +/- Stdev
Latency 17.13ms 19.24ms 252.88ms 90.65%
Req/Sec 1.54k 219.20 2.29k 69.35%
307068 requests in 20.03s, 38.07MB read
Socket errors: connect 0, read 85, write 0, timeout 0
Requests/sec: 15326.66
Transfer/sec: 1.90MB
------------------------------
FastAPI + uvicorn - async
--
Running 20s test @ http://localhost:8000/ping
10 threads and 200 connections
Thread Stats Avg Stdev Max +/- Stdev
Latency 10.55ms 7.91ms 142.84ms 85.13%
Req/Sec 2.06k 611.43 4.66k 63.60%
410659 requests in 20.06s, 50.91MB read
Requests/sec: 20473.65
Transfer/sec: 2.54MB
------------------------------
FastAPI + hypercorn
--
Running 20s test @ http://localhost:8000/ping
10 threads and 200 connections
Thread Stats Avg Stdev Max +/- Stdev
Latency 25.37ms 18.92ms 130.41ms 78.14%
Req/Sec 0.88k 342.42 1.91k 72.30%
174472 requests in 20.03s, 22.30MB read
Requests/sec: 8709.82
Transfer/sec: 1.11MB
------------------------------
FastAPI + hypercorn - async
--
Running 20s test @ http://localhost:8000/ping
10 threads and 200 connections
Thread Stats Avg Stdev Max +/- Stdev
Latency 19.60ms 10.87ms 61.90ms 71.20%
Req/Sec 1.05k 359.66 2.04k 60.55%
209097 requests in 20.03s, 26.72MB read
Requests/sec: 10436.90
Transfer/sec: 1.33MB
------------------------------
FastAPI + gunicorn + uvicorn
--
Running 20s test @ http://localhost:8000/ping
10 threads and 200 connections
Thread Stats Avg Stdev Max +/- Stdev
Latency 26.29ms 24.72ms 171.82ms 86.06%
Req/Sec 0.94k 484.69 2.06k 56.80%
186557 requests in 20.09s, 23.13MB read
Socket errors: connect 0, read 36, write 0, timeout 0
Requests/sec: 9283.99
Transfer/sec: 1.15MB
------------------------------
FastAPI + gunicorn + uvicorn - async
--
Running 20s test @ http://localhost:8000/ping
10 threads and 200 connections
Thread Stats Avg Stdev Max +/- Stdev
Latency 19.25ms 14.96ms 122.48ms 77.98%
Req/Sec 1.15k 291.98 2.04k 60.30%
230221 requests in 20.04s, 28.54MB read
Socket errors: connect 0, read 76, write 0, timeout 0
Requests/sec: 11485.82
Transfer/sec: 1.42MB
------------------------------
Falcon + gunicorn
--
Running 20s test @ http://localhost:8000/ping
10 threads and 200 connections
Thread Stats Avg Stdev Max +/- Stdev
Latency 21.58ms 33.17ms 401.51ms 99.21%
Req/Sec 572.84 260.23 1.08k 74.64%
16279 requests in 20.04s, 2.41MB read
Socket errors: connect 0, read 412, write 6, timeout 0
Requests/sec: 812.16
Transfer/sec: 122.93KB
------------------------------
Falcon + gunicorn + meinheld
--
Running 20s test @ http://localhost:8000/ping
10 threads and 200 connections
Thread Stats Avg Stdev Max +/- Stdev
Latency 2.90ms 4.33ms 134.42ms 91.90%
Req/Sec 9.20k 1.25k 15.58k 78.15%
1838302 requests in 20.10s, 278.75MB read
Socket errors: connect 0, read 39, write 0, timeout 0
Requests/sec: 91454.62
Transfer/sec: 13.87MB
------------------------------
Falcon cython + gunicorn
--
Running 20s test @ http://localhost:8000/ping
10 threads and 200 connections
Thread Stats Avg Stdev Max +/- Stdev
Latency 23.35ms 47.76ms 523.73ms 98.46%
Req/Sec 608.04 290.92 1.06k 69.66%
16387 requests in 20.09s, 2.42MB read
Socket errors: connect 0, read 357, write 4, timeout 0
Requests/sec: 815.64
Transfer/sec: 123.46KB
------------------------------
Falcon cython + gunicorn + meinheld
--
Running 20s test @ http://localhost:8000/ping
10 threads and 200 connections
Thread Stats Avg Stdev Max +/- Stdev
Latency 2.67ms 3.67ms 128.47ms 90.44%
Req/Sec 9.70k 1.38k 14.16k 66.70%
1931630 requests in 20.01s, 292.90MB read
Socket errors: connect 0, read 44, write 0, timeout 0
Requests/sec: 96540.80
Transfer/sec: 14.64MB
------------------------------
Flask + gunicorn
--
Running 20s test @ http://localhost:8000/ping
10 threads and 200 connections
Thread Stats Avg Stdev Max +/- Stdev
Latency 23.87ms 25.11ms 314.30ms 99.21%
Req/Sec 521.89 193.32 0.88k 76.62%
16528 requests in 20.06s, 2.57MB read
Socket errors: connect 0, read 341, write 5, timeout 0
Requests/sec: 823.77
Transfer/sec: 131.13KB
------------------------------
Flask + gunicorn + meinheld
--
Running 20s test @ http://localhost:8000/ping
10 threads and 200 connections
Thread Stats Avg Stdev Max +/- Stdev
Latency 8.20ms 4.24ms 46.84ms 75.44%
Req/Sec 2.49k 565.34 3.98k 77.55%
495311 requests in 20.02s, 78.89MB read
Socket errors: connect 0, read 46, write 0, timeout 0
Requests/sec: 24746.28
Transfer/sec: 3.94MB
------------------------------
Flask + gunicorn + gevent
--
Running 20s test @ http://localhost:8000/ping
10 threads and 200 connections
Thread Stats Avg Stdev Max +/- Stdev
Latency 1.49ms 2.73ms 104.37ms 97.91%
Req/Sec 1.31k 553.40 3.36k 60.99%
134625 requests in 20.08s, 21.57MB read
Socket errors: connect 0, read 0, write 0, timeout 23
Requests/sec: 6703.45
Transfer/sec: 1.07MB
@Kamrulhasan12345
Copy link

Kamrulhasan12345 commented Nov 30, 2020

Wow. Thank you very much for sharing this detailed benchmarking results.

@CTimmerman
Copy link

CTimmerman commented Feb 24, 2021

https://falconframework.org/ says:

PyPy3 7.0.0  speedup req/sec μs/req
Falcon (2.0.0) 75x 330.676 3.02
Bottle (0.12.16) 37x 162.274 6.16
Werkzeug (0.15.4) 11x 47.395 21.10
Flask (1.0.2) 8x 36.746 27.21
Django (2.2.1) 1x 4.436 225.41

@nhymxu
Copy link
Author

nhymxu commented Feb 24, 2021

@CTimmerman
This is just old bench for myself.
I'm not tested with PyPy, just Cpython ( install using Homebrew )

I guest that your result only test with default webserver.
You can increase performance on production by using other webserver and other worker.

I'm still using gunicorn + meinheld worker for both flask and django on production

@CTimmerman
Copy link

CTimmerman commented Feb 24, 2021

In Mint, this appears to work:

mkdir api_test && cd $_
apt install pypy3
pypy3 -m venv .venv_pypy3
source .venv_pypy3/bin/activate
pypy3 -m pip install fastapi uvicorn
nano fastapi_test.py

Paste:

import asyncio
import uvicorn
from fastapi import FastAPI
app = FastAPI()

@app.get('/')
async def root():
    # print('Sleeping for 10')
    # await asyncio.sleep(10)
    # print('Awake')
    return {'message': 'hello'}

if __name__ == "__main__":
    uvicorn.run(app, host="127.0.0.1", port=8000, workers=9)

Run:

pypy3 fastapi_test.py > /dev/null

Benchmark (2nd run):

$ siege --benchmark --time=60s --concurrent=20 http://127.0.0.1:8001
** SIEGE 4.0.4
** Preparing 20 concurrent users for battle.
The server is now under siege...
Lifting the server siege...
Transactions:		      164965 hits
Availability:		      100.00 %
Elapsed time:		       59.12 secs
Data transferred:	        2.99 MB
Response time:		        0.01 secs
Transaction rate:	     2790.34 trans/sec
Throughput:		        0.05 MB/sec
Concurrency:		       19.85
Successful transactions:      164965
Failed transactions:	           0
Longest transaction:	        0.11
Shortest transaction:	        0.00

Exit .venv_pypy3 and setup up Python3 version:

$ deactivate
$ apt install python3-venv
$ python3 -m venv .venv_python3
$ source .venv_python3/bin/activate
$ python3 -m pip install fastapi uvicorn
$ python3 fastapi_test.py > /dev/null

Benchmark (first run):

$ siege --benchmark --time=60s --concurrent=20 http://127.0.0.1:8001
** SIEGE 4.0.4
** Preparing 20 concurrent users for battle.
The server is now under siege...
Lifting the server siege...
Transactions:		      142478 hits
Availability:		      100.00 %
Elapsed time:		       59.38 secs
Data transferred:	        2.58 MB
Response time:		        0.01 secs
Transaction rate:	     2399.43 trans/sec
Throughput:		        0.04 MB/sec
Concurrency:		       19.89
Successful transactions:      142478
Failed transactions:	           0
Longest transaction:	        0.11
Shortest transaction:	        0.00

That's in a VirtualBox on my busy laptop.

@nhymxu
Copy link
Author

nhymxu commented Feb 28, 2021

Thanks for your bench.

I think difference environment, difference test tool will generate difference results.

Choose X over Y is hard. Feel free to choose. 😃

@CTimmerman
Copy link

CTimmerman commented Feb 28, 2021

I think difference environment, difference test tool will generate difference results.

Of courses, which is why i did both PyPy and CPython to show that PyPy is faster in this case.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment