Flask vs Falcon vs FastAPI benchmark

README-python-framework-benchmark.md

gunicorn run:app --workers=9
gunicorn run:app --workers=9 --worker-class=meinheld.gmeinheld.MeinheldWorker

Macbook Pro 2015 Python 3.7

Framework	Server	Req/s	Max latency	+/- Stdev
FastAPI	uvicorn	15326.66	252.88ms	90.65%
	hypercorn	8709.82	130.41ms	78.14%
	gunicorn + uvicorn	9283.99	171.82ms	86.06%
FastAPI - async	uvicorn	20473.65	142.84ms	85.13%
	hypercorn	10436.90	61.90ms	71.20%
	gunicorn + uvicorn	11485.82	122.48ms	77.98%
Falcon	gunicorn	812.16	401.51ms	99.21%
	gunicorn + meinheld	91454.62	134.42ms	91.90%
Falcon cython	gunicorn	815.64	523.73ms	98.46%
	gunicorn + meinheld	96540.80	128.47ms	90.44%
Flask	gunicorn	823.77	314.30ms	99.21%
	gunicorn + meinheld	24746.28	46.84ms	75.44%
	gunicorn + gevent	6703.45	104.37ms	97.91%

falcon-run.py

import falcon

class ThingsResource(object):
    def on_get(self, req, resp):
        """Handles GET requests"""
        resp.status = falcon.HTTP_200  # This is the default status
        resp.body = ('pong')

# falcon.API instances are callable WSGI apps
app = falcon.API()

# Resources are represented by long-lived class instances
things = ThingsResource()

# things will handle all requests to the '/things' URL path
app.add_route('/ping', things)

flask-run.py

from flask import Flask
app = Flask(__name__)

@app.route("/ping")
def ping():
    return "pong"

result1.txt

wrk --duration 20s --threads 1 --connections 100 http://localhost:8000/ping
--------------------------------
Falcon cython
--
Running 20s test @ http://localhost:8000/ping
  1 threads and 100 connections
  Thread Stats   Avg      Stdev     Max   +/- Stdev
    Latency     7.14ms    3.68ms  51.96ms   73.50%
    Req/Sec     9.33k     1.48k   12.44k    82.65%
  185796 requests in 20.05s, 27.46MB read
Requests/sec:   9266.09
Transfer/sec:      1.37MB

--------------------------------
Falcon cython + meinheld
--
Running 20s test @ http://localhost:8000/ping
  1 threads and 100 connections
  Thread Stats   Avg      Stdev     Max   +/- Stdev
    Latency     1.36ms    1.75ms  43.63ms   96.45%
    Req/Sec    85.32k    10.13k   95.71k    82.00%
  1697213 requests in 20.03s, 257.36MB read
Requests/sec:  84737.10
Transfer/sec:     12.85MB

--------------------------------
Falcon
--
Running 20s test @ http://localhost:8000/ping
  1 threads and 100 connections
  Thread Stats   Avg      Stdev     Max   +/- Stdev
    Latency     6.97ms    3.36ms  48.88ms   72.74%
    Req/Sec     9.51k     1.43k   12.42k    80.10%
  189142 requests in 20.06s, 27.96MB read
Requests/sec:   9430.63
Transfer/sec:      1.39MB

--------------------------------
Falcon + meinheld
--
Running 20s test @ http://localhost:8000/ping
  1 threads and 100 connections
  Thread Stats   Avg      Stdev     Max   +/- Stdev
    Latency     1.52ms    2.17ms  54.69ms   95.91%
    Req/Sec    80.24k    13.45k   92.23k    81.50%
  1596695 requests in 20.04s, 242.11MB read
Requests/sec:  79676.11
Transfer/sec:     12.08MB

--------------------------------
Flask + meinheld
--
Running 20s test @ http://localhost:8000/ping
  1 threads and 100 connections
  Thread Stats   Avg      Stdev     Max   +/- Stdev
    Latency     3.57ms    2.29ms  52.72ms   87.86%
    Req/Sec    29.47k     1.87k   30.87k    88.00%
  586316 requests in 20.02s, 93.38MB read
Requests/sec:  29285.76
Transfer/sec:      4.66MB

--------------------------------
Flask
--
Running 20s test @ http://localhost:8000/ping
  1 threads and 100 connections
  Thread Stats   Avg      Stdev     Max   +/- Stdev
    Latency    10.54ms    4.23ms  67.09ms   82.06%
    Req/Sec     7.93k     1.03k    9.36k    70.05%
  157485 requests in 20.01s, 24.48MB read
Requests/sec:   7872.26
Transfer/sec:      1.22MB

--------------------------------
Flask + pypy3.6-7.1.1
--
Running 20s test @ http://localhost:8000/ping
  1 threads and 100 connections
  Thread Stats   Avg      Stdev     Max   +/- Stdev
    Latency    28.93ms   14.93ms 241.74ms   66.13%
    Req/Sec     3.30k     1.49k    9.65k    83.92%
  66034 requests in 20.02s, 10.26MB read
Requests/sec:   3298.62
Transfer/sec:    525.07KB

--------------------------------
Falcon + pypy3.6-7.1.1
--
Running 20s test @ http://localhost:8000/ping
  1 threads and 100 connections
  Thread Stats   Avg      Stdev     Max   +/- Stdev
    Latency     9.11ms   10.42ms 146.96ms   83.78%
    Req/Sec     9.21k     5.13k   23.23k    55.78%
  184084 requests in 20.08s, 27.21MB read
Requests/sec:   9168.11
Transfer/sec:      1.36MB

result2.txt

wrk --duration 20s --threads 10 --connections 200 http://localhost:8000/ping
--------------------------------
Falcon cython
--
Running 20s test @ http://localhost:8000/ping
  10 threads and 200 connections
  Thread Stats   Avg      Stdev     Max   +/- Stdev
    Latency    16.30ms   60.85ms   1.68s    98.87%
    Req/Sec     1.14k   249.32     2.06k    71.67%
  225481 requests in 20.03s, 33.33MB read
  Socket errors: connect 0, read 0, write 0, timeout 14
Requests/sec:  11257.30
Transfer/sec:      1.66MB

--------------------------------
Falcon cython + meinheld
--
Running 20s test @ http://localhost:8000/ping
  10 threads and 200 connections
  Thread Stats   Avg      Stdev     Max   +/- Stdev
    Latency     3.14ms    2.76ms  48.32ms   82.92%
    Req/Sec     7.05k     3.73k   22.49k    83.59%
  1407587 requests in 20.08s, 213.44MB read
Requests/sec:  70082.22
Transfer/sec:     10.63MB

--------------------------------
Falcon
--
Running 20s test @ http://localhost:8000/ping
  10 threads and 200 connections
  Thread Stats   Avg      Stdev     Max   +/- Stdev
    Latency    16.00ms   50.71ms   1.67s    98.66%
    Req/Sec     1.10k   275.07     1.86k    68.70%
  218841 requests in 20.06s, 32.35MB read
Requests/sec:  10908.97
Transfer/sec:      1.61MB

--------------------------------
Falcon + meinheld
--
Running 20s test @ http://localhost:8000/ping
  10 threads and 200 connections
  Thread Stats   Avg      Stdev     Max   +/- Stdev
    Latency     3.59ms    3.95ms  79.25ms   90.61%
    Req/Sec     6.99k     3.49k   21.57k    76.60%
  1391987 requests in 20.03s, 211.07MB read
Requests/sec:  69499.75
Transfer/sec:     10.54MB

--------------------------------
Flask + meinheld
--
Running 20s test @ http://localhost:8000/ping
  10 threads and 200 connections
  Thread Stats   Avg      Stdev     Max   +/- Stdev
    Latency     8.55ms    7.00ms  87.67ms   74.76%
    Req/Sec     2.63k     1.24k    7.20k    69.95%
  523834 requests in 20.05s, 83.43MB read
Requests/sec:  26124.87
Transfer/sec:      4.16MB

--------------------------------
Flask
--
Running 20s test @ http://localhost:8000/ping
  10 threads and 200 connections
  Thread Stats   Avg      Stdev     Max   +/- Stdev
    Latency    21.15ms   61.75ms   1.68s    98.64%
    Req/Sec   823.05    209.44     1.34k    74.05%
  163927 requests in 20.05s, 25.48MB read
  Socket errors: connect 0, read 0, write 0, timeout 17
Requests/sec:   8174.99
Transfer/sec:      1.27MB

result3.txt

uvicorn run:app --workers 9
hypercorn run:app --workers 9

gunicorn run:app --workers=9 -k uvicorn.workers.UvicornWorker

gunicorn run:app --workers=9
gunicorn run:app --workers=9 --worker-class=meinheld.gmeinheld.MeinheldWorker

wrk --duration 20s --threads 10 --connections 200 http://localhost:8000/ping

------------------------------
FastAPI + uvicorn
--
Running 20s test @ http://localhost:8000/ping
  10 threads and 200 connections
  Thread Stats   Avg      Stdev     Max   +/- Stdev
    Latency    17.13ms   19.24ms 252.88ms   90.65%
    Req/Sec     1.54k   219.20     2.29k    69.35%
  307068 requests in 20.03s, 38.07MB read
  Socket errors: connect 0, read 85, write 0, timeout 0
Requests/sec:  15326.66
Transfer/sec:      1.90MB

------------------------------
FastAPI + uvicorn - async
--
Running 20s test @ http://localhost:8000/ping
  10 threads and 200 connections
  Thread Stats   Avg      Stdev     Max   +/- Stdev
    Latency    10.55ms    7.91ms 142.84ms   85.13%
    Req/Sec     2.06k   611.43     4.66k    63.60%
  410659 requests in 20.06s, 50.91MB read
Requests/sec:  20473.65
Transfer/sec:      2.54MB

------------------------------
FastAPI + hypercorn
--
Running 20s test @ http://localhost:8000/ping
  10 threads and 200 connections
  Thread Stats   Avg      Stdev     Max   +/- Stdev
    Latency    25.37ms   18.92ms 130.41ms   78.14%
    Req/Sec     0.88k   342.42     1.91k    72.30%
  174472 requests in 20.03s, 22.30MB read
Requests/sec:   8709.82
Transfer/sec:      1.11MB

------------------------------
FastAPI + hypercorn - async
--
Running 20s test @ http://localhost:8000/ping
  10 threads and 200 connections
  Thread Stats   Avg      Stdev     Max   +/- Stdev
    Latency    19.60ms   10.87ms  61.90ms   71.20%
    Req/Sec     1.05k   359.66     2.04k    60.55%
  209097 requests in 20.03s, 26.72MB read
Requests/sec:  10436.90
Transfer/sec:      1.33MB

------------------------------
FastAPI + gunicorn + uvicorn
--
Running 20s test @ http://localhost:8000/ping
  10 threads and 200 connections
  Thread Stats   Avg      Stdev     Max   +/- Stdev
    Latency    26.29ms   24.72ms 171.82ms   86.06%
    Req/Sec     0.94k   484.69     2.06k    56.80%
  186557 requests in 20.09s, 23.13MB read
  Socket errors: connect 0, read 36, write 0, timeout 0
Requests/sec:   9283.99
Transfer/sec:      1.15MB

------------------------------
FastAPI + gunicorn + uvicorn - async
--
Running 20s test @ http://localhost:8000/ping
  10 threads and 200 connections
  Thread Stats   Avg      Stdev     Max   +/- Stdev
    Latency    19.25ms   14.96ms 122.48ms   77.98%
    Req/Sec     1.15k   291.98     2.04k    60.30%
  230221 requests in 20.04s, 28.54MB read
  Socket errors: connect 0, read 76, write 0, timeout 0
Requests/sec:  11485.82
Transfer/sec:      1.42MB

------------------------------
Falcon + gunicorn
--
Running 20s test @ http://localhost:8000/ping
  10 threads and 200 connections
  Thread Stats   Avg      Stdev     Max   +/- Stdev
    Latency    21.58ms   33.17ms 401.51ms   99.21%
    Req/Sec   572.84    260.23     1.08k    74.64%
  16279 requests in 20.04s, 2.41MB read
  Socket errors: connect 0, read 412, write 6, timeout 0
Requests/sec:    812.16
Transfer/sec:    122.93KB

------------------------------
Falcon + gunicorn + meinheld
--
Running 20s test @ http://localhost:8000/ping
  10 threads and 200 connections
  Thread Stats   Avg      Stdev     Max   +/- Stdev
    Latency     2.90ms    4.33ms 134.42ms   91.90%
    Req/Sec     9.20k     1.25k   15.58k    78.15%
  1838302 requests in 20.10s, 278.75MB read
  Socket errors: connect 0, read 39, write 0, timeout 0
Requests/sec:  91454.62
Transfer/sec:     13.87MB

------------------------------
Falcon cython + gunicorn
--
Running 20s test @ http://localhost:8000/ping
  10 threads and 200 connections
  Thread Stats   Avg      Stdev     Max   +/- Stdev
    Latency    23.35ms   47.76ms 523.73ms   98.46%
    Req/Sec   608.04    290.92     1.06k    69.66%
  16387 requests in 20.09s, 2.42MB read
  Socket errors: connect 0, read 357, write 4, timeout 0
Requests/sec:    815.64
Transfer/sec:    123.46KB

------------------------------
Falcon cython + gunicorn + meinheld
--
Running 20s test @ http://localhost:8000/ping
  10 threads and 200 connections
  Thread Stats   Avg      Stdev     Max   +/- Stdev
    Latency     2.67ms    3.67ms 128.47ms   90.44%
    Req/Sec     9.70k     1.38k   14.16k    66.70%
  1931630 requests in 20.01s, 292.90MB read
  Socket errors: connect 0, read 44, write 0, timeout 0
Requests/sec:  96540.80
Transfer/sec:     14.64MB

------------------------------
Flask + gunicorn
--
Running 20s test @ http://localhost:8000/ping
  10 threads and 200 connections
  Thread Stats   Avg      Stdev     Max   +/- Stdev
    Latency    23.87ms   25.11ms 314.30ms   99.21%
    Req/Sec   521.89    193.32     0.88k    76.62%
  16528 requests in 20.06s, 2.57MB read
  Socket errors: connect 0, read 341, write 5, timeout 0
Requests/sec:    823.77
Transfer/sec:    131.13KB

------------------------------
Flask + gunicorn + meinheld
--
Running 20s test @ http://localhost:8000/ping
  10 threads and 200 connections
  Thread Stats   Avg      Stdev     Max   +/- Stdev
    Latency     8.20ms    4.24ms  46.84ms   75.44%
    Req/Sec     2.49k   565.34     3.98k    77.55%
  495311 requests in 20.02s, 78.89MB read
  Socket errors: connect 0, read 46, write 0, timeout 0
Requests/sec:  24746.28
Transfer/sec:      3.94MB

------------------------------
Flask + gunicorn + gevent
--
Running 20s test @ http://localhost:8000/ping
  10 threads and 200 connections
  Thread Stats   Avg      Stdev     Max   +/- Stdev
    Latency     1.49ms    2.73ms 104.37ms   97.91%
    Req/Sec     1.31k   553.40     3.36k    60.99%
  134625 requests in 20.08s, 21.57MB read
  Socket errors: connect 0, read 0, write 0, timeout 23
Requests/sec:   6703.45
Transfer/sec:      1.07MB

@CTimmerman
This is just old bench for myself.
I'm not tested with PyPy, just Cpython ( install using Homebrew )

I guest that your result only test with default webserver.
You can increase performance on production by using other webserver and other worker.

I'm still using gunicorn + meinheld worker for both flask and django on production

In Mint, this appears to work:

mkdir api_test && cd $_
apt install pypy3
pypy3 -m venv .venv_pypy3
source .venv_pypy3/bin/activate
pypy3 -m pip install fastapi uvicorn
nano fastapi_test.py

Paste:

import asyncio
import uvicorn
from fastapi import FastAPI
app = FastAPI()

@app.get('/')
async def root():
    # print('Sleeping for 10')
    # await asyncio.sleep(10)
    # print('Awake')
    return {'message': 'hello'}

if __name__ == "__main__":
    uvicorn.run(app, host="127.0.0.1", port=8000, workers=9)

Run:

pypy3 fastapi_test.py > /dev/null

Benchmark (2nd run):

$ siege --benchmark --time=60s --concurrent=20 http://127.0.0.1:8001
** SIEGE 4.0.4
** Preparing 20 concurrent users for battle.
The server is now under siege...
Lifting the server siege...
Transactions:		      164965 hits
Availability:		      100.00 %
Elapsed time:		       59.12 secs
Data transferred:	        2.99 MB
Response time:		        0.01 secs
Transaction rate:	     2790.34 trans/sec
Throughput:		        0.05 MB/sec
Concurrency:		       19.85
Successful transactions:      164965
Failed transactions:	           0
Longest transaction:	        0.11
Shortest transaction:	        0.00

Exit .venv_pypy3 and setup up Python3 version:

$ deactivate
$ apt install python3-venv
$ python3 -m venv .venv_python3
$ source .venv_python3/bin/activate
$ python3 -m pip install fastapi uvicorn
$ python3 fastapi_test.py > /dev/null

Benchmark (first run):

$ siege --benchmark --time=60s --concurrent=20 http://127.0.0.1:8001
** SIEGE 4.0.4
** Preparing 20 concurrent users for battle.
The server is now under siege...
Lifting the server siege...
Transactions:		      142478 hits
Availability:		      100.00 %
Elapsed time:		       59.38 secs
Data transferred:	        2.58 MB
Response time:		        0.01 secs
Transaction rate:	     2399.43 trans/sec
Throughput:		        0.04 MB/sec
Concurrency:		       19.89
Successful transactions:      142478
Failed transactions:	           0
Longest transaction:	        0.11
Shortest transaction:	        0.00

That's in a VirtualBox on my busy laptop.

Thanks for your bench.

I think difference environment, difference test tool will generate difference results.

Choose X over Y is hard. Feel free to choose. 😃

I think difference environment, difference test tool will generate difference results.

Of courses, which is why i did both PyPy and CPython to show that PyPy is faster in this case.

It would be supercool to re-run these benchmarks on Python 3.11 to see if its performance increases do anything for web frameworks.

For Python 3.11.3

% siege --benchmark --time=60s --concurrent=255 http://127.0.0.1:5009

and running uvicorn.run("app:app", host="127.0.0.1", port=5009, log_level="info", reload=False,workers=4)

Lifting the server siege...
Transactions: 33020 hits
Availability: 97.96 %
Elapsed time: 60.90 secs
Data transferred: 1.20 MB
Response time: 0.11 secs
Transaction rate: 542.20 trans/sec
Throughput: 0.02 MB/sec
Concurrency: 57.77
Successful transactions: 33020
Failed transactions: 687
Longest transaction: 19.52
Shortest transaction: 0.00

Thanks. How do these results compare with earlier versions of Python? Also vs. Python 3.12 release candidate 2?

• https://www.python.org/download/pre-releases