Skip to content

Instantly share code, notes, and snippets.

@agronholm
Last active June 2, 2024 10:58
Show Gist options
  • Save agronholm/1455506650e027b08ddf5e6639874499 to your computer and use it in GitHub Desktop.
Save agronholm/1455506650e027b08ddf5e6639874499 to your computer and use it in GitHub Desktop.
TCP file receiver performance comparison sockets vs asyncio vs anyio
import hashlib
import time
import anyio
from anyio import create_tcp_listener
from anyio.abc import SocketAttribute, SocketStream
async def handle_connection(stream: SocketStream):
with stream:
sha1 = hashlib.sha1()
print(f"Accepted connection from {stream.extra(SocketAttribute.remote_address)}")
start = time.monotonic()
async for data in stream:
sha1.update(data)
print("Elapsed:", time.monotonic() - start, "SHA1:", sha1.hexdigest())
async def main():
listener = await create_tcp_listener(local_port=10000)
await listener.serve(handle_connection)
anyio.run(main)
import asyncio
import hashlib
import time
async def handle_connection(reader, writer):
sha1 = hashlib.sha1()
print(f"Accepted connection from {writer.get_extra_info('peername')}")
start = time.monotonic()
while True:
if not (data := await reader.read(4096)):
break
sha1.update(data)
print("Elapsed:", time.monotonic() - start, "SHA1:", sha1.hexdigest())
writer.close()
async def main():
async with await asyncio.start_server(handle_connection, host="127.0.0.1", port=10000) as server:
await server.serve_forever()
asyncio.run(main())
Receiving a 6.3 GB file over a local TCP socket - 5 consecutive runs with each implementation
The file is sent locally to port 10000 using `netcat` with no parameters beyond the host and port.
Then another round is run with `strace -c python <script.py>` to get system call statistics.
The completion time of this run does not factor into the performance statistics.
Results are after one round of warm-up to populate the file system cache.
Blocking sockets:
Run 1: 4.8206517059588805
Run 2: 4.89901908300817
Run 3: 4.938847267010715
Run 4: 4.894235291983932
Run 5: 4.7170228459872305
Average: 4.853955238789785 seconds
System call statistics on the 6th run:
% time seconds usecs/call calls errors syscall
------ ----------- ----------- --------- --------- ----------------
99,12 0,342806 3 106375 recvfrom
0,17 0,000603 3 180 17 newfstatat
Asyncio streams:
Run 1: 9.582709847018123
Run 2: 9.6955241090036
Run 3: 9.520388889010064
Run 4: 9.404484694008715
Run 5: 9.737935993995052
Average: 9.58820870660711 seconds
System call statistics on the 6th run:
% time seconds usecs/call calls errors syscall
------ ----------- ----------- --------- --------- ----------------
60,25 1,182386 42 27553 recvfrom
28,92 0,567464 7 76511 brk
5,82 0,114124 2 50622 epoll_ctl
4,79 0,094048 1 53697 1 epoll_wait
0,05 0,001054 4 221 read
0,04 0,000719 1 438 32 newfstatat
AnyIO streams on asyncio:
Run 1: 6.748573081975337
Run 2: 6.034122936980566
Run 3: 6.254155839997111
Run 4: 5.921369992021937
Run 5: 6.073068528989097
Average: 6.206258076 seconds
System call statistics on the 6th run:
% time seconds usecs/call calls errors syscall
------ ----------- ----------- --------- --------- ----------------
81,98 1,187386 23 50525 1 recvfrom
9,10 0,131764 9 13622 brk
8,43 0,122147 0 126844 1 epoll_wait
0,12 0,001708 2 695 56 newfstatat
Trio streams:
Run 1: 6.363744358008262
Run 2: 6.337662931997329
Run 3: 6.294331677025184
Run 4: 6.304128275951371
Run 5: 6.274448422016576
Average: 6.314863133 seconds
System call statistics on the 6th run:
% time seconds usecs/call calls errors syscall
------ ----------- ----------- --------- --------- ----------------
71,54 0,243716 2 102465 recvfrom
25,22 0,085929 0 102475 epoll_wait
0,60 0,002059 2 821 56 newfstatat
import hashlib
import socket
import time
sock = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
sock.bind(('127.0.0.1', 10000))
sock.listen(5)
while True:
client, addr = sock.accept()
with client:
sha1 = hashlib.sha1()
print(f"Accepted connection from {addr}")
start = time.monotonic()
while True:
if not (data := client.recv(4096)):
break
sha1.update(data)
print("Elapsed:", time.monotonic() - start, "SHA1:", sha1.hexdigest())
import hashlib
import time
import trio
from trio import SocketStream
async def handle_connection(stream: SocketStream):
sha1 = hashlib.sha1()
print(f"Accepted connection from {stream.socket.getpeername()}")
start = time.monotonic()
async for data in stream:
sha1.update(data)
print("Elapsed:", time.monotonic() - start, "SHA1:", sha1.hexdigest())
async def main():
await trio.serve_tcp(handle_connection, 10000)
trio.run(main)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment