Last active
June 2, 2024 10:58
-
-
Save agronholm/1455506650e027b08ddf5e6639874499 to your computer and use it in GitHub Desktop.
TCP file receiver performance comparison sockets vs asyncio vs anyio
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
import hashlib | |
import time | |
import anyio | |
from anyio import create_tcp_listener | |
from anyio.abc import SocketAttribute, SocketStream | |
async def handle_connection(stream: SocketStream): | |
with stream: | |
sha1 = hashlib.sha1() | |
print(f"Accepted connection from {stream.extra(SocketAttribute.remote_address)}") | |
start = time.monotonic() | |
async for data in stream: | |
sha1.update(data) | |
print("Elapsed:", time.monotonic() - start, "SHA1:", sha1.hexdigest()) | |
async def main(): | |
listener = await create_tcp_listener(local_port=10000) | |
await listener.serve(handle_connection) | |
anyio.run(main) |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
import asyncio | |
import hashlib | |
import time | |
async def handle_connection(reader, writer): | |
sha1 = hashlib.sha1() | |
print(f"Accepted connection from {writer.get_extra_info('peername')}") | |
start = time.monotonic() | |
while True: | |
if not (data := await reader.read(4096)): | |
break | |
sha1.update(data) | |
print("Elapsed:", time.monotonic() - start, "SHA1:", sha1.hexdigest()) | |
writer.close() | |
async def main(): | |
async with await asyncio.start_server(handle_connection, host="127.0.0.1", port=10000) as server: | |
await server.serve_forever() | |
asyncio.run(main()) |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Receiving a 6.3 GB file over a local TCP socket - 5 consecutive runs with each implementation | |
The file is sent locally to port 10000 using `netcat` with no parameters beyond the host and port. | |
Then another round is run with `strace -c python <script.py>` to get system call statistics. | |
The completion time of this run does not factor into the performance statistics. | |
Results are after one round of warm-up to populate the file system cache. | |
Blocking sockets: | |
Run 1: 4.8206517059588805 | |
Run 2: 4.89901908300817 | |
Run 3: 4.938847267010715 | |
Run 4: 4.894235291983932 | |
Run 5: 4.7170228459872305 | |
Average: 4.853955238789785 seconds | |
System call statistics on the 6th run: | |
% time seconds usecs/call calls errors syscall | |
------ ----------- ----------- --------- --------- ---------------- | |
99,12 0,342806 3 106375 recvfrom | |
0,17 0,000603 3 180 17 newfstatat | |
Asyncio streams: | |
Run 1: 9.582709847018123 | |
Run 2: 9.6955241090036 | |
Run 3: 9.520388889010064 | |
Run 4: 9.404484694008715 | |
Run 5: 9.737935993995052 | |
Average: 9.58820870660711 seconds | |
System call statistics on the 6th run: | |
% time seconds usecs/call calls errors syscall | |
------ ----------- ----------- --------- --------- ---------------- | |
60,25 1,182386 42 27553 recvfrom | |
28,92 0,567464 7 76511 brk | |
5,82 0,114124 2 50622 epoll_ctl | |
4,79 0,094048 1 53697 1 epoll_wait | |
0,05 0,001054 4 221 read | |
0,04 0,000719 1 438 32 newfstatat | |
AnyIO streams on asyncio: | |
Run 1: 6.748573081975337 | |
Run 2: 6.034122936980566 | |
Run 3: 6.254155839997111 | |
Run 4: 5.921369992021937 | |
Run 5: 6.073068528989097 | |
Average: 6.206258076 seconds | |
System call statistics on the 6th run: | |
% time seconds usecs/call calls errors syscall | |
------ ----------- ----------- --------- --------- ---------------- | |
81,98 1,187386 23 50525 1 recvfrom | |
9,10 0,131764 9 13622 brk | |
8,43 0,122147 0 126844 1 epoll_wait | |
0,12 0,001708 2 695 56 newfstatat | |
Trio streams: | |
Run 1: 6.363744358008262 | |
Run 2: 6.337662931997329 | |
Run 3: 6.294331677025184 | |
Run 4: 6.304128275951371 | |
Run 5: 6.274448422016576 | |
Average: 6.314863133 seconds | |
System call statistics on the 6th run: | |
% time seconds usecs/call calls errors syscall | |
------ ----------- ----------- --------- --------- ---------------- | |
71,54 0,243716 2 102465 recvfrom | |
25,22 0,085929 0 102475 epoll_wait | |
0,60 0,002059 2 821 56 newfstatat |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
import hashlib | |
import socket | |
import time | |
sock = socket.socket(socket.AF_INET, socket.SOCK_STREAM) | |
sock.bind(('127.0.0.1', 10000)) | |
sock.listen(5) | |
while True: | |
client, addr = sock.accept() | |
with client: | |
sha1 = hashlib.sha1() | |
print(f"Accepted connection from {addr}") | |
start = time.monotonic() | |
while True: | |
if not (data := client.recv(4096)): | |
break | |
sha1.update(data) | |
print("Elapsed:", time.monotonic() - start, "SHA1:", sha1.hexdigest()) |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
import hashlib | |
import time | |
import trio | |
from trio import SocketStream | |
async def handle_connection(stream: SocketStream): | |
sha1 = hashlib.sha1() | |
print(f"Accepted connection from {stream.socket.getpeername()}") | |
start = time.monotonic() | |
async for data in stream: | |
sha1.update(data) | |
print("Elapsed:", time.monotonic() - start, "SHA1:", sha1.hexdigest()) | |
async def main(): | |
await trio.serve_tcp(handle_connection, 10000) | |
trio.run(main) |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment