IO bound thread performance in Perl and Python.

io-bound.md

-- Back to Index --

---

In this post I am doing IO bound performance comparison.

First in Perl:

[Source Code]

#!/usr/bin/env perl

use v5.38;
use threads;
use LWP::Simple;
use Time::HiRes qw(time);

sub io_worker($id) {
    my $url = "https://www.google.com";
    my $content = get($url);
    say "Thread $id finished.";
}

my $start_time = time();
my @threads = map { threads->create(\&io_worker, $_) } (1..9);
$_->join for @threads;
my $end_time = time();
printf("Total time taken: %.2f seconds\n", $end_time - $start_time);

Run the code now:

$ perl io-bound.pl
Thread 1 finished.
Thread 2 finished.
Thread 3 finished.
Thread 4 finished.
Thread 5 finished.
Thread 6 finished.
Thread 7 finished.
Thread 8 finished.
Thread 9 finished.
Total time taken: 0.34 seconds
$

Using HTTP::Tiny as suggested by @leont in the comments below:

[Source Code]

#!/usr/bin/env perl

use v5.38;
use threads;
use HTTP::Tiny;
use Time::HiRes qw(time);

sub io_worker($id) {
    my $url = "https://www.google.com";
    my $content = HTTP::Tiny->new->get($url);
    say "Thread $id finished.";
}

my $start_time = time();
my @threads = map { threads->create(\&io_worker, $_) } (1..9);
$_->join for @threads;
my $end_time = time();
printf("Total time taken: %.2f seconds\n", $end_time - $start_time);

Running the updated code now:

$ perl io-bound.pl
Thread 2 finished.
Thread 1 finished.
Thread 3 finished.
Thread 5 finished.
Thread 4 finished.
Thread 7 finished.
Thread 9 finished.
Thread 6 finished.
Thread 8 finished.
Total time taken: 0.30 seconds    $
$

Let's do the same in Python now:

[Source Code]

#!/usr/bin/env python3

import time
import requests
import threading

def io_worker(id):
    url = "https://www.google.com"
    response = requests.get(url)
    print(f"Thread {id} finished.")

start_time = time.time()
threads = []
for i in range(9):
    thread = threading.Thread(target=io_worker, args=(i + 1,))
    threads.append(thread)
    thread.start()
for thread in threads:
    thread.join()
end_time = time.time()
print(f"Total time taken: {end_time - start_time:.2f} seconds")

Run the code now:

$ py io-bound.py
Thread 3 finished.
Thread 8 finished.
Thread 2 finished.
Thread 7 finished.
Thread 6 finished.
Thread 5 finished.
Thread 4 finished.
Thread 1 finished.
Thread 9 finished.
Total time taken: 0.24 seconds
$

CONCLUSION: With the result above, Python with 0.24 seconds is a clear winner against Perl with 0.30 seconds using HTTP::Tiny and 0.34 seconds using LWP::Simple.

-- Back to Index --

---

Interestingly, if I run these with the time command, I get different results:

$ time perl io-bound.pl
...
Total time taken: 0.25 seconds
perl io-bound.pl  0.43s user 0.06s system 163% cpu 0.299 total

$ time python io-bound.py
...
Total time taken: 0.14 seconds
python io-bound.py  0.09s user 0.02s system 47% cpu 0.235 total

So while the in-process timing seems to favour Python, the interpreter overhead (either in starting or exiting phase, or both), pulls them much closer together.

@sbakker This is where it gets messy. I don't look too much into it.

This sort of use-case is much faster if you use HTTP::Tiny, because yes the thread startup is really slow.

@Leont, you are right, I have updated the code.

Actually, forking is faster than threading in Perl...

#!/usr/bin/env perl

use v5.38;
use POSIX ':sys_wait_h';
use HTTP::Tiny;
use Time::HiRes qw(time);

my $ua = HTTP::Tiny->new();

sub io_worker($id) {
    my $url = "https://www.google.com";
    my $response = $ua->get($url);
    say "Worker $id finished.";
}

my $start_time = time();
for my $id (1..5) {
    my $pid = fork;
    if ($pid < 0) {
        die "cannot fork: $!";
    }
    elsif ($pid == 0) {
        io_worker($id);
        exit(0);
    }
}

while (waitpid(-1, 0) > 0) {}

my $end_time = time();
printf("Total time taken: %.2f seconds\n", $end_time - $start_time);

This gives me:

$ time perl io-bound5.pl
Worker 5 finished.
Worker 1 finished.
Worker 4 finished.
Worker 6 finished.
Worker 7 finished.
Worker 2 finished.
Worker 3 finished.
Worker 8 finished.
Worker 9 finished.
Total time taken: 0.20 seconds
perl io-bound5.pl  0.17s user 0.05s system 103% cpu 0.218 total

$ time python io-bound.py
Thread 7 finished.
Thread 2 finished.
Thread 6 finished.
Thread 8 finished.
Thread 5 finished.
Thread 3 finished.
Thread 1 finished.
Thread 4 finished.
Thread 9 finished.
Total time taken: 0.13 seconds
python io-bound.py  0.10s user 0.02s system 50% cpu 0.233 total

Of course, there's variability here because of the response time of the Google web server and network latency.

And note that Parallel::ForkManager works, but the wait_all_children always waits for at least one second. 🤷