PyCon JP 2024 Talk Notes - Unlocking the Parallel Universe: Sub Interpreters and Free-Threading in Python 3.13
- PyCon 2023 – Eric Snow talk on sub interpreters
- EuroPython 2022 – Sam Gross talk on free-threading
- PyCon 2024 - “Sync vs Async in Python”
- PyCon 2024 - Building a JIT compiler for Cpython
- PyCon 2024 – Overcoming GIL with sub interpreters and immutability
- “Parallelism and Concurrency” chapter from CPython Internals)
- My Masters Thesis doi.org/10.25949/23974764.v1
Model | Execution | Start-up time | Data Exchange | Best for… |
---|---|---|---|---|
threads | Parallel * | small | Any | Small, IO-bound tasks that don’t require multiple CPU cores |
coroutines | Concurrent | smallest | Any | Small, IO-bound tasks that don’t require multiple CPU cores |
multiprocessing | Parallel | large | Serialization | Larger, CPU or IO-bound tasks that require multiple CPU cores |
Sub Interpreters | Parallel | medium** | Serialization or Shared Memory | Larger, CPU or IO-bound tasks that require multiple CPU cores |
Crude sample:
import numpy
# Create a random array of 100,000 integers
a = numpy.random.randint(0, 100, 100_000)
for x in a:
abs(x - 50)
Benchmark code to get the "2x slower" figure:
import numpy
import threading
# Create a random array of 100,000 integers between 0 and 100
a = numpy.random.randint(0, 100, 100_000)
def simple_abs_range(vec):
for x in vec:
abs(x - 50)
def f_linear():
# Calculate the distance for each value to 50
simple_abs_range(a)
def f_threaded():
threads = []
# Split array into blocks of 100 and start a thread for each
for ar in numpy.split(a, 100):
t = threading.Thread(target=simple_abs_range, args=(ar,))
t.start()
threads.append(t)
for t in threads:
t.join()
The Jupyter Notebook for this sample is here.
The demo code is here