Use c pickler extension to communicate data to workers

benchmark.py

import time
import loky


def process_dict(x):
    return len(x)


if __name__ == "__main__":
    e = loky.get_reusable_executor(max_workers=2)
    large_dict = dict(zip(range(500000), range(500000)))

    t0 = time.time()
    e.submit(process_dict, large_dict).result()
    total_time = round(time.time() - t0, 3)
    print(f'total time: {total_time}s')

using current cloudpickle:

total time: 3.591 s

using a cloudpickle that extends the C pickler:

total time: 0.256 s

We observe a 15x speedup here, but there is an irreductible startup time of 0.15 seconds (time required to send a negligable amount of data). If we subtract his startup time, we are talking about a x30 speedup here.

Wow, this is zippy!