Skip to content

Instantly share code, notes, and snippets.

@simonrw
Created January 11, 2017 22:48
Show Gist options
  • Save simonrw/a91e3e3f2e8d151b17e931850c2664ef to your computer and use it in GitHub Desktop.
Save simonrw/a91e3e3f2e8d151b17e931850c2664ef to your computer and use it in GitHub Desktop.
Display the source blob
Display the rendered blob
Raw
{
"cells": [
{
"cell_type": "markdown",
"metadata": {},
"source": [
"# Parallel processing example\n",
"\n",
"This notebook accompanies [this blog post][1], and includes an example of parallel code created with the `multiprocessing` module.\n",
"\n",
"First import the required functions from the standard library:\n",
"\n",
"[1]: https://mindriot101.github.io/blog/2017/01/10/python-parallelism-cheat-sheet/"
]
},
{
"cell_type": "code",
"execution_count": 2,
"metadata": {
"collapsed": true
},
"outputs": [],
"source": [
"# Bake in constant arguments to a function\n",
"from functools import partial\n",
"\n",
"# Package for running code in parallel \n",
"from multiprocessing import Pool"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Then we define our \"worker\" function, which performs the work of a single loop iteration."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"collapsed": true
},
"outputs": [],
"source": [
"def worker_fn(changing_stuff, const_value):\n",
" '''A function that takes as it's first parameter\n",
" the values that change for each loop iteration, and\n",
" the remaining parameters do not change with each\n",
" loop iteration\n",
" '''\n",
" a, b = changing_stuff\n",
" return (a + b) * const_value"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Set up an example problem to work with"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"collapsed": true
},
"outputs": [],
"source": [
"avalues = range(20)\n",
"bvalues = range(100, 120)\n",
"const = 100"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"\"Bake in\" our constant arguments to create our partially applied function, which takes a single parameter."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"collapsed": true
},
"outputs": [],
"source": [
"fn = partial(worker_fn, const_value=const)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Create an iterable of everything that changes for every loop iteration"
]
},
{
"cell_type": "code",
"execution_count": 3,
"metadata": {
"collapsed": true
},
"outputs": [],
"source": [
"# this must be an `iterable`, so a list or generator\n",
"zipped_args = zip(avalues, bvalues)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Finally run the code in parallel, using all of the processors on the machine"
]
},
{
"cell_type": "code",
"execution_count": 1,
"metadata": {
"collapsed": true
},
"outputs": [],
"source": [
"# Python 3\n",
"with Pool() as pool: # 3\n",
" results = pool.map(fn, zipped_args) # 4\n",
"\n",
"# or if you're stuck with Python 2:\n",
"# pool = Pool()\n",
"# results = pool.map(fn, zipped_args) # 4"
]
}
],
"metadata": {
"kernelspec": {
"display_name": "Python 3",
"language": "python",
"name": "python3"
},
"language_info": {
"codemirror_mode": {
"name": "ipython",
"version": 3
},
"file_extension": ".py",
"mimetype": "text/x-python",
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.5.1"
}
},
"nbformat": 4,
"nbformat_minor": 0
}
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment