Skip to content

Instantly share code, notes, and snippets.

@mittenchops
Last active August 29, 2015 13:58
Show Gist options
  • Star 0 You must be signed in to star a gist
  • Fork 0 You must be signed in to fork a gist
  • Save mittenchops/9962536 to your computer and use it in GitHub Desktop.
Save mittenchops/9962536 to your computer and use it in GitHub Desktop.
There are other ways of doing this, this is the one I used for where I was working with this for something that parallelized easily.
from __future__ import print_function
import numpy
import random
mylist = range(0,100)
random.shuffle(mylist)
myarray = numpy.array(mylist)
# via http://stackoverflow.com/questions/2130016/splitting-a-list-of-arbitrary-size-into-only-roughly-n-equal-parts
def chunkit(seq, N):
"""
Prints the bounds of a list divided into at most N chunks.
This is simple but useful in the case of spawning lots of workers from the
command line, where you need to process (0, x), (x+1, 2x+1), etc. records at a time.
and where you're going to have some irregular divisions.
"""
avg = len(seq) / float(N)
out = []
last = 0.0
while last < len(seq):
out.append((int(last), min(int(last + avg), len(seq))))
last += (avg + 1)
return(out)
f = lambda x: print("f --lower {} --higher {}".format(x[0],x[1]))
bounds = chunkit(mylist, 6)
print(bounds)
for b in bounds:
f(b)
print("and with map")
map(f,bounds)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment