Skip to content

Instantly share code, notes, and snippets.

View rishibarve's full-sized avatar

Rishi Barve rishibarve

  • Hike Limited
  • Bangalore
View GitHub Profile
@rishibarve
rishibarve / Multiprocessing Python With Monitoring
Created August 30, 2018 06:54
Code for multiprocessing python with feature to save work in middle to skip in case of failure. Helpful for processing large number of small files in a environment where failures are common.
import multiprocessing
# split a list into evenly sized chunks
def chunks(l, n):
return [l[i:i+n] for i in range(0, len(l), n)]
def do_job(job_id, data_slice):
processed = []
for i, item in enumerate(data_slice):