Python Multiprocessing

README.md

threading and multiprocessing are the modules that provide concurency facility in python. Both provide different utility. Google the difference to know more. But basically threading provides multiple threads (in a single process-context). This can you computing teeth while your data gets loaded. Although multiple threads can be started with this but only 1 thread can be active at a time. Other threads can be sleeping or loading data. Thus, this is not suitable when your threads need to all do number crunching (CPU-bound).

multiprocessing lets you create separate processes. These processes have different memory context. Data needs to be shared with semaphores. Easy way share data is using the Manager class and Manager.list() and Manager.dict() or the multiprocessing.Queues().

multiprocessing.py

## Two process prouduce numpy.arrays. These arrays are put in Queues to be consumed
#by another process. Also these arrays are put in a list for later reference in main. 
#Another process consumes the arrays to compute their Eigen values. 


from multiprocessing import Process, Queue, Manager
#from queue import Queue #dont use this. 
import time
import numpy as np
import os

def producer(queue, S_ ):
    for i in range(10):
        # if i%10 == 0:
        print '[producer-%d]' %(os.getpid()), i
        m = np.random.random( (1000,1000) )
        queue.put(m)
        S_.append( m )
        time.sleep( 1 )

def consumer(queue):
    print '[consumer]', 'sleeping for 10sec'
    time.sleep( 5 )

    while queue.qsize() > 0:

        print '[consumer-%d]' %(os.getpid()), 'get()', 'current qsize=%d' %(queue.qsize() )
        msg = queue.get()
        print '[consumer-%d]' %(os.getpid()), 'eig()'
        e = np.linalg.eig( msg )

if __name__ == "__main__":
    print 'MAIN PID', os.getpid()

    manager = Manager()
    q = Queue()
    S_ = manager.list()
    

    producer_p = Process( target=producer, args=( q,S_ )   )
    producer_p2 = Process( target=producer, args=( q,S_ )  )
    producer_p.start()
    producer_p2.start()


    consumer_p = Process( target=consumer, args=( q, )   )
    consumer_p2 = Process( target=consumer, args=( q, )  )

    consumer_p.start()
    consumer_p2.start()

    consumer_p.join()
    consumer_p2.join()
    producer_p.join()
    print 'Main Done!'