Skip to content

Instantly share code, notes, and snippets.

@zonca
Created June 16, 2012 17:54
Show Gist options
  • Save zonca/2942088 to your computer and use it in GitHub Desktop.
Save zonca/2942088 to your computer and use it in GitHub Desktop.
Spawn process to write in background
from multiprocessing import Pool
import numpy as np
from time import sleep
import logging as l
l.basicConfig(format = "%(asctime)-15s [Process: %(process)s] %(message)s", level=l.DEBUG)
def write_your_array(filename, a):
l.info("writing %s in background process" % filename)
# implement writing
sleep(10)
l.info("%s written" % filename)
if __name__ == "__main__":
# just need to have 1 background process
WritePool = Pool(1)
l.info("The main program started")
your_array = np.zeros(10)
WritePool.apply_async(write_your_array, ("file1.txt", your_array.copy()))
l.info("I am doing more computation in the main program (1)")
# queue up another
your_other_array = np.zeros(10)
WritePool.apply_async(write_your_array, ("file2.txt", your_other_array.copy()))
l.info("I am doing more computation in the main program (2)")
l.info("I am done in the main program, I wait for the writing to complete")
WritePool.close()
WritePool.join()
@zonca
Copy link
Author

zonca commented Jun 16, 2012

Example output on my machine:
run write_background.py
2012-06-16 11:59:18,948 [Process: 8988] The main program started
2012-06-16 11:59:18,949 [Process: 8988] I am doing more computation in the main program (1)
2012-06-16 11:59:18,951 [Process: 9052] writing file1.txt in background process
2012-06-16 11:59:18,952 [Process: 8988] I am doing more computation in the main program (2)
2012-06-16 11:59:18,952 [Process: 8988] I am done in the main program, I wait for the writing to complete
2012-06-16 11:59:28,956 [Process: 9052] file1.txt written
2012-06-16 11:59:28,957 [Process: 9052] writing file2.txt in background process
2012-06-16 11:59:38,964 [Process: 9052] file2.txt written

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment