Skip to content

Instantly share code, notes, and snippets.

@MrQianJinSi
Last active June 12, 2019 08:29
Show Gist options
  • Star 0 You must be signed in to star a gist
  • Fork 0 You must be signed in to fork a gist
  • Save MrQianJinSi/2daf5b6a9ef08b00facdfbea5200dd28 to your computer and use it in GitHub Desktop.
Save MrQianJinSi/2daf5b6a9ef08b00facdfbea5200dd28 to your computer and use it in GitHub Desktop.
do cleanup work when sub-process exit
import multiprocessing as mp
from multiprocessing.util import Finalize
import os
import time
def finalizer():
time.sleep(0.2) # some time consuming work
print('do cleanup work: {}'.format(os.getpid()))
def worker(_):
print('do some work: {}'.format(os.getpid()))
def initializer():
# ref: https://github.com/python/cpython/blob/master/Lib/multiprocessing/util.py#L147
Finalize(None, finalizer, exitpriority=1)
# atexit module don't work along with multiprocessing module
# because sub-process exit via os._exit()
# ref: https://docs.python.org/3/library/atexit.html
# atexit.register(finalizer) # don't work
print('main process ID: {}'.format(os.getpid()))
with mp.Pool(4, initializer=initializer) as executor:
executor.map(worker, range(20))
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment