Tornado Example: Delegating an blocking task to a worker thread pool from an asynchronous request handler

gistfile1.py

from time import sleep
from tornado.httpserver import HTTPServer
from tornado.ioloop import IOLoop
from tornado.web import Application, asynchronous, RequestHandler
from multiprocessing.pool import ThreadPool

_workers = ThreadPool(10)

def run_background(func, callback, args=(), kwds={}):
    def _callback(result):
        IOLoop.instance().add_callback(lambda: callback(result))
    _workers.apply_async(func, args, kwds, _callback)

# blocking task like querying to MySQL
def blocking_task(n):
    sleep(n)
    return n

class Handler(RequestHandler):
    @asynchronous
    def get(self):
        run_background(blocking_task, self.on_complete, (10,))

    def on_complete(self, res):
        self.write("Test {0}<br/>".format(res))
        self.finish()

HTTPServer(Application([("/", Handler)],debug=True)).listen(8888)
IOLoop.instance().start()

Note for other travellers - ThreadPoolExecutor works with @run_on_executor but ProcessPoolExecutor does not, because the calling class is not serializeable as required by ProcessPoolExecutor. This confused me for a while.

seems like there are some changes in Tornado 5+ related to Futures, so decorator above didn't work for me. So I've modified it to:

def blocking(method):
    """
    Wraps the method in an async method, and executes the function on `self.executor`
    """

    @wraps(method)
    async def wrapper(self, *args, **kwargs):
        def work():
            return method(self, *args, **kwargs)
        return await io_loop.run_in_executor(self.executor, work)

    return wrapper