tornado同步转异步几种方式

Tornado是一个异步非堵塞的web框架，但是在遇到数据库的io操作就会发现数据库的io操作是同步执行的，印象性能。

如不太了解tornado，可阅读下这篇文章 Tornado入门这一篇足以

可以三种方法可以使这个同步执行变成异步：

• 线程
• 进程
• 中间件（MQ、redis等）

编程时遇到的阻塞任务一般有两类：

• 等待 I/O 就绪（I/O 密集型），这种场景使用ThreadPoolExecutor
• 耗时的计算工作（CPU 密集型），这种场景使用ProcessPoolExecutor

本文对应的源码

1.线程

1-1.装饰器方式一

• @run_on_executor和ThreadPoolExecutor配合使用定义同步代码
• async和 await配合调用装饰后的方法

import time
from concurrent.futures.thread import ThreadPoolExecutor

from tornado import web, ioloop
from tornado.concurrent import run_on_executor


class SyncToAsyncThreadHandler(web.RequestHandler):

    executor = ThreadPoolExecutor(max_workers=2)

    @run_on_executor
    def sleep(self):
        print("休息1...start")
        time.sleep(5)
        print("休息1...end")
        return 'ok'

    async def get(self):
        res = await self.sleep()
        self.write(res)

url_map = [
    ("/?", SyncToAsyncThreadHandler)
]

if __name__ == '__main__':
    app = web.Application(url_map, debug=True)
    app.listen(8888)
    print('started...')
    ioloop.IOLoop.current().start()

1-2.装饰器方式二

• @run_on_executor和ThreadPoolExecutor配合使用定义同步代码
• @gen.coroutine和 yield 配合调用装饰后的方法

import time
from concurrent.futures.thread import ThreadPoolExecutor

from tornado import gen, web, ioloop
from tornado.concurrent import run_on_executor


class SyncToAsyncThreadHandler(web.RequestHandler):

    executor = ThreadPoolExecutor(max_workers=2)

    @run_on_executor
    def sleep(self):
        print("休息1...start")
        time.sleep(5)
        print("休息1...end")
        return 'ok'

    @gen.coroutine
    def get(self):
        res = yield self.sleep()
        self.write(res)

url_map = [
    ("/?", SyncToAsyncThreadHandler)
]

if __name__ == '__main__':
    app = web.Application(url_map, debug=True)
    app.listen(8888)
    print('started...')
    ioloop.IOLoop.current().start()

1-3.submit方式

可以使用executor.submit来提交函数

executor除了submit外，还有map方法,它们的返回值是Future，或者对Futureadd_done_callback的方法回调方式处理

import time
from concurrent.futures.thread import ThreadPoolExecutor

from tornado import gen, web, ioloop

executor = ThreadPoolExecutor(max_workers=2)

class SyncToAsyncThreadHandler(web.RequestHandler):

    def sleep(self):
        print("休息1...start")
        time.sleep(5)
        print("休息1...end")
        return 'ok'

    @gen.coroutine
    def get(self):
        res = yield executor.submit(self.sleep)
        self.write(res)

url_map = [
    ("/?", SyncToAsyncThreadHandler)
]

if __name__ == '__main__':
    app = web.Application(url_map, debug=True)
    app.listen(8888)
    print('started...')
    ioloop.IOLoop.current().start()

这executor.submit使用async和await方式无效

1-4.ioloop方式一

• 使用ioloop的IOLoop的run_in_executor函数，指定传入executor和要调用的同步函数
• 用@gen.coroutine和yield装饰成异步函数

import time
from concurrent.futures.thread import ThreadPoolExecutor

from tornado import gen, web, ioloop

executor = ThreadPoolExecutor(max_workers=2)

class SyncToAsyncThreadHandler(web.RequestHandler):

    def sleep(self):
        print("休息1...start")
        time.sleep(5)
        print("休息1...end")
        return 'ok'

    @gen.coroutine
    def get(self):
        rest = yield ioloop.IOLoop.current().run_in_executor(executor, self.sleep)
        self.write(rest)

url_map = [
    ("/?", SyncToAsyncThreadHandler)
]

if __name__ == '__main__':
    app = web.Application(url_map, debug=True)
    app.listen(8888)
    print('started...')
    ioloop.IOLoop.current().start()

1-5.ioloop方式二

• 跟ioloop方式一类似，只不过用async和await方式

import time
from concurrent.futures.thread import ThreadPoolExecutor

from tornado import gen, web, ioloop

executor = ThreadPoolExecutor(max_workers=2)

class SyncToAsyncThreadHandler(web.RequestHandler):

    def sleep(self):
        print("休息1...start")
        time.sleep(5)
        print("休息1...end")
        return 'ok'

    async def get(self):
        rest = await ioloop.IOLoop.current().run_in_executor(executor, self.sleep)
        self.write(rest)

url_map = [
    ("/?", SyncToAsyncThreadHandler)
]

if __name__ == '__main__':
    app = web.Application(url_map, debug=True)
    app.listen(8888)
    print('started...')
    ioloop.IOLoop.current().start()

2.进程

2-1.submit方式

import time
from concurrent.futures.process import ProcessPoolExecutor

from tornado import gen, web, ioloop

executor = ProcessPoolExecutor(max_workers=2)


def sleep():
    print("休息...start")
    time.sleep(5)
    print("休息...end")
    return 'ok'


class SyncToAsyncProcessHandler(web.RequestHandler):

    @gen.coroutine
    def get(self):
        rest = yield executor.submit(sleep)
        self.write(rest)


url_map = [
    ("/?", SyncToAsyncProcessHandler)
]

if __name__ == '__main__':
    app = web.Application(url_map, debug=True)
    app.listen(8888)
    print('started...')
    ioloop.IOLoop.current().start()

2-2.ioloop方式

import time
from concurrent.futures.process import ProcessPoolExecutor

from tornado import gen, web, ioloop

executor = ProcessPoolExecutor(max_workers=2)


def sleep():
    print("休息...start")
    time.sleep(5)
    print("休息...end")
    return 'ok'


class SyncToAsyncProcessHandler(web.RequestHandler):

    async def get(self):
        rest = await ioloop.IOLoop.current().run_in_executor(executor, sleep)
        self.write(rest)


url_map = [
    ("/?", SyncToAsyncProcessHandler)
]

if __name__ == '__main__':
    app = web.Application(url_map, debug=True)
    app.listen(8888)
    print('started...')
    ioloop.IOLoop.current().start()

3.中间件

• tornado-celery

基于AMQP and Redis实现的，所以要安装相关软件，有兴趣可以去阅读下说明，很简单

4.网站

本文对应的源码

• iworkh博客

个人博客刚开不久，主要用来辅助手册，写些零碎的知识点

• iworkh网站

注册下个人用户，就可以管理自己的书签/链接、享用各类学习手册，主要用来写手册，分享学习。