tornado同步轉異步幾種方式

Tornado是一個異步非堵塞的web框架，但是在遇到數據庫的io操作就會發現數據庫的io操作是同步執行的，印象性能。

如不太瞭解tornado，可閱讀下這篇文章 Tornado入門這一篇足以

可以三種方法可以使這個同步執行變成異步：

• 線程
• 進程
• 中間件（MQ、redis等）

編程時遇到的阻塞任務一般有兩類：

• 等待 I/O 就緒（I/O 密集型），這種場景使用ThreadPoolExecutor
• 耗時的計算工作（CPU 密集型），這種場景使用ProcessPoolExecutor

本文對應的源碼

1.線程

1-1.裝飾器方式一

• @run_on_executor和ThreadPoolExecutor配合使用定義同步代碼
• async和 await配合調用裝飾後的方法

import time
from concurrent.futures.thread import ThreadPoolExecutor

from tornado import web, ioloop
from tornado.concurrent import run_on_executor


class SyncToAsyncThreadHandler(web.RequestHandler):

    executor = ThreadPoolExecutor(max_workers=2)

    @run_on_executor
    def sleep(self):
        print("休息1...start")
        time.sleep(5)
        print("休息1...end")
        return 'ok'

    async def get(self):
        res = await self.sleep()
        self.write(res)

url_map = [
    ("/?", SyncToAsyncThreadHandler)
]

if __name__ == '__main__':
    app = web.Application(url_map, debug=True)
    app.listen(8888)
    print('started...')
    ioloop.IOLoop.current().start()

1-2.裝飾器方式二

• @run_on_executor和ThreadPoolExecutor配合使用定義同步代碼
• @gen.coroutine和 yield 配合調用裝飾後的方法

import time
from concurrent.futures.thread import ThreadPoolExecutor

from tornado import gen, web, ioloop
from tornado.concurrent import run_on_executor


class SyncToAsyncThreadHandler(web.RequestHandler):

    executor = ThreadPoolExecutor(max_workers=2)

    @run_on_executor
    def sleep(self):
        print("休息1...start")
        time.sleep(5)
        print("休息1...end")
        return 'ok'

    @gen.coroutine
    def get(self):
        res = yield self.sleep()
        self.write(res)

url_map = [
    ("/?", SyncToAsyncThreadHandler)
]

if __name__ == '__main__':
    app = web.Application(url_map, debug=True)
    app.listen(8888)
    print('started...')
    ioloop.IOLoop.current().start()

1-3.submit方式

可以使用executor.submit來提交函數

executor除了submit外，還有map方法,它們的返回值是Future，或者對Futureadd_done_callback的方法回調方式處理

import time
from concurrent.futures.thread import ThreadPoolExecutor

from tornado import gen, web, ioloop

executor = ThreadPoolExecutor(max_workers=2)

class SyncToAsyncThreadHandler(web.RequestHandler):

    def sleep(self):
        print("休息1...start")
        time.sleep(5)
        print("休息1...end")
        return 'ok'

    @gen.coroutine
    def get(self):
        res = yield executor.submit(self.sleep)
        self.write(res)

url_map = [
    ("/?", SyncToAsyncThreadHandler)
]

if __name__ == '__main__':
    app = web.Application(url_map, debug=True)
    app.listen(8888)
    print('started...')
    ioloop.IOLoop.current().start()

這executor.submit使用async和await方式無效

1-4.ioloop方式一

• 使用ioloop的IOLoop的run_in_executor函數，指定傳入executor和要調用的同步函數
• 用@gen.coroutine和yield裝飾成異步函數

import time
from concurrent.futures.thread import ThreadPoolExecutor

from tornado import gen, web, ioloop

executor = ThreadPoolExecutor(max_workers=2)

class SyncToAsyncThreadHandler(web.RequestHandler):

    def sleep(self):
        print("休息1...start")
        time.sleep(5)
        print("休息1...end")
        return 'ok'

    @gen.coroutine
    def get(self):
        rest = yield ioloop.IOLoop.current().run_in_executor(executor, self.sleep)
        self.write(rest)

url_map = [
    ("/?", SyncToAsyncThreadHandler)
]

if __name__ == '__main__':
    app = web.Application(url_map, debug=True)
    app.listen(8888)
    print('started...')
    ioloop.IOLoop.current().start()

1-5.ioloop方式二

• 跟ioloop方式一類似，只不過用async和await方式

import time
from concurrent.futures.thread import ThreadPoolExecutor

from tornado import gen, web, ioloop

executor = ThreadPoolExecutor(max_workers=2)

class SyncToAsyncThreadHandler(web.RequestHandler):

    def sleep(self):
        print("休息1...start")
        time.sleep(5)
        print("休息1...end")
        return 'ok'

    async def get(self):
        rest = await ioloop.IOLoop.current().run_in_executor(executor, self.sleep)
        self.write(rest)

url_map = [
    ("/?", SyncToAsyncThreadHandler)
]

if __name__ == '__main__':
    app = web.Application(url_map, debug=True)
    app.listen(8888)
    print('started...')
    ioloop.IOLoop.current().start()

2.進程

2-1.submit方式

import time
from concurrent.futures.process import ProcessPoolExecutor

from tornado import gen, web, ioloop

executor = ProcessPoolExecutor(max_workers=2)


def sleep():
    print("休息...start")
    time.sleep(5)
    print("休息...end")
    return 'ok'


class SyncToAsyncProcessHandler(web.RequestHandler):

    @gen.coroutine
    def get(self):
        rest = yield executor.submit(sleep)
        self.write(rest)


url_map = [
    ("/?", SyncToAsyncProcessHandler)
]

if __name__ == '__main__':
    app = web.Application(url_map, debug=True)
    app.listen(8888)
    print('started...')
    ioloop.IOLoop.current().start()

2-2.ioloop方式

import time
from concurrent.futures.process import ProcessPoolExecutor

from tornado import gen, web, ioloop

executor = ProcessPoolExecutor(max_workers=2)


def sleep():
    print("休息...start")
    time.sleep(5)
    print("休息...end")
    return 'ok'


class SyncToAsyncProcessHandler(web.RequestHandler):

    async def get(self):
        rest = await ioloop.IOLoop.current().run_in_executor(executor, sleep)
        self.write(rest)


url_map = [
    ("/?", SyncToAsyncProcessHandler)
]

if __name__ == '__main__':
    app = web.Application(url_map, debug=True)
    app.listen(8888)
    print('started...')
    ioloop.IOLoop.current().start()

3.中間件

• tornado-celery

基於AMQP and Redis實現的，所以要安裝相關軟件，有興趣可以去閱讀下說明，很簡單

4.網站

本文對應的源碼

• iworkh博客

個人博客剛開不久，主要用來輔助手冊，寫些零碎的知識點

• iworkh網站

註冊下個人用戶，就可以管理自己的書籤/鏈接、享用各類學習手冊，主要用來寫手冊，分享學習。