Python併發編程——concurrent.futures

 

Concurrent.futures 模塊

        Python標準庫爲我們提供了threading和multiprocessing模塊編寫相應的多線程/多進程代碼。從Python3.2開始，標準庫爲我們提供了concurrent.futures模塊，它提供了ThreadPoolExecutor和ProcessPoolExecutor兩個類，實現了對threading和multiprocessing的更高級的抽象，對編寫線程池/進程池提供了直接的支持。 concurrent.futures基礎模塊是executor和future。

 

 

Executor

Executor是一個抽象類，它不能被直接使用。它爲具體的異步執行定義了一些基本的方法。 
ThreadPoolExecutor和ProcessPoolExecutor繼承了Executor，分別被用來創建線程池和進程池的代碼。

class Executor(object):
    """This is an abstract base class for concrete asynchronous executors."""

    def submit(self, fn, *args, **kwargs):
        raise NotImplementedError()

    def map(self, fn, *iterables, timeout=None):
        if timeout is not None:
            end_time = timeout + time.time()

        fs = [self.submit(fn, *args) for args in zip(*iterables)]
        def result_iterator():
            try:
                for future in fs:
                    if timeout is None:
                        yield future.result()
                    else:
                        yield future.result(end_time - time.time())
            finally:
                for future in fs:
                    future.cancel()
        return result_iterator()

    def shutdown(self, wait=True):
        pass

    def __enter__(self):
        return self

    def __exit__(self, exc_type, exc_val, exc_tb):
        self.shutdown(wait=True)
        return False

submit()方法

Executor中定義了submit()方法，這個方法的作用是提交一個可執行的回調task，並返回一個future實例。future對象代表的就是給定的調用。 
通過下面的例子來理解submit對線程池/進程池的操作。

# coding: utf-8

from concurrent.futures import ThreadPoolExecutor
import time


def return_future(msg):
    time.sleep(3)
    return msg


# 創建一個線程池
pool = ThreadPoolExecutor(max_workers=2)

# 往線程池加入2個task
f1 = pool.submit(return_future, 'hello')
f2 = pool.submit(return_future, 'world')

print(f1.done())
time.sleep(3)
print(f2.done())

print(f1.result())
print(f2.result())

改寫爲進程池形式很簡單，把ThreadPoolExecutor替換爲ProcessPoolExecutor即可。如果需要提交多個task，可以通過循環多次submit()。

map()方法

除了submit，Exectuor還爲我們提供了map方法，這個方法返回一個map(func, *iterables)迭代器，迭代器中的回調執行返回的結果有序的。可以通過下面的例子來理解：

# coding: utf-8

from concurrent.futures import ThreadPoolExecutor as Pool
import requests

URLS = ['http://www.baidu.com', 'http://qq.com', 'http://sina.com']


def task(url, timeout=10):
    return requests.get(url, timeout=timeout)


pool = Pool(max_workers=3)
results = pool.map(task, URLS)

for ret in results:
    print('%s, %s' % (ret.url, len(ret.content)))

執行結果

http://www.baidu.com/, 2381
http://www.qq.com/, 252160
http://www.sina.com.cn/, 607265

 

 

Future

Future可以理解爲一個在未來完成的操作，這是異步編程的基礎。通常情況下，我們執行io操作，訪問url時（如下）在等待結果返回之前會產生阻塞，cpu不能做其他事情，而Future的引入幫助我們在等待的這段時間可以完成其他的操作。

 

The Future class encapulates the asynchronous execution of a callable. Future instances are created by Executor.submit().

用cancel(),可以終止某個線程和進程的任務，返回狀態爲 True False

 

Future.cancel()
Attempt to cancel the call. If the call is currently being executed then it cannot be cancelled and the method will return False, otherwise the call will be cancelled and the method will return True.


判斷是否真的結束了任務。
Future.cancelled()
Return True if the call was successfully cancelled.


判斷是否還在運行
Future.running()
Return True if the call is currently being executed and cannot be cancelled.


判斷是正常執行完畢的。 
Future.done()
Return True if the call was successfully cancelled or finished running.


針對result結果做超時的控制。 
Future.result(timeout=None)