Python Gearman 分析(1)

python gearman 分析：

偶然看到這個框架，發現他的用途還是挺廣泛的，讀了他的源碼並嘗試對其進行一個分析，加深自己的理解同時方便以後其他方面的應用和見解。以下是對python gearman的流程的一個深入的分析。


Server：
1. Server啓動並初始化一個TaskManager，之後通過asyncore.loop監控socket連接。
2. 每個新的連接都用GearmanServerClient包裝，並且將每個ServerClient連接加入TaskManager管理。
由於Server,ServerClient都繼承自asyncore.dispather，他們也將被納入loop監控體系中。

class GearmanServer(asyncore.dispatcher):
------------------------------------------------------------------
    def __init__(self, host="127.0.0.1", port=DEFAULT_PORT):
        self.manager = GearmanTaskManager()
    def handle_accept(self):
        GearmanServerClient(sock, addr, self, self.manager)
    def start(self):
        while self.running:
            asyncore.loop(timeout=1, use_poll=False, count=1)

class GearmanServerClient(asyncore.dispatcher):
    def __init__(self, sock, addr, server, manager):
        # add self to manager
        manager.register_client(self)


Worker:
1. Worker啓動並註冊工作函數，然後進入工作狀態。
2. 進入工作狀態需要以下一些步驟：
1) 檢查self.working並循環處理連接
2) 查找alive_connections，如果未連接，嘗試連接每個connection併發送本地can_do信息，加入alive並返回列表
3) 查詢連接是否有job，如果有則進行處理，否則發送pre_sleep消息並將connection狀態置爲sleep=True，爲防止有新job進入，select connection並確認沒有新數據接收，否則將其重新置爲sleep=Flase
4) 將沒有job的連接暫時休眠

class GearmanWorker(GearmanBaseClient):
------------------------------------------------------------------
    def work(self, stop_if=None, hooks=None):
  (1)   while self.working:
            need_sleep = True
            # Try to grab work from all alive connections
  (2)       for conn in self.alive_connections:
  (3)           worked = self._work_connection(conn, hooks)
                if worked:
                    need_sleep = False
            # If no tasks were handled then sleep and wait for the server
            # to wake us with a 'noop'
  (4)       if need_sleep:
                for conn in alive:
                    if not conn.sleeping:
                        conn.send_command("pre_sleep")
                        conn.sleeping = True


client:
1. Client初始化
2. 提交任務：do_task,do_taskset,dispatch_background_task
3. 查詢任務狀態或者對處理結果

class GearmanClient(GearmanBaseClient):
------------------------------------------------------------------
    # all three method will use do_taskset to invoke task
    def do_task(self, task):
    def dispatch_background_task(self, func, arg, uniq=None, high_priority=False):
    def do_taskset(self, taskset, timeout=None):
        # set of connections to which jobs were submitted
        taskset.connections = set(self._submit_task(task) for taskset ...)
        while not taskset.cancelled and not all(task.is_finished for ...)
            # select and deal with read write handle
            for conn in rd_list:
                for cmd in conn.recv():
                    self._command_handler(taskset, conn, *cmd)
            for conn in wr_list:
                conn.send()

下圖是Gearman各個組件的一個全貌，它們之間的交互都是通過消息傳遞來實現。