python發送zookeeper的kafka集羣消息

python發kafka消息

1. 目前技術一般採用基於zk的kafka集羣，傳統直連方式不可取。因此採用pykafka三方類庫來進行消息的發送和接收，本文實現了同步sync和異步async的消息發送produce，接收消息consume部分topic查詢不到消息，歡迎技術同學幫我補充（本文也是參考網上文章，基於python3.7）
2. 直接上代碼（實踐可用）

rom pykafka import KafkaClient
from pykafka.simpleconsumer import SimpleConsumer
import time
# import logging
# logging.basicConfig(level=logging.DEBUG)


# 創建類
class kafkaConfig:

    # 類中函數 連接
    def __init__(self):
        host_brokers = '172.13.34.168:9081,172.13.34.169:9081,172.13.34.170:9081'
        # 如果有chroot path 就放到端口號後面/kfk1
        zookeeper_hosts = '172.13.42.165:2181,172.16.42.170:2181,172.16.42.171:2181/kfk1'
        try:
            self.client = KafkaClient(hosts=host_brokers,zookeeper_hosts=zookeeper_hosts)
            print('[conn success!]')
        except Exception as e:
            print('[exception conn]:%s,[msg]:%s'%(e,self.client))

    # obtains all topics
    def topic_lists(self):
        try:
            topic_lists = self.client.topics;
            print('[topic lists]:%s'%topic_lists)
        except Exception as e:
            print('[exception topic lists]:%s'%(e,topic_lists))
        return topic_lists

    # produce 同步消息send 單條消息 需要等待
    def msg_sync_send(self,topic,content):
        try:
            topic = self.client.topics[topic]
            with topic.get_sync_producer() as produce:
                produce.produce(bytes(content,encoding='utf-8'), timestamp=round(time.time()*1000))
            print('[send sync success!]')
            return "send sync success!"
        except Exception as e:
            print('[exception sync send]:%s'%e)

    # produce 異步消息send 單條消息  但生產環境 高吞吐量
    def msg_async_send(self,topic,content):
        try:
            topic = self.client.topics[topic]
            with topic.get_producer() as produce:
                produce.produce(bytes(content,encoding='utf-8'),timestamp=round(time.time()*1000))
            print('[send async success!]')
        except Exception as e:
            print('[exception async send]:%s'%e)

    # consumer 消息receive  simple適用於需要消費指定分區且不需要自動的重分配(自定義)
    # 在consumer_timeout_ms內沒有任何信息返回，則中斷接受消息
    def msg_simple_receive(self,topic,param):
        try:
            topic = self.client.topics[topic.encode()]
            partitions = topic.partitions
            last_offset = topic.latest_available_offsets()
            print("info msg:%s,%s"%(partitions,last_offset))  # 查看所有分區
            message = topic.get_simple_consumer(consumer_timeout_ms=1000,consumer_group=None)
            print('[message type]:%s,%s'%(type(message),SimpleConsumer.consume(message)))
            msg_list=[]
            for msg in message:
                # if param in msg.value:
                print('[msg,offset,content]:%s,%s,%s'%(type(msg), msg.offset, msg.value))
                msg_list.append(msg)
            print('[receive simple success!]%s,%s'%(message,type(message)))
            return msg_list
        except Exception as e:
            print('[exception simple receive]:%s'%e)
            return {"code": 300, "msg": "[exception]:%s"%e}

3. 關於集羣host_brokers以及zookeeper_hosts地址 需要更換對應的地址，上述僅用例參考。
4. 上述msg_async_send以及msg_sync_send可以完成正常的消息生產發送，但是msg_simple_receive方法查詢獲取消息還有點問題，具體定位在get_simple_consumer方法，目前測試消息體可以獲取到消息列表，但是業務流程中一些消息topic獲取不到，會有報錯INFO:pykafka.simpleconsumer:Continuing in response to UnknownError報錯。希望有技術同學能幫助我。
5. 此外日誌打印方式：

import logging
logging.basicConfig(level=logging.DEBUG)