mac下kafka環境搭建 測試

kafka介紹：https://blog.csdn.net/see_you_see_me/article/details/78468108

1、安裝工具brew install kafka 會自動安裝依賴zookeeper

2、安裝配置文件位置 /usr/local/etc/kafka|zookeeper

3、啓動 zookeeper

cd /usr/local/Cellar/kafka/0.10.0.1

./bin/zookeeper-server-start /usr/local/etc/kafka/zookeeper.properties &

4、啓動kafka服務

./bin/kafka-server-start /usr/local/etc/kafka/server.properties &

5、創建topic 

./bin/kafka-topics --create --zookeeper localhost:2181 --replication-factor 1 --partitions 1 --topic test1

6、查看創建的topic

./bin/kafka-topics --list --zookeeper localhost:2181

7、生產數據

./bin/kafka-console-producer --broker-list localhost:9092 --topic test1

8、消費數據

./bin/kafka-console-consumer.sh --zookeeper localhost:2181 --topic test1--from-beginning

下面給出一個python代碼演示demo，首先保證你已經安裝python-kafak。

創建生產者：procucer.py

from kafka import KafkaProducer
import time
import logging
import sys

logging.basicConfig(
    stream=sys.stdout,
    level=logging.INFO,
    format='%(asctime)s.%(msecs)d %(filename)s[line:%(lineno)d] %(levelname)s: %(message)s',
    datefmt='%Y/%m/%d %H:%M:%S'
)

log = logging.getLogger('kafka demo')


def list_file():
    while True:
        log.info("===>>>running")
        time.sleep(2)
        producer.send('world',key=b'foo', value=b'bar')
        producer.flush()


if __name__ == '__main__':
    producer = KafkaProducer(bootstrap_servers='localhost:9092')
    list_file()
    producer.close()

創建消費者：consumer.py

# -*- encoding=utf-8 -*-
from kafka import KafkaConsumer
import logging
import sys

logging.basicConfig(
    stream=sys.stdout,
    level=logging.INFO,
    format='%(asctime)s.%(msecs)d %(filename)s[line:%(lineno)d] %(levelname)s: %(message)s',
    datefmt='%Y/%m/%d %H:%M:%S'
)

log = logging.getLogger('kafka demo')


if __name__ == '__main__':
    consumer = KafkaConsumer('world', group_id='consumer-20171017', bootstrap_servers=['localhost:9092'])
    for msg in consumer:
        recv = "%s:%d:%d: key=%s value=%s" % (msg.topic, msg.partition, msg.offset, msg.key, msg.value)
        log.info(recv)

然後，分別運行這2個文件就可以模擬了。

完整源代碼：https://github.com/sujeek/python_data_stract/tree/master/src/pds/python_kafka/py_kafka_demo1

參考：

1. https://kafka-python.readthedocs.io/en/master/index.html

2.https://github.com/dpkp/kafka-python/blob/master/docs/index.rst