docker kafka

kafka概念

broke
kafka集羣中包含一個或多個服務器，這種服務器被稱爲broker
topic
每條發佈到Kafka集羣的消息都有一個類別，這個類別被稱爲Topic。（物理上不同Topic的消息分開存儲，邏輯上一個Topic的消息雖然保存於一個或多個broker上但用戶只需指定消息的Topic即可生產或消費數據而不必關心數據存於何處）
Partition
Parition是物理上的概念，每個Topic包含一個或多個Partition.
producter
負責發佈消息到kafka broker
consumer
消息消費者，向kafka broker讀取消息的客戶端
consumer group
每個Consumer屬於一個特定的Consumer Group（可爲每個Consumer指定group name，若不指定group name則屬於默認的group）

kafka docker鏡像

參考網址：https://github.com/big-data-europe/docker-kafka
https://github.com/big-data-europe/pilot-sc6-cycle2/tree/master/sc6-kafka
https://github.com/big-data-europe/docker-zookeeper
https://github.com/big-data-europe/pilot-sc6-cycle2/tree/master/sc6-zookeeper

kafka集羣構建基於zookeeper集羣。zookeeper base鏡像中安裝了zookeeper,並且初始命令中啓動zookeeper集羣，但是配置文件還是默認的配置（https://github.com/big-data-europe/docker-zookeeper）。在zookeeper base鏡像基礎上，增加特定的配置文件，就可以構建自己需要的zookeeper鏡像。

zookeeper鏡像

FROM bde2020/zookeeper:latest

MAINTAINER Juergen Jakobitsch <[e-mail]>

ADD zoo.cfg /config/
ADD zoo_replicated1.cfg.dynamic /config/

zoo.cfg

standaloneEnabled=false
dataDir=/tmp/zookeeper
syncLimit=2
initLimit=5
tickTime=2000
dynamicConfigFile=/config/zoo_replicated1.cfg.dynamic

zoo_replicated1.cfg.dynamic

server.1=sc6_zoo_1:31200:31201:participant;31202
server.2=sc6_zoo_2:31200:31201:participant;31202
server.3=sc6_zoo_3:31200:31201:participant;31202

kafka鏡像

kafka base鏡像中安裝了kafka，並且在鏡像中定義了基於kafka-startup.json配置文件來啓動kafka，基於kafka-init.json來創建topic（實驗中沒有成功，還是需要手動執行腳本）。

FROM bde2020/kafka
ADD kafka-startup.json /config/
ADD kafka-init.json /config/

kafka-startup.json

[
  {
    "sh":"/app/bin/kafka-server-start.sh",
    "./config/server.properties":"",
    "--override":"-Djava.net.preferIPv4Stack=true",
    "--override":"delete.topic.enable=true",
    "--override":"advertised.host.name=$HOSTNAME",
    "--override":"advertised.port=9092",
    "--override":"zookeeper.connect=sc6_zoo_1:31202,sc6_zoo_2:31202,sc6_zoo_3:31202/kafka"
  }
]

kafka-init.json

[
  {
    "sh":"/app/bin/kafka-topics.sh",
    "--zookeeper":"sc6_zoo_1:31202,sc6_zoo_2:31202,sc6_zoo_3:31202/kafka",
    "--create":"",
    "--topic":"sampleTopic",
    "--partitions":"3",
    "--replication-factor":"1"
  }
]

docker-compose編排kafka集羣

version: "2"

services:
  sc6_zoo_1:
    image: "bde2020/sc6-zookeeper"
    ports:
      - 31200:31200
      - 31201:31201
      - 31202:31202
    container_name: sc6_zoo_1
    command: "bash -c /startup"
    hostname: "sc6_zoo_1"
  sc6_zoo_2:
    image: "bde2020/sc6-zookeeper"
    ports:
      - 32200:31200
      - 32201:31201
      - 32202:31202
    container_name: sc6_zoo_2
    command: "bash -c /startup"
    hostname: "sc6_zoo_2"
  sc6_zoo_3:
    image: "bde2020/sc6-zookeeper"
    ports:
      - 33200:31200
      - 33201:31201
      - 33202:31202
    container_name: sc6_zoo_3
    command: "bash -c /startup"
    hostname: "sc6_zoo_3"
  sc6-kafka-1:
    image: "xiongwen/kafka:v2"
    depends_on:
      - sc6_zoo_1
      - sc6_zoo_2
      - sc6_zoo_3
    ports:
      - 9092:9092
    container_name: sc6-kafka-1
    command: "bash -c /app/bin/kafka-init"
    hostname: "sc6-kafka-1"

進入sc6-kafka-1容器中，手動創建topic，運行生產者生產消息，運行消費者進行測試。
創建sampleTopic

/usr/bin/python3 /app/bin/kafka-bin.py /config/kafka-init.json

生產與消費消息

/usr/local/apache-kafka/current/bin/kafka-console-producer.sh --topic sampleTopic --broker-list sc6-kafka-1:9092

輸入消息。

/usr/local/apache-kafka/current/bin/kafka-console-consumer.sh --topic sampleTopic --zookeeper sc6_zoo_1:31202/kafka --bootstrap-server sc6-kafka-1:9092 --from-beginning

運行後顯示消息。