docker swam 安裝kafka集羣以及kfakamanger

前面搭建過單節點zookeeper的kafka集羣,但是操作起來很不方便,因此這次試着搭建一個帶有管理界面kafka-manager的kafka集羣.

1、創建docker-compose的yml文件
創建kafka文件夾,並在目錄下編寫docker-compose-zoosingle-manage-kafka.yml文件

version: '3'
services:
  zookeeper:
    image: zookeeper
    container_name: zookeeper
    ports:
      - "2181:2181"

  kafka1:
    image: wurstmeister/kafka
    ports:
      - "9092:9092"
    environment:
      KAFKA_ADVERTISED_HOST_NAME: localhost
      KAFKA_CREATE_TOPICS: TestComposeTopic:4:3
      KAFKA_ZOOKEEPER_CONNECT: zookeeper:2181
      KAFKA_BROKER_ID: 1
      KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://localhost:9092
      KAFKA_LISTENERS: PLAINTEXT://0.0.0.0:9092
    container_name: kafka01
    volumes:
      - /var/run/docker.sock:/var/run/docker.sock

  kafka2:
    image: wurstmeister/kafka
    ports:
      - "9093:9093"
    environment:
      KAFKA_ADVERTISED_HOST_NAME: localhost
      KAFKA_ZOOKEEPER_CONNECT: zookeeper:2181
      KAFKA_BROKER_ID: 2
      KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://localhost:9093
      KAFKA_LISTENERS: PLAINTEXT://0.0.0.0:9093
    container_name: kafka02
    volumes:
      - /var/run/docker.sock:/var/run/docker.sock

  kafka3:
    image: wurstmeister/kafka
    ports:
      - "9094:9094"
    environment:
      KAFKA_ADVERTISED_HOST_NAME: localhost
      KAFKA_ZOOKEEPER_CONNECT: zookeeper:2181
      KAFKA_BROKER_ID: 3
      KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://localhost:9094
      KAFKA_LISTENERS: PLAINTEXT://0.0.0.0:9094
    container_name: kafka03
    volumes:
      - /var/run/docker.sock:/var/run/docker.sock

   kafka-manager:  
    image: sheepkiller/kafka-manager
    environment:
        ZK_HOSTS: 106.54.1.220
    ports:  
      - "9000:9000"

2、啓動集羣

# docker-compose -f docker-compose-zoosingle-manage-kafka.yml up -d

docker會根據yml文件拉取相應的鏡像創建容器,並一鍵啓動集羣
如下

Creating network "kafka_default" with the default driver
Pulling zookeeper (zookeeper:latest)...
latest: Pulling from library/zookeeper
763de41d8bd6: Pull complete
c05f5d1c0b05: Pull complete
Digest: sha256:00e2815c75b95f4bfc55d8f2ded99b72c27aee5a8690788f09e9139695ec5e0d
Status: Downloaded newer image for zookeeper:latest
Creating kafka01 ... 
Creating kafka02 ... 
Creating zookeeper ... 
Creating kafka03 ... 
Creating kafka_kafka-manager_1 ... 
Creating kafka02
Creating zookeeper
Creating kafka01
Creating kafka03
Creating kafka03 ... done

查看容器啓動狀態

# docker ps -a

3、訪問管理界面
http://localhost:9000/

4、增加kafka節點

Cluster Zookeeper Hosts :宿主機ip:2181
選擇對應版本號的:2.4.0

保存可能會報錯
Yikes! KeeperErrorCode = Unimplemented for /kafka-manager/mutex Try again.
這主要是由於zookeeper的版本過低導致的, zookeeper至少要3.5.x version以上,纔可以,重新拉去最新版本的zookeeper,重啓集羣即可
查看zookeeper版本號

# echo stat|nc localhost 2181

Zookeeper version: 3.4.13-2d71af4dbe22557fda74f9a9b4309b15a7487f03, built on 06/29/2018 04:05 GMT
Clients:
 /172.19.0.2:38668[1](queued=0,recved=109,sent=112)
 /172.19.0.4:35832[1](queued=0,recved=66,sent=66)
 /172.19.0.3:44524[1](queued=0,recved=66,sent=66)
 /106.54.1.220:50420[1](queued=0,recved=29,sent=29)
 /172.19.0.1:39770[0](queued=0,recved=1,sent=0)

Latency min/avg/max: 0/2/35
Received: 281
Sent: 283
Connections: 5
Outstanding: 0
Zxid: 0x4f
Mode: standalone
Node count: 42

再次訪問