Dokcer部署Kafka集羣

原創 zer0black 2024-05-16 14:31

docker網絡規劃

docker network create kafka-net --subnet 172.20.0.0/16

docker network ls
  • zookeeper1（172.20.0.11 2184:2181）
  • zookeeper2（172.20.0.12 2185:2181）
  • zookeeper3（172.20.0.13 2186:2181）
  • kafka（172.20.0.14 內部9093:9093，外部9193:9193）
  • kafka（172.20.0.15 內部9094:9094，外部9194:9194）
  • kafka（172.20.0.16 內部9095:9095，外部9195:9195）
  • kafka manager（172.20.0.10 9000:9000）

部署中的配置和授權認證文件製作

準備一下兩個文件，他們的位置可以放到任意地方，只需要鏡像部署的配置文件中能引用到即可。

  1. 新建一個zookeeper和kafka共用的授權認證文件：server_jass.conf。按照本教程建議放到/root/kafka/kafka-sasl/server_jass.conf
Client {
    org.apache.zookeeper.server.auth.DigestLoginModule required
    username="test"
    password="test@QWER";
};

Server {
    org.apache.zookeeper.server.auth.DigestLoginModule required
    username="test"
    password="test@QWER"  
    user_admin="test@QWER"
    user_test="test@QWER"; # 賬號是test，密碼是test@QWER
};

KafkaServer {
    org.apache.kafka.common.security.plain.PlainLoginModule required
    username="test"
    password="test@QWER"
    user_test="test@QWER";
};

KafkaClient {
    org.apache.kafka.common.security.plain.PlainLoginModule required
    username="test"
    password="test@QWER";
};
  1. 新建一個kafka-run-class腳本文件，規避JMX衝突：kafka-run-class.sh。按照本教程建議放到/root/kafka/kafka-run-class.sh
#!/bin/bash
# Licensed to the Apache Software Foundation (ASF) under one or more
# contributor license agreements.  See the NOTICE file distributed with
# this work for additional information regarding copyright ownership.
# The ASF licenses this file to You under the Apache License, Version 2.0
# (the "License"); you may not use this file except in compliance with
# the License.  You may obtain a copy of the License at
#
#    http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.

if [ $# -lt 1 ];
then
  echo "USAGE: $0 [-daemon] [-name servicename] [-loggc] classname [opts]"
  exit 1
fi

# CYGWIN == 1 if Cygwin is detected, else 0.
if [[ $(uname -a) =~ "CYGWIN" ]]; then
  CYGWIN=1
else
  CYGWIN=0
fi

if [ -z "$INCLUDE_TEST_JARS" ]; then
  INCLUDE_TEST_JARS=false
fi

# Exclude jars not necessary for running commands.
regex="(-(test|test-sources|src|scaladoc|javadoc)\.jar|jar.asc)$"
should_include_file() {
  if [ "$INCLUDE_TEST_JARS" = true ]; then
    return 0
  fi
  file=$1
  if [ -z "$(echo "$file" | egrep "$regex")" ] ; then
    return 0
  else
    return 1
  fi
}

base_dir=$(dirname $0)/..

if [ -z "$SCALA_VERSION" ]; then
  SCALA_VERSION=2.13.5
  if [[ -f "$base_dir/gradle.properties" ]]; then
    SCALA_VERSION=`grep "^scalaVersion=" "$base_dir/gradle.properties" | cut -d= -f 2`
  fi
fi

if [ -z "$SCALA_BINARY_VERSION" ]; then
  SCALA_BINARY_VERSION=$(echo $SCALA_VERSION | cut -f 1-2 -d '.')
fi

# run ./gradlew copyDependantLibs to get all dependant jars in a local dir
shopt -s nullglob
if [ -z "$UPGRADE_KAFKA_STREAMS_TEST_VERSION" ]; then
  for dir in "$base_dir"/core/build/dependant-libs-${SCALA_VERSION}*;
  do
    CLASSPATH="$CLASSPATH:$dir/*"
  done
fi

for file in "$base_dir"/examples/build/libs/kafka-examples*.jar;
do
  if should_include_file "$file"; then
    CLASSPATH="$CLASSPATH":"$file"
  fi
done

if [ -z "$UPGRADE_KAFKA_STREAMS_TEST_VERSION" ]; then
  clients_lib_dir=$(dirname $0)/../clients/build/libs
  streams_lib_dir=$(dirname $0)/../streams/build/libs
  streams_dependant_clients_lib_dir=$(dirname $0)/../streams/build/dependant-libs-${SCALA_VERSION}
else
  clients_lib_dir=/opt/kafka-$UPGRADE_KAFKA_STREAMS_TEST_VERSION/libs
  streams_lib_dir=$clients_lib_dir
  streams_dependant_clients_lib_dir=$streams_lib_dir
fi


for file in "$clients_lib_dir"/kafka-clients*.jar;
do
  if should_include_file "$file"; then
    CLASSPATH="$CLASSPATH":"$file"
  fi
done

for file in "$streams_lib_dir"/kafka-streams*.jar;
do
  if should_include_file "$file"; then
    CLASSPATH="$CLASSPATH":"$file"
  fi
done

if [ -z "$UPGRADE_KAFKA_STREAMS_TEST_VERSION" ]; then
  for file in "$base_dir"/streams/examples/build/libs/kafka-streams-examples*.jar;
  do
    if should_include_file "$file"; then
      CLASSPATH="$CLASSPATH":"$file"
    fi
  done
else
  VERSION_NO_DOTS=`echo $UPGRADE_KAFKA_STREAMS_TEST_VERSION | sed 's/\.//g'`
  SHORT_VERSION_NO_DOTS=${VERSION_NO_DOTS:0:((${#VERSION_NO_DOTS} - 1))} # remove last char, ie, bug-fix number
  for file in "$base_dir"/streams/upgrade-system-tests-$SHORT_VERSION_NO_DOTS/build/libs/kafka-streams-upgrade-system-tests*.jar;
  do
    if should_include_file "$file"; then
      CLASSPATH="$file":"$CLASSPATH"
    fi
  done
  if [ "$SHORT_VERSION_NO_DOTS" = "0100" ]; then
    CLASSPATH="/opt/kafka-$UPGRADE_KAFKA_STREAMS_TEST_VERSION/libs/zkclient-0.8.jar":"$CLASSPATH"
    CLASSPATH="/opt/kafka-$UPGRADE_KAFKA_STREAMS_TEST_VERSION/libs/zookeeper-3.4.6.jar":"$CLASSPATH"
  fi
  if [ "$SHORT_VERSION_NO_DOTS" = "0101" ]; then
    CLASSPATH="/opt/kafka-$UPGRADE_KAFKA_STREAMS_TEST_VERSION/libs/zkclient-0.9.jar":"$CLASSPATH"
    CLASSPATH="/opt/kafka-$UPGRADE_KAFKA_STREAMS_TEST_VERSION/libs/zookeeper-3.4.8.jar":"$CLASSPATH"
  fi
fi

for file in "$streams_dependant_clients_lib_dir"/rocksdb*.jar;
do
  CLASSPATH="$CLASSPATH":"$file"
done

for file in "$streams_dependant_clients_lib_dir"/*hamcrest*.jar;
do
  CLASSPATH="$CLASSPATH":"$file"
done

for file in "$base_dir"/shell/build/libs/kafka-shell*.jar;
do
  if should_include_file "$file"; then
    CLASSPATH="$CLASSPATH":"$file"
  fi
done

for dir in "$base_dir"/shell/build/dependant-libs-${SCALA_VERSION}*;
do
  CLASSPATH="$CLASSPATH:$dir/*"
done

for file in "$base_dir"/tools/build/libs/kafka-tools*.jar;
do
  if should_include_file "$file"; then
    CLASSPATH="$CLASSPATH":"$file"
  fi
done

for dir in "$base_dir"/tools/build/dependant-libs-${SCALA_VERSION}*;
do
  CLASSPATH="$CLASSPATH:$dir/*"
done

for cc_pkg in "api" "transforms" "runtime" "file" "mirror" "mirror-client" "json" "tools" "basic-auth-extension"
do
  for file in "$base_dir"/connect/${cc_pkg}/build/libs/connect-${cc_pkg}*.jar;
  do
    if should_include_file "$file"; then
      CLASSPATH="$CLASSPATH":"$file"
    fi
  done
  if [ -d "$base_dir/connect/${cc_pkg}/build/dependant-libs" ] ; then
    CLASSPATH="$CLASSPATH:$base_dir/connect/${cc_pkg}/build/dependant-libs/*"
  fi
done

# classpath addition for release
for file in "$base_dir"/libs/*;
do
  if should_include_file "$file"; then
    CLASSPATH="$CLASSPATH":"$file"
  fi
done

for file in "$base_dir"/core/build/libs/kafka_${SCALA_BINARY_VERSION}*.jar;
do
  if should_include_file "$file"; then
    CLASSPATH="$CLASSPATH":"$file"
  fi
done
shopt -u nullglob

if [ -z "$CLASSPATH" ] ; then
  echo "Classpath is empty. Please build the project first e.g. by running './gradlew jar -PscalaVersion=$SCALA_VERSION'"
  exit 1
fi

# JMX settings
if [ -z "$KAFKA_JMX_OPTS" ]; then
  KAFKA_JMX_OPTS="-Dcom.sun.management.jmxremote -Dcom.sun.management.jmxremote.authenticate=false  -Dcom.sun.management.jmxremote.ssl=false "
fi

# JMX port to use
ISKAFKASERVER="false"
if [[ "$*" =~ "kafka.Kafka" ]]; then
  ISKAFKASERVER="true"
fi
if [  $JMX_PORT ] && [ -z "$ISKAFKASERVER" ]; then
  KAFKA_JMX_OPTS="$KAFKA_JMX_OPTS -Dcom.sun.management.jmxremote.port=$JMX_PORT "
fi

# Log directory to use
if [ "x$LOG_DIR" = "x" ]; then
  LOG_DIR="$base_dir/logs"
fi

# Log4j settings
if [ -z "$KAFKA_LOG4J_OPTS" ]; then
  # Log to console. This is a tool.
  LOG4J_DIR="$base_dir/config/tools-log4j.properties"
  # If Cygwin is detected, LOG4J_DIR is converted to Windows format.
  (( CYGWIN )) && LOG4J_DIR=$(cygpath --path --mixed "${LOG4J_DIR}")
  KAFKA_LOG4J_OPTS="-Dlog4j.configuration=file:${LOG4J_DIR}"
else
  # create logs directory
  if [ ! -d "$LOG_DIR" ]; then
    mkdir -p "$LOG_DIR"
  fi
fi

# If Cygwin is detected, LOG_DIR is converted to Windows format.
(( CYGWIN )) && LOG_DIR=$(cygpath --path --mixed "${LOG_DIR}")
KAFKA_LOG4J_OPTS="-Dkafka.logs.dir=$LOG_DIR $KAFKA_LOG4J_OPTS"

# Generic jvm settings you want to add
if [ -z "$KAFKA_OPTS" ]; then
  KAFKA_OPTS=""
fi

# Set Debug options if enabled
if [ "x$KAFKA_DEBUG" != "x" ]; then

    # Use default ports
    DEFAULT_JAVA_DEBUG_PORT="5005"

    if [ -z "$JAVA_DEBUG_PORT" ]; then
        JAVA_DEBUG_PORT="$DEFAULT_JAVA_DEBUG_PORT"
    fi

    # Use the defaults if JAVA_DEBUG_OPTS was not set
    DEFAULT_JAVA_DEBUG_OPTS="-agentlib:jdwp=transport=dt_socket,server=y,suspend=${DEBUG_SUSPEND_FLAG:-n},address=$JAVA_DEBUG_PORT"
    if [ -z "$JAVA_DEBUG_OPTS" ]; then
        JAVA_DEBUG_OPTS="$DEFAULT_JAVA_DEBUG_OPTS"
    fi

    echo "Enabling Java debug options: $JAVA_DEBUG_OPTS"
    KAFKA_OPTS="$JAVA_DEBUG_OPTS $KAFKA_OPTS"
fi

# Which java to use
if [ -z "$JAVA_HOME" ]; then
  JAVA="java"
else
  JAVA="$JAVA_HOME/bin/java"
fi

# Memory options
if [ -z "$KAFKA_HEAP_OPTS" ]; then
  KAFKA_HEAP_OPTS="-Xmx256M"
fi

# JVM performance options
# MaxInlineLevel=15 is the default since JDK 14 and can be removed once older JDKs are no longer supported
if [ -z "$KAFKA_JVM_PERFORMANCE_OPTS" ]; then
  KAFKA_JVM_PERFORMANCE_OPTS="-server -XX:+UseG1GC -XX:MaxGCPauseMillis=20 -XX:InitiatingHeapOccupancyPercent=35 -XX:+ExplicitGCInvokesConcurrent -XX:MaxInlineLevel=15 -Djava.awt.headless=true"
fi

while [ $# -gt 0 ]; do
  COMMAND=$1
  case $COMMAND in
    -name)
      DAEMON_NAME=$2
      CONSOLE_OUTPUT_FILE=$LOG_DIR/$DAEMON_NAME.out
      shift 2
      ;;
    -loggc)
      if [ -z "$KAFKA_GC_LOG_OPTS" ]; then
        GC_LOG_ENABLED="true"
      fi
      shift
      ;;
    -daemon)
      DAEMON_MODE="true"
      shift
      ;;
    *)
      break
      ;;
  esac
done

# GC options
GC_FILE_SUFFIX='-gc.log'
GC_LOG_FILE_NAME=''
if [ "x$GC_LOG_ENABLED" = "xtrue" ]; then
  GC_LOG_FILE_NAME=$DAEMON_NAME$GC_FILE_SUFFIX

  # The first segment of the version number, which is '1' for releases before Java 9
  # it then becomes '9', '10', ...
  # Some examples of the first line of `java --version`:
  # 8 -> java version "1.8.0_152"
  # 9.0.4 -> java version "9.0.4"
  # 10 -> java version "10" 2018-03-20
  # 10.0.1 -> java version "10.0.1" 2018-04-17
  # We need to match to the end of the line to prevent sed from printing the characters that do not match
  JAVA_MAJOR_VERSION=$("$JAVA" -version 2>&1 | sed -E -n 's/.* version "([0-9]*).*$/\1/p')
  if [[ "$JAVA_MAJOR_VERSION" -ge "9" ]] ; then
    KAFKA_GC_LOG_OPTS="-Xlog:gc*:file=$LOG_DIR/$GC_LOG_FILE_NAME:time,tags:filecount=10,filesize=100M"
  else
    KAFKA_GC_LOG_OPTS="-Xloggc:$LOG_DIR/$GC_LOG_FILE_NAME -verbose:gc -XX:+PrintGCDetails -XX:+PrintGCDateStamps -XX:+PrintGCTimeStamps -XX:+UseGCLogFileRotation -XX:NumberOfGCLogFiles=10 -XX:GCLogFileSize=100M"
  fi
fi

# Remove a possible colon prefix from the classpath (happens at lines like `CLASSPATH="$CLASSPATH:$file"` when CLASSPATH is blank)
# Syntax used on the right side is native Bash string manipulation; for more details see
# http://tldp.org/LDP/abs/html/string-manipulation.html, specifically the section titled "Substring Removal"
CLASSPATH=${CLASSPATH#:}

# If Cygwin is detected, classpath is converted to Windows format.
(( CYGWIN )) && CLASSPATH=$(cygpath --path --mixed "${CLASSPATH}")

# Launch mode
if [ "x$DAEMON_MODE" = "xtrue" ]; then
  nohup "$JAVA" $KAFKA_HEAP_OPTS $KAFKA_JVM_PERFORMANCE_OPTS $KAFKA_GC_LOG_OPTS $KAFKA_JMX_OPTS $KAFKA_LOG4J_OPTS -cp "$CLASSPATH" $KAFKA_OPTS "$@" > "$CONSOLE_OUTPUT_FILE" 2>&1 < /dev/null &
else
  exec "$JAVA" $KAFKA_HEAP_OPTS $KAFKA_JVM_PERFORMANCE_OPTS $KAFKA_GC_LOG_OPTS $KAFKA_JMX_OPTS $KAFKA_LOG4J_OPTS -cp "$CLASSPATH" $KAFKA_OPTS "$@"
fi

鏡像部署

  1. 新建zookeeper鏡像文件：zk-docker-compose.yml
services:
  zook1:
    image: zookeeper:latest
    #restart: always #自動重新啓動
    hostname: zook1
    container_name: zook1 #容器名稱，方便在rancher中顯示有意義的名稱
    ports:
    - 2183:2181 #將本容器的zookeeper默認端口號映射出去
    volumes: # 掛載數據卷 前面是宿主機即本機的目錄位置，後面是docker的目錄
    - "/Users/konsy/Development/volume/zkcluster/zook1/data:/data"
    - "/Users/konsy/Development/volume/zkcluster/zook1/datalog:/datalog"
    - "/Users/konsy/Development/volume/zkcluster/zook1/logs:/logs"
    - "/root/kafka/kafka-sasl/:/opt/zookeeper/secrets/" #映射賬號密碼配置文件
    environment:
        ZOO_MY_ID: 1  #即是zookeeper的節點值，也是kafka的brokerid值
        ZOO_SERVERS: server.1=zook1:2888:3888;2181 server.2=zook2:2888:3888;2181 server.3=zook3:2888:3888;2181
        ZOO_TLS_QUORUM_CLIENT_AUTH: need
        SERVER_JVMFLAGS: -Djava.security.auth.login.config=/opt/zookeeper/secrets/server_jass.conf #指定賬號密碼配置文件地址
    networks:
        kafka-net:
            ipv4_address: 172.20.0.11

  zook2:   
    image: zookeeper:latest
    #restart: always #自動重新啓動
    hostname: zook2
    container_name: zook2 #容器名稱，方便在rancher中顯示有意義的名稱
    ports:
    - 2184:2181 #將本容器的zookeeper默認端口號映射出去
    volumes:
    - "/Users/konsy/Development/volume/zkcluster/zook2/data:/data"
    - "/Users/konsy/Development/volume/zkcluster/zook2/datalog:/datalog"
    - "/Users/konsy/Development/volume/zkcluster/zook2/logs:/logs"
    - "/root/kafka/kafka-sasl/:/opt/zookeeper/secrets/"
    environment:
        ZOO_MY_ID: 2  #即是zookeeper的節點值，也是kafka的brokerid值
        ZOO_SERVERS: server.1=zook1:2888:3888;2181 server.2=zook2:2888:3888;2181 server.3=zook3:2888:3888;2181
        ZOO_TLS_QUORUM_CLIENT_AUTH: need
        SERVER_JVMFLAGS: -Djava.security.auth.login.config=/opt/zookeeper/secrets/server_jass.conf
    networks:
        kafka-net:
            ipv4_address: 172.20.0.12
            
  zook3:   
    image: zookeeper:latest
    #restart: always #自動重新啓動
    hostname: zook3
    container_name: zook3 #容器名稱，方便在rancher中顯示有意義的名稱
    ports:
    - 2185:2181 #將本容器的zookeeper默認端口號映射出去
    volumes:
    - "/Users/konsy/Development/volume/zkcluster/zook3/data:/data"
    - "/Users/konsy/Development/volume/zkcluster/zook3/datalog:/datalog"
    - "/Users/konsy/Development/volume/zkcluster/zook3/logs:/logs"
    - "/root/kafka/kafka-sasl/:/opt/zookeeper/secrets/"
    environment:
        ZOO_MY_ID: 3  #即是zookeeper的節點值，也是kafka的brokerid值
        ZOO_SERVERS: server.1=zook1:2888:3888;2181 server.2=zook2:2888:3888;2181 server.3=zook3:2888:3888;2181
        ZOO_TLS_QUORUM_CLIENT_AUTH: need
        SERVER_JVMFLAGS: -Djava.security.auth.login.config=/opt/zookeeper/secrets/server_jass.conf
    networks:
        kafka-net:
            ipv4_address: 172.20.0.13
networks:
  kafka-net:
    external: true
  1. 執行腳本部署zookeeper至Docker
docker compose -p zookeeper -f ./zk-docker-compose.yml up -d
  1. 新建Kafka集羣配置文件：kafka-docker-compose.yml
services:
  kafka1:
    image: docker.io/wurstmeister/kafka
    #restart: always #自動重新啓動
    hostname: 172.20.0.14
    container_name: kafka1
    ports:
      - 9093:9093
    volumes:
      - /Users/konsy/Development/volume/kafka/kafka1/wurstmeister/kafka:/wurstmeister/kafka
      - /Users/konsy/Development/volume/kafka/kafka1/kafka:/kafka
      - /root/kafka/kafka-sasl/:/opt/kafka/secrets/
      - /root/kafka/kafka-run-class.sh:/opt/kafka_2.13-2.8.1/bin/kafka-run-class.sh
    environment:
      KAFKA_BROKER_ID: 1
      KAFKA_LISTENERS: PLAINTEXT://:9093
      KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://192.168.198.131:9093
      KAFKA_INTER_BROKER_LISTENER_NAME: PLAINTEXT
      KAFKA_SASL_MECHANISM_INTER_BROKER_PROTOCOL: PLAIN
      KAFKA_SASL_ENABLED_MECHANISMS: PLAIN
      KAFKA_AUTHORIZER_CLASS_NAME: kafka.security.auth.SimpleAclAuthorizer
      KAFKA_ALLOW_EVERYONE_IF_NO_ACL_FOUND: "true"
      KAFKA_ZOOKEEPER_CONNECT: zook1:2181,zook2:2181,zook3:2181
      ALLOW_PLAINTEXT_LISTENER : 'yes'
      JMX_PORT: 9999 #開放JMX監控端口，來監測集羣數據
      KAFKA_OPTS: -Djava.security.auth.login.config=/opt/kafka/secrets/server_jass.conf
    external_links:
      - zook1
      - zook2
      - zook3
    networks:
      kafka-net:
        ipv4_address: 172.20.0.14

  kafka2:
    image: docker.io/wurstmeister/kafka
    #restart: always #自動重新啓動
    hostname: 172.20.0.15
    container_name: kafka2
    ports:
      - 9094:9094
    volumes:
      - /Users/konsy/Development/volume/kafka/kafka1/wurstmeister/kafka:/wurstmeister/kafka
      - /Users/konsy/Development/volume/kafka/kafka1/kafka:/kafka
      - /root/kafka/kafka-sasl/:/opt/kafka/secrets/
      - /root/kafka/kafka-run-class.sh:/opt/kafka_2.13-2.8.1/bin/kafka-run-class.sh
    environment:
      KAFKA_BROKER_ID: 2
      KAFKA_LISTENERS: PLAINTEXT://:9094
      KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://192.168.198.131:9094
      KAFKA_INTER_BROKER_LISTENER_NAME: PLAINTEXT
      KAFKA_SASL_MECHANISM_INTER_BROKER_PROTOCOL: PLAIN
      KAFKA_SASL_ENABLED_MECHANISMS: PLAIN
      KAFKA_AUTHORIZER_CLASS_NAME: kafka.security.auth.SimpleAclAuthorizer
      KAFKA_ALLOW_EVERYONE_IF_NO_ACL_FOUND: "true"
      KAFKA_ZOOKEEPER_CONNECT: zook1:2181,zook2:2181,zook3:2181
      ALLOW_PLAINTEXT_LISTENER : 'yes'
      JMX_PORT: 9999 #開放JMX監控端口，來監測集羣數據
      KAFKA_OPTS: -Djava.security.auth.login.config=/opt/kafka/secrets/server_jass.conf
    external_links:
      - zook1
      - zook2
      - zook3
    networks:
      kafka-net:
        ipv4_address: 172.20.0.15

  kafka3:
    image: docker.io/wurstmeister/kafka
    #restart: always #自動重新啓動
    hostname: 172.20.0.16
    container_name: kafka3
    ports:
      - 9095:9095
    volumes:
      - /Users/konsy/Development/volume/kafka/kafka1/wurstmeister/kafka:/wurstmeister/kafka
      - /Users/konsy/Development/volume/kafka/kafka1/kafka:/kafka
      - /root/kafka/kafka-sasl/:/opt/kafka/secrets/
      - /root/kafka/kafka-run-class.sh:/opt/kafka_2.13-2.8.1/bin/kafka-run-class.sh
    environment:
      KAFKA_BROKER_ID: 3
      KAFKA_LISTENERS: PLAINTEXT://:9095
      KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://192.168.198.131:9095
      KAFKA_INTER_BROKER_LISTENER_NAME: PLAINTEXT
      KAFKA_SASL_MECHANISM_INTER_BROKER_PROTOCOL: PLAIN
      KAFKA_SASL_ENABLED_MECHANISMS: PLAIN
      KAFKA_AUTHORIZER_CLASS_NAME: kafka.security.auth.SimpleAclAuthorizer
      KAFKA_ALLOW_EVERYONE_IF_NO_ACL_FOUND: "true"
      KAFKA_ZOOKEEPER_CONNECT: zook1:2181,zook2:2181,zook3:2181
      ALLOW_PLAINTEXT_LISTENER : 'yes'
      JMX_PORT: 9999 #開放JMX監控端口，來監測集羣數據
      KAFKA_OPTS: -Djava.security.auth.login.config=/opt/kafka/secrets/server_jass.conf
    external_links:
      - zook1
      - zook2
      - zook3
    networks:
      kafka-net:
        ipv4_address: 172.20.0.16
networks:
  kafka-net:
      external: true
  1. 執行腳本部署kafka至Docker
docker compose -f ./kafka-docker-compose.yml up -d
  1. 新建kafka-manager配置文件：kafka-manager-docker-compose.yml
services:
  kafka-manager:
    image: scjtqs/kafka-manager:latest
    restart: always
    hostname: kafka-manager
    container_name: kafka-manager
    ports:
      - 9000:9000
    external_links:  # 連接本compose文件以外的container
      - zook1
      - zook2
      - zook3
      - 172.20.0.14
      - 172.20.0.15
      - 172.20.0.16
    environment:
      ZK_HOSTS: zook1:2181,zook2:2181,zook3:2181
      KAFKA_BROKERS: 172.20.0.14:9093,172.20.0.15:9094,172.20.0.16:9095
      APPLICATION_SECRET: letmein
      KM_ARGS: -Djava.net.preferIPv4Stack=true
    networks:
      kafka-net:
        ipv4_address: 172.20.10.10
networks:
  kafka-net:
    external: true
  1. 執行腳本部署kafka-manager至Docker
docker compose -f ./kafka-manager-docker-compose.yml up -d
  1. 配置Cluster

發表評論
所有評論
還沒有人評論，想成為第一個評論的人麼? 請在上方評論欄輸入並且點擊發布.
相關文章

解密Prompt系列31. LLM Agent之從經驗中不斷學習的智能體

Agent智能體的工作流可以簡單分成兩種：一種是固定的靜態工作流，一種是智能體自主決策的動態工作流。 靜態流程的Agent舉幾個例子，例如新聞熱點追蹤推送Agent，每日新論文摘要總結Agent，它們的優點是可控，穩定，可復現，缺點是一種流

風雨中的小七 2024-06-11 14:22:16

第十二節：MySQL8.x版本新特性和變化

一.                二.                三.                  ! 作       者 : Yaopengfei(姚鵬飛) 博客地址 : http://www.cnblogs.com

Yaopengfei 2024-06-11 14:21:36

杭州的 IT 崩盤了麼？

大家好，我是R哥。 今天分享一個爽飛了的面試輔導 case： 這個杭州兄弟空窗期 1 個月+，面試了 6 家公司 0 Offer，不知道問題出在哪，難道是杭州的 IT 崩盤了麼？ 報名面試輔導後，經過一個多月的輔導打磨，現在成功入職某上市

Java技術棧 2024-06-11 14:21:26

Azure Virtual Network (22) 多訂閱使用Azure DNS解析問題 Windows Azure Platform 系列文章目錄

　　《Windows Azure Platform 系列文章目錄》   　　部署環境： 　　1.我們只有1個訂閱，1個Virtual Network 　　2.我們有1個Hub-VNet, 在Virtual Network上設置了DNS Se

Lei Zhang的博客 2024-06-11 14:20:36

VS2022 解決方案打不開 .NET Framework 4.0 、 4.5 等老項目

vs2022開發工具最低支持net4.8，以下的如net3.5、4.0、4.5項目，加載不上怎麼處理。 一、下載.NET Framework框架 .NET Framework 4.5.2 .NET Framework 4.5.1 .NET

鄭州-在路上 2024-06-11 14:20:26

Python 潮流週刊#55：分享 9 個高質量的技術類信息源！

大家好，我是貓哥，今天給大家分享幾個高質量的技術類信息源。 本文分享的信息源都是週刊類型的，所謂週刊類，就是以固定每週的頻率更新，每期分享很多精華內容的鏈接。它的特點是信息密度極高，可以節省你去查找信息的時間，高效的學習者都會喜歡這類內容。

豌豆花下貓 2024-06-11 14:19:16

統計vertica表的行數

select  anchor_table_schema || '.' || anchor_table_name as table_name,  row_countfrom PROJECTION_STORAGEorder by row_co

卡卡西村長 2024-06-11 14:19:16

開源高性能結構化日誌模塊NanoLog

  最近在寫數據庫程序，需要一個高性能的結構化日誌記錄組件，簡單研究了一下Microsoft.Extensions.Logging和Serilog，還是決定重造一個輪子。 一、使用方法   直接參考以下示例代碼: NanoLogger.St

白菜園 2024-06-11 14:18:16

Vue3 運行可以，build 打包發佈報錯，app.config.globalProperties 用法坑

目錄錯誤原因解決方案 app.config.globalProperties 用法坑Vue， 多環境配置 https://www.cnblogs.com/vipsoft/p/16696640.html main.js import conf

阿 軍 2024-06-11 14:15:15

Vue 打包 Error: error:0308010C:digital envelope routines::unsupported

這個錯誤通常與Node.js的加密模塊和OpenSSL版本有關 出現這個錯誤是因爲 node.js V17版本中最近發佈的OpenSSL3.0, 而OpenSSL3.0對允許算法和密鑰大小增加了嚴格的限制，可能會對生態系統造成一些影響. j

阿 軍 2024-06-11 14:15:05

Windows 允許空密碼遠程桌面

Windows 允許空密碼遠程桌面 開啓遠程 修改策略 輸入命令 secpol.msc 本地策略 -> 安全選項 -> 帳戶：使用空密碼的本地帳戶只允許進行控制檯登錄

阿 軍 2024-06-11 14:15:05

dubbo~全局異常攔截器的使用與設計缺陷~續

上一次的介紹，主要圍繞如何統一去捕獲異常，以及爲每一種異常添加自己的Mapper實現，並且我們知道，當在ExceptionMapper中返回非200的Response，不支持application/json的響應類型，而是寫死的text/p

張佔嶺 2024-06-11 14:12:35

如何安全地訪問互聯網

當你深夜在瀏覽器中輸入 www.baidu.com 時有沒有想過，除了月黑風高的夜和本機的瀏覽記錄，還有誰知道你訪問了它呢？要搞清楚這件事，首先我們要了解一下訪問網站時，這其中發生了什麼。 如果你在 10 年之前訪問網站，大概率會在瀏覽器的

張高興 2024-06-11 14:08:15

前端使用 Konva 實現可視化設計器（14）- 折線 - 最優路徑應用【代碼篇】

話接上回《前端使用 Konva 實現可視化設計器（13）- 折線 - 最優路徑應用【思路篇】》，這一章繼續說說相關的代碼如何構思的，如何一步步構建數據模型可供 AStar 算法進行路徑規劃，最終畫出節點之間的連接折線。 請大家動動小手，給

xachary 2024-06-11 14:08:05

[快速閱讀七] Halcon裏emphasize函數相關資料.

　　時不時有人問我我的SSE優化Demo裏emphasize（邊緣強調）的原理是啥，有沒有寫博客，其實不是我不願意寫博客，而是那個東西太過於簡單，我不想寫博客。但是耐不住問的人多了，我就乾脆複製點資料放在博客裏吧，省的每次我還要去找點資料複

Imageshop 2024-06-11 14:08:05