kafka2.1集羣部署

jdk部署

下載安裝包：wget wget --no-check-certificate --no-cookies --header "Cookie: oraclelicense=accept-securebackup-cookie" http://download.oracle.com/otn-pub/java/jdk/8u121-b13/e9e7ea248e2c4826b92b3f075a80e441/jdk-8u121-linux-i586.tar.gz  -C /var/tmp

解壓：tar zxf /var/tmp/jdk-8u121-linux-i586.tar.gz -C /usr/lib/

配置：

echo "

export JAVA_HOME=/usr/lib/jdk-8u121-linux-i586

export PATH=$JAVA_HOME/bin:$JAVA_HOME/jre/bin:$PATH

export CLASSPATH=$CLASSPATH:.:$JAVA_HOME/lib:$JAVA_HOME/jre/lib " > /etc/profile.d/jdk.sh

zookeeper部署

下載安裝包：wget https://archive.apache.org/dist/zookeeper/zookeeper-3.5.3-beta/zookeeper-3.5.3-beta.tar.gz

解壓：

tar zxvf /var/tmp/zookeeper-3.5.3-beta.tar.gz -C /data/src/

link -s /data/src/zookeeper-3.5.3-beta  /data/service/zookeeper

mkdir /data/zk_data && chown zookeeper:hadoop /data/zk_data && chmod 770 /data/zk_data

touch /data/zk_data/myid && chown zookeeper:hadoop /data/zk_data && chmod 770 /data/zk_data

echo 1|2|3 > /data/zk_data/myid 在各個節點運行

添加配置文件到/data/service/zookeeper/conf/zoo.cfg ：

clientPort=2181

tickTime=2000

initLimit=10

syncLimit=5

dataDir=/data/zk_data

maxClientCnxns=512

autopurge.purgeInterval=24

autopurge.snapRetainCount=3

server.1=ip:2900:3900

server.2=ip:2900:3900

server.3=ip:2900:3900

wget  http://mirrors.tuna.tsinghua.edu.cn/apache/kafka/2.3.1/kafka_2.11-2.3.1.tgz -C /var/tmp

touch /data/config/kafka/server.properties && chown root:hadoop /data/config/kafk/server.properties && chmod 770 /data/config/kafka/server.properties

添加配置文件到server.properties ：

port=9092

reserved.broker.max.id=0

broker.id.generation.enable=true

num.io.threads=60

num.partitions=20

background.threads=20

log.cleaner.threads=5

num.network.threads=48

log.retention.hours=168

num.replica.fetchers=30

log.cleaner.enable=true

delete.topic.enable=true

queued.max.requests=4096

log.cleanup.policy=delete

log.segment.bytes=1073741824

default.replication.factor=3

offsets.retention.minutes=1500

replica.fetch.wait.max.ms=3000

auto.create.topics.enable=false

replica.fetch.max.bytes=134217728

num.recovery.threads.per.data.dir=10

log.retention.check.interval.ms=300000

log.flush.interval.ms=10000

log.flush.interval.messages=200000

replica.socket.receive.buffer.bytes=33554432

socket.send.buffer.bytes=33554432

socket.receive.buffer.bytes=33554432

zookeeper.session.timeout.ms=10000

message.max.bytes=16777216

offsets.retention.check.interval.ms=600000

zookeeper.connect=

og.dirs=/mnt/storage00/datum/kafka_data

2.修改JVM參數、啓動服務

參照線上集羣修改參數：

1.vim /etc/profile.d/kafka-env.sh

export KAFKA_MM_OPTS="-server -XX:+UseG1GC -Xms24g  -Xmx24g -XX:MetaspaceSize=1024m -XX:ParallelGCThreads=10 -XX:ConcGCThreads=8"

export KAFKA_GC_OPT="-verbose.gc -XX:+PrintGC -XX:+PrintGCTimeStamps -XX:+PrintGCDateStamps -XX:+PrintHeapAtGC -XX:+PrintTenuringDistribution-XX:+PrintGCApplicationStoppedTime-XX:+PrintTenuringDistribution  -Xloggc:/data/log/kafka/kafka-gc.log -XX:+PrintGCDetails -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/data/log/kafka/java_pid%p.hprof"

export KAFKA_HEAP_OPTS="$KAFKA_MM_OPTS $KAFKA_GC_OPT -Dcom.sun.management.jmxremote=true -Dcom.sun.management.jmxremote.port=7070 -Dcom.sun.management.jmxremote.ssl=false

-Dcom.sun.management.jmxremote.authenticate=false"

2.vim /etc/profile.d/zk-env.sh

export ZK_MM_OPTS="-server -XX:+UseConcMarkSweepGC -XX:+UseParNewGC -Xms2024m -Xmx2048m -XX:ParallelGCThreads=33 -XX:ConcGCThreads=8 -XX:CMSInitiatingOccupancyFraction=85"

export ZK_GC_OPT="-verbose.gc -XX:+PrintGC -XX:+PrintGCTimeStamps -XX:+PrintGCDateStamps -XX:+PrintHeapAtGC -XX:+PrintTenuringDistribution

-XX:+PrintGCApplicationStoppedTime -XX:+PrintTenuringDistribution  -Xloggc:/data/log/zookeeper/zk-gc.log -XX:+PrintGCDetails -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/data/log/zookeeper/java_pid%p.hprof"

啓動服務：

zkServer.sh start

kafka-server-start.sh -daemon /data/config/kafka/server.properties

3.測試集羣

kafka-topics.sh --zookeeper zookeeper.connect  --create --partitions 6 --replication-factor 3 --topic bilibili

kafka-topics.sh --zookeeper zookeeper.connect --list

kafka-console-producer.sh --broker-list host:ip --topic bilibili

kafka-console-consumer.sh --bootstrap-server host:ip --topic bilibili

注意：

1.配置文件參數以實際任務佔比資源爲準

2.在修改jvm中，kafka-env.sh中的KAFKA_HEAP_OPTS參數需要在kafka啓動時放開，運行中屏蔽；xms和xmx安集羣任務啓動佔用資源設置