最近學校project項目需要一個消息系統，所以嘗試搭了一個集羣版kafka，過程中踩了非常多坑。。。這裏和大家分享一下搭建過程以及踩過的坑。
一、zookeeper集羣搭建
kafka使用zookeeper保存元數據，kafka自帶了zookeeper，不過爲了後續方便管理，我還是自己搭建了一個zookeeper集羣。
首先將下載好的zookeeper解壓
tar -zxf zookeeper-3.4.14.tar.gz
之後將zookeeper自帶的配置文件複製一份
進入zookeeper的/config目錄，然後執行復制命令
cp zoo_sample.cfg zoo.cfg
這時可以看到已經有zoo.cfg這個文件
root@VM-0-17-ubuntu:/usr/zookeeper/zookeeper-3.4.14/conf# ls
configuration.xsl log4j.properties zoo.cfg zoo_sample.cfg
打開它，修改配置
vim zoo.cfg
我是用兩臺服務器搭建集羣，這是server2的配置，切記這裏server.2一定要寫成0.0.0.0::的形式不然server1是連接不上這個節點的，同理，第一臺的配置中server.1也要寫成0.0.0.0::。
# The number of milliseconds of each tick
tickTime=2000
# The number of ticks that the initial
# synchronization phase can take
initLimit=10
# The number of ticks that can pass between
# sending a request and getting an acknowledgement
syncLimit=5
# the directory where the snapshot is stored.
# do not use /tmp for storage, /tmp here is just
# example sakes.
dataDir=/usr/zookeeper/data
# the port at which the clients will connect
clientPort=2181
server.1=server1:2888:3888 #server1爲第一臺服務器的ip
server.2=0.0.0.0:2888:3888
# the maximum number of client connections.
# increase this if you need to handle more clients
maxClientCnxns=60
#
寫好配置文件後:wq保存退出，然後進入dataDir指定的目錄，這裏我是/usr/zookeeper/data。然後創建一個名爲myid的文件，裏面只有一個數字，當前服務器是第幾個server就寫幾，例如，我當前的服務器是server.2，就寫2
echo "2" > myid
配置好了這些以後就可以啓動zookeeper了，進入zookeeper的/bin目錄，修改一下環境配置
vim zkEnv.sh
將ZOO_LOG_DIR寫成你生成日誌的目錄
if [ "x${ZOO_LOG_DIR}" = "x" ]
then
ZOO_LOG_DIR="/usr/zookeeper/log"
fi
之後啓動zookeeper，在/bin目錄下執行
./zkServer.sh start
然後執行
./zkServer.sh status
查看zookeeper節點狀況
root@VM-0-17-ubuntu:/usr/zookeeper/zookeeper-3.4.14/bin# ./zkServer.sh status
ZooKeeper JMX enabled by default
Using config: /usr/zookeeper/zookeeper-3.4.14/bin/../conf/zoo.cfg
Mode: follower
可以看到啓動成功，當前節點爲follower節點。按之前的步驟，修改一下必要配置後啓動server1。
至此，zookeeper集羣搭建完畢，之後開始搭建kafka集羣。
二、kafka集羣搭建
首先解壓kafka，
tar -zxf kafka_2.11-2.1.1.tgz
進入kafka的/config目錄,打開server.properties文件
vim server.properties
修改爲如下配置，這裏是server1的配置，注意borker.id每一個節點必須爲唯一值，同時zookeeper.connect裏填寫你zookeeper集羣的ip和端口號。advertised.listeners=PLAINTEXT://server.ip:9092這裏server.ip爲你當前節點的公網ip，以便於客戶端連接
# The id of the broker. This must be set to a unique integer for each broker.
broker.id=0
listeners = PLAINTEXT://0.0.0.0:9092
#server.ip爲你當前服務器的公網ip，如果不填寫公網ip將導致客戶端無法連接kafka
advertised.listeners=PLAINTEXT://server.ip:9092
# The number of threads that the server uses for receiving requests from the network and sending responses to the network
num.network.threads=3
# The number of threads that the server uses for processing requests, which may include disk I/O
num.io.threads=8
# The send buffer (SO_SNDBUF) used by the socket server
socket.send.buffer.bytes=102400
# The receive buffer (SO_RCVBUF) used by the socket server
socket.receive.buffer.bytes=102400
# The maximum size of a request that the socket server will accept (protection against OOM)
socket.request.max.bytes=104857600
############################# Log Basics #############################
# A comma separated list of directories under which to store log files
log.dirs=/usr/kafka/log
# The default number of log partitions per topic. More partitions allow greater
# parallelism for consumption, but this will also result in more files across
# the brokers.
num.partitions=1
# The number of threads per data directory to be used for log recovery at startup and flushing at shutdown.
# This value is recommended to be increased for installations with data dirs located in RAID array.
num.recovery.threads.per.data.dir=1
############################# Internal Topic Settings #############################
# The replication factor for the group metadata internal topics "__consumer_offsets" and "__transaction_state"
# For anything other than development testing, a value greater than 1 is recommended for to ensure availability such as 3.
offsets.topic.replication.factor=1
transaction.state.log.replication.factor=1
transaction.state.log.min.isr=1
# The minimum age of a log file to be eligible for deletion due to age
log.retention.hours=168
# A size-based retention policy for logs. Segments are pruned from the log unless the remaining
# segments drop below log.retention.bytes. Functions independently of log.retention.hours.
#log.retention.bytes=1073741824
# The maximum size of a log segment file. When this size is reached a new log segment will be created.
log.segment.bytes=1073741824
# The interval at which log segments are checked to see if they can be deleted according
# to the retention policies
log.retention.check.interval.ms=300000
# server1爲節點1的ip,server2爲節點2的ip
zookeeper.connect=server1:2181,server2:2181
# Timeout in ms for connecting to zookeeper
zookeeper.connection.timeout.ms=6000
group.initial.rebalance.delay.ms=0
配置好以後進入kafka的/bin目錄，啓動節點
./kafak-server-start.sh -daemon ../config/server.properties
查看是否啓動
netstat -anp | grep 9092
按以上的步驟修改一下配置，逐一啓動其他節點。所有節點啓動完成後就可以測試集羣了。進入/bin目錄，創建topic，這裏因爲我的集羣是兩個節點，所以設置複製因子爲2，分區爲2
./kafka-topics.sh --create --replication-factor 2 --partitions 2 --topic cluster-test --zookeeper zookeeper.server:2181
之後查看topic狀態
root@VM-0-13-ubuntu:/usr/kafka/kafka_2.11-2.1.1/bin# ./kafka-topics.sh --describe --topic cluster-test --zookeeper localhost:2181
Topic:cluster-test PartitionCount:2 ReplicationFactor:2 Configs:
Topic: cluster-test Partition: 0 Leader: 0 Replicas: 0,1 Isr: 0,1
Topic: cluster-test Partition: 1 Leader: 1 Replicas: 1,0 Isr: 1,0
可以看到該topic有兩個分區，各自的leader分別是節點1和節點2,各個分區的從節點爲節點2和節點1。
至此，kafka集羣搭建完成。最後通過java客戶端測試生產者消費者功能。
三、java客戶端測試消息系統
我使用的是springboot整合spring-kafka來測試，這裏貼上生產者和消費者的關鍵代碼
@Component
@Slf4j
public class KafkaSender {
private final KafkaTemplate kafkaTemplate;
@Autowired
public KafkaSender(KafkaTemplate kafkaTemplate) {
this.kafkaTemplate = kafkaTemplate;
}
public void send(Object data,String topic){
KafkaProducerMsg msg = new KafkaProducerMsg(data);
log.info("----------send message to topic :{}-------------",topic);
ListenableFuture<SendResult<String, String>> future = kafkaTemplate.send(topic, JSON.toJSONString(msg));
future.addCallback(new ListenableFutureCallback<SendResult<String, String>>() {
@Override
public void onFailure(Throwable throwable) {
log.info("--------------Fail to send message---------------");
}
@Override
public void onSuccess(SendResult<String, String> stringStringSendResult) {
log.info("-----------------success in sending message:{}-----------------------"
,stringStringSendResult.toString());
}
});
}
}
通過spring-test模塊發送消息
@Test
public void testSendMsgToKafka(){
for (int i = 0; i < 10 ; i++) {
Msg msg = new Msg();
msg.setMessage("test send msg to Kafka test1: "+i);
msg.setCode("0");
kafkaSender.send(msg,"cluster-test");
}
}
通過kafka客戶端提供的回調方法將發送結果輸出到日誌，測試成功。
2019-09-25 15:42:48.820 INFO 11608 --- [ad | producer-1] c.c.ksl.heart.service.impl.KafkaSender : -----------------success in sending message:SendResult [producerRecord=ProducerRecord(topic=cluster-test, partition=null, headers=RecordHeaders(headers = [], isReadOnly = true), key=null, value={"data":{"code":"0","message":"test send msg to Kafka cluster: 0"},"id":1569397368180,"sendDate":"2019-09-25 15:42:48"}, timestamp=null), recordMetadata=cluster-test-0@5]-----------------------
2019-09-25 15:42:48.824 INFO 11608 --- [ad | producer-1] c.c.ksl.heart.service.impl.KafkaSender : -----------------success in sending message:SendResult [producerRecord=ProducerRecord(topic=cluster-test, partition=null, headers=RecordHeaders(headers = [], isReadOnly = true), key=null, value={"data":{"code":"0","message":"test send msg to Kafka cluster: 2"},"id":1569397368604,"sendDate":"2019-09-25 15:42:48"}, timestamp=null), recordMetadata=cluster-test-0@6]-----------------------
2019-09-25 15:42:48.824 INFO 11608 --- [ad | producer-1] c.c.ksl.heart.service.impl.KafkaSender : -----------------success in sending message:SendResult [producerRecord=ProducerRecord(topic=cluster-test, partition=null, headers=RecordHeaders(headers = [], isReadOnly = true), key=null, value={"data":{"code":"0","message":"test send msg to Kafka cluster: 4"},"id":1569397368604,"sendDate":"2019-09-25 15:42:48"}, timestamp=null), recordMetadata=cluster-test-0@7]-----------------------
2019-09-25 15:42:48.824 INFO 11608 --- [ad | producer-1] c.c.ksl.heart.service.impl.KafkaSender : -----------------success in sending message:SendResult [producerRecord=ProducerRecord(topic=cluster-test, partition=null, headers=RecordHeaders(headers = [], isReadOnly = true), key=null, value={"data":{"code":"0","message":"test send msg to Kafka cluster: 6"},"id":1569397368604,"sendDate":"2019-09-25 15:42:48"}, timestamp=null), recordMetadata=cluster-test-0@8]-----------------------
2019-09-25 15:42:48.824 INFO 11608 --- [ad | producer-1] c.c.ksl.heart.service.impl.KafkaSender : -----------------success in sending message:SendResult [producerRecord=ProducerRecord(topic=cluster-test, partition=null, headers=RecordHeaders(headers = [], isReadOnly = true), key=null, value={"data":{"code":"0","message":"test send msg to Kafka cluster: 8"},"id":1569397368608,"sendDate":"2019-09-25 15:42:48"}, timestamp=null), recordMetadata=cluster-test-0@9]-----------------------
2019-09-25 15:42:48.824 INFO 11608 --- [ad | producer-1] c.c.ksl.heart.service.impl.KafkaSender : -----------------success in sending message:SendResult [producerRecord=ProducerRecord(topic=cluster-test, partition=null, headers=RecordHeaders(headers = [], isReadOnly = true), key=null, value={"data":{"code":"0","message":"test send msg to Kafka cluster: 1"},"id":1569397368604,"sendDate":"2019-09-25 15:42:48"}, timestamp=null), recordMetadata=cluster-test-1@5]-----------------------
2019-09-25 15:42:48.824 INFO 11608 --- [ad | producer-1] c.c.ksl.heart.service.impl.KafkaSender : -----------------success in sending message:SendResult [producerRecord=ProducerRecord(topic=cluster-test, partition=null, headers=RecordHeaders(headers = [], isReadOnly = true), key=null, value={"data":{"code":"0","message":"test send msg to Kafka cluster: 3"},"id":1569397368604,"sendDate":"2019-09-25 15:42:48"}, timestamp=null), recordMetadata=cluster-test-1@6]-----------------------
2019-09-25 15:42:48.824 INFO 11608 --- [ad | producer-1] c.c.ksl.heart.service.impl.KafkaSender : -----------------success in sending message:SendResult [producerRecord=ProducerRecord(topic=cluster-test, partition=null, headers=RecordHeaders(headers = [], isReadOnly = true), key=null, value={"data":{"code":"0","message":"test send msg to Kafka cluster: 5"},"id":1569397368604,"sendDate":"2019-09-25 15:42:48"}, timestamp=null), recordMetadata=cluster-test-1@7]-----------------------
2019-09-25 15:42:48.824 INFO 11608 --- [ad | producer-1] c.c.ksl.heart.service.impl.KafkaSender : -----------------success in sending message:SendResult [producerRecord=ProducerRecord(topic=cluster-test, partition=null, headers=RecordHeaders(headers = [], isReadOnly = true), key=null, value={"data":{"code":"0","message":"test send msg to Kafka cluster: 7"},"id":1569397368608,"sendDate":"2019-09-25 15:42:48"}, timestamp=null), recordMetadata=cluster-test-1@8]-----------------------
2019-09-25 15:42:48.828 INFO 11608 --- [ad | producer-1] c.c.ksl.heart.service.impl.KafkaSender : -----------------success in sending message:SendResult [producerRecord=ProducerRecord(topic=cluster-test, partition=null, headers=RecordHeaders(headers = [], isReadOnly = true), key=null, value={"data":{"code":"0","message":"test send msg to Kafka cluster: 9"},"id":1569397368608,"sendDate":"2019-09-25 15:42:48"}, timestamp=null), recordMetadata=cluster-test-1@9]-----------------------
之後測試消費者,
@Component
@Slf4j
public class KafkaReceiver {
@KafkaListener(topics = {"cluster-test"})
public void listen(@Payload String record,
@Header(KafkaHeaders.RECEIVED_TOPIC) String topic,
@Header(KafkaHeaders.RECEIVED_PARTITION_ID) String partitionId){
Optional<?> messageValue = Optional.ofNullable(record);
if(messageValue.isPresent()){
log.info("----------success in getting message " +
"from kafka topic:{},record:{},with partitionId:{}",topic,record,partitionId);
KafkaProducerMsg data = JSON.parseObject(record,KafkaProducerMsg.class);
System.out.println(data.toString());
}else{
log.info("---------Failure in getting message from kafka");
}
}
}
以上是消費者關鍵代碼，它會持續監聽kafka，有消息就會通過輪詢方法拉下來。啓動應用，查看測試結果
2019-09-25 15:45:47.074 INFO 2576 --- [ntainer#0-0-C-1] c.c.k.heart.service.impl.KafkaReceiver : ----------success in getting message from kafka topic:cluster-test,record:{"data":{"code":"0","message":"test send msg to Kafka cluster: 1"},"id":1569397368604,"sendDate":"2019-09-25 15:42:48"},with partitionId:1
KafkaProducerMsg(id=1569397547170, data={"code":"0","message":"test send msg to Kafka cluster: 1"}, sendDate=2019-09-25 15:45:47)
2019-09-25 15:45:47.174 INFO 2576 --- [ntainer#0-0-C-1] c.c.k.heart.service.impl.KafkaReceiver : ----------success in getting message from kafka topic:cluster-test,record:{"data":{"code":"0","message":"test send msg to Kafka cluster: 3"},"id":1569397368604,"sendDate":"2019-09-25 15:42:48"},with partitionId:1
KafkaProducerMsg(id=1569397547174, data={"code":"0","message":"test send msg to Kafka cluster: 3"}, sendDate=2019-09-25 15:45:47)
2019-09-25 15:45:47.174 INFO 2576 --- [ntainer#0-0-C-1] c.c.k.heart.service.impl.KafkaReceiver : ----------success in getting message from kafka topic:cluster-test,record:{"data":{"code":"0","message":"test send msg to Kafka cluster: 5"},"id":1569397368604,"sendDate":"2019-09-25 15:42:48"},with partitionId:1
KafkaProducerMsg(id=1569397547174, data={"code":"0","message":"test send msg to Kafka cluster: 5"}, sendDate=2019-09-25 15:45:47)
2019-09-25 15:45:47.174 INFO 2576 --- [ntainer#0-0-C-1] c.c.k.heart.service.impl.KafkaReceiver : ----------success in getting message from kafka topic:cluster-test,record:{"data":{"code":"0","message":"test send msg to Kafka cluster: 7"},"id":1569397368608,"sendDate":"2019-09-25 15:42:48"},with partitionId:1
KafkaProducerMsg(id=1569397547174, data={"code":"0","message":"test send msg to Kafka cluster: 7"}, sendDate=2019-09-25 15:45:47)
2019-09-25 15:45:47.174 INFO 2576 --- [ntainer#0-0-C-1] c.c.k.heart.service.impl.KafkaReceiver : ----------success in getting message from kafka topic:cluster-test,record:{"data":{"code":"0","message":"test send msg to Kafka cluster: 9"},"id":1569397368608,"sendDate":"2019-09-25 15:42:48"},with partitionId:1
KafkaProducerMsg(id=1569397547174, data={"code":"0","message":"test send msg to Kafka cluster: 9"}, sendDate=2019-09-25 15:45:47)
2019-09-25 15:45:47.178 INFO 2576 --- [ntainer#0-0-C-1] c.c.k.heart.service.impl.KafkaReceiver : ----------success in getting message from kafka topic:cluster-test,record:{"data":{"code":"0","message":"test send msg to Kafka cluster: 0"},"id":1569397368180,"sendDate":"2019-09-25 15:42:48"},with partitionId:0
KafkaProducerMsg(id=1569397547178, data={"code":"0","message":"test send msg to Kafka cluster: 0"}, sendDate=2019-09-25 15:45:47)
2019-09-25 15:45:47.178 INFO 2576 --- [ntainer#0-0-C-1] c.c.k.heart.service.impl.KafkaReceiver : ----------success in getting message from kafka topic:cluster-test,record:{"data":{"code":"0","message":"test send msg to Kafka cluster: 2"},"id":1569397368604,"sendDate":"2019-09-25 15:42:48"},with partitionId:0
KafkaProducerMsg(id=1569397547178, data={"code":"0","message":"test send msg to Kafka cluster: 2"}, sendDate=2019-09-25 15:45:47)
2019-09-25 15:45:47.178 INFO 2576 --- [ntainer#0-0-C-1] c.c.k.heart.service.impl.KafkaReceiver : ----------success in getting message from kafka topic:cluster-test,record:{"data":{"code":"0","message":"test send msg to Kafka cluster: 4"},"id":1569397368604,"sendDate":"2019-09-25 15:42:48"},with partitionId:0
KafkaProducerMsg(id=1569397547178, data={"code":"0","message":"test send msg to Kafka cluster: 4"}, sendDate=2019-09-25 15:45:47)
2019-09-25 15:45:47.178 INFO 2576 --- [ntainer#0-0-C-1] c.c.k.heart.service.impl.KafkaReceiver : ----------success in getting message from kafka topic:cluster-test,record:{"data":{"code":"0","message":"test send msg to Kafka cluster: 6"},"id":1569397368604,"sendDate":"2019-09-25 15:42:48"},with partitionId:0
KafkaProducerMsg(id=1569397547178, data={"code":"0","message":"test send msg to Kafka cluster: 6"}, sendDate=2019-09-25 15:45:47)
2019-09-25 15:45:47.178 INFO 2576 --- [ntainer#0-0-C-1] c.c.k.heart.service.impl.KafkaReceiver : ----------success in getting message from kafka topic:cluster-test,record:{"data":{"code":"0","message":"test send msg to Kafka cluster: 8"},"id":1569397368608,"sendDate":"2019-09-25 15:42:48"},with partitionId:0
KafkaProducerMsg(id=1569397547178, data={"code":"0","message":"test send msg to Kafka cluster: 8"}, sendDate=2019-09-25 15:45:47)
可以看到消費者收到了之前生產者發送到消息系統的數據。
至此，kafka集羣的測試結束。
如果你也剛開始學習kafka，希望這篇文章可以幫到你。