Linux安裝kafka 分佈式

下載kafka，地址：http://kafka.apache.org/downloads.html選擇Binary downloads下載

上傳到linux並解壓

$ tar -zxvf kafka_2.11-0.8.2.1.tgz

修改server.properties文件

$ vi config/server.properties

 

broker.id=1  (每臺機器不能一樣，我的另外兩臺機器寫的是2，3)
host.name=192.168.10.117 (寫服務器ip)
zookeeper.connect= 192.168.10.117:2181, 192.168.10.114:2181, 192.168.10.115:2181

啓動zookeeper (安裝略…..)

在每臺機器上啓動kafka 啓動腳本在bin目錄下

$ cd kafka_2.11-0.8.2.1/bin/
$ ./kafka-server-start.sh ../config/server.properties &

創建topic：創建topic：創建topic：

./kafka-topics.sh --create --zookeeper localhost:2181 --replication-factor 3 --partitions 1 --topic mykafka

查看Topic：

./kafka-topics.sh --list --zookeeper localhost:2181

 

單機模式收發消息

發送消息：

./kafka-console-producer.sh --broker-list localhost:9092 --topic mykafka

接收消息：

./kafka-console-consumer.sh --zookeeper localhost:2181 --topic mykafka --from-beginning

發送消息和接收消息是在兩個窗口

在發送消息窗口輸入字符串，在接收消息窗口查看

 

分佈式收發消息

發送消息：

./kafka-console-producer.sh --broker-list 192.168.10.114:9092 --topic mykafka

接收消息：

./kafka-console-consumer.sh --zookeeper 192.168.10.114:2181 --topic mykafka --from-beginning

 

寫--from-beginning會顯示歷史消息，如果只想顯示最新的可以不寫

停止服務：

./kafka-server-stop.sh

 

報錯和解決：

[2015-06-16 11:24:13,015] ERROR Failed to send requests for topics mykafka with correlation ids in [0,8] (kafka.producer.async.DefaultEventHandler)
[2015-06-16 11:24:13,015] ERROR Error in handling batch of 1 events (kafka.producer.async.ProducerSendThread)
kafka.common.FailedToSendMessageException: Failed to send messages after 3 tries.
	at kafka.producer.async.DefaultEventHandler.handle(DefaultEventHandler.scala:90)
	at kafka.producer.async.ProducerSendThread.tryToHandle(ProducerSendThread.scala:105)
	at kafka.producer.async.ProducerSendThread$$anonfun$processEvents$3.apply(ProducerSendThread.scala:88)
	at kafka.producer.async.ProducerSendThread$$anonfun$processEvents$3.apply(ProducerSendThread.scala:68)
	at scala.collection.immutable.Stream.foreach(Stream.scala:594)
	at kafka.producer.async.ProducerSendThread.processEvents(ProducerSendThread.scala:67)
	at kafka.producer.async.ProducerSendThread.run(ProducerSendThread.scala:45)

 這個錯誤是config/server.properties的host.name寫的不對，可能是前面的“#”沒有去掉或是寫的主機名稱，改成服務器ip地址就可以了，如果改成localhost單機模式不會有問題，但分佈式的時候會報下面錯誤。

 

[2015-06-16 14:20:59,519] WARN Fetching topic metadata with correlation id 9 for topics [Set(mykafka)] from broker [id:0,host:192.168.10.114,port:9092] failed (kafka.client.ClientUtils$)
java.nio.channels.ClosedChannelException
	at kafka.network.BlockingChannel.send(BlockingChannel.scala:100)
	at kafka.producer.SyncProducer.liftedTree1$1(SyncProducer.scala:73)
	at kafka.producer.SyncProducer.kafka$producer$SyncProducer$$doSend(SyncProducer.scala:72)
	at kafka.producer.SyncProducer.send(SyncProducer.scala:113)
	at kafka.client.ClientUtils$.fetchTopicMetadata(ClientUtils.scala:58)
	at kafka.producer.BrokerPartitionInfo.updateInfo(BrokerPartitionInfo.scala:82)
	at kafka.producer.async.DefaultEventHandler$$anonfun$handle$1.apply$mcV$sp(DefaultEventHandler.scala:67)
	at kafka.utils.Utils$.swallow(Utils.scala:172)
	at kafka.utils.Logging$class.swallowError(Logging.scala:106)
	at kafka.utils.Utils$.swallowError(Utils.scala:45)
	at kafka.producer.async.DefaultEventHandler.handle(DefaultEventHandler.scala:67)
	at kafka.producer.async.ProducerSendThread.tryToHandle(ProducerSendThread.scala:105)
	at kafka.producer.async.ProducerSendThread$$anonfun$processEvents$3.apply(ProducerSendThread.scala:88)
	at kafka.producer.async.ProducerSendThread$$anonfun$processEvents$3.apply(ProducerSendThread.scala:68)
	at scala.collection.immutable.Stream.foreach(Stream.scala:594)
	at kafka.producer.async.ProducerSendThread.processEvents(ProducerSendThread.scala:67)
	at kafka.producer.async.ProducerSendThread.run(ProducerSendThread.scala:45)
[2015-06-16 14:20:59,520] ERROR fetching topic metadata for topics [Set(mykafka)] from broker [ArrayBuffer(id:0,host:192.168.10.114,port:9092)] failed (kafka.utils.Utils$)
kafka.common.KafkaException: fetching topic metadata for topics [Set(mykafka)] from broker [ArrayBuffer(id:0,host:192.168.10.114,port:9092)] failed
	at kafka.client.ClientUtils$.fetchTopicMetadata(ClientUtils.scala:72)
	at kafka.producer.BrokerPartitionInfo.updateInfo(BrokerPartitionInfo.scala:82)
	at kafka.producer.async.DefaultEventHandler$$anonfun$handle$1.apply$mcV$sp(DefaultEventHandler.scala:67)
	at kafka.utils.Utils$.swallow(Utils.scala:172)
	at kafka.utils.Logging$class.swallowError(Logging.scala:106)
	at kafka.utils.Utils$.swallowError(Utils.scala:45)
	at kafka.producer.async.DefaultEventHandler.handle(DefaultEventHandler.scala:67)
	at kafka.producer.async.ProducerSendThread.tryToHandle(ProducerSendThread.scala:105)
	at kafka.producer.async.ProducerSendThread$$anonfun$processEvents$3.apply(ProducerSendThread.scala:88)
	at kafka.producer.async.ProducerSendThread$$anonfun$processEvents$3.apply(ProducerSendThread.scala:68)
	at scala.collection.immutable.Stream.foreach(Stream.scala:594)
	at kafka.producer.async.ProducerSendThread.processEvents(ProducerSendThread.scala:67)
	at kafka.producer.async.ProducerSendThread.run(ProducerSendThread.scala:45)
Caused by: java.nio.channels.ClosedChannelException
	at kafka.network.BlockingChannel.send(BlockingChannel.scala:100)
	at kafka.producer.SyncProducer.liftedTree1$1(SyncProducer.scala:73)
	at kafka.producer.SyncProducer.kafka$producer$SyncProducer$$doSend(SyncProducer.scala:72)
	at kafka.producer.SyncProducer.send(SyncProducer.scala:113)
	at kafka.client.ClientUtils$.fetchTopicMetadata(ClientUtils.scala:58)
	... 12 more