1下載並安裝JDK
參考jdk的安裝,需要jdk1.6+。
2 Kafka的安裝
2.1 下載kafka
下載地址:http://mirrors.hust.edu.cn/apache/kafka/0.10.0.1/kafka_2.11-0.10.0.1.tgz
2.2 解壓安裝包
$ tar zvxf kafka_2.11-0.10.0.1.tgz
2.3 複製解壓後的文件夾到安裝目錄
$ cp -r kafka_2.11-0.10.0.1 /software/
2.4 創建kafka安裝目錄的軟鏈接
$ ln -s /software/kafka_2.11-0.10.0.1/software/kafka
2.5 配置環節變量
修改/etc/profile文件內容:
export KAFKA_HOME=/software/kafka
export PATH=$KAFKA_HOME/bin:$FLUME_HOME/bin:$ZOOKEEPER_HOME/bin:$HBASE_HOME/bin:
$HIVE_HOME/bin:$JAVA_HOME/bin:$HADOOP_HOME/bin:$HADOOP_HOME/sbin:$PATH
export CLASSPATH=.:$KAFKA_HOME/libs/*:$FLUME_HOME/lib/*:$HBASE_HOME/lib/*:$JAVA_
HOME/lib/dt.jar:$JAVA_HOME/lib/tools.jar
2.6 啓動kafka
1)、啓動kafka前要先啓動zookeeper
$ zookeeper-server-start.sh/software/kafka/config/zookeeper.properties &
2)、啓動kafka
$ kafka-server-start.sh/software/kafka/config/server.properties &
3 體驗單節點Kafka
3.1 創建topic
$ kafka-topics.sh --create --zookeeperlocalhost:2181 --replication-factor 1 --partitions 1 --topic test
查詢創建的topic
$ kafka-topics.sh --list --zookeeperlocalhost:2181
3.2 發送消息
$ kafka-console-producer.sh --broker-listlocalhost:9092 --topic test
This is a message
This is another message
3.3 消費消息
$ kafka-console-consumer.sh --bootstrap-serverlocalhost:9092 --zookeeper localhost:2181 --topic test --from-beginning
This is a message
This is another message
4 搭建多broker的Kafka集羣
4.1 複製配置文件
$ cd /software/kafka/config/
$ cp server.properties s1.properties
$ cp server.properties s2.properties
$ cp server.properties s3.properties
4.2 修改配置文件
config/s1.properties:
broker.id=1
listeners=PLAINTEXT://:9093
log.dir=/tmp/kafka-logs-1
config/s2.properties:
broker.id=2
listeners=PLAINTEXT://:9094
log.dir=/tmp/kafka-logs-2
config/s3.properties:
broker.id=3
listeners=PLAINTEXT://:9095
log.dir=/tmp/kafka-logs-3
4.3 啓動kafka
$ kafka-server-start.sh/software/kafka/config/s1.properties &
$ kafka-server-start.sh/software/kafka/config/s2.properties &
$ kafka-server-start.sh /software/kafka/config/s3.properties&
4.4 創建topic
$ kafka-topics.sh --create--zookeeper localhost:2181 --replication-factor 3 --partitions 1 --topicmy-replicated-topic
查看topic描述
$ kafka-topics.sh --describe --zookeeper localhost:2181 --topicmy-replicated-topic
4.5 發送消息
$ kafka-console-producer.sh --broker-listlocalhost:9093 --topic my-replicated-topic
my test msg 1
my test msg 2
my test msg 3
4.6 消費消息
$ kafka-console-consumer.sh--zookeeper localhost:2181 --from-beginning --topic my-replicated-topic
5 容錯測試
5.1 找到s1進程
$ ps -ef| grep s1.properties
5.2 殺死進程s1
kill pid
5.3 查看主題描述
$>kafka-topics.sh --describe --zookeeperlocalhost:2181 --topic my-replicated-topic
Topic:my-replicated-topic PartitionCount:1 ReplicationFactor:3 Configs:
Topic: my-replicated-topic Partition: 0 Leader: 2 Replicas:1,2,0 Isr: 2,0
5.4 啓動消費者消費主題消息
$>kafka-console-consumer.sh--zookeeper localhost:2181 --from-beginning --topic my-replicated-topic
my test message 1
my test message 2
6 完全分佈式搭建
6.1 將kafka安裝文件夾分發到其餘集羣服務器
$ xsync kafka_2.11-0.10.0.1/
$ xsync kafka
6.2 分發環境變量
#xsync /etc/profile
6.3 修改server.properties屬性文件並分發
broker.id=203 #修改broker的id
log.dirs=/home/hadoop/kafka/logs #修改日誌的存放路徑
zookeeper.connect=s201:2181,s202:2181,s203:2181 #修改zookeeper的連接路徑
6.4 啓動Kafka集羣
到部署kafka每臺服務器上面啓動kafka集羣
$ kafka-server-start.sh/software/kafka/config/server.properties &
$ kafka-server-start.sh/software/kafka/config/server.properties &
$ kafka-server-start.sh/software/kafka/config/server.properties &