kafka 執行topic遷移到新建的分區(partition)

創建新的topic
kafka-console-producer.sh --broker-list 127.0.0.1:9092 --topic test1
kafka-console-producer.sh --broker-list 127.0.0.1:9092 --topic test2
kafka-console-producer.sh --broker-list 127.0.0.1:9092 --topic test3
kafka-console-producer.sh --broker-list 127.0.0.1:9092 --topic test4
查看所有topic列表
./kafka-topics.sh --zookeeper 127.0.0.1:2181 --list
增加topic分區數(越來越大,沒找到刪除)
./kafka-topics.sh --zookeeper 127.0.0.1:2181 --alter --topic test --partitions 4

1.創建遷移topic.json文件
{
"topics": [
{ "topic": "test5"},
{ "topic": "test6"},
{ "topic": "test7"},
{ "topic": "test8"},
{ "topic": "test9"}
],
"version": 1
}
2.遷移命令

# ./kafka-reassign-partitions.sh --zookeeper 127.0.0.1:2181 --topics-to-move-json-file topic.json --broker-list "2,3" --generate

[2018-04-02 16:50:49,031] WARN Connected to an old server; r-o mode will be unavailable (org.apache.zookeeper.ClientCnxnSocket)

打印出來的是分配規則
Current partition replica assignment
{"version":1,"partitions":[{"topic":"test7","partition":0,"replicas":[1001]},{"topic":"test9","partition":0,"replicas":[1001]},{"topic":"test8","partition":0,"replicas":[1001]},{"topic":"test5","partition":0,"replicas":[1001]},{"topic":"test6","partition":0,"replicas":[1001]}]}
Proposed partition reassignment configuration
{"version":1,"partitions":[{"topic":"test7","partition":0,"replicas":[3]},{"topic":"test9","partition":0,"replicas":[2]},{"topic":"test8","partition":0,"replicas":[3]},{"topic":"test5","partition":0,"replicas":[2]},{"topic":"test6","partition":0,"replicas":[3]}]}

註釋:Current partition replica assignment表示當前的消息存儲狀況 。
Proposed partition reassignment configuration表示遷移後的消息存儲狀況(複製生成topicExec.json文件,用於分區命令)


3.生成topicExec.json文件,把分配規則複製進去
{"version":1,"partitions":[{"topic":"test7","partition":0,"replicas":[1001]},{"topic":"test9","partition":0,"replicas":[1001]},{"topic":"test8","partition":0,"replicas":[1001]},{"topic":"test5","partition":0,"replicas":[1001]},{"topic":"test6","partition":0,"replicas":[1001]}]}
4.執行命令,開始分區重新分配
# ./kafka-reassign-partitions.sh --zookeeper 127.0.0.1:2181 --reassignment-json-file topicExec.json -execute
執行結果:
Current partition replica assignment
{"version":1,"partitions":[{"topic":"test7","partition":0,"replicas":[1001]},{"topic":"test9","partition":0,"replicas":[1001]},{"topic":"test8","partition":0,"replicas":[1001]},{"topic":"test5","partition":0,"replicas":[1001]},{"topic":"test6","partition":0,"replicas":[1001]}]}
Save this to use as the --reassignment-json-file option during rollback
Successfully started reassignment of partitions {"version":1,"partitions":[{"topic":"test7","partition":0,"replicas":[0]},{"topic":"test9","partition":0,"replicas":[2]},{"topic":"test6","partition":0,"replicas":[0]},{"topic":"test5","partition":0,"replicas":[3]},{"topic":"test8","partition":0,"replicas":[1]}]}

5.驗證分配結果 completed successfully表示成功
kafka-reassign-partitions.sh --zookeeper 127.0.0.1:2181 --reassignment-json-file topicExec.json -verify
執行結果:
Status of partition reassignment:
Reassignment of partition [test7,0] completed successfully
Reassignment of partition [test9,2] completed successfully
Reassignment of partition [test6,0] completed successfully
Reassignment of partition [test5,3] completed successfully
Reassignment of partition [test8,1] completed successfully





發表評論
所有評論
還沒有人評論,想成為第一個評論的人麼? 請在上方評論欄輸入並且點擊發布.
相關文章