kafka 自定義分區分配和遷移

自定義分區分配和遷移

分區重新分配工具還可用於選擇性地將分區的副本移動到特定的代理集。當以這種方式使用時,假設用戶知道重新分配計劃並且不需要工具生成候選重新​​分配,有效地跳過 - 生成步驟並直接移動到--execute步驟

例如,以下示例將主題foo1的分區0移動到代理5,6,將主題foo2的分區1移動到代理2,3:

第一步是在json文件中手工製作自定義重新分配計劃:

1

2

> cat custom-reassignment.json

{"version":1,"partitions":[{"topic":"foo1","partition":0,"replicas":[5,6]},{"topic":"foo2","partition":1,"replicas":[2,3]}]}

然後,使用帶有--execute選項的json文件來啓動重新分配過程:

1

2

3

4

6

7

8

9

10

11

12

13

14

> bin/kafka-reassign-partitions.sh --zookeeper localhost:2181 --reassignment-json-file custom-reassignment.json --execute

Current partition replica assignment

 

{"version":1,

"partitions":[{"topic":"foo1","partition":0,"replicas":[1,2]},

              {"topic":"foo2","partition":1,"replicas":[3,4]}]

}

 

Save this to use as the --reassignment-json-file option during rollback

Successfully started reassignment of partitions

{"version":1,

"partitions":[{"topic":"foo1","partition":0,"replicas":[5,6]},

              {"topic":"foo2","partition":1,"replicas":[2,3]}]

}

--verify選項可與該工具一起使用,以檢查分區重新分配的狀態。請注意,相同的expand-cluster-reassignment.json(與--execute選項一起使用)應與--verify選項一起使用:

1

2

3

4

> bin/kafka-reassign-partitions.sh --zookeeper localhost:2181 --reassignment-json-file custom-reassignment.json --verify

Status of partition reassignment:

Reassignment of partition [foo1,0] completed successfully

Reassignment of partition [foo2,1] completed successfully

 

增加複製因子

增加現有分區的複製因子很容易。只需在自定義重新分配json文件中指定額外副本,並將其與--execute選項一起使用,以增加指定分區的複製因子。

例如,以下示例將主題foo的分區0的複製因子從1增加到3.在增加複製因子之前,分區的唯一副本存在於代理5上。作爲增加複製因子的一部分,我們將添加更多副本經紀人6和7。

第一步是在json文件中手工製作自定義重新分配計劃:

1

2

3

> cat increase-replication-factor.json

{"version":1,

"partitions":[{"topic":"foo","partition":0,"replicas":[5,6,7]}]}

然後,使用帶有--execute選項的json文件來啓動重新分配過程:

1

2

3

4

6

7

8

9

10

> bin/kafka-reassign-partitions.sh --zookeeper localhost:2181 --reassignment-json-file increase-replication-factor.json --execute

Current partition replica assignment

 

{"version":1,

"partitions":[{"topic":"foo","partition":0,"replicas":[5]}]}

 

Save this to use as the --reassignment-json-file option during rollback

Successfully started reassignment of partitions

{"version":1,

"partitions":[{"topic":"foo","partition":0,"replicas":[5,6,7]}]}

--verify選項可與該工具一起使用,以檢查分區重新分配的狀態。請注意,相同的increase-replication-factor.json(與--execute選項一起使用)應與--verify選項一起使用:

1

2

3

> bin/kafka-reassign-partitions.sh --zookeeper localhost:2181 --reassignment-json-file increase-replication-factor.json --verify

Status of partition reassignment:

Reassignment of partition [foo,0] completed successfully

您還可以使用kafka-topics工具驗證複製因子的增加:

1

2

3

> bin/kafka-topics.sh --zookeeper localhost:2181 --topic foo --describe

Topic:foo   PartitionCount:1    ReplicationFactor:3 Configs:

  Topic: foo    Partition: 0    Leader: 5   Replicas: 5,6,7 Isr: 5,6,7

發表評論
所有評論
還沒有人評論,想成為第一個評論的人麼? 請在上方評論欄輸入並且點擊發布.
相關文章