在producer/consumer運行時刪除topic會怎麼樣

broker端有個參數（kafka_2.13-3.2.1/config/server.properties），auto.create.topics.enable，默認爲true。意思是，當生產者、消費者讀寫一個不存在的topic時，是否自動創建該topic。

我們使用kafka自帶的腳本（kafka_2.13-3.2.1/bin）進行測試。爲了方便使用，我把這些腳本封裝了一下，放在kafka-mate/scripts at master · whuwangyong/kafka-mate (github.com)。

1 auto.create.topics.enable=true

1.1 消費不存在的topic

（1）初始化環境，然後使用list-topics.sh，可見沒有任何topic：

wy@ship:~/dev/kafka-mate/scripts$ ./reset.sh 
kill kafka and zookeeper...
rm -rf /tmp/kafka-logs /tmp/zookeeper
start...
zookeeper started
waiting kafka starting...
waiting kafka starting...
waiting kafka starting...
waiting kafka starting...
waiting kafka starting...
kafka started
wy@ship:~/dev/kafka-mate/scripts$ ./list-topics.sh

（2）消費test topic，可見打出了一堆警告（行數不確定，2~10行）：LEADER_NOT_AVAILABLE

wy@ship:~/dev/kafka-mate/scripts$ ./consume-from.sh test
[2022-09-18 02:52:16,148] WARN [Consumer clientId=console-consumer, groupId=console-consumer-27797] Error while fetching metadata with correlation id 2 : {test=LEADER_NOT_AVAILABLE} (org.apache.kafka.clients.NetworkClient)
[2022-09-18 02:52:16,256] WARN [Consumer clientId=console-consumer, groupId=console-consumer-27797] Error while fetching metadata with correlation id 4 : {test=LEADER_NOT_AVAILABLE} (org.apache.kafka.clients.NetworkClient)
[2022-09-18 02:52:16,403] WARN [Consumer clientId=console-consumer, groupId=console-consumer-27797] Error while fetching metadata with correlation id 6 : {test=LEADER_NOT_AVAILABLE} (org.apache.kafka.clients.NetworkClient)
[2022-09-18 02:52:16,559] WARN [Consumer clientId=console-consumer, groupId=console-consumer-27797] Error while fetching metadata with correlation id 8 : {test=LEADER_NOT_AVAILABLE} (org.apache.kafka.clients.NetworkClient)
[2022-09-18 02:52:16,668] WARN [Consumer clientId=console-consumer, groupId=console-consumer-27797] Error while fetching metadata with correlation id 10 : {test=LEADER_NOT_AVAILABLE} (org.apache.kafka.clients.NetworkClient)
[2022-09-18 02:52:16,774] WARN [Consumer clientId=console-consumer, groupId=console-consumer-27797] Error while fetching metadata with correlation id 12 : {test=LEADER_NOT_AVAILABLE} (org.apache.kafka.clients.NetworkClient)

（3）此時新開一個終端再次list-topics.sh，會發現test topic已經被kafka自動創建了。（如果該topic沒有的話，上述消費線程將會一直打印Error while fetching metadata with correlation id ...）

（4）那麼消費線程是否正常？此時通過./produce-to.sh往test topic發消息，發現consumer能收到，說明該消費者仍正常工作

1.2 生產者寫不存在的topic

（1）清理環境

wy@ship:~/dev/kafka-mate/scripts$ ./reset.sh 
kill kafka and zookeeper...
rm -rf /tmp/kafka-logs /tmp/zookeeper
start...
zookeeper started
waiting kafka starting...
waiting kafka starting...
waiting kafka starting...
waiting kafka starting...
waiting kafka starting...
waiting kafka starting...
kafka started

（2）往不存在的testtopic發送消息：

wy@ship:~/dev/kafka-mate/scripts$ ./produce-to.sh test
>1
[2022-09-18 03:08:35,111] WARN [Producer clientId=console-producer] Error while fetching metadata with correlation id 4 : {test=LEADER_NOT_AVAILABLE} (org.apache.kafka.clients.NetworkClient)
[2022-09-18 03:08:35,220] WARN [Producer clientId=console-producer] Error while fetching metadata with correlation id 5 : {test=LEADER_NOT_AVAILABLE} (org.apache.kafka.clients.NetworkClient)
[2022-09-18 03:08:35,328] WARN [Producer clientId=console-producer] Error while fetching metadata with correlation id 6 : {test=LEADER_NOT_AVAILABLE} (org.apache.kafka.clients.NetworkClient)
>2
>

注意細節，敲完./produce-to.sh test回車後，未報警告。只有當輸入1，然後回車，在真正發送消息時，纔會打印Error while fetching metadata with correlation id ...

（3）驗證消息是否發送成功：

wy@ship:~/dev/kafka-mate/scripts$ ./consume-from.sh test
1
2

可見發送成功了。

1.3 消費過程中刪除topic

wy@ship:~/dev/kafka-mate/scripts$ ./consume-from.sh test
1
2
[2022-09-18 03:22:51,090] WARN [Consumer clientId=console-consumer, groupId=console-consumer-14043] Received unknown topic or partition error in fetch for partition test-0 (org.apache.kafka.clients.consumer.internals.Fetcher)
[2022-09-18 03:22:51,101] WARN [Consumer clientId=console-consumer, groupId=console-consumer-14043] Error while fetching metadata with correlation id 141 : {test=LEADER_NOT_AVAILABLE} (org.apache.kafka.clients.NetworkClient)
[2022-09-18 03:22:51,250] WARN [Consumer clientId=console-consumer, groupId=console-consumer-14043] Error while fetching metadata with correlation id 143 : {test=LEADER_NOT_AVAILABLE} (org.apache.kafka.clients.NetworkClient)
[2022-09-18 03:22:51,357] WARN [Consumer clientId=console-consumer, groupId=console-consumer-14043] Error while fetching metadata with correlation id 145 : {test=LEADER_NOT_AVAILABLE} (org.apache.kafka.clients.NetworkClient)

topic刪除後，consumer立即打印了幾行警告日誌。

1.4 生產過程中刪除topic

刪除topic時，producer暫時沒反應。在下次send消息時，纔會打印警告日誌。而且，通過再起一個consumer進行驗證，發現這個消息是發送成功了。

>11
>[2022-09-18 03:26:09,786] WARN [Producer clientId=console-producer] Got error produce response with correlation id 13 on topic-partition test-0, retrying (2 attempts left). Error: UNKNOWN_TOPIC_OR_PARTITION (org.apache.kafka.clients.producer.internals.Sender)
[2022-09-18 03:26:09,787] WARN [Producer clientId=console-producer] Received unknown topic or partition error in produce request on partition test-0. The topic-partition may not exist or the user may not have Describe access to it (org.apache.kafka.clients.producer.internals.Sender)
[2022-09-18 03:26:09,810] WARN [Producer clientId=console-producer] Error while fetching metadata with correlation id 14 : {test=LEADER_NOT_AVAILABLE} (org.apache.kafka.clients.NetworkClient)

1.5 警告爲什麼有多行？

猜測：不論是producer.send()還是consumer.poll，當topic不存在時，會進行多次嘗試，間隔100ms，每次失敗就打印一行日誌。另一邊，broker在創建該topic。當topic創建好後，producer或consumer可以正常使用該topic，則不再打印警告日誌。

那麼，topic不存在時，producer或consumer會嘗試多少次，或者嘗試多久呢？下面，將auto.create.topics.enable置爲false，進行測試。

2 auto.create.topics.enable=false

2.1 消費不存在的topic

消費者將持續不斷的嘗試，我測試一小時後還在嘗試。這估計是因爲consumer.poll邏輯寫在while(true)裏面的。

2.2 生產者寫不存在的topic

生產者在嘗試6分鐘後停止。

kafka自帶測試腳本的生產者和消費者的代碼在kafka-3.2.1-src\core\src\main\scala\kafka\tools\ConsoleConsumer.scala 和 ConsoleProducer.scala。可見ConsoleProducer.scala裏面是有3次重試的。

我自己寫的測試代碼，1分鐘後停止嘗試。

2.3 消費過程中刪除topic

由於不會自動創建topic，刪了就沒有了，consumer將一直包括。

2.4 生產過程中刪除topic

同上。

3 是否啓用自動創建topic

個人建議，不要啓用。topic應該嚴格管理，是運維操作。類似數據庫裏面的表，不能說讀寫表時，表不存在我就自動創建吧。

再者，自動創建的topic，其分區數爲1，可能並不符合預期。不應該在不符合預期的topic上運行，應該儘早報錯，failfast。

附：測試代碼

@Service
public class KafkaMate {

    private static final String SERVER = "192.168.191.128:9092";

    @PostConstruct
    public void init() {
//        consume();
        produce();
    }

    Properties consumerProperties() {
        Properties props = new Properties();
        props.setProperty("bootstrap.servers", SERVER);
        props.setProperty("group.id", "test-g-1");
        props.setProperty("enable.auto.commit", "true");
        props.setProperty("auto.commit.interval.ms", "1000");
        props.setProperty("key.deserializer", "org.apache.kafka.common.serialization.StringDeserializer");
        props.setProperty("value.deserializer", "org.apache.kafka.common.serialization.StringDeserializer");
        return props;
    }

    Properties producerProperties() {
        Properties props = new Properties();
        props.setProperty("bootstrap.servers", SERVER);
        props.put("linger.ms", 1);
        props.put("key.serializer", "org.apache.kafka.common.serialization.StringSerializer");
        props.put("value.serializer", "org.apache.kafka.common.serialization.StringSerializer");
        return props;
    }

    void consume() {
        try (KafkaConsumer<String, String> consumer = new KafkaConsumer<>(consumerProperties())) {
            consumer.subscribe(Collections.singleton("test"));
            while (true) {
                ConsumerRecords<String, String> records = consumer.poll(Duration.ofMillis(1000));
            }
        }
    }

    void produce() {
        try (KafkaProducer<String, String> producer = new KafkaProducer<>(producerProperties())) {
            ProducerRecord<String, String> record = new ProducerRecord<>("test", "hello" + System.currentTimeMillis());
            producer.send(record);
        }
    }

}