Kafka Java客戶端使用

文章目錄

3. spring-kafka

1. 常用配置

生產者

配置	描述	類型	默認值
bootstrap.servers	用於建立與kafka集羣的連接，僅影響用於初始化的hosts，來發現全部的servers。格式：host1:port1,host2:port2,…，數量儘量不止一個，以防其中一個down了。	list
acks	生產者要求Leader在確認請求完成之前已收到的確認數。 acks = 0：如果設置爲零，那麼生產者將完全不等待服務器的任何確認。該記錄將立即添加到套接字緩衝區中並視爲已發送。在這種情況下，不能保證服務器已收到記錄，並且重試配置不會生效（因爲客戶端通常不會知道任何故障）。爲每個記錄提供的偏移量將始終設置爲-1。 acks = 1：這將意味着Leader會將記錄寫入其本地日誌，但會在不等待所有Followers的完全確認的情況下做出響應。在這種情況下，如果領導者在確認記錄後立即失敗，但在追隨者複製記錄之前失敗，則記錄將丟失。 acks=all 或 acks=-1：這意味着Leader將等待完整的同步副本集確認記錄。這保證了只要至少一個同步副本仍處於活動狀態，記錄就不會丟失。這是最有力的保證。	string	1
request.timeout.ms	客戶端等待請求響應的最長時間。如果超時之前仍未收到響應，則客戶端將在必要時重新發送請求，如果重試已用盡，則會使請求失敗。	int	30000
retries	請求失敗時重試次數。	int	2147483647
batch.size	批次大小，批次就是一組消息，這些消息屬於同一個主題和分區，把消息分成批次傳輸可以減少網絡開銷。	int	163834
linger.ms	發送時延，即發送消息時，生產者等待給定的延遲，才允許發送，以便可以聚合更多的消息。	long	0
buffer.memory	生產者可以用來緩衝等待發送到服務器的記錄的總內存字節，即生產者所管理的最大內存。如果記錄的發送速度超過了將記錄發送到服務器的速度，則生產者將阻塞max.block.ms，此後它將引發異常。	long	33554432
key.serializer	消息key序列化方式	class
value.serializer	消息本身的序列化方式	class

消費者

配置	描述	類型	默認值
bootstrap.servers	用於建立與kafka集羣的連接，僅影響用於初始化的hosts，來發現全部的servers。格式：host1:port1,host2:port2,…，數量儘量不止一個，以防其中一個down了。	list
group.id	消費者所屬消費組的唯一標識	string	null
enable.auto.commit	如果爲true，則消費者的offset將在後臺定期提交。	boolean	true
auto.commit.interval.ms	如果將enable.auto.commit設置爲true，則消費者offset自動提交給Kafka的頻率（以毫秒爲單位）。	int	5000
session.timeout.ms	檢測消費者是否失效的超時時間，該值必須在group.min.session.timeout.ms和group.max.session.timeout.ms允許範圍內。	int	10000
group.min.session.timeout.ms	檢測消費者是否失效的超時最小時間。	int	6000
group.max.session.timeout.ms	檢測消費者是否失效的超時最大時間。	int	1800000
key.deserializer	消息key反序列化方式	class
value.deserializer	消息value反序列化方式	class

2. kafka-clients

添加依賴

<dependency>
    <groupId>org.apache.kafka</groupId>
    <artifactId>kafka-clients</artifactId>
    <version>2.5.0</version>
</dependency>

2.1 生產者

同步發送消息

KafkaProducer是線程安全的，可以在多個線程之間共享生產者實例。

@Test
public void syncProducer() {
    Properties properties = new Properties();
    properties.put("bootstrap.servers", "192.168.110.40:9092,192.168.110.40:9093,192.168.110.40:9094");
	properties.put("acks", "all");
    properties.put("retries", 0);
    properties.put("batch.size", 16384);
    properties.put("buffer.memory", 33554432);
    properties.put("key.serializer", "org.apache.kafka.common.serialization.StringSerializer");
    properties.put("value.serializer", "org.apache.kafka.common.serialization.StringSerializer");

    KafkaProducer<String, String> kafkaProducer = new KafkaProducer<>(properties);

    // 同步發送消息，發送10條
    for (int i = 1; i <= 10; i++) {
        kafkaProducer.send(new ProducerRecord<>("java-client-test", "test,test,test " + i));
    }

    try{
        // 同步發送消息，可獲取RecordMetadata，服務器已確認記錄的元數據
        Future<RecordMetadata> future = kafkaProducer.send(new ProducerRecord<>("java-client-test", "test,test,test"));
        RecordMetadata recordMetadata = future.get();
        System.out.printf("RecordMetadata : topic = %s, partition = %d, offset = %d, toString() = %s\n",
                recordMetadata.topic(),recordMetadata.partition(),recordMetadata.offset(),recordMetadata.toString());
    } catch(Exception e) {
        // 連接錯誤、No Leader錯誤都可以通過重試解決
        // 消息太大這類錯誤kafkaProducer不會進行任何重試，直接拋出異常
        e.printStackTrace();
    }

    kafkaProducer.close();
}

異步發送消息

發送消息時，傳遞一個回調對象

@Test
public void producer() {
    Properties properties = new Properties();
    properties.put("bootstrap.servers", "192.168.110.40:9092,192.168.110.40:9093,192.168.110.40:9094");
	properties.put("acks", "all");
    properties.put("retries", 0);
    properties.put("batch.size", 16384);
    properties.put("buffer.memory", 33554432);
    properties.put("key.serializer", "org.apache.kafka.common.serialization.StringSerializer");
    properties.put("value.serializer", "org.apache.kafka.common.serialization.StringSerializer");

    KafkaProducer<String, String> kafkaProducer = new KafkaProducer<>(properties);

    // 異步發送消息，發送10條
    for (int i = 1; i <= 10; i++) {
        // 發送消息時，傳遞一個回調對象
        kafkaProducer.send(new ProducerRecord<>("java-client-test", "test,test,test " + i), new Callback() {
            @Override
            public void onCompletion(RecordMetadata recordMetadata, Exception e) {
                System.out.printf("ProducerCallback RecordMetadata : topic = %s, partition = %d, offset = %d, toString() = %s\n",
                        recordMetadata.topic(),recordMetadata.partition(),recordMetadata.offset(),recordMetadata.toString());

                // 如果Kafka返回一個錯誤，onCompletion方法拋出一個non null異常。
                if (e != null) {
                    // 對異常進行一些處理，這裏只是簡單打印出來
                    e.printStackTrace();
                }
            }
        });
    }

    kafkaProducer.close();
}

2.2 消費者

KafkaConsumer 是線程不安全的。

@Test
public void consumer() {
	Properties properties = new Properties();
	properties.put("bootstrap.servers", "192.168.110.40:9092,192.168.110.40:9093,192.168.110.40:9094");
	properties.put("group.id", "test");
	properties.put("enable.auto.commit", "true");
	properties.put("auto.commit.interval.ms", "1000");
	properties.put("session.timeout.ms", "30000");
	properties.put("key.deserializer","org.apache.kafka.common.serialization.StringDeserializer");
	properties.put("value.deserializer","org.apache.kafka.common.serialization.StringDeserializer");

	KafkaConsumer<String, String> consumer = new KafkaConsumer<>(properties);
	
	consumer.subscribe(Arrays.asList("java-client-test"));

	try{
		while (true) {
			// 間隔100ms去拉取數據
			ConsumerRecords<String, String> records = consumer.poll(Duration.ofMillis(100));
			for (ConsumerRecord<String, String> record : records){
				System.out.printf("消費 : topic = %s, partition = %d, offset = %d, value = %s\n",
						record.topic(),record.partition(),record.offset(), record.value());
			}
		}
	}finally {
		consumer.close();
	}
}

2.3 多線程

生產者
將生產者定義爲一個工具類

public class BaseProducer {

    private static KafkaProducer<String, String> kafkaProducer;

    static {
        Properties properties = new Properties();
        properties.put("bootstrap.servers", "192.168.110.40:9092,192.168.110.40:9093,192.168.110.40:9094");
        properties.put("acks", "all");
        properties.put("retries", 0);
        properties.put("batch.size", 16384);
        properties.put("buffer.memory", 33554432);
        properties.put("key.serializer", "org.apache.kafka.common.serialization.StringSerializer");
        properties.put("value.serializer", "org.apache.kafka.common.serialization.StringSerializer");

        kafkaProducer = new KafkaProducer<>(properties);

        Runtime.getRuntime().addShutdownHook(new Thread() {
            public void run() {
                BaseProducer.kafkaProducer.close();
            }
        });
    }

    public static void send(String content, String... topics){
        if (topics != null && topics.length > 0) {
            for (int i = 0;i < topics.length; i++) {
                System.out.println("send " + topics[i] + " message " + content);
                kafkaProducer.send(new ProducerRecord<>(topics[i], content));
            }
        }
    }
}

多線程消費者
實現Runnable，定義一個抽象的消費者，每個線程裏面都有一個KafkaConsumer。

public abstract class AbstractConsumer implements Runnable {

    protected KafkaConsumer<String, String> consumer;

    protected String topic;

    public AbstractConsumer(String topic) {
        this.topic = topic;
    }

    /**
     * 創建消費者，訂閱topic
     */
    private void connect() {
        Properties properties = new Properties();
        properties.put("bootstrap.servers", "192.168.110.40:9092,192.168.110.40:9093,192.168.110.40:9094");
        properties.put("group.id", "test-user");
        properties.put("enable.auto.commit", "false");
        properties.put("session.timeout.ms", "30000");
        properties.put("key.deserializer","org.apache.kafka.common.serialization.StringDeserializer");
        properties.put("value.deserializer","org.apache.kafka.common.serialization.StringDeserializer");

        consumer = new KafkaConsumer<>(properties);
        consumer.subscribe(Arrays.asList("topic-user"));
    }

    @Override
    public void run() {
        this.connect();

        while(true) {
            try {
                ConsumerRecords<String, String> consumerRecords = this.consumer.poll(Duration.ofMillis(500));
                Iterator iterator = consumerRecords.iterator();
                while(iterator.hasNext()) {
                    ConsumerRecord<String, String> item = (ConsumerRecord)iterator.next();
                    this.handler(item.value());
                }
            } catch (Exception e) {
                e.printStackTrace();
            }
        }
    }

    public void handler(String message) {
        this.consumer.commitAsync();
    }

}

繼承AbstractConsumer，實現handler()方法，在裏面進行消息的消費。

public class UserConsumer extends AbstractConsumer {

    public UserConsumer(String topic) {
        super(topic);
    }

    @Override
    public void handler(String message) {
        try {
            // 消費數據
            List<User> userList = JSON.parseObject(message, new TypeReference<List<User>>(){});

            if (!userList.isEmpty()) {
                for (User user : userList) {
                    System.out.println(user);
                }
            }

            this.consumer.commitAsync();
        }catch (Exception e) {
            this.consumer.commitAsync();
            e.printStackTrace();
        }

    }
}

消費者組，定義線程池

public class UserConsumerGroup {

    private List<Runnable> consumerThreadList = new ArrayList<>();

    public UserConsumerGroup() {
        consumerThreadList.add(new UserConsumer("topic-user"));
    }

    public void start() {
        ThreadFactory factory = r -> new Thread(r, "ThreadPool thread: UserConsumerGroup");
        // 核心線程池大小；最大線程池大小；線程最大空閒時間；時間單位；線程等待隊列；線程創建工廠
        ThreadPoolExecutor executor = new ThreadPoolExecutor(3, 6, 200, TimeUnit.MILLISECONDS,
                new LinkedBlockingQueue<Runnable>(3), factory);
        for (Runnable item : consumerThreadList) {
            executor.execute(item);
        }
        executor.shutdown();
    }
}

3. spring-kafka

SpringBoot 2.3.0，spring-kafka 2.5.0

<dependency>
    <groupId>org.springframework.kafka</groupId>
    <artifactId>spring-kafka</artifactId>
</dependency>

application.yml

spring:
  kafka:
    bootstrap-servers: 192.168.110.40:9092,192.168.110.40:9093,192.168.110.40:9094
    producer:
      acks: all
      retries: 0
      batch-size: 16384
      buffer-memory: 33554432
      key-serializer: org.apache.kafka.common.serialization.StringSerializer
      value-serializer: org.apache.kafka.common.serialization.StringSerializer
    consumer:
      group-id: spring-test
      enable-auto-commit: true
      key-deserializer: org.apache.kafka.common.serialization.StringDeserializer
      value-deserializer: org.apache.kafka.common.serialization.StringDeserializer
      properties:
        auto.commit.interval.ms: 1000
        session.timeout.ms: 30000

生產者

@Component
public class Producer {

    @Autowired
    private KafkaTemplate<String, String> kafkaTemplate;

    /**
     * 同步
     * @param topic
     */
    public void syncSend(String topic){
        for (int i = 1; i <= 10; i++) {
            ListenableFuture<SendResult<String, String>> future = kafkaTemplate.send(topic, "test,test,test " + i);

            try {
                SendResult<String, String> sendResult = future.get();
                ProducerRecord<String, String> producerRecord = sendResult.getProducerRecord();
                System.out.printf("Sync ProducerRecord : topic = %s, partition = %d, toString() = %s\n",
                        producerRecord.topic(),producerRecord.partition(),producerRecord.toString());

                RecordMetadata recordMetadata = sendResult.getRecordMetadata();
                System.out.printf("Sync RecordMetadata : topic = %s, partition = %d, offset = %d, toString() = %s\n",
                        recordMetadata.topic(),recordMetadata.partition(),recordMetadata.offset(),recordMetadata.toString());
            }catch (Exception e) {
                e.printStackTrace();
            }
        }
    }

    /**
     * 異步
     * @param topic
     */
    public void asyncSend(String topic){
        for (int i = 1; i <= 10; i++) {
            ListenableFuture<SendResult<String, String>> future = kafkaTemplate.send(topic, "test,test,test " + i);
            future.addCallback(new ListenableFutureCallback<SendResult<String, String>>() {
                @Override
                public void onFailure(Throwable ex) {
                    ex.printStackTrace();
                }

                @Override
                public void onSuccess(SendResult<String, String> result) {
                    ProducerRecord<String, String> producerRecord = result.getProducerRecord();
                    System.out.printf("Async ProducerRecord : topic = %s, partition = %d, toString() = %s\n",
                            producerRecord.topic(),producerRecord.partition(),producerRecord.toString());

                    RecordMetadata recordMetadata = result.getRecordMetadata();
                    System.out.printf("Async RecordMetadata : topic = %s, partition = %d, offset = %d, toString() = %s\n",
                            recordMetadata.topic(),recordMetadata.partition(),recordMetadata.offset(),recordMetadata.toString());
                }
            });
        }
    }

}

消費者，這個併發3，啓動3個線程

@Component
public class Consumer {
	@KafkaListener(topics = "spring-topic", concurrency = "3")
	public void listen (ConsumerRecord<?, ?> record) {
		System.out.printf("消費 : topic = %s, partition = %d, offset = %d, value = %s\n",
				record.topic(),record.partition(),record.offset(),record.value());
	}
}

參考：
Kafka 官方文檔
 kafka-clients介紹
 kafka-clients API文檔 KafkaConsumer
創建多線程Kafka消費者
 spring-kafka 官方文檔

Kafka Java客戶端使用

文章目錄

1. 常用配置

2. kafka-clients

2.1 生產者

2.2 消費者

2.3 多線程

3. spring-kafka

MySQL 知識點整理

Java Calendar 使用

SpringMVC API統一JSON格式返回

Spring Session + Redis 使用

Java LocalDateTime 使用

Mac下配置sublime實現LaTeX

https://yachay.unat.edu.pe/blog/index.php?comment_area=format_blog&comment_component=blog&comment_co

linux以太網驅動總結