springboot kafka spring boot搭建單機集羣集成入門

搭建kafka (單機版)

使用虛擬機 + docker
參考 https://blog.csdn.net/qq_35394891/article/details/84349955
https://www.cnblogs.com/xiaohanlin/p/10078865.html

拉取鏡像 (kafka 依賴zookeeper ,所有兩個都要)

docker pull wurstmeister/zookeeper
docker pull wurstmeister/kafka:2.12-2.3.1

啓動

#先zookeeper
docker run -d --name zookeeper -p 2181:2181 -t wurstmeister/zookeeper
#此處需要修改KAFKA_ADVERTISED_HOST_NAME 爲實際IP地址 ,否則遠程訪問會出現 Connection to node {} ({}) could not be established. Broker may not be available. 此警告然後啓動失敗

docker run -d --name kafka -p 9092:9092 -e KAFKA_BROKER_ID=0 -e KAFKA_ZOOKEEPER_CONNECT=192.168.35.3:2181 -e KAFKA_ADVERTISED_LISTENERS=PLAINTEXT://192.168.35.3:9092 -e KAFKA_LISTENERS=PLAINTEXT://0.0.0.0:9092 -t wurstmeister/kafka:2.12-2.3.1

進入kafka 內部docker 測試

kafka 是容器id或容器別名 ,後面目錄改成自己對應版本,因爲此處拉取的是最新鏡像不保證相同

docker exec -it kafka /bin/bash
cd /opt/kafka_2.12-2.3.1/bin

創建topic

kafka-topics.sh --create --zookeeper zookeeper:2181 --replication-factor 1 --partitions 1 --topic mykafka

運行一個生產者

kafka-console-producer.sh --broker-list localhost:9092 --topic mykafka

如上再打開一個會話窗口 , 運行一個消費者

kafka-console-consumer.sh --bootstrap-server localhost:9092 --topic mykafka --from-beginning

在生產者端發送消息可以在消費者端看到消息

kafka 集羣+zookeeper 集羣 docker

首先安裝docker-compose 可以添加到path

curl -L https://github.com/docker/compose/releases/download/1.24.0/docker-compose-uname -s-uname -m -o /usr/local/bin/docker-compose
chmod +x /usr/local/bin/docker-compose
docker-compose --version

創建docker內部網絡,可以理解成容器中的host

docker network create --driver bridge --subnet 172.23.0.0/25 --gateway 172.23.0.1 zookeeper_network
這裏版本還是什麼關係我換成了官方版本
docker pull zookeeper:3.4.14
docker pull wurstmeister/kafka:2.12-2.3.1
docker pull hlebalbau/kafka-manager


version: '2'

services:
# 三個zookeeper
  zoo1:
    image:  zookeeper:3.4.14 # 鏡像
    restart: always # 重啓
    container_name: zoo1
    hostname: zoo1
    ports:
    - "2181:2181"
    environment:
      ZOO_MY_ID: 1 # id
      ZOO_SERVERS: server.1=zoo1:2888:3888 server.2=zoo2:2888:3888 server.3=zoo3:2888:3888
    networks:
      cc17223:
        ipv4_address: 172.23.0.2

  zoo2:
    image:  zookeeper:3.4.14
    restart: always
    container_name: zoo2
    hostname: zoo2
    ports:
    - "2182:2181"
    environment:
      ZOO_MY_ID: 2
      ZOO_SERVERS: server.1=zoo1:2888:3888 server.2=zoo2:2888:3888 server.3=zoo3:2888:3888
    networks:
      cc17223:
        ipv4_address: 172.23.0.3

  zoo3:
    image:  zookeeper:3.4.14
    restart: always
    container_name: zoo3
    hostname: zoo3
    ports:
    - "2183:2181"
    environment:
      ZOO_MY_ID: 3
      ZOO_SERVERS: server.1=zoo1:2888:3888 server.2=zoo2:2888:3888 server.3=zoo3:2888:3888
    networks:
      cc17223:
        ipv4_address: 172.23.0.4

  kafka1:
    image: wurstmeister/kafka:2.12-2.3.1 # 鏡像
    restart: always
    container_name: kafka1
    hostname: kafka1
    ports:
    - 9092:9092
    - 9999:9999
    environment:
      KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://192.168.35.3:9092 # 暴露在外的地址
      KAFKA_ADVERTISED_HOST_NAME: kafka1 # 
      KAFKA_HOST_NAME: kafka1
      KAFKA_ZOOKEEPER_CONNECT: zoo1:2181,zoo2:2181,zoo3:2181
      KAFKA_ADVERTISED_PORT: 9092 # 暴露在外的端口
      KAFKA_BROKER_ID: 0 # 
      KAFKA_LISTENERS: PLAINTEXT://0.0.0.0:9092
      JMX_PORT: 9999 # jmx
    volumes:
    - /etc/localtime:/etc/localtime
    links:
    - zoo1
    - zoo2
    - zoo3
    networks:
      cc17223:
        ipv4_address: 172.23.0.5

  kafka2:
    image: wurstmeister/kafka:2.12-2.3.1
    restart: always
    container_name: kafka2
    hostname: kafka2
    ports:
    - 9093:9092
    - 9998:9999
    environment:
      KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://192.168.35.3:9093
      KAFKA_ADVERTISED_HOST_NAME: kafka2
      KAFKA_HOST_NAME: kafka2
      KAFKA_ZOOKEEPER_CONNECT: zoo1:2181,zoo2:2181,zoo3:2181
      KAFKA_ADVERTISED_PORT: 9093
      KAFKA_BROKER_ID: 1
      KAFKA_LISTENERS: PLAINTEXT://0.0.0.0:9092
      JMX_PORT: 9999
    volumes:
    - /etc/localtime:/etc/localtime
    links:
    - zoo1
    - zoo2
    - zoo3
    networks:
      cc17223:
        ipv4_address: 172.23.0.6

  kafka3:
    image: wurstmeister/kafka:2.12-2.3.1
    restart: always
    container_name: kafka3
    hostname: kafka3
    ports:
    - 9094:9092
    - 9997:9999
    environment:
      KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://192.168.35.3:9094
      KAFKA_ADVERTISED_HOST_NAME: kafka3
      KAFKA_HOST_NAME: kafka3
      KAFKA_ZOOKEEPER_CONNECT: zoo1:2181,zoo2:2181,zoo3:2181
      KAFKA_ADVERTISED_PORT: 9094
      KAFKA_BROKER_ID: 2
      KAFKA_LISTENERS: PLAINTEXT://0.0.0.0:9092
      JMX_PORT: 9999
    volumes:
    - /etc/localtime:/etc/localtime
    links:
    - zoo1
    - zoo2
    - zoo3
    networks:
      cc17223:
        ipv4_address: 172.23.0.7

  kafka-manager:
    image: hlebalbau/kafka-manager
    restart: always
    container_name: kafka-manager
    hostname: kafka-manager
    ports:
    - 9000:9000
    links:
    - kafka1
    - kafka2
    - kafka3
    - zoo1
    - zoo2
    - zoo3
    environment:
      ZK_HOSTS: zoo1:2181,zoo2:2181,zoo3:2181
      KAFKA_BROKERS: kafka1:9092,kafka2:9092,kafka3:9092
      APPLICATION_SECRET: letmein
      KAFKA_MANAGER_AUTH_ENABLED: "true" # 開啓驗證
      KAFKA_MANAGER_USERNAME: "admin" # 用戶名
      KAFKA_MANAGER_PASSWORD: "admin" # 密碼
      KM_ARGS: -Djava.net.preferIPv4Stack=true
    networks:
      cc17223:
        ipv4_address: 172.23.0.8

networks:
  cc17223:
    external:
      name: zookeeper_network

啓動後首先觀察zookeeper 1/2/3 是否爲leader follower 如果是則成功,
docker logs zoo1/2/3
然後判斷kafka 1/2/3 看是否有連接拒絕或者異常報錯之類的,搭建時出現異常不斷重啓
docker logs kafka1/2/3
但是這裏好像因爲版本兼容 kafka 的管理界面 ,沒辦法添加集羣,升級版本之後又沒辦法啓動,所以暫時不使用web版本,還是使用kafka tools

項目代碼

此處參考 https://segmentfault.com/a/1190000015316875
此處我使用的是idea+lombok

首先創建一個spring boot 工程
修改配置文件我這裏使用的是yml結構的


spring:
  kafka:
    bootstrap-servers: 192.168.35.3:9092
    producer:
      retries: 0
      batch-size: 16384
      buffer-memory: 33554432
      key-serializer: org.apache.kafka.common.serialization.StringSerializer
      value-serializer: org.apache.kafka.common.serialization.StringSerializer
      properties:
        linger.ms: 1

    consumer:
      enable-auto-commit: false
      auto-commit-interval: 100ms
      key-deserializer: org.apache.kafka.common.serialization.StringDeserializer
      value-deserializer: org.apache.kafka.common.serialization.StringDeserializer
      properties:
        session.timeout.ms: 15000
      group-id: test-consume-group
    listener:
      missing-topics-fatal: false

kafka:
  topic:
    group-id: test-consume-group
    topic-name:
      - test

新增配置文件類

package com.example.springbootmqkafka.config;

import lombok.AllArgsConstructor;
import org.springframework.boot.autoconfigure.kafka.KafkaProperties;
import org.springframework.boot.context.properties.ConfigurationProperties;
import org.springframework.boot.context.properties.EnableConfigurationProperties;
import org.springframework.context.annotation.Bean;
import org.springframework.context.annotation.Configuration;
import org.springframework.kafka.annotation.EnableKafka;
import org.springframework.kafka.config.ConcurrentKafkaListenerContainerFactory;
import org.springframework.kafka.core.*;
import org.springframework.kafka.listener.ContainerProperties;

import java.io.Serializable;

/**
 * @author chen.chao
 * @version 1.0
 * @date 2020/4/2 16:13
 * @description
 */
@Configuration
@EnableConfigurationProperties(KafkaTopicConfiguration.KafkaTopicProperties.class)
public class KafkaTopicConfiguration {

    private final KafkaTopicProperties properties;

    /**
     * Topic 名稱
     */
    public static final String TOPIC_TEST = "test";

    public KafkaTopicConfiguration(KafkaTopicProperties properties) {
        this.properties = properties;
    }

    @Bean
    public String[] kafkaTopicName() {
        return properties.getTopicName();
    }

    @Bean
    public String topicGroupId() {
        return properties.getGroupId();
    }

    @ConfigurationProperties("kafka.topic")
    static class KafkaTopicProperties implements Serializable {

        private String groupId;
        private String[] topicName;

        public String getGroupId() {
            return groupId;
        }

        public void setGroupId(String groupId) {
            this.groupId = groupId;
        }

        public String[] getTopicName() {
            return topicName;
        }

        public void setTopicName(String[] topicName) {
            this.topicName = topicName;
        }
    }
} 

5. 新增業務方法
```java
package com.example.springbootmqkafka.service;

import lombok.extern.slf4j.Slf4j;
import org.apache.kafka.clients.consumer.ConsumerRecord;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.kafka.annotation.KafkaListener;
import org.springframework.kafka.core.KafkaTemplate;
import org.springframework.kafka.support.SendResult;
import org.springframework.stereotype.Service;
import org.springframework.util.concurrent.ListenableFuture;
import org.springframework.util.concurrent.ListenableFutureCallback;

/**
 * @author chen.chao
 * @version 1.0
 * @date 2020/4/2 16:13
 * @description
 */
@Slf4j
@Service
public class IndicatorService {

    private final KafkaTemplate<Integer, String> kafkaTemplate;

    /**
     * 注入KafkaTemplate
     * @param kafkaTemplate kafka模版類
     */
    @Autowired
    public IndicatorService(KafkaTemplate kafkaTemplate) {
        this.kafkaTemplate = kafkaTemplate;
    }
 
    public void sendMessage(String topic, String data) {
        log.info("kafka sendMessage start");
        ListenableFuture<SendResult<Integer, String>> future = kafkaTemplate.send(topic, data);
        future.addCallback(new ListenableFutureCallback<SendResult<Integer, String>>() {
            @Override
            public void onFailure(Throwable ex) {
                log.error("kafka sendMessage error, ex = {}, topic = {}, data = {}", ex, topic, data);
            }

            @Override
            public void onSuccess(SendResult<Integer, String> result) {
                log.info("kafka sendMessage success topic = {}, data = {}",topic, data);
            }
        });
        log.info("kafka sendMessage end");
    }
}

新增前臺入口

package com.example.springbootmqkafka.controller;

import com.alibaba.fastjson.JSON;
import com.example.springbootmqkafka.service.IndicatorService;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.stereotype.Controller;
import org.springframework.stereotype.Service;
import org.springframework.web.bind.annotation.*;

/**
 * @author chen.chao
 * @version 1.0
 * @date 2020/4/2 16:15
 * @description
 */
@RestController
public class ShopController {


    @Autowired
    private IndicatorService indicatorService;


    @GetMapping("/shop/order/{id}")
    public String shop(@PathVariable("id") Integer id , @RequestParam("name") String name){
        indicatorService.sendMessage("new_order", JSON.toJSONString(new Order(id,name)));
        return "預約下單成功,具體信息以短信提醒爲準";
    }

   static class Order{
        private Integer id;
        private String name;
        @Override
        public String toString() {
            return "Order{" +
                    "id=" + id +
                    ", name='" + name + '\'' +
                    '}';
        }

        public Order(Integer id, String name) {
            this.id = id;
            this.name = name;
        }

        public Integer getId() {
            return id;
        }

        public void setId(Integer id) {
            this.id = id;
        }

        public String getName() {
            return name;
        }

        public void setName(String name) {
            this.name = name;
        }
    }

}

消費者


package com.example.springbootmqkafka.consumer;

import com.example.springbootmqkafka.config.KafkaTopicConfiguration;
import lombok.extern.slf4j.Slf4j;
import org.apache.kafka.clients.consumer.ConsumerRecord;
import org.springframework.kafka.annotation.KafkaListener;
import org.springframework.stereotype.Component;

/**
 * @author chen.chao
 * @version 1.0
 * @date 2020/4/2 19:26
 * @description
 */
@Slf4j
@Component
public class OrderListener {


    @KafkaListener(topics = KafkaTopicConfiguration.TOPIC_TEST
                ,groupId = "test-consume-group"    )
    public void handleMessage(ConsumerRecord record) {
        try {
            String message = (String) record.value();
            log.info("收到消息: {}", message);
        } catch (Exception e) {
            log.error(e.getMessage(), e);
        }
    }

}

最後啓動 ,訪問http://localhost:8080/shop/order/2999?name=chenchao


2020-04-03 14:19:34.016  INFO 1120 --- [ntainer#0-0-C-1] c.e.s.consumer.OrderListener             : 收到消息: {"id":2999,"name":"chenchao"}

總結和可能出現的問題

選取版本的時候注意不要用最新的,因爲有可能最新的版本沒有對應的客戶端會出現無法消費消息的情況
搭建時如果出現失敗 , 最好把zookeeper 也重新刪掉 ,zookeeper 中有數據殘留,重新搭kafka的時候極可能步鄹正確無法得到預期結果

版本參考

https://spring.io/projects/spring-kafka

springboot kafka spring boot搭建單機集羣集成入門

搭建kafka (單機版)

kafka 集羣+zookeeper 集羣 docker

項目代碼

總結和可能出現的問題

ElementUI form 表單resetFields無法重置

Apollo 單環境與多環境搭建部署體驗

linux 服務器/虛擬機 centos7常用軟件安裝流程備忘

Elasticsearch入門到精通第二篇-查詢分析

springboot kafka spring boot搭建單機集羣集成入門

https://yachay.unat.edu.pe/blog/index.php?comment_area=format_blog&comment_component=blog&comment_co

linux以太網驅動總結

springboot kafka spring boot搭建 單機 集羣 集成入門

搭建kafka (單機版)

kafka 集羣+zookeeper 集羣 docker

項目代碼

總結和可能出現的問題

springboot kafka spring boot搭建單機集羣集成入門