Flink基礎系列13-Source之從Kafka讀取 一.環境介紹 二.代碼 三.打包代碼並執行 參考:

一.環境介紹

環境介紹
本地測試環境搭建了CDH 6.3集羣，集成了Kafka和Flink

Maven配置
從官網找到的maven配置如下:

<dependency>
  <groupId>org.apache.flink</groupId>
  <artifactId>flink-connector-kafka_2.11</artifactId>
  <version>1.9.0</version>
</dependency>

二.代碼

kafka_producer

package org.example;

/**
 * @author  只是甲
 * @date    2021-08-30
 * @remark  kafka生產者
 */

import java.util.Properties;
import java.util.Random;

import org.apache.kafka.clients.producer.KafkaProducer;
import org.apache.kafka.clients.producer.ProducerConfig;
import org.apache.kafka.clients.producer.ProducerRecord;
import org.apache.kafka.common.serialization.StringSerializer;


public class kafka_producer {
    public static String topic = "sensor3";//定義主題

    public static void main(String[] args) throws Exception {
        Properties p = new Properties();
        p.put(ProducerConfig.BOOTSTRAP_SERVERS_CONFIG, "10.31.1.124:9092,10.31.1.125:9092,10.31.1.126:9092");//kafka地址，多個地址用逗號分割
        p.put(ProducerConfig.KEY_SERIALIZER_CLASS_CONFIG, StringSerializer.class);
        p.put(ProducerConfig.VALUE_SERIALIZER_CLASS_CONFIG, StringSerializer.class);
        KafkaProducer<String, String> kafkaProducer = new KafkaProducer<>(p);

        try {
            while (true) {
                String msg = "Hello," + new Random().nextInt(100);
                ProducerRecord<String, String> record = new ProducerRecord<String, String>(topic, msg);
                kafkaProducer.send(record);
                System.out.println("消息發送成功:" + msg);
                Thread.sleep(10000);
            }
        } finally {
            kafkaProducer.close();
        }

    }
}

SourceTest3_Kafka

package org.example;

/*
  @author  只是甲
 * @date    2021-08-30
 * @remark  Flink Souce之Kafka
 */

import org.apache.flink.api.common.serialization.SimpleStringSchema;
import org.apache.flink.streaming.api.datastream.DataStream;
import org.apache.flink.streaming.api.environment.StreamExecutionEnvironment;
import org.apache.flink.streaming.connectors.kafka.FlinkKafkaConsumer;
import java.util.Properties;

import org.apache.flink.streaming.connectors.kafka.FlinkKafkaConsumerBase;


public class SourceTest3_Kafka {
    public static void main(String[] args) throws Exception{
        StreamExecutionEnvironment env = StreamExecutionEnvironment.getExecutionEnvironment();
        env.setParallelism(1);

        Properties properties = new Properties();
        properties.setProperty("bootstrap.servers", "10.31.1.124:9092,10.31.1.125:9092,10.31.1.126:9092");
        properties.setProperty("group.id", "consumer-group");
        properties.setProperty("key.deserializer", "org.apache.kafka.common.serialization.StringDeserializer");
        properties.setProperty("value.deserializer", "org.apache.kafka.common.serialization.StringDeserializer");
        properties.setProperty("auto.offset.reset", "latest");

        // 從Kafka讀取數據
        DataStream<String> dataStream = env.addSource( new FlinkKafkaConsumer<String>("sensor3", new SimpleStringSchema(), properties));


        // 打印輸出
        dataStream.print();

        env.execute();
    }

}

三.打包代碼並執行

運行命令:

flink run -m yarn-cluster -c org.example.SourceTest3_Kafka FlinkStudy-1.0-SNAPSHOT.jar

本地運行截圖:

遠程執行:
因爲遠程執行沒有輸出，這個地方不太好顯示，略過

參考:

1. https://ci.apache.org/projects/flink/flink-docs-release-1.9/zh/dev/connectors/kafka.html
2. https://blog.csdn.net/u013411339/article/details/103018379