kafka学习四：开发producer

procedure就是产生消息并将消息发布至broker的应用。

producer连接至任意的活动节点并请求获取某个topic的partition的leader元数据。这样producer可以直接将信息发给该partition的lead broker。

出于效率考虑，producer可以分批发布消息，但是只能在异步模式下。异步模式下，producer可以配置queue.time或`batch.size这两个参数其中一个来指定在一定数量或一定时间后批量发布消息。消息会在producer这一端积累，然后在一次请求中批量发布至broker。因此异步模式也带来了消息丢失的风险，当producer崩溃时，在内存中的积累的尚未发布的消息就丢失了。

对于异步模式的producer，回调函数可以用来注册捕捉错误的处理器。

Java producer API

Producer
Kafka提供了类kafka.javaapi.producer.Producer（class Producer<K, V>）用于向一个或多个topic创建消息，还可以制定消息的partition。K和V分别指定partiton key和消息的值的类型。
KeyedMessage
类kafka.producer.KeyedMessage的构造函数参数为topic名称、partition key和消息值：
```
class KeyedMessage[K,V](val topic: String, val key: K,  val message: V)
```
这里，K和V仍然分别是指定partiton key和消息的值的类型，topic始终是String类型的。
ProducerConfig
类kafka.producer.ProducerConfig封装了与broker建立连接所需要的参数，如borker list、partition类、消息序列化类、partiton key。

producer的API封装了同步模式下producer的实现，异步模式下producer基于producer.type。例如，异步模式的kafka.producer.Producer负责消息序列化和发送之前的数据缓存。在内部，kafka.producer.async.ProducerSendThread的实例从队列中读出该批次的消息，kafka.producer.EventHandler序列化并发送数据。配置event.handler这个参数还可以自定义处理器。

一个简单的Java producer

接下来，我们写一个类SimpleProducer来创建指定的topic对应的消息，并使用默认的partition。

1.引入以下类：

import kafka.javaapi.producer.Producer;
import kafka.producer.KeyedMessage;
import kafka.producer.ProducerConfig;

2.定义属性：

Properties props = new Properties();
props.put("metadata.broker.list", "localhost:9092, localhost:9093, localhost:9094");
props.put("serializer.class", "kafka.serializer.StringEncoder");
props.put("request.required.acks", "1");
ProducerConfig config = new ProducerConfig(props);  
Producer<String, String> producer = new Producer<String, String>(config);

看一下代码中提到的属性：

metadata.broker.list：该属性指定producer要连接的broker（格式为[<node:port>, <node:port>]）。Kafka producer会自动为topic选择lead broker，并且在发布消息时连接到正确的broker。
serializer.class：该属性指定准备发送消息时对消息进行序列化的类。在本例中使用的是Kafka提供的字符串编码器。默认情况下key和消息的序列化类是一样的。也可以通过扩展kafka.serializer.Encoder来实现自定义的序列化类。设置参数key.serializer.class就可以使用自定义编码器。
request.required.acks：该属性指示broker在收到消息后向producer发送回执。1表示只要lead副本接收到消息就发送回执。

3.构造消息并发送：

String runtime  = new Date().toString();
String msg = "Message Publishing Time - " + runtime;
KeyedMessage<String, String> data = new KeyedMessage<String, String>(topic, msg);
producer.send(data);

完整代码如下：

package kafka.examples.producer;

import java.util.Date;
import java.util.Properties;
import kafka.javaapi.producer.Producer;
import kafka.producer.KeyedMessage;
import kafka.producer.ProducerConfig;
public class SimpleProducer {
    private static Producer<String, String> producer;
    public SimpleProducer() {
        Properties props = new Properties();

        // Set the broker list for requesting metadata to find the lead broker
        props.put("metadata.broker.list",
                "192.168.146.132:9092, 192.168.146.132:9093, 192.168.146.132:9094");

        //This specifies the serializer class for keys
        props.put("serializer.class", "kafka.serializer.StringEncoder");

        // 1 means the producer receives an acknowledgment once the lead replica
        // has received the data. This option provides better durability as the
        // client waits until the server acknowledges the request as successful.
        props.put("request.required.acks", "1");

        ProducerConfig config = new ProducerConfig(props);
        producer = new Producer<String, String>(config);
    }
    public static void main(String[] args) {
        int argsCount = args.length;
        if (argsCount == 0 || argsCount == 1)
            throw new IllegalArgumentException(
                    "Please provide topic name and Message count as arguments");
        // Topic name and the message count to be published is passed from the
        // command line
        String topic = (String) args[0];
        String count = (String) args[1];
        int messageCount = Integer.parseInt(count);
        System.out.println("Topic Name - " + topic);
        System.out.println("Message Count - " + messageCount);
        SimpleProducer simpleProducer = new SimpleProducer();
        simpleProducer.publishMessage(topic, messageCount);
    }
    private void publishMessage(String topic, int messageCount) {
        for (int mCount = 0; mCount < messageCount; mCount++) {
            String runtime = new Date().toString();
            String msg = "Message Publishing Time - " + runtime;
            System.out.println(msg);
            // Creates a KeyedMessage instance
            KeyedMessage<String, String> data =
                    new KeyedMessage<String, String>(topic, msg);

            // Publish the message
            producer.send(data);
        }
        // Close producer connection with broker.
        producer.close();
    }
}

在运行上面的代码之前，确保已经创建了名为kafkatopic的topic：

bin/kafka-topics.sh --create --zookeeper localhost:2181 --replication-factor 1 --partitions 3 --topic kafkatopic

添加环境变量KAFKA_LIB指向Kafka的lib文件夹路径，并将lib文件夹下的jar包添加到classpath。

编译代码：

javac -d . kafka/examples/producer/SimpleProducer.java

运行程序，SimpleProducer接收两个参数，topic名称和消息数量：

java kafka.examples.producer.SimpleProducer kafkatopic 10

之后可以运行consumer接收消息了：

bin/kafka-console-consumer.sh --zookeeper localhost:2181 --from-beginning --topic kafkatopic

自定义partition的Java producer

上面的例子是一个非常简单的针对多broker集群的producer，没有明确指定消息的partition。接下来我们写一个带自定义消息partition的。例子的场景是，捕捉并发布从各个IP访问网站的日志消息。日志消息包含：网站被访问时的timestamp、网站的名称、访问网站的IP地址。

1.引用以下类

import java.util.Date;
import java.util.Properties;
import java.util.Random;
import kafka.javaapi.producer.Producer;
import kafka.producer.KeyedMessage;
import kafka.producer.ProducerConfig;

2.定义属性

Properties props = new Properties();
props.put("metadata.broker.list", "localhost:9092, localhost:9093, localhost:9094");
props.put("serializer.class", "kafka.serializer.StringEncoder");    
props.put("partitioner.class", "kafka.examples.producer.SimplePartitioner");
props.put("request.required.acks", "1");
ProducerConfig config = new ProducerConfig(props);  
Producer<Integer, String> producer = new Producer<Integer, String>(config);

属性partitioner.class指定用于决定消息发送的topic内partition的类。如果为null，则使用key的哈希值。

3.实现分区类

编写一个自定义分区类SimplePartitioner，它是抽象类Partitioner的实现。

package kafka.examples.producer;
import kafka.producer.Partitioner;
public class SimplePartitioner implements Partitioner {
    public SimplePartitioner (VerifiableProperties props) {

    }
    /*
        * The method takes the key, which in this case is the IP address, 
        * It finds the last octet and does a modulo operation on the number 
        * of partitions defined within Kafka for the topic.
        * 
        * @see kafka.producer.Partitioner#partition(java.lang.Object, int)
        */
    public int partition(Object key, int a_numPartitions) {
        int partition = 0;
        String partitionKey = (String) key;
        int offset = partitionKey.lastIndexOf('.');
        if (offset > 0) {
            partition = Integer.parseInt(partitionKey.substring(offset + 1))
                    % a_numPartitions;
        }
        return partition;
    }
}

4.构造消息并发送

完整代码如下：

package kafka.examples.producer;

import java.util.Date;
import java.util.Properties;
import java.util.Random;
import kafka.javaapi.producer.Producer;
import kafka.producer.KeyedMessage;
import kafka.producer.ProducerConfig;
public class CustomPartitionProducer {
    private static Producer<String, String> producer;
    public CustomPartitionProducer() {
        Properties props = new Properties();
        // Set the broker list for requesting metadata to find the lead broker
        props.put("metadata.broker.list",
                "192.168.146.132:9092, 192.168.146.132:9093, 192.168.146.132:9094");
        // This specifies the serializer class for keys 
        props.put("serializer.class", "kafka.serializer.StringEncoder");

        // Defines the class to be used for determining the partition 
        // in the topic where the message needs to be sent.
        props.put("partitioner.class", "kafka.examples.ch4.SimplePartitioner");

        // 1 means the producer receives an acknowledgment once the lead replica
        // has received the data. This option provides better durability as the  
        // client waits until the server acknowledges the request as successful.
                props.put("request.required.acks", "1");

        ProducerConfig config = new ProducerConfig(props);
        producer = new Producer<String, String>(config);
    }
    public static void main(String[] args) {
        int argsCount = args.length;
        if (argsCount == 0 || argsCount == 1)
            throw new IllegalArgumentException(
                    "Please provide topic name and Message count as arguments");
        // Topic name and the message count to be published is passed from the
        // command line
        String topic = (String) args[0];
        String count = (String) args[1];
        int messageCount = Integer.parseInt(count);
        System.out.println("Topic Name - " + topic);
        System.out.println("Message Count - " + messageCount);
        CustomPartitionProducer simpleProducer = new CustomPartitionProducer();
        simpleProducer.publishMessage(topic, messageCount);
    }
    private void publishMessage(String topic, int messageCount) {
        Random random = new Random();
        for (int mCount = 0; mCount < messageCount; mCount++) {

            String clientIP = "192.168.14." + random.nextInt(255);
            String accessTime = new Date().toString();
            String message = accessTime + ",kafka.apache.org," + clientIP;
            System.out.println(message);

            // Creates a KeyedMessage instance
            KeyedMessage<String, String> data =
                    new KeyedMessage<String, String>(topic, clientIP, message);

            // Publish the message
            producer.send(data);
        }
        // Close producer connection with broker.
        producer.close();
    }
}

在运行上面的代码之前，确保已经创建了名为website-hits的topic：

bin/kafka-topics.sh --create --zookeeper localhost:2181 --replication-factor 3 --partitions 5 --topic website-hits

编译代码：

javac -d . kafka/examples/producer/SimplePartitioner.java
javac -d . kafka/examples/producer/CustomPartitionProducer.java

运行程序：

java kafka.examples.producer.CustomPartitionProducer website-hits 100

运行consumer接收消息：
bash bin/kafka-console-consumer.sh --zookeeper localhost:2181 --from-beginning --topic kafkatopic

producer属性

metadata.broker.list：producer使用该属性获取元数据（topic、partition、、replica），格式为host1:port1,host2:port2。
serializer.class：指定消息的序列化类。默认值为kafka.serializer.DefaultEncoder，。
producer.type：指定消息发送是同步模式还是异步模式。可选值为async和sync。默认值为sync。
request.required.acks：指定producer请求完成时broker是否向producer发送回执。默认值为0。0表示producer不等待broker的回执，这样可以降低延迟，但可靠性降低。1表示在lead副本接收到数据后producer将立即收到回执，这提高了可靠性，因为客户端会等待服务器端处理请求完成的回执。-1表示在所有同步的副本都收到数据后producer将收到回执，这提供了最佳的可靠性。
key.serializer.class：指定对key的序列化类。默认值为${serializer.class}。
partitioner.class：指定在topic中对消息进行分区的类。默认值为kafka.producer.DefaultPartitioner，是基于key的哈希值。
compression.codec：指定producer压缩数据的格式，可选的值有none、gzip、snappy。默认值为none。
batch.num.messages：指定异步模式时批次发送消息的数量。默认值为200。producer会等到消息数量达到该值或者达到queue.buffer.max.ms后才会发送消息。

oO归去来兮Oo

发布了138 篇原创文章 · 获赞 65 · 访问量 62万+

他的留言板关注

kafka学习四：开发producer

Java producer API

一个简单的Java producer

自定义partition的Java producer

producer属性

linux的vi命令和查找命令

kafka學習七：kafka 運維

ElasticSearch學習一：搜索實例演示(Java API)

ElasticSearch學習五：實例展示elasticsearch集羣生態,分片以及水平擴展.

RESTful WebService入門

https://yachay.unat.edu.pe/blog/index.php?comment_area=format_blog&comment_component=blog&comment_co

linux以太網驅動總結