本文假設讀者已經對Storm的基礎結構有了全面理解,並知道Nimbus與supervisor在集羣之中所扮演的角色。之所以要理解Storm集羣的並行機制,是爲了能夠對數據流中數據地處理順序有一個深入地理解,這樣才能更有信心地使用工具。
首先是需要了解一些與Storm集羣並行機制相關地概念:
工作進程(worker process,就是一個JVM進程,通過在supervisor服務器上執行jps命令可以看到一個或多個的worker)的隔離性
一個Storm集羣可以運行多個Topologies,但是對於一個worker來說,它只爲一個Topology工作,這在一定程度上提供了Topologies之間的隔離性。
執行器(executor)
在工作進程JVM的的內部,可以啓動多個線程來一起工作,每個線程就是一個執行器(executor),一個執行器只能執行拓撲中的一個組件,要麼是一個Spout,要麼是一個Bolt。這裏,我們看到一個工作進程內部,某些線程可以執行Spout任務,而另外一些線程可以執行Bolt任務。
任務(tasks)
任務是Storm集羣中最細粒度的劃分,每個任務要麼是一個Spout任務,要麼是一個Bolt任務,再結合上面對於執行器的說明,我們可以看到這些任務被分配到一些執行器中去執行。我們在定義拓撲的時候,會定義好每個Spout和每個Bolt任務的數量,一旦定義好之後,是不能在拓撲運行的過程中動態變更相應組件的任務數的。
要理解Storm的並行機制,只需要理解Storm數據流中元組的處理順序即可。爲了更與生產環境接近,使用Maven作爲處理包依賴的處理工具,並通過其maven-shade-plugin插件,將需要的jar包依賴共同打包到最終的Storm任務jar包中,而不用對Storm的運行環境做環境變量做處理,這確實帶來了非常大的便利。首先,需要通過配置POM文件明確各種包的依賴:
<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
<modelVersion>4.0.0</modelVersion>
<groupId>com.example</groupId>
<artifactId>istorm</artifactId>
<version>0.0.1-SNAPSHOT</version>
<packaging>jar</packaging>
<name>istorm</name>
<url>http://maven.apache.org</url>
<properties>
<project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
</properties>
<dependencies>
<dependency>
<groupId>junit</groupId>
<artifactId>junit</artifactId>
<version>3.8.1</version>
<scope>test</scope>
</dependency>
<dependency>
<groupId>org.apache.storm</groupId>
<artifactId>storm-kafka</artifactId>
<version>0.9.5</version>
<scope>compile</scope>
</dependency>
<dependency>
<groupId>org.apache.storm</groupId>
<artifactId>storm-core</artifactId>
<version>0.9.5</version>
<scope>provided</scope>
</dependency>
<dependency>
<groupId>redis.clients</groupId>
<artifactId>jedis</artifactId>
<version>2.7.3</version>
</dependency>
<dependency>
<groupId>org.apache.kafka</groupId>
<artifactId>kafka_2.11</artifactId>
<version>0.8.2.1</version>
<scope>compile</scope>
<exclusions>
<exclusion>
<groupId>org.slf4j</groupId>
<artifactId>slf4j-log4j12</artifactId>
</exclusion>
<exclusion>
<groupId>log4j</groupId>
<artifactId>log4j</artifactId>
</exclusion>
</exclusions>
</dependency>
</dependencies>
<build>
<plugins>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-shade-plugin</artifactId>
<executions>
<execution>
<phase>package</phase>
<goals>
<goal>shade</goal>
</goals>
</execution>
</executions>
<configuration>
<finalName>withdepends-${artifactId}-${version}</finalName>
</configuration>
</plugin>
</plugins>
</build>
</project>
在剛剛建立好Maven任務之後,需要下載所有的依賴包,這可能需要一些時間。有了這些依賴之後,下一步就要規劃工程結構。爲了更能夠與生產環境接近,我們將Storm的數據源設定爲Kafka,Storm和Kafka相關的依賴包已經在上面的POM文件中有所體現了,至於Kafka集羣的搭建和使用,在此不做細緻的說明。整個工程的結構是這樣的:我們通過消費者向Kafka集羣的一個TOPIC發送一系列的順序的數字,1,2,3,4...N,而Storm(KafkaSpout)會讀取這些數據,將其交付給SequenceBolt,它負責爲序列數字添加上一些線程與任務相關的信息後,將其發送給PrinterBolt,這個PrinterBolt負責將收集到的數據發送給Kafka集羣中的一個結果隊列,我們能夠從終端中讀取這個這個隊列中的結果,以觀察最終的序列被處理的順序。爲了更能說明問題,我們的Topology能夠通過參數來控制總的worker數量以及各個組件的tasks的數量,這在下面的代碼示例中都會有所體現。
首先,是SequenceBolt的實現代碼:
package com.example.istorm;
import java.util.ArrayList;
import java.util.Date;
import java.util.List;
import java.util.Map;
import java.util.Random;
import org.apache.commons.lang.StringUtils;
import com.google.common.collect.ImmutableList;
import backtype.storm.task.TopologyContext;
import backtype.storm.topology.BasicOutputCollector;
import backtype.storm.topology.OutputFieldsDeclarer;
import backtype.storm.topology.base.BaseBasicBolt;
import backtype.storm.tuple.Fields;
import backtype.storm.tuple.Tuple;
public class SequenceBolt extends BaseBasicBolt {
private static final long serialVersionUID = -5735813963113079832L;
private int taskId;
private Random rand;
@Override
public void prepare(Map stormConf, TopologyContext context) {
this.taskId = context.getThisTaskId();
this.rand = new Random();
}
public void execute(Tuple input, BasicOutputCollector collector) {
String data = input.getString(0);
String timestamp = new Date().toString();
List<String> dataList = new ArrayList<String>();
dataList.add("sequenceBoltThreadID:" + Thread.currentThread().getId() + "@" + timestamp + "@sequnceTaskID:" + this.taskId);
dataList.add(data);
// simulate the work latency
try {
Thread.sleep(this.rand.nextInt(47));
} catch (InterruptedException e) {
e.printStackTrace();
}
collector.emit(ImmutableList.of((Object) StringUtils.join(dataList, '\t')));
}
public void declareOutputFields(OutputFieldsDeclarer declarer) {
declarer.declare(new Fields("sequence"));
}
public static void main(String[] args) throws InterruptedException{
List<String> dataList = new ArrayList<String>();
dataList.add("100");
dataList.add("200");
System.out.println(StringUtils.join(dataList, '\t'));
System.out.println(ImmutableList.of((Object) StringUtils.join(dataList, '\t')));
System.out.println("Sleep start");
Random rand = new Random();
Thread.sleep(rand.nextInt(1000));
System.out.println("Sleep end");
}
}
這個Bolt的邏輯非常簡單,僅僅是在接收到序列數據之後,在其上加上了事件戳、線程信息、任務信息之後,直接提交這個數據。這麼簡單的邏輯,相信理解起來非常簡單。下面是更簡單的PrinterBolt,它僅僅負責將過來的數據發送到Kafka的結果隊列之中:
package com.example.istorm;
import java.util.ArrayList;
import java.util.Date;
import java.util.List;
import java.util.Map;
import org.apache.commons.lang.StringUtils;
import com.example.utils.KafkaUtils;
import backtype.storm.task.TopologyContext;
import backtype.storm.topology.BasicOutputCollector;
import backtype.storm.topology.OutputFieldsDeclarer;
import backtype.storm.topology.base.BaseBasicBolt;
import backtype.storm.tuple.Tuple;
public class PrinterBolt extends BaseBasicBolt {
private static final long serialVersionUID = -6508964903665304388L;
// Every printer bolt task will hold a kafkaUtils object that will be used to
// post the result to the Kafka servers
private KafkaUtils kafkaUtils = null;
private int taskId;
@Override
public void prepare(Map stormConf, TopologyContext context) {
this.kafkaUtils = new KafkaUtils("resulttopic"); // we will send result to this topic
this.taskId = context.getThisTaskId();
}
@Override
public void cleanup() {
if(this.kafkaUtils != null){
this.kafkaUtils.close();
}
}
public void execute(Tuple input, BasicOutputCollector collector) {
String sequence = input.getStringByField("sequence");
String timestamp = new Date().toString();
List<String> dataList = new ArrayList<String>();
dataList.add("printerBoltThreadID:" + Thread.currentThread().getId() + "@" + timestamp + "@printerTaskID:" + this.taskId);
dataList.add(sequence);
this.kafkaUtils.publishMessage(StringUtils.join(dataList, '\t'));
}
public void declareOutputFields(OutputFieldsDeclarer arg0) {
// we don't emit anything
}
}
在這個程序中的prepare方法中,創建了Kafka的工具類,用來向Kafka的某個TOPIC發送消息,這個工具類的代碼也只有簡單幾行而已:
package com.example.utils;
import java.util.Properties;
import kafka.javaapi.producer.Producer;
import kafka.producer.KeyedMessage;
import kafka.producer.ProducerConfig;
public class KafkaUtils {
private Producer<String, String> producer;
private String topic;
public KafkaUtils(String topic){
Properties props = new Properties();
props.put("metadata.broker.list", "a00:9092,a01:9092,a02:9092");
props.put("serializer.class", "kafka.serializer.StringEncoder");
props.put("partitioner.class", "com.example.utils.SimplePartitioner");
props.put("request.required_acks", "1");
ProducerConfig config = new ProducerConfig(props);
this.producer = new Producer<String, String>(config);
this.topic = topic;
}
public void publishMessage(String msg){
KeyedMessage<String, String> data = new KeyedMessage<String, String>(this.topic, msg, msg);
this.producer.send(data);
}
public void close(){
this.producer.close();
}
public static void main(String[] args){
String topic = "datasrctopic";
KafkaUtils sp = new KafkaUtils(topic);
for(int i=0; i < 100; i++){
sp.publishMessage(Integer.toString(i));
}
sp.close();
}
}
其中的main方法,實現了向Kafka中發送數據的功能,我們以後的測試數據就是通過執行這個方法獲取的。爲了更能說明問題,我實現了一個Kafka簡單的分區函數,其實現也是非常簡單的,其代碼也是最簡短的:
package com.example.utils;
import kafka.producer.Partitioner;
import kafka.utils.VerifiableProperties;
public class SimplePartitioner implements Partitioner {
public SimplePartitioner(VerifiableProperties props){
}
public int partition(Object key, int a_numPartitions) {
String partitionKey = (String) key;
return partitionKey.hashCode() % a_numPartitions;
}
}
這個代碼,我們期望能夠將消息分散到Kafka相應TOPIC中不同的分區中。最後,是將整個Topology初始化:
package com.example.istorm;
import backtype.storm.spout.SchemeAsMultiScheme;
import backtype.storm.topology.TopologyBuilder;
import java.util.ArrayList;
import java.util.List;
import backtype.storm.Config;
//import backtype.storm.LocalCluster;
import backtype.storm.StormSubmitter;
import backtype.storm.generated.AlreadyAliveException;
import backtype.storm.generated.InvalidTopologyException;
import storm.kafka.KafkaSpout;
import storm.kafka.SpoutConfig;
import storm.kafka.StringScheme;
import storm.kafka.ZkHosts;
public class KafkaTopology {
public static void main(String[] args) throws AlreadyAliveException, InvalidTopologyException{
// Zookeeper hosts for the Kafka cluster
ZkHosts zkHosts = new ZkHosts("a00:2181,a01:2181,a02:2181");
// Create the KafkaSpout configration
// Second argument is the topic name
// Third argument is the Zookeeper root for Kafka
// Fourth argument is consumer group id
SpoutConfig kafkaConfig = new SpoutConfig(zkHosts, "datasrctopic","/tmp", "id7");
int workerNum = Integer.parseInt(args[0]);
int kafkaSpoutNum = Integer.parseInt(args[1]);
int kafkaSpoutTaskNum = Integer.parseInt(args[2]);
int sequenceBoltNum = Integer.parseInt(args[3]);
int sequenceBoltTaskNum = Integer.parseInt(args[4]);
int printerBoltNum = Integer.parseInt(args[5]);
int printerBoltTaskNum = Integer.parseInt(args[6]);
// final List<String> zkServerAddresses = new ArrayList<String>();
// zkServerAddresses.add("a00");
// zkServerAddresses.add("a01");
// zkServerAddresses.add("a02");
// kafkaConfig.zkServers = zkServerAddresses;
// kafkaConfig.zkPort = 2181;
// Specify that the kafka messages are String
kafkaConfig.scheme = new SchemeAsMultiScheme(new StringScheme());
// We want to consume all the first messages in
// the topic every time we run the topology to
// help in debugging. In production, this property should be false
kafkaConfig.forceFromStart = false;
// Now we create the topology
TopologyBuilder builder = new TopologyBuilder();
// Set the kafka spout class
builder.setSpout("KafkaSpout", new KafkaSpout(kafkaConfig), kafkaSpoutNum)
.setNumTasks(kafkaSpoutTaskNum);
// Configure the bolts
builder.setBolt("SequenceBolt", new SequenceBolt(), sequenceBoltNum)
.setNumTasks(sequenceBoltTaskNum)
.globalGrouping("KafkaSpout");
builder.setBolt("PrinterBolt", new PrinterBolt(), printerBoltNum)
.setNumTasks(printerBoltTaskNum)
.globalGrouping("SequenceBolt");
Config conf = new Config();
conf.setNumWorkers(workerNum);
StormSubmitter.submitTopology("KafkaTopology", conf, builder.createTopology());
/*
// Create an instance of LocalCluster class
// for executing in local mode.
LocalCluster cluster = new LocalCluster();
// Submit topology for execution
cluster.submitTopology("KafkaTopology", conf, builder.createTopology());
try{
// Wait for some time before exiting
Thread.sleep(100000);
} catch(Exception exception){
System.out.println("Thread interrupted exception: " + exception);
}
// Kill the KafkaTopology
cluster.killTopology("KafkaTopology");
// Shutdown the storm test cluster
cluster.shutdown();
*/
}
}
這裏需要注意的一點是,如果Storm是運行在LocalCluster的模式下,那麼我們必須顯式指定如下的代碼,才能讓Storm-Kafka記住上次的偏移量到Zookeeper服務中:
final List<String> zkServerAddresses = new ArrayList<String>();
zkServerAddresses.add("a00");
zkServerAddresses.add("a01");
zkServerAddresses.add("a02");
kafkaConfig.zkServers = zkServerAddresses;
kafkaConfig.zkPort = 2181;
當然,如果是在集羣模式下運行,就不需要這幾行代碼了。在我們的Topology的配置中,我們使用了globalGrouping作爲分發數據的方法,這種分發數據流的方法並不會將數據流分區,而是將整個數據流完整地分發到具有最小任務ID地BOLT中,有點兒類似於MapReduce中的Reduce階段的數據流。
場景1:數據源的順序確定。
對於數據源的順序,我們經常會忽視。對於像Kafka這樣的數據源,我們就不得不考慮其分區特性對於Storm數據源輸入順序的影響:Kafka在每個分區上,通過順序偏移量的機制爲所有消息提供了強的順序保證,但是在不同的分區之間並沒有這種順序的保證。爲了驗證這個想法,我們在Kafka的數據源隊列的TOPIC創建時,這樣創建:
bin/kafka-topics.sh --create --zookeeper a00:2181,a01:2181,a02:2181 --replication-factor 1 --partitions 10 --topic datasrctopic
數據源TOPIC創建時,指定其有10個分區,我們期待Storm在讀取這個TOPIC的數據時,總體上是無序的(雖然每個分區內的消息時有序的)。而最終的結果TOPIC,爲了消除多分區對輸出順序的影響,我們僅僅指定其有1個分區:
bin/kafka-topics.sh --create --zookeeper a00:2181,a01:2181,a02:2181 --replication-factor 1 --partitions 1 --topic resulttopic
在有了如上的準備之後,我們就可以提交Topology到Storm集羣上了:
bin/storm jar /tmp/withdepends-istorm-0.0.1-SNAPSHOT.jar com.example.istorm.KafkaTopology 1 1 1 1 1 1 1
爲了消除其它影響,我們將worker數量以及各個組件的執行器數量和任務數量都設置爲1,這樣整體的順序就取決於數據源的順序了。這個時候,我們去執行一下KafkaUtils中的main函數,我們從結果隊列中得到如下的輸出:
printerBoltThreadID:31@Mon Sep 14 15:31:01 CST 2015@printerTaskID:2 sequenceBoltThreadID:33@Mon Sep 14 15:31:01 CST 2015@sequnceTaskID:3 0
printerBoltThreadID:31@Mon Sep 14 15:31:01 CST 2015@printerTaskID:2 sequenceBoltThreadID:33@Mon Sep 14 15:31:01 CST 2015@sequnceTaskID:3 2
printerBoltThreadID:31@Mon Sep 14 15:31:01 CST 2015@printerTaskID:2 sequenceBoltThreadID:33@Mon Sep 14 15:31:01 CST 2015@sequnceTaskID:3 1
printerBoltThreadID:31@Mon Sep 14 15:31:01 CST 2015@printerTaskID:2 sequenceBoltThreadID:33@Mon Sep 14 15:31:01 CST 2015@sequnceTaskID:3 4
printerBoltThreadID:31@Mon Sep 14 15:31:01 CST 2015@printerTaskID:2 sequenceBoltThreadID:33@Mon Sep 14 15:31:01 CST 2015@sequnceTaskID:3 7
printerBoltThreadID:31@Mon Sep 14 15:31:01 CST 2015@printerTaskID:2 sequenceBoltThreadID:33@Mon Sep 14 15:31:01 CST 2015@sequnceTaskID:3 5
printerBoltThreadID:31@Mon Sep 14 15:31:01 CST 2015@printerTaskID:2 sequenceBoltThreadID:33@Mon Sep 14 15:31:01 CST 2015@sequnceTaskID:3 8
printerBoltThreadID:31@Mon Sep 14 15:31:01 CST 2015@printerTaskID:2 sequenceBoltThreadID:33@Mon Sep 14 15:31:01 CST 2015@sequnceTaskID:3 11
printerBoltThreadID:31@Mon Sep 14 15:31:01 CST 2015@printerTaskID:2 sequenceBoltThreadID:33@Mon Sep 14 15:31:01 CST 2015@sequnceTaskID:3 3
printerBoltThreadID:31@Mon Sep 14 15:31:01 CST 2015@printerTaskID:2 sequenceBoltThreadID:33@Mon Sep 14 15:31:01 CST 2015@sequnceTaskID:3 12
printerBoltThreadID:31@Mon Sep 14 15:31:01 CST 2015@printerTaskID:2 sequenceBoltThreadID:33@Mon Sep 14 15:31:01 CST 2015@sequnceTaskID:3 13
printerBoltThreadID:31@Mon Sep 14 15:31:01 CST 2015@printerTaskID:2 sequenceBoltThreadID:33@Mon Sep 14 15:31:01 CST 2015@sequnceTaskID:3 15
printerBoltThreadID:31@Mon Sep 14 15:31:01 CST 2015@printerTaskID:2 sequenceBoltThreadID:33@Mon Sep 14 15:31:01 CST 2015@sequnceTaskID:3 18
printerBoltThreadID:31@Mon Sep 14 15:31:01 CST 2015@printerTaskID:2 sequenceBoltThreadID:33@Mon Sep 14 15:31:01 CST 2015@sequnceTaskID:3 6
printerBoltThreadID:31@Mon Sep 14 15:31:01 CST 2015@printerTaskID:2 sequenceBoltThreadID:33@Mon Sep 14 15:31:01 CST 2015@sequnceTaskID:3 17
printerBoltThreadID:31@Mon Sep 14 15:31:01 CST 2015@printerTaskID:2 sequenceBoltThreadID:33@Mon Sep 14 15:31:01 CST 2015@sequnceTaskID:3 14
printerBoltThreadID:31@Mon Sep 14 15:31:01 CST 2015@printerTaskID:2 sequenceBoltThreadID:33@Mon Sep 14 15:31:01 CST 2015@sequnceTaskID:3 16
printerBoltThreadID:31@Mon Sep 14 15:31:01 CST 2015@printerTaskID:2 sequenceBoltThreadID:33@Mon Sep 14 15:31:01 CST 2015@sequnceTaskID:3 9
printerBoltThreadID:31@Mon Sep 14 15:31:01 CST 2015@printerTaskID:2 sequenceBoltThreadID:33@Mon Sep 14 15:31:01 CST 2015@sequnceTaskID:3 10
printerBoltThreadID:31@Mon Sep 14 15:31:01 CST 2015@printerTaskID:2 sequenceBoltThreadID:33@Mon Sep 14 15:31:01 CST 2015@sequnceTaskID:3 21
printerBoltThreadID:31@Mon Sep 14 15:31:01 CST 2015@printerTaskID:2 sequenceBoltThreadID:33@Mon Sep 14 15:31:01 CST 2015@sequnceTaskID:3 19
printerBoltThreadID:31@Mon Sep 14 15:31:01 CST 2015@printerTaskID:2 sequenceBoltThreadID:33@Mon Sep 14 15:31:01 CST 2015@sequnceTaskID:3 20
printerBoltThreadID:31@Mon Sep 14 15:31:01 CST 2015@printerTaskID:2 sequenceBoltThreadID:33@Mon Sep 14 15:31:01 CST 2015@sequnceTaskID:3 22
printerBoltThreadID:31@Mon Sep 14 15:31:01 CST 2015@printerTaskID:2 sequenceBoltThreadID:33@Mon Sep 14 15:31:01 CST 2015@sequnceTaskID:3 24
printerBoltThreadID:31@Mon Sep 14 15:31:01 CST 2015@printerTaskID:2 sequenceBoltThreadID:33@Mon Sep 14 15:31:01 CST 2015@sequnceTaskID:3 23
printerBoltThreadID:31@Mon Sep 14 15:31:01 CST 2015@printerTaskID:2 sequenceBoltThreadID:33@Mon Sep 14 15:31:01 CST 2015@sequnceTaskID:3 25
printerBoltThreadID:31@Mon Sep 14 15:31:01 CST 2015@printerTaskID:2 sequenceBoltThreadID:33@Mon Sep 14 15:31:01 CST 2015@sequnceTaskID:3 28
printerBoltThreadID:31@Mon Sep 14 15:31:02 CST 2015@printerTaskID:2 sequenceBoltThreadID:33@Mon Sep 14 15:31:01 CST 2015@sequnceTaskID:3 27
printerBoltThreadID:31@Mon Sep 14 15:31:02 CST 2015@printerTaskID:2 sequenceBoltThreadID:33@Mon Sep 14 15:31:02 CST 2015@sequnceTaskID:3 26
printerBoltThreadID:31@Mon Sep 14 15:31:02 CST 2015@printerTaskID:2 sequenceBoltThreadID:33@Mon Sep 14 15:31:02 CST 2015@sequnceTaskID:3 29
printerBoltThreadID:31@Mon Sep 14 15:31:02 CST 2015@printerTaskID:2 sequenceBoltThreadID:33@Mon Sep 14 15:31:02 CST 2015@sequnceTaskID:3 30
printerBoltThreadID:31@Mon Sep 14 15:31:02 CST 2015@printerTaskID:2 sequenceBoltThreadID:33@Mon Sep 14 15:31:02 CST 2015@sequnceTaskID:3 31
printerBoltThreadID:31@Mon Sep 14 15:31:02 CST 2015@printerTaskID:2 sequenceBoltThreadID:33@Mon Sep 14 15:31:02 CST 2015@sequnceTaskID:3 33
printerBoltThreadID:31@Mon Sep 14 15:31:02 CST 2015@printerTaskID:2 sequenceBoltThreadID:33@Mon Sep 14 15:31:02 CST 2015@sequnceTaskID:3 36
printerBoltThreadID:31@Mon Sep 14 15:31:02 CST 2015@printerTaskID:2 sequenceBoltThreadID:33@Mon Sep 14 15:31:02 CST 2015@sequnceTaskID:3 35
printerBoltThreadID:31@Mon Sep 14 15:31:02 CST 2015@printerTaskID:2 sequenceBoltThreadID:33@Mon Sep 14 15:31:02 CST 2015@sequnceTaskID:3 32
printerBoltThreadID:31@Mon Sep 14 15:31:02 CST 2015@printerTaskID:2 sequenceBoltThreadID:33@Mon Sep 14 15:31:02 CST 2015@sequnceTaskID:3 34
printerBoltThreadID:31@Mon Sep 14 15:31:02 CST 2015@printerTaskID:2 sequenceBoltThreadID:33@Mon Sep 14 15:31:02 CST 2015@sequnceTaskID:3 38
printerBoltThreadID:31@Mon Sep 14 15:31:02 CST 2015@printerTaskID:2 sequenceBoltThreadID:33@Mon Sep 14 15:31:02 CST 2015@sequnceTaskID:3 37
printerBoltThreadID:31@Mon Sep 14 15:31:02 CST 2015@printerTaskID:2 sequenceBoltThreadID:33@Mon Sep 14 15:31:02 CST 2015@sequnceTaskID:3 39
printerBoltThreadID:31@Mon Sep 14 15:31:02 CST 2015@printerTaskID:2 sequenceBoltThreadID:33@Mon Sep 14 15:31:02 CST 2015@sequnceTaskID:3 40
printerBoltThreadID:31@Mon Sep 14 15:31:02 CST 2015@printerTaskID:2 sequenceBoltThreadID:33@Mon Sep 14 15:31:02 CST 2015@sequnceTaskID:3 42
printerBoltThreadID:31@Mon Sep 14 15:31:02 CST 2015@printerTaskID:2 sequenceBoltThreadID:33@Mon Sep 14 15:31:02 CST 2015@sequnceTaskID:3 45
printerBoltThreadID:31@Mon Sep 14 15:31:02 CST 2015@printerTaskID:2 sequenceBoltThreadID:33@Mon Sep 14 15:31:02 CST 2015@sequnceTaskID:3 44
printerBoltThreadID:31@Mon Sep 14 15:31:02 CST 2015@printerTaskID:2 sequenceBoltThreadID:33@Mon Sep 14 15:31:02 CST 2015@sequnceTaskID:3 41
printerBoltThreadID:31@Mon Sep 14 15:31:02 CST 2015@printerTaskID:2 sequenceBoltThreadID:33@Mon Sep 14 15:31:02 CST 2015@sequnceTaskID:3 43
printerBoltThreadID:31@Mon Sep 14 15:31:02 CST 2015@printerTaskID:2 sequenceBoltThreadID:33@Mon Sep 14 15:31:02 CST 2015@sequnceTaskID:3 47
printerBoltThreadID:31@Mon Sep 14 15:31:02 CST 2015@printerTaskID:2 sequenceBoltThreadID:33@Mon Sep 14 15:31:02 CST 2015@sequnceTaskID:3 46
printerBoltThreadID:31@Mon Sep 14 15:31:02 CST 2015@printerTaskID:2 sequenceBoltThreadID:33@Mon Sep 14 15:31:02 CST 2015@sequnceTaskID:3 48
printerBoltThreadID:31@Mon Sep 14 15:31:02 CST 2015@printerTaskID:2 sequenceBoltThreadID:33@Mon Sep 14 15:31:02 CST 2015@sequnceTaskID:3 50
printerBoltThreadID:31@Mon Sep 14 15:31:02 CST 2015@printerTaskID:2 sequenceBoltThreadID:33@Mon Sep 14 15:31:02 CST 2015@sequnceTaskID:3 49
printerBoltThreadID:31@Mon Sep 14 15:31:02 CST 2015@printerTaskID:2 sequenceBoltThreadID:33@Mon Sep 14 15:31:02 CST 2015@sequnceTaskID:3 51
printerBoltThreadID:31@Mon Sep 14 15:31:02 CST 2015@printerTaskID:2 sequenceBoltThreadID:33@Mon Sep 14 15:31:02 CST 2015@sequnceTaskID:3 52
printerBoltThreadID:31@Mon Sep 14 15:31:02 CST 2015@printerTaskID:2 sequenceBoltThreadID:33@Mon Sep 14 15:31:02 CST 2015@sequnceTaskID:3 53
printerBoltThreadID:31@Mon Sep 14 15:31:02 CST 2015@printerTaskID:2 sequenceBoltThreadID:33@Mon Sep 14 15:31:02 CST 2015@sequnceTaskID:3 55
printerBoltThreadID:31@Mon Sep 14 15:31:02 CST 2015@printerTaskID:2 sequenceBoltThreadID:33@Mon Sep 14 15:31:02 CST 2015@sequnceTaskID:3 54
printerBoltThreadID:31@Mon Sep 14 15:31:02 CST 2015@printerTaskID:2 sequenceBoltThreadID:33@Mon Sep 14 15:31:02 CST 2015@sequnceTaskID:3 56
printerBoltThreadID:31@Mon Sep 14 15:31:02 CST 2015@printerTaskID:2 sequenceBoltThreadID:33@Mon Sep 14 15:31:02 CST 2015@sequnceTaskID:3 57
printerBoltThreadID:31@Mon Sep 14 15:31:02 CST 2015@printerTaskID:2 sequenceBoltThreadID:33@Mon Sep 14 15:31:02 CST 2015@sequnceTaskID:3 58
printerBoltThreadID:31@Mon Sep 14 15:31:02 CST 2015@printerTaskID:2 sequenceBoltThreadID:33@Mon Sep 14 15:31:02 CST 2015@sequnceTaskID:3 59
printerBoltThreadID:31@Mon Sep 14 15:31:02 CST 2015@printerTaskID:2 sequenceBoltThreadID:33@Mon Sep 14 15:31:02 CST 2015@sequnceTaskID:3 60
printerBoltThreadID:31@Mon Sep 14 15:31:02 CST 2015@printerTaskID:2 sequenceBoltThreadID:33@Mon Sep 14 15:31:02 CST 2015@sequnceTaskID:3 61
printerBoltThreadID:31@Mon Sep 14 15:31:02 CST 2015@printerTaskID:2 sequenceBoltThreadID:33@Mon Sep 14 15:31:02 CST 2015@sequnceTaskID:3 63
printerBoltThreadID:31@Mon Sep 14 15:31:02 CST 2015@printerTaskID:2 sequenceBoltThreadID:33@Mon Sep 14 15:31:02 CST 2015@sequnceTaskID:3 62
printerBoltThreadID:31@Mon Sep 14 15:31:02 CST 2015@printerTaskID:2 sequenceBoltThreadID:33@Mon Sep 14 15:31:02 CST 2015@sequnceTaskID:3 65
printerBoltThreadID:31@Mon Sep 14 15:31:02 CST 2015@printerTaskID:2 sequenceBoltThreadID:33@Mon Sep 14 15:31:02 CST 2015@sequnceTaskID:3 67
printerBoltThreadID:31@Mon Sep 14 15:31:02 CST 2015@printerTaskID:2 sequenceBoltThreadID:33@Mon Sep 14 15:31:02 CST 2015@sequnceTaskID:3 64
printerBoltThreadID:31@Mon Sep 14 15:31:02 CST 2015@printerTaskID:2 sequenceBoltThreadID:33@Mon Sep 14 15:31:02 CST 2015@sequnceTaskID:3 66
printerBoltThreadID:31@Mon Sep 14 15:31:02 CST 2015@printerTaskID:2 sequenceBoltThreadID:33@Mon Sep 14 15:31:02 CST 2015@sequnceTaskID:3 68
printerBoltThreadID:31@Mon Sep 14 15:31:02 CST 2015@printerTaskID:2 sequenceBoltThreadID:33@Mon Sep 14 15:31:02 CST 2015@sequnceTaskID:3 71
printerBoltThreadID:31@Mon Sep 14 15:31:02 CST 2015@printerTaskID:2 sequenceBoltThreadID:33@Mon Sep 14 15:31:02 CST 2015@sequnceTaskID:3 69
printerBoltThreadID:31@Mon Sep 14 15:31:02 CST 2015@printerTaskID:2 sequenceBoltThreadID:33@Mon Sep 14 15:31:02 CST 2015@sequnceTaskID:3 70
printerBoltThreadID:31@Mon Sep 14 15:31:02 CST 2015@printerTaskID:2 sequenceBoltThreadID:33@Mon Sep 14 15:31:02 CST 2015@sequnceTaskID:3 73
printerBoltThreadID:31@Mon Sep 14 15:31:02 CST 2015@printerTaskID:2 sequenceBoltThreadID:33@Mon Sep 14 15:31:02 CST 2015@sequnceTaskID:3 75
printerBoltThreadID:31@Mon Sep 14 15:31:02 CST 2015@printerTaskID:2 sequenceBoltThreadID:33@Mon Sep 14 15:31:02 CST 2015@sequnceTaskID:3 72
printerBoltThreadID:31@Mon Sep 14 15:31:02 CST 2015@printerTaskID:2 sequenceBoltThreadID:33@Mon Sep 14 15:31:02 CST 2015@sequnceTaskID:3 74
printerBoltThreadID:31@Mon Sep 14 15:31:03 CST 2015@printerTaskID:2 sequenceBoltThreadID:33@Mon Sep 14 15:31:02 CST 2015@sequnceTaskID:3 76
printerBoltThreadID:31@Mon Sep 14 15:31:03 CST 2015@printerTaskID:2 sequenceBoltThreadID:33@Mon Sep 14 15:31:03 CST 2015@sequnceTaskID:3 77
printerBoltThreadID:31@Mon Sep 14 15:31:03 CST 2015@printerTaskID:2 sequenceBoltThreadID:33@Mon Sep 14 15:31:03 CST 2015@sequnceTaskID:3 79
printerBoltThreadID:31@Mon Sep 14 15:31:03 CST 2015@printerTaskID:2 sequenceBoltThreadID:33@Mon Sep 14 15:31:03 CST 2015@sequnceTaskID:3 78
printerBoltThreadID:31@Mon Sep 14 15:31:03 CST 2015@printerTaskID:2 sequenceBoltThreadID:33@Mon Sep 14 15:31:03 CST 2015@sequnceTaskID:3 82
printerBoltThreadID:31@Mon Sep 14 15:31:03 CST 2015@printerTaskID:2 sequenceBoltThreadID:33@Mon Sep 14 15:31:03 CST 2015@sequnceTaskID:3 81
printerBoltThreadID:31@Mon Sep 14 15:31:03 CST 2015@printerTaskID:2 sequenceBoltThreadID:33@Mon Sep 14 15:31:03 CST 2015@sequnceTaskID:3 80
printerBoltThreadID:31@Mon Sep 14 15:31:03 CST 2015@printerTaskID:2 sequenceBoltThreadID:33@Mon Sep 14 15:31:03 CST 2015@sequnceTaskID:3 83
printerBoltThreadID:31@Mon Sep 14 15:31:03 CST 2015@printerTaskID:2 sequenceBoltThreadID:33@Mon Sep 14 15:31:03 CST 2015@sequnceTaskID:3 84
printerBoltThreadID:31@Mon Sep 14 15:31:03 CST 2015@printerTaskID:2 sequenceBoltThreadID:33@Mon Sep 14 15:31:03 CST 2015@sequnceTaskID:3 87
printerBoltThreadID:31@Mon Sep 14 15:31:03 CST 2015@printerTaskID:2 sequenceBoltThreadID:33@Mon Sep 14 15:31:03 CST 2015@sequnceTaskID:3 85
printerBoltThreadID:31@Mon Sep 14 15:31:03 CST 2015@printerTaskID:2 sequenceBoltThreadID:33@Mon Sep 14 15:31:03 CST 2015@sequnceTaskID:3 86
printerBoltThreadID:31@Mon Sep 14 15:31:03 CST 2015@printerTaskID:2 sequenceBoltThreadID:33@Mon Sep 14 15:31:03 CST 2015@sequnceTaskID:3 88
printerBoltThreadID:31@Mon Sep 14 15:31:03 CST 2015@printerTaskID:2 sequenceBoltThreadID:33@Mon Sep 14 15:31:03 CST 2015@sequnceTaskID:3 90
printerBoltThreadID:31@Mon Sep 14 15:31:03 CST 2015@printerTaskID:2 sequenceBoltThreadID:33@Mon Sep 14 15:31:03 CST 2015@sequnceTaskID:3 89
printerBoltThreadID:31@Mon Sep 14 15:31:03 CST 2015@printerTaskID:2 sequenceBoltThreadID:33@Mon Sep 14 15:31:03 CST 2015@sequnceTaskID:3 91
printerBoltThreadID:31@Mon Sep 14 15:31:03 CST 2015@printerTaskID:2 sequenceBoltThreadID:33@Mon Sep 14 15:31:03 CST 2015@sequnceTaskID:3 92
printerBoltThreadID:31@Mon Sep 14 15:31:03 CST 2015@printerTaskID:2 sequenceBoltThreadID:33@Mon Sep 14 15:31:03 CST 2015@sequnceTaskID:3 93
printerBoltThreadID:31@Mon Sep 14 15:31:03 CST 2015@printerTaskID:2 sequenceBoltThreadID:33@Mon Sep 14 15:31:03 CST 2015@sequnceTaskID:3 94
printerBoltThreadID:31@Mon Sep 14 15:31:03 CST 2015@printerTaskID:2 sequenceBoltThreadID:33@Mon Sep 14 15:31:03 CST 2015@sequnceTaskID:3 95
printerBoltThreadID:31@Mon Sep 14 15:31:03 CST 2015@printerTaskID:2 sequenceBoltThreadID:33@Mon Sep 14 15:31:03 CST 2015@sequnceTaskID:3 97
printerBoltThreadID:31@Mon Sep 14 15:31:03 CST 2015@printerTaskID:2 sequenceBoltThreadID:33@Mon Sep 14 15:31:03 CST 2015@sequnceTaskID:3 96
printerBoltThreadID:31@Mon Sep 14 15:31:03 CST 2015@printerTaskID:2 sequenceBoltThreadID:33@Mon Sep 14 15:31:03 CST 2015@sequnceTaskID:3 98
printerBoltThreadID:31@Mon Sep 14 15:31:03 CST 2015@printerTaskID:2 sequenceBoltThreadID:33@Mon Sep 14 15:31:03 CST 2015@sequnceTaskID:3 99
明顯看到,整個輸出時無序的。這個時候的無序完全是由於Kafka不能在多個分區上保持消息的有序性造成的,那麼如果要讓消息有序有兩種方法,一種方法就是修改Kafka的分區函數,將其全部消息發向同一個分區;第二種方法,就是在創建TOPIC的時候,指定只有1個分區。當然,作爲測試,我們採用第二種方法,在實際應用下,可能要利用第一種方法,採取某種Kafka的分區策略,比如按照用戶的uid分區到Kafka的各個分區上,保證每個用戶產生的消息是有序的。採用第二種方法就是簡單地重建TOPIC:
bin/kafka-topics.sh --delete --zookeeper a00:2181,a01:2181,a02:2181 --topic datasrctopic
bin/kafka-topics.sh --create --zookeeper a00:2181,a01:2181,a02:2181 --replication-factor 1 --partitions 1 --topic datasrctopic
在做好如上地處理之後,我們再執行先前地程序,最終地輸出就成爲:
printerBoltThreadID:31@Mon Sep 14 15:40:42 CST 2015@printerTaskID:2 sequenceBoltThreadID:33@Mon Sep 14 15:40:42 CST 2015@sequnceTaskID:3 0
printerBoltThreadID:31@Mon Sep 14 15:40:42 CST 2015@printerTaskID:2 sequenceBoltThreadID:33@Mon Sep 14 15:40:42 CST 2015@sequnceTaskID:3 1
printerBoltThreadID:31@Mon Sep 14 15:40:42 CST 2015@printerTaskID:2 sequenceBoltThreadID:33@Mon Sep 14 15:40:42 CST 2015@sequnceTaskID:3 2
printerBoltThreadID:31@Mon Sep 14 15:40:42 CST 2015@printerTaskID:2 sequenceBoltThreadID:33@Mon Sep 14 15:40:42 CST 2015@sequnceTaskID:3 3
printerBoltThreadID:31@Mon Sep 14 15:40:42 CST 2015@printerTaskID:2 sequenceBoltThreadID:33@Mon Sep 14 15:40:42 CST 2015@sequnceTaskID:3 4
printerBoltThreadID:31@Mon Sep 14 15:40:42 CST 2015@printerTaskID:2 sequenceBoltThreadID:33@Mon Sep 14 15:40:42 CST 2015@sequnceTaskID:3 5
printerBoltThreadID:31@Mon Sep 14 15:40:42 CST 2015@printerTaskID:2 sequenceBoltThreadID:33@Mon Sep 14 15:40:42 CST 2015@sequnceTaskID:3 6
printerBoltThreadID:31@Mon Sep 14 15:40:42 CST 2015@printerTaskID:2 sequenceBoltThreadID:33@Mon Sep 14 15:40:42 CST 2015@sequnceTaskID:3 7
printerBoltThreadID:31@Mon Sep 14 15:40:42 CST 2015@printerTaskID:2 sequenceBoltThreadID:33@Mon Sep 14 15:40:42 CST 2015@sequnceTaskID:3 8
printerBoltThreadID:31@Mon Sep 14 15:40:42 CST 2015@printerTaskID:2 sequenceBoltThreadID:33@Mon Sep 14 15:40:42 CST 2015@sequnceTaskID:3 9
printerBoltThreadID:31@Mon Sep 14 15:40:42 CST 2015@printerTaskID:2 sequenceBoltThreadID:33@Mon Sep 14 15:40:42 CST 2015@sequnceTaskID:3 10
printerBoltThreadID:31@Mon Sep 14 15:40:42 CST 2015@printerTaskID:2 sequenceBoltThreadID:33@Mon Sep 14 15:40:42 CST 2015@sequnceTaskID:3 11
printerBoltThreadID:31@Mon Sep 14 15:40:42 CST 2015@printerTaskID:2 sequenceBoltThreadID:33@Mon Sep 14 15:40:42 CST 2015@sequnceTaskID:3 12
printerBoltThreadID:31@Mon Sep 14 15:40:42 CST 2015@printerTaskID:2 sequenceBoltThreadID:33@Mon Sep 14 15:40:42 CST 2015@sequnceTaskID:3 13
printerBoltThreadID:31@Mon Sep 14 15:40:42 CST 2015@printerTaskID:2 sequenceBoltThreadID:33@Mon Sep 14 15:40:42 CST 2015@sequnceTaskID:3 14
printerBoltThreadID:31@Mon Sep 14 15:40:42 CST 2015@printerTaskID:2 sequenceBoltThreadID:33@Mon Sep 14 15:40:42 CST 2015@sequnceTaskID:3 15
printerBoltThreadID:31@Mon Sep 14 15:40:42 CST 2015@printerTaskID:2 sequenceBoltThreadID:33@Mon Sep 14 15:40:42 CST 2015@sequnceTaskID:3 16
printerBoltThreadID:31@Mon Sep 14 15:40:42 CST 2015@printerTaskID:2 sequenceBoltThreadID:33@Mon Sep 14 15:40:42 CST 2015@sequnceTaskID:3 17
printerBoltThreadID:31@Mon Sep 14 15:40:42 CST 2015@printerTaskID:2 sequenceBoltThreadID:33@Mon Sep 14 15:40:42 CST 2015@sequnceTaskID:3 18
printerBoltThreadID:31@Mon Sep 14 15:40:42 CST 2015@printerTaskID:2 sequenceBoltThreadID:33@Mon Sep 14 15:40:42 CST 2015@sequnceTaskID:3 19
printerBoltThreadID:31@Mon Sep 14 15:40:42 CST 2015@printerTaskID:2 sequenceBoltThreadID:33@Mon Sep 14 15:40:42 CST 2015@sequnceTaskID:3 20
printerBoltThreadID:31@Mon Sep 14 15:40:42 CST 2015@printerTaskID:2 sequenceBoltThreadID:33@Mon Sep 14 15:40:42 CST 2015@sequnceTaskID:3 21
printerBoltThreadID:31@Mon Sep 14 15:40:42 CST 2015@printerTaskID:2 sequenceBoltThreadID:33@Mon Sep 14 15:40:42 CST 2015@sequnceTaskID:3 22
printerBoltThreadID:31@Mon Sep 14 15:40:42 CST 2015@printerTaskID:2 sequenceBoltThreadID:33@Mon Sep 14 15:40:42 CST 2015@sequnceTaskID:3 23
printerBoltThreadID:31@Mon Sep 14 15:40:42 CST 2015@printerTaskID:2 sequenceBoltThreadID:33@Mon Sep 14 15:40:42 CST 2015@sequnceTaskID:3 24
printerBoltThreadID:31@Mon Sep 14 15:40:42 CST 2015@printerTaskID:2 sequenceBoltThreadID:33@Mon Sep 14 15:40:42 CST 2015@sequnceTaskID:3 25
printerBoltThreadID:31@Mon Sep 14 15:40:42 CST 2015@printerTaskID:2 sequenceBoltThreadID:33@Mon Sep 14 15:40:42 CST 2015@sequnceTaskID:3 26
printerBoltThreadID:31@Mon Sep 14 15:40:42 CST 2015@printerTaskID:2 sequenceBoltThreadID:33@Mon Sep 14 15:40:42 CST 2015@sequnceTaskID:3 27
printerBoltThreadID:31@Mon Sep 14 15:40:42 CST 2015@printerTaskID:2 sequenceBoltThreadID:33@Mon Sep 14 15:40:42 CST 2015@sequnceTaskID:3 28
printerBoltThreadID:31@Mon Sep 14 15:40:42 CST 2015@printerTaskID:2 sequenceBoltThreadID:33@Mon Sep 14 15:40:42 CST 2015@sequnceTaskID:3 29
printerBoltThreadID:31@Mon Sep 14 15:40:42 CST 2015@printerTaskID:2 sequenceBoltThreadID:33@Mon Sep 14 15:40:42 CST 2015@sequnceTaskID:3 30
printerBoltThreadID:31@Mon Sep 14 15:40:42 CST 2015@printerTaskID:2 sequenceBoltThreadID:33@Mon Sep 14 15:40:42 CST 2015@sequnceTaskID:3 31
printerBoltThreadID:31@Mon Sep 14 15:40:42 CST 2015@printerTaskID:2 sequenceBoltThreadID:33@Mon Sep 14 15:40:42 CST 2015@sequnceTaskID:3 32
printerBoltThreadID:31@Mon Sep 14 15:40:42 CST 2015@printerTaskID:2 sequenceBoltThreadID:33@Mon Sep 14 15:40:42 CST 2015@sequnceTaskID:3 33
printerBoltThreadID:31@Mon Sep 14 15:40:42 CST 2015@printerTaskID:2 sequenceBoltThreadID:33@Mon Sep 14 15:40:42 CST 2015@sequnceTaskID:3 34
printerBoltThreadID:31@Mon Sep 14 15:40:42 CST 2015@printerTaskID:2 sequenceBoltThreadID:33@Mon Sep 14 15:40:42 CST 2015@sequnceTaskID:3 35
printerBoltThreadID:31@Mon Sep 14 15:40:43 CST 2015@printerTaskID:2 sequenceBoltThreadID:33@Mon Sep 14 15:40:42 CST 2015@sequnceTaskID:3 36
printerBoltThreadID:31@Mon Sep 14 15:40:43 CST 2015@printerTaskID:2 sequenceBoltThreadID:33@Mon Sep 14 15:40:43 CST 2015@sequnceTaskID:3 37
printerBoltThreadID:31@Mon Sep 14 15:40:43 CST 2015@printerTaskID:2 sequenceBoltThreadID:33@Mon Sep 14 15:40:43 CST 2015@sequnceTaskID:3 38
printerBoltThreadID:31@Mon Sep 14 15:40:43 CST 2015@printerTaskID:2 sequenceBoltThreadID:33@Mon Sep 14 15:40:43 CST 2015@sequnceTaskID:3 39
printerBoltThreadID:31@Mon Sep 14 15:40:43 CST 2015@printerTaskID:2 sequenceBoltThreadID:33@Mon Sep 14 15:40:43 CST 2015@sequnceTaskID:3 40
printerBoltThreadID:31@Mon Sep 14 15:40:43 CST 2015@printerTaskID:2 sequenceBoltThreadID:33@Mon Sep 14 15:40:43 CST 2015@sequnceTaskID:3 41
printerBoltThreadID:31@Mon Sep 14 15:40:43 CST 2015@printerTaskID:2 sequenceBoltThreadID:33@Mon Sep 14 15:40:43 CST 2015@sequnceTaskID:3 42
printerBoltThreadID:31@Mon Sep 14 15:40:43 CST 2015@printerTaskID:2 sequenceBoltThreadID:33@Mon Sep 14 15:40:43 CST 2015@sequnceTaskID:3 43
printerBoltThreadID:31@Mon Sep 14 15:40:43 CST 2015@printerTaskID:2 sequenceBoltThreadID:33@Mon Sep 14 15:40:43 CST 2015@sequnceTaskID:3 44
printerBoltThreadID:31@Mon Sep 14 15:40:43 CST 2015@printerTaskID:2 sequenceBoltThreadID:33@Mon Sep 14 15:40:43 CST 2015@sequnceTaskID:3 45
printerBoltThreadID:31@Mon Sep 14 15:40:43 CST 2015@printerTaskID:2 sequenceBoltThreadID:33@Mon Sep 14 15:40:43 CST 2015@sequnceTaskID:3 46
printerBoltThreadID:31@Mon Sep 14 15:40:43 CST 2015@printerTaskID:2 sequenceBoltThreadID:33@Mon Sep 14 15:40:43 CST 2015@sequnceTaskID:3 47
printerBoltThreadID:31@Mon Sep 14 15:40:43 CST 2015@printerTaskID:2 sequenceBoltThreadID:33@Mon Sep 14 15:40:43 CST 2015@sequnceTaskID:3 48
printerBoltThreadID:31@Mon Sep 14 15:40:43 CST 2015@printerTaskID:2 sequenceBoltThreadID:33@Mon Sep 14 15:40:43 CST 2015@sequnceTaskID:3 49
printerBoltThreadID:31@Mon Sep 14 15:40:43 CST 2015@printerTaskID:2 sequenceBoltThreadID:33@Mon Sep 14 15:40:43 CST 2015@sequnceTaskID:3 50
printerBoltThreadID:31@Mon Sep 14 15:40:43 CST 2015@printerTaskID:2 sequenceBoltThreadID:33@Mon Sep 14 15:40:43 CST 2015@sequnceTaskID:3 51
printerBoltThreadID:31@Mon Sep 14 15:40:43 CST 2015@printerTaskID:2 sequenceBoltThreadID:33@Mon Sep 14 15:40:43 CST 2015@sequnceTaskID:3 52
printerBoltThreadID:31@Mon Sep 14 15:40:43 CST 2015@printerTaskID:2 sequenceBoltThreadID:33@Mon Sep 14 15:40:43 CST 2015@sequnceTaskID:3 53
printerBoltThreadID:31@Mon Sep 14 15:40:43 CST 2015@printerTaskID:2 sequenceBoltThreadID:33@Mon Sep 14 15:40:43 CST 2015@sequnceTaskID:3 54
printerBoltThreadID:31@Mon Sep 14 15:40:43 CST 2015@printerTaskID:2 sequenceBoltThreadID:33@Mon Sep 14 15:40:43 CST 2015@sequnceTaskID:3 55
printerBoltThreadID:31@Mon Sep 14 15:40:43 CST 2015@printerTaskID:2 sequenceBoltThreadID:33@Mon Sep 14 15:40:43 CST 2015@sequnceTaskID:3 56
printerBoltThreadID:31@Mon Sep 14 15:40:43 CST 2015@printerTaskID:2 sequenceBoltThreadID:33@Mon Sep 14 15:40:43 CST 2015@sequnceTaskID:3 57
printerBoltThreadID:31@Mon Sep 14 15:40:43 CST 2015@printerTaskID:2 sequenceBoltThreadID:33@Mon Sep 14 15:40:43 CST 2015@sequnceTaskID:3 58
printerBoltThreadID:31@Mon Sep 14 15:40:43 CST 2015@printerTaskID:2 sequenceBoltThreadID:33@Mon Sep 14 15:40:43 CST 2015@sequnceTaskID:3 59
printerBoltThreadID:31@Mon Sep 14 15:40:43 CST 2015@printerTaskID:2 sequenceBoltThreadID:33@Mon Sep 14 15:40:43 CST 2015@sequnceTaskID:3 60
printerBoltThreadID:31@Mon Sep 14 15:40:43 CST 2015@printerTaskID:2 sequenceBoltThreadID:33@Mon Sep 14 15:40:43 CST 2015@sequnceTaskID:3 61
printerBoltThreadID:31@Mon Sep 14 15:40:43 CST 2015@printerTaskID:2 sequenceBoltThreadID:33@Mon Sep 14 15:40:43 CST 2015@sequnceTaskID:3 62
printerBoltThreadID:31@Mon Sep 14 15:40:43 CST 2015@printerTaskID:2 sequenceBoltThreadID:33@Mon Sep 14 15:40:43 CST 2015@sequnceTaskID:3 63
printerBoltThreadID:31@Mon Sep 14 15:40:43 CST 2015@printerTaskID:2 sequenceBoltThreadID:33@Mon Sep 14 15:40:43 CST 2015@sequnceTaskID:3 64
printerBoltThreadID:31@Mon Sep 14 15:40:43 CST 2015@printerTaskID:2 sequenceBoltThreadID:33@Mon Sep 14 15:40:43 CST 2015@sequnceTaskID:3 65
printerBoltThreadID:31@Mon Sep 14 15:40:43 CST 2015@printerTaskID:2 sequenceBoltThreadID:33@Mon Sep 14 15:40:43 CST 2015@sequnceTaskID:3 66
printerBoltThreadID:31@Mon Sep 14 15:40:43 CST 2015@printerTaskID:2 sequenceBoltThreadID:33@Mon Sep 14 15:40:43 CST 2015@sequnceTaskID:3 67
printerBoltThreadID:31@Mon Sep 14 15:40:43 CST 2015@printerTaskID:2 sequenceBoltThreadID:33@Mon Sep 14 15:40:43 CST 2015@sequnceTaskID:3 68
printerBoltThreadID:31@Mon Sep 14 15:40:43 CST 2015@printerTaskID:2 sequenceBoltThreadID:33@Mon Sep 14 15:40:43 CST 2015@sequnceTaskID:3 69
printerBoltThreadID:31@Mon Sep 14 15:40:43 CST 2015@printerTaskID:2 sequenceBoltThreadID:33@Mon Sep 14 15:40:43 CST 2015@sequnceTaskID:3 70
printerBoltThreadID:31@Mon Sep 14 15:40:43 CST 2015@printerTaskID:2 sequenceBoltThreadID:33@Mon Sep 14 15:40:43 CST 2015@sequnceTaskID:3 71
printerBoltThreadID:31@Mon Sep 14 15:40:43 CST 2015@printerTaskID:2 sequenceBoltThreadID:33@Mon Sep 14 15:40:43 CST 2015@sequnceTaskID:3 72
printerBoltThreadID:31@Mon Sep 14 15:40:43 CST 2015@printerTaskID:2 sequenceBoltThreadID:33@Mon Sep 14 15:40:43 CST 2015@sequnceTaskID:3 73
printerBoltThreadID:31@Mon Sep 14 15:40:43 CST 2015@printerTaskID:2 sequenceBoltThreadID:33@Mon Sep 14 15:40:43 CST 2015@sequnceTaskID:3 74
printerBoltThreadID:31@Mon Sep 14 15:40:43 CST 2015@printerTaskID:2 sequenceBoltThreadID:33@Mon Sep 14 15:40:43 CST 2015@sequnceTaskID:3 75
printerBoltThreadID:31@Mon Sep 14 15:40:43 CST 2015@printerTaskID:2 sequenceBoltThreadID:33@Mon Sep 14 15:40:43 CST 2015@sequnceTaskID:3 76
printerBoltThreadID:31@Mon Sep 14 15:40:43 CST 2015@printerTaskID:2 sequenceBoltThreadID:33@Mon Sep 14 15:40:43 CST 2015@sequnceTaskID:3 77
printerBoltThreadID:31@Mon Sep 14 15:40:43 CST 2015@printerTaskID:2 sequenceBoltThreadID:33@Mon Sep 14 15:40:43 CST 2015@sequnceTaskID:3 78
printerBoltThreadID:31@Mon Sep 14 15:40:43 CST 2015@printerTaskID:2 sequenceBoltThreadID:33@Mon Sep 14 15:40:43 CST 2015@sequnceTaskID:3 79
printerBoltThreadID:31@Mon Sep 14 15:40:43 CST 2015@printerTaskID:2 sequenceBoltThreadID:33@Mon Sep 14 15:40:43 CST 2015@sequnceTaskID:3 80
printerBoltThreadID:31@Mon Sep 14 15:40:43 CST 2015@printerTaskID:2 sequenceBoltThreadID:33@Mon Sep 14 15:40:43 CST 2015@sequnceTaskID:3 81
printerBoltThreadID:31@Mon Sep 14 15:40:43 CST 2015@printerTaskID:2 sequenceBoltThreadID:33@Mon Sep 14 15:40:43 CST 2015@sequnceTaskID:3 82
printerBoltThreadID:31@Mon Sep 14 15:40:44 CST 2015@printerTaskID:2 sequenceBoltThreadID:33@Mon Sep 14 15:40:43 CST 2015@sequnceTaskID:3 83
printerBoltThreadID:31@Mon Sep 14 15:40:44 CST 2015@printerTaskID:2 sequenceBoltThreadID:33@Mon Sep 14 15:40:44 CST 2015@sequnceTaskID:3 84
printerBoltThreadID:31@Mon Sep 14 15:40:44 CST 2015@printerTaskID:2 sequenceBoltThreadID:33@Mon Sep 14 15:40:44 CST 2015@sequnceTaskID:3 85
printerBoltThreadID:31@Mon Sep 14 15:40:44 CST 2015@printerTaskID:2 sequenceBoltThreadID:33@Mon Sep 14 15:40:44 CST 2015@sequnceTaskID:3 86
printerBoltThreadID:31@Mon Sep 14 15:40:44 CST 2015@printerTaskID:2 sequenceBoltThreadID:33@Mon Sep 14 15:40:44 CST 2015@sequnceTaskID:3 87
printerBoltThreadID:31@Mon Sep 14 15:40:44 CST 2015@printerTaskID:2 sequenceBoltThreadID:33@Mon Sep 14 15:40:44 CST 2015@sequnceTaskID:3 88
printerBoltThreadID:31@Mon Sep 14 15:40:44 CST 2015@printerTaskID:2 sequenceBoltThreadID:33@Mon Sep 14 15:40:44 CST 2015@sequnceTaskID:3 89
printerBoltThreadID:31@Mon Sep 14 15:40:44 CST 2015@printerTaskID:2 sequenceBoltThreadID:33@Mon Sep 14 15:40:44 CST 2015@sequnceTaskID:3 90
printerBoltThreadID:31@Mon Sep 14 15:40:44 CST 2015@printerTaskID:2 sequenceBoltThreadID:33@Mon Sep 14 15:40:44 CST 2015@sequnceTaskID:3 91
printerBoltThreadID:31@Mon Sep 14 15:40:44 CST 2015@printerTaskID:2 sequenceBoltThreadID:33@Mon Sep 14 15:40:44 CST 2015@sequnceTaskID:3 92
printerBoltThreadID:31@Mon Sep 14 15:40:44 CST 2015@printerTaskID:2 sequenceBoltThreadID:33@Mon Sep 14 15:40:44 CST 2015@sequnceTaskID:3 93
printerBoltThreadID:31@Mon Sep 14 15:40:44 CST 2015@printerTaskID:2 sequenceBoltThreadID:33@Mon Sep 14 15:40:44 CST 2015@sequnceTaskID:3 94
printerBoltThreadID:31@Mon Sep 14 15:40:44 CST 2015@printerTaskID:2 sequenceBoltThreadID:33@Mon Sep 14 15:40:44 CST 2015@sequnceTaskID:3 95
printerBoltThreadID:31@Mon Sep 14 15:40:44 CST 2015@printerTaskID:2 sequenceBoltThreadID:33@Mon Sep 14 15:40:44 CST 2015@sequnceTaskID:3 96
printerBoltThreadID:31@Mon Sep 14 15:40:44 CST 2015@printerTaskID:2 sequenceBoltThreadID:33@Mon Sep 14 15:40:44 CST 2015@sequnceTaskID:3 97
printerBoltThreadID:31@Mon Sep 14 15:40:44 CST 2015@printerTaskID:2 sequenceBoltThreadID:33@Mon Sep 14 15:40:44 CST 2015@sequnceTaskID:3 98
printerBoltThreadID:31@Mon Sep 14 15:40:44 CST 2015@printerTaskID:2 sequenceBoltThreadID:33@Mon Sep 14 15:40:44 CST 2015@sequnceTaskID:3 99
正如,我們地預期,輸出確實變成了順序輸出。這樣就提醒我們,在需要順序處理某些數據地時候,一定不能忽視數據源的順序,具體到Kafka,就是採用合適的分區策略,利用單個分區的有序特性來達成自己的目的。
場景2:探究Global Grouping的特性。
Global Grouping不會對數據流做分區處理,而是將整個數據流完整發送到具有最小任務ID的BOLT任務上,也就是說,不論一個BOLT開幾個任務,只有一個任務會獲取數據流,也就是說,在我們的示例中,不論我們開啓幾個BOLT任務,我們得到也總是有序的輸出。爲測試,我們的想法,我們將以如下的方式來提交Topology到Storm集羣中:
bin/storm jar /tmp/withdepends-istorm-0.0.1-SNAPSHOT.jar com.example.istorm.KafkaTopology 2 2 2 2 2 2 2
這樣之後,我們向數據源TOPIC中發送數據,得到的輸出仍舊是有序的(爲了避免篇幅冗餘,不再貼具體輸出)。多個worker去讀取數據的時候,也是要參照Zookeeper中存儲的偏移量的,所以即使是多個worker去讀Kafka隊列(因爲只有一個分區)中的數據,得到的數據也是有序的,並不會在數據讀取時發生順序上的變動。
場景3:探究Shuffle Grouping的特性。
Shuffle Grouping的分發方式是把數據流中的元組隨機地在任務之間分發,既然這裏提到了隨機,那麼就有可能出現失序地問題。但是,這裏必須十分注意執行器和任務之間地關係。執行器就是一個獨立地線程,而任務就是一個Bolt地實例對象,而一個執行器可以執行多個任務。如果每個Bolt不管有多少個任務,但是隻有一個執行器來執行它們,那麼理論上是不會失序的,那麼實際上是怎樣的呢?爲了驗證,我們理論上的想法,可以在啓動Topology的時候,將每個Bolt的執行器都設置成1,而任務設置成多個,然後再做測試(不要忘了將Grouping改成Shuffle Grouping,並重新打包):
bin/storm jar /tmp/withdepends-istorm-0.0.1-SNAPSHOT.jar com.example.istorm.KafkaTopology 2 2 2 1 10 1 10
輸出如下:
printerBoltThreadID:38@Mon Sep 14 16:21:09 CST 2015@printerTaskID:3 sequenceBoltThreadID:38@Mon Sep 14 16:21:09 CST 2015@sequnceTaskID:13 0
printerBoltThreadID:38@Mon Sep 14 16:21:09 CST 2015@printerTaskID:10 sequenceBoltThreadID:38@Mon Sep 14 16:21:09 CST 2015@sequnceTaskID:21 1
printerBoltThreadID:38@Mon Sep 14 16:21:09 CST 2015@printerTaskID:7 sequenceBoltThreadID:38@Mon Sep 14 16:21:09 CST 2015@sequnceTaskID:15 2
printerBoltThreadID:38@Mon Sep 14 16:21:09 CST 2015@printerTaskID:11 sequenceBoltThreadID:38@Mon Sep 14 16:21:09 CST 2015@sequnceTaskID:17 3
printerBoltThreadID:38@Mon Sep 14 16:21:09 CST 2015@printerTaskID:8 sequenceBoltThreadID:38@Mon Sep 14 16:21:09 CST 2015@sequnceTaskID:14 4
printerBoltThreadID:38@Mon Sep 14 16:21:09 CST 2015@printerTaskID:4 sequenceBoltThreadID:38@Mon Sep 14 16:21:09 CST 2015@sequnceTaskID:19 5
printerBoltThreadID:38@Mon Sep 14 16:21:10 CST 2015@printerTaskID:9 sequenceBoltThreadID:38@Mon Sep 14 16:21:09 CST 2015@sequnceTaskID:22 6
printerBoltThreadID:38@Mon Sep 14 16:21:10 CST 2015@printerTaskID:6 sequenceBoltThreadID:38@Mon Sep 14 16:21:09 CST 2015@sequnceTaskID:16 7
printerBoltThreadID:38@Mon Sep 14 16:21:10 CST 2015@printerTaskID:5 sequenceBoltThreadID:38@Mon Sep 14 16:21:09 CST 2015@sequnceTaskID:18 8
printerBoltThreadID:38@Mon Sep 14 16:21:10 CST 2015@printerTaskID:12 sequenceBoltThreadID:38@Mon Sep 14 16:21:09 CST 2015@sequnceTaskID:20 9
printerBoltThreadID:38@Mon Sep 14 16:21:10 CST 2015@printerTaskID:3 sequenceBoltThreadID:38@Mon Sep 14 16:21:10 CST 2015@sequnceTaskID:13 10
printerBoltThreadID:38@Mon Sep 14 16:21:10 CST 2015@printerTaskID:5 sequenceBoltThreadID:38@Mon Sep 14 16:21:10 CST 2015@sequnceTaskID:21 11
printerBoltThreadID:38@Mon Sep 14 16:21:10 CST 2015@printerTaskID:9 sequenceBoltThreadID:38@Mon Sep 14 16:21:10 CST 2015@sequnceTaskID:19 12
printerBoltThreadID:38@Mon Sep 14 16:21:10 CST 2015@printerTaskID:7 sequenceBoltThreadID:38@Mon Sep 14 16:21:10 CST 2015@sequnceTaskID:18 13
printerBoltThreadID:38@Mon Sep 14 16:21:10 CST 2015@printerTaskID:4 sequenceBoltThreadID:38@Mon Sep 14 16:21:10 CST 2015@sequnceTaskID:17 14
printerBoltThreadID:38@Mon Sep 14 16:21:10 CST 2015@printerTaskID:12 sequenceBoltThreadID:38@Mon Sep 14 16:21:10 CST 2015@sequnceTaskID:16 15
printerBoltThreadID:38@Mon Sep 14 16:21:10 CST 2015@printerTaskID:6 sequenceBoltThreadID:38@Mon Sep 14 16:21:10 CST 2015@sequnceTaskID:20 16
printerBoltThreadID:38@Mon Sep 14 16:21:10 CST 2015@printerTaskID:8 sequenceBoltThreadID:38@Mon Sep 14 16:21:10 CST 2015@sequnceTaskID:15 17
printerBoltThreadID:38@Mon Sep 14 16:21:10 CST 2015@printerTaskID:11 sequenceBoltThreadID:38@Mon Sep 14 16:21:10 CST 2015@sequnceTaskID:22 18
printerBoltThreadID:38@Mon Sep 14 16:21:10 CST 2015@printerTaskID:10 sequenceBoltThreadID:38@Mon Sep 14 16:21:10 CST 2015@sequnceTaskID:14 19
printerBoltThreadID:38@Mon Sep 14 16:21:10 CST 2015@printerTaskID:8 sequenceBoltThreadID:38@Mon Sep 14 16:21:10 CST 2015@sequnceTaskID:17 20
printerBoltThreadID:38@Mon Sep 14 16:21:10 CST 2015@printerTaskID:9 sequenceBoltThreadID:38@Mon Sep 14 16:21:10 CST 2015@sequnceTaskID:22 21
printerBoltThreadID:38@Mon Sep 14 16:21:10 CST 2015@printerTaskID:11 sequenceBoltThreadID:38@Mon Sep 14 16:21:10 CST 2015@sequnceTaskID:13 22
printerBoltThreadID:38@Mon Sep 14 16:21:10 CST 2015@printerTaskID:4 sequenceBoltThreadID:38@Mon Sep 14 16:21:10 CST 2015@sequnceTaskID:15 23
printerBoltThreadID:38@Mon Sep 14 16:21:10 CST 2015@printerTaskID:6 sequenceBoltThreadID:38@Mon Sep 14 16:21:10 CST 2015@sequnceTaskID:19 24
printerBoltThreadID:38@Mon Sep 14 16:21:10 CST 2015@printerTaskID:5 sequenceBoltThreadID:38@Mon Sep 14 16:21:10 CST 2015@sequnceTaskID:20 25
printerBoltThreadID:38@Mon Sep 14 16:21:10 CST 2015@printerTaskID:3 sequenceBoltThreadID:38@Mon Sep 14 16:21:10 CST 2015@sequnceTaskID:21 26
printerBoltThreadID:38@Mon Sep 14 16:21:10 CST 2015@printerTaskID:7 sequenceBoltThreadID:38@Mon Sep 14 16:21:10 CST 2015@sequnceTaskID:14 27
printerBoltThreadID:38@Mon Sep 14 16:21:10 CST 2015@printerTaskID:12 sequenceBoltThreadID:38@Mon Sep 14 16:21:10 CST 2015@sequnceTaskID:16 28
printerBoltThreadID:38@Mon Sep 14 16:21:10 CST 2015@printerTaskID:10 sequenceBoltThreadID:38@Mon Sep 14 16:21:10 CST 2015@sequnceTaskID:18 29
printerBoltThreadID:38@Mon Sep 14 16:21:10 CST 2015@printerTaskID:11 sequenceBoltThreadID:38@Mon Sep 14 16:21:10 CST 2015@sequnceTaskID:16 30
printerBoltThreadID:38@Mon Sep 14 16:21:10 CST 2015@printerTaskID:8 sequenceBoltThreadID:38@Mon Sep 14 16:21:10 CST 2015@sequnceTaskID:13 31
printerBoltThreadID:38@Mon Sep 14 16:21:10 CST 2015@printerTaskID:9 sequenceBoltThreadID:38@Mon Sep 14 16:21:10 CST 2015@sequnceTaskID:22 32
printerBoltThreadID:38@Mon Sep 14 16:21:10 CST 2015@printerTaskID:5 sequenceBoltThreadID:38@Mon Sep 14 16:21:10 CST 2015@sequnceTaskID:18 33
printerBoltThreadID:38@Mon Sep 14 16:21:10 CST 2015@printerTaskID:12 sequenceBoltThreadID:38@Mon Sep 14 16:21:10 CST 2015@sequnceTaskID:14 34
printerBoltThreadID:38@Mon Sep 14 16:21:10 CST 2015@printerTaskID:10 sequenceBoltThreadID:38@Mon Sep 14 16:21:10 CST 2015@sequnceTaskID:21 35
printerBoltThreadID:38@Mon Sep 14 16:21:10 CST 2015@printerTaskID:7 sequenceBoltThreadID:38@Mon Sep 14 16:21:10 CST 2015@sequnceTaskID:20 36
printerBoltThreadID:38@Mon Sep 14 16:21:10 CST 2015@printerTaskID:4 sequenceBoltThreadID:38@Mon Sep 14 16:21:10 CST 2015@sequnceTaskID:15 37
printerBoltThreadID:38@Mon Sep 14 16:21:10 CST 2015@printerTaskID:3 sequenceBoltThreadID:38@Mon Sep 14 16:21:10 CST 2015@sequnceTaskID:19 38
printerBoltThreadID:38@Mon Sep 14 16:21:10 CST 2015@printerTaskID:6 sequenceBoltThreadID:38@Mon Sep 14 16:21:10 CST 2015@sequnceTaskID:17 39
printerBoltThreadID:38@Mon Sep 14 16:21:10 CST 2015@printerTaskID:3 sequenceBoltThreadID:38@Mon Sep 14 16:21:10 CST 2015@sequnceTaskID:14 40
printerBoltThreadID:38@Mon Sep 14 16:21:10 CST 2015@printerTaskID:8 sequenceBoltThreadID:38@Mon Sep 14 16:21:10 CST 2015@sequnceTaskID:16 41
printerBoltThreadID:38@Mon Sep 14 16:21:10 CST 2015@printerTaskID:10 sequenceBoltThreadID:38@Mon Sep 14 16:21:10 CST 2015@sequnceTaskID:13 42
printerBoltThreadID:38@Mon Sep 14 16:21:10 CST 2015@printerTaskID:9 sequenceBoltThreadID:38@Mon Sep 14 16:21:10 CST 2015@sequnceTaskID:20 43
printerBoltThreadID:38@Mon Sep 14 16:21:10 CST 2015@printerTaskID:6 sequenceBoltThreadID:38@Mon Sep 14 16:21:10 CST 2015@sequnceTaskID:17 44
printerBoltThreadID:38@Mon Sep 14 16:21:10 CST 2015@printerTaskID:7 sequenceBoltThreadID:38@Mon Sep 14 16:21:10 CST 2015@sequnceTaskID:18 45
printerBoltThreadID:38@Mon Sep 14 16:21:10 CST 2015@printerTaskID:4 sequenceBoltThreadID:38@Mon Sep 14 16:21:10 CST 2015@sequnceTaskID:22 46
printerBoltThreadID:38@Mon Sep 14 16:21:11 CST 2015@printerTaskID:12 sequenceBoltThreadID:38@Mon Sep 14 16:21:10 CST 2015@sequnceTaskID:19 47
printerBoltThreadID:38@Mon Sep 14 16:21:11 CST 2015@printerTaskID:5 sequenceBoltThreadID:38@Mon Sep 14 16:21:11 CST 2015@sequnceTaskID:21 48
printerBoltThreadID:38@Mon Sep 14 16:21:11 CST 2015@printerTaskID:11 sequenceBoltThreadID:38@Mon Sep 14 16:21:11 CST 2015@sequnceTaskID:15 49
printerBoltThreadID:38@Mon Sep 14 16:21:11 CST 2015@printerTaskID:5 sequenceBoltThreadID:38@Mon Sep 14 16:21:11 CST 2015@sequnceTaskID:17 50
printerBoltThreadID:38@Mon Sep 14 16:21:11 CST 2015@printerTaskID:6 sequenceBoltThreadID:38@Mon Sep 14 16:21:11 CST 2015@sequnceTaskID:21 51
printerBoltThreadID:38@Mon Sep 14 16:21:11 CST 2015@printerTaskID:4 sequenceBoltThreadID:38@Mon Sep 14 16:21:11 CST 2015@sequnceTaskID:16 52
printerBoltThreadID:38@Mon Sep 14 16:21:11 CST 2015@printerTaskID:10 sequenceBoltThreadID:38@Mon Sep 14 16:21:11 CST 2015@sequnceTaskID:22 53
printerBoltThreadID:38@Mon Sep 14 16:21:11 CST 2015@printerTaskID:12 sequenceBoltThreadID:38@Mon Sep 14 16:21:11 CST 2015@sequnceTaskID:13 54
printerBoltThreadID:38@Mon Sep 14 16:21:11 CST 2015@printerTaskID:8 sequenceBoltThreadID:38@Mon Sep 14 16:21:11 CST 2015@sequnceTaskID:15 55
printerBoltThreadID:38@Mon Sep 14 16:21:11 CST 2015@printerTaskID:9 sequenceBoltThreadID:38@Mon Sep 14 16:21:11 CST 2015@sequnceTaskID:20 56
printerBoltThreadID:38@Mon Sep 14 16:21:11 CST 2015@printerTaskID:11 sequenceBoltThreadID:38@Mon Sep 14 16:21:11 CST 2015@sequnceTaskID:19 57
printerBoltThreadID:38@Mon Sep 14 16:21:11 CST 2015@printerTaskID:3 sequenceBoltThreadID:38@Mon Sep 14 16:21:11 CST 2015@sequnceTaskID:18 58
printerBoltThreadID:38@Mon Sep 14 16:21:11 CST 2015@printerTaskID:7 sequenceBoltThreadID:38@Mon Sep 14 16:21:11 CST 2015@sequnceTaskID:14 59
printerBoltThreadID:38@Mon Sep 14 16:21:11 CST 2015@printerTaskID:7 sequenceBoltThreadID:38@Mon Sep 14 16:21:11 CST 2015@sequnceTaskID:22 60
printerBoltThreadID:38@Mon Sep 14 16:21:11 CST 2015@printerTaskID:9 sequenceBoltThreadID:38@Mon Sep 14 16:21:11 CST 2015@sequnceTaskID:16 61
printerBoltThreadID:38@Mon Sep 14 16:21:11 CST 2015@printerTaskID:6 sequenceBoltThreadID:38@Mon Sep 14 16:21:11 CST 2015@sequnceTaskID:19 62
printerBoltThreadID:38@Mon Sep 14 16:21:11 CST 2015@printerTaskID:12 sequenceBoltThreadID:38@Mon Sep 14 16:21:11 CST 2015@sequnceTaskID:18 63
printerBoltThreadID:38@Mon Sep 14 16:21:11 CST 2015@printerTaskID:5 sequenceBoltThreadID:38@Mon Sep 14 16:21:11 CST 2015@sequnceTaskID:17 64
printerBoltThreadID:38@Mon Sep 14 16:21:11 CST 2015@printerTaskID:11 sequenceBoltThreadID:38@Mon Sep 14 16:21:11 CST 2015@sequnceTaskID:13 65
printerBoltThreadID:38@Mon Sep 14 16:21:11 CST 2015@printerTaskID:3 sequenceBoltThreadID:38@Mon Sep 14 16:21:11 CST 2015@sequnceTaskID:15 66
printerBoltThreadID:38@Mon Sep 14 16:21:11 CST 2015@printerTaskID:4 sequenceBoltThreadID:38@Mon Sep 14 16:21:11 CST 2015@sequnceTaskID:20 67
printerBoltThreadID:38@Mon Sep 14 16:21:11 CST 2015@printerTaskID:8 sequenceBoltThreadID:38@Mon Sep 14 16:21:11 CST 2015@sequnceTaskID:21 68
printerBoltThreadID:38@Mon Sep 14 16:21:11 CST 2015@printerTaskID:10 sequenceBoltThreadID:38@Mon Sep 14 16:21:11 CST 2015@sequnceTaskID:14 69
printerBoltThreadID:38@Mon Sep 14 16:21:11 CST 2015@printerTaskID:8 sequenceBoltThreadID:38@Mon Sep 14 16:21:11 CST 2015@sequnceTaskID:17 70
printerBoltThreadID:38@Mon Sep 14 16:21:11 CST 2015@printerTaskID:4 sequenceBoltThreadID:38@Mon Sep 14 16:21:11 CST 2015@sequnceTaskID:13 71
printerBoltThreadID:38@Mon Sep 14 16:21:11 CST 2015@printerTaskID:11 sequenceBoltThreadID:38@Mon Sep 14 16:21:11 CST 2015@sequnceTaskID:20 72
printerBoltThreadID:38@Mon Sep 14 16:21:11 CST 2015@printerTaskID:7 sequenceBoltThreadID:38@Mon Sep 14 16:21:11 CST 2015@sequnceTaskID:18 73
printerBoltThreadID:38@Mon Sep 14 16:21:11 CST 2015@printerTaskID:12 sequenceBoltThreadID:38@Mon Sep 14 16:21:11 CST 2015@sequnceTaskID:19 74
printerBoltThreadID:38@Mon Sep 14 16:21:11 CST 2015@printerTaskID:5 sequenceBoltThreadID:38@Mon Sep 14 16:21:11 CST 2015@sequnceTaskID:21 75
printerBoltThreadID:38@Mon Sep 14 16:21:11 CST 2015@printerTaskID:3 sequenceBoltThreadID:38@Mon Sep 14 16:21:11 CST 2015@sequnceTaskID:14 76
printerBoltThreadID:38@Mon Sep 14 16:21:11 CST 2015@printerTaskID:9 sequenceBoltThreadID:38@Mon Sep 14 16:21:11 CST 2015@sequnceTaskID:15 77
printerBoltThreadID:38@Mon Sep 14 16:21:11 CST 2015@printerTaskID:6 sequenceBoltThreadID:38@Mon Sep 14 16:21:11 CST 2015@sequnceTaskID:16 78
printerBoltThreadID:38@Mon Sep 14 16:21:11 CST 2015@printerTaskID:10 sequenceBoltThreadID:38@Mon Sep 14 16:21:11 CST 2015@sequnceTaskID:22 79
printerBoltThreadID:38@Mon Sep 14 16:21:11 CST 2015@printerTaskID:11 sequenceBoltThreadID:38@Mon Sep 14 16:21:11 CST 2015@sequnceTaskID:14 80
printerBoltThreadID:38@Mon Sep 14 16:21:11 CST 2015@printerTaskID:7 sequenceBoltThreadID:38@Mon Sep 14 16:21:11 CST 2015@sequnceTaskID:21 81
printerBoltThreadID:38@Mon Sep 14 16:21:11 CST 2015@printerTaskID:10 sequenceBoltThreadID:38@Mon Sep 14 16:21:11 CST 2015@sequnceTaskID:13 82
printerBoltThreadID:38@Mon Sep 14 16:21:11 CST 2015@printerTaskID:9 sequenceBoltThreadID:38@Mon Sep 14 16:21:11 CST 2015@sequnceTaskID:22 83
printerBoltThreadID:38@Mon Sep 14 16:21:11 CST 2015@printerTaskID:3 sequenceBoltThreadID:38@Mon Sep 14 16:21:11 CST 2015@sequnceTaskID:16 84
printerBoltThreadID:38@Mon Sep 14 16:21:11 CST 2015@printerTaskID:6 sequenceBoltThreadID:38@Mon Sep 14 16:21:11 CST 2015@sequnceTaskID:18 85
printerBoltThreadID:38@Mon Sep 14 16:21:12 CST 2015@printerTaskID:8 sequenceBoltThreadID:38@Mon Sep 14 16:21:11 CST 2015@sequnceTaskID:20 86
printerBoltThreadID:38@Mon Sep 14 16:21:12 CST 2015@printerTaskID:5 sequenceBoltThreadID:38@Mon Sep 14 16:21:12 CST 2015@sequnceTaskID:15 87
printerBoltThreadID:38@Mon Sep 14 16:21:12 CST 2015@printerTaskID:4 sequenceBoltThreadID:38@Mon Sep 14 16:21:12 CST 2015@sequnceTaskID:19 88
printerBoltThreadID:38@Mon Sep 14 16:21:12 CST 2015@printerTaskID:12 sequenceBoltThreadID:38@Mon Sep 14 16:21:12 CST 2015@sequnceTaskID:17 89
printerBoltThreadID:38@Mon Sep 14 16:21:12 CST 2015@printerTaskID:8 sequenceBoltThreadID:38@Mon Sep 14 16:21:12 CST 2015@sequnceTaskID:17 90
printerBoltThreadID:38@Mon Sep 14 16:21:12 CST 2015@printerTaskID:10 sequenceBoltThreadID:38@Mon Sep 14 16:21:12 CST 2015@sequnceTaskID:20 91
printerBoltThreadID:38@Mon Sep 14 16:21:12 CST 2015@printerTaskID:11 sequenceBoltThreadID:38@Mon Sep 14 16:21:12 CST 2015@sequnceTaskID:19 92
printerBoltThreadID:38@Mon Sep 14 16:21:12 CST 2015@printerTaskID:12 sequenceBoltThreadID:38@Mon Sep 14 16:21:12 CST 2015@sequnceTaskID:15 93
printerBoltThreadID:38@Mon Sep 14 16:21:12 CST 2015@printerTaskID:5 sequenceBoltThreadID:38@Mon Sep 14 16:21:12 CST 2015@sequnceTaskID:16 94
printerBoltThreadID:38@Mon Sep 14 16:21:12 CST 2015@printerTaskID:6 sequenceBoltThreadID:38@Mon Sep 14 16:21:12 CST 2015@sequnceTaskID:18 95
printerBoltThreadID:38@Mon Sep 14 16:21:12 CST 2015@printerTaskID:9 sequenceBoltThreadID:38@Mon Sep 14 16:21:12 CST 2015@sequnceTaskID:21 96
printerBoltThreadID:38@Mon Sep 14 16:21:12 CST 2015@printerTaskID:7 sequenceBoltThreadID:38@Mon Sep 14 16:21:12 CST 2015@sequnceTaskID:22 97
printerBoltThreadID:38@Mon Sep 14 16:21:12 CST 2015@printerTaskID:4 sequenceBoltThreadID:38@Mon Sep 14 16:21:12 CST 2015@sequnceTaskID:14 98
printerBoltThreadID:38@Mon Sep 14 16:21:12 CST 2015@printerTaskID:3 sequenceBoltThreadID:38@Mon Sep 14 16:21:12 CST 2015@sequnceTaskID:13 99
可以看到,數據流中的元組被平均分配到了不同的任務中,但是輸出依舊是有序的。那麼繼續我們的思路,如果要讓輸出失序,只要將其中一個BOLT的執行器設置成2個即可:
bin/storm jar /tmp/withdepends-istorm-0.0.1-SNAPSHOT.jar com.example.istorm.KafkaTopology 2 2 2 2 10 1 10
這裏,我們將SequenceBolt的執行器設置成了2個,其對應的輸出是:
printerBoltThreadID:40@Mon Sep 14 16:24:37 CST 2015@printerTaskID:10 sequenceBoltThreadID:38@Mon Sep 14 16:24:37 CST 2015@sequnceTaskID:18 2
printerBoltThreadID:40@Mon Sep 14 16:24:37 CST 2015@printerTaskID:6 sequenceBoltThreadID:38@Mon Sep 14 16:24:37 CST 2015@sequnceTaskID:20 3
printerBoltThreadID:40@Mon Sep 14 16:24:37 CST 2015@printerTaskID:4 sequenceBoltThreadID:38@Mon Sep 14 16:24:37 CST 2015@sequnceTaskID:21 4
printerBoltThreadID:40@Mon Sep 14 16:24:37 CST 2015@printerTaskID:3 sequenceBoltThreadID:38@Mon Sep 14 16:24:37 CST 2015@sequnceTaskID:16 0
printerBoltThreadID:40@Mon Sep 14 16:24:37 CST 2015@printerTaskID:9 sequenceBoltThreadID:38@Mon Sep 14 16:24:37 CST 2015@sequnceTaskID:14 1
printerBoltThreadID:40@Mon Sep 14 16:24:37 CST 2015@printerTaskID:9 sequenceBoltThreadID:38@Mon Sep 14 16:24:37 CST 2015@sequnceTaskID:19 7
printerBoltThreadID:40@Mon Sep 14 16:24:37 CST 2015@printerTaskID:8 sequenceBoltThreadID:38@Mon Sep 14 16:24:37 CST 2015@sequnceTaskID:15 5
printerBoltThreadID:40@Mon Sep 14 16:24:37 CST 2015@printerTaskID:11 sequenceBoltThreadID:38@Mon Sep 14 16:24:37 CST 2015@sequnceTaskID:13 6
printerBoltThreadID:40@Mon Sep 14 16:24:37 CST 2015@printerTaskID:7 sequenceBoltThreadID:38@Mon Sep 14 16:24:37 CST 2015@sequnceTaskID:22 8
printerBoltThreadID:40@Mon Sep 14 16:24:37 CST 2015@printerTaskID:12 sequenceBoltThreadID:38@Mon Sep 14 16:24:37 CST 2015@sequnceTaskID:17 9
printerBoltThreadID:40@Mon Sep 14 16:24:37 CST 2015@printerTaskID:12 sequenceBoltThreadID:38@Mon Sep 14 16:24:37 CST 2015@sequnceTaskID:22 10
printerBoltThreadID:40@Mon Sep 14 16:24:37 CST 2015@printerTaskID:8 sequenceBoltThreadID:38@Mon Sep 14 16:24:37 CST 2015@sequnceTaskID:19 12
printerBoltThreadID:40@Mon Sep 14 16:24:37 CST 2015@printerTaskID:10 sequenceBoltThreadID:38@Mon Sep 14 16:24:37 CST 2015@sequnceTaskID:13 11
printerBoltThreadID:40@Mon Sep 14 16:24:37 CST 2015@printerTaskID:3 sequenceBoltThreadID:38@Mon Sep 14 16:24:37 CST 2015@sequnceTaskID:20 15
printerBoltThreadID:40@Mon Sep 14 16:24:37 CST 2015@printerTaskID:5 sequenceBoltThreadID:38@Mon Sep 14 16:24:37 CST 2015@sequnceTaskID:18 17
printerBoltThreadID:40@Mon Sep 14 16:24:37 CST 2015@printerTaskID:6 sequenceBoltThreadID:38@Mon Sep 14 16:24:37 CST 2015@sequnceTaskID:16 13
printerBoltThreadID:40@Mon Sep 14 16:24:37 CST 2015@printerTaskID:7 sequenceBoltThreadID:38@Mon Sep 14 16:24:37 CST 2015@sequnceTaskID:15 14
printerBoltThreadID:40@Mon Sep 14 16:24:37 CST 2015@printerTaskID:11 sequenceBoltThreadID:38@Mon Sep 14 16:24:37 CST 2015@sequnceTaskID:21 18
printerBoltThreadID:40@Mon Sep 14 16:24:37 CST 2015@printerTaskID:4 sequenceBoltThreadID:38@Mon Sep 14 16:24:37 CST 2015@sequnceTaskID:14 16
printerBoltThreadID:40@Mon Sep 14 16:24:37 CST 2015@printerTaskID:4 sequenceBoltThreadID:38@Mon Sep 14 16:24:37 CST 2015@sequnceTaskID:19 20
printerBoltThreadID:40@Mon Sep 14 16:24:37 CST 2015@printerTaskID:11 sequenceBoltThreadID:38@Mon Sep 14 16:24:37 CST 2015@sequnceTaskID:22 21
printerBoltThreadID:40@Mon Sep 14 16:24:37 CST 2015@printerTaskID:5 sequenceBoltThreadID:38@Mon Sep 14 16:24:37 CST 2015@sequnceTaskID:17 19
printerBoltThreadID:40@Mon Sep 14 16:24:37 CST 2015@printerTaskID:7 sequenceBoltThreadID:38@Mon Sep 14 16:24:37 CST 2015@sequnceTaskID:18 24
printerBoltThreadID:40@Mon Sep 14 16:24:37 CST 2015@printerTaskID:12 sequenceBoltThreadID:38@Mon Sep 14 16:24:37 CST 2015@sequnceTaskID:21 26
printerBoltThreadID:40@Mon Sep 14 16:24:37 CST 2015@printerTaskID:12 sequenceBoltThreadID:38@Mon Sep 14 16:24:37 CST 2015@sequnceTaskID:13 22
printerBoltThreadID:40@Mon Sep 14 16:24:37 CST 2015@printerTaskID:6 sequenceBoltThreadID:38@Mon Sep 14 16:24:37 CST 2015@sequnceTaskID:20 29
printerBoltThreadID:40@Mon Sep 14 16:24:37 CST 2015@printerTaskID:7 sequenceBoltThreadID:38@Mon Sep 14 16:24:37 CST 2015@sequnceTaskID:14 23
printerBoltThreadID:40@Mon Sep 14 16:24:37 CST 2015@printerTaskID:10 sequenceBoltThreadID:38@Mon Sep 14 16:24:37 CST 2015@sequnceTaskID:15 25
printerBoltThreadID:40@Mon Sep 14 16:24:37 CST 2015@printerTaskID:3 sequenceBoltThreadID:38@Mon Sep 14 16:24:37 CST 2015@sequnceTaskID:19 30
printerBoltThreadID:40@Mon Sep 14 16:24:37 CST 2015@printerTaskID:5 sequenceBoltThreadID:38@Mon Sep 14 16:24:37 CST 2015@sequnceTaskID:20 32
printerBoltThreadID:40@Mon Sep 14 16:24:37 CST 2015@printerTaskID:11 sequenceBoltThreadID:38@Mon Sep 14 16:24:37 CST 2015@sequnceTaskID:16 27
printerBoltThreadID:40@Mon Sep 14 16:24:37 CST 2015@printerTaskID:8 sequenceBoltThreadID:38@Mon Sep 14 16:24:37 CST 2015@sequnceTaskID:21 34
printerBoltThreadID:40@Mon Sep 14 16:24:37 CST 2015@printerTaskID:5 sequenceBoltThreadID:38@Mon Sep 14 16:24:37 CST 2015@sequnceTaskID:17 28
printerBoltThreadID:40@Mon Sep 14 16:24:37 CST 2015@printerTaskID:9 sequenceBoltThreadID:38@Mon Sep 14 16:24:37 CST 2015@sequnceTaskID:22 37
printerBoltThreadID:40@Mon Sep 14 16:24:37 CST 2015@printerTaskID:10 sequenceBoltThreadID:38@Mon Sep 14 16:24:37 CST 2015@sequnceTaskID:18 38
printerBoltThreadID:40@Mon Sep 14 16:24:37 CST 2015@printerTaskID:8 sequenceBoltThreadID:38@Mon Sep 14 16:24:37 CST 2015@sequnceTaskID:13 31
printerBoltThreadID:40@Mon Sep 14 16:24:37 CST 2015@printerTaskID:11 sequenceBoltThreadID:38@Mon Sep 14 16:24:37 CST 2015@sequnceTaskID:21 40
printerBoltThreadID:40@Mon Sep 14 16:24:37 CST 2015@printerTaskID:9 sequenceBoltThreadID:38@Mon Sep 14 16:24:37 CST 2015@sequnceTaskID:15 33
printerBoltThreadID:40@Mon Sep 14 16:24:37 CST 2015@printerTaskID:4 sequenceBoltThreadID:38@Mon Sep 14 16:24:37 CST 2015@sequnceTaskID:17 35
printerBoltThreadID:40@Mon Sep 14 16:24:37 CST 2015@printerTaskID:8 sequenceBoltThreadID:38@Mon Sep 14 16:24:37 CST 2015@sequnceTaskID:22 42
printerBoltThreadID:40@Mon Sep 14 16:24:38 CST 2015@printerTaskID:3 sequenceBoltThreadID:38@Mon Sep 14 16:24:37 CST 2015@sequnceTaskID:16 36
printerBoltThreadID:40@Mon Sep 14 16:24:38 CST 2015@printerTaskID:7 sequenceBoltThreadID:38@Mon Sep 14 16:24:37 CST 2015@sequnceTaskID:18 43
printerBoltThreadID:40@Mon Sep 14 16:24:38 CST 2015@printerTaskID:4 sequenceBoltThreadID:38@Mon Sep 14 16:24:38 CST 2015@sequnceTaskID:19 47
printerBoltThreadID:40@Mon Sep 14 16:24:38 CST 2015@printerTaskID:6 sequenceBoltThreadID:38@Mon Sep 14 16:24:38 CST 2015@sequnceTaskID:14 39
printerBoltThreadID:40@Mon Sep 14 16:24:38 CST 2015@printerTaskID:8 sequenceBoltThreadID:38@Mon Sep 14 16:24:38 CST 2015@sequnceTaskID:13 41
printerBoltThreadID:40@Mon Sep 14 16:24:38 CST 2015@printerTaskID:5 sequenceBoltThreadID:38@Mon Sep 14 16:24:38 CST 2015@sequnceTaskID:20 49
printerBoltThreadID:40@Mon Sep 14 16:24:38 CST 2015@printerTaskID:9 sequenceBoltThreadID:38@Mon Sep 14 16:24:38 CST 2015@sequnceTaskID:18 51
printerBoltThreadID:40@Mon Sep 14 16:24:38 CST 2015@printerTaskID:6 sequenceBoltThreadID:38@Mon Sep 14 16:24:38 CST 2015@sequnceTaskID:14 44
printerBoltThreadID:40@Mon Sep 14 16:24:38 CST 2015@printerTaskID:10 sequenceBoltThreadID:38@Mon Sep 14 16:24:38 CST 2015@sequnceTaskID:17 45
printerBoltThreadID:40@Mon Sep 14 16:24:38 CST 2015@printerTaskID:12 sequenceBoltThreadID:38@Mon Sep 14 16:24:38 CST 2015@sequnceTaskID:16 46
printerBoltThreadID:40@Mon Sep 14 16:24:38 CST 2015@printerTaskID:10 sequenceBoltThreadID:38@Mon Sep 14 16:24:38 CST 2015@sequnceTaskID:21 52
printerBoltThreadID:40@Mon Sep 14 16:24:38 CST 2015@printerTaskID:7 sequenceBoltThreadID:38@Mon Sep 14 16:24:38 CST 2015@sequnceTaskID:15 48
printerBoltThreadID:40@Mon Sep 14 16:24:38 CST 2015@printerTaskID:3 sequenceBoltThreadID:38@Mon Sep 14 16:24:38 CST 2015@sequnceTaskID:22 53
printerBoltThreadID:40@Mon Sep 14 16:24:38 CST 2015@printerTaskID:9 sequenceBoltThreadID:38@Mon Sep 14 16:24:38 CST 2015@sequnceTaskID:16 50
printerBoltThreadID:40@Mon Sep 14 16:24:38 CST 2015@printerTaskID:11 sequenceBoltThreadID:38@Mon Sep 14 16:24:38 CST 2015@sequnceTaskID:14 54
printerBoltThreadID:40@Mon Sep 14 16:24:38 CST 2015@printerTaskID:12 sequenceBoltThreadID:38@Mon Sep 14 16:24:38 CST 2015@sequnceTaskID:20 55
printerBoltThreadID:40@Mon Sep 14 16:24:38 CST 2015@printerTaskID:6 sequenceBoltThreadID:38@Mon Sep 14 16:24:38 CST 2015@sequnceTaskID:19 58
printerBoltThreadID:40@Mon Sep 14 16:24:38 CST 2015@printerTaskID:5 sequenceBoltThreadID:38@Mon Sep 14 16:24:38 CST 2015@sequnceTaskID:13 56
printerBoltThreadID:40@Mon Sep 14 16:24:38 CST 2015@printerTaskID:3 sequenceBoltThreadID:38@Mon Sep 14 16:24:38 CST 2015@sequnceTaskID:17 57
printerBoltThreadID:40@Mon Sep 14 16:24:38 CST 2015@printerTaskID:9 sequenceBoltThreadID:38@Mon Sep 14 16:24:38 CST 2015@sequnceTaskID:19 63
printerBoltThreadID:40@Mon Sep 14 16:24:38 CST 2015@printerTaskID:11 sequenceBoltThreadID:38@Mon Sep 14 16:24:38 CST 2015@sequnceTaskID:20 64
printerBoltThreadID:40@Mon Sep 14 16:24:38 CST 2015@printerTaskID:4 sequenceBoltThreadID:38@Mon Sep 14 16:24:38 CST 2015@sequnceTaskID:15 59
printerBoltThreadID:40@Mon Sep 14 16:24:38 CST 2015@printerTaskID:9 sequenceBoltThreadID:38@Mon Sep 14 16:24:38 CST 2015@sequnceTaskID:13 60
printerBoltThreadID:40@Mon Sep 14 16:24:38 CST 2015@printerTaskID:10 sequenceBoltThreadID:38@Mon Sep 14 16:24:38 CST 2015@sequnceTaskID:22 65
printerBoltThreadID:40@Mon Sep 14 16:24:38 CST 2015@printerTaskID:3 sequenceBoltThreadID:38@Mon Sep 14 16:24:38 CST 2015@sequnceTaskID:14 61
printerBoltThreadID:40@Mon Sep 14 16:24:38 CST 2015@printerTaskID:5 sequenceBoltThreadID:38@Mon Sep 14 16:24:38 CST 2015@sequnceTaskID:18 67
printerBoltThreadID:40@Mon Sep 14 16:24:38 CST 2015@printerTaskID:11 sequenceBoltThreadID:38@Mon Sep 14 16:24:38 CST 2015@sequnceTaskID:15 62
printerBoltThreadID:40@Mon Sep 14 16:24:38 CST 2015@printerTaskID:12 sequenceBoltThreadID:38@Mon Sep 14 16:24:38 CST 2015@sequnceTaskID:21 69
printerBoltThreadID:40@Mon Sep 14 16:24:38 CST 2015@printerTaskID:8 sequenceBoltThreadID:38@Mon Sep 14 16:24:38 CST 2015@sequnceTaskID:17 66
printerBoltThreadID:40@Mon Sep 14 16:24:38 CST 2015@printerTaskID:7 sequenceBoltThreadID:38@Mon Sep 14 16:24:38 CST 2015@sequnceTaskID:16 68
printerBoltThreadID:40@Mon Sep 14 16:24:38 CST 2015@printerTaskID:12 sequenceBoltThreadID:38@Mon Sep 14 16:24:38 CST 2015@sequnceTaskID:14 70
printerBoltThreadID:40@Mon Sep 14 16:24:38 CST 2015@printerTaskID:4 sequenceBoltThreadID:38@Mon Sep 14 16:24:38 CST 2015@sequnceTaskID:16 73
printerBoltThreadID:40@Mon Sep 14 16:24:38 CST 2015@printerTaskID:4 sequenceBoltThreadID:38@Mon Sep 14 16:24:38 CST 2015@sequnceTaskID:21 71
printerBoltThreadID:40@Mon Sep 14 16:24:38 CST 2015@printerTaskID:10 sequenceBoltThreadID:38@Mon Sep 14 16:24:38 CST 2015@sequnceTaskID:15 75
printerBoltThreadID:40@Mon Sep 14 16:24:38 CST 2015@printerTaskID:6 sequenceBoltThreadID:38@Mon Sep 14 16:24:38 CST 2015@sequnceTaskID:22 72
printerBoltThreadID:40@Mon Sep 14 16:24:38 CST 2015@printerTaskID:6 sequenceBoltThreadID:38@Mon Sep 14 16:24:38 CST 2015@sequnceTaskID:13 76
printerBoltThreadID:40@Mon Sep 14 16:24:38 CST 2015@printerTaskID:8 sequenceBoltThreadID:38@Mon Sep 14 16:24:38 CST 2015@sequnceTaskID:20 74
printerBoltThreadID:40@Mon Sep 14 16:24:38 CST 2015@printerTaskID:3 sequenceBoltThreadID:38@Mon Sep 14 16:24:38 CST 2015@sequnceTaskID:18 77
printerBoltThreadID:40@Mon Sep 14 16:24:38 CST 2015@printerTaskID:7 sequenceBoltThreadID:38@Mon Sep 14 16:24:38 CST 2015@sequnceTaskID:19 78
printerBoltThreadID:40@Mon Sep 14 16:24:38 CST 2015@printerTaskID:5 sequenceBoltThreadID:38@Mon Sep 14 16:24:38 CST 2015@sequnceTaskID:17 79
printerBoltThreadID:40@Mon Sep 14 16:24:38 CST 2015@printerTaskID:8 sequenceBoltThreadID:38@Mon Sep 14 16:24:38 CST 2015@sequnceTaskID:17 80
printerBoltThreadID:40@Mon Sep 14 16:24:38 CST 2015@printerTaskID:5 sequenceBoltThreadID:38@Mon Sep 14 16:24:38 CST 2015@sequnceTaskID:22 82
printerBoltThreadID:40@Mon Sep 14 16:24:38 CST 2015@printerTaskID:9 sequenceBoltThreadID:38@Mon Sep 14 16:24:38 CST 2015@sequnceTaskID:14 81
printerBoltThreadID:40@Mon Sep 14 16:24:38 CST 2015@printerTaskID:10 sequenceBoltThreadID:38@Mon Sep 14 16:24:38 CST 2015@sequnceTaskID:20 83
printerBoltThreadID:40@Mon Sep 14 16:24:38 CST 2015@printerTaskID:10 sequenceBoltThreadID:38@Mon Sep 14 16:24:38 CST 2015@sequnceTaskID:13 85
printerBoltThreadID:40@Mon Sep 14 16:24:38 CST 2015@printerTaskID:9 sequenceBoltThreadID:38@Mon Sep 14 16:24:38 CST 2015@sequnceTaskID:18 84
printerBoltThreadID:40@Mon Sep 14 16:24:38 CST 2015@printerTaskID:5 sequenceBoltThreadID:38@Mon Sep 14 16:24:38 CST 2015@sequnceTaskID:15 86
printerBoltThreadID:40@Mon Sep 14 16:24:38 CST 2015@printerTaskID:12 sequenceBoltThreadID:38@Mon Sep 14 16:24:38 CST 2015@sequnceTaskID:21 88
printerBoltThreadID:40@Mon Sep 14 16:24:38 CST 2015@printerTaskID:7 sequenceBoltThreadID:38@Mon Sep 14 16:24:38 CST 2015@sequnceTaskID:16 87
printerBoltThreadID:40@Mon Sep 14 16:24:38 CST 2015@printerTaskID:6 sequenceBoltThreadID:38@Mon Sep 14 16:24:38 CST 2015@sequnceTaskID:19 89
printerBoltThreadID:40@Mon Sep 14 16:24:38 CST 2015@printerTaskID:7 sequenceBoltThreadID:38@Mon Sep 14 16:24:38 CST 2015@sequnceTaskID:22 92
printerBoltThreadID:40@Mon Sep 14 16:24:38 CST 2015@printerTaskID:11 sequenceBoltThreadID:38@Mon Sep 14 16:24:38 CST 2015@sequnceTaskID:19 95
printerBoltThreadID:40@Mon Sep 14 16:24:38 CST 2015@printerTaskID:4 sequenceBoltThreadID:38@Mon Sep 14 16:24:38 CST 2015@sequnceTaskID:15 90
printerBoltThreadID:40@Mon Sep 14 16:24:38 CST 2015@printerTaskID:12 sequenceBoltThreadID:38@Mon Sep 14 16:24:38 CST 2015@sequnceTaskID:17 91
printerBoltThreadID:40@Mon Sep 14 16:24:38 CST 2015@printerTaskID:8 sequenceBoltThreadID:38@Mon Sep 14 16:24:38 CST 2015@sequnceTaskID:20 96
printerBoltThreadID:40@Mon Sep 14 16:24:38 CST 2015@printerTaskID:3 sequenceBoltThreadID:38@Mon Sep 14 16:24:38 CST 2015@sequnceTaskID:18 97
printerBoltThreadID:40@Mon Sep 14 16:24:38 CST 2015@printerTaskID:11 sequenceBoltThreadID:38@Mon Sep 14 16:24:38 CST 2015@sequnceTaskID:14 93
printerBoltThreadID:40@Mon Sep 14 16:24:38 CST 2015@printerTaskID:4 sequenceBoltThreadID:38@Mon Sep 14 16:24:38 CST 2015@sequnceTaskID:21 98
printerBoltThreadID:40@Mon Sep 14 16:24:38 CST 2015@printerTaskID:3 sequenceBoltThreadID:38@Mon Sep 14 16:24:38 CST 2015@sequnceTaskID:16 94
printerBoltThreadID:40@Mon Sep 14 16:24:38 CST 2015@printerTaskID:6 sequenceBoltThreadID:38@Mon Sep 14 16:24:38 CST 2015@sequnceTaskID:13 99
可以看到,在任務流中的元組被隨機分配到不同的任務中的同時,其最終的總體順序也被打亂了,這僅僅是因爲我們爲SequenceBolt分配了兩個執行器。同樣,如果我們只爲PrinterBolt分配1個以上的執行器,也會造成輸出的無序。
總結:希望通過這幾個例子,能夠有助於更深入理解Storm並行的各個組件,並在業務對流處理有序性有要求的時候能夠採取正確的處理方案。類似Kafka這樣的輸入源,一定要考慮其自身特點,其分區內有序,分區外無序的特性,使得使用個性化的Kafka分區策略是必要的。而要保證Storm處理流程中的有序性,可以使用Global Grouping來完成。
與Kafka分區機制類似的是,Storm的Fields Grouping,比如你可以指定按照用戶的uid字段進行Grouping,那麼同一個用戶的所有的數據都會進入到同一個任務中,這樣的話你在一個任務中才能統計到某一個用戶的完整信息。
當然,從另一個角度來說,要保證順序就可能要做對應的性能犧牲,最好能夠在業務設計的時候,儘量避免這種一致性的要求,但是如果不能繞過,那也是可以滿足需求的,只不過可能要犧牲一些性能來妥協了。