在做kafka割接測試消費程序只消費不做數據處理時編寫代碼遇到了No output streams registered, so nothing to execute異常的問題:
代碼如下:
public void execute() throws MintakaException {
// Create a Spark Configuration
final SparkConf sparkConf = new SparkConf();
// backpressure
sparkConf.set("spark.streaming.backpressure.enabled", "true");
sparkConf.set("spark.serializer", "org.apache.spark.serializer.KryoSerializer");
Class[] classes = new Class[14];
classes[0] = Long.class;
classes[1] = double[].class;
classes[2] = Set.class;
classes[3] = scala.Tuple2[].class;
classes[4] = long[].class;
classes[5] = HashSet.class;
classes[6] = CustomerflowXdrCommunicationRecord.class;
classes[7] = CustomerCellRecord.class;
classes[8] = CustomerCellRecordDb.class;
classes[9] = String.class;
classes[10] = SparkStreamingMapReduceJob.class;
classes[11] = SparkStreamingMapReduceCollection.class;
classes[12] = CustomerflowXdrTransformationMapReduce.class;
classes[13] = HigwayCell.class;
sparkConf.registerKryoClasses(classes);
// Create a Spark Streaming Context
JavaStreamingContext javaStreamingContext = new JavaStreamingContext(sparkConf, Durations.seconds(slideInSeconds)); javaStreamingContext.sparkContext().hadoopConfiguration().set("parquet.enable.summary-metadata", "false"); javaStreamingContext.sparkContext().hadoopConfiguration().set("spark.sql.parquet.output.committer.class", "org.apache.spark.sql.parquet.DirectParquetOutputCommitter");
javaStreamingContext.checkpoint(checkPointPath);
Map<String, Object> kafkaParams = new HashMap();
kafkaParams.put("bootstrap.servers", bootstrapServers);
kafkaParams.put("value.deserializer",
"org.apache.kafka.common.serialization.StringDeserializer");
kafkaParams.put("key.deserializer",
"org.apache.kafka.common.serialization.StringDeserializer");
kafkaParams.put("group.id", kafkaConsumerGroup);
kafkaParams.put("security.protocol", "SASL_PLAINTEXT");
kafkaParams.put("sasl.kerberos.service.name", "kafka");
kafkaParams.put("kerberos.domain.name", "hadoop.hadoop.com");
kafkaParams.put("enable.auto.commit", "true");
kafkaParams.put("auto.offset.reset", "latest");
LocationStrategy locationStrategy = LocationStrategies.PreferConsistent();
System.out.println("topics are : " + topicss);
Set<String> topicSet = new HashSet<>(Arrays.asList(topicss.split(",")));
ConsumerStrategy consumerStrategy = ConsumerStrategies.Subscribe(topicSet, kafkaParams);
JavaInputDStream<ConsumerRecord<String, String>> messages = KafkaUtils.createDirectStream(javaStreamingContext,
locationStrategy, consumerStrategy);
JavaPairDStream<String, String> xdrStreams = messages.mapToPair(new PairFunction<ConsumerRecord<String, String>, String, String>() {
@Override
public Tuple2<String, String> call(ConsumerRecord<String, String> record) {
return new Tuple2<>(record.key(), record.value());
}
});
JavaPairDStream<String, String> xdrs = xdrStreams.persist(StorageLevel.MEMORY_AND_DISK_SER());
//mapReduce.mapReduce(xdrs);
try {
// Spark Job Start
javaStreamingContext.start();
javaStreamingContext.awaitTermination();
// TODO: check whether need to close!
javaStreamingContext.close();
javaStreamingContext = null;
} catch (Exception e) {
javaStreamingContext.close();
}
}
執行程序時會報如下異常(如下代碼爲出現異常部分):
JavaPairDStream<String, String> xdrStreams = messages.mapToPair(new PairFunction<ConsumerRecord<String, String>, String, String>() {
@Override
public Tuple2<String, String> call(ConsumerRecord<String, String> record) {
return new Tuple2<>(record.key(), record.value());
}
});
JavaPairDStream<String, String> xdrs = xdrStreams.persist(StorageLevel.MEMORY_AND_DISK_SER());
問題原因:當發生輸出操作的時候,會觸發流計算操作,當在 DStream上沒有輸出操作或者計算時,通常需要執行某些操作。代碼中把該mapReduce.mapReduce(xdrs)方法註釋了,導致沒有輸出操作。
解決方法:爲了能夠正常運行程序,達到測試效果,只要使用以下方法之一觸發即可:
1.print()
2.foreachRDD()
3.saveAsObjectFiles()
4.saveAsTextFiles()
5.saveAsHadoopFiles()
我在代碼中加了xdrStreams.print(1)即可正常運行