Spark streaming 整合Kafka統計詞頻

**

Spark streaming 整合Kafka基於（Receiver-based Approach）統計詞頻

**

首先，啓動zookeeper

然後，啓動kafka

創建topic

啓動生產者

啓動消費者

開發Spark streaming 整合Kafka統計詞頻的程序

from pyspark import SparkContext
from pyspark.streaming import StreamingContext
from pyspark.streaming.kafka import KafkaUtils

'''Spark streaming 整合Kafka統計詞頻'''

sc = SparkContext(appName="KafkaReceiverWordCount",master="local[2]")
ssc = StreamingContext(sc,5)

kafkastream = KafkaUtils.createStream(ssc=ssc,zkQuorum="hadoop001:2181",topics={'kafka_streaming_topic':1},groupId="test")

counts = kafkastream.map(lambda x:x[1])\
    .flatMap(lambda line:line.split(" "))\
    .map(lambda word:(word,1))\
    .reduceByKey(lambda a,b:a+b)

counts.pprint()

ssc.start()
ssc.awaitTermination()

啓動pyspark

將開發好的代碼粘貼到這，開始執行，消費者結果

查看spark streaming執行結果

執行成功。

## Spark Streaming整合Kafka基於Direct Approach (No Receivers)統計詞頻

啓動zookeeper，啓動生產者，消費者步驟同上，開發spark streaming程序

#!/usr/bin/python
#coding=utf-8

from pyspark import SparkContext
from pyspark.streaming import StreamingContext
from pyspark.streaming.kafka import KafkaUtils

'''Spark Streaming整合Kafka基於Direct Approach (No Receivers)統計詞頻'''

sc = SparkContext(master="local[2]",appName="KafkaDirectWordCount")
ssc = StreamingContext(sc,10)

topics = ["kafka_streaming_topic"]

kafkaDStream = KafkaUtils.createDirectStream(ssc=ssc,\
                                      topics=topics,\
                                      kafkaParams={"bootstrap.servers":"hadoop001:9092"})

wordCounts = kafkaDStream.map(lambda x: x[1])\
    .flatMap(lambda line: line.split(" "))\
    .map(lambda word: (word,1))\
    .reduceByKey(lambda a,b: a+b)


wordCounts.pprint()

ssc.start()
ssc.awaitTermination()

使用spark-submit提交程序：

./spark-submit \
--master local[2] \
--num-executors 2 \
--executor-memory 1G \
/usr/soft/script/python/KafkaDirectWordCount.py

在生產者中，輸入數據進行測試

查看結果，正是我們想要的

這是spark streaming 整合kafka的兩種方式，在spark1.3之後spark streaming已經支持Direct Approach (No Receivers)的java API與scala API，在spark1.4之後就支持python API。實際生產環境中，大多數情況我們使用的也是Direct Approach (No Receivers)這種方式。