6-1 -课程目录
概述
发展史
应用场景
从词频统计功能着手入门
集成Spark生态系统的使用
工作原理
6-2 -Spark Streaming概述
官网：http://spark.apache.org/docs/latest/streaming-programming-guide.html
Spark Streaming is an extension of the core Spark API that enables scalable, high-throughput, fault-tolerant stream processing of live data streams. Data can be ingested from many sources like Kafka, Flume, Kinesis, or TCP sockets, and can be processed using complex algorithms expressed with high-level functions like map, reduce, join and window. Finally, processed data can be pushed out to filesystems, databases, and live dashboards. In fact, you can apply Spark’s machine learning and graph processing algorithms on data streams.
Internally, it works as follows. Spark Streaming receives live input data streams and divides the data into batches, which are then processed by the Spark engine to generate the final stream of results in batches.
Spark Streaming provides a high-level abstraction called discretized stream or DStream, which represents a continuous stream of data. DStreams can be created either from input data streams from sources such as Kafka, Flume, and Kinesis, or by applying high-level operations on other DStreams. Internally, a DStream is represented as a sequence of RDDs.
This guide shows you how to start writing Spark Streaming programs with DStreams. You can write Spark Streaming programs in Scala, Java or Python (introduced in Spark 1.2), all of which are presented in this guide. You will find tabs throughout this guide that let you choose between code snippets of different languages.
Note: There are a few APIs that are either different or not available in Python. Throughout this guide, you will find the tag Python API highlighting these differences.
Spak Streaming个人定义：
将不同的数据源经过spark streaming处理之后将结果输出到外部文件系统
特点：
低延时
能从错误中高效恢复
能够运行在成百上千的节点
能够将批处理，机器学习，图计算等子框架和spark Streaming综合使用
Spark Streaming 是否需要独立安装？
on stack to rule them all : 一栈式
6-3 -Spark Streaming应用场景
6-3 -Spark Streaming应用场景
6-5 -Spark Streaming发展史
6-6 -从词频统计功能着手入门Spark Streaming
GitHub
https://github.com/apache/spark
源码地址：
1、spark-submit执行
./spark-submit --master local[2] \
--lass org.apache.spark.examples.streaming.NetworkWordCout \在examples/jars里面的jar包 hadoop000 9999
2、spark-shell执行
拷贝本段代码，粘贴去终端运行，方法同spark-submit
6-7 -Spark Streaming工作原理(粗粒度)
工作原理：粗粒度
Spark Streaming 接受到实时数据流，把数据按照指定的时间段切分成一小片片小的数据块，然后把小的数据块传给Spark Engine处理
6-8 -Spark Streaming工作原理(细粒度)
