官網介紹
Spark Streaming is an extension of the core Spark API that enables scalable, high-throughput, fault-tolerant stream processing of live data streams. Data can be ingested from many sources like Kafka, Flume, Kinesis, or TCP sockets, and can be processed using complex algorithms expressed with high-level functions like
map,reduce,joinandwindow. Finally, processed data can be pushed out to filesystems, databases, and live dashboards. In fact, you can apply Spark’s machine learning and graph processing algorithms on data streams.
Internally, it works as follows. Spark Streaming receives live input data streams and divides the data into batches, which are then processed by the Spark engine to generate the final stream of results in batches.
在內部，它的工作原理如下。Spark streams接收實時輸入的數據流，並將數據分成批次，然後由Spark引擎對這些數據進行處理，以批量生成最終的結果流。
Spark流提供了一個高級抽象，稱爲離散流或DStream，它表示連續的數據流。
詳細解釋：
Spark Streaming中，會有一個接收器組件Receiver，作爲一個長期運行的task。Spark Streaming中，會有一個接收器組件Receiver，作爲一個長期運行的task跑在一個Executor上。Receiver接收外部的數據流形成input DStream。DStream會被按照時間間隔劃分成一批一批的RDD，當批處理間隔縮短到秒級時，便可以用於處理實時數據流。時間間隔的大小可以由參數指定，一般設在500毫秒到幾秒之間。對DStream進行操作就是對RDD進行操作，計算處理的結果可以傳給外部系統。park Streaming的工作流程像上面的圖所示一樣，接受到實時數據後，給數據分批次，然後傳給Spark Engine處理最後生成該批次的結果。