SparkSQL之DataSet

Dataset是具有強類型的數據集合，需要提供對應的類型信息。

創建一個DataSet吧先

1）創建一個樣例類

scala> case class Person(name: String, age: Long)
defined class Person

2）創建DataSet

scala> val caseClassDS = Seq(Person("Andy", 32)).toDS()
caseClassDS: org.apache.spark.sql.Dataset[Person] = [name: string, age: bigint]

RDD轉換爲DataSet

SparkSQL能夠自動將包含有case類的RDD轉換成DataFrame，case類定義了table的結構，case類屬性通過反射變成了表的列名。
1）創建一個RDD

scala> val peopleRDD = sc.textFile("examples/src/main/resources/people.txt")
peopleRDD: org.apache.spark.rdd.RDD[String] = examples/src/main/resources/people.txt MapPartitionsRDD[3] at textFile at <console>:27

2）創建一個樣例類

scala> case class Person(name: String, age: Long)
defined class Person

3）將RDD轉化爲DataSet

scala> peopleRDD.map(line => {val para = line.split(",");Person(para(0),para(1).trim.toInt)}).toDS()

DataSet轉換爲RDD

調用rdd方法即可。
1）創建一個DataSet

scala> val DS = Seq(Person("Andy", 32)).toDS()
DS: org.apache.spark.sql.Dataset[Person] = [name: string, age: bigint]

2）將DataSet轉換爲RDD

scala> DS.rdd
res11: org.apache.spark.rdd.RDD[Person] = MapPartitionsRDD[15] at rdd at <console>:28

DataFrame與DataSet的互操作

1. DataFrame轉換爲DataSet
   1）創建一個DateFrame

scala> val df = spark.read.json("examples/src/main/resources/people.json")
df: org.apache.spark.sql.DataFrame = [age: bigint, name: string]

2）創建一個樣例類

scala> case class Person(name: String, age: Long)
defined class Person

3）將DateFrame轉化爲DataSet

scala> df.as[Person]
res14: org.apache.spark.sql.Dataset[Person] = [age: bigint, name: string]

2. DataSet轉換爲DataFrame
   1）創建一個樣例類

scala> case class Person(name: String, age: Long)
defined class Person

2）創建DataSet

scala> val ds = Seq(Person("Andy", 32)).toDS()
ds: org.apache.spark.sql.Dataset[Person] = [name: string, age: bigint]

3）將DataSet轉化爲DataFrame

scala> val df = ds.toDF
df: org.apache.spark.sql.DataFrame = [name: string, age: bigint]

4）展示

scala> df.show
+----+---+
|name|age|
+----+---+
|Andy| 32|
+----+---+

DataSet轉DataFrame

這個很簡單，因爲只是把case class封裝成Row
（1）導入隱式轉換

import spark.implicits._

（2）轉換

val testDF = testDS.toDF

DataFrame轉DataSet

（1）導入隱式轉換

import spark.implicits._

（2）創建樣例類

case class Coltest(col1:String,col2:Int)extends Serializable //定義字段名和類型

（3）轉換

val testDS = testDF.as[Coltest]

這種方法就是在給出每一列的類型後，使用as方法，轉成Dataset，這在數據類型是DataFrame又需要針對各個字段處理時極爲方便。在使用一些特殊的操作時，一定要加上 import spark.implicits._ 不然toDF、toDS無法使用。