這裏簡單描述一下幾個rdd，key-values的使用
- groupByKey
- reduceByKey
- aggregateByKey
- foldByKey
- combineByKey
- mapValues
- join
groupByKey
返回((key,value數組),(key,value數組))
//定義兩個kv類型的rdd
val rdd1: RDD[(Int, String)] = spark.sparkContext.makeRDD(Array((1, "r"), (2, "n"), (3, "g")))
val rdd2: RDD[(Int, String)] = spark.sparkContext.makeRDD(Array((1, "e"), (2, "d"), (3, "g")))
val resultRdd: RDD[(Int, Iterable[String])] = rdd1.groupByKey()
//打印結果：((1,("r","e")),(2,("n","d")),(3,("g","g")))
reduceByKey
以相加舉例子
返回((key,value的和),(key,value的和))
val rdd1: RDD[(String, Int)] = spark.sparkContext.makeRDD(Array(("r", 1), ("n", 2), ("g", 3), ("r", 5)))
val resultRdd: RDD[(String, Int)] = rdd1.reduceByKey(_ + _)
//打印結果：List((n,2), (r,6), (g,3))
aggregateByKey
第一個參數0代表初始值
第二個參數代表自己分區內計算
第三個參數代表自己分區結果和其他分區結果計算
規則
以zhangsan，兩個分區舉例子
第一個分區(0 , 5)相加得5
第二個分區(0 , 5)相加得5
5和5相加等於10
val kvRdd: RDD[(String, Int)] = spark.sparkContext.parallelize(Array(("zhangsan", 5), ("zhangsan", 5), ("wangwu", 6), ("zhaoliu", 4), ("tianqi", 3)), 2)
val resultRdd: RDD[(String, Int)] = kvRdd.aggregateByKey(0)(_ + _, _ + _)
//打印結果：List((zhangsan,10), (zhaoliu,4), (tianqi,3), (wangwu,6))
foldByKey
aggregateByKey的簡化版
第一個參數，初始值
第二個參數，傳一個函數，函數做減法這個就做減法，不需要考慮自己分區和其他分區
val kvRdd: RDD[(String, Int)] = spark.sparkContext.parallelize(Array(("zhangsan", 5), ("zhangsan", 5), ("wangwu", 6), ("zhaoliu", 4), ("tianqi", 3)), 2)
val resultRdd: RDD[(String, Int)] = kvRdd.foldByKey(0)(_ + _)
//打印結果：List((zhangsan,10), (zhaoliu,4), (tianqi,3), (wangwu,6))
combineByKey
可以把原本的v做計算變成元組來計算
做一個計算每個k的v的平均數，需要v的sum和v的count
以zhangsan，兩個分區舉例子
第一個參數：把5變成(5,1)
第二個參數：把自己分區內的(5,1)和3做相加變成(8,2)
第三個參數：把一分區的(8,2)和二分區的(5,1)變成(13,3)
val kvRdd: RDD[(String, Int)] = spark.sparkContext.parallelize(Array(("zhangsan", 5),("zhangsan", 3), ("zhangsan", 5), ("wangwu", 6), ("zhaoliu", 4), ("tianqi", 3)), 2)
val resultRdd: RDD[(String, (Int, Int))] = kvRdd.combineByKey((_, 1), (x: (Int, Int), y) => (x._1 + y, x._2 + 1), (x: (Int, Int), y: (Int, Int)) => (x._1 + y._1, x._2 + y._2))
//打印結果：List((zhangsan,(13,3)), (zhaoliu,(4,1)), (tianqi,(3,1)), (wangwu,(6,1)))
mapValues
對v來操作
val kvRdd: RDD[(String, Int)] = spark.sparkContext.parallelize(Array(("zhangsan", 5), ("zhangsan", 5), ("wangwu", 6), ("zhaoliu", 4), ("tianqi", 3)), 2)
val resultRdd: RDD[(String, Int)] = kvRdd.mapValues(x => x * 10)
//打印結果：List((zhangsan,50), (zhangsan,50), (wangwu,60), (zhaoliu,40), (tianqi,30))
join
val rdd1: RDD[(String, Int)] = spark.sparkContext.makeRDD(Array(("r", 1), ("n", 2), ("g", 3), ("r", 5)))
val rdd2: RDD[(String, Int)] = spark.sparkContext.makeRDD(Array(("e", 1), ("d", 2), ("g", 3)))
val resultRdd: RDD[(String, (Int, Int))] = rdd1.join(rdd2)
//打印結果：List((g,(3,3)))