1.spark-shell 交互式編程
(1)該系總共有多少學生;
(2)該系共開設來多少門課程;
(3)Tom 同學的總成績平均分是多少;
(4)求每名同學的選修的課程門數;
(5)該系 DataBase 課程共有多少人選修;
(6)各門課程的平均分是多少;
(7)使用累加器計算共有多少人選了 DataBase 這門課。
第一問: val lines = sc.textFile("file:///usr/local/spark/sparksqldata/Data01.txt") val par = lines.map(row=>row.split(",")(0)) val distinct_par = par.distinct() //去重操作 distinct_par.count //取得總數 第二問 val lines = sc.textFile("file:///usr/local/spark/sparksqldata/Data01.txt") val par = lines.map(row=>row.split(",")(1)) val distinct_par = par.distinct() distinct_par.count 第三問: val lines = sc.textFile("file:///usr/local/spark/sparksqldata/Data01.txt") val pare = lines.filter(row=>row.split(",")(0)=="Tom") pare.foreach(println) Tom,DataBase,26 Tom,Algorithm,12 Tom,OperatingSystem,16 Tom,Python,40 Tom,Software,60 pare.map(row=>(row.split(",")(0),row.split(",")(2).toInt)).mapValues(x=>(x,1)).reduceByKey((x,y ) => (x._1+y._1,x._2 + y._2)).mapValues(x => (x._1 / x._2)).collect() //res9: Array[(String, Int)] = Array((Tom,30)) 第四問: val lines = sc.textFile("file:///usr/local/spark/sparksqldata/Data01.txt") val pare = lines.filter(row=>row.split(",")(1)=="DataBase") pare.count res1: Long = 126 第五問: val lines = sc.textFile("file:///usr/local/spark/sparksqldata/Data01.txt") val pare = lines.filter(row=>row.split(",")(1)=="DataBase") pare.count res1: Long = 126 第六問: val lines = sc.textFile("file:///usr/local/spark/sparksqldata/Data01.txt") val pare = lines.map(row=>(row.split(",")(1),row.split(",")(2).toInt)) pare.mapValues(x=>(x,1)).reduceByKey((x,y) => (x._1+y._1,x._2 + y._2)).mapValues(x => (x._1 / x._2)).collect() res0: Array[(String, Int)] = Array((Python,57), (OperatingSystem,54), (CLanguage,50), (Software,50), (Algorithm,48), (DataStructure,47), (DataBase,50), (ComputerNetwork,51)) 第七問: val lines = sc.textFile("file:///usr/local/spark/sparksqldata/Data01.txt") val pare = lines.map(row=>(row.split(",")(1),row.split(",")(2).toInt)) pare.mapValues(x=>(x,1)).reduceByKey((x,y) => (x._1+y._1,x._2 + y._2)).mapValues(x => (x._1 / x._2)).collect()