Spark內核天機解密(2016)第三講 Scala函數式編程

課程筆記:

變量與函數

>def fun1(name: String) {println(name)}

fun1: (name: String)Unit

>val fun1_v = fun1 _  //注意,是空格和下劃線

fun1_v: String => Unit = <function1>

>fun1("Spark")

Spark

>fun1_v("Spark")

Spark

>fun1_v("Scala")

Scala

//匿名函數

>val fun2 = (content: String) => println(content:)

fun2: String => Unit = <function1>

>fun2("Hadoop")

Hadoop


//函數的參數可以是函數


>val hiScala = (content : String) => println(context)

hiScala: String => Unit = <function1>

>def bigData(func: (String) => Unit, context: String)  {func(content)}

bigData:(func: String => Unit, content: String) Unit

>bigData(hiScala,"Spark")

Spark

//循環遍歷數組每個值

>val array = Array(1,2,3,4,5,6,7,8,9)

>array.map(item => 2 * item)

 Array[Int] = Array(2,4,6,8,10,12,14,16,18)

>array.map(item => print(item))


//函數作爲返回值

>def func_Returned(content : String) = (message : String) => println(message)

func_Returned: (content: String)String => Unit

>func_Returned("Spark")

res7: String => Unit = <function1>

>val returned = func_Returned("Spark")

returned: String => Unit = <function1>

//用變量調用執行

>returned("Scala")

Spark    Scala


>def spark(fun: (String) => Unit, name: String) {func(name)}

>spark((name: String) => println(name),"Scala")

Scala

>spark((name) => println(name),"Scala")

Scala

>spark(name => println(name),"Scala")

Scala

>spark(println(_), "Scala")

Scala

>spark(println,"Scala")

Scala


>array.map(2*_)

res17:Array[Int] = Array(2,4,6,8,10,12,14,16,18)

// print array elements

>array.map(2*_).foreach(println(_))

>array.map(2*_).foreach(println _)     


>array.map(2*_).filter(_ >12).foreach(println)

14

16

18


//閉包:函數的變量超出函數的作用域時,在外部還可以對它進行訪問; Scala內部維護了函數的參數(參數作爲函數的成員被保存)

>def scala(content : String) = (message : String) => println(content + "   :" + message)

scala: (content: String) String => Unit

>val funcResult = scala("Spark")

funcResult: String => Unit = <function1>

>funcResult("Flink")

Spark    :Flink

//柯里化:如果一個函數有兩個參數,那麼可以把這個函數轉換成兩個函數來實現,返回值也是一個函數。

>def sum(x: Int, y: Int) = x+y

>sum(1,2)

res24: Int = 3

>def sum_currying(x : Int) = (y: Int) => x + y

sum_currying: (x: Int) Int => Int

>sum_currying(1)(2)

也可以把兩個函數合併起來定義和使用。

>def sum_Currying_Better(x : Int) (y: Int) = x + y

sum_Currying_Better: (x: Int) (y: Int) Int

>sum_Currying_better(1)(3)

res27: Int = 4

//很奇妙的處理方式

>(1 to 100).reduceLeft(_+_)

res28: Int = 5050


集合

>val list = List("Scala","Spark","Flink")

list: List[String] = List(Scala,Spark,Flink)

>list.map("The content is : " + _)

res29: List[String] = List(The content is : Scala,The content is : Spark, The content is : Scalaala


作業:統計一個文件夾下面所有的單詞出現的總次數(轉)
http://blog.sina.com.cn/s/blog_7dc74d730102vyhv.html


object WordCounter {
  import scala.io.Source
  import java.io._
  

  // store word and counter
  var map = Map.empty[String, Int]
 
  def main(args: Array[String]): Unit = {
    scanDir(new File("D:/workspace"))
    map.foreach(f =>
      println(f)  
    )
  }
 
  def scanDir(dir: File): Unit = {
    dir.listFiles.foreach { file => 
      if(file.isFile()){
        readFile(file)
        println(file)
      }      
    }
  }
 
  def readFile(file: File){
    val f = Source.fromFile(file)
    for (line <- f.getLines()){
      count(line)
    }
  }
 
  def count(line: String) = {
    for(word <- line.split("[,:.!\\s?*\\/-=+]()><")){
      if(map.contains(word))
        map += (word -> (map(word)+1))
      else
        map += (word -> 1)
    }
  }
}

object WordCounter {
  //導入jar包
  import scala.io.Source
  import java.io._
  //存儲單詞和個數
  var map = Map.empty[String, Int]
 
  def main(args: Array[String]): Unit = {
    scanDir(new File("D:/TMPFOLDER"))
    map.foreach(f =>
      println(f)  
    )
  }
 
  def scanDir(dir: File): Unit = {
    dir.listFiles.foreach { file => 
      if(file.isFile()){
        readFile(file)
        println(file)
      }      
    }
  }
 
  def readFile(file: File){
    val f = Source.fromFile(file)
    for (line <- f.getLines()){
      count(line)
    }
  }
 
  def count(line: String) = {
    for(word <- line.split("[,:.!\\s?*\\/-=+]()><")){
      if(map.contains(word))
        map += (word -> (map(word)+1))
      else
        map += (word -> 1)
    }
  }
}

發表評論
所有評論
還沒有人評論,想成為第一個評論的人麼? 請在上方評論欄輸入並且點擊發布.
相關文章