在開始正式數據處理之前，我覺得有必要去學習理解下UDF。
UDF
UDF全稱User-Defined Functions，用戶自定義函數，是Spark SQL的一項功能，用於定義新的基於列的函數，這些函數擴展了Spark SQL的DSL用於轉換數據集的詞彙表。
我在databricks上找到一個比較簡單理解的入門栗子：
Register the function as a UDF
1val squared = (s: Int) => {
2 s * s
3}
4spark.udf.register("square", squared)Call the UDF in Spark SQL
1spark.range(1, 20).registerTempTable("test")
2%sql select id, square(id) as id_squared from test我理解就是先定義一個函數squared，返回輸入數字的平方，然後register，並綁定square方法名爲square，然後就在Spark SQL中直接使用square方法。
實例一：溫度轉化
1import org.apache.spark.sql.SparkSession
2import org.apache.spark.SparkConf
3
4object ScalaUDFExample {
5 def main(args: Array[String]) {
6 val conf = new SparkConf().setAppName("Scala UDF Example")
7 val spark = SparkSession.builder().enableHiveSupport().config(conf).getOrCreate()
8
9 val ds = spark.read.json("temperatures.json")
10 ds.createOrReplaceTempView("citytemps")
11
12 // Register the UDF with our SparkSession
13 spark.udf.register("CTOF", (degreesCelcius: Double) => ((degreesCelcius * 9.0 / 5.0) + 32.0))
14
15 spark.sql("SELECT city, CTOF(avgLow) AS avgLowF, CTOF(avgHigh) AS avgHighF FROM citytemps").show()
16 }
17}我們將定義一個 UDF 來將以下 JSON 數據中的溫度從攝氏度（degrees Celsius）轉換爲華氏度（degrees Fahrenheit）：
1{"city":"St. John's","avgHigh":8.7,"avgLow":0.6}
2{"city":"Charlottetown","avgHigh":9.7,"avgLow":0.9}
3{"city":"Halifax","avgHigh":11.0,"avgLow":1.6}
4{"city":"Fredericton","avgHigh":11.2,"avgLow":-0.5}
5{"city":"Quebec","avgHigh":9.0,"avgLow":-1.0}
6{"city":"Montreal","avgHigh":11.1,"avgLow":1.4}
7...實例二：時間轉化
1case class Purchase(customer_id: Int, purchase_id: Int, date: String, time: String, tz: String, amount:Double)
2
3val x = sc.parallelize(Array(
4 Purchase(123, 234, "2007-12-12", "20:50", "UTC", 500.99),
5 Purchase(123, 247, "2007-12-12", "15:30", "PST", 300.22),
6 Purchase(189, 254, "2007-12-13", "00:50", "EST", 122.19),
7 Purchase(187, 299, "2007-12-12", "07:30", "UTC", 524.37)
8))
9
10val df = sqlContext.createDataFrame(x)
11df.registerTempTable("df")自定義函數
1def makeDT(date: String, time: String, tz: String) = s"$date $time $tz"
2sqlContext.udf.register("makeDt", makeDT(_:String,_:String,_:String))
3
4// Now we can use our function directly in SparkSQL.
5sqlContext.sql("SELECT amount, makeDt(date, time, tz) from df").take(2)
6// but not outside
7df.select($"customer_id", makeDt($"date", $"time", $"tz"), $"amount").take(2) // fails如果想要在SQL外面使用，必須通過spark.sql.function.udf來創建UDF
1import org.apache.spark.sql.functions.udf 2val makeDt = udf(makeDT(_:String,_:String,_:String)) 3// now this works 4df.select($"customer_id", makeDt($"date", $"time", $"tz"), $"amount").take(2)
實踐操作
寫一個UDF來將一些Int數字分類
1val formatDistribution = (view: Int) => {
2 if (view < 10) {
3 "<10"
4 } else if (view <= 100) {
5 "10~100"
6 } else if (view <= 1000) {
7 "100~1K"
8 } else if (view <= 10000) {
9 "1K~10K"
10 } else if (view <= 100000) {
11 "10K~100K"
12 } else {
13 ">100K"
14 }
15}註冊：
1session.udf.register("formatDistribution", UDF.formatDistribution)SQL：
1session.sql("select user_id, formatDistribution(variance_digg_count) as variance from video")寫到這裏，再回顧UDF，我感覺這就像是去爲了方便做一個分類轉化等操作，和Python裏面的函數一樣，只不過這裏的UDF一般特指Spark SQL裏面使用的函數。然後發現這裏和SQL中的自定義函數挺像的:
1CREATE FUNCTION [函數所有者.]<函數名稱> 2( 3 -- 添加函數所需的參數，可以沒有參數 4 [<@param1> <參數類型>] 5 [,<@param1> <參數類型>]… 6) 7RETURNS TABLE 8AS 9RETURN 10( 11 -- 查詢返回的SQL語句 12 SELECT查詢語句 13)
1/*
2* 創建內聯表值函數，查詢交易總額大於1W的開戶人個人信息
3*/
4create function getCustInfo()
5returns @CustInfo table --返回table類型
6(
7 --賬戶ID
8 CustID int,
9 --帳戶名稱
10 CustName varchar(20) not null,
11 --身份證號
12 IDCard varchar(18),
13 --電話
14 TelePhone varchar(13) not null,
15 --地址
16 Address varchar(50) default('地址不詳')
17)
18as
19begin
20 --爲table表賦值
21 insert into @CustInfo
22 select CustID,CustName,IDCard,TelePhone,Address from AccountInfo
23 where CustID in (select CustID from CardInfo
24 where CardID in (select CardID from TransInfo group by CardID,transID,TransType,TransMoney,TransDate having sum(TransMoney)>10000))
25 return
26end
27go
28-- 調用內聯表值函數
29select * from getCustInfo()
30go好像有異曲同工之妙~