spark-IDEA之SBT打包

SBT打包代碼與依賴：

這裏我使用的是SBT進行的打包:

 

1、創建assembly.sbt並在裏面寫:

addSbtPlugin("com.eed3si9n" % "sbt-assembly" % "0.11.2")

 

 

2、Build.sbt中寫:

import AssemblyKeys._

name := "saprk-sbt"

version := "1.0"

scalaVersion := "2.10.4"

libraryDependencies ++= Seq(
  //spark依賴
  "org.apache.spark" % "spark-core_2.10" % "1.6.1" % "provided",
  "com.alibaba" % "fastjson" % "1.2.24",
  /*
   * kafka
   * */
  "org.apache.spark" % "spark-streaming_2.10" % "1.6.1" % "provided",
  "org.apache.spark" % "spark-streaming-kafka_2.10" % "1.6.1" % "provided",
  /*
   * commions io
   * */
  "commons-io" % "commons-io" % "2.4",
  /*
   * joda
   * */
  "joda-time" % "joda-time" % "2.3"
)
//這條語句打開了assembly的插件功能
assemblySettings

//執行assembly的時候忽略測試
test in assembly :={}

//指定類的名字
mainClass in assembly := Some("auto.DateManual")

assemblyOption in packageDependency ~= {
  _.copy(appendContentHash = true)
}
/**
  * MergeStrategy.first:默認取得第一個匹配項
  * MergeStrategy.last:默認取得最後一個匹配項
  * MergeStrategy.discard:近丟棄匹配的文件
  * */
//解決依賴重複的問題
mergeStrategy in assembly := {
  //  case x if x.startsWith("META-INF") => MergeStrategy.discard
  //  case x if x.endsWith(".class") => MergeStrategy.discard
  //    case x if x.contains("slf4j-api") => MergeStrategy.last
  //    case x if x.contains("org/cyberneko/html") => MergeStrategy.first
  //如果後綴是.properties的文件，合併策略採用（MergeStrategy.first）第一個出現的文件
  case PathList(ps@_*) if ps.last endsWith ".properties" => MergeStrategy.first
  case PathList(ps@_*) if ps.last endsWith "Absent.class" => MergeStrategy.first
  case PathList("com", "esotericsoftware", xs@_*) => MergeStrategy.first
  case x =>
    val oldStrategy = (mergeStrategy in assembly).value
    oldStrategy(x)
}
//定義jar包的名字
jarName in assembly := "sbt-solr-assembly.jar"

//把scala本身排除在Jar中，因爲spark已經包含了scala
assemblyOption in assembly :=

(assemblyOption in assembly).value.copy(includeScala = false)

3、

 

-----

中運行sbt 然後運行clean,compile,assembly(打包命令)