sbt發佈assembly解決jar包衝突問題 deduplicate: different file contents found in the following

一、問題定義

    最近在用sbt打assembly包時出現問題，在package的時候，發生jar包衝突/文件衝突問題，兩個相同的class來自不同的jar包在classpath內引起衝突。

具體是：我有一個self4j的jar， 還有一個hadoop-common-hdfs的jar包，其中hadoop-common-hdfs.jar內包含了self4j這個jar包，導致衝突。

此類異常一般是由於打包不規範和打包疏忽引起的。

（個人認爲正確的打包策略是：只打包自己核心功能，不將依賴打包在一起，但是有時爲了方便或者有時不得不打在一起，要注意可能會出現上述問題）

異常log如下：

[trace] Stack trace suppressed: run last *:assembly for the full output.
[error] (*:assembly) deduplicate: different file contents found in the following:
[error] C:\Users\shengli.victor\.ivy2\cache\org.slf4j\slf4j-api\jars\slf4j-api-1.7.7.jar:org/slf4j/IMarkerFactory.class
[error] C:\Users\shengli.victor\.ivy2\cache\com.xxx.xx.hdfsfile\hdfscommon\jars\hdfscommon-1.1.jar:org/slf4j/IMarkerFactory.class
[error] Total time: 4 s, completed 2014-11-20 19:07:33

異常很明顯，來自2個不同的jar包self4j,  hdfscommon-1.1.jar裏，在org/slf4j/IMarkerFactory.class這個類衝突了。

如下圖：

hdfscommon-1.1/jar

slf4j-api-1.7.2.jar

二、解決方案

解決jar包衝突有兩種方案：

1、刪除其中的某個jar,或者說，在打包的時候，不將2個相同的class打在同一個jar包內的classpath內，即exclude jar。

2、合併衝突

1. Excluding JARs and files

% "provided"

將相同的jar中排除一個，因爲重複，可以使用"provided"關鍵字。

例如spark是一個容器類，編寫spark應用程序我們需要spark core jar. 但是真正打包提交到集羣上執行，則不需要將它打入jar包內。

這是我們使用 % "provided" 關鍵字來exclude它。

libraryDependencies ++= Seq(
  "org.apache.spark" %% "spark-core" % "0.8.0-incubating" % "provided",
  "org.apache.hadoop" % "hadoop-client" % "2.0.0-cdh4.4.0" % "provided"
)

Maven defines "provided" as:

This is much like compile, but indicates you expect the JDK or a container to provide the dependency at runtime. For example, when building a web application for the Java Enterprise Edition, you would set the dependency on the Servlet API and related Java EE APIs to scopeprovided because the web container provides those classes. This scope is only available on the compilation and test classpath, and is not transitive.

2、Merge Strategy

如果在相對路徑下，有多個相同的文件或者jar，這時我們可以使用Merge策略。

在build.sbt中對assemblyMergeStrategy 進行定義。

如下：

mergeStrategy in assembly <<= (mergeStrategy in assembly) { (old) =>
  {
    case PathList("org", "slf4j", xs @ _*)         => MergeStrategy.first
    case PathList(ps @ _*) if ps.last endsWith "axiom.xml" => MergeStrategy.filterDistinctLines
    case PathList(ps @ _*) if ps.last endsWith "Log$Logger.class" => MergeStrategy.first
    case PathList(ps @ _*) if ps.last endsWith "ILoggerFactory.class" => MergeStrategy.first
    case x => old(x)
  }
}

解決方法：將org, slf4j 這個下的所有類和文件，都做合併， 採用的策略是：在classpath裏，2選1，選的是classpath順序裏第一個self4j。

這裏支持多種格式，例如ps.lat endsWith "axiom.xml" ，是以axiom.xml爲結尾的文件，都採用filterDistinctLines策略，即合併兩個文件，捨去重複的部分。

通過以上修改，終於解決了slf4j衝突的問題，即deduplicate: different file contents found in the following問題。

再次sbt assembly:

[warn] Merging 'META-INF\INDEX.LIST' with strategy 'discard'
[warn] Merging 'META-INF\MANIFEST.MF' with strategy 'discard'
[warn] Merging 'META-INF\maven\log4j\log4j\pom.properties' with strategy 'first'
[warn] Merging 'META-INF\maven\log4j\log4j\pom.xml' with strategy 'first'
[warn] Merging 'META-INF\maven\org.slf4j\slf4j-api\pom.properties' with strategy 'first'
[warn] Merging 'META-INF\maven\org.slf4j\slf4j-api\pom.xml' with strategy 'first'
[warn] Merging 'META-INF\maven\org.slf4j\slf4j-log4j12\pom.properties' with strategy 'first'
[warn] Merging 'META-INF\maven\org.slf4j\slf4j-log4j12\pom.xml' with strategy 'first'
[warn] Merging 'com\esotericsoftware\minlog\Log$Logger.class' with strategy 'first'
[warn] Merging 'com\esotericsoftware\minlog\Log.class' with strategy 'first'
[warn] Merging 'org\apache\log4j\helpers\LogLog.class' with strategy 'first'
[warn] Merging 'org\slf4j\ILoggerFactory.class' with strategy 'first'
[warn] Merging 'org\slf4j\IMarkerFactory.class' with strategy 'first'
[warn] Merging 'org\slf4j\Logger.class' with strategy 'first'
[warn] Merging 'org\slf4j\LoggerFactory.class' with strategy 'first'
[warn] Merging 'org\slf4j\MDC.class' with strategy 'first'
[warn] Merging 'org\slf4j\Marker.class' with strategy 'first'
[warn] Merging 'org\slf4j\MarkerFactory.class' with strategy 'first'
[warn] Merging 'org\slf4j\helpers\BasicMDCAdapter.class' with strategy 'first'
[warn] Merging 'org\slf4j\helpers\BasicMarker.class' with strategy 'first'
[warn] Merging 'org\slf4j\helpers\BasicMarkerFactory.class' with strategy 'first'
[warn] Merging 'org\slf4j\helpers\FormattingTuple.class' with strategy 'first'
[warn] Merging 'org\slf4j\helpers\MarkerIgnoringBase.class' with strategy 'first'
[warn] Merging 'org\slf4j\helpers\MessageFormatter.class' with strategy 'first'
[warn] Merging 'org\slf4j\helpers\NOPLogger.class' with strategy 'first'
[warn] Merging 'org\slf4j\helpers\NOPLoggerFactory.class' with strategy 'first'
[warn] Merging 'org\slf4j\helpers\NOPMDCAdapter.class' with strategy 'first'
[warn] Merging 'org\slf4j\helpers\NamedLoggerBase.class' with strategy 'first'
[warn] Merging 'org\slf4j\helpers\SubstituteLoggerFactory.class' with strategy 'first'
[warn] Merging 'org\slf4j\helpers\Util.class' with strategy 'first'
[warn] Merging 'org\slf4j\impl\Log4jLoggerAdapter.class' with strategy 'first'
[warn] Merging 'org\slf4j\impl\Log4jLoggerFactory.class' with strategy 'first'
[warn] Merging 'org\slf4j\impl\Log4jMDCAdapter.class' with strategy 'first'
[warn] Merging 'org\slf4j\impl\StaticLoggerBinder.class' with strategy 'first'
[warn] Merging 'org\slf4j\impl\StaticMDCBinder.class' with strategy 'first'
[warn] Merging 'org\slf4j\impl\StaticMarkerBinder.class' with strategy 'first'
[warn] Merging 'org\slf4j\spi\LocationAwareLogger.class' with strategy 'first'
[warn] Merging 'org\slf4j\spi\LoggerFactoryBinder.class' with strategy 'first'
[warn] Merging 'org\slf4j\spi\MDCAdapter.class' with strategy 'first'
[warn] Merging 'org\slf4j\spi\MarkerFactoryBinder.class' with strategy 'first'
[warn] Merging 'rootdoc.txt' with strategy 'concat'
[warn] Strategy 'concat' was applied to a file
[info] Strategy 'deduplicate' was applied to 373 files (Run the task at debug level to see details)
[warn] Strategy 'discard' was applied to 2 files
[warn] Strategy 'first' was applied to 38 files

[info] Done packaging.
[success] Total time: 84 s, completed 2014-11-20 19:04:52

合併策略有很多種：

可以參考官方sbt assembly文檔：https://github.com/sbt/sbt-assembly

http://stackoverflow.com/questions/19606243/resolving-dependencies-in-creating-jar-through-sbt-assembly

• MergeStrategy.deduplicate is the default described above
• MergeStrategy.first picks the first of the matching files in classpath order
• MergeStrategy.last picks the last one
• MergeStrategy.singleOrError bails out with an error message on conflict
• MergeStrategy.concat simply concatenates all matching files and includes the result
• MergeStrategy.filterDistinctLines also concatenates, but leaves out duplicates along the way
• MergeStrategy.rename renames the files originating from jar files
• MergeStrategy.discard simply discards matching files

更多的寫法，example：

assemblyMergeStrategy in assembly := {
  case PathList("javax", "servlet", xs @ _*)         => MergeStrategy.first
  case PathList(ps @ _*) if ps.last endsWith ".html" => MergeStrategy.first
  case "application.conf"                            => MergeStrategy.concat
  case "unwanted.txt"                                => MergeStrategy.discard
  case x =>
    val oldStrategy = (assemblyMergeStrategy in assembly).value
    oldStrategy(x)
}

Final Qucik Hack：

如果以上寫法都不奏效，還有最好一種，強制默認全部合併，不到萬不得已，不要用。。

mergeStrategy in assembly <<= (mergeStrategy in assembly) { mergeStrategy => {
 case entry => {
   val strategy = mergeStrategy(entry)
   if (strategy == MergeStrategy.deduplicate) MergeStrategy.first
   else strategy
 }
}}

三、總結

  碰到類似的問題不要慌張，仔細看log描述的是什麼意思。

  異常報出內容冗餘的衝突，在看路徑，發現在classpath內有完全相同的2個類，這是導致問題的根本原因。

  找出原因，解決方發，消除衝突兩種方法，一直是去除法，另一種是合併法。

  相對於maven和gradle，sbt的衝突解決方法還是比較接近底層。如果沒記錯的話，maven和gradle都能自動解決衝突。

  本文僅針對該問題提出解決方案和思路，具體的各個配置還需要繼續研究。

——EOF——

原創文章，轉載請註明出自：http://blog.csdn.net/oopsoom/article/details/41318599