spark在eclipse中遇到的問題及處理措施_標記下

環境：cdh5.9  spark1.6

本地開發環境：下載了最新的ide：scala-SDK-4.6.0-vfinal-2.12-win32.win32.x86_64 

開始沒有使用maven，而是使用eclipse直接加jar包的方式。

下面截圖報錯：

More than one scala library found in the build path (G:/scala-SDK-4.6.0-vfinal-2.12-win32.win32.x86_64/eclipse/plugins/org.scala-lang.scala-library_2.12.2.v20170412-161608-VFINAL-21d12e9.jar, G:/spark-assembly-1.6.0-cdh5.9.0-hadoop2.6.0-cdh5.9.0.jar).At least one has an incompatible version. Please update the project build path so it contains only one compatible scala library.

報錯有多個scala版本，spark1.6 scala版本是2.10.5  

下面修改了版本爲2.10.6

悲催的是問題反而嚴重了：

各種查詢很無語，沒有找到合適的方法，後來發現spark-assembly-1.6.0-cdh5.9.0-hadoop2.6.0-cdh5.9.0.jar  包中打包了scala的相關信息：

將scala包部分刪除，問題解決。

後來和朋友聊天發現使用maven沒有遇到這樣的問題，改進現有版本使用maven

下面貼下maven的pom.xml

 <repositories>
    <repository>
      <id>cloudera.com</id>
      <name>Cloudera Maven Repository</name>
      <url>https://repository.cloudera.com/artifactory/cloudera-repos/</url>
    </repository>
  </repositories>							


  <dependencies>
    <dependency>
      <groupId>org.scala-lang</groupId>
      <artifactId>scala-library</artifactId>
      <version>2.10.6</version>
    </dependency>
    <dependency>
      <groupId>junit</groupId>
      <artifactId>junit</artifactId>
      <version>4.11</version>
      <scope>test</scope>
    </dependency>
    <dependency>
      <groupId>commons-cli</groupId>
      <artifactId>commons-cli</artifactId>
      <version>1.2</version>
    </dependency>
    <dependency>
      <groupId>commons-configuration</groupId>
      <artifactId>commons-configuration</artifactId>
      <version>1.10</version>
    </dependency>
    <dependency>
      <groupId>org.apache.hbase</groupId>
      <artifactId>hbase-client</artifactId>
      <version>1.2.0</version>
    </dependency>
    <dependency>
      <groupId>org.apache.hadoop</groupId>
      <artifactId>hadoop-common</artifactId>
      <version>2.6.0</version>
    </dependency>
    <dependency>
      <groupId>org.apache.spark</groupId>
      <artifactId>spark-core_2.10</artifactId>
      <version>1.6.0</version>
    </dependency>
    <dependency>
      <groupId>org.apache.spark</groupId>
      <artifactId>spark-sql_2.10</artifactId>
      <version>1.6.0</version>
    </dependency>
    <dependency>
      <groupId>org.scalatest</groupId>
      <artifactId>scalatest_2.10</artifactId>
      <version>3.0.0</version>
      <scope>test</scope>
    </dependency>
    <dependency>
      <groupId>org.apache.hbase</groupId>
      <artifactId>hbase-spark</artifactId>
      <version>1.2.0-cdh5.9.0</version>
    </dependency>
    <dependency>
      <groupId>org.scala-tools</groupId>
      <artifactId>maven-scala-plugin</artifactId>
      <version>2.15.2</version>
    </dependency>
    <!--<dependency>-->
      <!--<groupId>it.unimi.dsi</groupId>-->
      <!--<artifactId>fastutil</artifactId>-->
      <!--<version>7.0.13</version>-->
    <!--</dependency>-->
  </dependencies>