1.下載idea社區版,安裝scala插件
2.下載spark和對應版本scala(重要http://spark.apache.org/docs/latest/index.html)jdk 8 sbt
3.安裝spark scala jdk sbt並添加相應環境變量
4.新建 scala項目 + sbt
5.點擊structure project 添加spark jar文件(spark/jar文件)
6.new ->scala class->object->添加代碼
兩個問題:1. Failed to locate the winutils binary in the hadoop binary path
2.Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
解決辦法:hadoop在文中缺少依賴庫,下載(https://github.com/4ttty/winutils),其中的2.7.1適用於已經安裝的hadoop2.7.7,把bin下的所有文件複製粘貼到hadoop/bin下,已有的就選擇跳過,即可解決這倆問題
運行代碼時錯誤:error: not found: value sqlContext
解決辦法:不同版本造成,把sqlContext換成spark.sqlContext解決問題,參見:https://stackoverflow.com/questions/42993521/apache-spark-error-not-found-value-sqlcontext