在Spark官網下載Spark時可以選擇下載帶Hadoop和不帶Hadoop的版本。
如果選擇下載不帶Hadoop的版本,需要在${SPARK_HOME}/conf/spark-env.sh
中export環境變量SPARK_DIST_CLASSPATH
,否則會在啓動時遇到A JNI error has occurred, please check your installation and try again;而下載帶Hadoop的Spark就不需要。
應該這樣配置SPARK_DIST_CLASSPATH
:
### in conf/spark-env.sh ###
# If 'hadoop' binary is on your PATH
export SPARK_DIST_CLASSPATH=$(hadoop classpath)
# With explicit path to 'hadoop' binary
export SPARK_DIST_CLASSPATH=$(/path/to/hadoop/bin/hadoop classpath)
# Passing a Hadoop configuration directory
export SPARK_DIST_CLASSPATH=$(hadoop --config /path/to/configs classpath)
官方的說法戳這裏