spark操作hive

1、下載spark並解壓

wget http://mirrors.tuna.tsinghua.edu.cn/apache/spark/spark-2.3.4/spark-2.3.4-bin-hadoop2.6.tgz

tar zxvf spark-2.3.4-bin-hadoop2.6.tgz

2、配置

1）將hive-site.xml拷貝到$SPARK_HOME/conf目錄下。

2）spark-env.sh增加內容：

export JAVA_HOME=/usr/java/jdk1.8.0
export CLASSPATH=$CLASSPATH:/opt/cloudera/parcels/CDH/lib/hive/lib
export SCALA_HOME=/data/scala-2.12.8
export HADOOP_CONF_DIR=/etc/hadoop/conf
export HIVE_CONF_DIR=/etc/hive/conf

3、測試

$SPARK_HOME/bin/spark-sql