Spark整合hive
1.hive的類庫需要在spark worker節點。2.複製core-site.xml(hdfs) + hdfs-site.xml(hdfs) + hive-site.xml(hive)三個文件
到spark/conf下。
如果spark-env裏面配置了hadoop的路徑,就不用複製core-site.xml(hdfs) + hdfs-site.xml(hdfs)
3.複製mysql驅動程序到/soft/spark/jars下
...
hive的jar包中有
4.啓動spark-shell,指定啓動模式
spark-shell --master local[4]
$scala>create table tt(id int,name string , age int)
row format delimited fields terminated by ','
lines terminated by '\n'
stored as textfile ;
//加載數據到hive表
$scala>spark.sql("load data local inpath 'file:///home/centos/data.txt' into table mydb.tt");
spark-sql讀取映射hbase數據的hive外部表
1.拷貝如下jar包到${spark_home}/jars（spark2.0之前是${spark_home}/lib）:
hbase的jar包中
- hbase-protocol-1.2.0.jar
- hbase-client-1.2.0.jar
- hbase-common-1.2.0.jar
- hbase-server-1.2.0.jar
hive的jar包中
1.複製hive的hive-hbase-handler-2.1.0.jar文件到spark/jars目錄下。
2.複製hive/下的metrics的jar文件到spark下。
$>cd /soft/hive/lib
$>ls | grep metrics | cp `xargs` /soft/spark/jars
3.啓動spark-shell 本地模式測試
$>spark-shell --master local[4]
$scala>spark.sql("select * from mydb.ext_calllogs_in_hbase").show();
$scala>spark.sql("select count(*) ,substr(calltime,1,6) from ext_calllogs_in_hbase where caller = '15778423030' and substr(calltime,1,4) == '2017' group by substr(calltime,1,6)").show();