今天工作量少,准备玩一下hive,,谁知。。。。一搭一个坑啊 闲话少说 进入正题~
配置系统环境的问题就不说了 copy一下就行 系统为centos7 64
export HADOOP_HOME=/apps/hadoop
export HADOOP_BIN_DIR=/apps/hadoop/bin
export HADOOP_HOME_WARN_SUPPRESS=1
export HADOOP_CONF_DIR=$HADOOP_HOME/etc/hadoop
export JAVA_HOME=/apprun/jdk
export PATH=/usr/local/mysql/bin:/usr/local/mysql/lib:/sbin:/bin:/usr/sbin:/usr/bin:/usr/X11R6/bin:$JAVA_HOME/bin:$HADOOP_HOME/bin:$HADOOP_HOME/sbin
HIVE_HOME=/apps/hive
export CLASSPATH=.:$JAVA_HOME/lib/tools.jar:$JAVA_HOME/lib/dt.jar
export PATH=$JAVA_HOME/bin:$HOME/bin:$HOME/.local/bin:/apps/hive/bin:$PATH
export HADOOP_CLASSPATH=/apps/hive/conf:/apps/hadoop/bin:/apps/hadoop/sbin
进行hive-site的配置(以下是hive连接mysql数据库的配置,如果直连derby数据库,这四个配置都需要修改)
<property>
<name>javax.jdo.option.ConnectionURL</name>
<value>jdbc:mysql://172.28.81.56:3306/myhive</value><!-- MySQL地址 -->
<description>
JDBC connect string for a JDBC metastore.
To use SSL to encrypt/authenticate the connection, provide database-specific SSL flag in the connection URL.
For example, jdbc:postgresql://myhost/db?ssl=true for postgres database.
</description>
</property>
<property>
<name>javax.jdo.option.ConnectionUserName</name>
<value>root</value>
<description>Username to use against metastore database</description><!-- MySQL用户名 -->
</property>
<property>
<name>javax.jdo.option.ConnectionPassword</name>
<value>haijintao</value><!-- MySQL密码-->
<description>password to use against metastore database</description>
</property>
<property>
<name>javax.jdo.option.ConnectionDriverName</name>
<value>com.mysql.cj.jdbc.Driver</value><!-- MySQL驱动类(我自用的mysql8.0,如果是之前的版本可能会不一样) -->
<description>Driver class name for a JDBC metastore</description>
</property>
然后就开始采坑了
初始化hive metastore_db:/hive/bin/hive schematool -initSchema -dbType mysql 报错
原因是配置的数据库中原来有表,需要清空重新建立一个数据库
清空重建数据库后初始化成功,然后在进入hive 报如下问题
Exception in thread "main" java.lang.RuntimeException: java.net.ConnectException: Call From skillbi0/172.28.81.56 to skillbi0:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see: http://wiki.apache.org/hadoop/ConnectionRefused
连接问题需要修改core-site.xml
<property>
<name>fs.defaultFS</name>
<value>hdfs://skillbi0:8020/</value><!-- 因为是伪分布 所以此处的skillbi0 配置和slaves中一样 -->
</property>
</configuration>
<property>
<name>hadoop.proxyuser.root.hosts</name>
<value>*</value>
</property>
<property>
<name>hadoop.proxyuser.root.groups</name>
<value>*</value>
</property>
敲重点! 在修改hadoop配置后 一定要先删除/tmp/hadoop* 文件以及 /hadoop/logs/下的所有文件,不然重启会造成冲突和异常。
hadoop namenode -format 格式化后重启
start-dfs.sh start-yarn.sh 分别运行
$> hive 进入hive中 报错 namenode 进入安全模式
Exception in thread "main" java.lang.RuntimeException: org.apache.hadoop.hdfs.server.namenode.SafeModeException: Cannot create directory /tmp/hive. Name node is in safe mode.
Resources are low on NN. Please add or free up more resourcesthen turn off safe mode manually. NOTE: If you turn off safe mode before adding resources, the NN will immediately return to safe mode. Use "hdfs dfsadmin -safemode leave" to turn safe mode off. NamenodeHostName:skillbi0
$> hdfs dfsadmin -safemode leave 关闭安全模式
$> hive
$> create database hive1;
$> use hive1;
$> create table t(eid int,name string);
$> insert into t(eid,name) values(1,'zhangsan');
然后进行下面的idea远程测试
========================================================================================
放一下maven的配置和启动类代码(简单测试) <dependency> <groupId>org.apache.hive</groupId> <artifactId>hive-jdbc</artifactId> <version>2.1.0</version> </dependency>
/** * @author haijintao */ public class App { public static void main(String[] args) throws ClassNotFoundException, SQLException { Class.forName("org.apache.hive.jdbc.HiveDriver"); Connection conn = DriverManager.getConnection("jdbc:hive2://172.28.81.56:10000/hive1","root","haijintao"); PreparedStatement ppst = conn.prepareStatement("select * from t"); ResultSet rs = ppst.executeQuery(); while(rs.next()){ int id = rs.getInt("eid"); String name = rs.getString("name"); System.out.println(id+","+name); } rs.close(); ppst.close(); conn.close(); } }