hadoop(僞分佈)+hive採坑集合

今天工作量少，準備玩一下hive，，誰知。。。。一搭一個坑啊 閒話少說 進入正題~

配置系統環境的問題就不說了 copy一下就行  系統爲centos7 64

export HADOOP_HOME=/apps/hadoop
export HADOOP_BIN_DIR=/apps/hadoop/bin
export HADOOP_HOME_WARN_SUPPRESS=1
export HADOOP_CONF_DIR=$HADOOP_HOME/etc/hadoop
export JAVA_HOME=/apprun/jdk
export PATH=/usr/local/mysql/bin:/usr/local/mysql/lib:/sbin:/bin:/usr/sbin:/usr/bin:/usr/X11R6/bin:$JAVA_HOME/bin:$HADOOP_HOME/bin:$HADOOP_HOME/sbin
HIVE_HOME=/apps/hive
export CLASSPATH=.:$JAVA_HOME/lib/tools.jar:$JAVA_HOME/lib/dt.jar 
export PATH=$JAVA_HOME/bin:$HOME/bin:$HOME/.local/bin:/apps/hive/bin:$PATH
export HADOOP_CLASSPATH=/apps/hive/conf:/apps/hadoop/bin:/apps/hadoop/sbin
 

進行hive-site的配置（以下是hive連接mysql數據庫的配置，如果直連derby數據庫，這四個配置都需要修改）

<property>
    <name>javax.jdo.option.ConnectionURL</name>
    <value>jdbc:mysql://172.28.81.56:3306/myhive</value><!-- MySQL地址 -->
    <description>
      JDBC connect string for a JDBC metastore.
      To use SSL to encrypt/authenticate the connection, provide database-specific SSL flag in the connection URL.
      For example, jdbc:postgresql://myhost/db?ssl=true for postgres database.
    </description>
  </property>
  <property>
    <name>javax.jdo.option.ConnectionUserName</name>
    <value>root</value>
    <description>Username to use against metastore database</description><!-- MySQL用戶名 -->
  </property>
  <property>
    <name>javax.jdo.option.ConnectionPassword</name>
    <value>haijintao</value><!-- MySQL密碼-->
    <description>password to use against metastore database</description>
  </property>
  <property>
    <name>javax.jdo.option.ConnectionDriverName</name>
    <value>com.mysql.cj.jdbc.Driver</value><!-- MySQL驅動類（我自用的mysql8.0，如果是之前的版本可能會不一樣） -->
    <description>Driver class name for a JDBC metastore</description>
  </property> 

 

然後就開始採坑了

初始化hive metastore_db:/hive/bin/hive schematool -initSchema -dbType mysql 報錯

原因是配置的數據庫中原來有表，需要清空重新建立一個數據庫

清空重建數據庫後初始化成功，然後在進入hive 報如下問題

Exception in thread "main" java.lang.RuntimeException: java.net.ConnectException: Call From skillbi0/172.28.81.56 to skillbi0:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see:  http://wiki.apache.org/hadoop/ConnectionRefused
 

連接問題需要修改core-site.xml

<property>
<name>fs.defaultFS</name>
<value>hdfs://skillbi0:8020/</value><!-- 因爲是僞分佈 所以此處的skillbi0 配置和slaves中一樣 -->
</property>
</configuration>
<property>     
  <name>hadoop.proxyuser.root.hosts</name>     
  <value>*</value> 
 </property> 
<property>     
  <name>hadoop.proxyuser.root.groups</name>    
    <value>*</value> 
</property>

敲重點！ 在修改hadoop配置後 一定要先刪除/tmp/hadoop* 文件以及 /hadoop/logs/下的所有文件，不然重啓會造成衝突和異常。 

hadoop namenode -format 格式化後重啓

start-dfs.sh  start-yarn.sh  分別運行

$> hive   進入hive中   報錯  namenode 進入安全模式

Exception in thread "main" java.lang.RuntimeException: org.apache.hadoop.hdfs.server.namenode.SafeModeException: Cannot create directory /tmp/hive. Name node is in safe mode.
Resources are low on NN. Please add or free up more resourcesthen turn off safe mode manually. NOTE:  If you turn off safe mode before adding resources, the NN will immediately return to safe mode. Use "hdfs dfsadmin -safemode leave" to turn safe mode off. NamenodeHostName:skillbi0

$>  hdfs dfsadmin -safemode leave 關閉安全模式

$> hive

$> create database hive1;

$> use hive1;

$> create table t(eid int,name string);

$> insert into t(eid,name) values(1,'zhangsan');

然後進行下面的idea遠程測試

========================================================================================

放一下maven的配置和啓動類代碼（簡單測試）
<dependency>
    <groupId>org.apache.hive</groupId>
    <artifactId>hive-jdbc</artifactId>
    <version>2.1.0</version>
</dependency>

/**
 * @author haijintao
 */
public class App {

    public static void main(String[] args) throws ClassNotFoundException, SQLException {
        Class.forName("org.apache.hive.jdbc.HiveDriver");
        Connection conn = DriverManager.getConnection("jdbc:hive2://172.28.81.56:10000/hive1","root","haijintao");
        PreparedStatement ppst = conn.prepareStatement("select * from t");
        ResultSet rs = ppst.executeQuery();
        while(rs.next()){
            int id = rs.getInt("eid");
            String name = rs.getString("name");
            System.out.println(id+","+name);
        }
        rs.close();
        ppst.close();
        conn.close();
    }

}