Hive 3.1.2
Table of Contents
2.1 Embedded Database + Embedded Metastore Server模式
2.2 Embedded Database + Remote Metastore Server 模式
2.3 Remote Database + Embedded Metastore Server 模式
2.4 Remote Database + Remote Metastore Server 模式
1 共用配置信息
1.1 配置Hadoop安全設置
$HADOOP_HOME/etc/hadoop/core-site.xml
<property> <name>hadoop.proxyuser.root.hosts</name> <value>*</value> </property> <property> <name>hadoop.proxyuser.root.groups</name> <value>*</value> </property>
這裏使用的root用戶。
1.2 創建數據存放目錄
$HADOOP_HOME/bin/hadoop fs -mkdir /tmp $HADOOP_HOME/bin/hadoop fs -mkdir /user/hive/warehouse $HADOOP_HOME/bin/hadoop fs -chmod g+w /tmp $HADOOP_HOME/bin/hadoop fs -chmod g+w /user/hive/warehouse
2. Hive 運行模式
2.1 Embedded Database + Embedded Metastore Server模式
服務器配置
不給定任何配置，默認情況就是該模式。
初始化 metastore 數據庫
$HIVE_HOME/bin/schematool -dbType derby -initSchema
啓動服務
使用root用戶來啓動hiveserver2
sudo $HIVE_HOME/bin/hiveserver2
測試連接
$HIVE_HOME/bin/beeline -u jdbc:hive2://localhost:10000 -n root
2.2 Embedded Database + Remote Metastore Server 模式
服務器配置
hive-site.xml配置文件中，需要指定以下信息：
<configuration> <!-- database configuration --> <property> <name>javax.jdo.option.ConnectionDriverName</name> <value>org.apache.derby.jdbc.EmbeddedDriver</value> <description>Driver class name for a JDBC metastore</description> </property> <property> <name>javax.jdo.option.ConnectionURL</name> <value>jdbc:derby:;databaseName=metastore_db;create=true</value> <description> JDBC connect string for a JDBC metastore. To use SSL to encrypt/authenticate the connection, provide database-specific SSL flag in the connection URL. For example, jdbc:postgresql://myhost/db?ssl=true for postgres database. </description> </property> <property> <name>javax.jdo.option.ConnectionUserName</name> <value>APP</value> <description>Username to use against metastore database</description> </property> <property> <name>javax.jdo.option.ConnectionPassword</name> <value>mine</value> <description>password to use against metastore database</description> </property> <!-- metastore mode definition --> <property> <name>hive.metastore.port</name> <value>9083</value> <description>Hive metastore listener port</description> </property> <property> <name>hive.metastore.uris</name> <value>thrift://localhost:9083</value> <description>Thrift URI for the remote metastore. Used by metastore client to connect to remote metastore.</description> </property> </configuration>
初始化 metastore 數據庫
$HIVE_HOME/bin/schematool -dbType derby -initSchema
啓動服務
啓動metastore服務
$HIVE_HOME/bin/hive –service metastore
使用root用戶來啓動hiveserver2
sudo $HIVE_HOME/bin/hive –service hiveserver2
測試連接
檢查metastore服務是否啓動成功
lsof -i:9083
如能查詢到信息表示啓動成功。
使用beeline來測試連接
$HIVE_HOME/bin/beeline -u jdbc:hive2://localhost:10000 -n root
2.3 Remote Database + Embedded Metastore Server 模式
依賴包準備
cp mysql-connector-java-5.1.41.jar $HIVE_HOME/lib/
服務器配置
hive-site.xml配置文件中，需要指定以下信息：
<configuration> <!-- database configuration --> <property> <name>javax.jdo.option.ConnectionDriverName</name> <value>com.mysql.jdbc.Driver</value> <description>Driver class name for a JDBC metastore</description> </property> <property> <name>javax.jdo.option.ConnectionURL</name> <value>jdbc:mysql://localhost:3306/metastore</value> <description> JDBC connect string for a JDBC metastore. To use SSL to encrypt/authenticate the connection, provide database-specific SSL flag in the connection URL. For example, jdbc:postgresql://myhost/db?ssl=true for postgres database. </description> </property> <property> <name>javax.jdo.option.ConnectionUserName</name> <value>root</value> <description>Username to use against metastore database</description> </property> <property> <name>javax.jdo.option.ConnectionPassword</name> <value>abc123</value> <description>password to use against metastore database</description> </property> <!-- metastore mode definition --> <property> <name>hive.metastore.uris</name> <value /> <description>Thrift URI for the remote metastore. Used by metastore client to connect to remote metastore.</description> </property> </configuration>
初始化 metastore 數據庫
$HIVE_HOME/bin/schematool -dbType mysql -initSchema
啓動服務
使用root用戶來啓動hiveserver2
sudo $HIVE_HOME/bin/hiveserver2
測試連接
使用beeline來測試連接
$HIVE_HOME/bin/beeline -u jdbc:hive2://localhost:10000 -n root
2.4 Remote Database + Remote Metastore Server 模式
依賴包準備
cp mysql-connector-java-5.1.41.jar $HIVE_HOME/lib/
服務器配置
hive-site.xml配置文件中，需要指定以下信息：
<configuration> <!-- database configuration --> <property> <name>javax.jdo.option.ConnectionDriverName</name> <value>com.mysql.jdbc.Driver</value> <description>Driver class name for a JDBC metastore</description> </property> <property> <name>javax.jdo.option.ConnectionURL</name> <value>jdbc:mysql://localhost:3306/metastore</value> <description> JDBC connect string for a JDBC metastore. To use SSL to encrypt/authenticate the connection, provide database-specific SSL flag in the connection URL. For example, jdbc:postgresql://myhost/db?ssl=true for postgres database. </description> </property> <property> <name>javax.jdo.option.ConnectionUserName</name> <value>root</value> <description>Username to use against metastore database</description> </property> <property> <name>javax.jdo.option.ConnectionPassword</name> <value>abc123</value> <description>password to use against metastore database</description> </property> <!-- metastore mode definition --> <property> <name>hive.metastore.port</name> <value>9083</value> <description>Hive metastore listener port</description> </property> <property> <name>hive.metastore.uris</name> <value>thrift://localhost:9083</value> <description>Thrift URI for the remote metastore. Used by metastore client to connect to remote metastore.</description> </property> </configuration>
初始化 metastore 數據庫
$HIVE_HOME/bin/schematool -dbType mysql -initSchema
啓動服務
啓動metastore服務
$HIVE_HOME/bin/hive –service metastore
使用root用戶來啓動hiveserver2
sudo $HIVE_HOME/bin/hive –service hiveserver2
測試連接
檢查metastore服務是否啓動成功
lsof -i:9083
如能查詢到信息表示啓動成功。
使用beeline來測試連接
$HIVE_HOME/bin/beeline -u jdbc:hive2://localhost:10000 -n root
遇到的問題：
MetaException(message:Hive Schema version 3.1.0 does not match metastore's schema version 1.2.0 Metastore is not upgraded or corrupt)
使用spark sql +remote metastore database連接後，在此啓動hive出現上面的錯誤。解決方案：
<property>
<name>hive.metastore.schema.verification</name>
<value>false</value>
<description>
Enforce metastore schema version consistency.
True: Verify that version information stored in is compatible with one from Hive jars. Also disable automatic
schema migration attempt. Users are required to manually migrate schema after Hive upgrade which ensures
proper metastore schema migration. (Default)
False: Warn if the version information stored in metastore doesn't match with one from in Hive jars.
</description>
</property>
生產環境，最後設定爲true。
參考文章：