Hive Server 2 調研，安裝和部署

背景

      我們使用Hive Server 1已經很長時間了，用戶ad-hoc query，hive-web, wormhole，運營工具等都是通過hive server來提交語句。但是hive server極其不穩定，經常會莫名奇妙假死，導致client端所有的connection都被block住了。對此我們不得不配置一個crontab檢查腳本，會不斷執行"show tables"語句來檢測server是否假死，如果假死，只能殺死daemon進程重啓。另外Hive Server 1的concurrency支持不好，如果一個用戶在連接中設置了一些環境變量，綁定到一個thrift worker thread, 用戶斷開連接，另一個用戶也創建了一個連接，他有可能也被分配到之前的worker thread，會複用之前的配置。這是因爲thrift不支持檢測client是否斷開鏈接，它也就無法清除session狀態信息。同時session綁定到worker thread的方式很難做HA。Hive Server 2中已經完美支持了session, client端每次RPC call的時候會帶上一個SessionID, Server端會mapping到保存狀態信息的Session State，使得任何一個worker thread都可以執行同一個Session的不同語句，而不會綁死在同一個上。

      Hive 0.11 包含了Hive Server 1 和 Hive Server 2，還包含1的原因是爲了做到向下兼容性。從長遠來看都會以Hive Server 2作爲首選。

配置

1. 配置hive server監聽端口和Host

<property>
  <name>hive.server2.thrift.port</name>
  <value>10000</value>
</property>
<property>
  <name>hive.server2.thrift.bind.host</name>
  <value>test84.hadoop</value>
</property>

2. 配置kerberos認證，這樣thrift client與hive server 2, hive server 2與hdfs交互 都由kerberos作認證

<property>
  <name>hive.server2.authentication</name>
  <value>KERBEROS</value>
  <description>
    Client authentication types.
       NONE: no authentication check
       LDAP: LDAP/AD based authentication
       KERBEROS: Kerberos/GSSAPI authentication
       CUSTOM: Custom authentication provider
               (Use with property hive.server2.custom.authentication.class)
  </description>
</property>
<property>
  <name>hive.server2.authentication.kerberos.principal</name>
  <value>hadoop/[email protected]</value>
</property>
<property>
  <name>hive.server2.authentication.kerberos.keytab</name>
  <value>/etc/hadoop.keytab</value>
</property>

3. 設置impersonation，這樣hive server會以提交用戶的身份去執行語句，如果設置爲false，則會以起hive server daemon的admin user來執行語句

<property>
  <name>hive.server2.enable.doAs</name>
  <value>true</value>
</property>

執行命令$HIVE_HOME/bin/hive --service hiveserver2或者$HIVE_HOME/bin/hiveserver2 會調用org.apache.hive.service.server.HiveServer2的main方法來啓動

hive log中輸出日誌信息如下：

2013-09-17 14:59:21,081 INFO  server.HiveServer2 (HiveStringUtils.java:startupShutdownMessage(604)) - STARTUP_MSG:
/************************************************************
STARTUP_MSG: Starting HiveServer2
STARTUP_MSG:   host = test84.hadoop/10.1.77.84
STARTUP_MSG:   args = []
STARTUP_MSG:   version = 0.11.0
STARTUP_MSG:   classpath = 略.................
2013-09-17 14:59:21,957 INFO  security.UserGroupInformation (UserGroupInformation.java:loginUserFromKeytab(633)) - Login successful for user hadoop/[email protected] using keytab file /etc/hadoop.keytab
2013-09-17 14:59:21,958 INFO  service.AbstractService (AbstractService.java:init(89)) - Service:OperationManager is inited.
2013-09-17 14:59:21,958 INFO  service.AbstractService (AbstractService.java:init(89)) - Service:SessionManager is inited.
2013-09-17 14:59:21,958 INFO  service.AbstractService (AbstractService.java:init(89)) - Service:CLIService is inited.
2013-09-17 14:59:21,959 INFO  service.AbstractService (AbstractService.java:init(89)) - Service:ThriftCLIService is inited.
2013-09-17 14:59:21,959 INFO  service.AbstractService (AbstractService.java:init(89)) - Service:HiveServer2 is inited.
2013-09-17 14:59:21,959 INFO  service.AbstractService (AbstractService.java:start(104)) - Service:OperationManager is started.
2013-09-17 14:59:21,960 INFO  service.AbstractService (AbstractService.java:start(104)) - Service:SessionManager is started.
2013-09-17 14:59:21,960 INFO  service.AbstractService (AbstractService.java:start(104)) - Service:CLIService is started.
2013-09-17 14:59:22,007 INFO  metastore.HiveMetaStore (HiveMetaStore.java:newRawStore(409)) - 0: Opening raw store with implemenation class:org.apache.hadoop.hive.metastore.ObjectStore
2013-09-17 14:59:22,032 INFO  metastore.ObjectStore (ObjectStore.java:initialize(222)) - ObjectStore, initialize called
2013-09-17 14:59:22,955 INFO  metastore.ObjectStore (ObjectStore.java:getPMF(267)) - Setting MetaStore object pin classes with hive.metastore.cache.pinobjtypes="Table,StorageDescriptor,SerDeInfo,Partition,Database,Type,FieldSchema,Order"
2013-09-17 14:59:23,000 INFO  metastore.ObjectStore (ObjectStore.java:setConf(205)) - Initialized ObjectStore
2013-09-17 14:59:23,909 INFO  metastore.HiveMetaStore (HiveMetaStore.java:logInfo(452)) - 0: get_databases: default
2013-09-17 14:59:23,912 INFO  HiveMetaStore.audit (HiveMetaStore.java:logAuditEvent(238)) - ugi=hadoop/[email protected] ip=unknown-ip-addr cmd=get_databases: default
2013-09-17 14:59:23,933 INFO  service.AbstractService (AbstractService.java:start(104)) - Service:ThriftCLIService is started.
2013-09-17 14:59:23,948 INFO  service.AbstractService (AbstractService.java:start(104)) - Service:HiveServer2 is started.
2013-09-17 14:59:24,025 INFO  security.UserGroupInformation (UserGroupInformation.java:loginUserFromKeytab(633)) - Login successful for user hadoop/[email protected] using keytab file /etc/hadoop.keytab
2013-09-17 14:59:24,047 INFO  thrift.ThriftCLIService (ThriftCLIService.java:run(435)) - ThriftCLIService listening on test84.hadoop/10.1.77.84:10000

可以看到在HiveServer2已經變成一個Compisite Service了，它包含了一組service，包括OperationManager，SessionManager，CLIService，ThriftCLIService。並且在初始化的時候會建立HiveMetaStore連接，並調用get_databases命令來測試。最後啓動thrift server(實際上是一個TThreadPool)，監聽在test84.hadoop/10.1.77.84:10000端口上

另外Hadoop FileSystem Cache會以uri schema, authority, ugi(CurrentUser)和unique的組合作爲Key來緩存文件系統對象。但是這會導致hive server memory leak，通過"jmap -histo pid"可以看出filesystem對象數量和所佔空間非常大，所以需要在啓動hive server的時候加上disable file system cache的參數。

$HIVE_HOME/bin/hive --service hiveserver2 --hiveconf fs.hdfs.impl.disable.cache=true --hiveconf fs.file.impl.disable.cache=true

1. Beeline訪問hive server 2

Beeline是hive 0.11引入的新的交互式CLI，它基於SQLLine，可以作爲Hive JDBC Client端訪問Hive Server 2，啓動一個beeline就是維護了一個session。

由於採用了kerberos認證方式，所以需要在本地有kerberos ticket，並且在connection url中指定hive server 2的service principal，此處爲principal=hadoop/[email protected]，另外用戶名和密碼可以不用填寫，之後的語句會以當前ticket cache中principal的用戶身份來執行。

-dpsh-3.2$ bin/beeline
Beeline version 0.11.0 by Apache Hive
beeline> !connect jdbc:hive2://test84.hadoop:10000/default;principal=hadoop/[email protected]
scan complete in 2ms
Connecting to jdbc:hive2://test84.hadoop:10000/default;principal=hadoop/[email protected]
Enter username for jdbc:hive2://test84.hadoop:10000/default;principal=hadoop/[email protected]:
Enter password for jdbc:hive2://test84.hadoop:10000/default;principal=hadoop/[email protected]:
Connected to: Hive (version 0.11.0)
Driver: Hive (version 0.11.0)
Transaction isolation: TRANSACTION_REPEATABLE_READ
0: jdbc:hive2://test84.hadoop:10000/default> select count(1) from abc;
+------+
| _c0  |
+------+
| 0    |
+------+
1 row selected (29.277 seconds)
0: jdbc:hive2://test84.hadoop:10000/default> !q
Closing: org.apache.hive.jdbc.HiveConnection

thrift client和server會建立一個session handler，有唯一的HandleIdentifier(SessionID)，由CLIService中的SessionManager統一管理(維護了SessionHandle對HiveSession的mapping關係)，HiveSession維護了SessionConf和HiveConf信息，用戶的每次執行語句會新建一個driver，將hiveconf傳進去後再執行語句，這也就是Hive server 2支持concurrency的方式。每次操作(會有不同的opType,比如EXECUTE_STATEMEN)會生成獨立的OperationHandle，也有各自的HandleIdentifier。用戶在beeline中輸入"!q"會銷燬該session，並且銷燬相應的資源。

ps : 用下來有點不太爽的是執行mapreduce job時候沒有執行過程信息，如果是一個執行時間很長的語句，會等很久而沒有任何信息反饋。

2. JDBC方式

hive server 1的driver classname是org.apache.hadoop.hive.jdbc.HiveDriver，Hive Server 2的是org.apache.hive.jdbc.HiveDriver，這兩個容易混淆。

另外可以在connectionUrl中指定HiveConf param和變量，params之間用';'分割，params和variables用'#'來隔開。這些都是session級別的，hive在建立完session後，會首先執行set hiveconf key value語句。

比如:

1. 帶hiveconf和variables: jdbc:hive2://test84.hadoop:10000/default?hive.cli.conf.printheader=true#stab=salesTable;icol=customerID

2. 帶variables:  jdbc:hive2://test84.hadoop:10000/default;user=foo;password=bar

示例代碼:

import java.sql.Connection;
import java.sql.DriverManager;
import java.sql.ResultSet;
import java.sql.ResultSetMetaData;
import java.sql.SQLException;
import java.sql.Statement;

public class HiveTest {

    public static void main(String[] args) throws SQLException {
        try {
            Class.forName("org.apache.hive.jdbc.HiveDriver");
        } catch (ClassNotFoundException e) {
            e.printStackTrace();
        }
        Connection conn = DriverManager
                .getConnection(
                        "jdbc:hive2://test84.hadoop:10000/default;principal=hadoop/[email protected]",
                        "", "");
        Statement stmt = conn.createStatement();
        String sql = "select * from abc";
        System.out.println("Running: " + sql);
        ResultSet res = stmt.executeQuery(sql);
        ResultSetMetaData rsmd = res.getMetaData();
        int columnCount = rsmd.getColumnCount();
        for (int i = 1; i <= columnCount; i++) {
            System.out.println(rsmd.getColumnTypeName(i) + ":"
                    + rsmd.getColumnName(i));
        }

        while (res.next()) {
            System.out.println(String.valueOf(res.getInt(1)) + "\t"
                    + res.getString(2));
        }
    }
}

HiveStatement現在支持取消語句，調用Statement.cancel()會終止並銷燬正在執行中的driver

注：如果kerberos認證有問題的話，可以在起client jvm時候增加JVM option "-Dsun.security.krb5.debug=true"來查看詳細信息

參考鏈接：

https://cwiki.apache.org/confluence/display/Hive/HiveServer2+Thrift+API

https://cwiki.apache.org/confluence/display/Hive/Setting+up+HiveServer2

https://github.com/apache/hive/blob/trunk/service/if/TCLIService.thrift

http://blog.cloudera.com/blog/2013/07/how-hiveserver2-brings-security-and-concurrency-to-apache-hive/

本文鏈接http://blog.csdn.net/lalaguozhe/article/details/11776055，轉載請註明