集羣突然nagios報出一臺regionserver掛了。頓時忙碌起來。
上去一看，從log中看到這樣一條信息
ERROR org.apache.hadoop.hbase.regionserver.HRegionServer: ZooKeeper session expired
之後， regionserver就理直氣壯地退出了。
於是查了下代碼，看到了在org.apache.hadoop.hbase.regionserver.HRegionSever.java下這樣一段代碼。
/**
* We register ourselves as a watcher on the master address ZNode. This is
* called by ZooKeeper when we get an event on that ZNode. When this method
* is called it means either our master has died, or a new one has come up.
* Either way we need to update our knowledge of the master.
* @param event WatchedEvent from ZooKeeper.
*/
public void process(WatchedEvent event) {
EventType type = event.getType();
KeeperState state = event.getState();
LOG.info(“Got ZooKeeper event, state: ” + state + “, type: ” +
type + “, path: ” + event.getPath());
// Ignore events if we’re shutting down.
if (stopRequested.get()) {
LOG.debug(“Ignoring ZooKeeper event while shutting down”);
return;
}
if (state == KeeperState.Expired) {
LOG.error(“ZooKeeper session expired”);
boolean restart =
this.conf.getBoolean(“hbase.regionserver.restart.on.zk.expire”, false);
if (restart) {
restart();
} else {
abort();
}
} else if (type == EventType.NodeDeleted) {
watchMasterAddress();
} else if (type == EventType.NodeCreated) {
getMaster();
// ZooKeeper watches are one time only, so we need to re-register our watch.
watchMasterAddress();
}
}
這段註釋寫的很清楚了。對於一個reigonserver，
他需要將自己註冊到Zookeeper上master的Znode上。這樣的目的，是當master
宕機或者新的master啓動的時候，能及時收到通知。對於regionserver來說，維持和Zookeeper的聯繫是非常重要的。因爲
regionserver需要定期的將心跳包發給master
server。如果regionserver不能及時的知道master的改變，就會導致regionserver和master失去聯繫，而成爲一個僵
死的進程。
於是，在默認情況下，regionserver遇到這種情況，就選擇退出。
爲什麼regionserver 和Zookeeper的session expired? 可能的原因有
1. 網絡不好。
2. Java full GC， 這會block所有的線程。如果時間比較長，也會導致session expired.
怎麼辦？
1. 將Zookeeper的timeout時間加長。
2. 配置“hbase.regionserver.restart.on.zk.expire” 爲true。 這樣子，遇到ZooKeeper session expired ， regionserver將選擇 restart 而不是 abort
具體的配置是，在hbase-site.xml中加入
<property>
<name>zookeeper.session.timeout</name>
<value>90000</value>
<description>ZooKeeper session timeout.
HBase passes this to the zk quorum as suggested maximum time for a
session. See http://hadoop.apache.org/zookeeper/docs/current/zookeeperProgrammers.html#ch_zkSessions
“The client sends a requested timeout, the server responds with the
timeout that it can give the client. The current implementation
requires that the timeout be a minimum of 2 times the tickTime
(as set in the server configuration) and a maximum of 20 times
the tickTime.” Set the zk ticktime with hbase.zookeeper.property.tickTime.
In milliseconds.
</description>
</property>
<property>
<name>hbase.regionserver.restart.on.zk.expire</name>
<value>true</value>
<description>
Zookeeper session expired will force regionserver exit.
Enable this will make the regionserver restart.
</description>
</property>
爲了避免java full GC suspend thread 對Zookeeper heartbeat的影響，我們還需要對hbase-env.sh進行配置。
將
export HBASE_OPTS="$HBASE_OPTS -XX:+HeapDumpOnOutOfMemoryError \ -XX:+UseConcMarkSweepGC -XX:+CMSIncrementalMode"
修改成
export HBASE_OPTS="$HBASE_OPTS -XX:+HeapDumpOnOutOfMemoryError -XX:+UseConcMarkSweepGC -XX:+CMSParallelRemarkEnabled -XX:+UseCMSInitiatingOccupancyOnly -XX:+UseParNewGC -Xmn256m"