一、引子
今天在跑一段很複雜而且涉及數據量10多年的N個表join的長SQL時，發生了OOM的異常。
由於一個map通常配置只有64MB或者128MB，則在Map階段出現OOM的情況很少見。所以一般發生在reduce階段。
但是今天這個異常詳細的看後，會發現既不是map階段，也不是reduce階段，發現不是執行過程，而是driver提交job階段就OOM了。
Hive中XMLEncoder序列化MapredWork引發OutOfMemoryError
XMLEncoder導致java.lang.OutOfMemoryError: GC overhead limit exceeded
二、概括回顧
先概括下，Hive中出現OOM的異常原因大致分爲以下幾種：
1. Map階段OOM。
2. Reduce階段OOM。
3. Driver提交Job階段OOM。
Map階段OOM：
1. 發生OOM的機率很小，除非你程序的邏輯不正常，亦或是程序寫的不高效，產生垃圾太多。
Reduce階段OOM：
1. data skew 數據傾斜
data skew是引發這個的一個原因。
key分佈不均勻，導致某一個reduce所處理的數據超過預期，導致jvm頻繁GC。
2. value對象過多或者過大
某個reduce中的value堆積的對象過多，導致jvm頻繁GC。
解決辦法：
1. 增加reduce個數，set mapred.reduce.tasks=300，。
2. 在hive-site.xml中設置，或者在hive shell裏設置 set mapred.child.java.opts = -Xmx512m
或者只設置reduce的最大heap爲2G，並設置垃圾回收器的類型爲並行標記回收器，這樣可以顯著減少GC停頓，但是稍微耗費CPU。
set mapred.reduce.child.java.opts=-Xmx2g -XX:+UseConcMarkSweepGC;
3. 使用map join 代替 common join. 可以set hive.auto.convert.join = true
4. 設置 hive.optimize.skewjoin = true 來解決數據傾斜問題
Driver提交job階段OOM：
job產生的執行計劃的條目太多，比如掃描的分區過多，上到4k-6k個分區的時候，並且是好幾張表的分區都很多時，這時做join。
究其原因，是 因爲序列化時，會將這些分區，即hdfs文件路徑，封裝爲Path對象，這樣，如果對象太多了，而且Driver啓動的時候設置的heap size太小，則會導致在Driver內序列化這些MapRedWork時，生成的對象太多，導致頻繁GC，則會引發如下異常:
java.lang.OutOfMemoryError: GC overhead limit exceeded
at sun.nio.cs.UTF_8.newEncoder(UTF_8.java:53)
at java.beans.XMLEncoder.createString(XMLEncoder.java:572)
三、診斷問題
如何診斷到了問題：
在網上搜異常，在Hive的IRA發現一個issues，和我的情況類似：
HIVE-2988
問題描述：
Use of XMLEncoder to serialize MapredWork causes OOM in hive cli
When running queries on tables with 6000 partitions, hive cli if configured with 128M runs into OOM. Heapdump showed 37MB occupied by one XMLEncoder object while the MapredWork was 500K which is highly inefficient. We should switch to using something more efficient like XStream.
比較相近的解釋：
I ran with 128M to investigate the OOM. We have resorted to running with 1G as XmX because we keep hitting OOM with bigger tables in hive. There were other things that contributed to the memory usage - mostly Path objects because of the higher number of partitions. But they are absolutely needed. XMLEncoder is something that created too much garbage in a very short span and caused GC. That would be something easy to change/fix without having to touch the core logic.
We should be looking at fixing the root cause of the problem instead of keeping on increasing the memory requirements. Ours is a highly multi-tenant system and there are lot of other programs(pig,etc) running too in the gateway. So running with a lower memory(256-512MB) will help.
Found two other reports of this issue:
http://mail-archives.apache.org/mod_mbox/hive-user/201106.mbox/%[email protected]%3E
https://issues.apache.org/jira/browse/HIVE-1316
This fix increased the max heap size of CLI client and disabled GC overhead limit.
於是親自確認一下問題處在哪個地方：
因爲這段複雜的sql有22個job，每次到第20個job的時候就會出現oom，所以我特意將數據置空，調試這在shell下跑。
發現跑到第20個job，掃描的正是掃描了14年的註冊表，推斷是分區太多，造成的序列化時對象太多OOM。
以下是調試圖：
將查詢計劃打印出來，會發現很大，主要是因爲有一個註冊表，掃描了近14年的歷史數據，14×365 = 5110 個分區。除去早些年份數據的不完整性，大約3138個分區。
每個分區都會生成一個Path對象，這樣僅僅是這一個表，生成的查詢計劃被序列化爲對象會耗去大半內存，如果在和其它的表好幾年的數據繼續做join的話，又會耗去更多內存來序列化成對象，這樣就會頻繁GC，GC不出來，就會拋出OOM的異常。
僅僅是一個查詢，都打印出了25MB，說明掃描的數據量是在是太多了。
explain extended a_complex_hql_query > /app/hadoop/shengli/large_plan
du -sh large_plan
25M large_plan
詳細日誌如下：
2014-11-21 13:13:28,334 INFO [Thread-2] AbstractBaseAction$StreamDrainer - Hadoop job information for Stage-23: number of mappers: 12; number of reducers: 1
2014-11-21 13:13:28,354 INFO [Thread-2] AbstractBaseAction$StreamDrainer - 2014-11-21 13:13:28,347 Stage-23 map = 50%, reduce = 0%, Cumulative CPU 42.04 sec
2014-11-21 13:13:48,370 INFO [Thread-2] AbstractBaseAction$StreamDrainer - 2014-11-21 13:13:48,363 Stage-23 map = 100%, reduce = 33%, Cumulative CPU 262.39 sec
2014-11-21 13:14:29,228 INFO [Thread-2] AbstractBaseAction$StreamDrainer - 2014-11-21 13:14:29,222 Stage-23 map = 100%, reduce = 69%, Cumulative CPU 262.39 sec
2014-11-21 13:14:49,248 INFO [Thread-2] AbstractBaseAction$StreamDrainer - 2014-11-21 13:14:49,239 Stage-23 map = 100%, reduce = 92%, Cumulative CPU 324.73 sec
2014-11-21 13:15:11,952 INFO [Thread-2] AbstractBaseAction$StreamDrainer - 2014-11-21 13:15:11,944 Stage-23 map = 100%, reduce = 100%, Cumulative CPU 360.1 sec
2014-11-21 13:15:36,740 INFO [Thread-2] AbstractBaseAction$StreamDrainer - 2014-11-21 13:15:36,734 Stage-23 map = 100%, reduce = 100%, Cumulative CPU 360.1 sec
2014-11-21 13:15:36,741 INFO [Thread-2] AbstractBaseAction$StreamDrainer - MapReduce Total cumulative CPU time: 6 minutes 0 seconds 100 msec
2014-11-21 13:15:36,749 INFO [Thread-2] AbstractBaseAction$StreamDrainer - Ended Job = job_201411141019_68657
2014-11-21 13:15:40,306 INFO [Thread-2] AbstractBaseAction$StreamDrainer - Launching Job 20 out of 21
2014-11-21 13:15:42,162 INFO [Thread-2] AbstractBaseAction$StreamDrainer - Number of reduce tasks not specified. Estimated from input data size: 252
2014-11-21 13:15:42,163 INFO [Thread-2] AbstractBaseAction$StreamDrainer - In order to change the average load for a reducer (in bytes):
2014-11-21 13:15:42,163 INFO [Thread-2] AbstractBaseAction$StreamDrainer - set hive.exec.reducers.bytes.per.reducer=<number>
2014-11-21 13:15:42,163 INFO [Thread-2] AbstractBaseAction$StreamDrainer - In order to limit the maximum number of reducers:
2014-11-21 13:15:42,163 INFO [Thread-2] AbstractBaseAction$StreamDrainer - set hive.exec.reducers.max=<number>
2014-11-21 13:15:42,163 INFO [Thread-2] AbstractBaseAction$StreamDrainer - In order to set a constant number of reducers:
2014-11-21 13:15:42,163 INFO [Thread-2] AbstractBaseAction$StreamDrainer - set mapred.reduce.tasks=<number>
2014-11-21 13:16:40,377 INFO [Thread-2] AbstractBaseAction$StreamDrainer - java.lang.OutOfMemoryError: GC overhead limit exceeded
2014-11-21 13:16:40,378 INFO [Thread-2] AbstractBaseAction$StreamDrainer - at sun.nio.cs.UTF_8.newEncoder(UTF_8.java:53)
2014-11-21 13:16:40,378 INFO [Thread-2] AbstractBaseAction$StreamDrainer - at java.beans.XMLEncoder.createString(XMLEncoder.java:572)
2014-11-21 13:16:40,378 INFO [Thread-2] AbstractBaseAction$StreamDrainer - at java.beans.XMLEncoder.outputValue(XMLEncoder.java:543)
2014-11-21 13:16:40,378 INFO [Thread-2] AbstractBaseAction$StreamDrainer - at java.beans.XMLEncoder.outputStatement(XMLEncoder.java:682)
2014-11-21 13:16:40,378 INFO [Thread-2] AbstractBaseAction$StreamDrainer - at java.beans.XMLEncoder.outputStatement(XMLEncoder.java:687)
2014-11-21 13:16:40,378 INFO [Thread-2] AbstractBaseAction$StreamDrainer - at java.beans.XMLEncoder.outputValue(XMLEncoder.java:552)
2014-11-21 13:16:40,378 INFO [Thread-2] AbstractBaseAction$StreamDrainer - at java.beans.XMLEncoder.outputStatement(XMLEncoder.java:682)
2014-11-21 13:16:40,378 INFO [Thread-2] AbstractBaseAction$StreamDrainer - at java.beans.XMLEncoder.outputStatement(XMLEncoder.java:687)
2014-11-21 13:16:40,379 INFO [Thread-2] AbstractBaseAction$StreamDrainer - at java.beans.XMLEncoder.outputValue(XMLEncoder.java:552)
2014-11-21 13:16:40,379 INFO [Thread-2] AbstractBaseAction$StreamDrainer - at java.beans.XMLEncoder.outputStatement(XMLEncoder.java:682)
2014-11-21 13:16:40,379 INFO [Thread-2] AbstractBaseAction$StreamDrainer - at java.beans.XMLEncoder.outputStatement(XMLEncoder.java:687)
2014-11-21 13:16:40,379 INFO [Thread-2] AbstractBaseAction$StreamDrainer - at java.beans.XMLEncoder.outputValue(XMLEncoder.java:552)
2014-11-21 13:16:40,379 INFO [Thread-2] AbstractBaseAction$StreamDrainer - at java.beans.XMLEncoder.outputStatement(XMLEncoder.java:682)
2014-11-21 13:16:40,379 INFO [Thread-2] AbstractBaseAction$StreamDrainer - at java.beans.XMLEncoder.outputStatement(XMLEncoder.java:687)
2014-11-21 13:16:40,379 INFO [Thread-2] AbstractBaseAction$StreamDrainer - at java.beans.XMLEncoder.outputValue(XMLEncoder.java:552)
2014-11-21 13:16:40,380 INFO [Thread-2] AbstractBaseAction$StreamDrainer - at java.beans.XMLEncoder.outputStatement(XMLEncoder.java:682)
2014-11-21 13:16:40,380 INFO [Thread-2] AbstractBaseAction$StreamDrainer - at java.beans.XMLEncoder.outputStatement(XMLEncoder.java:687)
2014-11-21 13:16:40,380 INFO [Thread-2] AbstractBaseAction$StreamDrainer - at java.beans.XMLEncoder.outputValue(XMLEncoder.java:552)
2014-11-21 13:16:40,380 INFO [Thread-2] AbstractBaseAction$StreamDrainer - at java.beans.XMLEncoder.flush(XMLEncoder.java:398)
2014-11-21 13:16:40,380 INFO [Thread-2] AbstractBaseAction$StreamDrainer - at java.beans.XMLEncoder.close(XMLEncoder.java:429)
2014-11-21 13:16:40,380 INFO [Thread-2] AbstractBaseAction$StreamDrainer - at org.apache.hadoop.hive.ql.exec.Utilities.serializeMapRedWork(Utilities.java:532)
2014-11-21 13:16:40,380 INFO [Thread-2] AbstractBaseAction$StreamDrainer - at org.apache.hadoop.hive.ql.exec.Utilities.setMapRedWork(Utilities.java:376)
2014-11-21 13:16:40,381 INFO [Thread-2] AbstractBaseAction$StreamDrainer - at org.apache.hadoop.hive.ql.exec.ExecDriver.execute(ExecDriver.java:418)
2014-11-21 13:16:40,381 INFO [Thread-2] AbstractBaseAction$StreamDrainer - at org.apache.hadoop.hive.ql.exec.MapRedTask.execute(MapRedTask.java:138)
2014-11-21 13:16:40,381 INFO [Thread-2] AbstractBaseAction$StreamDrainer - at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:144)
2014-11-21 13:16:40,381 INFO [Thread-2] AbstractBaseAction$StreamDrainer - at org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:57)
2014-11-21 13:16:40,381 INFO [Thread-2] AbstractBaseAction$StreamDrainer - at org.apache.hadoop.hive.ql.Driver.launchTask(Driver.java:1355)
2014-11-21 13:16:40,381 INFO [Thread-2] AbstractBaseAction$StreamDrainer - at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:1139)
2014-11-21 13:16:40,381 INFO [Thread-2] AbstractBaseAction$StreamDrainer - at org.apache.hadoop.hive.ql.Driver.run(Driver.java:945)
2014-11-21 13:16:40,381 INFO [Thread-2] AbstractBaseAction$StreamDrainer - at org.apache.hadoop.hive.cli.CliDriver.processLocalCmd(CliDriver.java:259)
2014-11-21 13:16:40,381 INFO [Thread-2] AbstractBaseAction$StreamDrainer - at org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:216)
2014-11-21 13:16:40,381 INFO [Thread-2] AbstractBaseAction$StreamDrainer - at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:413)
2014-11-21 13:16:40,381 INFO [Thread-2] AbstractBaseAction$StreamDrainer - FAILED: Execution Error, return code -101 from org.apache.hadoop.hive.ql.exec.MapRedTask
2014-11-21 13:16:40,382 INFO [Thread-2] AbstractBaseAction$StreamDrainer - MapReduce Jobs Launched:
2014-11-21 13:16:40,382 INFO [Thread-2] AbstractBaseAction$StreamDrainer - Job 0: Map: 33 Reduce: 5 Cumulative CPU: 903.59 sec HDFS Read: 4225680985 HDFS Write: 123129169 S
UCCESS
2014-11-21 13:16:40,382 INFO [Thread-2] AbstractBaseAction$StreamDrainer - Job 1: Map: 192 Reduce: 4 Cumulative CPU: 3215.94 sec HDFS Read: 3036813878 HDFS Write: 322647287
SUCCESS
2014-11-21 13:16:40,382 INFO [Thread-2] AbstractBaseAction$StreamDrainer - Job 2: Map: 52 Reduce: 4 Cumulative CPU: 1314.04 sec HDFS Read: 3278902794 HDFS Write: 435711878
SUCCESS
2014-11-21 13:16:40,382 INFO [Thread-2] AbstractBaseAction$StreamDrainer - Job 3: Map: 959 Reduce: 70 Cumulative CPU: 55831.24 sec HDFS Read: 69728403753 HDFS Write: 457377
8468 SUCCESS
2014-11-21 13:16:40,382 INFO [Thread-2] AbstractBaseAction$StreamDrainer - Job 4: Map: 154 Reduce: 1 Cumulative CPU: 1809.45 sec HDFS Read: 483233734 HDFS Write: 25999761 S
UCCESS
2014-11-21 13:16:40,382 INFO [Thread-2] AbstractBaseAction$StreamDrainer - Job 5: Map: 84 Reduce: 6 Cumulative CPU: 2466.01 sec HDFS Read: 5618486947 HDFS Write: 1649704865
SUCCESS
2014-11-21 13:16:40,382 INFO [Thread-2] AbstractBaseAction$StreamDrainer - Job 6: Map: 450 Reduce: 55 Cumulative CPU: 22239.14 sec HDFS Read: 54635746333 HDFS Write: 728231
5124 SUCCESS
2014-11-21 13:16:40,382 INFO [Thread-2] AbstractBaseAction$StreamDrainer - Job 7: Map: 359 Reduce: 27 Cumulative CPU: 14699.06 sec HDFS Read: 26702083597 HDFS Write: 236403
3090 SUCCESS
2014-11-21 13:16:40,382 INFO [Thread-2] AbstractBaseAction$StreamDrainer - Job 8: Map: 83 Reduce: 6 Cumulative CPU: 2410.03 sec HDFS Read: 5514015601 HDFS Write: 628742985
SUCCESS
2014-11-21 13:16:40,382 INFO [Thread-2] AbstractBaseAction$StreamDrainer - Job 9: Map: 84 Reduce: 6 Cumulative CPU: 2200.0 sec HDFS Read: 5618486947 HDFS Write: 853325331 S
UCCESS
2014-11-21 13:16:40,383 INFO [Thread-2] AbstractBaseAction$StreamDrainer - Job 10: Map: 365 Reduce: 27 Cumulative CPU: 13274.58 sec HDFS Read: 27143622450 HDFS Write: 29912
35104 SUCCESS
2014-11-21 13:16:40,383 INFO [Thread-2] AbstractBaseAction$StreamDrainer - Job 11: Map: 959 Reduce: 70 Cumulative CPU: 55145.65 sec HDFS Read: 69728403753 HDFS Write: 43358
51625 SUCCESS
2014-11-21 13:16:40,383 INFO [Thread-2] AbstractBaseAction$StreamDrainer - Job 12: Map: 172 Reduce: 1 Cumulative CPU: 1561.64 sec HDFS Read: 945050606 HDFS Write: 40445679
SUCCESS
2014-11-21 13:16:40,383 INFO [Thread-2] AbstractBaseAction$StreamDrainer - Job 13: Map: 260 Reduce: 9 Cumulative CPU: 5965.67 sec HDFS Read: 8639664681 HDFS Write: 38277094
9 SUCCESS
2014-11-21 13:16:40,383 INFO [Thread-2] AbstractBaseAction$StreamDrainer - Job 14: Map: 212 Reduce: 15 Cumulative CPU: 7573.68 sec HDFS Read: 14849298755 HDFS Write: 109518
4574 SUCCESS
2014-11-21 13:16:40,383 INFO [Thread-2] AbstractBaseAction$StreamDrainer - Job 15: Map: 16 Reduce: 2 Cumulative CPU: 981.37 sec HDFS Read: 1660614485 HDFS Write: 795259370
SUCCESS
2014-11-21 13:16:40,383 INFO [Thread-2] AbstractBaseAction$StreamDrainer - Job 16: Map: 27 Reduce: 3 Cumulative CPU: 1357.34 sec HDFS Read: 2991246795 HDFS Write: 238860030
1 SUCCESS
2014-11-21 13:16:40,383 INFO [Thread-2] AbstractBaseAction$StreamDrainer - Job 17: Map: 70 Reduce: 5 Cumulative CPU: 2138.48 sec HDFS Read: 4573808778 HDFS Write: 290957162
1 SUCCESS
2014-11-21 13:16:40,383 INFO [Thread-2] AbstractBaseAction$StreamDrainer - Job 18: Map: 12 Reduce: 1 Cumulative CPU: 360.1 sec HDFS Read: 853346317 HDFS Write: 705875733 SU
CCESS
三、Driver階段OOM解決方案：
原因找到了，是因爲掃描的表的分區太多，上到3千到6千個分區，這樣在對計劃進行序列化時，僅僅是路徑對象Path就會耗去大半Driver，如果Driver設置的heap太小，甚至都會OOM。
解決思路：
1. 減少分區數量，將歷史數據做成一張整合表，做成增量數據表，這樣分區就很少了。
2. 調大Hive CLI Driver的heap size, 默認是256MB，調節成512MB或者更大。
具體做法是在bin/hive bin/hive-config裏可以找到啓動CLI的JVM OPTIONS。
這裏我們設置
export HADOOP_HEAPSIZE=512
我的解決方法是，雙管齊下。
即做成了整合，方便使用，又調節了Hive CLI Driver的heap size，保證線上的運行穩定。
四、總結：
寫在結尾，今天只顧折騰這個問題了，遇到這種問題：
一是HiveQL的寫法上，儘量少的掃描同一張表，並且儘量少的掃描分區。掃太多，一是job數多，慢，二是耗費網絡資源，慢。
二是Hive的參數調優和JVM的參數調優，儘量在每個階段，選擇合適的jvm max heap size來應對OOM的問題。
——EOF——
原創文章，轉載請註明出自：http://blog.csdn.net/oopsoom/article/details/41356251