分区太多引起的内存溢出

执行一个hive sql时报了一个下面的错误,从错误堆栈上来看,是在SQL编译、解析、优化过程中出的错,还没有提交到YARN上执行。

从Utilities.getPartitionDesc这句来看是优化的时候,在读取分区信息时出现了内存溢出。所以想到是不是扫描的表,分区是不是太多了。

果然SQL中的一个表有36万个分区,这种分区太多的情况应该避免的。

 Exception in thread "main" java.lang.OutOfMemoryError: Java heap space
 	at java.util.Arrays.copyOf(Arrays.java:3332)
 	at java.lang.AbstractStringBuilder.expandCapacity(AbstractStringBuilder.java:137)
 	at java.lang.AbstractStringBuilder.ensureCapacityInternal(AbstractStringBuilder.java:121)
 	at java.lang.AbstractStringBuilder.append(AbstractStringBuilder.java:421)
 	at java.lang.StringBuilder.append(StringBuilder.java:136)
 	at org.apache.hadoop.hive.metastore.MetaStoreUtils.getDDLFromFieldSchema(MetaStoreUtils.java:871)
 	at org.apache.hadoop.hive.metastore.MetaStoreUtils.getSchema(MetaStoreUtils.java:1088)
 	at org.apache.hadoop.hive.metastore.MetaStoreUtils.getTableMetadata(MetaStoreUtils.java:881)
 	at org.apache.hadoop.hive.ql.metadata.Table.getMetadata(Table.java:236)
 	at org.apache.hadoop.hive.ql.exec.Utilities.getTableDesc(Utilities.java:1351)
 	at org.apache.hadoop.hive.ql.plan.PartitionDesc.<init>(PartitionDesc.java:86)
 	at org.apache.hadoop.hive.ql.exec.Utilities.getPartitionDesc(Utilities.java:1368)
 	at org.apache.hadoop.hive.ql.optimizer.GenMapRedUtils.setMapWork(GenMapRedUtils.java:684)
 	at org.apache.hadoop.hive.ql.optimizer.GenMapRedUtils.setTaskPlan(GenMapRedUtils.java:463)
 	at org.apache.hadoop.hive.ql.optimizer.GenMapRedUtils.setTaskPlan(GenMapRedUtils.java:441)
 	at org.apache.hadoop.hive.ql.optimizer.GenMapRedUtils.initPlan(GenMapRedUtils.java:170)
 	at org.apache.hadoop.hive.ql.optimizer.GenMRRedSink1.process(GenMRRedSink1.java:81)
 	at org.apache.hadoop.hive.ql.lib.DefaultRuleDispatcher.dispatch(DefaultRuleDispatcher.java:90)
 	at org.apache.hadoop.hive.ql.lib.DefaultGraphWalker.dispatchAndReturn(DefaultGraphWalker.java:95)
 	at org.apache.hadoop.hive.ql.parse.GenMapRedWalker.walk(GenMapRedWalker.java:54)
 	at org.apache.hadoop.hive.ql.parse.GenMapRedWalker.walk(GenMapRedWalker.java:65)
 	at org.apache.hadoop.hive.ql.parse.GenMapRedWalker.walk(GenMapRedWalker.java:65)
 	at org.apache.hadoop.hive.ql.parse.GenMapRedWalker.walk(GenMapRedWalker.java:65)
 	at org.apache.hadoop.hive.ql.parse.GenMapRedWalker.walk(GenMapRedWalker.java:65)
 	at org.apache.hadoop.hive.ql.lib.DefaultGraphWalker.startWalking(DefaultGraphWalker.java:110)
 	at org.apache.hadoop.hive.ql.parse.MapReduceCompiler.generateTaskTree(MapReduceCompiler.java:322)
 	at org.apache.hadoop.hive.ql.parse.TaskCompiler.compile(TaskCompiler.java:205)
 	at org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.analyzeInternal(SemanticAnalyzer.java:10524)
 	at org.apache.hadoop.hive.ql.parse.CalcitePlanner.analyzeInternal(CalcitePlanner.java:216)
 	at org.apache.hadoop.hive.ql.parse.BaseSemanticAnalyzer.analyze(BaseSemanticAnalyzer.java:237)
 	at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:465)
 	at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:321)
 jobId:58690e4949e84814965f0787b3b8ac0b执行hive sql失败
 java.lang.InterruptedException: 执行失败:执行hive sql失败
 	at com.yixin.bdp.execute.executor.JobExecutor.getCmd(JobExecutor.java:165)
 	at com.yixin.bdp.execute.executor.JobExecutor.run(JobExecutor.java:46)
 	at com.yixin.bdp.app.App.main(App.java:14)
發表評論
所有評論
還沒有人評論,想成為第一個評論的人麼? 請在上方評論欄輸入並且點擊發布.
相關文章