yarn 集羣部署，遇到的有關問題小結

優良自學吧提供yarn 集羣部署，遇到的有關問題小結,yarn 集羣部署，遇到的問題小結版本信息： hadoop 2.3.0 hive 0.11.0 1. Application Master 無法訪問點擊application mater 鏈接，出現 http 500 錯誤

yarn 集羣部署，遇到的問題小結

版本信息： hadoop 2.3.0 hive 0.11.0

1. Application Master 無法訪問

點擊application mater 鏈接，出現 http 500 錯誤，java.lang.Connect.exception：

問題是由於設定web ui時，50030 端口對應的ip地址爲0.0.0.0，導致application master 鏈接無法定位。

解決辦法:

yarn-site.xml 文件

<description>The address of the RM web application.</description>

<name>yarn.resourcemanager.webapp.address</name>

<value>xxxxxxxxxx:50030</value>

</property>

這是2.3.0 的裏面的一個bug 1811 ，2.4.0已經修復

2. History UI 無法訪問和 container 打不開

點擊 Tracking URL:History無法訪問

問題是 history service 沒有啓動

解決辦法:

配置：選擇（xxxxxxxxxx: 作爲history sever）

<name>yarn.log-aggregation-enable</name>

</property>

<name>mapreduce.jobhistory.address</name>

<value>xxxxxxxxxx::10020</value>

</property>

<name>mapreduce.jobhistory.webapp.address</name>

<value>xxxxxxxxxx:19888</value>

</property>

sbin/mr-jobhistory-daemon.sh

start historyserver

相關鏈接:http://www.iteblog.com/archives/936

3 yarn 平臺的優化

設置虛擬cpu的個數

<name>yarn.nodemanager.resource.cpu-vcores</name>

</property>

設置使用的內存

<name>yarn.nodemanager.resource.memory-mb</name>

<description>the amount of memory on the NodeManager in GB</description>

</property>

設置每個任務最大使用的內存

<name>yarn.scheduler.maximum-allocation-mb</name>

<source>yarn-default.xml</source>

</property>

4 運行任務提示： Found interface org.apache.hadoop.mapreduce.Counter, but class was expected

修改pom，重新install

<groupId>org.apache.hadoop</groupId>

<artifactId>hadoop-common</artifactId>

</dependency>

<groupId>org.apache.hadoop</groupId>

<artifactId>hadoop-mapreduce-client-core</artifactId>

</dependency>

<groupId>org.apache.mrunit</groupId>

<artifactId>mrunit</artifactId>

<classifier>hadoop2</classifier>

</dependency>

jdk 換成1.7

5 運行任務提示shuffle內存溢出Java heap space

2014-05-14 16:44:22,010 FATAL [IPC Server handler 4 on 44508] org.apache.hadoop.mapred.TaskAttemptListenerImpl: Task: attempt_1400048775904_0006_r_000004_0 - exited : org.apache.hadoop.mapreduce.task.reduce.Shuffle$ShuffleError: error in shuffle in fetcher#3

at org.apache.hadoop.mapreduce.task.reduce.Shuffle.run(Shuffle.java:134)

at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:376)

at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:168)

at java.security.AccessController.doPrivileged(Native Method)

at javax.security.auth.Subject.doAs(Subject.java:415)

at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1548)

at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:163)

Caused by: java.lang.OutOfMemoryError: Java heap space

at org.apache.hadoop.io.BoundedByteArrayOutputStream.<init>(BoundedByteArrayOutputStream.java:56)

at org.apache.hadoop.io.BoundedByteArrayOutputStream.<init>(BoundedByteArrayOutputStream.java:46)

at org.apache.hadoop.mapreduce.task.reduce.InMemoryMapOutput.<init>(InMemoryMapOutput.java:63)

at org.apache.hadoop.mapreduce.task.reduce.MergeManagerImpl.unconditionalReserve(MergeManagerImpl.java:297)

at org.apache.hadoop.mapreduce.task.reduce.MergeManagerImpl.reserve(MergeManagerImpl.java:287)

at org.apache.hadoop.mapreduce.task.reduce.Fetcher.copyMapOutput(Fetcher.java:411)

at org.apache.hadoop.mapreduce.task.reduce.Fetcher.copyFromHost(Fetcher.java:341)

at org.apache.hadoop.mapreduce.task.reduce.Fetcher.run(Fetcher.java:165)

來源： <http:/xxxxxxxxxx:19888/jobhistory/logs/ST-L09-05-back-tj-yarn15:8034/container_1400048775904_0006_01_000001/job_1400048775904_0006/hadoop/syslog/?start=0>

解決方法：

調低mapreduce.reduce.shuffle.memory.limit.percent的值默認爲0.25 現在調成0.10

參考：

http://www.sqlparty.com/yarn%E5%9C%A8shuffle%E9%98%B6%E6%AE%B5%E5%86%85%E5%AD%98%E4%B8%8D%E8%B6%B3%E9%97%AE%E9%A2%98error-in-shuffle-in-fetcher/

6 reduce 任務的log 中間發現：

2014-05-14 17:51:21,835 WARN [Readahead Thread #2] org.apache.hadoop.io.ReadaheadPool: Failed readahead on ifile

EINVAL: Invalid argument

at org.apache.hadoop.io.nativeio.NativeIO$POSIX.posix_fadvise(Native Method)

at org.apache.hadoop.io.nativeio.NativeIO$POSIX.posixFadviseIfPossible(NativeIO.java:263)

at org.apache.hadoop.io.nativeio.NativeIO$POSIX$CacheManipulator.posixFadviseIfPossible(NativeIO.java:142)

at org.apache.hadoop.io.ReadaheadPool$ReadaheadRequestImpl.run(ReadaheadPool.java:206)

at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)

at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)

at java.lang.Thread.run(Thread.java:745)

來源： <http://xxxxxxxxxx:8042/node/containerlogs/container_1400060792764_0001_01_000726/hadoop/syslog/?start=-4096>

ps：錯誤沒有再現，暫無解決方法

7 hive 任務

java.lang.InstantiationException: org.antlr.runtime.CommonToken

Continuing ...

java.lang.RuntimeException: failed to evaluate: <unbound>=Class.new();

參考:https://issues.apache.org/jira/browse/HIVE-4222s

8 hive 任務自動把join裝換mapjoin時內存溢出，解決方法：關閉自動裝換，11前的版本默認值爲false，後面的爲true;

在任務腳本里面加上：set hive.auto.convert.join=false;

或者在hive-site.xml 配上爲false；

出錯日誌：

SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory]

2014-05-15 02:40:58 Starting to launch local task to process map join; maximum memory = 1011351552

2014-05-15 02:41:00 Processing rows: 200000 Hashtable size: 199999 Memory usage: 110092544 rate: 0.109

2014-05-15 02:41:01 Processing rows: 300000 Hashtable size: 299999 Memory usage: 229345424 rate: 0.227

2014-05-15 02:41:01 Processing rows: 400000 Hashtable size: 399999 Memory usage: 170296368 rate: 0.168

2014-05-15 02:41:01 Processing rows: 500000 Hashtable size: 499999 Memory usage: 285961568 rate: 0.283

2014-05-15 02:41:02 Processing rows: 600000 Hashtable size: 599999 Memory usage: 408727616 rate: 0.404

2014-05-15 02:41:02 Processing rows: 700000 Hashtable size: 699999 Memory usage: 333867920 rate: 0.33

2014-05-15 02:41:02 Processing rows: 800000 Hashtable size: 799999 Memory usage: 459541208 rate: 0.454

2014-05-15 02:41:03 Processing rows: 900000 Hashtable size: 899999 Memory usage: 391524456 rate: 0.387

2014-05-15 02:41:03 Processing rows: 1000000 Hashtable size: 999999 Memory usage: 514140152 rate: 0.508

2014-05-15 02:41:03 Processing rows: 1029052 Hashtable size: 1029052 Memory usage: 546126888 rate: 0.54

2014-05-15 02:41:03 Dump the hashtable into file: file:/tmp/hadoop/hive_2014-05-15_14-40-53_413_3806680380261480764/-local-10002/HashTable-Stage-4/MapJoin-mapfile01--.hashtable

2014-05-15 02:41:06 Upload 1 File to: file:/tmp/hadoop/hive_2014-05-15_14-40-53_413_3806680380261480764/-local-10002/HashTable-Stage-4/MapJoin-mapfile01--.hashtable File size: 68300588

2014-05-15 02:41:06 End of local task; Time Taken: 8.301 sec.

Execution completed successfully

Mapred Local Task Succeeded . Convert the Join into MapJoin

Launching Job 2 out of 2

log出錯日誌：

2014-05-15 13:52:54,007 FATAL [main] org.apache.hadoop.mapred.YarnChild: Error running child : java.lang.OutOfMemoryError: Java heap space

at java.io.ObjectInputStream$HandleTable.grow(ObjectInputStream.java:3465)

at java.io.ObjectInputStream$HandleTable.assign(ObjectInputStream.java:3271)

at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1789)

at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1350)

at java.io.ObjectInputStream.readObject(ObjectInputStream.java:370)

at java.util.HashMap.readObject(HashMap.java:1183)

at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)

at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)

at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)

at java.lang.reflect.Method.invoke(Method.java:606)

at java.io.ObjectStreamClass.invokeReadObject(ObjectStreamClass.java:1017)

at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1893)

at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1798)

at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1350)

at java.io.ObjectInputStream.readObject(ObjectInputStream.java:370)

at org.apache.hadoop.hive.ql.exec.persistence.HashMapWrapper.initilizePersistentHash(HashMapWrapper.java:128)

at org.apache.hadoop.hive.ql.exec.MapJoinOperator.loadHashTable(MapJoinOperator.java:194)

at org.apache.hadoop.hive.ql.exec.MapJoinOperator.cleanUpInputFileChangedOp(MapJoinOperator.java:212)

at org.apache.hadoop.hive.ql.exec.Operator.cleanUpInputFileChanged(Operator.java:1377)

at org.apache.hadoop.hive.ql.exec.Operator.cleanUpInputFileChanged(Operator.java:1381)

來源： <http://xxxxxxxxxx:19888/jobhistory/logs/ST-L09-10-back-tj-yarn21:8034/container_1400064445468_0013_01_000002/attempt_1400064445468_0013_m_000000_0/hadoop/syslog/?start=0>

9 hive運行時提示： failed to evaluate: <unbound>=Class.new(); ，升級到0.13.0

參考https://issues.apache.org/jira/browse/HIVE-4222

https://issues.apache.org/jira/browse/HIVE-3739

SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory]OKTime taken: 2.28 secondsjava.lang.InstantiationException: org.antlr.runtime.CommonTokenContinuing ...java.lang.RuntimeException: failed to evaluate: <unbound>=Class.new();Continuing ...java.lang.InstantiationException: org.antlr.runtime.CommonTokenContinuing ...java.lang.RuntimeException: failed to evaluate: <unbound>=Class.new();Continuing ...java.lang.InstantiationException: org.antlr.runtime.CommonTokenContinuing ...java.lang.RuntimeException: failed to evaluate: <unbound>=Class.new();Continuing ...java.lang.InstantiationException: org.antlr.runtime.CommonTokenContinuing ...java.lang.RuntimeException: failed to evaluate: <unbound>=Class.new();Continuing ...java.lang.InstantiationException: org.antlr.runtime.CommonTokenContinuing ...

這個應該升級後能解決，不過不知道爲什麼我升級12.0 和13.0 ，一運行就報錯fileNotfundHIVE_PLANxxxxxxxxx 。ps （參考11）應該是我配置有問題，暫無解決方法。

10 hive 創建表或者數據庫的時候 Couldnt obtain a new sequence (unique id) : You have an error in your SQL syntax

解決方法：這個是因爲hive元數據庫的名字是yarn-hive, sql中中劃線是關鍵詞，所以sql錯誤。把數據庫名去掉中劃線，問題解決。

錯誤日誌：

FAILED: Error in metadata: MetaException(message:javax.jdo.JDOException: Couldnt obtain a new sequence (unique id) : You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near '-hive.`SEQUENCE_TABLE` WHERE `SEQUENCE_NAME`='org.apache.hadoop.hive.metastore.m' at line 1

at org.datanucleus.api.jdo.NucleusJDOHelper.getJDOExceptionForNucleusException(NucleusJDOHelper.java:596)

at org.datanucleus.api.jdo.JDOPersistenceManager.jdoMakePersistent(JDOPersistenceManager.java:732)

at org.datanucleus.api.jdo.JDOPersistenceManager.makePersistent(JDOPersistenceManager.java:752)

at org.apache.hadoop.hive.metastore.ObjectStore.createTable(ObjectStore.java:643)

at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)

at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)

at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)

at java.lang.reflect.Method.invoke(Method.java:606)

at org.apache.hadoop.hive.metastore.RetryingRawStore.invoke(RetryingRawStore.java:111)

at com.sun.proxy.$Proxy14.createTable(Unknown Source)

at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.create_table_core(HiveMetaStore.java:1070)

at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.create_table_with_environment_context(HiveMetaStore.java:1103)

at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)

at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)

at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)

at java.lang.reflect.Method.invoke(Method.java:606)

at org.apache.hadoop.hive.metastore.RetryingHMSHandler.invoke(RetryingHMSHandler.java:103)

at com.sun.proxy.$Proxy15.create_table_with_environment_context(Unknown Source)

at org.apache.hadoop.hive.metastore.HiveMetaStoreClient.createTable(HiveMetaStoreClient.java:466)

at org.apache.hadoop.hive.metastore.HiveMetaStoreClient.createTable(HiveMetaStoreClient.java:455)

at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)

at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)

at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)

at java.lang.reflect.Method.invoke(Method.java:606)

at org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.invoke(RetryingMetaStoreClient.java:74)

at com.sun.proxy.$Proxy16.createTable(Unknown Source)

at org.apache.hadoop.hive.ql.metadata.Hive.createTable(Hive.java:597)

at org.apache.hadoop.hive.ql.exec.DDLTask.createTable(DDLTask.java:3777)

at org.apache.hadoop.hive.ql.exec.DDLTask.execute(DDLTask.java:256)

at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:144)

at org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:57)

at org.apache.hadoop.hive.ql.Driver.launchTask(Driver.java:1362)

at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:1146)

at org.apache.hadoop.hive.ql.Driver.run(Driver.java:952)

at shark.SharkCliDriver.processCmd(SharkCliDriver.scala:338)

at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:413)

at shark.SharkCliDriver$.main(SharkCliDriver.scala:235)

at shark.SharkCliDriver.main(SharkCliDriver.scala)

NestedThrowablesStackTrace:

com.mysql.jdbc.exceptions.jdbc4.MySQLSyntaxErrorException: You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near '-hive.`SEQUENCE_TABLE` WHERE `SEQUENCE_NAME`='org.apache.hadoop.hive.metastore.m' at line 1

at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)

at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57)

at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)

at java.lang.reflect.Constructor.newInstance(Constructor.java:526)

at com.mysql.jdbc.Util.handleNewInstance(Util.java:406)

at com.mysql.jdbc.Util.getInstance(Util.java:381)

at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:1030)

at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:956)

at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:3558)

at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:3490)

at com.mysql.jdbc.MysqlIO.sendCommand(MysqlIO.java:1959)

at com.mysql.jdbc.MysqlIO.sqlQueryDirect(MysqlIO.java:2109)

at com.mysql.jdbc.ConnectionImpl.execSQL(ConnectionImpl.java:2648)

at com.mysql.jdbc.PreparedStatement.executeInternal(PreparedStatement.java:2077)

at com.mysql.jdbc.PreparedStatement.executeQuery(PreparedStatement.java:2228)

at org.apache.commons.dbcp.DelegatingPreparedStatement.executeQuery(DelegatingPreparedStatement.java:96)

at org.datanucleus.store.rdbms.ParamLoggingPreparedStatement.executeQuery(ParamLoggingPreparedStatement.java:381)

at org.datanucleus.store.rdbms.SQLController.executeStatementQuery(SQLController.java:504)

at org.datanucleus.store.rdbms.valuegenerator.SequenceTable.getNextVal(SequenceTable.java:197)

at org.datanucleus.store.rdbms.valuegenerator.TableGenerator.reserveBlock(TableGenerator.java:190)

at org.datanucleus.store.valuegenerator.AbstractGenerator.reserveBlock(AbstractGenerator.java:305)

at org.datanucleus.store.rdbms.valuegenerator.AbstractRDBMSGenerator.obtainGenerationBlock(AbstractRDBMSGenerator.java:170)

at org.datanucleus.store.valuegenerator.AbstractGenerator.obtainGenerationBlock(AbstractGenerator.java:197)

at org.datanucleus.store.valuegenerator.AbstractGenerator.next(AbstractGenerator.java:105)

at org.datanucleus.store.rdbms.RDBMSStoreManager.getStrategyValueForGenerator(RDBMSStoreManager.java:2019)

at org.datanucleus.store.AbstractStoreManager.getStrategyValue(AbstractStoreManager.java:1385)

at org.datanucleus.ExecutionContextImpl.newObjectId(ExecutionContextImpl.java:3727)

at org.datanucleus.state.JDOStateManager.setIdentity(JDOStateManager.java:2574)

at org.datanucleus.state.JDOStateManager.initialiseForPersistentNew(JDOStateManager.java:526)

at org.datanucleus.state.ObjectProviderFactoryImpl.newForPersistentNew(ObjectProviderFactoryImpl.java:202)

at org.datanucleus.ExecutionContextImpl.newObjectProviderForPersistentNew(ExecutionContextImpl.java:1326)

at org.datanucleus.ExecutionContextImpl.persistObjectInternal(ExecutionContextImpl.java:2123)

at org.datanucleus.ExecutionContextImpl.persistObjectWork(ExecutionContextImpl.java:1972)

at org.datanucleus.ExecutionContextImpl.persistObject(ExecutionContextImpl.java:1820)

at org.datanucleus.ExecutionContextThreadedImpl.persistObject(ExecutionContextThreadedImpl.java:217)

at org.datanucleus.api.jdo.JDOPersistenceManager.jdoMakePersistent(JDOPersistenceManager.java:727)

at org.datanucleus.api.jdo.JDOPersistenceManager.makePersistent(JDOPersistenceManager.java:752)

at org.apache.hadoop.hive.metastore.ObjectStore.createTable(ObjectStore.java:643)

11 安裝hive 12 和13 後，運行任務報錯提示：FileNotFoundException: HIVE_PLAN

解決方法：可能是hive一個bug，也可能那裏配置錯了，待解決

錯誤日誌

2014-05-16 10:27:07,896 INFO [main] org.apache.hadoop.mapred.MapTask: Processing split: Paths:/user/hive/warehouse/game_predata.db/game_login_log/dt=0000-00-00/000000_0:201326592+60792998,/user/hive/warehouse/game_predata.db/game_login_log/dt=0000-00-00/000001_0_copy_1:201326592+58503492,/user/hive/warehouse/game_predata.db/game_login_log/dt=0000-00-00/000001_0_copy_2:67108864+67108864,/user/hive/warehouse/game_predata.db/game_login_log/dt=0000-00-00/000001_0_copy_2:134217728+67108864,/user/hive/warehouse/game_predata.db/game_login_log/dt=0000-00-00/000002_0_copy_1:67108864+67108864InputFormatClass: org.apache.hadoop.mapred.TextInputFormat

2014-05-16 10:27:07,954 WARN [main] org.apache.hadoop.mapred.YarnChild: Exception running child : java.lang.RuntimeException: java.io.FileNotFoundException: HIVE_PLAN14c8af69-0156-4633-9273-6a812eb91a4c (沒有那個文件或目錄)

at org.apache.hadoop.hive.ql.exec.Utilities.getMapRedWork(Utilities.java:230)

at org.apache.hadoop.hive.ql.io.HiveInputFormat.init(HiveInputFormat.java:255)

at org.apache.hadoop.hive.ql.io.HiveInputFormat.pushProjectionsAndFilters(HiveInputFormat.java:381)

at org.apache.hadoop.hive.ql.io.HiveInputFormat.pushProjectionsAndFilters(HiveInputFormat.java:374)

at org.apache.hadoop.hive.ql.io.CombineHiveInputFormat.getRecordReader(CombineHiveInputFormat.java:540)

at org.apache.hadoop.mapred.MapTask$TrackedRecordReader.<init>(MapTask.java:168)

at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:409)

at org.apache.hadoop.mapred.MapTask.run(MapTask.java:342)

at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:168)

at java.security.AccessController.doPrivileged(Native Method)

at javax.security.auth.Subject.doAs(Subject.java:415)

at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1548)

at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:163)

Caused by: java.io.FileNotFoundException: HIVE_PLAN14c8af69-0156-4633-9273-6a812eb91a4c (沒有那個文件或目錄)

at java.io.FileInputStream.open(Native Method)

at java.io.FileInputStream.<init>(FileInputStream.java:146)

at java.io.FileInputStream.<init>(FileInputStream.java:101)

at org.apache.hadoop.hive.ql.exec.Utilities.getMapRedWork(Utilities.java:221)

... 12 more

2014-05-16 10:27:07,957 INFO [main] org.apache.hadoop.mapred.Task: Runnning cleanup for the task

來源： <http://sxxxxxxxxxx:19888/jobhistory/logs/ST-L10-10-back-tj-yarn10:8034/container_1400136017046_0026_01_000030/attempt_1400136017046_0026_m_000000_0/hadoop>

12java.lang.OutOfMemoryError: GC overhead limit exceeded

分析：這個是JDK6新添的錯誤類型。是發生在GC佔用大量時間爲釋放很小空間的時候發生的，是一種保護機制。解決方案是，關閉該功能，可以添加JVM的啓動參數來限制使用內存： -XX:-UseGCOverheadLimit
添加位置是：mapred-site.xml 裏新增項：mapred.child.java.opts 內容：-XX:-UseGCOverheadLimit

來源： <http://www.cnblogs.com/niocai/archive/2012/07/31/2616252.html>

參考14

13hive hive 0.10.0爲了執行效率考慮，簡單的查詢，就是隻是select，不帶count,sum,group by這樣的，都不走map/reduce，直接讀取hdfs文件進行filter過濾。這樣做的好處就是不新開mr任務，執行效率要提高不少，但是不好的地方就是用戶界面不友好，有時候數據量大還是要等很長時間，但是又沒有任何返回。

改這個很簡單，在hive-site.xml裏面有個配置參數叫

hive.fetch.task.conversion

將這個參數設置爲more，簡單查詢就不走map/reduce了，設置爲minimal，就任何簡單select都會走map/reduce。

來源： <http://slaytanic.blog.51cto.com/2057708/1170431>

參考14

14 運行mr 任務的時候提示：

錯誤日誌

Container [pid=30486,containerID=container_1400229396615_0011_01_000012] is running beyond physical memory limits. Current usage: 1.0 GB of 1 GB physical memory used; 1.7 GB of 2.1 GB virtual memory used. Killing container. Dump of the process-tree for container_1400229396615_0011_01_000012 : |- PID PPID PGRPID SESSID CMD_NAME USER_MODE_TIME(MILLIS) SYSTEM_TIME(MILLIS) VMEM_USAGE(BYTES) RSSMEM_USAGE(PAGES) FULL_CMD_LINE |- 30501 30486 30486 30486 (java) 3924 322 1720471552 262096 /opt/jdk1.7.0_55/bin/java -Djava.net.preferIPv4Stack=true -Dhadoop.metrics.log.level=WARN -Xmx1024m -XX:-UseGCOverheadLimit -Djava.io.tmpdir=/home/nodemanager/local/usercache/hadoop/appcache/application_1400229396615_0011/container_1400229396615_0011_01_000012/tmp -Dlog4j.configuration=container-log4j.properties -Dyarn.app.container.log.dir=/home/hadoop/logs/nodemanager/logs/application_1400229396615_0011/container_1400229396615_0011_01_000012 -Dyarn.app.container.log.filesize=0 -Dhadoop.root.logger=INFO,CLA org.apache.hadoop.mapred.YarnChild 30.30.30.39 47925 attempt_1400229396615_0011_m_000000_0 12 |- 30486 12812 30486 30486 (bash) 0 0 108642304 302 /bin/bash -c /opt/jdk1.7.0_55/bin/java -Djava.net.preferIPv4Stack=true -Dhadoop.metrics.log.level=WARN -Xmx1024m -XX:-UseGCOverheadLimit -Djava.io.tmpdir=/home/nodemanager/local/usercache/hadoop/appcache/application_1400229396615_0011/container_1400229396615_0011_01_000012/tmp -Dlog4j.configuration=container-log4j.properties -Dyarn.app.container.log.dir=/home/hadoop/logs/nodemanager/logs/application_1400229396615_0011/container_1400229396615_0011_01_000012 -Dyarn.app.container.log.filesize=0 -Dhadoop.root.logger=INFO,CLA org.apache.hadoop.mapred.YarnChild 30.30.30.39 47925 attempt_1400229396615_0011_m_000000_0 12 1>/home/hadoop/logs/nodemanager/logs/application_1400229396615_0011/container_1400229396615_0011_01_000012/stdout 2>/home/hadoop/logs/nodemanager/logs/application_1400229396615_0011/container_1400229396615_0011_01_000012/stderr Container killed on request. Exit code is 143 Container exited with a non-zero exit code 143

來源： <http://xxxxxxxxxx:50030/proxy/application_1400229396615_0011/mapreduce/attempts/job_1400229396615_0011/m/FAILED>

解決方法：

下面的參數是關於mapreduce任務運行時的內存設置，如果有的任務需要可單獨配置，就統一配置了。如果有container被kill 可以適當調高

mapreduce.map.memory.mb map任務的最大內存

mapreduce.map.java.opts -Xmx1024M map任務jvm的參數

mapreduce.reduce.memory.mb reduce任務的最大內存

mapreduce.reduce.java.opts -Xmx2560M reduce任務jvm的參數

mapreduce.task.io.sort.mb 512 Higher memory-limit while sorting data for efficiency.

摘自：http://hadoop.apache.org/docs/current/hadoop-project-dist/hadoop-common/ClusterSetup.html#Configuring_the_Hadoop_Daemons_in_Non-Secure_Mode

關閉內存檢測進程：

是在搞不清楚問什麼有的任務就物理內存200多MB ，虛擬內存就飆到2.7G了，估計內存檢測進程有問題，而且我有的任務是需要大內存的，爲了進度，索性關了，一下子解決所有內存問題。

yarn.nodemanager.pmem-check-enabled false

yarn.nodemanager.vmem-check-enabled false

15 yarn 的webUI 有關的調整：

1 cluser 頁面 application的starttime 和finishtime 都是 UTC格式，改成 +8區時間也就是北京時間。

./share/hadoop/yarn/hadoop-yarn-common-2.3.0.jar 裏面的webapps.static.yarn.dt.plugins.js

或者源碼包裏面：/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/resources/webapps/static/yarn.dt.plugins.js

添加代碼：

Date.prototype.Format = function (fmt) { //author: meizz 
    var o = {
        "M+": this.getMonth() + 1, //月份 
        "d+": this.getDate(), //日 
        "h+": this.getHours(), //小時 
        "m+": this.getMinutes(), //分 
        "s+": this.getSeconds(), //秒 
        "q+": Math.floor((this.getMonth() + 3) / 3), //季度 
        "S": this.getMilliseconds() //毫秒 
    };
    if (/(y+)/.test(fmt)) fmt = fmt.replace(RegExp.$1, (this.getFullYear() + "").substr(4 - RegExp.$1.length));
    for (var k in o)
    if (new RegExp("(" + k + ")").test(fmt)) fmt = fmt.replace(RegExp.$1, (RegExp.$1.length == 1) ? (o[k]) : (("00" + o[k]).substr(("" + o[k]).length)));
    return fmt;
};

同時按下面修改下的代碼

function renderHadoopDate(data, type, full) 
{ if (type === 'display' || type === 'filter') { if(data === '0') { return "N/A"; } 
return new Date(parseInt(data)).Format("yyyy-MM-dd hh:mm:ss"); }

16 MR1的任務用到DistributedCache 的任務遷移到MR2上出錯。原來我裏面使用文件名區分不同的緩存文件，MR2裏面分發文件以後只保留的文件名如：

application_xxxxxxx/container_14xxxx/part-m-00000
application_xxxxxxx/container_14xxxx/part-m-00001
application_xxxxxxx/container_14xxxx/00000_0

解決方法：每個緩存文件添加符號鏈接，鏈接爲父級名字+文件名

DistributedCache.addCacheFile(new URI(path.toString() + "#"+ path.getParent().getName() + "_" + path.getName()),
configuration);

這樣就會生成帶有文件名的緩存文件

iteye_19679

發佈了396 篇原創文章 · 獲贊 3 · 訪問量 2萬+

私信關注

yarn 集羣部署，遇到的有關問題小結

yarn 集羣部署，遇到的有關問題小結

優良自學吧提供yarn 集羣部署，遇到的有關問題小結,yarn 集羣部署，遇到的問題小結版本信息： hadoop 2.3.0 hive 0.11.0 1. Application Master 無法訪問點擊application mater 鏈接，出現 http 500 錯誤

通過HPA+CronHPA組合應對業務複雜彈性伸縮場景

Git URL 改變後，本地鏈接修改

storm-on-yarn

yum安裝maven2

MySQL記錄存在則更新，不存在則插入

yarn 集羣部署，遇到的有關問題小結

https://yachay.unat.edu.pe/blog/index.php?comment_area=format_blog&comment_component=blog&comment_co

linux以太網驅動總結

yarn 集羣部署，遇到的有關問題小結

yarn 集羣部署，遇到的有關問題小結優良自學吧提供yarn 集羣部署，遇到的有關問題小結,yarn 集羣部署，遇到的問題小結 版本信息： hadoop 2.3.0 hive 0.11.0 1. Application Master 無法訪問 點擊application mater 鏈接，出現 http 500 錯誤

yarn 集羣部署，遇到的有關問題小結

優良自學吧提供yarn 集羣部署，遇到的有關問題小結,yarn 集羣部署，遇到的問題小結版本信息： hadoop 2.3.0 hive 0.11.0 1. Application Master 無法訪問點擊application mater 鏈接，出現 http 500 錯誤