mapreduce書寫遇到的問題總結

1.mapreduce本地運行：

1）導入jar包，找到hadoop的安裝目錄HADOOP_HOME就可以找到這些jar包

HADOOP_HOME/share/hadoop/common目錄下的hadoop-common-2.7.1.jar和haoop-nfs-2.7.1.jar；
HADOOP_HOME/share/hadoop/common/lib目錄下的所有JAR包；
HADOOP_HOME/share/hadoop/hdfs目錄下的haoop-hdfs-2.7.1.jar和haoop-hdfs-nfs-2.7.1.jar；
HADOOP_HOME/share/hadoop/hdfs/lib目錄下的所有JAR包；
HADOOP_HOME/share/hadoop/mapreduce除hadoop-mapreduce-examples-2.7.1.jar之外的jar包；
HADOOP_HOME/share/hadoop/mapreduce/lib/所有jar包。
2）配置文件

將HADOOP_HOME/etc/hadoop目錄下的log4j.properties，core-site.xml，hdfs-site.xml文件放到工程目錄下的resources文件夾（如果沒有resources則在idea中新建並將該文件夾設置爲Sources Root）。

3）idea配置application

 4）啓動即可。

遇到的問題：

（1）發現hdfs連接的地址不對。不是yarn上的資源。可以看mapred-site.xml中的這個參數

local代表是本地。yarn代表是yarn資源器

（2）The ownership on the staging directory /tmp/hadoop-yarnis not as expectedThe directory mus

java.io.IOException: The ownership on the staging directory /tmp/hadoop-yarn/staging/root/.staging is not as expected. It is owned by hive. The directory must be owned by the submitter root or by root
	at org.apache.hadoop.mapreduce.JobSubmissionFiles.getStagingDir(JobSubmissionFiles.java:120)
	at org.apache.hadoop.mapreduce.JobSubmitter.submitJobInternal(JobSubmitter.java:146)
	at org.apache.hadoop.mapreduce.Job$11.run(Job.java:1341)
	at org.apache.hadoop.mapreduce.Job$11.run(Job.java:1338)
	at java.security.AccessController.doPrivileged(Native Method)
	at javax.security.auth.Subject.doAs(Subject.java:421)

執行hadoop fs -chown -R root:root /tmp即可

（3）自定義的類找不到

Error: java.lang.RuntimeException: java.lang.ClassNotFoundException: Class com.learning.mapreduce.WordMapper not found
at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:2195)
at org.apache.hadoop.mapreduce.task.JobContextImpl.getMapperClass(JobContextImpl.java:186)
at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:745)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:341)
at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:164)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1657)
at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:158)
Caused by: java.lang.ClassNotFoundException: Class com.learning.mapreduce.WordMapper not found
at org.apache.hadoop.conf.Configuration.getClassByName(Configuration.java:2101)
at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:2193)

自己定義的類運行的時候顯示找不到可以通過以下解決

conf.set("mapreduce.job.jar", "D:/work/workspace/12580/mr12580/target/chador-mapreduce-1.0-SNAPSHOT-jar-with-dependencies.jar");

2.idea運行調試遠程數據

  1）服務器開啓一個監聽使用jvm自帶的那種監聽

$export HADOOP_CLIENT_OPTS="$HADOOP_CLIENT_OPTS -agentlib:jdwp=transport=dt_socket,server=y,suspend=y,address=8000"

2）在hadoop的安裝路徑的bin目錄下修改hadoop的啓動文件

並添加HADOOP_CLIENT_OPTS參數值爲

agentlib:jdwp=transport=dt_socket,server=y,suspend=y,address=8000

3）啓動一個remote