win7 系統eclipse環境下測試 運行hadoop 的 wordcount mapreduce。

上篇介紹了在linux下測試運行 hadoop 的wordcount 例子後，就想着怎麼在eclipse 下編寫mapreduce函數，鏈接hadoop集羣計算呢。
linux下測試運行 hadoop 的wordcount 參考：https://mp.csdn.net/mdeditor/84143774#
linux 部署hadoop 集羣參考：https://mp.csdn.net/mdeditor/84073712#

1 下載eclipse的hadoop插件 hadoop2x-eclipse-plugin-2.6.0
https://download.csdn.net/download/qq_22830285/10792412
下載之後解壓，將relase 目錄下的hadoop-eclipse-plugin-2.6.0.jar 複製到eclipse的 plugin 目錄下、

2.重起運行eclipse，打開菜單Window->ShowView->Other,顯示如

3、new hadoop lacation 配置elcipse 與hadoop 的鏈接。填完之後，點擊finish.
/4、保存完配置之後，可以看到project explorer ，新增了一個 DFS location.
/5、如果出現下面錯誤的話，在系統環境變量添加HADOOP_USER_NAME=root環境變量，或者win系統的用戶名改爲root,又或者修改hadoop 的hdfs-site文件中添加以下內容，關閉權限檢查 ，即解決了上述問題。

<property>
        <name>dfs.permissions</name>
        <value>false</value>
 </property>

/6、新建立項目map/reduce 項目
創建 map 函數，WordCountMap 類如下

import java.io.IOException;
import java.util.StringTokenizer;

import org.apache.hadoop.io.IntWritable;
import org.apache.hadoop.io.LongWritable;
import org.apache.hadoop.io.Text;
import org.apache.hadoop.mapreduce.Mapper;

public class WordCountMap extends Mapper<LongWritable, Text, Text, IntWritable> {
	  private final IntWritable one = new IntWritable(1);
      private Text word = new Text();
      
      public void map(LongWritable key, Text value, Context context)
              throws IOException, InterruptedException {
          String line = value.toString();
          StringTokenizer token = new StringTokenizer(line);
          while (token.hasMoreTokens()) {
              word.set(token.nextToken());
              context.write(word, one);
          }
      }
}

創建reduce 函數，WordCountReduce

import java.io.IOException;

import org.apache.hadoop.io.IntWritable;
import org.apache.hadoop.io.Text;
import org.apache.hadoop.mapreduce.Reducer;

public class WordCountReduce extends Reducer<Text, IntWritable, Text, IntWritable> {
	  public void reduce(Text key, Iterable<IntWritable> values, Context context) throws IOException, InterruptedException {
          int sum = 0;
          for (IntWritable val : values) {
              sum += val.get();
          }
          context.write(key, new IntWritable(sum));
      }
}

創建wordcount mian ，WordCountTest

import org.apache.hadoop.fs.Path;
import org.apache.hadoop.io.IntWritable;
import org.apache.hadoop.io.Text;
import org.apache.hadoop.mapreduce.Job;
import org.apache.hadoop.mapreduce.lib.input.FileInputFormat;
import org.apache.hadoop.mapreduce.lib.input.TextInputFormat;
import org.apache.hadoop.mapreduce.lib.output.FileOutputFormat;
import org.apache.hadoop.mapreduce.lib.output.TextOutputFormat;

public class WordCountTest {
    @SuppressWarnings("deprecation")
    public static void main(String[] args) throws Exception{
       // Configuration conf = new Configuration();
        Job job = new Job();
        job.setJarByClass(WordCountTest.class);
        job.setJobName("wordcount");
        job.setOutputKeyClass(Text.class);
        job.setOutputValueClass(IntWritable.class);
        job.setMapperClass(WordCountMap.class); 
        job.setReducerClass(WordCountReduce.class);
        job.setInputFormatClass(TextInputFormat.class);
        job.setOutputFormatClass(TextOutputFormat.class);
        //輸入文件路徑，我的a.txt,b.txt 文件放在hdfs系統下的 user/root/input 目錄下
        FileInputFormat.addInputPath(job, new Path("hdfs://192.168.80.130:9000/user/root/input"));
       //計算結果輸出文件路徑，記住，此路徑不能存在，否則會報錯
        FileOutputFormat.setOutputPath(job, new Path("hdfs://192.168.80.130:9000/user/root/out3"));
        job.waitForCompletion(true)
    }
}

然後運行，WordCountTest 類。
run as -->run on hadoop
/7、也許，看不到控制檯的日誌，那麼將hadoop 的log4j.properties文件複製工程src 目錄下。

/8.運行之後DFS相應的目錄下看到輸出結果。