hadoop wordcount 示例

本示例假設已經基本配置好hadoop
如何配置hadoop不是本文的重點
如果在終端中輸入hadoop沒有報錯就可以繼續往下看

1 查看classpath

hadoop classpath

2 設定環境變量classpath

在終端輸入vim ~/.bashrc
在文件中添加export classpath=你的classpath
重啓終端

3 創建WordCount.java

    package org.myorg;  
    import java.io.IOException;
    import java.util.*;

    import org.apache.hadoop.fs.Path;
    import org.apache.hadoop.conf.*;
    import org.apache.hadoop.io.*;
    import org.apache.hadoop.mapred.*;
    import org.apache.hadoop.util.*;

    public class WordCount {

       public static class Map extends MapReduceBase implements Mapper<LongWritable, Text, Text, IntWritable> {
         private final static IntWritable one = new IntWritable(1);
         private Text word = new Text();

         public void map(LongWritable key, Text value, OutputCollector<Text, IntWritable> output, Reporter reporter) throws IOException {
           String line = value.toString();
           StringTokenizer tokenizer = new StringTokenizer(line);
           while (tokenizer.hasMoreTokens()) {
             word.set(tokenizer.nextToken());
             output.collect(word, one);
           }
         }
       }

       public static class Reduce extends MapReduceBase implements Reducer<Text, IntWritable, Text, IntWritable> {
         public void reduce(Text key, Iterator<IntWritable> values, OutputCollector<Text, IntWritable> output, Reporter reporter) throws IOException {
           int sum = 0;
           while (values.hasNext()) {
             sum += values.next().get();
           }
           output.collect(key, new IntWritable(sum));
         }
       }

       public static void main(String[] args) throws Exception {
         JobConf conf = new JobConf(WordCount.class);
         conf.setJobName("wordcount");

         conf.setOutputKeyClass(Text.class);
         conf.setOutputValueClass(IntWritable.class);

         conf.setMapperClass(Map.class);
         conf.setCombinerClass(Reduce.class);
         conf.setReducerClass(Reduce.class);

         conf.setInputFormat(TextInputFormat.class);
         conf.setOutputFormat(TextOutputFormat.class);

         FileInputFormat.setInputPaths(conf, new Path(args[0]));
         FileOutputFormat.setOutputPath(conf, new Path(args[1]));

         JobClient.runJob(conf);
       }
    }

4 生成class文件

首先在終端輸入mkdir wordcount_classes
現在的目錄大概是這樣的
WordCount.java(我們現在在這個目錄)
—–wordcount_classes
然後執行

javac -classpath ${classpath} -d wordcount_classes WordCount.java
// 把WordCount.java生成class放到wordcount_classes目錄下

5 打包.class生成.jar

在終端輸入

jar -cvf wc.jar wordcount_classes/ .
//將wordcount_classes目錄下所有文件打包成jar文件，jar名字不要與當前存在的目錄重名即可

現在的目錄
wordcount.java（我們在這裏
wc.jar
—-wordcount_classes

6 生成一些測試數據樣例

在終端中輸入

mkdir sample
cd sample
mkdir input
cd input
touch file1
echo helloword >> file1
cd ../..

現在的目錄
wordcount.java（我們在這裏
wc.jar
—-wordcount_classes
—-sample
——–input
————file1

7用hadoop運行jar文件

終端輸入
hadoop jar wc.jar org.myorg.WordCount sample/input sample/output
解釋一下
wc.jar是我們的包文件 org.myorg.WordCount是我們的class名字 後面兩個目錄是輸入輸出

（截圖中目錄名字與本示例不一樣，我在本文中將wordcount目錄換爲sample因爲這個名字更能讓人知道目錄的作用)

8 查看結果

終端輸入
cat sample/output/part-0000

(具體數字不太一樣是因爲我在示例中使用的數據和我真實使用的不一樣)