mapreduce中map方法一次讀取整個文件

最近有一個項目，需要將爬蟲獲取的衆多網頁解析出來，並將內容插入hbase，考慮採用mapreduce來實現。由於一個html文件最大不過幾M，所以一個文件將會交給一個map處理，mapreduce中最常見的是按行讀取文本文件，而我們需要的是一次讀取整個文件內容，然後在map方法中用jsoup解析內容。現將實現一次讀取整個文件內容的代碼貼出來。

1、定義一個FileInputFormat類

public class WholeFileInputFormat extends FileInputFormat<Text,Text>{

    @Override
    public RecordReader<Text, Text> createRecordReader(InputSplit arg0, TaskAttemptContext arg1) throws IOException,
            InterruptedException {
        // TODO Auto-generated method stub
        RecordReader<Text,Text> recordReader = new WholeFileRecordReader();
        return recordReader;
    }

}

2、自定義RecordReader方法

public class WholeFileRecordReader extends RecordReader<Text,Text>{

    private FileSplit fileSplit;
    private JobContext jobContext;
    private Text currentKey = new Text();
    private Text currentValue = new Text();
    private boolean finishConverting = false;
    @Override
    public void close() throws IOException {
        // TODO Auto-generated method stub

    }

    @Override
    public Text getCurrentKey() throws IOException, InterruptedException {
        // TODO Auto-generated method stub
        return currentKey;
    }

    @Override
    public Text getCurrentValue() throws IOException,
            InterruptedException {
        // TODO Auto-generated method stub
        return currentValue;
    }

    @Override
    public float getProgress() throws IOException, InterruptedException {
        // TODO Auto-generated method stub
        float progress = 0;
        if(finishConverting){
            progress = 1;
        }
        return progress;
    }

    @Override
    public void initialize(InputSplit arg0, TaskAttemptContext arg1)
            throws IOException, InterruptedException {
        this.fileSplit = (FileSplit) arg0;
        this.jobContext = arg1;
        String filename = fileSplit.getPath().getName();
        this.currentKey = new Text(filename);
    }

    @Override
    public boolean nextKeyValue() throws IOException, InterruptedException {
        // TODO Auto-generated method stub
        if(!finishConverting){
            int len = (int)fileSplit.getLength();
//          byte[] content = new byte[len];
            Path file = fileSplit.getPath();
            FileSystem fs = file.getFileSystem(jobContext.getConfiguration());
            FSDataInputStream in = fs.open(file);
            BufferedReader br = new BufferedReader(new InputStreamReader(in,"gbk"));
//          BufferedReader br = new BufferedReader(new InputStreamReader(in,"utf-8"));
            String line="";
            String total="";
            while((line= br.readLine())!= null){
                total =total+line+"\n";
            }
            br.close();
            in.close();
            fs.close();
            currentValue = new Text(total);
            finishConverting = true;
            return true;
        }
        return false;
    }

}

實現RecordReader接口，最核心的就是處理好迭代多行文本的內容的邏輯，每次迭代通過調用nextKeyValue()方法來判斷是否還有可讀的文本行，直接設置當前的Key和Value，分別在方法getCurrentKey()和getCurrentValue()中返回對應的值。

更多信息請參考