最近寫程序的時候要用到多文件輸出。但是在網上找了很多資料,要麼是老版本的方法,要麼就是新版本的方法。網上很多說0.20.203.0版本的Hadoop中有MultipleOutputs類——確實有,只不過是老版的。坑啊。而他們用的MultipleOutputs是在org.apache.hadoop.mapreduce.lib.output裏面的,但我筆者的Hadoop中偏偏就沒有這個類。取去了一個最新版的Hadoop,才發現了這個類。最後,在網上超找了半天,才發現,從0.21開始,org.apache.hadoop.mapreduce.lib.output中才有MultipleOutputs類。所以,只有自己重寫了- -。
又找了半天,終於找到自己想要的了。由於筆者還沒有正式學習Java,所以,還有很多地方不理解,但是大致能看懂,如有即使不周到或者沒有解釋的地方,請各位見諒。同時,筆者修改了一些地方以適應自己的程序。
首先是輸出格式的類,也就是job.setOutputFormatClass(……)參數列表中的類:
public class MoreFileOutputFormat extends Multiple<Text,Text>
{
@Override
protected String generateFileNameForKeyValue(Text key, Text value,Configuration conf)
{
return "Your name";
}
}
這裏,繼承Multiple類後必須重寫generateFileNameForKeyValue()方法,這個方法返回的字符串作爲輸出文件的文件名。內容有各位自己根據需要編寫。同時,key和value的值也根據自己的需要更換。
接下來是Multiple模板類的代碼:
import java.io.DataOutputStream;
import java.io.IOException;
import java.util.HashMap;
import java.util.Iterator;
import org.apache.hadoop.conf.Configuration;
import org.apache.hadoop.fs.FSDataOutputStream;
import org.apache.hadoop.fs.Path;
import org.apache.hadoop.io.Writable;
import org.apache.hadoop.io.WritableComparable;
import org.apache.hadoop.io.compress.CompressionCodec;
import org.apache.hadoop.io.compress.GzipCodec;
import org.apache.hadoop.mapreduce.OutputCommitter;
import org.apache.hadoop.mapreduce.RecordWriter;
import org.apache.hadoop.mapreduce.TaskAttemptContext;
import org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter;
import org.apache.hadoop.mapreduce.lib.output.FileOutputFormat;
import org.apache.hadoop.util.ReflectionUtils;
public abstract class Multiple<K extends WritableComparable<?>, V extends Writable>
extends FileOutputFormat<K, V>
{
// 接口類,需要在調用程序中實現generateFileNameForKeyValue來獲取文件名
private MultiRecordWriter writer = null;
public RecordWriter<K, V> getRecordWriter(TaskAttemptContext job)
throws IOException, InterruptedException
{
if (writer == null)
{
writer = new MultiRecordWriter(job, getTaskOutputPath(job));
}
return writer;
}
/**
* get task output path
*
* @param conf
* @return
* @throws IOException
*/
private Path getTaskOutputPath(TaskAttemptContext conf) throws IOException
{
Path workPath = null;
OutputCommitter committer = super.getOutputCommitter(conf);
if (committer instanceof FileOutputCommitter)
{
workPath = ((FileOutputCommitter) committer).getWorkPath();
}
else
{
Path outputPath = super.getOutputPath(conf);
if (outputPath == null)
{
throw new IOException("Undefined job output-path");
}
workPath = outputPath;
}
return workPath;
}
//繼承後重寫以獲得文件名
protected abstract String generateFileNameForKeyValue(K key, V value,Configuration conf);
//實現記錄寫入器RecordWriter類 (內部類)
public class MultiRecordWriter extends RecordWriter<K, V>
{
/** RecordWriter的緩存 */
private HashMap<String, RecordWriter<K, V>> recordWriters = null;
private TaskAttemptContext job = null;
/** 輸出目錄 */
private Path workPath = null;
public MultiRecordWriter(TaskAttemptContext job, Path workPath)
{
super();
this.job = job;
this.workPath = workPath;
recordWriters = new HashMap<String, RecordWriter<K, V>>();
}
@Override
public void close(TaskAttemptContext context) throws IOException,
InterruptedException
{
Iterator<RecordWriter<K, V>> values = this.recordWriters.values().iterator();
while (values.hasNext())
{
values.next().close(context);
}
this.recordWriters.clear();
}
@Override
public void write(K key, V value) throws IOException,
InterruptedException
{
// 得到輸出文件名
String baseName = generateFileNameForKeyValue(key, value,job.getConfiguration());
// 如果recordWriters裏沒有文件名,那麼就建立。否則就直接寫值。
RecordWriter<K, V> rw = this.recordWriters.get(baseName);
if (rw == null)
{
rw = getBaseRecordWriter(job, baseName);
this.recordWriters.put(baseName, rw);
}
rw.write(key, value);
}
// ${mapred.out.dir}/_temporary/_${taskid}/${nameWithExtension}
private RecordWriter<K, V> getBaseRecordWriter(TaskAttemptContext job,
String baseName) throws IOException, InterruptedException
{
Configuration conf = job.getConfiguration();
// 查看是否使用解碼器
boolean isCompressed = getCompressOutput(job);
RecordWriter<K, V> recordWriter = null;
if (isCompressed)
{
Class<? extends CompressionCodec> codecClass = getOutputCompressorClass(
job, GzipCodec.class);
CompressionCodec codec = ReflectionUtils.newInstance(codecClass, conf);
Path file = new Path(workPath, baseName + codec.getDefaultExtension());
FSDataOutputStream fileOut = file.getFileSystem(conf).create(file, false);
// 這裏我使用的自定義的OutputFormat
recordWriter = new MyRecordWriter<K, V>(new DataOutputStream(
codec.createOutputStream(fileOut)));
}
else
{
Path file;
System.out.println("workPath = " + workPath + ", basename = " + baseName);
file = new Path(workPath, baseName);
FSDataOutputStream fileOut = file.getFileSystem(conf).create(file, false);
// 這裏我使用的自定義的OutputFormat
recordWriter = new MyRecordWriter<K, V>(fileOut);
}
return recordWriter;
}
}
}
現在來實現Multiple的內部類MultiRecordWriter中的MyRecordWriter類以實現自己想要的輸出方式:
public class MyRecordWriter<K, V> extends RecordWriter<K,V>
{
private static final String utf8 = "UTF-8";//定義字符編碼格式
protected DataOutputStream out;
public MyRecordWriter(DataOutputStream out)
{
this.out = out;
}
private void writeObject(Object o) throws IOException
{
if (o instanceof Text)
{
Text to = (Text) o;
out.write(to.getBytes(), 0, to.getLength());
}
else
{
//輸出成字節流。如果不是文本類的,請更改此處
out.write(o.toString().getBytes(utf8));
}
}
/**
* 將mapreduce的key,value以自定義格式寫入到輸出流中
*/
public synchronized void write(K key, V value) throws IOException
{
writeObject(value);
}
public synchronized void close(TaskAttemptContext context) throws IOException
{
out.close();
}
}
這個類中還有其它集中方法,不過筆者不需要那些方法,所以把它們都刪除了,但最初的文件也刪除了- -,所以現在找不到了。請大家見諒。
現在,只需在main()或者run()函數中將job的輸出格式設置成MoreFileOutputFormat類就行了,如下:
job.setOutputFormatClass(MoreFileOutputFormatClass);
詳情請參考一下網址: