Hadoop & Hadoop Streaming 自定義輸出格式

    在用Hadoop處理大量的日誌文件的時候，有時候會將錯誤的或者不符合要求的日誌輸出到另外一個目錄，以備後來進行查驗，這裏給出個簡單的例子，並簡單說明下如何在Hadoop Streaming中使用這種方法將錯誤格式的日誌輸出到自定義的路徑。

    例子中類MultiFilesOutput繼承自MultipleTextOutputFormat<Text, Text>，可以自己定義不同的keyType，然後輸出到不同的目錄下。

package MultiFormats; import org.apache.hadoop.fs.Path; import org.apache.hadoop.io.Text; import org.apache.hadoop.mapred.lib.MultipleTextOutputFormat; /** * * @author xyl * */ public class MultiFilesOutput extends MultipleTextOutputFormat<Text, Text> { /** * Use the key as part of the path for the final output results * now we just define three types * err_log: the log format we can't parse * err_oth: the other log format * pvnormal: the right log format for output pv result * uvnormal: the right log format for output uvresult */ @Override protected String generateFileNameForKeyValue(Text key, Text value, String leaf) { String keyType = new String(key.toString()); if(keyType.equals("err_log")) { return new Path("err_log",leaf).toString(); } if(keyType.equals("err_oth")) { return new Path("err_oth",leaf).toString(); } /* * return new Path("normal",leaf).toString(); */ if(keyType.equals("pvnormal")) { return new Path("pvnormal",leaf).toString(); } return new Path("uvnormal", leaf).toString(); } /** * When actually writing the data, discard the key since it is already in * the file path. */ @Override protected Text generateActualKey(Text key, Text value) { return null; } } </text></text>

 在Hadoop Streaming，使用擴展的jar文件格式如下：

INPUT=/user/flume/${PATH} OUTPUT=/user/dplog/errlog/${DATE} #JOBNAME: displayed mapreduce jobname, better add job owner name JOBNAME="${DATE}-pvTest""(xyl)" ###################################################################### # Check if output directory aready exists # -------------------------------------------------------------------- #if $(/opt/hadoop/bin/hadoop fs -test -d ${OUTPUT}) #then # echo "Output Directory" $OUTPUT "already exists." # Other action code can be added at here # exit 100 #else # echo "Output Directory will be created!" #fi /opt/hadoop/bin/hadoop fs -rmr ${OUTPUT} ###################################################################### # Main hadoop streaming process # -------------------------------------------------------------------- /opt/hadoop/bin/hadoop jar ${STREAMING_JAR} \ -libjars /opt/home/hadoop/xyl/Jars/CustomOutputFormats.jar \ -Dmapred.job.name=${JOBNAME} \ -Dmapred.reduce.tasks=1 \ -input ${INPUT} \ -output ${OUTPUT} \ -mapper mapper.py \ -reducer reducer.py \ -file /opt/home/hadoop/xyl/mappers/mapper.py \ -file /opt/home/hadoop/xyl/reducers/reducer.py \ -outputformat MultiFormats.MultiFilesOutput \ ;

其中CustomOutputFormats.jar 爲生成的jar文件名稱，MultiFormats.MultiFilesOutput對應於package name和類名。