1 概述
1 SequenceFile是Hadoop爲例存儲二進制的<k,v>而設計的一種平面文件。2 SequenceFile的key和value是writable或者writable子類。
3 SequenceFile的存儲不按照key排序,內部類Writer提供了append方法。
4 SequenceFile作爲一個容器,可以將小文件打包到SequenceFile,高效對小文件進行存儲和處理。
2 壓縮類型
根據CompressionType的不同,有如下壓縮類型NONE:不壓縮。每個記錄有key長度、value長度、key、value組成,長度字段分別爲4字節。
RECORD: 記錄壓縮。結構與NONE非常類似,用定義在頭部的編碼器壓縮value,key不壓縮。
BLOCK:塊壓縮。一次壓縮多條記錄,當記錄字節數達到一個閾值則天際到塊,io.seqfile.compress.blocksize控制。格式爲:記錄數,鍵長度,鍵,值長度,值。
分別對應Writer:
Writer : Uncompressed recordsRecordCompressWriter : Record-compressed files, only compress values
BlockCompressWriter : Block-compressed files, both keys & values are collected in 'blocks' separately and compressed. The size of the 'block' is configurable
3 特點
優點
1 支持基於record和block的壓縮
2 支持splittable,能夠爲Mapreduce作爲輸入分片
3 修改簡單,只需要按照業務邏輯修改,不要考慮具體存儲格式
缺點
合併後的文件不易查看。hadoop fs -cat看不到,hadoop fs -text可以看到。
4 代碼
import java.io.IOException;
import org.apache.hadoop.conf.Configuration;
import org.apache.hadoop.fs.FileSystem;
import org.apache.hadoop.fs.Path;
import org.apache.hadoop.io.IOUtils;
import org.apache.hadoop.io.IntWritable;
import org.apache.hadoop.io.SequenceFile;
import org.apache.hadoop.io.Text;
import org.apache.hadoop.io.Writable;
import org.apache.hadoop.util.ReflectionUtils;
public class SequenceTest {
public static final String Output_path = "hdfs://192.x.x.x:9000/a.txt";
private static final String[] DATA = { "a", "b", "c", };
@SuppressWarnings("deprecation")
public static void write(String pathStr) throws IOException {
Configuration conf = new Configuration();
Path path = new Path(pathStr);
FileSystem fs = path.getFileSystem(conf);
SequenceFile.Writer writer = SequenceFile.createWriter(fs, conf, path,
Text.class, IntWritable.class);
Text key = new Text();
IntWritable value = new IntWritable();
for (int i = 0; i < DATA.length; i++) {
key.set(DATA[i]);
value.set(i);
System.out.printf("[%s]\t%s\t%s\n", writer.getLength(), key, value);
writer.append(key, value);
}
IOUtils.closeStream(writer);
}
@SuppressWarnings("deprecation")
public static void read(String pathStr) throws IOException {
Configuration conf = new Configuration();
Path path = new Path(pathStr);
FileSystem fs = path.getFileSystem(conf);
SequenceFile.Reader reader = new SequenceFile.Reader(fs, new Path(
pathStr), conf);
Writable key = (Writable) ReflectionUtils.newInstance(
reader.getKeyClass(), conf);
Writable value = (Writable) ReflectionUtils.newInstance(
reader.getValueClass(), conf);
while (reader.next(key, value)) {
System.out.printf("%s\t%s\n", key, value);
}
IOUtils.closeStream(reader);
}
public static void main(String[] args) throws IOException {
write(Output_path);
read(Output_path);
}
}