Hadoop I/O

- checksum, CRC-32C, for every 512 bits, 

• write, last datanode of the pipeline verifies checksum
• read, block verification on client read
• rawlocalfilesystem, to disable checksum

- compression, (default is DefaultCodec, DEFLATE)

• splittable compression for mapreduce, bzip2. also sequence file, avro file or parquet file
• CompressionCodec interface for all compression types

CompressionCodec compc = (CompressionCodec)ReflectionUtils.newInstance(
				Class.forName(compressClassname), cfg);

CompressionCodecFactory ccf = new CompressionCodecFactory(cfg);
CompressionCodec cc = ccf.getCodec(pp); // by filename extension

• mapreduce output compression, compression type for sequence file output, RECORD or BLOCK

FileOutputFormat.setCompressOutput(job, true);
FileOutputFormat.setOutputCompressorClass(job, GzipCodec.class);

• map output can also be compressed to save network traffic

Configuration conf = new Configuration();
conf.setBoolean(Job.MAP_OUTPUT_COMPRESS, true);
conf.setClass(Job.MAP_OUTPUT_COMPRESS_CODEC, GzipCodec.class,
CompressionCodec.class);

- serialisation for inter process communication and persistent storage

• serialisation format, needs to be compact, fast, extensible and interoperable
• one of the formats, Writable

public interface Writable {
    void write(DataOutput out) throws IOException;
    void readFields(DataInput in) throws IOException;
}

• Text is 

1. UTF-8,
2.  t.length is length of underlying bytes . 
3. mutable, getbytes.length may be different from t.getLength

• Custom Writable, WritableComparator, RawComparator, 

static {
WritableComparator.define(TextPair.class, new Comparator());
}

• Serialization Framework, Serialization interface, WritableSerialization and Avro Serialization are supported out of the box

 - File-based data structure

• Sequence File, binary key-value pairs, also as a container for multiple small files.

record and block compression.

• Map File (index for fast lookup. map entries need to be added in order)

index is fraction of keys to be loaded to memory

main data file contains all map entries in sorted key order. 

• sequence file, map file and avro file are row-oriented format.  column-oriented can skip unwanted columns.