一、Flink的分佈通信模型
flink做爲一個分佈式的應用，它基於的通信當然是分佈式的通信框架。在Flink中，可以分爲兩種通信方式，一種是通過網絡的傳輸通信，一種是基於本地的數據交換通信。網絡通信主要是用來連接節點間的通信（包括客戶端和服務端），本地主要是線程內交換一些數據。
本地主要是基於一系列的Oprator來實現的，這個在前面也已經看到過。這次主要分析這個。
二、線程間通信
在前面分析了任務和工作的啓動和分發，知道數據流DataSet的整體流轉的框架，現在分析一下本地的數據交換方法：
1、線程內
線程內使用Operator做爲一種通信手段，它也可以用於線程間的通信。當然，在分佈式中，跨線程的通信在上層是不區分的。在前面的學習過程中可以知道，Flink的數據模型其實是分爲數據和操作兩種類型的。Flink的各種工作和任務就圍繞着數據和操作展開不同的工作形式。不過，同一線程內的Operator互相通信有一個優勢，不需要進行序列化。直接可以進行數據的操作，它們通過共享的buffer來交換傳遞數據。比如KeyedStream.java中的函數sum:
public SingleOutputStreamOperator<T> sum(int positionToSum) {
return aggregate(new SumAggregator<>(positionToSum, getType(), getExecutionConfig()));
}
這個經常用於線程內的數據計算。
2、線程間
正如上面所講，線程間有本地和異地線程間，但在上層其實是會被抽象開來的。但是在線程間的通信，就必須得進行序列化了。它們同樣是通過buffer來傳遞數據，不過在緩衝區沒有數據後，需要通過一些方式來得到數據才能進行操作，本地的線程間，在阻塞後，等待新數據的Flush到Buffer，而遠程則需要發起RPC請求得到數據再返回給相關的Buffer。而運行不同的線程間的有：
public <R> SingleOutputStreamOperator<R> flatMap(FlatMapFunction<T, R> flatMapper) {
TypeInformation<R> outType = TypeExtractor.getFlatMapReturnTypes(clean(flatMapper),
getType(), Utils.getCallLocationName(), true);
return transform("Flat Map", outType, new StreamFlatMap<>(clean(flatMapper)));
}
當然，其它的如collection等也是經常用於線程間通信的。
三、內存的管理
Flink的優勢就在於它的數據計算，數據計算中數據交換的內存管理是一個非常難解決的問題，在前面提到過，Flink沒有直接使用JDK的內存管理，而是在其上又進行了一層的抽象，其中有兩個重要的數據結構：
1、MemorySegment
@Internal
public abstract class MemorySegment {
/**
* The unsafe handle for transparent memory copied (heap / off-heap).
*/
@SuppressWarnings("restriction")
protected static final sun.misc.Unsafe UNSAFE = MemoryUtils.UNSAFE;
/**
* The beginning of the byte array contents, relative to the byte array object.
*/
@SuppressWarnings("restriction")
protected static final long BYTE_ARRAY_BASE_OFFSET = UNSAFE.arrayBaseOffset(byte[].class);
/**
* Constant that flags the byte order. Because this is a boolean constant, the JIT compiler can
* use this well to aggressively eliminate the non-applicable code paths.
*/
private static final boolean LITTLE_ENDIAN = (ByteOrder.nativeOrder() == ByteOrder.LITTLE_ENDIAN);
protected final byte[] heapMemory;
/**
* The address to the data, relative to the heap memory byte array. If the heap memory byte
* array is <tt>null</tt>, this becomes an absolute memory address outside the heap.
*/
protected long address;
/**
* The address one byte after the last addressable byte, i.e. <tt>address + size</tt> while the
* segment is not disposed.
*/
protected final long addressLimit;
/**
* The size in bytes of the memory segment.
*/
protected final int size;
/**
* Optional owner of the memory segment.
*/
private final Object owner;
MemorySegment(byte[] buffer, Object owner) {
if (buffer == null) {
throw new NullPointerException("buffer");
}
this.heapMemory = buffer;
this.address = BYTE_ARRAY_BASE_OFFSET;
this.size = buffer.length;
this.addressLimit = this.address + this.size;
this.owner = owner;
}
MemorySegment(long offHeapAddress, int size, Object owner) {
if (offHeapAddress <= 0) {
throw new IllegalArgumentException("negative pointer or size");
}
if (offHeapAddress >= Long.MAX_VALUE - Integer.MAX_VALUE) {
// this is necessary to make sure the collapsed checks are safe against numeric overflows
throw new IllegalArgumentException("Segment initialized with too large address: " + offHeapAddress
+ " ; Max allowed address is " + (Long.MAX_VALUE - Integer.MAX_VALUE - 1));
}
this.heapMemory = null;
this.address = offHeapAddress;
this.addressLimit = this.address + size;
this.size = size;
this.owner = owner;
}
public int size() {
return size;
}
public boolean isFreed() {
return address > addressLimit;
}
public void free() {
// this ensures we can place no more data and trigger
// the checks for the freed segment
address = addressLimit + 1;
}
public boolean isOffHeap() {
return heapMemory == null;
}
public byte[] getArray() {
if (heapMemory != null) {
return heapMemory;
} else {
throw new IllegalStateException("Memory segment does not represent heap memory");
}
}
public long getAddress() {
if (heapMemory == null) {
return address;
} else {
throw new IllegalStateException("Memory segment does not represent off heap memory");
}
}
public abstract ByteBuffer wrap(int offset, int length);
......
}
//子類之一：分配堆內存
@SuppressWarnings("unused")
@Internal
public final class HeapMemorySegment extends MemorySegment {
private byte[] memory;
HeapMemorySegment(byte[] memory) {
this(memory, null);
}
HeapMemorySegment(byte[] memory, Object owner) {
super(Objects.requireNonNull(memory), owner);
this.memory = memory;
}
@Override
public void free() {
super.free();
this.memory = null;
}
@Override
public ByteBuffer wrap(int offset, int length) {
try {
return ByteBuffer.wrap(this.memory, offset, length);
}
catch (NullPointerException e) {
throw new IllegalStateException("segment has been freed");
}
}
}
//子類之二：即可分配堆內存也可分配非堆內存
@Internal
public final class HybridMemorySegment extends MemorySegment {
private final ByteBuffer offHeapBuffer;
HybridMemorySegment(ByteBuffer buffer) {
this(buffer, null);
}
}
這正好是剛剛提到的Flink自己抽象的內存數據結構。它們一般都有一個wrap()的方法，將自己打包成相關的內存的Buffer。在Flink中重點是第二個，這樣可以讓JVM優化掉相關的虛表的查找操作。
2、Buffer
Buffer是個什麼概念呢？在Flink中，數據交換是taskmanager管理task來進行的，爲了提高網絡的利用率，把Records的數據存儲到Buffer中，是一種非常通用的方式。就和實際的網絡通信一樣，在各個環節上，都分佈着一系列的緩衝區。要注意區分下面的兩個BUFFER,一個是JAVA本身的抽象類，一個是FLINK的接口。
//jdk中的buffer
public abstract class Buffer {
static final int SPLITERATOR_CHARACTERISTICS =
Spliterator.SIZED | Spliterator.SUBSIZED | Spliterator.ORDERED;
// Invariants: mark <= position <= limit <= capacity
private int mark = -1;
private int position = 0;
private int limit;
private int capacity;
// Used only by direct buffers
// NOTE: hoisted here for speed in JNI GetDirectBufferAddress
long address;
// Creates a new buffer with the given mark, position, limit, and capacity,
// after checking invariants.
//
Buffer(int mark, int pos, int lim, int cap) { // package-private
if (cap < 0)
throw new IllegalArgumentException("Negative capacity: " + cap);
this.capacity = cap;
limit(lim);
position(pos);
if (mark >= 0) {
if (mark > pos)
throw new IllegalArgumentException("mark > position: ("
+ mark + " > " + pos + ")");
this.mark = mark;
}
}
......
}
//子類--JDK
public abstract class ByteBuffer
extends Buffer
implements Comparable<ByteBuffer>
{
final byte[] hb; // Non-null only for heap buffers
final int offset;
boolean isReadOnly; // Valid only for heap buffers
// Creates a new buffer with the given mark, position, limit, capacity,
// backing array, and array offset
//
ByteBuffer(int mark, int pos, int lim, int cap, // package-private
byte[] hb, int offset)
{
super(mark, pos, lim, cap);
this.hb = hb;
this.offset = offset;
}
// Creates a new buffer with the given mark, position, limit, and capacity
//
ByteBuffer(int mark, int pos, int lim, int cap) { // package-private
this(mark, pos, lim, cap, null, 0);
}
public static ByteBuffer allocateDirect(int capacity) {
return new DirectByteBuffer(capacity);
}
}
//=========================================================================================
//FLINK中的BUFFER
public interface Buffer {
boolean isBuffer();
void tagAsEvent();
@Deprecated
MemorySegment getMemorySegment();
@Deprecated
int getMemorySegmentOffset();
BufferRecycler getRecycler();
void recycleBuffer();
boolean isRecycled();
Buffer retainBuffer();
Buffer readOnlySlice();
Buffer readOnlySlice(int index, int length);
int getMaxCapacity();
int getReaderIndex();
void setReaderIndex(int readerIndex) throws IndexOutOfBoundsException;
int getSizeUnsafe();
int getSize();
void setSize(int writerIndex);
int readableBytes();
ByteBuffer getNioBufferReadable();
ByteBuffer getNioBuffer(int index, int length) throws IndexOutOfBoundsException;
void setAllocator(ByteBufAllocator allocator);
ByteBuf asByteBuf();
}
public interface BufferPoolFactory {
BufferPool createBufferPool(int numRequiredBuffers, int maxUsedBuffers) throws IOException;
BufferPool createBufferPool(int numRequiredBuffers, int maxUsedBuffers, Optional<BufferPoolOwner> owner) throws IOException;
/**
* Destroy callback for updating factory book keeping.
*/
void destroyBufferPool(BufferPool bufferPool) throws IOException;
}
public class NetworkBuffer extends AbstractReferenceCountedByteBuf implements Buffer {
/** The backing {@link MemorySegment} instance. */
private final MemorySegment memorySegment;
/** The recycler for the backing {@link MemorySegment}. */
private final BufferRecycler recycler;
/** Whether this buffer represents a buffer or an event. */
private boolean isBuffer;
/** Allocator for further byte buffers (needed by netty). */
private ByteBufAllocator allocator;
/**
* The current size of the buffer in the range from 0 (inclusive) to the
* size of the backing {@link MemorySegment} (inclusive).
*/
private int currentSize;
/**
* Creates a new buffer instance backed by the given <tt>memorySegment</tt> with <tt>0</tt> for
* the <tt>readerIndex</tt> and <tt>writerIndex</tt>.
*
* @param memorySegment
* backing memory segment (defines {@link #maxCapacity})
* @param recycler
* will be called to recycle this buffer once the reference count is <tt>0</tt>
*/
public NetworkBuffer(MemorySegment memorySegment, BufferRecycler recycler) {
this(memorySegment, recycler, true);
}
......
}
public class NetworkBufferPool implements BufferPoolFactory {
private static final Logger LOG = LoggerFactory.getLogger(NetworkBufferPool.class);
private final int totalNumberOfMemorySegments;
private final int memorySegmentSize;
private final ArrayBlockingQueue<MemorySegment> availableMemorySegments;
private volatile boolean isDestroyed;
// ---- Managed buffer pools ----------------------------------------------
private final Object factoryLock = new Object();
private final Set<LocalBufferPool> allBufferPools = new HashSet<>();
private int numTotalRequiredBuffers;
......
}
看到AbstractReferenceCountedByteBuf沒，這個裏面有一個引用計數器，用來處理內存的GC的。NetworkBufferPool用來生產LocalBufferPool實現內存的池的管理 。
四、數據交換的方式
在Flink中，數據的交換方式主要還是以序列化的方式進行交換，這也是目前分佈系統中主流的數據通信方式。在Flink中主要有以下幾個類：
//記錄序列化器
public interface RecordSerializer<T extends IOReadableWritable> {
/**
* Status of the serialization result.
*/
enum SerializationResult {
PARTIAL_RECORD_MEMORY_SEGMENT_FULL(false, true),
FULL_RECORD_MEMORY_SEGMENT_FULL(true, true),
FULL_RECORD(true, false);
private final boolean isFullRecord;
private final boolean isFullBuffer;
SerializationResult(boolean isFullRecord, boolean isFullBuffer) {
this.isFullRecord = isFullRecord;
this.isFullBuffer = isFullBuffer;
}
/**
* Whether the full record was serialized and completely written to
* a target buffer.
*
* @return <tt>true</tt> if the complete record was written
*/
public boolean isFullRecord() {
return this.isFullRecord;
}
/**
* Whether the target buffer is full after the serialization process.
*
* @return <tt>true</tt> if the target buffer is full
*/
public boolean isFullBuffer() {
return this.isFullBuffer;
}
}
/**
* Starts serializing the given record to an intermediate data buffer.
*
* @param record the record to serialize
*/
void serializeRecord(T record) throws IOException;
/**
* Copies the intermediate data serialization buffer to the given target buffer.
*
* @param bufferBuilder the new target buffer to use
* @return how much information was written to the target buffer and
* whether this buffer is full
*/
SerializationResult copyToBufferBuilder(BufferBuilder bufferBuilder);
/**
* Clears the buffer and checks to decrease the size of intermediate data serialization buffer
* after finishing the whole serialization process including
* {@link #serializeRecord(IOReadableWritable)} and {@link #copyToBufferBuilder(BufferBuilder)}.
*/
void prune();
/**
* Supports copying an intermediate data serialization buffer to multiple target buffers
* by resetting its initial position before each copying.
*/
void reset();
/**
* @return <tt>true</tt> if has some serialized data pending copying to the result {@link BufferBuilder}.
*/
boolean hasSerializedData();
}
//反序列化器
public interface RecordDeserializer<T extends IOReadableWritable> {
/**
* Status of the deserialization result.
*/
enum DeserializationResult {
PARTIAL_RECORD(false, true),
INTERMEDIATE_RECORD_FROM_BUFFER(true, false),
LAST_RECORD_FROM_BUFFER(true, true);
private final boolean isFullRecord;
private final boolean isBufferConsumed;
private DeserializationResult(boolean isFullRecord, boolean isBufferConsumed) {
this.isFullRecord = isFullRecord;
this.isBufferConsumed = isBufferConsumed;
}
public boolean isFullRecord () {
return this.isFullRecord;
}
public boolean isBufferConsumed() {
return this.isBufferConsumed;
}
}
DeserializationResult getNextRecord(T target) throws IOException;
void setNextBuffer(Buffer buffer) throws IOException;
Buffer getCurrentBuffer();
void clear();
boolean hasUnfinishedData();
}
//事件序列化器
public class EventSerializer {
// ------------------------------------------------------------------------
// Constants
// ------------------------------------------------------------------------
private static final int END_OF_PARTITION_EVENT = 0;
private static final int CHECKPOINT_BARRIER_EVENT = 1;
private static final int END_OF_SUPERSTEP_EVENT = 2;
private static final int OTHER_EVENT = 3;
private static final int CANCEL_CHECKPOINT_MARKER_EVENT = 4;
private static final int CHECKPOINT_TYPE_CHECKPOINT = 0;
private static final int CHECKPOINT_TYPE_SAVEPOINT = 1;
// ------------------------------------------------------------------------
// Serialization Logic
// ------------------------------------------------------------------------
public static ByteBuffer toSerializedEvent(AbstractEvent event) throws IOException {
final Class<?> eventClass = event.getClass();
if (eventClass == EndOfPartitionEvent.class) {
return ByteBuffer.wrap(new byte[] { 0, 0, 0, END_OF_PARTITION_EVENT });
}
else if (eventClass == CheckpointBarrier.class) {
return serializeCheckpointBarrier((CheckpointBarrier) event);
}
else if (eventClass == EndOfSuperstepEvent.class) {
return ByteBuffer.wrap(new byte[] { 0, 0, 0, END_OF_SUPERSTEP_EVENT });
}
else if (eventClass == CancelCheckpointMarker.class) {
CancelCheckpointMarker marker = (CancelCheckpointMarker) event;
ByteBuffer buf = ByteBuffer.allocate(12);
buf.putInt(0, CANCEL_CHECKPOINT_MARKER_EVENT);
buf.putLong(4, marker.getCheckpointId());
return buf;
}
else {
try {
final DataOutputSerializer serializer = new DataOutputSerializer(128);
serializer.writeInt(OTHER_EVENT);
serializer.writeUTF(event.getClass().getName());
event.write(serializer);
return serializer.wrapAsByteBuffer();
}
catch (IOException e) {
throw new IOException("Error while serializing event.", e);
}
}
}
......
}
通過這三個數據結構，來組織序列化的過程，在此過程中，需要寫入和讀取，主要有以下幾個類：
//AbstractReader
public abstract class AbstractReader implements ReaderBase {
/** The input gate to read from. */
protected final InputGate inputGate;
/** The task event handler to manage task event subscriptions. */
private final TaskEventHandler taskEventHandler = new TaskEventHandler();
/** Flag indicating whether this reader allows iteration events. */
private boolean isIterative;
/**
* The current number of end of superstep events (reset for each superstep). A superstep is
* finished after an end of superstep event has been received for each input channel.
*/
private int currentNumberOfEndOfSuperstepEvents;
protected AbstractReader(InputGate inputGate) {
this.inputGate = inputGate;
}
@Override
public boolean isFinished() {
return inputGate.isFinished();
}
// ------------------------------------------------------------------------
// Events
// ------------------------------------------------------------------------
@Override
public void registerTaskEventListener(EventListener<TaskEvent> listener, Class<? extends TaskEvent> eventType) {
taskEventHandler.subscribe(listener, eventType);
}
@Override
public void sendTaskEvent(TaskEvent event) throws IOException {
inputGate.sendTaskEvent(event);
}
/**
* Handles the event and returns whether the reader reached an end-of-stream event (either the
* end of the whole stream or the end of an superstep).
*/
protected boolean handleEvent(AbstractEvent event) throws IOException {
final Class<?> eventType = event.getClass();
try {
// ------------------------------------------------------------
// Runtime events
// ------------------------------------------------------------
// This event is also checked at the (single) input gate to release the respective
// channel, at which it was received.
if (eventType == EndOfPartitionEvent.class) {
return true;
}
else if (eventType == EndOfSuperstepEvent.class) {
return incrementEndOfSuperstepEventAndCheck();
}
// ------------------------------------------------------------
// Task events (user)
// ------------------------------------------------------------
else if (event instanceof TaskEvent) {
taskEventHandler.publish((TaskEvent) event);
return false;
}
else {
throw new IllegalStateException("Received unexpected event of type " + eventType + " at reader.");
}
}
catch (Throwable t) {
throw new IOException("Error while handling event of type " + eventType + ": " + t.getMessage(), t);
}
}
public void publish(TaskEvent event){
taskEventHandler.publish(event);
}
// ------------------------------------------------------------------------
// Iterations
// ------------------------------------------------------------------------
@Override
public void setIterativeReader() {
isIterative = true;
}
@Override
public void startNextSuperstep() {
checkState(isIterative, "Tried to start next superstep in a non-iterative reader.");
checkState(currentNumberOfEndOfSuperstepEvents == inputGate.getNumberOfInputChannels(), "Tried to start next superstep before reaching end of previous superstep.");
currentNumberOfEndOfSuperstepEvents = 0;
}
@Override
public boolean hasReachedEndOfSuperstep() {
return isIterative && currentNumberOfEndOfSuperstepEvents == inputGate.getNumberOfInputChannels();
}
private boolean incrementEndOfSuperstepEventAndCheck() {
checkState(isIterative, "Tried to increment superstep count in a non-iterative reader.");
checkState(currentNumberOfEndOfSuperstepEvents + 1 <= inputGate.getNumberOfInputChannels(), "Received too many (" + currentNumberOfEndOfSuperstepEvents + ") end of superstep events.");
return ++currentNumberOfEndOfSuperstepEvents == inputGate.getNumberOfInputChannels();
}
}
//RecordReader：不可變讀取記錄器
public class RecordReader<T extends IOReadableWritable> extends AbstractRecordReader<T> implements Reader<T> {
private final Class<T> recordType;
private T currentRecord;
/**
* Creates a new RecordReader that de-serializes records from the given input gate and
* can spill partial records to disk, if they grow large.
*
* @param inputGate The input gate to read from.
* @param tmpDirectories The temp directories. USed for spilling if the reader concurrently
* reconstructs multiple large records.
*/
public RecordReader(InputGate inputGate, Class<T> recordType, String[] tmpDirectories) {
super(inputGate, tmpDirectories);
this.recordType = recordType;
}
@Override
public boolean hasNext() throws IOException, InterruptedException {
if (currentRecord != null) {
return true;
}
else {
T record = instantiateRecordType();
if (getNextRecord(record)) {
currentRecord = record;
return true;
}
else {
return false;
}
}
}
......
}
//MutableRecordReader：可變讀取記錄器
public class MutableRecordReader<T extends IOReadableWritable> extends AbstractRecordReader<T> implements MutableReader<T> {
/**
* Creates a new MutableRecordReader that de-serializes records from the given input gate and
* can spill partial records to disk, if they grow large.
*
* @param inputGate The input gate to read from.
* @param tmpDirectories The temp directories. USed for spilling if the reader concurrently
* reconstructs multiple large records.
*/
public MutableRecordReader(InputGate inputGate, String[] tmpDirectories) {
super(inputGate, tmpDirectories);
}
@Override
public boolean next(final T target) throws IOException, InterruptedException {
return getNextRecord(target);
}
@Override
public void clearBuffers() {
super.clearBuffers();
}
}
//RecordWriter：記錄寫入器
public class RecordWriter<T extends IOReadableWritable> {
private static final Logger LOG = LoggerFactory.getLogger(RecordWriter.class);
private final ResultPartitionWriter targetPartition;
private final ChannelSelector<T> channelSelector;
private final int numberOfChannels;
private final int[] broadcastChannels;
private final RecordSerializer<T> serializer;
private final Optional<BufferBuilder>[] bufferBuilders;
private final Random rng = new XORShiftRandom();
private Counter numBytesOut = new SimpleCounter();
private Counter numBuffersOut = new SimpleCounter();
private final boolean flushAlways;
/** Default name for teh output flush thread, if no name with a task reference is given. */
private static final String DEFAULT_OUTPUT_FLUSH_THREAD_NAME = "OutputFlusher";
/** The thread that periodically flushes the output, to give an upper latency bound. */
private final Optional<OutputFlusher> outputFlusher;
/** To avoid synchronization overhead on the critical path, best-effort error tracking is enough here.*/
private Throwable flusherException;
public RecordWriter(ResultPartitionWriter writer) {
this(writer, new RoundRobinChannelSelector<T>(), -1, null);
}
public RecordWriter(
ResultPartitionWriter writer,
ChannelSelector<T> channelSelector,
long timeout,
String taskName) {
this.targetPartition = writer;
this.channelSelector = channelSelector;
this.numberOfChannels = writer.getNumberOfSubpartitions();
this.channelSelector.setup(numberOfChannels);
this.serializer = new SpanningRecordSerializer<T>();
this.bufferBuilders = new Optional[numberOfChannels];
this.broadcastChannels = new int[numberOfChannels];
for (int i = 0; i < numberOfChannels; i++) {
broadcastChannels[i] = i;
bufferBuilders[i] = Optional.empty();
}
checkArgument(timeout >= -1);
this.flushAlways = (timeout == 0);
if (timeout == -1 || timeout == 0) {
outputFlusher = Optional.empty();
} else {
String threadName = taskName == null ?
DEFAULT_OUTPUT_FLUSH_THREAD_NAME :
DEFAULT_OUTPUT_FLUSH_THREAD_NAME + " for " + taskName;
outputFlusher = Optional.of(new OutputFlusher(threadName, timeout));
outputFlusher.get().start();
}
}
public void emit(T record) throws IOException, InterruptedException {
checkErroneous();
emit(record, channelSelector.selectChannel(record));
}
/**
* This is used to broadcast Streaming Watermarks in-band with records. This ignores
* the {@link ChannelSelector}.
*/
public void broadcastEmit(T record) throws IOException, InterruptedException {
checkErroneous();
serializer.serializeRecord(record);
boolean pruneAfterCopying = false;
for (int channel : broadcastChannels) {
if (copyFromSerializerToTargetChannel(channel)) {
pruneAfterCopying = true;
}
}
// Make sure we don't hold onto the large intermediate serialization buffer for too long
if (pruneAfterCopying) {
serializer.prune();
}
}
......
}
//ResultPartitionWriter：分區結果寫入器
public interface ResultPartitionWriter {
BufferProvider getBufferProvider();
ResultPartitionID getPartitionId();
int getNumberOfSubpartitions();
int getNumTargetKeyGroups();
void addBufferConsumer(BufferConsumer bufferConsumer, int subpartitionIndex) throws IOException;
/**
* Manually trigger consumption from enqueued {@link BufferConsumer BufferConsumers} in all subpartitions.
*/
void flushAll();
/**
* Manually trigger consumption from enqueued {@link BufferConsumer BufferConsumers} in one specified subpartition.
*/
void flush(int subpartitionIndex);
}
@ThreadSafe
public abstract class AbstractCollectingResultPartitionWriter implements ResultPartitionWriter {
private final BufferProvider bufferProvider;
private final ArrayDeque<BufferConsumer> bufferConsumers = new ArrayDeque<>();
public AbstractCollectingResultPartitionWriter(BufferProvider bufferProvider) {
this.bufferProvider = checkNotNull(bufferProvider);
}
@Override
public BufferProvider getBufferProvider() {
return bufferProvider;
}
@Override
public ResultPartitionID getPartitionId() {
return new ResultPartitionID();
}
@Override
public int getNumberOfSubpartitions() {
return 1;
}
@Override
public int getNumTargetKeyGroups() {
return 1;
}
@Override
public synchronized void addBufferConsumer(BufferConsumer bufferConsumer, int targetChannel) throws IOException {
checkState(targetChannel < getNumberOfSubpartitions());
bufferConsumers.add(bufferConsumer);
processBufferConsumers();
}
private void processBufferConsumers() throws IOException {
while (!bufferConsumers.isEmpty()) {
BufferConsumer bufferConsumer = bufferConsumers.peek();
Buffer buffer = bufferConsumer.build();
try {
deserializeBuffer(buffer);
if (!bufferConsumer.isFinished()) {
break;
}
bufferConsumers.pop().close();
}
finally {
buffer.recycleBuffer();
}
}
}
@Override
public synchronized void flushAll() {
try {
processBufferConsumers();
} catch (IOException e) {
throw new RuntimeException(e);
}
}
@Override
public void flush(int subpartitionIndex) {
flushAll();
}
protected abstract void deserializeBuffer(Buffer buffer) throws IOException;
}
通過這幾個類的互相組合，就可以完成數據的讀寫操作。其實應該畫幾個類圖和流程圖來看可能會更方便清楚一些，不過那個太佔地兒了，而且在Idea中可以自動生成，就不再傳上來了。
五、總結
通過上述的分析，對Flink中的分佈式通信框架就有一個整體上的宏觀認識，注重細節，並不是說沉溺於細節，整體把握，具體分析。這纔是學習別人的優秀的設計思想的好的辦法。
