13-ByteBuf

ByteBuf

• ByteBuf 是 Netty 中數據容器，它高效的實現了底層數據通信過程中所需要的字節序列的相關操作。

• ByteBuf 自身的源碼就非常多，而且還有很多子類，因此不可能面面俱到，我們分析一些主要的點，對比BIO中的ByteBuffer來對比會更好，ByteBuffer是Java Nio中的字節容器，但是使用上較爲複雜，Netty使用ByteBuf來實現了更加強大的功能，我們看看Netty 自己實現的ByteBuf具體有什麼優勢。

• 考慮到 ByteBuf 的內容非常多，會將其分爲幾個部分慢慢研究，本文主要關於ByteBuf的API；

一、ByteBuf

• ByteBuf 是一個抽象類，是Netty中具體的ByteBuf的抽象父類,實現了ReferenceCounted 和 Comparable 接口，ReferenceCounted 接口在後面文章分析

public abstract class ByteBuf implements ReferenceCounted, Comparable<ByteBuf> 

1.1 類註釋

• 先從類註釋獲取一些基本的信息,對其有一個大致的瞭解

/**
 * 一個零個或者多個字節序列，可以隨機並且有序訪問；此接口提供原始字節的抽象視圖
 *
 * A random and sequential accessible sequence of zero or more bytes (octets).
 * This interface provides an abstract view for one or more primitive byte
 * arrays ({@code byte[]}) and {@linkplain ByteBuffer NIO buffers}.
 *
 * <h3>Creation of a buffer</h3>
 *
 * 推薦使用幫助方法Unpooled 來創建一個實例，而不是使用構造方法
 * It is recommended to create a new buffer using the helper methods in
 * {@link Unpooled} rather than calling an individual implementation's
 * constructor.
 *
 * <h3>Random Access Indexing</h3>
 * 可以像數組一樣支持隨機訪問，下標 是: 0 ->  buffer.capacity()-1
 * Just like an ordinary primitive byte array, {@link ByteBuf} uses
 * <a href="http://en.wikipedia.org/wiki/Zero-based_numbering">zero-based indexing</a>.
 * It means the index of the first byte is always {@code 0} and the index of the last byte is
 * always {@link #capacity() capacity - 1}.  For example, to iterate all bytes of a buffer, you
 * can do the following, regardless of its internal implementation:
 *
 * <pre>
 * {@link ByteBuf} buffer = ...;
 * for (int i = 0; i &lt; buffer.capacity(); i ++) {
 *     byte b = buffer.getByte(i);
 *     System.out.println((char) b);
 * }
 * </pre>
 *
 * <h3>Sequential Access Indexing</h3>
 * 提供兩個指針變量來支持順序讀寫操作，readIndex()用於讀操作，writeIndex()用於寫操作，
 * 下圖展示了buffer如何被兩個指針分成三個部分：
 *  0  <=  readIndex <= writeIndex <= capacity
 *  0   丟棄部分 readIndex     可讀部分 writeIndex   可寫部分 capacity
 *
 * {@link ByteBuf} provides two pointer variables to support sequential
 * read and write operations - {@link #readerIndex() readerIndex} for a read
 * operation and {@link #writerIndex() writerIndex} for a write operation
 * respectively.  The following diagram shows how a buffer is segmented into
 * three areas by the two pointers:
 *
 * <pre>
 *      +-------------------+------------------+------------------+
 *      | discardable bytes |  readable bytes  |  writable bytes  |
 *      |                   |     (CONTENT)    |                  |
 *      +-------------------+------------------+------------------+
 *      |                   |                  |                  |
 *      0      <=      readerIndex   <=   writerIndex    <=    capacity
 * </pre>
 *
 * <h4>Readable bytes (the actual content)</h4>
 *
 * read開頭的操作和skip開頭的操作是讀取數據，讀取數據後readerIndex會自動增加讀取的
 * 部分長度，skip會跳過一部分讀取；
 *
 * 如果讀取操作的參數也是一個ByteBuf對象並且沒有指定目標index，那麼參數對象的writerIndex也
 * 會增長，如果空間不足，則會拋出：IndexOutOfBoundsException
 *
 * This segment is where the actual data is stored.  Any operation whose name
 * starts with {@code read} or {@code skip} will get or skip the data at the
 * current {@link #readerIndex() readerIndex} and increase it by the number of
 * read bytes.  If the argument of the read operation is also a
 * {@link ByteBuf} and no destination index is specified, the specified
 * buffer's {@link #writerIndex() writerIndex} is increased together.
 * <p>
 * If there's not enough content left, {@link IndexOutOfBoundsException} is
 * raised.  The default value of newly allocated, wrapped or copied buffer's
 * {@link #readerIndex() readerIndex} is {@code 0}.
 *
 * <pre>
 * // Iterates the readable bytes of a buffer.
 * {@link ByteBuf} buffer = ...;
 * while (buffer.isReadable()) {
 *     System.out.println(buffer.readByte());
 * }
 * </pre>
 *
 * <h4>Writable bytes</h4>
 *
 * write開頭的操作是寫數據，寫數據之後writerIndex會增加所寫部分的長度
 * 如果寫操作的參數也是ByteBuf，並且沒有指定index，那麼參數對應的ByteBuf的readerIndex也
 * 會一起自增，如果空間不足，則會拋出：IndexOutOfBoundsException
 *
 * 新創建的ByteBuf的writerIndex是0，wrapped的或者copied的Buffer的writerIndex等於capacity
 *
 * This segment is a undefined space which needs to be filled.  Any operation
 * whose name starts with {@code write} will write the data at the current
 * {@link #writerIndex() writerIndex} and increase it by the number of written
 * bytes.  If the argument of the write operation is also a {@link ByteBuf},
 * and no source index is specified, the specified buffer's
 * {@link #readerIndex() readerIndex} is increased together.
 * <p>
 *
 * If there's not enough writable bytes left, {@link IndexOutOfBoundsException}
 * is raised.  The default value of newly allocated buffer's
 * {@link #writerIndex() writerIndex} is {@code 0}.  The default value of
 * wrapped or copied buffer's {@link #writerIndex() writerIndex} is the
 * {@link #capacity() capacity} of the buffer.
 *
 * <pre>
 * // Fills the writable bytes of a buffer with random integers.
 * {@link ByteBuf} buffer = ...;
 * while (buffer.maxWritableBytes() >= 4) {
 *     buffer.writeInt(random.nextInt());
 * }
 * </pre>
 *
 * <h4>Discardable bytes</h4>
 * 已經讀取的部分稱爲丟棄部分，最初是0，隨着讀取操作的進行，這部分區域越來越大，可以通過
 * discardReadBytes方法來回收該部分的內存區域，調用之後這部分區域會變成可寫區域，如下圖所示
 *
 * This segment contains the bytes which were read already by a read operation.
 * Initially, the size of this segment is {@code 0}, but its size increases up
 * to the {@link #writerIndex() writerIndex} as read operations are executed.
 * The read bytes can be discarded by calling {@link #discardReadBytes()} to
 * reclaim unused area as depicted by the following diagram:
 *
 * <pre>
 *  BEFORE discardReadBytes()
 *
 *      +-------------------+------------------+------------------+
 *      | discardable bytes |  readable bytes  |  writable bytes  |
 *      +-------------------+------------------+------------------+
 *      |                   |                  |                  |
 *      0      <=      readerIndex   <=   writerIndex    <=    capacity
 *
 *
 *  AFTER discardReadBytes()
 *
 *      +------------------+--------------------------------------+
 *      |  readable bytes  |    writable bytes (got more space)   |
 *      +------------------+--------------------------------------+
 *      |                  |                                      |
 * readerIndex (0) <= writerIndex (decreased)        <=        capacity
 * </pre>
 *
 * 注意writable bytes是不能保證的，多數情況下不會被移動，這取決的底層實現
 * Please note that there is no guarantee about the content of writable bytes
 * after calling {@link #discardReadBytes()}.  The writable bytes will not be
 * moved in most cases and could even be filled with completely different data
 * depending on the underlying buffer implementation.
 *
 * <h4>Clearing the buffer indexes</h4>
 *
 * clear方法可以將讀寫索引置零，但是並不會清楚buffer的內容，注意clear操作的語義
 * 和ByteBuffer#clear()是不一樣的
 *
 * You can set both {@link #readerIndex() readerIndex} and
 * {@link #writerIndex() writerIndex} to {@code 0} by calling {@link #clear()}.
 * It does not clear the buffer content (e.g. filling with {@code 0}) but just
 * clears the two pointers.  Please also note that the semantic of this
 * operation is different from {@link ByteBuffer#clear()}.
 *
 * <pre>
 *  BEFORE clear()
 *
 *      +-------------------+------------------+------------------+
 *      | discardable bytes |  readable bytes  |  writable bytes  |
 *      +-------------------+------------------+------------------+
 *      |                   |                  |                  |
 *      0      <=      readerIndex   <=   writerIndex    <=    capacity
 *
 *
 *  AFTER clear()
 *
 *      +---------------------------------------------------------+
 *      |             writable bytes (got more space)             |
 *      +---------------------------------------------------------+
 *      |                                                         |
 *      0 = readerIndex = writerIndex            <=            capacity
 * </pre>
 *
 * <h3>Search operations</h3>
 * 使用indexOf，bytesBefore，forEachByte來搜索
 * For simple single-byte searches, use {@link #indexOf(int, int, byte)} and {@link #bytesBefore(int, int, byte)}.
 * {@link #bytesBefore(byte)} is especially useful when you deal with a {@code NUL}-terminated string.
 * For complicated searches, use {@link #forEachByte(int, int, ByteProcessor)} with a {@link ByteProcessor}
 * implementation.
 *
 * <h3>Mark and reset</h3>
 *  markReaderIndex 和 markWriterIndex分表標記讀寫索引的位置
 * There are two marker indexes in every buffer. One is for storing
 * {@link #readerIndex() readerIndex} and the other is for storing
 * {@link #writerIndex() writerIndex}.  You can always reposition one of the
 * two indexes by calling a reset method.  It works in a similar fashion to
 * the mark and reset methods in {@link InputStream} except that there's no
 * {@code readlimit}.
 *
 * <h3>Derived buffers</h3>
 * 派生緩衝區，下面方法會給一個已存在的buffer創建一個緩衝區視圖，(內部存儲的是一份數據，改動是關聯
 * 的，使用一個引用計數，但是讀寫索引和索引標記是相互獨立的)
 *
 * 如果需要一份獨立的拷貝，使用copy方法
 * You can create a view of an existing buffer by calling one of the following methods:
 * <ul>
 *   <li>{@link #duplicate()}</li>
 *   <li>{@link #slice()}</li>
 *   <li>{@link #slice(int, int)}</li>
 *   <li>{@link #readSlice(int)}</li>
 *   <li>{@link #retainedDuplicate()}</li>
 *   <li>{@link #retainedSlice()}</li>
 *   <li>{@link #retainedSlice(int, int)}</li>
 *   <li>{@link #readRetainedSlice(int)}</li>
 * </ul>
 * A derived buffer will have an independent {@link #readerIndex() readerIndex},
 * {@link #writerIndex() writerIndex} and marker indexes, while it shares
 * other internal data representation, just like a NIO buffer does.
 * <p>
 * In case a completely fresh copy of an existing buffer is required, please
 * call {@link #copy()} method instead.
 *
 * <h4>Non-retained and retained derived buffers</h4>
 *
 * 注意duplicate、slice、slice、readSlice這些方法返回派生緩衝區是不會調用retain增加引用計數器的，
 *
 * 如果需要創建一個buffer並且增加引用計數器，那麼使用retainedDuplicate、retainedSlice、retainedSlice、readRetainedSlice，
 * 它們返回的buffer會產生更少的垃圾
 *
 * Note that the {@link #duplicate()}, {@link #slice()}, {@link #slice(int, int)} and {@link #readSlice(int)} does NOT
 * call {@link #retain()} on the returned derived buffer, and thus its reference count will NOT be increased. If you
 * need to create a derived buffer with increased reference count, consider using {@link #retainedDuplicate()},
 * {@link #retainedSlice()}, {@link #retainedSlice(int, int)} and {@link #readRetainedSlice(int)} which may return
 * a buffer implementation that produces less garbage.
 *
 * <h3>Conversion to existing JDK types</h3>
 *
 * <h4>Byte array</h4>
 *
 * 如果ByteBuf內部使用byte array來保存數據，那麼可以通過array方法來獲取這個字節數組，
 * 使用hasArray可以判斷其是不是由byte array來保存數據的
 * If a {@link ByteBuf} is backed by a byte array (i.e. {@code byte[]}),
 * you can access it directly via the {@link #array()} method.  To determine
 * if a buffer is backed by a byte array, {@link #hasArray()} should be used.
 *
 * <h4>NIO Buffers</h4>
 * 如果一個ByteBuf能夠轉換爲一個NIO的ByteBuffer，並且內容共享，可以調用nioBuffer方法來做轉換，
 * 使用nioBufferCount判斷一個ByteBuf 是否可以轉換爲一個 NIO的ByteBuffer
 * If a {@link ByteBuf} can be converted into an NIO {@link ByteBuffer} which shares its
 * content (i.e. view buffer), you can get it via the {@link #nioBuffer()} method.  To determine
 * if a buffer can be converted into an NIO buffer, use {@link #nioBufferCount()}.
 *
 * <h4>Strings</h4>
 *
 * Various {@link #toString(Charset)} methods convert a {@link ByteBuf}
 * into a {@link String}.  Please note that {@link #toString()} is not a
 * conversion method.
 *
 * <h4>I/O Streams</h4>
 *
 * Please refer to {@link ByteBufInputStream} and
 * {@link ByteBufOutputStream}.
 */

• 小結如下：

1.ByteBuf內部是字節序列，支持隨機訪問和順序訪問，可以通過下標訪問，使用Unpooled創建
2.內部通過讀寫兩個索引來控制讀寫，兩個索引都要對應的標記方法，讀寫需要注意越界異常
3.已經讀取過的內容相當於廢棄內容，可以通過discardReadBytes來騰出該部分空間用於寫入，不過這取決於不同子類實現
4.派生緩衝區會共享數據，但是讀寫索引是獨立的，引用計數是共享的，對應的也有一些方法提供buffer並且增加引用計數
5.可以通過方法獲取內部的字節數組

1.2 方法

• 方法非常多，只能歸類熟悉

1.2.1 屬性相關

• ByteBuf 基本屬性相關的方法

public abstract int capacity(); // 容量
public abstract ByteBuf capacity(int newCapacity);
public abstract int maxCapacity(); // 最大容量

public abstract ByteBufAllocator alloc(); // 分配器，用於創建 ByteBuf 對象。

public abstract ByteBuf unwrap(); // 獲得被包裝( wrap )的 ByteBuf 對象。

public abstract boolean isDirect(); // 是否 NIO Direct Buffer

public abstract boolean isReadOnly(); // 是否爲只讀 Buffer

public abstract int readerIndex(); //獲取讀取位置
public abstract ByteBuf readerIndex(int readerIndex); //設置讀取位置
public abstract int writerIndex(); //獲取寫入位置
public abstract ByteBuf writerIndex(int writerIndex); //設置寫入位置
public abstract ByteBuf setIndex(int readerIndex, int writerIndex); // 設置讀取和寫入位置
public abstract int readableBytes(); // 剩餘可讀字節數
public abstract int writableBytes(); // 剩餘可寫字節數
public abstract ByteBuf markReaderIndex(); // 標記讀取位置
public abstract ByteBuf markWriterIndex(); // 標記寫入位置

1.2.2 讀寫數據

• 數據讀取方法，可以按照不同的類型讀取，讀取的字節數也不一樣,下面只給出Int類型的讀寫，其他類型的讀寫類似

// Int 4 字節
//讀取
//絕對讀，不改變索引
public abstract int   getInt(int index); //指定位置讀取int，不改變讀索引
public abstract int   getIntLE(int index); //指定位置讀取int，不改變讀索引，以小端的方式讀取

//普通讀，改變索引
public abstract int   readInt();//讀取int，readerIndex增加4
public abstract int   readIntLE();//讀取int，readerIndex增加4，以小端的方式讀取


//寫入
//絕對寫，不改變索引
public abstract ByteBuf setInt(int index, int value);//在指定位置寫入int，不改變讀寫索引
public abstract ByteBuf setIntLE(int index, int value);//在指定位置寫入int，不改變讀寫索引，以小端的方式寫入

//普通寫，改變索引
public abstract ByteBuf writeInt(int value);//寫入int，並且增加寫索引
public abstract ByteBuf writeIntLE(int value);//寫入int，並且增加寫索引

• 這裏還有很多其他類型的方法沒有給出，比如讀取無符號數等，但是討論都一樣，我們看到讀和寫都有兩類方法；直接通過get和set的讀寫方法是不改變索引的，因爲是隨機訪問直接通過index操作，而read/write對應的讀寫API的會改變索引的，相當於在前面索引的基礎之上再繼續操作；
• 其他還有很多類型，各種基本類型、數組、Medium 類型等操作都有類似的API

1.2.3 查找方法

// 指定值( value ) 在 ByteBuf 中的位置，包左不包右
public abstract int indexOf(int fromIndex, int toIndex, byte value); 


//查找第一次出現value的位置
public abstract int bytesBefore(byte value); 


//從讀索引開始查找length長度，查找value出現的位置，不改變讀寫索引
public abstract int bytesBefore(int length, byte value); 


//從index開始查找length的長度，查找value出現的位置，不改變讀寫索引
public abstract int bytesBefore(int index, int length, byte value);


// 遍歷 ByteBuf ，進行自定義處理
public abstract int forEachByte(ByteProcessor processor); 
public abstract int forEachByte(int index, int length, ByteProcessor processor);
//倒序遍歷處理
public abstract int forEachByteDesc(ByteProcessor processor);
public abstract int forEachByteDesc(int index, int length, ByteProcessor processor);

1.2.4 釋放操作

public abstract ByteBuf discardReadBytes(); // 釋放已讀的字節空間，參考前面的註釋裏面
public abstract ByteBuf discardSomeReadBytes(); // 釋放部分已讀的字節空間，可能會釋放也可能不釋放
public abstract ByteBuf clear(); // 清空字節空間。實際是修改 readerIndex=writerIndex=0，標記清空。

• discardReadBytes 方法會將剩餘的可讀部分移動到字節數組的頭部，由此騰出空間放在最後面寫入，這是一個時間換空間的操作，在AbstractByteBuf的實現如下：

@Override
    public ByteBuf discardReadBytes() {
        //1.校驗可訪問
        ensureAccessible();
        //2.無廢棄段，則直接返回
        if (readerIndex == 0) {
            return this;
        }

        //3.未讀取完
        if (readerIndex != writerIndex) {
            //4.將可讀段複製到ByteBuf頭部空間，由子類實現
            setBytes(0, this, readerIndex, writerIndex - readerIndex);
            //5.寫索引減小
            writerIndex -= readerIndex;
            //6.調整標記位
            adjustMarkers(readerIndex);
            //7.讀索引重置爲 0
            readerIndex = 0;
        } else {
            //8.全部讀取完，則調整標記位
            adjustMarkers(readerIndex);
            //9.索引重置
            writerIndex = readerIndex = 0;
        }
        //10.返回當前對象
        return this;
    }

• 在AbstractByteBuf 中discardSomeReadBytes方法會判斷廢棄的內容是否超過總空間的一半，超過則回收，反之不會回收

1.2.5 拷貝操作

• 按照前面類註釋的說明，拷貝方法有兩類，一類是拷貝後的內容是使用同一份的，相互關聯的，但是讀寫索引是獨立的，這種也叫派生buffer，注意這裏面有些方法不會自增引用計數，帶retained的方法，會自增引用計數

public abstract ByteBuf duplicate(); // 拷貝整個的字節數組。共享，相互影響。
public abstract ByteBuf slice(); // 拷貝可讀部分的字節數組。共享，相互影響。
public abstract ByteBuf slice(int index, int length);

public abstract ByteBuf readSlice(int length);
public abstract ByteBuf readRetainedSlice(int length);

public abstract ByteBuf retainedSlice();
public abstract ByteBuf retainedSlice(int index, int length);
public abstract ByteBuf retainedDuplicate();

• 另一類是內容一致，但是相互獨立，主要是copy 方法

public abstract ByteBuf copy(); // 拷貝可讀部分的字節數組。獨立，互相不影響。
public abstract ByteBuf copy(int index, int length);

1.2.6 其他

• 另外包括NIO轉換方法，在類註釋中說明了，主要是：nioBuffer() 和 nioBufferCount() 方法
• 還有獲取內部的字節數組，主要是：array() 方法和 hasArray() 方法，適用於Heap類型的 ByteBuf

二、ByteBuf和ByteBuffer對比

• 主要對比一下ByteBuf 和NIO 的byteBuffer 之間的區別

1.讀和寫使用了不同的索引，API更友好，因此也不需要flip 切換讀寫模式
2.容量可以按需增長，ByteBuffer 不能
3.支持引用計數，因此回收更及時，更靈活
4.支持池化，資源複用，支持零拷貝，提高性能
5.支持方法的鏈式調用，API更友好

• ByteBuffer 爲什麼不能擴容:如下所示其內部的hb數組是final類型的，因此在創建之後不能修改，因此無法擴容

public abstract class ByteBuffer extends Buffer implements Comparable<ByteBuffer>{

    final byte[] hb;                  // Non-null only for heap buffers
    final int offset;
    boolean isReadOnly;                 // Valid only for heap buffers

• 後續文章繼續分析ByteBuf 的一些特性，比如子類實現、擴容機制、分配方式等；