Algorithm description for Versatile Video Coding and Test Model 8 (VTM 8)
[譯自JVET-Q2002]
3.2 Partitioning 劃分
3.2.3 Partitioning of the CTUs using a tree structure CTU的QTMT劃分結構
In HEVC, a CTU is split into Cus by using a quaternary-tree structure denoted as coding tree to adapt to various local characteristics. The decision whether to code a picture area using inter-picture (temporal) or intra-picture (spatial) prediction is made at the leaf CU level. Each leaf CU can be further split into one, two or four Pus according to the PU splitting type. Inside one PU, the same prediction process is applied and the relevant information is transmitted to the decoder on a PU basis. After obtaining the residual block by applying the prediction process based on the PU splitting type, a leaf CU can be partitioned into transform units (Tus) according to another quaternary-tree structure similar to the coding tree for the CU. One of key feature of the HEVC structure is that it has the multiple partition conceptions including CU, PU, and TU.
- 在HEVC中，CTU通過四叉樹劃分成若干CU。
- 每個CU級別上做出intra / inter 預測的決定。
- CU可以進一步劃分爲一個，兩個或四個預測單元PU。PU內有相同的預測模式和預測過程。
- 預測後獲得殘餘信息，CU通過四叉樹劃分爲若干變換單元TU。
- HEVC結構的關鍵特徵之一是它具有多個分區概念，包括CU，PU和TU。
In VVC, a quadtree with nested multi-type tree using binary and ternary splits segmentation structure replaces the concepts of multiple partition unit types, i.e. it removes the separation of the CU, PU and TU concepts except as needed for CUs that have a size too large for the maximum transform length, and supports more flexibility for CU partition shapes. In the coding tree structure, a CU can have either a square or rectangular shape. A coding tree unit (CTU) is first partitioned by a quaternary tree (a.k.a. quadtree) structure. Then the quaternary tree leaf nodes can be further partitioned by a multi-type tree structure. As shown in Figure 7, there are four splitting types in multi-type tree structure, vertical binary splitting (SPLIT_BT_VER), horizontal binary splitting (SPLIT_BT_HOR), vertical ternary splitting (SPLIT_TT_VER), and horizontal ternary splitting (SPLIT_TT_HOR). The multi-type tree leaf nodes are called coding units (CUs), and unless the CU is too large for the maximum transform length, this segmentation is used for prediction and transform processing without any further partitioning. This means that, in most cases, the CU, PU and TU have the same block size in the quadtree with nested multi-type tree coding block structure. The exception occurs when maximum supported transform length is smaller than the width or height of the colour component of the CU.
- 在VVC中，使用QTMT劃分樹結構，包括四叉、二叉、三叉樹劃分。
- VVC去除了HEVC中CU，PU和TU概念的分隔。
- CU可以具有正方形或矩形形狀。
- CTU首先由四叉樹劃分，然後可以通過QTMT劃分結構進一步劃分四叉樹葉子節點。
- 如下圖所示，MT結構中有四種拆分類型：垂直二叉劃分（SPLIT_BT_VER），水平二叉劃分（SPLIT_BT_HOR），垂直三叉劃分（SPLIT_TT_VER）和水平三叉劃分（SPLIT_TT_HOR）。
- 在大多數情況下，CU，PU和TU具有相同的塊大小。當最大支持的變換長度小於CU的顏色分量的寬度或高度時，會發生例外。
![在這裏插入圖片描述]()
Figure 8 illustrates the signalling mechanism of the partition splitting information in quadtree with nested multi-type tree coding tree structure. A coding tree unit (CTU) is treated as the root of a quaternary tree and is first partitioned by a quaternary tree structure. Each quaternary tree leaf node (when sufficiently large to allow it) is then further partitioned by a multi-type tree structure. In the multi-type tree structure, a first flag (mtt_split_cu_flag) is signalled to indicate whether the node is further partitioned; when a node is further partitioned, a second flag (mtt_split_cu_vertical_flag) is signalled to indicate the splitting direction, and then a third flag (mtt_split_cu_binary_flag) is signalled to indicate whether the split is a binary split or a ternary split. Based on the values of mtt_split_cu_vertical_flag and mtt_split_cu_binary_flag, the multi-type tree slitting mode (MttSplitMode) of a CU is derived as shown in Table 3 1.
- 下圖展示了QTMT劃分的signal機制。
- CTU首先進行四叉樹劃分。
- 然後，每個四叉樹節點被MT進一步劃分。在MT結構中，先用第一個flag（ mtt_split_cu_flag）表示是否劃分；當mtt_split_cu_flag == true，再用第二個flag（mtt_split_cu_vertical_標誌）表示劃分方向，然後，用第三個flag（mtt_split_cu_binary_flag）表示二叉劃分還是三叉劃分。
- 基於mtt-split-cu-vertical-flag和mtt-split-cu-binary-flag，可以導出cu的MT劃分模式（mtt split mode），如下表所示。
![在這裏插入圖片描述]()
![在這裏插入圖片描述]()
In VVC, the maximum supported luma transform size is 64×64 and the maximum supported chroma transform size is 32×32. When the width or height of the CB is larger the maximum transform width or height, the CB is automatically split in the horizontal and/or vertical direction to meet the transform size restriction in that direction.
- 在VVC中，變換時支持的最大亮度塊大小爲64×64，最大色度塊大小爲32×32。
- 當CB的寬/高度大於最大變換寬/高度時，CB將自動進行水平和/或垂直劃分，以滿足該方向上的變換大小限制。
The following parameters are defined and specified by SPS syntax elements for the quadtree with nested multi-type tree coding tree scheme.
– CTU size: the root node size of a quaternary tree
– MinQTSize: the minimum allowed quaternary tree leaf node size
– MaxBtSize: the maximum allowed binary tree root node size
– MaxTtSize: the maximum allowed ternary tree root node size
– MaxMttDepth: the maximum allowed hierarchy depth of multi-type tree splitting from a quadtree leaf
– MinBtSize: the minimum allowed binary tree leaf node size
– MinTtSize: the minimum allowed ternary tree leaf node size
- SPS中包含了以下參數
– CTU size: CTU大小
– MinQTSize: 四叉樹劃分最小尺寸
– MaxBtSize: 二叉樹劃分最大尺寸
– MaxTtSize: 三叉樹劃分最大尺寸
– MaxMttDepth: 從四叉樹節點劃分後允許的最大MT劃分深度
– MinBtSize: 二叉樹劃分最小尺寸
– MinTtSize: 二叉樹劃分最小尺寸
In one example of the quadtree with nested multi-type tree coding tree structure, the CTU size is set as 128×128 luma samples with two corresponding 64×64 blocks of 4:2:0 chroma samples, the MinQTSize is set as 16×16, the MaxBtSize is set as 128×128 and MaxTtSize is set as 64×64, the MinBtSize and MinTtSize (for both width and height) is set as 4×4, and the MaxMttDepth is set as 4. The quaternary tree partitioning is applied to the CTU first to generate quaternary tree leaf nodes. The quaternary tree leaf nodes may have a size from 16×16 (i.e., the MinQTSize) to 128×128 (i.e., the CTU size). If the leaf QT node is 128×128, it will not be further split by the binary tree since the size exceeds the MaxBtSize and MaxTtSize (i.e., 64×64). Otherwise, the leaf qdtree node could be further partitioned by the multi-type tree. Therefore, the quaternary tree leaf node is also the root node for the multi-type tree and it has multi-type tree depth (mttDepth) as 0. When the multi-type tree depth reaches MaxMttDepth (i.e., 4), no further splitting is considered. When the multi-type tree node has width equal to MinBtSize and smaller or equal to 2 * MinTtSize, no further horizontal splitting is considered. Similarly, when the multi-type tree node has height equal to MinBtSize and smaller or equal to 2 * MinTtSize, no further vertical splitting is considered.
- 四叉樹劃分後的節點是MT樹劃分的根節點，其MT樹深度（mttDepth）爲0。
- 當MT樹深度達到MaxMttDepth時，劃分終止。
- 當MT樹劃分的節點寬度等於MinBtSize且小於或等於2*MinTtSize時，不考慮進一步的水平劃分。
- 當MT樹劃分的節點高度等於MinBtSize且小於或等於2*MinTtSize時，不考慮進一步的垂直拆分。
To allow 64×64 Luma block and 32×32 Chroma pipelining design in VVC hardware decoders, TT split is forbidden when either width or height of a luma coding block is larger than 64 , as shown in Figure 10. TT split is also forbidden when either width or height of a chroma coding block is larger than 32.
爲了在VVC硬件解碼器中允許64×64 亮度塊和32×32色度塊進行流水線設計，當亮度塊的寬度或高度大於64時，禁止三叉拆分。當色度編碼塊的寬度或高度大於32時，也禁止TT劃分。
In VVC, the coding tree scheme supports the ability for the luma and chroma to have a separate block tree structure. For P and B slices, the luma and chroma CTBs in one CTU have to share the same coding tree structure. However, for I slices, the luma and chroma can have separate block tree structures. When separate block tree mode is applied, luma CTB is partitioned into CUs by one coding tree structure, and the chroma CTBs are partitioned into chroma CUs by another coding tree structure. This means that a CU in an I slice may consist of a coding block of the luma component or coding blocks of two chroma components, and a CU in a P or B slice always consists of coding blocks of all three colour components unless the video is monochrome.
- VVC支持亮度和色度具有獨立的劃分樹(dual tree)。
- 對於P和B slice，一個CTU中的亮度和色度CTB必須共享相同的編碼樹。
- 對於I slice，亮度和色度可以有獨立的劃分樹。
- 當採用亮色度獨立劃分樹時，亮度CTB被一種編碼樹結構劃分爲luma-CUs，色度CTB被另一種編碼樹結構劃分爲chroma-CUs。這意味着I slice中的CU可以由亮度分量的編碼塊或兩個色度分量的編碼塊組成，並且P或B slice中的CU始終由所有三個顏色分量的編碼塊組成（除非視頻是單色的）。
3.2.4 CU splits on picture boundaries 圖像邊界的CU劃分
As done in HEVC, when a portion of a tree node block exceeds the bottom or right picture boundary, the tree node block is forced to be split until the all samples of every coded CU are located inside the picture boundaries.
HEVC中，當樹節點塊的一部分超過圖像邊界時，樹節點塊會被強行四叉分割，直到每個CU的所有像素都位於圖像邊界內。
The following splitting rules are applied in the VVC:
– If any portion of a tree node block exceeds the bottom or the right picture boundaries, and any of QT, BT and TT splitting is not allowed due to block size restriction, the block is forced to be split with QT split mode.
– Otherwise if a portion of a tree node block exceeds both the bottom and the right picture boundaries,
– If the block is a QT node and the size of the block is larger than the minimum QT size, the block is forced to be split with QT split mode.
– Otherwise, the block is forced to be split with SPLIT_BT_HOR mode
– Otherwise if a portion of a tree node block exceeds the bottom picture boundaries,
– If the block is a QT node, and the size of the block is larger than the minimum QT size, and the size of the block is larger than the maximum BT size, the block is forced to be split with QT split mode.
– Otherwise, if the block is a QT node, and the size of the block is larger than the minimum QT size and the size of the block is smaller than or equal to the maximum BT size, the block is forced to be split with QT split mode or SPLIT_BT_HOR mode.
– Otherwise (the block is a BTT node or the size of the block is smaller than or equal to the minimum QT size), the block is forced to be split with SPLIT_BT_HOR mode.
– Otherwise if a portion of a tree node block exceeds the right picture boundaries,
– If the block is a QT node, and the size of the block is larger than the minimum QT size, and the size of the block is larger than the maximum BT size, the block is forced to be split with QT split mode.
– Otherwise, if the block is a QT node, and the size of the block is larger than the minimum QT size and the size of the block is smaller than or equal to the maximum BT size, the block is forced to be split with QT split mode or SPLIT_BT_VER mode.
– Otherwise (the block is a BTT node or the size of the block is smaller than or equal to the minimum QT size), the block is forced to be split with SPLIT_BT_VER mode.
- VVC中邊界處理的規則：
– 1. 如果 : 樹節點塊的一部分超過圖像的右和底部邊界，且由於塊的大小限制而不允許QT、BT和TT劃分，則強制使用QT劃分。
– 2. 否則如果 : 樹節點塊的一部分超過圖像的右和底部邊界，繼續以下判斷：
– ① 如果 : 該塊是一個QT節點，並且塊的大小大於MinQTSize，則強制使用四叉劃分。
– ② 否則 : 強制使用水平二叉劃分。
– 3. 否則如果 : 樹節點塊的一部分超出圖像的底部邊界，繼續以下判斷：
– ① 如果 : 該塊是一個QT節點，並且塊的大小大於MinQTSize，並且大於MaxBtSize，則強制使用四叉劃分。
– ② 否則如果 : 該塊是一個QT節點，並且塊的大小大於MinQTSize，並且小於或等於MaxBtSize，則強制使用四叉或水平二叉劃分。
– ③ 否則（該塊是BTT節點或塊的大小小於或等於MinQTSize） : 強制使用水平二叉劃分。
– 4. 否則如果 : 樹節點塊的一部分超出圖像的右部邊界，繼續以下判斷：
– ① 如果 : 該塊是一個QT節點，並且塊的大小大於MinQTSize，並且大於MaxBtSize，則強制使用四叉劃分。
– ② 否則如果 : 該塊是一個QT節點，並且塊的大小大於MinQTSize，並且小於或等於MaxBtSize，則強制使用四叉或豎直二叉劃分。
– ③ 否則（該塊是BTT節點或塊的大小小於或等於MinQTSize） : 強制使用豎直二叉劃分。
3.2.5 Restrictions on redundant CU splits 冗餘CU劃分的限制
The quadtree with nested multi-type tree coding block structure provides a highly flexible block partitioning structure. Due to the types of splits supported the multi-type tree, different splitting patterns could potentially result in the same coding block structure. In VVC, some of these redundant splitting patterns are disallowed.
- QTMT會造成不同的劃分組合卻有相同的劃分結果。在VVC中，一些冗餘的劃分模式是不被允許的。
Figure 11 illustrates the redundant splitting patterns of binary tree splits and ternary tree splits. As shown in Figure 11, two levels of consecutive binary splits in one direction could have the same coding block structure as a ternary tree split followed by a binary tree split of the central partition. In this case, the binary tree split (in the given direction) for the central partition of a ternary tree split is prevented by the syntax. This restriction applies for CUs in all pictures.
- 下圖說明了二叉樹和三叉樹的冗餘劃分模式。“同個方向上連續的二叉劃分” 可以具有與 “三叉劃分後的中間區域再進行同方向二叉劃分”相同的編碼塊結構。在這種情況下，語法阻止了“三叉劃分後的中間區域再進行同方向二叉劃分” 。此限制適用於所有圖片中的CU。
When the splits are prohibited as described above, signalling of the corresponding syntax elements is modified to account for the prohibited cases. For example, when any case in Figure 11 is identified (i.e. the binary split is prohibited for a CU of a central partition), the syntax element mtt_split_cu_binary_flag which specifies whether the split is a binary split or a ternary split is not signalled and is instead inferred to be equal to 0 by the decoder.
- 當如上所述禁止劃分時，相應語法元素的信令將被修改，以考慮禁止的情況。
- 例如，當識別出上圖中的任何情況時，語法元素mtt_split_CU_binary_flag（指定分割是二叉還是三叉劃分）不發出信號，而是由解碼器推斷爲等於0。
3.2.6 Virtual pipeline data units (VPDUs)
Virtual pipeline data units (VPDUs) are defined as non-overlapping units in a picture. In hardware decoders, successive VPDUs are processed by multiple pipeline stages at the same time. The VPDU size is roughly proportional to the buffer size in most pipeline stages, so it is important to keep the VPDU size small. In most hardware decoders, the VPDU size can be set to maximum transform block (TB) size. However, in VVC, ternary tree (TT) and binary tree (BT) partition may lead to the increasing of VPDUs size.
- VPDU被定義爲圖片中不重疊的單元。
- 在硬件解碼器中，連續的VPDU由多個流水線同時處理。
- 在大多數流水線階段，VPDU大小大致與緩衝區大小成正比，因此保持VPDU大小較小很重要。
- 在大多數硬件解碼器中，VPDU大小可以設置爲最大變換塊（TB）大小。然而，在VVC中，三叉樹（TT）和二叉樹（BT）的劃分會導致VPDU大小的增加。
In order to keep the VPDU size as 64x64 luma samples, the following normative partition restrictions (with syntax signaling modification) are applied in VTM, as shown in Figure 12:
– TT split is not allowed for a CU with either width or height, or both width and height equal to 128.
– For a 128xN CU with N ≤ 64 (i.e. width equal to 128 and height smaller than 128), horizontal BT is not allowed.
– For an Nx128 CU with N ≤ 64 (i.e. height equal to 128 and width smaller than 128), vertical BT is not allowed.
- 爲了使VPDU的大小保持爲64x64（亮度像素），在VTM中應用了以下規範的分區限制（伴有語法信令修改）。
– 對於寬度或高度都等於128的CU，不允許三叉劃分。
– 對於N≤64（即寬度等於128，高度小於128）的128xN CU，不允許水平二叉劃分。
– 對於N≤64（即高度等於128，寬度小於128）的Nx128 CU，不允許垂直二叉劃分。
3.2.7 Intra chroma partitioning and prediction restriction 色差劃分與預測限制
In typical hardware video encoders and decoders, processing throughput drops when a picture has more small intra blocks because of sample processing data dependency between neighbouring intra blocks. The predictor generation of an intra block requires top and left boundary reconstructed samples from neighbouring blocks. Therefore, intra prediction has to be sequentially processed block by block.
- 在典型的硬件編解碼器中，當圖像具有較多的小塊（幀內預測）時，由於相鄰intra塊之間的數據依賴性，處理的吞吐量下降。
- intra塊的預測器生成需要使用相鄰塊的重構樣本。因此，幀內預測必須逐塊順序處理。
In HEVC, the smallest intra CU is 8x8 luma samples. The luma component of the smallest intra CU can be further split into four 4x4 luma intra prediction units (Pus), but the chroma components of the smallest intra CU cannot be further split. Therefore, the worst case hardware processing throughput occurs when 4x4 chroma intra blocks or 4x4 luma intra blocks are processed. In VVC, in order to improve worst case throughput, chroma intra CBs smaller than 16 chroma samples (size 2x2, 4x2, and 2x4) and chroma intra CBs with width smaller than 4 chroma samples (size 2xN) are disallowed by constraining the partitioning of chroma intra CBs.
- HEVC中，最小的intra CU是 8x8（亮度分量）。
- 最小的intra CU的亮度分量可以進一步分割爲4個4x4 PU，但最小的 intra CU的色度分量不能進一步分割。
- 因此，最壞情況下的硬件處理吞吐量發生在 4x4 色度 intra塊 或 4x4 亮度intra塊。
- VVC中，爲了提高最壞情況下的吞吐量，通過限制色度的劃分，不允許存在小於16個色度樣本（大小爲 2x2、4x2 和 2x4）的 CB，和寬度小於4個色度樣本（大小2xN）的CB。
In single coding tree, a smallest chroma intra prediction unit (SCIPU) is defined as a coding tree node whose chroma block size is larger than or equal to 16 chroma samples and has at least one child luma block smaller than 64 luma samples, or a coding tree node whose chroma block size is not 2xN and has at least one child luma block 4xN luma samples. It is required that in each SCIPU, all CBs are inter, or all CBs are non-inter, i.e, either intra or intra block copy (IBC). In case of a non-inter SCIPU, it is further required that chroma of the non-inter SCIPU shall not be further split and luma of the SCIPU is allowed to be further split. In this way, the small chroma intra CBs with size less than 16 chroma samples or with size 2xN are removed. In addition, chroma scaling is not applied in case of a non-inter SCIPU. Here, no additional syntax is signalled, and whether a SCIPU is non-inter can be derived by the prediction mode of the first luma CB in the SCIPU. The type of a SCIPU is inferred to be non-inter if the current slice is an I-slice or the current SCIPU has a 4x4 luma partition in it after further split one time (because no inter 4x4 is allowed in VVC); otherwise, the type of the SCIPU (inter or non-inter) is indicated by one flag before parsing the Cus in the SCIPU.
- 在單編碼樹中（亮色度具有相同的編碼樹結構），最小色度幀內預測單元（SCIPU）被定義爲其色度塊大小 大於或等於16個色度樣本 且 具有至少一個子亮度塊小於64個亮度樣本的編碼樹節點，或其色度塊大小不爲2xN且具有至少一個子亮度塊4xN 亮度樣本的編碼樹節點。
- 要求在每個SCIPU中，所有cb都是inter，或者所有cb都是非inter，即intra或IBC。
- 如果是非inter-SCIPU，則進一步要求 非inter-SCIPU 的色度不應進一步劃分，並且允許SCIPU的亮度進一步劃分。這樣，去除大小 小於16個色度樣本 或 大小爲2xN的小色度 intra CB。
- 此外，對於 非inter-SCIPU，不應用色度縮放。這裏，不需要傳輸額外的語法，並且 SCIPU是否是非inter 可以由SCIPU中的第一個亮度 CB的預測模式導出。如果當前是 I Slice，或者 當前SCIPU 在進一步劃分一次 後有 一個4x4 亮度分區（因爲VVC中不允許inter 4x4），則推斷SCIPU的類型爲非inter；否則，在解析SCIPU中的CU之前，SCIPU的類型（inter或non inter）由一個標誌指示。
For the dual tree in intra picture, the 2xN intra chroma blocks are removed by disabling vertical binary and vertical ternary splits for 4xN and 8xN chroma partitions, respectively. The small chroam blocks wtihsize 2x2, 4x2, and 2x4 are also removed by partitioning restrictions.
- 對於intra圖像中的dual tree (亮色度具有不同的編碼樹結構)，通過分別爲4xN和8xN色度分區 禁用垂直二叉和垂直三叉劃分 來 移除2xN色度塊。
- 大小爲2x2、4x2和2x4的小chroam塊也通過分區限制被移除。
In addition, a restriction on picture size is considered to avoid 2x2/2x4/4x2/2xN intra chroma blocks at the corner of pictures by considering the picture width and height to be multiple of max (8, MinCbSizeY).
- 此外，考慮到圖片寬度和高度是max（8，MinCbSizeY）的倍數，需要對圖片大小進行限制以避免圖片邊角處出現2x2/2x4/4x2/2xN intra 色塊。