Abstract—Existing deep convolutional neuralnetworks (CNNs) require a fixed-size (e.g., 224×224) input image. Thisrequirement is “artificial” and may reduce the recognition accuracy for theimages or sub-images of an arbitrary size/scale. In this work, we equipthe networks with another pooling strategy, “spatial pyramid pooling”, toeliminate the above requirement. The new network structure, calledSPP-net, can generate a fixed-length representation regardless of imagesize/scale. Pyramid pooling is also robust to object deformations. Withthese advantages, SPP-net should in general improve all CNN-basedimage classification methods. On the ImageNet 2012 dataset, we demonstratethat SPP-net boosts the accuracy of a variety of CNN architectures despitetheir different designs. On the Pascal VOC 2007 and Caltech101 datasets,SPP-net achieves state-of-the-art classification results using a singlefull-image representation and no fine-tuning.
The power of SPP-net is also significantin object detection. Using SPP-net, we compute the feature maps from theentire image only once, and then pool features in arbitrary regions(sub-images) to generate fixed-length representations for training thedetectors. This method avoids repeatedly computing the convolutional features.In processing test images, our method is 24-102×faster than the R-CNN method,while achieving better or comparable accuracy on Pascal VOC 2007.
In ImageNet Large Scale VisualRecognition Challenge (ILSVRC) 2014, our methods rank #2 in object detectionand #3 in image classification among all 38 teams. This manuscript alsointroduces the improvement made for this competition.
Index Terms—Convolutional NeuralNetworks, Spatial Pyramid Pooling, Image Classification, Object Detection
摘要
在ImageNet大規模視覺識別任務挑戰（ILSVRC）2014上，我們的方法在物體檢測上排名第2，在物體分類上排名第3，參賽的總共有38個組。本文也介紹了爲了這個比賽所作的一些改進。
關鍵詞：Convolutional NeuralNetworks, Spatial Pyramid Pooling, Image Classification, Object Detection
1 INTRODUCTION
We are witnessing a rapid, revolutionary change in ourvision community, mainly caused by deep convolutional neural networks (CNNs)[1] and the availability of large scale training data [2]. Deep-networks-based approacheshave recently been substantially improving upon the state of the art in imageclassification [3], [4], [5], [6], object detection [7], [8], [5],many other recognition tasks [9], [10], [11], [12],and even non-recognition tasks.
1.引言
我們看到計算機視覺領域正在經歷飛速的變化，這一切得益於深度卷積神經網絡（CNNs）[1]和大規模的訓練數據的出現[2]。近來深度網絡對圖像分類 [3][4][5][6]，物體檢測 [7][8][5]和其他識別任務 [9][10][11][12]，甚至很多非識別類任務上都表現出了明顯的性能提升。
However, there is a technical issue in the training andtesting of the CNNs: the prevalent CNNs require a fixed input image size (e.g.,224×224), which limits both the aspect ratio and the scale of the input image.Whenapplied to images of arbitrary sizes, current methods mostly fit the inputimage to the fixed size, either via cropping [3], [4] or via warping [13], [7],asshown in Figure 1 (top). But the cropped region may not contain the entireobject, while the warped content may result in unwanted geometric distortion.Recognitionaccuracy can be compromised due to the content loss or distortion. Besides, apre-defined scale may not be suitable when object scales vary. Fixing inputsizes overlooks the issues involving scales.
然而，這些技術再訓練和測試時都有一個問題，這些流行的CNNs都需要輸入的圖像尺寸是固定的（比如224×224），這限制了輸入圖像的長寬比和縮放尺度。當遇到任意尺寸的圖像時，都是先將圖像適應成固定尺寸，方法包括裁剪[3][4]和變形[13][7]，如圖1（上）所示。但裁剪會導致信息的丟失，變形會導致位置信息的扭曲，就會影響識別的精度。另外，一個預先定義好的尺寸在物體是縮放可變的時候就不適用了。固定輸入大小忽略了涉及比例的問題。
So why do CNNs require a fixed input size? A CNN mainly consists of two parts: convolutional layers,and fully-connected layers that follow. The convolutional layers operate in a sliding-window manner and output feature maps which represent the spatial arrangement of the activations (Figure2). In fact, convolutional layers do not require a fixed image size and cangenerate feature maps of any sizes. On the other hand, the fully-connected layers need to have fixed-size/length input by their definition. Hence, thefixed-size constraint comes only from the fully-connected layers, which existat a deeper stage of the network.
那麼爲什麼CNNs需要一個固定的輸入尺寸呢？CNN主要由兩部分組成，卷積層和其後的全連接層。卷積部分通過滑窗進行計算，並輸出代表激活的空間排布的特徵圖（feature map）（圖2）。事實上，卷積並不需要固定的圖像尺寸，他可以產生任意尺寸的特徵圖。而另一方面，根據定義，全連接層則需要固定的尺寸輸入。因此固定尺寸的問題來源於全連接層，也是網絡的最後階段。
In this paper, we introduce a spatial pyramid pooling(SPP)[14], [15] layer to remove the fixed-size constraint of the network.Specifically, we add an SPP layer on top of the last convolutional layer. The SPP layer pools the features and generates fixed-length outputs, which are then fed into the fully-connected layers (or other classifiers). In other words,we perform some information “aggregation” at a deeper stage of the networkhierarchy (between convolutional layers and fully-connected layers) to avoid theneed for cropping or warping at the beginning.Figure 1 (bottom) shows thechange of the network architecture by introducing the SPP layer. We call the newnetwork structure SPP-net.
本文引入一種空間金字塔池化( spatial pyramid pooling，SPP)層以移除對網絡固定尺寸的限制。尤其是，將SPP層放在最後一個卷積層之後。SPP層對特徵進行池化，併產生固定長度的輸出，這個輸出再餵給全連接層（或其他分類器）。換句話說，在網絡層次的較後階段（也就是卷積層和全連接層之間）進行某種信息“彙總”，可以避免在最開始的時候就進行裁剪或變形。圖1（下）展示了引入SPP層之後的網絡結構變化。我們稱這種新型的網絡結構爲SPP-net。
Spatial pyramid pooling [14], [15] (popularly known as spatial pyramid matching or SPM [15]), as an extension of the Bag-of-Words (BoW) model [16],is one of the most successful methods in computer vision. It partitions the image into divisions from finer to coarser levels, and aggregates local features in them. SPP has long been a key component in the leading and competition-winning systems for classification (e.g., [17], [18], [19]) and detection (e.g., [20])before the recent prevalence of CNNs. Nevertheless, SPP has not been considered in the context of CNNs.We note that SPP has several remarkable properties for deep CNNs: 1) SPP is able to generate a fixed-length output regardless of the input size, while the sliding window pooling used in the previous deep networks [3] cannot; 2) SPP uses multi-level spatial bins, while the sliding window pooling uses only a single window size. Multi-level pooling has been shown to be robust to object deformations [15]; 3) SPP can pool features extracted at variable scales thanks to the flexibility of input scales. Throughexperiments we show that all these factors elevate the recognition accuracy of deep networks.
空間金字塔池化[14][15]（普遍稱謂：空間金字塔匹配spatial pyramid matching, SPM[15]），是一種詞袋(Bag-of-Words, BoW)模型的擴展。詞袋模型是計算機視覺領域最成功的方法之一。它將圖像切分成粗糙到精細各種級別，然後整合其中的局部特徵。在CNN之前，SPP一直是各大分類比賽[17][18][19]和檢測比賽（比如[20]）的冠軍系統中的核心組件。對深度CNNs而言，SPP有幾個突出的優點：1）SPP能在輸入尺寸任意的情況下產生固定大小的輸出，而以前的深度網絡[3]中的滑窗池化(sliding window pooling)則不能；2）SPP使用了多級別的空間箱(bin)，而滑窗池化則只用了一個窗口尺寸。多級池化對於物體的變形十分魯棒[15]；3）由於其對輸入的靈活性，SPP可以池化從各種尺度抽取出來的特徵。通過實驗，我們將展示影響深度網絡最終識別精度的所有這些因素。
SPP-net not only makes it possible to generate representations from arbitrarily sized images/windows for testing, but also allows us to feedimages with varying sizes or scales during training. Training with variable-size images increases scale-invariance and reduces over-fitting. We develop a simple multi-size training method. For a single network to accept variable inputsizes, we approximate it by multiple networks that share all parameters, while each of these networks is trained using a fixed input size. In each epoch we train the network with a given input size, and switch to another input size for the next epoch. Experiments show that this multi-size training converges just as the traditional single-size training,and leads to better testing accuracy.
SPP-net不僅僅讓測試階段允許任意尺寸的輸入能夠產生表示(representations)，也允許訓練階段的圖像可以有各種尺寸和縮放尺度。使用各種尺寸的圖像進行訓練可以提高縮放不變性，以及減少過擬合。我們開發了一個簡單的多尺度訓練方法。爲了實現一個單一的能夠接受各種輸入尺寸的網絡，我們先使用分別訓練固定輸入尺寸的多個網絡，這些網絡之間共享權重（Parameters），然後再一起來代表這個單一網絡（譯者注：具體代表方式沒有說清楚，看後面怎麼說吧）。每個epoch，我們針對一個給定的輸入尺寸進行網絡訓練，然後在下一個epoch再切換到另一個尺寸。實驗表明，這種多尺度訓練和傳統的單一尺度訓練一樣可以瘦臉，並且能達到更好的測試精度。The advantages of SPP are orthogonal to the specific CNN designs. In a series of controlled experiments on the ImageNet 2012 dataset, we demonstrate that SPP improves four different CNN architectures in existing publications[3], [4], [5] (or their modifications), over the no-SPP counterparts. These architectures have various filter numbers/sizes, strides, depths, or other designs.It is thus reasonable for us to conjecture that SPP should improve more sophisticated(deeper and larger) convolutional architectures. SPP-net also shows state-of-the-art classification results on Caltech101 [21] and Pascal VOC 2007[22] using only a single full-image representation and no fine-tuning.
SPP的優點是與各類CNN設計是正交的。通過在ImageNet2012數據集上進行一系列可控的實驗，我們發現SPP對[3][4][5]這些不同的CNN架構都有提升。這些架構有不同的特徵數量、尺寸、滑動距離（strides）、深度或其他的設計。所以我們有理由推測SPP可以幫助提升更多複雜的（更大、更深）的卷積架構。SPP-net也做到了 Caltech101 [21]和Pascal VOC 2007 [22]上的最好結果，而只使用了一個全圖像表示，且沒有調優。
SPP-net also shows great strength in object detection. In the leading object detection method R-CNN[7],the features from candidate windows are extracted via deep convolutional networks. This method shows remarkable detection accuracy on both the VOC and ImageNet datasets. But the feature computation in R-CNN is time-consuming, because itrepeatedly applies the deep convolutional networks to the raw pixels of thousands of warped regions per image. In this paper, we show that we can runthe convolutional layers only once on the entireimage (regardless of the number of windows), and then extract features by SPP-net on the feature maps. This method yields a speed up of over one hundred times over R-CNN. Note that training/running a detector on the feature maps(rather than image regions) is actually a more popular idea [23], [24], [20],[5]. But SPP-net inherits the power of the deep CNN feature maps and also the flexibility of SPP on arbitrary window sizes, which leads to outstanding accuracy and efficiency. In our experiment, the SPP-net-based system (built upon the R-CNNpipeline) computes features 24-102×faster than R-CNN, while has better or comparable accuracy.With the recent fast proposal method of EdgeBoxes[25], oursystem takes 0.5 seconds processing an image(including all steps). This makes our method practical for real-world applications.
在圖像檢測方面，SPP-net也表現優異。目前領先的方法是R-CNN[7]，候選窗口的特徵是藉助深度神經網絡進行抽取的。此方法在VOC和ImageNet數據集上都表現出了出色的檢測精度。但R-CNN的特徵計算十分耗時，因爲他對每張圖片中的上千個變形後的區域的像素反覆調用CNN。本文中，我們展示了我們只需要在整張圖片上運行一次卷積網絡層（不關心窗口的數量），然後再使用SPP-net在特徵圖上抽取特徵。這個方法縮減了上百倍的耗時。在特徵圖（而不是圖像區域）上訓練和運行檢測器是一個很受歡迎的想法[23][24][20][5]。但SPP-net延續了深度CNN特徵圖的優勢，也結合了SPP兼容任意窗口大小的靈活性，所以做到了出色的精度和效率。我們的實驗中，基於SPP-net的系統（建立在R-CNN流水線上）比R-CNN計算特徵要快24-120倍，而精度卻更高。結合最新的推薦方法EdgeBoxes[25]，我們的系統達到了每張圖片處理0.5s的速度（全部步驟）。這使得我們的方法變得更加實用。
A preliminary version of this manuscript has been published in ECCV 2014. Based on this work, we attended the competition of ILSVRC 2014[26], and ranked #2 in object detection and #3 in image classification (bothare provided-data-only tracks) among all 38 teams. There are a few modifications made for ILSVRC 2014. We show that the SPP-nets can boost various networks that are deeper and larger (Sec. 3.1.2-3.1.4) over the no-SPP counterparts.Further, driven by our detection framework, we find that multi-view testing onfeature maps with flexibly located/sized windows (Sec. 3.1.5) can increase the classification accuracy. This manuscript also provides the details of these modifications.
We have released the code to facilitate future research(http://research.microsoft.com/ en-us/ um/ people/ kahe/).
本論文的一個早先版本發佈在ECCV2014上。基於這個工作，我們參加了ILSVRC 2014 [26]，在38個團隊中，取得了物體檢測第2名和圖像分類第3名的成績。針對ILSVRC 2014我們也做了很多修改。我們將展示SPP-nets可以將更深、更大的網絡的性能顯著提升。進一步，受檢測框架驅動，我們發現藉助靈活尺寸窗口對特徵圖進行多視角測試可以顯著提高分類精度。本文對這些改動做了更加詳細的說明。
另外，我們將代碼放在了以方便大家研究（http://research.microsoft.com/en-us/um/people/kahe/，譯者注：已失效）
2 DEEP NETWORKS WITH SPATIAL PYRA-MID POOLING
2.1 Convolutional Layers and Feature Maps Consider thepopular seven-layer architectures [3], [4].The first five layers are convolutional, some of which are followed by pooling layers. These pooling layers can also be considered as “convolutional”, in the sense that they are using sliding windows. The last two layers are fully connected, with an N-way softmax as the output, where N is the number of categories.
2. 基於空間金字塔池化的深度網絡
2.1 卷積層和特徵圖
在頗受歡迎的七層架構中[3][4]中，前五層是卷積層，其中一些後面跟着池化層。從他們也使用滑窗的角度來看，這些池化層也可以認爲是“卷積的”。最後兩層是全連接的，跟着一個N路softmax輸出，其中N是類別的數量。
The deep network described above needs a fixed imagesize. However, we notice that the requirement of fixed sizes is only due to the fully-connected layers that demand fixed-length vectors as inputs. On the other hand, the convolutional layers accept inputs of arbitrary sizes. The convolutional layers use sliding filters, and their outputs have roughly the same aspect ratio as the inputs. These outputs are known as feature maps [1] -they involve not only the strength of the responses, but also their spatial positions.
上述的深度網絡需要一個固定大小的圖像尺寸。然而，我們注意到，固定尺寸的要求僅僅是因爲全連接層的存在導致的。另一方面，卷積層使用滑動的特徵過濾器，它們的輸出基本保持了原始輸入的比例關係。它們的輸出就是特徵圖[1]-它們不僅涉及響應的強度，還包括空間位置。In Figure 2, we visualize some feature maps. They are generated by some filters of the conv layer. Figure 2(c) shows the strongest activated images of these filters in the ImageNet dataset. We see a filter can be activated by some semantic content. For example, the 55-th filter (Figure 2,bottom left) is most activated by a circle shape; the 66-th filter (Figure 2,top right) is most activated by a ∧-shape; and the 118-th filter (Figure 2, bottomright) is most activated by a ∨-shape.These shapes in the input images (Figure 2(a))activate the feature maps at the corresponding positions (the arrows in Figure2).
圖2中，我們可視化了一些特徵圖。這些特徵圖來自於conv5層的一些過濾器。圖2（c）顯示了ImageNet數據集中激活最強的若干圖像。可以看到一個過濾器能夠被一些語義內容激活。例如，第55個過濾器（圖2，左下）對圓形十分敏感；第66層（圖2，右上）對^形狀特別敏感；第118個過濾器（圖2，右下）對∨形狀非常敏感。這些輸入圖像中的形狀會激活相應位置的特徵圖（圖2中的箭頭）。
It is worth noticing that we generate the feature maps in Figure 2 without fixing the input size. These feature maps generated by deep convolutional layers are analogous to the feature maps in traditional methods[27], [28]. In those methods, SIFT vectors [29] or image patches [28] aredensely extracted and then encoded,e.g., by vector quantization [16], [15],[30],sparse coding [17], [18], or Fisher kernels [19].These encoded features consist of the feature maps,and are then pooled by Bag-of-Words (BoW) [16] or spatialpyramids [14], [15]. Analogously, the deep convolutional features can be pooled in a similar way.
值得注意的是，圖2中生成的特徵圖並沒有固定輸入尺寸。深度卷積層生成的特徵圖和傳統方法[27][28]中的特徵圖很相似。這些傳統方法中，SIFT向量[29]或圖像碎片[28]被密集地抽取出來，在通過矢量量化[16][15][30]，稀疏化[17][18]或Fisher核函數[19]進行編碼。這些編碼後的特徵構成了特徵圖，然後通過詞袋（BoW）[16]或空間金字塔[14][15]進行池化。類似的深度卷積的特徵也可以這樣做。
2.2 The Spatial Pyramid Pooling Layer
The convolutional layers accept arbitrary input sizes,but they produce outputs of variable sizes. The classifiers (SVM/softmax) or fully-connected layers require fixed-length vectors. Such vectors can be generated by the Bag-of-Words (BoW) approach [16] that pools the features together. Spatial pyramid pooling [14], [15] improves BoW in that it can maintain spatial information by pooling in local spatial bins. These spatialbins have sizes proportional to the image size, so the number of bins is fixed regardless of the image size. This is in contrast to the sliding window pooling of the previous deep networks [3], where the number of sliding windows depends on the input size.
2.2 空間金字塔池化層
To adopt the deep network for images of arbitrary sizes, we replace the last pooling layer (e.g.,pool5, after the last convolutional layer) with a spatial pyramid pooling layer.Figure 3 illustrates our method.In each spatial bin, we pool the responses of each filter (throughout this paper we use max pooling).The outputs of the spatial pyramid pooling are kM-dimensional vectors with the number of bins denoted as M(k is the number of filters in the last convolutional layer). The fixed-dimensional vectors are the input to the fully-connected layer.
爲了讓我們的神經網絡適應任意尺寸的圖像輸入，我們用一個空間金字塔池化層替換掉了最優一個池化層（最後一個卷積層之後的pool5）。圖3示例了這種方法。在每個空間塊中，我們池化每個過濾器的響應（本文中採用了最大池化法）。空間金字塔的輸出是一個kM維向量，M代表塊的數量，k代表最後一層卷積層的過濾器的數量。這個固定維度的向量就是全連接層的輸入。
With spatial pyramid pooling, the input image can 4 beof any sizes. This not only allows arbitrary aspect ratios, but also allows arbitrary scales. We can resize the input image to any scale (e.g.,min(w,h)=180,224,...) and apply the same deep network. When the input image is at different scales, the network (with the same filter sizes) will extract features at different scales. The scales play important roles in traditional methods,e.g.,the SIFT vectors are often extracted at multiple scales [29], [27] (determinedby the sizes of the patches and Gaussian filters). We will show that the scales are also important for the accuracy of deep networks.
有了空間金字塔池化，輸入圖像就可以是任意尺寸了。不但允許任意比例關係，而且支持任意縮放尺度。我們也可以將輸入圖像縮放到任意尺度(例如min(w;h)=180,224,…)並且使用同一個深度網絡。當輸入圖像處於不同的空間尺度時，帶有相同大小卷積核的網絡就可以在不同的尺度上抽取特徵。跨多個尺度在傳統方法中十分重要，比如SIFT向量就經常在多個尺度上進行抽取[29][27]（受碎片和高斯過濾器的大小所決定）。我們接下來會說明多尺度在深度網絡精度方面的重要作用。Interestingly, the coarsest pyramid level has a single bin that covers the entire image. This is in fact a “global pooling” operation,which is also investigated in several concurrent works. In [31], [32] a global average pooling is used to reduce the model size and also reduce overfitting; in [33],a global average pooling is used on the testing stage after all fc layers to improve accuracy; in [34], a global max pooling is used for weakly supervised object recognition. The global pooling operation corresponds to the traditional Bag-of-Words method.
有趣的是，粗糙的金字塔級別只有一個塊，覆蓋了整張圖像。這就是一個全局池化操作，當前有很多正在進行的工作正在研究它。[33]中，一個放在全連接層之後的全局平均池化被用來提高測試階段的精確度；[34]中，一個全局最大池化用於弱監督物體識別。全局池化操作相當於傳統的詞袋方法。
2.3 Training the Network
Theoretically, the above network structure can be trainedwith standard back-propagation [1], regardless of the input image size. But inpractice the GPU implementations (such as cuda-convnet[3] and Caffe[35]) are preferably run on fixed input images. Next we describe our training solution that takes advantage of these GPU implementations while still preserving the spatial pyramid pooling behaviors.
2.3 網絡的訓練
3 用於圖像分類的SPP-NET
3.1 ImageNet 2012分類實驗
3.1.1 基準網絡架構
3.1.2 多層次池化提升準確度
3.1.3 多尺寸訓練提升準確度
3.1.5 特徵圖上的多視圖測試
3.2 Experiments on VOC 2007 Classification
3.3 Experiments on Caltech101
4 SPP-NET用於物體檢測
4.1 檢測算法
4.2 檢測結果
表10中，我們進一步使用相同預訓練的SPPnet模型（ZF-5）和R-CNN進行比較。本例中，我們的方法和R-CNN有相當的平均成績。R-CNN的結果是通過預訓練模型進行提升的。這是因爲ZF-5比AlexNet有更好的架構，而且SPPnet是多層次池化（如果使用非SPP的ZF-5，R-CNN的結果就會下降）。表11表明了每個類別的結果。表也包含了其他方法。選擇性搜索（SS）[20]在SIFT特徵圖上應用空間金字塔匹配。DPM[23]和Regionlet[39]都是基於HOG特徵的[24]。Regionlet方法通過結合包含conv5的同步特徵可以提升到46.1%。DetectorNet[40]訓練一個深度網絡，可以輸出像素級的對象遮罩。這個方法僅僅需要對整張圖片應用深度網絡一次，和我們的方法一樣。但他們的方法mAP比較低(30.5%)。
4.3 複雜度和運行時間
4.4 用於檢測的多模型結合
4.5 ILSVRC 2014 Detection
5 結論
