博文參考standford UFLDL教程working with large images小節。
1、卷積特徵提取
之前做過的練習如sparse autoencoders、softmax regression、stacked autoencoders等處理的都是比較小的圖像，如8x8啊，28x28啊，那時用的是全聯通網絡（full connected networks），就是隱含層的每個單元都是與輸入層的全部單元連接的，如果圖像很大的話，比如96*96，那麼每個隱含層的單元都要有96*96個權重，如果要學習100個特徵的話，就有接近100w個權重了，這麼多權重參數學習速度會很慢，而且容易導致過擬合。一個解決辦法是使用部分聯通網絡（locally connected networks）。
部分聯通網絡的優點：
- 由於隱含單元與輸入單元的連接有了限制，每個隱含單元僅僅連接輸入層的一部分，要學習的參數大大減少；對於圖像而言，每個隱含單元僅僅連接圖像中的某一塊小區域；
- 這種部分聯通的網絡結構符合生物學裏面的視覺神經系統，視覺皮層的神經元是局部接受信息的（神經元只響應部分區域的刺激）。
用一個特徵矩陣從圖片中學習到卷積特徵，這個特徵矩陣只對圖片中大小相等的矩陣作卷積，所以特徵矩陣會從圖片的不同區域作卷積得到一個響應度矩陣即卷積特徵矩陣。如果圖片大小是rxc的話，有k個特徵矩陣，每個特徵矩陣大小爲axb，那麼每張圖片就可以學習到k個大小爲(r-a+1)x(c-b+1)的卷積特徵。
2、池化
我們卷積之後的特徵向量維數還是很大，用這樣的特徵去訓練分類器還是會過擬合，一種辦法是進行池化（pooling），池化是對卷積後的特徵進行聚合，可以有平均池化（mean pooling）和最大池化（max pooling），池化的作用是具有平移不變性，一個圖像某個區域平移到另一塊區域後，通過卷積特徵後池化還是具有一樣的效果的。
3、Exercise: Convolution and Pooling
該實驗是用之前Linear Decoders訓練出來的特徵來對數據進行卷積、池化的得到訓練集和測試集，然後用softmax訓練分類器，因爲在Linear Decoders中用的也是STL-10數據集，是8x8的RGB patches的特徵，這裏也是用STL-10數據集，是64x64的GRB圖片，可以進行8x8的卷積。
matlab基礎知識：
- randi([m n])返回[m,n]中的一個隨機整數；
- squeeze(a)是去掉a中維度大小爲1的維，比如a=rand(2,1,3)，squeeze(a)後變爲2x3，但是元素還是一樣的；
實驗重要函數說明：
- convolvedFeatures = cnnConvolve(patchDim, numFeatures, images, W, b, ZCAWhite, meanPatch);這個函數是求卷積特徵，卷積矩陣大小爲patchDim*patchDim,有numFeatures個特徵（隱含層單元數），images是要被卷積的圖片數據，W是權重矩陣，ZCAWhite是白化矩陣，b、meanPatch分別是長度爲patchDim*patchDim的偏置項、圖片patch平均值；
- pooledFeatures = cnnPool(poolDim, convolvedFeatures)；對poolDim*poolDim大小的矩陣進行池化。
要注意的是由於之前學特徵時用了白化處理，所以在進行卷積特徵提取時也要進行一樣的處理。
實驗結果：Accuracy: 80.281%
matlab代碼：
cnnExercise.m
%% CS294A/CS294W Convolutional Neural Networks Exercise
% Instructions
% ------------
%
% This file contains code that helps you get started on the
% convolutional neural networks exercise. In this exercise, you will only
% need to modify cnnConvolve.m and cnnPool.m. You will not need to modify
% this file.
%%======================================================================
%% STEP 0: Initialization
% Here we initialize some parameters used for the exercise.
imageDim = 64; % image dimension
imageChannels = 3; % number of channels (rgb, so 3)
patchDim = 8; % patch dimension
numPatches = 50000; % number of patches
visibleSize = patchDim * patchDim * imageChannels; % number of input units
outputSize = visibleSize; % number of output units
hiddenSize = 400; % number of hidden units
epsilon = 0.1; % epsilon for ZCA whitening
poolDim = 19; % dimension of pooling region
%%======================================================================
%% STEP 1: Train a sparse autoencoder (with a linear decoder) to learn
% features from color patches. If you have completed the linear decoder
% execise, use the features that you have obtained from that exercise,
% loading them into optTheta. Recall that we have to keep around the
% parameters used in whitening (i.e., the ZCA whitening matrix and the
% meanPatch)
% --------------------------- YOUR CODE HERE --------------------------
% Train the sparse autoencoder and fill the following variables with
% the optimal parameters:
optTheta = zeros(2*hiddenSize*visibleSize+hiddenSize+visibleSize, 1);
ZCAWhite = zeros(visibleSize, visibleSize);
meanPatch = zeros(visibleSize, 1);
%載入之前linear decoder練習中學到的參數
load 'STL10Features.mat'
% --------------------------------------------------------------------
% Display and check to see that the features look good
W = reshape(optTheta(1:visibleSize * hiddenSize), hiddenSize, visibleSize);
b = optTheta(2*hiddenSize*visibleSize+1:2*hiddenSize*visibleSize+hiddenSize);
displayColorNetwork( (W*ZCAWhite)');
%%======================================================================
%% STEP 2: Implement and test convolution and pooling
% In this step, you will implement convolution and pooling, and test them
% on a small part of the data set to ensure that you have implemented
% these two functions correctly. In the next step, you will actually
% convolve and pool the features with the STL10 images.
%% STEP 2a: Implement convolution
% Implement convolution in the function cnnConvolve in cnnConvolve.m
% Note that we have to preprocess the images in the exact same way
% we preprocessed the patches before we can obtain the feature activations.
load stlTrainSubset.mat % loads numTrainImages, trainImages, trainLabels
%% Use only the first 8 images for testing
convImages = trainImages(:, :, :, 1:8);
% NOTE: Implement cnnConvolve in cnnConvolve.m first!
convolvedFeatures = cnnConvolve(patchDim, hiddenSize, convImages, W, b, ZCAWhite, meanPatch);
%% STEP 2b: Checking your convolution
% To ensure that you have convolved the features correctly, we have
% provided some code to compare the results of your convolution with
% activations from the sparse autoencoder
% For 1000 random points
for i = 1:1000 %隨機挑選1000個patch進行驗證
featureNum = randi([1, hiddenSize]); %隨機選一個feature
imageNum = randi([1, 8]); %隨機選張圖
imageRow = randi([1, imageDim - patchDim + 1]); %隨機選valid的一行
imageCol = randi([1, imageDim - patchDim + 1]); %隨機選valid的一列
patch = convImages(imageRow:imageRow + patchDim - 1, imageCol:imageCol + patchDim - 1, :, imageNum); %取出那張圖RGB通道的那個patch
patch = patch(:); %組合成長向量
patch = patch - meanPatch; %預處理
patch = ZCAWhite * patch;
features = feedForwardAutoencoder(optTheta, hiddenSize, visibleSize, patch); %算出激活值
%與convoledFeatures比較是否相等
if abs(features(featureNum, 1) - convolvedFeatures(featureNum, imageNum, imageRow, imageCol)) > 1e-9
fprintf('Convolved feature does not match activation from autoencoder\n');
fprintf('Feature Number : %d\n', featureNum);
fprintf('Image Number : %d\n', imageNum);
fprintf('Image Row : %d\n', imageRow);
fprintf('Image Column : %d\n', imageCol);
fprintf('Convolved feature : %0.5f\n', convolvedFeatures(featureNum, imageNum, imageRow, imageCol));
fprintf('Sparse AE feature : %0.5f\n', features(featureNum, 1));
error('Convolved feature does not match activation from autoencoder');
end
end
disp('Congratulations! Your convolution code passed the test.');
%% STEP 2c: Implement pooling
% Implement pooling in the function cnnPool in cnnPool.m
% NOTE: Implement cnnPool in cnnPool.m first!
pooledFeatures = cnnPool(poolDim, convolvedFeatures);
%% STEP 2d: Checking your pooling
% To ensure that you have implemented pooling, we will use your pooling
% function to pool over a test matrix and check the results.
testMatrix = reshape(1:64, 8, 8);
expectedMatrix = [mean(mean(testMatrix(1:4, 1:4))) mean(mean(testMatrix(1:4, 5:8))); ...
mean(mean(testMatrix(5:8, 1:4))) mean(mean(testMatrix(5:8, 5:8))); ];
testMatrix = reshape(testMatrix, 1, 1, 8, 8);
pooledFeatures = squeeze(cnnPool(4, testMatrix)); %pool 4*4的矩陣
if ~isequal(pooledFeatures, expectedMatrix)
disp('Pooling incorrect');
disp('Expected');
disp(expectedMatrix);
disp('Got');
disp(pooledFeatures);
error('Pooled feature does not match expection.');
else
disp('Congratulations! Your pooling code passed the test.');
end
%%======================================================================
%% STEP 3: Convolve and pool with the dataset
% In this step, you will convolve each of the features you learned with
% the full large images to obtain the convolved features. You will then
% pool the convolved features to obtain the pooled features for
% classification.
%
% Because the convolved features matrix is very large, we will do the
% convolution and pooling 50 features at a time to avoid running out of
% memory. Reduce this number if necessary
stepSize = 50;
assert(mod(hiddenSize, stepSize) == 0, 'stepSize should divide hiddenSize');
load stlTrainSubset.mat % loads numTrainImages, trainImages, trainLabels
load stlTestSubset.mat % loads numTestImages, testImages, testLabels
pooledFeaturesTrain = zeros(hiddenSize, numTrainImages, ...
floor((imageDim - patchDim + 1) / poolDim), ...
floor((imageDim - patchDim + 1) / poolDim) );
pooledFeaturesTest = zeros(hiddenSize, numTestImages, ...
floor((imageDim - patchDim + 1) / poolDim), ...
floor((imageDim - patchDim + 1) / poolDim) );
tic();
%每次僅計算stepSize個特徵，之所以這樣是因爲卷積特徵矩陣太大了，爲了避免out of memory
for convPart = 1:(hiddenSize / stepSize)
featureStart = (convPart - 1) * stepSize + 1; %特徵起點
featureEnd = convPart * stepSize; %特徵終點
fprintf('Step %d: features %d to %d\n', convPart, featureStart, featureEnd);
Wt = W(featureStart:featureEnd, :); %取出特徵矩陣
bt = b(featureStart:featureEnd);
fprintf('Convolving and pooling train images\n');
%計算卷積特徵
convolvedFeaturesThis = cnnConvolve(patchDim, stepSize, ...
trainImages, Wt, bt, ZCAWhite, meanPatch);
pooledFeaturesThis = cnnPool(poolDim, convolvedFeaturesThis); %pooling
pooledFeaturesTrain(featureStart:featureEnd, :, :, :) = pooledFeaturesThis; %計算好的特徵放進去
toc();
clear convolvedFeaturesThis pooledFeaturesThis;
fprintf('Convolving and pooling test images\n');
convolvedFeaturesThis = cnnConvolve(patchDim, stepSize, ...
testImages, Wt, bt, ZCAWhite, meanPatch);
pooledFeaturesThis = cnnPool(poolDim, convolvedFeaturesThis);
pooledFeaturesTest(featureStart:featureEnd, :, :, :) = pooledFeaturesThis;
toc();
clear convolvedFeaturesThis pooledFeaturesThis;
end
% You might want to save the pooled features since convolution and pooling takes a long time
save('cnnPooledFeatures.mat', 'pooledFeaturesTrain', 'pooledFeaturesTest');
%load 'cnnPooledFeatures.mat';
toc();
%%======================================================================
%% STEP 4: Use pooled features for classification
% Now, you will use your pooled features to train a softmax classifier,
% using softmaxTrain from the softmax exercise.
% Training the softmax classifer for 1000 iterations should take less than
% 10 minutes.
% Add the path to your softmax solution, if necessary
% addpath /path/to/solution/
% Setup parameters for softmax
softmaxLambda = 1e-4;
numClasses = 4;
% Reshape the pooledFeatures to form an input vector for softmax
softmaxX = permute(pooledFeaturesTrain, [1 3 4 2]);
softmaxX = reshape(softmaxX, numel(pooledFeaturesTrain) / numTrainImages,...
numTrainImages);
softmaxY = trainLabels;
options = struct;
options.maxIter = 200;
softmaxModel = softmaxTrain(numel(pooledFeaturesTrain) / numTrainImages,...
numClasses, softmaxLambda, softmaxX, softmaxY, options);
%%======================================================================
%% STEP 5: Test classifer
% Now you will test your trained classifer against the test images
softmaxX = permute(pooledFeaturesTest, [1 3 4 2]);
softmaxX = reshape(softmaxX, numel(pooledFeaturesTest) / numTestImages, numTestImages);
softmaxY = testLabels;
[pred] = softmaxPredict(softmaxModel, softmaxX);
acc = (pred(:) == softmaxY(:));
acc = sum(acc) / size(acc, 1);
fprintf('Accuracy: %2.3f%%\n', acc * 100);
% You should expect to get an accuracy of around 80% on the test images.
cnnConvolve.m
function convolvedFeatures = cnnConvolve(patchDim, numFeatures, images, W, b, ZCAWhite, meanPatch)
%用稀疏自編碼學習到的W和b去卷積images
%W有400（numFeatures）行，每行是一個變換，可以對image進行卷積
%這樣每張圖片都能卷積到numFeatures個feature矩陣，叫feature map吧
%cnnConvolve Returns the convolution of the features given by W and b with
%the given images
%
% Parameters:
% patchDim - patch (feature) dimension
% numFeatures - number of features
% images - large images to convolve with, matrix in the form
% images(r, c, channel, image number)
% W, b - W, b for features from the sparse autoencoder
% ZCAWhite, meanPatch - ZCAWhitening and meanPatch matrices used for
% preprocessing
%
% Returns:
% convolvedFeatures - matrix of convolved features in the form
% convolvedFeatures(featureNum, imageNum, imageRow, imageCol)
numImages = size(images, 4); %樣本圖片
imageDim = size(images, 1); %圖片大小
imageChannels = size(images, 3); %圖片顏色通道
% 初始化images的卷積特徵矩陣，numImages張圖片，每張圖片numFeataures個卷積矩陣，每個矩陣size爲imageDim-patchDim+1
convolvedFeatures = zeros(numFeatures, numImages, imageDim - patchDim + 1, imageDim - patchDim + 1);
patchSize = patchDim * patchDim;
% Instructions:
% Convolve every feature with every large image here to produce the
% numFeatures x numImages x (imageDim - patchDim + 1) x (imageDim - patchDim + 1)
% matrix convolvedFeatures, such that
% convolvedFeatures(featureNum, imageNum, imageRow, imageCol) is the
% value of the convolved featureNum feature for the imageNum image over
% the region (imageRow, imageCol) to (imageRow + patchDim - 1, imageCol + patchDim - 1)
%
% Expected running times:
% Convolving with 100 images should take less than 3 minutes
% Convolving with 5000 images should take around an hour
% (So to save time when testing, you should convolve with less images, as
% described earlier)
% -------------------- YOUR CODE HERE --------------------
% Precompute the matrices that will be used during the convolution. Recall
% that you need to take into account the whitening and mean subtraction
% steps
%W是特徵矩陣，每行有patchDim*patchDim*3個元素
W = W*ZCAWhite;
b = b - W*meanPatch;
% --------------------------------------------------------
convolvedFeatures = zeros(numFeatures, numImages, imageDim - patchDim + 1, imageDim - patchDim + 1);
for imageNum = 1:numImages
for featureNum = 1:numFeatures
% convolution of image with feature matrix for each channel
convolvedImage = zeros(imageDim - patchDim + 1, imageDim - patchDim + 1);
for channel = 1:3
% Obtain the feature (patchDim x patchDim) needed during the convolution
% ---- YOUR CODE HERE ----
feature = zeros(8,8); % You should replace this
offset = (channel - 1) * patchSize; %patchSize = patchDim * patchDim
feature = reshape(W(featureNum, offset+1:offset+patchSize), patchDim, patchDim); %從W特徵矩陣中取出第featureNum個特徵的第channel個通道對應的特徵
% ------------------------
% Flip the feature matrix because of the definition of convolution, as explained later
feature = flipud(fliplr(squeeze(feature)));
% Obtain the image
im = squeeze(images(:, :, channel, imageNum));
% Convolve "feature" with "im", adding the result to convolvedImage
% be sure to do a 'valid' convolution
% ---- YOUR CODE HERE ----
convolvedImage = convolvedImage + conv2(im, feature, 'valid'); %把RGB通道的特徵響應加起來
% ------------------------
end
% Subtract the bias unit (correcting for the mean subtraction as well)
% Then, apply the sigmoid function to get the hidden activation
% ---- YOUR CODE HERE ----
convolvedImage = sigmoid(convolvedImage + b(featureNum));
% ------------------------
% The convolved feature is the sum of the convolved values for all channels
convolvedFeatures(featureNum, imageNum, :, :) = convolvedImage;
end
end
end
function sigm = sigmoid(x)
sigm = 1 ./ (1 + exp(-x));
end
cnnPool.m
function pooledFeatures = cnnPool(poolDim, convolvedFeatures)
%cnnPool Pools the given convolved features
%
% Parameters:
% poolDim - dimension of pooling region
% convolvedFeatures - convolved features to pool (as given by cnnConvolve)
% convolvedFeatures(featureNum, imageNum, imageRow, imageCol)
%
% Returns:
% pooledFeatures - matrix of pooled features in the form
% pooledFeatures(featureNum, imageNum, poolRow, poolCol)
%
numImages = size(convolvedFeatures, 2);
numFeatures = size(convolvedFeatures, 1);
convolvedDim = size(convolvedFeatures, 3);
pooledFeatures = zeros(numFeatures, numImages, floor(convolvedDim / poolDim), floor(convolvedDim / poolDim));
% -------------------- YOUR CODE HERE --------------------
% Instructions:
% Now pool the convolved features in regions of poolDim x poolDim,
% to obtain the
% numFeatures x numImages x (convolvedDim/poolDim) x (convolvedDim/poolDim)
% matrix pooledFeatures, such that
% pooledFeatures(featureNum, imageNum, poolRow, poolCol) is the
% value of the featureNum feature for the imageNum image pooled over the
% corresponding (poolRow, poolCol) pooling region
% (see http://ufldl/wiki/index.php/Pooling )
%
% Use mean pooling here.
% -------------------- YOUR CODE HERE --------------------
%對poolDim*poolDim的patch進行平均池化
numRows = convolvedDim / poolDim; %池化後總行數
numCols = convolvedDim / poolDim; %池化後總列數
for imageNum = 1:numImages
for featureNum = 1:numFeatures
for poolRow = 1:numRows
for poolCol = 1:numCols
pooledFeatures(featureNum, imageNum, poolRow, poolCol) = ...
mean(mean(convolvedFeatures(featureNum, imageNum, (poolRow-1)*poolDim+1:poolRow*poolDim, (poolCol-1)*poolDim+1:poolCol*poolDim)));
end
end
end
end
end