該項目的所有代碼在我的github上,歡迎有興趣的同學與我探討研究~
地址:Machine-Learning/machine-learning-ex3/
1. Introduction
手寫數字識別(Hand-written digits recognition),顧名思義,就是將帶有手寫數字的圖片輸入到已經訓練過的機器,且機器能夠很快識別圖片中的手寫數字,並將之作爲輸出打印出來。
實現原理:
現以我個人的理解,來描述一下手寫數字識別的實現原理。
首先,要實現機器學習,前提是要有充足的數據。對於本實驗,要有充足的手寫數字圖片作爲訓練集進行模型的訓練;
其次,要能夠提取手寫數字圖片的特徵,這就涉及到了數字圖像處理。對於本實驗,我使用的是已經提取好圖片特徵的訓練集。每張手寫數字圖片的分辨率20x20,將其特徵值以一維的形式存在文件中,即訓練集中每個example 有400個features。我使用的訓練集一共有5000行,即有5000個手寫圖片。
接着,就是本次實驗的核心了— 模型的訓練。本實驗我分別使用兩個模型實現了手寫數字識別:一個是正規化後的邏輯迴歸模型,另一個是神經網絡模型。但是,只有邏輯迴歸模型我實現了完整的過程,即包括模型的訓練。對於神經網絡模型,我直接使用的是訓練好的參數。對於神經網絡的訓練,我在下一篇博文中會實現。
對於模型的訓練,可以通過梯度下降法或者其它優化函數來訓練。當Cost Function接近0時,得到訓練後的參數。
完成模型的訓練後,要進行預測準確率的計算。將訓練集中的圖片feature作爲特徵值輸入目標函數,得到預測的結果。然後將預測的結果與訓練集中圖片feature對應的真實結果進行比較,計算預測的準確率。
對於手寫數字的預測,涉及到的是一個multi-outcomes的邏輯迴歸函數,使用的是oneVsAll的方法,即定義多個邏輯迴歸分類器,每個邏輯迴歸分類器只有兩個結果:1和0。0~9都有對應的分類器,使得它們在分類器中的輸出爲1。所以,每輸入一張圖片的features,都會得到10個分類器的值,而這10個分類器中擁有最高值的,其對應的數字便是圖片中的數字,因爲只有在對應分類器中才能得到較高的評分。
當準確率達到一個較高值時,就可以進行手寫數字識別了。
以上便是手寫數字識別的實現原理
對於two-outcomes的邏輯迴歸問題, 只需要訓練一個邏輯迴歸分類器;
對於multi-outcomes的邏輯迴歸問題,需要使用oneVsAll的方法,即同時訓練多個邏輯迴歸分類器。
而神經網絡模型,對解決Non-linear Hypotheses非常有效。
對於該模型,以下幾點值得關注:
神經網絡能夠構造複雜的Non-linear Hypotheses的原因(層數越多,函數構造越複雜);
Adding all these intermediate layers in neural works allows us to more elegantly produce interesting and more complex non-linear hypotheses.
神經網絡的分層:輸入層、隱藏層(一層或多層)、輸出層;
神經網絡層與層的關係也是基於Sigmoid函數,稱爲Sigmoid activation function;
神經網絡層與層之間的參數theta,尤其是它的規格mxn;
神經網絡中的“bias unit”一般設置爲1;
神經網絡的呈現:一般化表示和向量化表示;
2. 邏輯迴歸模型實現手寫數字識別
主函數:
%% Initialization
clear ; close all; clc
%% Setup the parameters you will use for this part of the exercise
input_layer_size = 400; % 20x20 Input Images of Digits
num_labels = 10; % 10 labels, from 1 to 10
% (note that we have mapped "0" to label 10)
%% =========== Part 1: Loading and Visualizing Data ============
% Load Training Data
fprintf('Loading and Visualizing Data ...\n')
% 該數據文件裏存儲了Matrix X(5000x400), 向量y。其中X的每一行代表一張20x20的圖片
% 即X包含5000張圖片的數據
load('ex3data1.mat'); % training data stored in arrays X, y
m = size(X, 1);
% Randomly select 100 data points to display
% 把1到m這些數隨機打亂得到的一個數字序列,是一個行向量。
% 在數列中取100個數據,即取X中100行數據,也就是100張20x20的圖
rand_indices = randperm(m);
sel = X(rand_indices(1:100), :);
displayData(sel);
fprintf('Program paused. Press enter to continue.\n');
pause;
%% ============ Part 2a: Vectorize Logistic Regression ===========
% Test case for lrCostFunction
fprintf('\nTesting lrCostFunction() with regularization');
theta_t = [-2; -1; 1; 2];
X_t = [ones(5,1) reshape(1:15,5,3)/10];
y_t = ([1;0;1;0;1] >= 0.5);
lambda_t = 3;
[J grad] = lrCostFunction(theta_t, X_t, y_t, lambda_t);
fprintf('\nCost: %f\n', J);
fprintf('Expected cost: 2.534819\n');
fprintf('Gradients:\n');
fprintf(' %f \n', grad);
fprintf('Expected gradients:\n');
fprintf(' 0.146561\n -0.548558\n 0.724722\n 1.398003\n');
fprintf('Program paused. Press enter to continue.\n');
pause;
%% ============ Part 2b: One-vs-All Training ============
fprintf('\nTraining One-vs-All Logistic Regression...\n')
lambda = 0.1;
[all_theta] = oneVsAll(X, y, num_labels, lambda);
fprintf('Program paused. Press enter to continue.\n');
pause;
%% ================ Part 3: Predict for One-Vs-All ================
pred = predictOneVsAll(all_theta, X);
fprintf('\nTraining Set Accuracy: %f\n', mean(double(pred == y)) * 100);
Part1: 將訓練集中圖片的features轉爲圖片呈現出來(隨機取100張圖片)
displayData.m
function [h, display_array] = displayData(X, example_width)
%DISPLAYDATA Display 2D data in a nice grid
% [h, display_array] = DISPLAYDATA(X, example_width) displays 2D data
% stored in X in a nice grid. It returns the figure handle h and the
% displayed array if requested.
% Set example_width automatically if not passed in
if ~exist('example_width', 'var') || isempty(example_width)
example_width = round(sqrt(size(X, 2)));
end
% Gray Image
colormap(gray);
% Compute rows, cols
[m n] = size(X);
example_height = (n / example_width);
% Compute number of items to display
display_rows = floor(sqrt(m));
display_cols = ceil(m / display_rows);
% Between images padding
pad = 1;
% Setup blank display
display_array = - ones(pad + display_rows * (example_height + pad), ...
pad + display_cols * (example_width + pad));
% Copy each example into a patch on the display array
curr_ex = 1;
for j = 1:display_rows
for i = 1:display_cols
if curr_ex > m,
break;
end
% Copy the patch
% Get the max value of the patch
max_val = max(abs(X(curr_ex, :)));
display_array(pad + (j - 1) * (example_height + pad) + (1:example_height), ...
pad + (i - 1) * (example_width + pad) + (1:example_width)) = ...
reshape(X(curr_ex, :), example_height, example_width) / max_val;
curr_ex = curr_ex + 1;
end
if curr_ex > m,
break;
end
end
% Display Image
h = imagesc(display_array, [-1 1]);
% Do not show axis
axis image off
drawnow;
end
Part2a: 檢驗邏輯迴歸損失函數與梯度計算是否正確;
lrCostFunction.m
function [J, grad] = lrCostFunction(theta, X, y, lambda)
%LRCOSTFUNCTION Compute cost and gradient for logistic regression with
%regularization
% J = LRCOSTFUNCTION(theta, X, y, lambda) computes the cost of using
% theta as the parameter for regularized logistic regression and the
% gradient of the cost w.r.t. to the parameters.
% Initialize some useful values
m = length(y); % number of training examples
% You need to return the following variables correctly
J = 0;
grad = zeros(size(theta));
% ====================== YOUR CODE HERE ======================
% Calulate the function h, we know it's sigmoid function
h = sigmoid(X*theta);
% Calculate the cost function
J = 1/m*(-y'*log(h)-(1-y)'*log(1-h))+lambda/(2*m)*(sum(theta.^2)-theta(1)^2);
% Calculate the Gradient
grad = 1/m*X'*(h-y)+lambda/m*theta;
% It must start at 1
grad(1) = (1/m*X'*(h-y))(1);
Part2b:One-vs-All Training
模型的訓練:同時訓練10個邏輯迴歸分類器,使用一個for循環實現。
oneVsAll.m
function [all_theta] = oneVsAll(X, y, num_labels, lambda)
%ONEVSALL trains multiple logistic regression classifiers and returns all
%the classifiers in a matrix all_theta, where the i-th row of all_theta
%corresponds to the classifier for label i
% [all_theta] = ONEVSALL(X, y, num_labels, lambda) trains num_labels
% logistic regression classifiers and returns each of these classifiers
% in a matrix all_theta, where the i-th row of all_theta corresponds
% to the classifier for label i
% Some useful variables
m = size(X, 1);
n = size(X, 2);
% You need to return the following variables correctly
all_theta = zeros(num_labels, n + 1);
% Add ones to the X data matrix
X = [ones(m, 1) X];
% ====================== YOUR CODE HERE ======================
for c = 1:num_labels
% Set Initial theta
initial_theta = zeros(n+1, 1);
% Set options for fminunc
options = optimset('GradObj', 'on', 'MaxIter', 50);
% Run fmincg to obtain the optimal theta
% This function will return theta and the cost
% fmincg 也是優化方法中的一種,而且效率較之前的梯度下降法與fminunc更高
[theta] = ...
fmincg (@(t) (lrCostFunction(t, X, (y == c), lambda)), ...
initial_theta, option)
% 因爲theta是列向量,需要轉置一下存到all_theta的每一行
all_theta(c, :) = theta';
end
% =========================================================================
end
Part3: Predict for One-Vs-All
根據訓練得到的參數進行預測。
predictOneVsAll.m
function p = predictOneVsAll(all_theta, X)
%PREDICT Predict the label for a trained one-vs-all classifier. The labels
%are in the range 1..K, where K = size(all_theta, 1).
% p = PREDICTONEVSALL(all_theta, X) will return a vector of predictions
% for each example in the matrix X. Note that X contains the examples in
% rows. all_theta is a matrix where the i-th row is a trained logistic
% regression theta vector for the i-th class. You should set p to a vector
% of values from 1..K (e.g., p = [1; 3; 1; 2] predicts classes 1, 3, 1, 2
% for 4 examples)
m = size(X, 1);
num_labels = size(all_theta, 1);
% You need to return the following variables correctly
p = zeros(size(X, 1), 1);
% Add ones to the X data matrix
X = [ones(m, 1) X];
% ====================== YOUR CODE HERE ======================
% max(A) 返回每列最大值
% max(A, [], 1) 也是返回每列最大值
% max(A, [], 2) 返回每行最大值
% 將訓練集直接帶入假設函數
[maxmum, p] = max(sigmoid(X*all_theta'), [], 2);
% =========================================================================
end
輸出結果:
從輸出結果可以看到正規化邏輯迴歸函數能夠正常運行,還可以看到One-Vs-All 對每個分類器訓練的具體細節。我們能發現,在迭代了50次之後,每個邏輯迴歸分類器的cost都接近於0,雖然還沒有達到忽略不計的程度,但已經能夠有較高的預測準確率了。如果有更多的數據,肯定能得到更小的cost,更高的預測準確率。
這裏訓練後的預測準確率爲95.02%
3. 神經網絡模型實現手寫數字識別
主函數:
%% Initialization
clear ; close all; clc
%% Setup the parameters you will use for this exercise
input_layer_size = 400; % 20x20 Input Images of Digits
hidden_layer_size = 25; % 25 hidden units
num_labels = 10; % 10 labels, from 1 to 10
% (note that we have mapped "0" to label 10)
%% =========== Part 1: Loading and Visualizing Data =============
% We start the exercise by first loading and visualizing the dataset.
% You will be working with a dataset that contains handwritten digits.
%
% Load Training Data
fprintf('Loading and Visualizing Data ...\n')
load('ex3data1.mat');
m = size(X, 1);
% Randomly select 100 data points to display
sel = randperm(size(X, 1));
sel = sel(1:100);
displayData(X(sel, :));
fprintf('Program paused. Press enter to continue.\n');
pause;
%% ================ Part 2: Loading Pameters ================
% In this part of the exercise, we load some pre-initialized
% neural network parameters.
fprintf('\nLoading Saved Neural Network Parameters ...\n')
% Load the weights into variables Theta1 and Theta2
load('ex3weights.mat');
%% ================= Part 3: Implement Predict =================
% After training the neural network, we would like to use it to predict
% the labels. You will now implement the "predict" function to use the
% neural network to predict the labels of the training set. This lets
% you compute the training set accuracy.
pred = predict(Theta1, Theta2, X);
fprintf('\nTraining Set Accuracy: %f\n', mean(double(pred == y)) * 100);
fprintf('Program paused. Press enter to continue.\n');
pause;
% To give you an idea of the network's output, you can also run
% through the examples one at the a time to see what it is predicting.
% Randomly permute examples
rp = randperm(m);
for i = 1:m
% Display
fprintf('\nDisplaying Example Image\n');
displayData(X(rp(i), :));
pred = predict(Theta1, Theta2, X(rp(i),:));
fprintf('\nNeural Network Prediction: %d (digit %d)\n', pred, mod(pred, 10));
% Pause with quit option
s = input('Paused - press enter to continue, q to exit:','s');
if s == 'q'
break
end
end
Part1: 加載並可視化數據,與第二部分的part1一致
Part2: 加載權重,即加載神經網絡參數theta
這裏沒有涉及到神經網絡的訓練,而是直接使用已經訓練好的神經網絡參數。目的是對神經網絡的入門。在下篇博文就會完成神經網絡的訓練。
Part3: 利用神經網絡模型進行手寫數字預測
predict.m
function p = predict(Theta1, Theta2, X)
%PREDICT Predict the label of an input given a trained neural network
% p = PREDICT(Theta1, Theta2, X) outputs the predicted label of X given the
% trained weights of a neural network (Theta1, Theta2)
% Useful values
m = size(X, 1);
num_labels = size(Theta2, 1);
% You need to return the following variables correctly
p = zeros(size(X, 1), 1);
% ====================== YOUR CODE HERE ======================
X = [ones(m,1) X];
% 第二層
temp = sigmoid(X*Theta1');
% 第三層
temp1 = sigmoid([ones(size(temp,1), 1) temp]*Theta2');
[maxmum,p] = max(temp1, [], 2);
% =========================================================================
end
以下是輸出樣例的舉例(由於訓練的圖片是20x20的,放大後有點模糊):
識別樣例一:數字8
機器能準確識別數字8
機器能準確識別數字3
機器能準確識別數字5
輸出結果:
從輸出結果上看,訓練過的該神經網絡模型預測的準確率達到了97.52,還是十分不錯的。
4. Conclusion
現在來總結該項目的要點:
首先,要了解手寫數字識別的原理;
其次,要了解手寫數字識別實現的全過程;
然後,要了解oneVsAll以解決multi-outcomes的邏輯迴歸問題;
還有,要了解神經網絡的相關要點;
最後,還是要學會向量化!
以上內容皆爲本人觀點,歡迎大家提出批評和指導,我們一起探討!