Deep learning：十五(Self-Taught Learning練習)

本文轉自http://www.cnblogs.com/tornadomeet/archive/2013/03/24/2979408.html

前言：

　　本次實驗主要是練習soft- taught learning的實現。參考的資料爲網頁：http://deeplearning.stanford.edu/wiki/index.php/Exercise:Self-Taught_Learning。Soft-taught leaning是用的無監督學習來學習到特徵提取的參數，然後用有監督學習來訓練分類器。這裏分別是用的sparse autoencoder和softmax regression。實驗的數據依舊是手寫數字數據庫MNIST Dataset.

　　實驗基礎：

　　從前面的知識可以知道，sparse autoencoder的輸出應該是和輸入數據尺寸大小一樣的，且很相近，那麼我們訓練出的sparse autoencoder模型該怎樣提取出特徵向量呢？其實輸入樣本經過sparse code提取出特徵的表達式就是隱含層的輸出了，首先來看看前面的經典sparse code模型，如下圖所示：

　　拿掉那個後面的輸出層後，隱含層的值就是我們所需要的特徵值了，如下圖所示：

　　從教程中可知，在unsupervised learning中有兩個觀點需要特別注意，一個是self-taught learning，一個是semi-supervised learning。Self-taught learning是完全無監督的。教程中有舉了個例子，很好的說明了這個問題，比如說我們需要設計一個系統來分類出轎車和摩托車。如果我們給出的訓練樣本圖片是自然界中隨便下載的（也就是說這些圖片中可能有轎車和摩托車，有可能都沒有，且大多數情況下是沒有的），然後使用的是這些樣本來特徵模型的話，那麼此時的方法就叫做self-taught learning。如果我們訓練的樣本圖片都是轎車和摩托車的圖片，只是我們不知道哪張圖對應哪種車，也就是說沒有標註，此時的方法不能叫做是嚴格的unsupervised feature，只能叫做是semi-supervised learning。

　　一些matlab函數：

　　numel:

　　比如說n = numel(A)表示返回矩陣A中元素的個數。

　　unique:

　　unique爲找出向量中的非重複元素並進行排序後輸出。

　　實驗結果：

　　採用數字5~9的樣本來進行無監督訓練，採用的方法是sparse autoencoder，可以提取出這些數據的權值，權值轉換成圖片顯示如下：

　　但是本次實驗主要是進行0~4這5個數字的分類，雖然進行無監督訓練用的是數字5~9的訓練樣本，這依然不會影響後面的結果。只是後面的分類器設計是用的softmax regression，所以是有監督的。最後據官網網頁上的結果精度是98%，而直接用原始的像素點進行分類器的設計不僅效果要差（才96%），而且訓練的速度也會變慢不少。

　　實驗主要部分代碼：

　　stlExercise.m:

%% CS294A/CS294W Self-taught Learning Exercise

%  Instructions
%  ------------
% 
%  This file contains code that helps you get started on the
%  self-taught learning. You will need to complete code in feedForwardAutoencoder.m
%  You will also need to have implemented sparseAutoencoderCost.m and 
%  softmaxCost.m from previous exercises.
%
%% ======================================================================
%  STEP 0: Here we provide the relevant parameters values that will
%  allow your sparse autoencoder to get good filters; you do not need to 
%  change the parameters below.

inputSize  = 28 * 28;
numLabels  = 5;
hiddenSize = 200;
sparsityParam = 0.1; % desired average activation of the hidden units.
                     % (This was denoted by the Greek alphabet rho, which looks like a lower-case "p",
                     %  in the lecture notes). 
lambda = 3e-3;       % weight decay parameter       
beta = 3;            % weight of sparsity penalty term   
maxIter = 400;

%% ======================================================================
%  STEP 1: Load data from the MNIST database
%
%  This loads our training and test data from the MNIST database files.
%  We have sorted the data for you in this so that you will not have to
%  change it.

% Load MNIST database files
mnistData   = loadMNISTImages('train-images.idx3-ubyte');
mnistLabels = loadMNISTLabels('train-labels.idx1-ubyte');

% Set Unlabeled Set (All Images)

% Simulate a Labeled and Unlabeled set
labeledSet   = find(mnistLabels >= 0 & mnistLabels <= 4);
unlabeledSet = find(mnistLabels >= 5);

%%增加的一行代碼
unlabeledSet = unlabeledSet(1:end/3);

numTest = round(numel(labeledSet)/2);%拿一半的樣本來訓練%
numTrain = round(numel(labeledSet)/3);
trainSet = labeledSet(1:numTrain);
testSet  = labeledSet(numTrain+1:2*numTrain);

unlabeledData = mnistData(:, unlabeledSet);%%爲什麼這兩句連在一起都要出錯呢？
% pack;
trainData   = mnistData(:, trainSet);
trainLabels = mnistLabels(trainSet)' + 1; % Shift Labels to the Range 1-5

% mnistData2 = mnistData;
testData   = mnistData(:, testSet);
testLabels = mnistLabels(testSet)' + 1;   % Shift Labels to the Range 1-5

% Output Some Statistics
fprintf('# examples in unlabeled set: %d\n', size(unlabeledData, 2));
fprintf('# examples in supervised training set: %d\n\n', size(trainData, 2));
fprintf('# examples in supervised testing set: %d\n\n', size(testData, 2));

%% ======================================================================
%  STEP 2: Train the sparse autoencoder
%  This trains the sparse autoencoder on the unlabeled training
%  images. 

%  Randomly initialize the parameters
theta = initializeParameters(hiddenSize, inputSize);

%% ----------------- YOUR CODE HERE ----------------------
%  Find opttheta by running the sparse autoencoder on
%  unlabeledTrainingImages

opttheta = theta; 
addpath minFunc/
options.Method = 'lbfgs';
options.maxIter = 400;
options.display = 'on';
[opttheta, loss] = minFunc( @(p) sparseAutoencoderLoss(p, ...
      inputSize, hiddenSize, ...
      lambda, sparsityParam, ...
      beta, unlabeledData), ...
      theta, options);


%% -----------------------------------------------------
                          
% Visualize weights
W1 = reshape(opttheta(1:hiddenSize * inputSize), hiddenSize, inputSize);
display_network(W1');

%%======================================================================
%% STEP 3: Extract Features from the Supervised Dataset
%  
%  You need to complete the code in feedForwardAutoencoder.m so that the 
%  following command will extract features from the data.

trainFeatures = feedForwardAutoencoder(opttheta, hiddenSize, inputSize, ...
                                       trainData);

testFeatures = feedForwardAutoencoder(opttheta, hiddenSize, inputSize, ...
                                       testData);

%%======================================================================
%% STEP 4: Train the softmax classifier

softmaxModel = struct;  
%% ----------------- YOUR CODE HERE ----------------------
%  Use softmaxTrain.m from the previous exercise to train a multi-class
%  classifier. 

%  Use lambda = 1e-4 for the weight regularization for softmax
lambda = 1e-4;
inputSize = hiddenSize;
numClasses = numel(unique(trainLabels));%unique爲找出向量中的非重複元素並進行排序

% You need to compute softmaxModel using softmaxTrain on trainFeatures and
% trainLabels


% You need to compute softmaxModel using softmaxTrain on trainFeatures and
% trainLabels

options.maxIter = 100;
softmaxModel = softmaxTrain(inputSize, numClasses, lambda, ...
                            trainFeatures, trainLabels, options);



%% -----------------------------------------------------


%%======================================================================
%% STEP 5: Testing 

%% ----------------- YOUR CODE HERE ----------------------
% Compute Predictions on the test set (testFeatures) using softmaxPredict
% and softmaxModel


[pred] = softmaxPredict(softmaxModel, testFeatures);


%% -----------------------------------------------------

% Classification Score
fprintf('Test Accuracy: %f%%\n', 100*mean(pred(:) == testLabels(:)));

% (note that we shift the labels by 1, so that digit 0 now corresponds to
%  label 1)
%
% Accuracy is the proportion of correctly classified images
% The results for our implementation was:
%
% Accuracy: 98.3%
%
%

　　feedForwardAutoencoder.m:

function [activation] = feedForwardAutoencoder(theta, hiddenSize, visibleSize, data)

% theta: trained weights from the autoencoder
% visibleSize: the number of input units (probably 64) 
% hiddenSize: the number of hidden units (probably 25) 
% data: Our matrix containing the training data as columns.  So, data(:,i) is the i-th training example. 
  
% We first convert theta to the (W1, W2, b1, b2) matrix/vector format, so that this 
% follows the notation convention of the lecture notes. 

W1 = reshape(theta(1:hiddenSize*visibleSize), hiddenSize, visibleSize);
b1 = theta(2*hiddenSize*visibleSize+1:2*hiddenSize*visibleSize+hiddenSize);

%% ---------- YOUR CODE HERE --------------------------------------
%  Instructions: Compute the activation of the hidden layer for the Sparse Autoencoder.
activation  = sigmoid(W1*data+repmat(b1,[1,size(data,2)]));

%-------------------------------------------------------------------

end

%-------------------------------------------------------------------
% Here's an implementation of the sigmoid function, which you may find useful
% in your computation of the costs and the gradients.  This inputs a (row or
% column) vector (say (z1, z2, z3)) and returns (f(z1), f(z2), f(z3)). 

function sigm = sigmoid(x)
    sigm = 1 ./ (1 + exp(-x));
end

　　參考資料:

http://deeplearning.stanford.edu/wiki/index.php/Exercise:Self-Taught_Learning

MNIST Dataset

Deep learning：十五(Self-Taught Learning練習)

進程與線程，TCP與UDP，虛函數面試題

winsocket學習筆記

Deep learning：十五(Self-Taught Learning練習)

判斷兩鏈表是否相交

mysql-load data local infile

https://yachay.unat.edu.pe/blog/index.php?comment_area=format_blog&comment_component=blog&comment_co

linux以太網驅動總結