caffe學習筆記4-- 手寫數字mnist訓練過程

通過mnist數據集的訓練，來看一下caffe的運行過程

這是caffe官網：examples裏的第二個例子，鏈接地址：http://caffe.berkeleyvision.org/gathered/examples/mnist.html

1. 準備數據

首先，我們需要從網上下載數據，在caffe根目錄下運行一下命令

./data/mnist/get_mnist.sh
./examples/mnist/create_mnist.sh

運行完後，會出現如下兩個文件：

mnist_train_lmdb, and mnist_test_lmdb

具體看一下這裏面寫了寫啥：

get_mnist.sh 用於獲取數據

#!/usr/bin/env sh
# This scripts downloads the mnist data and unzips it.

DIR="$( cd "$(dirname "$0")" ; pwd -P )"
cd $DIR

echo "Downloading..."　＃下載手寫數據集，http後面是手寫數據集的網址

wget --no-check-certificate http://yann.lecun.com/exdb/mnist/train-images-idx3-ubyte.gz
wget --no-check-certificate http://yann.lecun.com/exdb/mnist/train-labels-idx1-ubyte.gz
wget --no-check-certificate http://yann.lecun.com/exdb/mnist/t10k-images-idx3-ubyte.gz
wget --no-check-certificate http://yann.lecun.com/exdb/mnist/t10k-labels-idx1-ubyte.gz

echo "Unzipping..."　＃ 下載完成後解壓

gunzip train-images-idx3-ubyte.gz
gunzip train-labels-idx1-ubyte.gz
gunzip t10k-images-idx3-ubyte.gz
gunzip t10k-labels-idx1-ubyte.gz

# Creation is split out because leveldb sometimes causes segfault
# and needs to be re-created.

echo "Done."

/examples/mnist/create_mnist.sh　＃生成數據，這個腳本的作用是將數據轉換成lmdb格式或leveldb格式

具體如下：

#!/usr/bin/env sh
# This script converts the mnist data into lmdb/leveldb format,
# depending on the value assigned to $BACKEND.

EXAMPLE=examples/mnist
DATA=data/mnist
BUILD=build/examples/mnist

BACKEND="lmdb"

echo "Creating ${BACKEND}..."

rm -rf $EXAMPLE/mnist_train_${BACKEND}
rm -rf $EXAMPLE/mnist_test_${BACKEND}

$BUILD/convert_mnist_data.bin $DATA/train-images-idx3-ubyte \
  $DATA/train-labels-idx1-ubyte $EXAMPLE/mnist_train_${BACKEND} --backend=${BACKEND}　＃訓練數據
$BUILD/convert_mnist_data.bin $DATA/t10k-images-idx3-ubyte \
  $DATA/t10k-labels-idx1-ubyte $EXAMPLE/mnist_test_${BACKEND} --backend=${BACKEND}　＃測試數據

echo "Done."

convert_mnist_data.bin是由convert_minst_data.cpp編譯生成的可執行文件，這個編譯過程就是在caffe安裝的時候完成的，這個函數接受四個參數：

$DATA/train-images-idx3-ubyte: 手寫數字源文件
$DATA/train-labels-idx1-ubyte: 手寫數字標籤文件
$EXAMPLE/mnist_train_${BACKEND} : 轉換後數據的存儲位置
--backend=${BACKEND} : 宏定義，轉換數據的格式lmdb或leveldb

注：convert_mnist_data.cpp及用到的其他文件在下一篇中介紹

2. 網絡模型

這個實驗中使用的模型是LeNet,其模型結構如下：

這個網絡包含兩個卷積層，兩個池化層，兩個全連接層，最後一層用於分類

其結構定義在：$CAFFE_ROOT/examples/mnist/lenet_train_test.prototxt中,圖示如下

對於這個文檔的瞭解，需要查看caffe.proto相關的內容

首先，給出網絡名稱

name: "LeNet"

2.1. 數據層

接着，是數據層的寫入

這裏，我們從之前創建的lmdb文件夾中讀入數據

layer {
  name: "mnist"    //層名稱
  type: "Data"     // 層類型：數據層
  top: "data"       //該層產生兩個blobs,: data blobs
  top: "label"      //label blobs
  include {
    phase: TRAIN　// 僅用於訓練階段
  }
  transform_param {
    scale: 0.00390625　//將圖像的像素值歸一化，＝1/256
  }
  data_param {
    source: "examples/mnist/mnist_train_lmdb"   //數據所在的文件路徑
    batch_size: 64　//批尺寸，是每次迭代時輸入樣本的數量
    backend: LMDB　//輸入數據的類型爲lmdb
  }
}

文檔中還有另一個數據層，用於測試階段

2.2. 卷積層

layer {
  name: "conv1"
  type: "Convolution"      //層類型：　卷積層
  bottom: "data"     //該層的輸入爲data blobs, 由數據層產生
  top: "conv1"     //輸出爲卷積特徵conv1
  param {
    lr_mult: 1　　//權重w的學習率倍數
  }
  param {
    lr_mult: 2       //權重b的學習率倍數，
  }
  convolution_param {  //卷積參數
    num_output: 20　　//輸出爲20個特徵圖，其規模爲(data_size-kernel_size + stride)*(data_size -kernel_size + stride)
    kernel_size: 5    //卷積核爲5x5
    stride: 1    //卷積核的移動間隔爲１
    weight_filler {
      type: "xavier"  //權值初始化方法,這裏用xavier......
    }
    bias_filler {
      type: "constant"  //偏置初始化爲常量，默認爲０
    }
  }
}

2.3 池化層

layer {
  name: "pool1"
  type: "Pooling"　//層類型：池化層
  bottom: "conv1"　// 該層的輸入爲conv1的輸出結果
  top: "pool1"　//輸出爲pooling後的特徵
  pooling_param {
    pool: MAX　//pooling方式：最大值
    kernel_size: 2  //pooling核爲2x2
    stride: 2 //間隔爲2
  }
}

2.4　全連接層

layer {
  name: "ip1"
  type: "InnerProduct"
  bottom: "pool2"　//該層輸入爲上個池化層的輸出
  top: "ip1"    //輸出
  param {
    lr_mult: 1
  }
  param {
    lr_mult: 2
  }
  inner_product_param {
    num_output: 500　//500個輸出
    weight_filler {
      type: "xavier"
    }
    bias_filler {
      type: "constant"
    }
  }
}

2.5 非線性層

layer {
  name: "relu1"
  type: "ReLU"　//ReLU，限制線性單元，是一種激活函數，與sigmoid作用類似
  bottom: "ip1"　
  top: "ip1"
}

2.6 準確率層

layer {
  name: "accuracy"
  type: "Accuracy"
  bottom: "ip2"
  bottom: "label"
  top: "accuracy"
  include {
    phase: TEST  //僅用於測試數據，計算算法準確度
  }
}

2.7 損失估計層

layer {
  name: "loss"
  type: "SoftmaxWithLoss"　//多分類使用softMax迴歸計算損失
  bottom: "ip2"
  bottom: "label" //需要用到數據層產生的lable;
  top: "loss"
}

3.模型優化文件

這個文件在$CAFFE_ROOT/examples/mnist/lenet_solver.prototxt，配置一些參數信息等

文件內容如下：

# The train/test net protocol buffer definition
net: "examples/mnist/lenet_train_test.prototxt" #網絡模型文件路徑
# test_iter specifies how many forward passes the test should carry out.
# In the case of MNIST, we have test batch size 100 and 100 test iterations,
# covering the full 10,000 testing images.
test_iter: 100            #test的迭代次數，批處理大小爲100，　100*100爲測試集個數
# Carry out testing every 500 training iterations.
test_interval: 500         #訓練時每迭代500次測試一次
# The base learning rate, momentum and the weight decay of the network.
base_lr: 0.01        #學習率
momentum: 0.9      #動量
weight_decay: 0.0005       #權重衰減
# The learning rate policy#學習率策略
lr_policy: "inv" 
gamma: 0.0001
power: 0.75
# Display every 100 iterations     #每迭代100次顯示
display: 100
# The maximum number of iterations
max_iter: 10000　#最大迭代次數
# snapshot intermediate results#我感覺可能是每迭代5000次存儲一次參數什麼的。
snapshot: 5000
snapshot_prefix: "examples/mnist/lenet"   
# solver mode: CPU or GPU
solver_mode: GPU     #使用GPU 訓練

4. 訓練數據

在caffe根目錄下輸入：

./examples/mnist/train_lenet.sh

首先，程序讀取lenet_solver.prototxt和lenet_train_test.prototxt這兩個配置文件

sindyz@sindyz-desktop:~/caffe-master$ ./examples/mnist/train_lenet.sh
I0112 13:53:13.997781  3792 caffe.cpp:184] Using GPUs 0
I0112 13:53:14.093080  3792 solver.cpp:48] Initializing solver from parameters: 
test_iter: 100
test_interval: 500
base_lr: 0.01
display: 100
max_iter: 10000
lr_policy: "inv"
gamma: 0.0001
power: 0.75
momentum: 0.9
weight_decay: 0.0005
snapshot: 5000
snapshot_prefix: "examples/mnist/lenet"
solver_mode: GPU
device_id: 0
net: "examples/mnist/lenet_train_test.prototxt"
I0112 13:53:14.093252  3792 solver.cpp:91] Creating training net from net file: examples/mnist/lenet_train_test.prototxt
I0112 13:53:14.093545  3792 net.cpp:322] The NetState phase (0) differed from the phase (1) specified by a rule in layer mnist
I0112 13:53:14.093574  3792 net.cpp:322] The NetState phase (0) differed from the phase (1) specified by a rule in layer accuracy
I0112 13:53:14.093646  3792 net.cpp:49] Initializing net from parameters: 
name: "LeNet"
..........

之後，緊跟着創建每層網絡，這裏沒有一一貼出。只給出了一部分

4.1 數據層

I0112 13:53:14.093978  3792 layer_factory.hpp:77] Creating layer mnist　//創建數據層，文件中這層命名爲mnist
I0112 13:53:14.094444  3792 net.cpp:106] Creating Layer mnist
I0112 13:53:14.094468  3792 net.cpp:411] mnist -> data
I0112 13:53:14.094490  3792 net.cpp:411] mnist -> label
I0112 13:53:14.095192  3797 db_lmdb.cpp:38] Opened lmdb examples/mnist/mnist_train_lmdb
I0112 13:53:14.100249  3792 data_layer.cpp:41] output data size: 64,1,28,28   //輸出data blobs（number, channel, width, height）
I0112 13:53:14.101164  3792 net.cpp:150] Setting up mnist
I0112 13:53:14.101198  3792 net.cpp:157] Top shape: 64 1 28 28 (50176)
I0112 13:53:14.101207  3792 net.cpp:157] Top shape: 64 (64)
I0112 13:53:14.101213  3792 net.cpp:165] Memory required for data: 200960

4.2 卷積層（conv1）

I0112 13:53:14.101223  3792 layer_factory.hpp:77] Creating layer conv1　
I0112 13:53:14.101245  3792 net.cpp:106] Creating Layer conv1
I0112 13:53:14.101253  3792 net.cpp:454] conv1 <- data
I0112 13:53:14.101269  3792 net.cpp:411] conv1 -> conv1
I0112 13:53:14.101812  3792 net.cpp:150] Setting up conv1
I0112 13:53:14.101837  3792 net.cpp:157] Top shape: 64 20 24 24 (737280)
I0112 13:53:14.101845  3792 net.cpp:165] Memory required for data: 3150080

4.3 池化層(pooling1)

I0112 13:53:14.101857  3792 layer_factory.hpp:77] Creating layer pool1
I0112 13:53:14.101872  3792 net.cpp:106] Creating Layer pool1
I0112 13:53:14.101878  3792 net.cpp:454] pool1 <- conv1
I0112 13:53:14.101886  3792 net.cpp:411] pool1 -> pool1
I0112 13:53:14.101948  3792 net.cpp:150] Setting up pool1
I0112 13:53:14.101958  3792 net.cpp:157] Top shape: 64 20 12 12 (184320)
I0112 13:53:14.101963  3792 net.cpp:165] Memory required for data: 3887360

4.4 全連接層(ip1)

I0112 13:53:14.102401  3792 layer_factory.hpp:77] Creating layer ip1
I0112 13:53:14.102416  3792 net.cpp:106] Creating Layer ip1
I0112 13:53:14.102421  3792 net.cpp:454] ip1 <- pool2
I0112 13:53:14.102428  3792 net.cpp:411] ip1 -> ip1
I0112 13:53:14.105073  3792 net.cpp:150] Setting up ip1
I0112 13:53:14.105095  3792 net.cpp:157] Top shape: 64 500 (32000)
I0112 13:53:14.105101  3792 net.cpp:165] Memory required for data: 5039360

4.5 非線性層

I0112 13:53:14.105113  3792 layer_factory.hpp:77] Creating layer relu1
I0112 13:53:14.105124  3792 net.cpp:106] Creating Layer relu1
I0112 13:53:14.105130  3792 net.cpp:454] relu1 <- ip1
I0112 13:53:14.105139  3792 net.cpp:397] relu1 -> ip1 (in-place)
I0112 13:53:14.105152  3792 net.cpp:150] Setting up relu1
I0112 13:53:14.105161  3792 net.cpp:157] Top shape: 64 500 (32000)
I0112 13:53:14.105166  3792 net.cpp:165] Memory required for data: 5167360

4.6 損失層

I0112 13:53:14.105664  3792 layer_factory.hpp:77] Creating layer loss
I0112 13:53:14.105677  3792 net.cpp:106] Creating Layer loss
I0112 13:53:14.105685  3792 net.cpp:454] loss <- ip2
I0112 13:53:14.105692  3792 net.cpp:454] loss <- label
I0112 13:53:14.105700  3792 net.cpp:411] loss -> loss

接下來，開始計算反饋網絡，（內容不在一一貼出，運行是會看到）

之後，輸出測試網絡和創建過程

這一步完成後，開始優化參數

I0112 13:53:14.112908  3792 net.cpp:283] Network initialization done.
I0112 13:53:14.112957  3792 solver.cpp:60] Solver scaffolding done.
I0112 13:53:14.113140  3792 caffe.cpp:212] Starting Optimization
I0112 13:53:14.113149  3792 solver.cpp:288] Solving LeNet
I0112 13:53:14.113154  3792 solver.cpp:289] Learning Rate Policy: inv
I0112 13:53:14.113476  3792 solver.cpp:341] Iteration 0, Testing net (#0)
I0112 13:53:15.875176  3792 solver.cpp:409]     Test net output #0: accuracy = 0.0861
I0112 13:53:15.875221  3792 solver.cpp:409]     Test net output #1: loss = 2.34769 (* 1 = 2.34769 loss)
I0112 13:53:15.889724  3792 solver.cpp:237] Iteration 0, loss = 2.33398

當代碼運行到這裏時，進入迭代優化過程

待運行結束，輸出結果：

I0112 13:59:25.476456  3792 solver.cpp:409]     Test net output #0: accuracy = 0.9908
I0112 13:59:25.476516  3792 solver.cpp:409]     Test net output #1: loss = 0.02798 (* 1 = 0.02798 loss)
I0112 13:59:25.476526  3792 solver.cpp:326] Optimization Done.
I0112 13:59:25.476531  3792 caffe.cpp:215] Optimization Done.

其中，#0:爲準確度，　#1:爲損失

下一篇，將介紹一下mnist中的cpp代碼部分

參考資料：

http://caffe.berkeleyvision.org/gathered/examples/mnist.html

caffe初學，把邊學的東西記下來，希望大家多多指導。

caffe學習筆記4-- 手寫數字mnist訓練過程

caffe學習筆記1.1-- caffe的Makefile文件

caffe學習筆記9 -- Brewing Logistic Regression then Going Deeper

caffe中matlab接口配置

caffe學習筆記11 -- Net Surgery

caffe學習筆記3.1 -- caffe的三級結構

Mac下配置sublime實現LaTeX

https://yachay.unat.edu.pe/blog/index.php?comment_area=format_blog&comment_component=blog&comment_co

linux以太網驅動總結