Windows+caffe+vgg16訓練數據集分類

前言

網上有很多關於linux系統下，利用VGG16對圖片進行分類，但是運用到Windows下會出現很多問題，所以寫這篇博客做個總結。
關於配置caffe環境可以參考這篇博客win10+vs2013+caffe+gpu+python環境配置，在這裏就不在敘述了。下面將從以下5個步驟講起。
1.下載圖片，生成.txt標籤
2.把訓練圖片轉換成lmdb格式
3.計算圖片均值
4.修改網絡參數和網絡的圖片路徑等
5.訓練網絡

1 準備數據集

1.網上有很多數據集，可以下載這個數據集re.zip做爲參考，如果你要是訓練自己的數據集的話，可以模仿這個數據集製作，首先將你的數據集尺寸統一大小384*256（我有matlab代碼，想要的話可以私聊），然後以5:1比例分爲訓練集和測試集，訓練集命名爲train，測試集命名爲test，將數據拷貝到D:\caffe\caffe-master\data\re文件下（這是我caffe的路徑），如下圖所示。

2. 數據集準備好後，開始製作標籤。在D:/caffe/caffe-master/examples路徑下，新建文件夾myfile。
用python編寫代碼，直接運行即可

# -*- coding: utf-8 -*-

import os
data_path='D:/caffe/caffe-master/data/re/'      #數據集路徑
my='D:/caffe/caffe-master/examples/myfile/'    #文件輸出路徑
classes=[3,4,5,6,7]

def gen_txt(phase):
    f=open(my+phase+'.txt','w')
    for c in classes:
        folder=str(c)
        images=os.listdir(data_path+phase+'/')
        for img in images:
            f.write(phase+'/'+img+' '+folder+'\n')

gen_txt('train')
gen_txt('test')

程序運行後，會在你設置的路徑下，生成train.txt和test.txt文件，打開如下圖所示格式，表示你的標籤製作成功。

2. 將數據轉換成lmdb格式

這一步是最麻煩的，因爲網上很多教程都是基於linux系統的，要想運用到Windows下，在這我研究了好長時間。

新建txt文件，命名爲create_lmdb.sh，這裏的點是後綴，但是先彆着急把txt後綴去掉，因爲這樣寫代碼比較方便，然後寫入如下代碼

#!/usr/bin/env sh
MY=D:/caffe/caffe-master/examples/myfile                                         #文件輸出路徑

echo "Create train lmdb.."
rm -rf $MY/img_train_lmdb
D:/caffe/caffe-master/Build/x64/Release/convert_imageset \                       #caffe配置路徑
--shuffle \
--resize_height=256 \
--resize_width=256 \
D:/caffe/caffe-master/data/re/ \                                                  #數據集路徑
$MY/train.txt \
$MY/img_train_lmdb

echo "Create test lmdb.."
rm -rf $MY/img_test_lmdb
D:/caffe/caffe-master/Build/x64/Release/convert_imageset \                        #caffe配置路徑
--shuffle \
--resize_width=256 \
--resize_height=256 \
D:/caffe/caffe-master/data/re/ \                                                 #數據集路徑
$MY/test.txt \                                                                   #是test，不是val
$MY/img_test_lmdb                                                                #是test，不是val

echo "All Done.."

注意以上修改4個路徑，改爲你電腦的路徑。改完後，把漢字註釋去掉，因爲會影響運行，然後保存，把txt後綴改成sh。如下圖所示

然後運行程序，就是這最麻煩，因爲sh是linux系統命令，所以在運行程序前，先使用電腦管家的軟件管理，輸入git然後下載安裝，一直點next就行，安裝結束後，在環境變量中添加C:\Program Files\Git\bin路徑。這裏就不在介紹如何添加了路徑了。
添加完後，在程序中打開cmd命令窗口，然後cd到D:\caffe\caffe-master\examples\myfile路徑下，輸入指令 sh create_lmdb.sh
如下圖所示

如果在你的路徑下生成如下兩個文件夾，表示圖像數據已經成功製作成lmdb格式了。到這可以鬆一口氣了，因爲最難得已經過去了，後面的就很簡單了。

3.計算圖片均值

圖片減去均值再訓練，會提高訓練速度和精度。因此，一般都會有這個操作。
在D:\caffe\caffe-master\examples\imagenet路徑下找到make_imagenet_mean.sh文件拷貝到D:\caffe\caffe-master\examples\myfile路徑下，如下圖所示，

然後用記事本打開文件，修改代碼

#!/usr/bin/env sh
# Compute the mean image from the imagenet training lmdb
# N.B. this is available in data/ilsvrc12

EXAMPLE=D:/caffe/caffe-master/examples/myfile
DATA=D:/caffe/caffe-master/examples/myfile
TOOLS=D:/caffe/caffe-master/Build/x64/Release

$TOOLS/compute_image_mean $EXAMPLE/img_train_lmdb \
  $DATA/imagenet_mean.binaryproto

echo "Done."

EXAMPLE=D:/caffe/caffe-master/examples/myfile 爲lmdb文件目錄
DATA=D:/caffe/caffe-master/examples/myfile 生成文件所要存放的目錄
TOOLS=D:/caffe/caffe-master/Build/x64/Release caffe配置路徑

代碼修改後，在cmd命令窗口cd到D:\caffe\caffe-master\examples\myfile路徑下，然後輸入代碼 sh make_imagenet_mean.sh
如下圖所示

如果運行成功的話，會生成如下圖imagenet_mean.binaryproto文件

4.定義模型

在路徑D:\caffe\caffe-master\examples\myfile下新建vggnet_train_val.prototxt文件，用vs打開，複製以下代碼

name: "VGGNet"
layer {
  name: "data"
  type: "Data"
  top: "data"
  top: "label"
  include {
    phase: TRAIN
  }
  transform_param {
    crop_size: 224
    mean_value: 104
    mean_value: 117
    mean_value: 123
    mirror: true
  }
  data_param {
    source: "D:/caffe/caffe-master/examples/myfile/img_train_lmdb"    #注意訓練集文件的路徑
    batch_size: 20  #訓練批次大小根據自己的顯卡顯存而定
    backend: LMDB
  }
}
layer {
  name: "data"
  type: "Data"
  top: "data"
  top: "label"
  include {
    phase: TEST
  }
  transform_param {
    crop_size: 224
    mean_value: 104
    mean_value: 117
    mean_value: 123
    mirror: false
  }
  data_param {
    source: "D:/caffe/caffe-master/examples/myfile/img_test_lmdb" #注意驗證集文件的路徑
    batch_size: 10
    backend: LMDB
  }
}
layer {
  name: "conv1_1"
  type: "Convolution"
  bottom: "data"
  top: "conv1_1"
  param {
    lr_mult: 1
  }
  param {
    lr_mult: 1
  }
  convolution_param {
    num_output: 64
    kernel_size: 3
    pad: 1
    stride: 1
    weight_filler {
      type: "xavier"
    }
    bias_filler {
      type: "constant"
    }
  }
}
layer {
  name: "relu1_1"
  type: "ReLU"
  bottom: "conv1_1"
  top: "conv1_1"
}

layer {
  name: "conv1_2"
  type: "Convolution"
  bottom: "conv1_1"
  top: "conv1_2"
  param {
    lr_mult: 1
  }
  param {
    lr_mult: 1
  }
  convolution_param {
    num_output: 64
    kernel_size: 3
    pad: 1
    stride: 1
    weight_filler {
      type: "xavier"
    }
    bias_filler {
      type: "constant"
    }
  }
}
layer {
  name: "relu1_2"
  type: "ReLU"
  bottom: "conv1_2"
  top: "conv1_2"
}

layer {
  name: "pool1"
  type: "Pooling"
  bottom: "conv1_2"
  top: "pool1"
  pooling_param {
    pool: MAX
    kernel_size: 2
    stride: 2
  }
}

layer {
  name: "conv2_1"
  type: "Convolution"
  bottom: "pool1"
  top: "conv2_1"
  param {
    lr_mult: 1
  }
  param {
    lr_mult: 1
  }
  convolution_param {
    num_output: 128
    kernel_size: 3
    pad: 1
    stride: 1
    weight_filler {
      type: "xavier"
    }
    bias_filler {
      type: "constant"
    }
  }
}
layer {
  name: "relu2_1"
  type: "ReLU"
  bottom: "conv2_1"
  top: "conv2_1"
}
layer {
  name: "conv2_2"
  type: "Convolution"
  bottom: "conv2_1"
  top: "conv2_2"
  param {
    lr_mult: 1
  }
  param {
    lr_mult: 1
  }
  convolution_param {
    num_output: 128
    kernel_size: 3
    pad: 1
    stride: 1
    weight_filler {
      type: "xavier"
    }
    bias_filler {
      type: "constant"
    }
  }
}
layer {
  name: "relu2_2"
  type: "ReLU"
  bottom: "conv2_2"
  top: "conv2_2"
}

layer {
  name: "pool2"
  type: "Pooling"
  bottom: "conv2_2"
  top: "pool2"
  pooling_param {
    pool: MAX
    kernel_size: 2
    stride: 2
  }
}

layer {
  name: "conv3_1"
  type: "Convolution"
  bottom: "pool2"
  top: "conv3_1"
  param {
    lr_mult: 1
  }
  param {
    lr_mult: 1
  }
  convolution_param {
    num_output: 256
    kernel_size: 3
    pad: 1
    stride: 1
    weight_filler {
      type: "xavier"
    }
    bias_filler {
      type: "constant"
    }
  }
}
layer {
  name: "relu3_1"
  type: "ReLU"
  bottom: "conv3_1"
  top: "conv3_1"
}

layer {
  name: "conv3_2"
  type: "Convolution"
  bottom: "conv3_1"
  top: "conv3_2"
  param {
    lr_mult: 1
  }
  param {
    lr_mult: 1
  }
  convolution_param {
    num_output: 256
    kernel_size: 3
    pad: 1
    stride: 1
    weight_filler {
      type: "xavier"
    }
    bias_filler {
      type: "constant"
    }
  }
}
layer {
  name: "relu3_2"
  type: "ReLU"
  bottom: "conv3_2"
  top: "conv3_2"
}

layer {
  name: "conv3_3"
  type: "Convolution"
  bottom: "conv3_2"
  top: "conv3_3"
  param {
    lr_mult: 1
  }
  param {
    lr_mult: 1
  }
  convolution_param {
    num_output: 256
    kernel_size: 3
    pad: 1
    stride: 1
    weight_filler {
      type: "xavier"
    }
    bias_filler {
      type: "constant"
    }
  }
}
layer {
  name: "relu3_3"
  type: "ReLU"
  bottom: "conv3_3"
  top: "conv3_3"
}

layer {
  name: "pool3"
  type: "Pooling"
  bottom: "conv3_3"
  top: "pool3"
  pooling_param {
    pool: MAX
    kernel_size: 2
    stride: 2
  }
}

layer {
  name: "conv4_1"
  type: "Convolution"
  bottom: "pool3"
  top: "conv4_1"
  param {
    lr_mult: 1
  }
  param {
    lr_mult: 1
  }
  convolution_param {
    num_output: 512
    kernel_size: 3
    pad: 1
    stride: 1
    weight_filler {
      type: "xavier"
    }
    bias_filler {
      type: "constant"
    }
  }
}
layer {
  name: "relu4_1"
  type: "ReLU"
  bottom: "conv4_1"
  top: "conv4_1"
}

layer {
  name: "conv4_2"
  type: "Convolution"
  bottom: "conv4_1"
  top: "conv4_2"
  param {
    lr_mult: 1
  }
  param {
    lr_mult: 1
  }
  convolution_param {
    num_output: 512
    kernel_size: 3
    pad: 1
    stride: 1
    weight_filler {
      type: "xavier"
    }
    bias_filler {
      type: "constant"
    }
  }
}
layer {
  name: "relu4_2"
  type: "ReLU"
  bottom: "conv4_2"
  top: "conv4_2"
}

layer {
  name: "conv4_3"
  type: "Convolution"
  bottom: "conv4_2"
  top: "conv4_3"
  param {
    lr_mult: 1
  }
  param {
    lr_mult: 1
  }
  convolution_param {
    num_output: 512
    kernel_size: 3
    pad: 1
    stride: 1
    weight_filler {
      type: "xavier"
    }
    bias_filler {
      type: "constant"
    }
  }
}
layer {
  name: "relu4_3"
  type: "ReLU"
  bottom: "conv4_3"
  top: "conv4_3"
}

layer {
  name: "pool4"
  type: "Pooling"
  bottom: "conv4_3"
  top: "pool4"
  pooling_param {
    pool: MAX
    kernel_size: 2
    stride: 2
  }
}

layer {
  name: "conv5_1"
  type: "Convolution"
  bottom: "pool4"
  top: "conv5_1"
  param {
    lr_mult: 1
  }
  param {
    lr_mult: 1
  }
  convolution_param {
    num_output: 512
    kernel_size: 3
    pad: 1
    stride: 1
    weight_filler {
      type: "xavier"
    }
    bias_filler {
      type: "constant"
    }
  }
}
layer {
  name: "relu5_1"
  type: "ReLU"
  bottom: "conv5_1"
  top: "conv5_1"
}

layer {
  name: "conv5_2"
  type: "Convolution"
  bottom: "conv5_1"
  top: "conv5_2"
  param {
    lr_mult: 1
  }
  param {
    lr_mult: 1
  }
  convolution_param {
    num_output: 512
    kernel_size: 3
    pad: 1
    stride: 1
    weight_filler {
      type: "xavier"
    }
    bias_filler {
      type: "constant"
    }
  }
}
layer {
  name: "relu5_2"
  type: "ReLU"
  bottom: "conv5_2"
  top: "conv5_2"
}

layer {
  name: "conv5_3"
  type: "Convolution"
  bottom: "conv5_2"
  top: "conv5_3"
  param {
    lr_mult: 1
  }
  param {
    lr_mult: 1
  }
  convolution_param {
    num_output: 512
    kernel_size: 3
    pad: 1
    stride: 1
    weight_filler {
      type: "xavier"
    }
    bias_filler {
      type: "constant"
    }
  }
}
layer {
  name: "relu5_3"
  type: "ReLU"
  bottom: "conv5_3"
  top: "conv5_3"
}

layer {
  name: "pool5"
  type: "Pooling"
  bottom: "conv5_3"
  top: "pool5"
  pooling_param {
    pool: MAX
    kernel_size: 2
    stride: 2
  }
}

layer {
  name: "fc6"
  type: "InnerProduct"
  bottom: "pool5"
  top: "fc6"
  param {
    lr_mult: 1
  }
  param {
    lr_mult: 1
  }
  inner_product_param {
    num_output: 4096
    weight_filler {
      type: "xavier"
    }
    bias_filler {
      type: "constant"
    }
  }
}
layer {
  name: "relu6"
  type: "ReLU"
  bottom: "fc6"
  top: "fc6"
}
layer {
  name: "drop6"
  type: "Dropout"
  bottom: "fc6"
  top: "fc6"
  dropout_param {
    dropout_ratio: 0.5
  }
}
layer {
  name: "fc7"
  type: "InnerProduct"
  bottom: "fc6"
  top: "fc7"
  param {
    lr_mult: 1
  }
  param {
    lr_mult: 1
  }
  inner_product_param {
    num_output: 4096
    weight_filler {
      type: "xavier"
    }
    bias_filler {
      type: "constant"
    }
  }
}
layer {
  name: "relu7"
  type: "ReLU"
  bottom: "fc7"
  top: "fc7"
}
layer {
  name: "drop7"
  type: "Dropout"
  bottom: "fc7"
  top: "fc7"
  dropout_param {
    dropout_ratio: 0.5
  }
}
layer {
  name: "fc8"
  type: "InnerProduct"
  bottom: "fc7"
  top: "fc8"
  param {
    lr_mult: 1
  }
  param {
    lr_mult: 2
  }
  inner_product_param {
    num_output: 10              #注意將fc8層改成自己的圖像類別數目
    weight_filler {
      type: "xavier"
    }
    bias_filler {
      type: "constant"
    }
  }
}
layer {
  name: "accuracy"
  type: "Accuracy"
  bottom: "fc8"
  bottom: "label"
  top: "accuracy"
  include {
    phase: TEST
  }
}
layer {
  name: "loss"
  type: "SoftmaxWithLoss"
  bottom: "fc8"
  bottom: "label"
  top: "loss"
}

以上是VGG16模型，需改路徑和類別數後保存退出即可。
在當前路徑下新建文件vggnet_solver.prototxt，複製以下代碼，別忘了修改相對應的路徑

net: "D:/caffe/caffe-master/models/VGG16/vggnet_train_val.prototxt"                
test_iter: 10
test_interval: 500  #每經過500次訓練，進行一次驗證查看accuracy
base_lr: 0.001
lr_policy: "step"
gamma: 0.1
stepsize: 1000
display: 20
max_iter: 2000  #只是做做練習，2000次迭代就夠了
momentum: 0.9
weight_decay: 0.0005
snapshot: 1000  #每經過1000次迭代訓練保存一次快照
snapshot_prefix: "D:/caffe/caffe-master/examples/myfile/caffenet_train"    #在路徑下新建caffenet_train文件夾
solver_mode: GPU

5.訓練網絡

在當前路徑下新建train_vggnet.sh文件，同樣用記事本打開，複製以下代碼，同樣修改相對應的路徑

#!/usr/bin/env sh
set -e

D:/caffe/caffe-master/Build/x64/Release/caffe train \
    --solver=D:/caffe/caffe-master/models/VGG16/vggnet_solver.prototxt $@

修改後保存退出，然後在cmd命令窗口cd到當前路徑，然後輸入命令 sh train_vggnet.sh，如下圖所示

如果運行沒出錯的話，那恭喜你訓練成功，然後就是等待訓練結果就行

Windows+caffe+vgg16訓練數據集分類

前言

1 準備數據集

2. 將數據轉換成lmdb格式

3.計算圖片均值

4.定義模型

5.訓練網絡

致遠OA及相關OA系統集成與二次開發

EXCEL公式使用總結

System.Object未被引用的程序集中定義

Java 信號量（semaphore）搭配CountDownLatch 實現多線程處理循環內邏輯並限制創建線程數

[轉帖]linux命令top內存顯示M兆或者G

【面試準備】項目經驗——接口自動化項目

利用TensorBoad可視化網絡

LeetCode C++算法題持續跟新

劍指offer編程題持續跟新（c++）

計算機視覺/圖像算法/深度學習/AI算法崗位面試問題總結

ubuntu16.04+faster_rcnn+voc2007數據集訓練

https://yachay.unat.edu.pe/blog/index.php?comment_area=format_blog&comment_component=blog&comment_co

linux以太網驅動總結