Windows+caffe+vgg16訓練數據集分類

前言

網上有很多關於linux系統下,利用VGG16對圖片進行分類,但是運用到Windows下會出現很多問題,所以寫這篇博客做個總結。
關於配置caffe環境可以參考這篇博客win10+vs2013+caffe+gpu+python環境配置,在這裏就不在敘述了。下面將從以下5個步驟講起。
1.下載圖片,生成.txt標籤
2.把訓練圖片轉換成lmdb格式
3.計算圖片均值
4.修改網絡參數和網絡的圖片路徑等
5.訓練網絡

1 準備數據集

1.網上有很多數據集,可以下載這個數據集re.zip做爲參考,如果你要是訓練自己的數據集的話,可以模仿這個數據集製作,首先將你的數據集尺寸統一大小384*256(我有matlab代碼,想要的話可以私聊),然後以5:1比例分爲訓練集和測試集,訓練集命名爲train,測試集命名爲test,將數據拷貝到D:\caffe\caffe-master\data\re文件下(這是我caffe的路徑),如下圖所示。
在這裏插入圖片描述
2. 數據集準備好後,開始製作標籤。在D:/caffe/caffe-master/examples路徑下,新建文件夾myfile。
用python編寫代碼,直接運行即可

# -*- coding: utf-8 -*-

import os
data_path='D:/caffe/caffe-master/data/re/'      #數據集路徑
my='D:/caffe/caffe-master/examples/myfile/'    #文件輸出路徑
classes=[3,4,5,6,7]

def gen_txt(phase):
    f=open(my+phase+'.txt','w')
    for c in classes:
        folder=str(c)
        images=os.listdir(data_path+phase+'/')
        for img in images:
            f.write(phase+'/'+img+' '+folder+'\n')

gen_txt('train')
gen_txt('test')

程序運行後,會在你設置的路徑下,生成train.txt和test.txt文件,打開如下圖所示格式,表示你的標籤製作成功。
在這裏插入圖片描述

2. 將數據轉換成lmdb格式

這一步是最麻煩的,因爲網上很多教程都是基於linux系統的,要想運用到Windows下,在這我研究了好長時間。

  1. 新建txt文件,命名爲create_lmdb.sh,這裏的點是後綴,但是先彆着急把txt後綴去掉,因爲這樣寫代碼比較方便,然後寫入如下代碼
#!/usr/bin/env sh
MY=D:/caffe/caffe-master/examples/myfile                                         #文件輸出路徑

echo "Create train lmdb.."
rm -rf $MY/img_train_lmdb
D:/caffe/caffe-master/Build/x64/Release/convert_imageset \                       #caffe配置路徑
--shuffle \
--resize_height=256 \
--resize_width=256 \
D:/caffe/caffe-master/data/re/ \                                                  #數據集路徑
$MY/train.txt \
$MY/img_train_lmdb

echo "Create test lmdb.."
rm -rf $MY/img_test_lmdb
D:/caffe/caffe-master/Build/x64/Release/convert_imageset \                        #caffe配置路徑
--shuffle \
--resize_width=256 \
--resize_height=256 \
D:/caffe/caffe-master/data/re/ \                                                 #數據集路徑
$MY/test.txt \                                                                   #是test,不是val
$MY/img_test_lmdb                                                                #是test,不是val

echo "All Done.."

注意以上修改4個路徑,改爲你電腦的路徑。改完後,把漢字註釋去掉,因爲會影響運行,然後保存,把txt後綴改成sh。如下圖所示
在這裏插入圖片描述
然後運行程序,就是這最麻煩,因爲sh是linux系統命令,所以在運行程序前,先使用電腦管家的軟件管理,輸入git然後下載安裝,一直點next就行,安裝結束後,在環境變量中添加C:\Program Files\Git\bin路徑。這裏就不在介紹如何添加了路徑了。
添加完後,在程序中打開cmd命令窗口,然後cd到D:\caffe\caffe-master\examples\myfile路徑下,輸入指令 sh create_lmdb.sh
如下圖所示
在這裏插入圖片描述
如果在你的路徑下生成如下兩個文件夾,表示圖像數據已經成功製作成lmdb格式了。到這可以鬆一口氣了,因爲最難得已經過去了,後面的就很簡單了。
在這裏插入圖片描述

3.計算圖片均值

圖片減去均值再訓練,會提高訓練速度和精度。因此,一般都會有這個操作。
在D:\caffe\caffe-master\examples\imagenet路徑下找到make_imagenet_mean.sh文件拷貝到D:\caffe\caffe-master\examples\myfile路徑下,如下圖所示,
在這裏插入圖片描述
然後用記事本打開文件,修改代碼

#!/usr/bin/env sh
# Compute the mean image from the imagenet training lmdb
# N.B. this is available in data/ilsvrc12

EXAMPLE=D:/caffe/caffe-master/examples/myfile
DATA=D:/caffe/caffe-master/examples/myfile
TOOLS=D:/caffe/caffe-master/Build/x64/Release

$TOOLS/compute_image_mean $EXAMPLE/img_train_lmdb \
  $DATA/imagenet_mean.binaryproto

echo "Done."

EXAMPLE=D:/caffe/caffe-master/examples/myfile 爲lmdb文件目錄
DATA=D:/caffe/caffe-master/examples/myfile 生成文件所要存放的目錄
TOOLS=D:/caffe/caffe-master/Build/x64/Release caffe配置路徑

代碼修改後,在cmd命令窗口cd到D:\caffe\caffe-master\examples\myfile路徑下,然後輸入代碼 sh make_imagenet_mean.sh
如下圖所示
在這裏插入圖片描述
如果運行成功的話,會生成如下圖imagenet_mean.binaryproto文件
在這裏插入圖片描述

4.定義模型

在路徑D:\caffe\caffe-master\examples\myfile下新建vggnet_train_val.prototxt文件,用vs打開,複製以下代碼

name: "VGGNet"
layer {
  name: "data"
  type: "Data"
  top: "data"
  top: "label"
  include {
    phase: TRAIN
  }
  transform_param {
    crop_size: 224
    mean_value: 104
    mean_value: 117
    mean_value: 123
    mirror: true
  }
  data_param {
    source: "D:/caffe/caffe-master/examples/myfile/img_train_lmdb"    #注意訓練集文件的路徑
    batch_size: 20  #訓練批次大小根據自己的顯卡顯存而定
    backend: LMDB
  }
}
layer {
  name: "data"
  type: "Data"
  top: "data"
  top: "label"
  include {
    phase: TEST
  }
  transform_param {
    crop_size: 224
    mean_value: 104
    mean_value: 117
    mean_value: 123
    mirror: false
  }
  data_param {
    source: "D:/caffe/caffe-master/examples/myfile/img_test_lmdb" #注意驗證集文件的路徑
    batch_size: 10
    backend: LMDB
  }
}
layer {
  name: "conv1_1"
  type: "Convolution"
  bottom: "data"
  top: "conv1_1"
  param {
    lr_mult: 1
  }
  param {
    lr_mult: 1
  }
  convolution_param {
    num_output: 64
    kernel_size: 3
    pad: 1
    stride: 1
    weight_filler {
      type: "xavier"
    }
    bias_filler {
      type: "constant"
    }
  }
}
layer {
  name: "relu1_1"
  type: "ReLU"
  bottom: "conv1_1"
  top: "conv1_1"
}

layer {
  name: "conv1_2"
  type: "Convolution"
  bottom: "conv1_1"
  top: "conv1_2"
  param {
    lr_mult: 1
  }
  param {
    lr_mult: 1
  }
  convolution_param {
    num_output: 64
    kernel_size: 3
    pad: 1
    stride: 1
    weight_filler {
      type: "xavier"
    }
    bias_filler {
      type: "constant"
    }
  }
}
layer {
  name: "relu1_2"
  type: "ReLU"
  bottom: "conv1_2"
  top: "conv1_2"
}

layer {
  name: "pool1"
  type: "Pooling"
  bottom: "conv1_2"
  top: "pool1"
  pooling_param {
    pool: MAX
    kernel_size: 2
    stride: 2
  }
}

layer {
  name: "conv2_1"
  type: "Convolution"
  bottom: "pool1"
  top: "conv2_1"
  param {
    lr_mult: 1
  }
  param {
    lr_mult: 1
  }
  convolution_param {
    num_output: 128
    kernel_size: 3
    pad: 1
    stride: 1
    weight_filler {
      type: "xavier"
    }
    bias_filler {
      type: "constant"
    }
  }
}
layer {
  name: "relu2_1"
  type: "ReLU"
  bottom: "conv2_1"
  top: "conv2_1"
}
layer {
  name: "conv2_2"
  type: "Convolution"
  bottom: "conv2_1"
  top: "conv2_2"
  param {
    lr_mult: 1
  }
  param {
    lr_mult: 1
  }
  convolution_param {
    num_output: 128
    kernel_size: 3
    pad: 1
    stride: 1
    weight_filler {
      type: "xavier"
    }
    bias_filler {
      type: "constant"
    }
  }
}
layer {
  name: "relu2_2"
  type: "ReLU"
  bottom: "conv2_2"
  top: "conv2_2"
}

layer {
  name: "pool2"
  type: "Pooling"
  bottom: "conv2_2"
  top: "pool2"
  pooling_param {
    pool: MAX
    kernel_size: 2
    stride: 2
  }
}

layer {
  name: "conv3_1"
  type: "Convolution"
  bottom: "pool2"
  top: "conv3_1"
  param {
    lr_mult: 1
  }
  param {
    lr_mult: 1
  }
  convolution_param {
    num_output: 256
    kernel_size: 3
    pad: 1
    stride: 1
    weight_filler {
      type: "xavier"
    }
    bias_filler {
      type: "constant"
    }
  }
}
layer {
  name: "relu3_1"
  type: "ReLU"
  bottom: "conv3_1"
  top: "conv3_1"
}

layer {
  name: "conv3_2"
  type: "Convolution"
  bottom: "conv3_1"
  top: "conv3_2"
  param {
    lr_mult: 1
  }
  param {
    lr_mult: 1
  }
  convolution_param {
    num_output: 256
    kernel_size: 3
    pad: 1
    stride: 1
    weight_filler {
      type: "xavier"
    }
    bias_filler {
      type: "constant"
    }
  }
}
layer {
  name: "relu3_2"
  type: "ReLU"
  bottom: "conv3_2"
  top: "conv3_2"
}

layer {
  name: "conv3_3"
  type: "Convolution"
  bottom: "conv3_2"
  top: "conv3_3"
  param {
    lr_mult: 1
  }
  param {
    lr_mult: 1
  }
  convolution_param {
    num_output: 256
    kernel_size: 3
    pad: 1
    stride: 1
    weight_filler {
      type: "xavier"
    }
    bias_filler {
      type: "constant"
    }
  }
}
layer {
  name: "relu3_3"
  type: "ReLU"
  bottom: "conv3_3"
  top: "conv3_3"
}

layer {
  name: "pool3"
  type: "Pooling"
  bottom: "conv3_3"
  top: "pool3"
  pooling_param {
    pool: MAX
    kernel_size: 2
    stride: 2
  }
}

layer {
  name: "conv4_1"
  type: "Convolution"
  bottom: "pool3"
  top: "conv4_1"
  param {
    lr_mult: 1
  }
  param {
    lr_mult: 1
  }
  convolution_param {
    num_output: 512
    kernel_size: 3
    pad: 1
    stride: 1
    weight_filler {
      type: "xavier"
    }
    bias_filler {
      type: "constant"
    }
  }
}
layer {
  name: "relu4_1"
  type: "ReLU"
  bottom: "conv4_1"
  top: "conv4_1"
}

layer {
  name: "conv4_2"
  type: "Convolution"
  bottom: "conv4_1"
  top: "conv4_2"
  param {
    lr_mult: 1
  }
  param {
    lr_mult: 1
  }
  convolution_param {
    num_output: 512
    kernel_size: 3
    pad: 1
    stride: 1
    weight_filler {
      type: "xavier"
    }
    bias_filler {
      type: "constant"
    }
  }
}
layer {
  name: "relu4_2"
  type: "ReLU"
  bottom: "conv4_2"
  top: "conv4_2"
}

layer {
  name: "conv4_3"
  type: "Convolution"
  bottom: "conv4_2"
  top: "conv4_3"
  param {
    lr_mult: 1
  }
  param {
    lr_mult: 1
  }
  convolution_param {
    num_output: 512
    kernel_size: 3
    pad: 1
    stride: 1
    weight_filler {
      type: "xavier"
    }
    bias_filler {
      type: "constant"
    }
  }
}
layer {
  name: "relu4_3"
  type: "ReLU"
  bottom: "conv4_3"
  top: "conv4_3"
}

layer {
  name: "pool4"
  type: "Pooling"
  bottom: "conv4_3"
  top: "pool4"
  pooling_param {
    pool: MAX
    kernel_size: 2
    stride: 2
  }
}

layer {
  name: "conv5_1"
  type: "Convolution"
  bottom: "pool4"
  top: "conv5_1"
  param {
    lr_mult: 1
  }
  param {
    lr_mult: 1
  }
  convolution_param {
    num_output: 512
    kernel_size: 3
    pad: 1
    stride: 1
    weight_filler {
      type: "xavier"
    }
    bias_filler {
      type: "constant"
    }
  }
}
layer {
  name: "relu5_1"
  type: "ReLU"
  bottom: "conv5_1"
  top: "conv5_1"
}

layer {
  name: "conv5_2"
  type: "Convolution"
  bottom: "conv5_1"
  top: "conv5_2"
  param {
    lr_mult: 1
  }
  param {
    lr_mult: 1
  }
  convolution_param {
    num_output: 512
    kernel_size: 3
    pad: 1
    stride: 1
    weight_filler {
      type: "xavier"
    }
    bias_filler {
      type: "constant"
    }
  }
}
layer {
  name: "relu5_2"
  type: "ReLU"
  bottom: "conv5_2"
  top: "conv5_2"
}

layer {
  name: "conv5_3"
  type: "Convolution"
  bottom: "conv5_2"
  top: "conv5_3"
  param {
    lr_mult: 1
  }
  param {
    lr_mult: 1
  }
  convolution_param {
    num_output: 512
    kernel_size: 3
    pad: 1
    stride: 1
    weight_filler {
      type: "xavier"
    }
    bias_filler {
      type: "constant"
    }
  }
}
layer {
  name: "relu5_3"
  type: "ReLU"
  bottom: "conv5_3"
  top: "conv5_3"
}

layer {
  name: "pool5"
  type: "Pooling"
  bottom: "conv5_3"
  top: "pool5"
  pooling_param {
    pool: MAX
    kernel_size: 2
    stride: 2
  }
}

layer {
  name: "fc6"
  type: "InnerProduct"
  bottom: "pool5"
  top: "fc6"
  param {
    lr_mult: 1
  }
  param {
    lr_mult: 1
  }
  inner_product_param {
    num_output: 4096
    weight_filler {
      type: "xavier"
    }
    bias_filler {
      type: "constant"
    }
  }
}
layer {
  name: "relu6"
  type: "ReLU"
  bottom: "fc6"
  top: "fc6"
}
layer {
  name: "drop6"
  type: "Dropout"
  bottom: "fc6"
  top: "fc6"
  dropout_param {
    dropout_ratio: 0.5
  }
}
layer {
  name: "fc7"
  type: "InnerProduct"
  bottom: "fc6"
  top: "fc7"
  param {
    lr_mult: 1
  }
  param {
    lr_mult: 1
  }
  inner_product_param {
    num_output: 4096
    weight_filler {
      type: "xavier"
    }
    bias_filler {
      type: "constant"
    }
  }
}
layer {
  name: "relu7"
  type: "ReLU"
  bottom: "fc7"
  top: "fc7"
}
layer {
  name: "drop7"
  type: "Dropout"
  bottom: "fc7"
  top: "fc7"
  dropout_param {
    dropout_ratio: 0.5
  }
}
layer {
  name: "fc8"
  type: "InnerProduct"
  bottom: "fc7"
  top: "fc8"
  param {
    lr_mult: 1
  }
  param {
    lr_mult: 2
  }
  inner_product_param {
    num_output: 10              #注意將fc8層改成自己的圖像類別數目
    weight_filler {
      type: "xavier"
    }
    bias_filler {
      type: "constant"
    }
  }
}
layer {
  name: "accuracy"
  type: "Accuracy"
  bottom: "fc8"
  bottom: "label"
  top: "accuracy"
  include {
    phase: TEST
  }
}
layer {
  name: "loss"
  type: "SoftmaxWithLoss"
  bottom: "fc8"
  bottom: "label"
  top: "loss"
}

以上是VGG16模型,需改路徑和類別數後保存退出即可。
在當前路徑下新建文件vggnet_solver.prototxt,複製以下代碼,別忘了修改相對應的路徑

net: "D:/caffe/caffe-master/models/VGG16/vggnet_train_val.prototxt"                
test_iter: 10
test_interval: 500  #每經過500次訓練,進行一次驗證查看accuracy
base_lr: 0.001
lr_policy: "step"
gamma: 0.1
stepsize: 1000
display: 20
max_iter: 2000  #只是做做練習,2000次迭代就夠了
momentum: 0.9
weight_decay: 0.0005
snapshot: 1000  #每經過1000次迭代訓練保存一次快照
snapshot_prefix: "D:/caffe/caffe-master/examples/myfile/caffenet_train"    #在路徑下新建caffenet_train文件夾
solver_mode: GPU

5.訓練網絡

在當前路徑下新建train_vggnet.sh文件,同樣用記事本打開,複製以下代碼,同樣修改相對應的路徑

#!/usr/bin/env sh
set -e

D:/caffe/caffe-master/Build/x64/Release/caffe train \
    --solver=D:/caffe/caffe-master/models/VGG16/vggnet_solver.prototxt [email protected]

修改後保存退出,然後在cmd命令窗口cd到當前路徑,然後輸入命令 sh train_vggnet.sh,如下圖所示
在這裏插入圖片描述
如果運行沒出錯的話,那恭喜你訓練成功,然後就是等待訓練結果就行

發表評論
所有評論
還沒有人評論,想成為第一個評論的人麼? 請在上方評論欄輸入並且點擊發布.
相關文章