caffe學習2-LeNet網絡

LeNet網絡是Yann leCun在1989年提出的一個經典CNN網絡，主要用於手寫字體的識別，準確率可以達到99%以上。這裏採用caffe的python接口進行訓練，包括對網絡的定義、訓練參數的定義、觀察卷積效果和卷積核、記錄損失函數和測試精度並繪製相關圖形。

#採用caffe的python接口進行lenet經典網絡訓練
#第一步，加載相關模塊
from pylab import *
#在notebook內顯示圖像
%matplotlib inline

#定義caffe路徑，這裏視自己現在所在路徑爲準，進行合適的修改
caffe_root='../../'
import sys
sys.path.insert(0,caffe_root + 'python')
import caffe

#第二步，下載好LeNet網絡和數據
import os
os.chdir(caffe_root)
#下載數據
!data/mnist/get_mnist.sh
#準備數據，也就是將原始數據變成leveldb格式
!examples/mnist/create_mnist.sh
os.chdir('examples')

Downloading…
Creating lmdb…
Done.

#定義網絡，該網絡結構由python語言編寫
from caffe import layers as L, params as P

def lenet(lmdb, batch_size):
# our version of LeNet: a series of linear and simple nonlinear transformations
#NetSpec: Net Specifically  
    n = caffe.NetSpec()

    n.data, n.label = L.Data(batch_size=batch_size, backend=P.Data.LMDB, source=lmdb,
                             transform_param=dict(scale=1./255), ntop=2)
#定義網絡的結構 
    n.conv1 = L.Convolution(n.data, kernel_size=5, num_output=20, weight_filler=dict(type='xavier'))
    n.pool1 = L.Pooling(n.conv1, kernel_size=2, stride=2, pool=P.Pooling.MAX)
    n.conv2 = L.Convolution(n.pool1, kernel_size=5, num_output=50, weight_filler=dict(type='xavier'))
    n.pool2 = L.Pooling(n.conv2, kernel_size=2, stride=2, pool=P.Pooling.MAX)
    n.fc1 =   L.InnerProduct(n.pool2, num_output=500, weight_filler=dict(type='xavier'))
    n.relu1 = L.ReLU(n.fc1, in_place=True)
    n.score = L.InnerProduct(n.relu1, num_output=10, weight_filler=dict(type='xavier'))
    n.loss =  L.SoftmaxWithLoss(n.score, n.label)

    return n.to_proto()

#將上面敘述的網絡結構寫成了lenet_auto_train.prototxt和lenet_auto_test.prototxt文件 ，在此打開並將數據庫寫入
with open('mnist/lenet_auto_train.prototxt', 'w') as f:
    f.write(str(lenet('mnist/mnist_train_lmdb', 64)))

with open('mnist/lenet_auto_test.prototxt', 'w') as f:
    f.write(str(lenet('mnist/mnist_test_lmdb', 100)))

#顯示整個網絡用命令cat
!cat mnist/lenet_auto_train.prototxt

layer {
name: “data”
type: “Data”
top: “data”
top: “label”
transform_param {
scale: 0.00392156862745
}
data_param {
source: “mnist/mnist_train_lmdb”
batch_size: 64
backend: LMDB
}
}
layer {
name: “conv1”
type: “Convolution”
bottom: “data”
top: “conv1”
convolution_param {
num_output: 20
kernel_size: 5
weight_filler {
type: “xavier”
}
}
}
layer {
name: “pool1”
type: “Pooling”
bottom: “conv1”
top: “pool1”
pooling_param {
pool: MAX
kernel_size: 2
stride: 2
}
}
layer {
name: “conv2”
type: “Convolution”
bottom: “pool1”
top: “conv2”
convolution_param {
num_output: 50
kernel_size: 5
weight_filler {
type: “xavier”
}
}
}
layer {
name: “pool2”
type: “Pooling”
bottom: “conv2”
top: “pool2”
pooling_param {
pool: MAX
kernel_size: 2
stride: 2
}
}
layer {
name: “fc1”
type: “InnerProduct”
bottom: “pool2”
top: “fc1”
inner_product_param {
num_output: 500
weight_filler {
type: “xavier”
}
}
}
layer {
name: “relu1”
type: “ReLU”
bottom: “fc1”
top: “fc1”
}
layer {
name: “score”
type: “InnerProduct”
bottom: “fc1”
top: “score”
inner_product_param {
num_output: 10
weight_filler {
type: “xavier”
}
}
}
layer {
name: “loss”
type: “SoftmaxWithLoss”
bottom: “score”
bottom: “label”
top: “loss”
}

#查看訓練所用參數文件
!cat mnist/lenet_auto_solver.prototxt

# The train/test net protocol buffer definition
train_net: "mnist/lenet_auto_train.prototxt"
test_net: "mnist/lenet_auto_test.prototxt"
# test_iter specifies how many forward passes the test should carry out.
# In the case of MNIST, we have test batch size 100 and 100 test iterations,
# covering the full 10,000 testing images.
test_iter: 100
# Carry out testing every 500 training iterations.
test_interval: 500
# The base learning rate, momentum and the weight decay of the network.
base_lr: 0.01
momentum: 0.9
weight_decay: 0.0005
# The learning rate policy
lr_policy: "inv"
gamma: 0.0001
power: 0.75
# Display every 100 iterations
display: 100
# The maximum number of iterations
max_iter: 10000
# snapshot intermediate results
snapshot: 5000
snapshot_prefix: "mnist/lenet"

#加載sover文件，此處用SGD的方法來優化損失函數，當然也可以用其他優化方法
caffe.set_device(0)
caffe.set_mode_gpu

solver = None
solver = caffe.SGDSolver('mnist/lenet_auto_solver.prototxt')

#檢測網絡各個層的的featuremap情況，卷積核情況和參數情況
#blobs存儲的是【層的名字，(batchsize, 輸入數據個數，輸入數據大小)】
[(k,v.data.shape) for k,v in solver.net.blobs.items()]

[(‘data’, (64, 1, 28, 28)),
(‘label’, (64,)),
(‘conv1’, (64, 20, 24, 24)),
(‘pool1’, (64, 20, 12, 12)),
(‘conv2’, (64, 50, 8, 8)),
(‘pool2’, (64, 50, 4, 4)),
(‘fc1’, (64, 500)),
(‘score’, (64, 10)),
(‘loss’, ())]

#顯示參數層情況,分佈爲（層名字，（輸出feature-maps，輸入的feature-maps，卷積核大小））
[(k,v[0].data.shape) for k, v in solver.net.params.items()]

[(‘conv1’, (20, 1, 5, 5)),
(‘conv2’, (50, 20, 5, 5)),
(‘fc1’, (500, 800)),
(‘score’, (10, 500))]

solver.net.forward()                 #訓練網絡
solver.test_nets[0].forward()    #測試網絡

{‘loss’: array(2.288771152496338, dtype=float32)}

#用一行來顯示前八個圖像
imshow(solver.net.blobs['data'].data[:8, 0].transpose(1, 0, 2).reshape(28, 8*28), cmap='gray'); axis('off')
print 'train labels',solver.net.blobs['label'].data[:8]

train labels [ 5. 0. 4. 1. 9. 2. 1. 3.]

imshow(solver.test_nets[0].blobs['data'].data[:8, 0].transpose(1, 0, 2).reshape(28, 8*28), cmap='gray'); axis('off')
print 'test labels:', solver.test_nets[0].blobs['label'].data[:8]

test labels: [ 7. 2. 1. 0. 4. 1. 4. 9.]

#第三步，一步步操作去觀察
solver.step(2)

imshow(solver.net.params['conv1'][0].diff[:, 0].reshape(4, 5, 5, 5).transpose(0, 2, 1, 3).reshape(4*5, 5*5), cmap='gray'); axis('off')

(-0.5, 24.5, 19.5, -0.5)

#訓練網絡
%time
niter = 400
test_interval = 25
# losses will also be stored in the log
train_loss = zeros(niter)
test_acc = zeros(int(np.ceil(niter / test_interval)))
output = zeros((niter, 8, 10))

# the main solver loop
for it in range(niter):
    solver.step(1)  # SGD by Caffe

    # store the train loss
    train_loss[it] = solver.net.blobs['loss'].data

    # store the output on the first test batch
    # (start the forward pass at conv1 to avoid loading new data)
    solver.test_nets[0].forward(start='conv1')
    output[it] = solver.test_nets[0].blobs['score'].data[:8]

    # run a full test every so often
    # (Caffe can also do this for us and write to a log, but we show here
    #  how to do it directly in Python, where more complicated things are easier.)
    if it % test_interval == 0:
        print 'Iteration', it, 'testing...'
        correct = 0
        for test_it in range(100):
            solver.test_nets[0].forward()
            correct += sum(solver.test_nets[0].blobs['score'].data.argmax(1)
                           == solver.test_nets[0].blobs['label'].data)
        test_acc[it // test_interval] = correct / 1e4

PU times: user 0 ns, sys: 0 ns, total: 0 ns
Wall time: 5.96 µs
Iteration 0 testing…
Iteration 25 testing…
Iteration 50 testing…
Iteration 75 testing…
Iteration 100 testing…
Iteration 125 testing…
Iteration 150 testing…
Iteration 175 testing…
Iteration 200 testing…
Iteration 225 testing…
Iteration 250 testing…
Iteration 275 testing…
Iteration 300 testing…
Iteration 325 testing…
Iteration 350 testing…
Iteration 375 testing…

_, ax1 = subplots()
ax2 = ax1.twinx()
ax1.plot(arange(niter), train_loss)
ax2.plot(test_interval * arange(len(test_acc)), test_acc, 'r')
ax1.set_xlabel('iteration')
ax1.set_ylabel('train loss')
ax2.set_ylabel('test accuracy')
ax2.set_title('Test Accuracy: {:.2f}'.format(test_acc[-1]))

#經過最後一層raw，查看迭代次數與預估值。橫座標爲迭代次數，縱座標爲預估值，其中白色表示1，黑色表示0.越白表示屬於這個數的概率越大
for i in range(8):
    figure(figsize=(2, 2))
    imshow(solver.test_nets[0].blobs['data'].data[i, 0], cmap='gray')
    figure(figsize=(10, 2))
    imshow(output[:50, i].T, interpolation='nearest', cmap='gray')
    xlabel('iteration')
    ylabel('label')

其他省略。。共計8個！

#用softmax迴歸來做,可以看出0，1的效果更加明顯

for i in range(8):
    figure(figsize=(2, 2))
    imshow(solver.test_nets[0].blobs['data'].data[i, 0], cmap='gray')
    figure(figsize=(10, 2))
    imshow(exp(output[:50, i].T) / exp(output[:50, i].T).sum(0), interpolation='nearest', cmap='gray')
    xlabel('iteration')
    ylabel('label')


#根據已經定義好了的網絡結構配置文件.prototxt和訓練參數配置文件solver.prototxt文件進行訓練
train_net_path = 'mnist/custom_auto_train.prototxt'
test_net_path = 'mnist/custom_auto_test.prototxt'
solver_config_path = 'mnist/custom_auto_solver.prototxt'

### define net
def custom_net(lmdb, batch_size):
    # define your own net!
    n = caffe.NetSpec()

    # keep this data layer for all networks
    n.data, n.label = L.Data(batch_size=batch_size, backend=P.Data.LMDB, source=lmdb,
                             transform_param=dict(scale=1./255), ntop=2)

    # EDIT HERE to try different networks
    # this single layer defines a simple linear classifier
    # (in particular this defines a multiway logistic regression)
    n.score =   L.InnerProduct(n.data, num_output=10, weight_filler=dict(type='xavier'))

    # EDIT HERE this is the LeNet variant we have already tried
    # n.conv1 = L.Convolution(n.data, kernel_size=5, num_output=20, weight_filler=dict(type='xavier'))
    # n.pool1 = L.Pooling(n.conv1, kernel_size=2, stride=2, pool=P.Pooling.MAX)
    # n.conv2 = L.Convolution(n.pool1, kernel_size=5, num_output=50, weight_filler=dict(type='xavier'))
    # n.pool2 = L.Pooling(n.conv2, kernel_size=2, stride=2, pool=P.Pooling.MAX)
    # n.fc1 =   L.InnerProduct(n.pool2, num_output=500, weight_filler=dict(type='xavier'))
    # EDIT HERE consider L.ELU or L.Sigmoid for the nonlinearity
    # n.relu1 = L.ReLU(n.fc1, in_place=True)
    # n.score =   L.InnerProduct(n.fc1, num_output=10, weight_filler=dict(type='xavier'))

    # keep this loss layer for all networks
    n.loss =  L.SoftmaxWithLoss(n.score, n.label)

    return n.to_proto()


with open(train_net_path, 'w') as f:
    f.write(str(custom_net('mnist/mnist_train_lmdb', 64)))    
with open(test_net_path, 'w') as f:
    f.write(str(custom_net('mnist/mnist_test_lmdb', 100)))


### define solver
from caffe.proto import caffe_pb2
s = caffe_pb2.SolverParameter()

# Set a seed for reproducible experiments:
# this controls for randomization in training.
s.random_seed = 0xCAFFE

# Specify locations of the train and (maybe) test networks.
s.train_net = train_net_path
s.test_net.append(test_net_path)
s.test_interval = 500  # Test after every 500 training iterations.
s.test_iter.append(100) # Test on 100 batches each time we test.

s.max_iter = 10000     # no. of times to update the net (training iterations)

# EDIT HERE to try different solvers
# solver types include "SGD", "Adam", and "Nesterov" among others.
s.type = "Adam"

# Set the initial learning rate for SGD.
s.base_lr = 0.01  # EDIT HERE to try different learning rates
# Set momentum to accelerate learning by
# taking weighted average of current and previous updates.
s.momentum = 0.9
# Set weight decay to regularize and prevent overfitting
s.weight_decay = 5e-4

# Set `lr_policy` to define how the learning rate changes during training.
# This is the same policy as our default LeNet.
s.lr_policy = 'inv'
s.gamma = 0.0001
s.power = 0.75
# EDIT HERE to try the fixed rate (and compare with adaptive solvers)
# `fixed` is the simplest policy that keeps the learning rate constant.
# s.lr_policy = 'fixed'

# Display the current training loss and accuracy every 1000 iterations.
s.display = 1000

# Snapshots are files used to store networks we've trained.
# We'll snapshot every 5K iterations -- twice during training.
s.snapshot = 5000
s.snapshot_prefix = 'mnist/custom_net'

# Train on the GPU
s.solver_mode = caffe_pb2.SolverParameter.GPU

# Write the solver to a temporary file and return its filename.
with open(solver_config_path, 'w') as f:
    f.write(str(s))

### load the solver and create train and test nets
solver = None  # ignore this workaround for lmdb data (can't instantiate two solvers on the same data)
solver = caffe.get_solver(solver_config_path)

### solve
niter = 250  # EDIT HERE increase to train for longer
test_interval = niter / 10
# losses will also be stored in the log
train_loss = zeros(niter)
test_acc = zeros(int(np.ceil(niter / test_interval)))

# the main solver loop
for it in range(niter):
    solver.step(1)  # SGD by Caffe

    # store the train loss
    train_loss[it] = solver.net.blobs['loss'].data

    # run a full test every so often
    # (Caffe can also do this for us and write to a log, but we show here
    #  how to do it directly in Python, where more complicated things are easier.)
    if it % test_interval == 0:
        print 'Iteration', it, 'testing...'
        correct = 0
        for test_it in range(100):

            solver.test_nets[0].forward()
            correct += sum(solver.test_nets[0].blobs['score'].data.argmax(1)
                           == solver.test_nets[0].blobs['label'].data)
        test_acc[it // test_interval] = correct / 1e4

_, ax1 = subplots()
ax2 = ax1.twinx()
ax1.plot(arange(niter), train_loss)
ax2.plot(test_interval * arange(len(test_acc)), test_acc, 'r')
ax1.set_xlabel('iteration')
ax1.set_ylabel('train loss')
ax2.set_ylabel('test accuracy')
ax2.set_title('Custom Test Accuracy: {:.2f}'.format(test_acc[-1]))

Iteration 0 testing…
Iteration 25 testing…
Iteration 50 testing…
Iteration 75 testing…
Iteration 100 testing…
Iteration 125 testing…
Iteration 150 testing…
Iteration 175 testing…
Iteration 200 testing…
Iteration 225 testing…

具體實施步驟可以參照caffe官方網站：
http://nbviewer.jupyter.org/github/BVLC/caffe/blob/master/examples/01-learning-lenet.ipynb

caffe學習2-LeNet網絡

python學習11-集合和字典

python學習9-語句

Caffe學習1-圖像識別與數據可視化

caffe學習2-LeNet網絡

python學習1

https://yachay.unat.edu.pe/blog/index.php?comment_area=format_blog&comment_component=blog&comment_co

linux以太網驅動總結