有天導師突然找我,讓我搞一些關於人臉的應用,比如換個臉什麼的……沒辦法那就先把人臉自動檢測出來吧。人臉檢測,即檢測出圖像中存在的人臉,並把它的位置準確地框出來。是人臉特徵點檢測、人臉識別的基礎。可以谷歌Face Detection Benchmark尋找數據集和優秀論文,上thinkface論壇,蒐集人臉檢測數據集和方法。常用的人臉檢測數據集,包括FDDB、AFLW、WIDER FACE等。隨着近年來隨着深度學習的快速發展,涌現出來很多優秀的人臉檢測算法。
例如,FDDB數據庫就提交了很多出色的人臉檢測算法,例如採用級聯CNN網絡的人臉檢測方法:A Convolutioanal Neural Network Cascade,改進的faster rcnn做人臉檢測:Face Detection using Deep Learning:An Improved Faster RCNN Approach,還有對小臉檢測非常成功的Finding tiny faces等等,建議找個三篇左右認真研讀就行了,也不需要去一一實現,沒有太大意義。
另外,像opencv、dlib、libfacedetect等也提供了人臉檢測的接口。因爲人臉檢測是很基本的任務,所以很多公司都做了人臉檢測的工作,而且做得很牛,例如face++。
下面僅介紹本人嘗試並實現了的幾種常見的人臉檢測方法:
1.單個CNN人臉檢測方法
2.級聯CNN人臉檢測方法
3.OpenCV人臉檢測方法
4.Dlib人臉檢測方法
5.libfacedetect人臉檢測方法
6.Seetaface人臉檢測方法
1.單個CNN人臉檢測方法
該人臉檢測方法的有點在於,思路簡單,實現簡單;缺點是速度較慢(在一塊普通的gpu上對一副1000x600的圖像進行多尺度檢測也可能花上一兩秒),檢測效果還可以,但得到的人臉框不夠準確。
首先訓練一個判斷人臉非人臉的二分類器。例如採用卷積神經網絡caffenet進行二分類,可以在imagenet數據集訓練過的模型,利用自己的人臉數據集,進行微調。也可以自定義卷積網絡進行訓練,爲了能檢測到更小的人臉目標,我們一般採用小一點的卷積神經網絡作爲二分類模型,減小圖像輸入尺寸,加快預測速度。
然後將訓練好的人臉判斷分類網絡的全連接層改爲卷積層,這樣網絡變成了全卷積網絡,可以接受任意輸入圖像大小,圖像經過全卷積網絡將得到特徵圖,特徵圖上每一個“點”對應該位置映射到原圖上的感受野區域屬於人臉的概率,將屬於人臉概率大於設定閾值的視爲人臉候選框。
圖像上人臉的大小是變化的,爲了適應這種變化,最暴力的辦法就是使用圖像金字塔的方式,將待檢測的圖像縮放到不同大小,以進行多尺度人臉檢測。對多個尺度下檢測出來的所有人臉候選框,做非極大值抑制NMS,得到最後人臉檢測的結果。
這裏提供用caffe實現該方法的數據集、模型文件和代碼打包的 下載鏈接
下面介紹用caffe實現該方法的具體過程。因爲需要訓練判斷是否爲人臉的CNN分類器,準備好正負訓練樣本,然後得到caffe訓練所需的的數據集文件(由於採用的是48x48的網絡,原始數據集歸一化到了48x48)。
這裏CNN採用的是DeepID卷積神經網絡,網絡結構如下,它的輸入只有48x48大小,而採用AlexNet或CaffeNet網絡會增加時間開銷。
準備好網絡模型文件train_val.prototxt和超參數配置文件solver.prototxt之後(下載鏈接中都有),開始訓練,迭代10w次得到caffemodel。對測試集face_test文件夾中的圖像進行測試,準備好測試用的deploy.prototxt。
測試單張圖像的python腳本face_test.py如下:
# -*- coding: utf-8 -*-
"""
Created on Fri Mar 10 23:02:06 2017
@author: Administrator
"""
import numpy as np
import caffe
size = 48
image_file = 'C:/Users/Administrator/Desktop/caffe/data/face/face_test/0/253_faceimage07068.jpg'#測試圖片路徑
model_def = 'C:/Users/Administrator/Desktop/caffe/models/face/deploy.prototxt'
model_weights = 'C:/Users/Administrator/Desktop/caffe/models/face/_iter_10000.caffemodel'
net = caffe.Net(model_def, model_weights, caffe.TEST)
# 加載均值文件 也可指定數值做相應的操作
#mu = np.load('C:/Users/Administrator/Desktop/caffe/python/caffe/imagenet/ilsvrc_2012_mean.npy') ###caffe 自帶的文件
#mu = mu.mean(1).mean(1) # average over pixels to obtain the mean (BGR) pixel values
transformer = caffe.io.Transformer({'data': net.blobs['data'].data.shape}) ##設定圖片的shape格式(1,3,48,48),大小由deploy 文件指定
#transformer.set_mean('data', mu) # 每個通道減去均值
# python讀取的圖片文件格式爲H×W×K,需轉化爲K×H×W
transformer.set_transpose('data', (2,0,1)) #改變維度的順序,由原始圖片(48,48,3)變爲(3,48,48)
# python中將圖片存儲爲[0, 1],而caffe中將圖片存儲爲[0, 255],所以需要一個轉換
transformer.set_raw_scale('data', 255) # 縮放到【0,255】之間
transformer.set_channel_swap('data', (2,1,0)) #交換通道,將圖片由RGB變爲BGR
#net.blobs['data'].reshape(1,3,size, size) # 將輸入圖片格式轉化爲合適格式(與deploy文件相同)
#上面這句,第一參數:圖片數量 第二個參數 :通道數 第三個參數:圖片高度 第四個參數:圖片寬度
image = caffe.io.load_image(image_file) #加載圖片,始終是得到一副(h,w,3),rgb,0~1,float32的圖像
net.blobs['data'].data[...] = transformer.preprocess('data', image) #用上面的transformer.preprocess來處理剛剛加載圖片
caffe.set_device(0)
caffe.set_mode_gpu()
output = net.forward()
output_prob = output['prob'][0].argmax() # 給出概率最高的是第幾類,需要自己對應到我們約定的類別去
print output_prob
print output['prob'][0][0] #或print output['prob'][0,1]
批量測試計算準確度的matlab腳本face_test.m如下:
%注意:caffe中維度順序爲(N,C,H,W),而matcaffe中Blob維度順序爲(W,H,C,N),即完全相反
%matlab加載圖像爲(h,w,c),得到的是rgb,而caffe使用的是bgr
function test_face()
clear;
addpath('..');%添加上級目錄搜索路徑
addpath('.');%添加當前目錄搜索路徑
caffe.set_mode_gpu(); %設置gpu模式
caffe.set_device(0); %gpu的id爲0
%caffe.set_mode_cpu();
net_model = 'C:\Users\Administrator\Desktop\caffe\models\face\deploy.prototxt'; %網絡模型deploy.prototxt
net_weights = 'C:\Users\Administrator\Desktop\caffe\models\face\_iter_10000.caffemodel'; %訓練好的模型文件
%net_model = 'C:\Users\Administrator\Desktop\caffe\models\face2\deploy.prototxt'; %網絡模型deploy.prototxt
%net_weights = 'C:\Users\Administrator\Desktop\caffe\models\face2\_iter_100000.caffemodel'; %訓練好的模型文件
phase = 'test'; %不做訓練,而是測試
net = caffe.Net(net_model, net_weights, phase); %獲取網絡
tic;
error = 0;
total = 0;
%批量讀取圖像進行測試
datadir = 'C:\Users\Administrator\Desktop\caffe\data\face\face_test\0';
imagefiles = dir(datadir);
for i = 3:length(imagefiles)
im = imread(fullfile(datadir,imagefiles(i).name));
[input_data,flag] = prepare_image(im); %圖像數據預處理
if flag ~= 1
continue;
end
input_data ={input_data};
net.forward(input_data); %做前向傳播
scores = net.blobs('prob').get_data();
[best_score,best] = max(scores);
% fprintf('*****%.3f %d %d\n',best_score,best - 1,classid(i-2));
best = best - 1; %matlab中從1開始,減1變成從0開始
if best ~= 0
error = error + 1;
fprintf('-----error: %d\n',error);
errorfile = ['error\' imagefiles(i).name];
%imwrite(im,errorfile);
end
total = total + 1;
end
datadir_1 = 'C:\Users\Administrator\Desktop\caffe\data\face\face_test\1';
imagefiles_1 = dir(datadir_1);
for i = 3:length(imagefiles_1)
im_1 = imread(fullfile(datadir_1,imagefiles_1(i).name));
[input_data_1,flag] = prepare_image(im_1); %圖像數據預處理
if flag ~= 1
continue;
end
input_data_1 = {input_data_1};
net.forward(input_data_1); %做前向傳播
scores_1 = net.blobs('prob').get_data();
[best_score_1,best_1] = max(scores_1);
% fprintf('*****%.3f %d %d\n',best_score,best - 1,classid(i-2));
best_1 = best_1 - 1; %matlab中從1開始,減1變成從0開始
if best_1 ~= 1
error = error + 1;
fprintf('error: %d-----\n',error);
errorfile = ['face_error\' imagefiles_1(i).name];
%imwrite(im,errorfile);
end
total = total + 1;
end
total_time = toc;
%打印到屏幕上
fprintf('total_time: %.3f s\n',total_time);
fprintf('aver_time: %.3f s\n',total_time/total);
fprintf('error/total: %d/%d\n',error,total);
fprintf('accurary: %.4f\n',1.0 - (error*1.0)/total);
%disp(['error/total: ',num2str(error),'/',num2str(length(imagefiles)-2)]);
end
function [im_data,flag] = prepare_image(im)
%d = load('../+caffe/imagenet/ilsvrc_2012_mean.mat');
%mean_data = d.mean_data;
%resize to 227 x 227
im_data = [];
im = imresize(im,[227 227],'bilinear');
%im = imresize(im,[48 48],'bilinear');
[h,w,c] = size(im);
if c ~= 3
flag = 0;
return;
end
flag = 1;
%caffe的blob順序是[w h c num]
%matlab:[h w c] rgb -> caffe:[w h c] bgr
im_data = im(:,:,[3,2,1]); %rgb -> bgr
im_data = permute(im_data,[2,1,3]); %[h w c] -> [w h c]
[w,h,~] = size(im_data);
%ImageNet數據集的均值具有統計規律,這裏可以直接拿來使用
mean_data(:,:,1) = ones(w,h) .* 104; %b
mean_data(:,:,2) = ones(w,h) .* 117; %g
mean_data(:,:,3) = ones(w,h) .* 123; %r
im_data = single(im_data);
%im_data = im_data - single(mean_data); %因爲訓練集和測試集都沒有做去均值,所以這裏也不做(如果只是這裏做了去均值效果會變差)
end
在測試集上進行批量測試,準確率達到了98%。
爲了利用CNN分類器來檢測人臉,需要將CNN網絡中的全連接層替換爲卷積層得到全卷積網絡,修改好的全卷積網絡deploy_full_conv.prototxt內容如下:
name: "face_full_conv_net"
layer {
name: "data"
type: "Input"
top: "data"
input_param { shape: { dim: 1 dim: 3 dim: 48 dim: 48 } }
}
layer {
name: "conv1"
type: "Convolution"
bottom: "data"
top: "conv1"
convolution_param {
num_output: 20
kernel_size: 3
stride: 1
pad: 1
}
}
layer {
name: "relu1"
type: "ReLU"
bottom: "conv1"
top: "conv1"
}
layer {
name: "norm1"
type: "LRN"
bottom: "conv1"
top: "conv1"
lrn_param {
local_size: 5
alpha: 0.0001
beta: 0.75
}
}
layer {
name: "pool1"
type: "Pooling"
bottom: "conv1"
top: "pool1"
pooling_param {
pool: MAX
kernel_size: 2
stride: 2
}
}
layer {
name: "conv2"
type: "Convolution"
bottom: "pool1"
top: "conv2"
convolution_param {
num_output: 40
kernel_size: 3
pad: 1
}
}
layer {
name: "relu2"
type: "ReLU"
bottom: "conv2"
top: "conv2"
}
layer {
name: "norm2"
type: "LRN"
bottom: "conv2"
top: "conv2"
lrn_param {
local_size: 5
alpha: 0.0001
beta: 0.75
}
}
layer {
name: "pool2"
type: "Pooling"
bottom: "conv2"
top: "pool2"
pooling_param {
pool: MAX
kernel_size: 2
stride: 2
}
}
layer {
name: "conv3"
type: "Convolution"
bottom: "pool2"
top: "conv3"
convolution_param {
num_output: 60
kernel_size: 3
pad: 1
}
}
layer {
name: "relu3"
type: "ReLU"
bottom: "conv3"
top: "conv3"
}
layer {
name: "norm3"
type: "LRN"
bottom: "conv3"
top: "conv3"
lrn_param {
local_size: 5
alpha: 0.0001
beta: 0.75
}
}
layer {
name: "pool3"
type: "Pooling"
bottom: "conv3"
top: "pool3"
pooling_param {
pool: MAX
kernel_size: 2
stride: 2
}
}
layer {
name: "conv4"
type: "Convolution"
bottom: "pool3"
top: "conv4"
convolution_param {
num_output: 80
kernel_size: 3
pad: 1
}
}
layer {
name: "relu4"
type: "ReLU"
bottom: "conv4"
top: "conv4"
}
layer {
name: "norm4"
type: "LRN"
bottom: "conv4"
top: "conv4"
lrn_param {
local_size: 5
alpha: 0.0001
beta: 0.75
}
}
layer {
name: "pool4"
type: "Pooling"
bottom: "conv4"
top: "pool4"
pooling_param {
pool: MAX
kernel_size: 2
stride: 2
}
}
#修改爲卷積層
layer {
name: "fc5-conv" ### fc5
type: "Convolution" ### InnerProduct
bottom: "pool4"
top: "fc5-conv" ### fc5
#inner_product_param {
# num_output: 160
#}
convolution_param {
num_output: 160
kernel_size: 3
}
}
layer {
name: "relu5"
type: "ReLU"
bottom: "fc5-conv"
top: "fc5-conv"
}
layer {
name: "drop5"
type: "Dropout"
bottom: "fc5-conv"
top: "fc5-conv"
dropout_param {
dropout_ratio: 0.5
}
}
#修改爲卷積層
layer {
name: "fc6-conv" ### fc6
type: "Convolution" ### InnerProduct
bottom: "fc5-conv"
top: "fc6-conv"
#inner_product_param {
# num_output: 2
#}
convolution_param {
num_output: 2
kernel_size: 1
}
}
layer {
name: "prob"
type: "Softmax"
bottom: "fc6-conv"
top: "prob"
}
還需要將訓練好的_iter_100000.caffemodel模型文件也轉化爲全卷積的,得到的_iter_100000_full_conv.caffemodel,轉換腳本convert_full_conv.py如下:
# -*- coding: utf-8 -*-
"""
Created on Fri Mar 10 21:14:09 2017
@author: Administrator
"""
###首先需要手動將deploy.prototxt修改成全卷積的deploy_full_conv.prorotxt,特別要注意全連接層修改成卷積層的細節
###將訓練好的分類模型caffemodel轉換成可以接受任意輸入大小,最後輸出特徵圖的全卷積模型caffemodel
import numpy as np
import caffe
model_def = 'C:/Users/Administrator/Desktop/caffe/models/face/deploy.prototxt'
model_weights = 'C:/Users/Administrator/Desktop/caffe/models/face/_iter_100000.caffemodel'
net = caffe.Net(model_def,
model_weights,
caffe.TEST)
params = ['fc5', 'fc6']
# fc_params = {name: (weights, biases)}
fc_params = {pr: (net.params[pr][0].data, net.params[pr][1].data) for pr in params}
for fc in params:
print '{} weights are {} dimensional and biases are {} dimensional'.format(fc, fc_params[fc][0].shape, fc_params[fc][1].shape)
# Load the fully convolutional network to transplant the parameters.
net_full_conv = caffe.Net('./deploy_full_conv.prototxt',
'./_iter_100000.caffemodel',
caffe.TEST)
params_full_conv = ['fc5-conv', 'fc6-conv']
# conv_params = {name: (weights, biases)}
conv_params = {pr: (net_full_conv.params[pr][0].data, net_full_conv.params[pr][1].data) for pr in params_full_conv}
for conv in params_full_conv:
print '{} weights are {} dimensional and biases are {} dimensional'.format(conv, conv_params[conv][0].shape, conv_params[conv][1].shape)
for pr, pr_conv in zip(params, params_full_conv):
conv_params[pr_conv][0].flat = fc_params[pr][0].flat # flat unrolls the arrays
conv_params[pr_conv][1][...] = fc_params[pr][1]
net_full_conv.save('./_iter_100000_full_conv.caffemodel')
print 'success'
最後,就可以用deploy_full_conv.prototxt和_iter_100000_full_conv.caffemodel對任意輸入尺寸的圖像進行人臉檢測了。對單張圖像進行人臉檢測的python腳本face_detect如下:
# -*- coding: utf-8 -*-
import numpy as np
import cv2 #需要安裝opencv,然後將opencv安裝目錄下build\python\2.7\x64\cv2.pyd拷貝到python的安裝目錄下Anaconda2\Lib\site-packages文件夾下
from operator import itemgetter
import time
import caffe
caffe.set_device(0)
caffe.set_mode_gpu()
def IoU(rect_1, rect_2):
'''
:param rect_1: list in format [x11, y11, x12, y12, confidence]
:param rect_2: list in format [x21, y21, x22, y22, confidence]
:return: returns IoU ratio (intersection over union) of two rectangles
'''
x11 = rect_1[0] # first rectangle top left x
y11 = rect_1[1] # first rectangle top left y
x12 = rect_1[2] # first rectangle bottom right x
y12 = rect_1[3] # first rectangle bottom right y
x21 = rect_2[0] # second rectangle top left x
y21 = rect_2[1] # second rectangle top left y
x22 = rect_2[2] # second rectangle bottom right x
y22 = rect_2[3] # second rectangle bottom right y
x_overlap = max(0, min(x12,x22) -max(x11,x21))
y_overlap = max(0, min(y12,y22) -max(y11,y21))
intersection = x_overlap * y_overlap
union = (x12-x11) * (y12-y11) + (x22-x21) * (y22-y21) - intersection
return float(intersection) / union
def IoM(rect_1, rect_2):
'''
:param rect_1: list in format [x11, y11, x12, y12, confidence]
:param rect_2: list in format [x21, y21, x22, y22, confidence]
:return: returns IoM ratio (intersection over min-area) of two rectangles
'''
x11 = rect_1[0] # first rectangle top left x
y11 = rect_1[1] # first rectangle top left y
x12 = rect_1[2] # first rectangle bottom right x
y12 = rect_1[3] # first rectangle bottom right y
x21 = rect_2[0] # second rectangle top left x
y21 = rect_2[1] # second rectangle top left y
x22 = rect_2[2] # second rectangle bottom right x
y22 = rect_2[3] # second rectangle bottom right y
x_overlap = max(0, min(x12,x22) -max(x11,x21))
y_overlap = max(0, min(y12,y22) -max(y11,y21))
intersection = x_overlap * y_overlap
rect1_area = (y12 - y11) * (x12 - x11)
rect2_area = (y22 - y21) * (x22 - x21)
min_area = min(rect1_area, rect2_area)
return float(intersection) / min_area
def NMS(rectangles,threshold=0.3):
'''
:param rectangles: list of rectangles, which are lists in format [x11, y11, x12, y12, confidence]
:return: list of rectangles after local NMS
'''
rectangles = sorted(rectangles, key=itemgetter(4), reverse=True) #按照confidence降序排列
result_rectangles = rectangles[:] # list to return
'''
while not result_rectangles == []:
rect = result_rectangles[0]
for index in range(1,len(result_rectangles)):
iou = IoU(rect,result_rectangles[index])
if
'''
number_of_rects = len(result_rectangles)
#threshold = 0.3 # threshold of IoU of two rectangles
cur_rect = 0
while cur_rect < number_of_rects - 1: # start from first element to second last element
rects_to_compare = number_of_rects - cur_rect - 1 # elements after current element to compare
cur_rect_to_compare = cur_rect + 1 # start comparing with element after current
while rects_to_compare > 0: # while there is at least one element after current to compare
if (IoU(result_rectangles[cur_rect], result_rectangles[cur_rect_to_compare]) >= threshold or IoM(result_rectangles[cur_rect], result_rectangles[cur_rect_to_compare]) >= 0.3):
del result_rectangles[cur_rect_to_compare] # delete the rectangle
number_of_rects -= 1
else:
cur_rect_to_compare += 1 # skip to next rectangle
rects_to_compare -= 1
cur_rect += 1 # finished comparing for current rectangle
return result_rectangles
def face_detection(imgFile) :
#model_def = 'C:/Users/Administrator/Desktop/caffe/models/face/deploy_full_conv.prototxt'
#model_weights = 'C:/Users/Administrator/Desktop/caffe/models/face/_iter_10000_full_conv.caffemodel'
model_def = 'C:/Users/Administrator/Desktop/caffe/models/face2/deploy_full_conv.prototxt'
model_weights = 'C:/Users/Administrator/Desktop/caffe/models/face2/_iter_100000_full_conv.caffemodel'
net_full_conv = caffe.Net(model_def,
model_weights,
caffe.TEST)
mu = np.load('C:/Users/Administrator/Desktop/caffe/python/caffe/imagenet/ilsvrc_2012_mean.npy')
mu = mu.mean(1).mean(1) # average over pixels to obtain the mean (BGR) pixel values
#print 'mean-subtracted values:' , zip('BGR', mu)
start_time = time.time()
scales = [] #尺度變換和尺度變換因子
factor = 0.793700526
img = cv2.imread(imgFile) #opencv讀取的圖像爲(h,w,c),bgr,caffe的blob維度爲(n,c,h,w),使用的也是rgb
print img.shape
largest = min(2, 4000/max(img.shape[0:2])) #4000是人臉檢測的經驗值
scale = largest
minD = largest*min(img.shape[0:2])
while minD >= 48: #網絡的輸入是227x227??? #多尺度變換
scales.append(scale) #添加當前尺度
scale *= factor #乘以尺度變換因子
minD *= factor #得到新的尺度
true_boxes = []
for scale in scales:
scale_img = cv2.resize(img,((int(img.shape[1] * scale), int(img.shape[0] * scale)))) #將圖像縮放到各尺度
cv2.imwrite('C:/Users/Administrator/Desktop/caffe/scale_img.jpg',scale_img)
im = caffe.io.load_image('C:/Users/Administrator/Desktop/caffe/scale_img.jpg') #利用caffe的io接口加載圖像,始終是得到一副(h,w,3),rgb,0~1,float32的圖像
net_full_conv.blobs['data'].reshape(1,3,scale_img.shape[0],scale_img.shape[1]) #重新設置網絡data層Blob維度爲:1,3,height,width
transformer = caffe.io.Transformer({'data': net_full_conv.blobs['data'].data.shape}) #爲data層創建transformer
transformer.set_transpose('data', (2,0,1)) #(h,w,3)->(3,h,w)
#transformer.set_mean('data', mu) #設置均值,由於訓練集沒有去均值,這裏也不去均值
transformer.set_raw_scale('data', 255.0) #rescale from [0,1] to [0,255]
transformer.set_channel_swap('data', (2,1,0)) #RGB -> BGR
net_full_conv.blobs['data'].data[...] = transformer.preprocess('data', im)
out = net_full_conv.forward()
print out['prob'][0,0].shape #輸出層prob結果,行x列
#print out['prob'][0].argmax(axis=0)
featureMap = out['prob'][0,0] #out['prob'][0][0]屬於人臉的概率特徵圖
stride = 16 #特徵圖感受野大小
cellSize = 48 #網絡輸入尺寸
thresh = 0.95
for (y,x),prob in np.ndenumerate(featureMap):
if prob > thresh :
true_boxes.append([float(x*stride)/scale,
float(y*stride)/scale,
float(x*stride + cellSize - 1)/scale,
float(y*stride + cellSize - 1)/scale,
prob])
true_boxes = NMS(true_boxes,0.2) #非極大值抑制
for true_box in true_boxes:
(x1, y1, x2, y2) = true_box[0:4] #取出人臉框的座標
cv2.rectangle(img, (int(x1),int(y1)), (int(x2),int(y2)), (0,255,0)) #畫人臉框
end_time = time.time()
print (end_time-start_time)*1000,'ms'
cv2.imwrite('output.jpg',img)
cv2.namedWindow('test win')
cv2.imshow('test win', img)
cv2.waitKey(0)
cv2.destroyWindow('test win')
if __name__ == "__main__":
imgFile = 'C:/Users/Administrator/Desktop/caffe/matlab/demo/1.jpg'
face_detection(imgFile)