【目标检测】绘制FPPI、miss rate数据及代码例子

行人检测一般用FPPI而不是mAP作为评估指标，但是网上找了好久都没找到Python版本的代码，原来FPPI官方的代码是matlab的，这里记录一下用官方的代码来绘制FPPI曲线有哪些需要注意的

有两个方法:

1. 用matlab的evaluateDetectionMissRate（建议用这个）
2. 用官方的工具包（有一些参数我还没弄懂，欢迎清楚的小伙伴来填坑）

1. matlab的evaluateDetectionMissRate

感谢这篇文章提供思路：FPPI曲线绘制记录
matlab官方文档地址：evaluateDetectionMissRate

测试数据下载地址：matlab绘制FPPI测试数据

load fppi.mat
[am, fppi, missRate] = evaluateDetectionMissRate(results, a);
%%
% Plot log average miss rate - FPPI.
figure
loglog(fppi, missRate);
grid on
title(sprintf('log Average Miss Rate = %.5f',am))

成功绘制后图片长这个样子

如果要改成自己数据的话：
预测结果的格式（即代码中的result）:

1. 第一列为[x, y, width, height]
2. 第二列为预测的分数
3. 每一行中，第一列和第二列的物体数量要一样，例如第一列是6个物体，那他的shape为6*4，那么第二列也要是6个分数，shape为1*4（不懂的话可以打开fppi.mat看下）

label的格式（即代码中的a，名字我随便取的哈哈哈）：

1. 第一列为[x, y, width, height]
2. 如果有多个物体的话，就用分号;隔开
3. 预测结果和label的每一行都是指一张图片，所以两者的顺序要对应上

详情可以matlab源码中的注释：

function [logAverageMissRate, fppi, missRate] = evaluateDetectionMissRate(...
                                    detectionResults, groundTruthData, varargin)
%evaluateDetectionMissRate Evaluate the miss rate metric for object detection.
%   logAverageMissRate = evaluateDetectionMissRate(detectionResults,
%   groundTruthData) returns log-average miss rate to measure the detection
%   performance. For a multi-class detector, logAverageMissRate is a vector
%   of log-average miss rates for each object class. The class order
%   follows the same column order as the groundTruthData table.
% 
%   Inputs:
%   -------
%   detectionResults  - a table that has two columns for single-class
%                       detector, or three columns for multi-class
%                       detector. The first column contains M-by-4 matrices
%                       of [x, y, width, height] bounding boxes specifying
%                       object locations. The second column contains scores
%                       for each detection. For multi-class detector, the
%                       third column contains the predicted label for each
%                       detection. The label must be categorical type
%                       defined by the variable names of groundTruthData
%                       table.
%
%   groundTruthData   - a table that has one column for single-class, or
%                       multiple columns for multi-class. Each column
%                       contains M-by-4 matrices of [x, y, width, height]
%                       bounding boxes specifying object locations. The
%                       column name specifies the class label.
%  
%   [..., fppi, missRate] = evaluateDetectionMissRate(...) returns data
%   points for plotting the log-miss-rate/false-positives-per-image(FPPI)
%   curve. You can visualize the performance curve using loglog(fppi,missRate). 
%   For multi-class detector, recall and precision are cell arrays, where
%   each cell contains the data points for each object class.
%
%   [...] = evaluateDetectionMissRate(..., threshold) specifies the
%   overlap threshold for assigning a detection to a ground truth box. The
%   overlap ratio is computed as the intersection over union. The default
%   value is 0.5.
%
%   Example : Evaluate stop sign detector
%   -------------------------------------
%   % Load the ground truth table
%   load('stopSignsAndCars.mat')
%   stopSigns = stopSignsAndCars(:, 1:2);
%   stopSigns.imageFilename = fullfile(toolboxdir('vision'),'visiondata', ...
%       stopSigns.imageFilename);
%
%   % Train an ACF based detector
%   detector = trainACFObjectDetector(stopSigns,'NegativeSamplesFactor',2);
%
%   % Create a struct array to store the results
%   numImages = height(stopSigns);
%   results(numImages) = struct('Boxes', [], 'Scores', []);
%
%   % Run the detector on the training images
%   for i = 1 : numImages
%       I = imread(stopSigns.imageFilename{i});
%       [bboxes, scores] = detect(detector, I);
%       results(i).Boxes = bboxes;
%       results(i).Scores = scores;
%   end
%
%   results = struct2table(results);
%
%   % Evaluate the results against the ground truth data
%   [am, fppi, missRate] = evaluateDetectionMissRate(results, stopSigns(:, 2));
%
%   % Plot log-miss-rate/FPPI curve
%   figure
%   loglog(fppi, missRate);
%   grid on
%   title(sprintf('log Average Miss Rate = %.1f', am))
%
% See also evaluateDetectionPrecision, acfObjectDetector,
%          rcnnObjectDetector, trainACFObjectDetector, trainRCNNObjectDetector.

% Copyright 2016 The MathWorks, Inc.
%
% References
% ----------
%   [1] C. D. Manning, P. Raghavan, and H. Schutze. An Introduction to
%   Information Retrieval. Cambridge University Press, 2008.
%
%   [2] D. Hoiem, Y. Chodpathumwan, and Q. Dai. Diagnosing error in
%   object detectors. In Proc. ECCV, 2012.
%
%   [3] Dollar, Piotr, et al. "Pedestrian Detection: An Evaluation of the
%   State of the Art." Pattern Analysis and Machine Intelligence, IEEE
%   Transactions on 34.4 (2012): 743 - 761.

2. 用官方的工具包

非常感谢这篇文章：Caltech评估方法

评估的方法按照上面那篇文章可以跑通代码，需要注意的是：

1. 要下载annotations文件，并解压到../code3.2.1/data-USA中
2. 在此处下载别人预测的结果，并放在../code3.2.1/data-USA/res中（res文件夹需要自己你新建）

文件放置如图所示：
annotations文件内如图所示：

res文件内如图所示（ABC是我自己的算法名字）：

3. 修改dbEval.m中的algs变量（在第57行），保留要你对比的算法，他的的注释掉（名字要与你res文件夹中的算法名字相同）。同时，在最后面加上你自己算法的名字，前面的布尔变量表示是否将每个box的高度调整为100/128，clrs表示颜色，'-'表示线条类型

4. 然后运行dbEval.m即可。图会闪一下，因为最终图片会保存到pdf里面，所以需要安装Ghostscript和pdfcrop。当然也可以在savefig(fName1,1,'pdf','-r300','-fonts'); close(1); f1=[fName1 '.pdf'];设一个断点，然后手动保存

5. 自己的算法一定要放在前面，不然绘图的时候会显示不出来，这个坑以后再填
6. 自己的数据格式摹仿别人的预测就好了，第一列是图片id，然后是xmin、ymin、w、h，最后不知道是什么待填坑