【目標檢測】繪製FPPI、miss rate數據及代碼例子

行人檢測一般用FPPI而不是mAP作爲評估指標，但是網上找了好久都沒找到Python版本的代碼，原來FPPI官方的代碼是matlab的，這裏記錄一下用官方的代碼來繪製FPPI曲線有哪些需要注意的

有兩個方法:

用matlab的evaluateDetectionMissRate（建議用這個）
用官方的工具包（有一些參數我還沒弄懂，歡迎清楚的小夥伴來填坑）

1. matlab的evaluateDetectionMissRate

感謝這篇文章提供思路：FPPI曲線繪製記錄
matlab官方文檔地址：evaluateDetectionMissRate

load fppi.mat
[am, fppi, missRate] = evaluateDetectionMissRate(results, a);
%%
% Plot log average miss rate - FPPI.
figure
loglog(fppi, missRate);
grid on
title(sprintf('log Average Miss Rate = %.5f',am))

成功繪製後圖片長這個樣子

如果要改成自己數據的話：
預測結果的格式（即代碼中的result）:

第一列爲[x, y, width, height]
第二列爲預測的分數
每一行中，第一列和第二列的物體數量要一樣，例如第一列是6個物體，那他的shape爲6*4，那麼第二列也要是6個分數，shape爲1*4（不懂的話可以打開fppi.mat看下）

label的格式（即代碼中的a，名字我隨便取的哈哈哈）：

第一列爲[x, y, width, height]
如果有多個物體的話，就用分號;隔開
預測結果和label的每一行都是指一張圖片，所以兩者的順序要對應上

詳情可以matlab源碼中的註釋：

function [logAverageMissRate, fppi, missRate] = evaluateDetectionMissRate(...
                                    detectionResults, groundTruthData, varargin)
%evaluateDetectionMissRate Evaluate the miss rate metric for object detection.
%   logAverageMissRate = evaluateDetectionMissRate(detectionResults,
%   groundTruthData) returns log-average miss rate to measure the detection
%   performance. For a multi-class detector, logAverageMissRate is a vector
%   of log-average miss rates for each object class. The class order
%   follows the same column order as the groundTruthData table.
% 
%   Inputs:
%   -------
%   detectionResults  - a table that has two columns for single-class
%                       detector, or three columns for multi-class
%                       detector. The first column contains M-by-4 matrices
%                       of [x, y, width, height] bounding boxes specifying
%                       object locations. The second column contains scores
%                       for each detection. For multi-class detector, the
%                       third column contains the predicted label for each
%                       detection. The label must be categorical type
%                       defined by the variable names of groundTruthData
%                       table.
%
%   groundTruthData   - a table that has one column for single-class, or
%                       multiple columns for multi-class. Each column
%                       contains M-by-4 matrices of [x, y, width, height]
%                       bounding boxes specifying object locations. The
%                       column name specifies the class label.
%  
%   [..., fppi, missRate] = evaluateDetectionMissRate(...) returns data
%   points for plotting the log-miss-rate/false-positives-per-image(FPPI)
%   curve. You can visualize the performance curve using loglog(fppi,missRate). 
%   For multi-class detector, recall and precision are cell arrays, where
%   each cell contains the data points for each object class.
%
%   [...] = evaluateDetectionMissRate(..., threshold) specifies the
%   overlap threshold for assigning a detection to a ground truth box. The
%   overlap ratio is computed as the intersection over union. The default
%   value is 0.5.
%
%   Example : Evaluate stop sign detector
%   -------------------------------------
%   % Load the ground truth table
%   load('stopSignsAndCars.mat')
%   stopSigns = stopSignsAndCars(:, 1:2);
%   stopSigns.imageFilename = fullfile(toolboxdir('vision'),'visiondata', ...
%       stopSigns.imageFilename);
%
%   % Train an ACF based detector
%   detector = trainACFObjectDetector(stopSigns,'NegativeSamplesFactor',2);
%
%   % Create a struct array to store the results
%   numImages = height(stopSigns);
%   results(numImages) = struct('Boxes', [], 'Scores', []);
%
%   % Run the detector on the training images
%   for i = 1 : numImages
%       I = imread(stopSigns.imageFilename{i});
%       [bboxes, scores] = detect(detector, I);
%       results(i).Boxes = bboxes;
%       results(i).Scores = scores;
%   end
%
%   results = struct2table(results);
%
%   % Evaluate the results against the ground truth data
%   [am, fppi, missRate] = evaluateDetectionMissRate(results, stopSigns(:, 2));
%
%   % Plot log-miss-rate/FPPI curve
%   figure
%   loglog(fppi, missRate);
%   grid on
%   title(sprintf('log Average Miss Rate = %.1f', am))
%
% See also evaluateDetectionPrecision, acfObjectDetector,
%          rcnnObjectDetector, trainACFObjectDetector, trainRCNNObjectDetector.

% Copyright 2016 The MathWorks, Inc.
%
% References
% ----------
%   [1] C. D. Manning, P. Raghavan, and H. Schutze. An Introduction to
%   Information Retrieval. Cambridge University Press, 2008.
%
%   [2] D. Hoiem, Y. Chodpathumwan, and Q. Dai. Diagnosing error in
%   object detectors. In Proc. ECCV, 2012.
%
%   [3] Dollar, Piotr, et al. "Pedestrian Detection: An Evaluation of the
%   State of the Art." Pattern Analysis and Machine Intelligence, IEEE
%   Transactions on 34.4 (2012): 743 - 761.

2. 用官方的工具包

非常感謝這篇文章：Caltech評估方法

評估的方法按照上面那篇文章可以跑通代碼，需要注意的是：

要下載annotations文件，並解壓到../code3.2.1/data-USA中
在此處下載別人預測的結果，並放在../code3.2.1/data-USA/res中（res文件夾需要自己你新建）

文件放置如圖所示：
annotations文件內如圖所示：

res文件內如圖所示（ABC是我自己的算法名字）：

3. 修改dbEval.m中的algs變量（在第57行），保留要你對比的算法，他的的註釋掉（名字要與你res文件夾中的算法名字相同）。同時，在最後面加上你自己算法的名字，前面的布爾變量表示是否將每個box的高度調整爲100/128，clrs表示顏色，'-'表示線條類型

4. 然後運行dbEval.m即可。圖會閃一下，因爲最終圖片會保存到pdf裏面，所以需要安裝Ghostscript和pdfcrop。當然也可以在savefig(fName1,1,'pdf','-r300','-fonts'); close(1); f1=[fName1 '.pdf'];設一個斷點，然後手動保存

5. 自己的算法一定要放在前面，不然繪圖的時候會顯示不出來，這個坑以後再填
6. 自己的數據格式摹仿別人的預測就好了，第一列是圖片id，然後是xmin、ymin、w、h，最後不知道是什麼待填坑