因爲最近重新回爐深造Det細節了,就對cocoeval源碼進行了閱讀,發現這部分csdn上也沒有很詳細的註解,自己看了很久,就順帶寫了一下註解吧,希望給看着煩的朋友一點幫助。
首先我們瞭解下cocoeval .py的構成吧。
Params類:
對於COCO格式的數據檢測,我們主要分爲不同的IoU閾值,不同的面積範圍,單張圖片的最大檢測數量。在這些不同的參數下,會得到不同的AP與AR。
所以在這個類中,我們需要指定這些參數的數值範圍,具體可看下面貼出的代碼。
標準的即IoU閾值設置爲從0.5-0.95 間隔0.05,一共10個閾值
AR的閾值爲0-1 間隔0.01 ,一共101個閾值
面積範圍爲 small(0~32) medium(32~96) large(96~10**5)
檢測最大數,按照置信度分數排序後選擇最大檢測數範圍內的det結果。
COCOeval類:
創建COCOeval這個類的時候,我們需要傳入兩個COCO 類別的instance,一個是gt對應的COCO,一個是det對應的COCO,關於COCO的類別,那麼關於COCO類,在之前文章中有介紹,傳送門:COCO.py在det中的應用
OK,COCOeval類有三個方法是我們在det中會用到的,分別爲evaluate,accumulate,summarize
其中evaluate的作用就是得到單張圖片在特定類別,特定面積閾值內,特定最大檢測數下的所有閾值檢測結果。
accumulate是對這些單張圖片的結果進行積累計算。
summarize會根據傳入IoU閾值、面積閾值、最大檢測數這些參數返回對應的mAp與mAR。
好了有了上述分析,來一個整體的流程吧:
首先我們創建COCOeval類,傳入gt和det對應的兩個COCO類,COCOeval類的構造函數會把gt中對應的img id與 cat id添加至類變量中。
然後我們調用這個instance的evaluate方法,在這個方法裏調用_prepare方法,會生成gt與dt的字典列表,用[img_id,cat_id]作爲key,value即爲這個指定圖片指定類別對應的所有ann信息,是一個list形式。根據這兩個字典列表,我們可以生成iou計算,iou計算也以[img_id,cat_id]作爲key,value是一個M*N維的ndarry矩陣,m爲dt的個數,n爲gt的個數。
然後將會調用evaluateImg這個方法,這個方法傳入固定的img_id,cat_id,aRng,maxDet,我們可以得到對應的img在特定類別,特定面積閾值,特定最大檢測數下的檢測結果,(對於面積閾值來說,如果ann對應的bbox超過了就設置爲ignore,對於最大檢測數,按照置信度排序後取出前最大檢測數個即可)把這個檢測結果按照K,A,M的順序堆疊,可以得到self.evalImgs這個list,這個list包含了所有圖片在所有IoU閾值,面積閾值,最大檢測數下的所有檢測結果。
繼續調用instance的accumulate方法,可以根據上述得到的self.evalImgs返回所有圖片在不同IoU閾值、不同AR、不同類別、不同面積閾值、不同最大檢測數下的Ap與AR,以numpy數組的返回,即precision(T,R,K,A,M) recall(T,K,A,M)。
繼續調用instance的summarize方法,會根據傳入的具體的IoU閾值,面積閾值,最大檢測數的值返回上述precision和recall中對應維的檢測結果,我們就可以自定義形式返回我們想要的各種參數下的AP與AR啦。
上面說的可能比較簡單而且隱晦,下面貼註釋過的源碼,建議跟着上面說的順序看一下以便理解。
import numpy as np
import datetime
import time
from collections import defaultdict
from . import mask as maskUtils
import copy
class COCOeval:
# Interface for evaluating detection on the Microsoft COCO dataset.
#
# The usage for CocoEval is as follows:
# cocoGt=..., cocoDt=... # load dataset and results
# E = CocoEval(cocoGt,cocoDt); # initialize CocoEval object
# E.params.recThrs = ...; # set parameters as desired
# E.evaluate(); # run per image evaluation
# E.accumulate(); # accumulate per image results
# E.summarize(); # display summary metrics of results
# For example usage see evalDemo.m and http://mscoco.org/.
#
# The evaluation parameters are as follows (defaults in brackets):
# imgIds - [all] N img ids to use for evaluation
# catIds - [all] K cat ids to use for evaluation
# iouThrs - [.5:.05:.95] T=10 IoU thresholds for evaluation
# recThrs - [0:.01:1] R=101 recall thresholds for evaluation
# areaRng - [...] A=4 object area ranges for evaluation
# maxDets - [1 10 100] M=3 thresholds on max detections per image
# iouType - ['segm'] set iouType to 'segm', 'bbox' or 'keypoints'
# iouType replaced the now DEPRECATED useSegm parameter.
# useCats - [1] if true use category labels for evaluation
# Note: if useCats=0 category labels are ignored as in proposal scoring.
# Note: multiple areaRngs [Ax2] and maxDets [Mx1] can be specified.
#
# evaluate(): evaluates detections on every image and every category and
# concats the results into the "evalImgs" with fields:
# dtIds - [1xD] id for each of the D detections (dt)
# gtIds - [1xG] id for each of the G ground truths (gt)
# dtMatches - [TxD] matching gt id at each IoU or 0
# gtMatches - [TxG] matching dt id at each IoU or 0
# dtScores - [1xD] confidence of each dt
# gtIgnore - [1xG] ignore flag for each gt
# dtIgnore - [TxD] ignore flag for each dt at each IoU
#
# accumulate(): accumulates the per-image, per-category evaluation
# results in "evalImgs" into the dictionary "eval" with fields:
# params - parameters used for evaluation
# date - date evaluation was performed
# counts - [T,R,K,A,M] parameter dimensions (see above)
# precision - [TxRxKxAxM] precision for every evaluation setting
# recall - [TxKxAxM] max recall for every evaluation setting
# Note: precision and recall==-1 for settings with no gt objects.
#
# See also coco, mask, pycocoDemo, pycocoEvalDemo
#
# Microsoft COCO Toolbox. version 2.0
# Data, paper, and tutorials available at: http://mscoco.org/
# Code written by Piotr Dollar and Tsung-Yi Lin, 2015.
# Licensed under the Simplified BSD License [see coco/license.txt]
def __init__(self, cocoGt=None, cocoDt=None, iouType='segm'):
'''
Initialize CocoEval using coco APIs for gt and dt
:param cocoGt: coco object with ground truth annotations
:param cocoDt: coco object with detection results
:return: None
'''
if not iouType:
print('iouType not specified. use default iouType segm')
self.cocoGt = cocoGt # ground truth COCO API
self.cocoDt = cocoDt # detections COCO API
self.evalImgs = defaultdict(list) # per-image per-category evaluation results [KxAxI] elements
self.eval = {} # accumulated evaluation results
self._gts = defaultdict(list) # gt for evaluation
self._dts = defaultdict(list) # dt for evaluation
self.params = Params(iouType=iouType) # parameters
self._paramsEval = {} # parameters for evaluation
self.stats = [] # result summarization
self.ious = {} # ious between all gts and dts
if not cocoGt is None:
# 把GT中所有的img id 與 類別 id 加入 參數dict中
self.params.imgIds = sorted(cocoGt.getImgIds())
self.params.catIds = sorted(cocoGt.getCatIds())
def _prepare(self):
'''
Prepare ._gts and ._dts for evaluation based on params
在目標檢測中 _.gts 索引Ann的index爲 【圖片ip, 類別ip】,得到的是一個list數組,如果一張圖片的一個類別有多個bbox,
那麼list中將會有多個item ._dts同理
:return: None
'''
def _toMask(anns, coco):
# modify ann['segmentation'] by reference
for ann in anns:
rle = coco.annToRLE(ann)
ann['segmentation'] = rle
p = self.params
if p.useCats:
# 獲取特定圖片,特定類別的註釋,主要是清除檢測中出現gt中沒有的img id,class id
gts=self.cocoGt.loadAnns(self.cocoGt.getAnnIds(imgIds=p.imgIds, catIds=p.catIds))
dts=self.cocoDt.loadAnns(self.cocoDt.getAnnIds(imgIds=p.imgIds, catIds=p.catIds))
else:
gts=self.cocoGt.loadAnns(self.cocoGt.getAnnIds(imgIds=p.imgIds))
dts=self.cocoDt.loadAnns(self.cocoDt.getAnnIds(imgIds=p.imgIds))
# convert ground truth to mask if iouType == 'segm'
if p.iouType == 'segm':
_toMask(gts, self.cocoGt)
_toMask(dts, self.cocoDt)
# set ignore flag
for gt in gts:
# 部分比較小的物體,會設置忽略檢測 根據json中的註釋來定
gt['ignore'] = gt['ignore'] if 'ignore' in gt else 0
gt['ignore'] = 'iscrowd' in gt and gt['iscrowd']
if p.iouType == 'keypoints':
gt['ignore'] = (gt['num_keypoints'] == 0) or gt['ignore']
self._gts = defaultdict(list) # gt for evaluation
self._dts = defaultdict(list) # dt for evaluation
# 給對應img,類別 添加對應的bbox信息
for gt in gts:
self._gts[gt['image_id'], gt['category_id']].append(gt)
for dt in dts:
self._dts[dt['image_id'], dt['category_id']].append(dt)
#得到的是每張圖片,單個類別的檢測結果的集合。
self.evalImgs = defaultdict(list) # per-image per-category evaluation results
self.eval = {} # accumulated evaluation results
def evaluate(self):
'''
Run per image evaluation on given images and store results (a list of dict) in self.evalImgs
:return: None
'''
tic = time.time()
print('Running per image evaluation...')
p = self.params
# add backward compatibility if useSegm is specified in params
if not p.useSegm is None:
p.iouType = 'segm' if p.useSegm == 1 else 'bbox'
print('useSegm (deprecated) is not None. Running {} evaluation'.format(p.iouType))
print('Evaluate annotation type *{}*'.format(p.iouType))
# 取出GT中的,img cat id
p.imgIds = list(np.unique(p.imgIds))
if p.useCats:
p.catIds = list(np.unique(p.catIds))
p.maxDets = sorted(p.maxDets)
self.params=p
self._prepare()
# loop through images, area range, max detection number
catIds = p.catIds if p.useCats else [-1]
if p.iouType == 'segm' or p.iouType == 'bbox':
computeIoU = self.computeIoU
elif p.iouType == 'keypoints':
computeIoU = self.computeOks
# ious返回的是一個【M * N】的ndarry, 其中M是在這個img中,catId下有多少個預測的bbox, N是在這個img,catId下有多少個GT
self.ious = {(imgId, catId): computeIoU(imgId, catId) \
for imgId in p.imgIds
for catId in catIds}
evaluateImg = self.evaluateImg
maxDet = p.maxDets[-1]
# self.evalImages 順序是 K,A,M,I 一共K*A*M*I個單張圖片的檢測結果,單張圖片的特定類別,特定面積範圍,特定最大檢測個數下的檢測結果。
#我們可以按照這個來索引對應的檢測結果,在後續accumulate函數中有具體使用。
self.evalImgs = [evaluateImg(imgId, catId, areaRng, maxDet)
for catId in catIds
for areaRng in p.areaRng
for imgId in p.imgIds
]
self._paramsEval = copy.deepcopy(self.params)
toc = time.time()
print('DONE (t={:0.2f}s).'.format(toc-tic))
# 這塊用cython寫的,主要返回的就是 imgId,catId對應的M*N矩陣,每個值都是對應框的IoU值
def computeIoU(self, imgId, catId):
p = self.params
if p.useCats:
gt = self._gts[imgId,catId]
dt = self._dts[imgId,catId]
else:
#把這張圖片的所有類別的所有檢測結果進行一個數組的合併
gt = [_ for cId in p.catIds for _ in self._gts[imgId,cId]]
dt = [_ for cId in p.catIds for _ in self._dts[imgId,cId]]
if len(gt) == 0 and len(dt) ==0:
return []
#按照網絡預測的置信度score排序
inds = np.argsort([-d['score'] for d in dt], kind='mergesort')
dt = [dt[i] for i in inds]
#把超出最大檢測結果的bbox剔除
if len(dt) > p.maxDets[-1]:
dt=dt[0:p.maxDets[-1]]
if p.iouType == 'segm':
g = [g['segmentation'] for g in gt]
d = [d['segmentation'] for d in dt]
elif p.iouType == 'bbox':
g = [g['bbox'] for g in gt]
d = [d['bbox'] for d in dt]
else:
raise Exception('unknown iouType for iou computation')
# compute iou between each dt and gt region
iscrowd = [int(o['iscrowd']) for o in gt]
ious = maskUtils.iou(d,g,iscrowd)
return ious
def computeOks(self, imgId, catId):
p = self.params
# dimention here should be Nxm
gts = self._gts[imgId, catId]
dts = self._dts[imgId, catId]
inds = np.argsort([-d['score'] for d in dts], kind='mergesort')
dts = [dts[i] for i in inds]
if len(dts) > p.maxDets[-1]:
dts = dts[0:p.maxDets[-1]]
# if len(gts) == 0 and len(dts) == 0:
if len(gts) == 0 or len(dts) == 0:
return []
ious = np.zeros((len(dts), len(gts)))
sigmas = p.kpt_oks_sigmas
vars = (sigmas * 2)**2
k = len(sigmas)
# compute oks between each detection and ground truth object
for j, gt in enumerate(gts):
# create bounds for ignore regions(double the gt bbox)
g = np.array(gt['keypoints'])
xg = g[0::3]; yg = g[1::3]; vg = g[2::3]
k1 = np.count_nonzero(vg > 0)
bb = gt['bbox']
x0 = bb[0] - bb[2]; x1 = bb[0] + bb[2] * 2
y0 = bb[1] - bb[3]; y1 = bb[1] + bb[3] * 2
for i, dt in enumerate(dts):
d = np.array(dt['keypoints'])
xd = d[0::3]; yd = d[1::3]
if k1>0:
# measure the per-keypoint distance if keypoints visible
dx = xd - xg
dy = yd - yg
else:
# measure minimum distance to keypoints in (x0,y0) & (x1,y1)
z = np.zeros((k))
dx = np.max((z, x0-xd),axis=0)+np.max((z, xd-x1),axis=0)
dy = np.max((z, y0-yd),axis=0)+np.max((z, yd-y1),axis=0)
e = (dx**2 + dy**2) / vars / (gt['area']+np.spacing(1)) / 2
if k1 > 0:
e=e[vg > 0]
ious[i, j] = np.sum(np.exp(-e)) / e.shape[0]
return ious
def evaluateImg(self, imgId, catId, aRng, maxDet):
'''
perform evaluation for single category and image
計算本張圖片,特定類別,特定面積閾值,特定最大檢測結果下的result。
:return: dict (single image results)
'''
p = self.params
if p.useCats:
# 本張圖片特定類別的所有檢測結果與GT
gt = self._gts[imgId,catId]
dt = self._dts[imgId,catId]
else:
gt = [_ for cId in p.catIds for _ in self._gts[imgId,cId]]
dt = [_ for cId in p.catIds for _ in self._dts[imgId,cId]]
if len(gt) == 0 and len(dt) ==0:
return None
for g in gt:
#如果不符合特定面積的閾值,就忽略
if g['ignore'] or (g['area']<aRng[0] or g['area']>aRng[1]):
g['_ignore'] = 1
else:
g['_ignore'] = 0
# sort dt highest score first, sort gt ignore last
# gtind 前面都是 ignore爲0 的gt 後面都是 ignore爲1的gt
gtind = np.argsort([g['_ignore'] for g in gt], kind='mergesort')
#挑出滿足我們這個特定area閾值下的所有gt
gt = [gt[i] for i in gtind]
dtind = np.argsort([-d['score'] for d in dt], kind='mergesort')
#按照置信度大小挑出滿足這個最大檢測個數下的所有dt
dt = [dt[i] for i in dtind[0:maxDet]]
iscrowd = [int(o['iscrowd']) for o in gt]
# load computed ious
#得到滿足area閾值的gt與所有dt的iou結果 (M * n(gtind))
ious = self.ious[imgId, catId][:, gtind] if len(self.ious[imgId, catId]) > 0 else self.ious[imgId, catId]
#得到我們需要設置的IoU閾值,超過定義爲正樣本,不符合則爲負樣本
T = len(p.iouThrs)
G = len(gt)
D = len(dt)
#在每個閾值下的Gt是否得到匹配
gtm = np.zeros((T,G))
#在每個閾值下的Dt是否得到匹配
dtm = np.zeros((T,D))
#所有忽略的gt
gtIg = np.array([g['_ignore'] for g in gt])
#所有忽略的dt
dtIg = np.zeros((T,D))
#如果這張圖片存在這個類別的gt與dt
if not len(ious)==0:
for tind, t in enumerate(p.iouThrs): #IoU index, IoU閾值
#按照置信度大小排序好的前 max_Det個dt
for dind, d in enumerate(dt):
# 如果m= -1 代表這個dt沒有得到匹配 m代表dt匹配的最好的gt的下標
iou = min([t,1-1e-10])
m = -1
for gind, g in enumerate(gt):
# 如果這個gt已經被其他置信度更好的dt匹配到了,本輪的dt就不能匹配這個gt了。
if gtm[tind,gind]>0 and not iscrowd[gind]:
continue
# 因爲gt已經按照ignore排好序了,前面的爲0,於是當我們碰到第一個gt的ignore爲1時,判斷這個dt是否已經匹配到了
#其他的gt,如果m>-1證明並且m對應的gt沒有被ignore,就直接結束即可,對應的就是這個dt最好的gt。
if m>-1 and gtIg[m]==0 and gtIg[gind]==1:
break
# 如果計算dt與gt的iou小於目前最佳的IoU,忽略這個gt
if ious[dind,gind] < iou:
continue
# 超過當前最佳的IoU,更新IoU與m的值
iou=ious[dind,gind]
m=gind
# 如果這個dt沒有對應的gt與其匹配,繼續dt的下一個循環
if m ==-1:
continue
# 把當前dt與第m個gt進行匹配,修改dtm與gtm的值,分別一一對應
dtIg[tind,dind] = gtIg[m] # 如果這個dt對應的最佳gt本身就是被ignore的,就把這個dt也設置爲ignore。
dtm[tind,dind] = gt[m]['id']
gtm[tind,m] = d['id']
# set unmatched detections outside of area range to ignore
a = np.array([d['area']<aRng[0] or d['area']>aRng[1] for d in dt]).reshape((1, len(dt)))
dtIg = np.logical_or(dtIg, np.logical_and(dtm==0, np.repeat(a,T,0)))
# store results for given image and category
return {
'image_id': imgId,
'category_id': catId,
'aRng': aRng,
'maxDet': maxDet,
'dtIds': [d['id'] for d in dt],
'gtIds': [g['id'] for g in gt],
'dtMatches': dtm,
'gtMatches': gtm,
'dtScores': [d['score'] for d in dt],
'gtIgnore': gtIg,
'dtIgnore': dtIg,
}
def accumulate(self, p = None):
'''
Accumulate per image evaluation results and store the result in self.eval
:param p: input params for evaluation
:return: None
'''
print('Accumulating evaluation results...')
tic = time.time()
if not self.evalImgs:
print('Please run evaluate() first')
# allows input customized parameters
if p is None:
p = self.params
p.catIds = p.catIds if p.useCats == 1 else [-1]
T = len(p.iouThrs) # 多少個ioU的閾值
R = len(p.recThrs) #多少個recall的閾值
K = len(p.catIds) if p.useCats else 1 # 多少個類
A = len(p.areaRng) #多少個面積閾值
M = len(p.maxDets) #多少個最大檢測數
precision = -np.ones((T,R,K,A,M)) # -1 for the precision of absent categories
recall = -np.ones((T,K,A,M))
scores = -np.ones((T,R,K,A,M))
# create dictionary for future indexing
_pe = self._paramsEval
catIds = _pe.catIds if _pe.useCats else [-1]
setK = set(catIds)
setA = set(map(tuple, _pe.areaRng))
setM = set(_pe.maxDets)
setI = set(_pe.imgIds)
# get inds to evaluate
k_list = [n for n, k in enumerate(p.catIds) if k in setK] #對應不重複的K的id list 後續同此
m_list = [m for n, m in enumerate(p.maxDets) if m in setM]
a_list = [n for n, a in enumerate(map(lambda x: tuple(x), p.areaRng)) if a in setA]
i_list = [n for n, i in enumerate(p.imgIds) if i in setI]
I0 = len(_pe.imgIds) #多少個圖片
A0 = len(_pe.areaRng) #多少個面積閾值
# retrieve E at each category, area range, and max number of detections
# self.evalImgs 索引順序是 K,A,M,I 所以找到在特定K,A,M下的所有圖片,需要按照如下的三維索引
for k, k0 in enumerate(k_list):
Nk = k0*A0*I0 # 當前K0前面過了多少圖片與面積閾值
for a, a0 in enumerate(a_list):
Na = a0*I0 #在當前K0前面過了多少閾值
for m, maxDet in enumerate(m_list):
#k0,a0下的所有Images
E = [self.evalImgs[Nk + Na + i] for i in i_list]
E = [e for e in E if not e is None]
if len(E) == 0:
continue
#k0,a0,maxdet下的所有Images的得分
dtScores = np.concatenate([e['dtScores'][0:maxDet] for e in E])
# different sorting method generates slightly different results.
# mergesort is used to be consistent as Matlab implementation.
# k0,a0,maxdet下所有Images得分從高到底的索引 inds
inds = np.argsort(-dtScores, kind='mergesort')
#按照得分從高到低排序
dtScoresSorted = dtScores[inds]
# 在當前k0,a0下,每張圖片不超過MaxDet的所有det按照ind排序。 dtm[T,sum(Det) in every imges]
dtm = np.concatenate([e['dtMatches'][:,0:maxDet] for e in E], axis=1)[:,inds]
dtIg = np.concatenate([e['dtIgnore'][:,0:maxDet] for e in E], axis=1)[:,inds]
gtIg = np.concatenate([e['gtIgnore'] for e in E])
#有多少個正樣本
npig = np.count_nonzero(gtIg==0 )
if npig == 0:
continue
# 如果dtm對應的匹配gt不爲0,且對應的gt沒有被忽略,這個dt就是TP tips:[1,0,1,0,1,0]
tps = np.logical_and( dtm, np.logical_not(dtIg) )
#dtm對應的gt爲0, 並且這個dt也沒有被忽略,這個dt就是FP tips:[0,1,0,1,0,1]
fps = np.logical_and(np.logical_not(dtm), np.logical_not(dtIg) )
# 按照行的方式(每個Iou閾值下)進行匹配到的累加 每個index也就是到這個置信度的時候有多少個tp,有多少個fp
tp_sum = np.cumsum(tps, axis=1).astype(dtype=np.float)
fp_sum = np.cumsum(fps, axis=1).astype(dtype=np.float)
for t, (tp, fp) in enumerate(zip(tp_sum, fp_sum)):
tp = np.array(tp) #得到這個Iou下對應的tp tips:[1,0,2,0,3,0]
fp = np.array(fp) #得到這個IoU下對應的fp tips:[0,1,0,2,0,3]
nd = len(tp) #有多少個tp
rc = tp / npig #每個置信度分數下對應的recall 如上述例子 若有3個正樣本 則rc=[1/3,1/3,2/3,2/3,1,1]
pr = tp / (fp+tp+np.spacing(1)) #每個階段對應的精度
q = np.zeros((R,))
ss = np.zeros((R,))
if nd:
recall[t,k,a,m] = rc[-1]
else:
recall[t,k,a,m] = 0
# numpy is slow without cython optimization for accessing elements
# use python array gets significant speed improvement
pr = pr.tolist(); q = q.tolist()
#當前i下的最大精度
for i in range(nd-1, 0, -1):
if pr[i] > pr[i-1]:
pr[i-1] = pr[i]
#找到每個recall發生變化的時候的index,與p.recThrs一一對應,最接近其的值的index
inds = np.searchsorted(rc, p.recThrs, side='left')
try:
for ri, pi in enumerate(inds):
#得到每個recall閾值對應的最大精度,存入q中
q[ri] = pr[pi]
#得到這個recall值下的得分
ss[ri] = dtScoresSorted[pi]
except:
pass
precision[t,:,k,a,m] = np.array(q) # 按照recall的大小存入對應的精度
scores[t,:,k,a,m] = np.array(ss) #存入對應的分數
self.eval = {
'params': p,
'counts': [T, R, K, A, M],
'date': datetime.datetime.now().strftime('%Y-%m-%d %H:%M:%S'),
'precision': precision,
'recall': recall,
'scores': scores,
}
toc = time.time()
print('DONE (t={:0.2f}s).'.format( toc-tic))
def summarize(self):
'''
Compute and display summary metrics for evaluation results.
Note this functin can *only* be applied on the default parameter setting
'''
def _summarize( ap=1, iouThr=None, areaRng='all', maxDets=100 ):
p = self.params
iStr = ' {:<18} {} @[ IoU={:<9} | area={:>6s} | maxDets={:>3d} ] = {:0.3f}'
titleStr = 'Average Precision' if ap == 1 else 'Average Recall'
typeStr = '(AP)' if ap==1 else '(AR)'
iouStr = '{:0.2f}:{:0.2f}'.format(p.iouThrs[0], p.iouThrs[-1]) \
if iouThr is None else '{:0.2f}'.format(iouThr)
# 如果是'all' 就是所有尺度, 如果不是就是特定的尺度
aind = [i for i, aRng in enumerate(p.areaRngLbl) if aRng == areaRng]
mind = [i for i, mDet in enumerate(p.maxDets) if mDet == maxDets]
# 如果是ap,就從precision中得到對應面積閾值、最大檢測數下的精度
if ap == 1:
# dimension of precision: [TxRxKxAxM]
s = self.eval['precision']
# 得到特定IoU下的所有pr
if iouThr is not None:
t = np.where(iouThr == p.iouThrs)[0]
s = s[t]
s = s[:,:,:,aind,mind]
# 如果是recall,就取出recall的值
else:
# dimension of recall: [TxKxAxM]
s = self.eval['recall']
if iouThr is not None:
t = np.where(iouThr == p.iouThrs)[0]
s = s[t]
s = s[:,:,aind,mind]
if len(s[s>-1])==0:
mean_s = -1
#除去-1 其他的計算平均精度
else:
mean_s = np.mean(s[s>-1])
print(iStr.format(titleStr, typeStr, iouStr, areaRng, maxDets, mean_s))
return mean_s
def _summarizeDets():
stats = np.zeros((12,))
stats[0] = _summarize(1) # all iouThr, 所有recall下,所有面積下, 所有類別,在最大檢測數100下的的平均AP
# [1]:IoU閾值爲0.5 所有recall下,所有面積下, 所有類別,在最大檢測數100下的的平均AP
stats[1] = _summarize(1, iouThr=.5, maxDets=self.params.maxDets[2])
# [2]:IoU閾值爲0.75 所有recall下,所有面積下, 所有類別,在最大檢測數100下的的平均AP
stats[2] = _summarize(1, iouThr=.75, maxDets=self.params.maxDets[2])
#[3]: all iouThr, 所有recall下,small面積下, 所有類別,在最大檢測數100下的的平均AP
stats[3] = _summarize(1, areaRng='small', maxDets=self.params.maxDets[2])
#[4]: all iouThr, 所有recall下,medium面積下, 所有類別,在最大檢測數100下的的平均AP
stats[4] = _summarize(1, areaRng='medium', maxDets=self.params.maxDets[2])
#[5]: all iouThr, 所有recall下,large面積下, 所有類別,在最大檢測數100下的的平均AP
stats[5] = _summarize(1, areaRng='large', maxDets=self.params.maxDets[2])
#[6]: all iouThr,所有面積下, 所有類別,在最大檢測數1下的的平均recall
stats[6] = _summarize(0, maxDets=self.params.maxDets[0])
#[7]: all iouThr,所有面積下, 所有類別,在最大檢測數10下的的平均recall
stats[7] = _summarize(0, maxDets=self.params.maxDets[1])
# [8]: all iouThr,所有面積下, 所有類別,在最大檢測數100下的的平均recall
stats[8] = _summarize(0, maxDets=self.params.maxDets[2])
#[9]: all iouThr,small面積下, 所有類別,在最大檢測數100下的的平均recall
stats[9] = _summarize(0, areaRng='small', maxDets=self.params.maxDets[2])
# [10]: all iouThr,medium面積下, 所有類別,在最大檢測數100下的的平均recall
stats[10] = _summarize(0, areaRng='medium', maxDets=self.params.maxDets[2])
# [11]: all iouThr,large面積下, 所有類別,在最大檢測數100下的的平均recall
stats[11] = _summarize(0, areaRng='large', maxDets=self.params.maxDets[2])
return stats
def _summarizeKps():
stats = np.zeros((10,))
stats[0] = _summarize(1, maxDets=20)
stats[1] = _summarize(1, maxDets=20, iouThr=.5)
stats[2] = _summarize(1, maxDets=20, iouThr=.75)
stats[3] = _summarize(1, maxDets=20, areaRng='medium')
stats[4] = _summarize(1, maxDets=20, areaRng='large')
stats[5] = _summarize(0, maxDets=20)
stats[6] = _summarize(0, maxDets=20, iouThr=.5)
stats[7] = _summarize(0, maxDets=20, iouThr=.75)
stats[8] = _summarize(0, maxDets=20, areaRng='medium')
stats[9] = _summarize(0, maxDets=20, areaRng='large')
return stats
if not self.eval:
raise Exception('Please run accumulate() first')
iouType = self.params.iouType
if iouType == 'segm' or iouType == 'bbox':
summarize = _summarizeDets
elif iouType == 'keypoints':
summarize = _summarizeKps
self.stats = summarize()
def __str__(self):
self.summarize()
class Params:
'''
Params for coco evaluation api
'''
def setDetParams(self):
self.imgIds = []
self.catIds = []
# np.arange causes trouble. the data point on arange is slightly larger than the true value
self.iouThrs = np.linspace(.5, 0.95, int(np.round((0.95 - .5) / .05)) + 1, endpoint=True)
self.recThrs = np.linspace(.0, 1.00, int(np.round((1.00 - .0) / .01)) + 1, endpoint=True)
self.maxDets = [1, 10, 100]
self.areaRng = [[0 ** 2, 1e5 ** 2], [0 ** 2, 32 ** 2], [32 ** 2, 96 ** 2], [96 ** 2, 1e5 ** 2]]
self.areaRngLbl = ['all', 'small', 'medium', 'large']
self.useCats = 1
def setKpParams(self):
self.imgIds = []
self.catIds = []
# np.arange causes trouble. the data point on arange is slightly larger than the true value
self.iouThrs = np.linspace(.5, 0.95, int(np.round((0.95 - .5) / .05)) + 1, endpoint=True)
self.recThrs = np.linspace(.0, 1.00, int(np.round((1.00 - .0) / .01)) + 1, endpoint=True)
self.maxDets = [20]
self.areaRng = [[0 ** 2, 1e5 ** 2], [32 ** 2, 96 ** 2], [96 ** 2, 1e5 ** 2]]
self.areaRngLbl = ['all', 'medium', 'large']
self.useCats = 1
self.kpt_oks_sigmas = np.array([.26, .25, .25, .35, .35, .79, .79, .72, .72, .62,.62, 1.07, 1.07, .87, .87, .89, .89])/10.0
def __init__(self, iouType='segm'):
if iouType == 'segm' or iouType == 'bbox':
self.setDetParams()
elif iouType == 'keypoints':
self.setKpParams()
else:
raise Exception('iouType not supported')
self.iouType = iouType
# useSegm is deprecated
self.useSegm = None