maskrcnn_benchmark-----Step-by-step tutorial 如何訓練自己的數據集以及finetune

https://github.com/facebookresearch/maskrcnn-benchmark/issues/521 

稍後實驗改正

記錄方法一：

Steps

1) COCO format

將自己的數據集按照coco官方形式進行標註。這樣會好一點，測試時候直接用coco官方api就可以了。可用labelme等等，再進行轉換，還有對應的json文件。official coco format 

2) Creating a Dataset class for your data

根據 coco.py例子，從torchvision.datasets.coco.CocoDetection擴展一個新的類Class ，這個類封裝了pycoapi方法來管理coco數據集。類似於這樣：class  XXXDataset(torchvision.datasets.coco.CocoDetection):

寫在 maskrcnn-benchmark/maskrcnn_benchmark/data/datasets 文件夾下, 跟coco.py類似(比如還有voc格式的是voc.py)，並且寫進 __init__.py.（__all__ = ["COCODataset", "ConcatDataset", "PascalVOCDataset"]）

3) Adding dataset paths

此類將需要json文件的路徑作爲參數，它包含coco格式的數據集的metadata，以及圖像，它們所在的文件夾。

在 paths_catalog.py 改數據路徑，仿照人家原來的寫一下，然後可用在配置文件.yaml文件裏用字典形式的DATASETS指定，

比如

DATASETS:
  TRAIN: ("coco_2014_train", "coco_2014_valminusminival")
  TEST: ("coco_2014_minival",)

4) Evaluation file

因爲數據集是coco格式的，那麼可以使用用於COCODataset類的相同評估文件，在maskrcnn_benchmark/data/datasets/evaluation/__init__.py這裏加elif語句

 

    if isinstance(dataset, datasets.COCODataset):
        return coco_evaluation(**args)
    elif isinstance(dataset, datasets.PascalVOCDataset):
        return voc_evaluation(**args)
    elif isinstance(dataset, datasets.XXXXDataset):
        return XXX_evaluation(**args)
    else:
        dataset_name = dataset.__class__.__name__
        raise NotImplementedError("Unsupported dataset type {}.".format(dataset_name))

5) Training script

maskrcnn-benchmark/tools/  在這裏看訓練和測試的標準標本，添加自己的參數，更改輸出目錄（這個非常重要），等等。

6) Changing the hyper-parameters

引擎使用yacs config files配置文件，在repo中你可以找到不同的方法來改變超參數。
如果您使用的是單GPU，請查看README，有一個針對此案例的部分，您必須更改一些超參數，已爲多GPU（8 GPU）編寫了默認的超參數。 我使用單個模型訓練我的模型，所以沒有任何問題，只需更改SOLVER.IMS_PER_BATCH並調整其他SOLVER參數。

使用您在paths_catalog.py中使用的名稱更新DATASETS.TRAIN和DATASETS.TEST。 還要考慮更改最小/最大輸入大小超參數。還有學習率，不同GPU的base_lr是不同的，不然可能損失爆炸。比如1gpu約爲0.0025。

7) Finetuning the model

The issue #15 has all the explanation.

• 下載要設置的模型的官方權重。
• 更改配置文件 MODEL.ROI_BOX_HEAD.NUM_CLASSES = your_classes + background.
• 使用trim_detectron_model.py（附在後面） 刪除爲coco數據集設置的那些層, 如果不刪除就訓練，那麼那麼對於需要81個類（80個coco類+背景）的圖層將會出現問題，所以需要刪掉
• 此腳本將保存新權重，將路徑鏈接到MODEL.WEIGHT超參數上。

Now all it is ready for trainnig!!

this the general modifications to the code for a custom dataset, i made more changes according to my needs.

這是對自定義數據集代碼的一般修改，我根據自己的需要做了更多的修改。

Visualizing the results

Once the model finishes the training, the weights are saved, you can use the Mask_R-CNN_demo.ipynbnotebook to visualize the results of your model on the test dataset, but you have to change the class names in predictor.py, it has the coco classes by default, put them in the same order used for the annotations.

一旦模型完成訓練，權重被保存，你可以使用Mask_R-CNN_demo.ipynb  notebook來可視化模型在測試數據集上的結果，但你必須在predictor.py中更改類名，它默認是coco的類別，要將它們按照與註釋相同的順序放置。

 18 2 3

!python trim_detectron_model.py \
--pretrained_path /root/.torch/models/_detectron_35857345_12_2017_baselines_e2e_faster_rcnn_R-50-FPN_1x.yaml.01_36_30.cUF7QR7I_output_train_coco_2014_train%3Acoco_2014_valminusminival_generalized_rcnn_model_final.pkl \
--save_path ../weights/trimmed_weights/trimmed_e2e_faster_rcnn_R_50_FPN_1x_OURconfig.pth \
--cfg ../configs/caffe2/e2e_faster_rcnn_R_50_FPN_1x_caffe2.yaml

記錄方法二：
original = torch.load（' path / to / your / checkpoint.pth '）
new = { “ model ”：original [ “ model ” ]}
torch.save（new，' path / to / new / checkpoint.pth '）

1 -爲特定模型創建cfg
2 -使用load_c2_format函數，將爲您提供包含模型字段的dict。在那裏，可以通過刪除字段等來執行您想要的模型手術
3 - 使用 torch.save保存目標,結構應該還是按dict(model=state_dict).保存，並且請注意擴展名是.pth，而不是pkl
4 - 更改 MODEL.WEIGHT指向更改後的.pth文件地址。

from maskrcnn_benchmark.config import cfg
from maskrcnn_benchmark.utils.c2_model_loading import load_c2_format

cfg.merge_from_file("configs/caffe2/e2e_mask_rcnn_X_101_32x8d_FPN_1x_caffe2.yaml")
path = '/home/antonio/.torch/models/X-101-32x8d.pkl'
_d = load_c2_format(cfg, path)

keys = [k for k in _d['model'].keys()]
print(sorted(keys))

def _transfer_pretrained_weights(model, pretrained_model_pth):
    pretrained_weights = torch.load(pretrained_model_pth)['model']
    new_dict = {k.replace('module.',''):v for k, v in pretrained_weights.items()
                if 'cls_score' not in k and 'bbox_pred' not in k}
    this_state = model.state_dict()
    this_state.update(new_dict)
    model.load_state_dict(this_state)
    return model


def train(cfg, local_rank, distributed):
    old_model = build_detection_model(cfg)
    pretrained_model_pth = "/home/belhal/.torch/models/_detectron_35858933_12_2017_baselines_e2e_mask_rcnn_R-50-FPN_1x.yaml.01_48_14.DzEQe4wC_output_train_coco_2014_train%3Acoco_2014_valminusminival_generalized_rcnn_model_final.pkl"
    model = _transfer_pretrained_weights(old_model,pretrained_model_pth)
    device = torch.device(cfg.MODEL.DEVICE)
    model.to(device)
   ....

path='/Users/belhal/.torch/models/_detectron_35858933_12_2017_baselines_e2e_mask_rcnn_R-50-FPN_1x.yaml.01_48_14.DzEQe4wC_output_train_coco_2014_train%3Acoco_2014_valminusminival_generalized_rcnn_model_final.pkl'
from maskrcnn_benchmark.utils.c2_model_loading import load_c2_format

cfg.merge_from_file("../configs/e2e_mask_rcnn_X_101_32x8d_FPN_1x.yaml")
_d = load_c2_format(cfg, path)
newdict = _d

def removekey(d, listofkeys):
    r = dict(d)
    for key in listofkeys:
        del r[key]
    return r

newdict['model'] = removekey(_d['model'], ['cls_score.bias','cls_score.weight','bbox_pred.bias','bbox_pred.weight'])


w = torch.load("X-101-32x8d.pth")

然而，發生錯誤：UnicodeDecodeError：'ascii'編解碼器無法解碼位置2中的字節0xad：序數不在範圍內（128）
我可以通過使用pickle來解決這個錯誤：
with open("X-101-32x8d.pkl", "rb") as f: w = pickle.load(f, encoding='latin1')

 記錄方法三：該方法終結於：

 [loaded_state_dict]函數可以檢查函數中的哪些鍵 ，可以在這裏進行操作（maskrcnn-benchmark/maskrcnn_benchmark/utils/model_serialization.py ) 

不會影響repo中的其他部分。大概是

model_dict = model.state_dict()
loaded_state_dict = {k: v for k, v in loaded_state_dict.items() if k in model_dict and "roi_heads" not in k}
model_dict.update(loaded_state_dict)

如果是這樣，則無需經過4個步驟。只用編輯 [load_state_dict method] 
 align_and_update_state_dicts(model_state_dict, loaded_state_dict) 

**步驟1**。運行demo.py以測試您的軟件包是否已成功安裝。同時，您可以保存預先訓練的模型，該模型已從pkl格式轉換爲pth格式，可以通過pytorch模型加載。該模型是演示中COCODEmo對象的數據，因此您可以保存預訓練模型的權重，如下所示

    pretrained_model_path = "./pretrained_model/maskRCNN.pth"
    check_point = {'state': coco_demo.model.state_dict()}
    torch.save(check_point, pretrained_model_path)

這樣，無需查看[_load_file fun](https://github.com/facebookresearch/maskrcnn-benchmark/blob/f25c6cff92d32d92abe8965d68401004e90c8bee/maskrcnn_benchmark/utils/checkpoint.py#L117).另外還有個保存權重到OrderedDict的方法：(https://github.com/pytorch/pytorch/blob/828cb18fa35c7c132ab920de1c0dc6d859f152d6/torch/nn/modules/module.py#L602) 

retrained_model = torch.load(checkpoint_path)['model']
model_dict = self.model.state_dict()
pretrained_dict = {k: v for k, v in pretrained_model.items() if k in model_dict and "roi_heads" not in k}
model_dict.update(pretrained_dict)

 **step 2**. 使用configs文件夾中的yaml文件創建fastercnn或maskRCNN模型。執行此操作時，您需要調用cfg.merge_from_list方法來更改[_C.MODEL.ROI_BOX_HEAD.NUM_CLASSES = 81]到數據集中的類數（後臺在此實現中不計算）。(https://github.com/facebookresearch/maskrcnn-benchmark/blob/f25c6cff92d32d92abe8965d68401004e90c8bee/maskrcnn_benchmark/config/defaults.py#L182) 

**step 3**.需要創建兩個類。
第一個類：[torchvision.datasets.coco.CocoDetection] (
maskrcnn-benchmark/maskrcnn_benchmark/data/datasets/coco.py
 class COCODataset(torchvision.datasets.coco.CocoDetection): 
獲取一個圖像及其對應的bbox和標籤，而不將Bbox和標籤捆綁在BoxList對象中。也就是 return img, target, idx 
此類與[COCODataset](https://github.com/facebookresearch/maskrcnn-benchmark/blob/f25c6cff92d32d92abe8965d68401004e90c8bee/maskrcnn_benchmark/data/datasets/coco.py#L9).  在每次迭代中，該類生成一個元組（img，bbox，label）,其中img由Image.open(...).convert('RGB')生成;
 bbox 是 2d numpy 數組， label是1d 數組(mask的話就是掩碼信息). 
第二類繼承自第一類，這個類的主要職責是將一個圖像的真實包裝在一個BoxList對象中，類似於[line]

img, anno = super(COCODataset, self).__getitem__(idx)

(https://github.com/facebookresearch/maskrcnn-benchmark/blob/f25c6cff92d32d92abe8965d68401004e90c8bee/maskrcnn_benchmark/data/datasets/coco.py#L43).
**step 4**. 對於pytorch預訓練模型，權重在字典中被分類，其中鍵是layer的名稱。因此，您可以排除不存在的layer或者您想要從頭開始訓練，如下所示：pretrained_model = torch.load(checkpoint_path)['state']

trim_detectron_model.py

import os
import torch
import argparse
from maskrcnn_benchmark.config import cfg
from maskrcnn_benchmark.utils.c2_model_loading import load_c2_format


def removekey(d, listofkeys):
    r = dict(d)
    for key in listofkeys:
        print('key: {} is removed'.format(key))
        r.pop(key)
    return r


parser = argparse.ArgumentParser(description="Trim Detection weights and save in PyTorch format.")
parser.add_argument(
    "--pretrained_path",
    default="~/.torch/models/_detectron_35858933_12_2017_baselines_e2e_mask_rcnn_R-50-FPN_1x.yaml.01_48_14.DzEQe4wC_output_train_coco_2014_train%3Acoco_2014_valminusminival_generalized_rcnn_model_final.pkl",
    help="path to detectron pretrained weight(.pkl)",
    type=str,
)
parser.add_argument(
    "--save_path",
    default="./pretrained_model/mask_rcnn_R-50-FPN_1x_detectron_no_last_layers.pth",
    help="path to save the converted model",
    type=str,
)
parser.add_argument(
    "--cfg",
    default="configs/e2e_mask_rcnn_R_50_FPN_1x.yaml",
    help="path to config file",
    type=str,
)

args = parser.parse_args()
#
DETECTRON_PATH = os.path.expanduser(args.pretrained_path)
print('detectron path: {}'.format(DETECTRON_PATH))

cfg.merge_from_file(args.cfg)
_d = load_c2_format(cfg, DETECTRON_PATH)
newdict = _d

newdict['model'] = removekey(_d['model'],
                             ['cls_score.bias', 'cls_score.weight', 'bbox_pred.bias', 'bbox_pred.weight'])
torch.save(newdict, args.save_path)
print('saved to {}.'.format(args.save_path))

城市景觀數據集與coco

def clip_weights_from_pretrain_of_coco_to_cityscapes(f, out_file):
	""""""
	from maskrcnn_benchmark.config.paths_catalog import COCO_CATEGORIES
	from maskrcnn_benchmark.config.paths_catalog import CITYSCAPES_FINE_CATEGORIES
	coco_cats = COCO_CATEGORIES
	cityscapes_cats = CITYSCAPES_FINE_CATEGORIES
	coco_cats_to_inds = dict(zip(coco_cats, range(len(coco_cats))))
	cityscapes_cats_to_inds = dict(
		zip(cityscapes_cats, range(len(cityscapes_cats)))
	)

	checkpoint = torch.load(f)
	m = checkpoint['model']

	weight_names = {
		"cls_score": "module.roi_heads.box.predictor.cls_score.weight", 
		"bbox_pred": "module.roi_heads.box.predictor.bbox_pred.weight", 
		"mask_fcn_logits": "module.roi_heads.mask.predictor.mask_fcn_logits.weight", 
	}
	bias_names = {
		"cls_score": "module.roi_heads.box.predictor.cls_score.bias",
		"bbox_pred": "module.roi_heads.box.predictor.bbox_pred.bias", 
		"mask_fcn_logits": "module.roi_heads.mask.predictor.mask_fcn_logits.bias",
	}
	
	representation_size = m[weight_names["cls_score"]].size(1)
	cls_score = nn.Linear(representation_size, len(cityscapes_cats))
	nn.init.normal_(cls_score.weight, std=0.01)
	nn.init.constant_(cls_score.bias, 0)

	representation_size = m[weight_names["bbox_pred"]].size(1)
	class_agnostic = m[weight_names["bbox_pred"]].size(0) != len(coco_cats) * 4
	num_bbox_reg_classes = 2 if class_agnostic else len(cityscapes_cats)
	bbox_pred = nn.Linear(representation_size, num_bbox_reg_classes * 4)
	nn.init.normal_(bbox_pred.weight, std=0.001)
	nn.init.constant_(bbox_pred.bias, 0)

	dim_reduced = m[weight_names["mask_fcn_logits"]].size(1)
	mask_fcn_logits = Conv2d(dim_reduced, len(cityscapes_cats), 1, 1, 0)
	nn.init.constant_(mask_fcn_logits.bias, 0)
	nn.init.kaiming_normal_(
		mask_fcn_logits.weight, mode="fan_out", nonlinearity="relu"
	)
	
	def _copy_weight(src_weight, dst_weight):
		for ix, cat in enumerate(cityscapes_cats):
			if cat not in coco_cats:
				continue
			jx = coco_cats_to_inds[cat]
			dst_weight[ix] = src_weight[jx]
		return dst_weight

	def _copy_bias(src_bias, dst_bias, class_agnostic=False):
		if class_agnostic:
			return dst_bias
		return _copy_weight(src_bias, dst_bias)

	m[weight_names["cls_score"]] = _copy_weight(
		m[weight_names["cls_score"]], cls_score.weight
	)
	m[weight_names["bbox_pred"]] = _copy_weight(
		m[weight_names["bbox_pred"]], bbox_pred.weight
	)
	m[weight_names["mask_fcn_logits"]] = _copy_weight(
		m[weight_names["mask_fcn_logits"]], mask_fcn_logits.weight
	)

	m[bias_names["cls_score"]] = _copy_bias(
		m[bias_names["cls_score"]], cls_score.bias
	)
	m[bias_names["bbox_pred"]] = _copy_bias(
		m[bias_names["bbox_pred"]], bbox_pred.bias, class_agnostic
	)
	m[bias_names["mask_fcn_logits"]] = _copy_bias(
		m[bias_names["mask_fcn_logits"]], mask_fcn_logits.bias
	)

	print("f: {}\nout_file: {}".format(f, out_file))
	torch.save(m, out_file)

對最後一層進行手術的兩種示例：

Download the model from MODEL_ZOO

wget https://download.pytorch.org/models/maskrcnn/e2e_mask_rcnn_X_101_32x8d_FPN_1x.pth
 // Load the model in python
 python
 import torch
 model = torch.load("e2e_mask_rcnn_X_101_32x8d_FPN_1x.pth ")
 // Remove the previous training parameters. 
 del model['iteration']
 del model['scheduler']
 del model['optimizer']
 // Remove the output layers in COCO, these are the mismatched layers you saw.
 //Second stage prediction
 del model["model"]["module.roi_heads.box.predictor.cls_score.weight"]
 del model["model"]["module.roi_heads.box.predictor.cls_score.bias"]
 del model["model"]["module.roi_heads.box.predictor.bbox_pred.weight"]
 del model["model"]["module.roi_heads.box.predictor.bbox_pred.bias"]
 //mask prediction
 del model["model"]["module.roi_heads.mask.predictor.mask_fcn_logits.weight"]
 del model["model"]["module.roi_heads.mask.predictor.mask_fcn_logits.bias"]
 // RPN
 del model["model"]["module.rpn.head.cls_logits.weight"]
 del model["model"]["module.rpn.head.cls_logits.bias"]
 del model["model"]["module.rpn.head.bbox_pred.weight"]
 del model["model"]["module.rpn.head.bbox_pred.bias"]
 //save the model
 torch.save(model, "modified_model.pth")
Then use modified_model.pth in your MODEL.WEIGHT



def delete_net_weights_for_finetune(
	model_file, 
	out_file, 
	rpn_final_convs=False, 
	bbox_final_fcs=True, 
	mask_final_conv=True
):
	del_keys = []
	checkpoint = torch.load(model_file)
	print("keys: {}".format(checkpoint.keys()))
	m = checkpoint['model']

	if rpn_final_convs:
		# 'module.rpn.anchor_generator.cell_anchors.0', 
		# 'module.rpn.anchor_generator.cell_anchors.1', 
		# 'module.rpn.anchor_generator.cell_anchors.2', 
		# 'module.rpn.anchor_generator.cell_anchors.3', 
		# 'module.rpn.anchor_generator.cell_anchors.4'
		# 'module.rpn.head.cls_logits.weight', 
		# 'module.rpn.head.cls_logits.bias', 
		# 'module.rpn.head.bbox_pred.weight', 
		# 'module.rpn.head.bbox_pred.bias',
		del_keys.extend([
			k for k in m.keys() if k.find("rpn.anchor_generator") is not -1
		])
		del_keys.extend([
			k for k in m.keys() if k.find("rpn.head.cls_logits") is not -1
		])
		del_keys.extend([
			k for k in m.keys() if k.find("rpn.head.bbox_pred") is not -1
		])

	if bbox_final_fcs:
		# 'module.roi_heads.box.predictor.cls_score.weight', 
		# 'module.roi_heads.box.predictor.cls_score.bias', 
		# 'module.roi_heads.box.predictor.bbox_pred.weight', 
		# 'module.roi_heads.box.predictor.bbox_pred.bias',
		del_keys.extend([
			k for k in m.keys() if k.find(
				"roi_heads.box.predictor.cls_score"
			) is not -1
		])
		del_keys.extend([
			k for k in m.keys() if k.find(
				"roi_heads.box.predictor.bbox_pred"
			) is not -1
		])

	if mask_final_conv:
		# 'module.roi_heads.mask.predictor.mask_fcn_logits.weight', 
		# 'module.roi_heads.mask.predictor.mask_fcn_logits.bias',
		del_keys.extend([
			k for k in m.keys() if k.find(
				"roi_heads.mask.predictor.mask_fcn_logits"
			) is not -1
		])
	
	for k in del_keys:
		print("del k: {}".format(k))
		del m[k]

	# checkpoint['model'] = m
	print("f: {}\nout_file: {}".format(f, out_file))
	recursively_mkdirs(os.path.dirname(out_file))
	torch.save({"model": m}, out_file)