https://github.com/facebookresearch/maskrcnn-benchmark/issues/521
ps：如果只需要fintune的話，直接看第二部分，是可以查看model的各個鍵值的，然後根據需要進行刪減。
ps：後面有個亂七八糟的，是我記錄的，後期刪改
ps：還有個是可以凍結殘差網絡的訓練層的，eg：resnet50，我凍結了最大層，也就是：FREEZE_CONV_BODY_AT = 4
ResNet50FPNStagesTo5 = tuple(
StageSpec(index=i, block_count=c, return_features=r)
for (i, c, r) in ((1, 3, True), (2, 4, True), (3, 6, True), (4, 3, True))
)
FREEZE_CONV_BODY_AT = 1 train resnet only 16 last layers
FREEZE_CONV_BODY_AT = 2 train resnet only 13 last layers
FREEZE_CONV_BODY_AT = 3 train resnet only 9 last layers
FREEZE_CONV_BODY_AT = 4 train resnet only 3 last layers
一、不同數據fintune，不同類別。
Steps1) COCO format將自己的數據集按照coco官方形式進行標註。這樣會好一點，測試時候直接用coco官方api就可以了。可用labelme等等，再進行轉換，還有對應的json文件。official coco format 2) Creating a
|
18
2
3
二、網絡的fintune
需要的步驟是：
1 -爲特定模型創建cfg
2 -使用load_c2_format函數，將爲您提供包含模型字段的dict。在那裏，可以通過刪除字段等來執行您想要的模型手術
3 - 使用 torch.save保存目標,結構應該還是按dict(model=state_dict).保存，並且，請注意擴展名是.pth而不是pkl
4 - 更改 MODEL.WEIGHT指向更改後的.pth文件地址。
前三步分別對應下面的trim_detectron_model.py腳本，注意別忘了第4步說明如下：
pretrained_path：就是從MODEL_ZOO下載的作者訓練好的結果，（後綴.pkl）
save_path：自己修改後用於fintune的模型地址，請注意擴展名是.pth
cfg：與模型對應的配置文件。
可以看到自己網絡裏的參數，比如我的是(截取部分，對應刪除就好了)：[ # FAST RCNN層，2個FC和分類迴歸層 'fc1000.bias', 'fc1000.weight', 'fc6.bias', 'fc6.weight', 'fc7.bias', 'fc7.weight', 'bbox_pred.bias', 'bbox_pred.weight', 'cls_score.bias', 'cls_score.weight', # Mask層： 'mask_fcn1.bias', 'mask_fcn1.weight', 'mask_fcn2.bias', 'mask_fcn2.weight', 'mask_fcn3.bias', 'mask_fcn3.weight', 'mask_fcn4.bias', 'mask_fcn4.weight', 'conv5_mask.bias', 'conv5_mask.weight', 'mask_fcn_logits.bias', 'mask_fcn_logits.weight', # RPN層 'rpn.head.conv.bias', 'rpn.head.conv.weight', 'rpn.head.bbox_pred.bias', 'rpn.head.bbox_pred.weight', 'rpn.head.cls_logits.bias', 'rpn.head.cls_logits.weight' # FPN層 'fpn_inner1.bias', 'fpn_inner1.weight', 'fpn_inner2.bias', 'fpn_inner2.weight', 'fpn_inner3.bias', 'fpn_inner3.weight', 'fpn_inner4.bias', 'fpn_inner4.weight', 'fpn_layer1.bias', 'fpn_layer1.weight', 'fpn_layer2.bias','fpn_layer2.weight', 'fpn_layer3.bias', 'fpn_layer3.weight', 'fpn_layer4.bias', 'fpn_layer4.weight', # 殘差層 ps：爲啥只有bn1和bn2？？？？ 'conv1.bias', 'conv1.weight', 'bn1.bias', 'bn1.weight', 'layer1.0.downsample.0.bias', 'layer1.0.downsample.1.bias', 'layer1.0.downsample.1.weight', 'layer1.0.downsample.0.weight', 'layer1.0.conv1.bias', 'layer1.0.bn1.bias', 'layer1.0.bn1.weight', 'layer1.0.conv1.weight', 'layer1.0.conv2.bias', 'layer1.0.bn2.bias', 'layer1.0.bn2.weight', 'layer1.0.conv2.weight', ................. 'layer4.0.downsample.0.bias', 'layer4.0.downsample.1.bias', 'layer4.0.downsample.1.weight', 'layer4.0.downsample.0.weight', 'layer4.0.conv1.bias', 'layer4.0.bn1.bias', 'layer4.0.bn1.weight', 'layer4.0.conv1.weight', ................. 'layer4.2.conv3.bias', 'layer4.2.bn3.bias', 'layer4.2.bn3.weight', 'layer4.2.conv3.weight', ] '''
import os
import torch
import argparse
from maskrcnn_benchmark.config import cfg
from maskrcnn_benchmark.utils.c2_model_loading import load_c2_format
def removekey(d, listofkeys):
r = dict(d)
for key in listofkeys:
print('key: {} is removed'.format(key))
r.pop(key)
return r
parser = argparse.ArgumentParser(description="Trim Detection weights and save in PyTorch format.")
parser.add_argument(
"--pretrained_path",
default="~/.torch/models/_detectron_35858933_12_2017_baselines_e2e_mask_rcnn_R-50-FPN_1x.yaml.01_48_14.DzEQe4wC_output_train_coco_2014_train%3Acoco_2014_valminusminival_generalized_rcnn_model_final.pkl",
help="path to detectron pretrained weight(.pkl)",
type=str,
)
parser.add_argument(
"--save_path",
default="./pretrained_model/mask_rcnn_R-50-FPN_1x_detectron_no_last_layers.pth",
help="path to save the converted model",
type=str,
)
parser.add_argument(
"--cfg",
default="configs/e2e_mask_rcnn_R_50_FPN_1x.yaml",
help="path to config file",
type=str,
)
args = parser.parse_args()
#
DETECTRON_PATH = os.path.expanduser(args.pretrained_path)
print('detectron path: {}'.format(DETECTRON_PATH))
cfg.merge_from_file(args.cfg)
_d = load_c2_format(cfg, DETECTRON_PATH)
print([key for key in _d]) #['model'] 只有這一個鍵
print([so for so in newdict["model"]]) #或者執行keys = [k for k in _d['model'].keys()]也可以
newdict = _d
#把後面的值從前面移除
newdict['model'] = removekey(_d['model'],
['cls_score.bias', 'cls_score.weight', 'bbox_pred.bias', 'bbox_pred.weight'])
torch.save(newdict, args.save_path)
print('saved to {}.'.format(args.save_path))
# 檢查是否按設計輸出：
w = torch.load(args.save_path)
print([k for k in w['model'].keys()])
記錄一個錯誤：
w = torch.load("X-101-32x8d.pth")
然而，發生錯誤：UnicodeDecodeError：'ascii'編解碼器無法解碼位置2中的字節0xad：序數不在範圍內（128）
可以通過使用pickle來解決這個錯誤：
import pickle
with open("X-101-32x8d.pkl", "rb") as f: w = pickle.load(f, encoding='latin1')
[loaded_state_dict]函數可以檢查函數中的哪些鍵 ，可以在這裏進行操作（maskrcnn-benchmark/maskrcnn_benchmark/utils/model_serialization.py )
不會影響repo中的其他部分。大概是
model_dict = model.state_dict()
loaded_state_dict = {k: v for k, v in loaded_state_dict.items() if k in model_dict and "roi_heads" not in k}
model_dict.update(loaded_state_dict)
如果是這樣，則無需經過4個步驟。只用編輯 [load_state_dict method]
align_and_update_state_dicts(model_state_dict, loaded_state_dict)
**步驟1**。運行demo.py以測試您的軟件包是否已成功安裝。同時，您可以保存預先訓練的模型，該模型已從pkl格式轉換爲pth格式，可以通過pytorch模型加載。該模型是演示中COCODEmo對象的數據，因此您可以保存預訓練模型的權重，如下所示
pretrained_model_path = "./pretrained_model/maskRCNN.pth"
check_point = {'state': coco_demo.model.state_dict()}
torch.save(check_point, pretrained_model_path)
這樣，無需查看[_load_file fun](https://github.com/facebookresearch/maskrcnn-benchmark/blob/f25c6cff92d32d92abe8965d68401004e90c8bee/maskrcnn_benchmark/utils/checkpoint.py#L117).另外還有個保存權重到OrderedDict的方法：(https://github.com/pytorch/pytorch/blob/828cb18fa35c7c132ab920de1c0dc6d859f152d6/torch/nn/modules/module.py#L602)
retrained_model = torch.load(checkpoint_path)['model']
model_dict = self.model.state_dict()
pretrained_dict = {k: v for k, v in pretrained_model.items() if k in model_dict and "roi_heads" not in k}
model_dict.update(pretrained_dict)
**step 2**. 使用configs文件夾中的yaml文件創建fastercnn或maskRCNN模型。執行此操作時，您需要調用cfg.merge_from_list方法來更改[_C.MODEL.ROI_BOX_HEAD.NUM_CLASSES = 81]到數據集中的類數（後臺在此實現中不計算）。(https://github.com/facebookresearch/maskrcnn-benchmark/blob/f25c6cff92d32d92abe8965d68401004e90c8bee/maskrcnn_benchmark/config/defaults.py#L182)
**step 3**.需要創建兩個類。
第一個類：[torchvision.datasets.coco.CocoDetection] (
maskrcnn-benchmark/maskrcnn_benchmark/data/datasets/coco.py
class COCODataset(torchvision.datasets.coco.CocoDetection):
獲取一個圖像及其對應的bbox和標籤，而不將Bbox和標籤捆綁在BoxList對象中。也就是 return img, target, idx
此類與[COCODataset](https://github.com/facebookresearch/maskrcnn-benchmark/blob/f25c6cff92d32d92abe8965d68401004e90c8bee/maskrcnn_benchmark/data/datasets/coco.py#L9). 在每次迭代中，該類生成一個元組（img，bbox，label）,其中img由Image.open(...).convert('RGB')生成;
bbox 是 2d numpy 數組， label是1d 數組(mask的話就是掩碼信息).
第二類繼承自第一類，這個類的主要職責是將一個圖像的真實包裝在一個BoxList對象中，類似於[line]
| img, anno = super(COCODataset, self).__getitem__(idx) |
(https://github.com/facebookresearch/maskrcnn-benchmark/blob/f25c6cff92d32d92abe8965d68401004e90c8bee/maskrcnn_benchmark/data/datasets/coco.py#L43).
**step 4**. 對於pytorch預訓練模型，權重在字典中被分類，其中鍵是layer的名稱。因此，您可以排除不存在的layer或者您想要從頭開始訓練，如下所示：pretrained_model = torch.load(checkpoint_path)['state']
城市景觀數據集cityscapes與coco
def clip_weights_from_pretrain_of_coco_to_cityscapes(f, out_file):
""""""
from maskrcnn_benchmark.config.paths_catalog import COCO_CATEGORIES
from maskrcnn_benchmark.config.paths_catalog import CITYSCAPES_FINE_CATEGORIES
coco_cats = COCO_CATEGORIES
cityscapes_cats = CITYSCAPES_FINE_CATEGORIES
coco_cats_to_inds = dict(zip(coco_cats, range(len(coco_cats))))
cityscapes_cats_to_inds = dict(
zip(cityscapes_cats, range(len(cityscapes_cats)))
)
checkpoint = torch.load(f)
m = checkpoint['model']
weight_names = {
"cls_score": "module.roi_heads.box.predictor.cls_score.weight",
"bbox_pred": "module.roi_heads.box.predictor.bbox_pred.weight",
"mask_fcn_logits": "module.roi_heads.mask.predictor.mask_fcn_logits.weight",
}
bias_names = {
"cls_score": "module.roi_heads.box.predictor.cls_score.bias",
"bbox_pred": "module.roi_heads.box.predictor.bbox_pred.bias",
"mask_fcn_logits": "module.roi_heads.mask.predictor.mask_fcn_logits.bias",
}
representation_size = m[weight_names["cls_score"]].size(1)
cls_score = nn.Linear(representation_size, len(cityscapes_cats))
nn.init.normal_(cls_score.weight, std=0.01)
nn.init.constant_(cls_score.bias, 0)
representation_size = m[weight_names["bbox_pred"]].size(1)
class_agnostic = m[weight_names["bbox_pred"]].size(0) != len(coco_cats) * 4
num_bbox_reg_classes = 2 if class_agnostic else len(cityscapes_cats)
bbox_pred = nn.Linear(representation_size, num_bbox_reg_classes * 4)
nn.init.normal_(bbox_pred.weight, std=0.001)
nn.init.constant_(bbox_pred.bias, 0)
dim_reduced = m[weight_names["mask_fcn_logits"]].size(1)
mask_fcn_logits = Conv2d(dim_reduced, len(cityscapes_cats), 1, 1, 0)
nn.init.constant_(mask_fcn_logits.bias, 0)
nn.init.kaiming_normal_(
mask_fcn_logits.weight, mode="fan_out", nonlinearity="relu"
)
def _copy_weight(src_weight, dst_weight):
for ix, cat in enumerate(cityscapes_cats):
if cat not in coco_cats:
continue
jx = coco_cats_to_inds[cat]
dst_weight[ix] = src_weight[jx]
return dst_weight
def _copy_bias(src_bias, dst_bias, class_agnostic=False):
if class_agnostic:
return dst_bias
return _copy_weight(src_bias, dst_bias)
m[weight_names["cls_score"]] = _copy_weight(
m[weight_names["cls_score"]], cls_score.weight
)
m[weight_names["bbox_pred"]] = _copy_weight(
m[weight_names["bbox_pred"]], bbox_pred.weight
)
m[weight_names["mask_fcn_logits"]] = _copy_weight(
m[weight_names["mask_fcn_logits"]], mask_fcn_logits.weight
)
m[bias_names["cls_score"]] = _copy_bias(
m[bias_names["cls_score"]], cls_score.bias
)
m[bias_names["bbox_pred"]] = _copy_bias(
m[bias_names["bbox_pred"]], bbox_pred.bias, class_agnostic
)
m[bias_names["mask_fcn_logits"]] = _copy_bias(
m[bias_names["mask_fcn_logits"]], mask_fcn_logits.bias
)
print("f: {}\nout_file: {}".format(f, out_file))
torch.save(m, out_file)
對最後一層進行手術的兩種示例：
Download the model from MODEL_ZOO
wget https://download.pytorch.org/models/maskrcnn/e2e_mask_rcnn_X_101_32x8d_FPN_1x.pth
// Load the model in python
python
import torch
model = torch.load("e2e_mask_rcnn_X_101_32x8d_FPN_1x.pth ")
// Remove the previous training parameters.
del model['iteration']
del model['scheduler']
del model['optimizer']
// Remove the output layers in COCO, these are the mismatched layers you saw.
//Second stage prediction
del model["model"]["module.roi_heads.box.predictor.cls_score.weight"]
del model["model"]["module.roi_heads.box.predictor.cls_score.bias"]
del model["model"]["module.roi_heads.box.predictor.bbox_pred.weight"]
del model["model"]["module.roi_heads.box.predictor.bbox_pred.bias"]
//mask prediction
del model["model"]["module.roi_heads.mask.predictor.mask_fcn_logits.weight"]
del model["model"]["module.roi_heads.mask.predictor.mask_fcn_logits.bias"]
// RPN
del model["model"]["module.rpn.head.cls_logits.weight"]
del model["model"]["module.rpn.head.cls_logits.bias"]
del model["model"]["module.rpn.head.bbox_pred.weight"]
del model["model"]["module.rpn.head.bbox_pred.bias"]
//save the model
torch.save(model, "modified_model.pth")
Then use modified_model.pth in your MODEL.WEIGHT
def delete_net_weights_for_finetune(
model_file,
out_file,
rpn_final_convs=False,
bbox_final_fcs=True,
mask_final_conv=True
):
del_keys = []
checkpoint = torch.load(model_file)
print("keys: {}".format(checkpoint.keys()))
m = checkpoint['model']
if rpn_final_convs:
# 'module.rpn.anchor_generator.cell_anchors.0',
# 'module.rpn.anchor_generator.cell_anchors.1',
# 'module.rpn.anchor_generator.cell_anchors.2',
# 'module.rpn.anchor_generator.cell_anchors.3',
# 'module.rpn.anchor_generator.cell_anchors.4'
# 'module.rpn.head.cls_logits.weight',
# 'module.rpn.head.cls_logits.bias',
# 'module.rpn.head.bbox_pred.weight',
# 'module.rpn.head.bbox_pred.bias',
del_keys.extend([
k for k in m.keys() if k.find("rpn.anchor_generator") is not -1
])
del_keys.extend([
k for k in m.keys() if k.find("rpn.head.cls_logits") is not -1
])
del_keys.extend([
k for k in m.keys() if k.find("rpn.head.bbox_pred") is not -1
])
if bbox_final_fcs:
# 'module.roi_heads.box.predictor.cls_score.weight',
# 'module.roi_heads.box.predictor.cls_score.bias',
# 'module.roi_heads.box.predictor.bbox_pred.weight',
# 'module.roi_heads.box.predictor.bbox_pred.bias',
del_keys.extend([
k for k in m.keys() if k.find(
"roi_heads.box.predictor.cls_score"
) is not -1
])
del_keys.extend([
k for k in m.keys() if k.find(
"roi_heads.box.predictor.bbox_pred"
) is not -1
])
if mask_final_conv:
# 'module.roi_heads.mask.predictor.mask_fcn_logits.weight',
# 'module.roi_heads.mask.predictor.mask_fcn_logits.bias',
del_keys.extend([
k for k in m.keys() if k.find(
"roi_heads.mask.predictor.mask_fcn_logits"
) is not -1
])
for k in del_keys:
print("del k: {}".format(k))
del m[k]
# checkpoint['model'] = m
print("f: {}\nout_file: {}".format(f, out_file))
recursively_mkdirs(os.path.dirname(out_file))
torch.save({"model": m}, out_file)