總說

記錄一些比較有用的pytorch代碼(有些是自己寫的, 有些是從網上看到的)
本文和Pytorch學習(十五)------ 雜項知識彙總估計會經常更新, 記錄一些瑣碎的東西.

class LFTNet(nn.Module):
  def __init__(self, params, tf_path=None, change_way=True):
    super(LFTNet, self).__init__()
    #
    # Some code here
    #
    model_params, ft_params = self.split_model_parameters()
    self.model_optim = torch.optim.Adam(model_params)
    self.ft_optim = torch.optim.Adam(ft_params, weight_decay=1e-8, lr=1e-3)

    # total epochs
    self.total_epoch = params.stop_epoch

  # split the parameters of feature-wise transforamtion layers and others
  def split_model_parameters(self):
    model_params = []
    ft_params = []
    for n, p in self.model.named_parameters():
      n = n.split('.')
      if n[-1] == 'gamma' or n[-1] == 'beta':
        ft_params.append(p)
      else:
        model_params.append(p)
    return model_params, ft_params

這種方法，就是直接在定義網絡時，將參數進行分離，並用獨立的優化器。另一種方法可以參考：
Pytorch學習（二十三）---- 不同layer用不同學習率（高級版本）

提取網絡特徵

對於內部是sequence構建的網絡

class VGG16FeatureExtractor(nn.Module):
    def __init__(self):
        super().__init__()
        vgg16 = models.vgg16(pretrained=True)
        self.enc_1 = nn.Sequential(*vgg16.features[:5])
        self.enc_2 = nn.Sequential(*vgg16.features[5:10])
        self.enc_3 = nn.Sequential(*vgg16.features[10:17])

        # fix the encoder
        for i in range(3):
            # 這種寫法挺好的啊!!!!!!!!!!!
            for param in getattr(self, 'enc_{:d}'.format(i + 1)).parameters():
                param.requires_grad = False

    def forward(self, image):
        results = [image]
        for i in range(3):
            func = getattr(self, 'enc_{:d}'.format(i + 1))
            results.append(func(results[-1]))
return results[1:]

對於內部不是單純用sequential構建的網絡

import torch
import torchvision.models as models
import torch.nn as nn


class ResNetFeat(nn.Module):
    def __init__(self):
        super(ResNetFeat, self).__init__()
        resnet50_pretrained = models.resnet50(pretrained=True)
        conv_modules = list(resnet50_pretrained.children())[:-2]  # all layers expect Adaptive AvgPool2D and FC
        pool_modules = [list(resnet50_pretrained.children())[-2]]  # layers Adaptive AvgPool2D: there is only one.
        fc_modules = [list(resnet50_pretrained.children())[-1]]  # fc modules (1 only)
        self.model_conv = nn.Sequential(*conv_modules)
        # 可以看到這裏用.children()獲得所有的層， 做成list
        # 再用sequential包起來
        # 值得注意的是，這樣的可以適用於任意網絡的截取
        self.pool = nn.Sequential(*pool_modules)
        self.fc = nn.Sequential(*fc_modules)

    def forward(self, image):
        feat2d = self.model_conv(image)
        print(feat2d.shape)
        featedpooled = self.pool(feat2d)
        # flatten the feature before feeding the fc.
        featdense = featedpooled.view(featedpooled.size(0), -1)
        pred = self.fc(featdense)
        return featdense, pred


if __name__ == "__main__":
    x_image = torch.randn(1, 3, 224, 224)
    featurizer = ResNetFeat()
    feat, pred = featurizer(x_image)
    print(feat.shape, pred.shape)

修改複雜Pretrained的網絡(如ResNet等)

當你要稍微改改已有的大網絡時, 其實可以直接將github中torchvision的相關文件複製, 比如改進ResNet, 先複製 resnet.py ,
這種大的網絡, 裏面的網絡有多個屬性構建, 每個屬性都是一個block

class ResNet(nn.Module):

    def __init__(self, block, layers, num_classes=1000, zero_init_residual=False):
        super(ResNet, self).__init__()
        self.inplanes = 64
        self.conv1 = nn.Conv2d(3, 64, kernel_size=7, stride=2, padding=3,
                               bias=False)
        self.bn1 = nn.BatchNorm2d(64)
        self.relu = nn.ReLU(inplace=True)
        self.maxpool = nn.MaxPool2d(kernel_size=3, stride=2, padding=1)
        self.layer1 = self._make_layer(block, 64, layers[0])
        # self.layer2 = self._make_layer(block, 128, layers[1], stride=2)
        # self.layer3 = self._make_layer(block, 256, layers[2], stride=2)
        # self.layer4 = self._make_layer(block, 512, layers[3], stride=2)
        # self.avgpool = nn.AdaptiveAvgPool2d((1, 1))
        # self.fc = nn.Linear(512 * block.expansion, num_classes)

	...
	
	def forward(self, x):
        x = self.conv1(x)
        x = self.bn1(x)
        x = self.relu(x)
        x = self.maxpool(x)

        x = self.layer1(x)
        # x = self.layer2(x)
        # x = self.layer3(x)
        # x = self.layer4(x)

        # x = self.avgpool(x)
        # x = x.view(x.size(0), -1)
        # x = self.fc(x)

        return x
        
 def resnet101(pretrained=False, **kwargs):
    """Constructs a ResNet-101 model.
    Args:
        pretrained (bool): If True, returns a model pre-trained on ImageNet
    """
    model = ResNet(Bottleneck, [3, 4, 23, 3], **kwargs)
    if pretrained:
        # 這裏一定要`False`, 才能將預先訓練的ResNet101的前10層權重複制進去.
        model.load_state_dict(model_zoo.load_url(model_urls['resnet101']), False)
    return model

比如, 拿到ResNet101前面10層的卷積, 這樣改就行了. 然後在其他文件,

from .resnet import resnet101
...
class ResBase(nn.Module):
    def __init__(self):
        super(ResBase, self).__init__()
        # front_end has been truncated till conv2_3
        self.front_end = resnet101(pretrained=True)
        self.max_pool = torch.nn.MaxPool2d(3, stride=2, padding=1, dilation=1, ceil_mode=False)
        # Add some layers you want
        self.back_end = torch.nn.Sequential(
            torch.nn.Conv2d(256, 128, 3, dilation=1, padding=1),
            torch.nn.ReLU(),
            torch.nn.Conv2d(128, 64, 3, dilation=1, padding=1),
            torch.nn.ReLU(),
            torch.nn.Conv2d(64, 3, 3, dilation=1, padding=1)
        )
    
    def forward(self, x):
        x = self.front_end(x)
        x = self.max_pool(x)
        return self.back_end(x)

    # reinit front_end, especially for resnet.
    def _initialize_weights(self):
        self.front_end = resnet101(pretrained=True)

**稍微注意一下, 這樣構建的網絡, 基本是 frond_end用原始weights進行, 再後面的back_end用隨機初始化. **
所以, 先整個網絡進行隨機初始化, 再調用resnet._initialize_weights()就行.

預訓練模型的簡單更改(如ResNet等)

上面的方法可以進行任意的更改, 畢竟你都直接複製文件了, 不過你如果只是簡單的改改fc層, 比如之前是分類1000類的, 現在仍舊用ResNet改成分類100類.

model_conv = torchvision.models.resnet18(pretrained=True)
for param in model_conv.parameters():
    param.requires_grad = False

# 知道FC層的輸入是幾維
num_ftrs = model_conv.fc.in_features
# 原本是從 num_ftrs到1000, 現在直接替換成100類.
model_conv.fc = nn.Linear(num_ftrs, 100)

model_conv = model_conv.to(device)
criterion = nn.CrossEntropyLoss()

# 注意: 這裏只是把新的fc層參數輸入optimizer中
optimizer_conv = optim.SGD(model_conv.fc.parameters(), lr=0.001, momentum=0.9)

# Decay LR by a factor of 0.1 every 7 epochs
exp_lr_scheduler = lr_scheduler.StepLR(optimizer_conv, step_size=7, gamma=0.1)

這個真的沒啥毛病, 很厲害. 在外面直接用新建的fc層替換掉老fc.
另外一點, 你可能覺得奇怪, 就是前面的參數requires_grad都是False了, 都不會計算梯度, 那麼放入optimizer不也行嗎
Setting requires grad to False means that no gradients will be computed for this Tensor. And so your_tensor.grad will contain it’s previous value (None or a Tensor) and will never be updated.The thing is that when you have momentum (or l2 normalization), even having a gradient of 0 will make your weights change. So if you don’t want to optimize these weights, you want to exclude them from the optimizer to be sure (note that some optimizer will completely ignore tensors that have a grad field set to None and what you proposed will work. But this is not True for all optimizers and thus not safe).

Sequential網絡中插入自定義層

Sequential的好處就是, 順序執行裏面包含的一層層網絡.

class Mnist_CNN(nn.Module):
    def __init__(self):
        super().__init__()
        self.conv1 = nn.Conv2d(1, 16, kernel_size=3, stride=2, padding=1)
        self.conv2 = nn.Conv2d(16, 16, kernel_size=3, stride=2, padding=1)
        self.conv3 = nn.Conv2d(16, 10, kernel_size=3, stride=2, padding=1)

    def forward(self, xb):
        xb = xb.view(-1, 1, 28, 28)
        xb = F.relu(self.conv1(xb))
        xb = F.relu(self.conv2(xb))
        xb = F.relu(self.conv3(xb))
        xb = F.avg_pool2d(xb, 4)
        return xb.view(-1, xb.size(1))

這裏你會說, 咦, 有時候我們想, 要不把view也作爲一個網絡的層把, 這樣直接print(net)的時候, 就可以很清楚, 知道最後拉成了一個向量之類的了.

# 定義一個萬能的Lambda層, 繼承`nn.Module`類, 傳入是任意函數
# 將函數作爲一個類屬性, 在forward中, 用該函數對輸入進行處理.
class Lambda(nn.Module):
    def __init__(self, func):
        super().__init__()
        self.func = func

    def forward(self, x):
        return self.func(x)

def preprocess(x):
    return x.view(-1, 1, 28, 28)

model = nn.Sequential(
    Lambda(preprocess),
    nn.Conv2d(1, 16, kernel_size=3, stride=2, padding=1),
    nn.ReLU(),
    nn.Conv2d(16, 16, kernel_size=3, stride=2, padding=1),
    nn.ReLU(),
    nn.Conv2d(16, 10, kernel_size=3, stride=2, padding=1),
    nn.ReLU(),
    nn.AvgPool2d(4),
    Lambda(lambda x: x.view(x.size(0), -1)),
)

opt = optim.SGD(model.parameters(), lr=lr, momentum=0.9)

高斯模糊

import math
import numbers
import torch
from torch import nn
from torch.nn import functional as F

class GaussianSmoothing(nn.Module):
    """
    Apply gaussian smoothing on a
    1d, 2d or 3d tensor. Filtering is performed seperately for each channel
    in the input using a depthwise convolution.
    Arguments:
        channels (int, sequence): Number of channels of the input tensors. Output will
            have this number of channels as well.
        kernel_size (int, sequence): Size of the gaussian kernel.
        sigma (float, sequence): Standard deviation of the gaussian kernel.
        dim (int, optional): The number of dimensions of the data.
            Default value is 2 (spatial).
    """
    def __init__(self, channels, kernel_size, sigma, dim=2):
        super(GaussianSmoothing, self).__init__()
        if isinstance(kernel_size, numbers.Number):
            kernel_size = [kernel_size] * dim
        if isinstance(sigma, numbers.Number):
            sigma = [sigma] * dim

        # The gaussian kernel is the product of the
        # gaussian function of each dimension.
        kernel = 1
        meshgrids = torch.meshgrid(
            [
                torch.arange(size, dtype=torch.float32)
                for size in kernel_size
            ]
        )
        for size, std, mgrid in zip(kernel_size, sigma, meshgrids):
            mean = (size - 1) / 2
            kernel *= 1 / (std * math.sqrt(2 * math.pi)) * \
                      torch.exp(-((mgrid - mean) / std) ** 2 / 2)

        # Make sure sum of values in gaussian kernel equals 1.
        kernel = kernel / torch.sum(kernel)

        # Reshape to depthwise convolutional weight
        kernel = kernel.view(1, 1, *kernel.size())
        kernel = kernel.repeat(channels, *[1] * (kernel.dim() - 1))

        self.register_buffer('weight', kernel)
        
        # change it to 1 if DO NOT want to be channel-wise.
        self.groups = channels

        if dim == 1:
            self.conv = F.conv1d
        elif dim == 2:
            self.conv = F.conv2d
        elif dim == 3:
            self.conv = F.conv3d
        else:
            raise RuntimeError(
                'Only 1, 2 and 3 dimensions are supported. Received {}.'.format(dim)
            )

    def forward(self, input):
        """
        Apply gaussian filter to input.
        Arguments:
            input (torch.Tensor): Input to apply gaussian filter on.
        Returns:
            filtered (torch.Tensor): Filtered output.
        """
        return self.conv(input, weight=self.weight, groups=self.groups)


smoothing = GaussianSmoothing(3, 5, 1)
input = torch.rand(1, 3, 100, 100)
input = F.pad(input, (2, 2, 2, 2), mode='reflect')
output = smoothing(input)

自定義操作放入Transform中

https://discuss.pytorch.org/t/using-a-special-function-as-a-transform/42760

Pytorch學習(二十) ------ 可能有用的代碼合集

總說

目錄

網絡不同層用不同學習率的方法

提取網絡特徵

對於內部是sequence構建的網絡

對於內部不是單純用sequential構建的網絡

修改複雜Pretrained的網絡(如ResNet等)

預訓練模型的簡單更改(如ResNet等)

Sequential網絡中插入自定義層

高斯模糊

自定義操作放入Transform中

[轉帖]使用NMT和pmap解決JVM資源泄漏問題原創

Python實現大麥網搶票的四大關鍵技術點解析

Python 安裝庫指令大全

salesforce零基礎學習（一百三十八）零碎知識點小總結（十）

一款開源的.NET程序集反編譯、編輯和調試神器

關於接口協議，你必須要知道這些！

2020年上半年數據庫系統工程師考試

基於 Milvus + LlamaIndex 實現高級 RAG

【2024-05-21】以茶會友

Pytorch學習（二十二）soft label的交叉熵loss的實現

Pytorch學習(二十) ------ 可能有用的代碼合集

Gumbel-Max trick或者reparameterization

PATH刪除指定路徑

anaconda各種問題彙總

https://yachay.unat.edu.pe/blog/index.php?comment_area=format_blog&comment_component=blog&comment_co

linux以太網驅動總結