14損失函數

原創 G5Lorenzo 2020-06-16 05:40

一、損失函數概念

1.1 損失函數是什麼？

圖中綠色方塊爲真實數據點，藍色直線爲線性迴歸模型的結果，可以看模型輸出點與真實數據點存在一定的差距，而這個差距常用損失函數來進行描述
在這裏插入圖片描述
損失函數：衡量模型輸出與真實標籤的差異

損失函數(Loss Function): 計算一個樣本的損失值

  • Loss=f(y^,y)Loss=f(\hat y,y)

代價函數(Cost Function): 計算整個訓練集樣本的損失的平均值

  • Cost=1N∑iNf(y^i,yi)Cost=\frac{1}{N}\sum \limits _i ^N f(\hat y_i, y_i)

目標函數(Objective Function):

  • Obj=Cost+RegularizationObj = Cost + Regularization

1.2 pytorch中的Loss

class _Loss (Module):
	def __init__(self, size_average=None, reduce=None, reduction='mean'): 		
	
	super(_Loss, self).__init__() 
	
	if size_average is not None or reduce is not None: 
		self.reduction = _Reduction.legacy_get_string(size_average, reduce)
	else:self.reduction = reduction

pytorch中的loss也繼承於nn.Module，相當於一個網絡層，其中size_average和reduce這兩個參數已經被捨棄，不需要再使用

二、交叉熵的概念

交叉熵 = 信息熵 + 相對熵

信息熵：描述信息的不確定程度，熵越大，不確定性越大，數學描述爲自信息的期望

  • H(P)=E(x)∼P[I(x)]=−∑i=1NP(xi)logP(xi)H(P)=E(x)\sim P[I(x)]=-\sum \limits _{i=1}^NP(x_i)logP(x_i)

自信息：衡量單個輸出單個事件的不確定性

  • I(x)=−log[p(x)]I(x)=-log[p(x)]

交叉熵：衡量兩個分佈之間的相似度

  • H(P,Q)=−∑i=1NP(xi)logQ(xi)H(P, Q) = -\sum \limits _{i=1}^NP(x_i)logQ(x_i)

相對熵：又稱爲KL散度，用來衡量兩個分佈之間的差異(距離)

  • DKL(P,Q)=H(P,Q)−H(P)D_{KL}(P,Q)=H(P,Q)-H(P)
  • DKL(P,Q)=E(x)∼p[logP(x)Q(x)]=E(x)∼p[logP(x)−logQ(x)]=∑i=1NP(xi)[logP(xi)−logQ(xi)]=∑i=1NP(xi)logP(xi)−∑i=1NP(xi)logQ(xi)=H(P,Q)−H(P)D_{KL}(P,Q)=E(x)\sim p[log\frac{P(x)}{Q(x)}] \\ \qquad\qquad\quad =E(x)\sim p[logP(x)-logQ(x)] \\ \qquad\qquad\quad =\sum \limits _{i=1}^NP(x_i)[logP(x_i)-logQ(x_i)] \\ \qquad\qquad\quad=\sum \limits _{i=1}^NP(x_i)logP(x_i)-\sum \limits _{i=1}^NP(x_i)logQ(x_i) \\ \qquad\qquad\quad=H(P,Q)-H(P)

三、常用的損失函數

3.1 nn.CrossEntropyLoss

nn.CrossEntropyLoss(weight=None,
					size_average=None,
					ignore_index=-100,
					reduce=None,
					reduction='mean')

功能: nn.LogSoftmax()與nn.NLLLoss()結合,進行交叉熵計算
主要參數:

  • weight: 各類別的loss設置權值
  • ignore_index: 忽略某個類別，不計算損失函數值
  • reduction: 計算模式, 可爲none/sum/mean
    • none: 逐個元素或樣本計算
    • sum: 所有元素求和, 返回標量
    • mean: 加權平均, 返回標量

說明：

  • nn.LogSoftmax()將輸出歸一化爲概率取值範圍，即0到1之間，然後再使用nn.NLLLoss()計算交叉熵
  • 早期是使用size_average和reduce來設置計算模式，現在只使用reduction參數

交叉熵：

  • H(P,Q)=−∑i=1NP(xi)logQ(xi)H(P, Q) = -\sum \limits _{i=1}^NP(x_i)logQ(x_i)

交叉熵損失函數：
loss(x,class)=−log(exp(x[class])∑jexp(x[j]))=−x[class]+log(∑jexp(x[j]))loss(x,class)=-log(\frac{exp(x[class])}{\sum_jexp(x[j])})=-x[class]+log(\sum_jexp(x[j]))

  • 對比交叉熵公式，其中Q(x)就是softmax輸出
  • 因爲樣本已經取出來了，所以樣本P(x)=1，故沒有P(xi)項

loss(x,class)=weight[class](−x[class]+log(∑jexp(x[j])))loss(x,class)=weight[class](-x[class]+log(\sum_jexp(x[j])))

  • 當設置了weight參數時，各個類別的函數值就有了相應的權值

代碼：


import torch
import torch.nn as nn
import torch.nn.functional as F
import numpy as np

# fake data
inputs = torch.tensor([[1, 2], [1, 3], [1, 3]], dtype=torch.float)
target = torch.tensor([0, 1, 1], dtype=torch.long)

# ----------------------------------- CrossEntropy loss: reduction -----------------------------------
# flag = 0
flag = 1
if flag:
    # def loss function
    loss_f_none = nn.CrossEntropyLoss(weight=None, reduction='none')
    loss_f_sum = nn.CrossEntropyLoss(weight=None, reduction='sum')
    loss_f_mean = nn.CrossEntropyLoss(weight=None, reduction='mean')

    # forward
    loss_none = loss_f_none(inputs, target)
    loss_sum = loss_f_sum(inputs, target)
    loss_mean = loss_f_mean(inputs, target)

    # view
    print("Cross Entropy Loss:\n ", loss_none, loss_sum, loss_mean)

# --------------------------------- compute by hand
# flag = 0
flag = 1
if flag:

    idx = 0

    input_1 = inputs.detach().numpy()[idx]      # [1, 2]
    target_1 = target.numpy()[idx]              # [0]

    # 第一項
    x_class = input_1[target_1]

    # 第二項
    sigma_exp_x = np.sum(list(map(np.exp, input_1)))
    log_sigma_exp_x = np.log(sigma_exp_x)

    # 輸出loss
    loss_1 = -x_class + log_sigma_exp_x

    print("第一個樣本loss爲: ", loss_1)
 
# ----------------------------------- weight -----------------------------------
# flag = 0
flag = 1
if flag:
    # def loss function
    weights = torch.tensor([1, 2], dtype=torch.float)
    # weights = torch.tensor([0.7, 0.3], dtype=torch.float)

    loss_f_none_w = nn.CrossEntropyLoss(weight=weights, reduction='none')
    loss_f_sum = nn.CrossEntropyLoss(weight=weights, reduction='sum')
    loss_f_mean = nn.CrossEntropyLoss(weight=weights, reduction='mean')

    # forward
    loss_none_w = loss_f_none_w(inputs, target)
    loss_sum = loss_f_sum(inputs, target)
    loss_mean = loss_f_mean(inputs, target)

    # view
    print("\nweights: ", weights)
    print(loss_none_w, loss_sum, loss_mean)

# --------------------------------- compute by hand
flag = 0
# flag = 1
if flag:
    weights = torch.tensor([1, 2], dtype=torch.float)
    weights_all = np.sum(list(map(lambda x: weights.numpy()[x], target.numpy())))  # [0, 1, 1]  # [1 2 2]

    mean = 0
    loss_sep = loss_none.detach().numpy()
    for i in range(target.shape[0]):

        x_class = target.numpy()[i]
        tmp = loss_sep[i] * (weights.numpy()[x_class] / weights_all)
        mean += tmp

    print(mean)

運行結果：
在這裏插入圖片描述

3.2 nn.NLLLoss

nn.NLLLoss(weight=None,
		   size_average=None, 
		   ignore_index=-100, 
		   reduce=None, 
		   reduction='mean')

功能: 實現負對數似然函數中的負號功能
主要參數:

  • weight: 各類別的loss設置權值
  • ignore_index:忽略某個類別
  • reduction:計算模式,可爲none/sum/mean
    • none-逐個元素計算
    • sum-所有元素求和，返回標量
    • mean-加權平均,返回標量

損失函數：l(x,y)=L={l1,...,lN}T,   ln=−wynxn,ynl(x,y)=L=\{l_1,...,l_N\}^T,~~~l_n=-w_{y_n}x_{n,y_n}


import torch
import torch.nn as nn
import torch.nn.functional as F
import numpy as np

# fake data
inputs = torch.tensor([[1, 2], [1, 3], [1, 3]], dtype=torch.float)
target = torch.tensor([0, 1, 1], dtype=torch.long)

# ----------------------------------- 2 NLLLoss -----------------------------------
# flag = 0
flag = 1
if flag:

    weights = torch.tensor([1, 1], dtype=torch.float)

    loss_f_none_w = nn.NLLLoss(weight=weights, reduction='none')
    loss_f_sum = nn.NLLLoss(weight=weights, reduction='sum')
    loss_f_mean = nn.NLLLoss(weight=weights, reduction='mean')

    # forward
    loss_none_w = loss_f_none_w(inputs, target)
    loss_sum = loss_f_sum(inputs, target)
    loss_mean = loss_f_mean(inputs, target)

    # view
    print("\nweights: ", weights)
    print("NLL Loss", loss_none_w, loss_sum, loss_mean)

在這裏插入圖片描述
從運行結果中可看到，NNLLoss只不過是對輸入取了負號

3.3 nn.BCELoss

nn.BCELoss(weight=None, size_average=None, reduce=None, reduction='mean')

功能：二分類交叉熵
注意事項: 輸入值的取值應在[0,1]
主要參數:

  • weight: 各類別的loss設置權值
  • ignore_index: 忽略某個類別
  • reduction: 計算模式,可爲none/sum/mean
    • none-逐個元素計算
    • sum-所有元素求和,返回標量
    • mean-加權平均,返回標量

損失函數：ln=−wn[yn⋅logxn+(1−yn)⋅log(1−xn)]l_n=-w_n[y_n \cdot logx_n+(1-y_n)\cdot log(1-x_n)]

  • xnx_n是模型輸出概率取值
  • yny_n是標籤，取值爲0或1
import torch
import torch.nn as nn
import torch.nn.functional as F
import numpy as np

# fake data
inputs = torch.tensor([[1, 2], [1, 3], [1, 3]], dtype=torch.float)
target = torch.tensor([0, 1, 1], dtype=torch.long)
# ----------------------------------- 3 BCE Loss -----------------------------------
# flag = 0
flag = 1
if flag:
    inputs = torch.tensor([[1, 2], [2, 2], [3, 4], [4, 5]], dtype=torch.float)
    target = torch.tensor([[1, 0], [1, 0], [0, 1], [0, 1]], dtype=torch.float)

    target_bce = target

    # itarget
    inputs = torch.sigmoid(inputs)

    weights = torch.tensor([1, 1], dtype=torch.float)

    loss_f_none_w = nn.BCELoss(weight=weights, reduction='none')
    loss_f_sum = nn.BCELoss(weight=weights, reduction='sum')
    loss_f_mean = nn.BCELoss(weight=weights, reduction='mean')

    # forward
    loss_none_w = loss_f_none_w(inputs, target_bce)
    loss_sum = loss_f_sum(inputs, target_bce)
    loss_mean = loss_f_mean(inputs, target_bce)

    # view
    print("\nweights: ", weights)
    print("BCE Loss", loss_none_w, loss_sum, loss_mean)


# --------------------------------- compute by hand
# flag = 0
flag = 1
if flag:

    idx = 0

    x_i = inputs.detach().numpy()[idx, idx]
    y_i = target.numpy()[idx, idx]              #

    # loss
    # l_i = -[ y_i * np.log(x_i) + (1-y_i) * np.log(1-y_i) ]      # np.log(0) = nan
    l_i = -y_i * np.log(x_i) if y_i else -(1-y_i) * np.log(1-x_i)

    # 輸出loss
    print("BCE inputs: ", inputs)
    print("第一個loss爲: ", l_i)

在這裏插入圖片描述

3.4 nn.BCEWithLogitsLoss

nn.BCEWithLogitsLoss(weight=None, 
				     size_average=None, 
				     reduce=None, 
				     reduction='mean'
				     pos_weight=None)

功能: 結合Sigmoid與二分類交叉熵

注意事項: 網絡最後不加sigmoid函數
主要參數:

  • pos_weight : 正樣本的權值，用來均衡正負樣本
  • weight: 各類別的loss設置權值
  • ignore_index: 忽略某個類別
  • reduction: 計算模式,可爲none/sum/mea
    • none-逐個元素計算
    • sum-所有元素求和,返回標量
    • mean-加權平均,返回標量

ln=−wn[yn⋅logσ(xn)+(1−yn)⋅log(1−σ(xn))]l_n=-w_n[y_n \cdot log \sigma(x_n)+(1-y_n)\cdot log(1-\sigma(x_n))]

  • σ\sigma爲sigmoid函數

import torch
import torch.nn as nn
import torch.nn.functional as F
import numpy as np

# fake data
inputs = torch.tensor([[1, 2], [1, 3], [1, 3]], dtype=torch.float)
target = torch.tensor([0, 1, 1], dtype=torch.long)

# ----------------------------------- 4 BCE with Logis Loss -----------------------------------
# flag = 0
flag = 1
if flag:
    inputs = torch.tensor([[1, 2], [2, 2], [3, 4], [4, 5]], dtype=torch.float)
    target = torch.tensor([[1, 0], [1, 0], [0, 1], [0, 1]], dtype=torch.float)

    target_bce = target

    # inputs = torch.sigmoid(inputs)

    weights = torch.tensor([1, 1], dtype=torch.float)

    loss_f_none_w = nn.BCEWithLogitsLoss(weight=weights, reduction='none')
    loss_f_sum = nn.BCEWithLogitsLoss(weight=weights, reduction='sum')
    loss_f_mean = nn.BCEWithLogitsLoss(weight=weights, reduction='mean')

    # forward
    loss_none_w = loss_f_none_w(inputs, target_bce)
    loss_sum = loss_f_sum(inputs, target_bce)
    loss_mean = loss_f_mean(inputs, target_bce)

    # view
    print("\nweights: ", weights)
    print(loss_none_w, loss_sum, loss_mean)


# --------------------------------- pos weight

flag = 0
# flag = 1
if flag:
    inputs = torch.tensor([[1, 2], [2, 2], [3, 4], [4, 5]], dtype=torch.float)
    target = torch.tensor([[1, 0], [1, 0], [0, 1], [0, 1]], dtype=torch.float)

    target_bce = target

    # itarget
    # inputs = torch.sigmoid(inputs)

    weights = torch.tensor([1], dtype=torch.float)
    pos_w = torch.tensor([3], dtype=torch.float)        # 3

    loss_f_none_w = nn.BCEWithLogitsLoss(weight=weights, reduction='none', pos_weight=pos_w)
    loss_f_sum = nn.BCEWithLogitsLoss(weight=weights, reduction='sum', pos_weight=pos_w)
    loss_f_mean = nn.BCEWithLogitsLoss(weight=weights, reduction='mean', pos_weight=pos_w)

    # forward
    loss_none_w = loss_f_none_w(inputs, target_bce)
    loss_sum = loss_f_sum(inputs, target_bce)
    loss_mean = loss_f_mean(inputs, target_bce)

    # view
    print("\npos_weights: ", pos_w)
    print(loss_none_w, loss_sum, loss_mean)

在這裏插入圖片描述

3.5 nn.L1Loss

nn.L1Loss(size_average=None, reduce=None, reduction='mean')

功能: 計算inputs與target之差的絕對值

  • reduction: 計算模式, 可爲none/sum/mean
    • none: 逐個元素或樣本計算
    • sum: 所有元素求和, 返回標量
    • mean: 加權平均, 返回標量

損失函數：ln=∣xn−yn∣l_n=|x_n-y_n|


import torch
import torch.nn as nn
import torch.nn.functional as F
import matplotlib.pyplot as plt
import numpy as np
from tools.common_tools import set_seed

set_seed(1)  # 設置隨機種子

# ------------------------------------------------- 5 L1 loss ----------------------------------------------
# flag = 0
flag = 1
if flag:

    inputs = torch.ones((2, 2))
    target = torch.ones((2, 2)) * 3

    loss_f = nn.L1Loss(reduction='none')
    loss = loss_f(inputs, target)

    print("input:{}\ntarget:{}\nL1 loss:{}".format(inputs, target, loss))

在這裏插入圖片描述

3.6 nn.MSELoss

nn.MSELoss(size_average=None, reduce=None, reduction='mean')

功能: 計算inputs與target之差的平方
主要參數：

  • reduction: 計算模式, 可爲none/sum/mean
    • none: 逐個元素或樣本計算
    • sum: 所有元素求和, 返回標量
    • mean: 加權平均, 返回標量
import torch
import torch.nn as nn
import torch.nn.functional as F
import matplotlib.pyplot as plt
import numpy as np
from tools.common_tools import set_seed

set_seed(1)  # 設置隨機種子
# ------------------------------------------------- 6 MSE loss ----------------------------------------------

loss_f_mse = nn.MSELoss(reduction='none')
loss_mse = loss_f_mse(inputs, target)

print("MSE loss:{}".format(loss_mse))

運行結果：

在這裏插入圖片描述

3.7 nn.SmoothL1Loss

nn.SmoothL1Loss(size_average=None, reduce=None, reduction='mean')

功能：平滑的L1Loss

主要參數：

  • reduction: 計算模式, 可爲none/sum/mean
    • none: 逐個元素或樣本計算
    • sum: 所有元素求和, 返回標量
    • mean: 加權平均, 返回標量

損失函數：loss(x,y)=1n∑iziloss(x, y)=\frac{1}{n}\sum \limits _iz_i

  • 其中zi={0.5(xi−yi)2,    if∣xi−yi∣<1∣xi−yi∣−0.5,  otherwisez_i=\begin{cases} 0.5(x_i-y_i)^2,~~~~if|x_i-y_i|<1 \\ |x_i-y_i|-0.5, ~~otherwise \end{cases}
import torch
import torch.nn as nn
import torch.nn.functional as F
import matplotlib.pyplot as plt
import numpy as np
from tools.common_tools import set_seed

set_seed(1)  # 設置隨機種子

# ------------------------------------------------- 7 Smooth L1 loss ----------------------------------------------
# flag = 0
flag = 1
if flag:
    inputs = torch.linspace(-3, 3, steps=500)
    target = torch.zeros_like(inputs)

    loss_f = nn.SmoothL1Loss(reduction='none')

    loss_smooth = loss_f(inputs, target)

    loss_l1 = np.abs(inputs.numpy())

    plt.plot(inputs.numpy(), loss_smooth.numpy(), label='Smooth L1 Loss')
    plt.plot(inputs.numpy(), loss_l1, label='L1 loss')
    plt.xlabel('x_i - y_i')
    plt.ylabel('loss value')
    plt.legend()
    plt.grid()
    plt.show()

運行結果：
在這裏插入圖片描述

3.8 nn.PoissonNLLLoss

nn.PoissonNLLLoss(log_input=True, 
				  full=False, 
				  size_average=None, 
				  eps=1e-08, 
				  reduce=None,
				  eduction='mean')

功能：泊松分佈的負對數似然損失函數

主要參數:

  • log_input: 輸入是否爲對數形式,決定計算公式
    • log_input = True，loss(input, target) = exp(input)- target * input
    • log_input = False，loss(input, target) = input - target * log(input+eps)
  • full: 計算所有loss,默認爲False
  • eps: 修正項,避免log (input)爲nan
import torch
import torch.nn as nn
import torch.nn.functional as F
import matplotlib.pyplot as plt
import numpy as np
from tools.common_tools import set_seed

set_seed(1)  # 設置隨機種子

# ------------------------------------------------- 8 Poisson NLL Loss ----------------------------------------------
# flag = 0
flag = 1
if flag:

    inputs = torch.randn((2, 2))
    target = torch.randn((2, 2))

    loss_f = nn.PoissonNLLLoss(log_input=True, full=False, reduction='none')
    loss = loss_f(inputs, target)
    print("input:{}\ntarget:{}\nPoisson NLL loss:{}".format(inputs, target, loss))

# --------------------------------- compute by hand
# flag = 0
flag = 1
if flag:

    idx = 0

    loss_1 = torch.exp(inputs[idx, idx]) - target[idx, idx]*inputs[idx, idx]

    print("第一個元素loss:", loss_1)

在這裏插入圖片描述

3.9 nn.KLDivLoss

nn.KLDivLoss(size_average=None, reduce=None, reduction='mean')

功能: 計算KLD (divergence) , KL散度,相對熵

注意事項: 需提前將輸入轉換成probabilities，然後計算log,通過nn.logsoftmax()實現
主要參數:

  • reduction: none/sum/mean/batchmean-batchsize維度求平均值
    • none-逐個元素計算
    • sum-所有元素求和,返回標量
    • mean-加權平均,返回標量

在這裏插入圖片描述

import torch
import torch.nn as nn
import torch.nn.functional as F
import matplotlib.pyplot as plt
import numpy as np
from tools.common_tools import set_seed

set_seed(1)  # 設置隨機種子
# ------------------------------------------------- 9 KL Divergence Loss ----------------------------------------------
# flag = 0
flag = 1
if flag:

    inputs = torch.tensor([[0.5, 0.3, 0.2], [0.2, 0.3, 0.5]])
    inputs_log = torch.log(inputs)
    target = torch.tensor([[0.9, 0.05, 0.05], [0.1, 0.7, 0.2]], dtype=torch.float)

    loss_f_none = nn.KLDivLoss(reduction='none')
    loss_f_mean = nn.KLDivLoss(reduction='mean')
    loss_f_bs_mean = nn.KLDivLoss(reduction='batchmean')

    loss_none = loss_f_none(inputs, target)
    loss_mean = loss_f_mean(inputs, target)
    loss_bs_mean = loss_f_bs_mean(inputs, target)

    print("loss_none:\n{}\nloss_mean:\n{}\nloss_bs_mean:\n{}".format(loss_none, loss_mean, loss_bs_mean))

# --------------------------------- compute by hand
# flag = 0
flag = 1
if flag:

    idx = 0

    loss_1 = target[idx, idx] * (torch.log(target[idx, idx]) - inputs[idx, idx])

    print("第一個元素loss:", loss_1)

在這裏插入圖片描述

3.10 nn.MarginRankingLoss

nn.MarginRankingLoss(margin=0.0,
					 size_average=None, 
					 reduce=None, 
					 reduction='mean')

功能: 計算兩個向量之間的相似度,用於排序任務

特別說明: 該方法計算兩組數據之間的差異,返回一個n*n的loss矩陣

主要參數:

  • margin : 邊界值, x1與x2之間的差異值
  • reduction :計算模式,可爲none/sum/mean

損失函數：loss(x,y)=max(0,−y∗(x1−x2)+margin)loss(x,y)=max(0, -y*(x_1-x_2)+margin)

  • y=1時,希望x1比x2大,當x1>x2時,不產生loss
  • y=-1時,希望x2比x1大,當x2>x1時,不產生loss
import torch
import torch.nn as nn
import torch.nn.functional as F
import matplotlib.pyplot as plt
import numpy as np
from tools.common_tools import set_seed

# ---------------------------------------------- 10 Margin Ranking Loss --------------------------------------------
# flag = 0
flag = 1
if flag:

    x1 = torch.tensor([[1], [2], [3]], dtype=torch.float)
    x2 = torch.tensor([[2], [2], [2]], dtype=torch.float)

    target = torch.tensor([1, 1, -1], dtype=torch.float)

    loss_f_none = nn.MarginRankingLoss(margin=0, reduction='none')

    loss = loss_f_none(x1, x2, target)

    print(loss)

在這裏插入圖片描述

3.11 nn.MultiLabelMarginLoss

nn.MultiLabelMarginLoss(size_average=None, reduce=None, reduction='mean')

功能: 多標籤邊界損失函數
舉例: 四分類任務,樣本x屬於0類和3類,
標籤: [0, 3,-1,-1] ,不是[1,0,0,1]
主要參數:

  • reduction:計算模式,可爲none/sum/mean
    • none-逐個元素計算
    • sum-所有元素求和，返回標量
    • mean-加權平均,返回標量

損失函數：
在這裏插入圖片描述

import torch
import torch.nn as nn
import torch.nn.functional as F
import matplotlib.pyplot as plt
import numpy as np
from tools.common_tools import set_seed
# ---------------------------------------------- 11 Multi Label Margin Loss -----------------------------------------
# flag = 0
flag = 1
if flag:

    x = torch.tensor([[0.1, 0.2, 0.4, 0.8]])
    y = torch.tensor([[0, 3, -1, -1]], dtype=torch.long)

    loss_f = nn.MultiLabelMarginLoss(reduction='none')

    loss = loss_f(x, y)

    print(loss)

# --------------------------------- compute by hand
# flag = 0
flag = 1
if flag:

    x = x[0]
    item_1 = (1-(x[0] - x[1])) + (1 - (x[0] - x[2]))    # [0]
    item_2 = (1-(x[3] - x[1])) + (1 - (x[3] - x[2]))    # [3]

    loss_h = (item_1 + item_2) / x.shape[0]

    print(loss_h)

在這裏插入圖片描述

3.12 nn.SoftMarginLoss

nn.SoftMarginLoss(size_average=None, reduce=None, reduction='mean')

功能: 計算二分類的logistic損失
主要參數:

  • reduction:計算模式,可爲none/sum/mean
    • none-逐個元素計算
    • sum-所有元素求和，返回標量
    • mean-加權平均,返回標量

損失函數：
在這裏插入圖片描述

import torch
import torch.nn as nn
import torch.nn.functional as F
import matplotlib.pyplot as plt
import numpy as np
from tools.common_tools import set_seed

# ---------------------------------------------- 12 SoftMargin Loss -----------------------------------------
# flag = 0
flag = 1
if flag:

    inputs = torch.tensor([[0.3, 0.7], [0.5, 0.5]])
    target = torch.tensor([[-1, 1], [1, -1]], dtype=torch.float)

    loss_f = nn.SoftMarginLoss(reduction='none')

    loss = loss_f(inputs, target)

    print("SoftMargin: ", loss)

# --------------------------------- compute by hand
# flag = 0
flag = 1
if flag:

    idx = 0

    inputs_i = inputs[idx, idx]
    target_i = target[idx, idx]

    loss_h = np.log(1 + np.exp(-target_i * inputs_i))

    print(loss_h)

在這裏插入圖片描述

3.13 nn.MultiLabelSoftMarginLoss

nn.MultiLabelSoftMarginLoss(weight=None,size_average=None, reduce=None, reduction='mean')

功能：SoftMarginLoss的多標籤版本

主要參數:

  • weight:各類別的loss設置權值
  • reduction:計算模式,可爲none/sum/mean
    • none-逐個元素計算
    • sum-所有元素求和，返回標量
    • mean-加權平均,返回標量

損失函數：
在這裏插入圖片描述

import torch
import torch.nn as nn
import torch.nn.functional as F
import matplotlib.pyplot as plt
import numpy as np
from tools.common_tools import set_seed

# ---------------------------------------------- 13 MultiLabel SoftMargin Loss -----------------------------------------
# flag = 0
flag = 1
if flag:

    inputs = torch.tensor([[0.3, 0.7, 0.8]])
    target = torch.tensor([[0, 1, 1]], dtype=torch.float)

    loss_f = nn.MultiLabelSoftMarginLoss(reduction='none')

    loss = loss_f(inputs, target)

    print("MultiLabel SoftMargin: ", loss)

# --------------------------------- compute by hand
# flag = 0
flag = 1
if flag:

    i_0 = torch.log(torch.exp(-inputs[0, 0]) / (1 + torch.exp(-inputs[0, 0])))

    i_1 = torch.log(1 / (1 + torch.exp(-inputs[0, 1])))
    i_2 = torch.log(1 / (1 + torch.exp(-inputs[0, 2])))

    loss_h = (i_0 + i_1 + i_2) / -3

    print(loss_h)

在這裏插入圖片描述

3.14 nn.MultiMarginLoss

nn.MultiMarginLoss(p=1, margin=1.0, weight=None, size_average=None, reduce=None, reduction='mean')

功能: 計算多分類的摺頁損失

主要參數:

  • P: 可選1或2
  • weight: 各類別的loss設置權值
  • margin: 邊界值
  • reduction:計算模式,可爲none/sum/mean
    • none-逐個元素計算
    • sum-所有元素求和，返回標量
    • mean-加權平均,返回標量

損失函數：
在這裏插入圖片描述
在這裏插入圖片描述

import torch
import torch.nn as nn
import torch.nn.functional as F
import matplotlib.pyplot as plt
import numpy as np
from tools.common_tools import set_seed
# ---------------------------------------------- 14 Multi Margin Loss -----------------------------------------
# flag = 0
flag = 1
if flag:

    x = torch.tensor([[0.1, 0.2, 0.7], [0.2, 0.5, 0.3]])
    y = torch.tensor([1, 2], dtype=torch.long)

    loss_f = nn.MultiMarginLoss(reduction='none')

    loss = loss_f(x, y)

    print("Multi Margin Loss: ", loss)

# --------------------------------- compute by hand
# flag = 0
flag = 1
if flag:

    x = x[0]
    margin = 1

    i_0 = margin - (x[1] - x[0])
    # i_1 = margin - (x[1] - x[1])
    i_2 = margin - (x[1] - x[2])

    loss_h = (i_0 + i_2) / x.shape[0]

    print(loss_h)

在這裏插入圖片描述

3.15 nn.TripletMarginLoss

nn.TripletMarginLoss(margin=1.0, 
					 p=2.0, 
					 eps=1e-06, 
					 swap=False, 
					 size_average=None, 
					 reduce=None, 
					 reduction='mean')

功能: 計算三元組損失,人臉驗證中常用
主要參數:

  • p:範數的階,默認爲2
  • margin :邊界值
  • reduction: 計算模式, 可爲none/sum/mean
    • none-逐個元素計算
    • sum-所有元素求和，返回標量
    • mean-加權平均,返回標量
      損失函數：
      在這裏插入圖片描述
      在這裏插入圖片描述
      訓練時，希望anchor與positive的距離近，與negative的距離遠
import torch
import torch.nn as nn
import torch.nn.functional as F
import matplotlib.pyplot as plt
import numpy as np
from tools.common_tools import set_seed
# ---------------------------------------------- 15 Triplet Margin Loss -----------------------------------------
# flag = 0
flag = 1
if flag:

    anchor = torch.tensor([[1.]])
    pos = torch.tensor([[2.]])
    neg = torch.tensor([[0.5]])

    loss_f = nn.TripletMarginLoss(margin=1.0, p=1)

    loss = loss_f(anchor, pos, neg)

    print("Triplet Margin Loss", loss)

# --------------------------------- compute by hand
# flag = 0
flag = 1
if flag:

    margin = 1
    a, p, n = anchor[0], pos[0], neg[0]

    d_ap = torch.abs(a-p)
    d_an = torch.abs(a-n)

    loss = d_ap - d_an + margin

    print(loss)

在這裏插入圖片描述

3.16 nn.HingeEmbeddingLoss

nn.HingeEmbeddingLoss(margin=1.0, size_average=None, reduce=None, reduction='mean')

功能: 計算兩個輸入的相似性,常用於非線性embedding和半監督學習
特別注意: 輸入x應爲兩個輸入之差的絕對值
主要參數:

  • margin :邊界值
  • reduction:計算模式,可爲none/sum/mean
    • none-逐個元素計算
    • sum-所有元素求和，返回標量
    • mean-加權平均,返回標量

損失函數：
在這裏插入圖片描述

import torch
import torch.nn as nn
import torch.nn.functional as F
import matplotlib.pyplot as plt
import numpy as np
from tools.common_tools import set_seed

# ---------------------------------------------- 16 Hinge Embedding Loss -----------------------------------------
# flag = 0
flag = 1
if flag:

    inputs = torch.tensor([[1., 0.8, 0.5]])
    target = torch.tensor([[1, 1, -1]])

    loss_f = nn.HingeEmbeddingLoss(margin=1, reduction='none')

    loss = loss_f(inputs, target)

    print("Hinge Embedding Loss", loss)

# --------------------------------- compute by hand
# flag = 0
flag = 1
if flag:
    margin = 1.
    loss = max(0, margin - inputs.numpy()[0, 2])

    print(loss)

在這裏插入圖片描述

3.17 nn.CosineEmbeddingLoss

nn.CosineEmbeddingLoss(margin=0.0, size_average =None, reduce=None, reduction='mean')

功能: 採用餘弦相似度計算兩個輸入的相似性
主要參數:

  • margin :可取值[-1,1],推薦爲[0, 0.5]
  • reduction :計算模式,可爲none/sum/mean
    • none-逐個元素計算
    • sum-所有元素求和，返回標量
    • mean-加權平均,返回標量

損失函數：
在這裏插入圖片描述

import torch
import torch.nn as nn
import torch.nn.functional as F
import matplotlib.pyplot as plt
import numpy as np
from tools.common_tools import set_seed


# ---------------------------------------------- 17 Cosine Embedding Loss -----------------------------------------
# flag = 0
flag = 1
if flag:

    x1 = torch.tensor([[0.3, 0.5, 0.7], [0.3, 0.5, 0.7]])
    x2 = torch.tensor([[0.1, 0.3, 0.5], [0.1, 0.3, 0.5]])

    target = torch.tensor([[1, -1]], dtype=torch.float)

    loss_f = nn.CosineEmbeddingLoss(margin=0., reduction='none')

    loss = loss_f(x1, x2, target)

    print("Cosine Embedding Loss", loss)

# --------------------------------- compute by hand
# flag = 0
flag = 1
if flag:
    margin = 0.

    def cosine(a, b):
        numerator = torch.dot(a, b)
        denominator = torch.norm(a, 2) * torch.norm(b, 2)
        return float(numerator/denominator)

    l_1 = 1 - (cosine(x1[0], x2[0]))

    l_2 = max(0, cosine(x1[0], x2[0]))

    print(l_1, l_2)

在這裏插入圖片描述

3.18 nn.CTCLoss

torch.nn.CTCLoss(blank=0, reduction='mean', zero_infinity=F alse)

功能: 計算CTC損失,解決時序類數據的分類connectionist Temporal classification
主要參數:

  • blank: blank label
  • zero_infinity : 無窮大的值或梯度置0
  • reduction:計算模式,可爲none/sum/mean

損失函數可參考:
A. Graves et al.: Connectionist Temporal Classification:Labelling Unsegmented Sequence Data with RecurrentNeural Networks

import torch
import torch.nn as nn
import torch.nn.functional as F
import matplotlib.pyplot as plt
import numpy as np
from tools.common_tools import set_seed


# ---------------------------------------------- 18 CTC Loss -----------------------------------------
flag = 0
# flag = 1
if flag:
    T = 50      # Input sequence length
    C = 20      # Number of classes (including blank)
    N = 16      # Batch size
    S = 30      # Target sequence length of longest target in batch
    S_min = 10  # Minimum target length, for demonstration purposes

    # Initialize random batch of input vectors, for *size = (T,N,C)
    inputs = torch.randn(T, N, C).log_softmax(2).detach().requires_grad_()

    # Initialize random batch of targets (0 = blank, 1:C = classes)
    target = torch.randint(low=1, high=C, size=(N, S), dtype=torch.long)

    input_lengths = torch.full(size=(N,), fill_value=T, dtype=torch.long)
    target_lengths = torch.randint(low=S_min, high=S, size=(N,), dtype=torch.long)

    ctc_loss = nn.CTCLoss()
    loss = ctc_loss(inputs, target, input_lengths, target_lengths)

    print("CTC loss: ", loss)

在這裏插入圖片描述

發表評論
所有評論
還沒有人評論，想成為第一個評論的人麼? 請在上方評論欄輸入並且點擊發布.
相關文章

【pytorch】model.eval()和model.train()

eval() self.eval()和self.train(False)等價 eval()在測試之前加，否則有輸入數據即使不訓練，它也會改變權值 pytorch會自己把BatchNormalization和DropOut固定住，不

zxfhahaha 2020-07-05 06:33:33

Pytorch學習筆記【15】：Q-learning強化學習算法簡單實現

這個沒有基礎沒法看的，建議沒有基礎的先看看我的另一篇博客，會介紹強化學習以及Q-learng算法流程： https://blog.csdn.net/qq_36499794/article/details/103162841   一. 代碼

strong tyj 2020-07-03 12:43:26

Pytorch學習筆記【19】：利用dropout解決過擬合問題

如果對於什麼 叫過擬合不是很清楚，可以參考我的另一篇博客： https://blog.csdn.net/qq_36499794/article/details/103178812 注意 看代碼註釋以及 運行結果，最好自己去運行一遍 本篇代

strong tyj 2020-07-03 12:43:26

PyTorch保存網絡結構以及參數【 torch.save()、torch.load() 】

保存網絡結構以及參數 一、保存方式 二、pkl、pth文件區別 2.1 .pkl文件 2.2 .pth文件          對於pytorch保存網絡參數，大家一般可以看到有 .pkl文件 以及 .pth文件，對於這兩者有什麼區

Raywit 2020-07-02 05:11:29

PyTorch的inplace的理解

inplace的理解        我們平時看到的 nn.ReLU(inplace=True)、nn.LeakyReLU(inplace=True)，這些語句中的inplace是什麼意思？ inplace=True指的是進行原地

Raywit 2020-07-02 05:11:29

【PyTorch】常見錯誤: RuntimeError:one of the variables needed for gradient computation has been modified

【PyTorch】常見錯誤   錯誤： RuntimeError: one of the variables needed for gradient computation has been modified by an inpl

Raywit 2020-07-02 05:11:29

PyTorch中torch.utils.data.DataLoader加載數據

torch.utils.data.DataLoader使用方法 一、參數設置 二、實際應用        DataLoader是PyTorch中的一種數據類型，在PyTorch架構中訓練或者驗證模型經常要使用它，那麼怎麼生成以及使

Raywit 2020-07-02 05:11:29

pytorch資料總結

torch.optim.lr_scheduler：調整學習率 https://blog.csdn.net/qyhaill/article/details/103043637 Pytorch 使用不同版本的 cuda https://www

二狗diudiu 2020-07-01 15:51:57

Pytorch & Numpy 對照表

PyTorch 裏面處理的最基本的操作對象就是 Tensor（張量），表示的是一個多維的矩陣，比如零維就是一個點，一維就是向量，二維就是一般的矩陣，多維就相當於一個多維的數組，這和 numpy 是對應的，而且 PyTorch 的

ViatorSun 2020-07-01 05:00:09

PyTorch 大批量數據 如何訓練?

對於大多數的個人學習小夥伴來說，無法擁有一臺性能超強的深度學習主機，更沒有運算超羣的服務器來供自己訓練模型，但是又不得不對進行訓練時，矛盾就產生了！ 在深度學習訓練中，我們經常遇到 GPU 的內存太小的問題，如果我們的數據量比較大

ViatorSun 2020-07-01 05:00:09

詳解 MNIST 入門 Pytorch

MNIST雖然很簡單，但是值得我們學習的東西還是有很多的。 項目雖然簡單，但是個人建議還是將各個模塊分開創建，特別是對於新人而言，模塊化的創建更加清晰、易懂，我們看下項目都包含哪些部分： CNN模塊：卷積神經網絡的組成； tra

ViatorSun 2020-07-01 05:00:08

Pytorch模型 保存&恢復

keras中常使用 .h5 文件保存模型。而 Pytorch 保存數據的格式爲.t7文件 或者 .pt文件 或者 .pkl格式 .t7格式 是沿用 torch7 中讀取模型權重的方式 .pt格式 是Pytorch官方示例推薦使用

ViatorSun 2020-07-01 05:00:08

學習pytorch: 語義分割工具

文檔 Awesome-Pytorch-list: 提供了pytorch的各種工具，代碼，文檔等，下面的2,3是在裏面挑出來我使用過的工具。 可視化 tensorboardX: tensorboard的擴展，可以結合pytor

张王李刘赵孙杨 2020-06-29 10:48:01

學習pytorch: API 學習

torch torch.clamp(input, min, max, out=None) → Tensor Clamp all elements in input into the range[min, max] and re

张王李刘赵孙杨 2020-06-29 10:47:58

Pytorch控制打印矩陣的格式

原文地址 分類目錄——Pytorch 諸如長序列單行顯示，全部顯示（不縮略顯示），精度（保留小數點後幾位），是否科學計數法顯示等等。 直接用程序來說明 生成測試數據 import torch torch.random.man

BBJG_001 2020-06-27 23:31:17