用例子來學習Pytorch
第二篇教程主要是通過例子來介紹Pytorch 教程傳送門
文中主要拋出兩個關於Pytorch的主要特徵
1.An n-dimensional Tensor, similar to numpy but can run on GPUs
一個N緯度的張量，和Numpy很類似但是在GPU上運行
2.Automatic differentiation for building and training neural networks
自動的區分神經網絡的構建和訓練過程
張量
熱身活動：用numpy構建神經網絡
# -*- coding: utf-8 -*-
import numpy as np
# N is batch size; D_in is input dimension;
# H is hidden dimension; D_out is output dimension.
N, D_in, H, D_out = 64, 1000, 100, 10
# Create random input and output data
x = np.random.randn(N, D_in)
y = np.random.randn(N, D_out)
# Randomly initialize weights
w1 = np.random.randn(D_in, H)
w2 = np.random.randn(H, D_out)
learning_rate = 1e-6
for t in range(500):
# Forward pass: compute predicted y
h = x.dot(w1)
h_relu = np.maximum(h, 0)
y_pred = h_relu.dot(w2)
# Compute and print loss
loss = np.square(y_pred - y).sum()
print(t, loss)
# Backprop to compute gradients of w1 and w2 with respect to loss
grad_y_pred = 2.0 * (y_pred - y)
grad_w2 = h_relu.T.dot(grad_y_pred)
grad_h_relu = grad_y_pred.dot(w2.T)
grad_h = grad_h_relu.copy()
grad_h[h < 0] = 0
grad_w1 = x.T.dot(grad_h)
# Update weights
w1 -= learning_rate * grad_w1
w2 -= learning_rate * grad_w2
解析：這段代碼體現了一個三層神經網絡的構建和梯度下降過程，首先用np.rand.randn 構建一個輸入和一個真實值，前面64，1000，100，10分別爲batch_size，輸入的維度大小，隱層的節點數量，輸出的緯度大小；
綜上我們可以知道輸入爲 (batch_size,1000) ,隱藏層與輸入層之間的參數矩陣w1爲 (1000,100) ,隱藏層與輸出層之間的參數量爲w2爲 (100,10)
代碼中學習率設置爲
後面for循環開始就是前向傳播和反向傳播了，range(500)代表進行500此迭代(Step)，
前向傳播的過程就是不斷的做矩陣乘法
最終得到的Y_pred的大小爲(batch_size,10)
損失函數爲誤差的平方之和
所以在反向傳播第一步，輸出層的梯度爲grad_y_pred = 2.0 * (y_pred - y)
根據鏈式求導法則,隱層的梯度爲grad_w2 = h_relu.T.dot(grad_y_pred)
Torch-Pytorch的基本組件
下面我們是tensor同樣實現，前向傳播和反向傳播(Manually implement)
# -*- coding: utf-8 -*-
import torch
dtype = torch.float
device = torch.device("cpu")
# device = torch.device("cuda:0") # Uncomment this to run on GPU
# N is batch size; D_in is input dimension;
# H is hidden dimension; D_out is output dimension.
N, D_in, H, D_out = 64, 1000, 100, 10
# Create random input and output data
x = torch.randn(N, D_in, device=device, dtype=dtype)
y = torch.randn(N, D_out, device=device, dtype=dtype)
# Randomly initialize weights
w1 = torch.randn(D_in, H, device=device, dtype=dtype)
w2 = torch.randn(H, D_out, device=device, dtype=dtype)
learning_rate = 1e-6
for t in range(500):
# Forward pass: compute predicted y
h = x.mm(w1) #torch.mm就是矩陣乘法
h_relu = h.clamp(min=0) #clamp是夾子的意思，這裏可以看做，只要小於0的值都會被視作0
y_pred = h_relu.mm(w2)
# Compute and print loss
loss = (y_pred - y).pow(2).sum().item() #item用於把一個僅含有一個元素的tensor
#轉化爲Python數字,同種還有tolist()
if t % 100 == 99:
print(t, loss)
# Backprop to compute gradients of w1 and w2 with respect to loss
grad_y_pred = 2.0 * (y_pred - y)
grad_w2 = h_relu.t().mm(grad_y_pred)
grad_h_relu = grad_y_pred.mm(w2.t())
grad_h = grad_h_relu.clone()
grad_h[h < 0] = 0
grad_w1 = x.t().mm(grad_h)
# Update weights using gradient descent
w1 -= learning_rate * grad_w1
w2 -= learning_rate * grad_w2
關於clamp函數的示意圖
自動求導機制
上面我們已經自己實現了前向傳播和反向傳播，對於鏈式求導法則有個初步的應用的，但是對於複雜的網絡來說，進行反向傳播的算法是困難的(hairy)。
在原版的英文材料裏面我們注意到兩個概念
1.node (是tensor，我們的數據)
2.edge(是function，我們運算過程中的方法）
backpropagation 反向傳播
# -*- coding: utf-8 -*-
import torch
dtype = torch.float
device = torch.device("cpu")
# device = torch.device("cuda:0") # Uncomment this to run on GPU
# N is batch size; D_in is input dimension;
# H is hidden dimension; D_out is output dimension.
N, D_in, H, D_out = 64, 1000, 100, 10
# Create random Tensors to hold input and outputs.
# Setting requires_grad=False indicates that we do not need to compute gradients
# with respect to these Tensors during the backward pass.
x = torch.randn(N, D_in, device=device, dtype=dtype)
y = torch.randn(N, D_out, device=device, dtype=dtype)
# Create random Tensors for weights.
# Setting requires_grad=True indicates that we want to compute gradients with
# respect to these Tensors during the backward pass.
w1 = torch.randn(D_in, H, device=device, dtype=dtype, requires_grad=True)
w2 = torch.randn(H, D_out, device=device, dtype=dtype, requires_grad=True)
learning_rate = 1e-6
for t in range(500):
# Forward pass: compute predicted y using operations on Tensors; these
# are exactly the same operations we used to compute the forward pass using
# Tensors, but we do not need to keep references to intermediate values since
# we are not implementing the backward pass by hand.
y_pred = x.mm(w1).clamp(min=0).mm(w2)
# Compute and print loss using operations on Tensors.
# Now loss is a Tensor of shape (1,)
# loss.item() gets the scalar value held in the loss.
loss = (y_pred - y).pow(2).sum()
if t % 100 == 99:
print(t, loss.item())
# Use autograd to compute the backward pass. This call will compute the
# gradient of loss with respect to all Tensors with requires_grad=True.
# After this call w1.grad and w2.grad will be Tensors holding the gradient
# of the loss with respect to w1 and w2 respectively.
loss.backward()
# Manually update weights using gradient descent. Wrap in torch.no_grad()
# because weights have requires_grad=True, but we don't need to track this
# in autograd.
# An alternative way is to operate on weight.data and weight.grad.data.
# Recall that tensor.data gives a tensor that shares the storage with
# tensor, but doesn't track history.
# You can also use torch.optim.SGD to achieve this.
with torch.no_grad():
w1 -= learning_rate * w1.grad
w2 -= learning_rate * w2.grad
# Manually zero the gradients after updating weights
w1.grad.zero_()
w2.grad.zero_()
有幾個點需要注意:
1.計算圖(Computation Graph)
就是用來記錄一個反向傳播過程中的計算圖，相當於一個計算流程的記錄，可以用於反向傳播，文中也提到，我們所有反向傳播的梯度被維繫在tensor中
2.with toch.no_grad():
有個上面那個概念，下面就很好理解了，更新梯度的過程不是我們反向傳播或者正向傳播的流程，只是我們爲了更新一下係數
3.grad_zero
那麼下面就更好解釋了，既然參數更細了，之前傳出來的梯度就意義不大了，因爲是針對之前的係數算出來了，我們要清除一下，用新的，因爲Pytorch的梯度在每次計算是累加的
定義新的自動求導函數
# -*- coding: utf-8 -*-
import torch
class MyReLU(torch.autograd.Function):
"""
We can implement our own custom autograd Functions by subclassing
torch.autograd.Function and implementing the forward and backward passes
which operate on Tensors.
"""
@staticmethod
def forward(ctx, input):
"""
In the forward pass we receive a Tensor containing the input and return
a Tensor containing the output. ctx is a context object that can be used
to stash information for backward computation. You can cache arbitrary
objects for use in the backward pass using the ctx.save_for_backward method.
"""
ctx.save_for_backward(input)
return input.clamp(min=0)
@staticmethod
def backward(ctx, grad_output):
"""
In the backward pass we receive a Tensor containing the gradient of the loss
with respect to the output, and we need to compute the gradient of the loss
with respect to the input.
"""
input, = ctx.saved_tensors
grad_input = grad_output.clone()
grad_input[input < 0] = 0
return grad_input
dtype = torch.float
device = torch.device("cpu")
# device = torch.device("cuda:0") # Uncomment this to run on GPU
# N is batch size; D_in is input dimension;
# H is hidden dimension; D_out is output dimension.
N, D_in, H, D_out = 64, 1000, 100, 10
# Create random Tensors to hold input and outputs.
x = torch.randn(N, D_in, device=device, dtype=dtype)
y = torch.randn(N, D_out, device=device, dtype=dtype)
# Create random Tensors for weights.
w1 = torch.randn(D_in, H, device=device, dtype=dtype, requires_grad=True)
w2 = torch.randn(H, D_out, device=device, dtype=dtype, requires_grad=True)
learning_rate = 1e-6
for t in range(500):
# To apply our Function, we use Function.apply method. We alias this as 'relu'.
relu = MyReLU.apply
# Forward pass: compute predicted y using operations; we compute
# ReLU using our custom autograd operation.
y_pred = relu(x.mm(w1)).mm(w2)
# Compute and print loss
loss = (y_pred - y).pow(2).sum()
if t % 100 == 99:
print(t, loss.item())
# Use autograd to compute the backward pass.
loss.backward()
# Update weights using gradient descent
with torch.no_grad():
w1 -= learning_rate * w1.grad
w2 -= learning_rate * w2.grad
# Manually zero the gradients after updating weights
w1.grad.zero_()
w2.grad.zero_()
關於相關函數的解釋請參見
Pytorch筆記04-自定義torch.autograd.Function
非常感謝大神的講解，這裏再一次驗證了，鏈式求導法則在反向傳播中發揮得玄妙
ctx可以代替爲self，說明在python中指向自身對象的參數不一定限制死名字是self，可以自定義