【one way的pytorch學習筆記】(三)leaf 葉子(張量)

先導

在pytorch的tensor類中,有個is_leaf的屬性,姑且把它作爲葉子節點. is_leaf 爲False的時候,則不是葉子節點, is_leaf爲True的時候爲葉子節點(或者葉張量)

所以問題來了: leaf的作用是什麼?爲什麼要加 leaf?
我們都知道tensor中的 requires_grad()屬性,當requires_grad()爲True時我們將會記錄tensor的運算過程併爲自動求導做準備,但是並不是每個requires_grad()設爲True的值都會在backward的時候得到相應的grad.它還必須爲leaf.這就說明. leaf成爲了在 requires_grad()下判斷是否需要保留 grad的前提條件

具體的autograd的流程機制請移步下一篇:autograd的流程機制原理

is_leaf()

1. 按照慣例,所有requires_grad爲False的張量(Tensor) 都爲葉張量( leaf Tensor)
2. requires_grad爲True的張量(Tensor),如果他們是由用戶創建的,則它們是葉張量(leaf Tensor).這意味着它們不是運算的結果,因此gra_fn爲None
3. 只有是葉張量的tensor在反向傳播時纔會將本身的grad傳入的backward的運算中. 如果想得到當前tensor在反向傳播時的grad, 可以用retain_grad()這個屬性

例子:

>>> a = torch.rand(10, requires_grad=True)
>>> a.is_leaf
True
>>> b = torch.rand(10, requires_grad=True).cuda()
>>> b.is_leaf
False
# b was created by the operation that cast a cpu Tensor into a cuda Tensor
>>> c = torch.rand(10, requires_grad=True) + 2
>>> c.is_leaf
False
# c was created by the addition operation
>>> d = torch.rand(10).cuda()
>>> d.is_leaf
True
# d does not require gradients and so has no operation creating it (that is tracked by the autograd engine)
>>> e = torch.rand(10).cuda().requires_grad_()
>>> e.is_leaf
True
# e requires gradients and has no operations creating it
>>> f = torch.rand(10, requires_grad=True, device="cuda")
>>> f.is_leaf
True
# f requires grad, has no operation creating it