Sigmoid函數:
g(z) = a = 1 / (1 + e ^(-z))
g'(z) = g(z) * (1 - g(z)) = a * (1-a)
while z = 10 g(z) ≈ 1 g'(z) = 1 * (1 - 1) ≈ 0
z = -10 g(z) ≈ 0 g'(z) = 0 * (1 - 0) ≈ 0
z = 0 g'(z) ≈ 0.5 g'(z) = 0.5 * (1 - 0.5) = 0.25
Tanh函數:
g(z) = a = tanh(z) = (e^z - e^(-z)) / (e^z + e^(-z))
g'(z) = 1 - (tanh(z))^2 = 1 - a^2
while z = 10 g(z) ≈ 1 g'(z) ≈ 0
z = -10 g(z) ≈ -1 g'(z) ≈ 0
z = 0 g'(z) ≈ 0 g'(z) = 1
ReLU函數和Leaky ReLU函數:
ReLU函數:
g(z) = a = max(0 , z)
g'(z) = 0 if z < 0
g'(z) = z if z >= 0
Leaky ReLU函數:
g(z) = a = max(0.01z , z)
g'(z) = 0.01 if z < 0
g'(z) = z if z >= 0
爲什麼使用非線性的激活函數:
因爲如果使用線性的激活函數,那麼輸入x跟輸出y之間的關係爲線性的,便可以不需要網絡結構,直接使用線性組合便可以.只有在輸出層極小可能使用線性激活函數,在隱含層都使用非線性激活函數.