深度学习入门9787115485588 Chapter3

explain the activation function in neural network with an actual case and demonstrate it in python code 

In a neural network, an activation function is used to introduce non-linearity into the output of a neuron. It is applied to the weighted sum of the inputs and biases in a neuron神经元, which produces the neuron's output.

One commonly used activation function is the Rectified Linear Unit (ReLU), which is defined as:

f(x) = max(0, x)

 

The ReLU function returns the input value if it is greater than zero, and returns zero otherwise. This means that the function is "activated" only when the input is positive, and otherwise it has no effect. This non-linear behavior allows the neural network to model more complex functions than a purely linear model.

Here's an example of how to implement ReLU activation in Python using NumPy:

import numpy as np

# Define the ReLU function
def relu(x):
    return np.maximum(0, x)

# Example input vector
x = np.array([-1, 2, 3, -4, 0])

# Apply ReLU activation
y = relu(x)

# Print the output
print(y)

In this example, we define the ReLU function using NumPy's maximum function, which returns the element-wise maximum of two arrays. We then apply the function to an example input vector x and print the output. The ReLU activation function will set any negative input values to zero, while leaving positive values unchanged.

 

感知机的公式可以表示为

y=h(b+w1x1+w2x2)  式3.2

 

h(x)  式3.3

if x>0 h(x)=1

if x<=0  h(x)=0

h(x)将输入信号的总和转换为输出信号，这种函数一般称为激活函数activation function。

 

 

a=b+w1x1+w2x2  加权输入信号和偏置的总和。

y=h(a)  用h函数将a转换为输出y

 

 

上面的h表示的激活函数以阈值为界，一旦输入超过阈值，就切换输出。这样的函数称为阶跃函数。

因此，可以说感知机使用了阶跃函数作为激活函数。

 

How many function types are there for activation functions?

There are several types of activation functions that are commonly used in neural networks. Here are a few examples:

1. Sigmoid: This function is defined as f(x) = 1 / (1 + exp(-x)). It produces a smooth "S"-shaped curve that maps any input value to a value between 0 and 1. This function is often used in the output layer of a neural network to produce a probability value.

2. ReLU (Rectified Linear Unit): This function is defined as f(x) = max(0, x). It returns the input value if it is greater than zero, and returns zero otherwise. This function is commonly used in hidden layers of neural networks, as it introduces non-linearity and helps to prevent the vanishing gradient problem.

3. Tanh (hyperbolic tangent): This function is defined as f(x) = (exp(x) - exp(-x)) / (exp(x) + exp(-x)). It produces a smooth "S"-shaped curve that maps any input value to a value between -1 and 1. This function is similar to the sigmoid function, but it produces negative output values as well.

4. Softmax: This function is used in the output layer of a neural network that has multiple classes. It takes a vector of inputs and normalizes them so that the output values sum to 1. This produces a probability distribution over the classes.

There are also other types of activation functions, such as the Leaky ReLU, ELU, and Swish functions, among others. Each of these functions has different properties and may be more suitable for different types of neural networks or specific use cases.