What's Dropout?

Dropout is one of usual regularization methods in Machine Learning especially in Deep Learning. It could prevent model from over-fitting during train period.

Let's consider a condition that out data has some noisy. For example, a picture is added some noisy randomly, which results some areas becomes black from white. For people, that slightly change could be ignored and we would still realize its shape. But for the model which is trained from data, a bit change would impact its inference.

Naturally, we want our model have the ability to ignore noisy and do inference correctly. To get this ability, we could introduce some noisy to our model when it's training. It means that in our training period, we can add some noise to the data. In the other hand, we hope that distribution of the data would be changeless, which means its expectation value would be same.

\[E(x_i^{'}) = x_i \]

For example, we could change the value with probability.

\[\begin{equation*} x_i^{\prime}=\left\{\begin{array}{ll} 0 & \text { 概率爲 } p \\ \frac{x_i}{1-p} & \text { 其他情況 } \end{array}\right. \end{equation*} \]

As above equation displayed, we change \(x_i\) to 0 with probability p, and to \(\frac{x_i}{1-p}\) with probability 1- p. That's what Dropout does in training.