Dropout network, DropConnect network

Notations

• input $v$
• output $r$
• weight parameter $W \in \mathbb{R}^{d \times m}$
• activation function $a$
• mask $m$ for vector and $M$ for matrix

Dropout

• Randomly set activations of each layer to zero with probability $1-p$.
  $r = m \circ a(Wv),$
  $m_j \sim \text{\small Bernoulli}(p)$.
• As many activation functions have the property that $a(0)=0)$, we have
  $r = a(m \circ Wv).$

DropConnect

• Randomly set the weight of each layer to zero with probability $1-p$.
  $r = a(M \circ Wv),$
  $M_{ij} \sim \text{\small Bernoulli}(p)$.
• Each $M_{ij}$ is drawn independently for each example during training.
  The memory requirement for $M$'s grows with the size of each mini-batch, and therefore, the implementation needs to be carefully designed.
• overall model $f(x;\theta,M)$, where $\theta = \{W_g,W,W_s\}$
  $\begin{aligned} o=\mathbb{E}_M[f(x;\theta,M)]&=\sum_M p(M) f(x;\theta,M)\\ &=\frac{1}{|M|}\sum_M s(a(M \circ W) v); W_s) \quad \text{if } p = 0.5 \end{aligned}$

• inference (test stage)
  $\begin{aligned} r&=\frac{1}{|M|} \sum_M a((M \circ W)v))\\ r&\approx \frac{1}{Z} \sum_{z=1}^Z r_z \\ &\approx \frac{1}{Z} \sum_{z=1}^Z a(u_z), \end{aligned}$
  where $u_z \sim \mathcal{N}(pWv,p(1-p)(W \circ W)(v \circ v)$; $Z$ denotes the number of randoml samples drawn from the Gaussian distribution.
  Idea: approximate a sum of weighted Bernoulli random variables by a Gaussian random variable. Partially supported by the central limit theorem.

$\textcolor{red}{\text{\small 侷限性}}$:
Both techniques are suitable for fully connected layers only.