GAN网络中的Normalization汇总

引用¹

$\hat{x_i} = \frac{1}{\sigma_i}(x_i-\mu_i) \\ {y_i} = \gamma\hat{x_i} + \beta \\ \mu_i = \frac{1}{m}\Sigma_{k\in S_i}x_k \\ \sigma_i = \frac{1}{m}\sqrt{\Sigma_{x_k\in S_i}(x_k-\mu_i)^2+\epsilon}\\$

Batch Normalization

对于BN来说，其中$S_i$是沿着$N, H, W$三个方向的。

def Batchnorm_simple_for_train(x, gamma, beta, bn_param):
"""
param:x    : 输入数据，设shape(B,H,W,C)
param:gama : 缩放因子  γ
param:beta : 平移因子  β
param:bn_param   : batchnorm所需要的一些参数
	eps      : 接近0的数，防止分母出现0
	momentum : 动量参数，一般为0.9， 0.99， 0.999
	running_mean ：滑动平均的方式计算新的均值，训练时计算，为测试数据做准备
	running_var  : 滑动平均的方式计算新的方差，训练时计算，为测试数据做准备
"""
	running_mean = bn_param['running_mean']  #shape = [B]
    running_var = bn_param['running_var']    #shape = [B]
	results = 0. # 建立一个新的变量
    
	x_mean=np.mean(x, axis=(0,1,2), keepdim=True)  # 计算x的均值
    x_var=np.var(x, axis=(0,1,2), keepdim=True)    # 计算方差
    x_normalized=(x-x_mean)/np.sqrt(x_var+eps)       # 归一化
    results = gamma * x_normalized + beta            # 缩放平移

    running_mean = momentum * running_mean + (1 - momentum) * x_mean
    running_var = momentum * running_var + (1 - momentum) * x_var
    
    #记录新的值
    bn_param['running_mean'] = running_mean
    bn_param['running_var'] = running_var 
    
	return results , bn_param
	
# https://blog.csdn.net/qq_25737169/article/details/79048516

Layer Normalization

对于LN来说，其中$S_i$是沿着$C, H, W$三个方向。LN在RNN的效果比较理想。在CNN的效果不如BN

def Layernorm(x, gamma, beta):

    # x_shape:[B, H, W， C]
    results = 0.
    eps = 1e-5

    x_mean = np.mean(x, axis=(1, 2, 3), keepdims=True)
    x_var = np.var(x, axis=(1, 2, 3), keepdims=True)
    x_normalized = (x - x_mean) / np.sqrt(x_var + eps)
    results = gamma * x_normalized + beta
    return results

Instance Normalization

对于IN来说，其中$S_i$是沿着$H, W$两个方向。
IN主要用于计算风格迁移。

def Instancenorm(x, gamma, beta):

    # x_shape:[B, H, W, C]
    results = 0.
    eps = 1e-5

    x_mean = np.mean(x, axis=(2, 3), keepdims=True)
    x_var = np.var(x, axis=(2, 3), keepdims=True)
    x_normalized = (x - x_mean) / np.sqrt(x_var + eps)
    results = gamma * x_normalized + beta
    return results

Group Normalization

对于LN来说，其中$S_i$是沿着$C, H, W$三个方向。是LN和IN的一般化表示方法。LN和IN分别是GN的两种特殊情况。
GN对于batchsize较小的情况下表现仍然稳定。

def GroupNorm(x, gamma, beta, G=32):

    # x_shape:[B, H, W, C]
    results = 0.
    eps = 1e-5
    shape = np.shape(x)
    x = np.reshape(x, (x.shape[0], x.shape[2], x.shape[3], G, x.shape[-1]/32))

    x_mean = np.mean(x, axis=(1, 2, 4), keepdims=True)
    x_var = np.var(x, axis=(1, 2, 4), keepdims=True)
    x_normalized = (x - x_mean) / np.sqrt(x_var + eps)
    results = gamma * x_normalized + beta
    results = np.reshape(results, shape)
    return results

Spectral Normalization²

我们先说一下Spectral Normalization的实现方式，之后再说其推理过程以及作用。
SN是针对超参数W进行的处理。

def spectral_norm(w, iteration=1):
    w_shape = w.shape.as_list()
    w = tf.reshape(w, [-1, w_shape[-1]])

    # 
    u = tf.get_variable("u", [1, w_shape[-1]], 
        initializer=tf.random_normal_initializer(), trainable=False)

    u_hat = u
    v_hat = None
    for i in range(iteration):
        """
        power iteration
        Usually iteration = 1 will be enough
        """
        v_ = tf.matmul(u_hat, tf.transpose(w))
        v_hat = tf.nn.l2_normalize(v_)

        u_ = tf.matmul(v_hat, w)
        u_hat = tf.nn.l2_normalize(u_)

    u_hat = tf.stop_gradient(u_hat)
    v_hat = tf.stop_gradient(v_hat)

    sigma = tf.matmul(tf.matmul(v_hat, w), tf.transpose(u_hat))

    with tf.control_dependencies([u.assign(u_hat)]):
        w_norm = w / sigma
        w_norm = tf.reshape(w_norm, w_shape)

    return w_norm

为啥要对卷积使用的参数进行SN，这就要回到最初讨论GAN的不足上了。
Iran GoodFellow发明GAN网络之后，大家就对这网络进行了深入的研究，发现这个玩意儿容易蹦，而且参数稍微设置的不顺心，训练结果就跟没训练差不多。这到底是啥原因呢。
³

AdaIN（自适应实例归一化，StyleGANs的方法）

Total_Variation Regularization⁴

这种归一化的作用是减少椒盐噪声，使得画面平滑。
下面采用手写和调用Tensorflow的API来使用上述过程：

import tensorflow as tf
from PIL import Image
import numpy as np

img_path = '' # 图片地址
img = Image.open(img_paht)
image = np.array(img)
image = tf.cast(image, tf.int32)

# 调用API
tv = tf.reduce_mean(tf.image.total_variation(image))

# 手动实现
tv_h = tf.reduce_sum(tf.abs(image[1:,:,:] - image[:-1,:,:]))
tv_w = tf.reduce_sum(tf.abs(image[:,1:,:] - image[:,:-1,:]))
tv2 = (tv_h + tv_w)

with tf.Session() as sess:
    tv, tv2 = sess.run([tv, tv2])
    print('tv = ', tv)
    print('tv2 = ', tv2)

结果如下：
tv = 34654796
tv2 = 34654796

---

1. Group Normalization ↩︎

2. SPECTRAL NORMALIZATION FOR GENERATIVE ADVERSARIAL NETWORKS ↩︎

3. Spectral Normalization 谱归一化/https://zhuanlan.zhihu.com/p/55393813 ↩︎

4. Image Up-Sampling Using Total-Variation Regularization With a New Observation Model ↩︎