Continue
Representational Power, Layer Size and Depth
大多数自编码器都只有一层所谓的隐藏层,也就是所谓的码
单层已能够在给定精度表达任何函数
e.g. Principal Components Analysis(PCA)主元分析
一个多层自编码器
更难训练
, 但是如果训练适当,可以获得更牛叉的表达效果.1
随机自编码器
x−→−−−Q(h|x)h−→−−−P(x|h)output 随机自编码器的一般结构。编码和解码都不是简单的函数,都引入一些噪声,意味着他的输出可以看过是输入的分布取样,
Q(h|x) for the encoder andP(x|h) for the decoder.RBMs 是一个特殊情况,其中
P=Q
线性特征模型
关于数据生成的假设
- sample real-valued factors
h∼P(h) - sample the real-valued observable variables
x=Wh+b+mnoise
概率主元分析和特征分析
它们都是上面等式的特殊情况,不过他们先验和噪声分布的选择不同
h∼P(h)−→−−−P(x|h)x=Wh+b+noise
线性特征模型
的一般结构, 观测数据x→ 是通过隐性因素h 的线性组合加上一些噪声获得的
特征分析
2, 先验:h∼N(0,I)
其中假设xi 条件独立,噪声来自对角协方差的高斯分布,协方差矩阵其中ψ=mdiag(σ2) σ2=(σ21,σ22,...)
h 的角色是获取xi 的相互依赖。
其中x∼N(b,WWT+σ2I) xi 通过wik (for everyk )影响h^k=Wkx ,反过来,h^k 通过wkj 影响xj - 为了将PCA放进概率框架,令条件方差
σi 互相相当? . 不懂
这时
x∼N(b,WWT+σ2I)
也就是
x=Wh+b+σz
其中z∼N(0,I) 是白噪声.
概率 PCA
方差被h 获取, 使得 h成为一个小的重构残差
σ2 .
- 当
σ→0 , pPCA 成为 PCA.
Continue
Representational Power, Layer Size and Depth
Generally, most trained auto-encoders have had a single hidden layer which is also the representation layer or code.
approximator abilities of single hiddenlayer neural networks: a sufficiently large hidden layer can represent any function with a given accuracy
e.g. Principal Components Analysis(PCA)
training a deep neural network, and in particular a deep auto-encoder (i.e. with a deep encoder and a deep decoder) is
more difficult
than training a shallow one. If trained properly, such deep auto-encoders could yieldmuch better compression
than corresponding shallow or linear auto-encoders.3
stochastic auto-encoder
x−→−−−Q(h|x)h−→−−−P(x|h)output Basic scheme of a stochastic auto-encoder. Both the encoder and the decoder are not simple functions but instead involve some noise injection, meaning that their output can be seen as sampled from a distribution, Q(h|x) for the encoder and P (x|h) for the decoder.
RBMs are a special case where P = Q
but in general these two distributions are not necessarily conditional distributions compatible with a unique joint distribution P (x, h).
Liner factor Models
Assumpation of how data was generated
- sample real-valued factors
h∼P(h) - sample the real-valued observable variables
x=Wh+b+mnoise
Probabilistic PCA and Factor Analysis
Both are special cases of above equations and only differ in the choices made for the prior and noise distributions.
h∼P(h)−→−−−P(x|h)x=Wh+b+noise Basic scheme of a
linear factors model
, in which it is assumed that an observed data vector x is obtained by a linear combination of latent factors h, plus some noise. Different models, such as probabilistic PCA, factor analysis or ICA, make different choices about the form of the noise and of the prior P(h).
factor analysis
4, latent variable prior:h∼N(0,I)
wherexi are assumed to be conditionally independent and noise is assumed to be coming from a fiagonal covariance Gaussian distribution, with covariance matrixψ=mdiag(σ2)
whereσ2=(σ21,σ22,...)
The role of the latent variables is to capture the dependence amongxi .
x∼N(b,WWT+σ2I)
where
xi influencesh^k=Wkx viawik (for everyk ) andh^k influencessxj viawkj - In order to cast PCA in a probabilistic framework make the conditional variances
σi equal to each other? .
In that case
x∼N(b,WWT+σ2I)
also
x=Wh+b+σz
wherez∼N(0,I) is white nosie.
Probabilistic PCA
the covariance is mostlycaptured by the latent variables , up h to some small residual
reconstruction error
σ2 .
- if
σ→0 , pPCA becomes PCA.