第10章 無監督學習

自編碼器

---

目錄

---

• 在輸出層模擬輸入層的結果，中間層爲code

圖一自編碼器結構

其中:

• 輸入, x
• 編碼函數, f
• 內部表徵,“碼” h = f(x)
• 解碼函數 g
• 輸出, 重構 r=g(h)=g(f(x))
• 損失函數 L 計算 L(r, x) 的值來測量 r 對給定輸入 x 表達的效果. 目標是最小化 L 的值

稀疏編碼

By Olshausen and Field, 1996

經典表述:

h=f(x)=arghminL(g(h),x))+λΩ(h)

其中

• L 是損失函數
• f 是（無參）編碼函數
• g 是（參數化）解碼函數
• Ω(h)
  是稀疏正則項

稀疏編碼的一個有趣的變種

predictive sparse decomposition (PSD) (Kavukcuoglu et al., 2008)

懲罰函數

除了 L1 :

還有 Student_t

∑ilog(1+α2h2i)

和 KL-diverence

−∑i(tloghi+(1−t)log(1−hi))

其中 t 是目標稀疏指數

---

Auto-Encoders

---

• Try to copy its output to its input

figure 1 the schema of auto-encoders

Where:

• an input, x
• An encoder function, f
• A “code” or internal representation h = f(x)
• A decode function g
• An output, or “reconstruction” r=g(h)=g(f(x))
• A loss function L computing a scalar L(r, x) measuring how good of r of a given input x. The objective is to minimize the expected value of L over the training set of examples {x}

Sparse coding

By Olshausen and Field, 1996

a particular form of inference:

h=f(x)=arghminL(g(h),x))+λΩ(h)

Where

• L is the reconstruction loss
• f is the (non-parametric) encoder
• g is the(parametric) decoder
• Ω(h)
  is a sparsity regularizer

What’s more：

To achieve sparsity, the optimized objective function includes a term that is minimized when the representation has many zero or near-zero values, such as the L1 penalty

|h|1=∑|hi|

An interesting variation of sparse coing

predictive sparse decomposition (PSD) (Kavukcuoglu et al., 2008)

About spare auto-encoders

Besides L1 penalty:

Student_t penalty

∑ilog(1+α2h2i)

and KL-diverence penalty

−∑i(tloghi+(1−t)log(1−hi))

where t is a target sparsity level