機器學習基石 - Theory of Generalization

機器學習基石上 (Machine Learning Foundations)—Mathematical Foundations
Hsuan-Tien Lin, 林軒田，副教授 (Associate Professor)，資訊工程學系 (Computer Science and Information Engineering)

Theory of Generalization

Restriction of Break Points

• growth function mH(N)$m_{\mathcal{H}}(N)$ : max number of dichotomies
• 漏出一線曙光的點 break point
• break point k$k$ restricts maximum possible mH(N)$m_{\mathcal{H}}(N)$ a lot for N>k$N>k$

Bounding Function: Basic Cases

• B(N,k)$B(N,k)$ : maximum possible mH(N)$m_{\mathcal{H}}(N)$ when break point = k
  
• 表格
  

Bounding Function: Inductive Cases

• B(4,3)$B(4,3)$ 的估計
  
• Putting It All Together
  
• B(N,k)≤∑k−1i=0(Ni)$B(N,k)\le \sum _{i=0}^{k-1}\left(\begin{array}{c}N\\ i\end{array}\right)$
  
  • 數學歸納法
  • C iN−1+C i+1N−1=C i+1N$C_{N-1}^i+C_{N-1}^{i+1}=C_N^{i+1}$
  • actually ≤$\le$ can be =$=$
  • B(N,k)≥2B(N−1,k−1)+(B(N−1,k)−B(N−1,k−1))$B(N,k)\ge 2B(N-1,k-1)+(B(N-1,k)-B(N-1,k-1))$
• can bound mH(N)$m_{\mathcal{H}}(N)$ by only one break point

A Pictorial Proof

• Step 1: Replace Eout$E_{out}$ by E ′in$E_{in}^{{}^{\prime }}$
  
• Step 2: Decompose H$\mathcal{H}$ by Kind
  
• Step 3: Use Hoeffding without Replacement
  
• Vapnik-Chervonenkis (VC) bound