Fantope Projection and Selection: A near-optimal convex relaxation of sparse PCA

1. Goal

This paper mainly deals with Sparse Principal Component Analysis(PCA) using subspace method.

2. Theorey

2.1 How to get their formulation

Notation: λ1,λ2,⋯,λp are in decreasing order.
From Ky Fan’s maximum principal 1, we know that

∑i=1dλi(Σ)=tr(V∗′ΣV∗)=maxV′V=Idtr(V′ΣV)=maxV′V=Id⟨Σ,VV′⟩

If we regard the last formula as a function of VV′ , it is linear. So if we change the constrain to its convex hull does not change the optimization problem. From the less well known observation that

Fdp={H:trace(H)=d,0≤H≤Id}=conv({VV′:V′V=Id})

From all the analysis, we get

∑i=1dλi(Σ)=⟨Σ,Π⟩=maxH∈Fdp⟨Σ,H⟩

How to introduce the sparsity? And which norm is suitable to use? The goal of this paper is to get sparse PCs, then we should choose penalty making V∗∈Rp×d sparse. For matrix, there are two ways to get sparsity:

• columnwise sparsity: for matrix A , each of its column is sparse, i.e. only few elements of A∗i are nonzero.
• row sparsity: for matrix A , its rows are sparse, i.e. only few rows of A are sparse, which produce the group sparsity.

For sparse PCA, to select the import features, this paper uses row sparsity. An intuitive penalty is ∥V∥2,0 . But in high dimensional situation, ℓ0 norm is NP hard to deal. A common trick is replacing ℓ0 with ℓ1 . Then the penalty becomes ∥V∥2,1 . But our model is function of H=VV′ . So what sparsity on H can approximate well of ∥V∥2,1 . Note that

∥H∥1,1=∑ijk|Vik||Vjk|=∑ij∑k|Vik||Vjk|≤∑ij(∑kV2ik)12(∑kV2jk)12=∥V∥22,1

where ≤ uses Cauchy-Schiwarz Inequality. Final model:

H^=maxH∈Fdp⟨S,H⟩−ρ∥H∥1,1

where S is an estimation of Σ .

2.2 Consistency

Make assumption:

• assumption 1: (symmetry) S and Σ are symmetric matrices.
• assumption 2: (Identity) δ=λd(Σ)−λd+1(Σ)>0 .
• assumption 3: (Sparsity) ∥Π∥2,0=s≪p

Theorem (ℓ2 consistent): H^ is ℓ2 consistent for Π under some regular condition. In more detail, if ρ≥∥S−Σ∥∞,∞ ,

∥H^−Π∥F≤4sρδ

.

proof: The main tool is the curvature lemma2. Let Δ=H^−Π and W=S−Σ . From curvature lemma, we have

δ2∥Δ∥2F≤−⟨Σ,Δ⟩

.
Together with the optimality of H^ ,

⟨S,Δ⟩−ρ(∥H^∥1,1−∥Π∥1,1)≥0

we have

δ2∥Δ∥2F≤⟨W,Δ⟩−ρ(∥H^∥1,1−∥Π∥1,1)≤∥W∥∞,∞∥Δ∥1,1−ρ(∥Π+Δ∥1,1−∥Π∥1,1)≤∥W∥∞,∞∥Δ∥1,1−ρ(∥ΠS+ΔS∥1,1−∥ΠS∥1,1+∥ΔSc∥1,1)≤ρ(∥Δ∥1,1−∥ΠS+ΔS∥1,1+∥ΠS∥1,1−∥ΔSc∥1,1)≤ρ(∥ΔS∥1,1−∥ΠS+ΔS∥1,1+∥ΠS∥1,1)≤ρ(∥ΔS∥1,1−∥ΠS∥1,1+∥ΔS∥1,1+∥ΠS∥1,1)≤2ρ∥ΔS∥1,1≤2sρ∥Δ∥F

so

∥Δ∥F≤4sρδ.

3. Algorithm

The chief difficulty in solving this problem is the interaction between the penalty and the Fantope constraint. Without either of these features, the optimization problem would be much easier. ADMM an exploit this fact:

• rewrite the model as
  
  maxs.t.{−1Fdp(H)+⟨S,H⟩−ρ∥Z∥1,1+λ2∥H−Z∥2F}H−Z=0
• Lagrange function:
  
  L(H,Z,U)=−1Fdp(H)+⟨S,H⟩−ρ∥Z∥1,1+λ2∥H−Z∥2F−⟨U,H−Z⟩=−1Fdp(H)+⟨S,H⟩−ρ∥Z∥1,1+λ2(∥∥∥H−Z−1λU∥∥∥2F−∥∥∥1λU∥∥∥2F)=−1Fdp(H)+⟨S,H⟩−ρ∥Z∥1,1+λ2(∥∥H−Z−U′∥∥2F−∥∥U′∥∥2F)

R package code can be found, which is written in c++:
https://github.com/vqv/fps

where U′=1λU and then update (H,Z,U) interatively 3.

H+Z+U+=argmaxL(H,Z,U)=argmaxL(H+,Z,U)=U+H+−Z+

---

1. Ky Fan’s maximum principal:
   
   ∑i=1dλi(Σ)∑i=p−d+1pλi(Σ)=maxV′V=Idtr(V′ΣV)=minV′V=Idtr(V′ΣV)
   ↩
2. Curvature lemma: Let A be a symmetric matrix and E be the projection onto the subspace spanned by the eigenvectors of A corresponding to its d largest eigenvalues λ1≥λ2≥⋯ . If δA=λd−λd+1 , then
   
   ⟨A,E−F⟩≥δA2∥E−F∥2F
   
   for all F∈Fdp . ↩
3. H^ =argmaxL(H,Z,U∗)=argmaxH∈Fdp⟨S,H⟩+λ2∥H−Z−U∗∥2F=argminH∈Fdp∥∥∥H−(Z−U∗−Sλ)∥∥∥2F=PFdp(Z−U∗−Sλ)
   ↩