SaGAN:Generative Adversarial Network with Spatial Attention for Face Attribute Editing

#Problem

  1. Traditional GAN methods directly operate on the whole image, and inevitably change the attribute-irrelevant regions
  2. The performance of traditional regression methods heavily depends on the paired training data, which are however quite difficult to acquire

#Relative work

  • ResGAN: learning the residual image avoids changing the attribute-irrelevant region by restraining most regions of the residual image as zero.

  • Improvement:This work is quite insightful to enforce the manipulation mainly concentrate on local areas especially for those local attributes.

  • Drawback: the location and the appearance of target attributes are modeled in single sparse residual image which is actually hard for a favorable optimization than modeling them separately

#Method

  1. SaGAN: only alter the attribute specific region and keep the rest unchanged
  2. The generator contains an attribute manipulation network (AMN) to edit the face image, and a spatial attention network (SAN) to localize the attribute-speciic region which restricts the alternation of AMN within this region.

#Contribution

  1. The spatial attention is introduced to the GAN framework, forming an end-to-end generative model for face attribute editing (referred to as SaGAN),which can only alter those attribute-speciic region and keep the rest irrelevant region remain the same.
  2. The proposed SaGAN adopts single generator with attribute as conditional signal rather than two dual ones for two inverse face attribute editing.
  3. The proposed SaGAN achieves quite promising results especially for those local attributes with the attribute-irrelevant details well preserved. Besides, our approach also benefits the face recognition by data augmentation.

#Generative Adversarial Network with Spatial Attention

SaGAN

notation meaning
II input image
I^\hat{I} output image
IaI_a an edited face image output by AMN
cc attribute value
cgc_g ground truth attribute label of the real image II
Dsrc(I)D_{src}(I) probability of an image II to be a real one
Dcls(cI)D_{cls}(c\|I) probability of an image II with the attribute cc
FmF_m an attribute manipulation network (AMN)
FaF_a a spatial attention network(SAN)
bb a spatial attention mask, used to restrict the alternation of AMN within this region
λ1\lambda_1 balance parameters
λ2\lambda_2 balance parameters
λgpλ_{gp} hyper-parameters control the gradient penalty, default = 10
  • the goal of face attribute editing is to translate I into an
    new image I^\hat{I}, which should be realistic, with attribute c and look the same as the input image excluding the attribute-specific region

##Discriminator

  • Two objectives, one to distinguish the generated images from the real ones, and another to classify the attributes of the generated and real images
  • The two classifiers are both designed as a CNN with softmax function, denoted as DsrcD_{src} and DclsD_{cls} respectively.
  • The two networks can share the first few convolutional layers followed by distinct fully-connected layers for different classifications
    KaTeX parse error: Got function '\hat' with no arguments as subscript at position 61: …(I)]+\mathbb{E}_̲\hat{I}(log(1-D…
    LclsD=EI,cg[logDcls(cgI)]\mathcal{L}_{cls}^D = \mathbb{E}_{I,c^g}[-logD_{cls}(c^g|I)]
    discriminator D:
    minDsrc,DclsLD=LsrcD+LclsD\min \limits_{D_{src},D_{cls}} \mathcal{L}_D = \mathcal{L}_{src}^D+\mathcal{L}_{cls}^D

##Generator

  • G contains two modules, an attribute manipulation network(AMN) and a spatial attention network(SAN)
  • AMN focuses on how to manipulate and SAN focuses on where to manipulate.
  • The attribute manipulation network takes a face image II and an attribute value cc as input, and outputs an edited face image IaI_a
    Ia=Fm(I,c)I_a = F_m(I,c)
  • The spatial attention network takes the face image II as input, and predict a spatial attention mask bb, which is used to restrict the alternation of AMN within this region
  • Ideally, the attribute-specific region of bb should be 1, and the rest regions should be 0.
  • Regions with non-zeros attention values are all regarded as attribute-specific region, and the rest with zero attention values are regarded as attribute-irrelevant region
    b=Fa(I)b = F_a(I)
  • the attribute-specfiic regions are manipulated towards the target attribute while the rest regions remain the same
    I^=G(I,c)=Iab+I(1b)\hat{I} = G(I,c) = I_a \cdot b + I \cdot (1-b)
  1. To make the edited face image I^\hat{I} photo-realistic: an adversarial loss is designed to confuse the real/fake classifier
    (2)LsrcG=EI^[[logDsrc(I^)]]\mathcal{L}^G_{src} = \mathbb{E}_{\hat{I}}[[-logD_{src}(\hat{I})]]\tag{2}
  2. To make I^\hat{I} be correctly with target attribute cc: an attribute classification loss is designed to enforce the attribute prediction of I^\hat{I} from the attribute classifier approximates the target value cc
    KaTeX parse error: Got function '\hat' with no arguments as subscript at position 33: …^G = \mathbb{E}_̲\hat{I}[-logD_{…
  3. To keep the attribute-irrelevant region unchanged: a reconstruction loss is employed similar as CycleGAN and StarGAN
    LrecG=λ1EI,c,cg[(IG(G(I,c),cg)1]+λ2EI,cg[(IG(I,cg)1]\mathcal{L}_{rec}^G = \lambda_1\mathbb{E}_{I,c,c^g}[(||I-G(G(I,c),c^g)||_1]+\lambda_2\mathbb{E}_{I,c^g}[(||I-G(I,c^g)||_1]
  4. generator G
    minFm,FaLG=LadvG+LclsG+LrecG\min \limits_{F_m,F_a} \mathcal{L}_G = \mathcal{L}_{adv}^G+\mathcal{L}_{cls}^G+\mathcal{L}_{rec}^G

#Implementation
##Optimization
To optimize the adversarial real/fake classification more stably, in all experiments the objectives in Eq.(1) and Eq.(2) is optimized by using WGAN-GP
KaTeX parse error: Got function '\hat' with no arguments as subscript at position 59: …(I)]+\mathbb{E}_̲\hat{I}[D_{src}…

I~\tilde{I} is sampled uniformly along a straight line between the edited images I^\hat{I} and the real images II

##Network Architecture

  • For the generator, the two networks of AMN and SAN share the same network architecture except slight difference in the input and output:
Network Input Output Activation function
AMN 4-channel input, an input image and a attribute 3-channel RGB image Tanh
SAN 3-channel input, an input image 1-channel attention mask image Sigmoid
  • For the discriminator, the same architecture as PatchGAN, is used considering its promising performance.
發表評論
所有評論
還沒有人評論,想成為第一個評論的人麼? 請在上方評論欄輸入並且點擊發布.
相關文章