• Home

  • Technology

  • 論文輪読: Generative Adversarial Text to Image Synthesis
17
論文紹介 S. Reed et al. Genera-ve Adversarial Text-to-Image Synthesis (ICML’16) 2016/6/10 hFp://arxiv.org/abs/1605.05396

論文輪読: Generative Adversarial Text to Image Synthesis

  • Upload
    mmisono

  • View
    594

  • Download
    5

Embed Size (px)

Citation preview

Page 1: 論文輪読: Generative Adversarial Text to Image Synthesis

論文紹介S.Reedetal.

Genera-veAdversarialText-to-ImageSynthesis(ICML’16)

2016/6/10

hFp://arxiv.org/abs/1605.05396

Page 2: 論文輪読: Generative Adversarial Text to Image Synthesis

text-condi-onalconvolu-onalGAN

基本:textのencodingφ(t)で条件付けたGAN工夫:(1)encodingφ(t):char-CNN-RNN(2)Matching-awarediscriminator(3)ManifoldInterpola-on

hFp://www-personal.umich.edu/~reedscot/papersupplementcode

S.Reed+:Genera-veAdversarialText-to-ImageSynthesis(ICML’16)

Page 3: 論文輪読: Generative Adversarial Text to Image Synthesis

(1)char-CNN-RNN

•  以下の損失を最小化するようにencoderを訓練

•  このencodingは学習の速さに寄与

Imageencoder

textencoder(character-levelCNNorLSTM)

Δ:0-1loss,yn:classlabelvn:image,tn:text

S.Reed+:LearningDeepRepresenta-onsofFine-GrainedVisualDescrip-on(CVPR’16)

Page 4: 論文輪読: Generative Adversarial Text to Image Synthesis

(2)Matching-awareDiscriminator(GAN-CLS)

•  na-veGAN:以下のようになるようDを学習1.  D(本物の画像,正解文章)=12.  D(合成画像,任意の文章)=0

•  実際に判定すべき誤りには以下の二種類がある

–  (合成画像，任意の文章)–  (本物の画像,間違った文章)

•  そこで訓練時に(本物の画像，間違った文章)のペアを利用するようにアルゴリズムを修正

Page 5: 論文輪読: Generative Adversarial Text to Image Synthesis

(2)Matching-awareDiscriminator(GAN-CLS)

目標出力:1

目標出力:0

Page 6: 論文輪読: Generative Adversarial Text to Image Synthesis

(3)ManifoldInterpola-on(GAN-INT)

•  多様体仮説–  訓練データのembeddingsを補間したものもデータの多様体近くに存

在

•  φ(t)を重ね合わせたものを訓練に利用•  Generator の目的関数で以下を最小化する

•  β=0.5で十分うまくいくことを確認

Page 7: 論文輪読: Generative Adversarial Text to Image Synthesis

実験パラメータ

•  画像サイズ (64,64,3)•  textencoderφ(t):1024 次元•  φ(t)はFCで128次元にした後，leaky-ReLUを掛け，zと結合•  stride2convolu-on•  全てのconvolu-onの後にBNを適用•  discriminatorのspa-alな次元:4x4•  ADAMsolver,momentum0.5,minibatchsize64,600epoch

1x1conv,ReLU,4x4conv

Page 8: 論文輪読: Generative Adversarial Text to Image Synthesis

実験データ

•  CUBbirds–  1万枚程度，200カテゴリ

•  Oxfordflower102–  計8189枚

•  MSCOCO–  8万枚程度

•  CUBとflowersは著者らがキャプションをつけた?–  キャプションデータは公開されている–  MSCOCOは大きすぎるので，CUBやflowersは検証データとして良いか

も

Page 9: 論文輪読: Generative Adversarial Text to Image Synthesis

interpola-onの効果

実験結果(CUB)

Zero-shot(i.e.condi-onedontextfromunseentestsetcategories)generatedbirdimages

Page 10: 論文輪読: Generative Adversarial Text to Image Synthesis

実験結果(flowers)

Page 11: 論文輪読: Generative Adversarial Text to Image Synthesis

実験結果(MSCOCO)

Page 12: 論文輪読: Generative Adversarial Text to Image Synthesis

styleとcontentに関する考察

•  テキスト情報:contentinforma-on(鳥の種類，形，大きさ等)–  背景や鳥の姿勢等の情報(style)は一般に含まない–  ノイズzがstyleを表現するように学習されるべき

•  styleencoderを学習

•  評価実験 –  クラスタリングでposeとbackgroundに関して画像をグループ分け–  同一グループの画像のペアをstyleencoderに入力し，最終的に得ら

れた出力のコサイン類似度を計算–  AUC-ROCで評価

S:StyleEncoderNetwork

Page 13: 論文輪読: Generative Adversarial Text to Image Synthesis

styleとcontentに関する考察

•  styleencoderの評価実験 (CUBdataset)

-  text がベースライン(textencoderの特徴量のコサイン類似度)-  特にbackgroundについてzが寄与

Page 14: 論文輪読: Generative Adversarial Text to Image Synthesis

textのcontent(色など)を変えた場合

Page 15: 論文輪読: Generative Adversarial Text to Image Synthesis

2つのランダムなzを補間２つのtextencodingを補間

Page 16: 論文輪読: Generative Adversarial Text to Image Synthesis

訓練してみる

•  hFps://github.com/reedscot/icml2016

•  ベースはdcgan.torch

•  訓練データ(キャプション含む)も公開されている–  ただしCUB,flowersに関しては一部データが不完全なような..?

•  まだ自分で訓練できてはいない

Page 17: 論文輪読: Generative Adversarial Text to Image Synthesis

学習結果(MSCOCO,10epoch(4hr~))

(論文は600epoch?)


[Dl輪読会]semi supervised learning with context-conditional generative adversarial networks

[Dl輪読会]semi supervised learning with context-conditional generative adversarial networks

Technology
[DL輪読会]Wasserstein GAN/Towards Principled Methods for Training Generative Adversarial Networks

[DL輪読会]Wasserstein GAN/Towards Principled Methods for Training Generative Adversarial Networks

Technology
GENERATIVE ADVERSARIAL NETWORKS - Deep Learningdeeplearning.cs.cmu.edu/F20/document/recitation/GANS-ppt.pdf · 2021. 3. 22. · Generative Adversarial Networks Generative Models We

GENERATIVE ADVERSARIAL NETWORKS - Deep Learningdeeplearning.cs.cmu.edu/F20/document/recitation/GANS-ppt.pdf · 2021. 3. 22. · Generative Adversarial Networks Generative Models We

Documents
Triple Generative Adversarial Netspapers.nips.cc/paper/6997-triple-generative-adversarial-nets.pdf · the labels. To address the problems, we present triple generative adversarial

Triple Generative Adversarial Netspapers.nips.cc/paper/6997-triple-generative-adversarial-nets.pdf · the labels. To address the problems, we present triple generative adversarial

Documents
Unsupervised Learning – Generative Models (PixelRNNs, PixelCNNs) – Generative ... · 2019. 3. 28. · Generative Adversarial Networks Ian Goodfellow et al., “Generative Adversarial

Unsupervised Learning – Generative Models (PixelRNNs, PixelCNNs) – Generative ... · 2019. 3. 28. · Generative Adversarial Networks Ian Goodfellow et al., “Generative Adversarial

Documents
[DL輪読会]StackGAN: Text to Photo-realistic Image Synthesis with Stacked Generative Adversarial Networks

[DL輪読会]StackGAN: Text to Photo-realistic Image Synthesis with Stacked Generative Adversarial Networks

Technology
AnomiGAN: Generative Adversarial Networks forpsb.stanford.edu/psb-online/proceedings/psb20/Bae.pdf2.4. Generative Adversarial Networks GANs25 are designed to complement other generative

AnomiGAN: Generative Adversarial Networks forpsb.stanford.edu/psb-online/proceedings/psb20/Bae.pdf2.4. Generative Adversarial Networks GANs25 are designed to complement other generative

Documents
Generative Adversarial Networks (GANs)

Generative Adversarial Networks (GANs)

Documents
Generative Adversarial Transformers

Generative Adversarial Transformers

Documents
COMP9444 Neural Networks and Deep Learning 9b. Generative Adversarial … · 2020. 7. 25. · COMP9444 20T2 Generative Adversarial Networks 13 Generative Adversarial Networks GAN

COMP9444 Neural Networks and Deep Learning 9b. Generative Adversarial … · 2020. 7. 25. · COMP9444 20T2 Generative Adversarial Networks 13 Generative Adversarial Networks GAN

Documents
Conditioning of Generative Adversarial Networks · 2019-01-31 · Generative Adversarial Networks are: •Generative Models •Parametric –Latent Vector •Differentiable –Allow

Conditioning of Generative Adversarial Networks · 2019-01-31 · Generative Adversarial Networks are: •Generative Models •Parametric –Latent Vector •Differentiable –Allow

Documents
Unsupervised Generative Adversarial Alignment

Unsupervised Generative Adversarial Alignment

Documents
Adversarial Distillation of Bayesian Neural Network Posteriorsproceedings.mlr.press/v80/wang18i/wang18i.pdf · 2.3. Generative Adversarial Networks Generative Adversarial Networks

Adversarial Distillation of Bayesian Neural Network Posteriorsproceedings.mlr.press/v80/wang18i/wang18i.pdf · 2.3. Generative Adversarial Networks Generative Adversarial Networks

Documents
Generative Adversarial Perturbations - arXiv

Generative Adversarial Perturbations - arXiv

Documents
GENERATIVE ADVERSARIAL SELF-IMITATION LEARNING

GENERATIVE ADVERSARIAL SELF-IMITATION LEARNING

Documents
Wasserstein Generative Adversarial Networksproceedings.mlr.press/v70/arjovsky17a/arjovsky17a.pdf · Wasserstein Generative Adversarial Networks Martin Arjovsky1 Soumith Chintala2

Wasserstein Generative Adversarial Networksproceedings.mlr.press/v70/arjovsky17a/arjovsky17a.pdf · Wasserstein Generative Adversarial Networks Martin Arjovsky1 Soumith Chintala2

Documents
Image Representation Learning with Generative Adversarial ... · Image Representation Learning with Generative Adversarial Networks Keywords machine learning, generative models, image

Image Representation Learning with Generative Adversarial ... · Image Representation Learning with Generative Adversarial Networks Keywords machine learning, generative models, image

Documents
Generative Adversarial Imitation Learning

Generative Adversarial Imitation Learning

Documents
[DL輪読会]StarGAN: Unified Generative Adversarial Networks for Multi-Domain Image-to-Image Translation

[DL輪読会]StarGAN: Unified Generative Adversarial Networks for Multi-Domain Image-to-Image Translation

Technology
Generative adversarial networks

Generative adversarial networks

Software
Hypothetical Exploration on Generative Adversarial Networks · Convolutional Neural Networks, Generative Adversarial Net- works, Laplacian Pyramid of Adversarial Networks. I. INTRODUCTION

Hypothetical Exploration on Generative Adversarial Networks · Convolutional Neural Networks, Generative Adversarial Net- works, Laplacian Pyramid of Adversarial Networks. I. INTRODUCTION

Documents
Lecture 19: Generative Adversarial Networksrgrosse/courses/csc321_2018/readings/L19 GANs.pdfLecture 19: Generative Adversarial Networks Roger Grosse 1 Introduction Generative modeling

Lecture 19: Generative Adversarial Networksrgrosse/courses/csc321_2018/readings/L19 GANs.pdfLecture 19: Generative Adversarial Networks Roger Grosse 1 Introduction Generative modeling

Documents
Generative Adversarial Networks - HAW Hamburgubicomp/projekte/master... · Generative Adversarial Networks ... Outline 1. Basic idea 2. Generative Adversarial ... com/2017/01/generative-adversarial-networks-hot-topic-machine-learning

Generative Adversarial Networks - HAW Hamburgubicomp/projekte/master... · Generative Adversarial Networks ... Outline 1. Basic idea 2. Generative Adversarial ... com/2017/01/generative-adversarial-networks-hot-topic-machine-learning

Documents
Generative Adversarial Nets - courses.cs.washington.edu

Generative Adversarial Nets - courses.cs.washington.edu

Documents
Lecture 7: Generative Adversarial Networks

Lecture 7: Generative Adversarial Networks

Documents
Generative Adversarial Networks (GANs) - Emanuele Sansone · List of recent papers on Generative Adversarial Networks: PAPERs about GANs 20142016 Generative Adversarial Networks ICLR

Generative Adversarial Networks (GANs) - Emanuele Sansone · List of recent papers on Generative Adversarial Networks: PAPERs about GANs 20142016 Generative Adversarial Networks ICLR

Documents
Generative adversarial networks...Generative adversarial nets. NeurIPS. Longer version Goodfellow, (2016). NIPS 2016 Tutorial: Generative Adversarial Networks. Extra Radford et al,

Generative adversarial networks...Generative adversarial nets. NeurIPS. Longer version Goodfellow, (2016). NIPS 2016 Tutorial: Generative Adversarial Networks. Extra Radford et al,

Documents
Generative Adversarial Transformers - Stanford University

Generative Adversarial Transformers - Stanford University

Documents
Time-series Generative Adversarial Networks

Time-series Generative Adversarial Networks

Documents
[DL輪読会]Energy-based generative adversarial networks

[DL輪読会]Energy-based generative adversarial networks

Technology