View
15
Download
0
Category
Preview:
Citation preview
|X
simNet: Stepwise Image-Topic Merging Network for
Generating Detailed and Comprehensive Image Captions
1
Fenglin Liu¹, Xuancheng Ren²*, Yuanxin Liu¹, Houfeng Wang² and Xu Sun²
¹ Beijing University of Posts and Telecommunications, Beijing, China
² Peking University, Beijing, China
* Equal Contributions
|X
CONTENTS
1 Introduction
2 Approach
3 Experiment
4 Analysis
|X
Introduction
|X
simNet: Stepwise Image-Topic Merging Network for Generating Detailed and Comprehensive Image Captions
4
·Soft-Attention: Show, attend and tell: Neural image caption generation with visual attention. In PMLR 2015
·ATT-FCN : Image captioning with semantic attention. In CVPR 2016
Figure 1: Examples of using different attention mechanisms.
|X
Introduction: Soft-Attention
5
Image Image Features Caption
encode decodeSoft-Attention:
·Soft-Attention: Show, attend and tell: Neural image caption generation with visual attention. In PMLR 2015
omitting “dog” and “mouse”
|X
Introduction: ATT-FCN
6
Image Image Keywords Caption
encode decodeATT-FCN:
·ATT-FCN : Image captioning with semantic attention. In CVPR 2016
missing “open” and mislocating “dog”
|X
Introduction: SimNet
7
Image
Image Features
decode
Image Keywords
encode CaptionsimNet:
|X
Introduction: Main idea
8
The visual information captured by CNN
The topics extracted by a topic extractor
The merging gate then adaptively adjusts the weight between visual attention and topic attention
Figure 2: Illustration of the main idea.
|X
Contributions
9
We propose a novel approach that can effectively merge the information in the image and the topics.
The generated captions are both detailed and comprehensive.
The proposed approach outperforms previous works in terms of SPICE, which correlates the best with human judgments.
|X
Approach
|X 11
Overview
|X 12
Approach: Image Encoder
Image Encoder: ResNet152
He et al., 2016: Deep residual learning for image recognition. In CVPR 2016.
|X 13
Approach: Image Encoder
Feature map:
|X 14
Approach: Topic Extractor
Topic Extractor: Multiple Instance Learning
Zhang et al., 2006: Multiple instance boosting for object detection. In NIPS 2006.
Fang et al., 2015: From captions to visual concepts and back. In CVPR2015
|X 15
Approach: Input Attention
Input Attention:
Xu et al., 2015 : Show, attend and tell: Neural image caption generation with visual attention. In PMLR 2015
|X 16
Approach: Output Attention
Output Attention:
You et al., 2016 : Image captioning with semantic attention. In CVPR 2016
Lu et al ., 2017 : Knowing when to look: Adaptive attention via a visual sentinel for image captioning. In CVPR 2017
|X 17
Approach: Visual Information
the visual information:
Output Attention:
|X 18
Approach: Previous Topic Attention
Topic Attention (Previous work):
Lacking the attentive visual informationwhen selecting topic!
You et al., 2016 : Image captioning with semantic attention. In CVPR 2016
|X 19
Approach: Our Topic Attention
Topic Attention (Our):
|X 20
Approach: Contextual Information
Topic Attention (Our):
the contextual information:
|X 21
Approach: Merging Gate
How to make full use of the visual information and the contextual information?
|X 22
Approach: Merging Gate
Visual information(e.g., “behind”, “red” is better)
Contextual information(e.g., “people”, “table” is better)
|X 23
Approach: Merging Gate
( Where σ is the sigmoid function )
|X 24
Approach: Merging Gate
The scoring function S is designed to evaluate the importance of the topic attention.
|X 25
Approach: Merging Gate
|X 26
Approach: Merging Gate
Share Weights
Share Weights
|X 27
Generating Words
the contextual information:
|X
Experiments
|X
Experiments
29
Microsoft COCO(MSCOCO) and Flickr30k
DatasetEvaluation Metrics
Sparrow bird on branch, withbeak inspecting leaves on branch.
A bird sitting on the branch of atree near leaves.
A bird that is sitting in a tree. a bird sitting on a branch of a
tree. a bird that is on a small branch
of a tree.
SPICE
CIDEr
BLEU
METEOR
ROUGE
Correlates the best with human Judgments !
|X
Experiments: Results (MSCOCO)
30
ComparableModels
|X
Experiments: Results (MSCOCO)
31
Competitive
|X
Analysis
|X
Analysis: The Contributions of The Sub-modules
33
Comprehensiveness Detailedness
|X
Analysis: Output Attention
34
The output attention is much more effective than the input attention
|X
Analysis: Visual Attention
35
A combination of the input attention and the output attention makes the results even better
|X
Analysis: Topic Attention
36
The topic attention is better at identifying objects but worse at identifying attributes.
|X
Analysis: Visual Attention + Topic Attention
37
Combing the visual attention and the topic attention directly results in a huge boost in performance
|X
Analysis: Full Model
38
Applying the merging gate is essential to the overall performance.
|X
Analysis: Visualization
39
The upper part shows the attention weights of each of 5 extracted topics. Deeper color means larger in value. The middle part shows the value of the merging gate. Determines the importance of the topic attention. The lower part shows the visualization of visual attention. The blue shade indicates the output attention.
The red shade indicates the input attention.
|X
Analysis: Visualization
40
Visual information “chair” is more important than contextual information “bed”
|X
Analysis: Examples
41
erroneoustopic “kitchen”
lacking “mouse”
missing “wooden” error count
|X
Conclusion
42
Stepwise image-topic merging network can adaptively combine the visual and the semantic attention to achieve substantial improvements.
The generated captions are both detailed and comprehensive
Our approach outperforms previous works in terms of SPICE on COCO and Flickr datasets.
|X 43
Thank you!If you have any questions about our paper, you can send a email to lfl@bupt.edu.cn
|X
Experiments: Results (Flickr30k)
44
|X
Analysis: Topic Extraction
45
The reason of using objects as topics is that they are easier to identify so that the generation suffers less from erroneous predictions.
It proves that for semantic attention, it is also important to limit the absolute number of incorrect visual words instead of merely the precision or the recall.
|X
Analysis: Merging Gate
46
|X
Analysis: Error Analysis
47
There are mainly three types of errors, i.e. distance (32, 26%), movement (22,18%), and object(60, 49%), with 9 (7%) other errors.
|X
Analysis: Error Analysis
48
There are mainly three types of errors, i.e. distance (32, 26%), movement (22,18%), and object(60, 49%), with 9 (7%) other errors.
|X
Experiments: Results (MSCOCO)
49
|X
Analysis: Topic Attention
50
Recommended