A Generic Virtual Content Insertion SystemBased on Visual Attention Analysis

H. Liu1, 2, S. Jiang1, Q. Huang1, 2, C. Xu2, 3

1Institute of Computing Technology, Chinese Academy of Sciences2China-Singapore Institute of Digital Media

3National Lab of Pattern Recognition, Institute of Automation

Outline

Motivation Related work The proposed Virtual Content

Insertion (VCI) system Experimental results Conclusions

23/4/19 http://www.jdl.ac.cn 2

Virtual Content Insertion

23/4/19 http://www.jdl.ac.cn 3

Convenient Changeable Cost less

To construct a generic VCI system

Challenge

Advertisement insertion VS. Augmentation

Software based VS. Hardware based Challenge

Insertion time Insertion place Insertion method Insertion content

Related work– Insertion Time

Insert advertisements into video prologue Be neglected Insert the ad into

interesting segments Our method

Temporal attention Higher Attentive

Shots• K. Wan, C. Xu, “Automatic Content Placement in Sports Highlights”,

ICME, 2006.23/4/19 http://www.jdl.ac.cn 5

Related work– Insertion Place

Static region Color consistent

region Visual relevance

measure Lower informative

region Our method

Spatial attention Lower Attentive Region

• C. Xu, etc., “Implanting Virtual Advertisement into Broadcast Soccer Video”, PCM, 2004.

• K. Wan, etc., “Automatic Content Placement in Sports Highlights”, ICME, 2006.

• Y. Li, etc., “Real Time Advertisement Insertion in Baseball Video Based on Advertisement Effect”, ACM Multimedia, 2005.

23/4/19 http://www.jdl.ac.cn 6

Related work – Insertion Method

Challenge Camera parameters

unknown Existing methods

Structure of the scene Predefined landmarks

Our method Affine transformation Global Motion Estimation

• X. Yu, etc., “Inserting 3D Projected Virtual Content into Broadcast Tennis Video”, ACM Multimedia 2006.

• C. Xu, etc., “Implanting Virtual Advertisement into Broadcast Soccer Video”, PCM, 2004.

23/4/19 http://www.jdl.ac.cn 7

Related work – Insertion Content

Improve the ad effect Decrease intrusion VideoSense

Textual relevance Local visual and aural relevance

T. Mei, X-S. Hua, L. Yang, S. Li, “VideoSense-Towards Effective Online Video Advertising”, 16th ACM International Conference on Multimedia, pp: 1075-1084, 2007

23/4/19 http://www.jdl.ac.cn 8

Outline

Motivation Related work The proposed VCI system Experimental results Conclusions

23/4/19 http://www.jdl.ac.cn 9

23/4/19 http://www.jdl.ac.cn 10

T em poral A ttention A nalysis

Temporal Attention

Basic idea The more different a frame/shot/video

clip is to the preceding ones, the more probable for it to be attended

Measure Novelty

23/4/19 http://www.jdl.ac.cn 11

23/4/19 http://www.jdl.ac.cn 12

, 1 min ,i t i tbin

diff F F H H

1

, ,t

t t i t ii t l

Nol F diff F F w F F

HAS Detection

Shot novelty

The longer a shot is, the more it is probable to be attended

23/4/19 http://www.jdl.ac.cn 13

1

, ,t

t i t i ti t l

Nol S diff S S w S S

t t tAV S L Nol S

23/4/19 http://www.jdl.ac.cn 14

23/4/19 http://www.jdl.ac.cn 15

Spatial Attention Analysis

Static attention Spatio-temporal attention

Motion saliency Static novelty

Static saliency calculation

Motion saliency

calculation

Static novelty calculation

Static saliency maps

Motion saliency maps

Static novelty maps

+Video Frames

Attention Maps

Static Saliency (1)

Psychological basis Contrast Information theory

Our method Contrast and information theory

Calculation Property of receptive field

23/4/19 http://www.jdl.ac.cn 16

20, 0,DoG N N

Static Saliency (2)

Perceptive unit Pixel/block Region Object

Color quantization

23/4/19 http://www.jdl.ac.cn 17

23/4/19 http://www.jdl.ac.cn 18

Static Saliency (3)

Contrast

Information density

Saliency

1

, ,K

ki

Con k d f k f i G i k

logID k p f k

Sal k Con k ID k

23/4/19http://www.jdl.ac.cn 19

Motion Saliency

Motion Vector Space HSV color space Angle H

Magnitude S

Texture V

Static Novelty (1)

23/4/19 http://www.jdl.ac.cn 20

23/4/19 http://www.jdl.ac.cn 21

Static Novelty (2)

Static novelty: An event’s importance along temporal axis

Distance: KL 1X

log tt

t

M xNol t M x dx

M x

23/4/19 http://www.jdl.ac.cn 22

Static LAR Detection

23/4/19 http://www.jdl.ac.cn 23

Dynamic LAR Detection

1

L

M t tt

MAM P w t AM P

, 1

1

1k k

t

t Mk

P H P

23/4/19 http://www.jdl.ac.cn 24

23/4/19 http://www.jdl.ac.cn 25

Affine transformation

23/4/19 http://www.jdl.ac.cn 26

1

1 1

x

V y

f

2

2 2

x

V y

f

1 1 1r V V

2 2 2r V V

3 1 2r r r

1 2 3A r r r

1 10 0AP P A P A P

23/4/19 http://www.jdl.ac.cn 27

Global Motion Estimation

1

, 1 0 00

t

t k k Ak

P H AP P AP

Outline

Motivation Related work The proposed VCI system Experimental results Conclusions

23/4/19 http://www.jdl.ac.cn 28

Experiment Data Set – Test Video

23/4/19 http://www.jdl.ac.cn 29

No. Video Genre Shot Time

1 Friendssituation comedy

200 11:25

2Children at

Housesituation comedy

200 14:48

3A Date with

LuYuInterview 200 20:49

4Adventure to the

westOutdoor teleplay

200 25:48

Sum ---- ---- 800 72:50

23/4/19 http://www.jdl.ac.cn 30

Experiment Data Set -- Virtual Content

23/4/19 http://www.jdl.ac.cn 31

Temporal attention & HAS (1)

0 10 20 30 40 500

0.5

1

Shot

Attention curve Notice Rate

23/4/19 http://www.jdl.ac.cn 32

Temporal attention & HAS (2)

Noticing rate:

Consistency: the similarity between attention curve and noticing curve

in N

min ,k

cos AC k NC k

No. Consistency

1 0.75

2 0.79

3 0.84

4 0.82

23/4/19 http://www.jdl.ac.cn 33

Temporal attention & HAS (3)

Relationship between noticing rate and attention value

23/4/19 http://www.jdl.ac.cn 34

Spatial attention & LAR

Invited the users to evaluate the brands he/she has noticed in the video.

rate of GOOD 1 1

n n

i ii i

Rg Ng N

Video GOOD NEUTRAL BAD

1 72.25 19.13 8.62

2 70.87 23.13 6.00

3 66.25 25.00 8.75

4 70.38 25.62 4.00

Mean 69.94 23.22 6.84

Variance 6.67 8.55 5.20

Static Insertion Demo

23/4/19 http://www.jdl.ac.cn 36

Dynamic Insertion Evaluation

Subjective evaluation Criteria

1. Is the result’s deformation consistent with the scene?

2. Does the inserted VC follow the camera motion?

3. To what degree the user is satisfied with the result?

Scores: 15

1 2 3 4 5 60

2

4

VCI Result

Eva

luat

ion

Mean Variance

23/4/19 http://www.jdl.ac.cn 37

Dynamic Insertion Result

23/4/19 http://www.jdl.ac.cn 38

Conclusion

Main contribution A generic virtual content insertion system. A new method of temporal attention and HAS

detection A new method of spatial attention and LAR

detection A dynamic insertion method

Future work The attention change caused by content insertion The interaction between insertion time and place