Look, Listen and Learn Relja Arandjelović, Andrew...

Greg Rehm, Shahbaz Rezaei

Look, Listen and LearnRelja Arandjelović, Andrew Zisserman

February 20th, 2018

●●

●●

Misra, “Shuffle and Learn: Unsupervised Learning Using Temporal Order Verification.” 2016. Vol. 9905

Doersch “Unsupervised Visual Representation Learning by Context Prediction.” (2015):

● >4 young males in row

● Silly hair

● Dumb look on face

.01 sec. window

FFT

Half window overlap

FFT

●●●●

●●

●○○○

●○

■■■

○■■■

Train the entire network

FC:512*128 FC:128*34

Class Label

Class Label

Scalar product

Freeze weights

Train FCs

1 sec

1 sec......

Max pool

One-vs-Al

lSVM4*3*512=6144

Freeze weights

Train SVM

Pretrained on Flickr

mean over the class scores

Max pool

One-vs-Al

lSVM

4*3*512=6144

Max pool

One-vs-Al

lSVM

4*3*512=6144

... ...

1 sec

Max pool

One-vs-Al

lSVM4*3*512=6144

Randomly initialize weights

Train SVM

[Arandjelovic, Relja, and Andrew Zisserman. "Look, listen and learn." ICCV 2017]

Max pool FC

4*4*512=8192

Freeze weights

Train FC/Softmax

Soft max

1000

Pretrained on Flickr

Max pool FC

4*4*512=8192

Randomly initialize weights

Train FC/Softmax

Soft max

1000

[Arandjelovic, Relja, and Andrew Zisserman. "Look, listen and learn." ICCV 2017]

[Arandjelovic, Relja, and Andrew Zisserman. "Look, listen and learn." ICCV 2017]

[Arandjelovic, Relja, and Andrew Zisserman. "Look, listen and learn." ICCV 2017]

[Arandjelovic, Relja, and Andrew Zisserman. "Look, listen and learn." ICCV 2017]

[Arandjelovic, Relja, and Andrew Zisserman. "Look, listen and learn." ICCV 2017]

[Arandjelovic, Relja, and Andrew Zisserman. "Look, listen and learn." ICCV 2017]

[Arandjelovic, Relja, and Andrew Zisserman. "Look, listen and learn." ICCV 2017]

[Arandjelovic, Relja, and Andrew Zisserman. "Look, listen and learn." ICCV 2017]

Frequency

Time

[Arandjelovic, Relja, and Andrew Zisserman. "Look, listen and learn." ICCV 2017]

[Arandjelovic, Relja, and Andrew Zisserman. "Look, listen and learn." ICCV 2017]

[Arandjelovic, Relja, and Andrew Zisserman. "Look, listen and learn." ICCV 2017]

●○○

●○○

○○○○