Learning Techniques for Video Shot Detection

Under the guidance ofProf. Sharat Chandran

by M. Nithya

Outline

• Introduction• Types of Shot-break• Previous approaches to Shot Detection

General Approach - pixel comparison, histogram comparison… Recent Work – Temporal slice analysis, Cue Video

• Our Proposed approaches Supervised Learning using AdaBoost algorithm Unsupervised Learning using clustering Semi-supervised Learning combining AdaBoost &

clustering

• Conclusion

Introduction 9,000 hours of motion pictures are produced around the world every year. 3,000 television stations broadcasting for twenty-four hours a day produce

eight million hours of video per year.

Problems:• Searching the video

• Retrieving the relevant information

Solution: Break down the video into smaller manageable parts

called “Shots”

What is Shot?

Shot is the result of uninterrupted camera work

Shot-break is the transition from one shot

to the next

Types of Shot-Break

Shot-Break

Hard Cut Fade Dissolve Wipe

Hard Cut

Fade

Dissolve

Wipe

Shot Detection Methods

Shot Detection Methods

Goal:

To segment video into shots

Two ways:• Cluster the similar frames to identify shots• Find the shots that differ and declare it as shot-break

Pervious Approaches to Shot Detection

• General Approaches– Pixel Comparison– Block-based approach– Histogram Comparison– Edge Change Ratio

• Recent Work– Temporal Slice Analysis– Cue Video

Pixel Comparison

Frame N Frame N + 1

x=1 y=1 | Pi(x,y) – Pi+1(x,y) |D(i,i+1)=

X Y

XY

Block – Based Approach

Frame N Frame N + 1

Compares statistics of the corresponding blocks

Counts the number of significantly different blocks

Histogram Comparison

Edge Change Ratio

Comparison…

Method Advantages Disadvantages

Pixel-Comparison Simple, easy to implement

Computationally heavy,

Very sensitive to moving object or camera motion

Block based Performs better than pixel

Can’t identify dissolve, fade, fast moving objects

Histogram comparison Performance is better

Detects hard-cut, fade, wipe and dissolve

Fails if the two successive shots have same histogram. Can’t distinguish fast object or camera motion

Edge Change Ratios Detects hard-cut, fade, wipe and dissolve

Computationally heavy

Fails when there is large amount of motion

Problems with previous approaches

Can’t distinguish shot-breaks with• Fast object motion or Camera motion• Fast Illumination changes• Reflections from glass, water• Flash photography

Fails to detect long and short gradual transitions

Temporal – Slice Analysis

Temporal – Slice Analysis

Cue Video

Temporal – Slice Analysis

Cue Video

• Graph based approach• Each frame maps to a node• Connected upto 1, 3 or 7 frames apart• Each node is associated with

– color Histogram– Edge Histogram

• Weights of the edges represent similarity measure between the two frames

• Graph partitioning will segment the video into shots

Proposed Approaches

Proposed Approaches

Use learning techniques to distinguish between

shot-break and • Fast object motion or Camera motion• Fast Illumination changes• Reflections from glass, water• Flash photography

Supervised Learning

Feature Extraction

• 25 Primitive features like edge, color are extracted directly from the image

• These 25 features are used as input to next round of feature extraction yielding

25 x 25 = 625 features

• This 625 features can be used as input to compute 625 x 625 = 15, 625 features

How these features can be used to classify images?

Solution : Use AdaBoost to select these features.

Oops!! There are 15, 625 features!

Applying them to red, green and blue separately will result in 46, 875 features!

Can we find few important features that will help to distinguish the images?

Input: (x1,y1) (x2,y2) …(xm,ym) where x1,x2,…xm are the images

yi = 0,1 for negative and positive examples

Let n and p be the number of positive and negative examples

Initial weight w1,i = 1/2n if yi= 0 and

w1,I = 1/2p if yi = 1

For t= 1,…T:

Train one hypothesis hi(x) for each feature and find the error

Choose the hypothesis with low error value

update the weight:wt+1,i = wt,i * t

1-et

where ei=0,1for xi classified incorrectly or correctly

t=et/(1-et)

Normalize wt+1,I so that it is a distribution

Final hypothesis is calculated as

AdaBoost Algorithm

Supervised Learning

• Extract Highly selective features

• AdaBoost algorithm to select few important features

• Train the method to detect different shot-breaks

Unsupervised techniques Clustering

Unsupervised technique - clustering

Unsupervised technique - clustering

Hard Cut Dissolve

Unsupervised technique

• Clustering method to cluster into shots

• Relevance Feedback

Semi-supervised Learning

Semi-supervised Learning

Combination of Supervised and Unsupervised

Few labeled data are available, using which it works on large unlabeled video

Steps:

• AdaBoost algorithm to select features

• Clustering method to cluster into shots

• Relevance Feedback

Conclusion…

Conclusion…

Problems with previous approaches:• Can’t distinguish shot-breaks with

– Fast object motion or Camera motion – Fast Illumination changes– Reflections from glass, water– Flash photography

Fails to detect long and short gradual transitions

Planning to use AdaBoost learning based clustering scheme for shot-detection

Thank you…