From Semantically Annotated Media To Video DocumentariesStefano Bocconi

CWI Amsterdam, The Netherlands

Talk Outline

• Motivation• Example• Scenarios• Technical details– Annotations – Editing Process

• Conclusions

Talk Outline

Philips, 31-05-2006

Case Study

Subject: the opinion of American people after 9-11

Shooting: from 27-10-2001 to 01-11-2001 in Stamford (CT), New York (NY), Boston (MA) and Cleveland (OH)

Editing: approximately 3 months

http://www.interviewwithamerica.com/

Philips, 31-05-2006

The problems

Only one final version: what do we show?

Who is pro and who is against? What are the numbers?

Biased versus objective trade-off: a choice must be made

Philips, 31-05-2006

Authoring Documentaries

Traditional video editing: there is only one final version, what is shown is the choice of the author/editor

Proposed video authoring: ● Annotate the video

material semantics● Show automatically

what the user asks to see, using presentation forms a film editor would use

Philips, 31-05-2006

Example

“I am never a fan of military action, in the big picture I don’t think it is ever a good thing, but I think there are circumstances in which I certainly can’t think of a more effective way to counter this sort of thing

Claim

Concession

Claim contradict

supportClaim

I am not a fanof military actions

War has never solved anything

Two billions dollar bombs on tents

I cannot think of a

more effective solutionweaken

Analysis of the Example

Scenarios

Vox Populi: Augmenting one interview with man-on-the-street opinion

Overview of the content of video footage:

● Example: trailers ● Browse the content by

opinion

Voices of Iraq

Philips, 31-05-2006

• What question was asked• Who answered (social categories)• Filmic (e.g. location/time/framing/gaze)

Possible query: show me all the answers to question “what do you think of the war in Afghanistan” given by black men

Descriptive Annotations

• Statements formally annotated:– <subject> <modifier> <predicate>– E.g. “war best solution”

• A thesaurus containing: – Terms (155)– Relations between terms: similar (72), opposite

(108), generalization (10), specialization (10)– E.g. war opposite diplomacy

• Annotations: 1 hour annotated, 15 interviews, 60 interview segments, 120 statements

Rhetorical Annotations

ClaimData

Qualifier

Warrant

Backing

Condition

Concession

57 Claims, 16 Data, 4 Concessions, 3 Warrants, 1 Condition

Toulmin Model

• Using the thesaurus, generate related statements and query the repository– E.g. from “war best solution” “diplomacy best

solution”, “war not solution”

• Create a graph of related statements– Nodes are the statements (video segments), edges

are either support or contradict

The Generation Process

• Partially tunable: examining the Segment graph gives feedback on the quality of the annotations and the thesaurus

• The annotator can revise the annotations in a guided manner

S1

S2

S3 S5

S4

S7

S6

S8

S9

S10

= support

= contradict

The Semantic Graph

• Video documentaries are not neutral account of reality: the selection and editing of the footage expresses a point of view

• Editing strategy: Balanced, Pro opinion X and Against opinion X

• Vox Populi uses:– Logos (the statements)– Ethos (based on user profile)– Film editing (framing, transitions, counterpoint editing)

Controlling Bias

• Vox Populi automatically presents video interviews augmented with supporting and/or contradicting material

• The user can determine the subject and the bias of the presentation

• Extensive annotations are needed

Conclusions

This presentation and a Demo available at:http://www.cwi.nl/~media/demo/IWA

Philips, 31-05-2006

Discussion

Philips, 31-05-2006

Issues

Advantages

Reuse the reasoning of the humans, no need for a very complex system like Cyc

Disadvantages

Extensive annotation is needed

Need for a knowledge base

Validation

User studies?

Mimicking the reality

• This presentation and a Demo available at:http://www.cwi.nl/~media/demo/IWA/

• This research was funded by the Dutch national ToKeN2000 I2RP and CHIME projects.

Philips, 31-05-2006

Using the Pathos

How and what emotions can we use to persuade?

Cognitive structure of emotions (Ortony)

The expressions for happiness, sadness, disgust, surprise, anger and fear are culture-, gender- and age independent (Ekman & Friesen)

Open Documentary

PRO

AGAINST

The annotations

• Descriptive– Question asked– Interviewee (social)– Filmic (e.g. location/time/framing/gaze)

• Rhetorical– Rhetorical Statement (mostly verbal, but visual also

possible)– Argumentation model: Toulmin model

Annotations

Philips, 31-05-2006

Rhetorical Tools

According to Aristotle, rhetoric is "the ability, in each particular case, to see the available means of persuasion”

Three main forms of rhetoric: ● Ethos: appeal based on the character of

the speaker.● Logos: appeal based on logic or reason.● Pathos: appeal based on emotion.

Open Documentary