Workshop in Media Authoring Automatic Generation of Animation and Video

11Intelligent Tools for MediaIntelligent Tools for Media

Workshop in Media AuthoringAutomatic Generation of Animation and Video

ContactDoron Friedman

[email protected]: 054-461807


AgendaAgendaIntroductionIntroduction Motivation and BackgroundMotivation and Background Course MaterialCourse Material

Logistics and AdministrationLogistics and Administration

Description of ProjectsDescription of Projects


Motivation – Media Authoring Motivation – Media Authoring

3D:3D:Professional animators – several years of studyProfessional animators – several years of study

Professional softwareProfessional software

Low leve of abstaction Low leve of abstaction

Very little reuseVery little reuse VideoVideo

Professional editors Professional editors

Professional softwareProfessional software

Pixel based (rather than object based)Pixel based (rather than object based)


Smart GraphicsSmart GraphicsWhat this is NOT aboutWhat this is NOT about How to achieve efficient 3D renderingHow to achieve efficient 3D rendering

What this IS aboutWhat this IS about How to quickly generate animation from high level How to quickly generate animation from high level

inputinput


Making movies (Simplified)Making movies (Simplified)

Story/idea Script Shooting Script Animation File

•Story- understanding•Storytelling•Directing

•NLU •Reusable animation •Animation composition•Text-to-speech

•Theory of space and time •Camera-control principles•Model of actions


Script -> AnimationScript -> Animation

Formal Script Shooting Script Animation File

•Theory of space and time•Camera-control principles•Model of actions

•Reusable animation •Animation composition•Text-to-speech


Interactive Virtual EnvironmentsInteractive Virtual Environments

Linear AnimationLinear Animation AuthoringAuthoring

Interactive Virtual EnvironmentsInteractive Virtual Environments AuthoringAuthoring AgentAgent Real-time DirectorReal-time Director


ApplicationsApplicationsProduction IndustryProduction Industry

Rapid prototypingRapid prototyping StoryboardingStoryboarding

Gaming IndustryGaming Industry Engine for interactive Engine for interactive

gamesgames A movie-maker gameA movie-maker game Toys (Lego/AIBO)Toys (Lego/AIBO)

Education Education

TrainingTraining

SimulationsSimulations IndustrialIndustrial MilitaryMilitary Traffic Traffic ScientificScientific Medical Medical


State of the trade todayState of the trade today Leading tools (Maya, RenderMan, ..)Leading tools (Maya, RenderMan, ..) Leading standards/trends (VRML, Java3D, …)Leading standards/trends (VRML, Java3D, …) How authors work todayHow authors work today


SMIL-1SMIL-1- - TimeTime

ParPar SeqSeq

beginbegin

•Layout–Relative/absolute positioning–Z-index–Fit (meet, slice, scroll)


SMIL-2SMIL-2

45 Modules / Language Profiles45 Modules / Language ProfilesIntegration with other languagesIntegration with other languagesVector graphicsVector graphicsExcl (exclusive par)Excl (exclusive par)Relative synchronizationRelative synchronizationTime manipulation (speed, reverse)Time manipulation (speed, reverse)Animation Animation AudioLayoutAudioLayoutTransitionsTransitions


MPEG-4 as a Rich Media MPEG-4 as a Rich Media Representation SchemeRepresentation Scheme

Several goalsSeveral goals

Profile ApproachProfile Approach


Mpeg-4 Relevant FeaturesMpeg-4 Relevant Features

•TermCap node:

–Frame rate

–Memory/CPU load

•BIFS switch node


MPEG-7 – the Bits about the MPEG-7 – the Bits about the BitsBits

D – Descriptors – representation of features (syntax + D – Descriptors – representation of features (syntax + semantics)semantics)

DS – Description Schemes – structure and semantics of DS – Description Schemes – structure and semantics of relationships between componentsrelationships between components

DDL – Description Defintion Language – to create new DDL – Description Defintion Language – to create new DS, and perhaps new DDS, and perhaps new D


Example – Multiple Description Example – Multiple Description TypesTypesStructural – right, left, moving towardsStructural – right, left, moving towards

Physical – color, shape, motionPhysical – color, shape, motionConceptual – goalkeeper, player, ballConceptual – goalkeeper, player, ball


MPEG-21MPEG-21

Define a multimedia framework to enable Define a multimedia framework to enable transparent and augmented use of transparent and augmented use of multimedia resources across a wide range multimedia resources across a wide range of networks and devices of networks and devices 7 Items7 Items

Digital Item DeclarationDigital Item Declaration Digital Item Identification and DescriptionDigital Item Identification and Description Content Handling and UsageContent Handling and Usage Intellectual Property Management and ProtectionIntellectual Property Management and Protection Terminals and NetworksTerminals and Networks Content RepresentationContent Representation Event ReportingEvent Reporting


Course MaterialCourse MaterialBackground: OO Design and Programming (C++/Java, Background: OO Design and Programming (C++/Java, tbd)tbd)

3D technologies: 3D technologies: VRMLVRML HANIMHANIM Java3D ?Java3D ? MPEG-4/BIFS/XMTMPEG-4/BIFS/XMT

Video/Mixed TechnologiesVideo/Mixed Technologies MPEG-4MPEG-4 JMFJMF SMIL-1/SMIL-2SMIL-1/SMIL-2


Course LogisticsCourse LogisticsCommunication channelsCommunication channels

GroupsGroups

Development methodologyDevelopment methodology

DatesDates


Communication ChannelsCommunication ChannelsMailing list: tbaMailing list: tba

Web-site: Web-site: http://www.math.tau.ac.il/~doronf/workshop_200http://www.math.tau.ac.il/~doronf/workshop_2002.html2.html

Includes links to resourcesIncludes links to resources

My e-mail: My e-mail: [email protected]@post.tau.ac.il


GroupsGroups1-4 people in group (recommended: 2)1-4 people in group (recommended: 2)

Group gets one gradeGroup gets one grade

Inter-group integration – big bonus!Inter-group integration – big bonus!


Development MethodologyDevelopment MethodologySome thoughts on Software Engineering:Some thoughts on Software Engineering: Writing programs is easy, writing software is Writing programs is easy, writing software is

impossibleimpossible % of code used% of code used Legacy softwareLegacy software


Development Process (waterfall)Development Process (waterfall)

MRD (Marketing Requirements Document)MRD (Marketing Requirements Document)

HLD (High Level Design)HLD (High Level Design)

FRS (Functional Requirement Specification)FRS (Functional Requirement Specification)

ATP (Acceptance Testing Procedures)ATP (Acceptance Testing Procedures)

CodeCode

DebugDebug

Testing / Q.A.Testing / Q.A.


ProjectsProjects1.1. Animation LibraryAnimation Library

2.2. Automatic SummariesAutomatic Summaries

3.3. Video Template AuthoringVideo Template Authoring

4.4. Object-based VideoObject-based Video

5.5. Film & TV School projectsFilm & TV School projects


Animation LibraryAnimation LibraryObjects with animationObjects with animation HumanoidHumanoid CarCar DoorDoor 4-legged animal4-legged animal ElevatorElevator Bottle, glass and drinkBottle, glass and drink Cigarette and smokeCigarette and smoke WeaponsWeapons ……


Animation Library - HumanoidAnimation Library - HumanoidStandStand

Walk (from, to), (path)Walk (from, to), (path)

Run Run

JumpJump

Sit (on)Sit (on)

take (object)take (object)

Put (object, location)Put (object, location)

Touch (object/organ)Touch (object/organ)

WaveWave

TalkTalk

Eat/drinkEat/drink

Climb (wall, stairs,..)Climb (wall, stairs,..)


Animation Library - CarAnimation Library - CarDrive (from, to), (path)Drive (from, to), (path)

Open (door)Open (door)

Close (door)Close (door)

Traffic simulationTraffic simulation


Animation Library - SpeechAnimation Library - SpeechText-to-speech Text-to-speech

Lip animation (sync?)Lip animation (sync?)


Animation Library – Facial ExpressionsAnimation Library – Facial Expressions

Basic emotions (smile, angry, afraid) Low level animation

Eyes (look-at, close, open, blink) Eye-brows Mouth

Face features (hair color, hair style, eye color, glasses)


Animation Library - OtherAnimation Library - OtherDoor (open, close)Door (open, close)

4-legged (share animation with humanoid)4-legged (share animation with humanoid)

Elevator (open, close, go-to-floor)Elevator (open, close, go-to-floor)

Bottle, glass, drink (pour, person drinking, spill)Bottle, glass, drink (pour, person drinking, spill)

Gun (shoot, bullet/missile)Gun (shoot, bullet/missile)

……


Animation CompositionAnimation CompositionSequence (states)Sequence (states)

ParallelParallel (walk, run, Jump) x (hold, wave)(walk, run, Jump) x (hold, wave)

Compound Animations (hierarchy)Compound Animations (hierarchy) E.g: eat = take (food), touch(mouth)E.g: eat = take (food), touch(mouth)

Inverse KinematicsInverse Kinematics


Smart ObjectsSmart ObjectsDifferent Complexity Levels of BehaviorDifferent Complexity Levels of Behavior

Default behaviorsDefault behaviors Rules and constraints (trigger -> action)Rules and constraints (trigger -> action) Behavior model (perception, motivation, planning)Behavior model (perception, motivation, planning)

High level interface to expose capabilitiesHigh level interface to expose capabilities

Naïve physicsNaïve physics

ExamplesExamples AvatarAvatar Tamaguchi/petTamaguchi/pet CarCar ElevatorElevator


Problem specification -exampleProblem specification -example

Smart lift in crowd simulation Smart lift in crowd simulation


Defining object behaviors - Defining object behaviors - exampleexample

smart tablesmart table in in party simulationparty simulation

graspable fruitsgraspable fruits get_closest_ get_closest_

fruit fruit (and eat it)(and eat it)

replace_fruitsreplace_fruits


Script ParsingScript ParsingDefine syntaxDefine syntax

Define internal representationDefine internal representation

Parse input, create representation, and write to Parse input, create representation, and write to filefile

ChallengesChallenges ActionAction Space (near, behind, path planning, obstacles)Space (near, behind, path planning, obstacles) Time (before, after, concurrent, fast)Time (before, after, concurrent, fast)


Along Example


Animation GenerationAnimation GenerationDefine internal representationDefine internal representation

Read representation from fileRead representation from file

Generate VRMLGenerate VRML

ChallengesChallenges Re-use VRML codeRe-use VRML code Performance and optimizationPerformance and optimization scalabilityscalability


Automatic EditingAutomatic EditingFormalize cinematic principlesFormalize cinematic principles

Analyze screenplayAnalyze screenplay

Hints (Annotations) in screenplay Hints (Annotations) in screenplay

EditingEditing Separate into shotsSeparate into shots Decide shot typeDecide shot type Camera motionCamera motion LightLight SoundSound DissolvesDissolves EffectsEffects


Example - shotsExample - shotsCamera placementCamera placement

The Line

External ExternalInternal

Apex


ExamplesExampleshttp://www.math.tau.ac.il/~doronf/se-anim_files/http://www.math.tau.ac.il/~doronf/se-anim_files/frame.htmframe.htm


Template-based Video EditingTemplate-based Video Editing


Automated template-based generationAutomated template-based generationWizard approach: user fills in the slotsWizard approach: user fills in the slots

Performance: static elements of presentation are pre-cached Performance: static elements of presentation are pre-cached (unless very large)(unless very large)

Allows for dynamic, on-the-fly generation of presentationsAllows for dynamic, on-the-fly generation of presentations

DOM

MPEG-4

Abstract Template

XSL

material 1….

XML

XSL XSL

SMIL-1

caching

caching

material n

DB


Rich Media TranscodingRich Media Transcoding

Documents

Workshop in Media Authoring Automatic Generation of Animation and Video