Automated Staging for Virtual CinematographyAutomated Staging for Virtual Cinematography Amaury...

Automated Staging for Virtual

Cinematography

Amaury Louarn Marc Christie Fabrice Lamarche

2018-11-08

IRISA / Inria Rennes Atlantique

MimeTIC team

1

What is Staging?

Staging is the process of correctly

positioning actors, lights and viewpoints

in order to have a pleasing aesthetic

view for the spectator.

Viewpoint:

Theatre: Fixed (public)

Movies: Cameras in environment,

Image composition

Hamilton, by Lin-Manuel Miranda

Back to the future, by Robert Zemeckis

2

Staging in video games

In video games, staging concepts are mostly used for:

• Cutscenes

Figure 1: Battlefield 1,

by Electronic Arts

• Interactive drama

Figure 2: The Walking Dead,

by Telltale Games

3

Automated staging?

4

Automated staging?

• 7 DoF per camera (Position, Orientation, Focal Length)

4

Automated staging?

• 7 DoF per camera (Position, Orientation, Focal Length)

• 6+ DoF per actor (Position, Orientation, Rig joints)

4

Automated staging?

• 7 DoF per camera (Position, Orientation, Focal Length)

• 6+ DoF per actor (Position, Orientation, Rig joints)

• Environment constraints (Obstacles, occlusions, etc.)

4

Automated staging?

• 7 DoF per camera (Position, Orientation, Focal Length)

• 6+ DoF per actor (Position, Orientation, Rig joints)

• Environment constraints (Obstacles, occlusions, etc.)

High complexity:

Everything is interdependent

4

Staging in litterature - Camera placement

Multiple approaches to camera placement:

• Constraint solving (Bares et al. 2000)

• Numerical optimization (Drucker et al. 1992; Olivier et al.

1999; Ranon et al. 2014)

• Geometrical approach (Lino et al. 2010)

5

Staging in litterature - Actor placement (1)

Talbot 2015:

Actor placement for theatre plays

• Fixed viewpoint (public)

• Simple environment (theatre stage)

• Spring-mass system

Fig: Spring-mass

staging system

6

Staging in letterature - Actor placement (2)

Elson et al. 2007:

Actor and camera placement for machinimas

• Free viewpoint

• Precomputed library of actor-camera

configurations

Fig: Staging library

Fig: Stage placement

7

Limits of current approaches

Talbot 2015:

• No cameras

• Simple environments

Elson et al. 2007:

• Finite library of stagings

• Simple scenes

Common hypothesis:

Staging can be done in 2D with satisfactory results.

8

Limits of current approaches

Talbot 2015:

• No cameras

• Simple environments

Elson et al. 2007:

• Finite library of stagings

• Simple scenes

Common hypothesis:

Staging can be done in 2D with satisfactory results.

Goals of our contribution:

• No a priori knowledge

• Arbitrary complex scenes

• High-level specification

8

Overview

Two-fold contribution:

1. A staging description language

• Based on Prose Storyboard Language by Ronfard et al. 2015

• Staging constraints: distance, orientation, visibility, etc.

2. A staging resolution engine

9

Pre-process - Environment

2D Topological map extraction from lightly informed

3D environment:

10

Pre-process - Staging Description

Geometric constraints identification from staging description:

• Distance• (Entity) is close to (Element)

• (Entity) is at [least|most] (Value) from

(Element)

Actor is close to Bar

11

Pre-process - Staging Description

Geometric constraints identification from staging description:

• Distance

• Orientation• (Entity) is (facing | left of | ... ) (Entity)

• (Entity) looks at (Element)

A looks at B. B looks at A

11

Pre-process - Staging Description

Geometric constraints identification from staging description:

• Distance

• Orientation

• Visibility• (Entity) is [not] seeing (Element)

A is seeing bar

11

Pre-process - Staging Description

Geometric constraints identification from staging description:

• Distance

• Orientation

• Visibility

• Framing• (Camera), (Size) on (Entity) [(Profile)]

[(Screen position)] {and (Size) on (Entity)

[(Profile)] [(Screen Position)]}

Camera 1, MS on A and MS on B

11

Engine - Overview

Algorithm in 2 Steps:

1. Search-space pruning

with constraints

2. Progressive sampling

via elicitation

12

13

Engine - Search space pruning

Toy example:

• A near bar

• B near A

• B in chair

Fixed-point process:

1. Compute geometric regions where constraints are satisfied

2. Deduce allowed regions for entities from it

3. Propagate the new entity regions to the constraints

4. Repeat until all entities are at their smallest allowed regions

14

Engine - Search space pruning

Toy example:

• A near bar

• B near A

• B in chair

Fixed-point process:

1. Compute geometric regions where constraints are satisfied

2. Deduce allowed regions for entities from it

3. Propagate the new entity regions to the constraints

4. Repeat until all entities are at their smallest allowed regions

14

Engine - Search space pruning

Toy example:

• A near bar

• B near A

• B in chair

Fixed-point process:

1. Compute geometric regions where constraints are satisfied

2. Deduce allowed regions for entities from it

3. Propagate the new entity regions to the constraints

4. Repeat until all entities are at their smallest allowed regions

14

Engine - Search space pruning

Toy example:

• A near bar

• B near A

• B in chair

Fixed-point process:

1. Compute geometric regions where constraints are satisfied

2. Deduce allowed regions for entities from it

3. Propagate the new entity regions to the constraints

4. Repeat until all entities are at their smallest allowed regions

14

Engine - Search space pruning

Toy example:

• A near bar

• B near A

• B in chair

Fixed-point process:

1. Compute geometric regions where constraints are satisfied

2. Deduce allowed regions for entities from it

3. Propagate the new entity regions to the constraints

4. Repeat until all entities are at their smallest allowed regions

14

15

Engine - Sampling

Toy example:

• A near bar

• B near A

• B in chair

1. Choose an entity (using a heuristic)

2. Choose a sample using a uniform distribution on its region

3. Apply a fixed-point process to propagate its new position and

orientation on other entities

4. If this yields an unsolvable situation, discard everything and

start the sampling process again

5. Else, continue with another entity

16

Engine - Sampling

Toy example:

• A near bar

• B near A

• B in chair

1. Choose an entity (using a heuristic)

2. Choose a sample using a uniform distribution on its region

3. Apply a fixed-point process to propagate its new position and

orientation on other entities

4. If this yields an unsolvable situation, discard everything and

start the sampling process again

5. Else, continue with another entity

16

Engine - Sampling

Toy example:

• A near bar

• B near A

• B in chair

1. Choose an entity (using a heuristic)

2. Choose a sample using a uniform distribution on its region

3. Apply a fixed-point process to propagate its new position and

orientation on other entities

4. If this yields an unsolvable situation, discard everything and

start the sampling process again

5. Else, continue with another entity

16

Engine - Sampling

Toy example:

• A near bar

• B near A

• B in chair

1. Choose an entity (using a heuristic)

2. Choose a sample using a uniform distribution on its region

3. Apply a fixed-point process to propagate its new position and

orientation on other entities

4. If this yields an unsolvable situation, discard everything and

start the sampling process again

5. Else, continue with another entity

16

Results

17

Results - complex scenario (1)

Scenario:Scene 1: A faces B. B faces A.

18

Results - complex scenario (1)

Scenario:Scene 1: A faces B. B faces A. A and B are close to the bar and

in chair.

18

Results - complex scenario (1)

Scenario:Scene 1: A faces B. B faces A. A and B are close to the bar and

in chair. A and B and Camera 1 and Camera 2 are not seeing C.

18

Results - complex scenario (1)

Scenario:Scene 1: A faces B. B faces A. A and B are close to the bar and

in chair. A and B and Camera 1 and Camera 2 are not seeing C.

Camera 1, CU on B front screenright and BCU on A back screenleft.

18

Results - complex scenario (1)

Scenario:Scene 1: A faces B. B faces A. A and B are close to the bar and

in chair. A and B and Camera 1 and Camera 2 are not seeing C.

Camera 1, CU on B front screenright and BCU on A back screenleft.

Camera 2, MCU on A right screenleft and B left screenright.

18

Results - complex scenario (1)

Scenario:Scene 1: A faces B. B faces A. A and B are close to the bar and

in chair. A and B and Camera 1 and Camera 2 are not seeing C.

Camera 1, CU on B front screenright and BCU on A back screenleft.

Camera 2, MCU on A right screenleft and B left screenright. Camera

3, FS on A and B and C.

18

Results - complex scenario (1)

Scenario:Scene 1: A faces B. B faces A. A and B are close to the bar and

in chair. A and B and Camera 1 and Camera 2 are not seeing C.

Camera 1, CU on B front screenright and BCU on A back screenleft.

Camera 2, MCU on A right screenleft and B left screenright. Camera

3, FS on A and B and C. Camera 4, FS on C. Camera 4 is not seeing A

and B.

18

Results - complex scenario (1)

Scenario:Scene 1: A faces B. B faces A. A and B are close to the bar and

in chair. A and B and Camera 1 and Camera 2 are not seeing C.

Camera 1, CU on B front screenright and BCU on A back screenleft.

Camera 2, MCU on A right screenleft and B left screenright. Camera

3, FS on A and B and C. Camera 4, FS on C. Camera 4 is not seeing A

and B.

Scene 2: A and B are in same position as in scene 1. C is at

most 4 meters behind A and facing A. Camera 1, same position and

orientation as in scene 1. Camera 2, MLS on B screenleft and A

screencenter and C screenright. Camera 3, FS on A and B and C.

Camera 4, MS on C. Camera 4 is not seeing A and B.

18

Results - complex scenario (2)

Video

19

Results - complex scenario (3)

Scene 1:

CU on B front screenright and

BCU on A back screenleft

FS on A and B and C

MCU on A right screenleft and

B left screenright

FS on C. Camera 4 is not

seeing A and B

20

Results - complex scenario (2)

Scene 2:

Same position and orientation

as in scene 1

FS on A and B and C.

MLS on B screenleft and A

screencenter and C screenright

MS on C. Camera 4 is not

seeing A and B.

21

Results - Real-life comparison (1)

Comparison with Back to the future by Robert Zemeckis:

Scene 1: George is in chair

facing the bar. Marty is in

chair facing George. Camera

1, CU on Marty 3/4 left

screencenter and George left

screenright.

Scene 2: Marty is in same

position and orientation

as in Scene 1. George is

facing Marty. Camera 1, MCU

on George front screencenter

and Marty 3/4 backright

screenleft.

Scene 3: Marty and George

are in same position as in

Scene 2. Goldie is on the

left of Marty and on the

right of George. Camera

1, MLS on Goldie front

screencenter.

22

Results - Real-life comparison (1)

Comparison with Back to the future by Robert Zemeckis:

Scene 1: George is in chair

facing the bar. Marty is in

chair facing George. Camera

1, CU on Marty 3/4 left

screencenter and George left

screenright.

Scene 2: Marty is in same

position and orientation

as in Scene 1. George is

facing Marty. Camera 1, MCU

on George front screencenter

and Marty 3/4 backright

screenleft.

Scene 3: Marty and George

are in same position as in

Scene 2. Goldie is on the

left of Marty and on the

right of George. Camera

1, MLS on Goldie front

screencenter.

22

Results - Real-life comparison (2)

Scene 1: Scene 2: Scene 3:

23

Summary

Our contribution is:

• a high-level staging description language

• a resolution engine implementing our language

The resolution engine:

• is not limited to a precomputed set of actor-camera

configurations

• works on any arbitrary scene

• can propose alternative stagings

24

Future works

Example: Wes Anderson’s symmetry

Fantastic Mr. Fox Moonrise Kingdom The Grand Budapest Hotel

Example: Quentin Tarantino’s trunk shot

Reservoir Dogs Pulp Fiction Kill Bill vol. 1

• How can we extend our language to more aesthetic shots?

• How can we model a movie style and extract it?

• How can we apply a movie style to new scripts?

25

References

Bares, W., McDermott, S., Boudreaux, C., & Thainimit, S. (2000).

Virtual 3D Camera Composition from Frame Constraints. In

ACM International Conference on Multimedia.

Drucker, S., Galyean, T., & Zeltzer, D. (1992). CINEMA: A System

for Procedural Camera Movements. In SI3D ’92: Proceedings

of the 1992 symposium on Interactive 3D graphics

(pp. 67–70). Cambridge, Massachusetts, United States: ACM

Press.

26

Elson, D., & Riedl, M. (2007). A Lightweight Intelligent Virtual

Cinematography System for Machinima Production. In

Proceedings of the Third AAAI Conference on Artificial

Intelligence and Interactive Digital Entertainment (pp. 8–13).

AIIDE’07. Stanford, California: AAAI Press.

Lino, C., Christie, M., Lamarche, F., Schofield, G., & Olivier, P.

(2010). A Real-time Cinematography System for Interactive

3D Environments. In ACM SIGGRAPH/Eurographics

Symposium on Computer Animation.

Olivier, P., Halper, N., Pickering, J. H., & Luna, P. (1999). Visual

Composition as Optimisation. In AISB Symposium on on AI

and Creativity in Entertainment and Virsual Art (pp. 22–30).

Ranon, R., & Urli, T. (2014). Improving the Efficiency of Viewpoint

Composition. IEEE Transactions on Visualization and

Computer Graphics, 20(5), 795–807.

27

Ronfard, R., Gandhi, V., & Boiron, L. (2015). The Prose

Storyboard Language: A Tool for Annotating and Directing

Movies. CoRR, abs/1508.07593.

Talbot, C. (2015). Directing virtual humans using play-script

spatiotemporal reasoning. (Doctoral dissertation, University

of North Carolina at Charlotte).

28