Drone Cinematography - MultiDrone · Drone Cinematography • Value of current business services and labor that will be replaced in the near future by drone-based solutions, in the

Drone Cinematography

Presenter:

Nikos NikolaidisDepartment of Informatics, Aristotle University of Thessaloniki, Greece

[email protected]

Contributors:

Ioannis Mademlis, Chara RaptopoulouAristotle University of Thessaloniki, Greece

David BullUniversity of Bristol, UK

ICCV 2017 Tutorial on Drone Vision for Cinematography, 22/10/2017


• Unmanned Aerial Vehicles (UAVs, or “drones”) have

made their way in the media and entertainment industry.• Movies

• TV: sports, newsgathering, documentaries

• Advertising

• Drones allow filmmakers to capture shots that were

impossible, extremely difficult or costly to capture so far:• Wide shots of remote or hard-to-reach landscapes

• Follow-along close up views of sports or other action

• Impressive fly-throughs, fly-bys and fly-overs

• Essentially drones provide a level of camera motion

freedom that was so far available only in animation


• Value of current business services and labor that will be

replaced in the near future by drone-based solutions, in

the media & entertainment sector: $8.8bn (Source:

PricewaterhouseCoopers).

Drone Cinematography in Movies

• Example: rooftop motorbike chase scene from the J.

Bond film 007: Skyfall (MGM, Sony Pictures, Columbia

Pictures, 2012)

Drone Cinematography in movies

• Example: party scene from the film The Wolf of Wall

Street (Paramount Pictures, 2013)

Drone Cinematography in movies

and TV Series

• Numerous other films, TV series and documentaries

include scenes shot using drones:

• 007: Spectre (MGM, Columbia Pictures, 2015)

• Jurassic World (Universal Pictures, 2015)

• Captain America: Civil War (Walt Disney Studios, 2016)

• Game of Thrones (HBO, 2011-)

• Narcos (Netflix, 2015-)

• Planet Earth II (BBC, 2016)

GOT (HBO) Planet Earth II (BBC)

Drone Cinematography in Sports

Coverage

• Major channels including Fox Sports, BBC, and NBC

are using drones to cover outdoor sports events (live

or post-produced)

• Golf, auto-racing, skiing, rowing (2016 Olympics)

Dakar Rally 2016. (source Time.com, Franck

Fife—AFP/Getty Images)

Golf Channel

Drone Cinematography in News

• TV channels are using drones to cover large-scale

events (e.g. demonstrations), or capture videos from

war zones, natural disasters etc

War-torn Aleppo, Syria (Source: AMC) Moscow protest

https://dronelife.com/2014/04/01/

the-positives-and-negatives-of-drone-journalism/

https://dronelife.com/2014/04/01/


• Drones can replace spidercams, camera dollies, cranes

and helicopters.

https://commons.wikimedia.org/wiki/

File:2009_Emir_of_Qatar_Cup_Final_-_Spidercam_(3580955635).jpg

http://www.rent-jets.aero/aerial%20work/aerial-photography-details-big.jpg

https://www.proaim.com/media/wysiwyg/Images/Camera-Cranes/

Triangle/Proaim-40ft-Base-kit/Proaim-Wave-9-40ft-camera-crane-1.jpg

https://commons.wikimedia.org/wiki/

https://www.proaim.com/media/wysiwyg/Images/Camera-Cranes/


• UAVs advantages over dollies, cranes and helicopters:

• Reduced cost

• Cost of renting a helicopter for a few hours equals the cost of

buying a well-equipped drone.

• Fast and adaptive shot setup

• Access to narrow spaces, versatility, flights close to humans

and buildings

• Fewer legal and safety restrictions compared to helicopters

• Novel aerial shot types

• Continuous shootings with seamless transitions between very

different viewpoints/environments

• E.g.: the camera moves from ground level to several meters above

ground and then indoors, in a single shot


• The media and entertainment industry has already

praised the contribution of UAVs in modern

cinematography:

• A number of drone companies (DJI, PictorVision, Aerial MOB, Flying

Cam, etc) won a 2017 Emmy Award for Technology and

Engineering for their Low Latency Remote Controlled Airborne Video

Platforms for Television

• G. Hooper and P. George of HoverCam were awarded a 2013

Academy of Motion Picture Arts and Sciences (AMPAS) Technical

Achievement Award for the Helicam miniature helicopter camera

system.

• E. Prévinaire, J. Sperling, E. Brandt and T. Postiau were awarded a

2013 AMPAS Scientific and Engineering Award for the Flying-Cam

SARAH 3.0 system

UAV Cinematography

• UAV cinematography is mainly derived from traditional

ground and aerial cinematography, but must also take

into account UAV-specific limitations, capabilities,

properties

• A visual vocabulary of UAV cinematography can be

defined, consisting in:

• Shot types

• Combinations of framing shot types + UAV/camera motion types

• Composition principles

• Central Composition

• Rule of Thirds

• Lighting rules

• Depth-of-Field / Focus settings

© 2009, Jim Zuckerman

UAV Cinematography

• Need to define a standardized UAV shot type

taxonomy.

• Actual drone footage and related articles / guidelines

were used to this end:

• A total of 8 framing (static) shot types and 23 UAV/camera

motion types suitable for UAV media production have been

identified

• Camera motion types were clustered into groups according to

their characteristics

• Visually pleasing combinations of framing shot types and

camera motion types were identified

Framing Shot Types

• Framing Shot Types are more or less those of

traditional cinematography

• Most are defined based on the percentage of the video

frame width/height covered by the single

target/subject

FRAMING SHOT TYPE Percentage of frame

width/height covered by

target

Extreme Long Shot (ELS) <5%

Very Long Shot (VLS) 5-20%

Long Shot (LS) 20-40%

Medium Shot (MS) 40-60%

Medium Close Up (MCU) 60-75%

Close Up (CU) >75%

Framing Shot Types

• A couple of Framing Shot Types deal with two or more

subjects / targets:

• 2 Shot/3 Shot: 2/3 subjects appear on frame, equally visible

(typically LS or MS)

• Over the Shoulder (OTS): Adapted from traditional

cinematography OTS. Main target fully visible secondary target

visible at the video frame edge.

Framing Shot Types

• Example UAV shot types when shooting

boat targets from the side

Extreme Long Shot Long Shot

Medium Close Up Two Shot

UAV / Camera Motion Types

• Typical UAV trajectories along with coordinated camera

motions were identified

• UAV/camera motion types can be considered as either

“scene-oriented” or “target-oriented”

• Four groups of UAV/camera motion types were defined:

• Static shots (6): no UAV motion, target may or may not be

present

• Static Shot (SS)

• Static Shot of Still Target (SSST)

• Static Shot of Moving Target (SSMT)

• Static Aerial Pan (SAP)

• Static Aerial Tilt (SAT)


Source: Youtube: "5 Drone Moves

Every Flier Should Know",

https://www.youtube.com/watch?v

=1hz-lkx4o6c

Static Aerial Tilt (SAT)

• UAV hovers.

• Camera gimbal rotates slowly

around the pitch axis in order to

capture the scene context.

https://www.youtube.com/watch?v=1hz-lkx4o6c


• Dynamic shots (6): moving UAV, no target

• Moving Aerial Pan or Tilt (MAP, MAT)

• Pedestal/Elevator Shot (PS)

• Bird’s Eye Shot (BIRD)

• Moving Bird’s Eye Shot (MOVBIRD)

• Survey Shot (SURVEY)

• Fly-Through (FLYTHROUGH)


Source: Youtube: "How to

film amazing aerials with

your drone |

DroneFilmSchool",

https://www.youtube.com/wat

ch?v=zmQcSdj7vbE

Moving Bird’s Eye Shot

(MOVBIRD)

• Camera remains stable facing

vertically down.

• UAV is slowly flying parallel to

the terrain with constant velocity.

https://www.youtube.com/watch?v=zmQcSdj7vbE


Source: Youtube: "Manor

House Stables - Drone Film -

Horses",

https://www.youtube.com/wat

ch?v=56qOnxc28dw

copyright owner: M7 Aerial

Survey Shot

• Camera remains stable facing

ahead or backwards.

• UAV is slowly flying parallel to

the terrain with constant velocity.

https://www.youtube.com/watch?v=56qOnxc28dw


• Target Tracking shots (8): UAV motion depends on target

motion

• Moving Aerial Pan with Moving Target (MAPMT)

• Moving Aerial Tilt with Moving Target (MATMT)

• Lateral Tracking Shot (LTS)

• Vertical Tracking Shot (VTS)

• Orbit (ORBIT)

• Fly-Over (FLYOVER)

• Fly-By (FLYBY)

• Chase/Follow Shot (CHASE)


Source: Youtube: "5 Drone Moves

Every Flier Should Know",

https://www.youtube.com/watch?v

=1hz-lkx4o6c

Lateral Tracking Shot (LTS)

• Camera stays focused on the

moving target.

• UAV flies sideways/ in parallel to

the target, matching its speed

https://www.youtube.com/watch?v=1hz-lkx4o6c


Source: Youtube: "Drone

footage of Cycle-Racing",

https://www.youtube.com/watch

?v=vQ94R4LC9ig

Chase/Follow (CHASE)

• Camera stays focused on the

moving target.

• UAV follows/ leads the target

from behind/from the front,

matching its speed

https://www.youtube.com/watch?v=vQ94R4LC9ig

UAV Camera Motion Types

Source: Youtube: "Drone

Chasing Horses",

https://www.youtube.com/w

atch?v=O0uw8py9qmw

Orbit (ORBIT)

• Camera gimbal is slowly rotating,

so as to keep the still or moving

target properly framed

• UAV circles around the target while

following its trajectory (if any)

https://www.youtube.com/watch?v=O0uw8py9qmw


• Dynamic Target shots (3): a target exists, but UAV trajectory

also depends on other factors

• Constrained Lateral Tracking Shot (CONLTS)

• Pedestal/Elevator with Target (PST)

• Reveal Shot (REVEAL)

Constrained Lateral Tracking Shot (CONLTS)

• Camera remains stable, focused on the

moving target.

• UAV follows the target but it is constrained

to move onto a pre-defined “flight plane”

vertical to the ground plane

• Useful in sports, e.g. football

Multiple UAV Motion Types

• Motion types involving 2 or more drones in

orchestrated motion can be also considered

UAV Motion Type Modeling

• The drone trajectories and the camera gimbal motions

of the identified UAV motion types can be modeled

mathematically, using two coordinate systems:

• A global, fixed, World Coordinate System (WCS)

• A Target Coordinate System (TCS), attached and moving

along with the target.

• Such a modelling can prove useful for several tasks e.g

drone path planning and control.

UAV Motion Type Modeling

• Example: ORBIT• The camera always looks at the target while the drone moves

on a circle in the TCS

• Pt = [pt1, pt2, pt3]T : Target position in WCS

• Xt = [xt1, xt2, xt3]T : Drone position in TCS

• Lt: Camera Look-At Point in WCS

Tools of the Trade

• A UAV cinematographer has the following means for

achieving a certain framing shot type or UAV/camera

motion type or for transitioning between types:

• UAV location and orientation

• Camera orientation through gimbal control (usually 3 degrees

of freedom: roll, pitch, yaw)

• Camera focal length

Tools of the Trade

• Certain shot types can be obtained by more than one

camera / UAV configurations.

• The simplest, less energy consuming shall be chosen.

• Obtaining a certain framing shot type can simply be

achieved by setting the focal length f (zoom level)

• Zoom in to obtain a Medium Close Up

• The same shot type can be achieved by moving the

drone with respect to the subject

• More complex and consumes battery power

Tools of the Trade

• Certain UAV/camera motion types can also be

approximately materialized by means of changing only

the camera/gimbal configuration

• These approximations can/shall be preferred due to a

number of reasons:

• Inability to fly along the trajectory specified by the motion type

(e.g. due to proximity to humans)

• Inability to match the target speed

• Power consumption considerations

• Example: CHASE (follow) from behind a target moving

uphill.

• Due to energy consumption considerations, the same

shot can be approximately materialized by:

• Having the drone stop following the target and start hovering.

• Continuously changing the camera zoom and camera gimbal

orientation so as to always have the target in the frame center

and with the same size

Tools of the Trade

Current State of the Art,

Limitations & Challenges

• Current state of the art in UAV cinematography: single

drone, manually operated• Two persons are needed for professional productions: drone

operator, cameraman

• Single-UAV shooting limitations:

• A single target/subject can be captured at each time instance

• The target may only be captured from a specific viewpoint

and with specific shot type (e.g., Close-Up) at a given time

instance

• When UAV travels from one location to another to e.g. aim at a

different target the corresponding video might not be useful for

broadcasting.



• Multiple-UAVs (“swarm”) can alleviate these limitations

• Can cover multiple targets simultaneously or the same target

from different viewpoints, allowing the director to choose the

preferred video feed

• Can cover wide-area events

• Can be “relayed” in order to capture the moving target for a

longer period/distance

• Example: a UAV following a target is soon expected to enter a low-

battery mode and land.

• Another drone can take off and continue the task



• Multiple-UAVs shooting challenges:• Drone to drone collision avoidance

• Avoid having one drone entering the field of view of another

drone

• Coordination

• Crew size increase



• Manual control limitations:

• Difficult to achieve complex UAV motions especially when

combined with coordinated camera gimbal movement

• Orbit around a static target, camera always pointing to the target

• Additional difficulties when the target is moving

• Orbit around a moving target

• Follow a fast/unpredictably moving target physically or with the

camera



• Autonomous / intelligent drone/camera operation can

solve those issues or greatly facilitate drone/camera

control.

• Many technologies, sensors and research areas are

involved

• Cameras (standard, stereo, depth, infrared), Lidars, ultrasonic

sensors, GPS on the target

• Vision and machine learning on the generated data, path

planning, control, etc

• Moving target detection and 2D /3D tracking

• Detection of events (e.g. fall of a cyclist) and points of interest

• Automatic planning and execution of shots

• Automatic framing/camera control

• Rule of 3rds or central composition

• avoid having a drone in the field of view of the other

• 180 degrees rule

• Selection of the most informative or best framed video stream



• The technology is already here!

• Example: DJI Phantom 4 Pro can:

• Physically follow a moving target from behind or from the

side

• Orbit around a moving target by just using the roll stick

• Follow (lock) a moving target with the camera while the drone

moves under the operator commands

• Recognize various targets such as humans, bikes or

motorcycles

• Obvious questions that need to be answered: accuracy,

failure rates and robustness in:

• Cluttered environments

• Dynamic scenes with harsh/changing lighting conditions

• Fast and unpredictably moving targets


• Each UAV/camera motion type requires a certain

technology level to be performed autonomously

• Three technology levels can be identified:

• “Minimum”: No visual data analysis or 3D target localization

information is required for shooting such motion types.

• Example: scene-oriented shots with simplistic UAV trajectories

• “Vision”: Visual data analysis is required

• “Vision+3D”: Both visual data analysis and 3D target (and

drone) localization are needed



Research and Products

• Automated / intelligent UAV cinematography has

recently arise as an interesting R & D area

• Algorithms and software products have been developed

for automating aspects of drone flight and shooting,

while obeying cinematographic restrictions and director

guidelines

• The field lies at the intersection of:• Aerial cinematography

• Aerial robotics

• Computer vision/machine learning


• Joubert et al (2015) present a tool to support drone

video capture of static scenes.

• The tool enables users to:• Specify shots visually using keyframes for the drone camera

location and the camera look-at point;• The system then generates drone trajectories that satisfy the

equations of motion

• Precisely control the timing of shots using easing curves• When the user’s trajectory exceeds any physical limits the system

warns the user to modify it

N. Joubert, M. Roberts, A. Truong, F. Berthouzoz, and P. Hanrahan, “An interactive tool for designing quadrotor camera

shots,” ACM Transactions on Graphics (TOG), vol. 34, no. 6, pp. 238, 2015 (SIGGGRAPH Asia 2015).


• Preview the resulting shots in a 3D environment;

• Capture the resulting shots in the real world using a UAV.


• Joubert et al (2016) present a system to capture drone

video footage of human subjects performing rather

limited movements

• Subjects are tracked using RTK GPS and IMU sensors.

• The system automatically captures static (framing) shots

that respect visual composition principles (rule of 3rds).

• It also automatically calculates transitions (drone trajectories

and camera parameter configurations) between these shots

• Evaluated transitions are feasible, safe (e.g. not too close to

humans), and visually pleasing

N. Joubert, D. B. Goldman, F. Berthouzoz, M. Roberts, J. A. Landay, and P. Hanrahan, “Towards a drone

cinematographer: Guiding quadrotor cameras using visual composition principles,” arXiv:1610.01691, 2016.


• Roberts and Hanrahan (2016) propose a method that re-

times a user-defined physically infeasible UAV

trajectory

• Infeasible: it does not respect the dynamics and physical limits

of the UAV

• The output is a trajectory of the same geometry,

guaranteed to be physically feasible, as similar as

possible to the user’s input

• The user can preview the result in a 3D environment

and also capture the video in the real world using a

UAV.

[Roberts2016]: M. Roberts, P. Hanrahan. “Generating dynamically feasible trajectories for quadrotor cameras”, ACM

Transactions on Graphics, vol. 35, no. 4, 2016 (SIGGRAPH 2016).


• Nageli et al (2017) propose a method for aerial videography

planning in cluttered and dynamic environments.

• The method takes as input user specified, high-level plans (paths)

and framing objectives (position and size of filmed person in the

frame).

• The algorithm adapts the high-level plans in real-time to produce

feasible drones trajectories while also taking the motion of the

subjects into account, e.g. to avoid collisions

[Nageli2017b]: T. Nageli, L. Meier, A. Domahidi, J. Alonso-Mora, O. Hilliges, “Real-time Planning for Automated Multi-

view Drone Cinematography”, ACM Transactions on Graphics, vol. 36, no. 4, pp. 132:1-132:10, 2017.

[Nageli2017a]: T. Nageli, J. Alonso-Mora, A. Domahidi, D. Rus, O. Hilliges, “Real-Time Motion Planning for Aerial

Videography With Dynamic Obstacle Avoidance and Viewpoint Optimization”, IEEE Robotics and Automation Letters,

vol. 2, no. 3, pp. 1696-1703, 2017.


• The algorithm can also handle multiple drones

• Drone to drone collision avoidance

• Drone path modification to prevent a drone from entering the

field of view of other drones

[Nageli2017b]: T. Nageli, L. Meier, A. Domahidi, J. Alonso-Mora, O. Hilliges, “Real-time Planning for Automated Multi-

view Drone Cinematography”, ACM Transactions on Graphics, vol. 36, no. 4, pp. 132:1-132:10, 2017. (SIGGRAPH

2017)

[Nageli2017a]: T. Nageli, J. Alonso-Mora, A. Domahidi, D. Rus, O. Hilliges, “Real-Time Motion Planning for Aerial

Videography With Dynamic Obstacle Avoidance and Viewpoint Optimization”, IEEE Robotics and Automation Letters,

vol. 2, no. 3, pp. 1696-1703, 2017.


• Skywand:• A VR system allowing the user to explore a 3D model of the

scene and place desired, example key-frames in the

environment

• The system computes the UAV trajectory and the sequence of

camera motions, so as to capture smooth footage containing

the key-frames

• Previsualization of the trajectory and the video is available

• Freeskies CoPilot :• Similar functionality with a 3D map instead of VR

Open Issues

• What traditional cinematography rules persist in

UAV cinematography?

• What traditional rules can or should be broken to

create engaging / interesting videos?

• What opportunities are there for new rules of

cinematography, new shot types, camera

movements and visual effects?

• Shall preserve / enhance narrative and engagement,

creating a wow factor while preserving viewing quality and

comfort.

• Establish a grammar & taxonomy of drone

cinematography

• How these new visual paradigms can influence

visual narrative?

Open Issues

• What are the operating envelopes for the involved

parameters that lead to acceptable / engaging

visual results?

• Drone movement and speed relative to the target

• Camera zoom factor & orientation and their rate of

change

• Target dynamics

• What is the effect of errors (target tracking errors,

drone position and orientation errors) in the visual

results?

• Subjective and, if possible, objective tests and metrics are

needed

• Media professionals shall be consulted

Open Issues

• What are the appropriate types of transitions

between different shots?

• How one can use the new capture and display

formats (360 video, VR, plenoptic) within the scope

of UAV cinematography?

Tools and Research Needed

• R& D in computer vision and machine learning

• Intuitive and informative graphical user interfaces

for planning the drone and camera/gimbal

movements

• Tools (simulators) for the pre-visualization of the

drones trajectories and the videos acquired by the

cameras

Grazie Mille!

Documents

Drone Cinematography - MultiDrone · Drone Cinematography • Value of current business services and labor that will be replaced in the near future by drone-based solutions, in the