1

High Performance Video Processing for Military Applications

2

MotionDSP – introduction

• Sean Varah, Founder/CEO MotionDSP• San Mateo software company, founded 2005• Leading developer of real‐time image processing 

software• Customers include US Air Force, US Secret Service, 

Navy NCIS, Scotland Yard, DOD/IC customers

3

Video in the US Air Force

• Market Size– Systems: The US DoD expects to spend $64 BILLION on unmanned 

systems by 20211

– Analysts: The US Air Force has 5000 people devoted to “ISR” data analysis and dissemination. Add to that: Navy, Army, DIA, CIA, NGA, NRO . . .

• 3‐vector expansion in video– More missions, more vehicles, more cameras– 330% growth in Combat Air Patrols of UAVs since 2007– Growing number UAVs and ISR vehicles  – Number of cameras per vehicle going up 10x

1) Strategy Analytics

4

The Problem

Lieutenant General David DeptulaFormer Deputy Chief of Staff for ISR, Headquarters, US Air Force“We’re going to find ourselves in the not too distant future swimming in sensors and drowning in data”

General Norton A. SchwartzAir Force’s chief of staff “If automation can provide a cue for our people that would make better use of their time, that would help us significantly,” NY Times.

5

$64 Billion of Systems

The following slides are excerpts from a presentation by Gen Paul Dettmer, Asst Deputy Chief of Staff for Intelligence, Surveillance, and 

Reconnaissance (US Air Force)

66

NanoNavigate / communicate

inside buildings

MicroClose-in reconnaissance & situational awareness

Man-portable- ISR

- Time-Sensitive- Lethal

Family of Expendables- Close-In ISR- Expendable Jammers- Lethal- Counter Air- Precision Clandestine Resupply- Cyber attack

“SUAS Family of Transformers”- Personal ISR- Lethal- SIGINT- Cyber/EW- Counter-UAV- AutoSentries

Multi-Mission- ISR

- Force protection- FID

Air-Launched- Close-in ISR

- Lethal- SIGINT/DF

Irregular Warfare

Anti-Access Support

Increasing across all mission sets

Tier II Joint- ISR- Comm Relay- Lethal- SIGINT

Bio-Mechanicals- Indoor Reconnaissance

- Indoor Lethal- Indoor Comm- Cyber attack

Next Gen Multi-Mission- ISR- Communications Relay- Lethal / Non-lethal- Electronic/Cyber Attack/SEAD- SIGINT/Low Altitude Pseudo-Sats- = New Mission areas

Now Future

Nano

Raven B

Wasp III

Switchblade SUASTechnical Demonstration

Scan Eagle

FinderSUAS

Voyeur SUASTechnical Demonstration

Artist Conception

Future AL-SUAS

Bio-Mechanicals- Indoor Reconnaissance- Indoor Lethal/Non-lethal- Indoor Comm- Cyber attack- Swarming

Lite Machine’sConceptual SUAS

GT AeroConceptual Bandit SUAS

6

AF RPA Flight Plan:Small “Family of Systems”

7

Specialized ISR

CollectionDissemination

ISR

Comm Relay

EW

CAS

AAR-TSEAD

Air InterdictionAeromedical EvacuationPersonnel Recovery

Mod

ular

Pay

load

s

MQ-Xa

7

MQ-1- Collection – SIGINT/FMV

- CAS

MQ-9- Collection – SIGINT/FMV

- CASMQ-9- Collection

- SIGINT/FMV- Wide Area Airborne ISR- SAR/GMTI

- CAS

MQ-Xb

MQ-Xc

Now Future

Specialized ISR

CollectionDissemination

ISR

Comm Relay

EW

AAR-R

Mod

ular

Pay

load

s

CAS

AAR-R&T

DisseminationSpecialized ISR

CAS

Collection

EW/ISR

Comm Relay

SEADAir Interdiction

Counter AirCSAR

Strategic Attack

Mod

ular

Pay

load

s

Missile Defense

AF RPA Flight Plan:Medium Systems

8

Ground Moving Target IndicatorInformation Integration

ISR

Airborne Moving Target Indicator

EW

Airlift

Command and Control

AAR-R & TAirlift

Humanitarian AssistanceStrategic Attack

Global StrikeCAS

Air Mobility

Mod

ular

Pay

load

s8

MQ-Lc

RQ-4 (Blk 20/30/40)- Collection:

- Block 20- Enh SAR- Enh EO/IR

- Block 30- Adv SIGINT

- Block 40- MP-RTIP Radar- GMTI and concurrent SAR- High Range Resolution- No EO/IR or SIGINT

MQ-La

RQ-4 (Blk10)- Collection – ISR

- Basic SAR- Basic EO/IR

MQ-Lb

Ground Moving Target IndicatorInformation Integration

ISR

Airborne Moving Target Indicator

EW

AAR-RMod

ular

Pay

load

s

Command and Control

Now Future

Ground Moving Target IndicatorInformation Integration

ISR

Airborne Moving Target Indicator

EW

Humanitarian Assistance

Command and Control

AAR-R & TAirliftM

odul

ar P

aylo

ads

AF RPA Flight Plan:Large Systems

9

10

11

The increasing number of cameras on each platform

12

13

Where the video goes

Analysts

14

15

16

Challenges for analysis

17

• Analysts overwhelmed by “data crush”• Analysis and exploitation tools not keeping pace with

sensor development and deployment• “Copy & Paste” approach to ISR fusion becoming the

“norm”…revealing new ISR information becoming rare• “Hunt and not Gather” mentality pervasive in ISR

operations…few resources and tools available to “cast-the-net” approach to analysis and fusion

21st Century Challenges:A new emerging problem set

Swimming in sensors… Drowning in Data…

17

18

The “Big Picture” – everything is a sensor, thousands of analysts

19

Viewers/Analysts of FMV (as of 2007)

• Army + EUCOM, Southcom– 537,000 desktops

• Air Force + STRATCOM, NORTHCOM, TRANS, JCS, OSD– 525,000 desktops

• Intelligence Community– About 190,000 desktops– DHS: 170,000 desktops

• Navy– NMCI 330,000 desktops (Navy Marine Corps Intranet)– OCONUS 50,000 desktops (Outside the Continental US)

1.6 million desktops 

20

21

The Problem

Video is rarely recorded in ideal conditions

Poor resolution Noise Camera shake

• Poor cameras– Low‐resolution sources like cell phones capture videos with poor detail, shakiness, and noise 

artifacts

• Poor Conditions affect capture quality for all sensors– Medium/Small UAVs don’t have the payload for stabilized camera pods – extreme shake– Even large UAVs and ISR platforms suffer from atmospheric distortion

22

“Traditional” military image processing

Custom hardware

• Pro: small/low power consumption• Con: limited capabilities  “one task” devices

– Lighting/contrast– Color– Rudimentary stabilization

• Con: difficult to integrate and upgrade

23

MotionDSP’s GPU image processing software: Ikena ISR

• Real‐time image processing with GPU‐acceleration• Runs on off‐the‐shelf Windows hardware• Superior image processing to hardware‐based solutions• Easy to integrate and upgrade• Lots of memory, lots of compute• Capabilities scale with Moores Law

24

Demo Video

• Med/Large UAV source• Real‐time stabilization, super‐resolution enhancement, noise removal, 

dynamic contrast

25

Cessna 206

Android

Enhanced Imagery Distributed to Disadvantaged Users Via IP

• IP Video and Still Images sent to Rover5 and Android/iPad

• Higher NIIRS quality, Higher Resolution

Rover 5

Shore

R/T Stabilization/De‐JItter

R/T Super‐Resolution Re‐construction

COTS Embedded Windows PC (GE MAGIC1) with GPU runs Ikena ISR software for R/T 

image processing and change detection

Stiletto

On‐board Real‐time FMV Image Processing on Scalable Cloud• COTS Rack‐mount servers with GPUs• Multi‐channel video input and output

• Live, Real‐time Image Processing• Live, Real‐time automatic change detection

FMV Image Processing

IP Video

Stiletto Demonstration

26

Ikena ISR WorkFlow

EZ TV System

Live IP Video

Outputs

Flat Video Files

Still Imagery

JPEGNTIF

COTS Workstation(Windows XP/7)

Inputs

Archived Video

High‐Resolution Still Images

JPEGNTIF

• Includes metadata• JPEG, PNG, BMP, TIF • NTIF (coming soon)

Flat Video Files

JPEGNTIF

• Includes metadata• Windows Media (WMV) format

• Also H.264, MPEG2 formats

IP Video• Real‐time enhanced results

• Includes metadata• H.264 or MPEG2

Real‐Time FMV Processing• Super‐resolution  enhancement

• De‐noise; de‐jitter; de‐interlace;  light, color,  and haze correction• Real‐time mosaics

• Ground‐mover detection• View embedded metadata Metadata: Output products include 

embedded metadata from the original input

27

Air Force ESC/XR Project Roadmap Capabilities 

Detecting motion from video •Combine motion detection with tracking•Fuse video content information with metadata•Adapt algorithms for very large pixel dimensions of WAS•Fuse with other layers of INT (GMTI, SigInt)•Expected testing in spring 2011

28

The Future

• Trends– Sensors looking at everything. 

• "ability to survey any spot on earth, in any weather, at any time"– Tools to analyze data automatically will be key

• Tools we need– Multi‐GPU virtualization

• Run multiple GPU algorithms dynamically across GPUs– Cloud GPU virtualization

• High‐performance cloud environments and virtualization software which allow full access to GPU compute

29

Questions/Thank you

MotionDSP Inc.

700 Airport Blvd, Suite 270Burlingame, CA 94010sales@motiondsp.com(650) 288‐1164 (o)

30

Disclaimer & Attribution

The information presented in this document is for informational purposes only and may contain technical inaccuracies, omissions and typographical errors.

The information contained herein is subject to change and may be rendered inaccurate for many reasons, including but not limited to product and roadmap changes, component and motherboard version changes, new model and/or product releases, product differences between differing manufacturers, software changes, BIOS flashes, firmware upgrades, or the like. There is no obligation to update or otherwise correct or revise this information. However, we reserve the right to revise this information and to make changes from time to time to the content hereof without obligation to notify any person of such revisions or changes.

NO REPRESENTATIONS OR WARRANTIES ARE MADE WITH RESPECT TO THE CONTENTS HEREOF AND NO RESPONSIBILITY IS ASSUMED FOR ANY INACCURACIES, ERRORS OR OMISSIONS THAT MAY APPEAR IN THIS INFORMATION.

ALL IMPLIED WARRANTIES OF MERCHANTABILITY OR FITNESS FOR ANY PARTICULAR PURPOSE ARE EXPRESSLY DISCLAIMED. IN NO EVENT WILL ANY LIABILITY TO ANY PERSON BE INCURRED FOR ANY DIRECT, INDIRECT, SPECIAL OR OTHER CONSEQUENTIAL DAMAGES ARISING FROM THE USE OF ANY INFORMATION CONTAINED HEREIN, EVEN IF EXPRESSLY ADVISED OF THE POSSIBILITY OF SUCH DAMAGES.

AMD, the AMD arrow logo, and combinations thereof are trademarks of Advanced Micro Devices, Inc. All other names used in this presentation are for informational purposes only and may be trademarks of their respective owners.

The contents of this presentation were provided by individual(s) and/or company listed on the title page. The information and opinions presented in this presentation may not represent AMD’s positions, strategies or opinions. Unless explicitly stated, AMD is not responsible for the content herein and no endorsements are implied.