Livermore Labs - From Video to Knowledge

7/28/2019 Livermore Labs - From Video to Knowledge

1/8

4 Lawrence Livermore National Laboratory

S&TR April/May


2/8

5Lawrence Livermore National Laboratory

&TR April/May 2011 Persistent Surveillance

Livermores Persistics

data-processing pipeline

combines graphics-based

computer hardware

and clever software to

extract meaning from

wide-area overhead

surveillance video.

Unmanned ril vhicls, such sth ros Prtor Rpr, rrwritig th ruls of bttl i Irq

afghist r big cosir for

oitorig U.S. borrs. Oprt by

rot cotrol, sotis thouss of

kilotrs fro th bttlfil, th ircrft

us ultipl crs to trck stroy

th y.

Ths ros c fly svrl kilotrs

high, hovr ovr trgts for 24 hours

logr, r oft r with bobs

ir-to-grou issils. By fiig,

trckig, oitorig popl, vhicls,

vts of itrst o cotiuous

bsis with wi-r vio crs, th

ircrft provi prsistt itlligc,

survillc, rcoissc (ISR).

I thory, prsistt ISR covrg c

prvt is fro vig ovrh

survillc systs, thrby bligfst cisio kig for tcticl issios,

with ruc risk to U.S. forcs

ocobtts.

a w Livror coputtiol syst

is sig to hlp th dprtt of

dfs othr gcis oitor ts

of squr kilotrs of trri fro th

skis, with sufficitly high rsolutio

for trckig popl vhicls for

y hours t ti. Th syst, cll

Prsistics, proiss to ovrco svr

growig probl: th ovrwhlig

volu of vio t grt by

or ovrh igry. militry

wi-r vio survillc crs

such s Costt Hwk agl Fir

r collctig vr-icrsig outs

of vlubl vio igry ovr lrg

gogrphic rs. Howvr, th tchology

to proptly procss, stor, xtrct

ig fro t, th trsit this

ifortio ovr log istcs for furthr

lysis, hs lgg fr bhi.

Data Overwhelm Capabilities

Ssig cpbilitis for wi-r

vio rcoissc hv volv to

fr xc our wilst xpcttios,

sys physicistShil Viy, priciplivstigtor for Prsistics puty

progr irctor of dfs i

Livrors Globl Scurity Pricipl

dirctort. Thr xists orous

growiggp btw th out

of ifortio cotiuously collct

by xistig ssors our bility to

quickly tk vtg of tht ifortio

Unmanned aerial vehicles, such as the Predator shown

here, use multiple cameras for persistent surveillance.

(Courtesy of Department of Defense.)


3/8

S&TR April/May Persistent Surveillance

Th Prsistics t-procssig pipli

uss tchiqu cll pixl-lvl s

ig corrspoc to stbiliz vio,

liit prllx (th slight iffrcs i

th pprt positios of objcts viw fro

iffrt crs), coprss bckgrou,

provi uprct subpixl

rsolutio of ovig objcts of itrst.

ds ig corrspoc coprs th

sc fro o pixl to othr corrctsfor iiviul iffrcs.

Inside Persistics

avc ccurt lysis tchiqus

r urgtly by th popl who

r tsk with ctgorizig, ixig,

ottig, rwig coclusios fro

th ptbyts of t collct ily ithtr oprtios.

Coputr scitist mrk duchiu,

o of Prsistics ky rchitcts, sys,

Th t-procssig ifrstructur for

tiol scurity is ot sig for th

outs typs of t big grt

by u ril ros rltiv to

th scl of hu rsourcs vilbl to

lyz th through covtiol VCR

plybck styl of viwig. I itio,

th couictio bwith supportig

t trsissio fro ir to grou

th rchiv storg cpbility r uchtoo slow or too sll to support fst-

turrou hu lyss.

duchiu sys, Svrl yrs

go, dprtt of dfs grs

prst us with this itrstig

procssig chllg. Th Livror

scitists kw thy to work

o th softwr si to gt th ost

ifortio out of th strig vio

igry. Th grtst opportuity for

hlpig tckl this chllg ly i th

vlopt of vc lgoriths (short

coputr progrs) iplt o w

coputr rchitcturs.

Prsistics, th prouct of svrl

yrs ffort, is iovtiv t-

procssig pipli tht tks

riclly iffrt pproch to

rssig th vio-t ovrlo

chllg. (S th box t right.)

Th tchiqu rtis th lvl

of til cssry for tctig

olis whil t th s ti

coprssig th uchgig

bckgrou vrythigi otio by bout 1,000 tis

without losig prtit

ifortio. as such, th pproch

liorts th rth of couictio

bwith for trsportig vio without

losig ig fility. I, Prsistics


If w kow prcisly how y two v

igs rlt to o othr, w c r

prllx tri wht is bckgrou

wht is ovig, icrs sigl-to-o

rtio, icrs rsolutio to s ti

o vhicls, sys coputr scitist mr

duchiu. Th biggst bfit, howv

is th bility to coprss t without

throwig wy iportt ifortio.

Th tchiqu llows for igstbility, which is sstil for ccurt

(left) In the manual stitch image, the road does not line up between the collected images.

(right) Dense image correspondence corrects for intensity variations between cameras to

provide a seamless image.

tchology c prouc subpixl

rsolutio for th bckgrou y

ovrs (popl vhicls), thrby

llowig for itiol lyss of

suspicious ctivitis. a sigl pixl ccorrspo to ywhr fro svrl

squr trs to lss th 1 squr tr

of rl stt.

Th Prsistics rchitctur c support

r rl-ti oitorig for tcticl

cobt issios s wll s forsic

lysis of pst vts. Its lysis

lgoriths prit survillc systs to

str t ky popl, vhicls, loctios,

vts for hours v ys t

ti whil utoticlly srchig

with usurpss til for olis

or prslct trgts. Th Livror

brkthrough cobis optiiz

hrwr fturig th wst grti

of grphics chips (typiclly us forcoputr gig) with iovtiv

lgoriths. So lgoriths focus o

coprssig t whil othrs lyz

strig vio cott to utotic

xtrct its of itrst.

Th syst stbilizs icoig vi

igry through xtrly ccurt

clibrtio of obor crs

corrctio of pixl-to-pixl itsity

vritios. I this wy, osic

fro hurs of iiviul ssors


4/8


S&TR April/May 2011 Persistent Surveillance

(left) Persistics allows analysts to automatically follow individual

vehicles. In this scenario, a vehicle is tracked after a meeting

at a presumed safe house (red box) as it travels (green line) to

another building. (above) Persistics can extract a road network

from flowing traffic and show vehicle motion in a heat map

representation of traffic density to automatically discern if the traffic

pattern changes. Spurious tracks can be caused by the glint of

sunshine and other factors.

high-rsolutio objct itifictio

rckig. It lso prits slss stitchig,

procss tht opticlly cobis igs

ro ultipl crs to crt virtul

rg-fort cr. To o this, iiviul

rs ust b fily clibrt to ccout

or rltiv stitch positio itsity

iffrcs, ls istortio, pixl-to-pixl

ritios. Th rsultt osic bhvs lik

sigl ssor for Prsistics to prfor thcssry hug t ructio.

Persistics uses parallax-removing pixel-

level dense correspondence to register

two images and detect what has changed

in the time period between capturing

each image. The left image shows the

original aerial video view. In the center,

two images of the aerial view have been

corrected with the traditional method of

global registration. Because of parallax

motion, some buildings as well as movingvehicles are detected. The right image

corrected with dense correspondence

shows only vehicles in motion (the bright

areas); Persistics has removed the

static background.Global registration Dense correspondence

Spurious tracks

a rquirt for Prsistics thousfol

t ructio is th cotiul rvisit of

survillc pltfor ovr spcifi r. Th

icrsig cohrc of sc built up ovr

y psss proucs high-fility bckgrous

fro which it is sy to spot ovrs. By hvig

ircrft circl th s r, Prsistics

c clcult th ffct of prllx, which

cuss th prspctiv to costtly rott. Th

corrct ig provis stright-ow viwof highly til trri. Rovig prllx

ks it uch sir to tri wht is i

rlity ovig i th bckgrou, sys Prsistics

projct lr Holgr Jos. Currt procssig

tchiqus us globl rgistrtio, which grts

y fls-positiv ovrs (popl vhicls).

Prsistics is fficit i llottig procssig

rsourcs to whr thy r ost . For

xpl, it c utoticlly irct itiol

coputig powr to trck prticulr covoy of

vhicls i rl ti or kp wtch ovr group ofmris who r slp i poorly f r.


5/8

S&TR April/May


(a) The Autonomous Real-Time GroundUbiquitous Surveillance Imaging System

(ARGUS-IS) has four stabilized lenses

(telescopes), which together use 368 5-million-

pixel cell phone cameras. (b) A mosaic of the

368 individual images is stitched together to

form one wide-angle seamless image from which

details can be extracted.

Persistics software is currently housed on a high-density 12-node minicluster, providing an aggregate

40 trillion floating-point operations per second of computing capability. Each node contains a

combination of central processing units and seven graphics processing units. Pictured are (left) Sheila

Vaidya and Holger Jones.

c b rprst s ig fro

sigl cr. This procur llows t

syst to xploit th sptil tpo

rucy i th t ts out

ovrs fro th sttic bckgrou.

In Use at Ground Stations

I collbortio with th ntiol

Gosptil-Itlligc agcy, dfs

avc Rsrch Projcts agcy

(daRPa), air Forc, ary, svr

dprtt of dfs lbortoris,

Livror is icorportig th Prsistic

pipli ito t-procssig grou

sttios f with vio t fro Cost

Hwk vio crs obor both

u ircrft. Rct

of Costt Hwk igry hv vlitPrsistics pproch, ostrtig

1,000-tis coprssio of rw vio

collctios whil itiig high fi

I copriso, coprssio of still i

c ruc t cott tfol, whil

str vio coprssio c chiv

bst 30-tis t ructio.

alysts workig t grou sttio

will itrct with th trsitt irbo

vio t. For xpl, Prsistics

hs b itgrt ito th air Forc

Rsrch Lbortoryvlop Pursu

viwr to llow lysts to p, zoo

rwi, qury, ovrly ps o(a)

(b)

Persistent Surveillance


6/8



cll th autooous Rl-Ti Grou

Ubiquitous Survillc Igig Syst

(aRGUS-IS). This syst to tct

trck vts o bttlfils i urb

rs c covr 100 squr kilotrs,

sigifictly grtr ril footprit

th currt systs. Th aRGUS-IS

vio-collctio rt of 12 hrtz (frs

pr sco) is cosirbly grtr th

prvious systs, which oprt t 2 hrtz.

aRGUS-IS copriss 368 crs of

bout 5 illio pixls ch, iticl tothos us i cll phos. Th crs

oprt togthr bhi four high-qulity

tlscop lss. I ll, aRGUS-IS

hs 1.8 billio pixls copr with

4 illio pixls i th first Soo ssor

(th prcssor to aRGUS-IS) vlop

t Livror i 2003, 176 illio pixls

ifortio fro visul t. (S S&TR,

novbr 2004, pp. 1219.)

Persistics Coming On Board

although th currt Prsistics

cofigurtio is sig for grou-

sttio procssig, Livror rsrchrs

r workig to shrik th siz, wight,

powr rquirts to prit outig

of th syst obor survillc crft

withi two yrs. dvloprs r xplorig

th tr-off btw t coprssio,ig fility, coputig rquirts

to succssfully volv th xt-grtio

rchitctur tht will support obor

flight procssig of ssiv t sts.

Prsistics is big furthr hc to

work with daRPas wst grtio of

rl-ti prsistt survillc cpbility

tt. With this viwr, thy c us

rsistics to k rqusts such s, Giv

th frs tht rcor this vhicl

ro o to two oclock this ftroo, or

Show ll th vhicls tht stop t thisoctio toy. Sys Prsistics projct

r Holgr Jos, With Prsistics,

lysts c tri th rltioships

tw vhicls, popl, builigs,

vts.

Prsistics t-procssig ouls rly

corcil vio-procssig hrwr

sig for coputr grphics, vio

itig, gs. GPUs [grphics

rocssig uits] cotiu to grow i

owr, whil shrikig i siz rgy

quirts, sys Jos. I fct, thir

rocssig powr is outpcig tht oftrl procssig uits (CPUs), which

u ost coputr oprtios. Wr

iig th GPU tchology wv,

ys Jos.

Viy l forr projct tht

srch how GPUs ight b

rogr us i kowlg-

iscovry pplictios rlvt to tiol

curity. W rliz ths procssors

ritiolly sig for fst rrig

f visul siultios, virtul rlity,

coputr gigcoul provi

fficit solutios to so of th ost

hllgig coputig s fcig th

tlligc ilitry couitis,

ys Viy. (S S&TR, novbr 2005,

p. 1920.)

Th Prsistics softwr is currtly

ous o 12-o iiclustr,

ch o cotis cobitio

f CPUs GPUs. To hlp t

SR rquirts i cost-ffctiv

r, th Livror Prsistics

hs optiiz cobitio of

icroprocssors high- grphicsrs fou i both gig boxs

y prsol coputrs. Wh

obi with high-sp twork

oftwr tools writt i op-sourc (ot

or-propritry) co, th clustrs

utprfor lrgr or xpsiv

ropritry gis i xtrctig

ARGUS-IS imagery presentsnew challenges because of the

sheer amount of data collected.

The inset reveals people and

vehicles extracted from this

aggregate wide-angle view of

Quantico, Virginia.


7/8

S&TR April/May


Persistent Surveillance

i th ost vc Soo ssor

vlop i 2007, 800 illio pixls

i th ssor vlop i 2009 by th

msschustts Istitut of Tchologys

Licol Lbortory.Prsistics c siultously

cotiuously tct trck th otio

of thouss of trgts ovr th aRGUS-IS

covrg r of 100 squr kilotrs.

aRGUS-IS c grt svrl trbyts

of t pr iut, hurs of tis

grtr th prvious-grtio ssors.

Util ow, w h o prcticl wy to

stor tht uch t, sys Jos. With

Prsistics, w hv iovtiv tho

to coprss th quivlt of thouss of

hr rivs to just fw rivs.

Decades of Visualization Research

about 30 popl r ivolv i th

Prsistics ffort, icluig coputr

scitists, lctricl girs, optics

xprts, sttisticis, chi-lrig

spcilists who rw ig out of

visul t, cor Livror strgth. Th

Persistics researchers are

developing the capability to

produce three-dimensional images

from overhead video data.

Lbortory hs b couctig rsrch

i ISR for two cs. Prvious fforts

iclu sigig vc ssors

lrg, lightwight optics; lyzig spc

igry; ivtig istruts totct vrsril clsti ctivitis

ssocit with wpos of ss

structio. I itio, Livror

coputr scitists r vlopig

thos to g lrg volus of

visul t utot tchiqus to

xtrct ig k iscovris

fro tht ifortio for tiol

scurity pplictios.

aog othr ccoplishts,

rsrchrs hv pior thos to

loct rs of itrst i thr-isiol

siultios of uclr wpos physicsgrt by suprcoputrs prforig

trillios of oprtios pr sco.

Lbortory scitists hv lso vlop

coputr-utot tchiqus to srch

for icroscopic flws i glss optics

us i th ntiol Igitio Fcility, th

worls ost rgtic lsr.

my frl gcis r itrst

i Prsistics pplictios, icluig th

dprtt of ergy, dprtt of

dfs, dprtt of Hol

Scurity. Collbortios i ISR for ucsfgurs trty oitorig supp

xt to itrtiol orgiztios su

s th Itrtiol atoic ergy ag

th Coprhsiv nuclr-Tst-B

Trty Orgiztio.

Ovr th pst c, Livror

rsrchrs hv stui th vxig

issu of ssor output growth fr

outpcig hu cpbilitis to igst

rct to th collct t i ti

r. erly work, fu through

th Lbortory dirct Rsrch

dvlopt Progr, focus ovlopig softwr tchiqus tht

coul first quit th jittry vio tk

obor ovig irbor vhicl

bufft by th tosphr th

coprss th t. Th rsrch ffort

grw ito th dprtt of ergy

projct cll Soo, which vlop


8/8



b crri loft. Tht ffort shoul b

sir this yr bcus th siz of

GPUs is xpct to shrik yt gi, with

powr cosuptio ruc i hlf by lt

2011. as w ri th GPU tchologywv, w shoul hv th syst up i

th ir by 2013, sys Viy. W pl

to str to grou oly th ifortio

tht is prtit to th usr, which

y rquir syrgy of ultipl ssors

i th ir to provi robust stits of

whts iportt.

Prsistics tchology will soo b

vilbl s op sourc to th

govrt subcotrctors. Th

tchology is oulr to llow futur

plug ply, s Livror scitists

vlop itiol utot tchiqus.For xpl, thy r rsrchig wys

to k possibl th thr-isiol

viwig of trgts, which coul

furthr hc t coprssibility.

Thy r lso xplorig thos

to ovrly ultipl ssor iputs

icluig ifrr, rr, visul

t th rg t to obti

ultilyr sssst.

Viy ots tht u ircrft

hv ostrt thir ISR vlu for

yrs i afghist Irq. as U.S.

solirs rtur ho, th rol of ovrh

vio igry i by Prsistics

tchology is xpct to icrs.

Prsistics coul lso support issios

t ho, such s oitorig scurity

t U.S. borrs or gurig ports

rgy prouctio fcilitis. Clrly, with

Prsistics, vio s kowlg

strgth tiol scurity.

Arnie Helle

Key Words: agl Fir;autooousRl-Ti Grou Ubiquitous Survillc

Igig Syst(aRGUS-IS); Costt Hwk;grphics procssig uit (GPU); itlligc,survillc, rcoissc (ISR);Prsistics; pixl; Prtor; Rpr; Soo.

For further information contact Sheila Vaidya

(925) 423-5428 ([email protected]).

qulity. Viy rclls, Our ibility to

trsport th t to grou i fficit

r h bco bottlck.

Prsistics provis ot oly fficit

s of stbilizig coprssigvio to trsport it cross liit

bwith chls but lso bck-

oly tctio bhvior lysis

to iffrtit btw orl

borl pttrs of bhvior.

Fast Forward or Rewind

Prsistics vio procssig is vt-

riv, ig its lgoriths r

sig to tct i rl-ti pottilly

iportt vts i th strig vio

t. Th pipli c tct ovig

popl vhicls trck th ovry hours. autotic rcogitio

of crti scrios c provi rly

wrig of thrt grtly strgth

tcticl cobt support. For xpl,

lli forcs coul itrcpt trrorists i

th procss of pltig iprovis

xplosiv vic or sttig up for

lli bs ttck.Livror scitists r

lso vlopig utot thos for

itifyig pttrs of bhvior tht coul

iict vitios fro orl socil

culturl pttrs s wll s tworks

of subvrsiv ctivity.

at th othr of th spctru,

Prsistics supports forsic lyss.

Shoul vt such s trrorist ttck

occur, th rchivl igry of th public

spc coul b rviw to tri

iportt tils such s th ot

bob ws plc or wh suspct cs

th trgt r. With sufficitly high-

rsolutio igry, lw-forct or

ilitry usr coul o y zoo i o

iiviul fc i hvily popult

urb virot, thus itifyigth ttckr.

Looking Ahead

Livror rsrchrs cotiu to

optiiz th Prsistics pltfor so it c

wi-r ssors for oitorig uclr

oprolifrtio. Th Soo Prsistt

urvillc Syst ftur wi-r

iws t high rsolutio, rl-ti obor

t procssig, high-prforcisuliztio t th rcivig . I 2006,

h Soo Projct t rciv

R&d 100 awr for thir iovtio. (S

S&TR, Octobr 2006, pp. 45.)

Popl i th dprtt of dfs

ot xcit bcus thy rliz thy

oul us th s tchiqus to look

or trrorist ctivitis, sys duchiu.

2005, th vio-cr ffort tht

g with Soo ws pss to th

dprtt of dfs, which vlop

agl Fir for th air Forc mri

Corps Costt Hwk for th ary.aRGUS-IS, built by Bae Systs, Ic., is

h wst igig syst.

durig Soos vlopt,

daRPa grs tur to Livror

o xplor vus for rucig th

ssiv volu of t collct by ths

w ssors without scrificig ig

Documents

Livermore Labs - From Video to Knowledge