Robotic Cameras and Sensor Networks for High Resolution Environment Monitoring

Ken Goldberg and Dezhen Song

Alpha Lab, IEOR and EECS

University of California, Berkeley

NetworkedRobots

internettele-robot:

RoboMotes: Gaurav S. Sukhatme, USC

Smart Dust: Kris Pister, UCB (Image: Kenn Brown)

NetworkedCameras

Security Applications

Banks, Airports, Freeways, Sports Events, Concerts, Hospitals, Schools, Warehouses, Stores, Playgrounds, Casinos, Prisons, etc.

Conventional Security Cameras • Immobile or Repetitive Sweep• Low resolution

New Video Cameras:Omnidirectional vs. Robotic

• Fixed lens with mirror• 6M Pixel CCD• $ 20.0 K• 1M Pixel / Steradian

• Pan, Tilt, Zoom (21x)• 0.37M Pixel CCD• $ 1.2 K• 500M Pixel / Steradian

Where to look?

Sensornet detects activity

• “Motecams”

• Other sensors:audio, pressure switches,

light beams, IR, etc

• Generate bounding boxes

and motion vectors

• Transmit to PZT camera

Activity localization

Viewpoint Selection Problem

Given n bounding boxes, find optimal frame

Related Work• Facilities Location

– Megiddo and Supowit [84]– Eppstein [97]– Halperin et al. [02]

• Rectangle Fitting – Grossi and Italiano [99,00]– Agarwal and Erickson [99]– Mount et al [96]

• Similarity Measures – Kavraki [98]– Broder et al [98, 00]– Veltkamp and Hagedoorn [00]

Problem Definition

Requested frames: i=[xi, yi, zi], i=1,…,n

Problem Definition• Assumptions

– Camera has fixed aspect ratio: 4 x 3– Candidate frame = [x, y, z] t

– (x, y) R2 (continuous set)– z Z (discrete set)

(x, y)3z

4z

Problem Definition• “Satisfaction” for user i: 0 Si 1

Si = 0 Si = 1

= i = i

• Symmetric Difference

• Intersection-Over-Union

SDArea

AreaIOU

i

i

1)(

)(

)(

)()(

i

ii

Area

AreaAreaSD

Similarity Metrics

Nonlinear functions of (x,y)

• Intersection over Maximum:

),(

)(

),max(

)1,)/min(()/(),(

i

i

i

i

biiii

Max

Area

aa

p

zzaps

Requested frame i , Area= ai

Candidate frame

Area = a

Satisfaction Metrics

pi

Intersection over Maximum: si( ,i)

si = 0.20 0.21 0.53

Requested frame i

Candidate frame

),(),( yxpyxs iii

),( yxpi

Requested frame i

Candidate frame (x,y)

)1,)/min(()/(),( biiii zzaps

(for fixed z)

Satisfaction Function

– si(x,y) is a plateau

• One top plane• Four side planes• Quadratic surfaces at corners• Critical boundaries: 4 horizontal, 4 vertical

Objective Function• Global Satisfaction:

n

iii

n

i

biii

yxpyxS

zzapS

1

1

),(),(

)1,)/min(()/()(

for fixed z

Find * = arg max S()

S(x,y) is non-differentiable, non-convex, butpiecewise linear along axis-parallel lines.

Properties of Global Satisfaction

Approximation Algorithm

spacing zoom :

spacing lattice :

zd

dx

y

d

Compute S(x,y) at lattice of sample points:

Approximation Algorithm

– Run Time: – O(w h m n / d2)

* : Optimal frame

: Optimal at lattice ~

: Smallest frame at lattice that encloses *

)ˆ()~

()( * sss

)(

)ˆ(

)(

)~

(1

**

s

s

s

s

ddz

z

z

min

min...

Exact Algorithm• Virtual corner: Intersection between boundaries

– Self intersection:– Frame intersection:

y

x

Exact Algorithm

• Claim: An optimal point occurs at a virtual corner. Proof:– Along vertical boundary, S(y) is a 1D piecewise

linear function: extrema must occur at boundaries

Exact Algorithm

Exact Algorithm:

Check all virtual corners(mn2) virtual corners(n) time to evaluate S for each (mn3) total runtime

Improved Exact Algorithm

• Sweep horizontally: solve at each vertical – Sort critical points along y axis: O(n log n)– 1D problem at each vertical boundary O(nm) – O(n) 1D problems– O(n2m) total runtime

O(n) 1D problems

Examples

Summary• Networked robots

• High res. security cameras

• Omnidirectional vs. PTZ

• Viewpoint Selection Problem

• O(n2m) algorithm

Future Work

• Continuous zoom (m=)• Multiple outputs:

– p cameras – p views from one camera

• “Temporal” version: fairness– Integrate si over time: minimize accumulated

dissatisfaction for any user

• Network / Client Variability: load balancing• Obstacle Avoidance

Goldberg@ieor.berkeley.edu

Related Work• Facility Location Problems

– Megiddo and Supowit [84]– Eppstein [97]– Halperin et al. [02]

• Rectangle Fitting, Range Search, Range Sum, and Dominance Sum– Friesen and Chan [93] – Kapelio et al [95]– Mount et al [96]– Grossi and Italiano [99,00]– Agarwal and Erickson [99]– Zhang [02]

Related Work

• Similarity Measures – Kavraki [98]– Broder et al [98, 00]– Veltkamp and Hagedoorn [00]

• Frame selection algorithms – Song, Goldberg et al [02, 03, 04], – Har-peled et al. [03]

Problem Definition• Assumptions

– Camera has fixed aspect ratio: 4 x 3– Candidate frame c = [x, y, z] t

– (x, y) R2 (continuous set)– Resolution z Z

• Z = 10 means a pixel in the image = 10×10m2 area • Bigger z = larger frame = lower resolution

(x, y)3z

4z

Problem Definition

Requests: ri=[xli, yt

i, xri, yb

i, zi], i=1,…,n

(xli, yt

i) (xri, yb

i)

Optimization Problem

n

iii

zyxcrcsS

1],,[

),(max

User i’s satisfaction

Total satisfaction

Problem Definition• “Satisfaction” for user i: 0 Si 1

Si = 0 Si = 1

= c ri c = ri

• Measure user i’s satisfaction:

)1),/min(()/(

1,)(

)(min

)(

)(),(

zzap

cResolution

rResolution

rArea

rcAreacrs

iii

i

i

ii

Coverage-Resolution Ratio Metrics

Requested frame ri

Area= ai

Candidate frame c

Area = a

pi

Comparison with Similarity Metrics

• Symmetric Difference

• Intersection-Over-Union

SDcrArea

crAreaIOU

i

i

1)(

)(

)(

)()(

crArea

crAreacrAreaSD

i

ii

Nonlinear functions of (x,y), Does not measure resolution difference

Optimization Problem

n

iii

zyxcrcs

1],,[

),(max

),(),( yxpyxs iii

),( yxpi

Requested Frame ri Candidate

Frame c

)1,/min()/(),( zzapcrs iiii

(for fixed z)

Objective Function Properties

• si(x,y) is a plateau

• One top plane• Four side planes• Quadratic surfaces at corners• Critical boundaries: 4 horizontal, 4 vertical

Objective Function for Fixed Resolution

4z x

y

3z

4(zi-z)

Objective Function• Total satisfaction:

n

iii

n

iiii

yxpyxS

zzapcS

1

1

),(),(

)1),/min(()/()(

for fixed z

Frame selection problem: Find c* = arg max S(c)

S(x,y) is non-differentiable, non-convex, non-concave, but piecewise linear along axis-parallel lines.

Objective Function Properties

4z x

y

3z

4(zi-z)

3z y

si

3z

(z/zi)2

3(zi-z)

x

si

4z 4z

(z/zi)2

4(zi-z)

Plateau Vertex Definition• Intersection between boundaries

– Self intersection:– Plateau intersection:

y

x

Plateau Vertex Optimality Condition

• Claim 1: An optimal point occurs at a plateau vertex in the objective space for a fixed Resolution. Proof:– Along vertical boundary, S(y) is a 1D piecewise

linear function: extrema must occur at x boundaries

y

S(y)

Fixed Resolution Exact Algorithm

Brute force Exact Algorithm:

Check all plateau vertices (n2) plateau vertices(n) time to evaluate S for each (n3) total runtime

Improved Fixed Resolution Algorithm

• Sweep horizontally: solve at each vertical – Sort critical points along y axis: O(n log n)– 1D problem at each vertical boundary O(n) – O(n) 1D problems– O(n2m) total runtime

for m zoom levels

O(n) 1D problems

y

S(y)

x

y

A New Architecture

Activity and Video databaseActivity localization

Activities

Frame selection

Active surveillance

Control commands

Videos

Software diagram

TCP/IP

TCP/IP

Activity & video database

Core (with shared memory segments)

RPC module

RPC module

RPC module

Communication

Console/Log

Activity server

Activity generation

Motescam

Wireless Camera control

Calibration

Panoramic image generation

Video server

Panasonic HCM 280 Camera

Visual C++

NesC + Tiny OS

Gnu C++MySQL

Database: indexing video data

• Activity Index– Timestamp

– Speed (Or other sensor data)

– Range

• Query video data using activity – Show video clips of moving objects with speed faster than

1 meter per second in zone 1 in last 10 days

– Show video clips of zone 1 when CO2 concentration exceeded the threshold in Jan. 2004 (Assuming CO2 sensor is used in detecting activity)

Activity and Video database

Camera control