EV-Surv: Electronic and Visual Signals Integration for Efficient Surveillance

Dong Xuan, Ph.D.The Department of Computer Science and Engineering

The Ohio-State University

http://www.cse.ohio-state.edu/~xuan

Key Collaborators: Yuan F. Zheng, Jin Teng, Junda Zhu, Xinfeng Li, Boying Zhang and Qiang Zhai

Outline

Deficiency of Visual Surveillance Systems A Brief of Our EV-Surv System Case Studies A Broader View of Our EV-Surv System Final Remarks

2/55

Visual Surveillance Important for protecting public and personal security

3/55

Visual Surveillance Massive deployment in urban areas

Over 500 surveillance cameras in a Philadelphia neighborhood (below ) New York has 4176 video cameras in lower Manhattan area [1].

4/55

Surveillance in Action

Anomaly?Suspicious

Action?

Finding all white males in red, medium

stature, from Mon through Fri last week

Online monitoring

Offline retrieval

5/55

Failure Examples

Chicago police installed 10,000 surveillance cameras in the city, only 1 of 200 crimes is captured by the visual surveillance [2]!

In San Francisco, in the first three years after the city installed cameras, they helped police charge suspects in a grand total of six cases [2]!

One of the bombers in London bombing (July, 2005) is not identified by the surveillance system and escaped [3]!

6/55

Why fail?

Large volume of video dataTemporal: 2.07*106 frames per camera per daySpatial: tons of surveillance cameras in a city

Monitored objects may be visually occluded or

have multiple inconsistent appearance

Visual technologies are not efficient and accurate enough to do automatic localization and tracking, and a lot of human power is needed!

7/55

Outline

Deficiency of Visual Surveillance Systems A Brief of Our EV-Surv System Case Studies A Broader View of Our EV-Surv System Final Remarks

8/55

Our Methodology: E-V Integration

Combining electronic and visual signals, such as GSM, 3G, WiFi, Bluetooth and NFC signals together for efficient surveillance

E-V Integration makes it possible to efficiently and accurately localize and identify objects in large volume of video data

Indexing & Sorting Localization Accuracy

E-Signal Easy Low

V-Signal Hard High

9/55

Visual Signal-based Surveillance

Can accurately localize and continuously track a person But who is he or she?

Difficult to recognize people, e.g., through face recognition

Who are they?

10/55

Electronic Signal-based Surveillance Electronic Signals

Name Distance Frequency Data Rate (down)

GSM 35 km850, 900,

1800, 1900 MHz80 kb/s (GPRS), 236

kb/s (EDGE)LTE 30 km–100 km 700 MHz–2.6 GHz >100 Mb/s

WiFi 100 m2.4 GHz (802.11b/g), 5

GHz (802.11a)54 Mb/s

2.4 GHz, 5 GHz 450 Mb/s

Bluetooth 10 m2.4 GHz,

Frequency Hopping2.1 Mb/s

(up to 24 Mb/s)

NFC < 4 cm 13.56 MHz 106 kb/s–424 kb/s

Besides data exchanged, these communication channels also contain unique and identifiable electronic identities: IMEI (International Mobile Equipment Identity), IMSI (International Mobile

Subscriber Identity), WiFi, Bluetooth MAC addresses, RFID Number.11/55

Electronic signals are emitted by many mobile device Mobile device’s popularity is increasing

Smartphone as an example: 450 million shipped in 2011

Pervasiveness of Electronic Signals

Source: Technology Review, Sept/Oct 2011

Number Units Sold(millions)

12/55

Electronic Signal-based Surveillance

Cannot accurately localize a person with a mobile device Large error in localization due to interference

But can identify the device through electronic identifiers

Interferences, e.g., vehicles, building, humans etc.

-The with SIM card number 358985010745743 is here, -But the error can be as large as 100 meters

13/55

EV-Surv: A Bird’s Eye View

14/55

User Description:-Features, Clothing-Electronic Identity-Time Range- Area

User Description:-Features, Clothing-Electronic Identity-Time Range- Area

Visual signal

Visual signal

Visual signal

Electronic signal

Electronic signal

Electronic signal

E-V Integration

E-V Integration

E-V RetrievalE-V Retrieval

Backend Database

Gateway

Other Information Databases

All frames relevant to user inquiry

EV-Surv: the Workflow

15/55

Outline

Deficiency of Visual Surveillance Systems A Brief of Our EV-Surv System Case Studies

EV-Retrieval EV-Tracking

A Broader View of Our EV-Surv System Final Remarks

16/55

Case Study I: EV-Retrieval

Introducing electronic signals to help sort out videos for accurate and efficient person identification

Jin Teng, Junda Zhu, Boying Zhang, Dong Xuan and Yuan F. Zheng, “E-V: Efficient Visual Surveillance with Electronic Footprints”, IEEE INFOCOM 2012

17/55

Person Identification How a person of interest looks like at the time of surveillance

May be very different from any image or video of him in record A type of retrieval

Reference Photo

Appearance in video

A missing child A crime suspect

Problem:Given the identity of a person,

find his appearance

18/55

Traditional ways

Let people search through a huge pile of videos If done automatically, computers need to extract all

human figures in each frame and compare.

Too Costly

19/55

E Signal-assisted Retrieval With complementary electronic information, we can

find out the electronic identity of the person, and use that information to guide our visual search

In critical surveillance context, it is possible to acquire this information

E.g., the government can request service providers to disclose the user information in anti-terrorism operations

Or, sometimes, it may be the case that we only have an electronic identity

FBI gets a suspicious conversion through phone tapping

Much smaller search space after screening with E signals!

Less processing burden on the V side

20/55

Problem Formulation: Notations V-sensing: V-ID and V Frame

V-ID: Visual identity, such as human figure VID*: Our target V-ID V Frame: a set of V-IDs with some background captured

by visual sensors (cameras) in certain area and time

E-sensing: E-ID and E Frame E-ID: Electronic identity such as MAC address etc. EID*: Our target E-ID E Frame: a set of E-IDs captured by electronic sensors in

certain area and time

21/55

Problem Formulation Input: EID*, and a set of E frames and corresponding

V frames Output: VID* in video frames

We consider a baseline case with perfect E-IDs and V-IDs All E-IDs and V-IDs are clear and distinguishable. No

ambiguity. No false positives or negatives in the detection and

extraction of E-IDs and V-IDs We will discuss more practical cases later

22/55

A Basic Solution

Three steps:Step 1: Find out all E frames which include EID*Step 2: Find a subset of E frames, whose intersection

is EID*Step 3: Identify VID* in their corresponding V frames

Comments: Few V frames to process because V frames without VID* are filtered out, but there may be still many V frames

23/55

E frame 1

EID*

EID2

EID3

E frame 1

EID*

EID12

EID3

Example

E frame 1

EID*

EID2

EID3

E frame 1

EID*

EID2

EID7

E frame 1

EID*

EID2

EID3E frame 1

EID*

EID2EID3

E frame N+1

EID*

EID91

EID13

E frame 1

EID*

EID2

EID4

EID4E frame 1

EID*

EID2

EID3……

E frame 2

EID*EID2

EID4

E frame 1

EID* EID1

Millions of E Frames

E frame 3

EID* EID2

E frame 4

EID* EID5……

Step 1:Extract all E frames with EID*

24/55

E frame 1

EID* EID1

Example (cont’d)

E frame 2

EID* EID2

EID3

E frame 3

EID* EID2

E frame 4

EID* EID5

E frame 1EID1

E frame 2

EID2

EID3E frame 3

EID*EID2

V frame 1VID1

V frame 2

VID2

VID3V frame 3

VID*VID2

Step 1

Step 2

Step 3

25/55

Find the minimum number of E Frames, whose intersection is the given E-ID, i.e. EID*

Further less frames for V side processing

A Better Solution

EID*

E frame 1

EID* EID1

E frame 3

EID* EID2

E frame 2

EID* EID3

EID2Two E Frames are enough identify EID* through intersection.

E frame 1

EID1

E frame 2

EID3EID2

26/55

Nature of E-Filtering Finding the minimum number of frames, whose

intersection is EID* NP-complete: equivalent to the set cover problem

Whether each E-ID appears in each E frame is summarized in a matrix, with 1 meaning ‘appear’ and 0 ‘not appear’.

At least one 0 in each non-EID* column Use these 0s to ‘cover’ all non-EID* column (next page)

EID* EID1 EID2 EID3

e1 1 1 0 0

e2 1 0 1 1

e3 1 0 1 0

At least one 0 in each non-EID* column

27/55

Reduction to Set Cover ProblemEID* EID1 EID2 EID3

e1 1 1 0 0

e2 1 0 1 1

e3 1 0 1 0

E frame 1

EID3EID2

E frame 2

EID2

E frame 3

EID1 EID3

EID1

EID2

EID2

Set to be covered

Sets to cover

Set Cover Problem: Find the fewest sets from a pool of sets (left hand side), whose union includes the set to be covered (right hand side)

Cover

28/55

Solution: EDP Algorithm Element Distinguishing Problem (EDP)

The element to be distinguished is EID*

Greedily select E Frames in which the most number of E-IDs can be told apart from EID* In the example, the greedy algorithm will select e1 or e3

first, because we can tell two E-IDs are not EID* Repeat the greedy selection until EID* is distinguishable

EID* EID1 EID2 EID3

e1 1 1 0 0

e2 1 0 1 1

e3 1 0 1 0

29/55

EDP(cont’d) Approximation results can be achieved with the greedy

heuristic algorithm for the set cover problem

30/55

VID* Retrieval

Find the corresponding VID* from the frames selected by E-Filtering.

VID* is the only one that should appear in all the frames after E filtering.

Find all V-IDs in the selected frames, then an intersection operation can give VID*.

31/55

More Cases Vagueness and completeness of V-ID/E-ID

Vagueness: reflect how clearly a V-ID/E-ID can be identified Completeness: reflect if V-IDs/E-IDs are complete in a V/E frame

(false positive/negative)

√

√ The baseline case we have studied

□ practical case of our focus addressed

Input Target Input Frames

EID* VID* EIDs VIDs

Vagueness Clear □ □

Vague □ □

Completeness Complete

Incomplete □ □

√

√

√

√

√√

32/55

Practical Case I: Handling Vague V-IDs

Vague V-IDs Do not know for sure which person is which in different

frames

Difficulty in the intersection of V frames to find VID* Solution

nBM algorithm: find the VID with the largest probability of appearing in all V frames.

Same?

33/55

The nBM Algorithm n-partite Best Match Problem (nBM)

Put all VIDs in different frames in n different circles

n-partite graph (right)

Similarity matrix for all V-IDs which have appeared

1VID

1v

2v

1VID

2VID 4VID

3VID 5VID

3v

1VID

6VID

7VID 9VID

8VID

VID1 VID2 VID3 VID4 VID5 VID6

VID1 N/A 0 0.9 0.34 0.1 0.76

VID2 0 N/A 0.12 0.51 0.72 0.23

VID3 0.9 0.12 N/A 0 0.85 0.12

VID4 0.34 0.51 0 N/A 0.23 0.35

…

VID(m-1)

0.1 0.72 0.85 0.23 N/A 0

VIDm 0.76 0.23 0.12 0.35 0 N/A

v1 v2 v3

34/55

nBM (cont’d) Maximum Likelihood matching

Given the observed VID1 … VIDm Which VID is the best candidate

Calculate the probability of all VIDi across all V frames Select the VID with the largest probability

1VID

1v

2v

1VID

2VID 4VID

3VID 5VID

VID1 is not in v2

VID1 is in v2, and appears as VID2

35/55

Solutions to Other Practical Cases Careful Deployment

Make sure that the coverage of the camera and the wireless detectors are roughly the same

nBM is probability based, so it is naturally resistant to noises Select appropriate threshold in nBM for better tradeoff between noise

resistance and performance

Generalized EDP Handle missing/ghost E-ID Introduction of fuzzy logic to improve the robustness of EDP Use RSSI for estimation and smoothing

EID* EID1 EID2 EID3 EID4

e1' 0.98 0.95 0.1 0.01 0.06

e2' 0.9 0.01 1 0.94 0.04

e3' 0.88 0.99 0.03 0.1 0.12

e4' 0.99 0.02 0.89 0.27 0.23

EIDi 10

EIDi 1010

10

smoothing

smoothing

Time

EIDi

EIDi

36/55

Implementation

Real world implementationOne camera viewing from above to collect V frames1-3 laptops around sniffing the WiFi traffic to

collect E frames Tested on campus

GymnasiumLibrary

37/55

Combine E and V signals for more accurate localization and tracking

Preliminary work: EV-Loc

Case Study II: EV-Tracking

Boying Zhang, Jin Teng, Junda Zhu, Xinfeng Li, Dong Xuan and Yuan F. Zheng. “EV-Loc: Integrating Electronic and Visual Signals for Accurate Localization”, in ACM MobiHoc12.

38/55

EV-Loc for Localization Basic idea

Simultaneous E and V localization Same localization result E-IDi matches V-IDj Can collect localization results over time and perform

statistical matching

Visual Localization

Electronic Localization

V-IDj

E-IDi

39/55

Multiple Objects Localization

Minimize sum of localization differences

Localized E object

Localized V objectx1

x2

x3

y3

y1

y1

xi, yi: Localization results (coordinates)

x1 x2

y1

x3

y3y2

π=(3, 1, 2, 4)

x4

y4

x4

y4

40/55

Nature of The Problem Linear assignment problem (bipartite matching)

Minimum matching cost:argmin πi Σ||xi - yπi|| = ||x1 - y4|| + ||x2 - y3|| + ||x3 - y2|| + ||x4 – y1||

We can use the Hungarian algorithm to solve this optimization problemObjects’ EIDs Objects’ VIDs

e1

e2

e3

e4

v1

v2

v3

v4

41/55

In EV-Retrieval, we only use 0/1 to indicate E-IDs’ existence in E frames

However, we can improve it with EV-LocE-ID’s existence in a much smaller regionEasier for intersection

EV-Loc for EV-Retrieval

42/55

EV-Loc for EV-Retrieval (cont’d)

E frame 1

EID* EID1

E frame 2

EID*EID1

Original Scheme Now with Localization

EID1

EID*E frame 3

EID* EID1

EID1

EID0

Possible location

E frame 1

EID* EID1

E frame 2

EID* EID1

E frame 3

EID* EID1

Suppose all basestations can hear all mobile devices

43/55

Outline

Deficiency of Visual Surveillance Systems A Brief of Our EV-Surv System Case Studies A Broader View of Our EV-Surv System Final Remarks

44/55

EV Surveillance: Problem Space

TrackingOnsite Offline

Cooperative

Uncooperative

45/55

Problem Space (Cont’d) X: Tracking: offline or onsite Y: Object of monitoring:

Individual : non-coordinative Group: coordinative, tied by relation in terms of vicinity, social relationship,

appearance, action/behavior

Z: Object friendliness: Cooperative: E/V signals on purposely for easy tracking Uncooperative: neutral (E/V signal on/off follows its own cause), and even

misleading

Other possible dimensions: objects can be human and vehicles etc.

46/55

Typical Cases Case 1: <X: offline tracking, Y: individual object, Z: cooperative), i.e.

offline tracking of individual and cooperative object E.g. missing elders searching

Case 2: <X: offline tracking, Y: group objects, Z: cooperative) , i.e. offline tracking of group and cooperative objects E.g. public health monitoring

Case 3: <X: onsite tracking, Y: individual/group objects, Z: cooperative), i.e. onsite tracking of individual/group and cooperative objects E.g. sports training/traffic monitoring

Case 4: <X: offline/onsite tracking, Y: individual object/group objects, Z: uncooperative, i.e. tracking of uncooperative objects E.g. criminal tracking

47/55

Open Issues

E-V sensor deployment Electronic signal capturing E-V data analysis Privacy Real-world implementation and evaluation

48/55

Electronic and Visual Sensor Deployment

Electronic sensor deployment problem

How to do E/V sensors joint optimal deployment?

Visual sensor deployment problem

49/55

Mobile Devices emit electronic signals at different time

Electronic interference is always there Non-cooperative targets exist Different electronic signal capturing times vary

Electronic Signal Capturing

Wi-Fi Bluetooth RFID

Capturing Time 1~2 seconds 10.24 seconds 0.1 seconds

50/55

E-V Data Analysis

E-V integration discussed so far is simple E-Filtering aiming to minimize the number of V

frames not necessarily results in best performance E-Filtering an V-Retrieval should be integrated

together to get the “best” number of V frames for “best” VID* identification

Data mining on a huge, distributed but incomplete E-V data sets

51/55

Visual surveillance has been widely accepted (or tolerated)

How about electronic surveillance?E-IDs can be inferred to expose user’s real identity

Privacy

52/55

The key to successful E-V integration technologies

Small scale, medium scale and then large scale real-world implementation and evaluation

Real-World Implementation and Evaluation

53/55

Final Remarks Existing visual surveillance system is not efficient Our EV-Surv system

Integrates the E signals and V signals for efficient visual surveillance

Implemented in real world

Many open issues left, still a long way to go

54/55

References

[1] Big Apple is Watching You: http://www.slate.com/articles/news_and_politics/explainer/2010/05/big_apple_is_watching_you.html

[2] http://articles.chicagotribune.com/2010-05-06/news/ct-oped-0506-chapman-20100506_1_surveillance- cameras-vandalism-effect-on-violent-crime

[3] http://news.bbc.co.uk/2/hi/4659093.stm

[4] D. Smith, et.al, “Approaches to Multisensor Data Fusion in TargetTracking: A Survey”, Knowledge and Data Engineering, IEEE Transactionson, 2006.

[5] S. Cho, et.al, “Association and Identification in HeterogeneousSensors Environment with Coverage Uncertainty”, IEEE AdvancedVideo and Signal Based Surveillance, 2009.

55