Multimedia Surveillance Systems

University of Genova

Department of Biophysical and Electronic Engineering

Multimedia Surveillance Systems

Prof. C.Regazzoni

Data Fusion and Bayesian Interaction

Modeling for Cognitive Ambient Intelligence

vIdeo and SIgnal Processing for Telecommunications – ISIP40

Multimedia Surveillance Systems (MSS)

Multimedia Surveillance Systems (MSS) are

designed for assisting the human operator(s) in the

monitoring of various kinds of environments, e.g.:

Transport environments (railway stations, level crossings,

car parks, urban roads, highways, etc.);

Commercial and financial environments (supermarkets,

banks, jewel shops etc.);

Tourist environments (naturalistic parks, tourist villages,

historical town centres, etc.).


Multimedia video-surveillance systems

Characterization

A Multimedia Surveillance System is characterized by: Processing of heterogeneous data from different types of

sensors;

Transmission over heterogeneous communication channels;

Multiple signal representation and event detection abstraction levels for presentation to an improved human interface.

A MSS is a system able to process, transmit and present in different ways heterogeneous data transmitted over various communication channels.



Examples of functionalities (1)

Detection of thefts and vandalism;

Detection of unauthorized accesses (both of people

and vehicles) to restricted areas;

Prevention of dangerous situation related to the

presence of hidden bombs;

Prevention of dangerous situations related to the

presence of unauthorized persons in areas only

allowed to vehicles (e.g. railway tracks, highway

tollgates, airport strips etc.);



Examples of functionalities (2)

People counting and crowding estimations;

Detection of cars stopping within level crossing

areas;

Traffic monitoring and control.


Overall video surveillance market

Estimated marked size based on manufacturer total

revenue (not end user)


Overall surveillance market

2009 Global video surveillance market: $8.26 Billion

Asia: $3.28 Billion

Americas: $2.87 Billion

EMEA: $2.05 Billion

Projected 2014 market size and implicit Compound

annual growth rate (CAGR):

Americas: $4.48 Billion with CAGR of 8.9%

Asia: $6.6 Billion with CAGR of 15.1%

EMEA: $3.3 Billion with CAGR of 10%


IP Video adoption by segments


IP Video adoption by segments

The graph shows the percentage adoption of network

vs analog video by segment.

Residential: ~15%

Healthcare and Education: ~45%

Retail and Banking: ~12%

Transport and Government: ~35%

High correlation between network video adoption and

surveillance over larger physical area


Grow projections by segment

The table forecasts growth rates by segment


Grow projections by segment

Key claims to note:

Transportation market size projected to more than double to

$3.6 Billion (or 25% of the global market)

Government market size projected to double to $2.3 Billion

Banking/finance and retail projected for lowest growth rates


IP Video Surveillance growth

IP video surveillance product sales will increase by 200% total between 2010 and 2012 overtaking analog CCTV sales

Mainstream 720p/1.3MP cameras cut into the cost advantage of analog cameras

Mainstream hybrid DVRs remove the barriers of legacy deployments to migrate to IP

Emerging managed/hosted video undermines analog's advantages in small deployments

Emerging panoramic cameras further cut the cost advantage of analog cameras

Maturing IP camera standards will cut costs and reduce complexity


Video analytics

Only "moderate growth" projected this year and next

30% annual growth rate by 2012-2013

2008 sales volume: 38,000 channels server based,

40,000 channels edge based (cameras, encoders)

Government and Transportation about 50% of total

video analytics market

IMS states, "The market is still awaiting that “killer

application,” and no one seems certain yet what that

will be"



Scientific projects

US Projects (Defense Advanced Research Projects Agency-DARPA)

[Active] Mind’s Eye (ME - develop in machines a capability that exists only in animals: visual

intelligence)

[Active] Combat Zones That See (CTS - track everything that moves in a city by linking up a massive

network of surveillance cameras to a centralized computer system)

[Active] Heterogeneous Aerial Reconnaissance Team (HART – Aerial surveillance)

[1997-2000] VSAM Video Surveillance and Monitoring

European Projects

[2010-2013] SUPPORT Project - Autonomous Monitoring for Securing European Ports

[2010-2012] VANAHEIM Project - Autonomous Monitoring Of Underground Transportation

Environment

[2008-2011] COFRIEND Project - Cognitive Vision System able to adapt itself to unexpected

situations

[2005-2008] SERKET Project - Video Surveillance Architectures

[2004-2006] AVITRACK Project - Airport Apron Monitoring

[2002-2004] INMOVE Intelligent Mobile Video Environments

[2002-2004] REOST Railway Electro Optical System for Safe Transportation

[2000-2003] ADVISOR - Subway Monitoring

[1994-1997] PASSWORDS, AVS-PV - Video Surveillance



Scientific projects

Italian national interest projects

[1999-2000] Sistemi intelligenti per l’elaborazione e la trasmissione di segnali multidimensionali per applicazioni di video-sorveglianza in tempo reale (Ministero dell’Università e della Ricerca Scientifica);

[2002-2006] VICOM-FIRB Virtual Immersive Communications (Fondo Italiano per la Ricerca di Base - Ministero dell’Istruzione, dell’Università e della Ricerca);

[2002-2004] Industrial grant Elsag PSA (sistema di video-sorveglianza con l’uso di sensori video omnidirezionali);

[2002-2005] Architetture distribuite ed eterogenee per sistemi di sorveglianza multi-sensoriali (Prot. N. 7280/297 MIUR);

[2008-2010] SINTESIS (Integrated Security System with Distributed Intelligence)



Reference model for a generic MSS

Sensor

layer

Peripheral

intelligence

Network

access

HUB modem

Local processing

layer Network layer Operator layer

PC cluster

Network

interface



First generation (1960-1980)

Switching

devices

Local processing

layer Network layer (coax

75 ohm cable

network)

monitor cluster,

control panel and

analog transceiver

Sensor

layer

Analog

transceiver

Storage and

retrieval



Second generation (1980-2000)

Network

access

HUB WAN

and/or LAN

modem

Local

processing

layer

(channel

coding and

TX) Network layer Operator layer

PC cluster

(attention

driving)

Network

interface

VDSL

IEEE

802.11

Cable modem

Sensor

layer

Peripheral

intelligence:

source coding



Third generation (2000-?)

Peripheral

intelligence

(embedded image

processing and

understanding)

Network

access

HUB modem

Local

processing

layer Network layer

Operator layer

Network

interface

Sensor

layer

UMTS

Cable modem

ADSL

ISDN

Decision planning


Historical and technological evolution of

video-surveillance systems

Sensor type

Information processing

Information transmission

Information presentation

Relevant processing tools

First Generation (1960-1980)

Analog None Analog/Digital CCTV

Cyclic multiplexing of analog/digital video signals

Analog TV cameras, box for digital video encoding

Second Generation (1980-2000)

Analog/ Digital

Centralised at operator level

Analog CCTV, Analog over coax networks, wireless digital, wired digital

Automatic signalling of “interesting situations” (e.g. alert signalling)

Analog/digital cameras, frame grabbers, PCs, digital wireless standards (e.g, 802.11), digital wired standards (cable modems, xDSL, etc.)

Third Generation (2000 - ?)

Digital Centralised/ distributed (configurable)

Wireless digital, wired digital (configurable)

Configurable signalling at semantic level

Digital intelligent sensors, communtication interface language (e.g. JAVA), software radio technologies for radio link reconfigurability


Beyond 3rd Generation: cognitive

surveillance

Improved perception and reasoning over monitored events will allow efficient dynamic context representation within complex environments

Extended communications new research trends go in the direction of extended interaction capabilities: User will be able to interact with occurring events

• Directly

• Through virtual “agents”

Towards a new paradigm…. Cognitive Surveillance


Cognitive surveillance

Sensing Analysis &

Representation

Action &

Comms Decision

Physical/Virtual

World

Beyond 3rd generation

AVS systems

Data Fusion

--------

Learning

Extended comms


Beyond 3rd Generation

Research lines:

Extended Perception/reasoning

Fusion coordination problems (physical synchronization and

action timing)

Learning stage needed (increasing difficulty with increasing

complex interactions)

Extended communications management


Extended perception & reasoning

The described scene understanding basic methods

cooperate to achieve

extended

perception

extended

reasoning

Mainly based on low level

scene understanding

techniques

Increasing abstraction level

information processing


What can we do now

Intelligent sensors

Video Analytics

• Limited set of robust 24/7/12 functionalities

– E.g. Intrusion detection, People counting, Traffic analysis

Networking

Architecture

• Thousands of connected heterogeneous (e.g. analog, IP, etc.)

cameras

• High speed wireless connections

Coverage type

• On line metadata and video streaming

• First examples of networked embedded video analytics


What can we do now

Control Center

Architecture

• Coexistence of multiple vendors Video Management

Systems

• Audio-Video centered multimedia event

presentation

Functionalities

• A posteriori Event/Situation interactive retrieval

• Off-line Data fusion and Decision support


What would/could we have now/tomorrow

Cognitive sensors

Video analytics

• 24/7 extended set of robust functionalities (e.g. crowd analysis,

action recognition, etc.)

Cognitive surveillance/Operator in the loop

• Integrated on line anomaly detection and decision support

Networking

Architecture improvements

• Heterogeneous networks interoperability - Ad hoc fault

resistent networks - Scalable multilevel data transmission

Extended coverage with software/cognitive nodes networks

• Mature networked embedded nodes - On line understanding

and prediction of complex multicamera events


Control center

Architecture

• Interoperable standards for VMS

• Heterogeneous data source fusion and presentation

• Tools for operator reactions monitoring

Functionalities

• Improved performance retrieval techniques (time

and space)

• Cooperative distributed on line analysis and

prediction for safety mainteinance

• Cooperative and distributed automated learning

from experience

What would/could we have now/tomorrow


SIGNALS

FEATURES

OBJECTS

SITUATIONS

PREDICTIONS

DECISION

Info

rmation p

er

tim

e a

nd

space

unit

1995 2000 2005 2010 2015 2020

CHASING

?

?

?

?

Technological trends in Cognitive sensors


Control Centre

Intelligent Sensing

Node(s)

Mobile Terminal

Network

Cognitive Node

Cognitive surveillance system

architecture


Networking: architecture and extended

coverage


Info

rmation p

er

tim

e a

nd s

pace

unit

2000 2005 2010 2015 2020

RETRIEVAL

PREDICTION

DECISION

SUPPORT

?

?

?

?

SINGLE SENSOR

Technological trends in Control centers


Examples of existing video-surveillance

systems (Communications)

Wireless networks: CCTV(R) products from Wavelength (UK) (first generation commercial system which transmits JPEG compress images with IEEE 802.11 standard). About 11.000 USD;

Wireless networks: system for recognition of stolen car number plate with intelligent sensors processing in-situ the information acquired among highways traffic (prototypic system operating in USA) (S. S. Shyne, “Distributed surveillance network utilises neural networks for stolen vehicle detection”, Proceedings of the SPIE, Vol. 2938, 1997, pp. 186-190).

Wired Network: System for video surveillance over Closed Circuit Twisted Pair (CCTP) for the digital age; a security system for airports, offices, schools, healthcare facilities, and government & military.

Wired network: system for turistic villages monitoring, developed in the CEE project AVS-RIO, with information transmission over a coaxial cable CATV network (second generation prototypic system operating in Riomaggiore municipality).



systems (Video management systems)

Genetec

OnSSI

Milestone

Avigilon

Mirasys

March Networks

Axis



systems (Computer Vision)

Scene understanding (Multiprocessor and Multisensor): AxonX (http://www.axonx.com/): SigniFire (fire and smoke detection)

Acuity (http://acuitysystem.com/): AcuVI (waterside protection)

AgentVI (http://agentvi.com/): VI-System (Real-Time Detection & Alert Software)

VideoIQ (http://www.videoiq.com/): iCVR object tracking embedded/server based

VCATechnology (http://www.vcatechnology.com/): embedded and pc based traffic analysis

Bosch (http://www.boschsecurity.it): Intelligent Video Analysis (IVA)

IoImage (http://www.ioimage.com): embedded and server based object tracking

ObjectVideo (http://www.objectvideo.com/): embedded and server based intrusion detection, traffic analysis

TechnoAware (http://www.technoaware.com): server based or embedded object tracking, traffic management, people counting, intrusion detection


MSS main physical features

System architecture;

Environmental coverage: # and type of sensors;

System complexity: # and processing power of

processors;

Bandwidth and communication channels features.

Network characteristics: open or dedicated network

Processing capabilities: # frames/sec;


MSS main logical features

Multiple access characteristics: #of end users

simultaneously supported

System functionalities # and type of supported

functionalities;

System performances: misdetection and false alarms

probabilities;


MSS Methodologies

- Data alignment

- Attribute estimation

- Object identity estimation

Multi-sensor integration Data fusion

-Agent based systems

-Cooperative systems

-Understanding

-Intelligence distribution

Artificial intelligence

-Channel coding

-Multiple access techniques

- Reconfigurable devices and networks

Signal transmission Communications

- Compression algorithms

- Change detection

-Information detection and hiding

- Source Coding and transmission

- Object recognition

Signal processing

- Neural networks

-PCA Analysis

-Model learning

-Feature extraction and selection for event

classification

Pattern recognition

- Shape, motion or texture analysis

- Tracking

-Sensor calibration

–Object and behavior characterization

–Event classification

Computer Vision

Examples of methodologies Purpose Research field


Different methodologies in MSS

Computer Vision

Tracking example

1st detected blob

2nd detected blob



Data Fusion

Shape extraction without

using multisensor

information

Shape extraction using

multisensorial

information



Pattern Recognition

Retrieval and classification with SHOSLIF trees

Probed samples Retrieved samples

Voronoi space tessellation

Quasi-Voronoi space

tessellation



Artificial Intelligence

An agent framework

No matchingMatching



Communication

ADSL phone line

HFC net

Wireless network


Signal and Video Processing for MSS

Signal Processing for MSS must focus the system

attention on the most important information

(depending on the functionality) for:

Recognition

Communication

There are several constraints regarding:

Processing time / real-time requirements (for recognition);

Transmission time / available transmission band (for

communication).


Signal Processing for MSS

Signal Processing techniques vary on the basis of the

processed data nature (audio, video, tactile, etc.)

Video sensors will be considered (passive sensors)

Sensors layerData processingfor alarm generation


Signal Processing for MSS

Steps that follows Signal Processing techniques are mainly:

Transmission of data over heterogeneous channels to a remote site. Data can be transmitted over open networks (such as Internet) or dedicated channels. Main problems in these cases are related with protection of information against noise and unauthorized access and data integrity;

System intelligence: signal processing methods for MSS are able to automatically extract an high level symbolic representation of the processed data suitable for pattern recognition and computer vision steps in order to wake up the attention of the human operator.


Intrusion detection


People counter


Behavior analysis


Abandoned objects


Traffic management


PTZ Tracking


PTZ tracking


Other functions

Documents

Multimedia Surveillance Systems