Visual Intelligence Driven Smart Imaging Sensors

Video Content Distribution For a Smarter Network

Visual Intelligence (VI)

Tarik Hammadou

“BraVo” a Revolutionary Enabling Disruptive Smart Camera Technology Platform

12/19/2012 1

Background

• An Entrepreneur by practice and engineer by training

• Holds several patents in the area of imaging technologies

• Expert in imaging technologies with passion for algorithm development

• Great ability to deliver emerging technologies in a form of a solution platform, identify early adaptors and delivers a market entry strategy.

• Delivered business in Asia, USA, Mexico, UK, Canada and worked with different R&D groups around the world.

• Other activities: Building a disruptive enabling TV broadcasting platform to enable capturing, processing and delivering interactive photorealistic live (real-time) content.

Looking to challenge the deepest resources of my intelligence to deliver disruptive enabling technologies and

generate unconventional sources of revenues and innovative business models.

12/19/2012 2

Market

12/19/2012 3

Market Size

• Video analytics market is to be worth 600 million dollars in 2015

• Global network security camera market will exceed $4 billion in 2015.

• DSP video processing market projected to be $6 billion in 2012

12/19/2012 4

IP cameras Market Trend 2012 and beyond

• Refocusing on Image Quality in 2012

• Focus to Shift from the BRICs to the CIVETS

• HD over Coax—What Will Happen in 2012

• Increased Processor Power to Push Analytics to the Edge

• M&A in 2012: Video Companies Under Surveillance

• Looking Up—Where is Next for Cloud Base Video Surveillance

• The Turning Point for Spinning Disk

• It’s VMS, but not as we know it

• Beyond H.264

• Video Surveillance to Augment the Internet of Things in China

12/19/2012 5

Market Trend: Refocus on Image Quality

• By 2015, 70% of all network cameras will be megapixel.

• Misconception: More pixels = better quality imaging

• Reality: Image quality= number of pixels + quality of lens + image processing

• The market opportunity for “high” megapixel cameras remains relatively niche.

• Manufacturers will need to further develop their points of differentiation/USPs.

• Focus will be in: low light capability, wide dynamic range, P-iris technology, real time image enhancement

12/19/2012 6

Market Trend: From BRICS to CIVEST

• The BRICs (Brazil, Russia, India and China) have been the countries of choice for video surveillance vendors seeking new growth opportunities. $ 2.5 billion in 2010 with a growth rate exceeding 20% for the following 2 years.

• Whilst the BRICs have and continue to offer opportunities for growth, many of the leading market players have already established footprints in these countries, diminishing any potential for first mover advantage.

• The CIVETS (Colombia, Indonesia, Vietnam, Egypt, Turkey and South Africa) are being touted as the next set of tiger economies due to their rapidly industrializing economies.

• In the medium to long term, as infrastructure development and social mobility increases, the CIVETS will provide a strong opportunity for video surveillance vendors to grow.

• IMPORTANT: The security industry has historically benefited from instability and the threat of terrorism

12/19/2012 7

Market Trend: HD over Coax

• Transmission of high definition images over existing (or new) analogue infrastructure.

• Manufacturers will continue to push HD over Coax equipment to the end-user, increasing availability and choice.

• HDcctv alliance conformant equipment and HD-SDI video equipment standards will progress and evolve.

• HD over Coax category will still see strong growth, with the market size nearly doubling, as increasing numbers of manufactures begin to push this technology forward.

12/19/2012 8

Market Trend:

Increased Processor Power to Push Analytics to the Edge

• Video content analysis (VCA) software can be run on standard off-the-shelf computers or embedded in video surveillance devices such as network cameras and encoders.

• Two approaches centralized and embedded DSP solution.

• More powerful processors and innovation in SoC design will push more VCA processing to the edge.

• If Moore’s Law continues to hold then the power of processors will continue to increase quickly.

12/19/2012 9

M&A in 2012: Video Companies Under Surveillance

• Remote monitoring market has seen sizable deals. Example: Broadview Security by Tyco International in 2010. The deal was worth around $1.9 billion. Private equity firm GTCR acquired and Monitronics acquired protection one for $1.2 billion.

• Large, high profile M&A have occurred within the system integrators market. 2011, $1.2 billion acquisition of Niscayah by Stanley Black & Decker. SAFRAN’s acquisition of L-1 Identity Solutions ($1.09 billion value).

• Video surveillance market has not really seen deals of a similar magnitude. Generally, M&A activity has been on a much smaller scale. Most recently, March Networks announced that it had been acquired by Infinova in a deal worth around $90 million. Other notable activity includes DVTel’s acquisition of video analytics manufacturer ioimage in 2010 and mergers at Panasonic and Samsung to create unified video surveillance businesses.

12/19/2012 10

Looking Up—Where is Next Cloud Base Video Surveillance

• Increased traction of cloud-based video surveillance, also known as Video Surveillance as a Service (VSaaS).

• 2011, notable sales growth at some of the vendors in this space. Overall, the market grew by around 20-30%. However, the market is still emerging, developing and evolving.

• One application that could augment the functionality of VSaaS is the addition of cloud-based video content analysis. By running analytics in the cloud, users can take advantage of the large virtualized processing power available to them. This use of the cloud, named Infrastructure as a Service (IaaS), is common in other industries.

12/19/2012 11

The Turning Point for Spinning

Disk • Many of the world’s largest manufacturers of HDDs, and suppliers of

components to the HDD industry, have production facilities in Thailand. Nearly 60% of Western Digital’s production takes place in Thailand, while Toshiba, the fourth biggest hard drive producer, makes about half of its HDDs in the country.

• While average price of HDD has been falling for last two decades the 2011 flooding generated disturbance in the supply chain and an increase in cost.

• Once HDD production recovers in Thailand, or production in alternative manufacturing sites is increased, the trend towards lower HDD prices and the resulting lower price per TB of storage systems will continue. However, for the moment it appears that the trend of lower prices for spinning disks has been temporarily paused.

12/19/2012 12

VMS, but not as we know it

• Whilst the upper tier of the market is occupied by a combination of the large, well established VMS brands and VMS offered by security solution providers; the middle and lower tiers are a “free-for-all”.

• key selling feature of many VMS systems has been “openness”, and whilst the ability to integrate to a broad range of video surveillance brands is still desirable, this has become more of a basic expectation.

• Targeted innovation: User interface, Mobile applications, Situation awareness.

• Shift from pure VMS to PSIM (Physical Security Information Management) software market, there is a definite and tangible demand for situational awareness (a correlation of multiple sensor inputs to generate an actionable report and response) platforms in specific industry segments.

12/19/2012 13

Beyond H.264

• H.264 has become the de-facto compression technology for video

surveillance systems.

• Combination of factors in 2012 could lead to advancements in video surveillance compression. The three factors are: 1. Flooding in Thailand. 2. Economic climate. 3. Growth of HD and megapixel network cameras.

• Video surveillance industry is not on the cutting edge of technology advancements in video compression. Potential compression technologies: H.264 SVC (extension of H.264), WebM, High Efficiency Video Coding (HEVC).

• Longer term, HEVC is perhaps the most likely successor to H.264, but that will depend on just how much better it performs and its suitability for video surveillance applications.

12/19/2012 14

Video Surveillance to Augment

Internet of Things in China • The Chinese central government is strategically focusing on the development

of seven emerging industries. These industries are expected to enjoy preferential policy treatment in a number of respects. New Generation of Information Technology (NGIT) is one of these industries. And IOT is one of the most important parts of NGIT.

• China has installed millions of video surveillance cameras over the past 10 years and these cameras could provide a source of information to verify the categorization of objects for IOT. IOT includes three layers: the perception layer, the network layer, and the application layer. Video surveillance cameras would provide a source of information for the perception layer.

• It is questionable whether IOT will drive new investment into security cameras. However, the expected benefits of IOT will help justify public expenditure on video surveillance equipment.

12/19/2012 15

Innovation

BUILDING THE NEXT GENERATION NETWORK MEGAPIXEL INTELLIGENT

IMAGING PLATFORM AND A MARKET DRIVEN STRATEGY

12/19/2012 16

The Problem

• Camera video technology platform today does not address the main market challenges:

1. Limited features (mainly streaming and basic analytics).

2. Does not integrate well with modern IT infrastructure.

3. limited on board processing capability.

4. No integration flexibility between HDcctv and IP.

5. No clear differentiator in the megapixel sensor cameras

6. Need for better compression technologies

7. Need to fit within the big ideas of Iaas, Vsaas and IOT

8. Too expensive for cloud computing: in streaming, storage, processing.

12/19/2012 17

Key Technical Innovation and Features

• Color interpolation, noise reduction, low light capability, real time image quality enhancement, HDR.

• Platform output flexibility, HDcctv or IP.

• An advanced processing SoC capability with balance between soft cores (ARM, DSP..) and silicon cores.

• Key technology components (SDK) towards building IP cores for H.264 SVC, WebM, HEVC.

• Build an advanced video content feature extraction chip set

• Feature extraction algorithms

• API for VCA(Video Content Analysis) applications.

12/19/2012 18

BRAVO 1st Generation OEM Platform Specs

• Megapixel CMOS image sensor: HDR, low noise, high sensitivity.

• Processing (IP cores):

1. Front end image processing engine.

2. Compression engine: H264, and SDK for H.264 SVC, WebM, HEVC.

3. Feature extraction: Motion segmentation, Background estimation, tracking (preprocessing), color extraction

• Processing (soft cores):

1. Feature extraction coding to XML(flexible, any format)

2. Android OS, web services and apps

3. Post-processing applications (crowd estimation, object recognition, Vehicle detection and classification)

• Output/interface, Input: HD-SDI, HDMI, Ethernet, USB, WiFi, 3G, 4G,

• Input sensors: accelerometer, gyroscope, compass, temperature sensor, and barometer.

12/19/2012 19

Software Development Kit

• Matlab, Eclipse development environment/ Android Cam emulator

• Innovative design methodology.

• New generation of intelligent video surveillance services

• Streaming video driven by modern HTML5, javascript techniques

• Compatible with major video processing libraries: OpenCV

12/19/2012 20

Selecting the Image Sensor

12/19/2012 21

CMOS Image Sensor

• CMOS image sensors in the market suffer from low performance under low illumination, noise, motion blur in low illumination.

• Conventional CMOS image sensor camera architecture, more processing is required to produce high quality imaging. Higher is the noise level and and lower is the sensetivity more complex are the algorithms

12/19/2012 22

Back light illumination CMOS imagers

• The technology offers superior performance. Low noise, improved sensitivity, high dynamic range.

• Important to identify(partner) the right sensor/supplier and set the right requirements. Those features are the first requirements and differentiator within the main stream video surveillance.

http://www.sony.net/SonyInfo/News/Press/201201/12-009E/index.html

12/19/2012 23

Camera Front End Processing

12/19/2012 24

The Front end Processing Chain

Conventional processing chain used by most IP CMOS megapixel cameras 12/19/2012 25

Processing: Front end image processing

• The main issue for most megapixel IP cameras: low performance, and noise increase under low illumination.

• We propose a filtering method based on a trade-off between real time implementation with very low hardware logic and the usage of some human vision characteristics, texture and noise level estimation.

• The filter adapts its smoothing capability to local image characteristics yielding effective results in terms of visual quality.

• Algorithm development, Implementation, testing will lead to the development of a specific patent application. The patent and the claims need to be reflected in the product performance Differentiator.

12/19/2012 26

Proposed Algorithm for noise reduction/image enhancement

Signal analyzer

block

Filter 𝑃 𝑓

degree analyzer

Noise level estimator

Similarity thresholds 𝒕𝒉𝒍𝒐𝒘, 𝒕𝒉𝒉𝒊𝒈𝒉

computation block

Filter Mask

hvs

Noise level

Texture degrees (𝑻𝒅)

𝑻𝒅

𝒕𝒉𝒍𝒐𝒘, 𝒕𝒉𝒉𝒊𝒈𝒉

𝑲𝟏…… .𝑲𝒔𝒊𝒛𝒆

𝑫𝒎𝒂𝒙, 𝑫𝒎𝒊𝒏

𝑫𝒎𝒂𝒙

𝑫𝟏…… .𝑫𝒔𝒊𝒛𝒆

12/19/2012 27

Preliminary Results Noise patterns

Original Image in low illumination Processed Image

12/19/2012 28

Compression: Strategy • Adopt H264 to meet short term demand.

• Provide a road map towards building key modules related to

implementation of H.264 SVC (extension of H.264) and High Efficiency Video Coding (HEVC).

• The IP silicon compression module can be licensed to different verticals. Video surveillance, TV-setbox, mobile application video streaming.

• It is a valuable IP assets to have. With the HEVC compression emerging, a HW module will be in high demand.

• The opportunity is to develop new techniques combining video streaming, content extraction and build an adaptive video streaming technology specific to Video surveillance

http://arstechnica.com/business/2010/02/royalty-free-codec-still-needed-despite-no-cost-h264-license/ 12/19/2012 29

Surveillance application

The idea discussed in the slide is directly applicable in Video surveillance scenario because it requires video transmission to different users according to the content (in this case moving objects like a suspect car) and network bandwidth available to different users.

12/19/2012 30

Content Based Network Adaptive Video transmission

• At a conceptual phase:

12/19/2012 31

Content Based Network Adaptive Video transmission

The idea is simple: based on the content extracted and analyzed, based on the network resources we can adjust the bandwidth allocation, we are also able to adjust bit rate at the camera level. This technique will allow a fully adaptive video transmission. 12/19/2012 32

Video Content Analysis

• We are taking a complete different approach.

• Our method is based on learning machines and statistical analysis

• It is an evolution of the work I conducted over the last 15 years, moving from spatial processing/feature extraction to learning machines.

• This OEM program objective is to build self contained intelligent machines able to learn from examples and experience.

12/19/2012 33

Video Content Analysis Learning from Annotation Example

The idea is inspired from methods of counting bacterial cells in a fluorescence-light microscopy We are taking same approach for counting people in a surveillance video frame. Close-ups are shown along side the images. The bottom close-ups show examples of the dotted annotations (crosses). The method learns to estimate the number of objects in the previously unseen images based on a set of training Images of the same kind augmented with dotted annotations.

12/19/2012 34

Counting People: Annotation

• Totally unsupervised

• Reporting numbers

• Bounding boxes

• Pixel-accurate

• Dotted annotation

• ...arguably minimal for humans

• ...containing much more information than just numbers

12/19/2012 35

Video Content Analysis

12/19/2012 36

Application: Counting cells

12/19/2012 37

Results: Counting Cells

12/19/2012 38

Application 2: Counting People

12/19/2012 39

Counting People Results

12/19/2012 40

Demo

Ground truth count Count estimated by our method (trained on frames 1200-1600)

click to play 12/19/2012 41

People Count Conclusion

• Training to be done offline and training weights can be automatically updated or manually.

• Training data can be pushed to the camera processor to generate weights

• Possibility to build an automated annotation tool, to reuse positive detection as training data.

• Friendly HW implementation

• Very fast in training and in runtime. 95% accuracy in average

• Algorithm update every quarter.

12/19/2012 42

The Processing Module

• The heart of the smart camera or any smart machine is the processing module.

• Apple and Samsung dominance in the market of portable devices is the result of PoP (Package on Package) mainly stacking two wafers together.

• Apple uses A4 chip. It is an SoC and combines an ARM Cortex-A8 CPU with a PowerVR GPU.

• The future of smart camera will depends on those type of processors, a generation of SoCs combining CPU, GPU and specific IP cores.

PoP illustration integrated circuit packaging, first tape out samples under MIVG.

12/19/2012 43