"System-level Design for Embedded Vision with FPGA-based Programmable SoCs," a Presentation from Avnet Electronics Marketing

Copyright © 2015 Avnet 1

Mario Bergeron, Avnet

12 May 2015

System-Level Design for Embedded Vision

with FPGA-based Programmable SoCs


• Who We Are:

• We are one of the world’s largest global distributors of electronic

components, computer products and embedded technology serving

customers in more than 80 countries

Company Introduction—Avnet, Inc.

• What We Do:

• We connect the world's leading technology

companies with more than 100,000

customers by providing cost-effective,

value-added services and solutions

• Financial Scope:

• For the fiscal year ended June 28, 2014, we

generated revenue of $27.5 billion


• What are FPGA-based Programmable SoCs ?

• Advantages of using these devices

• Typical development cycle

• Challenges of using FPGA-based Programmable SoCs

• Balancing the load

• Building the infrastructure

• Choosing the right IP

• Solutions with Tools

Overview


FPGA-based Programmable SoCs

What are they ?


• The best of both worlds

• Software

• Operating Systems

• Leverage legacy code

• Hardware

• High performance

• Standard or custom interfaces

• Accelerators

What is an FPGA-based Programmable SoC ?

FPGA-based Programmable SoC

Software Hardware Processor

sub-system

FPGA

sub-system

• Disadvantages

• Complexity

• Hardware

design

• Advantages

• Differentiation

• Integration

• Flexibility/Scalability

• Performance



FPGA

sub-system

Typical Development Cycle

1 Hardware platform

2 Software application

3 System Profiling

4 Accelerator

Processor

sub-system

FPGA

sub-system

custom

interfaces

http://www.google.com/url?sa=i&rct=j&q=&esrc=s&frm=1&source=images&cd=&cad=rja&uact=8&ved=0CAcQjRw&url=http://pixshark.com/construction-sign.htm&ei=kM8vVfbyENDvoASL-ICwAQ&bvm=bv.91071109,d.cGU&psig=AFQjCNG0hEhih97144q_EylzJCCdppxu5Q&ust=1429283060493052

http://www.google.com/url?sa=i&rct=j&q=&esrc=s&frm=1&source=images&cd=&cad=rja&uact=8&ved=0CAcQjRw&url=http://www.logoeps.com/download-hdmi-logo-vector-ai-eps-crd/1699/&ei=Rc0vVcnqKsGoogSCy4HQCQ&bvm=bv.91071109,d.cGU&psig=AFQjCNHggvPo_FZ1VO8pYmX1sppBCkizYg&ust=1429282300552888

http://www.google.com/url?sa=i&rct=j&q=&esrc=s&frm=1&source=images&cd=&cad=rja&uact=8&ved=0CAcQjRw&url=http://www.clipartbest.com/cartoon-race-car-images&ei=_9IvVebdO9GVyASImoGoDA&bvm=bv.91071109,d.cGU&psig=AFQjCNGSvBB_Bh2B_WSxTxIWClm8QgoKwg&ust=1429283883244134


Challenge :

Balancing the Load


• Processing capabilities of software and hardware are very different

• Software

• ~ 1 GOPS per processor core

• 1 GHz ARM processor : capable of handling VGA @ 30 frames/sec

• Hardware

• from 100s of GOPS to several TOPS depending on device

• Very capable of handling 4K2K resolutions or 1000s frames/sec

Balancing the Load


Processor

sub-system

FPGA

sub-system

Processor

sub-system

FPGA

sub-system


Typical Computer Vision Requirements

100’s OPS/pixel

8MP×100 Pixels / frame

= 100’s GOPS

10000’s OPS/feature

1000’s of features/sec

= MOPS

4K×2K

1080p

720p

480p

Pixel based

Image Processing and Feature

Extraction

Frame based

Feature processing and

decision making

F1

F2

F3

…..


• XYLON—Face Analytics

• Real-time 3D head pose

• Facial expressions

• Eye closure

• Gaze direction

• FPGA-based programmable SoC implementation

Balancing the Load—A Real Example

face analytics

image

processing

hw

sw legend:

resizing

/csc

cross

correlation display

OSD

~ 54X

data rate

reduction

http://www.logicbricks.com/


Challenge :

Building the Infrastructure


Interconnect Bandwidth Latency Use Cases

General Purpose Ports (GP) Low configuration

High Performance Ports (HP) High High high rate data paths

Accelerator Coherency Port (ACP) High Low accelerators

Hardware Infrastructure

• Two worlds collide !

• Software, Hardware

• HWSW Interface

• Shared memory

• OCM, Cache, DDR

• DMA, Interrupts

• Interconnect

• Each change in architecture requires new hardware infrastructure


DDR

GP

ACP HP

Processor

sub-system FPGA

sub-system

Cache OCM


Hardware Infrastructure—A Real Example

face analytics

image

processing

hw

sw legend:

resizing

/csc

cross

correlation display

OSD

FPGA sub-system

Processor sub-system

L2 cache

DDR memory

ROI/

template video

ACP ACP

graphics

HP


• Software developers expect device drivers for custom hardware

• Often overlooked by hardware designers

Software Infrastructure—Device Drivers


Challenge :

Choosing the Right IP


• PYTHON—very scalable image sensor family

Use Case—PYTHON Image Sensor

Sensor Image Size Frame Rate Bandwidth Pixel Rate

PYTHON-300 640 x 480 840 fps 2.88 Gbps 288 MHz



PYTHON-25K 5120 x 5120 80 fps 19.84 Gbps 1.94 GHz

PYTHON

300

Serial

LVDS

Receiver

SYNC

DATA

Sync

Decoder

&

CRC

Checker

DATA

(40bits)

SYNC

(10 bits)

DATA

(40bits)

SYNC

4:1 PIXEL

(10bits)

SYNC

Serial LVDS

720Mbps 72MHz 72MHz 288MHz


• IP cores available from manufacturers

• Generic implementations

• Support single channel of pixels

• When used in the lower cost FPGA-based programmable SoCs

• Typical limit of 150MHz for single stream of pixels

• Corresponds to HD 1080P60

• When used in larger FPGA-based programmable SoCs

• Typical limit of 300MHz for single stream of pixels

• Corresponds to UHD 4K2K @ 60 fps

The Challenge—Single Channel IP Cores

300

150


• The Solution

• Parallel execution paths

• 4K2K designs are implemented with parallel data paths

• 4K2K @ 60fps => 600 MHz pixel rate (4 * 150MHz / 2 * 300MHz)

• Solution providers

• XYLON—Video IP cores supporting 1/2/4 pixels per clock

• Auviz Systems—IP cores supporting 1/8/16 pixels per clock

The Solution—Parallel Channel IP Cores


Solutions with Tools


• SDSoC

• Software based environment

• System optimizing compiler

• Software

• Hardware

(including device drivers)

• Connectivity

• HLS (High-Level Synthesis)

• C/C++ to hardware synthesizer

• pipelined / parallel accelerators

• Platforms

• Pre-built hardware infrastructure

Xilinx Software Defined Tools (SDSoC)


Conclusion


• Plan time to build infrastructure

• Hardware (interfaces, accelerators, interconnect)

• Software (device driver development)

• Choose the right IP

• Calculate your required pixel rate

• Support for parallel execution paths

• Device driver Support

• Consider using emerging software tools to accelerate productivity

• Pre-built infrastructure

• Automatically generate accelerator + interconnect + device driver

• Allows architecture exploration

Lessons Learned


• MicroZed Embedded Vision Kit

• Xilinx Zynq-7000 SoC

• ON Semiconductor PYTHON-1300-C camera

• Toshiba TCM3232PB industrial camera

• Demonstration of XYLON IP solutions

• Face Analytics

Summit Technology Showcase—Avnet Table

http://www.logicbricks.com/


• Avnet Development Kits

• MicroZed Embedded Vision Kit

• http://microzed.org/product/microzed-embedded-vision-kits

• Camera Modules

• http://microzed.org/product/python-1300-c-camera-module

• http://microzed.org/product/toshiba-industrial-1080p60-

camera-module

• XYLON IP Solutions

• Face Analytics

• http://www.logicbricks.com/Products/logiFDT.aspx

• Image Signal Processing Pipeline (ISP)

• http://www.logicbricks.com/Products/logiISP.aspx

References

http://microzed.org/product/microzed-embedded-vision-kits








http://microzed.org/product/python-1300-c-camera-module










http://microzed.org/product/toshiba-industrial-1080p60-camera-module










http://www.logicbricks.com/Products/logiFDT.aspx

http://www.logicbricks.com/Products/logiFDT.aspx

http://www.logicbricks.com/Products/logiISP.aspx

