Industry Pulse: Trends in Functional Verification

Industry Pulse:Trends in Functional Verification

Chief Scientist VerificationDesign Verification Technology

MemoCODE 2013

Harry Foster

www.mentor.com© 2013 Mentor Graphics Corp. Company Confidential

Extrapolating From Current Conditions Disregards Future Innovation

HF, MemoCODE, 20132

“In 1910, in the early history telephony, a Bell telephone statistician projected a massive ramp-up in switchboard operator jobs as telephone use grew, until “every woman in America” would be required.”

Source: Future Savvy: Identifying trends to Make Better Decisions, Manage Uncertainty, and Profit From Change Adam Gordon, 2008


Functional Verification Market According to EDAC

2010 20120

200

400

600

800

1000

700

965

Mill

ions

($)

HF, MemoCODE, 20133

38% Growth Between 2010 2012

EDAC: Market Statistics Service 2007 Annual Summary Report


Functional Verification Market According to EDAC

2010 20120

100

200

300

400 380430

190

365

130170

Simulation 13% GrowthEmulation 94% GrowthFormal 31% Growth

Mill

ions

($)

HF, MemoCODE, 20134

EDAC: Market Statistics Service 2007 Annual Summary Report

www.mentor.com© 2013 Mentor Graphics Corp. Company ConfidentialHF, MemoCODE, 20135

Conducted by Wilson Research Group— Commissioned by Mentor Graphics— Format followed 2002, 2004 Collett studies for trend

analysis, as well as the 2007 FarWest Research Study Worldwide study

— Overall confidence of 95% plus/minus 4.05% This was a blind study!

— To eliminate any bias in the results This was a balanced study!

— No single vendor dominated responsesWilson Research Group

2012 Wilson Research Group Functional Verification Study


Who Participated In The Survey Participant’s market segment

HF, MemoCODE, 20136

Wilson Research Group and Mentor Graphics, 2012 Functional Verification Study, Used with permission

ASIC Vendor

Fabless IC Vendor

IC Manu-facturer

Systems Company

Design Services Company

IP Vendor Company

Other0%

10%

20%

30%

40%

50%

60%

Non-FPGA 2012FPGA 2012

Stud

y Pa

rtic

ipan

ts


Who Participated In The Survey Participant’s job title

HF, MemoCODE, 20137


HARDWARE DES

IGNER

VERIFIC

ATION EN

GI...

SYST

EM ARCHITE

CT

TEAM LE

ADER

SOFTW

ARE ENGINEE

R

MANAGER

CAD SUPP

ORT ENGI...

OTHER

0%

10%

20%

30%

40%

50%

60%

Non-FPGA 2012FPGA 2012

Stud

y Pa

rtic

ipan

ts

www.mentor.com© 2013 Mentor Graphics Corp. Company Confidential8

Overview Beyond Theory Beyond Standards Beyond the Status Quo

HF, MemoCODE, 2013


BEYOND THEORY

Beyond Theory in Terms of Rising Complexity

HF, MemoCODE, 20139


Difference Between Theory and Practice

HF, MemoCODE, 2013

In theory there is no difference between theory and practice, but in

practice there is.



HF, MemoCODE, 2013

Theory: Everything is clear, but nothing works.



HF, MemoCODE, 2013

Practice: Everything works, but nothing is clear.



HF, MemoCODE, 2013

The problem is sometimes theory meets practice:

Nothing works and nothing is clear.

http://www.google.com/url?sa=i&rct=j&q=boat%20flipped&source=images&cd=&cad=rja&docid=jSiimmTOKzVhSM&tbnid=40AAk08FNA-Z2M:&ved=0CAUQjRw&url=http://jacksonkayak.com/blog/2012/09/15/deep-water-re-entry-in-a-coosa/&ei=feUUUoeALMKZiQLG9YHYCg&bvm=bv.51156542,d.cGE&psig=AFQjCNHEoS-ugzmRBc6fMGwTV1YF7MIM0g&ust=1377187570643316


Beyond Theory in Terms of Rising Complexity

HF, MemoCODE, 2013

What does this really mean? What makes things complex? How do we measure complexity?


What Makes Something Complex?

HF, MemoCODE, 2013

System consisting of many interconnected parts— Examining the individual parts tells you nothing about the

system Complex does not necessarily mean complicated


Designs are Getting More Complex

HF, MemoCODE, 2013


0.25µ

or LA

RGER0.1

8µ0.1

5µ0.1

3µ0.0

9µ

0.065

µ

0.045

µ

0.032

µ or sm

a...

0%

5%

10%

15%

20%

25%

30%

35%

40%

2007

Stud

y Pa

rtic

ipan

ts

2007: Mean 90nm2010: Mean 65nm2012: Mean 45nm

Process Geometry



HF, MemoCODE, 2013

100K

or LE

SS

>100K

- 50

0K

>500K

- 1M

>1M -

5M

>5M -

10M

>10M -

20M

>20M-40

M

>40M -

60M

>60M

0%

5%

10%

15%

20%

25%

30%

2002200720102012

Non

-FPG

A St

udy

Part

icip

ants

About a 1/3rd of designs below 5M gatesAbout a 1/3rd of design between 5M - 20M gatesAbout a 1/3rd of designs great than 20M gates


Number of gates of logic and datapath, excluding memories



HF, MemoCODE, 2013


2002 2007 2010 20120

2

4

6

8

10

12

0.4

2.7

6.1

11.1

Non

-FPG

A M

ean

Des

ign

Size

G

ates

(M

)

Mean number of gates of logic and datapath, excluding memories trends



NONE 1 2 3 4 5 or MORE0%

10%

20%

30%

40%

50%

60%

21% 22%

28%

6%

10% 13%

2004200720102012

Number of Embedded Processors for Non-FPGA Designs

Non

-FPG

A St

udy

Part

icip

ants

Source: Wilson Research Group and Mentor Graphics, 2012 Functional Verification Study

HF, MemoCODE, 2013

79% of designs contain one or more embedded processors



2004 2007 2010 20120

1

2

3

1.06

1.46

1.962.25

Non

-FPG

A M

ean

num

ber

of E

mbe

dded

Pro

cess

ors


HF, MemoCODE, 2013

Mean number of embedded processors continues to rise


FPGAs are Getting Complex Too!

HF, MemoCODE, 2013


NONE 1 2 3 4 5 or MORE0%

10%

20%

30%

40%44%

38%

11%

1%4%

2%

21% 22%

28%

6%10% 13%

FPGANon-FPGA

Number of embedded processors

Surv

ey P

arti

cipa

nts

56% of FPGAs contain one or more embedded processors


How do we measure complexity?

HF, MemoCODE, 2013

Computational complexity theory used in computer science

There are no generally accepted metrics!

O(1)O(logn)O(n)

O(nlogn)

O(n^2)O(2^n)O(n!)

Ope

rati

ons

Elements


Is bug density a good proxy?

Single, sequential data streams— Floating point unit— Graphics shading unit— DSP convolution unit — MPEG decode— . . .

Multiple, concurrent data streams— Cross bar— Bus traffic controller— DMA controller— Standard I/F (e.g., PCIe)— . . .

Channel

CompressedAudio

Data Link LayerTX

RX

PHY

Sequential data streams1x number of bugs

Concurrent data streams5x number of bugs

Encoder Decoder

HF, MemoCODE, 2013

-Ted Scardamalia, internal IBM study


Packet-Based Design

Transaction Layer Packet

Reformater

Data LinkLayer PacketReformater

Retry Buffer Arbiter

Tx

Rx

FromFabric To

PHY

From Rx Channel

Concurrency is Complicated to Verify

HF, MemoCODE, 2013


Maybe effort is a good proxy?

HF, MemoCODE, 2013

Affor

dabl

eEx

pens

ive

Proh

ibit

ive

Cost

/Eff

ort

Complexity

Critical Threshold


Verification Consumes Majority of Project Time

1%-20% 21%-30% 31%-40% 41%-50% 51%-60% 61%-70% 71%-80% >80%0%

5%

10%

15%

20%

25%

Total Project Time Spent in Verification

200720102012

Time (Percent)

Non

-FPG

A St

udy

Part

icip

ants 2007: Mean 49%

2010: Mean 56%2012: Mean 56%

Wilson Research Group and Mentor Graphics, 2012 Functional Verification Study

HF, MemoCODE, 2013


More and More Verification Engineers Mean peak number of design vs. verification engineers

2007 2010 2012

7.8 8.1 8.5

4.87.6 8.4

~ 1-to-1 ratio of peak design

and verification engineers58% 11%

4% 5%

Verification Engineers

Design Engineers


HF, MemoCODE, 2013


Where Verification Engineers Spend Their Time

HF, MemoCODE, 2013


16%

22%23%

36%4% Test Planning

Testbench DevelopmentCreating and Running TestDebugOther

More time spent in debug than any other task!


54%

2007

46%

2007

47%

2012

53%

2012

Doing Design Doing Verification

Design Engineer Project Time2007 - 2012

15%

Incr

ease

HF, MemoCODE, 2013

Designers Doing More and More Verification



2007 2012 2017 2022 2027 2032 20370

20

40

60

80

100

At this rate…In 25 years, ALL of a designer’s

time will be devoted to

verification

Project Time 2007 - 2037

DesignVerification

Time Design Engineers Spends Doing:

Tim

e (P

erce

nt)

HF, MemoCODE, 2013

Time Designers Spends in Design vs. Verification


Design Reuse Trends

HF, MemoCODE, 201331

0%

10%

20%

30%

40%41%

33%

13% 13%

28%

35%

22%

15%

20072012

Design Composition

Non

-FPG

A St

udy

Part

icip

ants

Source: Wilson Research Group and Mentor Graphics.


Verification Reuse

HF, MemoCODE, 2013


0%

10%

20%

30%

40%

50%50%

41%

8%

39%44%

17%

20072012

Testbench Composition

Non

-FPG

A St

udy

Part

icip

ants

Mean testbench composition trends


With All This Effort, How are We Doing?

HF, MemoCODE, 2013


Project’s Schedule Completion Trends

More than 10%

EARLY

10% EARLY

ON-SCHED

ULE

10% BEHIND SCHED

ULE

20% 30% 40% 50% >50% BEHIND SCHED

ULE

0%

5%

10%

15%

20%

25%

30%

35%

200720102012

Stud

y Pa

rtic

ipan

ts

2007: 67% behind schedule2010: 66% behind schedule2012: 67% behind schedule

Behind ScheduleAhead of schedule


HF, MemoCODE, 2013


FPGA vs. Non-FPGA Completion Trends

HF, MemoCODE, 2013


More than 10%

EARLY

10% EARLY

ON-SCHED

ULE

10% BEHIND SCHED

ULE

20% 30% 40% 50% >50% BEHIND SCHED

ULE

0%

5%

10%

15%

20%

25%

30%

35%

Non-FPGAFPGA

Non-FPGA vs. FPGA completion compared to project's original schedule

Stud

y Pa

rtic

ipan

ts

Non-FPGA: 67% behind schedule FPGA: 67% behind schedule


Required Number of Spins

HF, MemoCODE, 2013


1 FIRST

SILICON SUCCESS

2 3 4 5 6 7 SPINS or

MORE

0%

10%

20%

30%

40%

50%

2004200720102012

Number of Required Spins

Non

-FPG

A St

udy

Part

icip

ants


Types of Flaws

HF, MemoCODE, 2013


LOGIC O

R FUNCTIO

NAL

CLOCKIN

G

TUNIN

G ANALOG C

IRCUIT

CROSSTA

LK

POWER C

ONSUMPTIO

N

MIXED-SI

GNAL INTE

RFACE

YIELD

OR REL

IABILITY

TIMIN

G – PATH

TOO SL

OW

FIRMWARE

TIMIN

G – PATH

TOO FA

ST

IR DROPS

OTHER

0%

10%

20%

30%

40%

50%

60%

2004200720102012

Trends in Types of Flaws Resulting in Respins

Non

-FPG

A St

udy

Part

icip

ants

* Multiple answers possible


Root Cause of Functional Flaws

HF, MemoCODE, 2013


0%

10%

20%

30%

40%

50%

60%

70%

80%

90%

20022004200720102012

Root Cause of Functional Flaws

Non

-FPG

A S

tudy

Par

tici

pant

s


www.mentor.com© 2013 Mentor Graphics Corp. Company Confidential© 2010 Mentor Graphics Corp. Company Confidentialwww.mentor.comHF, MemoCODE, 201339

Cost of Find Functional Flaws

Silicon Debug, Doug Josephson and Bob Gottlieb, (Paul Ryan)D. Gizopoulos (ed.), Advances in Electronic Testing: Challenges and Methodologies, Springer, 2006

Relative Cost Of Finding Bugs

$1$10

$100$1,000

$10,000$100,000

$1,000,000$10,000,000

Design Cycle

Cos

t To

Fix


BEYOND STANDARDS

Beyond arguing over who won the standards war



Standardization of Languages

VHDL Verilog Synopsys Vera

System C SystemVer-ilog

Specman e C/C++ OTHER Testbench

0%

20%

40%

60%

80%

200720102012

Languages Used for Verification (testbenches)

Non

-FPG

A St

udy

Part

icip

ants

SystemVerilog grew 8.3% between 2010 and 2012


HF, MemoCODE, 2013


SystemVerilog Adoption by Design Size

< 5M 5 - 20M > 20M0%

20%

40%

60%

80%

100%

59%

71%

89%

SystemVerilog Adoption by Design Size (Gate Count Excluding Memories)

Non

-FPG

A St

udy

Part

icip

ants


HF, MemoCODE, 2013


Standardization in Base Class Libraries

0%

10%

20%

30%

40%

20102012

Testbench Methodologies and Base-Class Libraries

Non

-FPG

A St

udy

Part

icip

ants

486% UVM growth between 2010 and 201246% UVM projected growth in the next twelve months


HF, MemoCODE, 2013


Standardization of the SoC Verification Process

Ten years ago, IC/ASIC verification was partitioned into two main steps:

HF, MemoCODE, 2013

Integration

Verification

Block-Level

Verification

Block Full Chip


Standardization of the SoC Verification Process Emerging from ad hoc to systematic processes

HF, MemoCODE, 2013

Block-Level

Verification

Interconnect

Verification

Integration

Verification

Application / SW

Verification

IP Subsystem SoC System


BEYOND THE STATUS QUO

Beyond surviving by maintaining the status quo



The Verification Paradox A good verification process lets you get the most

out of best-in-class verification tools

Start Tools Ad Hoc

Processes6-9%Cost

Increase

Start Process Tools

20-30%Cost

Savings

Source: Cisco Momentum Research Group

HF, MemoCODE, 2013



HF, MemoCODE, 2013

Block-Level

Verification

Interconnect

Verification

Integration

Verification

Application / SW

Verification


Use of Advanced Verification Techniques

HF, MemoCODE, 2013

Constrained-Random Simu-lation

Functional coverage

Assertions

Code coverage

0% 10% 20% 30% 40% 50% 60% 70% 80%

62%

71%

68%

70%

41%

40%

37%

48%

20072012

Non-FPGA Study Participants



Directed vs Constrained-Random Simulation

HF, MemoCODE, 2013


2010 20120%

20%

40%

60%

80%

100%

44% 51%

56% 49%

Directed

Constrined-Random

Mean Directed vs. Constrained-Random Simulation Trends

Non

-FPG

A St

udy

Part

icip

ants

16%

Increase


Larger Designs Use More Formal

< 5M 5 - 20M > 20M0%

10%

20%

30%

40%

20%

26%

41%

Formal Property Checking Adoption by Design Size (Gate Count Excluding Memories)

Non

-FPG

A St

udy

Part

icip

ants


HF, MemoCODE, 2013


The Evolution of Formal Technology

FormalPropertyChecking

FullyAutomatic

Formal

AutomatedApplications

LowEffort

HighEffort

FormalExperts Everyone

1990s Today2000s

HF, MemoCODE, 2013



HF, MemoCODE, 2013

A57A57A57A57

Block-Level

Verification

Interconnect

Verification

Integration

Verification

Application / SW

Verification



HF, MemoCODE, 2013

Block-Level

Verification

Interconnect

Verification

Integration

Verification

Application / SW

Verification

— IP Blocks connectivity— Access all memories— Access all registers, such as control— Configurations work— Functional scenarios and use-cases— Verify multiple clock domain crossings


Mean Number of Clock Domains by Design Size

< 5M 5 - 20M > 20M0

2

4

6

8

10

12

14

5.21

9.36000000000001

11.48

Mean Number of Clock Domains by Design Size (Gates Excluding Memories)

Mea

n N

umbe

r of

Clo

ck D

omai

ns


HF, MemoCODE, 2013



HF, MemoCODE, 2013

Block-Level

Verification

Interconnect

Verification

Integration

Verification

Application / SW

Verification

— Boot OS— Load System Drivers— Run Application SW


SoC Design & Verification Involves Lots of SW


2000 2001 2002 2003 2004 2005 2006 2007 2008 2009 2010 2011 2012$0

$20

$40

$60

$80

$100

Total SW Engineering Costs + ESA Tool CostsTotal HW Engineering Costs + EDA Tool Costs

Source: ITRS 2010, Impact of Design Technology on SoC Consumer Portable Implementation Cost

($ M

)

It’s the software, stupid! -Gary Smith


As Design Sizes Increase…Emulation Up, FPGA Prototyping Down in 2012

HW Acceleration/Emulation FPGA Prototyping0%

10%

20%

30%

40%

50%

60%

18%

32%

50%

45%

57%

28%

< 5M 5 - 20M> 20M

Adoption by design size for those doing HW acceleration/emulation and FPGA Prototyping

Non

-FPG

A St

udy

Part

icip

ants


HF, MemoCODE, 2013


Integrated Simulation/Emulation/Software Verification Environments Emerge

HF, MemoCODE, 2013

Virtual Prototype

Processor Debug

JTAG

SW Debug

OVM/UVMSystemC/C++

Monitors

TestbenchAcceleration

Assertions & Checkers

Physical DevicesProtocol Solutions

System Level

Virtual Devices & Transactors

Protocol Solutions

Design under Test


Coverage and Power

HF, MemoCODE, 2013

Block-Level

Verification

Interconnect

Verification

Integration

Verification

Application / SW

Verification

Block Subsystem SoC System

Across all aspects of verification


Signoff criteria trends

The Rising Importance of Coverage

HF, MemoCODE, 2013



OTHER

WHEN THE PROJECT PLAN SAYS SIGN-OFF, REGARDLESS OF STATUS

WHEN WE CAN NO-LONGER THINK OF ANY MORE TESTS TO WRITE

WHEN THE RATE OF BUGS FOUND PER WEEK DROPS BELOW A SPECIFIED GOAL

WHEN THE EMULATED OR PROTOTYPED DESIGN IS WORKING IN-SITU

WHEN COVERAGE SAYS WE HAVE ACHIEVED A TARGET

WHEN THE PROJECT PLAN SAYS SIGN-OFF, ASSUMING VERIFICATION LOOKS OK

WHEN ALL TESTS DOCUMENTED IN THE VERIFICATION PLAN ARE COMPLETE AND PASS

0% 10% 20% 30% 40% 50%

20072012

Non-FPGA Study Participants


Unified Coverage Interoperability Standard

HF, MemoCODE, 2013

New Accellera UCIS Standard Announced at DAC 2012

UCIS API

Coverage Database

Simulation

FormalEmulatio

n

Analysis Testplan


Trends in Power Management Verification

Power domain power down/power up

Transitions between system

power states

Interactions between power domains

Application-level power management

0% 20% 40% 60% 80%

Aspects of Non-FPGA Power Managed Design That Are Verified

67% of the industry actively manages power

Source: Wilson Research Group and Mentor Graphics, 2012 Functional Verification Study, Used with permission

HF, MemoCODE, 2013


Power Trends

HF - January 2013 Master Set, WRG & MG Study Results


0%

5%

10%

15%

20%

25%

30%

35%

Percentage of total simulations that were power-aware

Non

-FPG

A St

udy

Part

icip

ants

About 10% of power managed designsperform no power-aware simulation!


Power Trends

HF - January 2013 Master Set, WRG & MG Study Results


0%

5%

10%

15%

20%

25%

30%

35%

Percentate of Verification Resources Focused on Power Management

Non

-FPG

A St

udy

Part

icip

ants

Median Verification Resources: 20% -29%


Power Trends


CPF 1.0 CPF 1.1 CPF 2.0 UPF 1.0 (Accellera)

UPF 2.0 (IEEE 1801-

2009)

Internal/Proprietary

0%

5%

10%

15%

20%

25%

6% 6%

24%

15%

27%

22%

Non

-FPG

A St

udy

Part

icip

ants


Notation used to describe power intent

Source: Wilson Research Group and Mentor Graphics, 2012 Functional Verification Study, Used with permission


Beyond the Status Quo

HF, MemoCODE, 2013

Block-Level

Verification

Interconnect

Verification

Integration

Verification

Application / SW

Verification

IP Subsystem SoC System



BACK TO THE FUTURE

The Productivity Gap



Design Productivity Gap

Silicon Density

Design Productivity

Capa

city

Time

Source: SEMATECH

HF, MemoCODE, 2013


Verification Productivity

Verification Productivity Gap

Source: SEMATECH

Silicon Density

Design Productivity

Capa

city

Time

HF, MemoCODE, 2013


Closing The Verification Gap


Reuse Abstraction

Acceleration Methodology

?


72

100s of lines of TLM

Billions of Transistors

100s of Millions of Gates

Millions of Lines of RTL

#include "systemc.h" SC_MODULE(adder) // module (class) declaration { sc_in<int> a, b; // ports sc_out<int> sum; void do_add() // process { sum = a + b; } SC_CTOR(adder) // constructor { SC_METHOD(do_add); // register do_add to kernel sensitive << a << b; // sensitivity list of do_add } };

Managing Complexity



73

RTLclk

Cycle Accurateclk

Timed TLM

Untimed TLM

C/C++Simulation

1x (7 days)

10,000x (1 min)

10x

100x

1,000x

Protocol

Protocol

Transaction

Functionarguments

Transaction

Productivity Gains Through Abstraction



What’s the advantage of SystemC compared with RTL?

There are two different aspects – quality and time schedule. Today, a full chip on SystemC will run around 10 MHz, and you will never reach that speed using RTL or a lower-level abstraction. It’s similar to a prototype speedup. Previously with RTL designs, our bug rate was in the range of 10 to 50 bugs per square millimeter. Now we are at less than one bug per millimeter squared. So we have both quality and speed of development.

Source: EETimes, 2007, Laurent Ducousso, who manages intellectual-property (IP) verification for STMicroelectronics’ Home Entertainment Division

Bug Prevention vs. Bug Hunting



Summary Beyond theory in terms of rising complexity Beyond arguing over who won the standards

wars Beyond surviving by maintaining the status

quo

HF, MemoCODE, 2013


Documents

Industry Pulse: Trends in Functional Verification