Functional Validation for Dependability

Functional Validation for Dependability

University of VeronaDepartment of Computer Science

Giuseppe Di Guglielmo

Agenda

• ESD Research Group• Functional Validation for Dependability

– A Methodology for Abstracting RTL Designs to TL Descriptions

– Functional Qualification of TLM verification– The Role of Parallel Simulation on Functional

Verification– HW/SW co-simulation for co-verification

2CREDES Kick-Off MeetingJune 5th, 2009

ESD RESEARCH GROUP

University of VeronaComputer Science Department


Verona


5

CS Department

• 8 research areas• 46 lecturers• 60 research assist.• 6 undergrad-

graduate degrees • 2 master degrees• > 1000 students

Best CS Dept. in Italy for teaching and research quality (CENSIS 2006-08)


6

ESD: research group• Permanent staff:

– prof. Franco Fummi– prof. Tiziano Villa– prof. Nicola Bombieri– prof. Damiano Carra– prof. Graziano Pravadelli– prof. Davide Quaglia

• 10 research assistants• 4 Post-doc• 2 PhD students• xx graduate students

22 People


7

ESD: research areas (I) • Embedded system

verification– Static verification– Dynamic verification– Semi-formal verification– Hybrid and real-time

systems

• Basic CAD algorithms– Synthesis of sequential and

combinational systems– Discrete event systems– Physical design

• Networked embedded systems– System/Network co-

simulation– System/Network co-design– QoS-enabled design– Sensor networks and M2M

systems


ESD: research areas (II) • Embedded SW

generation– TLM-RTL synthesis and

abstraction– RTL-to-SW abstraction– TLM transactor generation– Device-driver generation– Embedded SW for

multicore systems– Middleware-based design

• Networking systems– Protocol Design and

Architectures– Performance Evaluation– Network Measurement and

characterization– Overlay Networks

8

more than 250 papers


9

ESD: teaching• Some undergraduate courses in:

– Computer architecture – Operating systems, Networking– Real-time systems

• Graduate course of study with an embedded systems curriculum:– Electronic design automation– Embedded systems– Networked embedded systems– Multimedia embedded systems– Control embedded systems– Embedded systems architecture


ESD: projects• 2 European projects in FP6

– ANGEL (mobile gateway for sensors network)

– VERTIGO (HW formal verification)

• 2 European projects in FP7– COCONUT (embedded systems design and verification)

• best evaluation of the overall embedded systems track

– C4C (control for coordination of distributed systems)• 4 joint projects between University of Verona and private companies• 2 national funded projects• 10 research contracts with enterprises

2.5M€ projects funding in the last 5 years10CREDES Kick-Off MeetingJune 5th, 2009

11

ESD spin-off: EDALab• Focus:

– Networked embedded systems• Main products:

– EDA tools• HW/SW/Network co-simulation;• Static/dynamic validation;• RTL-TLM/C++ abstraction;• Sensor network design and management

• Etc;– Middleware

• Mobile terminals;• Sensor networks;• Etc.

– IP-cores• Factory automation;• Multimedia;• Etc; 10 year expertise in the development of hw/sw

models and tools


FUNCTIONAL VALIDATION FOR DEPENDABILITY

CREDESCentre of Research Excellence in Dependable Embedded Systems


State of the art (I)• Dependable computing first appears in the 1830’s

in the context of Babbage’s Calculating Engine.• Theories of using redundancy and masking to

build reliable logic structures, from less reliablecomponents, was proposed in ’60.

• Then, masking was integratewith the practical techniques of error detection, fault diagnosis,and recovery into the concept offault-tolerant systems.


State of the art (II)• The formation of the

– IEEE-CS TC on Fault-Tolerant Computing in 1970;– IFIP WG 10.4 Dependable Computing and Fault Tolerance

in 1980.accelerated the emergence of a consistent set of concepts and terminology.

• In 1992 intentional faults (malicious logic, intrusions) were listed along with accidental faults (physical, design, or interaction faults).

• Recently, the notion of dependability has been extended to incorporate, not only fault tolerance, but also safety.


State of the art (III)• Major event in measuring dependability was the

introduction of coverage concept.• Mutation analysis and mutation testing gained

consensus as important techniques for coverage estimation.

• What miss are– a widely accepted way of modeling faults (mutants) in

relation to soft and physical defects;– a methodology supported by tools to address the

trade-off among complexity, accuracy and speed;– a trusted methodology proving that solutions

developed for dependability are correct.


Motivations and Goals (I)• A great concern about having pervasive

electronics solutions into digital systems is related to their reliability– a key factor for their acceptance!

• The goal is to achieve the design of reliable circuits operating in harsh environmental conditions.

• The objectives includes the development of models, tools and methodologies for the validation of designs resistant to reliability detractors.


Motivations and Goals (II)

• Main tasks– accurate modeling of reliability detractors;– abstraction of their model at the highest level

• defects � faults � mutation operators

– concurrent evaluation of the reliability detractors, to enable realistic fault-injection campaigns;

– development of SoC validation environment at the highest abstraction level

• HW/SW abstraction & HW/SW co-simulation.


Why we want do this• The development of safe and dependable systems is

an actual challenge which is emphasized further by the high complexity.

• Existing techniques faces the reliability problems using hardware redundancy and guard-banding of key parameters– This approach may give drawbacks on performances

• Increased costs because of the employment of extra resources;• Preventive constraints on speed performances to anticipate

aging effects.• There is lack of tools and methods for the assessment

of reliability at the design level.


Today vs Tomorrow Design Flow

RT Level IPs &Custom logic

SynthesisGL fault injection

& simulation

HW acceleration

Validation of Safety/Dependable

Features at gate level

OK?

no

TL Abstraction OK?

Today

Tomorrow

Synthesis

TL Fault Injection

& simulation@ SoC level

Validation of Safety/Dependable

Features at TL

ValidationOf Safety/Dependable

Features at gate level

SoC Manufacturing

no


Tomorrow design flow

• There are three main advantages– allow SoC designers to face the evaluation of

the safety and dependable features early in the design flow;

– limit the need to perform validation of safety features of the SOC at gate-level and/or using HW accelerator solutions;

– concurrently evaluate a wide spectrum of possible reliability detractors (fault models).


UniVR Area of Contributions

SoCor

ApplicationSystem

Synthesis and

Abstraction

Fault Injection Solutions

HW/SW Co-Simulation

TLM and RTL

modeling

21

Faultsimulation

CREDES Kick-Off MeetingJune 5th, 2009

A METHODOLOGY FOR ABSTRACTING RTL DESIGNS TO TL DESCRIPTIONS


Transaction Level Modeling (TLM)• TL permits to start SW development very early in the

design flow

Functionalspecif.

Architect.specif Design Fab. Breadboard SW

develop.Sys

integrationSys

validation

Classical design flow

SWdevelop.

Sysintegration

Sysvalidation

GAIN!


TLM: Advantages• Implementation details are abstracted while

preserving the behavioral aspects of the system– this allows a faster simulation (up to 1,000x) than at

RTL.• System level design exploration and verification

are simplified– IP components and buses can be modified and

replaced in an easier way than at RTL.• An early platform for SW development can be

quickly developed.


TLM: Modeling Comparison• More emphasis on the data transfer functionality

– less on their implementation details at the early design stage.

clkreqgnt

data_0

data_31

addr_0

addr_31

……

RTL

write (data, addr);

TLM


TLM: TBV• Transactor-based Verification (TBV) has been

introduced to allow reuse in mixed TL-RTL designs.

M1

M9

M7

Existent RTL IPs

IP-c

ores

reus

e

Testbench& assertion/

propertyreuse

TL

M1

M3

M2 TL Testbenches& Assertions

T

T

T

RTL Testbenches& PropertiesT

RTL

M1

M3

M2

TL Testbenches& Assertions

RTL Testbenches& Properties

T


TLM: Transactor

RTLmodule

Control inputs

Data inputs

Control outputs

Data outputs

clk

write(addr,data)

…

read(addr,&res)

…

(write_status)

(read_status)

TLmodule Transactor

RTLsignals

clk

RTLsignals


TLM: Design Languages

Algo

RTL

Synthesis

TLMTLM

Simulink,

C/C++SystemC

VHDL,

Verilog

System

Verilog

--

--

--

--

SystemC OSCITLM library

Abs

tract

ion


Motivations and Goal (I)• The standard IP reuse

methodology is based on RTL IP reuse via transactor.

• Main problems– transactors coding is error

prone and time consuming;– global simulation/validation

time is bounded by RTL simulation time.

TL iTL i

BUS

MasterMaster

Transactions

Transactions

RTL Slave•Clock and pin accurate•Communication protocol between the transactor and the RTL slave well-defined


clk

TransactorTransactorclk


Motivations and Goal (II) • Main goal

– definition of an automatic abstraction methodology of RTL IPs towards TLM;

– compliant to the SystemC OSCI TLM library. • Advantages

– no error prone manual design phases (e.g., transactors coding);

– real TLM simulation/validation time;– distribution of simulation models of IP-cores without

discovering IP.


RTL SlaveClock and pin accurate


clk



TL iTL i

BUS

MasterMaster

Transactions

TransactorTransactor

Transactions

clk

clk

IPreuse

Sta

ndar

d IP

reus

e m

etho

dolo

gy

Pro

pose

d IP

abs

tract

ion

met

hodo

logy

TL 1 Slave•Clock accurate•Abstraction of data•Identification of Input/Elaboration/Output phases


TL 1TL 1MasterMaster

……

……

clk

clk

1

BUSclk

Methodology Overview (I)




clk

Pro

pose

d IP

abs

tract

ion

met

hodo

logy




……

……

clk

clk

1

BUSclk

Methodology Overview (I)

• Design is still clock-accurate.

• Interface pins are abstracted away.

• Communication protocol is unchanged.


TL 2 Slave•Abstraction of clock•Compaction of Input/Elaboration/Output phases•Abstraction of the communication protocol which is implemented in the driver

TL 2 Slave•Abstraction of clock•Compaction of Input/Elaboration/Output phases•Abstraction of the communication protocol which is implemented in the driver

TL 2TL 2

BUS

MasterMaster

DriverDriver

2

Methodology Overview (II)

• Clock is removed.• Communication protocol is

implemented by means of a FIFO approach.

• States composing an Input/Elaboration/Output sub-phases are collapsed into macro states.

• A driver is automatically generated to allow the master to control the slave execution.


Methodology Overview (III) • A TL3 module is a pure

function call-based interface with no communication events– FIFO channels are not used

anymore;– point-to-point communication is

established between master and slave.

• A slave is a passive module that implements only two functions– write() which is called by the

master to pass input data and start the slave computation;

– read() which is called by the master after a write() to retrieve the computation result.

TL 3 Slave•Removal of the bus•Abstraction of the driver

TL 3 Slave•Removal of the bus•Abstraction of the driver


DriverDriver

3


Experimental Results: Simulation Times

1

10

100

1000

Sim

ulat

ion

Tim

e (s

.)

RTL TL2 TL3 Man_TL3

37,910,511,19,34,6Man_TL3

38,911,813,214,95,1TL3

40,253,173,989,137,5TL2

68,2629,0647,2754,8439,1RTL

B01ADPCMDistDivRoot

37,910,511,19,34,6Man_TL3

38,911,813,214,95,1TL3

40,253,173,989,137,5TL2

68,2629,0647,2754,8439,1RTL

B01ADPCMDistDivRoot


Conclusions

• A methodology to automatically abstract RTL designs towards TLM levels.

• Alternative to TBV– NO slow down the overall simulation;– prevent error-prone manual abstractions;– prevent errors related to the design of

transactors;– allow the distribution of simulation models of

IP-cores without discovering IP.


FUNCTIONAL QUALIFICATION OF TLM VERIFICATION


Mutation Analysis

• Perturbation-based technique for software unit testing.

• M is known as a mutant of DUV D– M is formed by modifying a single statement

of D according to some predefined rule.

• Coupling effect – a test pattern that detects simple faults in a

program detects most complex faults.


Mutation Testing Process• Strong Mutation Analysis.• Weak Mutation Analysis.• Mutation score

– MS(D,T)= K / (M-E)

HDL Mutation operator setsCLR Constant Limit replacement. CSR Constant for Scalar Variable ReplacementSUR Signed/Unsigned ReplacementSAR Signal Assignment ReplacementCOR Conditional Operator ReplacementLER Level Replacement…

All mutants

dead?

Input test program

Create mutants

Input test cases

Program Patterns

Run T on each live mutant

Analyze and mark equivalent

mutants

T

F

Quit


Functional Qualification (I)• The creation of testbenches still poses the

question whether the testbenches are correct and sufficient.

• FQ is the first technology to provide an objective answer to the question– Quis custodiet ipsos custodes?– Who shall guard the guards?

• FQ provides an automated and objective measure of the quality of the (co-)verification environment.


Functional Qualification (II)

• FQ is based on Mutation Analysis.• Uses a meta-model to avoid the need to

recompile each mutation.• FQ considers a mutant to have been killed

if a testcase fails– CertitudeTM tool from SpringSoft.


Functional Qualification Framework• If there were a bug in your design, could the verification

environment find it?


TLM Mutation Fault Model (I)• A mutation model, targeting the primitives of SystemC

TLM 2.0 library, has been developed as follow1. Formalize the behavior of the TLM 2.0 communication

primitives by using EFSMs;2. Define the mutations for EFSMs, based on an extension of the

transition fault model;3. Identify relations between the proposed mutations and typical

design errors.

TLM communicationprimitives

SystemC TLM 2.0 library

1Mutations on

primitives

2TLM

designerrors

3


TLM Mutation Fault Model (II)• Starting point: Transition Fault Model

– for FSM (Chow ‘78); – target

• boolean functions on transitions;• destination states of transitions.

• Extensions to cover EFSM– Mutations on destination states;– Mutations on enabling functions;– Mutations on update functions.

TLM communicationprimitives

Mutations onprimitives


Conclusions• The concept of Functional Qualification could

be applied to– TLM 2.0 SystemC.

• A mutation model for communication primitives– primitives formalized by EFSMs;– mutation model affects state and transitions;– mutations correspond to design errors.

• A mutation model for C++ functionalities.


THE ROLE OF PARALLEL SIMULATION ON FUNCTIONAL VERIFICATION


• Fault simulation is used in test generation to determine the fault coverage of a test set.

• Complex design � large number of faults� fast and efficient fault simulation

Fault simulation

Test set

��

� �

��

�Faultlist


• Fault simulation at bit-level– Serial fault simulation

• Faults are simulated one at a time.– Parallel fault simulation

• Faulty circuits are simulated simultaneously; • One fault per faulty circuit.

– Concurrent fault simulation• More faults are simulated per faulty circuit.

– Deductive fault simulation• Fault free circuit is simulated and the list of faults is deduced.

– Fault models• stuck-at, bridge,…

�

�

State of the ArtFaulty circuitReference circuit

�

�

�

�

��

�

�

�

�

��

sa0


State of the Art• Fault simulation at functional level

– Serial fault simulation• Faults are simulated one at a time;

– Parallel fault simulation• Multiple instances of the design on

parallel architectures;– Cons

• Word vectorization is not possible as at bit-level;– Pros

• Simulation is faster than at bit-level;• Questions rise:

– Can bit-level parallelization produce better performance results than serial functional simulation? In which cases?

module( M )

port ( … )

begin…

end

module( M )

port ( ...fp…)

begin… fp …

end

Faulty circuitReference circuit


Goal

• Parallel functional fault simulation– How to benefit from the use of

bit-level parallel simulation techniques for simulating functional-level faults.


RTL code

Functional fault injection

Fault-injectedRTL code

Synthesis

Fault-injectedbit-level code

Functional ATPG

Testcases

bit-level simulation report

RTLsimulation report

bit-levelparallel fault simulation

RTLserial fault simulation

Comparison

Framework

Issues

Fault model?

Parallelization?

Fault engine? Simulation kernel?


The functional fault modelFunctional fault model

butparallel simulation at

bit-level

�Fault model has to be

synthesizable

RTL

fault injectionmodule( M )

port ( … )

begin…

end

module( M )

port ( ...fp… )

begin… fp …

end

reference copies

Net

list

synthesis

vectorization


The functional fault model

• Different fault models � proposed techniques can be widely adopted.

• Bit coverage fault model– Objective

• Correlate the RTL faults with the bit-level ones.– Similar to stuck-at fault model but on RTL data types.

• Mutant fault model– Objective

• Functional verification.– Different types of code mutations.


Functional fault parallelization• Vectorization

– Bit-blasted netlist;– Each bit of the netlist is substituted with a vector of bits;– Each operator with the corresponding bitwise operator;– C implementation: vectorized bit = machine word.

wire A ; wire [ 1 : WIDTH ] A ;

vectorization

Original Netlist Vectorized Netlist

Refe

renc

e cop

y

typedef uint32_t machine_word_t;

typedef uint64_t machine_word_t;

…

C


Functional fault parallelization• Concurrency

– Into each netlist-copy many faults are activated at the same time;– If a faults conflicts, it is disabled and outputs and registers are

reset to the reference values.

1

1

2 3

2

4

3

Node

Fault

Impact


Experimental resultsDesign Fault

modelFault # SSE time PSE time FC% Speedup

%b10 bitcov 244 9.647 7.504 91.0 22.21

b04 bitcov 398 2.816 14.194 99.0 -403.87

b10 mutant 185 25.319 7.984 81.1 68.46

b04 mutant 66 10.337 6.728 74.2 34.94

hc11 mutant 2245 811.115 204.341 49.8 74.81

am2910 bitcov 3608 755.82 505.23 83.3 33.15

am2910 mutant 2763 2219.46 636.98 76.5 71.3

SSE : serial simulation engine (RTL)PSE: parallel simulation engine (bit-level)FC%: fault coverage percentage

56 / 23CREDES Kick-Off MeetingJune 5th, 2009

Experimental results

• Why bad results for b04?– the greater number of faults are classified early– parallelism is not fully exploited

not y

et c

lass

ified

faul

ts

simulation time

PSE SSE

57 / 23CREDES Kick-Off MeetingJune 5th, 2009

Conclusions

• Performances depend on abstraction level– RTL simulation is faster than the bit-level

simulation;– The objective is the overall fault simulation

time• It could be lower thanks to parallel approach that is

not applicable at RT level.


HW/SW CO-SIMULATION FOR CO-VERIFICATION


HW/SW co-simulation

• An Instruction Set Simulator (ISS) and the corresponding simulator reproduces the behavior of the CPU and the SW modules.

• An HW Description Language (HDL) reproduces the behavior of HW components.

• Co-simulation HDL+ISS– Integrated use of HDL and ISS;– Needs communication between HDL and ISS.


HW/SW co-simulation architecture

SystemCKernel

SystemCModules

SystemC

iss_port

QEmu-SystemCWrapper

ISSGuest Machine

Device Drivers

ApplicationSoftware

QEMU User Space Emulator

Testbenches

I/O Memory

Host Machine Socket IPC


SystemC/QEMU co-simulation• Implementing co-simulation requires modification

to– the QEMU both to communicate with the SystemC

simulator and to manage the HW device;– the SystemC simulator kernel

• add the capability of reading and interpreting the QEMU messages;

• sending interrupts to QEMU whenever the HW models generate them.

• These operations must be transparent to the designer who just writes the HW/SW models.


The QEMU side• The actors involved in the QEMU side of the

communication are– Application

• a user space application that interacts with a device through a DD.

– Device driver• a kernel module that accesses a device through the mapped

memory.– I/O memory

• a memory region where the device registers are mapped. Accesses to must be caught by QEMU-SystemC Wrapper.

– QEMU-SystemC Wrapper• this module forwards the requests to the SystemC side of co-

simulation.


APPLICATION II LEV. DRIVER I LEV. DRIVER QEMU-SYSTEMC WRAPPER SYSTEMC

Invokes the ioctlfunction:

arg = { DATA, ADDR }

ioctl ( ecc,ECC_WRITE, arg ) ecc_ioctl( ecc, file,

ECC_WRITE, arg)

do_write( DATA, ADDR) Write to the DATAaddress the valuecontained bypacket

Write to the ADDRaddress the address containedby packet

Write to theCOMM addressthe operation type (write = 1)

First packet sent via socket:iss_message_t.type = 1;iss_messate_t.MemElem = value;iss_message_t.addr = DATA;iss_message_t.lenght = sizeof( value);

Second packet sent via socket:iss_message_t.type = 1;iss_messate_t.MemElem = address;iss_message_t.addr = ADDR;iss_message_t.lenght = sizeof( address);

Third packet sent via socket:iss_message_t.type = 1;iss_messate_t.MemElem = 1;iss_message_t.addr = COMM;iss_message_t.lenght =

sizeof( unsigned int);

Receives valueon the DATA port

Receives addresson the ADDR port

Receives 1 on the COMM port

The write operationis performed

QEMU catches MEM access request and forwards it to the SystemC side via sockets.


The SystemC side

control_register

address_register

data_register

read_iss_data_registerset data_register

read_iss_address_registerset address_register

read_iss_command_registerset command_registerset iss_control_register = 0run_io_process.notify()

entrycase WRITE:set request

(data_register,address_register)

case READ:set request

(data_register,address_register)

get responseset iss_data_register

set iss_control_register = 1

iss_data_register iss_address_register iss_command_register iss_control_register

response

request

As soon as a request from QEMU is received via socket, the kernel extracts information from the request packets and it sets the correct values on the ISS ports.


HW/SW co-verification of platforms

SoftwareModules

Testbench Generation

HardwareModules

Test Response Evaluation

Co-Simulation Environment

SW Fault Model HW Fault Model

Functional Co-Verification Environment

Fault CoverageEvaluation

TestbenchModification

Functional Qualification


Conclusions

• A HW/SW co-simulation framework has been proposed which supports all mechanisms used by devices drivers to communicate with HW devices.

• NO adaptation in device drivers and HW models to handle synchronization messages.

• HW/SW co-verification of platform models.67CREDES Kick-Off MeetingJune 5th, 2009

References• A Methodology for Abstracting RTL Designs into TL Descriptions

N. Bombieri, F. Fummi, G. PravadelliMEMOCODE 2006

• Functional Qualification of TLM VerificationN. Bombieri, F. Fummi, G. Pravadelli, M. Hampton, F. LetombeDATE 2009

• The Role of Parallel Simulation in Functional VerificationG. Di Guglielmo, F. Fummi, M. Hampton, G. Pravadelli, F. StefanniHLDVT 2007

• On the Functional Qualification of a Platform ModelG. Di Guglielmo, F. Fummi, G. Pravadelli, M. Hampton, F. LetombeDFT 2009 (submitted)


Thanks!


Documents

Functional Validation for Dependability