IV&V Facility 1 Pre-Software Assurance Symposium Facility Initiatives Briefing Independent Verification & Validation of Programmable Logic Devices 8 July

1

IV&V Facility

Pre-Software Assurance Symposium Facility Initiatives Briefing

Independent Verification & Validation of

Programmable Logic Devices

8 July 2005

NASA IV&V Facility

James Cercone, Ph.D., P.E.,WVU-Tech

Michael Beims, SAIC

July 8, 2005 IV&V of Programmable Logic Devices – Beims, Cercone

2

Outline

• Review IV&V of PLD Research Project Objectives and

Framework

• Review of detailed technical findings and VHDL defect

taxonomy

• Provide overview of Work Instruction development

IV&V Facility



3

NASA-STD-8739.8

Software V&V is concerned with ensuring that software being developed or maintained satisfies functional and other requirements and that each phase of the development process yields the right products.

….. IV&V is performed by an organization that is technically, managerially, and financially independent of the development organization. For NASA, IV&V is performed and/or managed by the NASA IV&V Facility.

…“Software includes programs and operational data contained in hardware (e.g. firmware, programmable logic, and programmable gate arrays).”

IV&V Facility


IV&V PLD Status

4

IEEE STD 1012-1998

IEEE Standard for Software Verification and Validation, provides supporting information regarding the integration of IV&V into every step of the Software Development Life Cycle. The IEEE standard, like the NASA Standard, also cites firmware and microcode in its definition of software: “This standard applies to software being developed, maintained, and reused …. The term Software also includes firmware, microcode, and documentation.”

IV&V Facility


IV&V PLD Status

5

IEEE Std 1076™-2002

Abstract:VHSIC Hardware Description Language (VHDL) is defined. VHDL is a formal notation intended for use in all phases of the creation of electronic systems. Because it is both machine readable and human readable, it supports the development, verification, synthesis, and testing of hardware designs; the communication of hardware design data; and the maintenance, modification, and procurement of hardware. Its primary audiences are the implementers of tools supporting the language and the advanced users of the language.Keywords:computer languages, electronic systems, hardware, hardware design, VHDL

IV&V Facility


IV&V PLD Status

6

IV&V Facility

NESC Project Activities performed by IV&V

From Project Plan (SAIC Document #ISTO-05-98-192), Section 3 – Activities

Completion Date

1.Identify the FPGA design logic faults from:• NASA and industrial sites,• Document Artifacts, and• Comparison of typical FPGA logic design methods with proven software engineering methodologies, including those used for design and peer review

Year 1, 2Q

2. Identify existing software engineering methodologies that can be directly applied to FPGA designs by tracing common defects to their underlying cause

Year 1, 3Q

3. Suggest enhancements to developers’ design and peer review methodologies

Year 2, 1Q

4. Provide field prototyped training materials for performing PL software V&V Year 2, 4Q

5. Successfully complete a pilot project Year 2, 4Q


Primary Goals:

• Develop an IV&V strategy for PLD’s

• Provide field proven PLD Work Instruction (WI) to the IV&V practitioner

7

IV&V Facility

Activities Thus Far• Better understanding of PLD’s at WVU Tech and SAIC

– Primarily via literature searches and attendance at workshops– Presentation to IV&V CAWG– WVU Tech obtained and learned IDE’s (Integrated development

Environment) for both Actel and Xilinx PLD’s. WVU Tech has also obtained and learned Active HDL

Limited analysis and simulation of NASA project data• IV&V has mapped PLD’s into a better framework for IV&V WI development• Identifying a taxonomy of defects in VHDL Domain

– Via Literature Search– By comparing VHDL releases for the same chip– Evaluating SW code defects that can be “mapped” to PLD VHDL defects

• Had initial discussions with JWST as candidate for VHDL Pilot Project


Activities/Results thus far (9-1-05 through 8-8-05)

8

IV&V Facility

Activities/Results thus far (9-1-05 through 8-8-05)

Results Thus Far• Scoped PLD IV&V Framework to understand 05 accomplishments and

identify potential future year activity– Based on increased understanding, and– Realization that existing IV&V Code Analysis WI is insufficient for PLD

analysis• Defect taxonomy, in process of refinement, for presentation at MAPLD


9

IV&V FacilityDevelopment Environments

Idealized Software/PLD Development

Requirements Design Code Test

• SRS is a CM’d document• Rigorous flowdown is common

• SDD is a CM’d document• Manual or tool generated

• Different types of code (C, C++, etc)• Mature tools avail to aid in development, verification

• Performed on implemented code• Unit, Subsystem, System testing follows req’ts flowdown• Rigorous processCommon PLD Development

Requirements Design/codeand simulateat functional

level

Design/code and simulate

After chip layout

Testing afterPLD is

programmed• Part of subsystem• Hardware artifact (e.g. EQ spec, product functional spec) • Performed on

implemented PLD• Unit, Subsystem, System testing follows req’ts flowdown

• It is at this stage that Idealized/Common development processes diverge

•Design process•IDE•Target

• PLD’s also have timing concerns that are rare in software development, such as

•Synthesized versus native components•Race Conditions•Adequately buffering data


10

IV&V Facility

Current Year (2005) Activities against PLD IV&V Framework

Our current WI activity focuses on verification of VHDL Design• Develop Work Instruction• Flesh out Work Instruction with Pilot Project• Deploy Work Instruction at Facility


level


After chip layout

Testing afterPLD is

programmed

Common PLD Development

Similar to FSWbut artifactexpectationsneed to bearticulated by IV&V

VerificationTasks

ValidationTasks

tbd

1) Ensure syntax is correct2) Identify typical errors3) Develop/deploy WI: Programming Standards andDefect ID• VHDL• Verilog• Schematics

Traceability of RequirementsIdentify key timing areas and independently simulate

Identify any known issueswith IDE and ensure potential errors not presentin developed product

Re-simulate key timing functions

Verify tests performedby developer (using simulation to generatetest cases)

tbd, independenttesting?


11

IV&V Facility

Ideas for Future Year Projects

Future Projects• Develop WI for verification of Verilog or Schematic designs (item “a”)• Provide WI for Requirements Analysis or Test Analysis of PLD’s (item “b”)

• This is probably a straightforward extrapolation of current WIs for FSW• Have the breadth of knowledge on development tools and all target PLD’s (item “c”)• Provide insights on timing/validation aspects of PLD implementation (item “d”)

d

b

b

a c


level


After chip layout

Testing afterPLD is

programmed



VerificationTasks

ValidationTasks

tbd








12

Complexity is a Challenge for all Design Representations


IV&V Facility

Functional Trace / Performance Test

Design Trace / Functional Test

?

13

Review of detailed technical findings

and

VHDL defect taxonomy

IV&V Facility



14

IV&V Facility

Entity

Are Signals defined in the port list as out type signals given values?

Are Signals that are defined in the port list as inout type signals used for both

– reading and writing?

Are Signals defined in the port list as in type signals used in the architecture?


Sample Findings Potential VHDL“Hot Spots” visible in semantics

15

IV&V Facility

Process

Is there a series of sequential statements followed by a branching structure?

Is there a branching structure followed by a series of sequential statements?

Is each process sensitive list made up of the signals from the Entity’s port list?



16

IV&V Facility

If Structures

Having elsif and no else statement

Having neither an elsif or else statement

Is there unreachable code inside an else statement?

When using a compound if statement, are all possible conditions covered in subsequent elsif and else statements.

How deep is the nesting of if structure?

Testing Signals in the condition that are not part of the process’s sensitive list



17

IV&V Facility

Signal Assignment

Is the same set of Signals assigned values in each of the if-elsif-else sections?

Is the same set Signals assigned values in each of the case structures

when and when others => clauses?

Are all Signals in a component’s port list mapped values during a

Component’s instantiation?



18

Number of Lines Number of Blank Lines 311 Number of Comment Lines 1157 Number of Library and Use lines 130 Number of Packages 1 packages 520 Number of Entities 25 entities 525 Number of Architectures 25 architectures 6291 Total Number of Lines Examined 8934

Sample Findings Static Metrics Analysis of Public Code

IV&V Facility


19

Process Summary 129Number of Processes with 0 <= line count < 10 13Number of Processes with 10 <= line count < 20 61Number of Processes with 20 <= line count < 30 22Number of Processes with 30 <= line count < 40 6Number of Processes with 40 <= line count < 50 5Number of Processes with 50 <= line count < 60 2Number of Processes with 60 <= line count < 70 3Number of Processes with 70 <= line count < 80 5Number of Processes with 80 <= line count < 90 2Number of Processes with 90 <= line count < 100 0

Process with line count >=100 Number of Lines101108109113117152166194217927Note !

IV&V Facility



20

IV&V Facility



Overall Comparison of LAT TD docs

Filename: 1880 1881 1882 1883 1885 2147

Version 50-1 50-1 50-1 50-1 50-3 50-1

Number of Files 13 25 20 21 30 5

# of Comment Changes 0 0 0 0 0 0

# of Functionality Changes 0 0 0 0 0 0

# of Lines 3460 9793 5887 7413 9603 813

# of Blank Lines 0 0 0 0 0 0

# of Commented Lines 561 1214 906 981 1269 124

# of Partial Comments 447 1165 555 781 559 49

# of Library 31 53 44 48 57 10

# of Use 44 77 64 70 109 19

# of Package 1 1 1 1 3 1

# of Entity 14 25 21 23 28 4

# of Architecture 14 25 21 23 28 4

# of Component 23 31 26 25 33 4

# of Signal 134 433 242 288 789 27

# of in 203 469 603 666 758 49

# of out 116 261 410 453 545 50

# of inout 37 75 18 24 17 0

# of if 70 321 139 205 293 13

# of elsif 42 283 100 91 137 3

# of else 41 201 78 132 216 4

# of case 7 33 17 18 18 1

…

21

IV&V Facility



Observations related to Code Changes (absolutes for the public code analyzed)

A code change occurred if:

•There were 12 or less files•Average number of lines per file was greater than 400•There were less than 3 “package” statements•There were less than 11 “entity” statements•There were less than 20 “component” statements•There were less than 13 “architecture” statements•There were less than 20 “clk’event” statements•There were any “while” statements•There were any “wait” statements•There were any “after” statements

22

IV&V Facility



Observations related to Code Changes (normalized for the public code analyzed)

A code change occurred if:

•5% or more of the total lines were “signal” statements•5% or less of the total lines were “in” statements•5% or less of the total lines were “out” statements•3½% or more of the total lines were “if” statements•¼% or more of the total lines were “case” statements

23

IV&V Facility



24

The Percent of "If" Statements Versus The Number of Functional Changes

0

100

200

300

400

500

600

700

0.0000 1.0000 2.0000 3.0000 4.0000 5.0000 6.0000

Percent of "If" Statements

Num

ber o

f Fun

ctiona

l Cha

nges

IV&V Facility



25

The Percent of "case" Statements Versus The Number of Functional Changes

0

100

200

300

400

500

600

700

0.0000 0.1000 0.2000 0.3000 0.4000 0.5000 0.6000 0.7000

The Percent of "case" Statements

The Number of Functional Chan

ges

IV&V Facility



26

3b Problem: wait statements with different conditions in the same process.

Example: processbegin wait until rising_edge(clk1); x <= a; wait until falling_edge(clk2); x <= a;end process;

Context: Synthesis: this construct may not be synthesizable

Explanation: Would require flip flops to be sensitive to different clock edges at differenttimes.

Processes with multiple wait statements are turned into finite statemachines. The wait statements denote transitions between states. Thetarget signals in the process are outputs of flip flops. Using different waitconditions would require the flip flops to use different clock signals atdifferent times.Multiple clock signals for a single flip flop would be difficult to synthesize inefficient to build fragile to operate.

Sample Findings

Sample taxonomy of semantic defects visible in VHDLIV&V Facility


27

Problem: Attempt to read an output port.

Example: entity DFF is port( D, CLK : in std_logic; Q : out std_logic )end DFF;

architecture badarchitecture of DFF isbegin case Q is when ‘0’ => … …end;

Context: Compilation: This should result in a compilation error.

Explanation: Most compilers flag this as an error, but Xilinx permitted the code abovewhere an out port is used as an argument to the case statement.

IV&V Facility


Sample Findings

Sample taxonomy of semantic defects visible in VHDL

28

Problem: In an asynchronous reset, the test for reset occurs outside of the test forthe clock edge.

Example: process (reset, clk)begin if (reset = ’1’) then q <= ’0’; elsif rising_edge(clk) then q <= d1; end if;end process;

Context: Synthesis: synthesizable, but not desirable

Explanation: Asynchronous resets are bad, because if a reset occurs very close to a clockedge, some parts of the circuit might be reset in one clock cycle and some inthe subsequent clock cycle. This can cause the circuit to be out of sync as itgoes through the reset sequence, potentially causing erroneous internal stateand output values.

Note: NASA experts’ recommended practice prevents an ‘out of sync’ by insuring that resets are never very close to a clock edge. This design is seen in NASA flight software as a D-FF with reset.

IV&V Facility


Sample Findings


29

Synthesis vs Simulation difference visible in VHDL

Multiplexer with missing sensitivity signal (signal “b”)

process(a,sel) if sel = '1' then out <=1; else out <= b; end if;end process

www.synplicity.com/literature/pdf/HDLDesignMethods.pdf

IV&V Facility


Sample Findings


30

IV&V Facility

Fault Detection Matrix

Actual

Tested

False True

Positive False/PositivePotential Hot Spot identified/ No defect

True/PositivePotential Hot Spot identified/ Design defect exists

Negative False/NegativeNo Hot Spot identified/ Defect exists

True/NegativeNo Hot Spot identified/ No Defect

High Confidence – IV&V success

Less Confidence - Mission Risk


31

IV&V Facility

True/Positive Potential Hot Spot identified/ Design defect exists

“Pre-processor directives and usage are often not controlled by coding standards.

1) #ifdef statements can be left active or inactive and permit non-flight code (e.g. test code) to be compiled. Instances of such errors have been found in Mars program code.

2) #define statements can be left in the code from testing leaving test values or conditions active in the flight code. Instances of such errors have been found in Mars program code.”


Fault Detection MatrixExamples

32

IV&V Facility

Fault Detection MatrixExamples

process (A,B)

begin

if (cond1)

X <= A + B;

elseif (cond2) X <= X – B;

end if;

end process;


If neither cond1 nor cond2 is true, then X will retain its value ...basically, X is stored in a latch

In general, latches are not usually recommended in synchronous designs

False/Negative No Hot Spot identified/ Defect exists

33

IV&V Facility

Taxonomy of Common Visible Defects


Type of Defect Definition Runtime Errors

Arithmetic Exception Dividing by zero, or returning a negative value instead of a positive one.

Logical Errors Includes the errors in state transition, timing, control, and data flow. Initialization or incorrect initialization of a value.

Abnormal Value Variables should not be assigned to signals. Duplicate Object Objects that have been defined more than one time (i.e.

functions, variables and macros). Compile Time Errors Syntax Errors Errors that are problematic in compilation. Scope and Linking Errors

These errors deal with visibility of functions and libraries, and a global declaration versus a declaration seen by a local function.

Other Defects Unused Objects These objects are defined but never used. Coding Standard Violation

This includes duplicate code, giving meaningful names to different variables and code fragments, and keeping track of the names given to different code packages to prevent two or more packages receiving the same name.

Comment and Spacing Errors

Comment and spacing errors can occur when changing formats such a PDF format to DOC format.

34

IV&V Facility

VHDL Code Severity Chart

Severity Description

A FPGA code will not perform desired tasks. Mission is jeopardized.

B Serious hindrance to mission accomplishment. A serious cost effect to project. I.E. (No “quick fix”)

C Adversely affects FPGA code performance. A minimal cost effect to project but a “quick fix” is possible.

D Annoying effect to user but FPGA code is operable.

E Any other effect.


35

Overview of Work Instruction

Development

IV&V Facility



36

Work Instruction Development

Background Considerations

VHDL specific considerations

Taxonomy of potential “Hot Spots”

Clock and Reset LinesSensitivity ListsFeatures not consistently supported between IDE’sNon-implemented features (i.e some attributes)

IV&V Facility


37

IV&V Facility

VHDL Code Severity Chart


Defects /Runtime Errors Impact of Error Defect Criticality (dependent on Functional Criticality)

High Medium Low Arithmetic Exception Termination of a function or

produce an erroneous behavior A-B A-B B-C

Logical Errors Cause erroneous behavior that can lead to mission failure

A-C A-C C-D

Abnormal Value Allocation Error

Variables assigned to signals can produce unexpected behavior

A-D A-D B-D

Duplicate Objects Unexpected results are produced if definitions are different.

C-D C-D D-E

Defects/Compile Time Errors

Syntax Errors C-D C-D C-D Scope/Linking Errors

Errors should never be seen, since they should have been fixed in the review and verification stage.

C-D C-D C-D

Other Defects Unused Objects Produce wasted memory space and

harder code to maintain D-E D-E D-E

Coding Standard Violation Lots of error producing potential and difficult to maintain

D-E D-E D-E

Comment and Spacing Errors

Can produce errors that make functional flow difficult to follow

D-E

D-E D-E

38

IV&V Facility

VHDL Code Severity Chart Examples


Example of High Functional Criticality -

The FPGA with this defect is the only functional capability for the Satellite to deploy the solar panels. If the FPGA does not perform this function, the satellite will run out of power, causing loss of the mission.

39

IV&V Facility

VHDL Code Severity Chart Examples


Example of Low Functional Criticality -

The FPGA controls one side of a dual redundant path to the telemetry transponder. If the FPGA fails, then the telemetry is routed via CPU control, causing a momentary delay in telemetry, if all telemetry is buffered through on-board storage. (Note: since this is a design (software) defect, if there were two identical FPGA's controlling this functionality, instead of an FPGA and the CPU, then the redundant FPGA can be expected to fail in the same manner and there is no functional redundancy, making this a High Functional Criticality defect.)

40

IV&V Facility

Example of Vendor Specific Degree of Compliance

• “major differences between XVHDL and Express is IEEE VHDL-93 compliance. XVHDL is a fully IEEE VHDL-93 compliant tool. Express supports many of the most commonly used VHDL-93 synthesis constructs, but is not yet fully compliant; it remains officially compliant with the IEEE VHDL-87 standard.”

• http://www.xilinx.com/xlnx/xil_ans_display.jsp?iLanguageID=1&iCountryID=1&getPagePath=5144 (7/21/2005)

Compliance Issues


http://www.xilinx.com/xlnx/xil_ans_display.jsp?iLanguageID=1&iCountryID=1&getPagePath=5144

41

IV&V Facility

Example of Vendor Specific Compliance

(two examples)

• Signal Declaration– Supported ("register" or "bus" type signals are not supported)

• Attribute– Only supported for some predefined attributes: HIGH, LOW, LEFT, RIGHT, RANGE, REVERSE_RANGE, LENGTH, POS,

ASCENDING, EVENT, LAST_VALUE– Otherwise, ignored.

• http://www.xilinx.com (7/21/2005)


Compliance Issues

http://www.xilinx.com/xlnx/xil_ans_display.jsp?iLanguageID=1&iCountryID=1&getPagePath=5144

42

IV&V Facility

Defect Taxonomy (VHDL Verification at Functional Design), and Pilot Deployment

• Draft of VHDL programming standards geared toward defect identification

– Defects commonly detected by compilers are not included– Includes syntax, timing margins, clock boundaries

• This draft is in process of update, peer review– Align defects with known coding defects– Test draft product against actual VHDL text

• Developer places multiple revs of VHDL on website

• The results will be presented at MAPLD in September, 05

Note: GLAST LAT used Actel VHDL for design and this served as basis for IR&D project.MRO project used Xilinx Verilog for design.


Work Instruction in 05 addressed VHDL verification. Future tasks needed to address:

Verilog and Schematic verification plus validation.

43

IV&V Facility

Preliminary Considerations for Defect Detection in VHDL Based Designs

• Materials needed to start the verification process are:

• Design Documentation to analyze performance• Actual VHDL Code • Code Pedigree

» (Reused modules, designers, level of experience…)• Development and Analysis Tools State diagrams. • Clock Trees• NASA and IEEE Standards


44

IV&V Facility

• V&V Process and Procedure at the Code Level

– Static Metric Analysis– Code Coverage (particularly for behavioral level designs)– Verification of Clock and Reset Tree’s (if provided)– Check for compliance to NASA Standards– Check for device resource usage

• (synthesized vs. board components such as MAC’s, SR, and DFF’s)

– Check of IDE specific restrictions – Check VHDL specific “Hot Spots”


Preliminary Considerations for Defect Detection in VHDL Based Designs

45

Conclusion

• Review IV&V of PLD Research Project Objectives and

Framework

• Review of detailed technical findings and VHDL defect

taxonomy

• Provide overview of Work Instruction development

IV&V Facility



46

IV&V Facility

Background Slides

47

IV&V FacilityIndependent• Technical: IV&V prioritizes its own efforts• Managerial: Independent reporting route to NASA Headquarters• Financial: Budget is allocated by NASA to the IV&V Facility such that

IV&V effectiveness is not compromised

Verification (Are we building the product right?)• The process of determining whether or not the products of a given phase of the

software development lifecycle fulfill the requirements established during the previous phase

• The product is internally complete, consistent and correct will support the next phase

Validation (Are we building the right product?) • The process of evaluating software throughout its development process

to ensure compliance with software requirements. This process ensures:– Expected behavior when subjected to anticipated events– No unexpected behavior when subjected to unanticipated events– System performs to the customer's expectations under all

operational conditions

What is IV&V ?


48

IV&V Facility

Maximizing Project V&V and IV&V

• The Project V&V goes end-to-end

– Needs sufficient depth to help ensure that they have build the product right and built the right product

• IV&V needs to be the second line of defense

– Select the most critical functionality and then IV&V to the appropriate depth -- not exceeding the IV&V point of diminishing returns (maintaining reasonability)

– The cut-off point should be where we have found the critical defects and also gained enough confidence in the software to support mission assurance requirements and launch recommendations

• Project Teams should compare our criticality rankings to their knowledge of the development as an independent source and explore differences

• Project Teams should look at activity just below the IV&V line to ensure adequate V&V resources


49

IV&V FacilityPLD Updated Framework

• Framework to perform IV&V was updated to– Take into consideration that development of PLD’s is different than development

of software,– Address verification and validation tasks explicitly

• PLD’s are developed in an environment that combines design, code and simulation simultaneously

– Timing aspects much more critical in PLD’s

• Major differences needed to be addressed:– PLD development is part of subsystem development – comparable artifacts not

consistently generated– PLD design can occur in many forms

• Schematic (representation similar to chip/board design)• VHDL (representation similar to Ada)• Verilog (representation similar to C/C++)

– Target system matters• Syntax different even when same language used

– Development environment matters• From a syntax standpoint• From a capability standpoint (e.g. software motivated or hardware motivated)• Which revision of IDE (multiple releases for hw motivated IDE’s)July 8, 2005 IV&V of Programmable Logic Devices – Beims, Cercone

50

IV&V Facility

Updated IV&V Framework for PLD Development


level


After chip layout

Testing afterPLD is

programmed



VerificationTasks

ValidationTasks

tbd







• The above updated IV&V framework has more detail– Allows us to clearly understand what we have accomplished, and what lies ahead– Strategy is to perform accomplished tasks well– For each task performed,

1. Develop Work Instruction (WI), flesh out internally2. Test on pilot project3. Deploy updated WI

• The fast pace of PLD product evolution requires additional considerations– Important to have cognizance of market trends– Update WI appropriately with trends that will be implemented in spacecraft in the near term


51

Simulink Example

http://www.transtech-dsp.com/software/simulink.asp


IV&V Facility

52

Small Microprocessor Example


IV&V Facility

53

Examples of Rare Software Defects


IV&V Facility

•PLD’s also have timing concerns that are rare in software development, such as:

–Synthesized versus native components – Seen in Stinger real time seeker simulations. Debate over whether a native compiler matrix multiplication routine was sufficiently predictable versus a matrix multiply built up in separate Fortran 77 instructions.–Race Conditions – Seen in the “Response to a Setting Satellite Vehicle” scenario in the Space Shuttle GPS’ firmware.–Adequately buffering data – Seen in the Space Shuttle Primary Avionics Software Systems’ Mid Frequency Executive where every variable must be analyzed for time homogeneity and treated accordingly.

•In General, “PLD–like” defects are seen in hard, real time software systems, which in turn are the primary candidates for migration to PLD’s in the near future (recent past?)

Documents

IV&V Facility 1 Pre-Software Assurance Symposium Facility Initiatives Briefing Independent Verification & Validation of Programmable Logic Devices 8 July