ERAU - FAA Research

Real Time Safety

ERAU College of Engineering page 1

ERAU the FAA Research2007-2009

CEH Tools QualificationContract DTFACT-07-C-00010

Dr. Andrew J. Kornecki, Dr. Brian ButkaEmbry Riddle Aeronautical University

Dr. Janusz ZalewskiFlorida Gulf Coast University

FAA SW&CEH Conference Denver, CO

August 20, 2008

Real Time Safety


OutlineContract DTFACT-07-C-00010 “A Study on Tool Qualification for Complex Electronic Hardware”, Mar 15, 2007 – May 14, 2009

o Introductiono CEH Tools and Development Processo Tool Qualificationo Concerns and Possible Solutionso Conclusions and Future Work

Real Time Safety


Introduction: CEH Tools Project

ERAU/FGCU Project: “A Study on Tool Qualification for Complex Electronic Hardware” with an objective to produce report on the state of art in the CEH tool market and the programmable logic tool qualification issuesExplore safety issues in the assessment and qualification of tools used in developing complex electronic hardware (CEH) for the aircraft Typical devices are programmable logic devices (PAL, PLA, GAL, FPGA, ASIC, etc.) used as components of programmable electronic hardware RTCA DO-254, “Design Assurance Guidance for Airborne Electronic Hardware”, Section 11.4, “Tool Assessment and Qualification” is the starting point

Real Time Safety


Literature StudyPerspectives:o A broad view of the issues in designing CEH (26)o A focus on safety issues in avionics applications

related to safety critical aspects of CEH (6)o An industry practices in qualification of CEH tools

and compliance with DO-254 standard (18)Literature review points:o Plan to develop and verify PLD programs in the

same way as software programso Plan the safety argument from the start, and build

up evidence throughout developmento Use mature tools, amenable to qualification and

supported throughout the project life timeo Investigate the use of formal notations and analysis

techniques to increase verifiabilityo Do not use programmable logic hardware just to

avoid developing safety-critical software

Real Time Safety


What is a tool?RTCA DO-254/ED-80 provides guidance for design assurance of airborne electronic hardware defining design assurance, lifecycle, processes (planning, design, V&V, CM, assurance, certification liaison), and the lifecycle dataNo clear definition of a tool in the RTCA DO-254 or associated CAST papers; paraphrasing DO-178B:“A computer program or a hardware device used to help develop, test, analyze, produce or modify hardware component, subsystem, system or its documentation”A tool reduces, eliminates, or automates the objectives of the design or verification process

Real Time Safety


RTCA DO-254DO-254 distinguishes between two primary types of tools:o Design Tools - Tools whose output is part of hardware

design and thus can introduce errors. For example, an ASIC router or a tool that creates a board or chip layout based on a schematic or other detailed requirement(used to generate the hardware item or the hardware design, thus an error in the tool could introduce an error in the hardware item)

o Verification Tools - Tools used to ensure performance against predetermined standards or requirements. These tools do not introduce errors, but may fail to detect them. For example, an analog or digital circuit simulator or an automated test that measures actual circuit performance.(used to verify the hardware item, an error in the tool may cause the tool to fail to detect an error in the hardware item or hardware design)

Real Time Safety


CEH Tools IssuesASIC/FPGA tools have a thick layer of abstraction between the user Verilog or VHDL input and the tools output: the tool interprets the input, synthesizes/optimizes the logic, creates net lists, and translates the net list into a hardware layout Synthesis includes typically the optimization of logic, timing, and power aspects - a black box, with design as an input and “synthesized” design as an outputFPGA/CPLD vendors are interested in developing software requiredto take a design (schematic or HDL) into a form that can be used to program a circuit – they offer cost-effective entry-level design environments (complete packages with design entry, simulators, libraries, and the back end)Tool industry is dynamic and hardware keeps evolving, the software developers for back-end tools have to attend to two primary activities, developing libraries for new EDA tools and simulators, while creating better fitters and routers for new hardware with more resources and more complex architecturesEvolving interchange standards such as EDIF and Verilog/VHDL help standardize interfaces to CAD tools and simulators in form of EDIF-compatible library of design elements using Verilog or VHDL to implement the models necessary for simulation environment

Real Time Safety


1. Identify the tool 2. Identify the processthe tool supports

6. Establish qualificationbaseline and

problem reporting

7. Basic toolqualification

9. Design toolqualification

yes

yes

no

no

no

10. Complete the process

DO-254/ED-80Tools

Assessment and Qualification

Process

3. Independent assessment?

4. Tool is design A/B/C or verification A/B?

5. Relevant toolhistory?

8. Tool is design tool A/B?

no

yes

yes

Real Time Safety


Is Tool Qualification Required?A company working on a design assurance level A project writes a test bench that runs on a simulator and produces a pass/fail outputo The test bench automates the verification process

and is therefore a tool and thus must be assessed and qualified

Can relevant product service history get us out of tool qualification?o Many design tool suites have hundreds of users

doing diverse designs o Relevant service history requires the tracking of

problem (bug) reports and their resolutionTraceability data for all functional failure paths

It is rare that all of the necessary data is available

Real Time Safety


Is Tool Qualification Required ?(2)

The other way to avoid tool qualification is if the tool outputs are independently assessedFrom DO-254:Independent assessment of a design tool’s output that is generated in whole or in part by the tool may be established by the verification activities performed on the item, such as component, netlistor assembly. In this case, the integrity of the end item does not depend upon the correctness of the design tool output alone

Real Time Safety


Is Tool Qualification Required ?(3)

The correctness of the design tool can be assessed by verification tools and in-system hardware verification tests that are run o Conventional debugging of the hardware using

logic analyzers etc. will also assess the correctness of the design tool

It seems that design tools would rarely need to be qualified since it is unlikely that the output of the design tool is used without being verified both in simulation and in hardware

Real Time Safety


CEH: Logic Design ActivitiesDesign Entry: HDL, schematic entry, integration of IP cores Synthesis: translation HDL design definition into the logical or physical representation for specific hardware platformImplementation & Configuration: assignment of the logic created during design entry and synthesis into specific physical resources of the target device.Verification: design verification ranging from simulation to static timing analysis to equivalency checking via formal verification.Board Level Integration: compatibility of programmable logic design with the entire system.

Real Time Safety


Design Entry

Synthesis

Place &Route

Programming Configuration

TimeSimulation

Time Analysis

PowerAnalysis

OtherVerification

BehavioralSimulation

FunctionalSimulation

Generic Design Flow Graph

Real Time Safety


The ConcernIt is possible for errors to be introduced that occur while translating the simulated design into hardware Normal verification techniques will catch most of these errors during hardware verificationBut … some circuits are designed to operate only if the hardware is malfunctioning

Examples are:o Triply redundant hardware with voting circuits

Used to mitigate failure in one patho Metastability detection circuits

Can be used to detect single-event upsets due to alpha particle radiation

These circuits are difficult to verify in working hardware

Real Time Safety


Example: Triply Redundant Hardware

Intended Implemented

In A

In B

The synthesis tool can “optimize” the design in an unexpected fashion

Real Time Safety


Example: Metastability Sensing Circuits

Intended Implemented

It is possible to detect metastability by monitoring the Q and Qbar outputs of a single flip-flop

Real Time Safety


A ProblemThe redundancy and metastability problems occur when: o design optimizations are applied during

synthesis, and o the tool would not generate explicit warnings

that the optimization had occurredThe circuits should not operate if the hardware is functioning normally, thus error like this may not be detected during verification of the hardware

Real Time Safety


Possible SolutionsSince the problems are caused by the synthesizer performing optimizations, we could turn off all synthesizer optimizationso It is a possible solution, but it could be difficult to

meet timing and area constraints without optimizations

An experienced designer would recognize these conditions ahead of time and configure the synthesizer appropriately o Possible but difficult to verify whether or not the

designer has handled all possible areas of concern

Real Time Safety


Possible Solutions (2)Catch the errors in hardware verificationo Add additional test circuitry to allow verification

of all circuitsDifficult to anticipate early in the design so that it can be written into the requirements

o Add additional verification tests to verify circuit operation

If the additional circuitry is in place to detect single-event upsets due to alpha particle radiation we could add radiation susceptibility testsNo current standard for alpha particle radiation testing. Should it be in DO-160 rather than handled in DO-254?

Real Time Safety


Possible Solutions (3)The problems introduced by the synthesizer could be detected by running simulations using the gate level netlist with back annotated timing data o Running the full test suite on a gate level netlist

is feasible but it will be very slow and expensiveHowever DO-254 guidance does not require to do this:o Analyzing the implementation below the level of

that specified by the designer, such as at the gate or transistor level, is not necessary ….(DO-254, Appendix B page B-9)

o The designer worked at the HDL level not the gate level

Real Time Safety


Other Likely ProblemsAt DAL level A and B elemental analysis is used to assure that all functional failure paths have been coveredHowever, some failure mechanisms can exist which occur at conditions that may not be identified as functional failure paths Some failures are related to where within the FPGA the actual hardware is located during the place and route phase

Real Time Safety


Non-Functional Failure Path ConcernsTwo problems can be caused if large numbers of devices within the FPGA connected to the same internal FPGA power supply switch at the same time Resistance in the internal FPGA power bussing can cause the voltage provided to circuits in a particular location to drop Inductance in the FPGA power bussing can create Simultaneous Switching Noise if many outputs switch simultaneously Both mechanisms can cause errors directly or change the circuit timing leading to errors elsewhere

Real Time Safety


Non-Functional Failure Path Concerns (2)

Much of the information needed to analyze these conditions is proprietary to the FPGA vendor Vendor supplied FPGA tools are the best at addressing these issues However our survey indicates that the majority of designs are implemented using third party tools such as Mentor Graphics and Synplicity

Real Time Safety


Industry Survey - Methodology

Questionnaire distributed during the 2007 FAA SW&CEH Conference, New Orleans, LA, July 24-26, 2007 on a dedicated CEH session attended by 54 individuals interested in the CEH and the application of DO-254 Posting the questionnaire on the web and linking from www.do254.com websiteFollow-up mailing to over 150 individuals engaged in development of the aviation software and hardware The questionnaire distributed internally in companies engaged in the design of programmable logic devices Collection of feedback at the Programmable Logic User Group meeting in Clearwater, FL, November 15, 2007As a result we obtained a sample of 45 recorded responses

http://www.do254.com/

Real Time Safety


Industry Survey - QualificationOnly eight out of 45 respondents indicated that tool qualification was attempted:The responses were inconsistent and incomplete; here is not verified list:o Aldec – level Ao Synplify – level C, level Ao Questa – level Ao ModelSim – level Ao Actel Libero – level A

Two respondents listed several tools but without clear statement if they were actually qualified and without specifying the qualification level/type (qualification activities have been performed and documented but never reviewed or challenged by the FAA)

Real Time Safety


Survey: Tool LandscapeType of Devices

fpga32%

asic18%

cpld18%

soc8%

pla/pal/gal24%

Vendors of Hardware Devices

actel20%

atmel6%

altera22%cypress

11%

lattice12%

quicklog5%

xilinx24%

Vendors of Software Tools

synopsis19%

synplicity19%

mentor27%

aldec9%

cadence8%

others18%

Errors Encountered

yes48%

no52%

Real Time Safety


Survey: PopulationEducational Background

SW17%

HW83%

Type of Work

tools3%

application36%

verification15%

proj.mgt25%

components21%

Years of Expertise

<3 years12%

3-6 years19%

7-12 years19%

>12 years50%

Work on DO-254 Projects

yes33%

no67%

Real Time Safety


Industry Survey – Selected CommentsWe qualified a tool by comparing a logic analyzer trace from the actual system with a simulation from the tool (including propagations delays and functionality). The DO-254 guidance is not useful and does not include what should be done with the data. Most of the tools are widely in use and do not have a negative safety impact. Mainly the repeatability and verification of the design itself affects safetyAll companies are attempting qualification differently and we have not seen any increase in safety by doing a qualification. These tools are in use in thousands of designs in the world and the probability of a tool qualification effort finding an error is very low or negligibleThe tools are designed for rapid freeform development and not for requirements based methodical design. Fine for commercial applications but not really targeted to high reliability design. Qualification is basically demonstrating the pedigree of the tool itself, most vendors either do not keep requirements based design and verification information or that information is considered proprietary or trades secret. Thus independent assessment of tool output will continue to be the primary, and sometimes only, methodology available for “qualification”. As long as that fact doesn’t change, there will be very little progressNot having specific guidelines, there is too much uncertainty on what needs to be done to assess or qualify the toolsThe issue is about the user competence about the task being performed - do they know how to appropriately apply the tool? Guidance is extremely vague on the objectives of qualification. The use of the word "tool" is somewhat misleading. How the tool is applied is sometimes more of a risk than the tool itself. E.g. if a verification environment consists of a simulation testbench running on Modelsim, the tool itself should be less of an issue for assessment/qualification than the user developed simulation testbench

Real Time Safety


ObservationsVendor supplied FPGA tools are best at identifying failures that require knowledge of the FPGA internalsVendor supplied design tools are not commonly used design toolsVendor supplied tools often cannot close timing on designs that are easily handled by third party tools Vendor supplied tools can take advantage of detailed knowledge of the hardware to implement unique features

Real Time Safety


SummaryFor many designs normal the normal verification process activities are sufficient to independently assess the tool outputsBut some design problems are difficult to assess and require extra effort to identify and address o Requires knowledgeable designers and DERso The risk is people do not know what they do not

know DAL level A FPGA design and verification requires an experienced team of design and verification engineersGate level verification of the design appears to be necessary

Real Time Safety


Moving ForwardWe are executing several test cases to investigate issues including:o Non Functional Failure Path issues such as

Simultaneous Switching Noiseo Comparing tool reported device timing to actual

measured timing to evaluate design robustnesso Un-programmed circuitry operation: observe

whether or not un-programmed logic blocks switch during normal device operation and what are the implications with respect to power supply and signal integrity

Real Time Safety


Questions?

Questions, comments, suggestions?

Andrew J. Kornecki [email protected] Butka [email protected] Zalewski [email protected]

Documents

ERAU - FAA Research