Embed Size (px)
Citation preview
Mapping MIL-HDBK-516B to DO-178C Safe and Secure Systems and Software Symposium (S5)
June 12, 2012
Scott Olson
Bill StClair
Todd White
What is MIL-HDBK-516?
• Airworthiness Certification Criteria
• This document establishes the airworthiness certification criteria to be used in the determination of airworthiness of all manned and unmanned, fixed and rotary wing air vehicle systems.
• It is a foundational document to be used by the system program manager, chief engineer, and contractors to define their air system’s airworthiness certification basis.
• This handbook is for guidance only
Mutual Acceptance of Airworthiness Data
MIL-HDBK-516 Airworthiness
Certification Criteria &
Joint Service MoA
USAF AFPD 62-6
USAF Aircraft Airworthiness Certification
USA AR 70-62
Airworthiness Qualification of U.S.
Army Aircraft Systems
USN NAVAIRINST 13034.1
Flight Clearance Policy for Air Vehicles and
Aircraft Systems
Creating a Basis of Certification
• Tailoring rules for MIL-HDBK-516 are as follows:
1) Identify non applicable criteria & provide rationale for exclusion
2) Criteria that only have partial applicability are to be identified along with supporting rationale
3) Supplement applicable criteria with measurable parameters
4) Develop additional criteria for characteristics not in the handbook
• A tailored Basis of Certification is a program level document
• TACC: Tailored Airworthiness Certification Criteria
• AQP: Airworthiness Qualification Plan
Basis of Certification
• Includes:
• Qualification requirements
• “Verify that…”
• Process Requirements
• Guidance on Design
• How do I use it?
– Performance Requirements
– Basis of Certification
What you need to design
How to design, build, and test
MIL-HDBK-516B
• Current Versions & Supplements:
• MIL-HDBK-516B Expanded Version – 26 SEP 2005 • ASC/EN Airworthiness Certification Criteria Expanded Version
of MIL-HDBK-516B
• ‘Standard’ and ‘Compliance’ further described
• MIL-HDBK-516B Change 1 – 29 FEB 2008
• Preliminary Unmanned Aircraft System (UAS) Airworthiness Certification Criteria – 31 MAY 2011 • Adds criteria for Unmanned Aircraft Systems
• “Fixes” MIL-HDBK-516B Change 1
What is DO-178C
• Software Considerations in Airborne Systems and Equipment Certification
• Provides the aviation community with guidance for determining that the software aspects of airborne systems and equipment comply with airworthiness requirements
• Includes: • Objectives
• Activities
• Descriptions of Evidence
• Tailoring Guidance
• Additional Considerations
• It is not a prescriptive process
• Relies on system level process • System Engineering
• System Safety Analyses
Required Processes for DO-178
System Safety (ARP4761 / MIL-STD-882E)
Aircraft and System Development Process
(ARP 4754A)
Electronic Hardware Development Life-Cycle
(DO-254)
Guidelines for Integrated Modular Avionics
(DO-297)
Software Development Life-Cycle (DO-178)
Integrated Safety Analysis Process
DO-178C Objectives >> DAL
Project Methodology
• Built a data base of certification criteria
• MIL-HDBK-516, UAS Supplement
• Mapped DO-178C and required entry processes, System Safety & Systems Engineering, to identified criteria to look for coverage completeness
• Considered USAF comments regarding DO-178
• Assembled a roundtable of experts to discuss:
• Software FAA DERs (Qty 2)
• Ph.D. Software Systems Engineering and System Safety
• Airworthiness Engineers
• Software Tools Expert
Project Assumptions
• DO-178C and DO-254 are to be used as the development approaches for software and electronic digital hardware
• Systems design is based on ARP 4754A
• The system safety process is based on MIL-STD-882E, ARP 4761 and the Joint Systems Software Safety Handbook
• The intent is to use these specification to define the development approaches and not to replace MIL-HDBK-516B or any part of MIL-HDBK-516B
• Any gaps that exist in coverage of MIL-HDBK-516B requirements will be treated within the "Additional Considerations" objective of the DO-XXX documents
High Level Mapping
• ARP 4754A Aircraft and System Development Process maps to MIL-HDBK-516B Section 4, Systems Engineering
• System & Software Safety maps to MIL-HDBK-516B Section 14, System Safety
• DO-178, DO-254, DO-297 maps to MIL-HDBK-516B Section 15, Computer Resources
USAF Perspective on DO-178
May 2010
November 2009
Key:
Item of major contention in DO-178B
Document partially addresses the task
Document does not address the task
Task MIL-HDBK-516B Expanded DO-178B USAF Rationale Alternate Opinion
Identify safety critical functions and hazards YES Partially
DO-178B: See page 111 in DO-178 - System safety assessment process briefly mentions functional hazard analysis.
In our opinion, it is not the intent of DO-178 to address hazards or how safety-critical functions are determined; this is the function of an integrated system and software safety processes.
Define architecture to meet safety critical function/hazard criticality YES NO
DO-178B: DO-178B has the opposite mentality; the architecture design is used to influence and dictate which software/hardware is critical.
In our opinion, DO-178 only states that architecture is a consideration in the determination of Level, which is consistent with standard hazard analyses, standard mitigation approaches (e.g. redundancy and partitioning). Note that architectural mitigation is very often used to reduce risk, as represented by AND gates in FTAs, and such a reduction in risk absolutely should be considered when assessing Level per DO-178.
Computer resource sizing Partially Partially
MIL-HDBK-516: 516 addresses memory, I/O and throughput, but not future growth analysis and requirements.
In our opinion, we are not proposing that MIL-HDBK-516 requirements be subsumed by DO-178. MIL-HDBK-516 requirements must be met, via DO-178 methods or otherwise.
DO-178B: Computer resource sizing is mainly a system-level and hardware aspect, both of which are not in the purview of DO-178B.
DO-178C sections 6.3.1 & 6.3.2 Requires that software requirements be reviewed for compatibility with the target computer.
Develop Software Development Plan (SDP) YES YES
Allocate requirements to hardware YES NO
DO-178B: DO-178B does not sufficiently address hardware. Requirement allocation to hardware is a topic of DO-254.
DO-178 does not address hardware; DO-254 addresses hardware requirements capture including system level requirements allocated to the hardware, safety requirements along with the derived requirements and hardware requirement validation and verification.
Allocate requirements to software YES YES
Horizontal/Vertical traceability (requirements and functions) YES Partially
DO-178B: DO-178 does address requirement allocation, which lends itself to vertical traceability, however this is only with regard to software. Horizontal function traceability, along with hardware traceability, is not addressed.
DO-178 does not address hardware; DO-254 address hardware requirement allocation and traceability. It includes vertical tracing (system requirements, code, test cases, etc. and horizontal tracing to design and developments standards, interfaces, group knowledge. (5.1.2 Requirements Capture Activities) In our opinion, DO-178 enforces the strongest traceability requirements of any standard available, in any industry. DO-178C section 6.3.1 & 6.3.2 requires that "software derived requirements and their reason for their existence are correctly defines be reviewed. This is normally interpreted that all derived requirements must be traced to their reason for existence. ARP 4754A requires that the hardware and software design/build processes provide traceability to the allocated system requirements. (4.6.2 Hardware and Software Design/Build).
Integrated Operational Flight Program (OFP) development YES NO
DO-178B: DO-178B does not address the integration of multiple CSCIs. As OFPs usually consist of multiple CSCIs, the document does not address OFP development.
In our opinion, we agree that this is the primary weakness of DO-178, that its emphasis on LRUs results in an unfortunate oversight of software/software integration and system integration. This is an area where we would look to MIL-HDBK-516 for requirements to drive a best-practice integration process.
Task MIL-HDBK-516B
Expanded DO-178B Rationale Alternate Opinion
Failure Modes and Effects Analysis (FMEA) YES NO
DO-178B: DO-178B does not address FMEA This is addressed as part of the safety program, which is entirely outside the scope of DO-178. DO-178 addresses the software development process once a Level determination has been made based on a safety process which it is dependent.
Failure Modes and Effects Criticality Analysis (FMECA) YES NO
DO-178B: FMECA is a criticality analysis of the failure rate of hardware used in a system; hardware is not a major aspect of DO-178B.
Addressed as part of the safety program.
Software Failure Modes and Effects Testing (FMET) YES YES
System Failure Modes and Effects Testing (FMET) YES NO
DO-178B: DO-178B addresses software only (except for target hardware), not system-level failure insertion testing.
In our opinion, these testing requirements are derived from the hazard analysis, which is out of scope for DO-178.
Software architecture risk mitigation strategy Partially Partially
MIL-HDBK-516: Redundancy management is a software form of architecture risk control, but little information on software techniques is provided.
Addressed in the same manner as currently under MIL-HDBK-516B
DO-178B: Redundancy management is a software form of architecture risk control, but little information on software techniques is provided.
In our opinion, this is a hazard risk reduction strategy that is out of scope for DO-178. Please mote that, in our opinion, industry does not want DO-178 to start describing safety analysis techniques. It only mentions architecture with respect to Level determination because it has to, because it incorrectly says that Level determination is based on severity instead of on risk. This is an area of improvement for DO-178. correction can be handled in planning documents.
System architecture risk mitigation strategy YES NO
DO-178B: DO-178B addresses software only (except for target hardware), not system-level redundancy or other fault-tolerant techniques.
Previously addressed (see above).
Safety interlocks mechanization YES NO
DO-178B: Safety interlocks are a form of software/system-level risk reduction technique; it is not addressed in DO-178B.
Previously addressed (see above).
Software engineering environment requirements YES YES
OFP build process YES NO
DO-178B: DO-178B does not address the integration of multiple CSCIs. As OFPs usually consist of multiple CSCIs, the document does not address OFP build requirements.
In our opinion we agree. Previously addressed (see above).
CSCI interface requirements YES YES
In our opinion, DO-178 does not adequately address interface requirements. It only focuses on software/hardware interface, and restricts that to single LRU integration. CSCI interface across multiple LRUs is not addressed.
Task MIL-HDBK-516B Expanded DO-178B Rationale Alternate Opinion
Does not allow lowering criticality of software based on redundancy YES NO
DO-178B: DO-178B allows for the lowering of criticality based on implementation of redundancy or other risk mitigation techniques.
Previously discussed (see above). It should be noted here that redundancy lowers risk, which absolutely should be considered during DO-178 Level determination. "Criticality" is an ambiguous word that should not be used in safety discussions.
Guidelines for regression/"delta" qualifications (software or system) YES NO
DO-178B: Nowhere in DO-178B is "delta" qualification addressed; the phrase "qualification testing" is primarily limited to tool qualification.
In our opinion, regression testing should be based on impact assessment of the "delta", which may or may not impact safety, determination of which is based on safety analysis, which is outside the scope of DO-178. Qualification is not used in DO-178 and DO-254 except for software tools and E
3 testing.
Digital hardware selection NO NO
MIL-HDBK-516: MIL-HDBK-516 is for airworthiness certification and does not address front end requirements, just if the hardware is safe and tested appropriately.
DO-178B: DO-178B does not address hardware. Previously addressed (see above) hardware is out of scope for DO-178. DO-178 only addresses hardware with respect to software/hardware integration on a LRU basis. DO-254 is the governing standard for complex electronic hardware. DO-178C section 6.3.2 and 6.3.3 requires review of software requirement compatibility with the target computer. DO-254 provides guidance for developing and certifying airborne electrical hardware(digital). Conformance to DO-254 would justify selection of the hardware. Also, DO-254 Section 11 "Other Considerations" provides guidance on Commercial-Off-The-Shelf (COTS) components usage and requires COTS management plans and that COTS be verified through the overall design process include the DO-254 supporting processes. Requirements based verification as required by d)-254 would imply that requirements be allocated to the COTS and verified by requirements based testing. DO-297 address integrated modular avionics development guidance, certification considerations, integral processes and continued airworthiness. Conformance to DO-297 would justify the section of the IMA system digital hardware.
Software selection NO NO
MIL-HDBK-516: 516 does not address coding language selection since it is an airworthiness standard, it assumes the language is already selected.
DO-178B: DO-178B does not have a process for determining the appropriate language for development, weapon system commonality or supportability
In our opinion, coding language should not be determined at this level. DO-178C gives the following very limited guidance. The basic principle is to choose programming languages that limit the opportunity for introducing errors, and verification methods that ensure that errors introduced are detected.
Integration of software on target hardware YES YES
Top-level software architecture design YES YES
System integrated schedule NO NO
MIL-HDBK-516B: Scheduling is not an airworthiness issue.
Task MIL-HDBK-516B Expanded DO-178B Alternate Opinion
Test plans/qualification
Structural coverage Partially YES
MIL-HDBK-516: Structural coverage test reports are acceptable verification artifacts; no guidance or specific requirement is given.
CSSIP: Currently, CSSIP addresses structural coverage testing in a limited manner; additional information will be added as the document matures.
CSU/CSC YES YES
CSCI YES Partially DO-178B: DO-178B allows for unqualified software to be released to the air vehicle
In our opinion, DO-178 does not allow "unqualified" software to be released to the air vehicle. In DO-178, 254 & ARO 4754 the terms qualified or qualification are only used in reference to software tools or E3 qualification.
System YES NO DO-178B: DO-178B has a software only (except for target hardware) focus and does not address system-level testing.
Previously addressed (see above); comments on software/software and system integration weaknesses of DO-178. ARP 4754A addresses system validation and verification testing. System level requirements based testing is required for airworthiness certification regardless of the development approach.
Hardware YES NO DO-178B: DO-178B has a software only (except for target hardware, but no description of its testing) focus and does not address system-level testing.
DO-254 address hardware validation and verification testing
Tool qualification plans YES YES
Software configuration management, control, and baseline establishment
YES YES
Software test procedures YES YES
Software test coverage analysis YES YES
Qualify tools Partially YES DO-254 specifies the requirements for hardware tool selection and qualification.
Future computer system modifications YES NO DO-178B: Future modification is mainly a system-level and hardware aspect, both of which are not in the purview of DO-178B.
In our opinion, DO-178 is very clear that systems are certified, not software alone. That is why you cannot purchase "certified" software from a vendor, not matter what that vendor tells you. Any change to the processing hardware upon which software is hosted (target hardware) would absolutely require re-certification per DO-178. ARP 4754 section 6 addresses modification to Aircraft or Systems. DO-254 or DO-297 are also applicable.
Software problem reporting YES YES
System problem reporting YES NO DO-178B: DO-178B has a software only (except for target hardware) focus.
DO-178 addresses software problem reporting and DO-254 addresses hardware problem reporting. In actual implementation a system level problem reporting is also required to achieve the DO-178, DO-254 required level of integration with the system level processes. System level problem reporting currently exists.
Software coding standards YES YES
Software requirements-based testing YES YES
Initial Opinion • Consideration must be given to the fact that a DO-178C approach requires that a set
of compliant plans be generated, judged to be compliant by the certification authority, and then the approved plans become the standards against which the project is judged
• The software development life-cycle is required by the civil regulations to be surrounded by the safety, hardware, and system (including integration) disciplines
• The USAF takes the role of the certification authority and is requiring that all of these disciplines be accounted for to create a complete life-cycle
• Once the plans are established, meaning the policies and procedures you are looking to develop, and once the air force is satisfied that the plans meet their requirements, then execution to the plans becomes the criterion for system approval
• This is an acknowledgement of the “concerns” listed in their presentation, which includes:
• This requires other processes be established and followed to address the verification criteria of MIL-HDBK-516 not addressed by DO-178B
• MIL-HDBK-516 is similar to, and can be a direct replacement for, the civil guidance that already surrounds DO-178C in its role in the certification of civil aircraft
• We are very interested in further understanding the USAF point-of-view and their insight on how to address either real or perceived gaps, and, the CSSIP development process under development
Areas of Interest & Recommendations
• Software FMEA & FMET • Hardware and Software Safety Analyses are different!
• Robustness Testing
• Tool Qualification
• Additional Considerations • Constraints
• Additional Objectives
• Specific Requirements due to weapons concerns • US Army AMCOM’s AR 385-17
• Software Safety Program Plan • Include as part of DO-178
• Interface Control Documents, … • Include as part of DO-178
Software FMEA & FMET
• Better methodologies exist
• Discussion paper was developed
• …Performing a hazard analysis as described within [the paper] (based on identifying hazardous software behavior beginning at external interfaces, performing FTA, etc.) is much more likely to identify potentially hazardous operating scenarios where the proper system (hardware and/or software) detection and response should be tested…
Focus on Robustness
“Robustness: The extent to which software can continue to operate correctly despite
abnormal inputs and conditions.”
Structural Coverage Analysis
• Purpose:
• Structural Coverage Analysis determines which code structure, including interfaces between components, was not exercised by the requirements-based test procedures.
• The requirements-based test cases may not have completely exercised the code structure, including interfaces, so structural coverage analysis is performed and additional verification produced to provide coverage
Functional Failure Path (FFP) Analysis
• DO-254 Objective:
• Identify the anomalous behaviors and functional failures that have been delegated to the software item from the system level
• Identify the FFPs, the effects of their anomalous behavior or functional failure, and decomposition level in the design hierarchy to which the analysis was performed and the type and location of the acceptable assurance data that should be available
• Describe the relationship between FFPs to determine their independence and inter-dependencies on other FFPs and components. Such relationships may be described using qualitative FTA or other top-down analysis, common mode analysis, F-FMEA or dependency diagrams. The relationship descriptions should identify those inter-related paths and components and the inter-dependencies
• Trace between the FFPs and the software requirements and derived requirements
Verification Tools in DO-178C
• Systematically and consistently apply coding standards
• Perform Data Control and Coupling Analysis based on the execution of requirements-based tests
• Automate the structural coverage analysis method
Additional Considerations
• Objective
• DO-178C Section 4.1d “Additional considerations, such as those described in Section 12, have been addressed, if necessary.”
• Activities
• DO-178C Sections 4.2f, 4.2h. 4.2i, 4.2j, 4.2k
• Output
• DO-178C Planning Documents
Section 12
Recommendation is to create a baseline template with all objectives required and/or constraints that
need to be imposed
Next Steps
• Expand database to include all known certification criteria • STANG 4671 for example
• Continue to identify supplemental requirements to be included in DO-178C’s ‘Additional Considerations’
• Evaluate Alternate Methods for additional constraints and objectives • USAF: Computer Systems and Software Integrity Program (CSSIP)
when published
• US Army: Software Engineering Evaluation System (SEES)
• Publish a universally tailorable Software Safety Program Plan Template • Pre-populated with their constraints and additional objectives
• Based on experience and expert input
• DID: DI-SAFT-81626
Contact Information
Scott Olson Airworthiness General Atomics Aeronautical Systems, Inc. [email protected] (858) 312-3282
Bill StClair Director LDRA Technology, Inc. & LDRA Certification Services [email protected] (408) 691-2442
Todd R. White FAA DER Qualtech Consulting, Inc. [email protected] (941) 331-5651
Thank you!
