DGTA-ADF

Migrating to a Software Assurance Standard

2008 ADF Software Symposium

FLTLT Patrick Redmond

SCI-DGTA

DGTA-ADF

Overview• The Problem with Legacy Software

• Migrating to a Software Assurance Standard

• The Potential Challenges- Bounding the Change- Low-Level Requirements- Traceability- Structural Coverage

• Case Study- MIL-STD-498 to DO-178B- Considered throughout.

DGTA-ADF

The Problem with Legacy Software• Numerous ADF platforms acquired where no software

assurance standard has been explicitly applied.

• Only MIL-STD-498 or DOD-STD-2167A.

• Development standards do not define how well software has to be constructed.

• During development, somebody made the decision to stop testing.

- Why?

• During development, somebody made the decision to review source code.

- Why?

• To what extent should the ADF rely on this software?

• To what extent does the ADF rely on this software?

DGTA-ADF

Migrating to a Software Assurance Standard• DGTA requires that a software assurance standard

be applied to legacy software systems.

• All development methods apply a level of assurance, but is that level acceptable?

• Applying a software assurance standard to each modification of in-service software progressively increases the integrity.

• Applies ‘Cancer Theory’ to Legacy Software

• Two methods:- Service History Argument and Application of Recognised

Standard e.g. RTCA DO-178B

- Software Assurance Task Matrix Negotiate a custom software assurance “standard” with DGTA

DGTA-ADF

How does migration work?

?

?

? ??

?? ?

?

?

? ?

?

?

?

?

?

? ?

Legacy Software of Unknown Integrity

Known Integrity

Software of Known(?) Integrity

DGTA considers the software to be compliant with the applied software assurance standard from this point forward.

Build x

Area Impacted byBuild x

Bui

ld y

DGTA-ADF

Service History Argument / Recognised Standard• Establish a Service History Argument that demonstrates that the

current software is acceptably safe.- Need to consider, among others:

Configuration Management Problem Reporting Change Control Relevance of Product Service History Operating Environments Safety Related Problems Design and Code Errors Error Rates

• No need to create a service history argument for in-service software, ADF has already determined that it is acceptably safe.

- Only applies to in-service, legacy software. Does not apply to acquisitions.- Does not apply if there is a substantial change of context in which the

software is used.

• Apply a recognised software assurance standard to the next modification.

• DGTA then consider the software to be compliant with that standard.

DGTA-ADF

Statement coverage.

High-level requirements coverage.

Software Assurance Task Matrix

Design Code Test

OFP B

OFP C

OFP D

Source code is verifiable.

Source code complies with low-level reqs.

Low-level requirements are defined.

High-level requirements are defined.

Decision coverage.

178B178B

Decision coverage.

Negotiate with SCI

Statement coverage plus untested code analysis.

DGTA-ADF

The Potential Challenges• Bounding the Change

- How much “re-assurance” needs to be done?

• Traceability

• Low-Level Requirements

• Structural Coverage

DGTA-ADF

The Easy Ones• Planning

• Additional Considerations- Tool Qualification

• Development of High-Level Requirements, Source Code and Executable Object Code

• Verification of High-Level Requirements

• Configuration Management

• Quality Assurance

DGTA-ADF

Bounding the Change• Software assurance standard only needs to be

applied to the scope of the modification.

• What is the scope of the modification?- The things that are changed, and- The things affected by the things that are changed.

• Change Impact Analysis:- Traceability- Memory Margin- Timing Margin- Data Flow- Control Flow- Input/Output- Development Environment and Process- Operational Characteristics- Certification Maintenance Requirements- Partitioning Analysis

DGTA-ADF

Traceability• Full traceability data was not recorded for many legacy

software systems.

• Software assurance standards generally require traceability to source code.

• When applying a software assurance standard to a legacy software system, how much traceability data needs to be generated?

• DGTA Position: Trace down and up once.

• Trace down from new or modified high-level requirements to affected and new low-level requirements, to affected and new code.

• Trace up from affected and new code to low-level requirements and to high-level requirements.

DGTA-ADF

Example – MIL-STD-498 to DO-178B• This example will be considered a number

of times.

• A legacy software system developed to MIL-STD-498 with typical artefacts.

• Can cause or contribute to Major hazards.

• Migrating to DO-178B Level C objectives.

DGTA-ADF

HLR1

LLR1 LLR2 LLR3

SCM1 SCM2 SCM3 SCM4

HLR1

LLR1 LLR2 LLR3

SCM1 SCM2 SCM3 SCM4

HLR1

LLR1 LLR2 LLR3

SCM1 SCM2 SCM3 SCM4

HLR1

LLR1 LLR2 LLR3

SCM1 SCM2 SCM3 SCM4

HLR1

LLR1 LLR2 LLR3

SCM1 SCM2 SCM3 SCM4

HLR1

LLR1 LLR2 LLR3

SCM1 SCM2 SCM3 SCM4

HLR1

LLR1 LLR2 LLR3

SCM1 SCM2 SCM3 SCM4

HLR1

LLR1 LLR2 LLR3

SCM1 SCM2 SCM3 SCM4

HLR1

LLR1 LLR2 LLR3

SCM1 SCM2 SCM3 SCM4

HLR1

LLR1 LLR2 LLR3

SCM1 SCM2 SCM3 SCM4

HLR1

LLR1 LLR2 LLR3

SCM1 SCM2 SCM3 SCM4

HLR1

LLR1 LLR2 LLR3

SCM1 SCM2 SCM3 SCM4

HLR1

LLR1 LLR2 LLR3

SCM1 SCM2 SCM3 SCM4

HLR1

LLR1 LLR2 LLR3

SCM1 SCM2 SCM3 SCM4

HLR1

LLR1 LLR2 LLR3

SCM1 SCM2 SCM3 SCM4

HLR1

LLR1 LLR2 LLR3

SCM1 SCM2 SCM3 SCM4

HLR1

LLR1 LLR2 LLR3

SCM1 SCM2 SCM3 SCM4

HLR1

LLR1 LLR2 LLR3

SCM1 SCM2 SCM3 SCM4

HLR1

LLR1 LLR2 LLR3

SCM1 SCM2 SCM3 SCM4

HLR1

LLR1 LLR2 LLR3

SCM1 SCM2 SCM3 SCM4

HLR1

LLR1 LLR2 LLR3

SCM1 SCM2 SCM3 SCM4

HLR1

LLR1 LLR2 LLR3

SCM1 SCM2 SCM3 SCM4

HLR1

LLR1 LLR2 LLR3

SCM1 SCM2 SCM3 SCM4

HLR1

LLR1 LLR2 LLR3

SCM1 SCM2 SCM3 SCM4

HLR1

LLR1 LLR2 LLR3

SCM1 SCM2 SCM3 SCM4

HLR1

LLR1 LLR2 LLR3

SCM1 SCM2 SCM3 SCM4

HLR1

LLR1 LLR2 LLR3

SCM1 SCM2 SCM3 SCM4

HLR1

LLR1 LLR2 LLR3

SCM1 SCM2 SCM3 SCM4

HLR1

LLR1 LLR2 LLR3

SCM1 SCM2 SCM3 SCM4

HLR1

LLR1 LLR2 LLR3

SCM1 SCM2 SCM3 SCM4

HLR1

LLR1 LLR2 LLR3

SCM1 SCM2 SCM3 SCM4

HLR1

LLR1 LLR2 LLR3

SCM1 SCM2 SCM3 SCM4

HLR1

LLR1 LLR2 LLR3

SCM1 SCM2 SCM3 SCM4

HLR1

LLR1 LLR2 LLR3

SCM1 SCM2 SCM3 SCM4

HLR1

LLR1 LLR2 LLR3

SCM1 SCM2 SCM3 SCM4

HLR1

LLR1 LLR2 LLR3

SCM1 SCM2 SCM3 SCM4

Traceability

HLR2

LLR4

HLR1

LLR1 LLR2 LLR3

SCM1 SCM2 SCM3 SCM4 SCM5

New High-Level Requirement

New Low-Level Requirement

ModifiedLow-Level Requirement

New Source CodeModified Source Code

DGTA-ADF

Low-Level Requirements• Some legacy systems have no low-level requirements.

• Others have design descriptions that are not refined enough to be low-level requirements.

• Others have design descriptions that do not drive source code development (source code is developed from requirements).

• How many low-level requirements need to be defined or redefined?

• DGTA Position: Each affected low-level requirement and each low-level requirement identified by the down and up trace.

DGTA-ADF

Low-Level Requirements

HLR1 HLR2

LLR1 LLR2 LLR3 LLR4

SCM1 SCM2 SCM3 SCM4 SCM5

HLR2

LLR4LLR3

SCM2 SCM3 SCM4 SCM5

LLR1

DGTA-ADF

Low-Level Requirements• Verify that low-level requirements LLR1,

LLR3 and LLR4:- Comply with high-level requirements.- Are accurate and consistent.- Conform to standards.- Are traceable to high-level requirements.

• Ensure that low-level requirements LLR1, LLR3 and LLR4 are sufficiently refined to be directly translatable to source code.

- May need to further refine one low-level requirement into several.

• Leave LLR2 as is.

DGTA-ADF

Structural Coverage• Software assurance standards generally require

assessment of structural coverage in order to demonstrate that testing is complete.

• Purpose of Structural Coverage Objectives is to:- Identify shortcoming in requirements based test cases or

procedures,- Identify inadequacies in software requirements,- Identify dead code, and- Identify deactivated code.

• How do these measures apply to modification of legacy software systems?

• DGTA Position:- For requirements based measures: each new, modified or

affected requirement needs to be tested.- For structure based measures: each new or modified source

code module.

DGTA-ADF

Structural Coverage

HLR1 HLR2

LLR1 LLR2 LLR3 LLR4

SCM1 SCM2 SCM3 SCM4 SCM5

HLR2

LLR4LLR3

SCM2 SCM3 SCM4 SCM5

LLR1

HLR1

Statement Coverage

Normal Range and Robustness Tests

Normal Range and Robustness Tests

DGTA-ADF

Structural Coverage• Normal Range and Robustness Tests

- High-level requirements that have been added or modified.

- High-level requirements where a dependent low-level requirement has been added or modified.

- High-level requirements where the implementation (source code) of a dependent low-level requirement has been added or modified.

- Low-level requirements that have been added or modified.

- Low-level requirements where the implementation (source code) has been added or modified.

• Structural Coverage Objectives:- Structural coverage of all new or modified source

code modules.

DGTA-ADF

Additional Testing• Also need to consider:

- Data Dependencies Has the change impacted data that other functions rely

upon?- Control Flow Dependencies

Has the change inadvertently/adversely disrupted control flow or coupling?

- Timing Dependencies Has the change violated a timing constraint?

- Memory Space Dependencies Has the change violated memory constraints or used

memory space assigned to other functions?

DGTA-ADF

What about bug fixes?• A large part of in-service support is the

rectification of bugs.

• Bug fixes may not commence at the requirements level, may start with an identified fault in the source code.

• To what extent should software assurance standards be applied to bug fixes?

• DGTA Position: Trace up from the modified source code.

DGTA-ADF

LLR7LLR7LLR5LLR5

SCM8SCM8

HLR4HLR4HLR3HLR3

What about bug fixes?

LLR6 LLR8

SCM6 SCM7 SCM9 SCM10

Additional traceability data

Structural Coverage

Normal Range and Robustness TestsLow-Level Requirements

DGTA-ADF

Conclusions• DGTA requires the application of a software assurance

standard to legacy software systems.

• All development methods provide a level of assurance, writing it down can identify gaps.

• A software assurance standard can be applied either by:- A Product Service History Argument and application of a recognised

software assurance standard, or- A Software Assurance Task Matrix.

• The software assurance standard need only be applied to the current modification.

- Determine extent through change impact analysis.

• For legacy software systems, a large number of assurance objectives are probably already being achieved.

• Meeting a recognised software assurance standard will probably require additional effort in the areas of:

- Traceability,- Low-Level Requirements, and- Structural Coverage

DGTA-ADF

Questions?