1

IV&V Facility

IV&V Status for THEMISOctober 26, 2006

Judi Connelly, IV&V Project Manager

2

IV&V Facility

Overview

• Delta status Bus Avionics Unit (BAU) Risks 1001 & 1002

• Delta status for BAU High Severity Technical Issues of Memoranda (TIMs ) and Overall BAU TIMs Status

• Summary of Instrument Data Processing Unit (IDPU) Analysis

• Ground System (GS) Analysis Status

• Backup Slides

3

IV&V Facility

THEMIS BAU Risks

Risk 1001: Configuration Management – Closed May 06

Risk 1002: BAU Testing Delta status: Closed Oct 06 The original driver for this risk has been mitigated. To date, the Project has mitigated

the technical aspect of this risk by resolving Testing TIMs. The Project will continue to address the majority of BAU testing TIMs over the next weeks. There no current concerns with cost and schedule aspects of risk.

• IV&V will continue to determine via objective evidence that remaining BAU FSW requirements have been adequately tested.

• Boot TIMs associated with this risk have either been resolved to closure or have a disposition to Project Accepts Risk (PAR).

4

IV&V FacilitySeverity 1 TIMs : None openSeverity 2 TIMs

TIM 1357: Incorrectly Deleted Requirement: ACS Req. 3.2.1.8.11 as a Trace to FSW.AC.09 Delta Status: ClosedTIM 1537: Lock flaw in function DS_clear_bulk : Critical Function: Data Storage

TIM 1536: Lock flaw in function DS_three_copy: Critical Function: Data Storage

TIM 1535: Memory-scrub lock management design flaws: Critical Function: Checksumming

Delta Status: Final disposition is reduced to Sev 3 & Project Accepts Risk

• IV&V continues to have limited concern on these 3 issues; however, the problem of memory management described is unlikely to occur. •If the problem does occur, there remains concern that if reboot does not correct the semaphore locking then recovery may be limited with little diagnostic information available. •The Project workaround to reboot is considered standard practice if such a problem manifests. IV&V considers this as sufficient to reduce these TIMs to Sev 3 from Sev 2 with disposition to Project Accept Risk state.

•Project accepts the risk, if any, associated with these TIMs.

BAU TIMs Status

5

IV&V Facility

BAU TIMs Summary

1 2 3 4 5Closed 280 1 13 222 30 14Project Accepts Risk(PAR) 32 0 0 29 3 0Submitted 103 0 0 87 16 0To Be Verified 27 0 0 27 0 0

Total 442 1 13 365 49 14

State Total Severity

Submitted: 16 new TIMs left to be submitted per 10 TIMs/ week

PAR Sev 3: PAR Sev 4: 17 = Boot Requirements 1 = BAU Test 1 = Boot Code 1 = BAU Requirement 2 = BAU Requirements 1 = Boot Requirement 3 = BAU Design 3 = BAU Code 3 = BAU Test

All BAU Analysis will be completed by 11/3/06.

The Project has been consistent in addressing these TIMs.

6

IV&V Facility

IDPU IV&V Analysis Summary

• IDPU Analysis was completed August 06.

• Overall, the code was determined to operate in the manner described in the Requirements & Design documentation.

• All coding concerns were addressed by the final version of code.

• There are no unresolved TIMs leveraged on the IDPU Flight Software. Of a total 150 IDPU TIMs none remain open.

• IV&V team also found that software development practices by the IDPU development team were robust.

• IV&V has no concerns regarding the operational readiness of the IDPU software.

7

IV&V Facility

Ground System IV&V Analysis Interface Requirements DocumentsPreliminary – 10/4/06 -reviewingDraft – 11/3/06 – will review; to be completed by 11/15Final – 12/8/06

Mission ReadinessTest Plans – reviewing

Mission Ops Plan Prelim – 10/4/6 – reviewingDraft – 11/3/06Final – 12/8/06

Flight Operations PlanPrelim – 10/4/6 – reviewingDraft – 11/3/06 Final – 12/8/06

Flight Operations ProceduresPrelim – 10/4/06 – reviewingDraft – 11/17/06Final – 12/15/06

With the exception of ICD all GS IV&V analysis will be completed by 11/7/06.

8

IV&V Facility

Summary

– Ongoing work with project on BAU TIMs continues to progress well. Current processes & communications are sufficient to address remaining IV&V concerns if the Project developer continues to address a minimum of 10 TIMs each week.

– While there remains IV&V concern on memory management in TIMs 1535-37, IV&V cannot verify the problem case will occur.

If the problem does occur, there remains concern that if reboot does not correct the semaphore locking then recovery may be limited with little diagnostic information available.

The Project workaround to reboot is considered standard practice if such a problem manifests. IV&V considers this as sufficient to reduce these TIMs to Sev 3 from Sev 2 with disposition to Project Accept Risk state.

– There are no unresolved TIMs leveraged on the IDPU Flight Software. IV&V has no concerns regarding the operational readiness of the IDPU software.

– Ground System analysis will be performed on listed artifacts. Analysis & Reporting will be completed by mid-December.

9

IV&V Facility

Backup Slides

10

IV&V Facility

Requirements AnalysisSystem and software requirements are verified to be, for example, complete, consistent, traceable and testable.

Software Design AnalysisSoftware design models and algorithms are, for example, checked to provide implementation of associated requirements and for handling off-nominal functionality.

Software Code AnalysisCode is verified that it is free of implementation errors and that it fulfills the requirements. Tools used, for example, Beyond Compare and Understand.

Test Program AnalysisTest artifacts are verified to cover all requirements levied on the software.

• Weekly meetings with Project are held to work through Technical Issues of Memoranda (TIMs). Other regular meetings attended include Spacecraft, Systems and Mission Ops.

• Work is driven by receipt of artifacts. Assessment of artifacts is tracked through TIMs and Risks.

• Phase Completion Reports are delivered to Mission Manager, Project Manager, IV&V Liaison

• Project Risks are reported to the Project first in written draft through weekly meetings, then formally in Monthly Software Status Report

IV&V for THEMIS

11

IV&V FacilityIV&V applies the following criteria

– Correct – accurately represents stakeholders’ needs– Complete – no unspecified or incomplete requirements– Consistent – no conflicts or incompatibilities between requirements.– Sponsored – traceable to system or user requirements– Testable – defined in terms of explicit, verifiable criteria; atomic requirements – Understandable – comprehensible to all stakeholders and self supporting;

proper use of standard terminology; definitions provided for all domain-specific terminology

– Precise – free from ambiguity– Design Independent – define what the system is to do, not how to do it; avoids

specifying requirements in terms of design elements – Organized – grouping of related requirements; partitioning of unrelated

requirements– Traced – each requirement is traced to an appropriate element of design

and/or test procedure

Requirements Analysis in General

12

IV&V Facility• IV&V verifies that the design specification

– Adequately provides for correct and complete implementation of the associated requirements

– Provides enough detail for a developer to correctly implement the design– Correctly accounts for all constraints levied upon the software by the system

design

• IV&V confirms that the design specification partitions the CSCI into appropriate design entities, defines the key relationships between entities, and provides the following attributes for all design entities

– A unique identifier, entity type, and purpose/requirements fulfilled– Function definition and interface definition– Subordinates, dependencies, resources used, and internal data

• In addition, IV&V verifies – Function – functionality provided by design entities will satisfy the applicable

requirements– Timing relationships – execution frequency meets needs, I/O homogeneity,

appropriate tasking priorities, and protected data sharing– Off-nominal functionality – design correctly handles each potential off-nominal

scenario– Interactions – interactions between design entities, including proper use of the

interfaces and the absence of non-deterministic behaviors and race conditions– Design quality – impacts maintainability and verifiability of the design

Design Analysis in General

13

IV&V Facility• IV&V analyzes source code

– Verify a correct implementation of design entities fulfilling all requirements – Verify that code is free of implementation errors

• Key code analysis objectives include– Verify that software will produce the intended result for combinations of

inputs and conditions • Confirm logic and algorithms are correct• Confirm data types and semantics are consistent and free from

errors across interfaces, function calls, computations, and assignments

• Identify cases of unintended functions or side effects– Verify that results are reliable/repeatable

• Investigate for race conditions and nondeterministic behaviors under a dynamic tasking environment

• Confirm data transmission timing, protection of shared resources, and the absence of type mismatches

– Confirm error handling is present and correctly detects and handles• Out-of-range, late, or missing inputs• Task overruns and hardware exceptions• Violations of preconditions, such as prerequisite hardware or

software states priority

Code Analysis in General

14

IV&V Facility• IV&V analysis of developer test programs is most often applied to software

requirements verification testing (a.k.a. Formal Qualification Testing)– In addition, IV&V often reviews integration or system level testing– Unit level tests are rarely reviewed due to resource constraints– Analysis often covers test plans, descriptions, procedures, scripts, and

results

• The primary objectives of IV&V test analysis are to ensure – Tests cover all requirements levied on the software– Tests fully exercise all critical functionality required of the software

• Key problem areas include– Coverage of requirements, especially in cases where requirements are

subject to change– Negative predicate testing (implicit converse scenarios)– Response to fault injection and off nominal conditions– Stress tests and extended duration tests– Negative interactions between components or software states– Regression testing

Test Analysis in General

15

IV&V Facility

IV&V Facility Severity definitions

1 a) Prevent the accomplishment of an essential capability b) Jeopardize safety, security, or other requirement designated critical.

2 a) Adversely affect the accomplishment of an essential capability and no work-around solution is known b) Adversely affect technical, cost or schedule risks to the project or life cycle support of the system, and no work-around solution is known

3 a) Adversely affect the accomplishment of an essential capability but a work-around solution is known b) Adversely affect technical, cost, or schedule risks to the project or life cycle support of the system, but a work-around solution is known

4 a) Result in user/operator inconvenience but does not affect a required operational or mission essential capability b) Result in inconvenience for development or maintenance personnel, but does not affect the accomplishment of these responsibilities

5Any other affect

16

IV&V Facility

Artifacts Reviewed

BAURequirements• FSW SRS Revs 1.1, 2.0, 2.2, 3.0 • Flight Software User’s Guide v1.0 • Boot SRS Revs 1.0, 2.1, 2.2

Design• BAU CDR Presentation (6/15/04)

Code• FSW Build 2• FSW Build 2.504• FSW Build 3.003• FSW Build 3.1 • FSW Build 3.19• Boot Build 0• Boot 2.510

Test• BAU FSW Build 2• BAU FSW Build 3.000/3.003/3.10 • BAU FSW Build 2.504 CPT• BAU FSW Test Plan v1.0

IDPURequirements• SRS Revs D, E, F

Design• THEMIS IDPU FSW Design document

Code• IDPU FSW Phase 1.04• IDPU FSW Phase 2.01• IDPU FSW Phase 3.03• IDPU FSW Phase 4

Test• IDPU CPT Plan

Ground• None

SystemRequirements• MRD Revs C, D, E, F, G, H

Below is a list of THEMIS artifacts which IV&V has reviewed or is currently reviewing.

17

IV&V Facility

IV&V Analysis Methods

Splin

t U

nder

stan

d Po

lysp

ace

Klo

kwor

k In

spec

tB

eyon

d C

ompa

re

McC

abe/

Hal

stea

d M

anua

l

BAUFSW Build 2 X X X XFSW Build 2.504 X X XFSW Build 3.003 X X X XFSW Build 3.1 X X X XBSW v0 X X X XBSW v2.51 X X

IDPUPhase 2 XPhase 3 XPhase 4 X

Ground SystemAttitude Determination XManeuver Planning X

18

IV&V Facility

Tool Descriptions

• PolySpace is a code analysis tool which utilizes Abstract Interpretation to provide exhaustive, automatic checking of the following run-time errors in C, C++, and Ada code.

– Attempt to read a non-initialized variable – Access conflicts for unprotected shared data in multi threaded

applications (if threading model is configured)– Referencing through null pointers – Out-of-bounds array access – Out-of-bounds pointers – Illegal type conversion (long to short, float to integer) – Invalid arithmetic operations (e.g. division by zero or the

square root of a negative number) – Overflow / underflow of arithmetic operations for integers and

floating point numbers – Unreachable code

19

IV&V Facility

Tool Descriptions

• Klokwork Inspect is a static code analysis tool which will identify the following errors in C and C++ code

– Array bounds violation– Assignment in condition– Buffer overflows– No-effect statements (i.e. k<j;)– Invalid memory de-allocations– Non-void functions not returning a value– Void functions returning a value – Return a reference to a local variable– Unused (labels, parameters, variables)– Memory leak– Null pointer– Semi-Colon misplaced for (int i=0; i<100; i++);– Using freeing memory (dereferencing, freeing, passing)– Uninitialized variables– Unreachable code

20

IV&V Facility

Tool Descriptions cont’d

• splint is a C/C++ static source code analyzer that detects coding errors and code constructs that are frequently associated with coding errors. It is particularly useful for detecting mixed-type operations and non-portable constructs

• Understand for C/C++ is an interactive C/C++ source code analyzer that generates dynamic cross-reference reports of code components.  Understand can detect language errors.  The tool uses an extensive data dictionary to underpin its interactive analysis of source code structure and its determination of source components relationships. 

• Beyond Compare is a utility tool for comparing things like text files, folders, zip archives, FTP sites, etc. Software developers and testers can use it to manage source code, keep folders in sync, compare program output, and validate CD copies. IV&V uses its ability to publish meticulously detailed and dynamic comparison reports between source code components of Build Releases as part of the code implementation analysis phase.

All of the analysis tools, especially if deployed in scripted mode, produce false positives under normal language and filter settings.  Manual analysis, guided by mature engineering judgment, is required to determine actual errors and significant issues in the extensive reports generated.