Automated Testing of NASA Software

CESE

Automated Testing of Large Multi-language SW Systems using Cloud

Computing

Executive Status Report

Principal Investigator (PI): Dr. Mikael Lindvall, CESENASA POC: Markland Benson, White Sands

Team members: Dharma Ganesan, Dr. Chris Ackermann(CESE)

GMSEC, CFS, Space Network, MSL

1© 2011 Fraunhofer USA, Inc. Center for Experimental Software Engineering

CESE

Motivation• Bugs can lead to deaths, injuries, or financial

loss• Software testing is necessary to find the

presence of bugs• Software testing consumes 50% - 75% of the

development effort• Many NASA projects could benefit from

automated test generation• In fact, several skeptical engineers noted that

the proposed work is illuminating!

© 2011 Fraunhofer USA, Inc. Center for Experimental Software Engineering

CESE

Technical Context• The system(s) under test (SUT) are complex• Multiple programming languages

– Ada, C, C++, Fortran, Java• Distributed legacy systems

– More than 20 yrs old– Runs on multiple machines and involves various

OSes,e.g. OpenVMS• GUI intensive

– Web, Java Swing, AWT, X Windows, Motif


CESE

Problems

• Test cases are developed manually• Some test execution is automated (e.g., JUnit)• Test cases miss valid “corner” cases• Test cases are also programs: not easy for non-

technical stakeholders to understand• Difficult to summarize what was tested

Approach: Lightweight Model-based Test Generation and Execution


CESE

Approach

• Develop a model of the system under test

• The model contains actions and expected results

• Automatically derive test cases from the model

• Execute the test cases

• Decide when to stop testing based on model coverage!


CESE

An Example Model

public interface IConnection{

public Status Connect();

public Status Disconnect();}

APIs of the module under test


CESE

Currently Targeted Projects• GMSEC – Reusable framework for ground systems

– Modeled the Core API and Applications– Generated executable test cases from the model– Confirmed defects/violations reported and fixed

• Core Flight Software – Reusable framework for flight systems– Modeled the OS abstraction layer (OSAL)– Generated executable test cases from the model– Confirmed defects/violations reported and fixed– Test cases delivered to the CFS team


CESE

Currently Targeted Projects• Space Network – White Sands

– Evaluating different test execution tools suitable for Open VMS environment

– E.g. Image-based test execution tools: EggPlant and Sikuli

– Challenges in this project:• SUT not possible to run at Fraunhofer• Must visit Goddard with an escort (secured building)• Not possible to connect to Internet


CESE

Currently Targeted Projects• Mars Science Laboratory (MSL) – JPL

– Can current test cases be modeled and used to generate new test cases?• New set cover scenarios not previously covered?• New approach more cost effective?

– Challenges in this project:• SUT not possible to run at Fraunhofer• Software is sensitive (ITAR)


CESE

Current Results• An end-to-end approach for test generation

– Successfully used on the GMSEC and the CFS and detected bugs

• Approach works well on different levels:– API (Module interface) level testing– GUI testing

• Approach transferred to an intern at GSFC– The intern found new bugs using the approach– Demonstrates the approach is lightweight


CESE

Summary and Next Steps• We’re building a new approach that

– Helps in test automation using lightweight models

• Prototype approach successfully applied to GMSEC and CFS/cFE

• Next steps are to apply to the Space Network and the MSL projects

• Examples and technical details in technical presentation!


Technology

Automated Testing of NASA Software