Title: Software Science: How Far Could Mathematics and Rigor Take Us?

Date: November 8, 2017

Researcher Name(s): Lan Lin

University: Ball State University

New Project Proposal Status Report Final Report

Long Term Goal(s)

Modern software development processes for safety- and mission-critical systems rely on

rigorous methods for code development and testing to support dependability claims and

assurance cases [1] that provide the justified and needed confidence. Sequence-based software

specification [2-4] (for requirements analysis and specification development) and Markov chain

usage-based statistical testing (for automated, model-based statistical testing and software

certification) [5-11] are two such methods that since inception have been successfully

combined and used in a variety of industry and government projects (including commercial

products), and have had a large measure of success. Previous collaborators include BNR,

Ericsson, CTI-PET Systems, IBM, Nortel, Oak Ridge National Laboratory [12], Raytheon, Verum

Consultants (in the Netherlands) [13-15], and Fraunhofer Institute for Experimental Software

Engineering (in Germany) [16]. The constructed rigorous specification automatically translates

to a formal model that can be used as a first pass at the state machine for a Markov chain usage

model for statistical testing. This integration defines a workflow and tool chain flowing from the

original, informal, and imperfect requirements, through model-based design and model-based

testing, to the final software certification, enabling a shift of focus up stream – one develops a

precise specification and code that meets the specification, and places in parallel development

and testing activities [5].

With our extensive experience with rigorous software specification and testing over the

last two decades [17-27], and an observation that although software-intensive systems have

become quite large over the years (with systems of 10 million lines of source code now

common), many systems are fielded without benefit of these methods (i.e., exhaustive analysis

of the system’s behavior in all possible scenarios of use, and testing based on a usage profile

that reveals faults in the order of their contribution to reliability, or demonstrates that the

highly likely use paths do not fail), we propose an adapted and augmented process for

developing safety- and mission-critical software-intensive systems. It incorporates our rigorous

specification and testing methods, as well as two other methods for design and testing that

have been widely used and shown great promise. The ultimate goal is to economically produce

zero- or nearly zero-defect safety- and mission-critical software utilizing a combination of these

mathematically sound methods. The resulting specification, design, and testing documentation,

as well as the accompanying quantitative analysis, will provide audit trails of evidence to

support a claim of dependability, and to demonstrate and certify that the system is fit for its

intended use.

Background for Long Term Goals

The proposed research follows the foundations of Cleanroom software engineering originated

with Harlan Mills [28, 29], and the work of Jesse H. Poore and his colleagues at the University of

Tennessee Software Quality Research Laboratory (UTK SQRL) in the past two decades [2-12, 16-

27]. Poore’s work has been focused on developing theory, engineering practices, working

methods and tools to implement Mills’ original concepts, and specifically on two areas: (1)

treating software development from a functional theory point of view, and (2) treating testing

as a problem that can be addressed by statistical science.

As prescribed by Mills [28, 29], every software program implements the mapping rule

for a mathematical function called the black box function. Simply put, the black box function

maps every possible history of system inputs to one and only one system output. Sequence-

based specification [2-5, 21, 22, 24] was developed to facilitate requirements analysis and to

discover this function definition through a systematic and constructive process called sequence

enumeration. The process requires one explicitly consider the software’s behavior in all possible

scenarios of use, and identifies all the control states of the system when it concludes. The result

is a fully documented, complete, consistent, and traceably correct black box specification, from

which one can easily generate a state box specification (or state machine), the code framework,

and much of the code itself [2-5, 24].

Meanwhile, the rigorous specification can also be used to generate a directed graph

that serves as the basis for a usage model for statistical testing [5-12]. One then assigns

probabilities on the arcs that represent the relative frequencies of taking different transitions

from each state, making it a Markov chain usage model. The model depicts the intended use of

the software in the field and represents the population of all possible use cases. Then all kinds

of statistics can be computed routinely from the model, providing a basis for model validation,

revision, and test planning. From the validated model one then generates test cases by walking

the graph, by applying graph algorithms, or by sampling. Test scripts can be associated with arcs

of the usage model which become instructions to manual testers or automated test runners.

Pass and fail data are recorded and analyzed for reliability estimation, coverage analysis, or

stopping decisions. The statistical testing process supports quantitative certification of software

by statistical protocol for standards compliance as well as for the construction and evaluation of

assurance cases for dependable systems.

Public domain tools supporting these rigorous methods are freely available [30-32].

Figure 1 shows the typical workflow combining rigorous specification with automated statistical

testing.

Intermediate Term Objectives

Our goal is to define an augmented process for developing safety- and mission-critical software

that builds on a number of mathematics- and theory-based formal/rigorous methods. In

contrast to the concept of “testing in quality” which is costly and ineffectual, we believe

software quality is achieved in the requirements, specification, architecture, design, code

generation, and coding activities. Our starting point is a precise black box and state box

specification derived from original functional requirements using sequence-based specification.

Then we refine it into a design that can be checked against desirable modularity properties

using conceptual integrity of software systems [33, 34], linear software models and the

modularity matrix [35], and develop code that meets the specification. Finally testing is a

statistical activity to demonstrate that there are no errors, rather than to find errors, and that

the software-intensive product, when released in the field, is fit for its intended use. By model-

based statistical testing, we place the software testing problem in a statistical context and

utilize statistical principles to guide testing strategy and data evaluation. We will also combine

statistical testing with another well-established rigorous testing method, i.e., combinatorial

testing that has proven to have high fault detection rate [36], and generate test suites that not

only target statistical analysis based on the expected software use but also provide good

combinatorial coverage. Figure 2 demonstrates our proposed work flow.

Figure 1. The typical workflow combining rigorous software specification with automated statistical testing

Although the theory for sequence-based specification and statistical testing has been

well established and combining these two rigorous methods was not a new idea, it remains

domain and application specific to integrate them into the existing development process, and

implement full test automation with no human intervention. On the other hand, the integration

of rigorous software specification with linear algebra-based modularity design, and the

integration of statistical testing with combinatorial testing are new research topics to explore

along the path; each could potentially become a major research result in itself. We will follow

the philosophy of theory-practice-tools in our research and exploration: first develop a theory

that has a sound mathematical foundation, then establish engineering practices that reinforce

the theory, and develop tool support that enforces the workflow but hides the mathematics.

Schedule of Major Steps

1. Select a real-world problem as the case study.

2. Derive a state-based specification from functional requirements using sequence-based

specification.

3. Refine the state-based specification into detailed design and implementation taking into

consideration the software architecture.

4. Validate the modular design using linear software models and the modularity matrix for

conceptual integrity.

5. Develop a Markov chain usage model for statistical testing based on the intended use in

the field, and validate the model.

6. Develop a test plan, and an automated test harness.

7. Augment the test suite with combinatorial test cases besides coverage sampling,

random sampling, and weighted sampling.

8. Automatically generate-execute-evaluate a large sample of test cases and record test

results, and project system reliability based on the testing experience.

Figure 2. The proposed workflow for the economical production of high-quality software

Dependencies

Steps 3 – 8 are based on the first two steps. Once a formal specification is derived, design

(Steps 3 – 4) and testing (Steps 5 – 8) could be carried out in parallel.

Major Risks

The selected case study should be small enough to fit the proposed time and budget, and to

make a feasible initial proof-of-concept, but large enough to convince future support in

applying rigorous methods to economically develop safety- and mission-critical software. The

integration of our rigorous methods with other mathematics-based methods might need more

time for the research and implementation. The proposed process needs to be adapted to the

working culture of the sponsoring affiliate.

Budget

Lan Lin, formal methods, mathematical analysis, application: $25,000

Graduate assistant, tool development, application: $25,000

---------------------------------

Total: $50,000

Staffing

Principal investigator: Dr. Lan Lin

One graduate assistant

We will work closely with the software engineers from the sponsoring affiliate and adapt our

methods to their specific process and work culture.

Category of Current Stage

New proposal

Contacts with Affiliates

N/A for the new proposal, however, it is in line with the following previously funded projects

(all through the NSF Security and Software Engineering Research Center):

Combining rigorous specification and testing methodologies to achieve high quality assurance

(single PI), $46,836, Lockheed Martin and Northrop Grumman, 01/01/13 – 12/31/13.

Towards scalable modeling for rigorous software specification and testing (single PI), $50,000,

Rockwell Collins, Air Force Research Laboratory, and Ontario Systems, 11/01/14 – 10/31/15.

Quantifying software quality through rigorous testing and test automation: From theory to

practice (single PI), $30,000, Ontario Systems, 11/01/16 – 10/31/17.

Publications and Other Research Products (actual or potential)

We anticipate 1-2 peer reviewed conference papers and 1-2 peer reviewed journal papers

reporting new research results in suitable venues.

We anticipate the following deliverables:

1. Complete documentation for the case study, including the black box and the state box

specifications, software architecture and modularization, the implemented state box

tables, code, the Markov chain usage model, model analysis, the test oracle, the

automated test harness, test plan, test cases, test scripts, test records, and test case

analysis.

2. Complete documentation for the developed theory that enables the integration of our

methods with other mathematics-based methods, in the form of technical/progress

reports. Released tool that implements the new theory.

References

[1] Jackson D, Thomas M, Millett LI ed., Software for Dependable Systems: Sufficient Evidence?

National Academies Press, 2009.

[2] Prowell SJ, Poore JH, Foundations of sequence-based software specification, IEEE

Transactions on Software Engineering 2003; 29(11): 417 – 429.

[3] Prowell SJ, Trammell CJ, Linger RC, Poore JH, Cleanroom Software Engineering: Technology

and Process, Addison-Wesley, 1999.

[4] Prowell SJ, Poore JH, Sequence-based software specification of deterministic systems,

Software – Practice and Experience 1998; 28(3): 329 – 344.

[5] Poore JH, Theory-practice-tools for automated statistical testing, DoD Software Tech News:

Model-Driven Development January 2010; 12(4): 20 – 24.

[6] Prowell SJ, Poore JH, Computing system reliability using Markov chain usage models, Journal

of Systems and Software 2004; 73(2): 219 – 225.

[7] Poore JH, Trammell CJ, Application of statistical science to testing and evaluating software

intensive systems, Statistics, Testing, and Defense Acquisition: Background Papers (Cohen ML,

Steffey DL, Rolph JE ed.), National Academies Press, 1999.

[8] Poore JH, Trammell CJ, Engineering practices for statistical testing, Crosstalk (DoD Software

Engineering Journal-Newsletter) April 1998; 24 – 28.

[9] Walton GH, Poore JH, Trammell CJ, Statistical testing of software based on a usage model,

Software – Practice and Experience 1995; 25(1): 97 – 108.

[10] Whittaker JA, Thomas MG. A Markov chain model for statistical software testing, IEEE

Transactions on Software Engineering 1994; 30(10): 812 – 824.

[11] Whittaker JA, Poore JH, Markov analysis of software specifications, ACM Transactions on

Software Engineering and Methodologies 1993; 2(1): 93 – 106.

[12] Sayre K, Poore JH, Automated testing of generic computational science libraries,

Proceedings of the 4th Hawaii International Conference on Systems Sciences, Waikoloa, HI,

2007, 277c.

[13] Bouwmeester L, Broadfoot GH, Hopcroft PJ, Compliance test framework, Proceedings of

the 2nd Workshop on Model-Based Testing in Practice, Enscede, The Netherlands, 2009, 97 –

106.

[14] Hopcroft PJ, Broadfoot GH, Combining the box structure development method and CSP for

software development, Electronic Notes in Theoretical Computer Science 2005; 128(6): 127 –

144.

[15] Broadfoot GH, Broadfoot PJ, Academia and industry meet: Some experiences of formal

methods in practice, Proceedings of the 10th Asia-Pacific Software Engineering Conference,

Chiang Mai, Thailand, 2003, 49 – 59.

[16] Bauer T, Beletski T, Boehr F, Eschbach R, Landmann D, Poore JH, From requirements to

statistical testing of embedded systems, Proceedings of the 4th International Workshop on

Software Engineering for Automotive Systems, Minneapolis, MN, 2007, 3 – 9.

[17] Lin L, Xue Y, Song F, A simpler and more direct derivation of system reliability using Markov

chain usage models, Proceedings of the 29th International Conference on Software Engineering

and Knowledge Engineering, Pittsburgh, PA, 2017, 462 – 466.

[18] Lin L, Scalable modeling for rigorous software specification and testing, 2016 Compendium

of Industry-Nominated NSF I/UCRC Technological Breakthroughs, 2016, 107 – 108.

[19] Lin L, Y. Xue, F. Song, An algorithm for forward reduction in sequence-based software

specification, International Journal of Software Engineering and Knowledge Engineering (Special

Issue on Best Papers from SEKE 2016) 2016; 9 & 10: 1431 – 1451.

[20] Lin L, He J, Xue Y, An automated testing framework for statistical testing of GUI

applications, Proceedings of the 27th International Conference on Software Engineering and

Knowledge Engineering, Pittsburgh, PA, 2015, 72 – 79.

[21] Eschbach R, Lin L, Poore, JH, Applying string-rewriting to sequence-based specification,

Formal Methods in System Design 2013; 43(3): 414 – 449.

[22] Lin L, Poore JH, Eschbach R, Hierons RM, Robinson-Mallett C, Augmenting sequence

enumeration with string-rewriting for requirements analysis and behavioral specification,

Proceedings of the 16th International Conference on Fundamental Approaches to Software

Engineering, Rome, Italy, Lecture Notes in Computer Science Volume 7793, 2013, 179 – 193.

[23] Poore JH, Lin L, Eschbach R, Bauer T, Automated statistical testing for embedded systems,

Model-Based Testing for Embedded Systems in the series on Computational Analysis, Synthesis,

and Design of Dynamic Systems (Zander J, Schieferdecker I, Mosterman PJ ed.), CRC Press –

Taylor & Francis, 2011.

[24]Lin L, Prowell SJ, Poore JH, An axiom system for sequence-based specification, Theoretical

Computer Science 2010; 411(2): 360 – 376.

[25] Lin L, Prowell SJ, Poore JH, The impact of requirements changes on specifications and state

machines, Software – Practice and Experience 2009; 39(6): 573 – 610.

[26] Lin L, Carter JM, Poore JH, Using state machines to model and manage requirements

changes and specification changes, Proceedings of the 51st IEEE International Midwest

Symposium on Circuits and Systems, Knoxville, TN, 2008, 523 – 526.

[27] Carter JM, Lin L, Poore JH, Automated functional testing of Simulink control models,

Proceedings of the 1st Workshop on Model-Based Testing in Practice, Berlin, Germany, 2008, 41

– 50.

[28] Mills HD, Dyer M, Linger RC, Cleanroom software engineering, IEEE Software September

1987; 19 – 24.

[29] Mills HD, The new math of computer programming, Communications of the ACM 1975;

18(1): 43 – 48.

[30] Protoseq, Prototype Sequence Enumeration Tool, Software Quality Research Laboratory,

The University of Tennessee, http://sourceforge.net/projects/protoseq, 2017.

[31]REALSBS, Requirements Elicitation and Analysis Sequence-Based Specification Tool,

Software Research Laboratory, The University of Tennessee,

http://sourceforge.net/projects/realsbs, 2017.

[32] J Usage Model Builder Library (JUMBL), Software Quality Research Laboratory, The

University of Tennessee, http://jumbl.sourceforge.net/jumblTop.html, 2017.

[33] Exman I, Conceptual integrity of software systems: Architecture, abstraction and algebra,

Proceedings of the 29th International Conference on Software Engineering and Knowledge

Engineering 2017, Pittsburgh, PA, 2017, 416 – 421.

[34] Exman I, Katz P, Conceptual Software Design: Algebraic axioms for conceptual integrity,

Proceedings of the 29th International Conference on Software Engineering and Knowledge

Engineering 2017, Pittsburgh, PA, 2017, 155 – 160.

[35] Exman I, Linear software models: Standard modularity highlights residual coupling,

International Journal of Software Engineering and Knowledge Engineering 2014; 24(2): 183 –

210.

[36] Kuhn DR, Kacker RN, Lei Y, Practical combinatorial testing, National Institute of Standards

and Technology Special Publication 800-142, October 2010.