1

Projects in Software TestingSoftware Testing, ETS 200Programvarutestning, 5p.

Version 1.3, 2007, Lp Vt2Carina Andersson, Telecom, LTH

http://serg.telecom.lth.se/education/SwTest/

1. Introduction

This document gives the practical details regarding the project in the course Software Testing.The project in the course is equivalent to 2 weeks of full time studies (2 credit points). Themain objective of the project is to gain a deeper understanding of a specific area within verifi-cation and validation (V&V) of software products. A project group consists of 4-5 persons,who perform the project together. All project members should be involved and the total effortshould be evenly distributed among participants.

2. Learning Objectives

The objective of the project is to learn a specific area of software testing. Furthermore, thestructure of the project makes it possible to learn important parts that will be practiced lateron in the education. Hence, the main learning points of the project are:

• Learn a specific area of software testing

• Collect and summarize research information

• Critical thinking beyond the written information

• Present information in a structured way

There are several types of projects to choose among. Examples of project types are:

• Research: solve a research problem; survey the state-of-the-art and identify the research problems in some area; develop and justify an extension to an existing technique; etc.

• Evaluation: apply and evaluate a technique or evaluate a commercial testing or, analysis tool.

• Practical: Use an existing technique to test a system or design and implement a prototype for a system.

2

3. Activities

The main activities in the project are:

1. Decide on a subject. There are some subjects specified in this document. Other sub-jects may be chosen, but has to be decided together with your project supervisor.

2. Find literature. In each subject area, there is a suggestion of research reports to startreading. Other research literature can be found on http://elin.lub.lu.se (within thedomain of LU, or by using your STIL password). For research projects, 7-10 reportsshould be chosen. For evaluation and practical projects, fewer reports are needed.

3. Outline the report. Specify heading one and two and write in short sentences whatwill be included in the sections.

4. Read; perform evaluation and practical work. Read and understand the new area ofsoftware testing; perform evaluation and practical work if these kind of projects havebeen chosen.

5. Write the report. The report is preferable written in English. The report shall be writ-ten in the IEEE template, see the homepage of the course.

6. Present the report. Communicate the area to other people who have knowledge ofsoftware testing, but not the specific area. This is done during the presentation session.You have about 15 minutes to describe your work. The presentation should either be inSwedish or English.

3

4. Schedule

5. Assessment

See the course program.

6. Report

The report shall be written using the IEEE template. The report shall consist of:

• Abstract: summary of all parts in the report. The purpose of the abstract is to attract peo-ple to read your report.

• Introduction: introduce the chosen area. The purpose is to give an introduction to the reader who is not familiar to the specific area, but knows software engineering and testing generally.

• Description of the chosen area: summary of the chosen area. The purpose is to describe why, how and for which purpose the area is used.

• Analysis: analysis, critical thinking and future plans within the chosen area. For research projects, the purpose is to describe the benefits, drawbacks, what research the area needs in the future. For evaluation and practical projects, the work should be described here together with a shorter analysis.

• Conclusion: main conclusion in the report. The purpose is to discuss the main points in the report again.

Another structure can be chosen after discussion with your project supervisor.

Week Activities Hand in to supervisor Deadline

11 Decide on a subject Decided subject Thu 15/3

Search literature

12-13 Search literature

Read literature

Outline the report Outline Tue 27/3

16-17 Read literature

Write report

Meeting with supervisor Book time

18 Write report

19 Report to supervisor Tue 8/5

Prepare presentation

20 Presentation Session Tue 15/5

Update report If needed, updated report to supervisor Wed 23/5

4

7. Presentation

The project will be presented in the project presentation sessions at the end of the course.Each project will be given 15 minutes to present their chosen area. The presentation shouldcover the main points of summary, analysis and conclusion.

8. Project areas

There are several types of projects to choose among. For research projects, 7-10 reports shouldbe chosen. For evaluation and practical projects, fewer reports are needed. Examples of projecttypes are:

• Research: solve a research problem; survey the state-of-the-art and identify the research problems in some area; develop and justify an extension to an existing technique; etc.

• Evaluation: apply and evaluate a technique or evaluate a commercial testing or, analysis tool.

• Practical: Use an existing technique to test a system or design and implement a prototype for a system. If you would like to choose another area, describe the area to your project supervisor

before starting the work. The keywords in Section 9 can be used when you chose an area andwhen you search for literature.

8.1 How do you know your test cases are correct?Test cases are developed to check whether a software system is implemented correctly accord-ing to the requirements specification. However, if the test cases are badly chosen, they will notdetect the failures. There is a method, called Mutation testing, which can be used to checkwhether the test cases are correct.

Mutation Testing: A. M. R. Vincenzi, J. C. Maldonado, E. F. Barbosa and M. E. Dela-maro, “Unit and Integration Testing Strategies for C Programs using Mutation”, SoftwareTesting, Verification and Reliability, 11(3):249–268, 2001.

8.2 How can you measure that the software system works?Software testing is performed to detect the failures in the software. There are different met-

rics to use in order to measure whether the software is correct. One such measure is reliability,which is defined as: “The probability for a failure-free operation of a program for a specifiedtime under a specified set of operating conditions.” (IEEE 610.12-1990). There are severalmodels and techniques that can be used, for example, reliability growth models, Markovmodels, statistical usage testing, usage-based testing and operational profile testing.

Reliability growth models: C. Stringfellow, A. A. Andrews, “An Empirical Method forSelecting Software Reliability Growth Models”, Empirical Software Engineering, 7(4): 319-343, 2002.

Markov models: J. A. Whittaker and M. G. Thomason, “A Markov Chain Model for Sta-tistical Software Testing”, IEEE Transactions on Software Engineering, 20(10):812-824, 1994.

8.3 What techniques are most effective to check the requirements and design documents?

Static verification is often used in the beginning of the development of software. There areseveral different techniques and methods used to check the static representation. The com-

5

mon feature of these is that they have to be manually checked by reviewers. Software inspec-tion, reviews, walkthroughs are common techniques, which are used together with readingtechniques.

Software inspection and reading techniques: A. Aybuke, H. Petersson, C. Wohlin, “State-of-the-art: Software Inspections after 25 Years”, Software Testing, Verification and Reliability,12(3):133-154, 2002.

8.4 How can you perform testing on an object-oriented software system?A special branch of software testing is application-based testing. Object-oriented is one

such application, which needs special treatment and special methods to test the system. Object-oriented testing: Y. Labiche, P. Thévenod-Fosse, H. Waeselynck, M.-H. Durand,

“Testing Levels for Object-Oriented Software”, Proceedings of the 22nd International Confer-ence on Software Engineering, pp. 136-145, 2000.

8.5 How can you perform testing on a distributed system?A special branch of software testing is application-based testing. Testing distributed sys-

tems need special treatment and special methods. Testing of distributed systems: S. Goeschl and H. M. Sneed, “Case study of testing a dis-

tributed internet-system”, Software Testing, Verification and Reliability, 12(2):77 - 92, 2002.

8.6 What techniques can be used to know when to stop testing?Stopping criteria are used in the testing phase to determine when a certain quality level has

been achieved. The quality can for example be defined as the reliability or just the number offaults left in the system. There are several techniques to use as a stopping criterion. Onecould, for example, estimate the number of faults left after an inspection or to estimate thenumber failures left in testing.

Fault content estimation; L. C. Briand, K. E. Emam, B. G. Freimut, O. Laitenberger, “AComprehensive Evaluation of Capture-recapture Models for Estimating Software DefectContent”, IEEE Transactions on Software Engineering, 26(6):518-540, 2000.

Stopping criterion in testing: J. D. Musa and A. F. Ackerman, “Quantifying Software Vali-dation: When to Stop Testing?”, IEEE Software, 6(3):19-27, 1989.

8.7 How is testing performed for Agile processes?Test-first is a principle that is often used in Agile processes. The principle is based on that

test cases are first derived and then these are used to specify how the system should work aswell as used as test cases when the system is developed. There are some questions, which are ofinterest within this area, for example, what is the relation between requirements and test cases,what is the final quality of such a system and how easy is it to maintain such a system?

Test-first principle: M. M. Muller, and O. Hagner, “Experiment about Test-first Program-ming”, IEE Proceedings Software, 149(5):131-136, 2002.

8.8 How can different testing techniques (methods) be compared?One research methodology is to use empirical methods in order to evaluate which testing

methods are best to use. The methodologies are often divided into experiments, surveys andcases studies. All these methods are important in order to help software organizations tochoose the right testing technique for their purpose.

6

Experiment: Sun Sup So, Sung Deok Cha, Timothy J. Shimeall and Yong Rae Kwon, “AnEmpirical Evaluation of Six Methods to Detect Faults in Software”, Software Testing, Verifica-tion and Reliability, 12(3):155–171, 2002.

Survey: P. Runeson, C. Andersson and M. Höst, “Test Processes in Software Product Evo-lution - A Qualitative Survey on the State of Practice”, Journal of Software Maintenance andEvolution, 15(1):41-59, 2003.

Case study: T. Berling and T. Thelin, “An Industrial Case Study of the Verification andValidation Activities”, International Software Metrics Symposium, pp. 226-238, 2003.

8.9 What quality models are used for the testing purpose?There are many general quality models developed within software engineering, and spe-

cific models for testing. The quality models try to capture the essential parts of software devel-opment (testing) so the final quality will be assured. Some of the general models are CMM,SPICE, ISO 9000, and some of the specific models are TPI and TIM.

CMM: M. Paulk, B. Curtis, M.B. Chrissis, C.V. Weber, “Capability Maturity Model, Ver-sion 1.1”, IEEE Software, pp. 18-27, 1993.

SPICE: J-M. Simon, “SPICE: Overview for software process improvement”, Journal of Sys-tems Architecture, 42(8):633-641, 1996.

TIM: T. Ericson, A. Subotic, S. Ursing, “TIM – A Test Improvement Model”, SoftwareTesting, Verification and Reliability, 7(4):229-246, 1997.

8.10 How is the software of a nuclear plant tested?Safety is concerned with the property of a system that it will endanger human life or the

environment. A safety-critical system is, hence, one by which the safety of equipment or plantis assured. An example of such a system is an aeroplane or nuclear plant system. Safety-criticalsoftware needs special attention when it comes to testing.

Safety: N. G. Leveson, S. S. Cha and T. J. Shimeall, “Safety Verification of ADA Programsusing Software Fault Trees”, IEEE Software, 8(4):48-59, 1991.

8.11 How can test cases be derived and generated?Software test cases are often derived from documents produced during the software devel-

opment. There are different techniques and languages that can be used. Use cases can be usedto derived test cases and two methods that use this approach are usage-based testing and oper-ation profile testing. Furthermore, there are techniques that are automated and use UML dia-grams.

Usage-based testing: B. Regnell, P. Runeson and C. Wohlin, “Towards Integration of UseCase Modelling and Usage-Based Testing”, Journal of Systems and Software, 50(2):117-130,2000.

Operational profile testing: J. Musa, “Operational Profiles in Software-Reliability Engi-neering”, IEEE Software, 10:(2):14-32, 1993.

Specification-based testing: J. Offutt, S. Liu, A. Abdurazik and P. Ammann, “GeneratingTest Data from State-based Specifications”, Software Testing, Verification and Reliability,13(1):25–53, 2003.

8.12 How can process simulation be used for project planning and learn-ing?

Software process simulation is used in a variety of fields in software engineering, for exam-ple, to support process improvement, software project management training, decision support

7

and to understand empirical research results. This project can either be practical by focusingon developing a simulation model of a software development/testing project with the puposeto assist in the project planning, or it could be focused on evaluating existing models’ useful-ness in the decision-making.

Simulation models for decision support: Donzelli, P., “A Decision Support System forSoftware Project Management”, IEEE Software, 23(4): 67-75, 2006.

Why using process simulation models: Kellner, M., Madachy, R., Raffo, D., "SoftwareProcess Simulation Modeling: Why? What? How?", Journal of Systems and Software, 46(2-3):91-105, 1999.

8.13 How can tools for software testing be evaluated?Tools are important in order to implement software testing effectively in an software

organization. However, although tools are needed, they do not solve the problems in the test-ing phase. Several tools exist and they have to be evaluated before a software organisation pur-chases one.

Evaluation of testing tools: Poston, R. M. and Sexton, M. P., “Evaluating and SelectingTesting Tools”, IEEE Software, pp. 33-42, 1992.

8.14 How can predictions of the software development project be useful during the testing process?

The project management is often interested in predicting the outcome of the ongoing soft-ware development projects. Both predictions of how much resources that will be required todeliver on time, and the quality of the delivered software product are valuable for the plan-ning process. What metrics are useful for predictions during the test process, and how canthese be used? One model to use for the whole development project is COCOMO, but otherapproaches could be applied when focusing on the test process.

COCOMO and other models: Boehm, B., Abts, C., Chulani, S., “Software DevelopmentCost Estimation Approaches – A Survey”, Annals of Software Engineering, 10(1-4):177-205,2000.

What is needed for making measurements: Schneidewind, N. E., “Body of Knowledge forSoftware Quality Measurement”, IEEE Computer, 35(2):77-83, 2002.

8.15 How can web applications be tested?The quality of web applications mostly relies on the skill of the individual developer. In

order to meet the quality decided by end-users, the processes of developing and testing webapplications need to be formalized.

Web application testing: Di Lucca G. and Fasolino, A., “Testing Web-based Applications:The State of the Art and Future Trends”, Information and Software Technology, 48(12):1172-1186, 2006.

8.16 How do you effectively regression test your system?Regression testing is the process of validating modified software to detect whether new

errors have been introduced into previously tested code. Because of time and resource con-traints for testing, regression test selection techniques have been proposed, to reduce theexpenses.

Regression testing: Rothermel, G. and Harrold, M.J., “Analyzing regression test selectiontechniques”, IEEE Transaction on Software Engineering, 22(8):529-551, 1996.

8

9. Keywords in Software Testing

The below keywords can be used if you want to decide on a subject not specified above.Another source that is appropriate to use is the Software Engineering Book of Knowledge.The latest version can be downloaded on http://www.swebok.org.

Black-box testing Gray-box testing White-box testing

Equivalence partitioningBoundary value analysisDecision tableFinite-state machine-basedTesting from formal specificationError guessingRandom testingOperational profileSRET

Reference models from code-based testing (flow-graph, call-graph)Control flow-based criteriaData-flow based criteriaMutation testingCoverage measures

Application testing Testing in Software Process Test Methods

Object-oriented testingComponent-basedWeb-basedGUI testingTesting of concurrent programsProtocol conformance testingTesting of distributed systemsTesting of real-time systemsTesting of scientific software

Connection between require-ments and testingArchitectureTestability

Statistical usage testing (usage-based)Specification-based software testingCode-based, Fault-basedTesting from formal specifica-tionRandom testing

Objectives of Testing Evaluation of the testingTesting metrics and measure-ments

Acceptance testingInstallationAlpha, betaConformance, functional, correct-ness RegressionPerformanceStressBack-to-backRecoveryConfigurationUsability

Coverage measuresFault seedingMutation scoreComparison and relative effec-tiveness of different techniquesStopping criteria

Attitudes and egoless program-mingTest processTest documentation and test-waresTest organisation vs. company SizeCost/effort estimation and other metricsTermination and stop criterionTest reuse and test patterns

Estimations in software testingEmpirical methods in software testing

Quality models and software testing

ReliabilityReliability growth models, types and classification of faultsRemaining faults and faults density

Compare testing techniquesCompare inspection and testing

CMMSPICETesting with SQATesting and certification

InspectionsTesting on the management level Testing targets

ProcessReading techniques

UnitIntegrationSystemAlphaBeta

9

10. Journals and Conferences

Below, there are some journals and conferences listed which publish results of softwaretesting research.

11. Test-Related References

Below, there are references to test and process-related conferences. These are almost thesame references as in Appendix I in the book by I. Burnstein.

Automated software testing (tools)

Simulation of the testing proc-ess

Testing connected to develop-ment process

Tools for different techniques XPCleanroomRisk-based

Journals Conferences

•ACM SIGSOFT software engineering notes / Soft-ware engineering notes•ACM transactions on software engineering and methodology•Automated Software Engineering•Empirical Software Engineering•IEEE Software•Information and Software Technology•Journal of Systems and Software•Software - Concepts & Tools•Software Engineering Journal•IEEE Transactions on Software Engineering•Software Engineering. IEE Proceedings•Software Quality Journal•Software Testing, Verification and Reliability

•International Conference on Software Engi-neering•International Conference & Workshop on the Engineering of Computer-Based Systems•International Symposium on Empirical Soft-ware Engineering•International Symposium on Software Metrics•International Conference on Automated Soft-ware Engineering•International Symposium on Software Reliabil-ity Engineering

Software Testing: Related Conferences•www.sqe.com•www.stickyminds.com•www.testinginstitute.com•www.stc-online.org•www.qualityweek.com

Software Process and Quality Information•www.sei.cmu.edu/collaborating/spins/spins.html•www.sei.cmu.edu•www.software.org•www.espi.org•www.sqi.gu.edu.au/spice/contents.html•www.asq.org•www.qaiworldwide.com

Test-Oriented Web Sites•www.rbsc.com•www.softwareqatest.com•www.ssq.org•standards.ieee.org•www.ondaweb.com•www.mtsu.edu/~storm•www.stqemagazine.com•www.soft.com•www.sast.se•www.aptest.com•www.testingeducation.com•www.opensourcetesting.org