CS4723Software

Validation and Quality Assurance

Lecture 01Introduction

2

Course Instructor     

Name:Dr. Xiaoyin Wang (Sean)

Office: NPB 3.208

Email: xiaoyin.wang@utsa.edu

Experiences Got my PhD from Peking University, China Did my postdoc in UC Berkeley Worked for Microsoft (.net project), and Ensighta (a startup comp

any at Berkeley with 7-8 people)       

3

Introduce yourselves!

4

Course Meetings, Web Pages, etc.       

Course Meetings: MWF 11:00pm – 11:45pm

MH 3.04.20

Office Hours:

MW 1:00pm - 2:30pm

Course Web Page: http://xywang.100871.net/CS4723.htm       

5

Course Textbooks      

Choose from the following reference books Software Testing and Quality Assurance: Theory

and Practice, by Kshirasagar Naik and Priyadarshi Tripathy

Software Testing: Principles and Practice, by Srinivasan Desikan and Gopalaswamy Ramesh

Introduction to Software Testing, by Paul Ammann and Jeff Offutt

6

Course Topics       

Software Testing Software Debugging Software Verification Anti-pattern Checking and Refactoring Bug Management Research Progress:

Automatic Oracles Automatic Program Repair

7

Grading Scheme       

Final Exam: 40% Assignments: 30% Projects: 20% Course participation and presentations: 10%

8

Final Exam

Time: May 7th: 9:45am to 11:45am

Location: MS 3.04.20

Content Selected important topics in the course

We will have a review during the two classes in the last week

9

Assignments

General Post and due date are in the schedule

Both paper and electronic submissions are acceptable is possible, electronic is preferred

Detailed evaluation plan will be introduced at the post time

Assignments Write unit test cases

To be posted

10

Project: Android UI Testing

2-3 people form a small team

Each team is going to do the UI script testing for an Android App

Steps Explore the Android App

Write Testing Reports

Write Scripts for Regression Testing

Deliverable A testing report and the testing scripts

11

Now, let’s go to the real lecture …

12

Why software quality matters?

Prevalent usage Billions of PCs and mobile phones

Chatting, Surfing, Gaming, …

About 1 million lines of code on a new car, MIT Tech Review 2013

Some software on almost all major or small appliances

ATMs

Counters at shopping centers, restaurants, …

Garage Entrances

…

13

Why software quality matters?

We are so dependent on software for critical tasks Traffic Control

Trains, air traffic, …

Transportation Control Ship / plane / car / elevator /…

Financial transactions Bank, stock, online shopping

Power supplies Electricity, Power stations, gas, …

Medical equipments

14

What can caused by software bugs

From inconvenience to disasters Ugly presentation

Not showing some results or output

Unable to do operation

Lost of a hour’s / day’s / week’s work…

Privacy leaks, lost accounts

Money loss

Injuries, Life-threatening dangers

Disasters

15

Some well-known disasters caused by software bugs

Mars Climate Orbiter ($165M, 1998) Sent to Mars to relay signal from the Mars Lander

Smashed to the planet: failing to convert

between English measure to metric values

Shooting down of Airbus A300 (290 death, 1988) US CG-49 shoot down a Airbus A300

Misleading output of the tracking software

THERAC-25 Radiation Therapy (1985) 2 cancer patients at East Texas Cancer

center received fatal overdoses

Miss-handling of race condition of the

software in the equipment

16

Facts about software bugs

On average, 1-5 bugs per KLOC (thousand lines of code) In mature software

More than 10 bugs in prototypes

Windows2000 35MLOC

63K known bugs at the time of release

2 bugs per KLOC

$59.5B loss due to bugs in US 2002 (estimation by NIST)

It is not feasible to remove all bugs But try to reduce critical bugs

17

Why so many bugs in software?

Errors can happen in any engineering discipline

Software is one of the most error-prone products of all engineering areas Requirements are often vague

Software can be really complex, undecidable problems are everywhere

Result Most product are put into the market with no or very

few errors Almost all software in the market has some number of

bugs (we will see that later)

UTSA CS510318

Aspects of software quality

Dependability availability, reliability, safety

Efficiency processing time, memory utilization, responsiveness

Usability appropriate user interface and adequate

documentation

Maintainability ease of change

UTSA CS510319

Dependability (I)

Availability For how large proportion of time the software

system is working

Recover from a crash?

Reliability For how large proportion of inputs the software is

giving correct outputs

Fault tolerance?

UTSA CS510320

Dependability (II)

Safety For critical software, e.g. insulin control

To what extent the software is threatening people?

Runtime fault detection?

Security To what extent the software is able to protect itself

from malicious inputs

Input sanitization?

UTSA CS510321

Efficiency

Processing time for time-consuming tasks Algorithm Complexity

Optimization

Maximal Memory utilization Avoid memory leaks

Release memory at proper time

Responsiveness Interleave UI threads and back-end threads

UTSA CS510322

Usability

User Interface Looks attractive

Easy to understand and convenient to use

Help Documentation Easy to search and read

Cover all the features of the software

UTSA CS510323

Maintainability

Ease to read Code quality

Comments

Ease to extend Design quality

Suitable design patterns

Ease to transplant Avoid hard coded elements

UTSA CS510324

Absolute Software Quality

Impossible Vague Requirements

Free of bugs can never be verified even for some simple programs

Practice on critical software The NASA example

UTSA CS510325

Each execution handles $4Billion equipments, human lives, dreams No prototypes

420k lines of code, 17 errors in 11 versions (is it a large software?)

Commercial equivalent would have at least 1000 bugs

NASA Example

UTSA CS510326

A third of the effort before coding starts

Long specifications, written down, fully discussed 40k pages of specification (longer than the code)

Change to add GPS support (2500 pages more specification)

Specification is almost pseudo code

NASA Example

UTSA CS510327

When fixing bugs, fix what allowed the mistake Unclear API

Insufficient tests

Improper use of tools

Validation and review at all levels 85% of bugs revealed before testing

NASA Example

UTSA CS510328

Cost 260 people

$32 million

A year

TOO EXPENSIVE!!!

Overkill for normal software

NASA Example

29

Quality Economics

Tradeoff of cost and requirements

30

Approaches to Enhance Software Quality

Bug Prevention Prevent bugs during requirement collection,

designing, coding

Bug Removal Remove bugs after coding, but before

release

Bug Tolerance Runtime mechanisms to avoid known or

unknown bugs in the software

31

Approach to prevent bugs

Requirement Engineering Remove ambiguity in requirements

Formal Specification

Design Reduce dependency between modules

Concentrate cross-cut concerns

Coding Proper coding styles

Well written comments

Avoid misleading interfaces

32

Approach to remove bugs

Testing Feed input to software and run it to see whether its

behavior is as expected

Limitations Impossible to cover all cases

Test oracles (what is expected)

Static checking Identify specific problems (e.g., memory leak) in the

software by scanning the code or all possible paths

Limitations Limited problem types

False positives

33

Approach to remove bugs

Formal Proof Formally prove that the program implements the

specification

Limitations Difficult to have a formal specification The proof cost a lot of human efforts

Inspection Manually review the code to detect faults

Limitations: Hard to evaluate Sometime hard to get progress

34

Approach to tolerate bugs

Exception handling Specify what the program should do when

unexpected results are generated

Spend time on writing exception handlers

Consider exceptions in the exception handler

Redundant code Have redundant modules to avoid runtime bugs by

checking consistency of results

Voting for a correct results when bug happens

35

In our course

Bug prevention: something you should learn in software

engineering course, briefly introduce it at the end of our course

Bug removal: The focus of this course

Bug tolerance We will introduce guidance for exception

handling at the end of our course, redundant code will not be introduced because it is rarely used

36

A short review

Software bugs affect every aspect of our life

Software Bugs may cause severe consequences

Software Bugs are prevalent

Aspects of software quality Dependability, Efficiency, Usability, Maintainability

(Bad smells -> bugs)

Approaches to enhance software quality Bug prevention, Bug removal, Bug avoiding