1

CEN 4072Software Testing

PPT2: Tracking the problem

2

What is a problem?

• A problem is a questionable property of a program run.• Life cycle of a problem:

- First, a developer is informed about the problem- The developer reproduces the problem to view the circumstance from his own eyes. The way a developer sees a problem may be different than the way the user has seen it- Developer then isolates the circumstance. In other words, they find the piece of code that is generating the problem.- Then the developer fixes the problem.- Lastly, the fix is delivered to the user.

3

Problem reports• Problem reports (a.k.a. bug reports, defect reports, fault

reports, problem reports, trouble reports, and change requests) contain the information required to fix a problem.

• Problem reports contain multiple elements.• Product release• Operating Environment• Problem History• Expected Behavior• Observed Behavior

4

Bugs and product release

• The version of the program that the problem occurs in.

• Could also be a different unique identifier that is needed to reproduce the exact version

• Must assess if the problem occurs only in that release or if it has a broader scope; could help isolate problem

5

Bugs and operating environments

• The operating environment in which the problem occurred (Ex. Windows 8.1 with the following packages installed)

• The operating system version information• Problem could be present in multiple

environments

6

Bugs and system resources

• Any other resources needed to duplicate the environment that the problem occurs in

• Ex. Programs that need to be running, external hardware, etc.

7

Bugs and problem history

• The steps that are necessary to reproduce the problem

• Must decide which steps are actually contributing to the problem and which are irrelevant

8

Bugs and expected and observed behavior

Expected:•What should have happened if the problem did not exist•What the problem is preventingObserved:•What is actually happening•What is the problem as observed by the user

9

Managing problems

To manage problems can either use…A problem file:•Does not scale•Only one person can work on the problem at a time•Previous fixes, history of the problem is lost•Less efficient for a company to use (what happens if a previous revision is more effective than the most current one?)A problem data base:•Allows for multiple people to work on problem at one time•Saves different revisions of problem so history is not erased

10

Bugzilla problem data base

• Bugzilla is a web-based general-purpose bug tracker and testing tool originally developed and used by the Mozilla project (Wikipedia definition)

•Features (as listed on Bugzilla website) :Optimized database structure for increased performance and scalability Excellent security to protect confidentiality Advanced query tool that can remember your searches Integrated email capabilities Editable user profiles and comprehensive email preferences Comprehensive permissions system Proven under fire as Mozilla's bug tracking system

11

Problems: severity

Bugs can be classified by many things- the first being severity. There are four main severity classifications:•Critical: The defect affects critical functionality or critical data. It does not have a workaround. Example: Unsuccessful installation, complete failure of a feature.•Major: The defect affects major functionality or major data. It has a workaround but is not obvious and is difficult. Example: A feature is not functional from one module but the task is doable if 10 complicated indirect steps are followed in another module/s.•Minor: The defect affects minor functionality or non-critical data. It has an easy workaround. Example: A minor feature that is not functional in one module but the same task is easily doable from another module.•Trivial: The defect does not affect functionality or data. It does not even need a workaround. It does not impact productivity or efficiency. It is merely an inconvenience. Example: Petty layout discrepancies, spelling/grammatical errors.*

*http://softwaretestingfundamentals.com/defect-severity/

12

Problems: priority

Problem priority is also a classification that is used. It is based on the urgency that the problem is resolved.Immediate – The bug should be resolved immediately.High - This bug should be resolved as soon as possible in the normal course of development activity, before the software is released.Medium – This bug should be repaired after serious bugs have been fixed.Low – It can be resolved in a future major system revision or not be resolved at all.*

*http://www.elementool.com/contact/articles/Bug_Tracking_severity_priority.html

13

Problems: identity and notification

• Every problem is assigned an identifier (also called a PR number or bug number) which is its own unique number that can be used when referring to the bug.

• A tester can also attach an email address to the problem so whenever the problem is worked on, an email is sent to the tester updating the status of the problem. This notifies the tester (or user) whenever the problem report is updated.

14

The problem life cycle

15

The problem life cycle

• Unconfirmed- A problem report first enters the data base as unconfirmed

16

The problem life cycle

New- If the report is invalid, or a duplicate, it goes straight to the resolved phase. If not, it is a new problem.

17

The problem resolution types

Assigned- The problem is assigned to a developer. The developer is in charge of producing the resolution•If the resolution is invalid, it is not a problem•Duplicate- the problem is already in existence•Fixed the problem has been solved by developer•Won’t fix- The problem will/can not be resolved by developer (ex. It is a feature)•Worksforme- The problem cannot be reproduced developer

18

The problem life cycle

Resolved- The problem has been processed ( all roads lead to this)

19

The problem life cycle

Verified- The problem has been satisfied with a successful fix. If not, the problem is reopened.

20

The problem life cycle

Closed- A new version with the fix is released.

Images and information obtained from: http://www.whyprogramsfail.com/

21

The charge of the Software Change Control Board (SCCB)

The SCCB is in charge of •assessing the impact of the problem•delegating tasks to the fix of the problem•assessing the “solved” problems •deciding if the problem can be closed.

22

Problem driven development

• The entire software development process can be approached with problem driven development.

• Initial problem: The product we need does not exist• Divide each this broad into sub-problems needed to develop the

software• Once all of these sub-problems are solved, the product is ready

23

Managing the clutter of problem reports

• With a large problem report database, it is easy for the database to become cluttered.

• This can be avoided in a number of ways.• Developers are encouraged to do a search for their problem

before they submit it as new and to simplify their bug reports• This eliminates duplicates in the system• Its is also important to “clean house” from time to time,

eliminating old problems that are obsolete and no longer occur

24

Problem fixes and releases

It is important to decipher problem fixes and releases by following naming conventions.

Ex. http://www.whyprogramsfail.com/

25

Problems and tests

• It is not necessary to insert a failed test into the problem data base.

• “Once we can repeat a problem, there is no need for a database entry” -http://www.whyprogramsfail.com/

26

Effective problem reports

An effective problem report…•Is unbiased and sticks with facts only•Is clear and simple; keeps cases general•Can be easily reproduced•Is well structured and readable for all parties

27

Other tools for bug tracking

• PHPBUGTRACKER- “a web-based bug tracker with functionality similar to other issue tracking systems, such as Bugzilla. Design focuses on separating the presentation, application, and database layers.” (http://phpbt.sourceforge.net/)

• ISSUETRACKER- “a support issue tracking system... designed to be user friendly, …includes many features include(ing) things like file uploads, email parsing, email and sms notifications, unlimited users and groups, and much more.” (http://issue-tracker.sourceforge.net/http://issue-tracker.sourceforge.net/)

• TRAC- “Trac is an enhanced wiki and issue tracking system for software development projects. Trac uses a minimalistic approach to web-based software project management. Our mission is to help developers write great software while staying out of the way. Trac should impose as little as possible on a team's established development process and policies.” (http://trac.edgewall.org/)

• SOURCEFORGE- “is a web-based service that offers a source code repository, downloads mirrors, bug tracking and other features. It acts as a central location that software developers can use to control and manage free and open-source software development.” (https://en.wikipedia.org/wiki/SourceForge)

• GFORGE- “a free software fork of the web-based project-management and collaboration software originally created for SourceForge, called Savane.” (https://en.wikipedia.org/wiki/GForge)

Recap Reports about problems encountered in the field are stored in a

problem database. A problem report must contain everything relevant to reproduce the

problem. It is helpful to set up a standard set of items that users must

provide (product release, operating environment…) A typical problem life cycle starts with an unconfirmed status and

ends with a closed status with a specified resolution Typically, a software change control board organizes priorities and

assignments Use version control to separate fixes and features during

development. Establish conventions to relate changes to problem reports and vice

versa. Make a problem report obsolete as soon as a test case exists.

*Recap and all other information obtained from Why Programs Fail: A Guide to Systematic Debugging, A. Zeller, Elsevier and coinciding website unless otherwise specified 28