Advanced Information Architecture- Fall 02 Testing and quality assurance I. Quality control and web site development The importance of testing Quality

Advanced Information Architecture- Fall 02

Testing and quality assurance

I. Quality control and web site development

• The importance of testing

• Quality assurance

II. Testing for quality assurance

• Phases of testing

• Types of testing

III. The test plan





Testing is a process of quality control and assurance

It is intended to improve the site by assessing how it works under a variety of conditions

Platforms, browsers,OS, monitors, digital devices, bandwidth

It ensures that the site does what it is supposed to do for the intended audience

Usability and accessibility

Navigation, search and retrieval


It is critical to gain client buy-in for testing

You must be ready with a clear justification for the time and costs of testing

Questions you should be ready to answer

What are the costs and how will testing save money?

How many people will be involved and how much time will it take?

How will it improve the site?

What is the client’s role in testing?

This should be determined in advance

The process should be clear and accounted for in the budget


Risks associated with testing

Slippage

Not allotting enough time for testing or taking time away because of problems elsewhere

Coverage: how much of the original plan did you cover?

Bug fixes

People have to be assigned to this and time has to be set aside

Typically this is at the end when people are tired and swamped

Metrics

These are unclear to testers


More risks associated with testing

Outdated specifications

Makes benchmarking difficult for testers

Poorly trained testers

Should have familiarity with the site and with the error management procedures

Unclear assessment of the targeted users

Leads to inappropriate testing scenarios

Component, module, application system integration

Testing should be taking place in a relatively stable environment


Quality assurance

What is quality?

Being able to understand and use a system to achieve desired goals and recover efficiently and gracefully from problem situations

Meeting requirements and conforming to target market needs and expectations

This means the requirements must be known to be valid

This means those needs and expectations are seen to be realistic

The delivery of a product that satisfies user expectations of functionality, performance, and security at previously agreed upon costs and timetables


Meanings of quality

Descriptive: being able to understand and utilize a system to achieve desired goals

Also, recovering efficiently and gracefully from problem situations

Prescriptive: being able to meet the requirements negotiated with the client

The requirements must be known to be valid

It also means conforming to target market needs and expectations

Those needs and expectations are known to be realistic)


Objective quality

The degree of compliance of a process/product or its outcome with a predetermined set of criteria

These criteria are presumed essential to the ultimate value the process or product provides

Subjective quality

The level of perceived value reported by the person(s) who benefits from a process/product or its outcome

Quality is the delivery of a product that

Satisfies user expectations of functionality, performance, and security

The site is delivered at previously agreed upon costs and in a timely manner


Quality involves building applications and formulating methodologies that add value to the business and customers

It is a valuation perceived by organizations, customers, and users

This is based on how a product or service meets their collective and individual needs

Information quality is a strategic resource for an organization and supports “managing with facts”

It is an ongoing process of building and sustaining relationships by assessing, anticipating, and fulfilling stated and implied needs

It encompasses business concerns (i.e., the producers and customers) and the concerns of the end-user as well


There is a distinction between business needs and market needs

This mirrors the distinction between a “customer” and a “user”

A customer will be focused on the product meeting certain objectives and including a certain amount of functionality

A user will focus on issues such as ease-of-use and overall operability

How does the user perceive the quality of the product?

The view the customer has of the product is the view that the organization must have

The bottom-line of the organization is dependent upon its view of the customer or user


The main goal of quality is to enable competitive advantage and business effectiveness

It involves building applications and processes that add value to the business and customers

Quality is a tradeoff between time and money

The goal is to make the site as good as it can be given the constraints of the project

The site meets the stated objectives

These are laid out in the scope document

It is delivered on time an budget

Quality can be empirically measured

This requires standards, standard methods, and accepted processes


Quality involves the creation and maintenance of relationships between the business and its customers

In a practical sense you are trying to make quality visible

Showing tangible aspects of quality by focusing on how customer experiences with a site impact the bottom line

The tangible result of quality is the degree to which the stability and market viability of a product

Increases the probability of outcomes desired by the customer

Reduces the probability of undesired outcomes for the organization


Customers judge the quality according to stated and implied needs and expectations and the availability of alternatives

The site should be

Reliable: ease of use

Responsive: allows efficient use

Easier to use than the alternatives

Compared to the competition


What is quality assurance?

It is a function that identifies, documents, and reviews for improvement the processes that deliver products

Systematic actions that constantly optimize productivity, communication, and value within an organization

Measures attributes, properties, and characteristics of a product

Nyman, J. (2002). What is Quality Assurance? http://www.globaltester.com/sp2.html

Another view of QA (ISO 9001)

What you need to do, to demonstrate that your product or service will satisfy your customers


The “practice of quality assurance” is auditing, monitoring, and testing

How it functions is the means by which you do that auditing, monitoring, and testing

Testing is so much a part of quality assurance that it is really inseparable

Context of testing

Your company is in the position of making some sort of product

That product will have various projects placed against it

Either to create new functionality to the product or to modify existing functionality


The quality focus, overall, is looking at the product lifecycle

The means by which it is produced

Requirements documents

Design specifications, source code, executable binaries, etc.

At each phase of the lifecycle, QA is going to be doing some sort of testing

Testing documents (to check for things like consistency)

Testing code itself (in the form of code reviews or inspections)

Testing the direct execution of code without a front-end (unit testing)


QA testing

Testing the full system, after all units have been integrated and work with a front-end (system testing)

Testing to be sure that new functionality did not affect existing functionality and is not, itself, broken (regression testing)

QA is also auditing and monitoring

What QA is doing

Establishing certain processes: defect tracking solution or a standard change control process

Audit that which you establish in order to make sure that what you are doing is effective


What is a QA system?

The organizational structure, the processes and procedures necessary to ensure that the overall intentions and direction of an organization as regards quality are met

The quality of the product or service is assuredISO 9001 Quality Assurance. (2002). Home Page. http://www.iso9001qualityassurance.com

A well developed ISO 9001 QA system should:

Improve your product and service quality

Give customers confidence that their needs will be met

Standardize your business by giving it a consistent approach to its operations

Improve work processes, efficiencies, morale and reduce waste









III. The test plan



• What is testing?

Creating manual or automated sequences that are structured, methodical, and repeatable

These methods are designed to probe suspected product or system weaknesses and to find defects

This provides an indication of product reliability

The degree to which the product fits the specifications

Response of client and customer

How well the product meets their needs

Both are necessary for QA

A reliable product that satisfies stakeholders and users


Testing involves two distinct phases

Before the milestone

This is verification testing

The process of evaluating a system or component to determine whether the products of a given development phase satisfy the conditions imposed at the start of that phase (IEEE/ANSI)

Given the requirements, are we doing the right thing?

Testing up to this point is necessary to determine which among a range of alternatives to select

As new content and functionalities are added, they should be tested

This is iterative or developmental testing and is intended to point out bugs


Phases of testing

After the milestone

This is validation testing

The process of evaluating a system or component during or at the end of the development process to determine whether it satisfies specified requirements

Given the requirements, did we do what we said we were going to do?

No new content or functionality is accepted

Testing after this milestone is to determine the usability and quality assurance (QA) of the site that will be launched

Goal is to uncover operational or functionality problems


Sample metrics

These determine the ideal test environment and should resemble the typical or majority user environment

Browser/OS configurations

Hardware requirements

Software requirements

Memory size requirements

File and application size

Bandwidth and connection speed

Response times for download


Sample testers

They don’t (and should not) care about what’s inside the black box

They should just be using the site

The test is based on the site specifications

Tests reflect the functional requirements set out in the scope document

Testers are given a representative range of scenarios

This is based on typical uses of the site

Testers use valid and invalid inputs

They should intentionally try to break the system

This provides information about error handling


Eight validation axioms

1. Testing can be used to show the presence of errors, but never their absence

2. One of the most difficult problems in testing is knowing when to stop

3. Avoid unplanned, non-reusable, throwaway test cases

4. A necessary part of a test case is a definition of the expected output or result

5. A comparison should be made of the actual versus the expected result

6. Test cases must be written for invalid and unexpected, as well as valid and expected, input conditionsNyman, J. (2002). What is Quality Assurance? http://www.globaltester.com/sp2_a6.html


5. Invalid input

Invalid input is a condition outside the set of valid conditions and should be diagnosed as such by the program being tested

6. Cases must be written to generate desired output

Do not think just in terms of inputs

Determine the input required to generate a pre-designed set of outputs

7. A program should not be tested (except in unit and integration phases) by the person or organization that developed it

8. The number of undiscovered errors is directly proportional to the number of errors already discovered


Types of testing

User acceptance/usability testing

To determine whether the site meets the specified requirements

Whether the intended audience can use the site

Useful when the pre-production prototype is finished (pre-milestone)

Also when launch version is ready (post-milestone)

Can use a variety of methods

Observation, accompanied browsing

Heuristic evaluation, card sorting, focus groups, peer review


Functional testing

Going through the entire site to ensure that everything works as it is supposed to

Compatibility test: testing configurations as described in the project document’s technical specifications

# of browsers x # of operating systems x # of screen resolutions = the # of tests to be performed

Start with the configuration that is likely to cause the most problems (and use Netscape)

Try to break it (unusual characters in forms)

Regression test: retest with the most common configuration after finishing the changes


Functional testing

Compatibility test

Elements include:

Links, forms, navigation, download times, readability, content management, transactions, push functions, personalization, printing

Error management

What is your process for logging errors and problems uncovered in testing?

How much detail should be recorded?

Testers and fixers need access to this system so they they can record problems and fixes in real time


Functional testing

Error management

Resolution of the error or problem should also be logged

The resolver log: incident number, date, logged by, priority, category, description (with error message), date work began, owner, comments, status, date finished

Priority: 1 Fatal (crash or freeze) 2 Serious (works but not the way it should) 3 Significant (can be fixed, interferes with user experience) 4 Minor (typos…)

Tester should then try to recreate the error after it has been resolved


Operational testing

Testing the back end of the site

Also the service components involved in the site

Focus: delivery times, shipping accuracy, customer service, data integrity

Goal is to find the problem before the customer or user does

Stress/load testing

Simulate possible scenarios of use

Testing the stability of the programming and the capacity of the hardware

Can be done with scripting that simulates multiple HTTP requests


Security testing

Test the stability of the firewall and intrusion detection software

Check the protection of sensitive data (especially passwords and credit card information)

This will probably require outside expertise

Proofreading

Read every page carefully and make sure there are no typos and the grammar is correct

Check the contact information and verify the content









III. The test plan


III. The test plan

This is a document that outline the range of tests that will be conducted throughout the project lifecycle

It specifies types of risks that can be expected and the test that will be used to assess these risks

It outlines procedures, timeline, and costs

It defines the acceptable metrics that will be used

These can be in the form of benchmarks

How many seconds will it take for the home page to download?

A secondary page?

How long will that average search return take?


Structure of the test plan

Scope

What kinds of tests will be conducted?

What types of test environments will be used?

Who are the testers?

What are the main metrics that will be used

Types and severity of bugs

Functional: problems with programming

Usability: problems with navigation

Creative: differences in color rendering across platforms

Severity: use a simple ranking system



Version control

Explaining the system in place

Risks and assumptions

What is the range of expected problems that could compromise testing?

What resources are needed in order to carry out the testing?

Testing schedule and cycles

When they occur, how long they take, and when and how often they are repeated



Test cases

Include the scenarios used as cases

Bug reports

Listing of all the bug found in testing

Description of how they were handled

QA summary

General observations

Discussion of major problems and strategies for resolving them

The test plan is an important addition to the project archives

Documents

Advanced Information Architecture- Fall 02 Testing and quality assurance I. Quality control and web site development The importance of testing Quality