Quality Management and Testing Quality throughout the Lifecycle Testing as a Quality Practice Types...
Preview:
Citation preview
- Slide 1
- Quality Management and Testing Quality throughout the Lifecycle
Testing as a Quality Practice Types of Testing Agile Testing
Practices
- Slide 2
- 2 Review: Reliability Terminology Failure Incorrect or
unexpected output Symptom of a fault Fault Invalid execution state
Symptom of an error May or may not produce a failure Error Defect
or anomaly in source code Commonly referred to as a bug May or may
not produce a fault Defects may be injected at any time in the
lifecycle Recall Watts Humphrey (father of PSP): (paraphrase) A
defect is anything that necessitates a change in the code
- Slide 3
- 3 Review: The Cost of Defects Cost Development Phases Diagram
below is the assumed curve (Capers Jones, 2000) Software problems
are 100 to 1000 times more costly to find and repair after
deployment Diagram below is the assumed curve (Capers Jones, 2000)
Software problems are 100 to 1000 times more costly to find and
repair after deployment Cost to Repair Software Costs to Re-test,
Re-deploy, Re-* Cost of Lost Opportunities Cost of Lost Customers
Cost to Repair Software Costs to Re-test, Re-deploy, Re-* Cost of
Lost Opportunities Cost of Lost Customers Courtesy IBM Academic
Initiative
- Slide 4
- 4 Software Qualities Correctness Reliability Robustness
Performance User friendliness Verifiability Maintainability
Repairability Safety Evolvability Reusability Portability
Understandability Interoperability Productivity Size Timeliness
Visibility Notes courtesy Andre Van Der Hoek UC Irvine Copyright
2001 Quality is concerned about more than finding and fixing
traditional bugs testing is in essence a preventive process whereas
you build quality in; you create it deals with measurable and
verifiable ilities of software
- Slide 5
- 5 Verification and Validation Verification Ensure software
meets specifications Internal consistency Are we building the
product right? Validation Ensure software meets customers intent
External consistency Are we building the right product? Formal
Technical Reviews Test Planning & Review Measurement Analysis
& Reporting Process Definition & Standards V & V is a
whole life-cycle process
- Slide 6
- 6 Views of QA Design, in formal notation Executable machine
code Execution on verified hardware Code, in verifiable language
Complete formal specification of problem to be solved
Correctness-preserving transformation Notes courtesy Andre Van Der
Hoek UC Irvine Copyright 2001 Design, in mixed notation Pentium
machine code Execution on commercial hardware Code, in C++, Java,
Ada, Mixture of formal and informal specifications Manual
transformation Compilation by commercial compiler Commercial
firmware An Idealistic View of QA A Realistic View of QA
- Slide 7
- 7 Quality Practices in the Lifecycle Quality Practices Pervade
Lifecycle Activities: Requirements 8 quality measures (remember
these? correct, unambiguous, complete, consistent, prioritized,
verifiable, modifiable, traceable) Design Does your design satisfy
all the requirements? What constitutes a quality design anyway?
Coding What practices can developers do to improve quality? Test
Seen as the principle quality practice - but does it create
quality? Deploy The ability to get the software into the hands of
users Maintain What happens to quality as the system evolves over
time?
- Slide 8
- 8 Complicating the Process Four inherent complexities to
building quality software: 1.Requirements 2.Inherent problem
complexity 3.Accountability 4.Inertia Note that only #2 is a
function of the domain The others are principally
social/communicative
- Slide 9
- 9 First Complication - Requirements No matter how sophisticated
the QA process, the problem of creating the initial specification
remains Goes all the way back to requirements! Real needs Actual
Specification Correct Specification Notes courtesy Andre Van Der
Hoek UC Irvine Copyright 2001
- Slide 10
- 10 Second Complication - Complexity Complex data communications
Electronic fund transfer Distributed processing Web search engine
Stringent performance objectives Air traffic control system Complex
processing Medical diagnosis system Sometimes, the software system
is extremely complicated making it tremendously difficult to
perform QA Notes courtesy Andre Van Der Hoek UC Irvine Copyright
2001
- Slide 11
- 11 Third Complication - Accountability It is difficult to
divide the particular responsibilities involved when performing
quality assurance Project Management Development Group Quality
Assurance Group Notes courtesy Andre Van Der Hoek UC Irvine
Copyright 2001
- Slide 12
- 12 Fourth Complication - Inertia Quality assurance lays out the
rules You will check in your code every day You will comment your
code You will Quality assurance also uncovers the faults Taps
developers on their shoulders Creates image of competition Quality
assurance is viewed as cumbersome Just let me code You cant tell me
what to do Quality assurance has a negative connotation Notes
courtesy Andre Van Der Hoek UC Irvine Copyright 2001
- Slide 13
- 13 Quality Practices in Testing FURPS Model - sets expectations
for testing Actually a general quality model, this figure shows how
it fits for testing Reliability Test that the application behaves
consistently and predictably. Performance Test online response
under average and peak loading Functionality Test the accurate
workings of each usage scenario Usability Test application from the
perspective of convenience to end-user. Supportability Test the
ability to maintain and support application under production
use
- Slide 14
- Types of Testing Unit Testing (not covered) Acceptance Testing
Installation Testing Regression Testing Functional (System) Testing
Automated Testing Integration Testing Stress Testing
- Slide 15
- Acceptance Tests Purpose: Determine if the product is ready for
final release Different levels of acceptance based on the user
population Maps to the types of acceptance tests (next slide) May
be part of a contractual basis If software development under
contract specifications, then passing acceptance tests may be
considered a final milestone on your deliverable. Not dedicated
solely to finding implementation defects Also used to identify
requirements, deployment, maintenance defects, etc.
- Slide 16
- Acceptance tests Benchmark Test: real-world conditions Actual
or simulated real user base Commonly used for systems w/ special
requirements Evaluate multiple offerings against the benchmark The
benchmark represents ground truth Pilot test: install on
experimental basis Uses actual end users Focuses on mainstream
requirements Specialized training and education of the user base
not required Alpha test: in-house pilot test Or sometimes with most
trusted partners Beta test: customer pilot Parallel testing: new
system operates in parallel with old Useful for side-by-side
comparison Release Candidate: early access in the community
Community approval that the product is ready for final release
- Slide 17
- Installation / Regression Testing Installation Testing Does the
system work in the field as it does in the lab? How easy or hard is
it to deploy and maintain? Are migration procedures accurate? Does
the documentation match? This is a big deal nowadays! Regression
Test Once system has passed a system test, future system tests must
guarantee new functionality doesnt break working functionality When
you run the Nth test, you have to run the N-1 tests before it Fits
particularly well with iterative and incremental processes
- Slide 18
- Functional/Automated Tests Functional: Test of the system as a
whole Functionally-oriented (black-box) Can include non-functional
aspects Typically though, these are done as part of stress testing
Not concerned with location of defects or interface problems These
are tasks for integration testing Defect expensive to fix when
found at this step Automated Testing and Test Tools Simulations
Record-and-play tools Workflow support for test scripts Tracking
and reporting results Nightly build and test tools Example:
http://public.kitware.com/dashboard.php
- Slide 19
- Integration testing Tests complete systems or subsystems
composed of integrated components Integration testing should be
black-box testing with tests derived from the specification Main
difficulty is localising errors Incremental integration testing
reduces this problem Interface Testing Takes place when modules or
sub-systems are integrated to create larger systems Objectives are
to detect faults due to interface errors or invalid assumptions
about interfaces Particularly important for object-oriented
development as objects are defined by their interfaces
- Slide 20
- Interfaces types Procedural interfaces Sub-system encapsulates
set of procedures called by other sub-systems Message passing
interfaces Sub-systems request services from other sub-systems
Parameter interfaces Data passed from one procedure to another
Shared memory interfaces Block of memory is shared between
procedures Types of Interface Errors Interface misuse: A calling
component calls another component and makes an error in its use of
its interface e.g. parameters in the wrong order Interface
misunderstanding: A calling component embeds assumptions about the
behaviour of the called component which are incorrect Timing
errors: The called and the calling component operate at different
speeds and out-of-date information is accessed
- Slide 21
- Stress testing Exercises the system beyond its maximum design
load. Stressing the system often causes defects to come to light
Stressing the system test failure behaviour. Systems should not
fail catastrophically. Stress testing checks for unacceptable loss
of service or data Particularly relevant to distributed systems
which can exhibit severe degradation as a network becomes
overloaded
- Slide 22
- The Testing Process Test data: inputs devised to test the
system Test cases: Test Data plus the predicted outputs from these
inputs if the system operates according to its specification Test
suite: A collection of Test Cases organized around some principle
or objective.
- Slide 23
- Test documentation Test plan describes system and plan for
exercising all functions and characteristics Test specification and
evaluation details each test and defines criteria for evaluating
each feature Test description test data and procedures for each
test Test script The actual test procedures (process description)
Test analysis report results of each test
- Slide 24
- Example Test Script
- Slide 25
- Example Test Script (cont.)
- Slide 26
- Agile Practices Change Roles Traditional Testing role The Last
Defender of quality Enforce a strict change management policy w/
rigid entry/exit criteria Heavy documentation & upfront
planning ( but a dynamic project plan?! ) Agile Testing role A team
member primarily focused on quality I was Loved and Appreciated
overheard on a testers 1 st agile project Integrate quality
practices into development process Test provides value to help
reduce risk Agile Principles: Quality is everyones responsibility
Developers write tests too (test-driven) Testers work in same
bullpen as developers No separate, closed-door testing group Track
quality with rest of development process Coverage, metrics, etc. on
each build
- Slide 27
- Agile Testing: Continuous Testing Agile likes transparency,
adaptability to change Dashboards used to show constant quality
progress Include results of unit, system, load, etc. tests As well
as quality measures from your metrics Compatible with other
transparency measures like burndowns
- Slide 28
- 28 Agile Testing Top 10 Best Practices 1.Write tests first:
Code is not delivered to build unless it passes tests 2.Write
acceptance tests: Every story defines its acceptance criteria
3.Embrace change: Agile development means frequent change 4.Reduce
Test Documentation: R ecord the test and track results, thats it!
Use a Wiki to capture essence, not details (must must be
repeatable) 5.There is no handoff activity: Testing integrated in
the dev process 6.There is separate no integration testing
activity: Integration is continuous, supported with (automated)
regression testing 7.Automate everything you can! Make test scripts
the documentation 8.Test early (or Test First!) Add tests to
development build process before feature is completed Test features
before they are complete Test features before they are accessible
from its external interface (GUI) 9.Testing is not a phase, its a
way of life. Agile teams are test infected. How will we test it? is
as important as How will we build it? 10.Testing and development
are not separate teams
- Slide 29
- Quality and Testing Conclusions Dr. Garys Things to Remember:
1.Tested software != Quality software 2.Quality is injected, not
something defined in absentia 3.But sure, testing is important, and
important to do well Have clear testing objectives Dont treat it as
a last step checklist 4.Respect your testers you are on the same
team Dont hate the playa, hate the game--Ice T 5.Include your
testers early in the process 6.Start testing early in the process
7.Verification is important, but Validation is everything 29
Program testing can be used to show the presence of bugs, but never
to show their absence! --E.W. Dijkstra, 1972