Upload
dangdat
View
217
Download
1
Embed Size (px)
Citation preview
© CapgeminiSogeti 2
Presenter
• Ole Chr. Hansen • Managing Consultant • Fellow – SogetiLABS – Global Innovation Team
Blog - http://ochansen.blogspot.com LinkedIn: www.linkedin.com/in/ochansen Twitter: www.twitter.com/Ole_Chr_Hansen
• ISTQB Expert Level – Improving the Test Process (Assessing) • ISTQB Accredited Trainer in Software Testing • ISEB Practitioner Certificate in Software Testing • ISTQB Foundation Certificate in Software Testing • TMap NEXT® Test Engineer Certified • TPI NEXT® Foundation Certificate • PRINCE2 Foundation Certificate • Certified Scrum Master • Certified Lead Assessor (ISO 9000)
• Sogeti, ATP, Nordea, BRFkredit, WM-data, CRI, LEC • 18+ years in Test Management, 10+ years in Project Management
© CapgeminiSogeti 4
ATTA Contents
1) The Technical Test Analyst’s Tasks in Risk-Based
Testing
2) Structure-Based Testing
3) Analytical Techniques
4) Quality Characteristics for Technical
Testing
5) Reviews
6) Test Tools and Automation
© CapgeminiSogeti 5
Quality Characteristics for Technical Testing
SYSTEM
What the system does
Stakeholder expectations
Stakeholder compares
The closer they match – the higher the quality
© CapgeminiSogeti 6
General Planning Issues
• Issues to consider when planning non-functional testing
– Stakeholder requirements
– Required tool acquisition and training
– Test environment requirements
– Organizational considerations
– Data security considerations
The Technical Test
Analyst may be
requested by the
Test Manager….
….to identify the
principal risks for
the relevant
quality…..
….to address any
planning issues
associated with the
proposed tests
© CapgeminiSogeti 7
STAKE-HOLDERS
Customers
Maintenance
Users
Operators
Helps to ensure no important
requirements are missed
Helps to identify risks and priorities
Stakeholder requirements
© CapgeminiSogeti 8
Required tool acquisition and training
• Tool support often needed
– Particularly for some performance and security testing
• Tools not available
– Need to acquire tool
– Learn to use the tool
• Training time
• Hire tool specialists (external)
– Implement tool
• Pilot project
• Incremental rollout
– Allow time and ressource
• Learning curve
© CapgeminiSogeti 9
Test environment requirements
Many technical tests require a production-like test
environment in order to provide realistic measures
e.g., security tests, performance tests
DTAP Concept
DEV TEST ACCEPTANCE PRODUCTION
© CapgeminiSogeti 10
Organizational considerations
• Technical tests may involve measuring the behavior of several
components in a complete system (e.g., servers, databases,
networks)
© CapgeminiSogeti 11
Data security considerations
• Access to secure systems
– Physical access / system access
– May require pre-authorization, user id and
password, id card etc.
• Specific security measures implemented for a
system should be taken into account at the
test planning stage to ensure that all testing
activities are possible.
– For example, the use of data encryption
may make the creation of test data and
the verification of results difficult.
© CapgeminiSogeti 12
Efficiency Testing – Sub-characteristics
Efficiency Testing
Performance Testing
Load Testing
Stress Testing
Scalability Testing
Ressource Utilization
© CapgeminiSogeti 13
Efficiency Testing
Load testing Stress testing
Scalability testing
Ressource utilization
Efficiency testing
Operational profiles
Tools
Environments
Analysis Reports:
Server
Network Database
Application
Test Plan:
Goal Objectives
Risks Tests
Response Times | Break Point | Peak Load | Memory Leaks | Uptime | Downtime
© CapgeminiSogeti 14
Types of performance testing
Performance testing
Load testing
Stress testing
Scalability testing
© CapgeminiSogeti 15
Load testing
• Concurrent users or processes
– Handle increasing levels of
anticipated realistic loads
resulting from the transaction
• Average response times for users
– Under different scenarios of
typical use (operational profiles)
– Measured and analyzed
• Supported by tools
– Generate loads
© CapgeminiSogeti 16
Stress testing
• Handle peak loads at or beyond the limits
– Anticipated or specified workloads
– Reduced availability of resources
• Assessing risk posed by extreme conditions
• Expect performance levels to be degraded
slowly and predictably
– the functional integrity of the system should be
tested
• The ‘weakest link in the chain’ can be
determine
• Important for web applications
– E.g. self service at the Danish tax authorities
• Supported by tools
© CapgeminiSogeti 17
Scalability testing
• Ability of a system to meet future
efficiency requirements
– May be beyond those currently required
– System’s ability to grow
• E.g., with more users, larger amounts of
data stored
• More assessment than testing
• Related to capacity planning
– Simulation of a scaled down production
environment
• Supported by tools
© CapgeminiSogeti 18
Resource Utilization Testing
• Sub-characteristic of efficiency
– See appendix for an overview
• Tests relating to resource utilization evaluate the usage of system
resources
– usage of memory
– disk capacity
– network bandwidth
– connections
• Compare with benchmark
– Requirements from user and other stakeholders
– Unnatural growth in usage
© CapgeminiSogeti 19
Resource Utilization Testing
• Memory resource usage is relevant for real-
time embedded systems
– If the memory footprint exceeds the allowed
measure, the system may have insufficient
memory needed to perform its tasks within
the specified time periods
– This may slow down the system or even lead
to a system crash
• Dynamic analysis may also be applied
– Investigating resource utilization and
detecting performance bottlenecks
© CapgeminiSogeti 20
Trend in test automation approaches
Capture/Playback
Data-driven
Keyword-driven
Each builds on the previous approaches
© CapgeminiSogeti 21
Objectives of scripting approaches
• To reduce costs
– Make it easier to build automated tests
– Avoid duplicated specs
– Avoid excessive maintenance costs
• Higher degree of reuse of modular scripting
• Higher RoI
– Better testing support
– Better portability
• Environments and hardware
• Enhance testing capabilities
– Achieve more testing for the same effort (or less)
– Testing beyond traditional manual approaches
© CapgeminiSogeti 23
Captured script
Test Analysts Technical Test Analysts
Record/ Edit
Test Case Test Case Test Case Test Case Test Case Test Case Test Case
Record/ Edit
Test Script Test Script Test Script Test Script Test Script Test Script Test Script
Statements and test data
Manually
Test Tool
SUT
© CapgeminiSogeti 24
Example captured script
• Test case:
1. Open application login to network
2. Enter username and password
3. Click Ok
• Login screen – scripted by capture/playback tool:
– Login.Username.Enter(”ohansen”)
– Login.Password.Enter(”ata2015”)
– Login.OK.Press
© CapgeminiSogeti 25
PROS
• No initial effort
• Can start automating
quickly
• No programming skills
required
– Some developer
knowledge would be
beneficial
CONS
• Redundant statements for redundant actions – No reuse
• Difficult to understand – May be in an proprietary
language
• Change to the software can cause many scripts to be updated – Higher maintenance costs
Captured script – Pros/Cons
© CapgeminiSogeti 26
Data-driven
Test Analysts Technical Test Analysts
Test Case Test Case Test Case Test Case
Test Script Library
Low-level statements
Manually
Test Tool
SUT
Create
Data Files Data Files Data Files
Control Scripts
High-level statements
© CapgeminiSogeti 27
Data-driven
• Test data extracted from test cases / test procedures
– Placed into separate data files
• Control scripts reads data from the data files
– One script implements several tests by reading different data
• Reduces maintenance
• Multiple control scripts needed
– Per type of test
• Possible to automate more tests than it would be possible
to run manually
– Costs of automating some tests may be less than the cost of
running them manually
© CapgeminiSogeti 28
Example data-driven script
Login screen – test script – data-driven:
DataFile = Openfile(”Logindata”)
Read DataFile.LoginRec // reading the first row in the file
For each LoginRec in DataFile
Login.Username.Enter(LoginRec.Username)
Login.Password.Enter(LoginRec.Password)
Login.OK.Press
Verify (MessageBox.Text = LoginRec.Message)
Read DataFile.Login // reading the next row in the file
End Loop
Username Password Message ohansen atta2016 Welcome hbonde atta1963 Unknown user nlangkilde Xyz346 Incorrect password
© CapgeminiSogeti 29
PROS
• Reduced build cost
– Faster and easier to
automate similar test
procedures
– Test variations using
different test data
• Tests defined by data files
– Separate from scripts
– Test analysts can create
and maintain data files
CONS
• More initial work is needed – Development the control
and shared scripts
• Developer skills required – Create scripts
• Control/Shared
• Data files may not be an intuitive way to specify tests – Layout and form has to be
appropriate for the tester
Data-driven – Pros/Cons
© CapgeminiSogeti 30
Keyword-driven
Test Analysts Technical Test Analysts
Test Case Test Case Test Case Test Case
Test Script Library
Low-level statements
Manually
Test Tool
SUT
Create
Data Files Data Files Test Files
Control Script
Single control script
High-level and test data
© CapgeminiSogeti 31
Keyword-driven
• Keyword / Action Word
• Single control script
– ‘Interpreter’
• Test files (or test definitions)
– Independent of scripting language
– ‘Language’ tailored to testers business domain and business
processes
© CapgeminiSogeti 32
Example keyword-driven script
Keyword Username Password Result
LOGIN_KNOWN_USER ohansen atta2016 Welcome
LOGIN_UNKNOWN_USER hbonde atta1963 Unknown user
LOGIN_INCORRECT_PASSWORD nlangkilde Xyz346 Incorrect password
….. …. …. ….
© CapgeminiSogeti 33
PROS
• Reduced build cost
– Faster and easier to
automate similar test
procedures
– Test variations using
different test data
• Tests files provide more
natural way to specify the
tests
– Can be tailored
CONS
• More initial work is needed – Development the control
and shared scripts
• Developer skills required – Create scripts
• Control/Shared
• Keyword use needs control – Must avoid too many
Keyword-driven – Pros/Cons
© CapgeminiSogeti 34
ATTA Contents
1) The Technical Test Analyst’s Tasks in Risk-Based
Testing
2) Structure-Based Testing
3) Analytical Techniques
4) Quality Characteristics for Technical
Testing
5) Reviews
6) Test Tools and Automation