ASQ Seminar PresentationBy Denice Wong
ASQ Seminar Deck. Contact
[email protected]
Seminar is 3 hours, includes 3 demos for RMT, RFT and RPT. Demos
run in seminar VM from
ftp://pokgsa.ibm.com/projects/v/vm_images/050610_BBryson_ASQ_Seminar.
IBM Software Group | Rational software
Challenges Associated With Testing
Unable to perform full regression testing
Inaccurate performance prediction on production environment
Here are some of the challenges of testing – automated or
otherwise:
Insufficient Requirements – I think we’ve all experienced this…and
especially for newcomers to a testing team. We’ve all be assigned
to a project where we’ve been given a PC and told to “test the
app”…no help, no requirements…well the results of those tests, as
you would expect are not usually great. Without some guidance on
what to test, and what results should be expected for a given set
of inputs, the tests are not likely to find any significant
problems beyond “Alt+F doesn’t open up the File menu”.
“Changing requirements” is the next variation of that problem. When
system specs are changing so rapidly that the QA team can’t keep
up, tests that used to be perfectly valid may be of little or no
use – or worse they may find “false positives or false defects” –
defects that aren’t true defects that waste time as they get
entered into the defect tracking system, and consume project
resources.
The waterfall process model is a development practice that leaves
all QA to the end of a project. This cultivates multiple issues.
First of all, there is no validation of the quality as you go
along. It is not until very late in the cycle where you will
discover potentially critical bugs and architectural issues that
might take a lot of rework to correct. If it weren’t for schedule
slips, this might be manageable, but virtually all projects
experience some schedule slip that inevitably takes away from the
time allotted for testing.
Poor unit testing is frustrating for QA teams, as they often cause
road-blocking defects. Defects that should have been found earlier
and were not impede other integrated tests.
And, of course, time. I’ve already brought up the schedule slip,
and it’s worth it’s own point here – as no QA team ever really has
the time they need to do the job they want.
IBM Software Group | Rational software
An Example: Challenges of Siebel Implementations
a
Testing costs vary widely depending on project type and
complexity
“Customers are spending close to $½ Million on testing complex
projects”1
1 Source: Siebel Test Automation Datasheet
Survey of more than 130 Siebel projects showed that:
Not unusual for customers to spend ~25% of total project costs on
testing
Testing costs vary widely depending on project type and
complexity
Customers are spending close to $½ Million on testing complex
projects
Chart1
109998
212400
462000
10%
9.17
17.70
38.50
8.25
10%
16.82
5%
37.54
3%
2.44
73%
10.09
43%
33.65
13%
10%
16.00
16.67
17.34
14.40
10%
15.83
5%
16.90
3%
4.25
73%
9.50
43%
15.15
13%
10%
9.34
14.50
53.50
8.40
10%
13.78
5%
52.16
3%
2.48
73%
8.27
43%
46.76
13%
5%
4.67
7.25
26.75
4.20
10%
6.53
10%
24.08
10%
2.94
37%
4.57
37%
16.85
37%
Total
93.35
145.00
535.00
86.51
7%
137.28
5%
511.86
4%
43.55
53%
79.85
45%
339.64
37%
ahan: This number is estimated since there was no survey data in
this group. Estimated based on linear extrapolation from medium and
complex.
ahan: Percentage breakdown based on survey responses
ahan: Percentage breakdown based on survey responses
ahan: Percentage breakdown based on survey responses
ahan: Percentage breakdown based on survey responses
ahan: Percentage breakdown based on survey responses
ahan: Percentage breakdown based on survey responses
Cost Model by Feature
10%
9.17
17.70
38.50
9.17
17.70
38.50
calc
0.92
8.85
34.65
9.17
17.70
38.50
9.17
17.70
38.50
95%
8.71
16.82
36.58
9.17
17.70
38.50
80%
7.33
14.16
30.80
9.17
17.70
38.50
10%
16.00
16.67
17.34
16.00
16.67
17.34
calc
1.60
8.33
15.60
16.00
16.67
17.34
16.00
16.67
17.34
95%
15.20
15.83
16.47
16.00
16.67
17.34
80%
12.80
13.33
13.87
16.00
16.67
17.34
10%
9.34
14.50
53.50
9.34
14.50
53.50
calc
0.93
7.25
48.15
9.34
14.50
53.50
9.34
14.50
53.50
95%
8.87
13.78
50.83
9.34
14.50
53.50
80%
7.47
11.60
42.80
9.34
14.50
53.50
5%
4.67
7.25
26.75
95%
4.43
6.89
25.41
4.67
7.25
26.75
4.67
7.25
26.75
4.67
7.25
26.75
4.67
7.25
26.75
90%
4.20
6.53
24.08
4.67
7.25
26.75
90%
4.20
6.53
24.08
1
2
93.35
145.00
535.00
83.64
129.92
479.36
75.87
129.92
523.87
76.63
119.03
439.17
90.62
140.76
519.35
90.55
140.65
518.95
88.22
137.03
505.58
84.95
131.95
486.85
90.25
140.18
517.21
TOTAL
93.35
145.00
535.00
10%
10%
10%
19%
10%
2%
18%
18%
18%
3%
3%
3%
3%
3%
3%
5%
6%
6%
9%
9%
9%
3%
3%
3%
ahan: These Numbers are based on Testing customer survey. 130 total
respondents
ahan: These Numbers are based on Testing customer survey. 130 total
respondents
ahan: These Numbers are based on Testing customer survey. 130 total
respondents
ahan: This number is estimated since there was no survey data in
this group. Estimated based on linear extrapolation from medium and
complex.
ahan: Percentage breakdown based on survey responses
ahan: Percentage breakdown based on survey responses
ahan: Percentage breakdown based on survey responses
ahan: Percentage breakdown based on survey responses
ahan: Need PS to validate these numbers
ahan: Percentage breakdown based on survey responses
ahan: Percentage breakdown based on survey responses
ahan: Need PS to validate these numbers
ahan: Need PS to validate these numbers
ahan: 90% savings for simple 50% savings for medium 10% savings for
complex
Cost Model by Type
10%
9.17
17.70
38.50
10%
16.00
16.67
17.34
10%
9.34
14.50
53.50
ahan: Percentage breakdown based on survey responses
ahan: Percentage breakdown based on survey responses
ahan: Percentage breakdown based on survey responses
ahan: Percentage breakdown based on survey responses
ahan: Percentage breakdown based on survey responses
ahan: Percentage breakdown based on survey responses
ahan: This number is estimated since there was no survey data in
this group. Estimated based on linear extrapolation from medium and
complex.
Cost Model by Type
0
0
0
Do you ever have enough time to test everything???
Trade coverage for risk
Reduce scope of testing
Introduce / enhance automation
…and that’s often why people come to test automation.
They run out of time, and look for ways to get more done with
less…usually they’ll examine ways of cutting tests (“Just test what
is really REALLY important.”), cutting cycles (“We only have time
to test every other milestone build.”), or configurations (“We’ll
test on Windows XP, and just assume it works on Windows 2000”) – or
they’ll consider adding people to a project (few projects have the
resources or budget to simply increase staff, however) – or they’ll
look to become more efficient by using automation.
IBM Software Group | Rational software
Automation Adds Options!
Improve human productivity
Broader test coverage
Tie testing to requirements
…And automation certainly offers the hope of becoming more
efficient with the resources you have. You have the opportunity to
improve human productivity – by running test overnight. You can
remove the tedium: Let the computer handle repetitive tasks and
thus enable your team to spend more time on complex tests and test
designs. You also remove the human error factor, which is bound to
creep into any human-intensive process.
Automation also offers the hope of process improvements – enabling
you to execute more tests on every build of your application.
Continuously regression testing each build helps ensure that things
that worked in one build are not broken by new features or bug
fixes in the next build.
So – there are obviously some great benefits to automation – the
devil, as they say, is in the details…so let’s dive into the
details of implementing test automation into your
organization.
IBM Software Group | Rational software
What Does Test Automation Mean?
Do my testing for me!
Of course, this is what we’d like test automation to mean. Wouldn’t
that be nice?! Unfortunately, the tool has yet to be invented that
will read the mind of the designer to determine how he intended to
build it and read the mind of the end user to determine how he
intended to use it. This is just one misconception of automation.
Automated test tools do not simply do your testing for you. They
help to automate the time-consuming, error-prone, mundane tasks,
but there is certainly still some thinking required.
IBM Software Group | Rational software
What is Automated Functional
Using automation tools to record actions taken against an
application.
Test scripts are derived from these recordings which are then used
for subsequent test execution.
Record
Enhance
Execute
BBryson: Introduce here the three main steps of creating a script.
Note that few vendors talk about the enhance step. It’s a critical
stage, but often overlooked and underestimated by those taking a
first step into automation. Hopefully today we can steer you in the
right direction and set some proper expectations for you. [WE’RE
ATTEMPTING TO BUILD CREDIBILITY AGAINST MERCURY’S RECOCRD-PLAYBACK
CLAIMS, BY BEING MORE UP FRONT AND HONEST ABOUT THE REALITY OF
AUTOMATION].
IBM Software Group | Rational software
What Is Automated Performance Testing?
The process of exercising an application by emulating actual
users
with a load generation tool for the purpose of finding system
bottlenecks
Controller
First, what is performance testing?
It is the process of exercising an application by emulating actual
users putting a load on your system for the purpose of finding
system bottlenecks. This load is generally created by a load
generation tool. You typically have a master computer that is
controlling other computers – as well as ‘virtual testers’ – to
emulate this production load.
Once the test run is complete all the data is returned to the
master where the user analyzes the data to find the performance
bottlenecks.
There are probably a million different definitions which we could
use, but I’m hoping we can just start from here and build on this.
From here we’ll move forward take a look at the various challenges
of performance testing.
IBM Software Group | Rational software
What is Automated Test Management?
Using automation tools to accelerate the process of test planning,
test creation, test execution and test results analysis
Plan
Create
Execute
Analyze
There’s probably as many valid and arguable definitions for
automated test management as there are people in this room today,
but just for the purposes of out presentation, let’s go with the
one we have here. [READ DEFINITION FROM SLIDE]
It’s quite a narrow definition, and I’m doing myself somewhat of a
disservice by using this – as it ignores many of the areas of
automation that IBM Rational can help out with, but we want this
narrow focus for today [YOU’RE TEASING AND ALLUDING TO ROBOT,
PURIFYPLUS, ANY OTHER TOOL THAT THEY MAY BE INTERESTED IN HERE BY
SAYING THIS] , so that we can drill a little deeper on just
plan-create-execute-analyze areas.
IBM Software Group | Rational software
IBM Software Group | Rational software
Challenges of Functional Testing
Inexperienced testers face steep learning curve for tools
Test maintenance outpaces testing
“Our testers are not experienced in building automated
tests.”
“Application UI changes are always breaking our test
scripts.”
The main point on this slide is that test maintenance becomes a
significant cost for test teams as they grow their testing
projects. Test maintenance is the death of automation
projects.
The text in the bubbles illustrates sample comments/complaints that
your customers may have heard from their teams.
IBM Software Group | Rational software
Challenges of Functional Testing
Inexperienced testers face steep learning curve for tools
Test maintenance grows with project
Testing Tools Lack Flexibility
Power users hands are tied by tool limitations
“I need one tool for both my expert and novice testers.”
“Our script language limits our ability to overcome some automation
obstacles.”
The main point on this slide is that in other test tools (like
Mercury’s QuickTestPro) the product’s efforts to simplify testing
make it difficult for power users to overcome inevitable challenges
of automation.
The text in the bubbles illustrates sample comments/complaints that
your customers may have heard from their teams.
IBM Software Group | Rational software
Functional Testing with IBM Rational Functional Tester
Test automation for the novice and the professional
IBM Rational Functional Tester
Key Benefits
Minimize test maintenance with scripts resilient to application
changes
Wizard enhanced automation to speed test creation for the new
user
Powerful scripting language and IDE for the professional
tester
Supports Team oriented parallel development
IBM Software Group | Rational software
Effective Test Automation
Execute
Script Enhancing
Script Execution
Enhance
Record
BBryson: When it comes to script execution, you can run locally on
your test development machine or leverage TestManager and run
remotely on distributed machines, even Linux machines.
IBM Software Group | Rational software
Recording Scripts
Separates test data from test script
Enables a single script to run multiple tests by using multiple
data sets
Wizard driven process
No programming involved
Recording Scripts
Validating Dynamic Data
Dynamic Data/Content Matching
Use pattern matching technique to verify dynamic data and create
robust tests
E.g. Instead of validating “Order ID 230”, validate “Order ID
###”
or Order ID 2##, etc.
This allows for a wide variety of acceptable responses as well as
restrictions on acceptable responses when validating the
application’s behavior
Order ID changes with each order placed
Test Fails when comparing static baseline to dynamic data
Test Passes when comparing pattern baseline to dynamic data
Dynamic Data Validation solves another very common problem with
java and web applications.
In many applications, there will be dynamic data that changes with
every run of the application. In the case we have here, you can see
that every time an order is placed a new order number is generated.
Any attempt to compare this order number to a static baseline will
result in a test failure.
So how does one validate ever changing data in an
application.
<<< CLICK >>>
The answer is to use Pattern Matching technology. Instead of trying
to match a dynamic value to a static baseline, you match it to a
pattern. In our case here, we match simply against 3 digits. If we
get a three digit number, the test passes. This of course is a very
simple pattern. Using industry standard Regular Expression notation
(common in popular parsing languages such as PERL), you can create
any imaginable pattern to validate any imaginable type of dynamic
data!
IBM Software Group | Rational software
Enhancing Scripts
Visual Studio .net
Code assist editors
Reduce Test Script Maintenance
Object Map
The object map is a collection of descriptions of test
objects
It is automatically populated during test script recording
It contains properties for each test object
Recognition properties enable Rational XDE Tester to find the test
object during playback
Changes to an object need only be made once – in the map
Central Object Map
Panel
The object map is a collection of descriptions of test objects that
is automatically populated during test script recording – but also
allows you to manually add objects later if you so desire.
For each object, the map stores properties and values that identify
the object. It is the recognition properties that enable Rational
XDE Tester to find the test object during execution of a test
script.
In the event the application changes substantially and and changes
to an object‘s identification are needed, the change can be made in
the central object map and all test scripts that reference that
object are automatically updated. Keeping script maintenance, when
required, to a minimum.
So, now we know how objects are stored, and how centralized object
maintenance can be achieved. The next question is how ScriptAssure
can virtually eliminate the need for script maintenance?
Objective
Key Points
Recognition properties are used to find (recognize) the
object.
Administrative properties are all the properties not used to find
the object (mostly used to specify what can be done to the
object).
IBM Software Group | Rational software
Reduce Test Script Maintenance
Version 1.0
Version 2.0
No User
DeterminesMatch
Tester
Sees
Tool
Sees
Let’s take a closer look at the ScriptAssure technology.
ScriptAssure is the technology that makes test scripts immune to
object name changes between software builds. This is a critical
feature to reduce script maintenance and increase script
reliability.
Here we have two builds of the same application.
<<< CLICK >>>
The scripting tool sees the underlying attributes which allow the
tool to uniquely identify the objects being testing. The problem
is, when the attributes are changed build to build as the
application gets constructed, the test scripts can no longer
uniquely identify the object to be tested, thus breaks. In the
past, the Tester would need to manually intervene using a GUI
Mapper to identify to the script what it is was supposed to
test.
<<< CLICK >>>
Unique to Rational, the new technology tool can identify objects
even after the identifiers have changed. It uses many different
attributes, weighs them proportionately, then uses a sensitivity
setting to identify the object. This way, should one or a few
properties – such as an object name – change between builds, it can
still recognize the correct object.
<<< WAIT FOR CHECK MARK TO APPEAR >>>
The benefit is that even as object properties change between
builds, the scripts will still playback. This decreases script
failures by making scripts more resilient to change. Of course, all
of this translates to less script maintenance and more time for the
Tester.
IBM Software Group | Rational software
Summarizing Key Features in IBM Functional Tester
Broad Environment Support
Web, Java. .net
Static data and properties verification
Dynamic data validation without coding
What can’t or shouldn’t be automated?
Flexible and Extensible Scripts
IBM Software Group | Rational software
IBM Software Group | Rational software
Challenges of Performance Testing
Complex Tools & Complex Task
Tool complexity and lack of experience intimidates many first time
users
“We don’t have a lot of experience building good load tests.”
“Most load test tools are much more complicated than we
require.”
BBryson: Emphasize this point to sell our ease of use benefit.
Visual editor is the feature we’re getting at. The though bubbles
are common symptoms that audience may have heard and resonate
with.
IBM Software Group | Rational software
Challenges of Performance Testing
Complex Tools & Complex Task
Tool complexity and lack of experience intimidates many first time
users
Testing Tools Lack Flexibility
Power users hands are tied by tool limitations
“We don’t have a good understanding of what exactly is occurring
during a load test.”
“We have highly technical performance testers that need to
customize their tests.”
BBryson: Emphasize this point to sell benefit of being able to get
at all page details and add custom java code
IBM Software Group | Rational software
Challenges of Performance Testing
Complex Tools & Complex Task
Tool complexity and lack of experience intimidates many first time
users
Testing Tools Lack Flexibility
Hidden Costs
Hardware intensive tools and inability to share information with
team mates increases cost of testing
“Our developers have no way of performing load tests.”
“We don’t have enough hardware to run our tests.”
BBryson: Emphasize this point to sell benefit of low memory
footprint of tool
IBM Software Group | Rational software
Performance Testing with IBM Rational Performance Tester
Test automation for the novice and the professional
IBM Rational Performance Tester
Key Benefits
Mask complexity to get the job done
Advanced Data Access & Manipulation
Low resource requirements
Desktop IDE integration
Creating a Performance Test
Execute & Analyze
Schedule Workload
Build Scripts
Scheduling Considerations
Execute and Analyze Considerations
BBryson: Analysis is relatively strong in the tool.
IBM Software Group | Rational software
No-Code Tests
Tests are represented in a tree view as a list of web pages and
their elements, such as images.
No programming necessary to create a test
Underlying detail is accessible but hidden to simplify test
creation and maintenance
IBM Software Group | Rational software
Creating a Performance Test
Literal values used during recording are replaced with
variables
At playback time, variables are populated with data from a
datapool
Data for playback can be imported from external data sources
No user coding involved
User Name: JSmith
User Name: CBryson
User Name: TJones
Session No. 100
Session No. 101
Session No. 102
Datapooling is the process of making sure that each user submits
unique data to the server. If I run a test for 1,000 users, I want
to make sure that each user submits a different user name, e-mail,
zip code etc… Why? If I don’t – the server may cache the response
resulting in overly optimistic response times. The good news is
that Performance Tester handles this automatically for me. It
generates scripts that are already pre-programmed to feed off of a
datapool of values. I just need to create that data and attach it
to the test.
IBM Software Group | Rational software
LoadSmart scheduling
scheduling
Historically, load testing has been very difficult. That’s one of
the reasons more companies don’t do it. You’ve got very complex
scenarios to model and multiple transactions and data exchanges to
emulate. Rational Suite has taken the ‘difficult’ out of this
task.
Rational Suite TestStudio provides a graphical point-and-click
scheduler that lets you model your user population and activity
quickly and easily.
Your schedule can include already-recorded functional scripts (as
you might find in your test plan) and/or virtual user scripts used
solely for load testing..
And because all user groups are percentages of the whole, changing
your load test from 100 to 10,000 plus users is as easy as changing
one number. Any data that is passed during these scripts can be
specific (read from existing data tables) or parameterized, i.e.,
read from a random table of data that is created on the fly. There
is no programming required here and that will help you get started
much faster.
IBM Software Group | Rational software
Creating a Performance Test
Coordinate timing and dependencies
Dynamically increase the load during the run
BBryson: A nice, but not unique, feature of the tool is the ability
to dynamically add users during a test run. Other tools can do
this, so don’t over sell…also, we cannot schedule additional users
– it has to be done ad hoc during test execution…however, it does
give prospects the ability to crank up the load during a test to
break a server.
IBM Software Group | Rational software
Creating a Performance Test
All activity coordinated and monitored
from master console
Key considerations for analysis
Find the slow pages
Find resource bottlenecks
BBryson: Resource monitoring is better in RPT v6.1 than it was with
Robot. We now have access to all perfmon counters. Provides lots of
data.
IBM Software Group | Rational software
Find Slow Pages
Easily determine slow performing pages of your web site
Reports are available live during the run and for post run
analysis
BBryson: This is the first report prospects will see when they use
the tool. It shows the average page load response times for all
users.
IBM Software Group | Rational software
Find Resource Bottlenecks
Capture and analyze a long list of performance monitoring
statistics
Determine your resource bottlenecks
BBryson: This is a screenshot of the resource monitoring ability.
On the left is a list of counters to monitor. The graph shows which
ones have been selected and are being measured.
IBM Software Group | Rational software
Identify the Root Cause of Performance Problems
IBM Performance Optimization Toolkit for Rational Performance
Tester
System
under
test
Analyze
Speed delivery
Developer
Tester
Speed changes by providing detailed data to developers
User
Load
1
2
b
c
[NOTE TO SPEAKER] The IBM Performance Optimization toolkit is a
free developerWorks download for all customer with Rational
Performance Tester 6.1
KEY POINTS FOR IBM PERFORMANCE OPTIMIZATION TOOLKIT:
Monitors web applications running on the application server during
performance testing under load
Provides analysis of the collected performance trace data in
Rational Performance Tester to identify where the root cause of the
application bottleneck is located
Enables analysis and correlation of log files of the tiers of the
tested J2EE system leveraging the Self -Managing Autonomic
Capabilities from Tivoli
------------------------------------------------------
GENERAL PROBLEM STATEMENT FOR BOTH TOOLKITS:
Once an application goes to production, it becomes increasing
difficult and expensive to isolate any performance or functionality
problems that may come up. Up until now, the best the IT Operator
could do was attempt to appropriately tune the system to
"alleviate" the problem as best as they could, then point to a
suspect transaction, and take a guess at what was happening at the
point in time the Performance or Functional flaw surfaced and
provide this information back to the development team. The
development team would eventually pick up on the problem and take a
shot at reproducing it in their test environment. It could take
days or weeks to isolate the problem, especially those related to
memory leaks or code bottlenecks that only become "problematic"
under load, or after hours of continuous execution - both of which
happen in production.
IBM SOLUTION (S)
IBM Problem Resolution Toolkit for Rational Application Developer
links the developer directly to data that has been collected in a
production environment, allowing the developer to quickly and
efficiently identify the root cause of performance bottlenecks and
provide a resolution.
and
IBM Performance Optimization Toolkit for Rational Performance
Tester collects performance trace data for the applications running
on the application server in the test lab. It provides analysis
capabilities of the collected data and makes them available for the
developer for efficient root cause analysis and resolution.
IBM Problem Resolution Toolkit helps reduce problems once they
entered production and IBM Performance Optimization Toolkit helps
the Performance Tester to detect and with the help of the Developer
fix the problems before they get to the production.
These toolkits enable a faster, automated and more coherent process
for application problem resolution. These toolkits link the
developer directly to data recorded in either the production
environment or the test lab.
KEY POINTS FOR IBM PERFORMANCE OPTIMIZATION TOOLKIT:
Monitors web applications running on the application server during
performance testing under load
Provides analysis of the collected performance trace data in
Rational Performance Tester
Enables analysis and correlation of log files of the tiers of the
tested J2EE system leveraging the Self -Managing Autonomic
Capabilities from Tivoli
Enables sharing of the recorded performance trace data with the
developer using Rational Application Developer or Rational Software
Architect
* Includes self-managing autonomic capabilities
HOW DOES IBM PERFORMANCE OPTIMIZATION TOOLKIT WORK:
Tivoli Monitoring for Transaction Performance (TMTP) Software
monitors application performance and quality in the production
environment, tracing the application execution and saving away
application and system details when it identifies that a
performance or quality issue exists. The IBM Performance
Optimization Toolkit uses Data Collection Infrastructure (DCI) that
is based on the TMTP technology in the test lab. This enables the
Performance Tester to easily and quickly identify the root cause of
performance or quality problems uncovered during the test phase.
Through the use of IBM's self-managing Autonomic Capabilities, the
toolkit can suggest probable causes and resolution. If the probable
cause is linked to the application source code, the recorded data
can then be shared with the developer for quick resolution
FLOW:
1: Need to pinpoint/recreate performance problems in test Lab
1a: Tester initiates multi-user test with RPT against system in
test lab
1b: Results about slow performing pages and page elements captured
and analyzed
1c: Root Cause identification of bottlenecks done to the method
level with data collection infrastructure (from Tivoli)
2: Isolated defect submitted into ClearQuest for Developer to take
immediate action
In the test lab, the tester is using Rational Performance Tester
(RPT) to apply a user load on the target application environment.
The user takes advantage of the real time reporting capabilities of
RPT to view the response times being achieved. If the response
times are acceptable, the user takes advantage of the Dynamic Load
capabilities of RPT and dynamically increases the user load until
they recognize that certain transactions (page hits) are now
exceeding a target response time threshold. The Tester engages IBM
Performance Optimization Toolkit ( which is powered by Tivoli
Monitoring for Transaction Performance tools) to turn on
application tracing, gathering all of the execution data associated
with the slow transaction that was observed. Once the data is
gathered, the tester reviews the data to identify the root cause.
They compare the data they have obtained to data contained in the
Symptoms Database for Autonomic Computing. This Symptoms Database
is a cross reference to commonly known problems that are linked to
specific types of data being observed. In some cases, a tuning
recommendation will come out of the Symptoms Database, in other
cases, the tester will have to seek additional help to drill into
the root cause. Under this scenario, the tester will file a defect
in ClearQuest, and link that defect to the application trace data
they have recorded/stored. The developer will see the defect and
review the trace data recorded by the Performance Tester in the
test lab. After analyzing the test results the Developer the
Developer starts the debugging/isolation activity just like they
did with the data from the production environment. When the
developer has drilled down to the line of code that is the root
cause, they will resolve the problem and create a new build to push
into the test lab.
WHAT IS UNIQUE TO IBM:
Consider these steps used in the two scenarios above:
1. Synthetic Transaction Generation in the production environment:
- Keynote, Mercury, IBM and many more
2. Record and store detailed trace data associated with the slow
transaction - Wily, Mercury, IBM and more – unlike Wily Tivoli is
integrated in the production environment and remains active
24x7
3. Package that data up to be shipped to the Developer - Wiley,
IBM
4. Import the data into the Developer IDE and perform Jump to
Source Code Analysis - IBM ( Wiley has some jump to code
capabilities, but not in the Developer’s IDE). The data recorded by
IPOT is ready for the Development environment. That’s a unique IBM
capability – both RPT and RAD/RSA use the same data.
5. Use the IDE to fix the problem - IBM (Borland can also do
this)
6. Use ClearCase to perform another build for the Change Management
tools in the production Space – IBM, CVS (open source)
IBM Software Group | Rational software
Identify the Root Cause of Performance Problems
The IBM Performance Optimization Toolkit decomposes poor response
times identified during a performance test…
into the amount of time spent within each application
component.
IBM Software Group | Rational software
Summarizing Key Features in Performance Tester
Built for Day 1 Productivity
Create code free tests
Advanced Data Access & Manipulation
Create code free schedules
Low resource requirements
Desktop IDE integration
BBryson: Wrap up here referring back to the challenges of Complex
Tools, Lack of Tool Flexibility and High cost of performance
testing.
IBM Software Group | Rational software
IBM Software Group | Rational software
Why Should I Test?
Establishes a link between the test and the motivator
Motivator can be RequisitePro Requirements, Rose Model, Excel
Spreadsheet, or custom
Establishing link enables requirements coverage reporting
Test Input
…let’s examine these last two items in more detail, starting with
the “why” – Why should you create and run a given test?
The “why” of the test defines what is driving the need for the
test. There are many reasons you could be running the test – but
generally the most common reason is because there is a system
requirement that says “The system should do X”, therefore you
should have a test “for X”.
Establishing this link accomplishes two key objectives for you.
First, should the underlying requirement or motivator change –
you’ll get a notification in TestManager that the test is now
“suspect”. A “suspect” test is one who’s motivator has changed. The
idea here is that if the requirement changes, you may need to
re-evaluate the test to determine if it needs to change as
well.
Establishing the reason for the test case also will enable you to
do requirement coverage reporting. With the link established you’ll
be able to see which of your requirements are passing and which
have failed…and which have not been tested. Invaluable information
when it comes to release time.
IBM Software Group | Rational software
Suspect Test Cases
Suspect Test Cases show up with red slashes in the Test Inputs
View
Suspect Test Cases show up in bold in the Test Plan View
AND
Within TestManager:
Suspect test cases display in both the Test Inputs View and the
Test Case view.
IBM Software Group | Rational software
Automatically Track Impact of Changing Requirements
Automatic Notification of Changed Requirements that Affect Test
Cases
With TestManager and RequisitePro, that’s easy. Any change to a
requirement automatically causes the test case to be marked as
“suspicious”. That means that the test case need to be re-evaluated
to see if the change in the requirement impacts the testing. And
TestManager makes that very easy.
There are several ways that TestManager notifies the test team
about a change. Here is just one way. This report marks in Red any
test case whose associated requirement has been changed. And, by
drilling down right from this report, you can find the details of
the change to understand what impact this change will have on
testing.
By TestManager and RequisitePro working together, no changes to
requirements will go undetected. This ensures that your testing
team is always working with the latest information – no matter how
often or how quickly requirements changes.
IBM Software Group | Rational software
Comprehensive Change/Defects Management
Script
Requirement
One of the greatest advantages of using Rational Suite TestStudio
is the team unification power you gain. Let’s take the example of
requirements. You may have formal requirements or a product spec in
a database or a Word doc. You can look at this requirement from
Rational TestManager, the test planning and test management tool.
From here you can assign or create test scripts that will be used
to measure functional compliance against that requirement.
But, requirements change. They are dynamic, right? No problem with
Rational Suite TestStudio. It doesn’t matter where you update your
requirement - work wherever you are most comfortable. Because the
requirement is stored in a central repository, you can edit it from
Word, RequisitePro, or TestManager - and it gets automatically
updated everywhere.
More importantly, when logging a defect, you can note which
requirement this test script was running against. So when the
developer or analyst views the defect, they know immediately which
business requirement is broken. Rational ClearQuest, the integrated
defect management component, automatically pulls in that info when
you generate the defect. The script, verification point,
input/requirement, log results, etc. are ALL attached automatically
to the defect note, ensuring that the developer can reproduce it
quickly and easily.
Rational ClearQuest has dozens of built-in reports you can run with
a single mouse click, so anyone on the team can find out the status
of the project at that moment in time!
=======================
If you are using the built-in defect tracking tool, Rational
ClearQuest, to handle your change requests, a simple right-click
lets you submit a defect. The nice part here is that when you
choose submit defect, any relevant test data that went into
generating that fail message automatically gets attached to the
defect – including the test script, results, machine configuration,
test input or requirement that’s being failed, … everything. Again,
extreme traceability is the key here.
IBM Software Group | Rational software
Control Your Process
Track Test Thoroughness
Defect Tracking (ClearQuest)
One of the greatest advantages of using Rational Suite TestStudio
is the team unification power you gain. Let’s take the example of
requirements. You may have formal requirements or a product spec in
a database or a Word doc.
You can look at this requirement from Rational TestManager, the
test planning and test management tool. From here you can assign or
create test scripts that will be used to measure functional
compliance against that requirement.
But, requirements change. They are dynamic. No problem with
Rational Suite TestStudio. It doesn’t matter where you update your
requirement - work wherever you are most comfortable. Because the
requirement is stored in a central repository, you can edit it from
Word, RequisitePro, or TestManager - and it gets automatically
updated everywhere.
IBM Software Group | Rational software
Control Your Process
Track Test Thoroughness
Finally, to close the loop, you can generate reports on
requirements coverage. Can’t release the product until it meets all
its business requirements, right? Either that or you’ve got to
start dropping features. Well, with Rational Suite TestStudio you
know where you stand at any point in time.
IBM Software Group | Rational software
Summarizing Key Features
Simulate Real World Workload
Create real world workload model to make sure the test result is
meaningful
Effortless Analysis
test reports
Hopefully I’ve given you a realistic view of what you can expect as
you take your first step towards automated test asset management.
Let’s review some of the keys to delivering quality on time…
Crucial is the ability to support all types of testing…performance,
functional and manual. By automating all phases of the testing life
cycle, from planning through execution to data analysis, the tools
will ensure your test group runs at maximum efficiency.
Configuration awareness ensures that TestManager only runs tests on
the appropriate platform. Tracking this effort manually can consume
many needless hours as testers maintain complex checklists of what
has and has not been done.
Finally, the tools enable you to effortlessly track your testing
progress. By taking care of all the data gathering and reporting
for you, TestManager can instantly keep you up to date on test
progress reporting, requirement changes and configuration testing …
to name a few.
IBM Software Group | Rational software
The IBM Rational Software Development Platform
A complete, open, modular, and proven solution for Software Quality
Assurance
Customer
Extensions
The IBM Software Development Platform is based on an architectural
foundation called the Eclipse universal framework. This framework
enables unprecedented tool integration and artifact traceability
throughout the development lifecycle.
What is Eclipse? It is both an open source software development
project—with over 100 active vendor participants—and a rich
foundation for providing shared services across an application
tooling environment.
Part of the Eclipse Project¹ is the Eclipse Modeling Framework
(EMF), which offers essential capabilities, notably code
generation, meta-data management, and editor support.
Eclipse offers customers:
Role-based user interfaces that allows each member of the team to
view information in the way that is optimized for their technical
specialty.
It provides common models for integration across the lifecycle.
This makes it easier for all vendors—including IBM—to integrate
their products by mapping meta-data to a shared set of common
models.
-----------------------------
“The Eclipse Project is an open source software development project
dedicated to providing a robust, full-featured, commercial-quality,
industry platform for the development of highly integrated tools.
The Eclipse Tools Project provides a focal point for diverse tool
builders to ensure the creation of best of breed tools for the
Eclipse Platform. The mission of the Eclipse Technology Project is
to provide new channels for open source developers, researchers,
academics and educators to participate in the on-going evolution of
Eclipse.”* Part of the Eclipse project is the Eclipse Modeling
Framework (EMF), which offers essential capabilities, notably code
generation, meta-data management, default serialization, and editor
support. For more information on the eclipse platform, please visit
eclipse.org.
*From eclipse.org FAQ on www.eclipse.org
What is the planned architectural direction of the IBM Rational
solution?
We intend to invest in and leverage shared components based on the
Eclipse open source platform. This architectural direction reflects
our deep commitment to open standards– including UML2, Java, J2EE,
and XML, and Web service standards. In concert with our customers
and partners, we envision a vibrant tools and services ecosystem
supported and extended by a rich community of developers, partners,
and Independent Software Vendors
IBM Software Group | Rational software
What’s Next?
Rapid Web Application Deployment Seminar (26th May, 2006)
Please fill in the feedback form and get IBM Rational evaluation CD
kit !!
Cost of testing new Siebel installation
$0
$100,000
$200,000
$300,000
$400,000
$500,000