View
78
Download
1
Category
Preview:
Citation preview
a p p e n d i x b
Test Tool Evaluations
Anyone who has contemplated the implementation of an automated test tool has
quickly realized the wide variety of options on the market in terms of both the kinds
of test tools being offered and the number of vendors. The best tool for any partic-
ular situation depends on the system engineering environment that applies and the
testing methodology that will be used, which in turn will dictate how automation
will be invoked to support the process.
This appendix evaluates major tool vendors on their test tool characteristics,
test execution capability, tool integration capability, test reporting capability, per-
formance testing and analysis, and vendor qualification. The following tool vendors
evaluated are Compuware, Empirix/RSW, Mercury, Rational, and Segue.
243
s
R
L
D
28206 03 pp. 217-282 r11kp.ps 7/26/01 3:10 PM Page 243
S
R
L
Appendix B Test Tool Evaluations244
Intuitive and easy to use for users newto automated testing tools
Intuitive and easy to use for users newto this particular tool but who haveused others
Easy to install; tool may not be used ifdifficult to install
Tasks can be accomplished quickly,assuming basic user proficiency
Easy to maintain automated tests witha central repository whereby users canseparate GUI object definitions from thescript
Can vary how designs and documentsare viewed (zooming, multipagediagrams easily supported, multipleconcurrent views up at the same time);basic windowing
QACenter—YesPerformance Edition—Yes
QACenter—YesPerformance Edition—Yes
QACenter—YesPerformance Edition—Yes
QACenter—YesPerformance Edition—Yes
QACenter—YesPerformance Edition—Yes
NA
eTest Suite—YesUses Visual Scripts whichrequire no programming.
eTest Suite—YesMaintainable VisualScripts, fully extensibleprogramming
eTest Suite—YesNo proxy servers or specialconfigurations
eTest Suite—Yes
eTest Suite—YesUse of shared workspace,VBA code shared betweenscripts
eTest Suite—YesView test results/datain MDI windows, andswitch between
Table B.1 Test Tool Characteristics:Capture/Playback and Script Creation
Criterion/Feature Compuware Empirix/RSW
Ease of Use
Tool Customization
Fully customizable toolbars to reflectany commonly used tool capabilities
Tool customizable: fields added,deleted
Performance Edition—Yes
NA
eTest Suite—Notcustomizable, butcommon tool capabilitiesin the toolbar
NA
28206 03 pp. 217-282 r11kp.ps 7/26/01 3:10 PM Page 244
s
R
L
245B.1 Test Tool Characteristics
Astra QuickTest—Yes
Astra QuickTest—YesWinRunner—Yes
Astra QuickTest—YesWinRunner—Yes
Astra QuickTest—YesWinRunner—Yes
Astra QuickTest—YesWinRunner—Yes
Astra QuickTest—YesWinRunner—Yes
Rational Suite TestStudio, RationalRobot—YesGUI interface, popup icon bars, tool tipsand on-line help.
Rational Suite TestStudio, RationalRobot—YesSQA Robot used for both GUI functional& performance tests.
Rational Suite TestStudio, RationalRobot—YesAn administrator should be assigned tohelp set up test asset datastore &license server.
Rational Suite TestStudio, RationalRobot—Yes
Rational Suite TestStudio, RationalRobot—YesScripts are maintained in a central datastore.
Rational Suite TestStudio, RationalRobot—YesDockable windows, toolbars and otherattributes allow interfacecustomization.
SilkTest—Yes
SilkTest—Yes
SilkTest—Yes
SilkTest—Yes
SilkTest—Set up so that bydefault, object declarations are inseparate include file soscript can have minimum codeand intuitive steps inside the testcases.
SilkTest—Uses format similar toWord’s outlining. Can zoom in andout of scoping by clicking onpluses and minuses. Can viewmany documents.
B.1 Test Tool Vendors: Compuware, Empirix and Mercury
Mercury Rational Segue
Ease of Use
(continued )
Tool Customization
Astra QuickTest—YesWinRunner—Yes
Astra QuickTest—YesWinRunner—YesTestDirector—Yes
Rational Suite TestStudio—YesToolbars, menu bars and icon bars arecustomizable.
Rational Suite TestStudio—YesThree custom fields used to elaborateon any test asset.
SilkTest—No
Silk Test—YesFeatures are customizable, but notthe GUI.
28206 03 pp. 217-282 r11kp.ps 7/26/01 3:10 PM Page 245
S
R
L
Appendix B Test Tool Evaluations246
Fully customized editor with formatsand colors for better readability
Tool support for required test procedurenaming convention
QACenter—YesPerformance Edition—Yes
QACenter—YesPerformance Edition—Yes
eTest Suite—YesCode editor fullycustomizable
eTest Suite—Yes
Table B.1 Test Tool Characteristics: Capture/Playback and Script Creation (continued)
Criterion/Feature Compuware Empirix/RSW
Tool Customization
Platform Support
Multiple platform support: UNIX,XWindows, Windows CE, Win3.1,Win95, Win98, NT, Win2000, WinME
Browser support: all versions ofNetscape, Internet Explorer
Cross-browser testing (record on IE,replay on NS, or vice versa) capability
Technology support for one or more ofHTML, DHTML, JavaScript, XML, Javaapplications and applets, MFC (C/C++),Visual Basic, PowerBuilder, Delphi,Terminal Emulator, Oracle, SAP,PeopleSoft, Siebel
QACenter—Supports variousWindows versions. PerformanceEdition supports Unix and variousWindows. Can also test serversserving content for mobile devices.
QACenter—YesPerformance Edition—Yes
NA
QACenter—YesPerformance Edition—YesSupports Java apps and applets ina customer’s JVM. Supports HTML,DHTML, JavaScript, XML, SAP,Oracle, PeopleSoft, Siebel
Supports Unix andvarious Windows.
eTest Suite—Yes
eTest Suite—Yes
eTest Suite—YesSupports Java apps andapplets w/in MicrosoftJVM/Sun JVM. HTML,DHTML, JavaScript, XML,MFC, Visual Basic,Oracle, SAP, Delphi,Terminal Emulator,PowerBuilder, Oracle,PeopleSoft
28206 03 pp. 217-282 r11kp.ps 7/26/01 3:10 PM Page 246
s
R
L
247B.1 Test Tool Characteristics
Astra QuickTest—YesWinRunner—Yes
Astra QuickTest—YesWinRunner—Yes
Rational Suite TestStudio, RationalRobot—YesRobot editor is color-coded. Reservedwords highlighted—blue, comments—green.
Rational Suite TestStudio, RationalRobot—YesNames up to 40 characters.Dashes / Numeric okay.
SilkTest—Good editor
SilkTest—YesAlphanumeric with no space.
B.1 Test Tool Vendors: Compuware, Empirix and Mercury
Mercury Rational Segue
Tool Customization
(continued )
Platform Support
WinRunner—variousWindowsAstra QuickTest—supportsvarious Windows.Astra LoadTest—NTXRunner—HP, IBM, SunLoadRunner—NT, Unix
WinRunner—YesAstra QuickTest—Yes
WinRunner—YesAstra QuickTest—Yes
WinRunner—Yes HTML, DHTML, JavaScript,VBScript, XML, Java applets,ActiveX, Visual Basic,PowerBuilder, Delphi,Terminal Emulator, Oracle,SAP, PeopleSoft, SiebelAstra QuickTest—HTML,DHTML, JavaScript, VBScript,XML, Java applets, ActiveX
Rational Suite TestStudio, RationalRobot—YesSupports UNIX and various Windows
Rational Suite TestStudio, RationalRobot—YesPresently not Netscape 6.0
Rational Suite TestStudio, RationalRobot—Yes
Rational Suite TestStudio, RationalRobot—YesHTML, Dynamic HTML, JavaScript,VBScript, ActiveX, Java apps, Javaapplets
Supports UNIX and variousWindows
SilkTest—Yes
SilkTest—Yes
SilkTest: Yes—HTML, DHTML,JavaScript, XML, Java appsand applets, MFC, VisualBasic, Oracle, PowerBuilder,Delphi, SAP
28206 03 pp. 217-282 r11kp.ps 7/26/01 3:10 PM Page 247
S
R
L
Appendix B Test Tool Evaluations248
Allows add-ins/extensions compatiblewith third-party controls
Does not involve additional cost foradd-ins and extensions
Test editor/debugger feature
Test scripting language flexible yetrobust; allows for modular scriptdevelopment
Scripting language not too complex
Scripting language allows for variabledeclaration and use and for parametersto be passed between functions
A test script compiler or an interpreterused
TestPartner—Yes
NA
QACenter—YesPerformance Edition—Yes
QACenter—YesPerformance Edition—Yes
TestPartner uses VBA for scriptinglanguage.Performance Edition uses C, acommon language.QARun has its own scriptinglanguage.
QACenter—YesPerformance Edition—YesLanguage also supports COMtesting
QARun and TestPartner use aninterpreter.Performance Edition—Scripts arecompiled for performance benefits.
eTest Suite—Yes
eTest Suite—Yes
eTest Suite—YesExtensive debugger
eTest Suite—YesVisual Scripts do notrequire a programminglanguage, but can beextended by using VB. Canuse script libraries.
VB—a common language
eTest Suite—Yes
eTest Suite—YesScripts are compiled forperformance benefits.
Table B.1 Test Tool Characteristics: Capture/Playback and Script Creation (continued)
Criterion/Feature Compuware Empirix/RSW
Test Language Features
28206 03 pp. 217-282 r11kp.ps 7/26/01 3:10 PM Page 248
s
R
L
249B.1 Test Tool Characteristics
WinRunner—Yes
WinRunner—Additional costsfor some environments
WinRunner—YesAstra QuickTest—Yes
WinRunner—Yes uses TSL(Test Script Language)developed for testing.Astra QuickTest usesVBScript.
WinRunner TSL is easy touse, English-like interpreted,procedural language.Astra QuickTest usesVBScript also interpretedlanguage.
WinRunner—YesAstra QuickTest—Yes
Mercury Interactive tools useinterpreted languages.
Rational Suite TestStudio, RationalRobot—YesSupported via followingutilities/functions: object mapping,object data test, definition, Java ProxyAPI, Custom Control API.
Rational Suite TestStudio, RationalRobot—No additional costs.
Rational Suite TestStudio, RationalRobot—YesColor-coded editor and debugger toinclude a compile option, debug runmode, go until cursor mode, step mode,breakpoints and a watch variableswindow.
Rational Suite TestStudio, RationalRobot—YesTwo scripting languages, both robustand extensible. Users can writefunctions, create libraries & callexternal API’s. Robot can also be utilizedusing Visual Basic and Java.
Rational Suite TestStudio, RationalRobot—SQABasic is a VB-likelanguage. VU is C-like language usedfor performance testing.
Rational Suite TestStudio, RationalRobot—Yes
Rational Suite TestStudio, RationalRobot—YesSQABasic scripts compiled and theninterpreted at runtime. VU scripts arecompiled to an executable format.
SilkTest—YesAccomplished via object mapping,dll calls, or by adding an extensionvia our Extension Kit.
SilkTest—Yes
SilkTest—Yes
SilkTest—Yes
SilkTest—Not complex.Object Oriented languagestructure with ability to usefunctions. Very much like C++,not as rigid as Java.
SilkTest—YesStrong feature of SilkTest.
SilkTest—YesCompiler used for performance,manipuate the actual objects, andexercise actual AUT code.
B.1 Test Tool Vendors: Compuware, Empirix and Mercury
Mercury Rational Segue
Test Language Features
(continued )
28206 03 pp. 217-282 r11kp.ps 7/26/01 3:10 PM Page 249
S
R
L
Appendix B Test Tool Evaluations250
Interactive test debugging: allowsviewing variable values, steps throughthe code, integrates test procedures, orjumps to other external procedures
Allows recording at the widget level(object recognition level)
Allows for interfacing and testing ofexternal .dll and .exe files
Published APIs: Language InterfaceCapabilities
Tool is not intrusive: source code ofapplication needs to be expanded byinserting additional statements ordlls for the application to becompatible with the tool
Allows for data-driven testing
Allows for automatic data generation
Allows for adding timers for timingtransaction start and end
QARun—LimitedTestPartner—VBA full debuggerPerformance Edition—Limited
NA
TestPartner—Yes Performance Edition—Yes
TestPartner—Yes
QACenter—YesPerformance Edition—Yes
QACenter—Yes Performance Edition—Yes
QACenter—YesPerformance Edition—Yes
Performance Edition—YesQACenter—YesQALoad—Yes
eTest Suite—YesVBA debugger combinedwith Visual Scriptdebugger.
eTest Suite—YesObjects are recognized atthe DOM level.
eTest Suite—YesInterface to call anyexternal program or .dll
eTest Suite—YesCapability is called e-PI
eTest Suite—Yes
eTest Suite—YesWizard driven
eTest Suite—No
eTest Suite—Yes
Table B.1 Test Tool Characteristics: Capture/Playback and Script (continued)
Criterion/Feature Compuware Empirix/RSW
Test Language Features
28206 03 pp. 217-282 r11kp.ps 7/26/01 3:10 PM Page 250
s
R
L
251B.1 Test Tool Characteristics
Mercury Interactive toolsoffer full debuggingcapabilities.
WinRunner—YesAstra QuickTest—Yes
Mercury Interactive tools—Yes
Mercury Interactive tools—YesPublished APIs.
Mercury Interactive tools donot require any modificationof the application under test.
Mercury Interactive tools—Yes
Mercury Interactive tools—Do not support datageneration but allow forautomatic retrieval andimport of real test data.
WinRunner—YesAstra QuickTest—Yes
Rational Suite TestStudio, RationalRobot—YesDebugging to include a compile option,debug run mode, go until cursor mode,step mode, breakpoints and a watchvariables window.
Rational Suite TestStudio, RationalRobot—YesObject recognition is available andcustomizable.
Rational Suite TestStudio, RationalRobot—YesExternal dll and exe files can be calledfrom within test scripts. Also a GUIscript can detect if a dll or exe isloaded. Also has file existence and filecompare verification.
Rational Suite TestStudio, RationalRobot, Rational TestManager—Yes
Rational Suite TestStudio, RationalRobot—YesSpecific environments like Delphirequire code insertion (running theEnabler utility) to achieve objectrecognition.
Rational Suite TestStudio, RationalRobot—YesBoth GUI and VU scripts can beaugmented to handle data variation.
Rational Suite TestStudio, RationalRobot—YesDatapools can be created for use withGUI or VU scripts.
Rational Suite TestStudio, RationalRobot—Yes
SilkTest—Yes
SilkTest—YesStrong feature of SilkTest.
SilkTest—Yes
SilkTest—YesExtension Kit allows directcommunication with app.
SilkTest—YesTool is not invasive.
SilkTest—YesObject oriented featureimplemented to supportdata driven testing.
SilkTest—Yes
SilkTest—Yes
Table B.1 Test Tool Vendors: Capture/Playbackand Script Creation (continued)
Mercury Rational Segue
Test Language Features
(continued )
28206 03 pp. 217-282 r11kp.ps 7/26/01 3:10 PM Page 251
S
R
L
Appendix B Test Tool Evaluations252
Allows for adding comments duringrecording
Allows for automatic or specifiedsynchronization between clientand server
Allows for verification of objectpropertiesAllows for object data extraction andverification
Allows for database verification
Allows for text (alphanumeric)verification
Allows for wrappers (shells) wherebymultiple procedures can be linked andcalled from one procedure
Allows for automatic data retrieval fromany data source—RDBMS, legacysystem, spreadsheet—for data-driventesting
Allows for use of common spreadsheetfor data-driven testing
QACenter—YesPerformance Edition—Yes
QACenter—YesPerformance Edition—Yes
TestPartner—Yes
Performance Edition—Yes
QACenter—Yes
QACenter—Yes
TestPartner—Yes
QACenter—YesPerformance Edition—Yes
eTest Suite—Yes
eTest Suite—Yes
eTest Suite—YesAll properties of DOMobjects
eTest Suite—YesVia ODBC functions in VB.
eTest Suite—Yes
eTest Suite—Yes
eTest Suite—YesVia ODBC in scripts ordump to .csv file formatto populate a databank
eTest Suite—YesSupports .csv format
Table B.1 Test Tool Characteristics: Capture/Playback and Script Creation (continued)
Criterion/Feature Compuware Empirix/RSW
Test Language Features
28206 03 pp. 217-282 r11kp.ps 7/26/01 3:10 PM Page 252
s
R
L
253B.1 Test Tool Characteristics
WinRunner—YesAstra QuickTest—Yes
WinRunner—YesAstra QuickTest—Yes
WinRunner—YesAstra QuickTest—Yes
WinRunner—YesVia ODBC and nativedatabase connectivity.Astra QuickTest—Yes viaVBScript functions.
WinRunner—YesAstra QuickTest—Yes
WinRunner—YesAstra QuickTest—Yes
WinRunner—Yes
WinRunner—YesAstra QuickTest—YesBoth support Excel format.
Rational Suite TestStudio, RationalRobot—YesComments may be added duringrecording.
Rational Suite TestStudio, RationalRobot—YesWait States for GUI playback For VU scripts, synchronization isautomatic.
Rational Suite TestStudio, RationalRobot—Yes
Rational Suite TestStudio, RationalRobot—YesVia ODBC functions in SQABasic
Rational Suite TestStudio, RationalRobot—Yes
Rational Suite TestStudio, RationalRobot, Rational TestManager—YesUses either shell scripts (Robot),LoadTest v2000 schedules,TestManager v2001 Suites.
Rational Suite TestStudio, RationalRobot—YesVia ODBC in scripts or dump to .csvfile format. SQABasic can also read.csv files directly. Also can use .csvfiles to create datapool source data.
Rational Suite TestStudio, RationalRobot—YesUses .csv format
SilkTest—Yes
SilkTest—YesCan automatically connectto a particular agent, oruse a script to connect.Can change from oneparticular agent to another.
SilkTest—Yes
SilkTest—YesVia database Tester 4Test functions.
SilkTest—Yes
SilkTest—Yes
SilkTest—YesDatabase Tester functionsused to supply an ODBChookup, or can scriptparsing for any files.
SilkTest—Yes.Scripts can access andmanipulate content withinall common spreadsheets.
Table B.1 Test Tool Vendors: Capture/Playbackand Script Creation (continued)
Mercury Rational Segue
Test Language Features
(continued )
28206 03 pp. 217-282 r11kp.ps 7/26/01 3:10 PM Page 253
S
R
L
Appendix B Test Tool Evaluations254
Ease of maintaining script whenapplication changes
QACenter—YesPerformance Edition—Yes
eTest Suite—YesAutomatic update ofscripts, no recodingrequired.
Table B.1 Test Tool Characteristics: Capture/Playback and Script Creation (continued)
Criterion/Feature Compuware Empirix/RSW
Test Language Features
Test Tool Database
Tool database scalability when testrepository grows
Network-based test repository,necessary when multiple access torepository required
Supports ANSI SQL execution
NA
QACenter—Yes
TestPartner—YesPerformance Edition—Yes
eTest Suite—YesSupports MS Access,MS SQL Server, andOracle
eTest Suite—YesShared workspaces
eTest Suite—YesSQL execution can bedone through VB.
28206 03 pp. 217-282 r11kp.ps 7/26/01 3:10 PM Page 254
s
R
L
255B.1 Test Tool Characteristics
WinRunner—YesAstra QuickTest—YesVia GUI map or objectrepository
Rational Suite TestStudio, RationalTeamTest, Rational Robot—YesRobot editor and debug tools enableeasy maintenance.
SilkTest—Yes.Include file can be updatedwhenever objects change in nameor nature.
B.1 Test Tool Vendors: Compuware, Empirix and Mercury
Mercury Rational Segue
Test Language Features
Test Tool Database
WinRunner & Astra QuickTestdo not use a database.TestDirector support for allmajor database vendors.
TestDirector—Yes
WinRunner—Yes
Rational Suite TestStudio—YesFor v2001—database repositoryreplaced with an XML—baseddatastore. Within the datastore, Accessdatabase used for queries and reports.
Rational Suite TestStudio—Yes
Rational Suite TestStudio, RationalRobot—YesVia SQL commands in SQABasic.
SilkTest—Yes.Doesn't have limitations orslowdowns based on the sheervolume of test code.
SilkTest—Yes.Files can be placed anywhere,provided there is networkdirectory access to test code.
SilkTest—Yes
28206 03 pp. 217-282 r11kp.ps 7/26/01 3:10 PM Page 255
S
R
L
Appendix B Test Tool Evaluations256
Ability to kick off scripts at a specifiedtime; scripts can be scheduled and rununattended
Has a complete graphical palette todesign and to schedule test runs basedon dates and time, or logic betweentests
Stateless playback: Application doesnot have to be in a preset state beforescript playback
Centralized execution and control
Standalone test execution automation
QACenter—YesPerformance Edition—YesQADirector allows scheduling forspecific dates/times and /or regulartime periods.
PE/ QADirector do not havegraphical palette.
Inclusion of File—AID/CS allows PEto test application DBMS states.QADirector allows for setup,execution and restore
QADirector supports access,execution, review of results from a central location.
QACenter—YesPerformance Edition—Yes
eTest Suite—YesSchedules Visual Scripts orsets of scripts to rununattended.
eTest Suite—Yes
eTest Suite—Yes
eTest Suite—Yese-Test console providescentralized monitoring ofall application activity.
eTest Suite—Yes
Table B.2 Test Execution Capability
Criterion/Feature Compuware Empirix/RSW
Test Control Features
Distributed Test Execution
Distributed test control,synchronization, execution
Execution of manual orautomated tests via a browserfor anywhere/anytime test execution,with Web-based test management
Supports synchronization of multitestthreads
QACenter—YesQADirector supports distributedexecution where app is tested as itruns in productionPerformance Edition—Yes
Manual test execution is web-based. Web interface is underwayfor other features.
For QADirector tests can be parallelor singularPerformance Edition—Yes
eTest Suite—Yes
eTest Suite—No
eTest Suite—Yes
28206 03 pp. 217-282 r11kp.ps 7/26/01 3:10 PM Page 256
s
R
L
257B.2 Test Execution Capability
TestDirector—allowsscheduling.LoadRunner has built-inschedulers for loadscenarios.
TestDirector—Yes
WinRunner—Yes
TestDirector—Yes
WinRunner—YesAstra QuickTest—Yes
SilkTest—Yes No separate tool required
SilkTest—No
SilkTest—YesAny script can be run at any time
SilkTest—Yes
SilkTest—Yes
Table B.2 Test Execution Capability
Mercury Rational Segue
Test Control Features
Distributed Test Execution
TestDirector—allows forlocal or remote test executioncontrol via networks.
TestDirector—Yes. All partsof TestDirector can beaccessed via a browser.
WinRunner—Yes
Rational Suite TestStudio, RationalTestManager—Yes
Rational Suite TestStudio—Notcurrently
Rational Suite TestStudio—Yes
SilkTest—Yes
SilkTest—No No web front end at present. Onecould be created, since tool canrun from the command line.
SilkTest—Yes
(continued )
Rational Suite TestStudio, RationalRobot—YesRobot script checks the time, LoadTestv2000 & TestManager v2001 scheduledto run at a specific time.
Rational Suite TestStudio, RationalRobot—YesSuite creation in TestManager v2001 isall graphical.
Rational Suite TestStudio, RationalRobot—YesRobot GUI scripts can locate anywindow, by using window set context tolocate a window to begin scriptplayback. VU scripts are completelystateless.
Rational Suite TestStudio, RationalTestManager—Yes
Rational Suite TestStudio, RationalRobot, Rational TestManager—Yes
28206 03 pp. 217-282 r11kp.ps 7/26/01 3:10 PM Page 257
Table B.2 Test Execution Capability (continued)
Criterion/Feature Compuware Empirix/RSW
Distributed Test Execution
S
R
L
Appendix B Test Tool Evaluations258
Headless back-end server testing
Allows for test execution across LANsand WANs
Test results analysis: automaticcreation of test results log
Multiplatform testing support
Performance Edition—YesQADirector tests can be routed toany server or workstation attachedto the network.
Performance Edition—YesQADirector supports a centralrepository that can be accessed viathe network.
QACenter—YesQADirector supports both pass/failas well as drill down.Performance Edition—Yes
Performance Edition—YesQADirector supports Unix, Windowsand OS/390
eTest Suite—Yes
eTest Suite—Yes
eTest Suite—Yes
eTest Suite—YesSupports Unix and variousWindows.
Test Suite Recovery Logic
Unexpected error recovery: Can handleerror recovery and unexpected activewindows, logs the discrepancy, andcontinues playback (automaticrecovery from errors)
Can restore the application to its initialstate or provides ways for theapplication to be restored
QACenter—YesPerformance Edition—Yes
QACenter—YesPerformance Edition—YesQADirector—Yes
eTest Suite—Yes
eTest Suite—Yes
28206 03 pp. 217-282 r11kp.ps 7/26/01 3:10 PM Page 258
s
R
L
259B.2 Test Execution Capability
WinRunner—Yes
TestDirector—Yes
WinRunner—YesAstra QuickTest—YesVisually displayed for ease ofscanning.
Mercury Interactive toolsprovide support for over 28different platforms.
Rational Suite TestStudio, RationalTestManager—Yes
Rational Suite TestStudio, RationalTestManager—Yes
Rational Suite TestStudio, RationalTestManager—YesTest results are viewed fromTestManager. Defects andenhancements may be reported directlyfrom the Test Log via ClearQuestintegration.
Rational Suite TestStudio, RationalTestManager—YesSupports Unix and various Windows
SilkTest—Yes
SilkTest—YesNetwork connectivity needed tothe particular machine with theSilkTest agent.
SilkTest—Yes
SilkTest—Yes
Table B.2 Test Execution Capability
Mercury Rational Segue
Distributed Test Execution
Test Suite Recovery Logic
WinRunner—YesAstra QuickTest—YesVia exception handler.
WinRunner—YesAstra QuickTest—Yes
Rational Suite TestStudio, RationalRobot—YesError recovery for script failures andverification point failure. Alsounexpected active window handler canbe set. When fatal error is encountered,trap feature can be used to obtain astack trace, module lists, restartWindows or run a procedure.
Rational Suite TestStudio, RationalRobot—Yes
SilkTest—Yes
SilkTest—Yes
(continued )
28206 03 pp. 217-282 r11kp.ps 7/26/01 3:10 PM Page 259
Table B.2 Test Execution Capability (continued)
Criterion/Feature Compuware Empirix/RSW
Test Management
S
R
L
Appendix B Test Tool Evaluations260
Supports test execution management
Supports industry standards in testingprocess (SEI/CMM, ATLM, ISO)
Application requirements managementsupport integrated with the testmanagement tool
Requirements management capabilitysupports the trace of requirements totest plans to provide requirementcoverage metrics
Test plans can be importedautomatically into test managementrepository from standard text files
Tool can be customized toorganization’s test process
Supports planning, managing,and analyzing testing efforts;can reference test plans,matrices, product specifications,in order to create traceability
Supports manual testing
Supports the migration from manual toautomated scripts
QACenter—YesPerformance Edition—YesQADirector—Yes
QACenter—YesPerformance Edition—YesQADirector—Yes
QADirector—Integrates withReconcile/DOORS for RMPerformance Edition—NA
QADirector Integrated withReconcile—Yes
QADirector—Yes Performance Edition—NA
QADirector supports various testingmethodologies.Performance Edition—NA
QADirector—plan and report testactivity.Performance Edition—NA
QADirector—provides manual testcreation.Performance Edition—NA
QADirector—can replace manualtests with automated testing scriptsPerformance Edition—NA
eTest Suite—YesThrough e-Manager
eTest Suite—Yes
eTest Suite—No
eTest Suite—No
eTest Suite—No
eTest Suite—Yes
eTest Suite—Yese-Manager and d-Trackerprovide the ability to trackand manage the testingprocess
eTest Suite—Yes
eTest Suite—No
28206 03 pp. 217-282 r11kp.ps 7/26/01 3:10 PM Page 260
s
R
L
261B.2 Test Execution Capability
TestDirector—Yes
TestDirector—Yes
TestDirector—has built-inrequirements management.Also integrates with 3rd partyRM tools like TBI Caliber,DOORS and RequisitePro
TestDirector—Yes
TestDirector—Yes
TestDirector—Yes
TestDirector—Yes
TestDirector—Yes
TestDirector—YesGenerates commented scriptfor ease of recordingbusiness processes.
Rational Suite TestStudio, RationalTestManager—Yes
Rational Suite TestStudio, RationalTeamTest, Rational TestManager,Rational Unified Process—Yes
Rational Suite TestStudio, RationalRobot—YesTestManager can utilize RequisitePro orRational Rose. Other RM tools used viathe creation of Test Input Adaptersusing TestManager’s open API.
Rational Suite TestStudio, RationalRobot—YesRequirements can be traced to TestCases within a Test Plan inTestManager.
Rational Suite TestStudio—YesVia RequisitePro integration.
Rational Suite TestStudio, RationalTeamTest, Rational TestManager,Rational Unified Process—Yes
Rational Suite TestStudio, RationalTestManager—Yes
Rational Suite TestStudio, RationalTestManager—Yes
Rational Suite TestStudio—No
SilkTest—Yes
SilkTest—Yes
SilkTest—No
SilkTest—No
SilkTest—Yes
SilkTest—YesBuilt-in test plan functionality canbe customized to process.
SilkTest—YesVia use of extensive results fileswith collapsability features andthrough using test plans.
SilkTest—YesSpecify manual tests via a testplan or usingAgent.Display() methodwithin a script.
SilkTest—YesCan replace manual tests withscripts to perform steps for you.
Table B.2 Test Execution Capability
Mercury Rational Segue
Test Management
(continued )
28206 03 pp. 217-282 r11kp.ps 7/26/01 3:10 PM Page 261
Table B.2 Test Execution Capability (continued)
Criterion/Feature Compuware Empirix/RSW
Test Management
S
R
L
Appendix B Test Tool Evaluations262
Can track the traceability of tests totest requirements
Has built-in test requirements modules
Tool can check for duplicate defectsbefore logging newly found defects
Allows for measuring test progress
Allows for various reporting activities
Allows for tracking of manual andautomated test cases
QADirector—Integrates withReconcile/DOORS for RMPerformance Edition—NA
Reconcile provides full RMPerformance Edition—NA
QADirector—YesPerformance Edition—NA
QADirector—YesPerformance Edition—NA
QADirector—YesAlso supports 3rd party reportgenerators & reports to the webPerformance Edition—NA
QADirector—YesAlso provides file/data mgmt Performance Edition—NA
eTest Suite—No
eTest Suite—No
eTest Suite—Yes
eTest Suite—Yes
eTest Suite—Yes
eTest Suite—No
28206 03 pp. 217-282 r11kp.ps 7/26/01 3:10 PM Page 262
s
R
L
263B.2 Test Execution Capability
TestDirector—Yes
TestDirector—YesAlso integrates with 3rd partyRM tools
TestDirector—Yes
TestDirector—Yes
TestDirector—Yes
TestDirector—Yes
Rational Suite TestStudio, RationalTestManager—Yes viaTestManager coverage reports.
Rational Suite TestStudio, RationalTestManager—YesVia RequisitePro
Rational Suite TestStudio, RationalClearQuest—Yes
Rational Suite TestStudio, RationalTestManager—Yes
Rational Suite TestStudio, RationalTeamTest, Rational TestManager—Yes
Rational Suite TestStudio, RationalTestManager—YesTestManager is 100% test case centricto allow for manual & automatedimplementation.
SilkTest—Yes
SilkTest—No
SilkTest —NA Does not support defect tracking.
SilkTest—Yes
SilkTest—Yes
SilkTest—Yes
Table B.2 Test Execution Capability
Mercury Rational Segue
Test Management
28206 03 pp. 217-282 r11kp.ps 7/26/01 3:10 PM Page 263
S
R
L
Appendix B Test Tool Evaluations264
Interface to softwarearchitecture/modeling tool
Integration to unit testing tools
Interface to test management tool
Interface to requirements managementtool
Interface to defect tracking tool
Interface to configuration managementtool
Does not have direct integration
QADirector—YesWith TestPartner, TrueTime,ActiveAnalysis, TrueCoverage,BoundsChecker.
QADirector—YesWith Reconcile & TrackRecord
QADirector—YesWith Reconcile & DOORS
QADirector—YesWith TrackRecord
QADirector—YesVersion control tests within CM toolof choice
Does not have directintegration
eTest Suite—Supports unittesting of MiddlewareObjects.
eTest Suite—Yes
eTest Suite—No
eTest Suite—Yes
eTest Suite—No
Table B.3 Tool Integration Capability
Criterion/Feature Compuware Empirix/RSW
28206 03 pp. 217-282 r11kp.ps 7/26/01 3:10 PM Page 264
s
R
L
265B.3 Tool Integration Capability
Mercury Interactive tools—Yes
Mercury Interactive tools—YesSuch as McCabe tools.
Mercury Interactive tools—YesWinRunner, LoadRunner,Astra QuickTest and AstraLoadTest integrate withTestDirector.
TestDirector—YesIntegrates with 3rd party RMtools
Mercury Interactive tools—YesBuilt-in defect managementtool. Also integrate withClearQuest and PVCS Tracker
TestDirector—YesBuilt-in interface to MSSourcesafe, PVCS VersionManager and ClearCase
Rational Suite Enterprise, Rational SuiteDevelopmentStudio, Rational SuiteTestStudio, Rational TestManager—YesRational Rose model elements can beused as test inputs using TestManagerv2001. Can generate scripts from UMLmodels in Rose.
Rational Suite Enterprise, Rational SuiteDevelopmentStudio—YesRational Quality Architect v2001 (EJB &COM)
Rational Suite TestStudio, RationalTestManager—Yes
Rational Suite TestStudio, RationalTestManager—YesTestManager fully integrates withRequisitePro and Rational Rose. OtherRM tools integrated via the Test InputAdapter API.
Rational Suite TestStudio, RationalTestManager—YesTest Log is integrated with ClearQuest
Rational Suite TestStudio—YesClearCase LT shipping with testingproducts v2001 and beyond
SilkTest—No
SilkTest—Unit testing supportedby SilkTest.
SilkTest—Yes
SilkTest—No
SilkTest—Yes
SilkTest—Yes
Table B.2 Test Execution Capability
Mercury Rational Segue
28206 03 pp. 217-282 r11kp.ps 7/26/01 3:10 PM Page 265
S
R
L
Appendix B Test Tool Evaluations266
Error filtering and review features
Metric collection and metric analysisvisualization
Performance Edition—Yes
Performance Edition—YesMetric collection & reporting
eTest Suite—Yes
eTest Suite—Yes
Table B.4 Test Reporting Capability
Criterion/Feature Compuware Empirix/RSW
Summary-Level Reporting
Test Report Presentation
Predefined reports can be modified andnew reports created
Most widely used views can be createdand made public for others to view attheir leisure with automatically updatedinformation
Generate graphs and reports from testresults
Graphs and reports fully customizable,supporting more than just limitedmodification, such as allowing choiceof axis labels
Reports are exportable to HTML,.csv, or .txt files
Performance Edition—YesReports modified & created
Performance Edition—YesFields & views customizable
Performance Edition—YesLibrary of graphs/reports
Performance Edition—Yes
Performance Edition Outputs to HTML format
eTest Suite—Yes
eTest Suite—YesHTML output to web.Complete flexibility withWeb Reporter.
eTest Suite—Yes
eTest Suite—Yes
eTest Suite—Yes
28206 03 pp. 217-282 r11kp.ps 7/26/01 3:10 PM Page 266
s
R
L
267B.4 Tool Reporting Capability
TestDirector—Yes
TestDirector—Yes
Rational TestManager—YesUsers create Test Log filters and sortthe log items
Rational Suite TestStudio, RationalTestManager—Yes
SilkTest—Yes
SilkTest—Yes
Table B.2 Test Execution Capability
Mercury Rational Segue
Summary-Level Reporting
Test Report Presentation
TestDirector—YesDocument Generatorprovides full customization ofreports.
TestDirector—Yes
TestDirector—Yes
TestDirector—Yes
TestDirector—YesLoadRunner—YesAutomatic HTML reportgeneration setting
Rational TestManager—Yes
Rational TestManager—Yes
Rational TestManager—Yes
Rational TestManager—Yes
Rational TestManager—Yes
SilkTest—Yes
SilkTest—YesAbility to view work of others andcan control write-abilty throughuse of permissions.
SilkTest—Yes
SilkTest—No
SilkTest—Yes
28206 03 pp. 217-282 r11kp.ps 7/26/01 3:10 PM Page 267
S
R
L
Appendix B Test Tool Evaluations268
All users can be queued to execute aspecified action at the same time
Automatic generation of summary loadtesting analysis reports
Ability to change recording of differentprotocols in the middle of load-recording session
Actions in a script can be iterated anyspecified number of times withoutprogramming or rerecording of thescript
Different modem connection speedsand browser types can be applied to ascript without any rerecording
Load runs and groups of users withinload runs can be scheduled to executeat different times
Automatic load scenario generationbased on load testing goals:hits/second, number of concurrentusers before specified performancedegradation, and so on
Cookies and sessions IDs automaticallycorrelated during recording andplayback for dynamically changing Webenvironments
Performance Edition—Yes
Performance Edition—Yes
Performance Edition—YesCan record multiple middlewareand protocols during samerecording session.
Performance Edition—Yes
Performance Edition—Yes
Performance Edition—Yes
Performance Edition—Yes
Performance Edition—Yes
eTest Suite—Yes
eTest Suite—Yes
eTest Suite—YesCan change protocols aslong as within the webtransaction and can berecorded.
eTest Suite—YesWith or without variabledata.
eTest Suite—Yes
eTest Suite—Yes
eTest Suite—No
eTest Suite—YesNever requiresprogramming, entirelyautomatic
Table B.5 Performance Testing and Analysis Capability
Criterion/Feature Compuware Empirix/RSW
Load and Stress Test Features
28206 03 pp. 217-282 r11kp.ps 7/26/01 3:10 PM Page 268
s
R
L
269B.5 Performance Testing and Analysis Capability
LoadRunner—YesAstra LoadTest—Yes
LoadRunner—YesAstra LoadTest—Yes
LoadRunner—Yes for someprotocols.
LoadRunner—YesAstra LoadTest—YesA simple runtime setting.
LoadRunner—YesAstra LoadTest—YesA simple runtime setting.
LoadRunner—YesAstra LoadTest—Yes
LoadRunner—Yes
LoadRunner—YesAstra LoadTest—YesDoes not requireprogramming, entirelyautomatic.
Rational Suite TestStudio, RationalTestManager—YesVia sync points
Rational Suite TestStudio, RationalTestManager—Yes
Rational Suite TestStudio, RationalRobot—YesAll protocols can be captured during asingle recording session, so “changing”protocols is not necessary. Afterrecording, the multiple protocols can beput into a single script, or, filtering couldbe used to put a single, or acombination of multiple protocols into asingle script.
Rational Suite TestStudio, RationalRobot, Rational TestManager—YesMost other tools require that all useractions be put in a single script———Rational Robot allows a user session tobe split into multiple scripts, each ofwhich can be iterated any number oftimes via the graphical schedulingmechanism in TestManager.
Rational Suite TestStudio, RationalTestManager—Yes
Rational Suite TestStudio, RationalTestManager—Yes
SiteLoad, Rational Suite TestStudio—Yes (in TestManager via Transactors)
Rational Suite TestStudio, RationalTestManager, Rational Robot—Yes
SilkPerformer—Yes
SilkPerformer—Yes
SilkPerformer—Yes
SilkPerformer—Yes
SilkPerformer—Yes
SilkPerformer—Yes
SilkPerformer—Yes
SilkPerformer—Yes
B.1 Test Tool Vendors: Compuware, Empirix and Mercury
Mercury Rational Segue
Load and Stress Test Features
(continued )
28206 03 pp. 217-282 r11kp.ps 7/26/01 3:10 PM Page 269
S
R
L
Appendix B Test Tool Evaluations270
Allows for variable access methods andability to mix access methods in asingle scenario: modem simulation orvarious line speed simulation
Ability to have data-driven scripts thatcan use a stored pool of data
Allows for throttle control for dynamicload generation
Allows for automatic service-levelviolation (boundary value) checks
Allows for variable recording levels(network, Web, API, and so on)
Allows for transaction breakdown/drill-down capabilities for integrityverification at the per client, persession, and per instance level forvirtual users
Allows for Web applicationserver integration: ColdFusion,NetDynamics, Dynamo, and so on
Supports workload, resource, and/orperformance modeling
Can run tests on various hardware andsoftware configurations
Performance Edition—Yes
Performance Edition—Yes
Performance Edition—Yes
Performance Edition—Yes
Performance Edition—Yes
Performance Edition—YesOffers integrated load testing andserver monitoring.
Performance Edition—Yes
Performance Edition—Yes
Performance Edition—Yes
eTest Suite—Yes
eTest Suite—Yeswith DataBank Wizard
eTest Suite—Yes
eTest Suite—Yes
eTest Suite—No
eTest Suite—Yes
eTest Suite—Yes
eTest Suite—Yes
eTest Suite—Yes
Table B.5 Performance Testing and Analysis Capability (continued)
Criterion/Feature Compuware Empirix/RSW
Load and Stress Test Features
28206 03 pp. 217-282 r11kp.ps 7/26/01 3:10 PM Page 270
s
R
L
271B.5 Performance Testing and Analysis Capability
LoadRunner—YesAstra LoadTest—Yes
LoadRunner—YesAstra LoadTest—Yes
LoadRunner—Yes
LoadRunner—YesAstra LoadTest—Yes
LoadRunner—Yes
LoadRunner—YesAstra LoadTest—Yes
LoadRunner—YesAstra LoadTest—YesIncluding ability to record andreplay against theapplications, and monitor theapp server performanceduring load test.
LoadRunner—YesAstra LoadTest—Yes
LoadRunner—YesAstra LoadTest—Yes
Rational TestManager Suite— YesAny combination of tests is possible.
Rational Suite TestStudio, RationalTestManager, Rational Robot—YesVia Datapools.
Rational Suite TestStudio, RationalTestManager—YesWith Transactors (graphical point andclick creation) or Shared Variables(hand coded)
Rational Suite TestStudio, RationalTestManager—Yes
Rational Suite TestStudio, RationalRobot—Yes
Rational Suite TestStudio, RationalTestManager—Yes
Rational Suite TestStudio, RationalRobot, Rational TestManager—Yes
Rational Suite TestStudio, RationalTestManager—Yes
Rational Suite TestStudio, RationalTestManager—Yes
SilkPerformer—Yes
SilkPerformer—Yes
SilkPerformer—Yes
SilkPerformer—Scriptscan be deployed to supportthese checks.
SilkPerformer—YesAllows variable levels andmixing of protocols for ascript, user or transaction.
SilkPerformer—Yes
SilkPerformer—Yes
SilkPerformer—Yes
SilkPerformer—Yes
Table B.1 Test Tool Vendors: Capture/Playbackand Script Creation (continued)
Mercury Rational Segue
Load and Stress Test Features
(continued )
28206 03 pp. 217-282 r11kp.ps 7/26/01 3:10 PM Page 271
S
R
L
Appendix B Test Tool Evaluations272
Support headless virtual user testingfeature
Requires low overhead for virtual userfeature (Web, database, other?)
Scales to 500–1,000 virtual users
Simulated IP addresses for virtual users
Thread-based virtual user simulation
Process-based virtual user simulation
Centralized load test controller
Allows for reusing scripts fromfunctional test suite
Support for WAP protocol testingagainst WAP Gateway or Web server
Compatible with SSL recording
Performance Edition—Yes
Performance Edition—YesExceptionally low overhead
Performance Edition—Yes
Performance Edition—Yes
Performance Edition—Yes
Performance Edition—Yes
Performance Edition—Yes
Performance Edition—YesFunctional/load test scripts can becreated at same time. Two usedifferent languages.
Performance Edition—Yes
Performance Edition—Yes
eTest Suite—Yes
eTest Suite—YesSmall footprint per Virtual User
eTest Suite—Yes
eTest Suite—Yes
eTest Suite—YesDefault and most efficientmode of testing
eTest Suite—YesOptional
eTest Suite—Yes
eTest Suite—YesVisual Scripts createdduring functional testingreused for load testing andmonitoring
eTest SuiteYes for the web server
eTest Suite—Yes
Table B.5 Performance Testing and Analysis Capability (continued)
Criterion/Feature Compuware Empirix/RSW
Load and Stress Test Features
28206 03 pp. 217-282 r11kp.ps 7/26/01 3:10 PM Page 272
s
R
L
273B.5 Performance Testing and Analysis Capability
LoadRunner—YesAstra LoadTest—Yes
LoadRunner—YesVery high scalability ratingsusing TurboLoad technology.
LoadRunner—YesLargest load recorded to dategenerated more than 1.35Mconcurrent connections.Astra LoadTest—Yes
LoadRunner—YesAstra LoadTest—YesBoth use IPSpoofer feature.
LoadRunner—YesAstra LoadTest—Yes
LoadRunner—YesIncluded since some appsrequire it for accuracy.
LoadRunner—YesAstra LoadTest—Yes
Astra QuickTest—YesWinRunner scripts used asGUI Vusers in LoadRunner.
LoadRunner—Yes
LoadRunner—YesAstra QuickTest—Yes
Rational Suite TestStudio, RationalTestManager—Yes
Rational Suite TestStudio, RationalTestManager—Yes
Rational Suite TestStudio, RationalTestManager—Yes
Rational Suite TestStudio, RationalTestManager—Yes
Rational Suite TestStudio, RationalTestManager—Yes
Rational Suite TestStudio, RationalTestManager—YesEach virtual user has it own processspace and unimpeded access to allresources within a process. Multi-threaded model supported whenneeded.
Rational Suite TestStudio, RationalTestManager—Yes
Rational Suite TestStudio, RationalTestManager—Yes
Rational Suite TestStudio, RationalRobot—Yes
Rational Suite TestStudio, RationalRobot—Yes
SilkPerformer—Yes
SilkPerformer—Yes
SilkPerformer—Yes
SilkPerformer—Yes
SilkPerformer—Yes
SilkPerformer—YesCan set any number of virtualusers as a universal defaultparameter or set number withina script.
SilkPerformer—Yes
SilkPerformer—Yes
SilkPerformer—Yes
SilkPerformer—Yes
Table B.1 Test Tool Vendors: Capture/Playbackand Script Creation (continued)
Mercury Rational Segue
Load and Stress Test Features
(continued )
28206 03 pp. 217-282 r11kp.ps 7/26/01 3:10 PM Page 273
Table B.5 Performance Testing and Analysis Capability (continued)
Criterion/Feature Compuware Empirix/RSW
Load and Stress Test Features
S
R
L
Appendix B Test Tool Evaluations274
Compatible with one or more of therelevant technologies: streamingmedia, COM, EJB, RMI, CORBA, Siebel,Oracle, SAP
Compatible with one or more of therelevant technologies: Linux, UNIX, NT,XWindows, Windows CE, Win3.1,Win95, Win98, Win2000, WinME
Performance Edition—Yes
Performance Edition—Yes
eTest Suite—Yes
eTest Suite—Yes
Performance Monitor Test Features
Monitors various tiers: Web server,database server, and app serverseparately
Supports monitoring for one or more ofColdFusion, Broadvision, BEAWebLogic, Silverstream, ATG Dynamo,Apache, IBM Websphere, OracleRDBMS, MS SQL Server, Real MediaServer, IIS, Netscape Web Server
Supports monitoring for one or more ofthe relevant technologies: Linux, NT,UNIX, XWindows, Windows CE, Win3.1,Win95/98, Win2000
Monitors network segments
Supports resource monitoring
Synchronization ability in orderto determine locking, deadlockconditions, and concurrencycontrol problems
Ability to correlate any metrics from allmonitors to identify performancebottlenecks
Performance Edition—Yes
Performance Edition—Yes
Performance Edition—Yes
Performance Edition—YesWith EcoSystems and ApplicationExpert
Performance Edition—Yes
Performance Edition—Yes
Performance Edition—Yes
eTest Suite—Yes
eTest Suite—Yes
eTest Suite—Yes
eTest Suite—Yes
eTest Suite—Yes
eTest Suite—Yes
eTest Suite—Yes
28206 03 pp. 217-282 r11kp.ps 7/26/01 3:10 PM Page 274
Table B.1 Test Tool Vendors: Capture/Playbackand Script Creation (continued)
Mercury Rational Segue
Load and Stress Test Features
s
R
L
275B.5 Performance Testing and Analysis Capability
LoadRunner—Yes
LoadRunner—YesAstra QuickTest—Yes
Rational Suite TestStudio, RationalRobot—Yes
Rational Suite TestStudio, RationalRobot, Rational TestManager—Yes
SilkPerformer—Yes
SilkPerformer—Yes
(continued )
Performance Monitor Test Features
LoadRunner—YesAstra QuickTest—Yes
LoadRunner—YesAstra QuickTest—Yes
LoadRunner—YesAstra QuickTest—Yes
LoadRunner—YesAstra QuickTest—Yes
LoadRunner—YesAstra QuickTest—Yes
LoadRunner—YesAstra QuickTest—Yes
LoadRunner—YesAstra QuickTest—Yes
Rational Suite TestStudio, RationalRobot, Rational TestManager—Yes
Rational Suite TestStudio, RationalTestManager—Yes
Rational Suite TestStudio, RationalTestManager—Yes
Rational Suite TestStudio, RationalTestManager—Yes
Rational Suite TestStudio, RationalTestManager—Yes
Rational Suite TestStudio, RationalTestManager—Yes
Rational Suite TestStudio, RationalTestManager—Yes
SilkPerformer—Yes
SilkVision—YesSilkVision component providesmonitoring capability.
SilkVision—YesSilkVision component providesmonitoring capability.
SilkVision—YesSilkVision component providesmonitoring capability.
SilkPerformer—Yes
SilkPerformer—Yes
SilkPerformer—Yes
28206 03 pp. 217-282 r11kp.ps 7/26/01 3:10 PM Page 275
S
R
L
Appendix B Test Tool Evaluations276
Ability to detect when events havecompleted in a reliable fashion
Ability to provide client-to-serverresponse times
Ability to provide graphical results andexport them to common formats
Ability to provide performancemeasurements of data loading
Performance Edition—Yes
Performance Edition—Yes
Performance Edition—Yes
Performance Edition—Yes
eTest Suite—Yes
eTest Suite—Yes
eTest Suite—Yes
eTest Suite—Yes
Table B.5 Performance Testing and Analysis Capability (continued)
Criterion/Feature Compuware Empirix/RSW
Performance Monitor Test Features
28206 03 pp. 217-282 r11kp.ps 7/26/01 3:10 PM Page 276
s
R
L
277B.5 Performance Testing and Analysis Capability
LoadRunner—YesAstra QuickTest—Yes
LoadRunner—YesAstra QuickTest—Yes
LoadRunner—YesAstra QuickTest—Yes
LoadRunner—Yes
Rational Suite TestStudio, RationalTestManager—Yes
Rational Suite TestStudio, RationalTestManager—Yes
Rational Suite TestStudio, RationalTestManager—YesResults can be exported to .csv.
Rational Suite TestStudio, RationalTestManager—Yes
SilkPerformer—Yes
SilkPerformer—Yes
SilkPerformer—Yes
SilkPerformer—Yes
Table B.1 Test Tool Vendors: Capture/Playbackand Script Creation (continued)
Mercury Rational Segue
Performance Monitor Test Features
28206 03 pp. 217-282 r11kp.ps 7/26/01 3:10 PM Page 277
S
R
L
Appendix B Test Tool Evaluations278
Maturity of vendor
Market share of vendor
On the market since 1996
Growing market share
On the market since 1996
Growing market share
Table B.6 Vendor Qualifications
Criterion/Feature Compuware Empirix/RSW
Consulting Requirements
Vendor Support
Software patches provided, if sodeemed necessary
Upgrades provided on a regular basis
Upgrades backward compatible: scriptsfrom previous version can be reusedwith later version
Training available
Help feature available; tool welldocumented
Tech support reputation throughoutindustry
Yes
YesMajor releases once per year, pointreleases once per quarter, bug fixesas necessary.
Yes
YesOnsite and public training.
Yes
Highly rated technical support.
Yes
YesMajor releases once peryear, point releases onceper quarter, bug fixes asnecessary.
Yes
YesOnsite and publictraining.
Yes
Highly rated technicalsupport from customerservice surveys.
Vendor Qualifications
Financial stability of vendor
Time existed
Profitable company
Since 1973
Former subsidiary ofTeradyne. Formed by thecombination of HammerTechnologies and RSWSoftware and funded byMatrix Partners.
Since 1996
28206 03 pp. 217-282 r11kp.ps 7/26/01 3:10 PM Page 278
Continued/consistent growthpattern over last severalyears.
Company incorporated in1989
s
R
L
279B.6 Vendor Qualifications
Tools have been supportingindustry since early 1990s.
Considerable market share
Various testing products have been onthe market since early 1980s.
Sustaining market share
SilkTest/QA Partner products onthe market since early 1990s.
Sustaining market share
Table B.2 Test Execution Capability
Mercury Rational Segue
Consulting Requirements
(continued )
Vendor Support
Yes
YesBuilt-in updater in theproduct to check for newpatches and features via theInternet.
Yes
Onsite and public training.Public training facilitiesaround the world.
Yes
Given prestigious SSPA STARaward last two years forexcellence in support.
Yes
YesEvery six months.
YesUpgrade and conversion utilities arealways provided.
YesRational UniversityOnsite and public training.
Yes
YesService has received many awards
Yes
Yes
YesAlways within major versionreleases.
Yes
Yes
Good Tech support.
Vendor Qualifications
Stable/profitable company
Since early 1980s
Stable/profitable company
Since early 1990s
28206 03 pp. 217-282 r11kp.ps 7/26/01 3:10 PM Page 279
S
R
L
Appendix B Test Tool Evaluations280
No consulting needed
Availability of and access to tool usergroups
Consulting not needed
Several discussion forums
Consulting not needed
Several discussion forums
Table B.6 Vendor Qualifications
Criterion/Feature Compuware Empirix/RSW
Vendor Support
Pricing
Price consistent within estimated pricerange
Price consistent with comparablevendor products
NA
Yes
NA
Yes
Licensing
Allows for floating license
Allows for node-locked license
Licensing used
Licensing rigid
Floating and node-locked licenses.
Yes
Yes
Flexibility is available.
Floating and node-locked licenses.
Yes
Yes
Flexibility is availableand is a commercialdecision
28206 03 pp. 217-282 r11kp.ps 7/26/01 3:10 PM Page 280
s
R
L
281B.6 Vendor Qualifications
Consulting not needed forAstra QuickTest or AstraLoadTest. A QuickStart isrecommended for others toobtain maximum return ontool investment.
Complete network of localusers groups throughoutworld.
Consulting generally not needed.Depends on type of testing to beperformed.
Several discussion forums
None needed. Jump startprograms available andrecommended for teams withtight time constraint.
User Groups exist.
Table B.2 Test Execution Capability
Mercury Rational Segue
Vendor Support
Pricing
NA
Yes
NA
Yes
NA
Yes
Licensing
Completely flexible licensingfrom pay-per-use toengagement based toperpetual pricing
Yes
Yes
Completely flexible licensing
Floating and node-locked licenses.
Yes
Yes
Yes
Floating and node-lockedlicenses.
Yes
Yes
No. Licensing can be based onmost need-based models forcustomers.
28206 03 pp. 217-282 r11kp.ps 7/26/01 3:10 PM Page 281
28206 03 pp. 217-282 r11kp.ps 7/26/01 3:10 PM Page 282
Recommended