Embed Size (px)
Citation preview
A Publication
VOLUME 5 • ISSUE 11 • NOVEMBER 2008 • $8.95 • www.stpmag.com
BESTPRACTICES:
Web Services, SOA
Like Duh, It’s the Network:Latency Test, Excellent!
When Automation Projects Fail Because of YOU
Dial ‘P’ For Performance
Contents A Publication
VOLUME 5 • ISSUE 11 • NOVEMBER 2008
1144CCOOVVEERR SSTTOORRYYGo Behind the Scenes At The Testers Choice Awards
This year’s most celebrated products as selected by you, our audience oftesters in the mosh pit of corporate QA teams. By Edward J. Correia
Sometimes performance problems areso obvious that they’re overlooked byeven the sharpest minds. Learn every-day techniques to determine if your net-work is the tortoise or the hare.
By Nels Hoenig
2288 Automation Project Failure—
Not On Your Watch!
3333 Network Slow? How to Know
2222 Press ‘P’ For Performance
Managementexpectationsdon’t alwaysjive with reali-ty. The timeto set them
straight isthe moment
you know there’sa mismatch. In part
two of a three-partseries on project auto-mation failures, learn
how to approachthese sometimesdelicate matters. By Elfriede Dustin
Distinguishing between componentsin a large system is not always easy; test-ing them all is even harder. Here’s astep-by-step guide to performance test-ing simple enough for school child.
By Keven Lui
DDeeppaarrttmmeennttss7 • EditorialCheck out CMG, a great non-profit resourcewith hundreds of documents about per-formance analysis.
8 • ContributorsGet to know this month’s experts and thebest practices they preach.
9 • FeedbackIt’s your chance to tell us where to go.
11 • Out of the BoxNews and products for testers.
13 • ST&PediaIndustry lingo that gets you up to speed.
37 • Best PracticesSOA testing is as much about business as itis about technology. By Joel Shore
38 • Future TestTake apart SOAs to prevent costly perform-ance rework. By John Michelsen
NOVEMBER 2008 www.stpmag.com • 5
I’ve stumbled upon aresource that I believeyou’ll want to know aboutand use. It’s called theComputer MeasurementGroup (www.cmg.org),and in essence, it’s wherethe world’s performancedata goes to retire. Wellthat’s the goal, at least.
According to CMG dir-ector Michael Salsburg, thenon-profit CMG started as a group ofIBM mainframe users doing perform-ance analysis. It soon expanded intoLinux and Unix. “In a simple way, itdelivers a service to businesses.Members of CMG have a reputation asexperts at making sure that levels ofservice end-to-end arewhat the business wants.”The site contains hun-dreds of white papers,submitted by members,chronicling their per-formance measurementexperiences.
“You can get any levelof performance you wantif you have enoughmoney,” says Salsburg.“The real balance is to getmost performance forleast money.” That’s whatCMG aims to help compa-nies do. For example, oneof this month’s most pop-ular articles is called “Xenvs. VMware – The Battle ofBrands,” which compares performanceof the two.
Another is “Using ITIL Best Practicesto Create a Capacity ManagementProcess,” detailing the InformationTechnology Infrastructure Library forservice management adopted by BritishAirways, Hewlett Packard, IBM,Microsoft, Proctor and Gamble andmany other top-flight companies.
Does your organizationemploy telecommuters? Pe-rhaps home-office workersshould read “Remote ButNot Forgotten – Workingfrom Home Successfully,”which contains advice forhome-office workers aboutbuilding and maintainingtrust relationships with co-workers and bosses.
People doing specifictypes of testing might align themselveswith particular authors. “A person whowrites and presents a lot of software per-formance engineering papers is Dr.Connie Smith. Or when you look at soft-ware test and quality assurance, it’sdevoted to function testing,” Salsburg
says. But at some point, hesaid, your department willhave to guarantee a levelof quality of service for theapplication. “That’s wherewe come in. We see our-selves as a go-to communi-ty where you can seewhat’s out there and howproducts are performing.People write papers andthey present them, andtheir companies pay theirway. We offer a communityand that’s what we wantpeople to know about.”
More than the big pic-ture, CMG also can helptesters get into the nutsand bolts of testing. “We
would help you understand the tech-niques of how to instrument yourapplication, to understand perform-ance and forecast how it would per-form on a production system andunder heavy load.” That’s somethingyou might not be able to create in yourtest environment and can’t afford toduplicate your production environ-ment, he added. ý
CMG: Data ForYour Thoughts
NOVEMBER 2008 www.stpmag.com • 7
VOLUME 5 • ISSUE 11 • NOVEMBER 2008
Ed Notes
President Ted Bahr
Executive Vice President Alan Zeichick
Software Test & Performance (ISSN- #1548-3460) ispublished monthly by BZ Media LLC, 7 High Street,Suite 407, Huntington, NY, 11743. Periodicals postagepaid at Huntington, NY and additional offices.
Software Test & Performance is a registered trade-mark of BZ Media LLC. All contents copyrighted2008 BZ Media LLC. All rights reserved. The priceof a one year subscription is US $49.95, $69.95 inCanada, $99.95 elsewhere.
POSTMASTER: Send changes of address to SoftwareTest & Performance, PO Box 2169, Skokie, IL 60076.Software Test & Performance Subscribers Servicesmay be reached at [email protected] or by calling 1-847-763-9692.
Cover Photo Illustration by The Design Diva, NY
Director of Circulation
Agnes [email protected]
EDITORIAL
SALES & MARKETING
READER SERVICE
Art DirectorLuAnn T. Palazzo
ART & PRODUCTION
BZ Media LLC7 High Street, Suite 407Huntington, NY 11743+1-631-421-4158fax [email protected]
EditorEdward J. Correia+1-631-421-4158 [email protected]
Copy DeskAdam LoBeliaDiana Scheben
Editorial DirectorAlan [email protected]
Contributing EditorsMatt HeusserChris McMahonJoel Shore
Publisher
Ted Bahr+1-631-421-4158 x101
Associate Publisher
David Karp+1-631-421-4158 x102
Advertising Traffic
Liz Franklin+1-631-421-4158 [email protected]
List Services
Lisa [email protected]
Reprints
Lisa [email protected]
Accounting
Viena Ludewig+1-631-421-4158 [email protected]
Customer Service/Subscriptions
Edward J. Correia
•‘We’re a go-to
community’ for
performance
testers.
•
KEVEN LIU graduated fromShanghai Jiao Tong University inChina. Over the past 10 years,Keven has gained experiencewith software development, project management, qualityassurance and testing, workingclosely with numerous clients todevelop and deploy softwareapplications for Hong Kong AirCargo Terminals Ltd. He cur-rently serves as the company’s testleader.
Keven has published articles in Chinese about softwaremanagement, CMM/CMMI and testing. Starting on page 22,he discusses how to simplify complex testing.
Contributors
TO CONTACT AN AUTHOR, please send e-mail to [email protected].
8 • Software Test & Performance NOVEMBER 2008
ELFRIEDE DUSTIN, is author of“Effective Software Testing” (Add-ison-Wesley, 2002) and leadauthor of numerous other works.She is also a testing consultant cur-rently employed by InnovativeDefense Technologies, a Virginia-based software testing consultingcompany specializing in automat-ed testing.
In her role as consultant,Elfriede has evaluated scores of soft-ware testing tools. This month, shepresents the second installment of her three-part series on soft-ware automation testing, beginning on page 28.
NELS HOENIG has 15 years expe-rience in the IT industry as aproven software quality engineer,project leader, business analyst andapplication support specialist.
Nels has worked both in thedevelopment of new applicationsand the implementation of exist-ing applications.
In his role as senior qualityassurance leader at DSW, he's anexpert at listening to user con-cerns and developing a trust rela-
tionship that focuses on issues and allows the project to succeed.In this issue of Software Test & Performance, he writes abouthow network performance problems can sometimes be so obvi-ous that they’re overlooked by the test team. Read all about itbeginning on page 33.
THAT’S SO 1990’SRegarding “Which Functional Testers Dothe Best Job?” July 15, 2008, (http://www.sdtimes.com/link/32536): The fact is,most of the products you mentioned weredeveloped in the 1990s for the applica-tion context of the 1990s. Beginning in2001 we here at Software Research tooka new tact, based on the recognition (well,OK, the bet!) that Web browser enabledapplications would win out. The bet wasa good one, of course.
The result of our efforts was our eValidsolution– it's now patented that puts ALLof the test functionality in a browser. It isa very simple idea: test Web applicationsfrom where they are normally driven. Butdoing it is somewhat more complex. Butwhat we have is simple, easy to use, inex-pensive, and 100 percent capable.
eValid, now in Version 8, is in use in1000s of companies worldwide.
In monitoring mode—functional testsplayed back regularly—eValid accountsfor [about] 10,000,000 monitoring test
playbacks per year.Edward Miller, Presidentwww.e-valid.co, San Francisco, CA
While the sentence “You can think moreclearly when you are busy if you planahead” is okay, more clarity and com-pleteness of the thought is conveyed bythe following: “You can think more clear-ly when you are busy if you have plannedahead.”
Michael F. Stanton
A PAT ON THE BACKRegarding “Security & Quality: A UnionOf Equals,” Software Test & Performance,Oct. 2008: When you have a few sparemoments please give a well deserved paton the back to Danny Allan for hisremarkable piece of journalism and your“guts” (or other anatomical parts) forprinting it.
I’d like your permission to pass thisarticle around especially to Steve Lasky,Publisher/Editor-in-Chief of “Security
Technology and Design.”Attached for your reading pleasure or
lining your wife’s bird cage is somethingI have worked at since 1962 to help mebe a better Air Force NCO/security per-son and now a security consultant, onewho chooses his projects with the great-est of care.
Bill Warnock
WHERE THE ACTION ISRegarding 'Ten Concepts for SuccessfulAutomation Scripts,” Test & QA Report,Sept. 6, 2008 (http://www.sdtimes.com/link/32824): Although it is essential forthe test automation engineer to havedevelopment skills it is not necessarily thecase for those who enter and maintainthe test scripts. This is possible if your test
automation approach follows ahigh level framework drivenapproach.
The script editors of our testautomation scripts are businessanalysts, testers and managers whotype keywords or (as Hans Buwaldawould say) action words - into anExcel form. Those people do notneed development skills, but Iabsolutely agree that without sucha framework any script editorwould need those skills and it isessential – as you say – to have a
good understanding of how software isorganized into re-usable pieces. In our caseit is of great benefit that only a small num-ber of specialists are needed to keep thecore framework running for all users.
My contribution to this test automa-tion article today is also a new cartoon ofmine. If automation is done right, bugswill learn to fear automation (see attachedpicture). =;O)
Best regardsT.J. ZelgerTest lead, COREAudatex Systems, Switzerland
Editor’s Note: We will include the picture in afuture edition of the Test & QA Report.
Feedback
NOVEMBER 2008 www.stpmag.com • 9
FEEDBACK: Letters should include the writer’sname, city, state, company affiliation, e-mailaddress and daytime phone number. Send yourthoughts to [email protected]. Lettersbecome the property of BZ Media and may beedited for space and style.
‘TOPS’ NOT ALWAYS ‘BEST’
Regarding “China's Top Coders” editorial in Sept. 2008, issue of Software Test &Performance:
As always I enjoyed your editorial comments.The TopCoder’s piece was very inter-esting and thought provoking. It is particularly interesting is taking a game like atmos-phere and turning it into an opportunity to deliver value to a company…should I sayBRILLIANT?!
Not sure how many of these 114,000+ programmers are formally trained (from thesurvey’s it looks like less than 35%), and if the distribution is such that the majorityare either Eastern European or Asian then is it a true barometric measure for pro-gramming excellence. Programming application is not cultural, nor is it hereditary, it’spragmatic and with well defined rule sets that are used. But let’s talk the businessvalue proposition and what makes the outcome successful.
Simply,TopCoder put out narrow scope challenges (or open ended when it comes todoing an efficiency improvement project,when then creates a research atmosphere) thatallow these people to deliver. In a typical IT setting the scope is constantly changing, thebusiness dynamics are substantial, and the requirements are seldomadequate.For these reasons, the Agility paradigm has become an attrac-tive solution to these issues. I’m wondering if we put these pools oftalent into these settings whether the delivery outcome would be thesame.
One might argue that delivery is all that matters. But I’m not surethat I would trust strategically important applications to this approach.
Respectfully,Jerry E. Durant, Chairman EmeritusThe International Institute for Outsource Management Winter Springs, FL
Oracle Corp.’s new Application TestingSuite is an addition to its Oracle Enter-prise Manager product line. The newsuite provides an integrated solution forload testing, functional testing and testmanagement, letting customers testpackaged, Web and SOA-based apps andtheir underlying infrastructure.
The Oracle Application Testing Suiteemploys an integrated scripting plat-form for load and functional testing thatthe company says can help cut scriptingtime in half and reduce weeks from atypical project’s testing schedule. Thesuite, which works with both Oracle andnon-Oracle applications, includes:
• Load Testing for Web Applications:By simulating tens of thousands ofvirtual users accessing the appli-
cation simultaneously, Load Testingfor Web Applications measures theeffect of the load on applicationperformance providing a way to val-idate the performance and scala-bility of applications and Web serv-ices. The tool can test against syn-thetic and real-world test loads.
• Functional Testing for Web App-lications: Lets developers andtesters execute functional tests ofapplications and Web services.According to Oracle, the transac-tion engine simplifies automatedtest script generation and enablesout-of-the-box automation of themost complex Web applicationsand associated technologies toquantifiably validate the end user’s
experience.• Test Manager for Web Applications:
Helps users manage the Web appli-cation testing process, enabling fulltest coverage, by allowing users todefine testing requirements, spec-ify and execute manual or auto-mated tests to validate require-ments, and then manage thedefects that those tests uncover.
The company says that the OracleApplication Testing Suite provides acommon, standards-based scripting envi-ronment that links functional testingand load testing to simplify test processand accelerate completion of tests by asmuch as 50 percent. The test suite alsoincludes a functional and load testingaccelerator for Oracle's Siebel CRM.
Active Streaming XF is a new servicefrom that measures the speed and qual-ity of streamed video and audio deliv-ered over the Web. Self-service featuringa highly intuitive diagnostics interfaceand Active Streaming XF rapidly identi-fies and troubleshoots streaming mediaperformance issues so that businesseslike advertisers, broadcasters, retailersand social networks can deliver contentwithout delay or disruption to audiencesaround the world.
Nikki Baird, managing partner at RSRResearch, a research company run byretailers for the retail industry, said:"Interest in video as part of the onlinecommerce experience is definitely onthe rise. Between product demos andreviews alone, the potential for the medi-um is huge. A solution, like ActiveStreaming XF, that can help sites ensurea high quality video experience for visi-tors is very timely."
According to Gomez, possible uses
for the ActiveStreaming XF serv-ice include measur-ing the perform-ance and quality ofhigh-value streamsas experienced byend users in differ-ent geographies. Itcan also be used forverifying if syndi-cated or third partycontent sources arenegatively affectingthe end-user expe-rience; validating the performanceclaims of hardware vendors or contentdelivery networks prior to purchase; andobjectively managing service level agree-ments with streaming service providers.
The company explained that theActive Streaming XF interface providesactionable, near real-time insight intometrics such as start up and buffer time.
Entirely self-serving users can set up,execute and analyze tests within min-utes, making the Active Streaming XFservice flexible and easy to administer,and speeding time to resolution, saidGomez.
The service is sold on an annual sub-scription basis, with the price varyingdepending on the features needed.
Gomez MonitorsMultimedia WebExperience
Out of the Box
Oracle Delivers App Testing Suite
Active Streaming XF can validate the effectiveness of content deliv-ery networks and multimedia delivery infrastructure.
NOVEMBER 2008 www.stpmag.com • 11
12 • Software Test & Performance NOVEMBER 2008
OpenMake AndSerena PromoteMeister BuildsOpenMake Software, the companybehind the open-source Meister build-management system, is working withSerena Software to offer a migrationpath for customers using SerenaBuilder and Serena Builder Enter-prise. Serena customers now have thechoice to move their Builder supportand maintenance from Serena Soft-ware to OpenMake Software, provid-ing them immediate access to the lat-est version of OpenMake Meister.
The last release of Serena Builder,version 6.41, uses an older version ofthe Meister code, which is currently upto version 7.2.
"Moving to OpenMake Meister 7.2will provide the Serena Builder cus-tomers with many new features andenhancements," states Stephen King,CEO of OpenMake Software. “Themost important of these featuresinclude Meister's improved continu-ous integration server, its ability to per-form build mash-ups that synchronizeIDE builds with the Continuous Inte-gration builds and support for Micro-soft Team Build and Visual Studio2008.”
“OpenMake Software and SerenaSoftware will work together to supportinteroperability between Serena Build-er and OpenMake Meister,” saidSteven Mitzenmacher, Vice Presidentof Corporate Development at SerenaSoftware.
New SOAP TestSoftware Knows The ScoreHow standards compliant is yourSOAP and WSDL? Check the score tosee if it passes the test, says CrosscheckNetworks. The company’s SOAPSonar4.0 test software incorporates modulesthat enable QA and Developmentteams to quickly test the quality oftheir enterprise WSDLs based on aneasy-to-understand scoring system.The scoring system provides the teams
a valuable feedback loop to improvethe quality of their WSDLs, thusincreasing interoperability, securityand integrity of services.
"This release addresses one of themajor pain points that SOA users face,that is, how to test for quality ofWSDLs," said Mamoon Yunus, CTO atForum Systems and an advisor toCrosscheck Networks. "The latestSOAPSonar release offers uniquecapabilities that provide strong valida-tion features that helps strengthen thequality of enterprise services."
Version 4.0 of the software, nowshipping, offers WS-Policy Consum-ption and Policy Generation report-ing, WSDL scoring, custom XSDSchema support, Schematron valida-tion, SmartCard certificates, and asyn-chronous performance testing. It’s afree upgrade for customers runningSOAPSonar 3.5 or later.
HP and Surgient PlayVirtually Together
Surgient, a virtualization company, isworking with Hewlett-Packard to improvethe speed and accuracy of software devel-opment and testing.
The Surgient Virtual AutomationPlatform optimizes IT's ability to supportcritical business initiatives, effectively man-aging diverse virtual resources, accord-ing to Surgient. Integrated with HP ServerAutomation software, the solutionempowers IT teams to automate the cre-ation of identical production environ-ments, whether physical or virtual, allow-ing the highest quality software applica-tions.
“With our pre-existing integration withHP Quality Center and new integrationwith HP Server Automation software, wecan provide organizations with a com-prehensive solution for the creation andmanagement of cost- effective virtual envi-ronments for the testing of production-ready software applications,” said SameerJagtap, Surgient’s vice president of prod-uct strategy. “Customers can acceleratetest cycles by consolidating test infra-structure and automating the set-up ofclean, identical production environmentson-demand.”
IT operations and development testteams can now ose Surgient to define thevirtual blank- slate deployment environ-ment, and then populate that environ-ment by leveraging existing, production-ready operating system sequences fromHP Server Automation to build replicasof the production systems in the virtualenvironment.
“Customers can manage change moreeffectively and meet compliance needswith this integrated solution,” said MichelFeaster, director of products and BusinessService Automation Software at HP. “Bycreating identical, clean copies of pro-duction systems and leveraging the sys-tem of record for production deploy-ments to be used for testing purposes,customers gain greater flexibility in get-ting new applications out to market.”
Registration Opensfor FutureTest 2009Come to FutureTest 2009, Feb. 24-29 inNew York City, to focus on what it takesto deliver. Whether you’re a vice presi-dent of software test, director of soft-ware quality, project manager or a sen-ior development manager with respon-sibility for test, FutureTest 2009 is whereyou want to be.
If you’re evaluating Web software builtfor use by your customers or by employ-ees within your company, or if you’recharged with vetting Web apps built forinternal consumption, FutureTest is foryou. If you want to keep your orgnaizationon the cutting edge of software testingand quality assurance, FutureTest 2009 isthe must-attend event of the year for youand your senior test professionals.
At FutureTest 2009, the editors ofSoftware Test & Performance assembledthe greatest minds in software test/QA tohelp you look into the future so that youcan better manage the present, whilealways being ready for change.
The eXtreme Early Bird registrationprice of $995 is a $500 savings from thefull conference price. This discountexpires on Nov. 21, so register soon atfuturetest.net. Follow the conference asit evolves at blog.futuretest.net.
Send product announcements to [email protected]
Taster’s, er, Testers Choice
SSTT&&PPeeddiiaaTranslating the jargon of testing into plain English
NOVEMBER 2008 www.stpmag.com • 13
Matt Heusser andChris McMahon
Matt Heusser and Chris McMahonare career software developers,testers and bloggers. They’re col-leagues at Socialtext, where theyperform testing and quality assur-ance for the company’s Web-based collaboration software.
Q:What would your answers be?
Did you exhaustively test
this? Are we doing an SVTafter our BVT? Does the
performance testing pass?
What are your equivalenceclasses? Which heuristics
are you using?
A: ST&Pedia will help you answer questions
like these and earn therespect you deserve.
Upcoming topics:
DecemberTest Automation
January 2009Change Management, ALM
FebruaryTuning SOA Perfomance
MarchWeb Perf. Management
AprilSecurity & Vuln.Testing
MayUnit Testing
So you’re a tester. How will you test? Put sim-ply, here are the main things you will need:
1. Some test data.2. A way to send that data to whatever
you’re testing.3. A means of analyzing the result.4. A way to figure out what went wrong.5. A way to report about what you did.6. To see what happens when you use a
whole lot of test data.7. You will need a way to report bugs and
how they were fixed.Doing all this manually can take a lot of
time, and in the case of changing versionsover time, can lead to mistakes. To simplifythe process and minimize mistakes, compa-nies and testers have developed tools to dosome of this work for you.
This month’s issue of Software Test &Performance—the Testers Choice issue—describes the tools that readers like best anduse most. And this month’s ST&Pedia is devot-ed to some of the Testers Choice categories, tohelp you understand how they’re defined andhow they fit into your daily life as a tester.
DATA TEST/PERFORMANCE tools allowfor setup, population, migration, testing,performance measurement, and tuning ofdatabases, Web servers and all forms of data-driven services.
FUNCTIONAL TEST tools evaluate appli-cations and components through executingcode. They typically “run” automated tests,which may drive code at the unit level, runbusiness logic at the acceptance level, ordirectly drive a userinterface.
STATIC/DYNAMIC CODE ANALYSIStools look at the code, either examining it astext looking for flaws (static), or providinginstrumentation on the code as it is execut-ed (dynamic). Instrumented code can pro-vide information about the software undertest, such as the percentage of lines of code,or possible code branches, that were cov-ered by the tests and where uncovered(missed) spots lie.
TEST/QA MANAGEMENT tools provideplanning, organizing, and oversight tools for
the test process, such as the work breakdownstructure, the requirements that have beentested, what remains to be tested, knownissues, and creation of reports and summaries.
DEFECT/ISSUE MANAGEMENT toolstrack the status of known issues and providea history of when bugs were discovered, bywhom, how to reproduce the issue, and whenthe issue was fixed.
LOAD/PERFORMANCE TEST tools helpmeasure the average experience for a clientand many generate and measure perform-ance under load.
SOA/WEB SERVICES TEST tools helpevaluate a Web services stack, such as SOAP,XML, and WSDL, typically by sending (simu-lated) messages to the Web server and evalu-ating responses.
SECURITY TEST tools identify potentialvulnerabilities of applications, generally byeither hardening the application or operat-ing system (such as a firewall) or by generat-ing multiple attacks on the system and listingvulnerabilities. Some also spot “vulns” usingstatic code analysis.
TEST AUTOMATION tools execute com-puter programs and evaluate the results. Thetwo most common approaches to GUI testautomation are record/playback/comparescreenshots and drive/compare GUI throughobject IDs, known as keyword driven testing.(More on test automation next month).
EMBEDDED/MOBILE TEST/PERFOR-MANCE tools specialize in the testing ofembedded or wireless applications and sys-tems, either with support for embedded oper-ating systems and/or with specific interfaces toand features for testing mobile devices.
SCM/BUILD MANAGEMENT toolscreate an automated library of codechanges, including differences. Build man-agement systems assist or automate builds,integrate components, and automate thedevelop and deploy cycle by running buildsand automating buildtime tasks. ý
hands together for LoadRunner, QuickTest Profes-sional and TestDirector for Quality Center.
I mean like last year, the players were the same,dude, but the awards went to Hewlett-Packard,which acquired Mercury in November 2006.Bonus! These tools have received our standing ova-tions for four years running. They are like trulyamazing and excellent.
Just as last year’s tallies bestowed the awesome
LoadRunner with our Grand Prize for most votesoverall, so too does the versatile tool receive thisyear’s top honor, leading in the individual categoriesof Java/Test Performance, Data/Test Performanceand Load/Test Performance. Schwing!
So here’s your backstage pass to party with the besttesting tools to take the stage in 2008, along with thetwo closest warm-up bands. This year’s categoriesincluded defect management, automation, security, per-formance, free solutions and those from new players—seventeen categories in all. Maybe you’ll seeyour Testers Choice. Party on, Garth.
By Edward J. Correia
E x-squeeze me! OK, pop quiz: In 2006, did youvote for Mercury Interactive as your starring
supplier of testing tools? Chyeah! You all put your
14 • Software Test & Performance
Party Time. Excellent.Take a Trip Backstage to Hang With This Year’s Test-Tool Heroes (We’re Not Worthy!)
LOADRUNNER took the most votes in the data/test perform-ance category, which groups products in terms of their abilityto apply test data to an application or system and evaluate howit handles the processing.
At the core of LoadRunner’s power is the Virtual User gen-erator (VuGen), which creates test scripts for playback and
test case simulation. Script parameters can be modified asneeded to adapt to different cases, data parameters (such asfor keyword-driven testing), correlation and error handling.A controller module runs the scripts and can simulate largenumbers of users while the analysis module tracks and graphsbehavior and performance results.
If you’re new to LoadRunner or need to pick up a few newtechniques, you might refer to the March 2007 issue ofSoftware Test & Performance for an excellent tutorial by con-sultant Francisco Sambade on using the tool to test multipleprotocols.
Runners-up in this category were Compuware File-AID/CSand Red Gate’s SQL Data Generator. File-AID is an enterprisedata management tool that permits testers to quickly build
The tester’s perennial favorite is LoadRunner, amulti-faceted load and performance testing tool.LoadRunner examines application behavior andperformance of running systems as it applies load to sim-ulate application activities. The tool is capable of simu-lating real-world scenarios with hundreds or thousands ofsimultaneous nodes simulating an application, Web serv-ice, Web server, database or other program in Windows orUnix.
HP’s most recent major update of LoadRunner camein June 2007, when it enhanced the tool to better addressthe testing needs of companies deploying applicationsbased on so-called Web 2.0 technologies.
www.stpmag.com • 15
Data/TestPerformance
GGRRAANNDD PPRRIIZZEE WWIINNNNEERR 22000088
HHeewwlleetttt--PPaacckkaarrdd LLooaaddRRuunnnneerr
test data environments across a multitude of systems, includ-ing mainframes, MVS, DB2 and distributed systems (as thefinalist CS edition).
Introduced just this past April, Red Gate’s SQL DataGenerator has generated enough attention from testers toand unseat Intel’s VTune Performance Analyzer, last year’srunner-up in the category. The Red Gate tool’s claim to fameis the ability to populate “two million rows of intelligent datafor 10 tables in less time that it takes to get a cup of coffee.”What’s more, it does so with a single mouse-click, accordingto the company.
One again, HP’s QUICKTESTPROFESSIONAL takes the top prize for functional testing. The2006 Grand Prize winner, QuickTest Pro is a Windows-only record-and-playback tool for functional and regres-sion GUI testing, and it includes a scripting languagebased on VBScript that can control and manipulateprogram objects and allow testers to script out testprocedures.
Leading it to victory in 2006 were enhamcementsto the tool’s team collaboration capabilities, a newobject repository manager and the ability to share functionlibraries across tester workgroups. It also added keyword man-agement and drag-and-drop test-step construction, XML out-put for test-result reporting, and a new and more accuratedebugger that identifies errors while building or maintainingtest cases. HP has released no major updates recently.
Runners-up on the functional test category were IBMRational Functional Tester and Borland Silk Test. Neither oflast year’s finalists—Parasoft SOAtest and Compuware OptimalQuality Management—came close this time.
Functional Tester is an automated function and regressiontest tool for GUI and data-driven testing. It allows scriptingwith its own ScriptAssure environment or can be driven byVisual Basic .NET, Java from within Eclipse, or custom con-trols built with Java or .NET through a proxy SDK.
SilkTest is also an automated function and regression testingtool, with cross-platform testing capabilities for applications builtfor client/server, Java, .NET and the Web. It also includes a script-ing framework to help facilitate reuse of test assets.
Testers handed the top honor in the the code analysis cate-gory to IBM’s RATIONAL SOFTWARE ANALYZER DEVELOPEREDITION, unseating three-time imcumbent PurifyPlus, a use-ful tool with long and storied roots, now also developed andmarketed by IBM Rational. The automated runtime analysis
tool spots memory leaks, profiles application performanceand analyzes code coverage. Supported languages includeC/C++, Java, the .NET languages, Visual Basic, and Visual C++.Versions are available for Linux, Unix and Windows.
Also unseated by this year’s runners-up—the Eclipse Testand Performance Tools Platform and HP’s DevInspect—wereCompuware’s DevPartner Studio and Parasoft’s Jtest, last year’sfinalists in the category.
Eclipse’s TPTP project for Linux and Windows got an infu-sion of new features last June as part of the Ganymede releasetrain. Major new capabilities include increased test coveragethrough test creation, automation and expanded run timeexecution; the ability to execute multiple instances of the toolin parallel; simplified handling of test assets, including theability to cut, copy, paste and rename. Also new are animproved thread profiler and new profiler API for Java 1.5and above.
Inherited along with HP’s acquisition of SPI Dynamics nearthe end of 2007, DevInspect was widely praised this year bytesters. The tool is intended primarily for automated securi-ty testing and remediation during development.
HP is once again on top with TESTDIRECTOR FOR QUALITY CENTER, which testersvoted this year and last their favorite tool for test and QA man-agement. TestDirector for Quality Center includes modulesfor requirements management, test planning, test lab anddefect management.
TestDirector allows testers and managers to gather require-ments, design and schedule manual and automated tests, andanalyze the results, all through a browser. Also included aregraphical reporting and integration with HP’s WinRunnerand QuickTest Professional tools. Options include extensionsfor Oracle, SAP and SOA.
Finalists in this category were Borland’s SilkCentral TestManager (as last year) and IBM’s Optim Test Data ManagementSolution, kicking VMware Lab Manager from its third-placeperch last year.
Since being acquired by Borland along with Segue Softwarein February 2006, SilkCentral Test Manager has been inte-grated with VMware Lab Manager to streamline cross-plat-
form testing, and with Eclipse to facilitate capturing oftest results in a centralized console for impact analysis.
The browser-based environment for remote, simulta-neous test execution and management of JUnit/NUnitand other third-party testing frameworks also includes amanual test client to ensure repeatable data collectionand reporting. Reporting is provided by BIRT.
Optim Test Data Management Solution is IBM’s way to man-age test data and improve application quality. The tool per-mits testers to apply specific standards of coverage, createerror or boundary conditions and create an environment thatclosely simulated production.
16 • Software Test & Performance NOVEMBER 2008
Static/DynamicCode Analysis
Test/QAManagement
FunctionalTest
NOVEMBER 2008 www.stpmag.com • 17
The same tool can be used across enterprise apps, includ-ing PeopleSoft and Siebel; databases from IBM, Microsoft,Oracle, Sybase and others; and Linux, Unix, Windows andZ/OS.
Last year there were two winners in the defect andissue management category: HP TestDirector for QualityCenter and the Mozilla Foundation’s Bugzilla were tied. Thisyear, TESTDIRECTOR alone took the top spot and Bugzillamoved to third, edged out by Microsoft’s Visual Studio TeamEdition for Software Testers.
The defect tracking module in TestDirector automaticallychecks its central defect database for similar defects each timea new defect is logged, helping to reduce duplicates. The tooltabulates defect statistics to aid management deployment deci-sions and identify trends.
A relative newcomer, Visual Studio Team Edition forSoftware Testers can perform functional and load testing onWeb applications and Web sites, and manages and permitsreuse of test cases, all without writing a single line of code.
Bugzilla began its life in 1998 and was originally written inTcl by Terry Weissman. Thinking another language might getmore traction within the community, Weissman decided toport it to Perl, which resulted in Bugzilla 2.0. In April 2000,the project was handed off to Tara Hernandez, who gainedmore developer participation, including that of its currentcustodian, Dave Miller. Bugzilla last year won the top spot inthe free test/performance tools category, but this year slippedto runner-up.
This year’s winners in the loadand performance testing category are a carbon copy of lastyear’s. LOADRUNNER again topped the list, with capabilitiescentered around its Virtual User Generator record-and-play-back tool. VuGen generates an editable script of user actionsthat are routed through a protocol-dependant proxy duringrecording. So when scripts are generated using the Web/HTTPedition, testers can set LoadRunner to generate either URL- orHTML-based scripting, for example.
Runners-up in this category again were IBM RationalPerformance Tester and Borland SilkPerformer. PerformanceTester stresses Web applications and tests their scalability.Integrated with IBM’s Tivoli management environment, itenables large, multi-user tests while using minimal hardwareresources. Distributed controller agents are available for Linux,Windows and z/OS. Test results are displayed in a Linux or
Windows application with high-level and detailed tree views anda tree-based text editor.
SilkPerformer is an automated, sharable environment forexecuting load and stress performance tests across Java and .NETapplications with server-side analysis. The tool can simulate thou-sands of users with no licensing restrictions for controllers orprotocols. A visual script recorder enables script editing andmanipulation. A plug-in extends test creation to Eclipse. There’salso a SOA edition.
In the SOA and Web services testing category, last year’s winnerwas this year’s no-show. LoadRunner has fallen out of favor fortesting these types of applications, replaced in 2008 by IBM’sRATIONAL TESTER FOR SOA QUALITY.
When your app relies on someone else’s Web service, youcan’t leave availability to chance. For example, if you’re depend-ing on a credit authorization service that’s accessed via the Web,how can you ensure that it won’t fail on Black Friday, traditionallyAmerica’s busiest shopping day? While you can’t control thecredit agency’s service, you can use Tester for SOA Quality tosimulate thousands of hits against its service and watch how yourapplication behaves while simultaneously trying to conduct hun-dreds or thousands of transactions. The tool also works withservices that present no GUI.
Runners-up in the category were HP’s QuickTest Professionaland Parasoft SOA Quality Solution.
A perennial favorite of testers, QuickTest Pro is an auto-mated functional and regression testing tool for GUI applica-tions that records and plays back user actions on Web or nativeapplications. Scripts are recorded in a proprietary scripting lan-guage based on Microsoft’s VBScript and can be modified, playedback and reused.
Parasoft’s SOA Quality Solution incorporates the company’smulti-layer workflow approach to testing and quality assurance.Capabilities include policy enforcement with interoperabilityand consistency checking across SOA layers; multi-layer verifi-cation of the underlying implementation for auditing; serviceemulation with end-to-end testing and emulation of businesslogic or transactions; regression testing; penetration testing; andload and performance testing.
Last year, SPI Dynamics took the top prize in security testingwith WebInspect, its security scanning and assessment tool forWeb applications. Now owned by HP, which took both runner-up slots in the category, WebInspect this year was displaced byIBM’s RATIONAL APPSCAN STANDARD EDITION.
Defect/IssueManagement
Load/PerformanceTest
SOA/WebServices Test
SecurityTest
18 • Software Test & Performance NOVEMBER 2008
Inherited along with its acquisition of Watchfire, AppScan is anautomated security auditing tool for developers, QA staff and secu-rity auditors. It’s also available in multi-user and SaaS versions. IBMin October introduced AppScan Express Edition, a versionof the utility aimed at small and mid-sized businesses withsupport for apps made with AJAX, Flash and other so-called Web 2.0 technologies.
HP’s Assessment Management Platform, a first-timerfor the Testers Choice awards, is an automated, central-ly controlled security scanning and testing platform forWeb applications. It permits distributed teams to accessand remediate security vulnerabilities through a dashboard-styleinterface.
WebInspect last year, prior to its acquisition by HP, got itselfa shiny new profiler that scans Web apps and suggests configu-ration settings for the most effective testing. It also received a real-time traffic monitor that reports HTTP activity during a scan. Aresults window displays requests and responses sent by WebInspectduring crawls and security audits. The tool was completely rewrit-ten in January 2007, and SPI claimed performance improvementsand compatibility with modern technologies and techniques.WebInspect 7.5 reportedly further improved auditing capabili-ties for spotting vulnerabilities in AJAX-based applications andstrengthened its support for Windows Vista.
QUICKTEST PROFESSIONALagain comes out on top, this time in the
test automation category. QuickTest Pro includesa pluggable architecture to allow customization. The toolincludes plug-ins for ActiveX controls, Web applications andVB objects. Optional .NET-language objects and multimediaplug-ins are available. The tool also saves the AUT’s screensalong with the recorded script, highlighting the portion of thescreen being tested as an aid to regression testing. These arealso useful for creating checkpoints, which can apply to images,tables and pages, and permit verification of expected applica-tion behavior during automated tests. Other features includevirtual objects, output value for data verification, data- and trans-action-driven testing, and exception handling.
Compuware TestPartner is an automated GUI function test-ing tool that uses Microsoft’s Visual Basic for Applicationsfor scripting, debugging and error handling. Tests alsocan be created using "Visual Navigator," a storyboardenvironment. By displaying screens of the AUT, testersselect from lists of automation steps to assemble story-boards. They also can add test logic, variables and oth-er particulars.
Another newcomer to the Testers Choice awards is HPBusiness Process Testing, a Web-based test design solutionthat the company says is intended to allow subject-matterexperts to build and execute manual and automated testswithout any programming knowledge. It automates testingand creation of documentation, and is also said to reducetest maintenance.
Microsoft VISUALSOURCESAFE was tops in the
SCM/build management category for the sec-ond year running. Originally developed by One Tree Software
as SourceSafe, this tool was first released in the early 1990s asversion 3.1, which a Wikipedia author speculates was meant tomatch the current version of Windows at the time. Microsoft’sSCM at the time was called Delta and was said to be inferior.Microsoft acquired the 16-bit SourceSafe and released a 32-bitversion as Visual SourceSafe 4.0 around 1995. It would be 10years before the company released VSS 2005, the first client-server version; until then it was limited to local use.
Among VSS’ strongest attributes are its tight integration withVisual Studio, relative ease of use and price; it’s free with cer-tain subscriptions to MSDN. It also offers numerous expan-sion possibilities thanks to a pluggable architecture. Its futurein the enterprise seems uncertain, however, since many of itscapabilities are implemented in Visual Studio Team FoundationServer and several good rival systems are available from the open-source community.
One such rival system is Subversion, which again this year isa runner up along with ElectricCommander from Electric Cloud.
The Subversion version control system, particularly popularamong open-source developers, was launched in 2000 byCollabNet. Subversion commits are atomic and do not cause cor-ruption if interrupted. All moves, adds, changes and deletionsare versioned, including files, directories, trees, file metadataand symbolic links. It supports binary file storage with file-lock-ing for unmerged files, and sends diffs in both directions.
Build automation tool maker Electric Cloud in lateSeptember began shipping ElectricCommander 3.0, whichincluded the ability to "preflight" newly modified source codeto determine if it will correctly build into an application. Thenew version also integrates with Eclipse and Visual Studio.ElectricCommander automates program component assem-bly without regard to development language or build utility.It supports numerous scripting languages, including Bash,perl, Python and Tcl. It also works with AccuRev, ClearCase,Perforce, Subversion and Synergy software configuration man-agement systems.
Parasoft took the top prize in the embed-ded/mobile test and performance category with its C++ TEST,putting IBM’s Rational Test RealTime out of the top spot. C++Test is a solution for code analysis and review, coverage analysis,regression testing and automation of unit and component test-ing. Parasoft offers versions for embedded and non-embedded
TestAutomation
Embedded/MobileTest/Performance
SCM/BuildManagement
NOVEMBER 2008 www.stpmag.com • 19
apps for Linux, Solaris and Windows. C++ Test works with Eclipseand Visual Studio as well as in batch processes.
Runners-up in the category were TestShell from QualiSystemsand Prevent from Coverity.
TestShell, the flagship of tool maker QualiSystems, is a lineof products that includes tools for planning, building, execut-ing and controlling tests and analyzing the results. The heart ofthe suite is TestShell Foundation, the engine and database thatprovide a pluggable infrastructure to grow with the needs of thetesting organization.
Aptly named for its intended purpose is Prevent, Coverity’scode scanning tool for finding programming errors using stat-ic analysis before compilation. It works with C, C++ and Javasource code, and integrates with an existing build environmentand development process.
For testing performance of .NET applications, testers this yearchose Microsoft VISUAL STUDIO TEAM SYSTEM EDITIONSOFTWARE TESTER, moving it up from its second-place fin-ish last year to trade places with HP LoadRunner.
Team System is an assemblage of server- and client-side com-ponents that trickled onto the market since 2005. The primaryserver-side components are Team Foundation Server and SQLServer. The server stores all source code and supports multi-ple simultaneous checkouts, branching, merging, conflict res-olution and tracking of all changes. Security can be applied atany level.
Reporting is handled by SQL Server Reporting Services.Included reports can show code change over time, lists of bugswithout test cases and regressions on previously passing tests.There’s also a build server with tracking. On the client side,Visual Studio enables code analysis, code coverage and testtools for build validation.
Support for .NET languages was added to LoadRunnerbeginning with version 8, which also added a diagnostics serv-er, commander, mediator and probes for Java EE and .NET.Other improvements included a LoadRunner 8 that was morescalable, and easier to install, configure and use, accordingto the company.
LoadRunner permits hundreds of simulated users tobe launched from one or several machines with theLoadRunner agent installed. These load generators, or“injectors,” are instructed on how and when to launchscripts using LoadRunner scenarios so that simulationscan be ramped up.
The .NET version of Compuware DevPartner Studioalso was a runner-up in this category. The company alsooffers versions for C/C++ and Java.
DevPartner analyzes quality and complexity of code andcan detect memory leaks, timing issues and coverage. The soft-ware also can help with memory optimization, CPU, disk andnetwork resource management, fault simulation, and errordetection.
LOADRUNNER is the testers choice in the Java test and perform-ance category for the third year running. During load tests, thetool uses monitors to gauge the performance of the applicationsor components under test. Available monitors include those forvirtual users, transaction rate, network latency, Web and data-base server responsiveness (these are server-specific) and for serv-er resources.
Monitor data is saved for examination by the analysis tool, whichprocesses and graphs the completed scenario results. By analyz-ing these graphs and combining them into reports, testers gainan understanding of the big performance picture and can make
or suggest adjustments to tune performance. JUnit and the Eclipse Test and Performance Tools
Platform also received high marks. Often heralded as the grandfather of unit testing frame-
works is JUnit, one of the earliest and most successful of itskind. Celebrating its 10th birthday in 2007, JUnit has spawnedports to numerous languages including C# (NUnit), C++(CPPUnit), Fortran (fUnit), Perl (Test::Class and Test::Unit),
PHP (PHPUnit) and Python (PyUnit). There’s even a versionfor JavaScript (JSUnit). JUnit has been downloaded more than 2million times in 10 years, according to one study, and is includedas a plug-in to all major IDEs, including Eclipse.
JUnit 4.5 was released in August and simplifies JUnit extensionswith new public extension points for inserting behavior into thestandard JUnit 4 class runner. Advances in JUnit 4.4 included anew API with an “extensible and readable syntax,” according tothe release notes. The API enables features like assumptions orthe declaration of explicit dependencies when a tester has no con-trol of forces that might cause a test to fail. Assumptions also giverise to theories, which can capture some aspect of behavior in “pos-sibly infinite numbers of potential scenarios,” reads the notes.
When it was updated to version 4.5.1 in September, the EclipseTest and Performance Tools Platform (TPTP) workbench andAgent Controller received Java 1.6 support, numerous bug fixesand increased test coverage through test creation, automation andexpanded run-time execution. The update includes numerousenhancements over 4.5, its Ganymede release in June.
While awaiting Visual Studio Team System 2008, testers in2008 told us they are quite fond of VISUAL STUDIO TEAMEDITION SOFTWARE TESTER, this year’s winner in theIntegrated Test and Performance Suite category. Meanwhile,much is already known of Rosario, code name for Visual StudioTeam System 2010.
Microsoft reportedly plans to build more application life-
.NET Test/Performance
JavaTest/Performance
Integrated Test/Performance Suite
20 • Software Test & Performance NOVEMBER 2008
cycle management capabilities into the tool, will integrate it withits Office Project Server, and add full traceability across projectsand resources as well as metrics and dashboards.
Testers last year chose HP Performance Center as theirfavorite integrated test/performance suite; this year it was run-ner-up. The tool combines all the capabilities of LoadRunnerwith test-asset and human-resource management features andreporting in a centralized repository accessible through a brows-er with role-based security. Load-test configurations and testscripts can be created, edited and scheduled for execution.Progress can be tracked, graphed and compared with data frompast projects or releases. Performance Center also can rebootand deploy patches to remote nodes.
Also a runner-up was TestComplete from AutomatedQA.TestComplete is a record and playback tool that automates UIfunctional testing of Flash, Java, .NET, Web and Windows apps. Italso can handle database and back-end HTTP load testing.Recorded tests can be modified and reused for regression testing.
The prime choice for sub-$500 commercial testing solutions wasSOFTWARE PLANNER, making 2008 a peak in the roller coasterride that was the past three years for Pragmatic. The SaaS toolearned top marks this year, was a runner up in 2007 and took thetop spot in 2006. Software Planner is a Web-based hierarchicalproject planner that uses task linking to prevent one task frombeginning before another is completed.
Project managers can use the system to set baseline dates, costsand track deviations as the project proceeds. Team members arealerted to new tasks via e-mail; multi-level security limits access toauthorized people. Billed as an application life-cycle managementtool, it helps IT departments manage requirements, project deliv-erables, test cases, defects and trouble tickets. The tool inte-grates a calendar, dashboards, document collaborationand sharing, a knowledge base, and threaded discus-sions. Software Planner also is available as a self-hostedsolution.
Runners-up were Mindreef SOAPscope Developer andShunra Virtual Enterprise Desktop Standard.
Mindreef has been earning awards almost since comingonto the scene in 2002. SOAPscope 5.0, a tool for testingSOAP-based apps was awarded best solution from a new compa-ny in 2006. SOAPscope Developer allows teams to create, test, deliv-er, support Web services and SOA components, and automateXML-oriented tasks. The tool is included with SOAPscope Server6.1, a server-based solution intended for use and collaboration byall members of the SOA and Web services team, including analystsand managers.
Shunra Virtual Enterprise creates a virtual network for use asa simulation environment for application testing. It permits testersto determine application performance and user experience undera variety of network conditions and “impairments,” such as highnetwork load or traffic, excessive collisions, broadcast storms, andother types of real-world production scenarios. Test results are dis-played graphically, and testers can drill into particular results forissue specifics and analysis.
Of the free testing and performance tools available today, testersthis year chose the TEST AND PERFORMANCE TOOLS PLATFORMproject of Eclipse, bouncing Bugzilla, last year’s winner, into a run-ner-up. TPTP is a test and performance life-cycle tool, address-ing a project’s needs from early testing to production applica-tion monitoring, and everything in between. Test editing and exe-cution, monitoring, tracing and profiling, and log analysis capa-bilities all are core parts of this tool, which supports embeddedsystems, standalone applications, enterprise-grade apps as well ashigh-performance computing systems.
One of Eclipse’s earliest and most successful projects is stillTPTP, and it has enjoyed tremendous support from the commu-nity. Now at version 4.5.1, its subprojects include those for build-ing tools for monitoring and logging performance, and resourceallocation of application servers; for building testing tools that edit,manage, deploy and execute tests; and for application trace andprofiling.
Runners up were JUnit and Bugzilla. It’s safe to say you’vesucceeded when your name is widely used as a verb. For instance:“Google that term and see what you get,” or “Be sure to JUnit yourcode before checking it in.” JUnit has become synonymous withunit testing and has been built into Eclipse for years now. Its suc-cess has even exceeded the expectations of co-creator Kent Beck.
Bugzilla, the open-source defect tracking system, celebratedversion 3.0 last year, nine years after v2. Chief among the enhance-ments are huge performance gains, a Web services interface,and the ability to control and customize far more of the environ-ment than in prior versions, according to project developers.
VMLogix LABMANAGER 3.5took this year’s top spot for Best Solution from
a New Player, which we define as a company less than five yearsold. Released in May, version 3.5 added support for Citrix XenServer4.1 to its existing support for virtualization systems from Microsoftand VMware. LabManager 3.5 also includes some major user inter-face enhancements and to virtual network management.
Last year’s winner in this category was Defender from FortifySoftware. This year the company reached runner-up with anoth-er new product: Fortify 360, which takes aim at the software devel-opment life cycle with the ability to identify, prioritize and repairvulnerabilities during every phase of development.
Requirements management company Blueprint grabbed a run-ner-up slot with Requirements Center, which brings cloud-basedcollaboration to application users and business analysts. As thecompany puts it, the solution puts these people at the heart of thedevelopment process, “allowing them to collaboratively define,visualize and validate their requirements in business terms, allbefore a single line of code has been developed.” ý
Commercial Test/Performance under $500/seat
Free Test/Performance
Best Solution From A New Player
“STP Con has produced a conference that is jampacked with ‘mustknow’ topics by thebest presenters intheir field.”
“Great material, greatspeakers, all around agreat experience.”
“The courses wereexcellent and veryinformative. The otherattendees were a vastsource of knowledgeregarding techniquesand experiences.”
“If you don’t learnsomething you are notpaying attention!”
SPRING
San Mateo MarriottSan Mateo, CA
March 31-April 2, 2009www.stpcon.com
The biggest conference ever:more classes!more speakers!more fun!
22 • Software Test & Performance
Breaking Down LargeSystems to ComponentParts Turns ComplexTesting Into Child’s Play
includes all sub-systems? Do you know howto distinguish between them? You will bythe time you finish reading this.
You know, the goal of performancetesting is not to find bugs, it is importantto guarantee the system is able to supportspecified load, as well as properly recoverfrom the excess use of resources. By per-formance testing, we try todetermine a system’s per-formance, validate the qual-ity attributes (such reliabili-ty, stability) of the system,expose the bottle neck, andmeasure the peak of loads.
Many of today’s systemscan be considered huge—such as those of Amazonand eBay—e-commerceWebsites that service thou-sands of users simultaneous-ly all over the world. Howdo administrators of thosesites ensure high efficiencyand stability? Of course,that’s done with perform-ance testing. But do youknow how to define thetesting scenario, whichfunctions should be includ-ed, the approach and steps,the areas of coverage, etc.?Do you know how to touch all systemcomponents and exercise its full-scaleperformance? What follows is my methodfor doing that.
Before you do any testing, please con-sider some basic principles of perform-ance testing:
• Ensure the test environment is stableand that test conditions are as similar
to the production environment aspossible.
• Try to simulate the random nature ofthe workload.
• Estimate the workload’s growth everyyear and factor that into your tests.
• Ensure the version of software wasdeployed and tested is correct.
• During a performance test, ramp upthe system workload gradually, withadequate time between steps to allowfor the system to stabilize.
• Always monitor and record systemsstats, including CPU memory utiliza-
tion of clients and all serversused by the AUT.
In my practice, full-scaleperformance test consists ofthe Response Time Test, theLoad Test, the Stress Test, theStability Test and the DatabaseVolume Test. Many times peo-ple use these terms inter-changeably. But I don’t agreethat they are interchange-able. A few definitions:
System Performance: Thedegree to which a system orcomponent accomplishes itsdesignated functions withingiven constraints regardingprocessing time and through-put rate.
Performance Test: A test todetermine how fast someaspect of a system performsunder a particular work-load; the validation and ver-
ification of other quality attributes of thesystem, such as scalability, reliability andresource usage.
Response Time Test: The basic perform-ance test and one of most concern by end-user. The response time test evaluates thecompliance of a system or componentwith specified performance require-ments. It usually measures the end-to-endtiming of critical business processesand/or transactions. I sometimes refer toit as the end-to-end response time test, ore2e time test, and often assign benchmark
A ten-year software developer and QA profes-sional, Keven Lui is test leader at Hong KongAir Cargo Terminals, Ltd.
By Keven Lui
M ost experienced testers arefamiliar with perform-
ance testing. But have you evertested full-scale performance that
www.stpmag.com • 23
•For stability,
simulate work,
estimate growth,
validate versions
and ramp up
performance.
•
24 • Software Test & Performance NOVEMBER 2008
values. For example, the home page ofyour Website should display completelyin less than 5 seconds.
Load Test: You can call this perform-ance testing with loads. It tests anapplication under normal loads, suchas testing of a Web site under a rangeof actual loads to determine theresponse times for various criticaltransactions or business processes todetermine whether it meets a require-ment (or Service Level Agreements -SLA). It also tests the capability of theapplication to function correctly underload by measuring transactionpass/fail/error rates.
Stress Test: This evaluates a system atthe specified limitation of load to deter-mine the load under which it fails andhow. It is important to know in advanceif a ‘stress’ situation will result in a cata-strophic system failure, or even “slowdown.”
Stability Test: Tests a system at differ-ent levels of loads for prolonged peri-
ods of time. It simulates normal execu-tion and multiplies transactions by sev-eral times in one day as would beexpected in a single day, week or otherperiod. It is intended to identify per-formance problems that might appear
after several large numbers of transac-tions have been executed, and is espe-cially useful for servers intended tooperate 24/7.
Database Volume Test: The databasevolume test is designed specifically forapplications that access huge stores ofdata. It’s important to know how theapplication will operate not only whenit’s new, but also as the databasebecomes populated and nears orexceeds its maximum capacity.Problems over time might includedegraded response times for searching
and report generation, added time orresource requirements for backup andrestore functions, and an increase intransaction errors.
Once you execute all these tests withpositive results, you can be confident in
your system’s performance and stability.
Activity 1: Response Time TestingThis is the narrow definition of per-formance testing, and it focuses ondetermining the application client’sresponse time.
Put simply, all you need to do islaunch the application and capture theresponse times for each function andloading of all UI pages from a clientmachine. This testing should also berepeatable and consistent. If you re-exe-cute several times, you should get thesame or similar response times for eachfunction. Of course, automation toolscan simplify this job, and should beused if available.
Traditionally, response time isdefined as the interval between when auser initiates a request to the server andwhen the first part of the response isreceived by the application. However,there’s a sometimes a difference fromthe customer’s perspective, and longdelays should prompt a message. Inpractice, you can define the responsetime of a Web application system includ-ing the processing time on the clientmachine and the time taken to down-load and display all contents. Table 1displays time specifications for typicalWeb pages to display.
When defining response times, youwill find that some are unacceptableaccording the SLA. Generally, as the Websystem, in a single screen the time whichperforms the business logic transactionshould be between five and 10 seconds,UI showing time limits of five to eightseconds. If you are the customer, willyou want to wait for the next page formore than 15 seconds? Certainly thereare some principles that will help you tomake the decision of which componentsor functions should be tuned:
• Is it critical to the business?• Is there heavy throughput or func-
tions on a the page?• Is this a function or page with a
large percentage of hits?• Is the function related to the data-
base?• Is this a most complex function?• Is this a page with a large picture or
resource?Also, the application should meet
the SLA or otherwise satisfy the cus-tomer. This type of testing will help youdetermine whether clients will be satis-fied or if functions should be tuned bythe developer.
PERFORMANCE TUNING
TABLE 1: LOAD TIMESFunction Name
HomeAboutContact UsNewsAvailable PositionProducts & ServicesOnline Support
Min8.3430.4310.5700.7210.6560.5270.566
Max 14.422
1.8390.7996.0021.9269.4403.849
Response Time(s)
TABLE 2: KNOW WHAT TO TEST
CCaannddiiddaattee FFuunnccttiioonnCritical Business Function
High Frequency Function
Query Function
Create or Update Function
Heavy Throughput Function (Optional)
CCoommmmeennttThe most important functions to the business; those
that will impact revenue or profit.Example: in a ticket sales system, the critical func-tions might be pricing and discounts, revenue col-lection, statistical analysis and reporting.
The function or UI that is most frequently used.Example: in a travel Website, the most frequentlyused UI/functions might include airline, airport, cal-endar, flight schedules, resorts near destination andprices.
The query function is the basic function of the appli-cation. At least one query should be involved in theselected list.Example: in a human resources Website, “SearchJobs” and “Search Candidates” are the basic queryfunctions.
At least one create or update function should beincluded, and should verify correct behavior in cas-es an update attempt is made when the databasetable (or record) is locked.
Example: in a blog site,“Register” might be the firststep before submitting a comment is allowed.If the testing will be impacted by the network orbandwidth, the Heavy Throughput Function shouldbe included.Example: Streaming or downloading a video.
NOVEMBER 2008 www.stpmag.com • 25
Activity 2: Load TestingSimilar to Activity 1, load testing will cap-ture a series of response times underload (sometimes called backgroundloading or background noise). These arevarious load levels below the daily maxi-mum load expectation to determine ifthe application still functions properly.
You will find there is no differencebetween performance testingand load testing except thelevel of background loading.Although the methods are sim-ilar, their goals are quite differ-ent. At this testing, you will val-idate that response times and transaction pass/fail/errorrates under a range of loadsmeets the SLA. Usually, thetransactions per second (TPS),HTTP connections, concurrentuser or throughput measure-ments are used to represent forthe load level. For example,one useful test might be donewith a workload of 100 concur-rent users.
From this testing, you will determinethe performance thresholds of the appli-cation and server resources such asmemory, CPU utilization and disk usage.
Activity 1 performance testing gaveyou the response times of every func-tion. In Activity 2, we load-test a nucle-us of functions based on the judgmentsin Table 2.
You can separate the backgroundloading into three levels of the maxi-mum load expectation: none or lowerthan 1 percent, 50 percent and 100 per-cent. As in performance testing, startthe background load first and measurethe response times after the backgroundload stabilizes. For efficiency, I suggestcovering all functions, especially trivialones. You can perform the testing of theten most important functions in order.
In this testing, the same importantthing should be done, that is checkingthe transaction pass/fail/error rates.Load test tools such as HP MercuryLoadRunner, IBM Rational Perfor-mance Tester are useful here to simu-late multiple users accessing the appli-cation concurrently. I have found that asetting of 60-120 minutes for runtimeduration is good enough to ensure astable simulation environment.
As for exit criteria for this testing,the transaction failure rates should belower than 3 percent for most systems.For financial systems or others in which
transactions are mission-critical, 0.01percent might be closer to optimal.
Activity 3: Stress TestingFollowing Activity 2, stress testing initi-ates the maximum expected systemload, then raises it in an attempt toreach the system’s absolute limit.Typically, stress testing starts with load
testing, and then additional users oractivities are gradually increased untilthe application server “slows down” orbegins to work abnormally. For data-base applications, you can try morecomplex database queries, continuous
system input or heavy database load.With stress testing, we want to see
what happens when the “acceptable”load is exceeded. Does the system crash?How long does it take to recover oncethe load is reduced? Does it fail in a waythat causes loss of data or other damage?In particular, the goals of such tests maybe to ensure the software doesn’t crash
in conditions of insufficientcomputational resources (suchas memory or disk space),unusually high concurrency, ordenial of service attacks.
When the system’s load limi-tation is reached, the systemperformance will usuallydegrade, and the system re-sponds more and more slowly.But when the excessive load isremoved, the performanceshould return to normal. Sowhen you execute the stress test-ing with initial loads, you canadd additional users or activitieswith gradual ramp-up (e.g. add
1 more user after 5 minutes until thesystem’s load stable), at the same time,monitor the CPU and memory usage inthe application server and databaseserver. It’s also useful to monitor per-formance on the test clients and all
PERFORMANCE TUNING
TABLE 3: SERVER PERFORMANCE COUNTERS
OObbjjeeccttProcessor
Memory
PhysicalDisk
MMeemmooIf the server’s processor utilizationstays high for extended periods, itmight indicate a bottleneck and thatmore CPU power is needed.
The average number of processeswaiting for execution, in general,should be less than the number ofCPU cores. For a system with a sin-gle CPU core, a load average lessthan two is optimal; three is accept-able; more than five is sub-optimal.
The amount of physical memoryavailable for use by the server. If theserver does not have enough mem-ory to handle its workload, it willspend more time reading from disk.Sometimes, Page Faults/sec couldindicate a memory shortage.
Shows the percentage of elapsed timethat the disk is busy with read/writeactivity. If this counter is high andthe processor and network bandwidthare not saturated, then there is like-ly a disk bottleneck. A number con-sistently above 80% may indicate amemory leak.
IInnddiiccaattoorr% TotalProcessor Time
Load Average
Available bytes(usually repre-sented as a per-centage of phys-ical memory)
% Disk time
RReeffeerreennccee< 75%
<3 / per CPU core
> 10% physicalmemory
< 80%
•When excessive load
is removed, performance
should return to normal.
•
other Web servers and applicationservers touched by the application.
Performance counter data is usedto determine when a test client, Webserver or application server hasreached its maximum CPU or mem-ory use. These performance coun-ters will be the easiest way to deter-mine where the bottleneck is occur-ring. For example, the total proces-sor time is the best counter for view-ing processor saturation. A numberconsistently above 90 percent onone or more processors indicatesthat the test is too intense for thehardware.
Generally, there are four levels bot-tleneck: application level, databaselevel, operation system level and net-work level. Table 3 shows some typicalcounters used in server performancetests, and can be helpful when trying tofind less obvious causes of performanceproblems. Table 4 contains a stress-testresults from a Web server test.
Activity 4: Stability TestingAfter Activities one to three, you will testthe permanency of the system, that is sta-bility testing. In this testing, the systemwill be running at various loads for pro-
longed periods of time.Why should we do this testing? For
real-time and embedded systems, thiscan help ensure that systems will runnormally through many days andnights. Simply put, this test simulatesmore and varied workloads across awhole day or longer period than wouldbe expected in a working day. Stabilitytesting can identify problems that onlyoccur after the system has been running
a large number of transactions overa long period of time.
The memory leak is generally themost common issue. Gradual degra-dation of response time of somefunctions and reduction in efficien-cy, serious memory leaks will almostalways eventually result in a largercrisis or failure. This testing pro-vides an opportunity to identifydefects that load and stress tests maynot due to their relatively shortduration.
A stability testing should run for aslong as possible, so the weekend isoften an opportune time for this test.
Of course, the stability test exe-cutes one of the performance tests,but it is also important to measureCPU and physical memory utiliza-
tion, and system response times. If aparticular server process needs to beavailable for the application to operate,it is necessary to record the server’smemory usage at the start and the end,as well as the client’s (or JVM’s) memo-ry usage.
Activity 5: Database Volume Test:If you’re deploying a database applica-tion, it’s recommended that you add the
26 • Software Test & Performance NOVEMBER 2008
PERFORMANCE TUNING
TABLE 4: STRESSFUL RESULTSObject Processor
PhysicalDisk
Memory
Reference<75%
<3 / perCPU core
<80%
> 10%PhysicalMemory
Pass
Pass
Pass
Pass
Max:86%, 6sMin:76%, 3s
Max:1.324Min:0.338
34%
Max:1024113, 25%Min:614408,15%
Indicator%TotalProcessorTime
LoadAverage
%DiskTime
AvailableBytes
Actual Result
Searching for the next step in your career? A new job? Just want a peek at what’s out there?
BZ Media
O v e r2 , 8 9 5
J o b P o s t i n g s !
P o s t Y o u r R e s u m e T O D A Y !
Visit SDTimes.com to Find a Job Near You!www.sdtimes.com/jobboard
The SD Times Job Boardinstantly connects you to employers looking for someone just like you!
The SD Times Job Board offers services,resources and a networking community for software developmentmanagers and professionals. In our NEWonline Career Center, you’ll find a hugebank of job postings and resumes, careerresources, coaching and advice services,and so much more!
Post Your Resume AnonymouslyYou can keep your identity secret, sending only your contact information to the employers whose opportunities you are interested in!
Search Over 2,895 Software Development Job PostingsFind a software development, testing,engineering or programming positiontoday!
Access a Library of Career-Related Articlesand ResourcesWe offer Career Coaching, Ask the Job Search Expert, and an entire libraryof career-advancement articles.
NOVEMBER 2008 www.stpmag.com • 27
database volume test to your required list.It’s important to know how the databasewill behave when it’s loaded to its maxi-mum with data, or even beyond. This testwill let you know of problems with datacorruption, degraded response time forsearching or generating reports, slowbackup and restore processing or anincreased number of transaction errors.
This testing is similar to the responsetime test (Activity 1). The idea is torecord the response times when the sys-tem is running with and without maxi-mum amounts of data. The volume testmight also reveal problems of functionsrunning abnormally, slow or noresponse, no disk space for writing thelog file, no space in databases or datafiles or complex searches with sortingthrough many tables that lead to over-flow, to name just a few.
Table 5 contains some results of atest with various amounts of test data.Of course, when you test a huge data-base system, you should try to generaterealistic data in all tables. For exam-ple, if the “customers” table involvesover million records but 999,000 ofthose are identical, you will get unreal-istic results when searching for a cus-tomer name.
It’s helpful to keep a few things inmind when doing these types of tests:Concerns (or complaints) aboutresponse time and application stabilitytypically come from end-users; inspec-tion of the hardware system is whereyou’ll find network throughput andresource utilization bottlenecks; sourcecode and database optimization are thedeveloper’s domain. But the tester hasresponsibility for all these areas. And afull-scale performance test is the onlyway to ensure that the whole system ishealthy and will perform well over time.Though the test activities explainedhere each have quite different mean-ings and scopes, there is no clear
boundary between them. They must betaken together to properly evaluate anentire system. ý
REFERENCES• Performance vs. load vs. stress testing http://
agiletesting.blogspot.com/2005/02/performance-vs-load-vs-stress-testing.html. Grig Georghiu,February 28, 2005
• Patterns & Practices Performance TestingGuidance for Web Applications by J.D. Meier, ScottBarber, Carlos Farre, Prashant Bansode, DennisRea
• Software performance testing http://en.wikipedia.org/wiki/Software_performance_testing
• Performance Testing for Ajax-based ApplicationsAztecsoft Limited, May 14, 2008
• General Functionality and Stability Test ProcedureJames Bach
• Performance Testing Considerations. http://www.visibleprogress.com/software_performance_testing.htm
TABLE 5: VARIED RESULTS
IItteemmDatabase Volume Setting
Database Volume Utilization
Table Record Count (CARGO)
Table Record Count (CONTAINER)
Table Record Count (CUSTOMER)
Response Time Result
CCaassee 3320 GB
3%
10,873
7,769
1,265
1.837
4.062
3.765
2.125
CCaassee 11
20 GB
99.5%
1,200,920
1,504,842
40,981
2.282
21.102
5.473
15.023
CCaassee 2220 GB
42%
450,760
600,765
9,864
2.165
6.125
5.314
11.904
Add Cargo Record
Search Cargo Record
Assign Cargo inContainer
Search Customer
PERFORMANCE TUNING
Without oversight, software projects can creepout of control and cause teams to freak. But withSoftware Planner, projects stay on course.Track project plans, requirements, test cases,and d efects via the web. Share documents, holddiscussions, and sync with MS Outlook®. VisitSoftwarePlanner.com for a 2-week trial.
lead. The manager says that the project will use an automatedtest tool and that due to this planned automation, the testeffort is not expected to be significant. The project managerconcludes the meeting with a request that you submit withinthe next week a recommendation of the specific test toolrequired, together with a cost estimate for the procurement ofthe tool.
Huh? You’re caught totally by surprise. What in the worldjust happened? You sit alone in the meeting room, wonderingwhere the project manager got his expectations with regard toautomated testing. You decide they must have been developedafter reading a vendor brochure.
A scenario similar to this is one that I have actually experi-enced. If this ever happens to you, my suggestion would be toclear up and correct these misconceptions immediately. Theidea of automated testing often brings high expectations, anda lot is demanded from the technology. But some people havethe notion that an automated test tool does everything fromtest planning through execution, without manual interventionof any kind. And while it would be great if such a tool existed,that is pure fantasy; and no such tool exists.
In fact, automated test tools in most cases do not initially
reduce the test effort nor the test schedule. While it has beenproven that automation is valuable and can produce a returnon investment, there isn’t always an immediate payback.Such misconceptions of test automation are manyand persistent in the software industry. Here aresome of the more common ones along with guid-ance on how to manage.
Automatic Test Tools Do Everything Currently, there is no commercially avail-able tool that can create a comprehensivetest plan while also supporting test designand execution.
Throughout a software test career,the test engineer can expect to witnesstest tool demonstrations and review anabundant amount of test tool literature.Often the test engineer will be asked tostand before a senior manager or anumber of managers to give a test toolfunctionality overview. As always thepresenter must bear in mind the audi-ence. In this case, the audience mayrepresent individuals with justenough technical knowledge tomake them enthusiastic aboutautomated testing, while notenough to be aware of thecomplexity involved with an
Elfriede Dustin is currently employed by Innovative Defense Tech-nologies (IDT), a software testing consulting company specializing inautomated testing.
28 • Software Test & Performance
By Elfriede Dustin
W hile at a new project kickoff meeting, theproject manager introduces you as the test
automated test effort. Specifically, the man-agers may have obtained information aboutautomated test tools third hand, and may havereached the wrong interpretation of the actu-al capability of automated test tools.
What the audience at the managementpresentation may be waiting to hear is that thetool that you are proposing automaticallydevelops the test plan, designs and createsyour test procedures, executes all the test pro-cedures and analyzes the results for you. Youmeanwhile start out the presentation byinforming the group that automated test toolsshould be viewed as enhancements to manualtesting, and that automated test tools will notautomatically develop the test plan, designand create the test procedures and executethe test procedures.
Soon into the presentation and after sev-eral management questions, it becomesapparent just how much of a divide existsbetween the reality of the test tool capabili-ties and the perceptions of the individuals inthe audience. The term automated test toolseems to bring with it a great deal of wishfulthinking that is not closely aligned with real-ity. An automated test tool will not replacethe human factor necessary for testing aproduct. The proficiencies, analytical capa-bilities and subject matter expertise of testengineers and other quality assuranceexperts will still be needed to keep the test-ing machinery running. A test tool can be viewed as an additional part of themachinery that supports the release of agood product.
One Tool Fits All Currently not one single test tool exists that can beused to support all operating system environments.
Generally, a single test tool will not fulfill allthe testing requirements for an organization,nor is it likely to run on all its platforms.Consider the experience of one test engineerencountering such a situation. The test engi-neer, Dave, was asked by a manager to find atest tool that could be used to automate alltheir real-time embedded system tests. Thedepartment was using various operating sys-tems, including VxWorks and Integrity, plusmainframe, Linux and Windows XP.Programming languages included Java andC++, and various n-tier server and Web tech-nologies also were involved.
Expectations had to be managed. It had tobe made clear that there did not exist any sin-gle tool that was compatible with all of thosevariables. Granted more vendors have provid-ed cross-platform compatibility since then, butthe capabilities of such tools might not meetyour needs and often more than one tool andfeature is required to test the various plat-forms and technologies.
Test Efforts are Immediately ReducedIntroduction of automated test tools will not imme-diately reduce the test effort, and will usuallyincrease them at first.
A primary impetus for introducing an auto-mated test tool onto a project is to reduce thetest effort. Experience has shown that a learn-ing curve is associated with the attempts toapply automated testing to a new project and
Part II: Myths, Realities AndHow to Know the Difference
www.stpmag.com • 29
30 • Software Test & Performance NOVEMBER 2008
to the effective use of automated test-ing. Test effort savings do not alwayscome immediately. Still, test or projectmanagers have read the test tool litera-ture and are anxious to realize thepotential of the automated tools.
There is a good chance that the testeffort will actually increase initiallywhen an automated test tool is firstbrought into an organization. Whenintroducing an automated test tool to anew project, a whole new level of com-plexity is being added to the test pro-gram. And while there may be a learn-ing curve for the test engineers tobecome smart and efficient in the use ofthe tool, there are still manual tests tobe performed on the project as well.The reasons why an entire test effortgenerally cannot be automated are out-lined later in this article.
Initial introduction of automatedtesting also requires careful analysis ofthe application-under-test in order todetermine which sections of the appli-cation can be automated. Test automa-tion also requires careful attention toautomated test procedure design anddevelopment. The automated test effortcan be viewed as a mini-development
lifecycle, complete with the planningand coordination issues that comealong with a development effort. (Tounderstand what a team has to learnbefore an automated test tool can beeffective, don’t miss part III of this arti-cle series.)
Schedules are ImmediatelyCompressedAn automated test tool will not immediatelyminimize the testing schedule, and will usu-ally lengthen it at first.
Another misconception is that auto-mated test tools will immediately mini-mize the test schedule. Since the testingeffort actually increases initially, thetesting schedule will not experience theanticipated decrease at first, and anallowance for schedule increase shouldbe built in whenever automation isintroduced. This is because whenrolling out an automated test tool, anentirely new testing process has to bedeveloped and implemented. Theentire test team and possibly the devel-opment team also needs to learn newskills and become familiar with (and fol-low) this new automated testingprocess. Once an automated testing
process has been established and effec-tively implemented, the project canexpect gains in productivity and turn-around time that can have a positiveeffect on schedule and cost.
Automation Tools are Easy to UseAn automated tool requires new skills, andadditional training is usually required.Plan for costs and time of training and alearning curve.
Many tool makers market their toolsby exaggerating the ease of use of thetool. They deny any learning curve thatis associated with the use of the new tool.The vendors are quick to point out thatthe tool can simply capture (record) testengineer keystrokes and (like magic) ascript is created in the background,which can then simply be reused forplayback and edited as necessary.
Efficient automation is not that sim-ple. The test scripts generated duringrecording need to be modified manual-ly to make them reusable and to allowplayback under various applicationstates and with variable data. Thisrequired language and scripting knowl-edge. Scripts also need to be modifiedas to be maintainable. For a test engi-neer to be able to modify the scripts,they need to be trained on the tool andits built-in scripting language. Newtraining requirements and a learningcurve can be expected with the use ofany new tool.
All Tests Can be AutomatedNot all tests can be automated.
Automated testing is an enhance-ment to manual testing. It is unreason-able to expect that 100 percent of thetests on a project can be automated. Forexample, when an automated GUI testtool is first introduced, it is beneficial toconduct some compatibility tests on theAUT to determine whether the tool willbe able to recognize all objects andthird party controls.
The performance of compatibilitytests is especially important for GUI testtools, because such tools have difficultyrecognizing some custom controls fea-tures within the application. Theseinclude the little calendars or spin con-trols that are incorporated into manyapplications, especially in Windowsapplications. These controls or widgetsare often written by third parties, andmost test tool manufacturers can’t keepup with the hundreds of clever controlschurned out by various companies.
AUTOMATION FACT AND FICTION
TABLE 1:TO AUTOMATE OR NOT TO AUTOMATE?
Number of test executions:• Is test executed more than once?
• Is test run on a regular basis, i.e. often reused, such as partof regression or build testing?
Criticality of test:• Does test cover most critical feature paths?
• Does test cover most complex area (often most error-pronearea)?
Cost of test:• Is test impossible or prohibitively expensive to perform man-
ually, such as concurrency, soak/endurance testing, perform-ance and memory leak detection testing?
• Is test very time consuming, such as expected results analy-sis of hundreds or thousands of outputs?
Type of test:• Does test require many data combinations using the same test
steps (i.e. multiple data inputs for same feature)?
• Does test need to be verified on multiple software and hard-ware configurations?
• Are the expected results constant, i.e. expected results do notchange with each test and don’t vary?
Application/System under Test:• Is test run on a stable application, i.e. the features of the
application are not in constant flux?
• Uses compatible Technology and open architecture
Yes No
NOVEMBER 2008 www.stpmag.com • 31
AUTOMATION FACT AND FICTION
It might be that the test tool is com-patible with all releases of C++ and Java,for example, but if an incompatible thirdparty custom control is introduced intothe application, the results will be thatthe tool might not recognize the objecton the screen. It might be the case thatmost of the application uses a third partygrid that the test tool does not recognize.The test engineer will have to decidewhether to automate this part of theapplication by defining a custom objectwithin the automation tool, finding awork around solution or whether to testthis control manually only.
There are other tests that are physi-cally impossible to automate, such asverifying a printout. The test engineercan automatically send a document tothe printer, but then has to verify theresults by physically walking over to theprinter to make sure the document real-ly printed. The printer could have beenoff line or was out of paper.
Neither can a test tool automate 100percent of the test requirements of anygiven test effort. Given the endless num-ber of permutations and combinationsof system and user actions possible with n-tier (client/middle-layer/server)architecture and GUI-based applica-tions, a test engineer or team does nothave enough time to test every possibili-ty – manually or otherwise.
It also should be noted that no testteam can possibly have enough time orresources to support 100 percent testautomation of an entire application. Itis not possible to test all inputs or allcombinations and permutations of allinputs. It is impossible to test exhaus-tively all paths of even a moderate sys-tem. As a result, it is not feasible toapproach the test effort for the entireAUT with the goal of testing 100 per-cent of the application.
Another limiting factor is cost. Sometests are more expensive to automatethan others, and it might make sensesimply to test those manually. A test thatis only executed once per developmentcycle is often not worth automating. Forexample, an end of year report of ahealth claim system might only be runonce, because of all the setup activityinvolved to generate this report. Sincethis report is executed rarely, it’s morecost effective not to automate it. Whendeciding which test procedures to auto-mate, a test engineer needs to evaluatethe cost-effectiveness of developing anautomated script.
When performing this cost analysis,the test engineer will need to weed outredundant tests. The goal for test pro-cedure coverage using automated test-ing is for each single test to exercisemultiple items, while avoiding duplica-tion of tests. For each test, an evaluationshould be performed to ascertain thevalue of automating the test.
As a starting point, Table 1 providesa list of questions to ask when decidingwhether or not to automate a test. If theanswers are “Yes” to most of the ques-tions, chances are the test lends itself toautomation.
Automation Provides 100 Percent Test CoverageEven with automation, not everything can betested. It is impossible to perform a 100 per-cent test of all the possible simple inputs to asystem. It is also impossible to exhaustivelytest every combination and path of a system.
One of the major reasons why testinghas the potential to be aninfinite task is that inorder to know that thereare no problems with afunction, it must be testedwith all possible data,valid and invalid.Automated testing mayincrease the breadth anddepth of test coverage, yetwith automated testingthere still isn’t enoughtime or resources to per-form a 100 percentexhaustive test.
The sheer volume ofpermutations and combi-nations is simply too stag-gering. Take for examplethe test of a function thathandles the verification ofa user password. Each useron a computer system has apassword, which is general-ly six to eight characterslong, where each characteris an uppercase letter or adigit. Each password must contain at leastone digit. How many possible passwordcharacter combinations do you thinkthere are in this example? According toKenneth H. Rosen in “DiscreteMathematics and its Application” thereare 2,684,483,063,360 possible variationsof passwords.
Even if it were possible to create atest procedure each minute, or sixty testprocedures per hour, equaling 480 test
procedures per day, it would still take155 years to prepare and execute a com-plete test. Therefore, not all possibleinputs could be exercised during a test.With this rapid expansion it would benearly impossible to exercise all inputsand in fact it has been proven to beimpractical.
Let’s take for example the test of thetelephone system in North America.The format of the telephone numbersin the North America is specified by anumbering plan. A telephone numberconsists of ten digits, which are split intoa three-digit area code, a three-digitcentral office code and a four-digit sta-tion code. Because of signaling consid-erations, there are certain restrictionson some of these digits.
To specify the allowable format, let Xdenote a digit that can take any of thevalues of 0 through 9, let N denote adigit that can take any of the values of 2through 9. The formats of these codes
are NXX, NXX, andXXXX, respectively. Howmany different NorthAmerican telephone num-bers do you think are pos-sible under this plan?There are 8 x by 8 x by 10= 640 office codes with for-mat NNX and 8 x by 10 xby 10 = 800 with formatNXX. There are also 10 x10 x 10 x 10 = 10,000 sta-tion codes with formatXXXX. Consequently itfollows that there are 800x 800 x 10,000 =6,400,000,000 differentnumbers available. Thisnumber would only testthe valid numbers andinputs to the system, andhasn’t even touched onthe invalid numbers thatcould be applied. This isanother example, whichshows how it is impracticalbased upon development
costs versus ROI associated with play-back - to test all combinations of inputdata for a system1.
In view of this, random code reviewsof critical modules are often done. It isalso necessary to rely on the testingprocess to discover defects early. Suchtest activities, which include require-ments, design and code walkthroughs,support the process of defect preven-tion. Both defect prevention and detec-
•It is not feasable
to approach the
test effort
with the goal
of testing
100 percent
of the
application.
•
32 • Software Test & Performance NOVEMBER 2008
tion technologies need to be part of aneffective automated testing process.Given the potential magnitude of anytest, the test team needs to rely on testprocedure design techniques, such asboundary, equivalence testing and com-binatorics, where only representativedata samples are used.
Test Automation is the Same AsCapture and PlaybackHitting a record button doesn’t produce aneffective automated script.
Many companies and automatedtesters still equate automated softwaretesting with using capture/playback
tools. Capture/playback in itself is ineffi-cient automated software testing at best,usually creating non-reusable scripts.
Capture/playback tools record thetest engineer’s keystrokes in the tool’sprovided scripting language and allowfor script playback for baseline verifica-tion. Automated test tools mimicactions of the test engineer. During test-ing, the engineer uses the keyboard andmouse to perform some type of test oraction. The testing tool captures all key-strokes or images and subsequentresults, which are baselined in an auto-mated test script. During test playback,scripts compare the latest outputs withprevious baseline. Testing tools oftenprovide built-in, reusable test functions,which can be useful and most test toolsprovide for non-intrusive testing; i.e.,they interact with the AUT as if the testtool was not involved and won’t modi-fy/profile code used in the AUT.
Capture/playback generated scriptsprovide challenges. For example, thecapture/playback record hard-coded val-ues; i.e. if an input called “First Name” is
recorded, that “First Name” will be hard-coded, rendering test scripts usable onlyfor that “First Name.” If variables are tobe added, scripts must be modified. Toread in more than one “First Name,” theability to read data from a file or data-base would have to be added, as wouldthe ability to include conditional state-ments or looping constructs.
Additionally, capture/playback toolsdon’t implement software developmentbest practices right out of the box;scripts need to be modified to be main-tainable and modular. Additionally, ven-dor-provided capture/playback toolsdon’t necessarily provide all testing fea-
tures required; code enhancements areoften required to meet testing needs.Finally, vendor-provided tools are notnecessarily compatible with system engi-neering environment, and software test-ing frameworks need to be developedin-house to enhance some of the exist-ing tools’ capabilities.
Automated Software Testing is AManual Tester ActivityUse of capture/playback tools and scriptrecording does not an automated tester make.
While automated software testingdoes not immediately replace the man-ual testing and analytical skills requiredfor effective and efficient test case devel-opment, automated testing skillsrequired are different skill from manualsoftware testing skills required. Oftenhowever, companies buy into the ven-dor and marketing hype that automatedsoftware testing is simply hitting arecord button to automate a script.That is simply not the case.
It is important to distinguish be-tween the skills required for manual
testing versus those for automatedsoftware testing. In essence, an auto-mated software tester needs softwaredevelopment skills. Different skills arerequired and a manual tester withoutany training or background in soft-ware development will have a difficulttime implementing successful auto-mated testing programs.
Losing Site of the Testing Goal:Finding DefectsOften during automated software test-ing activities, the focus is on creatingthe best automation framework andthe best automation software and welose sight of the testing goal: to finddefects. It’s important that testingtechniques, such as boundary value,risk based testing, equivalence parti-tioning, deriving the best suitable testcases, etc. are applied.
You might have employed the latestand greatest automated software devel-opment techniques and used the bestdevelopers to implement your automa-tion framework. It performs fast withtens of thousands of test case results,and it’s getting rave reviews. No matterhow sophisticated your automated test-ing framework is, if defects slip into pro-duction and the automated testingscripts didn’t catch the defects it wassupposed to, your automated testingeffort will be considered a failure.
The most successful automated test-ing efforts require a mix of tools, frame-works and skills. Capture/playback solu-tions get the biggest hype, but are onlyone piece of the automated testing solu-tion. There is no magic bullet to auto-mated testing and not one tool exists onthe market that can help solve all yourautomated testing vows. Automatedframeworks developed in-house, variousscripting languages, skills and knowledgeof testing techniques, analytical skills anddevelopment experience are all requiredfor automated testing to be successful.
Next TimeIn the next and final part of this series I’lldiscuss additional pitfalls to avoid such aspicking the wrong tool; how to betterintegrate tools, by asking for standardsacross tool vendors; and how an auto-mated software testing process can helpimplement a successful automated test-ing effort. ý
REFERENCES1. Rosen, Kenneth H. (1991) Discrete Mathematics
and Its Application.
AUTOMATION FACT AND FICTION
applications?Network latency is composed of two factors;
the latency that is native to the connection(transmission speed and distance) and latencydue to other traffic on the network. Nativelatency makes the assumption that the entirenetwork and Internet pipe is yours for the ask-ing. The cold reality is that you share this pipewith all the other users in your office and alsoall the other traffic on the Internet. We know
that the Internet is used for many things andthat the latency can vary tremendously duringthe day due to legitimate and not so legitimateactivities.
No matter how fast your application runs andthe speed of your servers, transaction speed islimited by the speed of the network used tomove data between the client and the server.
This article covers my recent experience imple-menting a global Web-based solution and some
NOVEMBER 2008 www.stpmag.com • 33
By Nels Hoenig
Have you considered the impact oflatency on your Web-based internal
Are Latency Issues Holding Back Your App? A Few Simple Tests Will Keep You In Front
Nels Hoenig is a lead software quality engineer for ITconsulting firm Princeton Information.
Pho
togr
aph
by J
ames
Ste
idl
34 • Software Test & Performance NOVEMBER 2008
latency issues found in a remote loca-tion. It also includes DOS code samplesthat can be used to create simple datacollection tools to measure latency at agiven location to determine if a prob-lem exists.
“Your application sucks” was not thebest subject line in myinbox that morning, butbased on the other peo-ple copied, it was appar-ently the first oneopened. The messagearrived just after we hadextended manual testingto users in India, aftertesting using remotelogins to machines inIndia. The remote testingwas OK, but not as fast asin the U.S. In manualremote testing from thislocation, processing timewas longer than in U.S.locations, but the timeshad been acceptable.
So how were we tointerpret this somewhatinflammatory message?
Step 1: Define “Sucks”As we had a sample data-base and the transactionwas Web-based, we hadthe remote users perform manual testsidentical to the tests we had doneusing desktop remote. Based on theanswers that came back the next morn-
ing, suck would seem a valid term. Theusers were reporting transaction timesas much as to 500 percent worse thanour testing results using remote desk-top to run the same test on machineslocated in the same location. Problemconfirmed.
Step 2: InvestigationWe checked the server tosee if it was having per-formance issues or if back-ups were running whenthe issue was reported. Noissues; keep looking. Nextcheck the users’ PCs foradequate resources orunexpected constraints.No issues; keep looking.We had machines locatedin India that we couldremote into. So we set thealarm for 3:00 am and test-ed it ourselves. This testingconfirmed the issue, butalso showed that the prob-lem existed only in Indiaand only during the Indiaworking day (Figure 1).
To further understandthe issue, we used an auto-mated testing tool to run asample transaction setfrom India for 11 hours to
get a perspective of performance andhow it was fluctuating. Now we had realproof that this was a problem to solve.We had eliminated the application, the
computers, the user and server as possi-ble bottlenecks. Attention now focusedon the network.
Step 3:TestingNow that we understood the problemwas rooted in the network and that itonly happened during working hours,it was time to take a closer look at thenetwork. We had previously donesome sample trace route testing tomeasure the latency (the number ofhops and time needed to get a packetfrom India to our server), A latency of300 - 350 ms was considered reason-able based on the distance involved.This also approximated (but some-times exceeded) the ISP vendor’sService Level Agreement, which prom-ised latency of less than 330 ms.
Performing a trace route is straight-forward. Simply open a DOS window,type in the command “tracert” fol-lowed by the target URL and hit enter.The standard command will return upto 30 hops and on numbered lineswith the URL, IP address and the time(in milliseconds) needed to completeeach hop. Your results should looksimilar to this:
C:\>tracert stpmag.com
Tracing route to stpmag.com [198.170.245.48]overa maximum of 30 hops:
1 1 ms <1 ms <1 ms 192.168.1.12 8 ms 9 ms 13 ms cpe-75-185-104-
1.insight.res.rr.com [75.185.113.1]3 7 ms 9 ms 9 ms network-065-024-192-
197.insight.rr.com [65.24.192.197]4 10 ms 9 ms 9 ms pos13-2.clmboh1-rtr2.colum-
bus.rr.com [65.24.192.38]5 10 ms 9 ms 9 ms pos13-1.clmboh1-rtr1.colum-
bus.rr.com [24.95.86.165]6 20 ms 18 ms 19 ms tge5-3-0.ncntoh1-
rtr0.mwrtn.rr.com [65.25.137.249]7 118 ms 207 ms 203 ms te-3-
4.car1.Cleveland1.Level3.net [64.156.66.97]8 21 ms 19 ms 19 ms ae-2-
4.bar1.Cleveland1.Level3.net [4.69.132.194]9 25 ms 35 ms 35 ms ae-6-
6.ebr1.Washington1.Level3.net [4.69.136.190]10 40 ms 35 ms 36 ms ae-61-
61.csw1.Washington1.Level3.net [4.69.134.130]11 27 ms 26 ms 28 ms ae-1-
69.edge2.Washington4.Level3.net [4.68.17.19]12 40 ms 27 ms 28 ms 4.68.63.18613 30 ms 28 ms 29 ms ae-
1.r21.asbnva01.us.bb.gin.ntt.net [129.250.2.180]14 33 ms 28 ms 28 ms xe-1-
2.r03.stngva01.us.bb.gin.ntt.net [129.250.2.19]15 30 ms 28 ms 28 ms ge-0-0-
0.r01.stngva01.us.wh.verio.net [129.250.27.218]16 34 ms 28 ms 28 ms ge-
26.a0719.stngva01.us.wh.verio.net [204.2.125.122]17 28 ms 30 ms 26 ms www.stpmag.com
[198.170.245.48]
Trace complete.
Complete Process
Clear Filter
Login
800
700
600
500
400
300
200
100
0
INDIA BUSINESS DAY
0:00
10:00
1:00
11:00
2:00
12:00
3:00
13:00
4:00
14:00
5:00
15:00
6:00
16:00
7:00
17:00
8:00
18:00
9:00
10:00
10:00
20:00
11:00
21:00
FIG. 1:WORKING DAY,THE INDIA WAY
•Automated tools
ran transactions
from India for
11 hours to see
performance
flucuations...
we had proof
of a problem.
•
IT’S THE NETWORK, STUPID
NOVEMBER 2008 www.stpmag.com • 35
Since the performance resultsseemed to vary so much during the day,we decided to run an extended latencytest over a 24-hour period. Some verysimple DOS-command scripts wereused to collect the data and run a trac-ert every 10 minutes. When the resultsare viewed in a raw form, it is hard tosee the pattern, but when the results arechanged to show an hourly average viewof the latency, the issue comes intofocus (see Figure 2).
Step 4: Fix the IssueNow that we understood that the net-work was the source of the perform-ance issue, we worked with the net-work team to make changes.
Efforts to improve this situationapproached several fronts:
• We informed the ISP serving thislocation that we were measuringlatency against the terms of theSLA. The ISP made some im-provements to the network con-nection.
• We increased the bandwidth ofour Internet connection, allowingmore traffic to flow without beingbuffered.
• We upgraded the enforcement ofproxy controls to limit networkusage for non-business relatedactivities.
• We told users that they were con-tributing to the performanceproblem, and asked that they bepart of the solution.
Once these changes were applied,the same test was repeated. The resultsare shown in Figure 3. The changesgreatly reduced the amount of vari-ability in the latency seen by the usersand this gave them an improved expe-rience for all networked applications.
Step 5: Learn Your LessonWhat we learned from this project wasthat application performance dependsnot only on the program, the clientand the server. The network beingused also can be a source of the prob-lem. You should never assume any-thing is fine without conducting a testto confirm. We were happy to find thatthis problem was not with our applica-tion and that resolving it improvedperformance for all the remote usersat our India location.
The ScriptsThis process used three DOS Scripts to
collect the data into a text file, we didhave to do some manual analysis of theresults but it is not that hard. All you arelooking for are hops where the latencyperiod is above your maximum allowed.If you are going to run these tests, pleasecheck with your network/security ad-ministration before hand. This kind oftest could appear like an attempt tohack the server and you don’t want tohave one of “those meetings.”
We used one script to load the dateof the test, one script to load the time ofthe test and finally the tracert script todo the actual test. By testing this way wedidn’t need to use any fancy applica-tions and it could be used on anymachine that has the XP operating sys-tem on it. We used Windows scheduler
to call the scripts.
Date Script (date.bat)date /t >>test.txt
Time Script (time.bat)time /t >>test.txt
Tracert Script (tracert.bat)The format for the tracert command has severaloptions we set them as follows:
Tracert -h 10 -w 500 www.stpmag.com >>test.txt
tracert -h maximum_hops -w timeout What the parameters do:-h maximum_hops
Specifies the maximum number ofhops to search for the target. Thedefault is 30 and if your site resolves inless, save some paper and set this to amore valid value.
IT’S THE NETWORK, STUPID
600
550
500
450
400
350
300
250
200
8:00 pm
SLA330 ms
5:50
pm
7:50
pm
9:50
pm
11:5
0 pm
1:50
am
3:50
am
5:50
am
7:50
am
9:50
am
11:5
0 am
1:50
pm
3:50
pm
5:50
pm
7:50
pm
9:50
pm
11:5
0 pm
1:50
am
3:50
am
5:50
am
7:50
am
9:50
am
11:5
0 am
1:50
pm
3:50
pm
5:50
pm
8:00
pm
10:0
0 pm
12:0
0 am
4:00
am
6:00
am
10:0
0 am
12:0
0 pm
2:00
pm
4:10
pm
FIG. 2: SLA, BLOWN AWAY
B a n g a l o re L a te n c y
0 1 2 3 4 5 6 7 8 13 14 15 16 17 18 19 20 21 22
430
410
390
370
350
330
310
290
270
250
Bangalore Time
25 April15 May
FIG. 3: AFTER FIX, PERFORMANCE CLICKS
-w timeout
Waits the number of millisecondsspecified by timeout for each, you canadjust this value to allow more time ifneeded.
Using Windows SchedulerThe windows scheduler is located insidethe Accessories folder under systemtools and is labeled Task Scheduler. Youwill select DOS command as the trans-action type, be sure the user you use hasexecute permissions on the machine.
You will want to use the Advancedtab when scheduling a task to repeat.If it is going to repeat every 10 minutesfor a 24-hour period, this tab shouldlook like the example in Figure 5.
Notes: The “/t” used in the date andtime scripts causes the command toreport the current value, leaving it outwill cause the program to consider thisan update command.
The “>>” causes the results toappend to the end of the existing infor-mation in the results file.
We located all the scripts and datafile in a common folder on the desk-top but they could be anywhere on thetesting PC.
The scripts needto be saved as a(Batch) “bat” file sothe scheduler cancall them.
You may find thatonce your tracereaches the data cen-ter (firewall) thetrace will no longerresolve. By this pointyou have completedthe connection tothe data center andresponse time insidethe data centershould not be theissue. Your networkadministrators canassist should youneed to measureresponse time inside the firewall.
Please make sure your .bat files areexecutable and that the results file isnot read only.
Leave a minute or so between whenyour scripts execute. For example, havethe date script run at 00:03 once a day,the time stamp run twice per hour at 05and 35 after the hour and the trace com-mand run at 00, 10, 20,…every ten min-
utes. This ensures the results file is alwaysavailable and that no conflicts will occur.
If you have a load balancer or otherdevice that routes traffic to differentservers, you can use the IP address ofthe target as an alternative to the URL.This can be handy if you are measur-ing the impact of the load balancercompared to going direct to the appli-cation server. ý
Advertiser URL Page
Automated QA www.testcomplete.com/stp 10
Froglogic www.froglogic.com/evaluate 36
FutureTest 2009 www.futuretest.net 2, 3
Hewlett-Packard hp.com/go/quality 40
Mu Dynamics www.mudynamics.com/voip 6
Reflective Solutions www.stresstester.net/stp 8
Seapine www.seapine.com/optiframe 4
SD Times Job Board www.sdtimes.com/content/sdjobboard.aspx 26
Software Planner www.softwareplanner.com 27
Software Test & Performance www.stpcon.com 21Conference
TechExcel www.techexcel.com 39
Index to Advertisers
AUTOMATED GUI TESTINGDONE RIGHT.SQUISH
®.
CROSS-PLATFORMGUI TESTING
Get a free evaluation at www.froglogic.com/evaluate
03 RZ Ad squish_Froglogic_SWTP:Layout 1 17.09.2008 13:44 Uhr Seite 1
36 • Software Test & Performance NOVEMBER 2008
IT’S THE NETWORK, STUPID
If one thing is clear fromtalking to developers andtesters of SOA applications,it’s that SOA’s very struc-ture dictates that theapproach used in testingmust differ from methodstaken with more traditionalprojects and constructs.
Doron Reuveni, CEOand co-founder of uTest, acloud-based testing servicesWebsite, says that since SOA systems arebased on multiple Web services compo-nents, a multifaceted approach to test-ing is crucial.
Unit testing of code functionalitywithout any interfaces should come firstto verify that each component functionsas it should on its own, he says. Systemtesting to verify the interfaces betweenthe various components follows.Reuveni notes that because system test-ing touches components within the ITecosystem and infrastructure, develop-ers must verify correct application andinfrastructure behaviors. “It is impor-tant to always test applications in theirentirety,” he says. Load testing is last. Itverifies the system load capabilitiesacross the end-to-end functionality.
This was true for VetSource, an onlinesupplier of pharmaceuticals to veterinar-ians and pet owners. The company is inthe midst of implementing several SOA-based applications that handle paymentprocessing and order fulfillment onbehalf of hundreds of veterinary clinics.For products not inventoried in its ownfacilities, VetSource passes orders direct-ly to manufacturers or other distributors.Of the 13 people on its developmentteam, two are dedicated to QA.
“We are learning how to merge auto-mated testing at build time as a way tosmooth out our development cycle,” saystechnology director Craig Sutter. “We’rea fast-moving group with a fairly large
plate of development tasks,so we address this up frontto eliminate iterations.” Inthe process, the team relieson QA for requirementschecking during each two-to three-week developmentcycle. VetSource is using acombination of tools andmethods, including Ajax,Mule, RESTClient for test-ing RESTful Web services,
and IBM’s Rational Robot.SOA applications represent a shift in
culture that requires thinking about theoverall business process, not just theprocessing logic of individual services.That advice comes from DavidKapfhammer, practice director for theQuality Assurance and Testing SolutionsOrganization at IT services providerKeane Inc. “SOA was supposed addressthe need for getting to market faster.Looking from that perspective, youbuild your architecture around servicesand you become more business drivenand less technology driven.”
Among the miscalculations thatReuveni sees in testing is taking a viewthat’s too narrow. “It’s important toremember that testing a SOA system onlythrough the GUI layer is not sufficient.”
He also points to SOA’s modularity,among its chief advantages, as also hav-ing a down side. “In most cases, SOA sys-tems are built based on off-the-shelfcomponents combined with self-devel-oped components. Most developmentteams find it hard to verify the interfacebetween the two.” In many cases he seesdevelopers finding bugs in code theydidn’t write but which still influenceapplication behavior dramatically.
At outsource developer TeraCode themethodology is to test early and test often.“One of the advantages of SOA is that thecomponents are discreet and can be test-ed independently of the completed prod-
uct, so you should be testing each compo-nent as early as possible,” said CEOMichael Sack. Yet, testing later in thedevelopment cycle remains equally essen-tial. “Given our agile methodology, weoften get hit with changes during the finalweeks of a project, and that necessitates alot of rapid releases which in turnrequires a lot of QA resources and QAcycles to fully regress each build.”TeraCode relies on a dual-strategyapproach: automated unit tests andextensive manual execution of test scripts.
For testing, Sack’s observations pointto coverage as a key challenge, question-ing whether it’s possible to test everyaspect adequately and fully test every-thing, and get it done in the time allot-ted. His solution is turning to uTest. Ithas helped his developers run more testscripts, get better bug reporting and,though it increases bug count, ultimate-ly leads to a better, more stable release.
So, what’s to be avoided? Everyoneagrees on several perils:
• Choosing the wrong “off-the-shelf”components.
• Testing primarily the presentationlayer.
• Not performing sufficient regres-sion testing between builds.
• Not doing load testing early.From technical perspective, SOA
apps demand a shift in testing manage-ment. “It’s a system of systems and inde-pendent services, and the developergroup often doesn’t have control over allof those services,” says Kapfhammer. “If aservice changes because the businesschanges, and 100 different apps use thisservice, you need a strategy for testingevery one.”ý
Testing SOA’s About Business,Not Just Technology
Best Practices
Joel Shore
Joel Shore is a 20-year industry veteran andhas authored numerous books on personal com-puting. He owns and operates Reference Guide,a technical product reviewing and documenta-tion consultancy in Southboro, Mass.
NOVEMBER 2008 www.stpmag.com • 37
The phrase “test early, testoften” is a worn-out man-tra by now. But in the per-formance lab, testing isstill considered a near-deployment step. Thattiming worked fine forhomegrown monolithicclient-server apps, butthat doesn’t cut it fortoday’s highly distributed,service-oriented businessapplications.
No matter what technology you’retesting, or testing with, high perform-ance will never happen if you’re notsigning up to performance-test indi-vidual technologies and componentsearly in the development lifecycle.
Take for instance a lab testing asolution with an expected service levelof a 2.1-second response time. The ini-tial UI is finally ready for load testing,which means the app is already about95 percent completed.
If performance comes in at lessthan 2.1 seconds fine. But if it doesn’t,the entire team has major concerns, asmost of the connections within theapp are already integrated. Pin-pointing the culprit service or compo-nent is nearly impossible.
To move performance testing intothe lifecycle, each contributing devel-opment team should be assigned aspecific metric and must perform theirown performance testing. A qualityengineering resource should be taskedwith making that happen.
If my solution has an expectedresponse time of 2.1 seconds, I need todecompose that SLA into perform-ance expectations at a componentlevel. In this example, the app servermay take 0.1 seconds, while we give 0.3seconds to the Verify Order process,0.5 seconds to Order Lookup, and 1.2seconds to Order Response. The key is
to notify the developmentteams what their expectedresponse times are early inthe lifecycle, so perform-ance risks can be mitigat-ed at the component level.If we wait for the entireapplication to exceed theSLA, it will be much hard-er to pinpoint the culpritin the integrated applica-tion.
Enterprise applications are fre-quently too complex and varied to beconsidered “fully tested”unless they’re built usingsuch a test-enabledmodel. For years, ourhardware engineeringcounterparts have under-stood the need for a testharness.
These test beds en-abled electronics testersto execute the instru-mented device in a vari-ety of ways, and thedevice itself reports itsfailures such that engi-neers can quickly under-stand the failures andcorrect them.
For every order hittingthe application, thou-sands of lines of codefrom many differentauthors over years ofdeployments are execut-ed. It isn’t feasible for my solution-level per-formance-testing team toprove that every possible error condi-tion or performance issue that mightoccur over time has been avoided.Therefore, a test-enabled componentis needed for the test lab to providevalue.
Virtualization has helped mitigate
some of the hardships of trying to testcomponents early, be it virtualizingserver configurations for faster setupsand teardowns or simulating thebehavior of other components youdepend on.
In the waterfall approach, youmight test the three components inorder, aiming for that 2.1-secondbenchmark. However, the completesolution isn’t tested until the integra-tion is completed, and at that point,the performance testing results mightballoon to a 4-second response time.The first temptation after getting suchterrible results might be to throwmoney at the hardware and band-width.
Aside from the real monetary con-straint and time delay of provisioningeach project with its own high-capacitylab, buying additional hardware andbandwidth is like adding oil to a leaky
engine; it buys time with-out fixing the problem.Running a system onfaster hardware may buyyou a 10 percent increasein response time, but theworkflow will reach thesame choke points, just alittle faster. Changinghardware rarely fixesthreading issues, criticalprocess failures or badcoding.
We can better manageend-user performanceadherence if we don’twait for a completedapplication to test.Furthermore, there arefew, if any, enterprise sys-tems that are still devel-oped and tested by a sin-gle team. Today’s distrib-uted applications offerincreased functionalityand flexibility, but thatcomes with the cost of a
higher degree of variability, and lesspredictable performance as a result.Here’s to getting the whole teamsigned up for better performance.
Future Test
38 • Software Test & Performance NOVEMBER 2008
Getting a GripOn SOA Speed
John Michelsen
Future Test
John Michelsen is CTO of iTKO, a maker ofQA tools for SOA applications.
•Buying additonal
hardware and
bandwidth is like
adding oil to a
leaky engine;
it buys time
without fixing
the problem.
•
OK
Prepared by Goodby, Silverstein & Partners 2008. All rights reserved. 415.392.0669
Released on 07.24.08Print Output at 100% Reader 3
ClientJob Number
Ad Number
Ad-ID
Job Title
File Name
File Format
Start Date
Color / Media
1st Close
1st Insertion
Vendor
Pubs
B
T
L
G
S
PeopleCreative Director
Associate CD
Art Director
Copywriter
Proofreader
Account Manager
Asst. Account Manager
Print Producer
Project Manager
Client
Production Artist
Mechanical SpecsHP TSGHPTSG-322
000085
EQUP13390000
Demand Gen - Quality Mgmt REV
HPTSG322_DGen_Qual Mgmt_REV.indd
Adobe InDesign CS3
7/23/08 9:27 AM
4/c Mag
None
September
XYZ Graphics
Better Software
Software Test and Performance
8.25 in x 10.75 in
7.875 in x 10.5 in
7 in x 9.75 in
None
1 in = 1 in
For Regions Only
Keith Anderson
None
Brian Gunderson
Brian Thompson
Shannon / Sage / Jen
Liz Clark
Leigh McDaniel
Katie Borzcik
Lisa Nicolay
None
Kirsten Finkas @ 7/24/08 10:00 AM
Notes
1735207.31.08
GSPRT
n/aOrisM17352_HPTSG322DGenQtyMgmt_REV
Cyan Magenta Yellow Black
Technology for better business outcomes. hp.com/go/quality
A LT E R N A T I V E T H I N K I N G A B O U T Q U A L I T Y M A N A G E M E N T S O F T W A R E :
Make Foresight 20/20.Alternative thinking is “Pre.” Precaution. Preparation. Prevention.Predestined to send the competition home quivering.
It’s proactively designing a way to ensure higher quality in yourapplications to help you reach your business goals.
It’s understanding and locking down requirements ahead oftime—because “Well, I guess we should’ve” just doesn’t cut it.
It’s quality management software designed to remove theuncertainties and perils of deployments and upgrades, leavingyou free to come up with the next big thing.
©2008 Hewlett-Packard Development Company, L.P.
S:7 inS:9.75 in
T:7.875 inT:10.5 in
B:8.25 inB
:10.75 in
![[Webinar] Test First, Fail Fast - Simplifying the Tester's Transition to DevOps](https://img.pdfslide.net/doc/110x75/58eb898d1a28ab99078b46db/webinar-test-first-fail-fast-simplifying-the-testers-transition.jpg)
