Embed Size (px)
Citation preview
1Automated Performance Testing of Desktop Applications
Introduction
For the most part, performance testing is associated with Web applications. This area is more or less covered by a lot of articles and videos about how to use JMeter and other tools, which allow load generation and performance metrics capturing.
But how do you test the performance of a Desktop application? Long story short, here are some basic requirements:
•Nosourcesofapplication(blackbox)
•AnyWindowsapplication(usuallytheseareresourceintensiveapplicationssuchasarhivators,backuputilities,etc.)
•Differentoperationswiththeapplicationundertestandsystemperformancetracking(RAM,CPU,andothermetrics)
•Notificationaboutthesystemstatewhentheuserperformsanyaction(forexample,buttonclicks,messageboxpop-out(GUIevents),etc.)
By Ostap Elyashevskyy
Automated Performance Testing of Desktop Applications
2Automated Performance Testing of Desktop Applications
To show how everything works, here is one of the possible approaches to test the performance of Desktop applications.
Tool selection
UI testing toolInordertocaptureperformancedataduringuseractions,weneedtohandletheapplication under test and interact with it, in particular click on controls, emulate mouse and keyboards events, etc.
The selected tool should satisfy the following requirements:
•InteractwithanyDesktopapplication:Java,.NET,nativeWindowsapp,andothers
•Emulateuseractions,clicks,keyboard/mouseevents
•Recognizecontrolsbytext/image(languageandtechnologyagnosticapproach)
•Captureperformanceinformationofactions:whencontrolbecomesavailable,etc.
Afterashortinvestigation,Ihavefoundtwopossibletools:SikuliandAl’exa.Bothareopen source projects, with more detailed features description and comparison to be found here.Asit’scapableofgrabbingperformancedataandintegratingwiththeNagiosmonitoringsystem,Al’exawasselectedasthemaintoolforUIperformancetesting.
Resource monitoring toolToanalyzemetricslikeCPU/RAMusage,I/OHDDoperations,etc.,aresource-monitoringtoolisneeded.Basically,Al’exasupportsintegrationwithNagios,anopensourceresourcemonitoringtoolwhichcanbeinstalledonLinuxbasedOS.IhavetrieditonUbuntuandfoundinstallation/configurationandusageverycumbersome,sinceitrequiresinstallationofNSClient++ontheclientside,alongwithSQL/PHP/WebServerandotherpackagesontheserverside.Ineededtogetresultsfasterandsimplifymonitoringtoolconfigurationanddeployment,soIusedWindowsPerformanceMonitorandintegrateditwithAl’exa.
Integration of UI and resource monitoring toolWindowsPerformanceMonitor(WPM)allowscustomcounterstobecreatedandsetthoughaproxywhichwillsendperformancedatafromAl’exatoWPM.Implementationofsuchaproxyisdescribedlater.
3Automated Performance Testing of Desktop Applications
Solution implementation
ThissectiondwellsuponimplementationofasolutionforDesktopPerformanceTestingbychoosingaspecificDesktopApplicationandtryingtoretrievemetrics.
Ioptedforaninstalled7-zipactivatorthatcangenerateanumberofsystemresources(benchmarkingfeature).Here’sasimpletestcasefor7-zip:
1.Start7-zipFilemanager
2.ClickTools
3.ClickBenchmark
4. Wait for 5 seconds
5.PressStop
6.Closethewindow
The system metrics are measured within the following operations order:
a.Navigatemenuandopenbenchmarkwindow
b.Performbenchmarkingfor10seconds
c.Stopbenchmarking
4Automated Performance Testing of Desktop Applications
Performance Monitor Counters set-upInordertotrackcustomactions,createperformancegroupsandaddperformancecountersforeachuserworkflow.IfyouarenotfamiliarwithWindowsPerformanceMonitor, take a glimpse at this short articledescribingthebasicsofWPM.
OpenPerformanceMonitorandcheckanexampleofMemorycounterswhicharealreadydefined.Inaddition,therearealotofothercountersrelatedtoCPU,HDD,Network,protocols,orspecificprocesses,threads,etc.
Inthiscase,addthecountersrelatedtoaspecifictestcasetoseehowsystemperformancechangesovertimeduringuserinteractionswithapplicationGUI.
Havinganalyzedseveralsolutionsabouthowtosetcustomcounters,Ifoundit’sbettertodoitprogrammaticallyonC#.Therefore,Ihaveimplementedasmallcommand line utility based on this cool article.Downloadthefileshere:C#sources/executable.
Let’srunutilitywiththeseparameters:
> CreatePerfCounters.exe ALEXA open_window benchmark stop_benchmark
Thefirstparametersetsagroupandanotheronedefinescounters.Bylaunching
5Automated Performance Testing of Desktop Applications
this command again, you delete an old group and create a new one with the new counters.Then,openPerformanceMonitorandcheckthenewlycreatedgroupwithcustom counters:
The operation of creating counters was separated for two reasons:
•It’snotnecessarytocreateacounterforeverytestexecution
•Thisisaresource-consumingoperation,soitisfastertocreatecountersseparately
Proxy implementationNowit’stimetosetthecountersprogrammatically.Tosetthecounters’values,implementproxyin.NET(thereareprobablyothersolutionsimplementedonPython,so if you know any libs or better solutions, please share in comments or write me anemail).Fornow,downloadsources and build them or use a compiled library PerfMonitorProxy.dll.
Putthislibraryunderthepath:C:\PerfMonitorProxy.dll
This.NETlibrarywillbecalledbythescriptsinAl’exa(Python),soit’snecessary“tore-call”itintoa.NETcode.Moreinformationaboutbuildingandcallingunmanagedcode can be found here and here.Fornow,it’senoughtodownloadandusethelibrary.
Getting started with Al’exaTestsandinteractionwiththeGUIwillbeperformedbytheAl’exatool.Firstofall,installAl’exa.CreatingscriptsinAl’exaIDEisintuitiveandthelinkedgettingstartedmanual covers this aspect. Let me elaborate on the most important moments and comment on some code snippets.
6Automated Performance Testing of Desktop Applications
Performance monitor configuration and capturing results 1.SetupPerformancemonitortoshowonlyimportantmetrics:
2.Addnextcounters:
3.Setupgrid:
Action->Properties->Graph-> Check Vertical and Horizontal grid:
4.(Optional)ClickonDataandsetWidthforcustomcounterstobold:
5.ClickonMemoryandsetappropriatescalingindex(formy
computerwith8GbofRAM,Iset0,01)
This scaling allows Free Memory to
7Automated Performance Testing of Desktop Applications
bedisplayedonthesamegraph,sincethereisamarkofonly100ontheXaxis,zoomdownMemorycounter.Itmeansthatthereare~35*100=3500MBfree.
6.Trytoexecutethescriptandanalyzetheresultsandcodestructure(youmayneedtochangeimagesforcontrolsintheprojectforyourresolution):
Afterscriptexecution,youshouldbeabletoseeasimilargraph:
•CPUconsumptionstartedincreasingafterweclickedtheBenchmarkbuttonanddroppeddownonlyafterwestoppedit(blueline).
•Beforewestartedthetest,therewasabove3Gbofavailablememoryandafterthetestitdroppedto2.5Gb
•DuringBenchmarking,thehighestloadofCPUwas87%andjustbeforethestopithadincreasedto100%.
8Automated Performance Testing of Desktop Applications
Now,let’sgothroughthecodeandcheckthemostessentialcodesnippets:
proxy.perfmon(“ALEXA”, “open_window”, 50)
Thiscommandexecutesfunctionsfromourproxydlllibraryandsetstheperformancecounteropen_windowwithvalue50(youcansetanyvalueasitisusedtovisualizethestartoftheaction).
Iamcallingtheendaction:
proxy.perfmon(“ALEXA”, <any counter>, 0) to set the counter’s value to 0.
Incaseyouneedtowaituntilthedynamiccontrolsappear,useBindfunction:
performanceValue = ControlToWait.Bind(15) # 15 seconds - wait timeout
9Automated Performance Testing of Desktop Applications
Afterthat,senditsvaluetooutput.Here’showitmaylooklike:
OK: Open Tools time is 0.842000007629 sec.
Recording performance data to log fileThe previous chapter described capturing and showing results in real time mode. Usuallythisisneededtoscheduletestsandcollectresultstoanalyzeitafterwards.
Here’showtorecordperformancetestdatainthefile:
1.RightclickonDataCollectionSets->UserDefined->New->DataCollectorSet
2.Nameit:AlexaMetrics,clickNext
3.ChoseBasic,clickNext
4.ChooselocationforlogsorleaveasisandclickNext
5.ClickFinish
6.Nowaddcounters:
10Automated Performance Testing of Desktop Applications
7.Clickstarttocapturemetrics
8. Launch the script
9.Stoprecordingperformancedata
Nowlet’scheckwhathasbeenrecorded:
1.ClickonPerformanceMonitor
2.Chosepropertiesandchooselogfileyouhavecaptured:
11Automated Performance Testing of Desktop Applications
3.AfterclickingApply,youshouldseesomethingsimilartothis:
Summary
PerformanceWindowsMonitorenablestrackingofalltheimportantmetricsandsystem/processparameters.Inaddition,itcancaptureperformancemetricstoCSV,log,binary,tabulatedfiles,orSQL.
AlsoWPMprovidestheopportunitytoschedulemonitoringexecution,definestopconditions,executealltypesofactionsonspecificconditions,etc.
ThecombinationofAl’exaandWPMresultsinapowerfulflexibletestandmonitoringsystem for Windows desktop applications. When tracking applications from different hosts,optforintegrationwithNagiosorothermonitoringsystems.Scriptsandcodeprovided in this article were created as a proof of concept and can be improved and applied according to your needs.
USAHQTollFree:866-687-3588Tel:+1-512-516-8880
UkraineHQTel:+380-32-240-9090
Bulgaria Tel:+359-2-902-3760
GermanyTel:+49-69-2602-5857
NetherlandsTel:+31-20-262-33-23
PolandTel:+48-71-382-2800
SwedenTel:+46-844-685-084
UKTel:+44-207-544-8414
WEBSITE:www.softserveinc.com
About Author
OstapElyashevskyyisaTestAutomationExpertatSoftServe. Hehasmorethan7yearsofexperience,2ofwhichareinsoftware development and 5 are in test automation testing. ThemainprojectsOstapworkedonaremobiletestautomationprojectsforbothAndroidandiOSplatforms,DekstopandWebprojectswherevarioustypesoftestingwereimplemented-fromfunctionaltoperformance/loadandstaticanalysis.Ostapbelieves
that automation should be applied not only for software testing, but also other activitieswheretimecanbesavedandautomationbringsbenefits:collectingmetricsof different kinds, implementation of helpful utilities such as bug migration utility or testresultssynchronizationwithtestmanagementsystem,etc.
About SoftServe
SoftServeisaleadingtechnologysolutionscompanyspecializinginsoftwaredevelopmentandconsultancyservices.Since1993we’vebeenpartneringwithorganizationsfromstart-upstolargeenterprisestohelpthemaccelerategrowthandinnovation,transformoperationalefficiency,anddelivernewproductstomarket.
Toachievethiswe’vebuiltastrongteamofthebrightest,mostinquiringmindsinthe industry, and we form close, collaborative relationships with our clients so we canreallyunderstandtheirneedsanddeliverintuitivesoftwarethatexceedstheirexpectations.
OurexperiencestretchesfromCloud,SecurityandUXDesigntoBigAnalyticsandtheInternetofThings,wehaveofficesacrosstheglobeanddevelopmentcentersacrossEasternEurope.Formoreinformationpleasevisitwww.softserveinc.com
