Human testing of mobile apps
Human testing of mobile appsFor Vistacon 2013 conference25 April 2013JULIAN HARTYContact me: [email protected]: 29 April 2013Creative Commons LicenseHow to design your mobile apps by Julian Harty is licensed under a Creative Commons Attribution-ShareAlike 3.0 Unported License.
http://creativecommons.org/licenses/by-sa/3.0/deed.en_US 1automationDrive for efficiencyState of affairsThe industry is driving towards automated testsManual testing is downgradedAgile projects need automated tests to provide fast & frequent feedbackDevelopment environments for Mobile Apps (SDKs) include Test Automation frameworksWindows Phone 8 Test Automationhttp://blogs.msdn.com/b/windowsappdev/archive/2012/09/04/automating-the-testing-of-windows-8-apps.aspx (from BJ Rollisons answer on SO site)http://mobile.dzone.com/articles/windows-phone-8-unity-testing
3Automated testsUnit TestsContinuous BuildsUI Test AutomationScale out to run tests on multiple devicesVirtual devices, remote devices, local devices
Complex, sometimes fragile, ecosystemsChallenges: device security models, limitations in frameworks, connectivity, immature domain, lack of support for multi-modal interfaces, etc.People need People tooHuman-centric testingOr mobile apps need people too?5Beyond the reach of automated testsTest frameworks focus on code, API, or GUI,
Not:Movement: acceleration, rotation, locationTactile, Haptic, audio Inputs or OutputsAesthetics: look & feel, transitionsUser Experience (UX)Highly dynamic updates: video, audio outputComplex inputs: Gestures, multi-touch, shakingLight: e.g. camera as an input device
So how do we test these?QOXQuality Of Experience (QOX)
From the standpoint of a users holistic experience of using the mobile app.
Essentially subjectivehttp://en.wikipedia.org/wiki/Quality_of_experienceUX Research, Evaluation, and Testing: strategies, tips and tools Mark J. Williams
Ive got feelings too6 hats of software testingThe red hat emotions, intuitions, feelingsI've Got a Feeling: Emotions in Testinghttp://www.stickyminds.com/s.asp?F=S17922_COL_2 Here's an article I wrote a few years ago on the six hats of software testing http://archive.newsweaver.com/qualtech/newsweaver.ie/qualtech/e_article000666359.html and if you really have lots of time on your hands, here's a video of a talk I gave on the same topic at StarWest 2008 http://www.stickyminds.com/Media/Video/Detail.aspx?WebPage=117 An inconsistency between the product and the document might be surprising, frustrating, or annoying to that person, but what really matters is inconsistency between the product and the desire.What do we need to do?SoFast, efficient interactive testingBugfestsMove and rotate the phoneBe mobileBugfest: A group of people get together to test in parallel and in common. Developers on-hand for direct feedback.Lunch outtesting a video appAround 2 hoursAround 10 testersTest the app while going out for lunch
Examples of what the testers didTourist trip on the top of a double-decker busLondon UndergroundOut for a walk with sandwichesEstablish strong foundationsLearn:The platforms, the norms, what users expectHow the platform was designed to be usedThe application lifecycleThe devicesCommon, and unusual problemsBlack BoxWhite Box12Insightful testingRAPID softwareHeuristicsMnemonics to guide the testing
Session-Based Test Management techniquesCharterTime-boxedSimple, lightweight test reportingDebrief
http://moolya.com/blog/2012/04/18/test-mobile-applications-with-cop-who-flung-gun/Extended testingExtended testingRemote devices
Remote testing
Remote testers
Credit: "Ken Banks, kiwanja.net"
http://www.kiwanja.net/gallery/texting/kiwanja_kenya_texting_17.jpghttp://qz.com/49916/indian-smartphone-users-are-gobbling-up-mobile-data/http://www.imedicalapps.com/2012/09/researchers-smartphone-spirometer-mobile-app/16Challenges of extended testingLoss of informationSensory deprivationMissing or incomplete data
Loss of fidelityLags, delays
Loss of controlHow do we test rich interactive apps unless you have the device in your hand?
In theory, theory and practice are the same. In practice, they are not. A device in your hand is worth 2 in the cloud, and 100 virtual devices. More contextMore contextAdditional considerationsCultural norms & differencesGenerationsWhats important to them?How can we empathize?Mobile Apps & PlatformsInternationalCulture?Culture?Culture?GSMA REVEALS FEARS OVER MOBILE PRIVACY ARE HOLDING BACK THE GROWTH OF MOBILE APPS IN LATIN AMERICAhttp://www.gsma.com/newsroom/gsma-reveals
A South Korean woman using her mobile phone to buy a product at the virtual retail shop in Seoul.www.guardian.co.ukExamples of what can go wrongWhat this means is that unlike the iPhone, which is most certainly the closest competitor on the market to this phone, the Storm's UI is not custom built for touch navigation -- touch navigation is added after the fact. Things which flow naturally on an iPhone -- flicking through lists, scrolling for a contact, moving around in a webpage or looking through photos -- feel inelegant and uncomfortable on the Storm. There's no inertia to movement, no assurance that your finger is the lynchpin to control of the device. The screen is sensitive enough, surely, but how its software reacts to those touches makes all the difference, and here the feeling is that you're never completely in charge of the phone.http://www.engadget.com/2008/11/19/blackberry-storm-review/
Image: http://worldwide.blackberry.com/gettingstarted/storm-9500.png! XOR[*]Practical outcomes[*] Not Exclusive ORCombining strengthsAutomated TestsFast feedback, technology and developer centric
Interactive TestsExploratory, rich interactions and dynamicsRemote testing e.g. in other countries
CombinedAugmented, Cyborg testing e.g. automated preparation, data collection, etc.Learn to delegate & learn when to take controlSome further readings
http://www.wipconnector.com/download/GuideToTheParallelUniverse_3rdEdition.pdfhttp://www.enough.de/fileadmin/uploads/dev_guide_pdfs/Guide_12thEdition_WEB.pdf
Testing and Test Automation for Mobile AppsJulian Harty
Summer 2013CRC PressDevelopmentMarketingTesting(2009)Testing(2013)Q & A ?Misc references & linkshttp://techcrunch.com/2012/06/02/android-qa-testing-quality-assurance/http://techcrunch.com/2012/03/19/applover-android-testing-platform/
24If you want the draft bookDraft book available for review & feedback
Email me [email protected] not to share or distributeComment to get the next available draft