Embed Size (px)
Citation preview
1
Mobile TestingMobile TestingKaren N. JohnsonKaren N. Johnson © Karen N. Johnson, 2012© Karen N. Johnson, 2012
2
who am I?who am I?
Karen N. JohnsonKaren N. Johnson
software test consultant based software test consultant based in Chicagoin Chicago
website: website: www.karennjohnson.com
twitter: @karennjohnsontwitter: @karennjohnson
email: email: [email protected]
3
the mobile worldthe mobile world
where are we?where are we?
4
Are you testing with Are you testing with mobile browsers or a mobile browsers or a mobile app?mobile app?
5
manufacturers, models, operating systems, network carriers, operating
systems, firmware upgrades, mobile apps & mobile websites and yet ...
so little time to test.
6
mobile manufacturersmobile manufacturers
Manufacturers Manufacturers change continually.change continually.
Manufacturers vary Manufacturers vary by country. by country.
Understand the Understand the market your app is in market your app is in vs. the overall global vs. the overall global market.market.
Expect to update your Expect to update your strategy frequently.strategy frequently.
7
device modelsdevice models
Device models Device models change frequently change frequently as well.as well.
Devices also vary by Devices also vary by country.country.
Device detection: do Device detection: do you know what your you know what your users have?users have?
8
operating systemsoperating systems
Android from Google, Inc.Android from Google, Inc.
BlackBerry OS from RIMBlackBerry OS from RIM
iOS from AppleiOS from Apple
Symbian OS from the Symbian OS from the Symbian FoundationSymbian Foundation
WebOS from HPWebOS from HP
Microsoft Windows Mobile Microsoft Windows Mobile - Source Wikipedia- Source Wikipedia
9
mobile browsersmobile browsersUsers can download a variety of browsers in addition to the native browser on the device.
Android browser Android browser
BlackBerry browserBlackBerry browser
SafariSafari
Internet Explorer Internet Explorer
FirefoxFirefox
OperaOpera
DolphinDolphin
10
network carriersnetwork carriers
Connection is based Connection is based on country, carrier, on country, carrier, signal and the signal and the device.device.
Network carriers can Network carriers can impact SMS texting impact SMS texting and access to and access to websites.websites.
11
mobile statisticsmobile statistics
Source: Which Source: Which source do you trust?source do you trust?
Country: Usage Country: Usage statistics vary statistics vary greatly by country.greatly by country.
Analytics: Are you Analytics: Are you capturing device capturing device usage for your own usage for your own application?application?
12
find your mobile sanityfind your mobile sanity
Understand the global mobile market.Understand the global mobile market.
Keep watching the market.Keep watching the market.
Know where your app is within the market.Know where your app is within the market.
Keep watch on what your competitors offer.Keep watch on what your competitors offer.
Realize what you can do and what you cannot do.Realize what you can do and what you cannot do.
13
Quiz Time !Quiz Time !
What is a walled garden? What is a walled garden? or perhaps you refer to it as a or perhaps you refer to it as a
carrier deck?carrier deck?
14
walled garden/carrier walled garden/carrier deckdeck
““...seasoned mobile developers suffer from carrier ...seasoned mobile developers suffer from carrier (operator) myopia. That is to say, they believe (operator) myopia. That is to say, they believe carriers (e.g. Vodafone, Orange, AT&T, Verizon, carriers (e.g. Vodafone, Orange, AT&T, Verizon, etc.) have, and will continue to have, complete etc.) have, and will continue to have, complete control over the availability of web content through control over the availability of web content through their networks. Traditionally this has been the case, their networks. Traditionally this has been the case, as most users had restricted access only to as most users had restricted access only to websites approved by the operator or carrier, often websites approved by the operator or carrier, often referred to as a carrier deck, closed portal, or referred to as a carrier deck, closed portal, or walled garden.”walled garden.”
Cameron Moll, Mobile Web FundamentalsCameron Moll, Mobile Web Fundamentals
mobile-centric testingmobile-centric testing
UX: user experienceUX: user experience
connectivityconnectivity
installationinstallation
user scenariosuser scenarios
OS & firmware updatesOS & firmware updates
app permissionsapp permissions
device settingsdevice settings
mobile interruptionsmobile interruptions
SD CardsSD Cards
SIM CardsSIM Cards
15
16
UXUXUser interface considerations on mobile are radically different than on the desktop.
17
connectivitconnectivityy
We are not stationary when we use mobile.
So why are we testing from the inside of our quiet well-connected offices with fully charged batteries?
18
installationinstallationApplication install & updates combined with OS updates combined with device or browser settings.
19
user user scenariosscenarios
The navigational The navigational path users take path users take on mobile is not on mobile is not the same as on the same as on a desktop. a desktop.
20
OS & firmware updatesOS & firmware updates
21
app app permissionpermissionss
Your app asks for what?
22
device settingsdevice settingsuser interface considerations on mobile are radically different than on the desktop.
23
Quiz Time !Quiz Time !
What does WURFL mean?What does WURFL mean?
24
WURFLWURFL
WURFL = Wireless Universal Resource FileWURFL = Wireless Universal Resource File
The WURFL is an "ambitious" configuration file that The WURFL is an "ambitious" configuration file that contains info about all known Wireless devices on contains info about all known Wireless devices on earth.earth.
http://wurfl.sourceforge.net/backgroundinfo.php
http://www.scientiamobile.com
Mobile device detectionMobile device detection, http://www.hand-interactive.com/resources/detect-mobile-javascript.htm
Device Atlas, Device Atlas, http://deviceatlas.com//
25
mobile interruptionsmobile interruptions
Forced updatesForced updates
Incoming calls & Incoming calls & textstexts
Battery issues Battery issues
Connectivity issuesConnectivity issues
26
SD CardsSD Cards
Does your app store Does your app store data on a card?data on a card?
What if the card is What if the card is removed?removed?
What if a different What if a different card is inserted?card is inserted?
27
SIM cardsSIM cardsA SIM card or A SIM card or Subscriber Identity Subscriber Identity Module is a chip. Module is a chip.
SIM holds personal SIM holds personal identity information, identity information, cell number, phone cell number, phone book, text messages book, text messages and other data.and other data.
Again, what Again, what happens if the card happens if the card is full? Changed? is full? Changed? Removed?Removed?
28
mobile test challengesmobile test challenges
1.1. What devices should we test with?What devices should we test with?
2.2. We don’t have any analytics yet.We don’t have any analytics yet.
3.3. Screen shots are hard to get.Screen shots are hard to get.
4.4. Replicating defects is not the same as it used to Replicating defects is not the same as it used to be.be.
5.5. The testing tool market is not there yet.The testing tool market is not there yet.
29
choosing what devices choosing what devices to testto test
30
screen capture toolsscreen capture tools
WebCaptureWebCapture
Apple. DJM DevelopmentApple. DJM Development
http://itunes.apple.com/us/app/webcapture/id331452784?mt=8
Apple iPhone and iPod TouchApple iPhone and iPod Touch
Hold the "Home" button, then press the "Power/Sleep" Hold the "Home" button, then press the "Power/Sleep" button. The screenshot will automatically appear in your button. The screenshot will automatically appear in your photo gallery.photo gallery.
Dolphin Screen CutDolphin Screen Cut
Android. Dolphin Browser Add-onAndroid. Dolphin Browser Add-on
https://market.android.com/details?id=mobi.mgeek.ScreenCut&hl=en
31
testing toolstesting tools
TestFlightTestFlight
FoneMonkey: iOSFoneMonkey: iOS
Apple xCode: Mac, iPhone and iPadApple xCode: Mac, iPhone and iPad
Robotium: AndroidRobotium: Android
eggPlant: automation tool to work across all eggPlant: automation tool to work across all devicesdevices
Perfecto MobilePerfecto Mobile
Device AnywhereDevice Anywhere
32
Quiz Time !Quiz Time !
What is jailbreaking?What is jailbreaking?
33
iOS jailbreakingiOS jailbreaking
iOS jailbreaking, or simply iOS jailbreaking, or simply jailbreaking, is the process of jailbreaking, is the process of removing the removing the limitations imposed imposed by by Apple on devices running the on devices running the iOS operating system through through use of custom use of custom kernels..- wikipedia- wikipedia
34
relevant “traditional” relevant “traditional” testingtesting
functionalfunctional
searchsearch
navigationnavigation
securitysecurity
e-commerce (now m-commerce)e-commerce (now m-commerce)
cookiescookies
interface testinginterface testing
35
functional testingfunctional testingHere’s an example of a mobile app from UPS. Notice how much traditional functional testing there is to do.
Track packagesTrack packages
Ship packagesShip packages
Locate a storeLocate a store
Get a quoteGet a quote
Give feedbackGive feedback
36
searchsearchClassic testing with a new twist: search with barcodes or where device location is factored into the results.
37
navigationnavigationWe are still navigating sites and apps but rarely do we have a visual cookie crumb trail.
RotateRotate
Pinch/zoomPinch/zoom
AccelerometerAccelerometer
Vertical/horizontalVertical/horizontal
Paging forward & Paging forward & backback
38
securitysecurityUser interface considerations on mobile are radically different than on the desktop.
39
m-commercem-commerceThe shopping cart with barcodes, location-based (NFC) coupons, card scanning/swiping and Google wallet.
40
Quiz Time !Quiz Time !
What is rooting?What is rooting?
41
rootingrooting
Rooting is a process allowing users of mobile Rooting is a process allowing users of mobile phones, tablets PCs, and other devices running phones, tablets PCs, and other devices running the Android operating system to attain privileged the Android operating system to attain privileged control (known as “root access.”) within control (known as “root access.”) within Android's subsystem.Android's subsystem.
- wikipedia- wikipedia
42
cookiescookies
Remember me?Remember me?
Forgotten passwordForgotten password
Auto-loginAuto-login
Third party login Third party login access (i.e. Use access (i.e. Use Facebook account to Facebook account to login)login)
Password changed Password changed on the desktop.on the desktop.
43
interface testinginterface testing
Watch for apps Watch for apps working together. working together. Like Yelp! and Open Like Yelp! and Open Table.Table.
Consider app-Consider app-dependencies on dependencies on phone settings - like phone settings - like location-based location-based searching.searching.
44
where are we now?where are we now?
develop a strategydevelop a strategy
watch the marketwatch the market
watch competitorswatch competitors
watch “other” apps watch “other” apps and sites for and sites for innovationinnovation
develop mobile develop mobile testing skills & testing skills & experiencesexperiences
46
resourcesresources
mobile usage statsmobile usage stats
http://mobithinking.com/mobile-marketing-tools/latest-mobile-stats
http://www.cloudfour.com/a-comprehensive-guide-to-mobile-statistics/
Google offers free web statistics. To use the service, you need to apply and setup the Google offers free web statistics. To use the service, you need to apply and setup the service – from there you can track device statistics.service – from there you can track device statistics. http://www.google.com/analytics/
mobile manufacturers, mobile manufacturers, http://www.mobileforum.com/
mobile browsers, mobile browsers, http://webtrends.about.com/od/mobileweb20/tp/list_of_mobile_web_browsers.htm
mobile testing communities, mobile testing communities, http://www.mobileqazone.com/
47
resourcesresources
Product offerings to test mobile devices remotely. Product offerings to test mobile devices remotely.
http://www.deviceanywhere.com
http://www.perfectomobile.com
quick tests for mobile readinessquick tests for mobile readiness
http://ready.mobi/launch.jsp?locale=en_EN
http://www.gomez.com/mobile-readiness-instant-test/
http://validator.w3.org/mobile/
http://ipadpeek.com/
48
resourcesresourcesmobile manufacturers, mobile manufacturers, http://www.mobileforum.com/
mobile browsers, mobile browsers, http://webtrends.about.com/od/mobileweb20/tp/list_of_mobile_web_browsers.htm
A worldwide community with local chapters for A worldwide community with local chapters for mobile design - Mobile Monday, known as MoMo. mobile design - Mobile Monday, known as MoMo. http://www.mobilemonday.net/
49
jonathan kohljonathan kohl
Designing Mobile Apps: Using Old School Tools for New School Designing Mobile Apps: Using Old School Tools for New School TechnologyTechnology Article, Modern Analyst, April 2011 Article, Modern Analyst, April 2011
Mobile Challenges for Project Management, Part 2: Human FactorsMobile Challenges for Project Management, Part 2: Human Factors Article, Stickyminds, January 2011Article, Stickyminds, January 2011
Mobile Challenges for Project Management, Part 1: Project FactorsMobile Challenges for Project Management, Part 1: Project Factors Article, Stickyminds, January 2011. Re-published January 2012 by Article, Stickyminds, January 2011. Re-published January 2012 by Better Software MagazineBetter Software Magazine
Test Mobile Apps with I SLICED UP FUN!Test Mobile Apps with I SLICED UP FUN! Article, self-published, Article, self-published, October 2010. Republished December 2010 by October 2010. Republished December 2010 by Mobile Developer Mobile Developer MagazineMagazine
50
julian hartyjulian harty
A Practical Guide to Testing Mobile Smartphone A Practical Guide to Testing Mobile Smartphone ApplicationsApplications
http://www.mobileapptesting.com/julian-harty-http://www.mobileapptesting.com/julian-harty-mobile-app-testing-tutorial/2011/03/mobile-app-testing-tutorial/2011/03/
http://www.ebaytechblog.com/2011/06/06/the-http://www.ebaytechblog.com/2011/06/06/the-8th-edition-of-mobile-developers-guide/8th-edition-of-mobile-developers-guide/
twitter: @julianhartytwitter: @julianharty
51
mobile articlesmobile articles
Karen N Johnson, “The Mobile Environment”Karen N Johnson, “The Mobile Environment”The Testing Planet, November 2011 The Testing Planet, November 2011
Karen N Johnson, “What If” in Mobile TestingKaren N Johnson, “What If” in Mobile TestingTea-Time with Testers, July 2011 Tea-Time with Testers, July 2011
Stephen Janaway, Stephen Janaway, http://www.thetestingplanet.com/2011/11/mobile-http://www.thetestingplanet.com/2011/11/mobile-testing-that-s-just-a-smaller-screen-right/testing-that-s-just-a-smaller-screen-right/
Brad Frost, Brad Frost, http://bradfrostweb.com/blog/mobile/test-http://bradfrostweb.com/blog/mobile/test-on-real-mobile-devices-without-breaking-the-bank/on-real-mobile-devices-without-breaking-the-bank/
•
52
mobile booksmobile books
Programming the Mobile Web, by Maximiliano FirtmanProgramming the Mobile Web, by Maximiliano Firtman
A Practical Guide to Testing Wireless Smartphone Applications, by A Practical Guide to Testing Wireless Smartphone Applications, by Julian Harty and Mahadev SatyanarayananJulian Harty and Mahadev Satyanarayanan
Strategic Mobile Design: Creating Engaging Experiences, by Strategic Mobile Design: Creating Engaging Experiences, by Joseph Cartman and Richard TingJoseph Cartman and Richard Ting
Mobile Design and Development, by Brian FlingMobile Design and Development, by Brian Fling
Mobile Web Design, by Cameron MollMobile Web Design, by Cameron Moll
Designing Mobile Interfaces,by Steven Hoober and Eric BerkmanDesigning Mobile Interfaces,by Steven Hoober and Eric Berkman
53
twittertwitter
Luke Wroblewski, @lukewLuke Wroblewski, @lukew
Maximiliano Firtman,@firtMaximiliano Firtman,@firt
Cameron Moll, Cameron Moll, @cameronmoll@cameronmoll
tomiahonen, @tomiahonentomiahonen, @tomiahonen
Pete Cashmore, Pete Cashmore, @mashable@mashable
@karennjohnson - see who @karennjohnson - see who I followI follow
