Embed Size (px)
DESCRIPTION
With more and more applications competing for users attention it is getting increasingly important to ensure that your app works on all Android devices as it should. If you launch your app without proper testing you risk wasting your marketing and development efforts by getting bad ratings and having your users disappointed just because your app does not work on their device. In his presentation Jouko Kaasila from Bitbar goes through the most common approaches to tackle the testing challenge from application developer's point of view and explain why testing for device compatibility is a must for any serious Android application developer. He will also give examples of most common urban myths about testing and go through some best practices to minimize the cost and effort related to setting up your test infrastructure. Jouko Kaasila is a co-founder of Bitbar, the company behind Testdroid Android testing tools. He has more than 14 years of experience in mobile software from early mobile services through Smartphone operating systems to Android platform. He is a serial entrepreneur with strong passion to build and help others to build businesses around mobile technologies and he is always keen to exchange views and experiences in this field.
Citation preview
PRESENTATION OVERVIEWPRESENTATION OVERVIEW
− Why to test at all?
− Richness of Android devices – a real opportunity
− Android success story in numbers
− Urban myths about mobile testing
− Options for professional grade testing
− Typical application implementations pitfalls
− Some automated testing frameworks for Android
1
WHY TO TEST AT ALL?WHY TO TEST AT ALL?
Beta test approach
2
Systematic testing
Testing criteria: Some level of basic acceptance criteria needs to be created: What devices are supported, what use cases are important, what is the expected performance
Setting up test automation: Test cases are converted to test scripts, that are executed every time when the app changes
Testing throughout the development cycle: Testing starts when first features are implemented and continues even when the app is released
Pre launch activities: Light manual testing for catching most obvious bugs. Usually done on one or max 2-3 devices.
Beta phase: App is distributed to 10-100 beta users who will report bugs and any other issues
Launch: Based on the feedback from beta testing a new version is created and launched to the app stores.
WHY TO TEST AT ALL?WHY TO TEST AT ALL?
3
Some quotes:
Summary: Developers need to avoid treating end-users as beta testers and conduct pre-launch app testing to avoid critical reviews due to poor usability once released to market
“If there is a major bug that is not caught before the launch, then a lot of expensive development and marketing work is quickly wasted”
"The first wave of users gives the app bad ratings, which then undermine [the number of] downloads and the app would [eventually] sink into oblivion."
Source: http://www.zdnet.com/mobile-app-testing-imperative-but-not-foolproof-7000000787/
WHY TO TEST AT ALL?WHY TO TEST AT ALL?
4
2012 Cap Gemini Quality Report (Mobile Testing)
“Mobile design and development fits nicely with iterative development. Looping around the cycle of requirements, design, build and test in smaller chucks means that testing need not simply be a cost “tacked-on” to the end of a project. There are some types of such as penetration and security testing that should only be done towards the end of the project, but for functionality, performance and ease-of-use testing we recommend that this be done iteratively during the mobile project.”
Source: http://www.capgemini.com/technology-blog/2012/09/quality-report-2012-mobile-testing/
WHY TO TEST AT ALL?WHY TO TEST AT ALL?
5
2012 Cap Gemini Quality Report (Mobile Testing)
“Notable from the survey is that 18% of responding companies say that they do not have enough time to test mobile apps, and 65% do not have the right tools. Also 52% cite lack of devices as a reason not to do mobile testing”
Source: http://www.capgemini.com/technology-blog/2012/09/quality-report-2012-mobile-testing/
THE RICHNESS OF ANDROID ECOSYSTEMTHE RICHNESS OF ANDROID ECOSYSTEM
Fragmentation is innovation
Source: Open Signal Maps, May 2012
6Graphical representation of fragmentation in the Android ecosystem. Each rectangle represents the market share of one Android device. Samsung Galaxy S2, the most successful Android to date, has less than 10% market share.
Samsung Galaxy S2
< 10% market share
ANDROID SUCCESS STORY IN NUMBERSANDROID SUCCESS STORY IN NUMBERS
7
− Over 500m Android devices shipped since Android 1.0
− About 220m iOS devices have been shipped since 2007
OS Q2/2012 shipments
Q2/2012 market share
Q2/2011 shipments
Q2/2011 market share
YoY change
Android 104.8 68.1% 50.8 46.9% 106.5%
iOS 26 16.9% 20.4 18.8% 27.5%
Blackberry 7.4 4.8% 12.5 11.5% -40.9%
Symbian 6.8 4.4% 18.3 16.9% -62.9%
Windows 5.4 3.5% 2.5 2.3% 115.3%
Source: IDC, 8.8.2012
URBAN MYTHS ABOUT ANDROID TESTINGURBAN MYTHS ABOUT ANDROID TESTING
8
Myth #1: Testing on emulators is enough Myth #1: Testing on emulators is enough
This app works perfectly on Android 2.3 and 4.0 emulators but on some of the most popular devices looks like this:
URBAN MYTHS ABOUT ANDROID TESTINGURBAN MYTHS ABOUT ANDROID TESTING
9
Myth #2: Testing on 5-8 most common devices is enoughMyth #2: Testing on 5-8 most common devices is enough
Yes, if reaching every every fourth Android user is ok for your CEO:
URBAN MYTHS ABOUT ANDROID TESTINGURBAN MYTHS ABOUT ANDROID TESTING
10
Myth #3: Exploratory testing just before launch is enoughMyth #3: Exploratory testing just before launch is enough
Exploratory testing is not very good for catching any regressions
Exploratory testing is not very repeatable
Exploratory testing (any manual testing) does not scale very well However, exploratory testing has its own merits – it is not enough just by itself
URBAN MYTHS ABOUT ANDROID TESTINGURBAN MYTHS ABOUT ANDROID TESTING
11
Myth #4: Users will understand if there are some issuesMyth #4: Users will understand if there are some issues
Application not working and having bugs is #1 reason for uninstalling the app
Users don’t update their apps very frequently
Quality issues are #1 reason for bad reviews in Google Play (about 80%), Application content related issues are only (20%)
TYPICAL APPLICATION IMPLEMENTATION PITFALLSTYPICAL APPLICATION IMPLEMENTATION PITFALLS
12
Mix different technologies and frameworks
Extensive use of web views− Webkits differ from one device to another− All sorts of timeout due to most content coming from server− Issues in http headers across devices− CSS&Javascript cause their own problems
Use cross platform tools− Especially the ones that generate the code on server side are very difficult to debug− Wrapping HTML5 to an app is quite ok− The time you save on development, you loose on debugging
HOW TO SET UP A PROFESSIONAL GRADE TESTINGHOW TO SET UP A PROFESSIONAL GRADE TESTING
13
Set up Continuous Integration and version control
Automate most common use cases− Address the basics first− Establish a repeatable baseline− Keep adding new tests as your functionality grows− Do not try to automate everything – there is still time and place for manual checks as well− Maintain your test library as your code evolves (store both in version control)
Define your target devices:− Depending your ambition level anything from 20 to 50 will do.− Test automatically and often with these devices− To get a reference point run occasionally tests with all possible devices (especially before any
release)
TESTDROID PRODUCTSTESTDROID PRODUCTS
14
Testdroid Cloud− Device cloud for developers to run their tests on 100+ real
Android devices, in parallel
− Provides detailed test results and analytics
Testdroid Private Cloud− Private on-premise device cloud for Enterprises to run their
tests on Android devices in their own development lab
− Provides detailed test results and analytics
Testdroid Recorder− Developer tool to simplify and accelerate the creation of
test scripts
Testdroid Cloud
INTRODUCING TESTDROID APP CRAWLERINTRODUCING TESTDROID APP CRAWLER
15
Intelligent systematic app exerciser
− Device cloud for developers to run their tests on 100+ real Android devices, in parallel
− Provides detailed test results and analytics
Visual easy to read results− Visual browser for inspecting the crawler's path
through the app
− Screenshots, logs and performance info
Android testing will never be the same!
See video: http://youtu.be/eYvhyjf7HoY
TEST AUTOMATION FRAMEWORKS FOR ANDROID - TEST AUTOMATION FRAMEWORKS FOR ANDROID - OVERVIEWOVERVIEW
16
This is an overview of OPEN SOURCE Android testing frameworks – there are tens of closed proprietary testing tools/frameworks that are not covered here
Selected frameworks:
- Monkey exerciser
- Monkey Runner
- Robotium
- Roboelectric
- Calabash
- Monkey Talk
AUTOMATION FRAMEWORKS FOR ANDROID - AUTOMATION FRAMEWORKS FOR ANDROID - OVERVIEWOVERVIEW
17
Two main approaches:
Android Instrumentation Framework Instrumentation wrapped to the app itself
AUTOMATION FRAMEWORKS FOR ANDROID – AUTOMATION FRAMEWORKS FOR ANDROID – MONKEY EXERCISERMONKEY EXERCISER
18
What:
- A UI/Application exerciser monkey for creating stream of random user events such as clicks, touches, gestures and some system level events
Usage:
Pros:
- Useful way to stress test your application to find non-obvious usage patterns and corner cases
Cons:
- No way to create any other inputs than clicks which make for instance logging in to an app impossible
$ adb shell monkey -p your.package.name -v 500
AUTOMATION FRAMEWORKS FOR ANDROID – AUTOMATION FRAMEWORKS FOR ANDROID – MONKEY RUNNERMONKEY RUNNER
19
What:
- An API toolkit for writing programs that control an Android device or emulator from outside of Android code
Usage:
Pros:
- Only way to create tests that span across application boundaries
Cons:
- UI elements can only be clicked with x,y clicks that do not scale => you have to create one script for each resolution. There is no validation mechanism for any events created by clicks
device.press('KEYCODE_MENU','DOWN_AND_UP')
AUTOMATION FRAMEWORKS FOR ANDROID – AUTOMATION FRAMEWORKS FOR ANDROID – ROBOTIUMROBOTIUM
20
What:
- It's like Selenium for Android. A JUnit based helper library/framework
Usage:
Pros:
- Powerful UI, functional and system testing capabilities (because it's based on JUnit)
Cons:
- Cannot cross the application boundaries in test scripts (because it is using Android Instrumentation Framework)
solo.clickOnText("Configuration");
AUTOMATION FRAMEWORKS FOR ANDROID – AUTOMATION FRAMEWORKS FOR ANDROID – CALABASHCALABASH
21
What:
- Cucumber based test automation framework for Android and iOS
Usage:
Pros:
- Creating scripts does not require any programming skills, Same scripts work across platforms
Cons:
- Wraps the application under test with an http server so you will be testing a different binary that the one you are going to ship.
And I select ”Options" from the menu
AUTOMATION FRAMEWORKS FOR ANDROID – AUTOMATION FRAMEWORKS FOR ANDROID – MONKEY TALKMONKEY TALK
22
What:
- Cross platform testing tool/framework for Android and iOS
Usage:
Pros:
- Same script works for your Android and iOS app (if the apps are identical)
Cons:
- Uses proprietary scripting language. Requires you to wrap your app with a testing library so you will be testing a different binary that the one you are going to ship.
Input address Verify “123 Main St.” %timeout=500
AUTOMATION FRAMEWORKS FOR ANDROID – AUTOMATION FRAMEWORKS FOR ANDROID – ROBOELECTRICROBOELECTRIC
23
What:
- Running unit tests on virtual machine – instead of an emulator or real devices
Usage:
Pros:
- Ultra fast! Can be run in every code commit
Cons:
- It is just for unit testing – it doesn't give you any info on how your app runs on target devices
activity.findViewById(R.id.press_me_button);
FURTHER INFORMATIONFURTHER INFORMATION
Additional information on web:
− Company and Products: http://www.bitbar.com
– Testdroid Cloud: http://cloud.testdroid.com
– Testdroid App Crawler: http://youtu.be/eYvhyjf7HoY
− Demo video (3min): http://youtu.be/zj11lzUmScg?hd=1
24
