About Myself• Rajdeep Varma

• 5 years in Quality Engineering. I work at ThoughtWorks

• Passionate about test automation and agile practices

• Tested a number of web apps and recently started testing mobile apps

I will talk about

• Our Problem:

A. Long running automation

B. Running Automation on Multiple devices

• Our Solution: parallel_calabash

Typical CI Pipeline Of Mobile Project

Git PushCompile

+ Unit Test

Build App UI Tests Distribute

“Fast Feedback”

Stats

• 5 git repo

• Avg. 30 commits a day

• Approx. 100 Automated UI tests.

• “GitPush” TO “TestedOK” > 1 Hrs

With slow tests

Git Push

UI Tests

Deploy to next stage

Umm…slow tests—Chai Break—damn! test failed :(

Fixed!

<pain>its so slow<pain/>— Pray —

LOL

:(

Patience Is Not Always A Virtue

Choose One

a) Lots of automated tests for good coverage

b) Faster automation run

c) BOTH!

How?

http://sharpreflections.com/wp-content/uploads/2012/06/multi_core_cpu.png

Our tool-set

• MobileWeb: Cucumber + waitr-webdriver + parallel_tests

• AndroidApp: Calabash

Options

• Multiple parallel builds in CI

• Proprietary cloud based tools

• Create a tool that matches our need

http://images.clipartpanda.com/choice-clipart-fork-choice-hi.png

parallel_calabash

Demo

Single Device Parallel

Time

CPU

Usa

ge

How it works? <feature_grouper>

1_digit_addition.feature

2_digit_addition.feature

3_digit_addition.feature

4_digit_addition.feature

5_digit_addition.feature

6_digit_addition.feature

7_digit_addition.feature

8_digit_addition.feature

Detect Devices

Feature Grouper

1_digit_addition 2_digit_addition

3_digit_addition 4_digit_addition

5_digit_addition 6_digit_addition

7_digit_addition 8_digit_addition

How it works? <process spawner>

1_digit_addition 2_digit_addition

3_digit_addition 4_digit_addition

5_digit_addition 6_digit_addition

7_digit_addition 8_digit_addition

ENV[ADB_DEVICE_ARG] = emulator-5554ENV[TEST_PROCESS_NUMBER] = 1

ProcessSpawner

ENV[ADB_DEVICE_ARG] = emulator-5556ENV[TEST_PROCESS_NUMBER] = 2

ENV[ADB_DEVICE_ARG] = emulator-5558ENV[TEST_PROCESS_NUMBER] = 3

ENV[ADB_DEVICE_ARG] = emulator-5560ENV[TEST_PROCESS_NUMBER] = 4

SUMMARIZE

RESULT

stdout

Reportingconfig/cucumber.yml

default: --format html --out my_report_<%=ENV['DEVICE_INFO']%>.html

Reports from 3 devices:

my_report_GT_N7100.html

my_report_Samsung_Galaxy_Note_2___4_3___API_18___720x1280_1.html

my_report_Samsung_Galaxy_Note_2___4_3___API_18___720x1280_2.html

Optimization in test distribution

• Issue with distribution based on cucumber “featue” files

• Distribution based on “Scenario” instead of features

• --group-by-scenarios

Advantages of fast tests

• Fast Feedback

• Faster development cycle for BDD environment

• Modern machines have multiple cores and parallelization can make Maximum use of Hardware

• Faster tests makes developers & Testers happy

I will talk about

• Our Problem:

A. Long running automation

B. Running Automation on Multiple devices

• Our Solution: parallel_calabash

Device Fragmentation

parallel_calabash -a <path/to/apk> features --concurrent

what did we achieve?

• 70% increase in testing velocity.

• Scalability

• Reporting

• Process specific action using ENV[TEST_PROCESS_NUMBER]

Choose One

a) Lots of automated tests for good coverage

b) Faster automation run

c) BOTH!

TIME

https://github.com/rajdeepv/parallel_calabash https://rubygems.org/gems/parallel_calabash

parallel_calabash

References:

• https://github.com/grosser/parallel_tests

• http://martinfowler.com/

• http://continuousdelivery.com/

• http://watirmelon.com/

?