iOS Test Automation

The Good, the Bad and the Ugly

KIF,Appium,Calabash,EarlGreyorXcode7automation?�

HowtochoosethebestUITestframeworkforyour

project.�

Test frameworks under evaluation

Name: AppiumDeveloped by: Sauce LabsLanguage: Ruby, Java, C#, Python, JS, PHPSetup: Works out of the box after all needed tooling is installed (brew, node.js, appium)Status: Used and developed actively

Name: KIFDeveloped by: Square, Inc.Language: Objective C, SwiftSetup: Doesn’t work out of the boxSupport: Poor

Name: CalabashDeveloped by: Xamarin (currently Microsoft)Language: RubySetup: One-click setup when running existing tests but need some coding for seVing it up from scratch. Status: : Used and developed actively

Name: XCUITestDeveloped by: AppleLanguage: Objective C, SwiftSetup: Easy setup. Built-in in XcodeStatus: Used and developed actively

Name: EarlGreyDeveloper by: GoogleLanguage: Objective C, SwiftSetup: Easy setup when using CocoaPodsStatus: Used and developed actively

Questions to be answered before evaluation begins

  Who will write UI Tests?

  Do we want to have a possibility to write cross-platform tests?

  Do we want to change project structure and code in order to run UI Automation?

Questions to be answered while evaluating

  How responsive is community and how fast are new features developing?

  How easy will searching and defining APP’s elements be?

  What are the possibilities to start tests on physical devices and build a device farm?

#5 KIFAdvantages:

Supports iOS 8.1 and above (according documentation)Transitions are fastIt is possible to access APP’s assets (e.g. LocalizedStrgings)

Disadvantages:

Poor supportCucumber is not supported:

1. It could be hard to write tests for non-devs 2. Shiny Cucumber test reports are not available. Note: it is possible to use other plugins to get test reports

Accesses elements only by accessibility aVributesHas to be in the same repo as main APPDeeplinks (Universal links) are not supportedNecessity to manage schemes to run tests on CI

#5 KIF code example

SummaryTo make test runs stable, coding on top of framework

is needed. You can checkout example from Linkedin:

link

Disadvantages:

①  Poor support

②  Hard to write tests for non-devs as it is incompatible with tools like Cucumber

③  Accesses elements only by accessibility attributes

#4 XCUITestAdvantages:

Developed by ApplePerform gestures fast and stableRecord functionHas ‘wait APP to idle’ mechanism (but it is mostly waiting for animation and not listening the APP)

Disadvantages:Cucumber is not supported:

1. It could be hard to write tests for non-devs 2. Shiny Cucumber test reports are not available. Note: it is possible to use other plugins to get test reports

Record function is not stable. Recorded steps could be wrongSupport exists but it is really hard to get something done (e.g. bug)Limited by iOS9Access elements mostly by accessibility aVributesProvides access to native elements via accessibility classes, not the concrete classes. Xcode could crash trying to access elements with concrete classes. Necessity to touch APP code (like add accessibility ids/labels or disable animations during the test) to make it compatible with framework requirementsHas to be in the same repo as main APPDeeplinks are not supportedNecessity to manage schemes to run tests on CI

#4 XCUITest code example

Summary XCUITest could be a tool that is really nice to work with. But lack of possibility to get in contact with Apple representatives and constantly broken features like type text take away all the pleasure. There is also an amount of limitations because Apple has their vision on how tests should look like.

Disadvantages:

①  Changes in APP code is needed

②  May be hard to write tests for non-devs as it doesn’t support tools like Cucumber

③  Accesses elements mostly by accessibility attributes

#3 EarlGrey from GoogleAdvantages:

Google supportFast and stable performance of gestures (because EarlGrey synchronizes with the app and waits for it to idle)

It is possible to access APP’s assets (e.g. LocalizedStrgings)

Supports iOS 8 and above Could see concrete classes of the elements

Possible to test deeplinks (Universal Links) using workaround

Disadvantages:

Cucumber is not supported: 1. It could be hard to write tests for non-devs 2. Shiny Cucumber test reports are not available. Note: it is possible to use other plugins to get test reports

May require additional coding in your APP for using EarlGrey feature like ‘wait APP to idle’Has to be in the same repo as main APPNecessity to manage schemes to run tests on CI

#3 EarlGrey code example

SummaryEarlGrey is young but promising tool widely used by Google to automate their APPs. Absence of own element inspector can slow you down at the start but it is worth trying. Note: It is possible to use Facebook inspector this one does not ideally pass because it uses different hierarchy then EarlGrey.

Disadvantages:

①  Changes in APP code may be needed

②  Hard to write tests for non-devs as it is incompatible with tools like Cucumber

#2 Appium Advantages:

Active communityTests are stableRecord function Note: You can’t rely only on record function to write your tests

Easy to write tests for non-devs where it comes with Cucumber

Not limited by iOS versionsGood overall report where it comes with Cucumber

Independent from APP codeTests are tagged, easy to run single testNo additional frameworks should exist in APP to run tests

Cross-platform

Disadvantages:

Accesses elements mostly by accessibility aVributesProvides access to native elements via UIAutomation accessibility classes, not the concrete classesGestures performance is slowDeeplinks are not supported

#2 Appium code example

Summary Currently Appium is the most used test framework. Appium has UI Interface where a user can easily configure the framework. But Selenium Webdriver API is not fun to work with and was not specifically designed for Mobile UI testing.

Disadvantages:

①  Provides access to native elements via UIAutomation accessibility classes, not the concrete classes.

②  Accessibility attributes should be used all over the APP

③  Impossible to access APP’s classes and methods

#1 CalabashAdvantages:

Easy to write tests for non-devs as it comes with CucumberTests are stablePossible to use APP’s classes and methods

Provides access to native elements via concrete classesAccessibility aVributes are not needed to find an elementEasy to find elements (console is just awesome)Possible to identify when all the animations on the screen are doneWorks with physical devicesNot limited by iOS versionsGood overall reportIndependent from APP codeTests are tagged, easy to run single testCross-platform

Disadvantages:

Calabash framework should be inside the APPCommunity is not so bigSome coding is required to set up frameworkGestures performance is slow

Although waiting for animation methods are good and really usable, it is a pity that ‘wait APP to idle’ mechanism doesn’t exist

#1 Calabash code example

1

2 3

Summary Calabash tools like backdoors and direct Objective-C/Swift selector calls can make testing faster by putting your application into a state where it can be tested. It is called "Power over Purity”. It is easy to write tests for non-devs and find elements on the screen using hierarchy (child, sibling, descendant, parent).

Disadvantages:

①  Calabash framework should be inside the APP

②  Community is not so big

③  Some coding is required to set up framework

④  Gestures performance is slow (should change with XCUITest support)

iOS 10 support

Apple announced dropping of UIAutomation in iOS 10.

The fact that Apple will no longer support UIAutomation becomes a problem for frameworks

based on it.

Recent results  QA write tests where no programming skills are

needed

  As android also use it for automation, we actively share code between Android and iOS

  We were able to automate Push Notifications tests

  We have proof of concept for tracking tests automation

  We have a good base of implemented steps. Now writing a new test take some minutes in most cases.

for your attention!

This work is licensed under a Creative Commons Attribution-ShareAlike 4.0

International License.

Created by: Serghei Moret @xJoeSSx