Exercising and Scaling Up Mobile DevOps in the Enterprise

The Mobile DevOps Company

Ville-Veikko Helppi Head of Demand Generation

[email protected]

Exercising and Scaling Up Mobile DevOps in the Enterprise

WEBINAR

Mobile DevOps

•  The ‘Mobile DevOps’, Process and Tools

•  Infrastructure & Architecture •  Tips & Tricks for Scaling Up •  Troubleshooting Device Lab •  Demo •  Summary – Q&A

Agenda

More information about Mobile DevOps can be found at bitbar.com

©BitbarTechnologies2017–h7p://bitbar.com

The Mobile DevOps

•  On-Demand Mobile Device Infrastructure •  Support Any Development & Testing Framework •  Automate, Automate, Automate – To Save Time, Money and Hassle •  Provide Unprecedented Scale and Throughput •  Allow For Creative Use (Open API, Customizable Integrations) •  Adopt Mobile Development Thought Leadership and Expertise

Support all development stages:

Prototyping Development Testing Production Monitoring

Successful Adoption of Mobile DevOps

•  Modern mobile app development is Agile, but infrastructure does not support Agility well

•  DevOps practices are emerging in to the intersection of Development, QA and Operations

•  DevOps practice emphasizes the collaboration and communication of both software developers and operations while automating the process of software delivery and infrastructure changes

•  DevOps practice establishes a culture and environment where building, testing, and releasing software, can happen rapidly, frequently, and more reliably.

Mobile DevOps and Agile

Development QA

Operations

Dev Ops


SPEED WINS It’s a simple fact. The faster you can deliver new applications, features and upgrades to market, and collect feedback, the better your business performs. For that you need higher development pace. And a lot of automation. THERE IS A DEMAND FOR MOBILE DEVOPS DevOps practices and tools have revolutionized the pace and quality of service delivery – but only for traditional server-based apps. FRAGMENTATION WITH TOOLS In mobile the delivery pipelines are more complex – in terms of tools and used methods. Cohesive and well-integrated development tool flow derives from Mobile DevOps practices. MANUAL DOING IS NOT EFFECTIVE Every stage requires a lot of manual work and silo’ed solutions from various vendors and/or open source tools.

Why Mobile DevOps


Delivery Pipeline for Mobile Apps

Development Source Code Management

SCM Build Test Deploy Production

Dev Tools: •  IDEs •  JIRA •  Slack •  etc.

SCM: •  Git/GitHub/

GitLab •  Bitbucket •  MSFT •  etc.

Continuous Integration and Build

Automated Testing

Automation for app store deployment, beta testers & enterprise app stores

Production Monitoring and feedback to Dev


Delivery Pipeline with Mobile DevOps

Development Source Code Management

SCM Build Test Deploy Production

Dev Tools: •  IDEs •  JIRA •  Slack •  etc.

SCM: •  Git/GitHub/

GitLab •  Bitbucket •  MSFT •  etc.

Continuous Integration and

Build

Automated Testing Automation for app store deployment,

beta testers & enterprise app

stores

Production Monitoring and feedback to Dev

TheScopeofMobileDevOps


Mobile DevOps - User Experience

Development/IDE SCM

Build

Automated Testing

Deploy / Release

“Production”

Automated builds for every code

change

Built binaries automatically tested

Tested applications, Screenshots for various devices

and languages, certificates, metadata ready to push to App stores

Deploy To Devices

Test apps in real world (Synthetic)

Apps Used by Users •  Synthetic Monitoring •  Real User Monitoring

Beta Testing (Humans)


Exercising Mobile DevOps

Test

Deploy

Monitoring

SCM

Development

Build


Continuous Integration

CONTINUOUS INTEGRATION


The ‘Shift-Left’ – What People Think It Is

Shi>-Le>TesAng(happensearlier)

TradiAonalTesAng(happensbeforerelease)


The ‘Shift-Horizontal’ Is The Way Forward


Hardware & Software for Efficient and Scalable

Mobile DevOps

•  Parallelism, and Lots of Real Mobile Devices (Plug-and-Play) •  Mobile Test Automation Frameworks and Appropriate Use of Those •  Easy to Adopt, Deploy and Use Admin Configurability •  Use of Virtual Machines & Containers (Cloud, On-Premise) •  Open API and Ability to Integrate with Any Tools •  Efficient Use of Resources

Support all development stages:

Prototyping Development Testing Production Monitoring

What Enables Scalability?

Three Deployment Options for Test Automation with Devices

PublicCloud PrivateCloud

On-Premise

Reserved&User-Specific

Devices

Shared&PubliclyAvailableDevices

HostedbyVendor

HostedLocally©BitbarTechnologies2017–h7p://bitbar.com

UNPRECEDENTED SCALE By automating the creation, validation and deployment of the containers we are able to deploy large number of identical device servers in matter of minutes. ALWAYS CLEAN EXECUTION ENVIRONMENT New pristine container is started for each test session so that the environment is always identical from one session to another. FAST AND EASY TO UPDATE AND MAINTAIN Any updates need to be made only to the scripts that create the container images and the changes are orchestrated to all device servers ALLOWS RUN TIME CUSTOMIZATION OF THE ENVIRONMENT Because the tests are executed inside the container we can allow users to customize the environment any way they like during the test session.

Why Containers

Host

Container1

Container2

Container3

Containern

…


Because the tests are executed inside the container we can allow users to customize the environment any way they like during the test session. This is done by using the run.sh script that is used to launch the the container session. Any shell command or tool that is installed on the container can be called from run.sh script. For instance you can build anything with Maven: mvn clean test –DexecutionType=serverside or install some library or tool with Brew brew install python

Customizing The Environment

Host

Container1

Container2

Container3

Containern

…


#!/bin/bash startAppium(){

if [ "$(uname)" == "Darwin" ]; then startAppiumOSX elif [ "$(expr substr $(uname -s) 1 5)" == "Linux" ]; then startAppiumLinux else echo "Unknown OS system, exiting..." exit 1 fi

} executeTests(){

echo "Extracting tests.zip..." unzip tests.zip if [ "$(uname)" == "Darwin" ]; then echo "Running iOS Tests..." mvn clean test -Dtest=IosAppiumExampleTest -DexecutionType=serverside elif [ "$(expr substr $(uname -s) 1 5)" == "Linux" ]; then echo "Running Android Tests..." mvn clean test -Dtest=AndroidAppiumExampleTest -DexecutionType=serverside fi echo "Finished Running Tests!" cp target/surefire-reports/junitreports/TEST-*.xml TEST-all.xml

} startAppium executeTests

Example of run.sh Script

Host

Container1

Container2

Container3

Containern

…


Example: iOS Virtual Machine Components

1.3.6,1.4.3,1.4.13,1.4.16,1.6.0,1.6.3+Appium_Helper:1.3.6,1.4.3,1.4.13

0.20.4

TestAutomaAonFrameworks

Dependenciesand‘Middleware’iTunes

ProgrammingLanguagesCoreTools


Example: iOS Virtual Machine Components

1.3.6,1.4.3,1.4.13,1.4.16,1.6.0,1.6.3+Appium_Helper:1.3.6,1.4.3,1.4.13

WGETideviceinstaller

Ios-webkit-debug-proxy

usbmuxd/iproxy

0.20.4

AddiAonalframeworks

2.4.9

8.x/7.x

Settings & Config to VM •  Disable software updates,

Disable screensaver, Disable sleeping, Disable Spotlight mds agent, Remove dock items, Disable hibernation, Remove sleepimage

•  Remove swap files, Zero out free space, Shrink disk

•  Set up /etc/hosts, Remove old Keychains, Import keychain

•  Create ~/Library/MobileDevice directory, Create ~/Library/MobileDevice/Provisioning Profiles directory

•  Import mobileprovision


Mobile Test Automation Frameworks

XCTEST/XCUITEST

UIAUTOMATOR ESPRESSO APPIUM CALABASH

ANDROID No Yes Yes Yes Yes

IOS Yes No No Yes Yes

MOBILE WEB Yes

Limited to x.y clicks No Yes No

LANGUAGE Obj-C/Swift Java Java Almost any Ruby

TOOL FOR TEST CREATION

Xcode UIAutomator viewer

Hierarchy viewer

Appium.app CLI (Human-Readable syntax)

COMMUNITY Apple Google Google Community Community



•  Easy to setup & easily extendable •  Time-independent •  Backward compatibility (supporting

various API levels) •  A custom Instrumentation TestRunner

with special privileges •  Works on any API levels •  Thin layer on top of Android

Instrumentation Framework


•  Integral part of Xcode, works with both real devices and simulator

•  XCTest allows developers to write tests for components at any level

•  XCUITest provides rich UI testing capabilities for apps

•  Objective-C / Swift •  Nearly all frameworks (Appium, EarlGrey)

are working in conjuction with XCTest •  Bitbar Testing products support XCTest +

XCUITest with all deployment options

XCTest / XCUITest


•  Different Deployment Options o  Hosted in various locations (data centers) o  Co-location o  On-premise

•  Infrastructure Support o  Wireless carrier support o  Rooted devices (optional) o  Support for hardware add-ons (i.e. Bluetooth,

Wearables, etc)

•  Security (seriously!) o  Isolated network o  Dedicated hardware resources o  VPN Support o  OAuth/SSO

What Also Helps Scaling Up?


Private Cloud / Enterprise Support

•  Network Test Support o  Network Performance Optimization o  Charles Proxy o  Fiddler

•  Performance o  Built-in CPU / Memory tracking

•  Graphics Benchmarking Integration o  FPS, GPU utilization, battery usage, etc

•  Value-Added Services o  API Implementations o  On-demand scripting o  Video recording o  Custom reporting / formatting


iOS Device Cluster Server Mac OS 1.9 or later, i5 or greater CPU, 4GB RAM or 16GM Minimum for Parallel Device Runs, 256GB Hard Disk Minimum Android Device Cluster Server Ubuntu Linux 12.04, i5 or Greater CPU, 4GB RAM Minimum, 500MB Hard Disk, 10 Port USB 3.0 Hub or 1USB 3.0 Port per Device, Motherboard with BIOS that supports turning off USB Power Saving. Up to 10 Android devices per Machine. Web Front End Server Ubuntu Server 12.04 LTS 64-Bit (Virtual or Physical), 4-Core CPU Minimum, 16GB of RAM Minimum, 500GB Hard Disk Space, 1GBit Ethernet Connectivity Database Server Ubuntu Server 12.04 LTS 64-Bit (Virtual or Physical), 4-Core CPU Minimum, 8GB of RAM Minimum, 2TB Minimum of Hard Disk, Storage in RAID Array, 1GBit Ethernet Connectivity Other Equipment To Consider Server Racks, USB Cables, PDUs, WiFi Routers, UPS

Hardware Infrastructure

n


Troubleshooting A Device Lab

•  Devices o  So many things can go wrong …

•  Infrastructural Hardware o  Device Control Hardware

•  ADB and Instruments/Tools •  Hard Disks/Storage •  RAM

o  USB •  Hubs •  Cables

•  Network o  WiFi o  Cellular

•  Software o  Jenkins/CI o  SCM o  Test Automation Frameworks

80%The Most Typical Problems

10%

5%

5%©BitbarTechnologies2017–h7p://bitbar.com

The most typical problems •  Device doesn’t show up

o  CI/On-Premise Solution/etc •  Device doesn’t react •  Device doesn’t start tests/run •  Device is out of battery/battery level low •  Device shows notifications

Troubleshooting Mobile Devices Daily Check-Ups for Mobile Devices In order to maintain device farm, environment and all aspects of SW & HW, the following daily checks are recommended: •  Are devices powered up? •  Are all devices WiFi connected? •  Do all devices have an active USB connection? •  Are battery levels at least 50% (for every

device)? •  Are there any pop-ups, system notifications, or

other system dialogs open on device screen? •  Are screens unlocked?

We recommend checking every device 3 times/day.


Too Many Devices Connected / Device Server •  Too large device cluster •  Maximum of ...

o  3 iOS devices per Mac Mini o  10 Android devices per Linux server

•  Enough harddisk/storage for all servers We recommend checking & cleaning harddisks (with a shell script) every day.

Typical Mistakes (and How To Tackle) Device Server Check-Ups In addition to checking logs on server machines, it’s highly recommended to check following things: •  Diskspace on all machines (script) •  Available RAM (script) •  Do all devices have an active USB connection? •  Available and consumption of CPU for peaks •  Network conditions

We recommend automated scripts for checking above-listed items every day.


Network Not Working Properly •  Need to check if network is working •  Network for ALL connected devices •  Take WiFi down and up – try connection

with your browser

We recommend checking network connection 6-9 times every day.

Typical Mistakes (and How To Tackle) Other Software Not Working Properly Again, check all logs, that contain important information about the system and health of the environment: •  Syslog (on all server machines) •  Catalina out (on application server) •  Jenkins logs (on device servers) •  MySQL errors (and their logs)

We recommend checking these items every day. In addition, it’s recommended to implement alarm script to notify about any occurring problems.


PROBLEM: If none of the devices connected to one device server are running tests (and you can’t initiate any manual session) SOLUTION: Majority of issues seen with non-running tests are related to Jenkins •  Check Jenkins log (https://hostname:8080/log/all) •  Restart Jenkins – https://hostname:8080/safeRestart •  Restart the device server

Troubleshooting - Mobile Devices PROBLEM: Device is not running any tests. SOLUTION: First, try if you can establish a manual test session with any of your devices. If not, follow these steps: •  Check that device is powered and visible on Jenkins

o  https://hostname:8080/android-devices or http://hostname:8080/ios-devices

•  Check that device is available in Bitbar On-Premise o  https://cloudservername:9080/#service/admin/configuration


PROBLEM: Tests fail without running the test script successfully. Fail can happen at init or any phase of the script execution. SOLUTION: Find the exact error and follow these steps: •  Retrying test run is a good idea to see if the error is random or reproducable •  Check test run and device run views for logs and errors •  Check console.log for the test run •  Common failures: 1) app installation failed, 2) test zip structure is invalid, 3) Xcode didn’t start (iOS)

Troubleshooting - Mobile Devices PROBLEM: None of devices are running any tests. SOLUTION: Start with logs. In all cases, the logs contain information about errors, wrong behavior and root cause of issues can be found in those. Start with these: •  First check application server log (catalina.out) •  Check available diskspace •  Then restart Application server (sudo reboot now)


Troubleshooting – Mobile Devices


Test On Real Devices Automation - Quickly and Efficiently

Use Existing Assets Now and Future Tools

Platform Designed For Agile Development

•  Test real world scenarios on real, non-emulated devices with 100% certainty

•  See exactly what the user experiences

•  Reduce time and headache of device acquisition and management

The Key Things to Boost Mobile DevOps

•  Use any automation framework or test exercisers (e.g. Testdroid App Crawler) to get instant reports and results

•  Hit the ground running and get actionable results on a wide range of devices

•  Run more tests across more devices in less time, with fewer resources

•  Bring your own hardware, tools and frameworks and the Bitbar team will integrate them into the cloud

•  Bitbar solutions will not lock you in to any tool or automation framework

•  No vendor lock-in means that you have a long term and future proof platform for all of your mobile development and testing needs

•  Bitbar integrates seamlessly to your development tools and CI process

•  Instant results give immediate feedback to the development team

•  Faster iterations mean more efficient development, faster time to market and more satisfied users


Demonstration

Summary – Q&A

More information about mobile app testing, mobile monitoring and mobile devops at

bitbar.com

Technology

Exercising and Scaling Up Mobile DevOps in the Enterprise