Colorful world-of-visual-automation-testing-latest

COLORFUL WORLD of

VISUAL TEST AUTOMATIONOnur Başkırt

Who am I?

▪ I am Onur Başkırt

▪ Head of Software Testing at

▪ Co-owner of

▪ Formerly: Teknosa, Ericsson, Huawei, STMicroelectronics, BAU & USCS

Online Employment and Recruitment Market Leader in TURKEY since 1999

5.6 millionVisitors

180 millionPage views

25 millionResumes

75.000+Employers

Technical Software Testing Blog

Automation Performance Security DevOps

Outline• What is Visual Test Automation?

• Why Visual Testing?

• Why Automation?

• Challenges

• Visual Test Automation Tools

• Brief Overviews

• Detailed Reviews (Applitools Eyes, Percy.io, Galen Framework)

• Selenium & ImageMagick & AShot Example

• Q&A

What is Visual Test Automation?• Verifies the GUI is displayed as expected to the end users.

• Focuses on visual contents: • Colors• Shapes• Sizes• Locations of elements

UI Test vs Visual Test

Why do We do Visual Testing?

WordPress Template

CSS animations Widgets Sliders etc.

Mobile App Generation

Engine

.ipa

.apk

GUI MUST APPEAR CORRECTLY

USER INTEARCTIONS

COMPANY BRAND

CRITICAL PRODUCTS and SERVICES

Why do We Automate Visual Testing?

Variety of Test Environments• Browsers – Mobile Devices – Resolutions etc.

Time to Market Pressure• Release cycles getting shorter.

Manual Visual Testing is Cumbersome• Human eyes error-prone and getting blind.

In Mobile World, you will live with visual bugs longer• Can not update mobile app daily.

What are the Challenges? • Anti-aliasing

What are the Challenges? • Dynamic Content (CSS Animations, GIFs)

What are the Challenges?

Dynamic content

Dynamic Content

Auto-Kerning Offsets

BRIEF OVERVIEW of Well-Known

VISUAL AUTOMATION TOOLS

PhantomCSS (Open-Source by Huddle, 4197+ Stars on GitHub)

• PhantomJS (Webkit) and SlimerJS (Gecko)Browsers

• CasperJSNavigation

• ResembleJSComparison

• Ignores Colors, Antialiasing, Moving ElementsFeatures

WebdriverCSS (Open-Source, 447+ Stars on GitHub)

• Chrome, Firefox, PhantomJSBrowsers

• WebdriverIO v2.0 Selenium Bindings for NodeJSNavigation

• GraphicsMagickComparison

• Ignores Color, AntialiasingFeatures

Wraith (Open-Source by BBC, 3883+ Stars on GitHub)

• PhantomJS, SlimerJSBrowsers

• Configuration Based .yaml file• CasperJSNavigation

• ImageMagickComparison

• Fuzz Option• Antialiasing and small changesFeatures

• and many tools exists in the market both open-source and commercial.

• FBSnapshotTestCase• Needle• Rspec Page Regression• Pix-Diff • Selenium Visual Diff• Vizregress • VisualCeption• Specter• Fighting Layout Bugs• dpdxt• Huxley• SiteEffect• Shoov• Screener.io

• BaskstopJS• CSSCritic• Grunt Photobox• VIFF• GreenOnion• Kobold• CSS Visual TestAsd• Snap And Compare• Grunt-Vigo• Hardy• Browsershots• Screenbeacon• VisualReview• Gemini

APPLITOOLS EYES

What is Applitools Eyes?• Automated Visual Testing Tool for Web sites and Mobile apps

• Finds Visual Differences with Several Match Levels

• Eyes SDK for Selenium & Appium

• Cloud-Based Clean and Informative Dashboard

How Does it Work?

Visually notifies the differences

At 2nd run and more it compares actual screenshots with the baselines.

At 1st run it saves all screenshots as baseline

Takes screenshots with Eyes SDK and sends to the Applitools cloud

Match Levels and Their Meanings?

• Pixel by Pixel ComparisonEXACT

• Mimics the human eyes. • Ignores trivial mismatches (antialiasing, small pixel movements, etc.)STRICT

• Similar with STRICT plus it ignores color changes.CONTENT

• Compares the layouts.• Checks the elements locations.LAYOUT

4 Main Steps of Eyes SDK

Close the Testeyes.close();

Add UI validation Check Pointseyes.checkWindow("Validation Window Name");

Start a Testeyes.open(driver, APP_NAME, TEST_NAME);

Create and Configure an Eyes ObjectEyes eyes = new Eyes();

Mobile Visual Testing ExampleOpen BitbarSampleApp.apk

* Visually check that start screen as expected?

Click second option “Use Testdroid Cloud”

*Visually check that Second Option is clicked?

Write “SW Test Academy” to text bar

Click Answer button

Wait until the second page appear

*Visually check that the answer is correct?

Close the driver.

Mobile Visual Testing ExampleStep-1: Install Eyes SDK with Maven Dependency

<dependency> <groupId>com.applitools</groupId> <artifactId>eyes-selenium-java</artifactId> <version>RELEASE</version></dependency>

Step-2: Do Appium, Android SDK and GenyMotion Settings

Mobile Visual Testing ExampleStep-3: Configure Eyes SDK in your Setup Method

Mobile Visual Testing ExampleStep-4: Open Eyes Do Visual Check Close Eyes

Open Eyes!

Perform a Visual Check!

Close Eyes!

Abort if Eyes not closed!

Mobile Visual Testing ExampleStep-5: Run and Check Results – First RUN (BASELINES)

Mobile Visual Testing ExampleStep-5: Run and Check Results – PASSED RESULTS

Mobile Visual Testing ExampleStep-5: Run and Check Results – FAILED RESULTS

Applitools Settings

GALEN FRAMEWORK

What is Galen Framework?Automated LAYOUT and RESPONSIVE testing framework.

• Runs in Selenium Grid (SauceLabs & BrowserStack)

• Supports Parallel Testing

Programming Languages

• Basic Syntax

• JavaScript

• JAVA & Selenium Webdriver & TestNG/Junit

Galen Reporting• Error Reporting

Generates HTML reports where you can see all your test objects

• ScreenshotsHighlights the misaligned elements

• Image ComparisonCompares images and shows differences

Galen Example with JAVA & Webdriver & TestNG

Logo: 31 pixel below from the

header

Header: 0px to the right, left, top

Navigation Links: Horizontally aligned.

Logo should displayed correctly.

Galen Example with JAVA & Webdriver & TestNGStep-1: Installation with Maven Dependency

<dependency> <groupId>com.galenframework</groupId> <artifactId>galen-java-support</artifactId> <version>2.3.2</version> </dependency>

Galen Example with JAVA & Webdriver & TestNGStep-2: Write a Galen Specifications (.gspec)

web elementsfor desktop

Declaring locations

tagging

Image Comparison

loops

5

Galen Example with JAVA & Webdriver & TestNGStep-3: Write your Test Code

Get Layout Report

Get Galen Test Information from

Layout Report

Generate HTML Report based on Galen Test Information

Galen Example with JAVA & Webdriver & TestNGStep-4: Folder Structure

Galen Example with JAVA & Webdriver & TestNGStep-5: Run the Test and Check The Results

Galen Test Report

Galen Example with JAVA & Webdriver & TestNGHomepage Layout Report

Galen Example with JAVA & Webdriver & TestNGLAYOUT ERRORS

Galen Example with JAVA & Webdriver & TestNGHEATMAP

Galen Example with JAVA & Webdriver & TestNGIMAGE COMPARISON ERRORS

What is Percy?Visual regression testing and review platform.

• COMPARES images, FINDS differences.

• We can REVIEW and APPROVE changes.

• Supports RESPONSIVE Design testing.

• Prevents FALSE-POSITIVES.

• PARALLEL TESTING support.

• Integrated with GitHub and many CI tools.

Which Technologies Does it Support?CI Tool Integrations Languages/ClientsTravis CI EmberCircleCI Ruby – CapybaraSemaphore Python – Selenium (BETA)Codeship Static websitesBuildkite Drone Jenkins

How Does it Work?

When commit any code changes.

Trigger Jenkins by git push or Pull Request then it starts to build the project and run the tests

Percy takes DOM screenshots and send them to the Percy Server

For 1st Run: Percy creates BASELINE IMAGES (No Comparison)

For 2nd + Runs: Percy compares Actual & Baseline images and notifies us.

At last step, we review the results. (Approve/Not Approve)

Installation Ember PercyRequirements for Ember Percy with Jenkins and GitHub

git node.js ember.js ember-percy

Phantom.js bower Jenkins GitHub

Percy Installation Steps (Percy Side)• Go to Percy.io and click Sign In WITH GITHUB.

• Go to Percy Dashboard and Create an Organization

• Install GitHub Integration

• Create a project and get your CI integration variables:

PERCY_TOKEN & PERCY_PROJECT

Percy Installation Steps (Jenkins Side)• Set your PERCY_TOKEN and PERCY_PROJECT environment variables

• Do GitHub Integration settings and Create a Jenkins Job.

• Do your Ember Project’s Repository settings.

• Add build and test Windows Batch commands:

• For BUILD:

• For TEST:

npm install && bower install

npm test

How to Write Test with Ember-Percy?

percySnapshot(name, options);

scope For element to snapshots. Default is full page.

percySnapshot('homepage', {scope: '#header'});

Takes ScreenShots

Sample Test Code Official TODO Example of Percy

Navigate the homepage

Take ScreenShot

Write sth to the bar and Press Enter Assertions

Take Final ScreenShot

Responsive Design Test Code

Define Resolutions by Screen Widths

First Run No Comparison (BASELINES)

Second Run PASSED!

Third Run FAILED!

Visual Test Automation by UsingImageMagick & Selenium & AShot

What is ImageMagick?• Open-source software suite that manipulates

images in a variety of formats (over 200) including PNG, JPEG, GIF, etc.

• Edits, Converts, Resizes, Compares, Rotates, Adjusts Colors etc.

• For JAVA it has Im4java which is a pure-java interface.

What is AShot?• Yandex's screenshot utility which takes, crops, prettifies and

compares screenshots.

• Entire Page

• Web Element

Example Test Scenario (Basic CSS Animation Check)

• Open www.kariyer.net

• Unhide text area of Uzman Photo

• Hover on the “UZMAN” weblement on main page.

• Wait 2 seconds to finish CSS animation

• Take screenshot of an element and save as

Baseline for the 1st Run.

• For 2nd and more runs COMPARE baseline image

with actual image.

• Put all differences in "Differences folder"

HOVER

BASELINE

Example Test Scenario (Basic CSS Animation Check)• Step-1: Install ImageMagick with all options and add to system path.

Example Test Scenario (Basic CSS Animation Check)• Step-2: Create a Selenium & TestNG project and add dependencies

Example Test Scenario (Basic CSS Animation Check)Test Method’s Code

AShot

ImageMagick

JavascriptExecutor

Example Test Scenario (Basic CSS Animation Check)How to Take Screenshot with AShot

Example Test Scenario (Basic CSS Animation Check)If Baseline Exists, Do Comparison!

ImageMagick Comparison

ImageMagickConfigurations

Add images to ImageMagick Operation object for comparison

Do Comparison!

First Run

Test ScreenShot Folder is created by using Test Name

Second Run

Change The Baseline

Test Folder

Third Run!Arithmetic Error

Number

Diffe

renc

es F

olde

r

Test

Fol

der

Test

Fai

led!

THANKS QUESTIONS?

onurb@kariyer.net

@swtestacademy

@onurbaskirt