Advanced Visual Test Automation with Selenium

Adam Carmi (@carmiadam)Co-Founder and VP R&D at Applitools

adam.carmi@applitools.com

ADVANCED VISUAL TEST AUTOMATION

WITH SELENIUM

YOU CAN AND SHOULD AUTOMATE YOUR VISUAL

TESTS!

AGENDA

Why automated visual testing?Tools & TechnologyWhere does it fit?

WHAT IS VISUAL TESTING?

A quality assurance activity aimed to verify that a

Graphical User Interface appears correctly to users

A VISUAL BUG

AND ANOTHER…

WHY SHOULD IT BE AUTOMATED?

THE TEST MATRIX IS TOO BIG TO APPROXIMATE MANUALLY

Web browsers Devices Operating systems Screen resolutions Responsive designs L10n

WHY SHOULD IT BE AUTOMATED?

WHY SHOULD IT BE AUTOMATED?

NATIVE / HYBRID MOBILE APPS

Harder to roll back changesCan’t push dailyUpdates take battery and data

Higher quality bar

WHY SHOULD IT BE AUTOMATED?

Many are already doing it (and sharing)…

PhantomCSSFighting Layout Bugs

CSS Critc Wraith

Needle

Grunt PhotoBoxdpdxt

WebdriverCSS

EyesHuxley

FBSnapshotTestCase

GeminiSelenium Visual Diff

VisualCeption

Specter

Snap And Compare

kobold

AGENDA

Why automated visual testing?Tools & TechnologyWhere does it fit?

THE WORKFLOW Drive the AUT and take screenshots

Compare screenshots with baseline images

Report differences

Update the baseline

DEMOhttps://github.com/webdriverio/webdrivercss

THE WORKFLOW Drive the AUT and take screenshots

Compare screenshots with baseline images

Report differences

Update the baseline

DRIVING AND CAPTURING

OTHER• Wraith (URLs)• PhantomCSS (CJS)• dpdxt (URLs)• Grunt PhotoBox

(URLs)• Snap And Compare

(URLs)• Specter (JS)• Gemini (JS)• Grunt-Vigo (URLs)• BackstopJS (URLs)• Ghost Inspector (RP)• Applitools Eyes (All)

• WebDriverCSS (JS)• Fighting Layout Bugs

(Java)• Wraith-Selenium

(Ruby)• Selenium Visual Diff

(Java)• Needle (Python)• VisualCeption (PHP)• Huxley (RP)• Gemini (JS)• Vizregress (.NET)• Visualci (Java)• Viff (JS)• Pix-Diff (JS)• Applitools Eyes (All)

• CSS Critic (URLs)• FBSnapshotTestC

ase (XCTest)• Kobold (folders)• Applitools Eyes

(All)

SCREENSHOTS

Real browsers?Full page / Viewport?Regions / Frames?Device pixel ratio?Rotation?Viewport size?Page stabilization?Page preparation?

THE WORKFLOW Drive the AUT and take screenshots

Compare screenshots with baseline images

Report differences

Update the baseline

WHY NOT?

FALSE POSITIVES

ANTI-ALIASING?

ANTI ALIASING 1/2

ANTI ALIASING 2/2

BRIGHTNESS 1/2

BRIGHTNESS 2/2

SUB PIXEL SHIFTS 1/2

SUB PIXEL SHIFTS 2/2

IMAGE SCALING 1/2

IMAGE SCALING 2/2

AND MORE…1 pixel offsets in element positioning

Dynamic contentMoving elementsImages of different sizePerformance

Image Comparison APIs

ImageMagick A powerful command line tool for image processing. APIs are available for most programming languages. Fuzzing is used to eliminate slight color differences Useful for validating individual UI components An error ratio is usually used to determine a match

$ compare –metric AE –fuzz 5% img1.png img2.png diff.png

2246

Resemble.js An image comparison library

implemented in Javascript Used by PhantomCSS and other

tools. Good antialiasing support Useful for validating individual UI

components An error ratio is usually used to

determine a match http://huddle.github.io/Resemble.js/

Blink-Diff An image comparison library

implemented in Javascript Can be operated from the command

line using node Perceptual color distance

computation Anti-aliasing support Useful for validating individual UI

components An error ratio is used to determine a

match https://github.com/yahoo/blink-diff/

Applitools Eyes A specialized image processing stack

designed to compare computer generated UI images Anti-aliasing, pixel offsets, and image scaling

with low false negative rate Dynamic and moving content Compare images of different sizes No error ratio configuration required Validates full UI pages Fast! Layout matching

Available as a cloud service Free for open-source projects

DEMO

THE WORKFLOW Drive the AUT and take screenshots

Compare screenshots with baseline images

Report differences

Update the baseline

REPORT DIFFERENCESAs files on the file system (combined with source control)

REPORT DIFFERENCESAs a Gallery (example from Selenium Visual Diff)

THE WORKFLOW Drive the AUT and take screenshots

Compare screenshots with baseline images

Report differences

Update the baseline

UPDATE THE BASELINE

Rename or commit individual image files

GUI (Gemini GUI)

UPDATE THE BASELINE

Overwrite mode

Automatic maintenance (Applitools Eyes)

AGENDA

Why automated visual testing?Tools & TechnologyWhere does it fit?

WHERE DOES IT FIT?

• Components

• Code review

• Developers

• Designers• QA

WHERE DOES IT FIT?

• Pages• Page

sections

• Developers

• Designers• QA• Others

WHERE DOES IT FIT?

• Staging vs. Production

• Ops• QA

WHERE DOES IT FIT?

• Monitoring

• Ops• QA

WHERE DOES IT FIT?

VISUAL FUNCTIONALVALIDATION

FULLCOVERAGE

LOWMAINTENANCE

QUESTIONS?

Adam Carmi (@carmiadam)Co-Founder and VP R&D at Applitools

adam.carmi@applitools.com

THANK YOU