Embed Size (px)
Citation preview
Adam Carmi (@carmiadam)Co-Founder and VP R&D at Applitools
ADVANCED VISUAL TEST AUTOMATION
WITH SELENIUM
YOU CAN AND SHOULD AUTOMATE YOUR VISUAL
TESTS!
AGENDA
Why automated visual testing?Tools & TechnologyWhere does it fit?
WHAT IS VISUAL TESTING?
A quality assurance activity aimed to verify that a
Graphical User Interface appears correctly to users
A VISUAL BUG
AND ANOTHER…
WHY SHOULD IT BE AUTOMATED?
THE TEST MATRIX IS TOO BIG TO APPROXIMATE MANUALLY
Web browsers Devices Operating systems Screen resolutions Responsive designs L10n
WHY SHOULD IT BE AUTOMATED?
WHY SHOULD IT BE AUTOMATED?
NATIVE / HYBRID MOBILE APPS
Harder to roll back changesCan’t push dailyUpdates take battery and data
Higher quality bar
WHY SHOULD IT BE AUTOMATED?
Many are already doing it (and sharing)…
PhantomCSSFighting Layout Bugs
CSS Critc Wraith
Needle
Grunt PhotoBoxdpdxt
WebdriverCSS
EyesHuxley
FBSnapshotTestCase
GeminiSelenium Visual Diff
VisualCeption
Specter
Snap And Compare
kobold
AGENDA
Why automated visual testing?Tools & TechnologyWhere does it fit?
THE WORKFLOW Drive the AUT and take screenshots
Compare screenshots with baseline images
Report differences
Update the baseline
THE WORKFLOW Drive the AUT and take screenshots
Compare screenshots with baseline images
Report differences
Update the baseline
DRIVING AND CAPTURING
OTHER• Wraith (URLs)• PhantomCSS (CJS)• dpdxt (URLs)• Grunt PhotoBox
(URLs)• Snap And Compare
(URLs)• Specter (JS)• Gemini (JS)• Grunt-Vigo (URLs)• BackstopJS (URLs)• Ghost Inspector (RP)• Applitools Eyes (All)
• WebDriverCSS (JS)• Fighting Layout Bugs
(Java)• Wraith-Selenium
(Ruby)• Selenium Visual Diff
(Java)• Needle (Python)• VisualCeption (PHP)• Huxley (RP)• Gemini (JS)• Vizregress (.NET)• Visualci (Java)• Viff (JS)• Pix-Diff (JS)• Applitools Eyes (All)
• CSS Critic (URLs)• FBSnapshotTestC
ase (XCTest)• Kobold (folders)• Applitools Eyes
(All)
SCREENSHOTS
Real browsers?Full page / Viewport?Regions / Frames?Device pixel ratio?Rotation?Viewport size?Page stabilization?Page preparation?
THE WORKFLOW Drive the AUT and take screenshots
Compare screenshots with baseline images
Report differences
Update the baseline
WHY NOT?
FALSE POSITIVES
ANTI-ALIASING?
ANTI ALIASING 1/2
ANTI ALIASING 2/2
BRIGHTNESS 1/2
BRIGHTNESS 2/2
SUB PIXEL SHIFTS 1/2
SUB PIXEL SHIFTS 2/2
IMAGE SCALING 1/2
IMAGE SCALING 2/2
AND MORE…1 pixel offsets in element positioning
Dynamic contentMoving elementsImages of different sizePerformance
Image Comparison APIs
ImageMagick A powerful command line tool for image processing. APIs are available for most programming languages. Fuzzing is used to eliminate slight color differences Useful for validating individual UI components An error ratio is usually used to determine a match
$ compare –metric AE –fuzz 5% img1.png img2.png diff.png
2246
Resemble.js An image comparison library
implemented in Javascript Used by PhantomCSS and other
tools. Good antialiasing support Useful for validating individual UI
components An error ratio is usually used to
determine a match http://huddle.github.io/Resemble.js/
Blink-Diff An image comparison library
implemented in Javascript Can be operated from the command
line using node Perceptual color distance
computation Anti-aliasing support Useful for validating individual UI
components An error ratio is used to determine a
match https://github.com/yahoo/blink-diff/
Applitools Eyes A specialized image processing stack
designed to compare computer generated UI images Anti-aliasing, pixel offsets, and image scaling
with low false negative rate Dynamic and moving content Compare images of different sizes No error ratio configuration required Validates full UI pages Fast! Layout matching
Available as a cloud service Free for open-source projects
DEMO
THE WORKFLOW Drive the AUT and take screenshots
Compare screenshots with baseline images
Report differences
Update the baseline
REPORT DIFFERENCESAs files on the file system (combined with source control)
REPORT DIFFERENCESAs a Gallery (example from Selenium Visual Diff)
THE WORKFLOW Drive the AUT and take screenshots
Compare screenshots with baseline images
Report differences
Update the baseline
UPDATE THE BASELINE
Rename or commit individual image files
GUI (Gemini GUI)
UPDATE THE BASELINE
Overwrite mode
Automatic maintenance (Applitools Eyes)
AGENDA
Why automated visual testing?Tools & TechnologyWhere does it fit?
WHERE DOES IT FIT?
• Components
• Code review
• Developers
• Designers• QA
WHERE DOES IT FIT?
• Pages• Page
sections
• Developers
• Designers• QA• Others
WHERE DOES IT FIT?
• Staging vs. Production
• Ops• QA
WHERE DOES IT FIT?
• Monitoring
• Ops• QA
WHERE DOES IT FIT?
VISUAL FUNCTIONALVALIDATION
FULLCOVERAGE
LOWMAINTENANCE
QUESTIONS?
Adam Carmi (@carmiadam)Co-Founder and VP R&D at Applitools
THANK YOU
