Embed Size (px)
Citation preview
Fullstack End-to-End Test Automation with Node.js
One year later
SF Selenium Meetup @ Saucelabs
Chris Clayman
Mek Srunyu Stittri
2
Agenda
● Background○ What we presented last year
● Async / Await - an alternative to Webdriver’s built-in control flow.○ Limitations with control flow○ Use Babel to write the latest ES6 JavaScript syntax.
ES6 Pageobjects
● Extending MochaJS○ Custom reporter with screenshots from Sauce Labs○ Parallel tests and accurate reporting
● Type-safe JavaScript with Facebook’s Flow-type library.● Robust visual diffs● What’s next
3
Background
Back in 2015, we started looking at node.js for end-to-end functional test framework.● Kept Node.js adoption in mind
○ More and more company moving to node.js○ Share code with fullstack developers
● Team presented at San Francisco Selenium Meetup in Nov 2015○ Event - http://www.meetup.com/seleniumsanfrancisco/events/226089563/
○ Recording - https://www.youtube.com/watch?v=CqeCUyoIEo8
○ Slides - http://www.slideshare.net/MekSrunyuStittri/nodejs-and-selenium-webdriver-a-journey-from-the-java-side
4
Why we chose node.js
QA, Automation engineers
Frontend engineers
Backend engineers
Java
Javascript
Java
Python
Javascript
Java
Ruby
Javascript
Node.js
Company A Company B Company C
Node.js
Javascript
Node.jsGo, Python
Airware
5
What we presented last year
Input / update data
Get data
Input / update data
Get data
UI Framework : node.js● selenium-webdriver● mocha + wrapper● Applitools● co-wrap for webclient● chai (asserts)
Rest API Framework : node.js
● co-requests● mocha● co-mocha● chai (asserts)● json, jayschema
WebClients
Pageobjects
Webclient adaptor
Database
Backend : Go, Python
Browser
Frontend : Javascript
Microservice #1
Microservice #2 ... #n
Rest APIs
6
Selenium-webdriver usage
7
Mocha usage
8
Heavily dependent on selenium-webdriver control flows http://seleniumhq.github.io/selenium/docs/api/javascript/module/selenium-webdriver/lib/promise.html
What is the promise manager / control flow● Implicitly synchronizes asynchronous actions
● Coordinate the scheduling and execution of all commands.
Maintains a queue of scheduled tasks and executing them.
What we presented last year
driver.get('http://www.google.com/ncr');driver.findElement({name: 'q'}).sendKeys('webdriver');driver.findElement({name: 'btnGn'}).click();
driver.get('http://www.google.com/ncr') .then(function() { return driver.findElement({name: 'q'});}) .then(function(q) { return q.sendKeys('webdriver');}) .then(function() { return driver.findElement({name: 'btnG'});}) .then(function(btnG) { return btnG.click();});
The core Webdriver API is built on top of the control flow, allowing users to write the below.
Instead of that This
9
Achieving Sync-like Code
Code written using Webdriver Promise Manager
JavaScript selenium tests using Promise Manager
driver.get("http://www.google.com");
driver.findElement(webdriver.By.name('q')).sendKeys('webdriver');
driver.findElement(webdriver.By.name('btnG')).click();
driver.getTitle().then(function(title) {
console.log(title);
});
Equivalent Java code
driver.get("http://www.google.com");
driver.findElement(By.name("q")).sendKeys("webdriver");
driver.findElement(By.name("btnG")).click();
assertEquals("webdriver - Google Search", driver.getTitle());
Hey, we look similar now!
10
MochaJS with Webdriver WrapperProvided Mocha test wrapper with Promise ManagerSelenium-webdriver’s wrapper for mocha methods that automatically handles all calls into the promise manager which makes the code very sync like.
var test = require('selenium-webdriver/testing');var webdriver = require('selenium-webdriver');var By = require('selenium-webdriver').By;var Until = require('selenium-webdriver').until;
test.it('Login and make sure the job menu is there', function() { driver.get(url, 5000); driver.findElement(By.css('input#email')).sendKeys('[email protected]'); driver.findElement(By.css('input#password')).sendKeys(password); driver.findElement(By.css('button[type="submit"]')).click(); driver.wait(Until.elementLocated(By.css('li.active > a.jobs'))); var job = driver.findElement(By.css('li.active a.jobs')); job.getText().then(function (text) { assert.equal(text, 'Jobs', 'Job link title is correct'); });});
Mocha wrapper makes the code very “synchronous” like.
11
test.it('Verify data from both frontend and backend', function() { var webClient = new WebClient(); var projectFromBackend; // API Portion of the test var flow = webdriver.promise.controlFlow(); flow.execute(function *(){ yield webClient.login(Constants.USER001_EMAIL, Constants.USER_PASSWORD); var projects = yield webClient.getProjects(); projectFromBackend = projectutil.getProjectByName(projects, Constants.QE_PROJECT); }); // UI Portion of the test var login = new LoginPage(driver); login.enterUserInfo(Constants.USER001_EMAIL, Constants.USER_PASSWORD); var topNav = new TopNav(driver); topNav.getProjects().then(function (projects){ Logger.debug('Projects from backend:', projectsFromBackend); Logger.debug('Projects from frontend:', projects); assert.equal(projectsFromBackend.size, projects.size);});
Heavily dependent on the promise manager / control flowHere we handle execution order including a generator call
What We Presented Last Year
Context switching to REST API calls
Alternatives to Webdriver’s Built-in Control Flow
Async / Await
JS and Webdriver: the Good, the Bad...
Good Stuff● Functional programming● More Collaboration with
front-end development teams● JavaScript Developers writing
Selenium tests● Fast paced open source
community● Able to build things really quick● JavaScript is fun!
Bad Stuff● Webdriver’s Control Flow & Promise
Manager○ Not agnostic○ Parent variable declarations○ Iteration can be hacky
● Context switching between Webdriver and non-Webdriver asynchronous function calls
14
...and the Ugly
const promise = require('selenium-webdriver').promise
PageObject.prototype.getMessages = function() {
const els = this.driver.findElements(By.css('.classname');
const defer = promise.defer();
const flow = promise.controlFlow();
flow.execute(function* () {
const textArray = yield els.map((el) => {
return el.getText();
});
defer.fulfill(textArray);
});
return defer.promise;
}
We want our tests to be:● Readable / Flat structure ✔● Agnostic / Context-free ❌● De-asynchronous ✔● In line with ECMA standards ❌
Context Switch
15
...and the (kind of) Ugly
test.it('Archives job', function () {
const flow = promise.controlFlow();
let job;
flow.execute(function* () {
// Create a job through API
job = yield webClient.createJob();
driverutil.goToJob(driver, job.id);
});
const jobInfo = new ViewJob(driver);
jobInfo.clickArchive();
jobInfo.isModalDisplayed().then((displayed) => {
assert.isTrue(displayed, 'Modal should be displayed');
});
flow.execute(function* () {
yield webClient.deleteJob(job.id);
});
});
We want our tests to be:● Readable / Flat structure ● Agnostic / Context-free ❌● De-asynchronous ✔● In line with ECMA standards ❌
Context Switch
Context Switch
16
JobsPage.prototype.getJobList = function () { this.waitForDisplayed(By.css('.job-table')); const jobNames = []; const defer = promise.defer(); const flow = promise.controlFlow(); const jobList = this.driver.findElement(By.css('.job-table')); // get entries flow.execute(() => { jobList.findElements(By.css('.show-pointer')).then((jobs) => { // Get text on all the elements jobs.forEach((job) => { let jobName; flow.execute(function () { job.findElement(By.css('.job-table-row-name')).then((element) => { element.getText().then((text) => { jobName = text; });
// look up more table cells... }); }).then(() => { jobNames.push({ jobName: jobName }); }); }); }); }).then(() => { // fulfill results defer.fulfill(jobNames); }); return defer.promise;};
...and the Really Ugly
Not planning for complexity + promise chaining =
We want our tests to be:● Readable / Flat structure ❌● Agnostic / Context-free ❌● ‘De-asynchronous’ ✔● In line with ECMA standards ❌
17
As if we had trainer wheels on...
Doing complex things with selenium-webdriver promise manager ended up taking more time and being more cumbersome.
What It Felt Like
18
What We Wanted
19
Async/Await
Introducing Async/Await● ES2016 language specification grabbed from C#● Async functions awaits and returns a promise● No callbacks, no control flow libraries, no promise
chaining, nothing but simple syntax.● ‘De-asynchronous’ done easy
20
function notAsync () {
foo().then((bar) => {
console.log(bar)
});
}
Async/Await
async function isAsync() {
const bar = await foo();
console.log(bar);
}
Given function foo() that returns a promise...
ES5 Javascript ES2016 Latest Javascript
21
PageObject.prototype.getMessages = function() {
const els = this.driver.findElements(By.css('.classname');
const defer = promise.defer();
const flow = promise.controlFlow();
flow.execute(function* () {
const textArray = yield* els.map((el) => {
return el.getText();
});
defer.fulfill(textArray);
});
return defer.promise;
}
async getMessages() {
const els = await this.driver.findElements(By.css('.classname');
return Promise.all(els.map((el) => {
return el.getText();
}));
}
We want our tests to be● Readable / Flat structure ✔● Portable / Context-free ✔● De-asynchronous ✔● In line with ECMA standards ✔
Promise Manager vs. Async/Await
22
test.it('Archives job', function () { const flow = promise.controlFlow(); let job; flow.execute(function* () { // Create a job through API job = yield webClient.createJob(); driverutil.goToJob(driver, job.id); }); const jobInfo = new ViewJob(driver); jobInfo.clickArchive(); jobInfo.isModalDisplayed().then((displayed) => { assert.isTrue(displayed, 'Modal should be displayed'); }); flow.execute(function* () { yield webClient.deleteJob(job.id); }); });
Promise Manager vs. Async/Await
it('Archives job', async function () { const job = await webClient.createJob(); await driverutil.goToJob(driver, job.id); const jobInfo = new ViewJob(driver); await jobInfo.clickArchive(); const displayed = await jobInfo.isModalDisplayed(); assert.isTrue(displayed, 'Modal should be displayed'); await webClient.deleteJob(job.id); });
We want our tests to be● Readable / Flat structure ✔● Portable / Context-free ✔● De-asynchronous ✔● In line with ECMA standards ✔
23
How to use Async / Await
How did we get Async / Await?
● Babel compiles latest JS syntax to ES5 compatible code
● Babel-register can be a pre-runtime compiler in Mocha.
● See the repo!
24
'use strict';var BasePage = require('./BasePage');var By = require('selenium-webdriver').By;
//Constructor for the Top Navigation Barfunction TopNav(webdriver) { BasePage.call(this, webdriver); this.isLoaded();}
//BasePage and Constructor wiringTopNav.prototype = Object.create(BasePage.prototype);TopNav.prototype.constructor = TopNav;
TopNav.prototype.isLoaded = function () { this.waitForDisplayed(By.css('.options')); return this;};
TopNav.prototype.openProjectDropdown = function () { this.waitForDisplayed(By.css('.options')); this.waitForEnabled(By.css('.options ul:nth-of-type(1)')); this.click(By.css('.options ul:nth-of-type(1)')); return this;};
'use strict';import BasePage from './BasePage';import { By } from 'selenium-webdriver';import ui from './../util/ui-util';
export default class TopNav extends BasePage {
//Constructor for the Top Navigation Bar constructor(webdriver: WebDriverClass) { super(webdriver); }
async isLoaded(): Promise<this> { await ui.waitForDisplayed(this.driver, By.css('.options')); return this; }
async openProjectDropdown(): Promise<this> { await ui.waitForDisplayed(this.driver, By.css('.options')); await ui.waitForEnabled(this.driver, By.css('.options ul:nth-of-type(1)')); await ui.click(this.driver, By.css('.options ul:nth-of-type(1)')); return this; }}
ES6 Pageobjects with flow annotation
25
What we presented last year
Input / update data
Get data
Input / update data
Get data
UI Framework : node.js● selenium-webdriver● mocha + wrapper● Applitools● co-wrap for webclient● chai (asserts)
Rest API Framework : node.js
● co-requests● mocha● co-mocha● chai (asserts)● json, jayschema
WebClients
Pageobjects
Webclient adaptor
Database
Backend : Go, Python
Browser
Frontend : Javascript
Microservice #1
Microservice #2 ... #n
Rest APIs
26
Current
Database
Backend : Go, Python
BrowserRead & Write- Click, Drag- Enter text- Get text
Frontend : Javascript
Microservice #1
Microservice #2 ... #n
Rest APIs
Read & WriteAPI CallsGet, Post, Put, Delete
UI tests● selenium-webdriver● requests
Rest API tests● requests● Json, jayschema
WebClients- Job Client- Project Client- File Client- etc..
UI Pageobjects- Project List - Job List- Job info- Maps- Annotations
Node.js common tooling● Mocha● Babel (ES6)● Bluebird● Asserts (Chai)● Flow type (Facebook)
Visual tests● Applitools (visual diffs)
27
Async / Await Cons
● Error handling / stack tracing● Forces devs to actually understand promises● Async or bust - difficult to incrementally refactor● Promise Wrapper optimizations gone● Chewier syntax than control flow - `await`
everywhere
28
● Async / await was almost designed for browser automation ‘desync’ing. The glove fits
● Refactoring out of Promise Manager is cumbersome
● Simplifying test syntax -> less dependencies and opinion -> happy and efficient devs
The Bottom Line...
Custom reporter with screenshots from Saucelabs
Extending MochaJS
30
Why Roll Your Own Reporter?
● Start a conversation with developers:○ What is the most important data you need to see in
order to be efficient and successful?
● 5 things important to Airware cloud team:○ Failures First○ Flakiness vs. Failure○ Assertion messaging ○ Screenshots○ Video
31
Keeping Reporting Simple
32
Parallelization Try mocha-parallel-tests. But be careful!
33
● Take advantage of scalability to deliver what other devs want
● Keep an eye on the OS-community for new solutions
The Bottom Line...
Type-safe JavaScript
Facebook FlowType
35
Flowtype and Eslint
Don’t like the willy-nilly-ness of JavaScript? Lint! Type check!
● Static type analysis is available with Flow and TypeScript● Both have awesome IDE / editor plugins● We picked Flow in order to start annotating our code piece by
piece.● We added Webdriver, Mocha, and Chai type definitions● ESlint has some great plugins related to test structuring. We’ve
also written our own● The bottom line: the best time to capture test errors is as you write
them
And best practices
Robust visual diffs
37
Robust automated visual tests
Powered by
38
Size of test
The test pyramid*
Cost
Time
Coverage
Martin Fowler - Test Pyramid, Google Testing Blog2014 Google Test Automation Conference
BigE2E tests
MediumIntegration tests
SmallUnit tests
As you move up the pyramid, your tests gets bigger. At the same time the number of tests(pyramid width) gets smaller.
● Cost: Test execution, setup time and maintenance is less as you come down
● Feedback: Detection cycle (time) is less as you come down
● Stability: Smaller tests are less flaky
● Coverage: E2E workflow tests have better coverage but with tradeoffs; longer time, flakiness
# Number of tests
39
Google suggests a 70/20/10 ratio for test amount allocation. (Of a total 100 tests)
○ 70% unit tests, ○ 20% integration tests○ 10% end-to-end tests
The exact mix will be different for each team, but we should try to retain that pyramid shape.
The ideal ratio
Google Testing Blog2014 Google Test Automation Conference
10
20
70
40
Visual diff test pyramid
Visualtests
UI Seleniumtests
REST API tests
QA test pyramid
BigE2E tests
MediumIntegration tests
SmallUnit tests
Engineeringtest pyramid Browser
Screenshots
Backend image diffs
Visual difftest pyramid
Apply the test pyramid concept to automated visual test suite
● Browser screenshot tests are big E2E tests
● Add smaller visual testing with commandline image diffs. No browsers involved.
41
Applitools commandline image diff
Powered by
Using Applitools eyes.images SDKvar Eyes = require('eyes.images').Eyes;await image = getImage("store.applitools.com","/download/contact_us.png/" + version);// Visual validation point #1await eyes.checkImage(img, 'Contact-us page');
● Downloads or streams the images as PNG format● Uploads images to validate against Applitools service
More info - https://eyes.applitools.com/app/tutorial.html?accountId=m989aQAuq8e107L5sKPP9tWCCPU10JcYV8FtXpBk1pRrlE110
Thanks Liran Barokas !!
42
Visual diff test pyramid
BrowserScreenshots
Backend image diffs
Powered by
Test trend analysis and etc.
Whats next ?
44
Important attributes of CI/CD systems
● Trustworthy results● Tests that do not add value
are removed● Tests have the privilege
(not the right) to run in CI
● Metadata from build and test results
● Trend analysis● Is the feature ready to ship
● Cutting edge technology○ Visual diff
■ Applitools○ Kubernetes○ Containers○ Unikernels
Denali Lumma (Uber), testing in 2020
45
Test trend analyticsWork in Progress!
46
Testability as a product requirement
Collaboration between Frontend and Automation teams Surface data attributes in the UI DOM
● uuids - test can cross validate by making API calls in UI tests● Image group, image names, photo clustering attributes● etc.. Testability as a
product requirement! :)
47
Github links
The Airware Github repos https://github.com/airware
Our Async / Await Examplehttps://github.com/airware/webdriver-mocha-async-await-example
Flowtype interfaces for Webdriver, Chai, Mochahttps://github.com/airware/forseti-flow-interfaces
Thank you
Questions ?
