Stabilizing SE Build - Selenium conf 2013

Stabilizing Selenium Build

Dima Kovalenkodima@groupon.com

@dimacus

We Are Hiring

• dima@groupon.com• @dimacus

Love My Job

• Love testing• Love hacking• Love talking about above

Partner In Crime

Darko MarinovAssociate ProfessorUniversity of Illinois

Stability is a Journey not destination

Culture

Culture Changes

• Convince Management that flakey tests waste a lot of money

• Discourage developers from relying on tests• Prevent red build from merging into

master/trunk• Red master is NOT OK!

Fix Ajax Waiting

• jQuery– evaluate_script("jQuery.active") == 0– evaluate_script('$(":animated").length') == 0

Flakey Tests

Self contained tests

• Separate Databases• Self reliant tests– Setup/Teardown

Faster Feedback Loop

• Fight to get build to 10 mins• Notify of failures before build is finished– Rspec/Cucumber Early Notification Formatters– Blog post to follow– engineering.groupon.com

Flakey Detector

• Check Jenkins or Git diff• If a test was modified, kick of flake detector

build– Run changed test 5k times in 10 mins– In random order– They should always be green!

• engineering.groupon.com

Fix Environment Problems!!

• Tests get blamed if environment fails• Standardize test boxes– Chef, Puppet, w/e

Flakey Environment

Windows Tips

• Don’t use IE• Move mouse out of browser in IE• Maximize test window• Don’t use RPD/RDC– Use VNC

Linux Tips

• Run Firefox in separate Desktop

Speed up tests

• Parallel more• Does it have to be Selenium?– Can this be a unit test?– Can it be headless?– Can you run Jasmine instead?

Black Hole Proxy

• Capture any external requests, return 200• Speeds tests up• Reliability goes up• em-proxy– https://gist.github.com/dimacus/5757573

Gem Cacher

• Cache any external dependencies locally• Don’t rely on rubygems.org to always be up!– github.com/dimacus/gem-cacher

• Local Maven repo

Credit on Grid idea

• David Burns• @AutomatedTester

Selenium Grid Extras

• github.com/groupon/Selenium-Grid-Extras

Basic Idea

• Help maintain the Environment to produce stable tests

• Turn Selenium Grid Nodes into dumb browsers

Control OS from tests

• Clean up old sessions• Kill browsers• Take full OS Screenshots• Move mouse• Reboot windows

Future plan

• Record videos• Share files with grid nodes• Spin up/down VMs on the fly• Image comparison• Auto update Selenium WebDriver• Black-hole proxy integration• Improve Grid API• Restart SE Node• Idle machine utilization

Need your help

• https://github.com/groupon/Selenium-Grid-Extras

Questions?

42!