TestIstanbul Conferences 2012 Test Automation at the Scale of Google Baran Ozgul Sr. Test Engineer - Google AdSense Olivier Gaillard Sr. Software Engineer

TestIstanbul Conferences 2012

Test Automation at the Scale of Google

Baran Ozgul

Sr. Test Engineer - Google AdSense

Olivier Gaillard

Sr. Software Engineer in Test - Google AdSense

Guessing Game

How many test cases are executed per year at Google?

100 x 10^9

~14 000 years of test execution per year


Agenda

• Case study: AdSense Frontend

• Google's Innovation Culture

• The Google Scale

• Developer Engagement

• Testing Infrastructure

• Web Testing at Google

• Key learnings

• Q&A

GOOGLE'S INNOVATION CULTURE


Google's Innovation Culture

"Launch early, launch often"

"The only true failure is not attempting the audacious."

• Take risks

• Launch and iterate


Launch and iterate

Google+: 100 new features in first 90 days

Search: Idea-to-user in less than 48 hours

Chrome: A new version every 6 weeks

THE GOOGLE SCALE


The "Google Scale"

• 60 offices, 30+ countries

• 1 giant code base everything built at latest check-in

• 11K people in R&D

• 10s of thousands of changed code lines / day

• 10s of products, 100s of feature teams

• Weekly to sub-daily release cycles

How is this possible?


Paradigm Shifts!

• Developer owns code

• Test engineering = quality assurance

• Reliance on people to scale

• Test to prevent bugs

• Spread thinly

• Developer owns code + test • Test engineering =

Productivity enablers

• Reliance on common test infrastructure to scale

• Capacity to fix bugs fast

• Staff high impact projects

DEVELOPER ENGAGEMENT


Developer engagement






Test Engineering

"Engineering productivity"

• Test leadership• Process automation• Code health signals• Test evangelism


Test Leadership

• Make existing code testable

• Implement the first N tests, lead by example

• Use data to define testing priorities

• Remove testing bottlenecks

• Make it faster to write and run the tests


Process automation

• Continuous integration

• Automated releases

• Release tracking

• Production monitoring

• Automated code quality checks


Code health signals

Latency (ms)

Code coverage

If you don't measure it, you can't control it.


Metrics: Incremental Code Coverage


Test Evangelism

• Codelabs & Techtalks ("yt: code health talks")

• Testing on the Toilet (TotT)

• Internal & external test automation events (GTAC)

• Testing intergrouplet

• Test certification

Make testing part of the development culture


Test Certified Programme

• An indication test automation maturity• Show value every stage

• “Test Certified” Levels

1: Setup tools – get familiar with testing

2: Write tests for all new code

3: Get all code under test

4: Improve test coverage

5: Invest in testable code

ADSENSE FRONTEND

Case Study


Case Study: AdSense Frontend

Before

• Time until next release: months

• Build broken for weeks

• 5+ release candidates

• Reliance on manual testing for release confidence

• Near absent test automation coverage

• One man to release them all


Case Study: AdSense Frontend

Now

• Developers write tests for their features

• Affected tests run at every code change

• Manual testing: only exploratory

• Build gets fixed in hours

• weekly releases

• <2 release candidates per release

• 6 auto-staging releases per day

• Push-button releases

TEST INFRASTRUCTURE


Life of a code change (AdSense Frontend)

minutes hours days

code review

Push to production



code ready for submission

Submit code through presubmit queue

tests fail?

minutes hours days

Push to production

code review





tests fail?

code submittedimpacted tests run

minutes hours days

Push to productionbuilcops/build breakers

notified

code review





tests fail?

code submittedimpacted tests run

minutes hours days

Push to productionbuilcops/build breakers

notified

push to production dry runcode review


Test enablers

• test engineers worked on optimizing this process

• makes it easier to create/maintain testso developers are more likely to create more tests

• lots of these steps are using a common infrastructureo search for "continuous integration google"

WEB TESTING


Browser Automation

Selenium 2.0/WebDriver

• Core browser automation platform (others exist)

• A collection of language specific bindings to control a browser

• Emulate true user actions

• Support for major browser manufacturers


Browser Automation

WebDriver driver = new FirefoxDriver(); driver.get("http://www.google.com"); WebElement element = driver.findElement(By.name("q")); element.sendKeys("Cheese!"); element.submit(); assertTrue(driver.getTitle().contains("Cheese"));

How does this scale?


Browser automation

Test Runner

Shard 1Browser 1

Test Browser

Shard 1Browser 2

Test Browser

Shard NBrowser M

Test Browser

Shard NBrowser1

Test BrowserOutput

Aggregator


Browser automation (AdSense Frontend)

Test Runner

Shard 1Chrome

Test(s) Browser(s)

Server

Database

...

Shard 1Firefox

Test(s) Browser(s)

Server

Database

...

Shard 1InternetExplorer

Test(s) Browser(s)

Server

Database

...


Browser Automation: Test enablers

• Test engineers worked on making it easy to write/maintain browser testso developers are more likely to more tests

• Browser tests run at every affecting code changesome costly tests are throttled

KEY LEARNINGS


Key Learnings

• Push testing up-streamo Early developer testing brings biggest winso Position ourselves as productivity enablerso Make/Hire developers passionate about testing

• Invest in a great testing infrastructureo Reduce the cost of writing tests to zeroo Automate as much as possible


Thanks

"Google Testing Blog""Google Test Automation Conference (GTAC)""Google Engineering Tools""WebDriver" or "Selenium"

"GTAC""Clean code talks""Web Testing at Google GTAC""Build Software at Google Scale GTAC"

Documents

TestIstanbul Conferences 2012 Test Automation at the Scale of Google Baran Ozgul Sr. Test Engineer - Google AdSense Olivier Gaillard Sr. Software Engineer