In the land of the startup

Joseph Wilk

Acceptance Testing

What kind of land is this?

What kind of land is this?

Do startups need acceptance tests?

Measure

Learn

Build

Context is king

Toolset

Rails, MongoDB, Mysql, RabbitMQ, Memcache

Cucumber, Rspec

Cucumber

• Behaviour Driven Development framework

• Plaintext

• Promotes communication

Given /^I have trained monkeys$/ do @test_pilots << Monkeys.newend

Feature: Filling Cucumbers with rocket fuel

Scenario: Rocket fuel Given I have trained monkeys When I click the launch button Then the monkeys should not die

Why Acceptance?

• Developer centric tool

• Behaviour Driven Development

• Conversations

Where acceptance tests are born

ProductOwner

Developer

QAUser

interaction

Where acceptance tests are born

ProductOwner

Developer

QA

Just In TimeTalking

User interaction

Non-technical people writing

Gherkin?

Feature: Ajax pagination for upcoming events In order to reduce friction finding concerts I want to go to As a logged in user I want an Ajax button which loads more concerts

Scenario: Events 60 days in the futures Given ... When ... Then ... Scenario: No events outside of the next 60 days Given ... When ... Then ... Scenario: No events Given ... When ... Then ...

Are we done yet?

“A story isn't done until it is being used by real users in production and has been validated to be a useful part of a product.”

Joshua Kerievsky

LearningsGood and bad

Asynchronous Testing is messy

Scenario: Upload image ... Given all the messages have been processed ...

Feature Overload

254 Feature files

1257 Scenarios/Examples

10807 Steps

Slicing Features

Tagging

Filenames

$ cucumber --tags @media

Subsystems

$ cucumber visitor_*

$ cucumber features/admin/*

Profiles $ cucumber --profile admin

@media, @publicFeature: Visitor views artist’s media

@feed, @adminFeature: Logged in user views views their activity feed

Acceptance testsUnit tests

vs

The Test Build

254 Features, 1257 scenarios, 10807 steps

The Test Build

254 Features, 1257 scenarios, 10807 steps

1 build server ~ 4 hours

The Test Build

254 Features, 1257 scenarios, 10807 steps

1 build server ~ 4 hours

19 EC2 build servers ~ 13:40 minutes

The Test Build

254 Features, 1257 scenarios, 10807 steps

$2068.99/$3000

1 build server ~ 4 hours

19 EC2 build servers ~ 13:40 minutes

Divide and Concuquer

Divide and Concuquer

Divide and Concuquer

Divide and Concuquer

Divide and Concuquer

Divide and Concuquer

Scenario: Related artists Given an artist “SYGC” And “M.Bison” is related to “SYGC” When I visit the artist Then I should see within Similar Artists a link to “M.Bison”

Divide and conquer

Rails

Scenario: Related artists Given an artist “SYGC” And “M.Bison” is related to “SYGC” When I visit the artist Then I should see within Similar Artists a link to “M.Bison”

Divide and conquer

RelatedArtistsService

Rails

Scenario: Related artists Given an artist “SYGC” And “M.Bison” is related to “SYGC” When I visit the artist Then I should see within Similar Artists a link to “M.Bison”

Divide and conquer

related(artist)

RelatedArtistsService

Rails

Scenario: Related artists Given an artist “SYGC” And “M.Bison” is related to “SYGC” When I visit the artist Then I should see within Similar Artists a link to “M.Bison”

Divide and conquer

related(artist)

RelatedArtistsService

HTMLRails

Scenario: Related artists Given an artist “SYGC” And “M.Bison” is related to “SYGC” When I visit the artist Then I should see within Similar Artists a link to “M.Bison”

Divide and conquer

related(artist)

RelatedArtistsService

HTMLRails

Scenario: Related artists Given an artist “SYGC” And “M.Bison” is related to “SYGC” When I visit the artist Then I should see within Similar Artists a link to “M.Bison”

Divide and conquer

related(artist)

RelatedArtistsService

HTMLRails

Scenario: Related artists Given an artist “SYGC” And “M.Bison” is related to “SYGC” When I visit the artist Then I should see within Similar Artists a link to “M.Bison”

unit testunit test

Tests that never fail

Tests that never fail

Tests that never fail

Tests which regularly fail

Daily Build

Tests that never fail

Tests which regularly fail

Daily Build Nightly Build

Tests that never fail

Tests which regularly fail

Daily Build Nightly Build

Tests that never fail

Tests which regularly fail

$cucumber --tags ~@nightly $cucumber --tags @nightly

Flickering Confidence

Flickering Confidence

FAIL

Flickering Confidence

FAIL

FAIL

Flickering Confidence

FAIL

PASS

FAIL

Flickering Confidence

FAIL

PASS

FAIL

Main Flicker@flicker

PairwiseScenario: Testing Cucumber against different environments Given I have a rails app <Rails version> And I’m using Ruby <Ruby version> And I am using the <Test Framework> And I am using the driver <Driver> Then Cucumber should install and play nicely Examples: | Rails version | Ruby version | test framework | Driver | | 3.0 | 1.8.7 | Rspec | Webrat | | 3.0 | 1.8.7 | Testunit | Capybara | | 3.0 | 1.8.7 | Rspec | Capybara | | 3.0 | 1.8.7 | Testunit | Webrat | | 3.0 | 1.9 | Rspec | Webrat | | 3.0 | 1.9 | Testunit | Webrat | | 2.3.1 | 1.9 | Rspec | Webrat | ....

| Rails version | Ruby version | test framework | Driver | | 2.3.1 | 1.8.7 | Rspec | Webrat | | 2.3.1 | 1.8.7 | Rspec | Capybara | | 2.3.1 | 1.8.7 | Testunit | Webrat | | 2.3.1 | 1.8.7 | Testunit | Capybara | | 2.3.1 | 1.9 | Rspec | Capybara | | 2.3.1 | 1.9 | Rspec | Webrat | | 2.3.1 | 1.9 | Testunit | Capybara | | 2.3.1 | 1.9 | Testunit | Webrat |

....

“most faults are caused by interactions of at most two factors”

Run Just Enough Tests

# * Test files must be stored in test/# * Test files names must start with test_# * Test class names must start with Test# * Implementation files must be stored in lib/# * Implementation files must match up with a # test file named# test_.*implementation.rb

@analyzer = Rcov::CodeCoverageAnalyzer.new

Cucover

Autotest

The Art of War Plaintext

, , @"===, ,_____cctI "?AAAAAAAAAAAAAAAA,,,,,,,,,,,,,,,,,,,,,;LLLLLLLLLL ~",,, 1""""""""""""###OOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOO '"EEEEE, !'"***"~~~~~~"OOOIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIII ,EEEEE)>"'''???????"WWW!MMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMM "E.,)+="WWW~~~~~~#"OOO1OOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOO ,~:#")LLL!"+++???????"$$$1==========##/ &LLLLLLLLL;;;;;;;;;;;;,,,/ 1#LLLLLLLLLLLLLLLLLLLLLL! ,!###LLLLLL"'EEEE,'"LLLLL! !######LLL" "EEE" "LLLL" !#########L! "EEJ. "LL! !##########1 "JJ*,l" !############"! ,l" 1##########" 1"~~,~~" !##########" !###########! !###########1 !############! 1############# !"#############" !##############! 1##########"' A magnum, By calendron 1#####"' """""

Don’t force structure

Feature: Title In order to <value> As a <role> I want <feature>

Avoid Noise! Given I am able to loginGiven I am able to login

Avoid Inconsistency

Given I loginGiven I authenticateGiven I gain access

BalanceAbstraction

Given I go to the login pageAnd I fill in "username" with "cuke"And I fill in "password" with "cuker"And I click "login"

Given I'm logged in

Use Language Building blocks

Given /I’m logged in/ do User.create!(:user => 'josephwilk', :password => "pass") Given 'I fill in "password" with "josephwilk"' Given 'I fill in "password" with "pass"' Given 'I click "login"'end

Continuous Integration(WIP)Work in progress Done

list movies

add movies

edit movies

delete movies

Pending PassFail

$ cucumber --wip --tags @WIP features/

Limiting Tags in FlowWork in progress Done

add movies

add movies

add movies

add movies

add movies

add movies

add movies

editmovies

add movies

add movies

add movies

deletemovies

$ cucumber --tags @WIP:3 features/

Further reading

• http://cukes.info

• http://wiki.github.com/aslakhellesoy/cucumber

• http://www.cuke4ninja.com/

• http://blog.josephwilk.net

Thanks!Joseph Wilk

@josephwilkhttp://blog.josephwilk.net