Upload
reed-pretty
View
219
Download
3
Tags:
Embed Size (px)
Citation preview
Acceptance Testing vs. Unit Testing:
A Developer’s Perspective
Owen Rogers
Rob Styles
2
What are Acceptance Tests?
• Tests owned and defined by the customer to verify that a story is complete and correct.
3
Why we use Acceptance Tests:
• Increase team confidence that the system is correct.• Structures communication between customer and
developers.• Help customer think through requirements.• Informs developer estimates.• Indicates how to split a story.• Concrete definition of story completion (“knowing when
we’re done”).• Early involvement of QA.• Automated regression testing for free.
4
Why other people don’t:
• Customer participation.• Management buy-in.• QA support.• More tests for developers to implement.• No common framework.
5
ATs and UTs are the same though, right?
• Both can and should be written first, before implementation.
• Both should be automated by containing assertions for validating expected results.
• Both typically use fixture setup and teardown code to run before and after the execution of the tests.
• Both can be implemented using an xUnit framework.
6
Difference: ATs are written by the customer.
• UTs only need to be read and understood by other developers.
• ATs must be defined in a language familiar to the customer.
• ATs require a customer meta-language:• Generic enough to capture requirements• Abstract enough to be maintainable• Simple enough to be easily understood
7
Example Acceptance Tests.
fit.ActionFixture
start eg.Page
enter location http://google.com
check title Google
enter link Jobs
check title About Google
enter link Press
enter link Review
check title Google Press Room
FIT Acceptance Test: FAT Acceptance Test:
• Request: “http://google.com”
• Check title: “Google”
• Click link: “Jobs”
• Check title: “About Google”
• Click link: “Press”
• Click link: “Review”
• Check title: “Google Press Room”
8
Difference: Customers don’t use an IDE.
• UTs are implemented in the language of development using an IDE.
• ATs should be defined using a tool accessible to the customer (Excel, web page, wiki).
• ATs must be retrieved from the tool and parsed.• Ideally, the customer should be able to run the ATs from
this environment.
9
Interacting with customer tool.
Excel
Wiki
AT Framework
Extract Test (.html)
Extract Test (.csv)
Execute TestWrite Test
10
Difference: ATs are not directly executable.
• UTs invoke code-under-test directly.• AT definition is abstracted from its implementation.• ATs specify what needs to be done, but not how to do it.• ATs require an implementation layer to invoke the
application.• AT definitions must be interpreted – parsed and mapped
onto corresponding implementation.• Tests must be definable and executable prior to
implementation.
11
Test Definition / Test Implementation
• Request: “http://google.com”
• Check title: “Google”
• Click link: “Jobs”
• Check title: “About Google”
• Click link: “Press”
• Click link: “Review”
• Check title: “Google Press Room”
Interpreter
public void Request(string url) { }
public void CheckTitle(string title) { }
public void ClickLink(string link) { }
Test: Verify Google Links Class: WebPageFixtureAT Framework
12
Difference: ATs test through the same interface.
• UTs test at level of a single unit (class).• ATs test at level of user/client interaction.• All tests go through a common gateway (Http, Soap, UI
controls or application façade)• Considerable opportunity for reuse across test
implementations.• ATs need a framework in the technology of the interface
it drives: Web, GUI, Web Service, Messaging etc.
13
Testing through the application gateway.
Application G
ateway
ApplicationDefinition Implementation
Application G
ateway T
est Client
Acceptance T
est Interpreter
ATD
ATD
ATimpl
ATimpl
ATimpl
ATD
14
Difference: ATs test the Users’ Experience.
• UTs should test all application logic in detail.• ATs encourage the customer to consider the all aspects
of user experience.• ATs should test:
• Interaction & Flow• Performance• Error Handling• Security
• ATs don’t test:• Usability• Look & Feel
15
Difference: ATs are end-to-end.
• UTs test single unit in isolation.• ATs test interaction of all layers in the system.• ATs require external systems to be operating correctly.
16
Difference: ATs are process-driven.
• Each UT typically tests a single public method.• ATs consist of a sequence of steps simulating a client
interacting with the system.• ATs may test the same steps repeatedly in different
variations with different data.• Considerable overlap across different ATs.• Single point of failure will break multiple ATs.
17
Difference: ATs have serious side-effects.
• For UTs, setup and teardown code should be common only to a single fixture.
• ATs interact extensively with external systems and the environment.
• ATs typically require separate setup and teardown per test as different tests will have different side-effects.
• Setup and teardown routines are commonly shared across tests and even across stories.
18
Difference: ATs suffer concurrency issues.
• UTs should be largely independent of their environment.• ATs require that the environment is setup as expected
by the tests. • Problems can result from different users running ATs
concurrently.• Ensure sufficient variation in the data used for each test
to prevent collisions.
19
Difference: Failing ATs don’t break the build.
• UTs must pass when checked in – failure breaks the build.
• ATs may be written at the start of a story and may not pass until the very end of the iteration.
• We mark ATs as [Ignore] until they have been fully implemented.
20
Difference: ATs don’t go quietly
• UTs are designed to run silently, with all verification encoded in the assertions.
• Customer requires visual feedback to give reassurance that test does what is expected.
• ATs need to provide step-by-step visibility of the contents and results of running the tests.
21
Difference: ATs are useful outside of dev environment.
• UTs are primarily useful to developers for ensuring the code does what was intended of it.
• ATs on the other hand can provide a good assessment of the overall health of the system.
• This makes them useful for assuring deployments to testing environments are sound.
• To do this usually requires environmental parameters to be extracted to a configurable location.
22
Difference: ATs mock out systems, not units.
• Introducing mocks for dependencies in UTs helps to decouple the code.
• We often have the same needs in ATs to mock third party interfaces that are not always available.
• Mocking is very hard because the AT is outside of the application context so a specific interface for the system being mocked has to be created.
• Mocking is still important to maintain customer’s trust in the solution.
• Introduce mocks to help simulate failure conditions.
Implementation
Mo
ck
Application Interface
Unit TestFixture
Setup
Teardown
UnitImplementation
impl
impl
impl
impl
impl
impl
UT
UT
UT
UT
UT
UT
AcceptanceTest Definition
ATimpl
ATD
ATimpl
ATD
ATimpl
ATD
setup
teardown
setup
teardown
setup
teardown
Acceptance T
est Client