Ngo Hoang Minh

TEST DESIGN WITH ACTION-

BASED TESTING

METHODOLOGY

Who is your speaker?

2

• Software testing company, around since 1994

• Testing and test automation services:– consultancy, training

– test development and automation services

– "test integrated" development services

• Products:– TestArchitect™, TestArchitect for Visual Studio™

– integrating test development with test management and automation

– based on modularized keyword-driven testing

• LogiGear Magazine:– themed issues, non-commercial

LogiGear Corporation

Background in computer science, management

Since 2009 focusing on automated testing keywords, agile testing

Ngo Hoang Minh

3

Test Design With Action-Based

4

• Introduction

• Action Based Testing, Test Modules

• Success factors of test design (for automation)

• Test case design

• Action design

• Final

Agenda

5

Introduction

Testing Under Pressure

6

specification development test

DEADLINE

Testing Under Pressure –

"Shift Left"

7

specification development test

DEADLINE

Develop tests in time:• Test design• Auditing, acceptance• Preparations• Automation

8

Action Based Testing

Risks of keyword and scenario approaches

9

• Often seen as silver bullet, complications are underestimated– often treated as a technical "trick"

– testers can get squeezed and marginalized

• developers and users dictating tests

• automation engineers dictating actions

– testers can end up with an automation responsibility, thus becoming pseudo programmers

• The method needs understanding and experience to be successful– pitfalls are many, and can have a negative effect on the outcome

• Lack of method and structure can risk manageability– maintainability not as good as hoped

– results can be disappointing, approach will be blamed

Action Based Testing (ABT)

10

• Uses the keyword based "actions" as basis for a method

– Covers test management, test development and automation

– With a large focus on test design as the main driver for automation

success

– Method is specific, but concepts are generic

• The central product in ABT is the "Test Module", not the test

case

– Like chapters in a book• test cases are part of the test modules,

– Test development is seen as having both analytical and creative aspects

– Developed as spread sheets, external from the automation

– Each test module is a separate (mini) project, each test module can

involve different stake holders

Test Module

11

Test Module

Initial - setup

Objectives

Test cases

Final - cleanup

Test Modules versus Test Cases

12

• The test module is a bigger unit in the test design– Easier to identify– A chapter rather than a paragraph

• Better flow of execution– Each test case can set up for the next one– Keep test modules independent, test cases can be dependent

• Test cases become creative output, rather than stifling input– Avoids having to define all test cases at once early in the process

• Clear scope helps to identify cases, actions and checks

13

Test Design Success Factors

Why Better Test Design?

14

• Better test design can improve quality of test

– Many tests are often quite "mechanical" now, no surprises

– One to one related to specifications, user stories or requirements,

which often is ok, but lacks aggression

– No combinations, no unexpected situations, lame and boring

– Such tests have a hard time finding (interesting) bugs

• Better test design can give (much) better automation

– Unneeded details are left out of tests

– Avoiding "over checking“

– Limit the impact of system changes on tests, making such impact

more manageable

The Three “Holy Grails” of Test

Design

15

• Metaphor to depict three main steps in test design

• Using "grail" to illustrate that there is no single perfect solution, but that it matters to pay attention

Organization of tests into test modules

Right approach for each test module

Proper level of detail in the test specification

What's the trick...

16

What's the trick...

17

• Have or acquire facilities to store and organize your content

• Select your stuff

• Decide where to put what– assign and label the shelves

• Put it there

• If the organization is not sufficient anymore, add to it or change it

Properties of a good Breakdown

18

Reflects the level of tests

Well differentiated and clear in scope

Balanced in size and amount

Modules mutually independent

Fitting the priorities and planning of the project

Breakdown Criteria

19

• Common Criteria

– Functionality (customers, finances, management information, UI, ...)

– Architecture of the system under test (client, server, protocol, sub systems,

components, modules, ...)

– Kind of test (navigation flow, negative tests, response time, ...)

• Additional Criteria

– Stakeholders (like "Accounting", "Compliance", "HR", ...)

– Complexity of the test (put complex tests in separate modules)

– Execution aspects (special hardware, multi-station, ...)

– Project planning (availability of information, timelines, sprints, ...)

– Risks involved (extra test modules for high risk areas)

– Ambition level (smoke test, regression, aggressive, …)

Approach 1: Workshop

20

• Gather a meeting with relevant participants– test developers

– domain experts

– automation engineer (focus on efficiency of automation)

– experienced moderator

– also consider: developers, managers

• If necessary, provide training of participants before the discussion

• I prefer this approach, in particular to start off

Approach 2: Design and Feedback

21

• One or two experienced test designers create a

first draft

• The draft is delivered/discussed to relevant

parties

• Ask the parties to verify:

1. Structure: does it make sense

2. Completeness: are all relevant areas covered

• Based on feedback, further modify the design

22

Test Case Design

Grail 2: Approach per Test Module

23

When to develop: do we have enough information? When to execute: make sure lower level stuff working first

Plan the test module:

Do an intake: understand what is needed and devise an approach Analyze requirements, formulate "test objectives", create tests

Process :

See the test development as a "learning process", about the business domain, the application structure, the interaction, etc

Talk about your tests, make them strong

Exploratory approach:

Users, subject matter experts, developers, auditors, etc

Identify stakeholders and their involvement:

Users, subject matter experts, developers, auditors, etc

Identify stakeholders and their involvement:

Clarify your objectives with your tests . . .

24

requirement,specification,

…test case

requirement,specification,

…test objective test case

direct relation indirect relation via a test objective

Linking through test objectives can help easier traceability:

...TO-3.51 The exit date must be after the entry date...

test objective TO-3.51

name entry date exit date

enter employment Bill Goodfellow 2016-10-02 2016-10-01check error message The exit date must be after the entry date.

Test Objectives

25

• Keep test objectives short and simple

• Focus on what to test, not how

• Split longer texts into atomic sentences

• Typically test objectives will be like:

– cause and effect ("clicking clear clears all fields")

– condition and effect ("if all fields filled, 'ok' is enabled")

Eye on the ball, Scope

26

• Always know the scope of the test module

• The scope should be unambiguous

• The scope determines many things:– what the test objectives are

– which test cases to expect

– what level of actions to use

– what the checks are about and which events should generate a warning or error (if a “lower” functionality is wrong)

Use of "Anti-patterns"

27

• Informal way to classify typical test design problems

• Use with care, can come across offensive

• The point of view in the following list is automation: test

design choices that can be counter-productive to

automation

• However, lack of good organization can also effect the

quality of manual tests

28

Action Design

Grail 3: Specification Level, choosing

actions

29

• Scope of the test determines the action level

• As high level as appropriate, as little arguments as possible

• Clear names for actions

• Avoid "engineer" styles for names of actions and arguments

– tests are not source code

– like no spaces, uppercase, camel-case or underlines

• Manage and document the Actions

• Try to treat as a "by-product of test design"

Low-level, high-level, mid-level

actions

30

enter customer

enter address fields

enter select set . . .. . .

"Low level": detailed interaction with the UI (or API)- generic, do not show any functional or business logic

- examples: "click", "expand tree node", "select menu"

"Mid level": common sequences at a more

detailed application level- usually to wrap a form or dialog

- for use in high level actions

- greatly enhance maintainability

- example: "enter address fields"

"High level": a business domain operation or check on the

application under testhide the interaction

examples: "enter customer", "rent car", "check balance"

Use the right level actions

31

Low level of UI interaction detail makes sense only with the module

scope is to test the UI

window tree tree item path

click tree item main projects /Projects/Drill Assembly

window list item

check list item exists main tasks Plan of Approach

Better to have a business level action and hide the details in an action definition:

But do show details when it matters. The example below is too high level, requires drill down into action definition to understand what is tested.

32

Finally

Takeaways

33

• Test design is a major contributor to automation success,

often more than technical prowess

• Domain language approaches like Actions and BDD allow

for efficient communication and driving of automation

• Test modules can help organize the tests, and focus their

scopes

• Focusing tests, checks and actions on a clear and

differentiated scope will make for better tests, but also

better automation

© 2014 HCMC Software Testing Club

THANK YOU