Testing ‘slow flows’ fast

Eurostar 2010

Copenhagen

Automated End-2-End testing using interrupts

Agenda

2

Hi there !

The Challenge

The Quest for a solution

The Solution

Demonstration

Conclusion

Q & A

Agenda

3

Hi there !

The Challenge

The Quest for a solution

The Solution

Demonstration

Conclusion

Q & A

The Challenge...

4

Suppose you have 10.000 orders placed every day in your online store. Every order results in several steps: Checking the available quantities in the warehouse,

processing the command,

ordering things that are not readily available,

making invoices,

sending confirmations to the customer,

checking money transfers,

etc...

All these processes are connected and use a central database

A step can take up several minutes/hours of processing time

Let's visualize the workflow

5

Process AOnline purchase

10 mins

Process BCheck Supply

Max 3 hrs

Process CCheck client status

Max 20 mins

Process DOrder missing items

Max 2 hrs

Process ESend items

5 mins

Process FInvoice10 mins

Max total time: 3h45minMin amount of time: 55 min

Processes B,C and D are other systems using the same Db’s but working

independently. These three processes take minimum

15 min each to complete.

Processes A, E and F are on the system under test.

Starting each process is a matter of seconds

Agenda

6

Hi there!

The Challenge

The Quest ...

The Solution

Demonstration

Conclusion

Q & A

7

Economical needs

Bigger, better, faster software

Constant evolving software (RAD)

Advantages

More work enjoyment Regression testing = Tedious work

Higher test quality (avoid human errors)

Increased productivity

Extension of test options

Why automate your testing?

Test automation Pitfalls

8

Automating chaos only results in more chaosFirst structure, then tool !

A fool with a tool is still a fool !

A Test automation project is a development project Test tools can test everything Test tools make testing cheaper Testing with a press on a button We want to start test automation on this new

application… Let’s run our tests once with the tool

What is available on Frameworks?

9

What kind of frameworks are there?

0) Plain Record and playback

1) Data driven

2) Modularity driven

3) Keyword driven

4) Model based testing

5) Hybrid testing

A lot of commercial frameworks on the market

Classic Frameworks fail because....

10

Conventional test automation frameworks are:

too much based on ‘online’ input & output

waiting till application responds, then they continue

not capable of dealing with time related checks

Agenda

11

Hi there!

The Challenge

The Quest ...

The Solution

Demonstration

Conclusion

Q & A

Magic ingredient(s)

12

We ‘cut’ the process-flow in subprocesses.

We start off using a ‘standard’ modular, keyword driven and data-driven framework with a modular structure for maintenance.

And we make it ‘interupt-driven’

We add an Action Queue consisting of:

Execution timestamp

Action to perform

Different arguments/parameters

Step 1: Cut your process flow into pieces

13

Business workflow is usually modular

In test scripts, handling different(sub-) processes is mostly modular

Process AOnline purchase

10 mins

Process BCheck Supply

Max 3 hrs

Process CCheck client status

Max 20 mins

Process DOrder missing items

Max 2 hrs

Process ESend items

5 mins

Process FInvoice10 mins

14

Step 2: Built in Interrupt Mechanism

System startup

System initialization

Central system

Scheduler

Context restore

Context save

Task : Process A

Switch

Context restore

Context save

Task: Process B

Context restore

Context save

Task : Process E

Context restore

Context save

Task : Process F

calls

Reporting

Generic functions

Error Handling

15

Add a timestamp check in your scheduler

If Timestamp(x) ≥ Current System Time

Then Start Action(x)

Step 2: Built in Interrupt Mechanism

16

Adapt your functions or modules inside your automation tool to the new mechanism.

Build in a ‘waiting time’ or a ‘check regularly’

Add a line similar to this one at the end of a task (context save)

Add_to_Action_queue(Execution time, Action name,

Argument 1, Argument 2,…)

Step 2: Built in Interrupt Mechanism

Step 3: Add an 'Action Queue'

17

Make sure you have an action queue and add an ‘execution timestamp’ into it.

Action Param 1 Param 2 ….

Add Order 12455 Mr. Smith ….

Add Order 12456 Ms. Simpson ….

Add Order 12457 Mr. Doe …

Execution Timestamp Action Param 1 Param 2 ….

1/12/2010 - 08:00:00 Add Order 12455 Mr. Smith ….

1/12/2010 - 08:01:00 Add Order 12456 Ms. Simpson ….

1/12/2010 - 08:20:30 Add Order 12457 Mr. Doe …

Agenda

18

Hi there!

The Challenge

The Quest ...

The Solution

Let’s Rock ‘n Roll !

Conclusion

Q & A

Demo preparation

19

Start off with your action queue:

Execution Timestamp Action Param 1 Param 2 ….

1/12/2010 - 08:00:00 Add Order 12455 Mr. Smith ….

1/12/2010 - 08:01:00 Add Order 12456 Ms. Simpson ….

1/12/2010 - 08:20:30 Add Order 12457 Mr. Doe …

Demo preparation

20

Check of first action is added in action queue

Execution Timestamp Action Param 1 Param 2 ….

1/12/2010 - 08:00:00 Add Order 12455 Mr. Smith ….

1/12/2010 - 08:01:00 Add Order 12456 Ms. Simpson ….

1/12/2010 - 08:20:30 Add Order 12457 Mr. Doe …

1/12/2010 - 08:10:15 Check Order 12455 Mr. Smith …

Demo preparation

21

Sorted action queue before action 2

Execution Timestamp Action Param 1 Param 2 ….

1/12/2010 - 08:01:00 Add Order 12456 Ms. Simpson ….

1/12/2010 - 08:10:15 Check Order 12455 Mr. Smith …

1/12/2010 - 08:20:30 Add Order 12457 Mr. Doe …

Demo preparation

22

Performing second action in list

Execution Timestamp Action Param 1 Param 2 ….

1/12/2010 - 08:01:00 Add Order 12456 Ms. Simpson ….

1/12/2010 - 08:10:15 Check Order 12455 Mr. Smith …

1/12/2010 - 08:20:30 Add Order 12457 Mr. Doe …

1/12/2010 - 08:11:15 Check Order 12456 Ms. Simpson …

Demo preparation

23

Before start of third action

Execution Timestamp Action Param 1 Param 2 ….

1/12/2010 - 08:10:15 Check Order 12455 Mr. Smith …

1/12/2010 - 08:11:15 Check Order 12456 Ms. Simpson …

1/12/2010 - 08:20:30 Add Order 12457 Mr. Doe …

Demo preparation

24

Start of Second (sub)process for first Order

Execution Timestamp Action Param 1 Param 2 ….

1/12/2010 - 08:10:15 Check Order 12455 Mr. Smith …

1/12/2010 - 08:11:15 Check Order 12456 Ms. Simpson …

1/12/2010 - 08:20:30 Add Order 12457 Mr. Doe …

1/12/2010 - 08:10:30 Start Invoice 12455 Mr. Smith …

Demo

25

Agenda

26

Hi there!

The Challenge

The Quest ...

The Solution

Let’s Rock ‘n Roll !

Conclusion

Q & A

Framework Drawbacks

27

When flows can not be cut into(sub-) processes

Timing accuracy can be tricky

Depending on ‘input-times’

Trade off between ‘Quantity’ and ‘Accuracy’

Needs to be ‘fine-tuned’

Processing will not be the same in different runs

Luckily we ‘log’ the right order of processing

Summary

28

Starting with a‘plain’ record and playback tool

Built on a modular, data-driven and keyword-driven framework

Add ‘time-driven’ to make your framework more versatile

Agenda

29

Hi there!

The Challenge

The Quest ...

The Solution

Let’s Rock ‘n Roll !

Conclusion

Q & A

Thank you !

30