IBM Labs in Haifa

Automatic Noise Simulation in UDP

Eitan Farchi, Yoel Krasny, Yarden Nir

IBM Labs in Haifa

Outline

� UDP transport

� Difficulties in testing UDP based application

� ConTest/UDP – Automatic noise simulation in UDP

� Comparing ConTest/UDP to white box testing in testing of group communication

IBM Labs in Haifa

� Transport protocols over IP

� TCP� Connection based� Reliable� Analogous to making a telephone call� Usage examples: FTP, HTTP

� UDP� Not connection based� Unreliable� Analogous to sending a letter� Usage examples: Ping, Multicast, Stocks Quote server

� Java programs use TCP or UDP through the java.net package

UDP VS TCP

Java.netTransport(TCP/UDP)Network

(IP)Link

(device driver)

IBM Labs in Haifa

Link

IP

UDP

Application

Java

Link

IP

UDP

Application

Java

NetworkNetwork

1

Unreliable flow in UDP – Losing messages

AB

IBM Labs in Haifa

Link

IP

UDP

Application

Java

Link

IP

UDP

Application

Java

NetworkNetwork

Unreliable flow in UDP – Duplicating messages

22

2 2

2

AB

IBM Labs in Haifa

Link

IP

UDP

Application

Java

Link

IP

UDP

Application

Java

NetworkNetwork

Unreliable flow in UDP – Changing the order

44

2 42 3

3

4334

3

AB

IBM Labs in Haifa

Link

IP

UDP

Application

Java

Link

IP

UDP

Application

Java

NetworkNetwork

Unreliable flow in UDP – Delaying messages

55

2 42 3 5

AB

IBM Labs in Haifa

Testing UDP based applications

� A UDP based application must handle network noises�For example – Any reliable component on top of UDP

� Testing scenarios of network noise is compulsory

� The probability to encounter such scenarios during testing in normal environment is low

IBM Labs in Haifa

Automatic Simulation of Noise by ConTest/UDP

� ConTest/UDP addresses the difficulties in testing UDP applications

� ConTest/UDP simulates UDP noise and network failures

� ConTest/UDP creates an intermediary layer above the Java API

� ConTest/UDP is part of the ConTest suite, that aims to increase the probability of exposing concurrent bugs

IBM Labs in Haifa

ConTest/UDP Options

� Remote Nodes – apply noise on messages from/to certain nodes

� Direction – incoming and/or outgoing messages

� Mode �No noise�Delay – packets are accumulated by the tool�Block – packets are lost�Random noise – packets are tampered randomly

� Strength – determines the random noise level

IBM Labs in Haifa

Mode = DelayDirection = Out

Link

IP

UDP

Application

Java

Tool

Link

IP

UDP

Application

Java

ConTest/UDP

Link

IP

UDP

Java

Tool

Link

IP

UDP

Application

Java

ConTest/UDP

Delaying messages with ConTest/UDP

NetworkNetwork

123

AB

IBM Labs in Haifa

Mode = No-Noise

Link

IP

UDP

Application

Java

Tool

Link

IP

UDP

Application

Java

Link

IP

UDP

Java

Link

IP

UDP

Application

Java

Delaying messages with ConTest/UDP

NetworkNetwork

123

1 2 3 4

4

AB

ConTest/UDP ConTest/UDP

IBM Labs in Haifa

Mode = BlockDirection = Out

Link

IP

UDP

Application

Java

Tool

Link

IP

UDP

Application

Java

Link

IP

UDP

Java

Link

IP

UDP

Application

Java

Losing messages with ConTest/UDP

NetworkNetwork

123 XXX

AB

ConTest/UDP ConTest/UDP

IBM Labs in Haifa

Mode = No-Noise

Link

IP

UDP

Application

Java

Tool

Link

IP

UDP

Application

Java

Link

IP

UDP

Java

Link

IP

UDP

Application

Java

Losing messages with ConTest/UDP

NetworkNetwork

123 XXX

4

4

AB

ConTest/UDP ConTest/UDP

IBM Labs in Haifa

Outline

� UDP transport

� Difficulties in testing UDP based application

� ConTest/UDP – Automatic noise simulation in UDP

� Comparing ConTest/UDP to white box testing of a group communication application

IBM Labs in Haifa

� Multi-point to multi-point communication for a dynamic group of processes

� Supplies a group membership service that tracks the set of groupmembers – view

� Supports various guarantees for its messages delivery:� FIFO delivery� Virtual Synchrony� Total Order

Group Communication

IBM Labs in Haifa

� Multi-point to multi-point communication for a dynamic group of processes.

� Supplies a group membership service that tracks the set of groupmembers – view

� Supports various guarantees for its messages delivery:� FIFO delivery� Virtual Synchrony� Total Order

Group Communication

A

BC

View Id 1{A}View Id 2{A, B}View Id 3{A, B, C}

IBM Labs in Haifa

Transport layer

Membership Layer� VRI – Java implementation based on Clue (AS400)

� All layers support a generic interface

� Every layer is responsible for a certain taskVirtual Synchrony

Application Layer

Versatile Replication Infrastructure

IBM Labs in Haifa

� The Transport Layer provides reliable Multicast delivery over UDP

� Testing the reliability is possible using Contest/UDP and a Verification Layer

� Running with the Random mode provides powerful automatic testing

Testing the reliability of the Transport Layer

Transport Layer

Virtual Synchrony

Application Layer

Membership Layer

ConTest/UDP

Verification Layer

IBM Labs in Haifa

VRI

UDP

D

VRI

UDP

C

VRI

UDP

B

VRI

UDP

A

{A, B, C, D}

{A, B, C, D} {A, B, C, D}

{A, B, C, D}

Network connection is broken

X

Testing partitions – required scenario

IBM Labs in Haifa

VRI

UDP

D

VRI

UDP

C

VRI

UDP

B

VRI

UDP

A

{A, B, C, D}

{A, B, C, D} {A, B, C, D}

{A, B, C, D}

X

After some time the group should be

partitioned

Testing partitions – required scenario

{A, B}

{A, B}

{C, D}

{C, D}

IBM Labs in Haifa

VRI

ConTest

UDP

B

VRI

ConTest

UDP

C

VRI

ConTest

UDP

D

VRI

ConTest

UDP

A

{A, B, C, D}

{A, B, C, D}

{A, B, C, D}

{A, B, C, D}

Connection is brokenBlock

(C, D)

Block(C, D)

Block(A, B)

Block(A, B)

Testing partitions – simulated scenario

IBM Labs in Haifa

{A, B, C, D} {A, B, C, D}{A, B} {C, D}

After some time the group should be

partitioned

{A, B, C, D} {A, B, C, D}{A, B} {C, D}

Testing partitions – simulated scenario

VRI

ConTest

UDP

A

VRI

ConTest

UDP

B

VRI

ConTest

UDP

C

VRI

ConTest

UDP

D

IBM Labs in Haifa

� Follows the white box testing strategy

� Responsible for injecting noise into the VRI stack

� Implements the generic interface of a layer in the stack

The Manipulator Layer

Transport Layer

Virtual Synchrony

Application Layer

Membership Layer

Manipulator

Manipulator

IBM Labs in Haifa

� This scenario illustrates that sometimes it is hard to create certain scenario with ConTest/UDP.

� When a new view is established in a certain member C before another member B, C should queue view messages from B until the new view is established

View message synchronization

IBM Labs in Haifa

NewView

VRI

Transport

A

NewView

View N{A, B, C, D}

View N{A, B, C, D}

View N{A, B, C, D}

View N{A, B, C, D}

View message synchronization – required scenario

VRI

Transport

B

VRI

Transport

C

VRI

Transport

DX

IBM Labs in Haifa

View N{A, B, C, D}

View N{A, B, C, D}

View N+1{A, B, C}

View N+1{A, B, C}

VRI

Transport

A

VRI

Transport

B

VRI

Transport

D

View N{A, B, C, D}

View N{A, B, C, D}

NewView

View message synchronization – required scenario

VRI

Transport

C

X

IBM Labs in Haifa

VRI

Transport

C

View N{A, B, C, D}

View N{A, B, C, D}

View N+1{A, B, C}

View N+1{A, B, C}

VRI

Transport

A

VRI

Transport

B

NewView

View N{A, B, C, D}

11

1

1

View message synchronization – required scenario

VRI

Transport

D

View N{A, B, C, D}

X

IBM Labs in Haifa

View N{A, B, C, D}

View N+1{A, B, C}

VRI

Transport

C

View N+1{A, B, C}

View N+1{A, B, C}

NewView

NewView

1

1

View message synchronization – required scenario

VRI

Transport

A1

VRI

Transport

B

VRI

Transport

D

View N{A, B, C, D}

X

IBM Labs in Haifa

VRI

Transport

B

VRI

Transport

C

NewView

NewView

View N{A, B, C, D}

View N{A, B, C, D}

View N{A, B, C, D}

View message synchronization – Simulated scenario

Manip

View N{A, B, C, D}

VRI

Transport

DX

VRI

Transport

A

IBM Labs in Haifa

View N{A, B, C, D}

View N+1{A, B, C}

VRI

Transport

B

View N{A, B, C, D}

View N+1{A, B, C}

VRI

Transport

A

VRI

Transport

C

View N{A, B, C, D}

View message synchronization – Simulated scenario

Manip

Delay New View Call

View N{A, B, C, D}

VRI

Transport

DX

IBM Labs in Haifa

VRI

Transport

C

Manip

View N{A, B, C, D}

View N+1{A, B, C}

VRI

Transport

A

View N{A, B, C, D}

View N+1{A, B, C}

View N{A, B, C, D}

11

1

1

View message synchronization – Simulated scenario

VRI

Transport

B

View N{A, B, C, D}

VRI

Transport

DX

IBM Labs in Haifa

View N{A, B, C, D}

View N{A, B, C, D}

View N+1{A, B, C}

View N+1{A, B, C}

VRI

Transport

A

VRI

Transport

B

Transport

C

View N{A, B, C, D}

1

1

1

View message synchronization – Simulated scenario

Manip

VRIDelay Over

View N{A, B, C, D}

VRI

Transport

DX

IBM Labs in Haifa

� ConTest/UDP - provides a powerful utility to test UDP-based programs

� ConTest/UDP - easy to use with no effect on the application code

� ConTest/UDP - not always suitable to test application specific scenarios�Such scenarios require specifically tailored testing code

Conclusions