WHO WILL TEST YOUR TESTS ?

Yahya Poonawala @meetykp

Priti Biyani @pritibiyani

2https://flic.kr/p/9uZhDS 2

TESTSCode works

Reduce Bugs

Reducecost of change

AllowRefactoring

Faster development

Reduce fear

3

NOW

WHEN YOU HEAR ….

4

Oh, it failed?

Don’t worry. Just re-trigger it!

5

Oh, it failed?

Strange; it passed on my machine!

6

Oh, it failed?

Run it on agent 007!

It’s an agent issue!

7

Oh, it failed?

I know. It fails at 12AM!

8

9

Now, again,

WHY DO WE WRITE TESTS?

1

0

Tests prove that the code actually works

Tests prove that the code

sometimes works

1

1

Tests reduce bugs

Tests itself might have bugs

1

2

Tests reduce the cost of change

Change? Are you kidding?

1

3

Tests helps in faster development.

Testing is slowing me down

1

4

NON-DETERMINISTIC

TESTS

A.K.A “Flaky Tests”

1

5

1

6http://www.trovatten.com/long-game/

1

7

1

8

http://tom-doherty.com/wp-content/uploads/2014/02/craigslist-waste-time.jpg

1

9

ILLUSION OF SUCCESS!

2

0

ILLUSION OF FAILURE!

2

1

2

2

2

3

DO NOT DELETE YOUR

FLAKY TESTS!!

2

4

CAUSES OF FLAKINESS

2

5

Thread 1::

if( name is not Yahya ) {

…

fputs( name, file )

…

}

Thread 2::

Initialize name to Yahya

CONCURRENCY

2

7

Initialize lock;

Thread 1::

Acquire lock

if( name is not Yahya) {

. . .

fputs( name, file )

. . .

}

Release lock

Thread 2::

Acquire lock

Initialize name to Yahya

Release lock

TIME

2

9

http://imgs.xkcd.com/comics/bug.png

EXTERNAL SYSTEM CALLS

3

0

UNORDERED COLLECTIONS

3

1

3

2https://www.vardot.com/sites/default/files/2.jpg

# pseudo-code

makeAsyncCall;

sleep(2000);

readResponse;

ASYNCHRONOUS

WAITING

3

4

make asynchronous call;

While (response received?)

if timeout then

throw Test Timeout Exception

else

sleep(10)

. . .

read response

RESOURCE LEAKS

3

6

# //pseudo-code

FILE_NAME = “content.txt”

// Test 1:

FileReader.read(FILE_NAME).to be empty

// Test2: Write to the file

data = “XP Conf 2015”

FileWriter.write(FILE_NAME, data)

FileReader.read(FILE_NAME) == data

// Test 3: Updates the file

append_data = “Schedule for talk”

complete_data = “XP Conf 2015. Schedule for talk ”

FileWrite.append(FILE_NAME, append_data)

FileReader.read(FILE_NAME) == complete_data

TEST ORDER DEPENDENCY

3

8

INPUT / OUTPUT

3

9

4

0

WHAT’S YOUR STRATEGY?

4

1

STOP CALLING YOUR BUILD

FLAKY!!

4

2

Identify Flaky tests

Quarantine

Plan Fix Tests

Monitor

4

3

IT’S NOT ALWAYS YOUR

TESTS

4

4

DO NOT GET

OVERWHELMED BY FAILING

TESTS

4

5

REMEMBER!

4

6

LOVE YOUR TEST CODE !

4

7

There is no such thing as

FLAKY TEST!

If its flaky, its not a test !4

8

AND THE FUTURE…

4

9

REFERENCES

• An Empirical Analysis of Flaky tests

• Eradicating Non-Determinism in Tests – Martin

Fowler

• No more flaky tests on the GO team - Pavan

Sudarshan

• Top 12 reasons to write unit tests

• http://tech.just-eat.com/2015/03/30/how-to-fix-

flaky-tests/

5

0

THANK YOU

yahya.poonawala@gmail.com

priti.biyani6@gmail.com