White Box Testing: It’s Not Just for Developers Any More

BT6Session6/9/1611:30AM

WhiteBoxTesting:It'sNotJustforDevelopersAnyMore

Presentedby:

RobertVanderwall

UltimateSoftware

Broughttoyouby:

350CorporateWay,Suite400,OrangePark,FL32073888---268---8770··[email protected]://www.techwell.com/

RobertVanderwallUltimateSoftwareAsatestarchitectatUltimateSoftware,RobertVanderwallprovidesguidance,training,andtechniquestoengineersperformingtestingactivities.PriortojoiningUltimateSoftware,Robertwasahardwareengineer,asoftwareprogrammer,asoftwaretestengineer,andaneducator.Hehasworkedonprojectsrangingfromtinyembeddedsystemstowebscaleapplications.Inadditiontomanyyearsofexperienceinindustry,RobertholdsaPh.D.incomputerscience,specializinginsoftwaretesting.

5/10/16

1

June 5–10, 2016 | Las Vegas, NV Robert Vanderwall 1

Unit Tes)ng: Not just for developers anymore

Robert Vanderwall, Ul)mate So?ware Group

June 5–10, 2016 | Las Vegas, NV Robert Vanderwall

Agenda

•  Current Trends •  Test Pyramid •  Code Patterns •  Metrics •  Adding Value •  Wrap-up

5/10/16

2


Current Trends

•  Shift left •  Death of Testing •  Deep Collaboration


Shift Left

Many activities are moving earlier, including testing.

•  Testing begins at inception. •  Testing is an integral part of the

development process.

Especially

5/10/16

3


Death of Testing

Several companies have adopted a “No QA” approach

•  Engineers are becoming hybrids. •  Generalists with area of specialization. •  Developers do their own testing. •  Ownership through to production.


Deep Collaboration

• Cross functional teams

• Continuous conversations

• Group ownership of quality

5/10/16

4


Quality Engineer

•  Understand ways to stay relevant

•  Enhance testing skills

•  Test with an understanding of what information developers would find useful


Software Developer

•  Better appreciation for the value of exploratory testing

•  Boost ability to deliver value by collaborating well with testers

•  Improve ability to turn testing observations into useful ideas for unit testing

5/10/16

5


Test Pyramid

•  Most of the tests are unit tests

•  Fewer tests are integra)on tests

•  Least number of tests at system level


Purpose

•  Unit tests are cheap and fast, they provide fast feedback.

•  Integration tests verify interactions between components.

•  System tests verify value is realized.

5/10/16

6


Inverted Pyramid

•  Many UI tests but few unit tests.

•  Automation effort without development support.

•  Lack of development practices for unit testing.


Test Space Needle

•  Lots of unit tests, plenty of UI tests. •  Almost no integration tests.

•  Overconfidence in unit testing. •  Ends up with big-bang integration.

5/10/16

7


Coding Patterns

•  A few easy patterns •  A few not so easy patterns •  Pointers


Easy Pattern - Sequence

What could go wrong? •  Operator may be wrong

•  z = x * y •  Bad test x =2, y=2 •  No discriminatory power

def seq(x,y) : z = x + y return z

5/10/16

8


Test Pattern

X Y Expected Output def seq(x,y) : z = x + y return z

0 07 0

0 9

6 5

Are these tests any good? What is the business need being sa)sfied? Total exemp)ons. these may be fine Total sales. these are not representa)ve.

3 8

Number, 0, >


Test Pattern - combinations

X Y Expected Output def seq(x,y) : z = x + y return z

0 07 0

0 9

6 5

What is the expected output? 0, 7, 9, 11, 11 ? Did you get the expecta)ons from the code? Could the tests ever fail?

3 8

5/10/16

9


Easy Pattern - Condition

What could go wrong? •  Relation may be wrong

•  x < 10, x >= 10 •  Value may be wrong

•  x > 20

def condition (x) : if x > 10: z = x + 10 else: z = x -10 return z


Test Pattern – Boundary tests

X Expected Output

-‐10

0

9

10

11

20

Edge -‐1, Edge, edge +1, mid-‐range

def condition (x) : if x > 10: z = x + 10 else: z = x -10 return z

5/10/16

10


Test Pattern – Boundary tests

X Y Expected Output

0 0

10 10

10 11

11 10

8 -‐8

-‐5 -‐6

def condition (x,y) : if x > y: z = x - y else: z = x + y return z

Edge, edge +1, edge -‐1


Easy Pattern - Loop

What could go wrong? •  Off-by-one

•  Didn’t iniPalize properly •  Didn’t finalize properly

def loop(L): for l in L: do_something(l) return results

5/10/16

11


Test Pattern – Loop

L Expected Output

empty

1 item

2 items

Typical number of items

Max number of items

Max+1 items

def loop(L): for l in L: do_something(l) return results


Lessons

•  Test inputs should reflect expected uses. •  Expected output should come from business

function NOT from the code. •  Try <, =, > for sequences. •  Try edge-1, edge, edge+1, mid-range for boundary. •  Try 0, 1,2, t, max. max+1 for loops.

5/10/16

12


References

https://msdn.microsoft.com/en-us/library/cc514239.aspx http://www.exampler.com/testing-com/test-patterns/patterns/index.html


Metrics

If you can’t measure it, you can’t improve it

– Peter Drucker

•  Code Coverage •  Code Complexity •  Mutation Score

5/10/16

13


Code Coverage

Most IDEs have a way to measure coverage. Look for high coverage, esp:

critical areas, complicated areas. Statement coverage first, then branch coverage.


Code Coverage

If a line of code is not executed, it is not tested. If a line of code is not tested, it may have a bug. Inspections help, a lot!

5/10/16

14


Code Coverage

1. def condition (items, y) : 2. for item in items: 3. if item.x > y: 4. z += a(item) 5. else: 6. z += b(item) 7. return z

How many statements? How many tests to cover them?


Code Coverage

1. def condition (items, y) : 2. for item in items: 3. if item.x > y: 4. z += a(item) 5. else: 6. z += b(item) 7. return z

How many branches? How many tests to cover them? 1

2

3

4 6

7

5/10/16

15


Coverage caveats •  I noPced that the good developers drink lots of coffee.

•  SO •  I have all developers drink 6 cups a

day.

•  Surely, I will get beber code, right?

•  NO: CorrelaPon is not causality.


Code Coverage Caution

Low coverage tells you that some code has not been tested. High coverage tells you nothing! Therefore, it should be a guidepost, NOT be a goal. hbp://www.exampler.com/tesPng-‐com/wriPngs/coverage.pdf

5/10/16

16


Code Complexity

Many IDEs can measure Cyclomatic Complexity. 1-5 : Simple and easily understandable. 6-15: Rather complicated, better test it a lot. > 15: Quite complicated, probably hard to test and likely a haven for many bugs.


Mutation Testing

If I change the code, does a test fail? If not, then there are ways the code can be wrong

that the tests will not find. A Few tools available, but manually altering the code works well,

esp. for JS.

5/10/16

17


X Y Expected Output

0 0

10 10

10 11

11 10

8 -‐8

-‐5 -‐6

def condition (x,y) : if x > y: z = x - y else: z = x + y return z


Complexity Cautions

If every module has very low complexity, where is the complex business logic? Not all statements are equally complex.

A complex linq query may show CC=1.

5/10/16

18


CRISP tests

C – Clear purpose R – Repeatable I – Independent S – Simple P – Performant

DRY – Don’t repeat yourself Tests may have duplicate code


Adding Value

In a left shifted highly collaborative environment, how can we add value? We just saw an overview of the test pyramid and a few testing patterns. How can we make that work for us?

5/10/16

19


Leverage Left Shift

• Review unit tests for coverage, patterns, etc.

• Pull together the tester-facing discipline of Exploratory testing and the developer-facing discipline of Unit testing.

• Provide insights into the product and code through exploring and leverage these insights via unit tesPng.


Leverage collaboration

• Become more helpful for developers: •  CommunicaPng bugs found •  AND ideas from exploraPon that could be used to improve unit tests

5/10/16

20


Leverage Automation

• Combine the forces of machines and humans.

• Machines are parPcularly well suited for exploring large computaPonal space. • What they can’t do is have any insight or creaPve ideas.
