TestNG… and improving use of JUnit

Dec 11, 2008

C. Gordon Huffman

2

Agenda (not chronological)

• TestNG testing framework

• Improving use of JUnit

• Testing integrated classes vs. 1 class

3

What is TestNG?

• Automated testing framework• NG = Next Generation• Similar to JUnit (especially JUnit 4)• Not a JUnit extension (but inspired by JUnit)• Designed to be better than JUnit, especially

when testing integrated classes• Created by Dr. Cédric Beust (of Google)• Open source (http://testng.org)

4

History

• sUnit for Smalltalk (~1998, Kent Beck)

• JUnit (~2000, Kent Beck & Erich Gamma)– Latest 3.x: 3.8.2 (March 2006)

• TestNG (~2004)– Latest: 5.8 (March 2008)

• JUnit 4.x (2006)– Latest: 4.5 (August 2008)

5

TestNG philosophy

• Use more Java and OO features

• Feature-rich (JUnit: simplicity is valued)

• Support testing integrated classes (e.g., by default, don’t create a new test class instance for every test method).

• Separate compile-time test code from run-time configuration/data info.

6

Basic Testing Terms

Test class Class(es) under test(“Production” class)

“CUT” (class under test)“SUT” (system under test)

7

“Rhythm” of an Automated Test

• Setup (most of it in a separate method)

• Exercise (call) production class(es)

• Assertion(s); true => silence; false => fail

• Teardown (if needed, in a separate method)

8

Test class/method (JUnit 3 vs. 4)

// JUnit 3import junit.framework.TestCase;public class MyTestClass extends TestCase { public void testSomething1() throws ... { ... } public void testSomething2() throws ... { ... }}

// JUnit 4import org.junit.Test;public class MyTestClass { @Test public void aTest1() throws ... { ... } @Test public void aTest2() throws ... { ... }}

9

Test class/method (JUnit 4 vs. TestNG)

import org.junit.Test; // JUnit 4... orimport org.testng.annotations.Test; // TestNG

public class MyTestClass { @Test public void aTestMethod() throws ... { ... }}

10

TestNG Assertions

import static org.testng.Assert.*;import org.testng.annotations.Test;

public class MyTest { @Test public void myTestMethod() { // ... Possibly some setup // ... Call production method(s) assertTrue(boolExpression); // ... more assertions }}

11

Assertions: TestNG vs. JUnit 3/4

• JUnit 3 assertions inherited by base TestCase• JUnit 4: similar to TestNG• TestNG assertEquals(…) assertion signatures

are different than JUnit’s:– JUnit: [msg,] expected, actual– TestNG: actual, expected [, msg]

• Can use JUnit 3/4 assertions (library issues)• Can use TestNG’s copy of JUnit 3 assertions

(org.testng.AssertJUnit): migration aid.• JUnit4/TestNG assertions richer than JUnit 3;

each contains features that the other doesn’t.

12

Expected Exceptions

JUnit 4 (can simplify test code)@Test(expected = AThrowable.class)

TestNG (can have more than one exception)@Test(expectedExceptions = AThrowable.class)

Or ...

@Test(expectedExceptions = { T1.class, ... })

13

TestNG “Groups”• Each test method is tagged with any # of groups.

– @Test // no groups– @Test (groups = “group1”)– @Test (groups = { “g1”, “g2”, ... })

• A group therefore contains any # of test methods.• Can also annotate test class this way; test method groups are union

of any explicit test method groups, and any of its test class.• Groups can span classes.• Groups can also be externally defined (TestNG xml configuration

file).• A group is identified by a unique string (don’t use white space).

– There are no pre-defined group names.– E.g., “slow”, “db”, “ui”, “unit”, “integration”, “broken.unknownReason”,

“check-in”, “week-end”, “functional.billing”

14

TestNG Groups (continued)

• TestNG community suggests hierarchical names from more general to less. E.g.:– db.table.CUSTOMER– state.broken.fix-in-progress

• Design group names so that you can select them with prefix patterns.

• Groups complement other features (as we’ll see).

15

Setup/Teardown (JUnit 3/4)

JUnit 3• protected void setUp() throws Exception { ... }• protected void tearDown() throws Exception { ... }

JUnit 4 (also, @After*; annotation package: org.junit)

• @BeforeClass public static void before1() ...• @Before public void before2() ...• Multiple per class (order not defined)• These 4 annotations are “inherited” (order is [kind of] defined).

16

Setup/Teardown (TestNG)

@BeforeMethod

@BeforeClass (no need to be static)

@BeforeGroups ({“group1”, …})

@BeforeTest

@BeforeSuite

• And @After*

17

TestNG run-time concepts

• A org.testng.TestNG instance runs tests.– Invoked by IDE, ant, or on command line.– Run params specified as setters, or in xml.

• Hierarchy of suite/test/class/method.

• A TestNG run executes 1 or more suites.– Can be transparent, e.g., run 1 test method via Eclipse.

18

testng.xml

• Controls test runs

• Optional• File can have any name (.xml suffix advised)

– TestNG creates “testng-failed.xml”, for easy re-run.

• Root: <suite> (then <test>/<classes>/<method>)

• Suite can also point to a list of additional xml suite files.• Some of the available specifications are described next.

19

testng.xml (continued)

• Which potential test methods to run: suite/test: list of java package name patterns to include/exclude; test: list of fully qualified class names to include; within a class, optional (default = include all) list of method names to include/exclude; test: list of group name patterns to include/exclude (and can define new groups for this file with a list of existing group names).

• Parameter name/value pairs: Suite/test: (primitive values) for @Parameters (test values override suite values).

• JUnit semantics: test-level flag (default: false); migration aid.

20

Disable Tests

• TestNG– @Test(enabled = false)– Add to a group which is excluded– Exclude in other ways in testng.xml

• JUnit 4: @Ignore, @Ignore(“reason”)– Can also annotate test class.– Runners report # ignored

21

Test Timeouts

TestNG

• @Test(timeout = 1000)

• testng.xml <suite|test> time-out attribute

JUnit 4

• @Test(timeout = 1000)

22

The Costs of Testing

• Time, money, schedule, production scope.• Doubled (very roughly) code base.

– Maintenance (production and test).

• Sloppily written tests??• Is test code quality as good as production?• Slow tests?• True buy in from business sponsors/owners,

project managers, user reps?

23

“High View” of Test Benefits

• “High View”: using tests to best advantage• “Do the thing right”• “Do the right thing” (acceptance tests)• Safety net for modifications/refactoring• Defect localization (1 fail is more info than 100)

• Documentation/training• Minimize costs of testing• This view leads to better tests

24

Integration vs. Unit Tests

• Unit test: tests 1 invocation of 1 method of 1 isolated class with 1 thread

• Integration test: tests more than 1 method invocation of an isolated class, often much more, or more than 1 thread

• Distinction: unit test vs. a test written in JUnit• Examples

– Test hits db– Life-cycle of an object (e.g., a customer order)– Testing units by testing integrated classes

25

Integration vs. Unit (continued)

• Should both be tested?• Unit

– How big do you want the holes in your refactoring safety net?

– How many permutations can integration tests handle?

• Integration– If the units work independently, will they work

together?

26

If Unit Tests are Really Integration Tests

• Slow tests.• False sense of security for refactoring safety net.• Permutations of business rules likely not

covered.• Poor defect isolation: 100 tests failing is less

information than 1 test failing.• High maintenance for test/production

modification.• Counter to the goals of TDD and Agility.

27

Test Method Dependencies

• JUnit 3: Antithetical concept; each test method gets its own test class instance; high isolation; unit oriented by intentional design

• TestNG:– @Test(dependsOnMethods = {“m1”, “m2”, ...})– @Test(dependsOnGroups = {“g1”, “g2”, ...})– E.g.: configuration dependencies; production lifecycle

• JUnit 4: tests ignored if “assumptions” (similar to assertions) are violated: assumeTrue(bool), assumeNoException(Throwable), assumeNotNull(refs …), assumeThat(…) high-readability method

28

Parallel (multiple threads)

@Test(threadPoolSize = #,

invocationCount = #,

timeout = #)

testng.xml:– <suite|test

parallel =“tests|methods|none”

thread-count=“#” (5); 1 if not parallel

time-out =“#” ms (default: 0=none)

...>

• Testing production code with multiple threads (unless not supported); stress testing; speeding up tests!

29

Test Method Parameters I

@Test

@Parameters({“name1”, “name2”})

public void aTest(String name1, int name2) {

...

}

• Values come from testng.xml (<test> is checked first, then <suite>).

• Only basic types (String, [Bb]ool, #’s, …).

30

Test Method Parameters II

@Test(dataProvider = “myDataProvider”)public void aTestMethod(/* params! */) { ...}

@DataProvider(name = “myDataProvider”)public Object[][] getSomeTestData() { ...}• Runs (possibly multiple: first []) parameterized tests.• Multiple parameters: second []• Return type can also be Iterator<Object[]>• Data provider itself can take 0, 1, or 2 args: Method, and test

context

31

Test Class Factories

@Factorypublic static Object[] myTestFactory() {

// return instances of test classes

}

• Those instances will also be interrogated for @Factory methods, and so on. (Each such method invoked once.)

• Example use: you want some of your test classes to have non-standard constructors; the factory constructs them.

32

Migrating JUnit 3 => TestNG

• TestNG tool that modifies source code by adding imports, @Test, @BeforeTest, @AfterTest, and an optional list of desired group names; tool accessible via command line, ant, or Eclipse/IDEA.

• Can use testng.xml <test junit=“true”> for tests that need further migration analysis.

• That’s not necessarily enough; for other warnings and tips, see Appendix D of [TestNG].

33

More TestNG and JUnit 4

• TestNG and JUnit 4 each have other features not mentioned here.

• For more TestNG features, see [TestNG].

• For more JUnit 4 features, see JUnit 4 docs and the source code (e.g., Hamcrest).

34

Reference

• [TestNG] Next Generation JavaTM Testing; TestNG and Advanced Concepts; Cédric Beust and Hani Suleiman; Addison-Wesley; ©2008