A Framework For Software Component Testing Tool In

A Framework For Software Component Testing Tool In Distributed Environment

A Project Report Presented to

The Faculty of the College of Engineering

San Jose State University In Partial Fulfillment

Of the Requirements for the Degree Master of Science in Engineering

By Pham Quang Cuong

December 2002

Copyright © 2002 Pham Quang Cuong

ALL RIGHTS RESERVED

APPROVED FOR THE COLLEGE OF ENGINEERING Dr. Jerry Gao, Project Advisor Dr. Dan Harkey Committee Member Dr. Javad (Jim) Dorosti, MSE Program Advisor

ABSTRACT

A Framework For Software Component Testing Tool In Distributed Environment By

Pham Quang Cuong

In recent years the use of object-oriented analysis, design and programming has

increased significantly. With increasing pressures on time and money, the concept of

component based software development originated. An important issue in software

development cycle is how to test software components which are the building blocks for

larger systems, especially distributed software systems in which operability, integrability,

reliability are always the major concerns of software vendors and users. However, testing

a component in a distributed environment has been a difficult task because of its

complexity and dependency. This report addresses an enterprise methodology and

seamlessly integrated testing framework for software components with Black Box

approach in a distributed environment.

Acknowledgments

The author is deeply indebted to Professor Jerry Gao of Computer Engineering

Department, San Jose State University for his invaluable comments and

assistance in the preparation of this study.

v

Table of Contents

Chapter 1. Project Overview............................................................................................... 1

1.1 Introduction............................................................................................................... 1

1.1.1 Current Issues..................................................................................................... 1

1.1.2 Background Theory .......................................................................................... 4

1.1.2.1 Software Component .................................................................................. 4

1.1.2.2 Testable Software Component.................................................................... 5

1.1.2.3 Built-in-Test................................................................................................ 7

1.2 Proposed Areas of Study and Academic Contribution ............................................. 8

1.3 Current State of the Art............................................................................................. 9

1.3.1 Java Reflection................................................................................................... 9

1.3.2 Java Native Interface (JNI) ................................................................................ 9

1.3.3 Remote Method Invocation (RMI) .................................................................... 9

1.3.4 EasyCharts.................................................................................................... 10

Chapter 2. Solutions.......................................................................................................... 11

2.1 Built-In-Test Solution ............................................................................................. 11

2.2 Test Driver Solution................................................................................................ 13

Chapter 3. Project Architecture......................................................................................... 16

3.1 Introduction............................................................................................................. 16

3.2 Architecture Subsystems......................................................................................... 22

vi

vii

3.3 System Package diagram ........................................................................................ 22

Chapter 4. Technology Descriptions................................................................................. 26

4.1 Client Technologies ............................................................................................... 26

4.2 Middle-Tier Technologies ..................................................................................... 26

4.3 Data-Tier Technologies ......................................................................................... 26

Chapter 5. Project Design ................................................................................................. 27

5.1 Client Design .......................................................................................................... 27

5.2 Middle-Tier Design................................................................................................ 45

5.3 Data-Tier Design..................................................................................................... 64

Chapter 6. Project Implementation ................................................................................... 65

6.1 Client Implementation ............................................................................................ 65

6.1.1 Execute test in the from client request (result.jsp)........................................... 65

6.1.2 Load component profile from test repository (loadcomponentprofile.jsp)...... 65

6.2 Middle-Tier Implementation................................................................................... 66

6.2.1 Create java wrapper class for C component (CAdapter.java).......................... 66

6.2.2 Create header file for C component (CAdapter.java) ...................................... 66

6.2.3 Get Method of test component for invocation (JavaAdapter.java).................. 67

6.2.4 Create instance of the testing component (Runner.java) ................................. 68

6.2.5 Run test for testing component (Runner.java) ................................................. 68

6.3 Data-Tier Implementation....................................................................................... 69

6.3.1 Create table for test suite.................................................................................. 69

viii

6.3.2 Create table for test case .................................................................................. 69

6.3.3 Create tables for test components .................................................................... 69

6.3.4 Create tables for test constructor...................................................................... 69

6.3.5 Create tables for test method............................................................................ 69

6.3.6 Create tables for test argument ........................................................................ 70

6.3.7 Create tables for test set XML ......................................................................... 70

6.3.8 Create tables for test user................................................................................. 70

6.3.9 Create tables for test user roles ........................................................................ 70

6.3.10 Create tables for test result............................................................................. 70

Chapter 7. Illustration of Testing MathComponent with SCTF ....................................... 71

7. 1 Pre-test operation ................................................................................................... 71

7.2 Test Operations ....................................................................................................... 73

Chapter 8. Performance and Benchmarks......................................................................... 84

8.1 Traditional testing Efforts Estimation..................................................................... 84

8.1.1 Cost to develop test cases ................................................................................ 85

8.2 Testing effort with JUnit unit testing tool............................................................... 87

8.2.1. Cost to develop test cases ............................................................................... 87

8.2.2. Number of test drivers .................................................................................... 87

8.2.3. Number of test cases ....................................................................................... 87

8.2.4. Time to develop test cases .............................................................................. 87

8.2.5. Testing time .................................................................................................... 87

ix

8.3 Testing effort with the Software Component Testing Framework (SCTF)............ 90

8.3.1. Cost to develop test cases ............................................................................... 90

8.3.2. Number of test drivers .................................................................................... 90

8.3.3. Number of test cases ....................................................................................... 90

8.3.4. Time to develop test cases .............................................................................. 90

8.3.5. Testing time .................................................................................................... 90

8.4. Comparative Analysis of the different testing methods......................................... 91

8.4.1. The Traditional Testing Method (adhoc testing) ............................................ 91

8.4.2. The JUnit Testing Method .............................................................................. 91

8.4.3. Testing Using the Software Component Testing Framework......................... 92

Chapter 9. Deployment, Operations, Maintenance ........................................................... 95

9.1 Software Deployment ............................................................................................ 95

9.2 Hardware deployment............................................................................................ 96

Chapter 10. Summary, Conclusions, and Recommendations ........................................... 98

10.1 Summary ............................................................................................................... 98

10.2 Conclusions........................................................................................................... 98

10.3 Recommendations for Further Research............................................................... 98

Glossary .......................................................................................................................... 100

References....................................................................................................................... 103

Appendices...................................................................................................................... 105

x

A.1. Java files:............................................................................................................. 105

A.2. JSP files:.............................................................................................................. 107

A.3. SQL files: ............................................................................................................ 108

xi

List of Figures

Figure 1. Component model........................................................................................ 7

Figure 2. Test Framework Model For Distributed System ....................................... 17

Figure 3. Test Framework model for simple system................................................. 18

Figure 4. Distributed System Architecture Diagram................................................. 20

Figure 5. Simple System Architecture Diagram ....................................................... 21

Figure 6. Sub-system: Component Loader................................................................ 22

Figure 7. System Package Diagram .......................................................................... 23

Figure 8. Client Use Case.......................................................................................... 27

Figure 9. Client-side presentation ............................................................................. 28

Figure 10. Flash screen of the client user interface. ................................................... 29

Figure 11. Login dialog box ....................................................................................... 29

Figure 12. Main screen............................................................................................... 30

Figure 13. Component Profile Viewer screen ............................................................ 31

Figure 14. Component Loader screen ........................................................................ 32

Figure 15. System Configuration screen .................................................................... 33

Figure 16. Create user screen ..................................................................................... 34

Figure 17. Test suite screen........................................................................................ 35

Figure 18. Test Case screen........................................................................................ 36

Figure 19. Test component setup screen .................................................................... 37

Figure 20. Test Bed setting screen ............................................................................. 38

xii

Figure 21. Test Result screen ..................................................................................... 39

Figure 22. Test Report screen..................................................................................... 40

Figure 23. Test Suite Coverage screen....................................................................... 41

Figure 24. Test Component Coverage screen ............................................................ 42

Figure 25. Defect Report screen................................................................................. 43

Figure 26. Defect Detail screen.................................................................................. 44

Figure 27. Test Case Activity chart............................................................................ 48

Figure 28. Test data set up in client side sequential diagram..................................... 49

Figure 29. Test Sequential diagram............................................................................ 50

Figure 30. ComponentAdapter class diagram ............................................................ 51

Figure 31. Wrapper class generation process............................................................. 52

Figure 32. TestDriver class diagram .......................................................................... 53

Figure 33. JavaAdapter class diagram........................................................................ 54

Figure 34. Adapter class diagram............................................................................... 55

Figure 35. Test Resource package class diagram....................................................... 56

Figure 36. Test Invocation diagram ........................................................................... 58

Figure 37. Test component process............................................................................ 60

Figure 38. Process of invocation C component.......................................................... 63

Figure 39. Entity Relationship Diagram (ERD)......................................................... 64

Figure 40. Source code snap shot of TestableComponentTwo.................................. 84

Figure 41. Screen shot of traditional testing method component run ........................ 86

Figure 42. Source code snap shot for traditional testing method component ............ 86

xiii

Figure 43. Screen shot of a JUnit testing method component run ............................. 88

Figure 44. Source code snap shot for JUnit testing method component .................... 89

Figure 45. Software deployment diagram for distributed system .............................. 95

Figure 46. Software deployment for simple system................................................... 95

Figure 47. Hardware Deployment for distributed system diagram............................ 96

Figure 48. Harware deployment for simple system ................................................... 97

List of Tables

Table 1. Testability ....................................................................................................... 6

Table 2. Comparisons of BIT component in built-in test case and

built-in test interface ..................................................................................... 13

Table 3. Characteristics of testing component with Test Driver Solution .................. 14

Table 4. Summary Evaluation of Testing Approaches ............................................... 15

Table 5. Estimated time to perform one test case for a component with

the Software Component Testing Framework .............................................. 93

Table 6. Estimate time to perform one test component with

different testing approach (for a component with 3 test cases) .................... 93

Table 7. Test case coverage for function AddNumber of TestableComponentTwo

component with Boundary Method of BlackBox Approach ........................ 94

xiv

1

Chapter 1. Project Overview

1.1 Introduction

1.1.1 Current Issues

In recent years the use of object-oriented analysis, design, and programming has

increased significantly. With increasing pressures on time and money, the concept of

component based software development originated. However, many of the challenges of

using component-based software in critical applications cannot be effectively addressed

via traditional software assurance technologies.

One important issue in object-oriented programming is how to test software

components, which are the building blocks for larger systems, especially distributed

software systems. In addition, testing a component in a distributed environment has been

a difficult task because of its complexity and dependency. Variety of testing approaches

such as Black Box and White Box have been used for many years to show that software

will function as intended.

In the software industry today, software components might not be developed by

the organization using the component, therefore White Box testing is generally not

possible due to source code is not provided by software vendors or manufacturers. In

addition, component testing is less effective when a component's configuration,

environment, and dependencies can be changed at integration or deployment time.

Performing component-level Black Box testing can increase assurance, but developing

the associated test oracles can be very expensive. System and subsystem level testing can

be effective, but they have a high associated cost, and cannot cover all potential behaviors

for most software. Unit testing for software components may reduce integration cost if

2

software components are assured before system integration. However, component unit

testing also cannot cover all component behavior in the system because tester can not

envisioned the complexity and dependency of software component when they are in the

system.

It will be more difficult to test all functionalities and performance of a software

component alone because testing capabilities essentially depend on software component

testability. Therefore, software component testability is one of important concepts in

design and testing of software program and components. There is a set of program

characteristics that lead to testable software, including operability, observability,

controllability, understandability, and so on. (Roger 1997). However, it is difficult to

understand component behaviors in a system. In system testing and maintenance, system

testers usually have the difficulty in understanding and monitoring the component

behaviors in a system due to the following reasons (Gao et al., 2000):

• Test engineers use ad hoc mechanisms to track the behaviors of in-house

components, causing problems for system testers to understand the behavior

of a system due to inconsistent trace messages, formats, and diverse tracking

methods.

• No built-in tracking mechanisms and functions in third-party components for

monitoring their external behaviors.

• No configuration functions for component clients to control and configure the

built-in tracking mechanisms.

• No systematic methods and technologies to control and monitor the external

behaviors of the components.

3

With all these reasons, cost to maintenance and validate software components

usually high due to setup and running time for testing. Although suffering high cost in

testing software components, test engineers today have to face with components that are

not testable. According to Jerry Gao, a professor of San Jose State University, component

testability indicates how well a component is structured to facilitate software testing,

including component auto-test, component integration, and component-based software

testing. To make a component facilitates with all characteristics, software development

engineers have to follow a development standard in order to produce testable

components.

It is obviously seen that a demand of testing component-based software is the

most challenging for many primary software vendors. It is also noted that currently there

is no outstanding software component-testing tool available in the market. Some major

software companies such as Microsoft, Borland, Sun Micro Systems etc… have tried to

promote their software testing tools but most of them cannot introduce a global standard

that can apply for all existing software components. In the other words, their testing tool

cannot be interchangeably used for their software components.

Therefore, most current software development teams use an ad-hoc approach to

create component test suites. Also it is difficult to come up with a uniform and consistent

test suite technology to support different requirements like different information formats,

repository technologies, database schema and test access interfaces of the test tools for

testing such diverse software components. With increasing use of software components,

the tests used for these components should also be reused. Development of systematic

4

tools and methods are required to set up these reusable test suites and to organize,

manage and store various component test resources like test data and test scripts.

In challenging the current issues, this project introduces a testing framework in

which a standard for software component is applied to assist the testing process. The

standard proposes every software component released must be coupled with its profile or

called component descriptor that is a XML based file. The component profile will

provide the standard testing tool enough information of the testing component during test

process. If every software component complies with this standard, the testing framework

will be able to test the component without discriminating what language that component

was written. Therefore, the framework will be a solution for the current problem that has

been raised by different programming language using to develop software components.

1.1.2 Background Theory

1.1.2.1 Software Component

Software components are more or less independent units with a specific task or

functionality. Components are put together in order to build more complex components

or “composable software systems”. For this purpose, they should have clearly defined

interfaces. The idea of component-based development is the assembling of software end

products from reusable components (Szyperski 1997). Each component is designed to

work in a variety of contexts and in conjunction with a variety of other components. As

stated in (Will 1999), many end products can be designed with the help of one

component. A component must be capable of being connected to other components

(through a communications interface) to form a larger group.”

5

As further described in (Paul et al. 1998), “components provide a means of

packaging related objects together into prefabricated pieces of software from which

solutions are constructed.” The distinction between specification, implementation and

binary code aspects of components is discussed in (Keith 1997): “A software component

is an independently deliverable package of reusable software services. In general, a

component has three facets:

• A specification that describes the semantics. This explains what the

component does, and how a client should use it.

• An implementation design, which describes how an implementer has chosen

to design and construct software and data stores to meet the intended

specification.

• An executable that delivers the component’s capability on a designated

platform.

1.1.2.2 Testable Software Component

Software component testability indicates how well a component is structured to

facilitate software testing, including component auto-test, component integration, and

component-based software testing (Gao, 2000). The most important issue for most

software test engineers today is how to make the component testable. According to Gao,

there are three different mechanisms that can use to construct the testability of software

components:

• Framework-based testing facility: needs component source code and well-

defined class library to add test code. This way is simple and flexible.

6

• Built-In-Test: needs extra programming overhead during component

development, well define interface to add test code.

• Automatic component wrapping for testing: needs a well-defined framework

to interact with test tool and components are wrapped inside program for

testing, but it requires low programming overhead.

The following table shows the comparisons among three mechanisms:

Table 1. Testability (Source: On Building Testable Software Components – Gao et al., 2002)

These mechanisms so far have been utilized upon the demands on testing

component functionality, and most of the time they are applied for in-house testing.

Associating with these mechanisms, under tested component architecture can be divided

into two groups and described in figure 1.

7

F1

Fn

BIT T1

BIT Tn

Component

Test

Inte

rface F1

Fn

Component

Test

Inte

rface

Component Test Driver

(a) Component with Built-in-Test (b) Component interacts with a test driver

F1

Fn

BIT T1

BIT Tn

Component

Test

Inte

rface F1

Fn

Component

Test

Inte

rface

Component Test Driver

F1

Fn

BIT T1

BIT Tn

Component

Test

Inte

rfaceF1

Fn

BIT T1

BIT Tn

Component

Test

Inte

rface F1

Fn

Component

Test

Inte

rface

Test

Inte

rface

Component Test DriverComponent Test Driver

(a) Component with Built-in-Test (b) Component interacts with a test driver

Figure 1. Component model

1.1.2.3 Built-in-Test

Built-in-Test (BIT) is defined as a new kind of maintenance-oriented software

testing (YingXu et al. 2000), which is explicitly described in software source code as

member functions. BIT is a test mechanism embedded in a software entity. A BIT has the

capability to be “visible” outside its implementing software entity. The testing of

conventional software focuses on the generation of tests for existing systems; the BIT

method draws attention to building testability into systems, so that software testing and

maintenance can be self-contained. From the component structure concept, a component

consists of two structure parts: interface and implementation. The interface of component

is the only means of external access to the member functions of the component. The

implementation describes external behavior or internal behavior of the component.

A component is reusable because of its natural encapsulation and inheritability.

According to Yingxu (et al. 2000), to create BITs, it is noteworthy that the standard

functions of constructor and destructor contained in an object are interestingly reusable

structures. The authors have found that these standard structures can be extended further,

to be able to incorporate reusable BITs in an object. A prototype of a test-built-in object

is developed as shown in the following code:

8

class class-name {

//interface

Data declaration;

Constructor declaration;

Destructor declaration;

Function declarations;

Tests declaration;

// Built-in test implementation

Constructor;

Destructor;

Functions;

TestCases; // Built-in test cases

} BITObject;

The test declarations in the interface and the test cases in the implementation have

been embedded into a standard object structure. In this way, the BITs may be inherited

and reused in the same way as that of standard and application-specific member functions

within the object.

1.2 Proposed Areas of Study and Academic Contribution

The project proposes a framework for a software component test tool that is based

on client-server design pattern. This framework will fit for all components written by

Java or C language.

In this project, a prototype of the software tool is introduced. This prototype is

capable to test all Java or C components that have interfaces supported primitive-data

types. The prototype is a web-based application that provide testers graphic user

interfaces to set up test cases, access component profile, and retrieve statistical data of

test results in bar and pipe charts.

9

The proposed testing framework also aims to give students a clearer picture in

testing software components that has not been focused significantly in academic but

industrial training courses. The framework can give students a prototype that they can

have some hands-on experience in software component testing with Black Box method.

1.3 Current State of the Art

1.3.1 Java Reflection

Java Reflection enables Java code to discover information about the fields,

methods and constructors of loaded classes, and to use reflected fields, methods, and

constructors to operate on their underlying counterparts on objects, within security

restrictions. The API accommodates applications that need access to either the public

members of a target object (based on its runtime class) or the members declared by a

given class.

1.3.2 Java Native Interface (JNI)

The Java Native Interface is the native programming interface for Java that is part

of the JDK. By writing programs using the JNI, it ensures that the code is completely

portable across all platforms.

The JNI allows Java code that runs within a Java Virtual Machine (VM) to

operate with applications and libraries written in other languages, such as C, C++, and

assembly. In addition, the Invocation API allows programmer to embed the Java Virtual

Machine into native applications.

1.3.3 Remote Method Invocation (RMI)

RMI enables the programmer to create distributed Java technology-based to Java

technology-based applications, in which the methods of remote Java objects can be

10

invoked from other Java virtual machines, possibly on different hosts. A Java technology-

based program can make a call on a remote object once it obtains a reference to the

remote object, either by looking up the remote object in the bootstrap naming service

provided by RMI or by receiving the reference as an argument or a return value. A client

can call a remote object in a server, and that server can also be a client of other remote

objects. RMI uses object serialization to marshall and unmarshall parameters and does

not truncate types, supporting true object-oriented polymorphism.

1.3.4 EasyCharts

EasyCharts is a complete library of java chart components, chart applets, and

chart servlets that enable programmers to add charts and graphs in java applications, web

applications, and web pages with just a few lines of code. The java chart library includes

bar charts, line charts, and pie charts and is highly configurable. The java chart library

supports charts with multiple data series, overlay charts, drilldown charts, and interactive

features such as zooming and scrolling of chart data. The charts are compatible with JDK

1.1 or newer so it works on all major web browsers with no additional add-ons other than

a standard Java Virtual Machine installed.

11

Chapter 2. Solutions

As mentioned in chapter 1, testing approaches can be based on the testable

component models that are described with built-in test and test driver. It is very

important to choose the correct approach for test tool. It is not only for technical

feasibility, but also timing and resources spend for testing process. For both

approaches, they are addressing both pros and cons. That make software test tool

designers and developers have to consider very carefully, and balance out their

needs to approach an appropriate method.

2.1 Built-In-Test Solution

Based on the BIT, software component is proposed for operation in two

modes: a normal and a maintenance mode. In the normal mode, the software has

the same behavior as the conventional system; in the maintenance/testing mode,

the BITs can be activated by calling the BITs as member functions in the

component.

The Built-In-Tests can be anything from assertions to complete test cases

that have been built into the component. However, it is generally not a feasible

solution to have the complete test cases built-in. Complete test cases add a lot of

overhead to a component, while many test cases, like most functional tests, does

not add any value to the component once executed in the context where the

component has been deployed. Significantly fewer overheads have to be added to

the component if the test cases are placed outside the component and the built-in

tests provide the information needed in order to test it. This approach to built-in

tests also provides a more flexible solution since test cases can be added and

12

removed according to the needs without changing the internals of the component

under test.

Therefore a BIT component can be make in two ways: built-in test cases

as Yingxu (et al. 2000) mentioned or API that only visible in test mode for

component functions. A Built-In-Test (BIT) component requires both extra effort

of developers and a standardized development process among software

component manufacturers.

Most BIT components have to suffer “overhead” due to extra

implementation for built-in test case or testing APIs. The “overhead” will lead

extra cost for component development due to time of developers, time of

component compilation and time of component execution. If the test case is built

in, the testing will be limited in the test paradigm of the component itself. Testers

will be limited in creation of new test case to serve for their own verification and

validation.

The benefit of BIT component is to provide tester a completed predefined

test cases and API so that testers can follow the test case scenario or utilize the

built-in-test API to generate their test cases. In short, the pros and cons of BIT

approach can me summarized in the following table:

13

Pros Cons Component with

Built-In Test API

• Provide tester better

information about component

behavior.

• More flexible in creating test

case.

• May require re-compilation

• Suffer overhead in coding

• Current test cases may be

impacted if the code

changed.

• Higher cost in development

and design.

Component with

Built-In-Test Test

Case

• Requires less effort from

tester since test cases are

pre-defined.

• No or simple test driver

required.

• May require re-compilation

• Rigid, lack of flexibility for

tester. Pre-defined test cases

may not support for testers’

purposes

• Significant overhead in

source code.


and design.

Table 2. Comparisons of BIT component in built-in test case and built-in test interface

2.2 Test Driver Solution

The traditional way of constructing test drivers is to create it such that it works for

a specific component. But with the advent of the component world and systems using

reusable third party components, such traditional construction will not work. This is

because it is inefficient to cope with the diverse software components and their

customizable functions. Therefore, a test driver of a “perfect testing tool” should be

generic or it at least can take minor static or dynamically modifications during the testing

process.

14

A successful test driver should be reusable. Therefore to construct a test driver,

the test tool should have significant information about the under test component before

the test driver is statically or dynamically generated. The generic test driver is made

based on the assumption that every tested component must be coupled with its profile (a

component profile is a set of information presented in a system file, object, or and

communication mean that expose the tested component’s interface and inside behavior.)

Such a test driver will make a relationship between the tested component and the testing

environment that support for test case execution.

The generic test driver can help testers reduce the testing time significantly from

modifying test case for each component. However, making a test driver requires a highly

sophisticated implementation of the test tool. In short the pros and cons of this solution

can be summarized in the following table:

Pros Cons Component API

with test driver

• No re-compilation needed.

• No coding overhead

• Test case is independent

from code change of

component.

• Less constraint in creation of

test cases.

• No extra cost in design and

development of components.

• Requires component profile

support.

• Requires knowledge of

testing environment

• Higher cost to develop the

test tool (higher complexity)


and design.

• Cannot guarantee that will

make all components

testable.

Table 3. Characteristics of testing component with Test Driver Solution

15

Each approach has pros and cons. However, the ultimate goal of software testing

is to assure that a software product is delivered with highest confidence of bug free and

lowest cost. The following table is showing a comparison among the discussed testing

approaches.

Estimate cost and effort Test Driver Approach

BIT Test Case

BIT API

Code recompilation No Yes No

Coding overhead No Yes Yes

Development cost Low High High

Testing cost High Low High

Impact on test case when code change No High High

Impact on test case when component specification change

Low High High

Test environment dependency Low High Low

Test case customization Yes No Yes

Increase testability of software component No Yes Yes

Test case reusability Yes No Yes

Test driver generation Yes No Yes

Table 4. Summary Evaluation of Testing Approaches

In this project, the test driver approach is selected as applied method for the

testing framework. The testing components are presumably always coupled with

component profiles. The test driver will bind with other testing environment components

to make the component become testable from understanding the component external and

internal behaviors from its profile.

16

Chapter 3. Project Architecture

3.1 Introduction

Software Component Testing Framework (SCTF) is a client-server based

application (figure 2). The system composed of a three-tier layers sub system and remote

repository servers. The three-tier layer system includes client, middleware, and back-end

tier.

Client tier is a web user interface that communicates with test server via well-

defined http/xml protocol. The communication protocol can be a SOAP

protocol or regular http request (to Apache web server). Test data are

maintained their persistence and manipulation via JSP engine (Jakarta

Tomcat) and JavaBeans.

Middleware tier is a set of Java programs that handles all test logic and

communications with other component repository servers.

Back-end tier handle data transaction between test server and database. It is

also responsible for loading component profile from component repositories

with test database.

Most of component repositories are placed behind firewalls therefore it is

necessary to make a “punchout”, which is a breakthrough allowance through firewall, to

access component profile. All remote request and invocation from test server to

component repositories are handled by java RMI (if component repository locates in a

machine that is different from the test server machine). Database transactions are

processed via JDBC connection pool that can handle multiple threads from test clients.

17

SCTF can be viewed as centric web-based application in which application users

can control and access multiple platforms that serve for testing purposes. The testing

framework model can simply be depicted as follow:

Case Studies

Database

Test Agent

RMI

ComponentProfileLoader

Data Connector

Repository Server

Database

Web Server

Test Server

Java Adapter CAdapter

Test Library

DBAccesor

JSP engine

Test Client

Case Studies

Database

Test Agent

RMI

ComponentProfileLoader

Data Connector

Repository Server

Database

Web Server

Test Server

Java Adapter CAdapter

Test Library

DBAccesor

JSP engine

Test Client

TestD

atabase

Component Repository

DBAccessor

Figure 2. Test Framework Model For Distributed System

18

Test Server

JavaAdapter

Test Library

DBAccessor

Web Server

TestD

atabaseCom

ponent Repository CAdapter

Component ProfileLoader

JSP Engine

Test Client

Test Server

JavaAdapter

Test Library

DBAccessor

Web Server

TestD

atabaseCom

ponent Repository CAdapter

Component ProfileLoader

JSP Engine

Test Client

Figure 3. Test Framework model for simple system

The SCTF design can fit for both distributed (figure 2) and non-distributed

(figure 3) system. The system basically composed the following modules:

• Test client: is the web interface that allows users perform testing activities.

• Test server: is the heart of the test framework. The test server of a simple

system may contain different modules from the test server of a distributed

system.

• DBAccessor: is a java component that has JDBC support to read and write

test data.

• Component Profile Loader: is a java component that can read component

profile and insert its information in appropriate tables in the test database.

• JavaAdapter, CAdapter: are pluggable components that allow test server to

generate test dirver for Java or C component respectively.

19

• Test Agent: only available in distributed testing framework version. The

test agent is a RMI program locates in the remote repository server that

communicate remotely with test server to call tested methods of the under

tested components.

The test client can access component repository and then invoke method calls via

the test server that centrally control all activity of the testing framework. The test server

provides a sophisticated test library to handle all test requests from the test client via

XML communication channel. In the distributed system model and the test server will

send invocation request via test agent to execute the components in the repository servers

by passing data to its counterpart test agent that resides in the repository server (remote

invocation). In the simple system, the test server will directly invoke the component that

locates in the same physical machine.

The detailed modules of the SCTF is depicted in the following architecture

diagram:

20

CVS Server

ComponentCVS

Repository

Test Server

TestBed(Web-based application)

Test Agent

JSP Engine(Tomcat)

TestReport(Web-based application)

ComponentViewer(Web-based application)

http http

http

Http/XML (SOAP)

DatabaseServer

(Oracle 8i)

Test client

ApplicationServer

JavaBeans,TestLibrary

Apache webserver

DatabaseAccessor(JDBC)

TestAdapterJava/C++

TestController

TestManager GUIWeb-basedApplication

Server Admin

RMI

JDBC

http

http

XML Interface

Firewall

gatewaypunchout

gatewaypunchout

TestAgent

Pipeline(Pipelets)

Test Manager

TestRunner

Internet

TestDriver

Figure 4. Distributed System Architecture Diagram

21

Test Server

TestBed(Web-based application)

JSP Engine(Tomcat)

TestReport(Web-based application)

ComponentViewer(Web-based application)

http http

http

Http/XML (SOAP)

DatabaseServer

(Oracle 8i)

Test client

ApplicationServer


Apache webserver

DatabaseAccessor(JDBC)

TestAdapterJava/C++

TestController

TestManager GUIWeb-basedApplication

Server Admin

JDBC

http

http

XML Interface

Pipeline(Pipelets)

Test Manager

TestRunner

Internet

ComponentCVS

Repository

TestDriver

Figure 5. Simple System Architecture Diagram

22

3.2 Architecture Subsystems

In order to access testing components and make their information available for the

test server, component profiles need to be loaded in the test database. A subsystem of this

framework is designed to gather a component's information and to maintain its profile in

the test database (figure 6). A web inteface is provided for system administrator to send

the request to the remote repository so that the system can down load component profiles

and save them in the test database for later testing.

Server Admin

Com

pone

nt P

rofil

e Lo

ader

Inte

face

Test Server

http

Apache webserver

JSP Engine(Tomcat)

Pipeline(Pipelets)

ApplicationServer


ComponentLoader

DatabaseServer

(Oracle 8i)

ComponentRepositoryjdbc

ftp

Figure 6. Sub-system: Component Loader

3.3 System Package diagram

In order to facilitate with three tiers designed for the system architecture, the test

server library provides its components that function on appropriate layers that are mapped

to system tiers. The following system package diagram depicts the relationship of java

components in the test library that supports system tiers’ functionality.

23

com.testcom.testbean com.testcom.report com.testcom.test.repository.profile

com.testcom.test.testbed com.testcom.test.manage com.testcom.test.control

com.testcom.dbaccess

com.testcom.test.invocation

Under testedcomponent

com.testcom.pipeline

User Interface Layer

Test Service Layer

Test Data Layer

com.testcom.admin

com.testcom.config

com.testcom.test.repository

com.testcom.server

Figure 7. System Package Diagram

Each package supports for one or more modules in the test server. They are

designed in such a way that test data are logically sent across application tiers. Basically,

system package design is to map system tiers with services of test server. The system

package composes of three layers:

24

User interface layer: this layer contains components that handle test request

from test client and support GUI presentation in the front-end via JSPs. This

layer contains the following packages

•

•

com.testcom.testbean: handles GUI in the test client

com.testcom.report: handles statistical data of test report

com.testcom.test.repository.profile : handles component profile

presentation in the test client

com.testcom.config: handles system configuration

Test service layer: this layer handles all logical implementation such as define

test adapter, invoke test component, and access data from test data layer. This

layer contains the following packages:

com.testcom.pipeline : handles test request from client side to test server

and vice versa.

com.testcom.server: handles test request queing and test result response.

com.testcom.admin: handles user administration, create, update users.

com.testcom.test.testbed: core or the test library, handles component

profile information, test data and build in wrapper class for testing.

com.testcom.test.manage: handles test cases, test reports and test adapters.

com.testcom.test.control: handles test adapter usage, system configuration.

com.testcom.test.repository: handles component loading to the test server.

com.testcom.test.invocation: handles communication between test server

and repository server.

25

Test data layer: this layer handles all data transaction from test server to test

database. Data transactions include test result, test report and component

profile. This layer contains only one package:

•

com.testcom.dbaccess: accesses and retrieves data between test server and

database server.

26

Chapter 4. Technology Descriptions

4.1 Client Technologies

The system client is based on HTML presentation that is generated dynamically

by Java Server Page 2.0 (JSP).

4.2 Middle-Tier Technologies

The test server and library is composed of java components that employs

JavaBeans, Java Reflection, Java Native Interface (JNI), and Java RMI (Remote Method

Invocation) technology of Sun Micro Systems (JDK 1.3).

The test report , coverage, and statistical data presentation employs Java

component EasyCharts 1.0.

4.3 Data-Tier Technologies

The test database is oracle server 8.1.7. All data transactions between test server

and database server via JDBC 1.3.1

27

Chapter 5. Project Design

5.1 Client Design

The client functionalities are illustrated as following use cases:

Tester

«» «»

Log in

«» «»

View componentprofile

«» «»

LoadComponent

«» «»

Report testresult

<<uses>>

<<uses>>

<<uses>>

«» «»

Run Test Case

<<uses>>

«» «»View Defect

<<uses>>

«» «»

Create Test User

<<uses>>

<<uses>>

«» «»

Config System

Figure 8. Client Use Case

System user may have different roles: admin, manager or tester. Upon user’s role

the client functionalities may be visible or hidden. The client user interface is presented

dynamically by server-side scripts in Java Server Page. The JSP pages that provide client

functionality are presented as following chart:

28

x

admin.jspinde .jsp frame.jsp

fra

login.jsp menu.jsp

testsuite.jsp

testcase.jsp

testcomponent.jsp

testbed.jsp test.jsp result.jsp

configadmin.jsp report.jsp

reportdetail.jsp

defect.jsp

defectdetail.jsp

createuser.jsp

componentprofile.jsp

Componentprofiledetail.jsp

Loadcomponentprofile.jsp

listuser.jsp

modifyuser.jsp

Figure 9. Client-side presentation

29

The web page of test client is formatted with embedded CSS (Cascading Style

Sheet) in JSP pages in order to generate well-defined and consistent HTML layout.

Figure 10. Flash screen of the client user interface.

Figure 11. Login dialog box

30

Figure 12. Main screen

31

Figure 13. Component Profile Viewer screen

32

Figure 14. Component Loader screen

33

Figure 15. System Configuration screen

34

Figure 16. Create user screen

35

Figure 17. Test suite screen

36

Figure 18. Test Case screen

37

Figure 19. Test component setup screen

38

Figure 20. Test Bed setting screen

39

Figure 21. Test Result screen

40

Figure 22. Test Report screen

41

Figure 23. Test Suite Coverage screen

42

Figure 24. Test Component Coverage screen

43

Figure 25. Defect Report screen

44

Figure 26. Defect Detail screen

45

5.2 Middle-Tier Design

The middle-tier is the test application server to handle all test requests from client

side. A tester accesses the test server via common login. Based on the client side

presentation, the user can perform different features of test server. Upon user logon

information sent from the client side, test library in the middle tier decides what privilege

that user can have in the test system. The common user activity in the test system is

described in figure 27. The communication channel between test client and test server is

achieved by a well-defined xml protocol. The DTD of test set that is used to generate

XML content in a test suite is designed as follows:

   <!— Test Suite includes multiple Test case and environment information-->  <!ELEMENT TestSet (TestSuite?)> <!ELEMENT TestSuite (TestEnv*, TestCase*, TestSuiteID*, TestSuiteName*, ReleaseVersion*, Project*)> <!ELEMENT TestSuiteID (#PCDATA)> <!ELEMENT TestSuiteName (#PCDATA)> <!ELEMENT ReleaseVersion (#PCDATA)> <!ELEMENT Project (#PCDATA)> <!ELEMENT TestEnv (#PCDATA)> <!ELEMENT OperatingSystem (#PCDATA)> <!ELEMENT IPAddress (#PCDATA)> <!ELEMENT Domain (#PCDATA)>    <!ELEMENT TestCase (TestComponent?)> <!ELEMENT TestCaseID (#PCDATA)> <!ELEMENT TestCaseName (#PCDATA)> <!ELEMENT TesterName (#PCDATA)> <!ELEMENT TestDate (#PCDATA)> <!ELEMENT TestCaseDescription (#PCDATA)> <!ELEMENT ActualResult (Value*, DataType*)> <!ELEMENT Value (#PCDATA)> <!ELEMENT DataType (#PCDATA)> <!ELEMENT ExpectedResult (Value*, DataType*)> <!ELEMENT Value (#PCDATA)> <!ELEMENT DataType (#PCDATA)>

46

  



    <!ELEMENT TestComponent (Constructor*, Method*, ID*, Name*, FullyQualifiedName*,Version*, Package*, Company*, Author*, HasConstructor*, HasMethod*, Modifier*, Extends*, Implement*, Testable*, ComponentType*, Signature*)> <!ELEMENT ID (#PCDATA)> <!ELEMENT Name (#PCDATA)> <!ELEMENT FullyQualifiedName (#PCDATA)> <!ELEMENT Version (#PCDATA)> <!ELEMENT Package (#PCDATA)> <!ELEMENT Company (#PCDATA)> <!ELEMENT Author (#PCDATA)> <!ELEMENT HasConstructor (#PCDATA)> <!ELEMENT HasMethod (#PCDATA)> <!ELEMENT Modifier (#PCDATA)> <!ELEMENT Extends (#PCDATA)> <!ELEMENT Implement (#PCDATA)> <!ELEMENT Testable (#PCDATA)> <!ELEMENT ComponentType (#PCDATA)> <!ELEMENT Signature (#PCDATA)>    <!ELEMENT Constructor (Arg*, ID*, Name*,

NumberOfArg*, Signature*, ThrowsException*, HasParameter*, ExceptionClass*)> <!ELEMENT ID (#PCDATA)"> <!ELEMENT Name (#PCDATA)> <!ELEMENT NumberOfArg (#PCDATA)> <!ELEMENT Signature (#PCDATA)> <!ELEMENT ThrowsException (#PCDATA)> <!ELEMENT HasParameter (#PCDATA)> <!ELEMENT ExceptionClass (#PCDATA)> <!ELEMENT Arg (ID*, DataType*, isArray*, Value*, IsOutput*)> <!ELEMENT ID (#PCDATA)> <!ELEMENT DataType (#PCDATA)> <!ELEMENT isArray (#PCDATA)> <!ELEMENT Value (#PCDATA)> 

47

  <!ELEMENT Method (Arg*, ID*, Name*, NumOfArg*, Modifier* , ThrowsException* ReturnType*, HasParameter*,

Signature *, ActualException*)> <!ELEMENT ID (#PCDATA)> <!ELEMENT Name (#PCDATA)> <!ELEMENT NumOfArg (#PCDATA)> <!ELEMENT Modifier (#PCDATA)> <!ELEMENT ThrowsException (#PCDATA)> <!ELEMENT ReturnType (#PCDATA)> <!ELEMENT HasParameter (#PCDATA)> <!ELEMENT Signature (#PCDATA)> <!ELEMENT ExceptionClass (#PCDATA)> <!ELEMENT Arg (ID*, DataType*, isArray*, Value*, IsOutput*)> <!ELEMENT ID (#PCDATA)> <!ELEMENT DataType (#PCDATA)> <!ELEMENT isArray (#PCDATA)> <!ELEMENT Value (#PCDATA)> <!ELEMENT IsOutput (#PCDATA)>

User’s test data are kept persistent in the user’s session via Javabeans (figure 28).

Test application server is designed in plug-in module approach. Testing components that

are written by different programming languages require using different adapters. The

adapter is a pluggable module in the test server that is designed for a particular language.

For example, Java components require Java adapter that is set from system configuration.

Upon the component type, the test process will select the test adapter accordingly (figure

29).

48

Figure 27. Test Case Activity chart

49

TestSuiteBean TestCaseBean TestComponentBean MethodBeanConstructorBean ArgumentBean DataBean DBManager Server

ProcessRequest

SetTestCaseBean

ProcessRequest

ProcessRequest

SetComponentBean

ProcessRequest

ProcessRequest

ProcessRequest

SetConstructorBean

SetMethodBean

SetArgumentBean

SetArgumentBeanProcessRequest

SaveTestSuite

SaveTestCase

getTestRequest

Figure 28. Test data set up in client side sequential diagram

50

Server Contro ller Driver TestRunner

getTestDocument

d ispatchTestDriver

TestValidator DBManager

run

getActua lResult

getTestResult

saveTestResult

getTestResponse

ComponentAdapter Com.testcom.test.testbed.resource

getComponentAdapter

loadComponent

getDrivenComponent

saveTestSchema

getComponentType

Figure 29. Test Sequential diagram

51

Figure 30. ComponentAdapter class diagram

52

Each pluggable adapter, a Java class, needs to implement from an interface

ComponentAdapter (figure 30). The overloaded LoadComponent() function is

implemented so that the testing component written by a particular language will be

stimulated in a wrapper class (figure 31). In this prototype, there are two adapters

provided for Java (figure 33) and C (figure 34) components. The wrapper class is

generated during test server builds test suite (figure 35).

<<Wrapper class>>

TestEntityTestcomponent

ComponentProfile

ComponentProfileComponentDescriptor

<<Wrapper class>>

TestEntityTestcomponent

Figure 31. Wrapper class generation process

53

Figure 32. TestDriver class diagram

54

Figure 33. JavaAdapter class diagram

55

Figure 34. Adapter class diagram

56

-setTestEntities() : void+getTestCaseID() : int+getTestCaseName() : String+getTestComponent() : TestEntity+getTestDuration() : double+getTestEndTime() : double+getTestStartTime() : double+getTesterName() : String+getTestStubs() : TestEntity+setTestCaseID() : void+setTestCaseName() : void+setTestDate() : void+setTestEndTime() : void+setTestStartTime() : void+testDateToString() : String

-TestCaseID : int-TestCaseName : String-TestDate : Date-TestDescription : String-TestedEntities : TestEntity-TestEndTime : double-TestStartTime : double-TesterName : String

TestCase

+getDomain() : String+getIPAddress() : String+getProjectName() : String+getReleaseVersion() : String+getTestCase() : TestCase+getTestOperatingSystem() : String+getTestSuiteName() : String+setTestCase() : void

-TestSuiteID : int-Domain : String-IPAddress : String-ProjectName : String-ReleaseVersion : String-TestOS : String-TestSuiteName : String-testCase : TestCase-document : Document

TestSuite

+getAvailableMethods() : TestMethod+getCompany() : String+getComponentAuthor() : String+getComponentID() : int+getComponentName() : String+getComponentVersion() : String+getConstructors() : ComponentConstructor+getTestedMethod() : TestMethod+hasConstructor() : boolean+hasMethod() : boolean+isStatic() : boolean+isStub() : boolean

-Company : String-ComponentAuthor : String-ComponentID : int-ComponentName : String-ComponentVersion : String-Constructors : ComponentConstructor-HasConstructor : boolean-HasMethod : boolean-IsStatic : boolean-IsStub : boolean-TestedMethods : TestMethod

TestEntity

+getArgSet() : Argument+getArgumentTypes() : Class+getArgumentValues() : Object+getMethodName() : String+getNumOfArg() : int+hasOutput() : boolean+hasReturn() : boolean+isExceptionThrown() : boolean+isStatic() : boolean+isTested() : boolean

-ArgSet : Argument-ExceptionThrown : boolean-HasOutput : boolean-HasReturn : boolean-IsStatic : boolean-MethodName : String-NumOfArg : int-Tested : boolean

TestMethod

+addProperties() : void+containProperty() : boolean+containsKey() : boolean+getProperties() : Object+removeProperties() : Object+setProperties() : void

-PropHash : HashtableComponentProperties

+getArguments() : Argument+getNumOfArguments() : int+isExceptionThrown() : boolean+isInherited() : boolean+isInstantiable() : boolean

-Arguments : Argument-ExceptionThrown : boolean-Inherited : boolean-Instantiable : boolean-Name : String

ComponentConstructor

+getDatatype() : Class+getValue() : Object+isArray() : boolean+isOutput() : boolean

-ArgValue : Object-DataType : Class-IsArray : boolean-IsOutput : boolean

Argument

1

1..*

1

1..*

1

1..*

1

1..*

1 1..*

11..*

1

1

11

<<instantiate>>

<<setup>>

<<set test method>>

<<set test entity>>

<<has agument>>

<<set constructor>>

<<has argument>>

Figure 35. Test Resource package class diagram

57

Test driver (figure 32) will generate appropriate wrapper class that encapsulates

the test component data and then invoke a function call from the tested component. The

process of invocation (figure 36) in the test server will be achieved by employing java

reflection technology via Java wrapper class. The result from the invocation will be

validated and compared with the expected result that provided by the tester. The test

report will be generated and saved in the database upon the tester’s request.(figure 27)

58

0..1

1

11 +getDomain() : String

+getIPAddress() : String+getProjectName() : String+getReleaseVersion() : String+getTestCase() : TestCase+getTestOperatingSystem() : String+getTestSuiteName() : String+setTestCase() : void

-TestSuiteID : int-Domain : String-IPAddress : String-ProjectName : String-ReleaseVersion : String-TestOS : String-TestSuiteName : String-testCase : TestCase-document : Document

resource::TestSuite

1

1

TestRunner

doTest() : voidinitArrayObject() : ObjectinitArrayClass(): Class[]run() : void

Idriver : _driver

bean : TestSuiteBean

getId() : StringgetName() : StringgetDomain() : StringgetIpAddress() : StringgetVersion() : StringgetProject() : StringgetOperatingSytem() : StringgetTestCaseBean : StringProcessRequest() : voidresetBean() :voidsetId() : voidsetName() : voidsetDomain() : voidsetIpAddress() : voidsetProject() : voidsetVersion() : voidsetOperatingSystem() : voidsettestCasebean() : void

_Id: String_name : String_ domain : String_ipaddress : String_version : String_project : String_operatingsystem : String_testcasebean : TestCaseBean

1

1

TestDriver

createWrapperClass() : booleangetComponentInstance() : ObjectgetComponentMethod() : MethodgetDrivenComponent() : TestEntity

ComponentAdapter: comAdapter

ComponentAdapter <<interface>>

getComponentInstance() : ObjectgetComponentMethod() : MethodgetComponentProperties() : ComponentProfilePropertiesgetComponentType() : StringgetTestEntity() : TestEntitygenerateJavaCode() :StringloadComponent() : voidwriteToJavaFile() : void

Figure 36. Test Invocation diagram

59

The process of invocation component method is done by the direct call of the

selected method in the wrapper class for tested component. The system has knowledge

about testing component from its profile. The component profile is a XML based system

file that is loaded in the test database during the testing process. When user creates a test

case and loads a component to test, the wrapper class will be dynamically created based

on the selected component, method, constructor and test parameter for the selected

method and constructor (figure 37).

Different from testing Java component that is directly invoked by Java Reflection,

test server has to use Java Native Interface (JNI) to create an interface Java class for a C

component, then call the method of the C component via Java Reflection.

60

Component Profile(XML)

ComponentProfileProperties

ComponentXMLDescriptor

TestEntity (Wrapper class)

Testcomponent

JavaAdapter

TestRunner(Java Reflection)

Load

Build Invoke

Assert Driver

(a) test Java component process

(b) test C component process

ComponentProfileProperties

ComponentXMLDescriptor

TestEntity (Wrapper class)

Testcomponent

CAdapterTestRunner

(Java Reflection)

Load

Build

Invoke

Assert Driver

Make DLLRecompile

Component Profile(XML)

TestableComponent

TestableComponent

Figure 37. Test component process

The process of executing C component is different from executing Java

component in using intermediate files and external process. The C program is coupled

with its header file that is generated from the JDK of the repository environment. Context

in the header file of C program is mapped with the wrapper class generated by the test

server. The system does not directly invoke the C program but it will call C function from

61

the generated .dll file that is also generated by Java Native interface. The following

processes describe (figure 35) how the test server invoke a C program from the wrapper

class:

a. The system generates a wrapper class (HelloWorld.java) that declares the native

method.

public class HelloWorld{ public native String printMsg(String msg); public native int doubleNumber(int param_0); public native void print(); static { System.loadLibrary("HelloWorld"); } } b. Use javac to compile the HelloWorld source file, resulting in the class file

HelloWorld.class. The javac compiler is supplied with JDK or Java 2 SDK

releases.

javac HelloWorld.java

In the test library the code should be

Process P1 = Runtime.getRuntime().exec("javac " + C_ComponentDir + comProp.getComponentName() + ".java"); c. Use javah -jni to generate a C header file (HelloWorld.h) containing the

function prototype for the native method implementation. The javah tool is

provided with JDK or Java 2 SDK releases.

javah -jni HelloWorld

In the test library the code should be

Process P = Runtime.getRuntime().exec("javah -jni " + comProp.getComponentName());

d. Generate the C implementation (HelloWorld.c) of the native method.

62

#include <jni.h> #include <stdio.h> #include "HelloWorld.h" JNIEXPORT jstring JNICALL Java_HelloWorld_printMsg(JNIEnv *env, jobject obj, jstring string) { const char *str = (*env)->GetStringChars(env, string, 0); jstring string = (*env)->NewStringUTF(env, str); return string; } JNIEXPORT jint JNICALL Java_HelloWorld_doubleNumber(JNIEnv *env, jobject obj, jint i) { return i*2; } JNIEXPORT void JNICALL Java_HelloWorld_print(JNIEnv *env, jobject obj) { printf("from JNI Test String"); } e. Compile the C implementation into a native library, creating HelloWorld.dll. Use the C

compiler and linker available on the host environment.

Note that different operating systems support different ways to build native

libraries. On Solaris, the following command builds a shared library called

libHello-World.so:

cc -G -I/java/include -I/java/include/solaris HelloWorld.c -o libHelloWorld.so

The -G option instructs the C compiler to generate a shared library instead of a

regular Solaris executable file. On Win32, the following command builds a dynamic link

library (DLL) HelloWorld.dll using the Microsoft Visual C++ compiler:

cl -Ic:\java\include -Ic:\java\include\win32 -MD -LD HelloWorld.c -FeHelloWorld.dll

In the test library the code should be (note that the prototype only support for win32)

String str = " cl -I" + TestKeys.JDK_PATH + " -I" + TestKeys.JDK_PATH_WIN32 + " -MD -LD " + comProp.getComponentName() + ".c -Fe" + comProp.getComponentName() + ".dll"; Process P = Runtime.getRuntime().exec(str);

63

f. Execute the HelloWorld class using the Java Reflection. Both the class file

(HelloWorld.class) and the native library (HelloWorld.dll) are loaded at runtime.

Figure 38. Process of invocation C component

64

5.3 Data-Tier Design

Figure 39. Entity Relationship Diagram (ERD)

65

Chapter 6. Project Implementation

The project implementation include HTML layout, java programming (core

components of test library and JSP server–side scripts) and SQL scripts generating.

6.1 Client Implementation

6.1.1 Execute test in the from client request (result.jsp) ... // get component adapter of the test component ComponentAdapter adapter = testController.getComponentAdapter(testsuite); // get test driver IDriver driver = testController.dispatchTestDriver(adapter); // set expecresult for test comparision databean.setExpectResult(request); //save test suite information int suite = databean.saveTestSuite(testsuite); // save test case information int tcase = databean.saveTestCase(testcase,Integer.parseInt(testsuite.getId())); // run test TestRunner runner = new TestRunner(driver); Object obj = runner.run(); //save result in the session object for report, result display session.setAttribute("returnObj", obj); // validate the result TestValidator validator = new TestValidator(databean,obj); testresult = validator.getTestResult(); expectedresult = validator.getExpectedResult().toString(); ...

6.1.2 Load component profile from test repository (loadcomponentprofile.jsp) ... // get the profile name String comfile = request.getParameter("comfile"); // create new file object for component profile file File file = new File(comfile); // get component profile info from XML component profile

66

ComponentXMLDescriptor comdesc = new ComponentXMLDescriptor(file); // create component properties object ComponentProfileProperties comprop = new ComponentProfileProperties(comdesc); // create component profile loader ProfileLoader proloader = new ProfileLoader(comprop); // load component information to test data base. int loadResult = proloader.loadComponent(); ...

6.2 Middle-Tier Implementation

6.2.1 Create java wrapper class for C component (CAdapter.java) public boolean createWrapperClass(){

ComponentProfileProperties comProp = comAdapter.getComponentProperties(); if(comAdapter.getComponentType().equals("C")){ Config.load(TestKeys.CONFIG_FILE); String C_ComponentDir = Config.getValue(TestKeys.C_REPOSITORY_PATH); try { comAdapter.writeToJavaFile(comAdapter.javaCodeGenerate()); Process P1 = Runtime.getRuntime().exec("javac " + C_ComponentDir + comProp.getComponentName() + ".java"); Process P2 = Runtime.getRuntime().exec("java – Djava.library.path=. " + comProp.getComponentName()); String str = " cl -I" + TestKeys.JDK_PATH + " -I" + TestKeys.JDK_PATH_WIN32 + " -MD -LD " + comProp.getComponentName() + ".c -Fe" + comProp.getComponentName() + ".dll"; Process P3 = Runtime.getRuntime().exec(str); return true; } catch (IOException ex) { return false; } } else{

setComponentInstance(); setComponentMethod();

return true; }

} 6.2.2 Create header file for C component (CAdapter.java)

private boolean generateHeaderCComponent(){

67

ComponentProfileProperties comProp = comAdapter.getComponentProperties(); Config.load(TestKeys.CONFIG_FILE); String C_ComponentDir = Config.getValue(TestKeys.C_REPOSITORY_PATH); try { Process P = Runtime.getRuntime().exec("javah -jni " + comProp.getComponentName()); File wrapperFile = new File(C_ComponentDir + "\\" + comProp.getComponentName()); if(wrapperFile.exists()) return true; return false; } catch (IOException ex) { return false; } } 6.2.3 Get Method of test component for invocation (JavaAdapter.java)

public Method getComponentMethod(){

Method runMethod; try{ TestMethod testMethod = testEntity.getTestedMethods(); if(testMethod.getModifier(). indexOf(ComponentProfileKeys.STATIC)>0) { Class classObj = Class.forName(testEntity.getComponentName()); runMethod = classObj.getMethod(testMethod.getMethodName(), testMethod.getArgumentTypes() ); } else{ Object testObj = getComponentInstance(); runMethod = testObj.getClass().getMethod(testMethod.getMethodName(), testMethod.getArgumentTypes() ); } } catch(NoSuchMethodException ne){ Log.ERROR("NoSuchMethodException", this.getClass().toString()); return null; } catch(ClassNotFoundException ce){ Log.ERROR("ClassNotFoundException", this.getClass().toString()); return null; } return runMethod; }

68

6.2.4 Create instance of the testing component (Runner.java)

public Object getComponentInstance(){

TestMethod method = testEntity.getTestedMethods(); // if the method is static, no constructor need to be instantiated if(method.getModifier().indexOf(ComponentProfileKeys.STATIC)>0) return null; try{ Class componentClass = Class.forName(testEntity.getComponentName()); ComponentConstructor constructor = testEntity.getContructors()[0]; Constructor cons = componentClass.getConstructor(constructor.getArgumentTypes()); return cons.newInstance(constructor.getArgumentValues()); } catch(ClassNotFoundException ce){ Log.ERROR("ClassNotFoundException", this.getClass().toString()); return null; } catch(IllegalAccessException ie){ Log.ERROR("IllegalAccessException ", this.getClass().toString()); return null; } catch(InstantiationException ie){ Log.ERROR("InstantiationException", this.getClass().toString()); return null; } catch(NoSuchMethodException ne){ Log.ERROR("NoSuchMethodException", this.getClass().toString()); return null; } catch(InvocationTargetException ie){ Log.ERROR("InvocationTargetException", this.getClass().toString()); return null; } } 6.2.5 Run test for testing component (Runner.java)

public Object doTest(){

Method runMethod = _driver.getComponentMethod(); Object objrun = _driver.getComponentInstance(); if(runMethod != null){ try{ TestMethod testeMethod = _driver.getDrivenComponent().getTestedMethods(); Object[] values = testeMethod.getArgumentValues(); return runMethod.invoke(objrun, testeMethod.getArgumentValues()); }

69

catch(InvocationTargetException ie){ Log.ERROR("InvocationTargetException", this.getClass().toString()); return null; } catch(IllegalAccessException ie){ Log.ERROR("IllegalAccessException ", this.getClass().toString()); return null; } } return null; }

6.3 Data-Tier Implementation

6.3.1 Create table for test suite CREATE TABLE MS_TESTSUITE (TestSuiteID INTEGER PRIMARY KEY, TestSuiteName VARCHAR2(50), TestOS VARCHAR2(20), IPAddress VARCHAR2(20), DOMAIN VARCHAR2(50), ReleaseVersion VARCHAR2(20), ProjectName VARCHAR2(100))

6.3.2 Create table for test case CREATE TABLE MS_TESTCASE(TestCaseID INTEGER PRIMARY KEY, TestCaseName VARCHAR2(150), TesterName VARCHAR2(50), Description VARCHAR2(4000), TestDate VARCHAR2(100), TestSuiteID INTEGER, ComponentID INTEGER, ConstructorID INTEGER, MethodID INTEGER)

6.3.3 Create tables for test components CREATE TABLE MS_COMPONENTPROFILE (ComponentProfileID INTEGER PRIMARY KEY,Name VARCHAR2(50),PACKAGE VARCHAR2(100),FullyQualifiedName VARCHAR2(200),Version VARCHAR2(20),Company VARCHAR2(50),ComponentType VARCHAR2(10),Modifier VARCHAR2(30),HasConstructor VARCHAR2(1),HasMethod VARCHAR2(1),Extends VARCHAR2(30),Implement VARCHAR2(30),Signature VARCHAR2(500), Testable VARCHAR2(1), Author VARCHAR2(50))

6.3.4 Create tables for test constructor CREATE TABLE MS_CONSTRUCTORPROFILE (ConstructorID INTEGER PRIMARY KEY,ComponentProfileID INTEGER,Name VARCHAR2(50), Modifier VARCHAR2(30),HasParameters VARCHAR2(1), NumberOfArgument INTEGER, ExceptionThrown VARCHAR2(1), ExceptionClass VARCHAR2(500), Signature VARCHAR2(500))

6.3.5 Create tables for test method CREATE TABLE MS_METHODPROFILE (MethodID INTEGER PRIMARY KEY, ComponentProfileID INTEGER,Name VARCHAR2(50), NumberOfArg INTEGER, Modifier VARCHAR2(30),ExceptionThrown VARCHAR2(200),HasOutPut

70

VARCHAR2(1),ReturnType VARCHAR2(100),HasParameters VARCHAR2(1),Signature VARCHAR2(500),ExceptionClass VARCHAR2(500))

6.3.6 Create tables for test argument CREATE TABLE MS_ARGUMENTPROFILE (ArgumentID INTEGER PRIMARY KEY, ComponentID INTEGER, OperationName VARCHAR2(20), OperationID INTEGER, DataType VARCHAR2(20), IsArray VARCHAR2(1), IsOutput VARCHAR2(1), Value VARCHAR2(4000));

6.3.7 Create tables for test set XML CREATE TABLE MS_SERIALIZATION (ID INTEGER primary key, Name VARCHAR2(100), TestCaseID INTEGER, Content VARCHAR2(4000))

6.3.8 Create tables for test user CREATE TABLE MS_USER(UserID INTEGER PRIMARY KEY, UserName VARCHAR2(12), Password VARCHAR2(12), FirstName VARCHAR2(25), LastName VARCHAR2(25), Mi VARCHAR2(1), UserRole INTEGER)

6.3.9 Create tables for test user roles CREATE TABLE MS_USER_ROLES(RoleId INTEGER PRIMARY KEY, Name VARCHAR2(20))

6.3.10 Create tables for test result CREATE TABLE MS_TESTRESULT (ResultID INTEGER primary key, TestSuiteId INTEGER, TestCaseId INTEGER, ComponentId INTEGER, MethodId INTEGER, ExpectResult VARCHAR2(200), ActualResult VARCHAR2(100), ResultType VARCHAR2(50), ResultDate VARCHAR2(100), Result VARCHAR2(8))

71

Chapter 7. Illustration of Testing MathComponent with SCTF

The purpose of this chapter is to explain in details of implementation for SCTF

operations.

7. 1 Pre-test operation

Before the test component is tested by SCTF, its profile from the component

repository has to be loaded in the test server database. User can select the tested

component under Load Component Profile menu.

The list of component profile is encapsulated in an Enumeration object that is created

from ComponentProfileSorter class.

ComponentProfileSorter ns = new ComponentProfileSorter(); Enumeration enu = ns.getComponentProfileList(comtype);

The ComponentProfileSorter basically run a process to list component profile in an

assigned directory in the component repository server.

72

. . . Vector fileList = new Vector();

try {

String[] cmd = new String[3]; cmd[0]="cmd.exe"; cmd[1]="/c"; cmd[2]="dir " + path + "*.xml /b"; Process p = Runtime.getRuntime().exec(cmd);

BufferedReader stdInput = new BufferedReader( new InputStreamReader(p.getInputStream()));

String s; while((s=stdInput.readLine())!=null){ fileList.addElement((Object)s); } }

File names of component profiles (.xml) are passed in a Vector object and then

converted to Enumeration.

When a component profile is selected, the user submits the form to pass the

component name to the test server. The test server will receive the component name and

insert component profile information to the test database by using loadComponent in

ProfileLoader class.

. . . String comfile = request.getParameter("comfile"); File file = new File(comfile); ComponentXMLDescriptor comdesc = new ComponentXMLDescriptor(file); ComponentProfileProperties comprop = new ComponentProfileProperties(comdesc); ProfileLoader proloader = new ProfileLoader(comprop); int loadResult = proloader.loadComponent(); . . .

The loadComponent is implemented with SQL Insert statement to insert

appropriate component profile fields in the MS_ComponentProfile,

MS_ConstructorProfile, MS_MethodProfile and MS_ArgumentProfile tables. After

loading component profile, user will see details of the loaded component in the next

screen as follows:

73

7.2 Test Operations

A component testing process in SCTF is start with a defined setting hierarchy:

TestSuite → TestCase → TestComponent → TestConstructor → TestMethod

→ TestArgument.

Open the Setup Test menu, the TestSuite Administration will appear as follows:

74

Select Create New TestSuite. The main TestSuite screen will appear as follows:

75

Enter the test suite information and then click the Create TestSuite button to

submit the test suite information to the test server. The test suite information will be kept

persistent in the session by declaring TestSuiteBean in the JSP session

<jsp:useBean id="testsuite" scope="session" class="com.testcom.testbean.TestSuiteBean" />

The testsuite information is set in the attributes of TestSuiteBean by function

ProcessRequest.

. . . _testSuiteId = request.getParameter("testsuiteid"); _name = request.getParameter("testsuitename"); _version = request.getParameter("version"); _project = request.getParameter("projectname"); _ipaddress = request.getParameter("ipaddress"); _domain = request.getParameter("domain"); _operatingSystem = request.getParameter("operatingSystem"); . . .

After test suite form is submitted, the test case screen will appear as follows:

76

The test suite information (from TestSuiteBean) is displayed in the test case

screen. Enter test case information and click on the Create Test Case button to submit the

test case information to the test server. Similar to creation of test suite, test case

information will be kept persistent in the JSP session by declaring TestCaseBean. The

test case information is set in the attributes of TestCaseBean by function

ProcessRequest.

<jsp:useBean id="testcase" scope="session" class="com.testcom.testbean.TestCaseBean"/>

. . . _testCaseId = request.getParameter("testcaseid"); _name = request.getParameter("testcasename"); _tester = request.getParameter("tester"); _description = request.getParameter("description"); _createdDate = request.getParameter("createdate"); . . . and the TestCaseBean is also set as an attribute of TestSuiteBean.

After the test case form is submitted, the component screen will

appear as follows:

77

Tester can select different constructor (if any) and methods of the test component.

The component information will automatically displayed based on the tester’s selection

in the test case setup. Like test suite and test case setting, component information is also

kept persistent in the session by declaring TestComponentBean in the JSP session

<jsp:useBean id="testcomponent" scope="session"

class="com.testcom.testbean.TestComponentBean"/>

The test component information is set in the attributes of TestComponentBean by

function ProcessRequest, and the TestComponentBean is set as an attribute in the

TestCaseBean.

. . . _testComponentId = request.getParameter("testcomponentid"); _name = request.getParameter("testcompname"); _fullyQualifiedName = request.getParameter("fullyqualifiedname"); _package = request.getParameter("package"); _version = request.getParameter("version"); _company = request.getParameter("company"); _author = request.getParameter("author"); _extends = request.getParameter("extends");

78

_modifier = request.getParameter("modifier"); _hasConstructor = request.getParameter("hasconstructor"); _hasMethod = request.getParameter("hasmethod"); _implement = request.getParameter("implement"); _testable = request.getParameter("testable"); _componentType = request.getParameter("componenttype"); signature = request.getParameter("signature"); . . .

After the component form is submitted, the screen of constructor and method setting will

appear as follows:

The test constructor and test method are displayed based on testers’ selection.

Tester only needs to insert the test data such as parameters of constructor and method.

After click on the Continue button, test constructor and method information will

be kept in the TestConstructorBean, TestMethodBean with declaration these beans in

the JSP session.

<jsp:useBean id="testconstructor" scope="session" class="com.testcom.testbean.TestConstructorBean"/> <jsp:useBean id="testmethod" scope="session" class="com.testcom.testbean.TestMethodBean"/>

79

Likewise, the test constructor and method are set as attributes of the

TestComponentBean. Constructor and method information of the component are set in

the attributes of TestConstructorBean, TestMethodBean by function

ProcessRequest.

. . . // for TestConstructorBean _constructorid = request.getParameter("c_testconstructorid"); _name = request.getParameter("c_testconstructname"); _numberOfArg = request.getParameter("c_numberofarg"); _signature = request.getParameter("c_signature"); _throwsException = request.getParameter("c_throwexception"); _hasParameter = request.getParameter("c_hasparameter"); _exceptionClass = request.getParameter("c_exceptionclass"); _modifier = request.getParameter("c_modifier"); . . . // for TestMethodBean _methodid = request.getParameter("m_testmethodid"); _name = request.getParameter("m_testmethodname"); _numberOfArg = request.getParameter("m_numberofarg"); _signature = request.getParameter("m_signature"); _throwsException = request.getParameter("m_throwexception"); _hasParameter = request.getParameter("m_hasparameter"); _exceptionClass = request.getParameter("m_exceptionclass"); _modifier = request.getParameter("m_modifier"); _hasOutput = request.getParameter("m_hasoutput"); _returnType = request.getParameter("m_returntype"); . . . The complete test bed after settings will be displayed as follows:

80

When user enter the expected value, and click Run Test button, the information of

the test bed will be submitted to the test server. The test server controller determines what

component adapter should be used for this testing process.

. . . ComponentAdapter adapter = testController.getComponentAdapter(testsuite); . . . // excerpt from getComponentAdapter() function String comType = testsuiteBean.getTestCaseBean().getTestComponentBean().getComponentType(); String result = this.getTestRequest(testsuiteBean); Document document = (Document)XMLUtil.getXMLDoc(result); TestSuite testSuite = new TestSuite(document); TestCase testCase = testSuite.getTestCase(); TestEntity testEntity = testCase.getTestEntities()[0]; ComponentProfileProperties comProp = null; try { comProp = new ComponentProfileProperties( new ComponentXMLDescriptor( this.getSerializedComponent(testsuiteBean))); } . . .

Then the test driver is generically built for the component.

81

. . . IDriver driver = testController.dispatchTestDriver(adapter); . . . The adapter associated with the test driver creates a wrapper class.

//excerpt from createWrapperClass() method in TestDriver class. . . . if(comAdapter.getComponentType().equals("C")){

Config.load(TestKeys.CONFIG_FILE); String C_ComponentDir = Config.getValue(TestKeys.C_REPOSITORY_PATH); try { comAdapter.writeToJavaFile(comAdapter.javaCodeGenerate()); Process P1 = Runtime.getRuntime().exec("javac " + C_ComponentDir + comProp.getComponentName() + ".java"); Process P2 = Runtime.getRuntime().exec("java " + comProp.getComponentName()); String str = " cl -I" + TestKeys.JDK_PATH + " -I" + TestKeys.JDK_PATH_WIN32 + " -MD -LD " + comProp.getComponentName() + ".c -Fe" + comProp.getComponentName() + ".dll"; Process P3 = Runtime.getRuntime().exec(str); return true; } catch (IOException ex) { return false; } } else{ setComponentInstance(); setComponentMethod();

return true; } . . . //excerpt for getComponentInstance() method in JavaAdapter class . . . Class componentClass = Class.forName(testEntity.getComponentName()); ComponentConstructor constructor = testEntity.getContructors()[0]; Constructor cons = componentClass.getConstructor(constructor.getArgumentTypes()); return cons.newInstance(constructor.getArgumentValues()); . . . //excerpt for getComponentMethod() method in JavaAdapter class . . . Method runMethod; try{

TestMethod testMethod = testEntity.getTestedMethods(); if(testMethod.getModifier(). indexOf(ComponentProfileKeys.STATIC)>0){

Class classObj =

82

Class.forName(testEntity.getComponentName()); runMethod = classObj.getMethod(testMethod.getMethodName(), testMethod.getArgumentTypes()); } else{ Object testObj = getComponentInstance(); runMethod = testObj.getClass().getMethod(testMethod.getMethodName(), testMethod.getArgumentTypes()); } } . . .

After the test driver and wrapper class are generated, test server will call

TestRunner to invoke the tested component via the wrapper class.

. . . //excerpt for run() method in TestRunner. Method runMethod = _driver.getComponentMethod(); Object objrun = _driver.getComponentInstance(); if(runMethod != null){ try{ TestMethod testeMethod = _driver.getDrivenComponent().getTestedMethods(); Object[] values = testeMethod.getArgumentValues(); return runMethod.invoke(objrun, testeMethod.getArgumentValues()); } . . .

The TestRunner return the test result as an Object. The expected result is set in

DataBean object for later comparison with the test result via TestValidator.

. . . TestValidator validator = new TestValidator(databean,obj); testresult = validator.getTestResult(); expectedresult = validator.getExpectedResult().toString(); . . . In addition, test bed information also save in the database for next testing.

. . . int suite = databean.saveTestSuite(testsuite); int tcase = databean.saveTestCase(testcase,Integer.parseInt(testsuite.getId())); . . .

and the test result is also save for test report.

...

83

int save_success = testController.saveTestResult(testsuite,validator); ...

The test result page will be displayed as follows:

84

Chapter 8. Performance and Benchmarks

A simple testable component (figure 40) was used in evaluation the effectiveness

of this test tool versus other test ting approach.

/** * Title: Master Project Software Component Test Tool * Description: Software Component Test Library * Copyright: Copyright (c) 2002 * Company: San Jose State University * @author Pham Quang Cuong * @version 1.0 */ public class TestableComponentTwo { public TestableComponentTwo() { } public String DuplicateString(String dup){ return dup+dup; } public int getStringLength(String str){ return str.length(); } public int AddNumber(int one, int two){ return one + two; } public char getFirstChar(String str){ return str.charAt(0); } }

Figure 40. Source code snap shot of TestableComponentTwo

8.1 Traditional testing Efforts Estimation

The traditional testing was carried out using assert statements provided in the Sun

JDK 1.4.0. Figure 40 shows a screen shot of an example run of the traditional testing

method component. Examples of both a successful and a failed run can be seen. Figure

41 shows a source code snap shot of the traditional testing module developed.

85

8.1.1 Cost to develop test cases

The effort involved in developing a single test case (unit test) using traditional

efforts was the equivalent of .5 man-hours for each component.

8.1.2 Number of test drivers

The main function in each component was used for driving the test cases. Thus,

one test driver per component was used to drive multiple test cases.

8.1.3 Number of test cases

Number of test cases per testable component was limited to three.

8.1.4 Time to develop test cases

The time to develop a single test case was 30 minutes i.e. approx. 1.5 hours of

development time for writing three test cases for each testable component.

8.1.5 Testing time

The total testing time, on average, per test case, was about 35 minutes. This

included development time, running time, and analysis time.

86

Figure 41. Screen shot of traditional testing method component run

/** * Title: Master ProjectProject Software Component Test Tool * Description: Software Component Test Library * Copyright: Copyright (c) 2002 * Company: San Jose State University * @author Pham Quang Cuong * @version 1.0 */ public class ClassicTestMethodComponentTwo { public ClassicTestMethodComponentTwo() { } public static void main(String[] args) { TestableComponentTwo tct = new TestableComponentTwo(); // assert is a java 1.4 feature. To use assert // javac -source 1.4 <javasource> assert tct.DuplicateString("bye").equals("byebye"); assert new Character(tct.getFirstChar("cba")).charValue() == 'c'; assert tct.AddNumber(4, 6) == 10; assert tct.getStringLength("123456") == 6; } }

Figure 42. Source code snap shot for traditional testing method component

87

8.2 Testing effort with JUnit unit testing tool

JUnit is an open source industry standard tools used widely in software

development circles. It provides the unit tester with an infrastructure to create test cases

for individual/suite of modules. As part of this case study, JUnit was used for comparison

analysis with the tool written as part of this project. Figure 42 shows a screen shot of a

run of the JUnit testing method component. Examples of a failed run can be seen. Figure

43 shows a source code snap shot of the JUnit testing module developed.

8.2.1. Cost to develop test cases

The effort involved in developing a single test case (unit test) using the JUnit tool

and its infrastructure was the equivalent of 1.5 man-hours for each component. Once the

infrastructure was set up and familiarized, the effort required progressively decreased.

8.2.2. Number of test drivers

JUnit provides the complete infrastructure for setting up and running the test

cases. Thus, a single test driver was used to drive test cases for all the components

8.2.3. Number of test cases

Only one test case was developed purely for comparative analysis

8.2.4. Time to develop test cases

The time to develop a single test case was 90 minutes.

8.2.5. Testing time

The total testing time, on average, per test case, was about 100 minutes.

88

Figure 43. Screen shot of a JUnit testing method component run

89

Figure 44. Source code snap shot for JUnit testing method component

import junit.framework.*; /** * Title:Master Project Software Component Test Tool * Description: Software Component Test Library * Copyright: Copyright (c) 2002 * Company: San Jose State University * NetBeans JUnit based test * @author Pham Quang Cuong * @version 1.0 */ public class TestableComponentTwoTest extends TestCase { public TestableComponentTwoTest(java.lang.String testName) { super(testName); } public static void main(java.lang.String[] args) { junit.textui.TestRunner.run(suite()); } public static Test suite() { TestSuite suite = new TestSuite(TestableComponentTwoTest.class); return suite; } /** Test of DuplicateString method, of class TestableComponentTwo. */ public void testDuplicateString() { System.out.println("testDuplicateString"); TestableComponentTwo tct = new TestableComponentTwo(); assertTrue(tct.DuplicateString("bye").equals("byebye")); } /** Test of getStringLength method, of class TestableComponentTwo. */ public void testGetStringLength() { System.out.println("testGetStringLength"); TestableComponentTwo tct = new TestableComponentTwo(); assertTrue(tct.getStringLength("12345") == 6); } /** Test of AddNumber method, of class TestableComponentTwo. */ public void testAddNumber() { System.out.println("testAddNumber"); TestableComponentTwo tct = new TestableComponentTwo(); assertTrue(tct.AddNumber(4, 6) == 10); } /** Test of getFirstChar method, of class TestableComponentTwo. */ public void testGetFirstChar() { System.out.println("testGetFirstChar"); TestableComponentTwo tct = new TestableComponentTwo(); assertTrue(new Character(tct.getFirstChar("cba")).charValue() == 'c'); } }

90

8.3 Testing effort with the Software Component Testing Framework (SCTF)

An illustrated sample demo of this case study can be seen in the section 4.1.

8.3.1. Cost to develop test cases

Using this testing tool, the effort involved in developing a single test case was the

equivalent of .05 man-hours of entering information through an interactive GUI interface.

This is a fraction of the time taken using traditional testing techniques. Additionally, a

non-developer user can carry out the testing, as it does not involve writing any code. This

drastically reduces the costs and protects the engineering human resources for more

important tasks.

8.3.2. Number of test drivers

The same test driver is used to drive all the test cases i.e. the tester module.

8.3.3. Number of test cases

Using the tool, a large number of test cases per component were developed,

largely because of ease and speed.

8.3.4. Time to develop test cases

The time to develop a single test case was 3 minutes or approximately about 10

minutes of development time for writing three test cases for each testable component.

8.3.5. Testing time

The total testing time, on average, per test case, was about 3 minutes. This

included development time, running time, and analysis time. An automatic report was

generated after test completion, which reduced analysis time. Additionally the detailed

report generated makes it easy for a novice tester to be able to record and understand the

test results.

91

8.4. Comparative Analysis of the different testing methods

The three testing methods used in this case study for a comparative analysis all

performed to the required standards and accomplished the task. Yet, each tool

demonstrated its own capabilities and advantages over other tools.

8.4.1. The Traditional Testing Method (ad hoc testing)

This method was the quickest solution to a problem of quickly testing

components. It requires the user to have good prior knowledge of the Java language, local

access to the testable component binary and prior understanding and knowledge of the

testable component API.

The time required for development was reasonable but the results analysis was

problematic because of minimal information. On error, a Java exception is generated. On

success, this method prints nothing i.e. a blank line. No reports are generated that could

be saved, moved around or run repeatedly.

8.4.2. The JUnit Testing Method

JUnit provided the most reliable and standardized infrastructure used in the

industry for testing. It provides a lot of functionality at hand that could be used to create

suites of tests for different testable components. It requires the user to have good prior

knowledge of the Java language, local access to the testable component binary, prior

understanding and knowledge of the testable component API and prior understanding and

knowledge of the JUnit infrastructure.

92

The time required for development of test cases for JUnit was orders more than

the other methods. It provides a good success/failure test reporting format and the ability

to save and run the test repeatedly. Yet, it requires a lot of fundamental Java and JUnit

knowledge from the user. This can tie up costly resources in long testing runs.

8.4.3. Testing Using the Software Component Testing Framework

Testing using the tool developed, as part of this project required the minimum

amount of resources, in time and in costly human interaction and yet provided with the

simplest and easiest user interface and most information to help the tester understand and

analyze the results. This tool does not require the user to have prior knowledge of Java,

C, C++, etc… or prior understanding and knowledge of the testable component API. The

only requirement is local or remote access to the testable component binary.

This tool does not make any expectations of the user. It allows a novice to

develop test cases with a clean interactive GUI. The user is provided with the API

specifications for each component and is allowed to make dynamic changes to the test

case. This can result in huge cost savings, as expensive developers are not tied up in

creating and running test cases. It provides an excellent success/failure test reporting

format and the ability to save and run the test repeatedly through the GUI. The test case

and the results are merged together into a human readable XML document. This tool also

provides a client/server architecture that allows multiple clients to run test cases on the

server. This means that more than one tester can execute test cases on a single testable

component at the same time resulting timesavings.

93

As indicated, each testing method has its own benefits and drawbacks, yet, the

tool as part of this project is leaps and bounds ahead of the other alternatives. An

adoption of this tool by any development group will show immediate results in monetary

and human savings.

Set up one test case (test data, select component, method etc..)

Build driver Execute method Total

2-3 minutes ~ 368 ms ~ 80 ms 3-5 minutes

Table 5. Estimated time to perform one test case for a component with the Software Component Testing Framework

Test Tool Method

Number of test case Number of test drivers Total Time (minutes)

SCTF 15 1 45 Junit 15 1 450 Ad hoc 15 15 450

Table 6. Estimate time to perform one test component with different testing approach (for a component with 3 test cases)

94

No Test case description Number 1 Number 2 Expected

Value 1 Parameter 1 upper boundary

(valid) 2147483647 0 2147483647

2 Parameter 2 upper boundary (valid)

0 2147483647 2147483647

3 Parameter 1 upper boundary (invalid)

2147483648 0 Exception

4 Parameter 2 upper boundary (invalid)

0 2147483648 Exception

5 Parameter 1 lower boundary (valid)

-2147483648 0 -2147483648

6 Parameter 2 lower boundary (valid)

0 -2147483648 -2147483648

7 Parameter 1 lower boundary (invalid)

-2147483649 0 Exception

8 Parameter 2 lower boundary (invalid)

0 -2147483649 Exception

9 Total boundary (valid) 2147483647 -2147483648 -1 10 Total boundary (valid) -2147483648 2147483647 -1 11 Total boundary (invalid) 2147483647 1 Exception 12 Total boundary (invalid) -2147483648 -1 Exception 13 Invalid parameter 1 0.5 1 Exception 14 Invalid parameter 2 1 0.5 Exception 15 Total zero 0 0 0

Table 7. Test case coverage for function AddNumber of TestableComponentTwo component with Boundary Method of BlackBox Approach

95

Chapter 9. Deployment, Operations, Maintenance

9.1 Software Deployment

Applicationservernode

Databaseservernode

WebserverApacheTomcat

Agent inCVS

server

Test Clientweb

browser

Http port 8080

classpath or selected port

Tcp port 1515

usebean

Figure 45. Software deployment diagram for distributed system

Applicationservernode

Databaseservernode

WebserverApacheTomcat

Test Clientweb

browser

Http port 8080

Tcp port 1515

usebean

Figure 46. Software deployment for simple system

96

9.2 Hardware deployment

Test client PCWin2K, IE 5.5 Web server Pentium 4

1.6 GHz, 1 GB DDRWin2k

Apache Tomcat 1.4

Database serverPentium 4

1.6 GHz, 2GB DDRWin2k, Oracle 8.1.7

Failover serverPentium 4

1.6 GHz, 2GB DDR,win2K, Oracle 8.1.7

Test server Pentium 41.6 GHz, 2GB DDR

Win2k CVS Servers

HttpPort 8080

TCPPort 1515

TCPPort 1515

TCP/IPPort 9000

Comm Hub

Internet

TCPPort 1515

HttpPort 8080

Printer Report

Figure 47. Hardware Deployment for distributed system diagram

97

Test client PCWin2K, IE 5.5 Web server Pentium 4

1.6 GHz, 1 GB DDRWin2k

Apache Tomcat 1.4

Database serverPentium 4

1.6 GHz, 2GB DDRWin2k, Oracle 8.1.7

Failover serverPentium 4

1.6 GHz, 2GB DDR,win2K, Oracle 8.1.7

Test server Pentium 41.6 GHz, 2GB DDR

Win2k

HttpPort 8080

TCPPort 1515

TCPPort 1515

Comm Hub

Internet

TCPPort 1515

HttpPort 8080

Printer Report

Figure 48. Hardware deployment for simple system

98

Chapter 10. Summary, Conclusions, and Recommendations

10.1 Summary

Software Component Testing Framework (SCTF) is a testing tool for software

components that provides a cross-technology to test component written by different

language. The SCTF aims to reduce the cost of unit and integration test and improve the

reliability of testing quality. SCTF design also aims to resolve issues that most software

component vendors have challenged. That is to establish a standard in development and

testing for software components.

10.2 Conclusions

The Software Component Testing Framework (SCTF) in this project is a

proposed framework and it provides a prototype of a full-blown deployment application

of that should be put under intensive design and development. The current prototype only

supports for components that have a single java class or a single C program file with

parameters of tested method having a primitive type.

The most advantage feature of SCTF is to provide a generic test driver for all test

cases. The test driver only needs to couple with the test adapter to build a wrapper class

for a tested component. This can help testers to reduce the testing time significantly. In

addition, SCTF can reduce the time and cost in testing component by providing a feature

to record test cases for later testing in the same components, and it also provides a test

coverage statistically analytical feature with graphical reports.

10.3 Recommendations for Further Research

The Software Component Testing Framework (SCTF) can be leverage to higher

phase that can provide higher functional requirements with capabilities:

99

• To test components with non-primitive type parameters.

• To handle all exception thrown from the tested components.

• To trace component defects

• To measure component performance in a particular test environment.

• To test component with multiple classes or functions.

• To test distributed component such as CORBA or COM.

100

Glossary

Ad Hoc Testing: Testing carried out using no recognized test case design technique

Benchmarks: Programs that provide performance comparison for software, hardware,

and systems.

BIT component: is a software component with built-in test mechanisms, which are

provided through one or more interfaces

Black box testing: A testing method where the application under test is viewed as a

black box and the internal behavior of the program is completely ignored. Testing occurs

based upon the external specifications. Also known as behavioral testing, since only the

external behaviors of the program are evaluated and analyzed.

Condition Coverage: A test coverage criteria requiring enough test cases such that each

condition in a decision takes on all possible outcomes at least once, and each point of

entry to a program or subroutine is invoked at least once. Contrast with branch coverage,

decision coverage, multiple condition coverage, path coverage, and statement coverage.

Defect: The difference between the functional specification (including user

documentation) and actual program text (source code and data). Often reported as

problem and stored in defect tracking and problem-management system. Defect also

called a fault or a bug, a defect is an incorrect part of code that is caused by an error. An

error of commission causes a defect of wrong or extra code. An error of omission results

in a defect of missing code. A defect may cause one or more failures

Integration testing : testing of combined parts of an application to determine if they

function together correctly. The 'parts' can be code modules, individual applications,

101

client and server applications on a network, etc. This type of testing is especially relevant

to client/server and distributed systems.

Software Component: is an identifiable part of a larger program or construction.

Usually, a component provides a particular function or group of related functions.

Test Oracle: a mechanism to produce the predicted outcomes to compare with the actual

outcomes of the software under test

Test Bed: An environment containing the hardware, instrumentation, simulators,

software tools, and other support elements needed to conduct a test.

Test Case: A set of test inputs, executions, and expected results developed for a

particular objective.

Test Coverage: The degree to which a given test or set of tests addresses all specified

test cases for a given system or component.

Test Driver: A software module or application used to invoke a test item and, often,

provide test inputs (data), control and monitor execution. A test driver automates the

execution of test procedures.

Test Harness: A system of test drivers and other tools to support test execution (e.g.,

stubs, executable test cases, and test drivers). See: test driver.

Test Item: A software item that is the object of testing.

Test Log: A chronological record of all relevant details about the execution of a test.

Test Plan: A high-level document that defines a testing project so that it can be properly

measured and controlled. It defines the test strategy and organized elements of the test

life cycle, including resource requirements, project schedule, and test requirements

102

Test Procedure: A document, providing detailed instructions for the [manual] execution

of one or more test cases. Often called - a manual test script.

Test Stub: A dummy software component or object used (during development and

testing) to simulate the behavior of a real component. The stub typically provides test

output.

Test Suites: A test suite consists of multiple test cases (procedures and data) that are

combined and often managed by a test harness.

Testing: The execution of tests with the intent of providing that the system and

application under test does or does not perform according to the requirements

specification.

103

References 1. Chen, C., Gao, J., Hsia, P., Kung, D., & Toyoshima, Y., “Object State Testing for

Object Oriented Programs.” Proc. IEEE COMPSAC ’95, 1995, 232-238.

2. Daniels, F. J. & Tai, K.C. “Test Order for Inner Class Integration Testing of Object-

Oriented Software.” IEEE Software, 1997, 601-606.

3. Gao, J., Gupta, K., Gupta, S., & Shim, S. “On Building Testable Software

Components.” Proc. Cost-Based Software System, 2002, 108-121.

4. Gao, J., “Design Specifications – A Systematic Solution For Program Testing”

v.1. Unpublished Project Report, San Jose State University, June 2002.

5. Gao, J. “Component Testability and Component Testing Challenges.” Proceedings

of Star’99, SQE, 1999.

6. Keith, S, “Component Based Development and Object Modeling”. Version 1,

Sterling Software, February 1997.

7. McGregor, J. D.“Component Testing.” JOOP Column, 1996.

8. Paul A., & Stuart F, “Component-Based Development for Enterprise Systems –

Applying”. The SELECT Perspective. Cambridge University Press, 1998.

9. Perrone, J.P. & Venkata S.R. Chaganti. Building Java Enterprise Systems with

J2EE: The Authoritative Solution. Sams Publishing, 2002.

10. Pressman, R. S., Software Engineering: A Practitioner's Approach, 4th Ed.,

McGraw-Hill, 1997.

11. Smith, M.D & Robinson, D. J. “A Frame Work for Testing Object-Oriented

Programs.” Journal of Object-Oriented Programming, 1992, 45-53.

104

12. Szyperski, C. Component Software- Beyond Object Oriented Programming.

Addison Wesley, 1997.

13. Turner, C. D & Robson, D.J. “The State-Based Testing of Object Oriented

Programs.” Proc. IEEE Conf. Software Maintenance, 1993, 302-310.

14. Tse, T. H., & Xu, Z. “Test Case Generation for Class-Level Object-Oriented

Testing.” Proc. 9th International Software Quality Week, 1996, (4T4), 1-12.

15. Voas, J. M. & Miller, K.W.," Software Testability: The New Verification", IEEE

Software, Vol. 12, No. 3: May 1995, pp. 17-28.

16. Wills, A.C., “Designing Component Kits and Architectures with Catalysis”,

Technical Report, TriReme International Ltd., 1999.

17. Yingxu W., Graham K., & Hakan W., “A Method for Built-in Tests in Component-

based Software Maintenance”, Research Centre for Systems Engineering,

Southampton Institute.

18. EasyCharts, http://www.objectplanet.com/EasyCharts

19. Java Remote Invocation, http://java.sun.com/products/jdk/rmi/

20. Java Native Interface, http://java.sun.com/products/jdk/1.2/docs/guide/jni/

http://www.objectplanet.com/EasyCharts

http://java.sun.com/products/jdk/rmi/

http://java.sun.com/products/jdk/1.2/docs/guide/jni/

105

Appendices

DESCRIPTION OF CDROM CONTENTS A.1. Java files: Argument.java

CAdapter .java

ClassUtil .java

ComponentAdapter .java

ComponentBinaryDescriptor .java

ComponentConstructor .java

ComponentCoverage .java

ComponentProfileInfo .java

ComponentProfileKeys .java

ComponentProfileProperties .java

ComponentProfileSorter .java

ComponentProperties .java

ComponentProperties .java

ComponentXMLDescriptor .java

Config .java

ConfigAdmin.java

Controller .java

DataBean .java

DBConfig .java

106

DBManager .java

DBUtil .java

DictionaryMgr.java

HashProperties .java

IDriver .java

JavaAdapter.java

Log .java

NotSupportClassException .java

NotTestableComponentException .java

NullResultSetException .java

PipeLineDictionary .java

ProfileIndexOutOfBoundsException .java

ProfileLoader .java

ProgressBarBean .java

Server .java

StringUtil .java

TestArgumentBean .java

TestCase .java

TestCaseBean.java

TestComponentBean .java

TestConstructorBean .java

TestDriver .java

TestEntity .java

107

TesterBean .java

TestKeys .java

TestMethod .java

TestMethodBean.java

TestResourceMgr .java

TestRunner .java

TestSuite .java

TestSuiteBean.java

TestSuiteCoverage.java

TestValidator .java

UserBean .java

Utils .java

UtilsStringTokenizer .java

XMLNodeDoesNotExistException .java

XMLUtil .java

A.2. JSP files: admin.jsp

componentprofile.jsp

ComponentProfileDetail.jsp

configadmin.jsp

createuser.jsp

defect.jsp

108

defectdetail.jsp

detailreport.jsp

frame.jsp

index.jsp

LoadComponentProfile.jsp

login.jsp

menu.jsp

print.jsp

report.jsp

result.jsp

screen.jsp

test.jsp

testbed.jsp

testcase.jsp

testcomponent.jsp

testsuite.jsp

A.3. SQL files: msproject.sql

ms_argumentprofile.sql

ms_componentprofile.sql

ms_constructorprofile.sql

ms_methodprofile.sql

109

ms_serialization.sql

ms_testcase.sql

ms_testresult.sql

ms_testsuite.sql

ms_user.sql

ms_userrole.sql

Documents

A Framework For Software Component Testing Tool In