LO-5: Testing

Lecturer: Ricardo Gonçalves

2

• Testing Overview• Conformance Testing

– Definition– Methodologies – Application Architecture

• Interoperability Checking– Definition– Methodologies – Application Architecture

Contents

3

What is Testing?

Testing is the process used to help identify: – Correctness – Completeness– Security– Quality

… of developed computer software.

Testing is a process of technical investigation, performed on behalf of stakeholders, that is intended to reveal quality-related information about the product with respect to the context in which it is intended to operate.

4

Testing life cycle

• Software testing life cycle:

Requirement Analysis

Test Planning

Test Development

Test Execution

Test Reporting

Retesting the Defects

5

Types of Testing

There are many types of tests, like:

• Performance;

• Robustness;

• Behaviour;

• Functions;

• Conformance;

• Interoperability;

Conformance Testing (CT)

7

Conformance Testing (CT)

• ISO/IEC TR 13233 defined conformance and conformity as “fulfilment by a product, process or service of all relevant specified conformance requirements”.

• ISO/IEC DIS 10641 defined conformance testing as "test to evaluate the adherence or no adherence of a candidate implementation to a standard”.

8

Conformance Testing (CT)

To apply the conformance testing to an implementation, the standards or specifications, must specify the requirements or criteria for conformance. In general this are done in a conformance clause or conformance statement.

Also, to help in the conformance testing, some standards describe methodologies and platforms to implement the conformance testing.

9

CT - Methodologies

• ISO 9646: “Conformance Testing Methodology and Framework”

• ISO10303 30’s parts a Conformance testing methodology and framework

• ETS 300 406 (Methods for Testing and Specification (MTS), Protocol and profile conformance testing specifications, Standardization methodology

10

CT General concepts

• System Under Test (SUT)

• Implementation Under Test (IUT)

• Abstract Test Suits (ATS).

SUT

IUTIs verified and

certified via

adopts

ATS Standard

11

CT Abstract Test Suits(ATS)

• ATS is defined by a set of Abstract Test Cases (ATC) describing in a conceptual format

• With the ATS defined for one AP, the Conformance Testing procedures can be executed on the IUT

• ATS needs to be instantiated creating the Executable Test Cases (ETC), which are the data and commands to be used to execute the test described in computer format.

12

CT Abstract Test Suits(ATS)

APGenerate set ofTest Purposes (TP)

Identify set ofTest Objectives (TO)

AP domainexperts

Derived fromInformation Requirements (IR)

ImplementationRequirements (IR)

+

for PreandPostprocessors

For PostProcessorsonly

Plan Abstract Test Cases (ATC) & Map TP to each ATCPlan Abstract Test Cases (ATC) & Map TP to each ATC

Input Application Semantics (IAS)Test Case (TC) for bothPost and Pre processorsGenerates

TC =(Input Data Specification – IDS,Veredict Criteria - VC)

TCs=(IDS, VC)UsingConstruct simple ATCs for Post and Pre processors

TCs=(IDS, VC)UsingConstruct simple ATCs for Post and Pre processors

Add ATC to ATSAdd ATC to ATS

ATS covers the TPsAs a whole for eachConformance Class ?

No

ATSdone!

TP coverageisadequateYes

13

CT – “A Methodology”

Test preparation

Analysis of the results

Report production

Test execution

Reports

SUT

ATS

ISO 10303, Conformance testing methodology

14

CT – Application example

EXPRESSModel

XMLDATA

Is in conformance??

Retailer

Manufacturer

Model of exchanged messages

Conformance testing of messages

Request quotation received well format

<Request_for_quotation> <buyer>Tomas Smith</buyer> <sellller>Furniture Shop</sellller> <product>Blue Chair</product> <quantity>20</quantity></Request_for_quotation>

ENTITY Request_for_quotation buyer: STRING seller: STRING product: STRING quantity: STRINGRequest_for_quotation

<Request_for_quotation> <buyer>Tomas Smith</buyer> <seller>Furniture Shop</seller> <product>Blue Chair</product> <quantity>20</quantity></Request_for_quotation>

15

CT – Application Architecture

Knowledge Base

Schematron Rules

XSD Model

XSL Math Functions

JAVAFunctions

ApplicationEngine

ApplicationEngine

SAX

JAV

A V

M

XALAN

ReportReport

Conformance Test Service

WSDL InterfaceWSDL Interface

Client Application

Execute Test

Test Result

Interoperability Checking (IC)

17

Interoperability Checking

• IEEE define interoperability as "the ability of two or more systems or components to exchange information and to use the information that has been exchanged.”

• Interoperability testing is the act of determining if end-to-end functionality between (at least) two communicating systems is as required by those base systems’ standards. “ETSI”

18

IC Methodology – General concepts

• System Under Test (SUT) • Implementation Under Test (IUT)• Abstract Test Suits (ATS)• Test Suite Structure and Test Purposes

(TSS&TP) • Interoperable Functions Statement (IFS) • Implementation eXtra Information for

Testing (IXIT) • Executable Test Suite (ETS)

19

IC – “A Methodology”

Base

(Protocol)

Standards

TSS & TP ATS

IFS and IXIT

Manual TestingImplementationTest selection and

parameterization

Development of Interoperability Test Specifications

ETSI activities

non-ETSI activities

Filled-in IFS

and IXIT

RE EUT

SUT

• TS 102 237-1; Interoperability test methods & approaches; Part 1: Generic approach to interoperability testing

From: In Pursuit of Interoperability (ITJ2582) - Scott Moseley; Steve Randall; Anthony Wiles.

20

IC – Application Architecture

Control, Logging, Coordination etc.

Encoding/DecodingSystem

TTCN-3 Runtime System

Adaption Layer

TTCN-3 Runtime Interface (TRI)

TTCN-3 Control Interface (TCI)

TTCN-3 Tester

Test Manager

ExecutableTest Suite

Underlying Protocol Stack

TTCN-3 Test Suite

TTCN-3Compiler

Completed ICS

From: In Pursuit of Interoperability (ITJ2582) - Scott Moseley; Steve Randall; Anthony Wiles.