Upload
thomasine-preston
View
228
Download
0
Embed Size (px)
Citation preview
Testing Techniques
Conformance Testing Methodology
and Framework
ISO IS-9646
TT 2004, ISO Conformance 2© Ed Brinksma/Jan Tretmans
Overview
Conformance Testing Methodology and Framework
ISO IS-9646
Literature: “An Overview of OSI Conformance Testing”
TT 2004, ISO Conformance 3© Ed Brinksma/Jan Tretmans
Overview: This Lecture
Protocol conformance testing: why and what Standardization Overview Standard IS-9646:
Conformance Testing Methodology and Framework
Conformance: static and dynamic Options, protocol implementation conformance statement Test generation: test purposes, abstract-, executable tests Test methods, test architectures, Test classification and -structuring Test notation Test implementation and -execution
TT 2004, ISO Conformance 4© Ed Brinksma/Jan Tretmans
Types of Testing
TT 2004, ISO Conformance 5© Ed Brinksma/Jan Tretmans
Protocol Conformance Testing
distributed systems
exchange of information
rules for communication : protocol
standardized protocol specifications
conforming implementations
testing for protocol conformance
TT 2004, ISO Conformance 6© Ed Brinksma/Jan Tretmans
Protocol Conformance Testing
International Standard ISO IS-9646 :
Conformance Testing Methodology and Framework
How to test protocol entity implementations
for conformance with respect to their specifications
Originally for OSI systems
Now more generally applied
Defines test language TTCN
Standard accepted by ITU-T, ETSI, ATM-Forum, ……
TT 2004, ISO Conformance 7© Ed Brinksma/Jan Tretmans
Protocol Conformance Testing
Starting point :a correct and valid (standardized) protocol specification
Original goal :standardization of test suites in TTCN consistency between specification and test suite ?
Conformance testing interoperability testing
TT 2004, ISO Conformance 8© Ed Brinksma/Jan Tretmans
The Conference Protocol
TT 2004, ISO Conformance 9© Ed Brinksma/Jan Tretmans
CPE
UDP Layer
CPECPE
The Conference Protocol
joinleavesend
receive
Conference Service
TT 2004, ISO Conformance 10© Ed Brinksma/Jan Tretmans
Conference ProtocolTest Architecture
CPE= IUT
UT-PCO = C-SAP
LT-PCO
UDP Layer
U-SAP LT-PCO
TesterTorX
B C
A
TT 2004, ISO Conformance 11© Ed Brinksma/Jan Tretmans
(N)-ASPs (N)-ASPs
(N)-service provider
(N)-ProtocolEntity
(N)-PDUs
(N-1)-ASPs
(N-1)-service provider
(N)-ProtocolEntity
protocol
(N-1)-ASPs
Protocol Conformance Testingvs. Interoperability Testing
TT 2004, ISO Conformance 12© Ed Brinksma/Jan Tretmans
Protocol Conformance Testingvs. Interoperability Testing
Protocol Conformance Testing: correct protocol (design and validation) conforming protocol entities (testing) interoperability
Interoperability testing test directly whether each pair of protocol entities can
work together (i.e. deliver (N)-service)
Internet: interoperability testing
with (standard) reference entities
TT 2004, ISO Conformance 13© Ed Brinksma/Jan Tretmans
Standard Conformance Testing
Testing by: manufacturers users organizations telecom operators independent test laboratories
Standard IS-9646 for certification of implementations accreditation of testing processes and test laboratories international acceptance of test results
TT 2004, ISO Conformance 14© Ed Brinksma/Jan Tretmans
ISO IS - 9646
Part 1 : Introduction and general concepts
Part 2 : Abstract test suite specification
Part 3 : TTCN
Tree and Tabular Combined Notation
Part 4 : Execution of tests
Part 5 : Requirements on test laboratories and
administrative conformance assessment
process
TT 2004, ISO Conformance 15© Ed Brinksma/Jan Tretmans
testgeneration
testexecution
testimplementation
standardprotocol
specification
standardconformance
test suite
verdict
implementationprocess
protocolimplementation
IUT
process ofconformance testing
executabletest suite
TT 2004, ISO Conformance 16© Ed Brinksma/Jan Tretmans
Conforming Implementation
Specifications in natural language
What is a conforming implementation ?
“ … exhibits conformance
if it complies with the conformance requirements
of the applicable specification standard …”
Satisfaction of conformance requirements
Conformance requirements :
“the implementation shall do ….. “
“the implementation shall not do …..”
Complication : options
TT 2004, ISO Conformance 17© Ed Brinksma/Jan Tretmans
Conforming Implementation
(P)ICS -
(Protocol) Implementation Conformance Statement :
list of implemented options
supplied by manufacturer
(P)ICS proforma
fill-in questionnaire with all possible options
attachment to protocol standard
to be completed by manufacturer to make PICS
Restrictions on combinations of implemented options
documented in standard in static conformance requirements
TT 2004, ISO Conformance 18© Ed Brinksma/Jan Tretmans
Example : Static Conformance
Requirement
Feature A is optional
Feature B is optional
But A may only be implemented
if B is also implemented
TT 2004, ISO Conformance 19© Ed Brinksma/Jan Tretmans
Conforming Implementation
Requirements on dynamic behaviour: dynamic conformance requirements
Messages : structure coding
Ordering : sequence of sent and receive messages relation between information content of different messages
Example: When in state Enable, upon receiving a Data-Request,
the implementation shall send a Data-PDUor it shall enter the abort phase.
TT 2004, ISO Conformance 20© Ed Brinksma/Jan Tretmans
Conforming Implementation
A conforming implementation is onewhich satisfies both static and dynamic conformance requirements, consistent withthe capabilities stated in the PICS
Conformance testing: checking static conformance by reviewing checking dynamic conformance requirements by testing
TT 2004, ISO Conformance 21© Ed Brinksma/Jan Tretmans
Test Generation
dynamic conformance requirement
test purpose(s)
generic test case
abstract test case
executable test case
test case execution
TT 2004, ISO Conformance 22© Ed Brinksma/Jan Tretmans
Test Generation
Test purpose :precise description (natural language)of what is to be tested in order to decide aboutsatisfaction of a particular conformance requirement
Example: When in state Enable, upon receiving a Data-Request,
the implementation shall send a Data-PDUor it shall enter the abort phase.
Check whether in state Enable, after receivinga Data-Request, a Data-PDU is sent.
(but implementation may also enter the abort phase!)
TT 2004, ISO Conformance 23© Ed Brinksma/Jan Tretmans
Test Generation
Generic test case:operationalization of test purpose;description of actions to be performedto reach test purpose
Example:
Check whether in state Enable, after receivinga Data-Request, a Data-PDU is sent.
Go to state Enable, send a Data-Request,and observe a Data-PDU, and if not check that theimplementation starts entering the abort phase
TT 2004, ISO Conformance 24© Ed Brinksma/Jan Tretmans
Test Generation
Abstract test case:take test method into account
Test method ( test architecture, test environment ) :abstract description of environmentin which testing will be performed,in terms of PCO - Points of Control and Observation
Example:Do Connect-Request,
check Connect-Response (now in state Enable),send Data-Requestobserve Data-PDUor observe Start-Abort
TT 2004, ISO Conformance 25© Ed Brinksma/Jan Tretmans
IUT
LT
UT
(N)-ASPs
(N)-PDUs
PCO
PCO(N-1)-ASPs
testcoordination
Local test method
Test Methods
TT 2004, ISO Conformance 26© Ed Brinksma/Jan Tretmans
IUT
LT
UT
(N)-ASPs
(N)-PDUs
PCO
PCO (N-1)-ASPs
testcoordination
service provider
test system SUT
Distributed test method
Test Methods
TT 2004, ISO Conformance 27© Ed Brinksma/Jan Tretmans
Test Notation
Abstract test cases are written in standardized notation :TTCN : Tree and Tabular Combined Notation
Specification language for Abstract Test Cases
Behaviour specified as tree of sequences of possible events
Data specified in tables or in ASN.1
ISO IS-9646 Part 3
With: input ? , output ! (from tester perspective!) ,timers / time out, ordering of events,default behaviour, alternative behaviours,tree attachment (=subroutines),parallel component testers,
verdicts: pass, fail, inconclusive
TT 2004, ISO Conformance 28© Ed Brinksma/Jan Tretmans
behaviour
Test Case Dynamic Behaviour
Test Case Name: Conn_EstabGroup: transport/connectionPurpose: Check connection establishment with remote initiative
constraints verdict
+preamble LT ! T-PDU-connect-request UT ? T-SP-connect-indication UT ! T-SP-connect-response LT ? T-PDU-connect-confirm OTHERWISE LT ? T-PDU-disconnect-request OTHERWISE
passfailinconclusivefail
Example : TTCN
TT 2004, ISO Conformance 29© Ed Brinksma/Jan Tretmans
Classification of Tests
Basic interconnection tests
Capability tests
Behaviour tests
Conformance resolution tests
TT 2004, ISO Conformance 30© Ed Brinksma/Jan Tretmans
Structuring of Test Suites
Test suite
Test caseTest case Test case
Test stepTest step Test step
Test eventTest event Test event
PCO ? X PCO ! A
Test groupTest group Test group
TT 2004, ISO Conformance 31© Ed Brinksma/Jan Tretmans
Test Implementation
Transform abstract test suite into executable test suite by compilation interpretation a test slave …..
Select test cases from abstract test suite based on PICS Incorporate (P)IXIT -
(Protocol) Implementation eXtra Information for Testing : address information timer information specific parameter values …..
PIXIT proforma supplied by test laboratory
TT 2004, ISO Conformance 32© Ed Brinksma/Jan Tretmans
Test Execution
Execute / interpret executable test suite
Collect and analyse the resultsbased on verdicts pass, fail, inconclusiveof each test case execution
Collect all results of static conformance review
dynamic testing
into Protocol Conformance Test Report
TT 2004, ISO Conformance 33© Ed Brinksma/Jan Tretmans
implementationprocess
staticconformance
review
testexecution
analysisof results
testselection
testimplementation
standardizedtest methods
testpurposes
generictest suite
testnotation
verdicttest report
certification
PIXITproforma
standard protocolspecification
PICSproforma
dynamic conformance req’s
static conformance req’s
protocolimplementation
IUT
PIXIT
PICS
(standardized)abstracttest suite
Basic interconn.capabilitybehaviour
executable
test suite