Upload
letruc
View
217
Download
1
Embed Size (px)
Citation preview
World Class Standards
Achieving Interoperability -the ETSI Approach
Achieving Interoperability - the ETSI Approach
FMCA Interoperability WeekJune 2008
Anthony WilesDirector
Centre for Testing and Interoperability© ETSI 2008. All rights reserved
June 2008
World Class Standards
Achieving Interoperability - the ETSI Approach 2
Download from: http://www.etsi.org/WebSite/AboutETSI/HowWeWork/Tes tingandinteroperability.aspx
World Class Standards
Why we are here – Interoperability!
� IOP is one of the ultimate aims of ICT standardisat ion
� IOP is the red thread running through the entire standards development process, it’s not an isolated issue
Achieving Interoperability - the ETSI Approach 3
issue� Not something to be somehow fixed at the end
� ETSI philosophy� Interoperability is built-in!
World Class Standards
Poor Interoperability is Expensive
� In the past, it meant� Bad publicity in trade magazines� Embarrassment for the manufacturer� Annoyance of the end customer
� Today, interoperability failures in the field means
Achieving Interoperability - the ETSI Approach 4
� Today, interoperability failures in the field means� Front page headlines in the Financial Times� Fall in manufacturers stock price� Loss of investor confidence� Unrecoverable damage to brand name� Irretrievable loss of customers
� We can no longer afford to get it wrong!
World Class Standards
Different ‘levels’ of Interoperability� Many definitions but in the context of ETSI/3GPP
interoperability is generally defined as� " the ability of two or more systems or components to
exchange data and use information"
Achieving Interoperability - the ETSI Approach 5
TechnicalTechnicalTechnicalTechnicalSyntacticSyntacticSyntacticSyntacticSemanticSemanticSemanticSemanticOrganisationalOrganisationalOrganisationalOrganisational
World Class Standards
Typical Symptoms of Non-interoperability
Where are you?Where are you?Where are you?Where are you?
What did you say?What did you say?What did you say?What did you say?
Achieving Interoperability - the ETSI Approach 6
Why did you do that?Why did you do that?Why did you do that?Why did you do that?
Protocol standards must ensure total internal consistency, robustness and efficiency. Testing ensures correct implementation.
Clear, efficient and unambiguous specification of data formats and encodings go a long way to eliminating non-interoperability. Testing ensures correct implementation.
Protocols should be designed (repeat, designed!) to be flexible, robust and predictable. Uncontrolled evolutionary development of a protocol should be avoided. Testing ensures correct implementation.
World Class Standards
Interoperability and Complex Systems
� ICT standards increasingly specified by ‘islands of
Achieving Interoperability - the ETSI Approach 7
� ICT standards increasingly specified by ‘islands of standards’� whole system not specified in detail
� Multiple sources of standards for one system� i.e. from several different SDOs� e.g. NGN is: TISPAN, 3GPP, IETF, ITU-T etc.
� Multiple specifications (100’s docs) for one protoc ol� e.g. IPv6 (many RFCs)
World Class Standards
Typical Causes of Standards Failing to Provide Interoperability
� Inadequately defined interfaces (reference points)
� Missing requirements
� Poor handling of options
Achieving Interoperability - the ETSI Approach 8
� Lack of clarity, requirements not well identified
� Lack of clear system overview
� Poor maintenance
� Using standards beyond their original purpose
� Varying quality of standards
World Class Standards
Building Interoperability into ETSI standards
� Manage for interoperability!
� Specify for interoperability!
� Validate for interoperability!
� Test for interoperability!
Achieving Interoperability - the ETSI Approach 9
� Test for interoperability!
� Maintain for interoperability!
MaintenanceSpecificationManagement Validation Testing
World Class Standards
Manage for Interoperability!
� Good project management and overview� Important in any standardisation project, but� Essential in the new world of multi-standards organ isation,
multi-standards projects
Achieving Interoperability - the ETSI Approach 10
� ETSI has a team of Technical Officers, Coordinators and Assistants to support ETSI Technical Bodies and pro vide project management
� Team of Editing Assistants dedicated to publishing
� ETSI has other unique resources dedicated to improv ing the quality of standards and interoperability …
World Class Standards
ETSI Initiatives and Support for Interoperability
� Board Champion for Interoperability� Co-ordination for interoperability issues� Chair of OCG-IOP ad hoc group
� Technical Committee MTS� Methods for Testing and Specification
Achieving Interoperability - the ETSI Approach 11
� Methods for Testing and Specification� Standardised frameworks and methodologies� Making Better Standards: http://portal.etsi.org/mbs
� Centre for Testing and Interoperability (CTI)� Support on protocols and testing to ETSI TBs� Interoperabilty events ( Plugtests TM)
• Organises and runs interop test events• Open to members and non-members of ETSI• For technologies inside and outside of ETSI
� Commercial services to fora and other external bodi es
World Class Standards
Centre For Testing and Interoperability
CTI
Protocol Design
Achieving Interoperability - the ETSI Approach 12
CTIStandards Engineering based on methodology and best working
practices
Training
Validation Activities Interoperability Events
Test Specifications
World Class Standards
Centre For Testing and Interoperability
CTI
Training• Best Practices• Methodology
Test Specification• Test Planning & Frameworks• Extraction of Requirements
Protocol Design
Protocol Design and Specification• Application of best practice• 3-Stage approach• Techniques such as:
UML, MSC, SDL, ASN.1, XML etc.
Achieving Interoperability - the ETSI Approach 13
CTIStandards Engineering based on methodology and best working
practices
Training
Validation Activities Interoperability Events
• Methodology• Languages• Implementation issues
Validation Activities• Validation of Standards• Validation of products• Peer-review• Simulation by modelling• Interoperability events
Test Specifications
Interoperability Events
• Extraction of Requirements• Test Purposes• Test Descriptions• TTCN-3 code• Tool support
Interoperability Events• Organizational &Technical• Logistics, website, press• Registration, NDA, promo• Test platform and IT• Monitoring, test results• Feedback to TC
World Class Standards
Achieving Interoperability - the ETSI Approach 14
INTEROPOLIS is the ETSI product enabling service. It helps develop specifications and interoperable products with confidence in a timely and cost-effective manner
World Class Standards
Specify for interoperability!
� Best working practices (guides from TC MTS)
Achieving Interoperability - the ETSI Approach 15
� ITU-T I.130 3-stage model for protocol specificatio n� Used extensively in 3GPP� Stage 1: High-level requirements on the system� Stage 2: Functional architecture and Information Flows
• Standardise interoperable interfaces, not internal behaviour
� Stage 3: Detailed protocol specification• Use most appropriate techniques
• text, UML, SDL, ASN.1, XML etc.
• Clearly identify requirements, options and dependen cies
World Class Standards
Validate for interoperability!
� EG 202 107 V1.1.1, validation through
Achieving Interoperability - the ETSI Approach 16
� EG 202 107 V1.1.1, validation through
� Technical reviews
� Modelling and simulation
� Interoperability events
� Test specification development
� Testing
� Ensure validation results are fed-back into the sta ndard
World Class Standards
Validation through Plugtests TM events
Achieving Interoperability - the ETSI Approach 17
World Class Standards
Validation/Verification Activities
Products mature from prototypes to commercial produ cts
Certification
Achieving Interoperability - the ETSI Approach 18
Conformance Testing Interoperability Testing
Development of base standards
Interoperability events
Peer Review Modelling and Simulation Prototyping
World Class Standards
Test for interoperability!
Test Purposes
Prose or TPLan
Reqs Catalogue
ICS
Test Scripts
TTCN-3
Validation of Tests
Testbed
Testing
Test Descriptions
Prose, TPLan Tables
e.g. C++
Implementation of tests Aceptance
Achieving Interoperability - the ETSI Approach 19
� Plan for testing!� Usually Conformance Testing and Interoperability Te sting
� Use existing methodologies� ISO/IEC 9646 Conformance Testing Methodology� ETSI Interoperability Testing Methodology (EG 202 2 37)� TPLan (ES 202 553) and TTCN-3 (ES 201 873)
� Validate test specifications� Use of external test platforms e.g., UMTS, WiMAX, I MS� In simpler cases some ETSI in-house facilities e.g. , SIP, IPv6
World Class Standards
Different Kinds of ETSI Test Specifications
ConformanceRobustness
Achieving Interoperability - the ETSI Approach 20
PerformanceInteroperability
Interoperability + Conformance Verification RF/EMC
s1
World Class Standards
Interdependencies Between Specification and Testing
Interoperability Testing
Products mature from prototypes to commercial produ cts
Certification
Industry Iterative
feedback
Interoperability events & other validation activities
Achieving Interoperability - the ETSI Approach 21
(Unit) Conformance Testing
Development of base standards
Industry
time
Conformance Test Specifications
Interoperability Test SpecificationsIterative feedback
Standards
Body
Testing, validation, simulation etc. provides vital feedback! e.g., holes, errors, inconsistencies etc. in base standards
World Class Standards
Test Specification Development
Standard
Successive Levels of Abstraction
Test Purposes
Test Descriptions
TTCN-3 Test Suite
Achieving Interoperability - the ETSI Approach 22
Purposes Descriptions Test Suite
Executable Tests
CompilationCompilationCompilationCompilation
Requirements Catalogue
and/or ICS/IXITTest Case Parameterisation Test Case Parameterisation Test Case Parameterisation Test Case Parameterisation
and Selectionand Selectionand Selectionand Selection
World Class Standards
Conformance and Interoperability Testing
1 2 3
4 5 6
7 8 9
* 8 #
latigid
Achieving Interoperability - the ETSI Approach 23
1 2 3
4 5 6
7 8 9
* 8 #
1 2 3
4 5 6
7 8 9
* 8 #
World Class Standards
Combining Interoperability Testing with Conformance Verification
Achieving Interoperability - the ETSI Approach 24
World Class Standards
Conformance Testing and Interoperability Testing are Complementary
� ETSI experience� As you move up a system stack the emphasis should c hange
from conformance to IOT� Moving from component testing, to more complex
interoperability issues
� Lower layer protocols, infrastructure� Emphasis on conformance
Achieving Interoperability - the ETSI Approach 25
� Emphasis on conformance� Middleware, enablers
� Combination of Conformance + IOT� Services, applications, systems
� Emphasis on IOT
� Conformance testing as a pre-requisite to IOT� Ensure interoperability through standardised interf aces
� Interoperability testing with conformance verificat ion� E.g. end-to-end conformance tests with intermediate reference
point verification
World Class Standards
The CTI Testing STFs Write Test Specifications
� CTI experts plus …� Specialist Task Forces (STFs)
� 15 – 20 STFs per year� Typical total yearly resource = 20-25 man-years � Experts seconded from the ETSI membership
Achieving Interoperability - the ETSI Approach 26
� Experts seconded from the ETSI membership• STF Budget for Testing• Assignment of this budget is done by ETSI members ( ETSI
Board)� Various complexity
• e.g., 2 man-months maintenance of VoIP tests• e.g., UMTS testing 58 man-months per year over 4-5 years
� Commercial contracts� Negotiated on an individual basis
World Class Standards
Typical Test Specification Projects� Cellular: GSM, 3G UMTS (including IMS/SIP),
� Wireless: HiperMAN, HiperACCESS, WiMax
� VoIP: H.323, IETF SIP, SIGTRAN
� Service Creation: OSA/Parlay (API, IDL, Java)
� IPv6: Core, Security, Mobility, v4-v6
� Radio communications: DECT , TETRA, DMR
Achieving Interoperability - the ETSI Approach 27
� Radio communications: DECT , TETRA, DMR
� Access terminals: FSK, SMS
� Broadband: ISDN, DSL
� Smartcards: Readers, cards, security modules
� Intelligent Transport Systems (ITS): DSRC
� NGN: IMS Interoperability (interworking)
� Future: More Security, more NGN, GRID ...
World Class Standards
Typical
� SIGTRAN
� Bluetooth
� IPv6 (Core, IPSEC, etc)
� Triple Play over xDSL
� SIM/Handset
� SIPit
� J2ME
� HDMI
� Air Traffic Control (EUROCAE)
� Electronic Signature (XadES)
Achieving Interoperability - the ETSI Approach 28
� SIM/Handset
� WLAN IRAP
� RFID
� STQ (Speech Quality)
� WiMAX
� ENUM
� OSA/Parlay (ParlayX)
� Electronic Signature (XadES)
� Lawful Interception
� Optical Fibre (GPON)
� Power Line (PLT)
� GRID
� H.323
� FMCA
World Class Standards
More on the development of More on the development of test specifications…
Achieving Interoperability - the ETSI Approach 29
World Class Standards
The Requirements Catalogue
� Database of all extracted requirements� REQ name� Reference to base standard� Context information
Achieving Interoperability - the ETSI Approach 30
� Context information� Status: Mandatory (M), Optional (O) etc.� Dependencies between optional requirements
� Links to Test Purposes� Links to Test Cases
� Powerful search and grouping capabilities
Requirements Catalogue
and/or ICS/IXIT
World Class Standards
The Requirements Catalogue
� Database of all extracted requirements� REQ name� Reference to base standard� Context information
Achieving Interoperability - the ETSI Approach 31
� Context information� Status: Mandatory (M), Optional (O) etc.� Dependencies between optional
requirements� Links to Test Purposes� Links to Test Cases
� Powerful search and grouping capabilities
Requirements Catalogue
and/or ICS/IXIT
World Class Standards
Test Purposes� Test Purposes (TP) are precise descriptions of the
purpose of the test for a particular requirement� Specify WHAT is being tested
� Do not define HOW to test� They are not test code
� Grouped into a logical structure
Achieving Interoperability - the ETSI Approach 32
� Grouped into a logical structure� Test Suite Structure (TSS & TP)� One Requirement may spawn several TPs
� Specified in � Natural language, or� ETSI’s Test Purpose Language (TPLan)� http://www.tplan.info/home.htm
Test Purposes
World Class Standards
Example TPLAN Test Purpose
TP id : TP_COR_0047_01Summary : ‘hop limit of one'RQ Ref : RQ_COR_0047Config : CF_02_CTC Ref : TC_COR_0047_01ensure that {
Achieving Interoperability - the ETSI Approach 33
ensure that {--Stimulus
when { IUT receives ‘Ipv6 packet' from ‘Host'containing ‘IPv6 Header'indicating ‘Hop limit' set to ‘1‘ }
--Expected responsethen { IUT sends ‘ICMPv6 Time Exceeded' to ‘Host‘
containing ‘ICMP code' set to ‘ZERO‘
}
World Class Standards
Test Descriptions
� More detailed than Test Purposes� But not directly executable
� Act as a design specification for test cases� Or for manual execution
Achieving Interoperability - the ETSI Approach 34
� E.g. Interoperability testing
� Contains more information� Configuration details� Postamble and Preamble sequences� Detailed message sequences for test body� Parameter values� Etc.
Test Descriptions
World Class Standards
Example Test Description
Identifier:
Test Purpose: TP_COR_1100_01 Reference: RQ_COR_1100 Configuration:
TD_COR_1100_01
Summary EUT reassembles a fragmented packet of an original length less than 1500 octets
with { 'the MTU on Link1 set to 1400 octets' }
ensure that {
when { QE is requested to 'send data requiring a packet
length greater than 1500 octets' }
then { EUT indicates 'receipt of the same data without
modification' }
}
CF_011_I
Test Description
Achieving Interoperability - the ETSI Approach 35
Step Step
}
Pre-Test
Conditions:MTU set to 1400 octets on link1
Verdict
1
2
3
Cause QE to send an Echo Request to EUT with a
packet size of 1450 octets and with each octet set
to the hexadecimal value "F0"
Check: Does protocol monitor show that the Echo
Request was sent from QE to EUT?Yes No
Check: Does QE receive an Echo Reply from EUT
with the packet length the same as the
Echo Request and with each octet
containing the hexadecimal value "F0"?
Yes No
Pass Fail
Observations
World Class Standards
TTCN-3 Test Cases
� Detailed TTCN-3 test script that implements test pu rpose� Can be compiled and executed
� Specifies HOW to test not WHAT to test� Preamble� Test body (i.e., implementation of the Test Purpose )
Achieving Interoperability - the ETSI Approach 36
� Test body (i.e., implementation of the Test Purpose )� Postamble
� Assigns test verdicts� Handles unexpected behaviour as well as the behavio ur
in the test purpose� Can be distributed over parallel test components� Can be entirely automated� Configurable at run-time
TTCN-3 Test Suite
World Class Standards
Example TTCN-3 Test Casetestcase TC_COR_0047_01() runs on Ipv6Node system EtherNetAdapter {
f_cf02Up(); // Configure test system for HS->RT// No preamble required in this case
f_TP_HopsSetToOne(); // Perform test// No postamble required in this case
f_cf02Down(); // Return test system to initial state}function f_TP_HopsSetToOne() runs on Ipv6Node {
var Ipv6Packet v_ipPkt;
Achieving Interoperability - the ETSI Approach 37
var Ipv6Packet v_ipPkt;var FncRetCode v_ret := f_echoTimeExceeded( 1, v_ipPkt );if ( v_ret == e_success and v_ipPkt.icmpCode == 0 ) { setverdict ( pass );}else { setverdict ( fail ); }
}function f_echoTimeExceeded( in UInt8 p_hops, out Ipv6Packet p_ípPkt ) runs on Ipv6Node return FncRetCode {
var Ipv6Packet v_ipPacket; var FncRetCode v_ret;ipPort. send ( m_echoReqWithHops(p_hops) );alt {
[] ipPort. receive ( mw_anyTimeExceeded ) -> value p_ipPkt { return e_success }
[] ipPort. receive { return e_error } }}
World Class Standards
TTCN-3 is an ETSI Standardhttp://www.ttcn-3.org
� ES 201 873-1 (Z.140)� TTCN-3 Core Language
� ES 201 873-2 (Z.141)� TTCN-3 Tabular Presentation Format (TFT)
Achieving Interoperability - the ETSI Approach 38
� TR 101 873-3 (will eventually be ES 201 873-3) (Z.142)� TTCN-3 Graphical Presentation Format (GFT)
� ES 201 873-4 (Z.143)� TTCN-3 Operational Semantics
� ES 201 873-5� TTCN-3 Runtime Interface (TRI)
� ES 201 873-6� TTCN-3 Control Interfaces (TCI)
� ES 201 873-7 and upwards� Using ASN.1, XML, IDL, C/C++, with TTCN-3
World Class Standards
Why TTCN-3?� Specifically designed for testing
� Concentrates on the test not the test system� Independent of the execution environment
� Wide range of applications� Mobile communications to Internet to software to …
� Standardised
Achieving Interoperability - the ETSI Approach 39
� Standardised� Commonly understood syntax and operational semantics� Constantly maintained and developed� Off-the-shelf tools and TTCN-based test systems
� Unifies different (all) testing activities� Education and training costs can be rationalized� Maintenance of test suites (and products) is easier� Facilitates a common methodology and style� Both on a corporate level and within standardization
http://eu.wiley.com/WileyCDA/WileyTitle/productCd-0 470012242.htm
World Class Standards
TTCN-3 Test System
TCI TCI TCI TCI
PICS etcPICS etcPICS etcPICS etc
(reqs. (reqs. (reqs. (reqs. catalogue)catalogue)catalogue)catalogue)
ParameterisationParameterisationParameterisationParameterisation
SelectionSelectionSelectionSelectionControl / LoggingControl / LoggingControl / LoggingControl / Logging
Achieving Interoperability - the ETSI Approach 40
ENCODERENCODERENCODERENCODER
DECODERDECODERDECODERDECODER
(N(N(N(N----protocol protocol protocol protocol specific)specific)specific)specific)
Adaptation LayersAdaptation LayersAdaptation LayersAdaptation LayersTRITRITRITRITTCNTTCNTTCNTTCN----3 Runtime Interface3 Runtime Interface3 Runtime Interface3 Runtime Interface
TCI TCI TCI TCI TTCNTTCNTTCNTTCN----3 Control Interface3 Control Interface3 Control Interface3 Control InterfaceTest Suite Test Suite Test Suite Test Suite
in TTCNin TTCNin TTCNin TTCN----3333
(source)(source)(source)(source)
TTCNTTCNTTCNTTCN----3 3 3 3 Test SuiteTest SuiteTest SuiteTest Suite
(object)(object)(object)(object)
Underlying Protocol StackUnderlying Protocol StackUnderlying Protocol StackUnderlying Protocol Stack
(N(N(N(N----1)1)1)1) Connection to the SUT
CompilationCompilationCompilationCompilation
CompilationCompilationCompilationCompilation
World Class Standards
Maintain for Interoperability!
� Maintenance
Achieving Interoperability - the ETSI Approach 42
� Maintenance� The black sheep – but its what ETSI is good at!
� Good standards can be broken by poor maintenance� Or no maintenance!� Corrections to be made with care
� Requires well-defined Change Request (CR) process� Feedback needs to be sought and captured � Implementation of CRs should follow same process as
original development
World Class Standards
Conclusions
� Standards can be designed for interoperability
� Standards should be engineered not hacked
� Plan for validation and testing (early)
Achieving Interoperability - the ETSI Approach 43
� Plan for validation and testing (early)
� Do the right kind of testing and test in parallel
� Interoperability events: Plugtests TM
� CTI can help you achieve the above!
World Class Standards
Thank you!
Achieving Interoperability - the ETSI Approach
Anthony WilesDirector
Centre for Testing and [email protected]
Whitepaper can be downloaded from
http://www.etsi.org/WebSite/NewsAndEvents/whitepape rs.aspx