View
219
Download
1
Category
Tags:
Preview:
Citation preview
LTE CONFORMANCE TESTING: EXPERIENCES IN A LONG TERM PROJECTTTCN-3 User Conference 2011 – Bled, Slovenia
© ETSI 2011. All rights reserved
Presented by MCC TF160: Wolfgang Seka
Authors: Wolfgang Seka, Hellen Griffiths, Shicheng Hu
wolfgang.seka@etsi.org , hellen.griffiths@etsi.org , shicheng.hu@etsi.org
LTE Conformance Testing: Experiences in a long term project
The Project
Technical Issues
Common Aspects of Long Term Projects
Conclusions
2 © ETSI 2011. All rights reserved
THE PROJECT
3 © ETSI 2011. All rights reserved
MCC TF160 - General
Task Force – Mobile Competence Centre: Project Group at ETSI• Pool of TTCN expertise used by 3GPP
3GPP: 3rd Generation Partnership Project (http://www.3gpp.org)→ Telecommunication Standardisation Bodies
→ TSG RAN: Radio Access Network→ WG RAN5: Mobile terminal conformance testing
Conformance Tests• Specification (Prose): RAN5• Implementation (TTCN): MCC TF160• Verification and Validation: Test Industry
MCC TF160: Signalling Conformance Tests for 3GPP (RAN5: Testing)• Task: Develop Conformance Test Suites for UE world-wide certification• since 2000 Conformance Tests for UMTS Signalling (TTCN-2)
since 2006 Conformance Tests for IMS (TTCN-3)2007..2008 Pre-evaluation of TTCN-3 for LTE Signalling2008..now 3GPP LTE/SAE UE Conformance Test
© ETSI 2011. All rights reserved4
MCC TF160 – LTE/SAE Project
Size: 17 experts all over the worldDuration: since 2008 and at least until 2015 (LTE-advanced will follow)Deliveries2011: 8 deliveries planned Test cases: 264 approved
391 fully implemented468 in total80..100 planned for IMS
Code size:• nearly 200 Modules• nearly 190 000 Lines of code (TTCN-3)
Type Definitions• 25 TTCN-3 modules, 3 ASN.1 modules, 6 XML modules
• TTCN-3: > 17 500 lines of code • ASN.1: > 38 000 lines of code• XML: < 1 000 lines of code
Tools:• 6 different compilers (all available at ETSI)• quality checks (naming conventions, template restrictions etc.)• code generation (top-level test case definitions, parameters, etc.)• code analysis (structure, approved objects)
TTCN-3 code is officially published and widely used© ETSI 2011. All rights reserved5
Component Structure - 1
MTC• Start of PTCs, monitoring of “done” and “killed”• RAT independent interfaces (e.g. AT-/MMI-commands to control the UE)• In general no pass/fail verdicts
PTCs for each RAT (radio access technology)• LTE, UMTS, GSM/GPRS, CDMA2000• May be connected to any other RAT PTC• Only RAT specific interfaces• Assignment of pass/fail verdicts
PTCs for other protocols • IP data (e.g. DHCP, ICMPv6), IMS and “protocol layer” underneath LTE PTC (NAS Emulator)• To keep system interface simple and deal with “parallel behaviour”• In general no test characteristic ( no pass/fail verdicts)
Ports and interfaces• Connections: only one-to-one
( no “send to”, “receive from”; no use of address data type)• No duplicated interfaces: e.g. only one interface for AT-/MMI-commands hosted by the MTC• Message based communication only
© ETSI 2011. All rights reserved6
Component Structure - 2
7 © ETSI 2011. All rights reserved7
7System Interface
E_SYSINDE_DRB E_SYS E_SRB
EUTRA PTC(LTE)
NAS Emulator
CTRL TC_SRB
SYS_SRB
SYSINDDRB SYS NASCTRL SRB
UT
UT
RA
NG
ER
AN
MTCE
_Ut
Ut
Ut
G_U
tU
_Ut UTRAN
PTCC
2K_U
t
GERANPTC
CDMA2000PTC
UTRANPTC
GERANPTC
CDMA2000PTCC
2K
Coordination
Upper Tester (AT , MMI)
Configuration
Signalling
User Data
SUT
Test System
© ETSI 2011. All rights reserved8
Host PCSystem Simulator HW
complex configurationdelay of messagesno matter what test purpose is
UE
RF
AT/MMI
Test Executable
system
specificSystem/PlatformAdaptor
Codec
e.g. requirements regarding real-time behaviour for System Simulator and TTCN-3 code
Test Control,Logging
TECHNICAL ISSUES
9 © ETSI 2011. All rights reserved
Real Time Issues - 1
Requirements• Time resolution at the air interface: 1ms• Due to significant delay between air interface and test case executable:
Timing needs to be monitored/controlled at the air interface (not at the TRI)• All test equipment shall have similar behaviour at any time• Test results shall not depend on
• how fast or slow a UE or system simulator is• specific combinations of UEs and system simulators
System Restrictions• Delay by codec and matching: up to some milliseconds;
typically < 5ms• Host system: typically Windows or Linux
no real-time OS• Architecture: Host-PC + System Simulator
Estimation• delay between test executable and air interface:
1ms < delay < 80ms© ETSI 2011. All rights reserved10
Real Time Issues - 2
Impact on System InterfaceLTE timing information • is part of all LTE primitives• is provided by the System Simulator for all received messages• is used to schedule sending of messages in the system simulator• can be retrieved from the system simulator
Impact on writing TTCN-3 • Scheduled and non-scheduled sending shall not be mixed up• When there is no reaction (e.g. from the UE) it is hard to know in the TTCN code when the
message has been sent out• Scheduling vs. TTCN timers
TTCN timers are not as accurate as scheduling Rules are required when to use TTCN timers or scheduled sending
Restrictions and Issues• No common system time: uncorrelated timing between different RATs• Legacy primitive definitions for UMTS and GSM/GPRS: less support of timing information• Short duration only: For LTE, wrap-around after 10s• No “bell” mechanism (yet):
to schedule a “wake-up call” in the system simulator to send back a trigger after a given time© ETSI 2011. All rights reserved11
Race Conditions
Race conditions occur when• messages appear at the same time at different ports • messages appear in any order at one port
Impact on TTCN implementation and Design considerations• TTCN implementation
• interleave (in general not more than two messages)• check operation• evaluation of timing information
• Clever port definitions (few ports rather than many)• Component structure
(e.g. signalling and data at different PTCs)
© ETSI 2011. All rights reserved12
COMMON ASPECTS OF LONG TERM PROJECTS
13 © ETSI 2011. All rights reserved
Extendibility
Extensions of TTCN-3 code• Modification of common objects shall not cause changes in the whole test suite
reduced maintenance effort• Impact on writing TTCN-3
• foresighted TTCN implementation• introduction of additional optional parameters
• Example:
© ETSI 2011. All rights reserved14
template MyType c_MyType :={ field1 := 1, field2 := 2};
template MyType c_MyType(integer p_Int := 1) :={ field1 := p_Int, field2 := 2};
Note: Optional Parameters vs. Modified Templates• Optional parameters need to be added to modified templates too • modification of templates and addition of optional parameters may serve the
same purpose possible conflicts, potential contradiction of both approaches
Backward Compatibility - 1
Backward Compatibility at the System Interface• Modifications are caused by
• Release upgrade for signalling messages may have impact on configuration of the system simulator
• New requirements (due to new test cases)• Corrections
• Backward Compatibility Aspects• Old TTCN-3 code on new system simulator implementation• New TTCN-3 code on old system simulator implementation
• Impact • system adaptor and codec • interface definition
• top-level records or unions (even when not needed in the first place)• definitions of place holders
© ETSI 2011. All rights reserved15
Backward Compatibility - 2
Examples: Backward Compatibility at System Interface
© ETSI 2011. All rights reserved16
type union MyUnion_Type := { R8_Type R8 };
type union MyUnion_Type := { R8_Type R8, R9_Type R9};
New branch: branches are mutual exclusive; requires new templates
type record MyRecord_Type := { R8_Type R8Only };
type record MyRecord_Type := { R8_Type R8Only, R9Extension_Type R9Ext optional};
Extension: templates may be enhanced by optional parameter
Tools
Large, long-term project code reviews and manual checks are not sufficient anymore
Requirements• Quality checks• Approved objects: objects used by approved test cases shall not be changed
without change request• Analysis of module dependencies• Recursive selection
Enhanced tool support• Quality checks by T3Q (on behalf of ETSI)• Common tool to get dependencies based on T3D (on behalf of ETSI)• Project-specific front-ends (approved objects, recursive selection)• Further project specific tools:
replace tabs, find non-ASCII characters in comments
© ETSI 2011. All rights reserved17
CONCLUSIONS
18 © ETSI 2011. All rights reserved
Conclusions (TTCN-3 language)
Important TTCN-3 language features
• optional parameters backward compatibility, maintenance
• template restrictions in combination with naming conventions and appropriate tools no problems with receive templates used in send direction anymore
• encvalue/decvalue maintenance (no external functions needed)
• check operation, interleave race conditions
• testcase.stop well-defined termination of a test case in case of unrecoverable error situations
• visibility: "private", "friend“ not used yet (may improve quality regarding module dependencies)
© ETSI 2011. All rights reserved19
Common Conclusions
Feedback of long-terms projects to the core language is vital
Tool Compatibility is needed
Shortened release cycle of TTCN-3 standards
TTCN-3 development is SW Development SW Engineering System Engineering
Long-term characteristic of a project needs to be considered from the beginning Conceptual errors will be expensive
Tool support is vital Common and project-specific tools
© ETSI 2011. All rights reserved20
ADDITIONAL SLIDES
21 © ETSI 2011. All rights reserved
Release Upgrades - 1
Release Upgrade for Protocol and Signalling(„Baseline Moving“)• Typically every year• In general changes are backward compatible• Critical and non-critical extensions (TS 36.331 Annex A)
• Critical extensions• redefinition of message content or parts of it
(branching, e.g. r8 and r9 branch)• sender needs to know which release the receiver supports
• Non-critical extensions• Provision of additional information• e.g. new fields at the end of a record, new enumerated value• when not supported receiver shall skip the information
• Messages in general contain hooks for critical and non-critical extensions
© ETSI 2011. All rights reserved22
Release Upgrades - 2
Impact on writing TTCN-3• Non-critical extensions
• In general no code duplication is necessary• Enhancement of existing templates by new optional parameters
(initialised with “*” for receiving and “omit” for sending)• Critical extensions
• New templates are needed• Challenge: minimised code duplication• Example:
template of a plain RRC PDU for UMTS up to several hundred lines of code clever template structure reduces duplication of code
• parameterised templates• modified templates
© ETSI 2011. All rights reserved23
Recommended