LTE CONFORMANCE TESTING: EXPERIENCES IN A LONG TERM PROJECT TTCN-3 User Conference 2011 – Bled, Slovenia © ETSI 2011. All rights reserved Presented by

LTE CONFORMANCE TESTING: EXPERIENCES IN A LONG TERM PROJECTTTCN-3 User Conference 2011 – Bled, Slovenia

© ETSI 2011. All rights reserved

Presented by MCC TF160: Wolfgang Seka

Authors: Wolfgang Seka, Hellen Griffiths, Shicheng Hu

[email protected] , [email protected] , [email protected]

mailto:[email protected]

mailto:[email protected]

LTE Conformance Testing: Experiences in a long term project

The Project

Technical Issues

Common Aspects of Long Term Projects

Conclusions

2 © ETSI 2011. All rights reserved

THE PROJECT


MCC TF160 - General

Task Force – Mobile Competence Centre: Project Group at ETSI• Pool of TTCN expertise used by 3GPP

3GPP: 3rd Generation Partnership Project (http://www.3gpp.org)→ Telecommunication Standardisation Bodies

→ TSG RAN: Radio Access Network→ WG RAN5: Mobile terminal conformance testing

Conformance Tests• Specification (Prose): RAN5• Implementation (TTCN): MCC TF160• Verification and Validation: Test Industry

MCC TF160: Signalling Conformance Tests for 3GPP (RAN5: Testing)• Task: Develop Conformance Test Suites for UE world-wide certification• since 2000 Conformance Tests for UMTS Signalling (TTCN-2)

since 2006 Conformance Tests for IMS (TTCN-3)2007..2008 Pre-evaluation of TTCN-3 for LTE Signalling2008..now 3GPP LTE/SAE UE Conformance Test

© ETSI 2011. All rights reserved4

MCC TF160 – LTE/SAE Project

Size: 17 experts all over the worldDuration: since 2008 and at least until 2015 (LTE-advanced will follow)Deliveries2011: 8 deliveries planned Test cases: 264 approved

391 fully implemented468 in total80..100 planned for IMS

Code size:• nearly 200 Modules• nearly 190 000 Lines of code (TTCN-3)

Type Definitions• 25 TTCN-3 modules, 3 ASN.1 modules, 6 XML modules

• TTCN-3: > 17 500 lines of code • ASN.1: > 38 000 lines of code• XML: < 1 000 lines of code

Tools:• 6 different compilers (all available at ETSI)• quality checks (naming conventions, template restrictions etc.)• code generation (top-level test case definitions, parameters, etc.)• code analysis (structure, approved objects)

TTCN-3 code is officially published and widely used© ETSI 2011. All rights reserved5

Component Structure - 1

MTC• Start of PTCs, monitoring of “done” and “killed”• RAT independent interfaces (e.g. AT-/MMI-commands to control the UE)• In general no pass/fail verdicts

PTCs for each RAT (radio access technology)• LTE, UMTS, GSM/GPRS, CDMA2000• May be connected to any other RAT PTC• Only RAT specific interfaces• Assignment of pass/fail verdicts

PTCs for other protocols • IP data (e.g. DHCP, ICMPv6), IMS and “protocol layer” underneath LTE PTC (NAS Emulator)• To keep system interface simple and deal with “parallel behaviour”• In general no test characteristic ( no pass/fail verdicts)

Ports and interfaces• Connections: only one-to-one

( no “send to”, “receive from”; no use of address data type)• No duplicated interfaces: e.g. only one interface for AT-/MMI-commands hosted by the MTC• Message based communication only


Component Structure - 2

7 © ETSI 2011. All rights reserved7

7System Interface

E_SYSINDE_DRB E_SYS E_SRB

EUTRA PTC(LTE)

NAS Emulator

CTRL TC_SRB

SYS_SRB

SYSINDDRB SYS NASCTRL SRB

UT

UT

RA

NG

ER

AN

MTCE

_Ut

Ut

Ut

G_U

tU

_Ut UTRAN

PTCC

2K_U

t

GERANPTC

CDMA2000PTC

UTRANPTC

GERANPTC

CDMA2000PTCC

2K

Coordination

Upper Tester (AT , MMI)

Configuration

Signalling

User Data

SUT

Test System


Host PCSystem Simulator HW

complex configurationdelay of messagesno matter what test purpose is

UE

RF

AT/MMI

Test Executable

system

specificSystem/PlatformAdaptor

Codec

e.g. requirements regarding real-time behaviour for System Simulator and TTCN-3 code

Test Control,Logging

TECHNICAL ISSUES


Real Time Issues - 1

Requirements• Time resolution at the air interface: 1ms• Due to significant delay between air interface and test case executable:

Timing needs to be monitored/controlled at the air interface (not at the TRI)• All test equipment shall have similar behaviour at any time• Test results shall not depend on

• how fast or slow a UE or system simulator is• specific combinations of UEs and system simulators

System Restrictions• Delay by codec and matching: up to some milliseconds;

typically < 5ms• Host system: typically Windows or Linux

no real-time OS• Architecture: Host-PC + System Simulator

Estimation• delay between test executable and air interface:

1ms < delay < 80ms© ETSI 2011. All rights reserved10

Real Time Issues - 2

Impact on System InterfaceLTE timing information • is part of all LTE primitives• is provided by the System Simulator for all received messages• is used to schedule sending of messages in the system simulator• can be retrieved from the system simulator

Impact on writing TTCN-3 • Scheduled and non-scheduled sending shall not be mixed up• When there is no reaction (e.g. from the UE) it is hard to know in the TTCN code when the

message has been sent out• Scheduling vs. TTCN timers

TTCN timers are not as accurate as scheduling Rules are required when to use TTCN timers or scheduled sending

Restrictions and Issues• No common system time: uncorrelated timing between different RATs• Legacy primitive definitions for UMTS and GSM/GPRS: less support of timing information• Short duration only: For LTE, wrap-around after 10s• No “bell” mechanism (yet):

to schedule a “wake-up call” in the system simulator to send back a trigger after a given time© ETSI 2011. All rights reserved11

Race Conditions

Race conditions occur when• messages appear at the same time at different ports • messages appear in any order at one port

Impact on TTCN implementation and Design considerations• TTCN implementation

• interleave (in general not more than two messages)• check operation• evaluation of timing information

• Clever port definitions (few ports rather than many)• Component structure

(e.g. signalling and data at different PTCs)


COMMON ASPECTS OF LONG TERM PROJECTS


Extendibility

Extensions of TTCN-3 code• Modification of common objects shall not cause changes in the whole test suite

reduced maintenance effort• Impact on writing TTCN-3

• foresighted TTCN implementation• introduction of additional optional parameters

• Example:


template MyType c_MyType :={ field1 := 1, field2 := 2};

template MyType c_MyType(integer p_Int := 1) :={ field1 := p_Int, field2 := 2};

Note: Optional Parameters vs. Modified Templates• Optional parameters need to be added to modified templates too • modification of templates and addition of optional parameters may serve the

same purpose possible conflicts, potential contradiction of both approaches

Backward Compatibility - 1

Backward Compatibility at the System Interface• Modifications are caused by

• Release upgrade for signalling messages may have impact on configuration of the system simulator

• New requirements (due to new test cases)• Corrections

• Backward Compatibility Aspects• Old TTCN-3 code on new system simulator implementation• New TTCN-3 code on old system simulator implementation

• Impact • system adaptor and codec • interface definition

• top-level records or unions (even when not needed in the first place)• definitions of place holders


Backward Compatibility - 2

Examples: Backward Compatibility at System Interface


type union MyUnion_Type := { R8_Type R8 };

type union MyUnion_Type := { R8_Type R8, R9_Type R9};

New branch: branches are mutual exclusive; requires new templates

type record MyRecord_Type := { R8_Type R8Only };

type record MyRecord_Type := { R8_Type R8Only, R9Extension_Type R9Ext optional};

Extension: templates may be enhanced by optional parameter

Tools

Large, long-term project code reviews and manual checks are not sufficient anymore

Requirements• Quality checks• Approved objects: objects used by approved test cases shall not be changed

without change request• Analysis of module dependencies• Recursive selection

Enhanced tool support• Quality checks by T3Q (on behalf of ETSI)• Common tool to get dependencies based on T3D (on behalf of ETSI)• Project-specific front-ends (approved objects, recursive selection)• Further project specific tools:

replace tabs, find non-ASCII characters in comments


CONCLUSIONS


Conclusions (TTCN-3 language)

Important TTCN-3 language features

• optional parameters backward compatibility, maintenance

• template restrictions in combination with naming conventions and appropriate tools no problems with receive templates used in send direction anymore

• encvalue/decvalue maintenance (no external functions needed)

• check operation, interleave race conditions

• testcase.stop well-defined termination of a test case in case of unrecoverable error situations

• visibility: "private", "friend“ not used yet (may improve quality regarding module dependencies)


Common Conclusions

Feedback of long-terms projects to the core language is vital

Tool Compatibility is needed

Shortened release cycle of TTCN-3 standards

TTCN-3 development is SW Development SW Engineering System Engineering

Long-term characteristic of a project needs to be considered from the beginning Conceptual errors will be expensive

Tool support is vital Common and project-specific tools


ADDITIONAL SLIDES


Release Upgrades - 1

Release Upgrade for Protocol and Signalling(„Baseline Moving“)• Typically every year• In general changes are backward compatible• Critical and non-critical extensions (TS 36.331 Annex A)

• Critical extensions• redefinition of message content or parts of it

(branching, e.g. r8 and r9 branch)• sender needs to know which release the receiver supports

• Non-critical extensions• Provision of additional information• e.g. new fields at the end of a record, new enumerated value• when not supported receiver shall skip the information

• Messages in general contain hooks for critical and non-critical extensions


Release Upgrades - 2

Impact on writing TTCN-3• Non-critical extensions

• In general no code duplication is necessary• Enhancement of existing templates by new optional parameters

(initialised with “*” for receiving and “omit” for sending)• Critical extensions

• New templates are needed• Challenge: minimised code duplication• Example:

template of a plain RRC PDU for UMTS up to several hundred lines of code clever template structure reduces duplication of code

• parameterised templates• modified templates


Documents

LTE CONFORMANCE TESTING: EXPERIENCES IN A LONG TERM PROJECT TTCN-3 User Conference 2011 – Bled, Slovenia © ETSI 2011. All rights reserved Presented by