Drive Testing Benchmarking v4b

NSO Service Delivery

Motorola Confidential

Page 1

UMTS Drive Testing & Benchmarking

-release 4-

Applies to TEMS 2.3.1 20 July 2004

Razvan Georgescu



Page 2

Table of Content

Table of Content ......................................................................................................................................... 2 1. Introduction ...................................................................................................................................... 3 2. Executive Summary ....................................................................................................................... 3

2.1 Summary Report Format - Example ........................................................................................ 3 2.2 Deliverables ................................................................................................................................. 3 2.3 Tools ............................................................................................................................................. 3

3. Drive Test Setup ............................................................................................................................. 4 3.1 Call scenarios .............................................................................................................................. 4 3.2 Drive test routes .......................................................................................................................... 5 3.3 Duration ........................................................................................................................................ 5 3.4 Test Case Description ................................................................................................................ 5

4. Drive Test Analysis ......................................................................................................................... 7 4.1 TEMS Outputs ............................................................................................................................. 7 4.2 Formulas, Counters, Measurements ...................................................................................... 7 4.2.1 Formulas - Voice calls MO .................................................................................................... 7 4.2.2 Formulas - Data calls MO ..................................................................................................... 7 4.2.3 Voice calls MO TEMS Raw Output ................................................................................... 8 4.2.4 Data Calls TEMS Raw Output ........................................................................................... 9 4.2.5 Radio Health Indicators ...................................................................................................... 12 4.3 SMAP Outputs .......................................................................................................................... 14 4.4 NRG Outputs ............................................................................................................................. 14 4.5 Pareto Analyis Outputs ............................................................................................................ 14

5. Equipment List .............................................................................................................................. 15 6. Appendix ........................................................................................................................................ 16



Page 3

1. Introduction

This document intends to give a general guideline for collecting drive test data with the primary scope of collecting KPI ( benchmarking) and secondary to collect basic call log data

2. Executive Summary

Summary Report Format - Example

CSSR (average) LOAD

RAB 0% Call Attempts Requirement

AMR MOC 98,25 343 97

AMR MTC 98,58 212 97

AMR MTM 97,78 270 97

AMR Total 98.18 825 97

CS 64 MTM 97,76 268 95

PS64 100,00 44 97

PS128 97.16 423 97

PS384 (static) 100,00 154 97

DCR (average) LOAD

RAB 0% Successful Call Setups GSD Req.

AMR 1,2219 1064 3

CS 64 0,57416 406 5

PS64 0 44 3

PS128 0.97 411 3

PS384 (static) 0 154 3

Deliverables

KPI counters

Logs for offline analyisis and post processing stored at a Motorola location and available on the intranet

Pareto for call failure causes

Tools

Note: At this date TEMS 2.4 is available, however no time for tests yet, release notes do not suggest changes in the event definition which could impact the KPI calculation described further in thei document.



Page 4

Tools to be used:

TEMS Investigation WCDMA 2.3.1 , Motorola A835 phones with NETMON - UE logging and call counting

SMAP RNC logging NRG post processing

System Retainability & System Accessibility statistics will be generated from calls which are 2 minutes long for all types of calls, voice, video, data in order to align to the test calls done in Porto by GSD. This is to have KPIs deducted from the same type of calls in each location subjected to GSD aligned benchmarking. If this alignment is not needed, then the TEMS Practices document describes an alternative procedure which is also used by other major UMTS test equipment manufacturers.

3. Drive Test Setup

Call scenarios

All tests done in car and in motion. Voice calls & Video calls: 2 phones simultaneously MS1 calls MS2 both in the same car. This is an atipycal subscriber behaviour. This scenario is adopted though for testing purposes ,because we need to log with 1 TEMS kit both the call originating mobile and the paged mobile with the scope of collecting logs for debugging. KPIs only for paging can be deduced much easily only using UTRAN stats.

120 sec calls + 30 sec breaks

Data calls: - 1 phone

FTP, the data quantity adapted for each bearer speed. 2 minutes long + 30 seconds break

UL FTP tests also to be included

UDP as required



Page 5

Drive test routes To ensure proper functionality validation of the UTRAN, the drive test routes for KPI benchmarking should be done in areas with proper coverage. For orientation we define the following values to be measured outdoor. o Good RF coverage, UTRAN Carrier RSSI > -70 dBm, CPICH Ec/Io > - 10 dB, 1/2/3 way SHO o Degraded RF coverage, UTRAN Carrier RSSI < -90 dBm, CPICH Ec/Io = -10 dB to 14 dB

Duration TBD. Depending on call volume. Can be from minimum 3 days to more than 1 week for the complete set of tests. Remark: Reports must be delivered weekly. This will be a live table where KPIs for different calls must be updated daily as the revelant test was performed.

Test Case Description All tests done in car and in motion. with the possible exception of 384K A description of the initial test cases is in the follwing table. Please treat this as a guideline to be adapted locally according to the needs and resources.



Page 6

Which mobile

Scope of Calls

Call Sequence Description

Call Sequence Handling

Log Ue Log UTRAN

Post Processing

Minimum time needed for execution

Comments

Day 1, Voice Calls

5 hrs absolute minimum effective drive time

Practically takes whole day.

100 MTM voice call

MS1 Motion to MS2 Motion

CSSR, DCR, Pareto

120 sec call + 30 sec break

automatic

TEMS logging both mobiles

SMAP TEMSNRG

Day 2, Video Calls


100 MTM video call

MS1 Motion to MS2 Motion

CSSR, DCR, Pareto


manual

TEMS logging both mobiles

SMAP TEMS,NRG

Day 3, Voice Calls MO to fixed + Downloads


100 MO voice call

MS1 Motion to fixed line (answering machine, or TEST number )

CSSR, DCR, Pareto


automatic TEMS SMAP TEMSNRG

Need fixed line and answering machine or TEST number

100 Download sessions

MS2 Motion ftp ownload 128 K (or preffered service)

CSSR, DCR, Pareto

120 sec download + 30 sec break

automatic TEMS

Day 4, Uploads (+ optional MO to fixed)


100 Upload sessions

MS2 Motion ftp upload 64 K

CSSR, DCR, Pareto

120 sec upload + 30 sec break

automatic SMAP TEMSNRG

TOTAL= 6 working days



Page 7

Notes: 1. Day 5 & 6, Additional tests to be performed as needed on non preffeded data bearers - 64 K 384 K, UDP 2. TOTAL= 6 working days

3. The TEMS logging and UE control is done according to the general TEMS practices, for details please review Drive Testing and Benchmarking TEMS practices. 4. SMAP, NRG usage are subject to usage guidelines from GSD who are the tool developers. Any issues related to these tools should be directed to GSD. 5 .FULL BENCHMARKING of all services requires more than one working week with 1 team or alternative 2 teams or reduce call volume as per TEMS Practices if the performance is good and no alignment is required.

4. Drive Test Analysis

TEMS Outputs

Note: All these reports are generated using the Report Generator function in TEMS. Reports can be generated by anyone having TEMS instabbeld on his PC, without the dongle.

Formulas, Counters, Measurements

Formulas - Voice calls MO

CSSR = [Call Established / Call Initialization ]* 100 rounded at 2 decimals DCR = [Dropped Call / Call Established ] *100 rounded at 2 decimals

Formulas - Data calls MO

CSSR = [Session Start / RAS Dial ]* 100 rounded at 2 decimals DCR = [Session Error / Session Start] *100 rounded at 2 decimals



Page 8

Voice calls MO TEMS Raw Output

Event #[no.of] #Log

Blocked Call 6 2

Call Established 75 1 ,2

Call Initialization 82 1 ,2

Call End 66 1 ,2

Call Setup 76 1 ,2

Dropped Call 9 2

Missing Neighbor Detection

0 -

Call Attempt 85 1 ,2

RRC Established 76 1 ,2

CSSR = [Call Established / Call Initialization]* 100 = [75 / 82 ]*100 = 91.5 % rounded at 2 decimals DCR = [Dropped Call / Call Established ] *100 = [9 / 75 ]*100 =12 % , rounded at 2 decimals

Setup Times



Page 9

Data Calls TEMS Raw Output

Event #[no.of] #Log

Packet Service Attach 1 1

Packet Service Attach Failure

0 -

Packet Service Detach

0 -

PDP Context Activation

7 1

PDP Context Activation Failure

0 -

PDP Context Deactivation

7 1

RACH Error 0 -

RRC Connection Abnormal Release

5 1

RRC Connection Reject

1 1

RRC Established 6 1

RAS Error 0 -

RAS Dial 7 1

RAS Hangup 7 1

IP Address Assigned 7 1

Session Start 7 1

Session End 6 1

Session Error 1 1

A normal Call Sequence for Data Calls is:

DATA CALL SETUP PART



Page 10

RAS Dial .. . based on the fact that the phone responds properly to the RAS Access from the PC PDP Context Activate IP Address Assigned Session Start .. .. FTP Get starts here .. I have successfully set up a Data Call, this is including also FTP server authentication which prooves that my data pipe is usable and I do not just have a blind PDP Context. If the call fails during server authentication before Session Start, this is technically a dropped call. The user may interpret it as a setup failure, that is why if the counters for IP Address Assigned and Session Start are not equal, these cases are recommended to be analysed by replaying the logs.

DATA CALL END PART Session End . The Data Transfer was successfull / or Session Error- Call is Dropped RAS Hangup release RAS from PC PDP Context Deactivated . end PS connection If Session Start counter = IP Address Assigned counter CSSR = [Session Start / RAS Dial ]* 100 =xy.z % rounded at 2 decimals If Session Start < IP Address Assigned , analyse and adjust the Call Setup figure and the DRC figure accordingly to compensate. Note: Low variance is to be expected here. Generally if IP Address Assigned but Session cannot Start then there is a functionality issue in the PS core or FTP server.

DCR = [Session Error / Session Start] *100 = xy.z % , rounded at 2 decimals Link Reestablishments :

these are meant to save the PS call if the RL fails, due to the specifics of the user data, such a procedure is not realistic for voice / video CS calls.

Reestablishments are not to be considered into the DRC statistics.

If the Reestablishments are too often, causes must be analysed. Visual investigation of the data transfer behaviour:



Page 11

The 64 K bearer. On the graph it can be seen at 58 kbps app throughput (ftp) The 128 K bearer is reaching above 110 k the horizontal blue line is showing that. The 384K bearer is not really stable like the 128k and 64k. It suffers either from reconfiguration back to 128 triggered every few seconds ( without dropping, which is good) or when it is stable assigned the throughput stays usually above 250 kbps, mostly above 300. Setup Times Attach



Page 12

PDP Context Activation

Radio Health Indicators

Below only some examples which can help you grossly evaluate the quality of the radio environment in a network.



Page 13



Page 14

SMAP Outputs

Documented by GSD on request.

NRG Outputs

Documented by GSD on request.

Pareto Analyis Outputs

Attached the original Pareto template click to open.



Page 15

5. Equipment List

Item Equipment Description Qty Comments

1. Car 1 Renault Espace or similar MPV

2. DC/ AC 12V >> 100240 V inverter 1 For use with the equipment not running on accus or requiring a usage time longer than possible with own accus

3. USB Cables for A835 for Flash and Data 2 For Flashing and Data Transfer

4. Portable USB Hub 1

5. Serial GPS for TEMS 1

6. TEMS kits + 2 x A835 TEMS phones 1 For Drive Testing

7. A835 phones 1 per engineer

8. Digital Maps for TEMS

9. Laptops for drive testing 1 Min PIII with accus lasting more than 2 hrs.

10. Iu Analyser 1 For IuPS, CS, IuB Agilent

11. SIM cards 3 ( CS& 64 + CS&128 + CS&384)

12. Additional accus for A835 2 1 extra per phone

13. Dongles for RTA/PST/Radicom 4 1 per Motorola location

14. PSTN line 1 For Mobile Terminated Calls

15. NRG 1

16. SMAP 1

17. Laptop Holder 1 For incar

18.



Page 16

6. Appendix Tools package available in compass at http://compass.mot.com/go/140807456

Documents

Drive Testing Benchmarking v4b