Practical QoS Testing of Open Source and Commercial IMS Solution for Various Wireless Access Networks

Emina Luckin BH Telecom d.d. Sarajevo

Sarajevo, Bosnia and Herzegovina emina.luckin@bhtelecom.ba

Vlatko Lipovac University of Dubrovnik

Dubrovnik, Croatia vlatko.lipovac@unidu.hr

Abstract—IP multimedia subsystem (IMS) is the base for convergence of mobile and fixed networks as it offers possibilities to create open infrastructure of services based on IP protocol, which will enable simpler development of a variety of multimedia services. In this paper, we present testing of a voice call performance over various wireless networks, especially UMTS, using Open IMS Core system, and analyze the functionality of selected scenarios for voice calls, comparing the so obtained results with the performance of the commercial IMS system that provides built-in enhanced QoS and other functionalities, as well. We also analyze how coding rate affects the quality of voice calls in our test systems, presenting the measurements’ results for relevant QoS parameters and stationary scenario of the observed system. The results show that the open source solution for the IMS system, while having the QoS functionalities and the server with lower performance than those of the commercial solution, still achieves excellent performance, especially for the narrowband codec, while for the wideband speech codec, the performance remains somewhat lower than the one obtained for the commercial solution. All observed solutions exhibited good results when used for speech transmission, except the GPRS and the EDGE network when used with the wideband codec.

Keywords-IMS; Open IMS Core; UMTS; GPRS; EDGE; WLAN

I. INTRODUCTION IP multimedia subsystem (IMS) is the architecture for IP

based core network, which is supposed to enable fixed-mobile network convergence. Open IMS Core is the project of the Fraunhofer Institute for Open Communication Systems (FOKUS) [13] and is completely based on open-source software. It offers core functionality of the IMS and some extra QoS control functionality. In this paper, the results of testing and comparison of the QoS parameters for the Open IMS Core system and the commercial solution are presented. In addition, the influence of involving IMS system in communication via UMTS network was tested by means of measuring and comparing the QoS parameters for voice calls via SIP server on Internet using UMTS and for voice calls via UMTS access network and IMS system. With regard to tests in [1], which were done for signaling traffic, in this paper, we focused on testing of QoS

performance for unloaded system, while the tested RTP stream was using various IMS systems.

Main objectives of our research were: (i) To measure the performance of voice calls via Open

IMS Core system and various wireless access networks (WLAN, UMTS, EDGE, GPRS), in order to find out which one out of these achieved the best performance, and so conclude what is the performance of the UMTS access network for IMS call,

(ii) To compare QoS mechanisms of Open IMS Core (PCRF and PCEF implemented) with mechanisms implemented within the commercial solution of IMS system (UMTS K label on the related diagrams), namely: access control, echo cancellation, jitter buffer, packet loss concealment and dynamic reservation of resources, and to conclude if these advanced QoS mechanisms have significant influence on the performance of the tested system class,

(iii) To conclude which of the measured and derived QoS parameters provide the best overview of network conditions,

(iv) To compare the performance for accessing an Open IMS Core system while using various codecs, and so conclude about the influence of the codec selection on call quality,

(v) To examine the performance of UMTS access network and investigate the impact of the IMS system on call quality.

General characteristics of our test systems presumed: (i) Unloaded systems for the Open IMS Core and the

commercial solution, while for testing of voice calls via UMTS access network and SIP server (UMTS S label on diagrams), the server load was unknown. Evaluation of loaded system for signaling traffic is described in ETSI recommendations [11], [12] and [13],

(ii) The Open IMS Core, installed on the PC with 1024/128 kbit/s access link,

(iii) Stationary conversation participants, (iv) Testing voice calls, as a commonly used IMS

service. The GPRS, EDGE, UMTS and WLAN networks were

firstly tested in combination with Open IMS Core system. Then tests with UMTS network in combination with the

2010 Third International Conference on Communication Theory, Reliability, and Quality of Service

978-0-7695-4070-2/10 $26.00 © 2010 IEEE

DOI 10.1109/CTRQ.2010.36

169

commercial solution of the IMS system and SIP server were accomplished to provide the evidence for conclusions about the advantages of introducing 3G technology in mobile communications as confirmed in [2]. The UMTS will be compared to WLAN.

In this paper, we will first present selected QoS parameters that will be measured. Then, we will test the Open IMS Core and its QoS mechanisms for various types of wireless access. Using this test we will find out how much progress has been achieved by introducing 3G to mobile communications, and then test UMTS access network for SIP call, with aim to compare performances and determine the effect of introducing IMS system on the overall performance of voice calls. Furthermore, testing commercial solution of IMS system and its QoS mechanisms using UMTS technology will be performed. Then, results will be compared with those previously obtained.

II. STATE OF THE ART Papers related to this topic covered the following

research: (i) Vingarzan and Weik presented performance results

of the signaling plane based upon the Open IMS Core in the most common application of an IMS network – transmission of a voice call - when the clients are communicating over various wireless technologies testing of signalization traffic with combination of Open IMS Core and various access networks [1]. The obtained results are presented in ETSI TS 186.008 parts 1 to 3. However, no comparison to commercial solution of an IMS system has been made so far.

(ii) Prokkola, Perala, Hanski and Piri presented testing of HSDPA and WCDMA networks, performed in order to investigate the advantages of HSDPA, as a more advanced technology than the WCDMA [2]. No comparison for network neither with IMS, nor with any other wireless networks was performed.

As it can be seen, the current work lacks a comparison of service quality is achieved by using the Open Source IMS system with those achieved with commercial solution. In this work testing has been done for voice call using different codecs and different wireless networks. Using this aproach helps in getting very good overview of network quality of service, so it can be concluded if commercial solution improves this quality compared to Open Source IMS solution. Further improvements would have been done by testing systems loaded with traffic originating from fixed and mobile networks.

III. TEST RESULTS The performance analysis was accomplished for various

codecs of interest, separately for the narrowband and the wideband ones, including the following QoS parameters: the mean-opinion-score (MOS), sequence errors, 95 percentile jitter, 95 percentile delay, percentage of call durations with jitter<30 ms, percentage of call durations with delay<150 ms, packet loss, maximum delay, average delay, maximum

jitter, average jitter and bandwidth. However, in this paper, we presented only the results for MOS, 95 percentile jitter and 95 percentile delays. Complete measurement results are shown in Tables I, II and III.

TABLE I. MEASUREMENT RESULTS FOR MOS, R-FACTOR, 95% JITTER AND 95% DELAY

Access technology Codec MOS R-

Factor 95 % jitter

95 % delay

GPRS GSM FR 1.70 32.60 32.24 41.01

GPRS G.711 1.10 11.30 91.53 77.51

EDGE GSM FR 3.30 63.00 40.51 74.85

EDGE G.711 1.60 29.30 29.03 42.04

UMTS GSM FR 3.70 73.20 16.42 22.00

UMTS G.711 3.30 64.50 57.79 23.58

WLAN GSM FR 3.70 73.20 4.36 25.13

WLAN G.711 4.40 90.90 2.90 23.49

UMTS K* GSM FR 3.70 73.20 14.02 22.83

UMTS K* G.711 4.40 92.30 23.14 40.08

UMTS S** GSM FR 3.70 73.20 13.55 41.07

UMTS S** G.711 4.30 88.60 23.13 41.04

* Testing over commercial solution. ** Testing over SIP server.

TABLE II. MEASUREMENT RESULTS FOR JITTER <30 MS, DELAY<150 MS, PACET LOSS AND BANDWIDTH

Access technology Codec Jitter

<30 ms

Delay <150 ms

Pct. loss [%]

BW (kbit/s)

GPRS GSM FR 92.62 97.62 11.80 26.65

GPRS G.711 3.29 98.36 58.90 32.06

EDGE GSM FR 84.17 96.86 0.12 31.92

EDGE G.711 96.19 98.73 20.56 51.71

UMTS GSM FR 99.87 99.15 0.00 33.98

UMTS G.711 85.98 99.16 0.00 63.07

WLAN GSM FR 100.00 100.00 0.00 33.99

WLAN G.711 100.00 100.00 0.25 83.45

UMTS K* GSM FR 99.48 99.69 0.00 34.03

UMTS K* G.711 97.60 99.80 0.10 83.55

UMTS S** GSM FR 100.00 100.00 0.00 33.17

UMTS S** G.711 100.00 99.09 0.51 81.04

* Testing over commercial solution. ** Testing over SIP server.

170

TABLE III. MEASUREMENT RESULTS FOR MAXIMUM DELAY, MEAN DELAY, MAXIMUM JITTER AND MEAN JITTER

Access technology Codec

Max. delay [ms]

Mean delay [ms]

Max. jitter [ms]

Mean jitter [ms]

GPRS GSM FR 468.97 25.50 48.25 14.85

GPRS G.711 420.01 52.06 129.63 54.69

EDGE GSM FR 1022.03 19.41 107.04 23.42

EDGE G.711 339.18 32.28 44.58 16.19

UMTS GSM FR 266.05 19.99 34.71 4.86

UMTS G.711 1576.26 30.56 99.90 13.37

WLAN GSM FR 30.27 19.98 5.13 3.98

WLAN G.711 101.62 20.03 3.04 2.85

UMTS K** GSM FR 406.98 19.95 33.51 4.13

UMTS K** G.711 620.75 20.00 58.26 12.90

UMTS S** GSM FR 61.59 40.96 18.10 11.52

UMTS S** G.711 1058.05 20.61 31.32 13.80

* Testing over commercial solution. ** Testing over SIP server.

The values in Tables I, II and III will be discussed in the following sections.

A. Comparison of Codecs We selected the GSM FR as a narrowband codec

representative, and the G.711 A as a wideband one. These codecs are selected as the most widely used, while being supported by the client application.

The MOS reached its maximum, but got degraded in accordance with the codec performance, for the calls via WLAN and Open IMS Core, the UMTS call via commercial IMS System, and for the SIP call via UMTS network. MOS exhibited larger values for the GSM FR codec for UMTS, GPRS and EDGE networks in combination with Open IMS Core (Fig. 1).

Figure 1. MOS for the GSM FR and G.711 codecs.

The results of measuring 95 percentile jitter show better

values for the GSM FR codec, with exception of EDGE via Open IMS Core, which exhibits poorer results than the ones achieved with the G.711 codec, Fig. 2.

The results of measuring 95 percentile delay show better values for the GSM FR codec, with exception of EDGE and WLAN networks, which exhibit poorer values than the ones

achieved with the G.711 codec. For SIP calls via UMTS network, this parameter was found to have similar values for both codecs, Fig. 3.

Figure 2. 95 percentile jitter for the GSM FR and G.711 codecs.

After comparison of performances of voice calls via various access technologies and IMS system, it can be concluded that, for most of QoS parameters, excellent GSM FR codec’s results are evident, with small deviation when accessing Open IMS Core via GPRS, EDGE and UMTS (for MOS parameter). For other parameters characteristic deviation from this statement is evident for EDGE network.

Figure 3. 95 percentile latency for the GSM FR and G.711 codecs.

B. Test Results for the GSM FR Codec When MOS was measured, UMTS and WLAN for the

Open IMS Core system showed best results, followed by UMTS via commercial IMS, and the SIP call via UMTS access network, and tailing EDGE and GPRS. The Open IMS Core system influence on voice call was not notable, as only the impact of access network performance on call quality was found to be visible, Fig. 4.

Figure 4. MOS for various call scenarios (GSM FR codec).

When 95 percentile jitter was measured, we found out that WLAN exhibited the best performance in combination with Open IMS Core, followed by the SIP calls via UMTS network, the UMTS calls via commercial IMS, and, finally,

171

the calls via Open IMS Core and UMTS, GPRS and EDGE, Fig. 5.

The poor results for EDGE access network can, most likely, be attributed to link congestion, as coming out from the delay diagram. In accordance with the test results, we could conclude that the Open IMS Core introduced performance degradation of 2.4 ms; however, as this degradation could be the consequence of larger number of users in the UMTS cell at a particular time, the conclusion is not really significant.

Figure 5. 95 percentile jitter for various call scenarios (GSM FR codec).

When 95 percentile delay was measured, the calls via Open IMS Core and UMTS exhibited the best performances, while the calls via the commercial IMS were with slightly inferior values, which cannot be a reliable indicator of any influence that the Open IMS Core system could introduce. The values for WLAN, SIP via UMTS, GPRS and EDGE follow, Fig. 6.

Figure 6. 95 percentile latency for various call scenarios (GSM FR codec).

It can be concluded that the MOS reaches its maximum values when accessing Open IMS Core via UMTS and WLAN, and then when accessing the commercial solution and SIP server via UMTS. For other tests, this value is lower, as expected. Generally, the Open IMS and the commercial solution achieve similar results, while accessing SIP server via UMTS, which achieves slightly poorer results for latency than other technologies.

C. Test Results for G.711 Codec The maximum MOS values were obtained for WLAN

with the Open IMS Core system, and the UMTS access network for commercial IMS solution. The values achieved

for the SIP calls via UMTS, and UMTS, EDGE and GPRS via Open IMS Core, follow, Fig. 7.

In this case, the influence of Open IMS Core on call quality can be discussed, because the QoS is visibly degraded for the UMTS access with respect to the Open IMS Core system, as well as for the UMTS access with regard to the commercial IMS (for 25 %). The impact of introducing IMS system for voice calls is not notable.

Figure 7. MOS for various call scenarios (G .711 codec).

When 95 percentile jitter was measured, the Open IMS Core via WLAN access network showed the best performance, while the UMTS access to commercial solution and SIP calls via UMTS and EDGE exhibited better performances than UMTS (via Open IMS Core system), while the worst results were the ones achieved with GPRS.

The results for the calls via UMTS and Open IMS Core were 40% degraded with respect to those achieved when using the commercial solution, Fig. 8.

Figure 8. 95 percentile jitter for various call scenarios (G.711 codec).

When measuring 95 percentile delay, the Open IMS Core via UMTS and WLAN access networks exhibited the best performance, while the commercial solution - SIP calls via UMTS and EDGE via Open IMS Core, showed certain degree of performance degradation, topping for GPRS. The UMTS via Open IMS Core showed better results for 42 % than the commercial IMS, Fig. 9.

Except for measuring 95 percent delay, it can be concluded that the performance via UMTS access network and Open IMS Core system is degraded compared to the performance of voice calls via UMTS and the commercial solution of the IMS system. In addition, based on the measured MOS results and the objective analysis, it can be concluded that using voice calls via UMTS access network

172

and Open IMS Core is acceptable from the point of the end user quality of service.

It can also be concluded that the expectedly positive influence of introducing the IMS system via UMTS network is not significant for a voice call as in case of the GSM FR codec.

Figure 9. 95 percentile latency for various call scenarios (G.711 codec).

D. Other Conclusions As coming out from the measured results, when using

Open IMS Core system for voice calls via various access wireless networks, the best performance was achieved by WLAN, as expected, while for some QoS measurements, the UMTS performance comes close. The disadvantage of using WLAN for voice calls is relatively small area of coverage of a WLAN access point, so it can be only used for conversation with restricted movement of conversation participants. (The performance of the UMTS network for a moving call participants could be the subject of other research.)

According to [7] the recommended values for the QoS parameters in VoIP networks are:

(i) maximum packet loss of 1%, (ii) maximum one-way delay of 200 ms, (iii) maximum jitter of 30 ms.

However, these parameters do not necessarily reflect the real quality of service. So, for example, according to our tests results, some calls with the values of maximum jitter exceeding the above mentioned limit also exhibited the maximal MOS value, and so can be subjectively highly rated (e.g. the results for calls via UMTS network and Open IMS Core, via commercial IMS, G.711 codec). Similar conclusions can be derived for maximum delay results. On the other hand, the appropriate results for ranking quality of voice call require measuring 95 percentile jitter and delay, which is the percentage of call duration for which the relevant parameter is lesser than the certain critical value:

(i) packet loss <1%, (ii) 95 percentile delay < 150 ms, (iii) 95 percentile jitter < 30 ms.

From the table in Appendix A, it can be seen that all examined technologies, except GPRS, fulfill these conditions, as well as the EDGE access network in combination with the G.711 codec.

For the G.711 codec, the best results are associated with WLAN, while UMTS via commercial IMS and UMTS via Open IMS Core follow with close number of good results, and the SIP calls via UMTS network exhibit poor results.

For the GSM FR codec, the best results were also related to WLAN, while UMTS via commertial IMS and UMTS via Open IMS Core follow with the same number of good results, and the SIP calls via UMTS network enabled much better results than for the G.711 codec.

For unloaded system, the QoS mechanisms implemented on both Open IMS Core and the commercial solution provided very good results. The Open IMS Core for small number of users provided very good quality calls, while the advance mechanisms of the commercial solution would probably perform the best way for larger system load, which was not focused by this research.

IV. CONCLUSION After testing of performances of voice calls via Open IMS

Core and GPRS, EDGE, UMTS and WLAN wireless access networks, and comparisons of the obtained QoS parameters’ values with those for testing commercial IMS solution via UMTS network, and also, after testing calls via SIP server on Internet network and UMTS access, it can generally be concluded that Open IMS Core achieved results that follow the ones for using commercial IMS solution, despite the fact that Open IMS Core was installed on a PC with 1024/128 kbit/s access link only.

The Open IMS Core showed very good performance with respect to the advanced commercial solution, especially for the narrowband codec, while the performance was partly degraded when the wideband codec was used. It can also be concluded that using the IMS system generally increases the quality of calls, compared to SIP calls via Internet network.

The best performance by all of the above addressed criteria was achieved by WLAN access network, but coming out from the restriction of coverage of the Wi-Fi technology, it is possible to use it only for stationary users. All tested technologies could be successfully used for transmitting voice calls, except the GPRS for both codecs, and EDGE for the G.711 one.

As a major shortcoming of using the Open IMS Core system, we can point out the problem of using public and private IP addresses. If these are not within the same subnet, which is often the case, calls cannot be established between various networks. In addition, Open IMS Core would likely show weaknesses of its QoS mechanisms with larger number of users, and the commercial solution has few built-in functionalities aimed for this case.

REFERENCES [1] D. Vingarzan and P. Weik, “End-to-end Performance of the IP

Multimedia Subsystem over Various Wireless Networks”, Wireless Communications and Networking Conference, WCNC IEEE , 2006.

[2] J. Prokkola, P.H.J. Perala, M. Hanski, and E. Piri, “3G/HSPA Performance in Live Networks from the End User Perspective”, IEEE International Conference on Communications, 2009.

[3] T. Braun, M. Diaz, J. Enríquez-Gabeiras, and T. Staub, „End-to-End Quality of Service Over Heterogeneous Networks“, Springer, 2008.

[4] Z. Wang, “Internet QoS Architectures and Mechanisms for Quality of Service“, Morgan Kaufmann Publishers, 2001.

[5] G. R. Ash, „Traffic Engineering and QoS Optimization of Integrated Voice & Data Networks”, Morgan Kaufmann Publishers, 2007.

173

[6] G. Camarillo and M. A. Garcia – Martin, "The 3G IP Multimedia Subsystem (IMS)", Merging the Internet and the Cellular Worlds, John Wiley & Sons, 2006.

[7] M. Marchese, »QoS Over Heterogeneous Network«s, JohnWiley & Sons, Ltd, 2007.

[8] D. Soldani, M. Li, and R. Cuny, »QoS and QoE Management in UMTS Cellular Systems«, JohnWiley & Sons, Ltd, 2006.

[9] W. C. Hardy, »QoS Measurement and Evaluation of Telecommunications Quality of Service«, JohnWiley & Sons, Ltd, 2001.

[10] 3GPP TS 23.228 V8.5.0, »Digital cellular telecommunications system (Phase 2+), Universal Mobile Telecommunications System (UMTS), IP Multimedia Subsystem (IMS)«, Stage 2 (Release 8), last access date: 30.10.2009.

[11] ETSI TS 186 008, “Telecommunications and Internet converged services and Protocols for Advanced Networking (TISPAN) IMS/NGN Performance Benchmark Part 1: Core Concepts”, last access date: 30.10.2009.

[12] ETSI TS 186 008: “Telecommunications and Internet converged services and Protocols for Advanced Networking (TISPAN) IMS/NGN Performance Benchmark Part 2: Subsystem Configurations and Benchmarks”, last access date: 30.10.2009.

[13] http://www.openimscore.org/, last access date: 05.04.2010.

174