1 © 2005 Nokia Thesis_presentation.ppt / 2007-02-06 Benchmarking process as a part of risk identification 06.02.2007 Author: Marita Harju Supervisor: Docent

1 © 2005 Nokia Thesis_presentation.ppt / 2007-02-06

Benchmarking process as a part of risk identification

06.02.2007

Author: Marita Harju

Supervisor: Docent Timo O. Korhonen

Instructor: BSc. Jari A. Juntunen


Content

• Thesis context and objectives

• Process definition

• Implementation

• Conclusion


Thesis context

• Benchmarking • Definition: Continuous process of measuring products, services, and

practices against the company’s toughest competitors (Juran 1999)

• Effective method of establishing performance goals and quality improvement projects based on industry best practices

• Can be used to identifying gaps between the own product and the competitors, helping to identify what must be changed and to showing what is possible

• Performance testing• The goal of performance tests is to evaluate the compliance of a system or

component with specified performance requirements (BS 7925-1 1998). Level of performance is the degree to which the needs are satisfied, represented by a specific set of values for the quality characteristics (ISO/IEC 9126-1 2001).

• Usability• The degree to which the software is easy to use (ISO 9126 standard 2001)


Thesis objectives

• The purpose of this Thesis was to

1. define benchmarking process for mobile devices

2. find out how the benchmarking process can be used as a part of risk and fault identification

• Research questions:

1. “How to select main use cases for benchmarking in question?” The follow-up for this question is to consider how to create relevant test assets according to use cases.

2. “How benchmarking can be used as a part of risk and fault identification?”

• The objectives are discussed in scope of selected quality attributes:

1. performance (main attribute)

2. usability


Process definition

•The process is defined for benchmarking of S60 based mobile devices against competitors

•The process is based on formal benchmarking process and fundamental testing process


Implementation: Test setting

• Testing scope• Whole phone

• Quality attributes: Performance & usability

• Phones• Nokia E60

• Motorola RAZR V3

• i-mate JAMin


Implementation: User data & test execution

• User profile was defined using data from S60 device user panel

• Use cases were defined according to user profile

• Main use cases = the most used 20 percent of all use cases

• Test cases were defined according to use cases and user profile

• In this case, only PIM (Personal Information Managements) and messaging related cases are developed

• Test execution• Performance: Response times• Usability: Heuristic evaluation

pim29%

multimedia9%

messaging29%

infotainment0%

games2%

config6%

browsing4%

utitity4%

unknown1%

productivity1%

voice calls15%

Phone usage activities by average S60 phone user


Implementation: Performance test results, PIM

Test ID Test case name Data Unit Nokia E60 Motorola V3i-mate

JAMin Notes

PIM

1 Opening calendar application 255 meetings s 1.4 0.1 0.6

2 Opening clock application - s 1.5 0.1 0.8

3 Opening logs application 20 logs s 0.1 0.1 0.5

4 Opening phonebook application 200 items s 0.5 0.6 0.6

5 Saving a new contact to phonebook 200 items s 0.5 4.7 0.6

6Calendar view switch from month view

to day view 255 meetings s 0.6 - 0.1

7Calendar view switch from week view

to month view 255 meetings s 0.6 0.8 0.1

8Calendar view switch from day view to

week view 255 meetings s 0.8 0.6 0.6

9 Opening calendar item 255 meetings s 1.2 0.1 0.1

10Initializing a phone call using

phonebook application 200 items s 3.2 3.4 4.4

11 Initializing a phone call using logs 20 logs s 3.4 2.9 4.1

12 Opening a contact item 200 items s 1.0 0.6 0.6


Implementation: Performance test results, messagingTest ID Test case name Data Unit Nokia E60 Motorola V3

i-mate JAMin Notes

Messaging

13 Opening messaging application40 SMS + 5

MMS s 1.4 0.7 0.1

14

Opening a received SMS from idlenew SMS and

other 40 SMS + 5MMS s 3.0 1.0 0.7

i-mate opened inbox view, not the SMS

15

Creating a new SMS message 40 SMS + 5 MMS s 1.2 0.8 0.8

16

Deleting the first SMS message from inbox 40 SMS + 5 MMS s 0.6 2.5 0.1

17

Deleting the first MMS message from inbox 40 SMS + 5 MMS s 0.8 2.2 0.1

18

Opening a received SMS message 40 SMS + 5 MMS s 1.4 0.8 1.0

19

Opening a received MMS message 40 SMS + 5 MMS s 2.8 4.3 2.3

20

Opening messaging inbox 40 SMS + 5 MMS s 0.1 1.1 0.1

21

Adding image to a MMS message 40 SMS + 5 MMS s 2.6 2.7 0.9

22

Creating a new MMS message. 40 SMS + 5 MMS s 1.7 0.7 0.8


Implementation: Result summary & next steps

Nokia E60

Motorola V3

i-mate JAMin

Feature Score Score Score

Calendar 3.0 2.0 5.0

Clock 5.0 1.0 2.0

Phonebook 5.0 3.0 2.0

Logs 2.0 5.0 2.0

PIM average 3.8 2.8 2.8

Messaging 3.0 4.0 2.0

Total (0.51 x PIM+ 0.49 x messaging) 3.4 3.4 2.4

Nokia E60

Motorola V3

i-mate JAMin

Average PIM index 0.6 0.8 0.7

Average messaging

index 0.4 0.5 1.0

Overall average

index 0.5 0.6 0.8

• Performance results are summarized using performance index (calculated from measured response times)

• Usability evaluation was done using heuristic evaluation (scores from 0 to 5). Summarizing is done by calculating averages.

• Results in short:• Poor message opening performance in

S60 device (E60), especially SMS opening performance is weak

• S60 device’s usability is in average level

• Next steps• Performance: More testing on SMS area

• Usability: More thorough evaluation with several evaluators

Usability, score scale 0-5

Performance index, scale 0-1


Conclusion1/2The purpose of this Thesis was to

1.define benchmarking process for mobile devices2.find out how the benchmarking process can be used as a part of risk and fault identification

1. The test execution according to the defined benchmarking process proves that in a general level the process is working

2. It seems that the process is able to fulfill its main target; it can find gaps and therefore is able to identify improvement areas, risks and faults in S60 devices


Conclusion2/2Research questions:

1. “How to select main use cases for benchmarking in question?”. The follow-up for this question is to consider how to create relevant test assets according to use cases.

2. “How benchmarking can be used as a part of risk and fault identification?”

1. Main use cases are the most used 20 % of all use cases. Test asset, including e.g. test cases, is defined according to use cases and user profile (end user point of view).• The selection to use S60 device based user profiles as a basis for main use

case definition makes the cases a bit S60 device specific, but… • Results are easy to compare, because the same profile is used for all products• It is possible to establish quality goals to S60 devices in their main use areas

according to results obtained for other device.

2. Several iterations of the benchmarking process is the answer to the question how the benchmarking process can be used as a part of risk and fault identification. • In the first iteration the selected quality attribute of the device is tested as a

whole • If there are some gaps found in the monitoring results- step, those gaps can

be then further investigated in the second iteration of the process


Questions?


References

• Juran, J.M., 1999. Juran’s Quality Handbook. McGrawHill.

• British Standard (BS) 7925-1:1998. Software testing Part 1: Vocabulary.

• ISO/IEC 9126-1: 2001. Information technology- Software Product Quality – Part 1: Quality model.

Documents

1 © 2005 Nokia Thesis_presentation.ppt / 2007-02-06 Benchmarking process as a part of risk identification 06.02.2007 Author: Marita Harju Supervisor: Docent