Quality of Service and Quality of Experience Correlations in a Location-Based Mobile Multiplayer Role-Playing Game Maarten Wijnants 1, Wouter Vanmontfort

Quality of Service and Quality of Experience Correlations in a

Location-Based Mobile Multiplayer Role-Playing Game

Maarten Wijnants1, Wouter Vanmontfort1, Jeroen Dierckx1, Peter Quax1, Wim Lamotte1, Katrien De Moor2, Jeroen Vanattenhoven3

1 Hasselt University – Expertise Centre for Digital Media (EDM) - IBBT

{firstname.lastname}@uhasselt.be

2 Ghent University – Media and ICT (MICT) - [email protected]

3 KULeuven - Centre for User Experience Research (CUO) - IBBT

[email protected]

Outline

• Introduction and Motivation• Research Context

– Location-Based Role-Playing Game

• Investigated Technological Factors• User Research

– Practical Approach and Participant Demography– Test Procedure and Data Collection

Methodology– Quantitative and Qualitative Findings

• Conclusions

06/10/2011 ICEC2011 2

Introduction and Motivation

• Quality of Service (QoS)– Well-established concept in software and system

engineering– Measure of technological performance

• E.g., performance guarantees in computer network

– Technical dimensions Objectively measurable

• Quality of Experience (QoE)– Fairly novel active research topic– Semantic variant of QoS

• Denotes overall experience or satisfaction of end-user• Humane, multi-disciplinary and highly subjective metric

and therefore hard to quantify objectively

06/10/2011 ICEC2011 3

Introduction and Motivation

• QoS and QoE are irrefutably related– Strong attention for technological factors remains

explicit in user research

• However, exact connection to date remains largely uninvestigated and therefore unclear– Such information is highly valuable for developers of

(distributed) applications• Decent understanding of their mutual impact and interaction

likely enables user experience optimization • Important asset in Customer Relationship Management

• This paper unveils QoS/QoE correlations via a hybrid user research methodology

06/10/2011 ICEC2011 4

Research Context

• The field of mobile gaming was selected as context to conduct our research– Increasingly popular application domain

• Focus on distributed multiplayer games– Network operation and performance might

substantially influence user experience

• Further scope reduction by concentrating on (outdoor) location-based games– Revolve around location determination: Position of

players/objects in the physical world is tracked and plays prominent role in gameplay

06/10/2011 ICEC2011 5

Research ContextLocation-Based Role-Playing Game

• In-house developed mobile location-based multiplayer game with WW I setting

• Played outdoors between two rivaling teams (Allied forces and German army)

• Location-based game– Players walk around in the physical world while

carrying a handheld device (netbook)– Player location tracking via GPS receiver

• Data dissemination via 3G connection• Game presents satellite view of the outdoor

environment

06/10/2011 ICEC2011 6


• Interaction occurs by “scanning” QR tags– Each player carries (unique) QR code– Codes in playing area represent virtual items

• Role-Playing Game (RPG)– Profile determines actions player can undertake,

his objectives and type of collectable virtual items• Commander: Remains at a fixed indoor location; has a

global overview of his team’s actions, permanently sees discovered objects and can instruct players on the field

• Soldier: Collect weapons; engage enemies• Spy: Collect intelligence items; reveal enemy players• Medic: Collect medical supplies; heal injured teammates

06/10/2011 ICEC2011 7


06/10/2011 ICEC2011 8

Investigated Technological Factors

• 3 technical/QoS-related parameters studied for their impact on player QoE– Each parameter acted as separate test condition

• Parameter 1: Player Location Determination Precision– Investigate impact of imprecisions on player QoE– Position of players artificially and randomly varied

in 10 meter radius around actual location

• Parameter 2: Player Location Refresh Rate– Important design variable for location-based apps– Refresh player locations only per 6 seconds

06/10/2011 ICEC2011 9

Investigated Technological Factors

• Parameter 3: Voice communication delay– Communication is affected by network delay– ITU recommends 1-way latency < 400 ms– 2 seconds of additional artificial delay

• Test conditions are representative, but the used magnitudes might be exaggerated– To elicit pronounced and unambiguous reactions

from test subjects

06/10/2011 ICEC2011 10

User ResearchPractical Approach and Participant

Demography

• 4 separate user studies with 8 subjects each– Participants evenly divided in 2 teams– Each participant assumed different role (random)– Deliberately not informed about QoS manipulation

• Test population demography– 12.5 % was female; average age 29 (SD = 9.04)– Over 80 % had played video game in past month– Professional occupation: mostly students (40 %) and

employees (50 %)

• 7000 m2 science park as playing area– 25 virtual items (QR codes) strategically dispersed

06/10/2011 ICEC2011 11

User ResearchTest Procedure and Data Collection

Methodology

• Test session of approximately 2 hours, logically enveloping 3 consecutive phases

• Pre-usage phase– Game objectives and mechanics verbally outlined– Paper-based questionnaire

• Basic socio-demographical characteristics• Previous digital gaming experiences and expectations

• During-usage phase– 40 minute duration; 4 intervals of equal length

• Three intervals exclusively devoted to 1 test condition– Investigate influence of corresponding QoS parameter

• 1 interval with “perfect” gameplay (reference scenario)

06/10/2011 ICEC2011 12

User ResearchTest Procedure and Data Collection

Methodology

• In course of play session, 4 identical in-game surveys needed to be electronically filled in– At end of each conceptual interval of the session– Acquire detailed QoE data in each test condition

• In-situ data sampling at the very moment of experience• Reduces risk of recall bias effect Accurate assessments• 11 statements measured on 5-point Likert scale

• Post-gaming phase– Second paper-based questionnaire

• Inquiries about overall experiences, feelings, thoughts

– Participant group conversation (focus group)• Share experiences, reflect on (non-)reported QoE factors

06/10/2011 ICEC2011 13

User ResearchQuantitative and Qualitative Findings

Global characteristics of non-instrumental dimensions•Users felt least ‘absorbed’ in role in interval 1; feeling monotonously increased as game progressed (same for ‘fun’)

– Users needed to get used to the game?

•Expectations adjusted in course of the gaming session– Pre-usage expectations were high; adapted according to context

06/10/2011 ICEC2011 14

Mean scores for in-game survey statements per test condition


Player location distortion test condition•Accuracy of location-related information concerning other players as well as objects was rated lowest•Influenced users’ perception of location update rate (both players and objects) Unjust!

06/10/2011 ICEC2011 15



Voice delay test condition•Participants felt most ‘frustrated’ + most ‘effort’ required•High rating for ‘experiencing technical problems in the communication with team members’ + lowest average score for ‘I can communicate smoothly with my team members’

06/10/2011 ICEC2011 16



Normal gameplay test condition (no QoS tampering)•Highest degree of ‘absorption’ in the played role + maximal score for ‘fun’•Highest rating for accuracy of player location information + for position refresh rate of both players and objects

06/10/2011 ICEC2011 17



Limited player location refresh test condition•Minimal ‘frustration’ and required ‘effort’ + high score for ‘fun’•Technical manipulation no notable influence on QoE

– Claim is supported by qualitative data from the post-gaming stage– Players were near the end rather centrally clustered (entire playing area already

discovered) less reliance on the data displayed on their device?

06/10/2011 ICEC2011 18



• During-usage study succeeded in relating modifications in technical circumstances to the users’ (subjective) experiences

• Example results from post-usage stage– QoS manipulation in 1st and 2nd playing condition were

experienced as sources of frustration• Consistent with findings from in-game surveys

– Quotes from focus groups• “The GPS update rate wasn't really a problem; most of the

time they weren't running, so you could easily track the other players”

• “As Commander I expect to know the precise location of my team; the faulty positions made this impossible”

06/10/2011 ICEC2011 19


Did player role influence QoE?•Statistically significant differences between roles for 5 aspects

– ‘Absorption’: Soldiers felt notably more immersed than Commanders– ‘Fun’: Soldiers had more pleasure than the Medic and Commander roles– ‘Effort’: Significantly higher for Soldier profile (esp versus Commander)– ‘Expectations’: Opposite of ‘effort’ (Commander > Soldier)– Player location update rate: Clearly lowest for Commander (cfr post-usage)

06/10/2011 ICEC2011 20

Mean scores for in-game survey statements per player role

Conclusions

• Two high-level conclusions– Modifications in 3 specific technical parameters

held implications for quality of users’ experiences• E.g., test condition without QoS impairments had highest

average scores for ‘absorption’ and ‘fun’

– For some sub-dimensions, player reported QoE diverged depending on assumed role

• E.g., maximal ‘absorption’ and ‘fun’ for Soldier profile

– Complementary qualitative and quantitative data

06/10/2011 ICEC2011 21

Conclusions

• Lower-level, specific findings illustrate extent of impact of QoS performance on QoE– E.g., voice communication latency complicated

coordination of players’ actions• Low ‘absorption’ + highest mean ‘frustration’ and ‘effort’

– Valuable information for software developers• How best to cope with QoS irregularities?

• ‘Absorption’, ‘fun’ monotonously improved while playing + users adjusted expectations– QoE not merely determined by technical params

• QoE exceeds QoS in terms of scope; importance and potential influence of non-instrumental aspects on QoE

06/10/2011 ICEC2011 22

Questions

06/10/2011 ICEC2011 23


• Findings from pre-usage surveys– Covered (among other things) same 11 QoE-

related statements as the in-game surveys – Pre-usage responses indicated that test subjects

attached most importance to• “Having fun” (Mean = 4.58)• “Not having technical problems while communicating

with team members” (M = 4.42)• “Being able to communicate smoothly with team

members” (M = 4.23)

– Analysis of post-usage data resulted in the same top 3 list

06/10/2011 ICEC2011 24

In-Game Questionnaire

1. I feel totally absorbed in my role in the game2. With the information that I am receiving about the position of the

other players, I am able to localize them without a problem3. I am having fun while playing the game4. I have to put a lot of effort in the game5. The kind of information that I am getting about the location of

objects, allows me to localize these objects without a problem6. I expected more from the game7. While communicating with team members I experienced technical

problems8. I feel frustrated9. The location of the other players is updated sufficiently10.I am able to communicate smoothly with my team members11.The location of objects is updated sufficiently

06/10/2011 ICEC2011 25

Documents

Quality of Service and Quality of Experience Correlations in a Location-Based Mobile Multiplayer Role-Playing Game Maarten Wijnants 1, Wouter Vanmontfort