Confidential

3/152 83-LXE 110 0261 Uen Rev PA4 2008-04-22 1

OSE Chinese Text Input Benchmark Test Report

Chen, JohnSony Ericsson Experience Lab BJ2008.Mar.-Apr.

Confidential

3/152 83-LXE 110 0261 Uen Rev PA4 2008-04-22 3

Purpose

• To benchmark the Chinese text entry speed, accuracy (on 12 key pad phone) and user’s subjective satisfaction between Sony Ericsson (OSE) and main competitors’ mobile phones.

• To identify what are good designs and what are bad designs regarding Chinese text input on keypad phone.

• And to provide data support to push changes of Chinese text input on Central platform.

Confidential

3/152 83-LXE 110 0261 Uen Rev PA4 2008-04-22 4

Executive Summary

• Nokia had the best usability because of the highly consistent UI, simple design and considerate association words, phrases and even symbols.

• A4 showed the fastest texting speed, but a little lower on usability than Nokia due to the slight inconsistency and the way of  choosing single candidate Pinyin.

• Sony Ericsson had both the slowest texting speed and also the worst usability due to the obvious system lag, lack of new features, and hidden functionalities for long press operations (switch input mode and choose candidates related number).

• Samsung had the relative fast texting speed, but lower usability ratings due to the hidden functionality on “1” key, and hidden long press on asterisk key to input symbols.

• Chinese users has very limited experience in Latin input. Almost all the test users got confused while entering uppercase “MSN” in SonyEricsson, Samsung and Motorola, due to the design of separated key to switch Latin input mode.

Confidential

3/152 83-LXE 110 0261 Uen Rev PA4 2008-04-22 5

Content

• Methodology

• Results: Entry Speed Analysis

• Results: SUS Analysis

• Design Recommendations

• Recap

Confidential

3/152 83-LXE 110 0261 Uen Rev PA4 2008-04-22 6

Methodology

Confidential

3/152 83-LXE 110 0261 Uen Rev PA4 2008-04-22 7

Research Design

• Hardware keypad performance benchmark• The results as a controlled factor

• 2 (user experience level) x 5 (text entry interfaces) x 4 (text type) design• User experience level as between subject factor• Text entry interfaces & text type as within subject factors

Test participants(10 normal Pinyin users)

Test phones(the order of phone presented counterbalanced)

p1,p2 A B C D

P3,p4 A B C E

p,5,p6 A B D E

p7,p8 A C D E

p9,p10 B C D E

Test participants(9 experienced Pinyin users)

Test phones

p11,p12 A B C D

p13,p14 A B C E

p15,p16 A B D E

p17,p18 A C D E

p19,p20 B C D E

Confidential

3/152 83-LXE 110 0261 Uen Rev PA4 2008-04-22 8

Apparatus

• Test phones• ZTA4 (Nokia 3230 S60 with ZTA4) • Nokia 2610 S40• Motorola E398 (iTap)• Samsung J218 • Sony Ericsson K530 (OSE)

• Video camera / stopwatch• Stopwatch used as primary timer• Video recording used as backup timing

Confidential

3/152 83-LXE 110 0261 Uen Rev PA4 2008-04-22 9

Users Profile

• 19 participants in all, aged from 20-30, 8 females and 11 males, mixed

• 10 Normal Pinyin users, SMS usage: <10 SMS/day• Text entry speed on own phone is 12 wpm (words per minute) on average,

varies between 7wpm to 16wpm

• 9 Experienced Pinyin users, SMS usage: >15 SMS/day• Text entry speed on own phone is 20 wpm on average, varies between

17wpm to 26wpm, average is 20 wpm

Confidential

3/152 83-LXE 110 0261 Uen Rev PA4 2008-04-22 10

Measurement

• Chinese Pinyin Text entry speed• Keystroke level• Character level

e.g. Ni xian zai mang ma (15 characters, 21 keystrokes in OSE phone K660) 你 现 在 忙 吗 (5 words)

Note: 405 Pinyin strings are used for a total of 7513 common Chinese characters, the average Pinyin length is 3.21, and the average Pinyin length weighted by number of characters having the same pronunciation is 3.07 (US Patent No. US006822585B1)

• SUS (System Usability Scale)• 10 items of 5 point Likert Scale (e.g. “I thought the input method was easy to use”)

• Qualitative data collection (observations & user’s comments etc.)• Interview after test • Affinity diagram to highlight user’s behavior and needs

Confidential

3/152 83-LXE 110 0261 Uen Rev PA4 2008-04-22 11

Test tasks

• Text copy task• Entry speed measurement

• Have user enter the same sentence in their own phone, and record time“ 王总，产品报价发到您的邮箱了，请查收，如有任何疑问，请随时联系我。”

• Practice task (have users get familiar with the key mapping & keypad feeling etc)• Have user enter 2 messages which he/she actually sent in his daily life (keep th

e 2 messages the same for each phone practice task)

• Test task (have users complete the task as quick & accurate as possible)

• Chinese content: (from SMS investigation*)• 先到附近的嘉禾一品或者肯德基吃点东西 (Only Chinese text)• 我在单位呢 现在上不了 MSN (Mixed Chinese & letters)• 晚上去哪儿 happy ？ (Mixed Chinese, English& symbols)• 请回电话 58659394 (Mixed Chinese & numbers)

* An online SMS content investigation was conducted before the test to extract typical SMS expressions/sentences for text copy task.

Confidential

3/152 83-LXE 110 0261 Uen Rev PA4 2008-04-22 12

Results: Entry Speed analysis

Confidential

3/152 83-LXE 110 0261 Uen Rev PA4 2008-04-22 13

Entry Speed - time spent on sentences

• ZTA4 has the fastest entry speed on all sentences

• SE spent longer time than competitors on all sentences

Confidential

3/152 83-LXE 110 0261 Uen Rev PA4 2008-04-22 14

Entry Speed- time spent on Chinese only sentence

• Sony Ericsson Chinese entry speed was significantly lower than other competitors.

• ZTA4 was almost as fast as user’s entry speed on their own phones even it was new for them.

“ 先到附近的嘉禾一品或者肯德基吃点东西”

Confidential

3/152 83-LXE 110 0261 Uen Rev PA4 2008-04-22 15

Why? - Observations and comments

• Why Sony Ericsson was the slowest one?• Lag of showing Pinyin (system response was slow).• System lag made a lot of mistakes • Unintuitive scrolling of candidates• Pinyin candidates were hidden• “C” key and “Back” key confusion made mistakes

• Why ZTA4 was the fastest?• Easy to get started in spite of previous texting experience due to the good compatibility• Phrase input, even for non-dictionary phrase entry• Fast word frequency adjustment• Word frequency adjustment effect (due to NOT reset the phone after each single test session)

Confidential

3/152 83-LXE 110 0261 Uen Rev PA4 2008-04-22 16

Entry speed- time spent on Chinese +”MSN”

• Sony Ericsson was significantly slower than other competitor phones.

• ZTA4 was fastest due to the uppercase and lowercase was in the input mode switch loop, which was quite clear for Chinese users.

“ 我在单位呢 现在上不了 MSN”

Confidential

3/152 83-LXE 110 0261 Uen Rev PA4 2008-04-22 17

Entry Speed- time spent on Chinese + “happy?”

• Sony Ericsson was significantly slower than other competitor phones.

• ZTA4 was the fastest, partly because after entering “happy”, the symbols bar appeared automatically.

“ 晚上去哪儿 happy ？”

Confidential

3/152 83-LXE 110 0261 Uen Rev PA4 2008-04-22 18

Why? - Observations and comments

• Most Chinese users got confused and very frustrated to enter uppercase “MSN” in some models.

• According to the SMS online survey, most Chinese had very limited experience with Latin input.

• SonyEricsson and Samsung has separated switch key for Latin input (Lowercase, uppercase and initial letter capitalized), while ZTA4 and Nokia has lowercase and uppercase in the input mode switch loop which was quite clear.

Confidential

3/152 83-LXE 110 0261 Uen Rev PA4 2008-04-22 19

Entry Speed- time spent on “Chinese + 58659394”

• Sony Ericsson and Motorola was slower than the other 3 phones significantly.

请回电话 58659394

Confidential

3/152 83-LXE 110 0261 Uen Rev PA4 2008-04-22 20

Why? - Observations and comments

• Phrase input, association words and the way of switch were the main influence factors.

• ZTA4, Nokia and Samsung had phrase input, good association, which made this simple Chinese sentence input very quick.

• SonyEricsson and Motorola lack these new features to facilitate texting speed. And SonyEricsson needed long press to switch input mode, which made it slower than Motorola.

Confidential

3/152 83-LXE 110 0261 Uen Rev PA4 2008-04-22 21

Results: SUS analysis

Confidential

3/152 83-LXE 110 0261 Uen Rev PA4 2008-04-22 22

SUS Scores Comparison

• Both ZTA4 and Nokia were rated as good usability. Normally SUS* scores above 60 shows good usability.

• No significant difference between experienced Pinyin users and normal Pinyin users. Although experienced users rated higher on Samsung than normal users.

* SUS (System Usability Scale) is a ten item attitude scale, developed by John Brooke in 1996. Here it was used to measure the usability of several user interfaces of Chinese text input.

Confidential

3/152 83-LXE 110 0261 Uen Rev PA4 2008-04-22 23

Why?- Observations and comments

• Why Nokia and ZTA4 had higher usability?

• Nokia had the most consistent operation.• “Anytime I can use asterisk key to input symbols and use pound key to switch inpu

t mode, it’s easy”

• A4 was very easy to get started for most test users, in spite of his/her previous texting experience. e.g. allow users to choose candidates with 5-way Nav. key, numerical keypad, and also using number "1" to indicate number index etc (like Samsung)..• “I can use it in any way what I want. It’s just what I want.”

Confidential

3/152 83-LXE 110 0261 Uen Rev PA4 2008-04-22 24

Why?- Observations and comments

• Why SonyEricsson got the worst usability?

• Obvious System lag • “I want to type quickly, but the system is too slow to support quick typi

ng. I need to wait.”

• Hidden functions due to long press

• Long press to switch between Chinese and Latin

• Long press to chose candidates related number

Confidential

3/152 83-LXE 110 0261 Uen Rev PA4 2008-04-22 25

Perceived keypad Performance Impact

• There’s no statistical significance among the test phone’s perceived hardware performance impact on text entry speed.

• Although ZTA4 keypad was perceived to have slightly bigger impact on text entry speed, but it still showed the fastest texting speed.

- SUS item 11 analysis

“Keypad handfeel , screen size etc have big impact on my text entry speed.”

Confidential

3/152 83-LXE 110 0261 Uen Rev PA4 2008-04-22 26

Design Recommendations

Confidential

3/152 83-LXE 110 0261 Uen Rev PA4 2008-04-22 27

How these recommendations comes- Affinity Diagram

Confidential

3/152 83-LXE 110 0261 Uen Rev PA4 2008-04-22 28

6 Most Important

• Consistence: Keep consistence of single key’s functionality and keep consistent with user’s former experience as well

• Features: Advanced features facilitate text speed

• Flexible: Allow users to make mistake and recover from mistakes easily and costless

• Simple: Less eye movement and fewer clicks

• Feedback: Provide clear and in-time feedback

• Help: Provide tips on-demand

Confidential

3/152 83-LXE 110 0261 Uen Rev PA4 2008-04-22 29

Consistence

• Keep consistence of keys (especially *, #, and “1” key) in different input mode

• Keep consistent in different situation (especially for “space” key and switch input mode key)• When there’re predictions on screen, I can still

enter space or switch input mode without back or delete the predictions firstly.

• Enter symbols easily and keep consistent in different input mode

Confidential

3/152 83-LXE 110 0261 Uen Rev PA4 2008-04-22 30

Consistence (cont.)

• Simplify uppercase and lowercase entry for Chinese• Put uppercase and lowercase separately in the switc

h loop

• Do not use long press in frequently used functions, e.g. switch input mode, enter symbols, choose number index etc.• Provide users more ways to choose candidates, e.g.

navigation and number index.

• Simplify the definition of “C” key and “Back” key in Chinese text input• Most users confused about “C” key and “back” k

ey• Whenever users need to ‘delete’ or ‘back’, they

tend to press “C” Key

Confidential

3/152 83-LXE 110 0261 Uen Rev PA4 2008-04-22 31

Features

• Phrasal input• Capability of dictionary phrase input and also non-dic

tionary phrase input• User dictionary: Memorize user’s non-dictionary phr

ase

• Fast frequency adjustment

• Abundant and smart association

• Not only word association, but also phrase and symbols association like Nokia does which was very appreciated by most test users.

Confidential

3/152 83-LXE 110 0261 Uen Rev PA4 2008-04-22 32

Flexible

• Allow users to make mistakes and recover from mistakes easily and costless.

• Allow users choose candidate Pinyin and candidates word in a loop

• Allow users move forward and backward freely using Navigation keys

• Keep my association words after I back from typing mistakes.• Samsung didn’t dismiss the association words w

hich made users feel good

Confidential

3/152 83-LXE 110 0261 Uen Rev PA4 2008-04-22 33

Simple

• Keep Pinyin and candidates bar close on the screen

• Don’t force user use menu to switch input mode like Samsung does.

• Fewer clicks to get the intended candidate• Allow user to choose number index with short press

Confidential

3/152 83-LXE 110 0261 Uen Rev PA4 2008-04-22 34

Feedback

• Provide in-time visual feedback when typing

• See exactly what the user is typing, no lag of word and phrase input

• Error alert for wrong typing is good (Sony Ericsson has it))

• Provide clear indicator

• Clear indicator on the keypad• “# 中” is a clear indicator on the keys• No toggling indicator on Sony Ericsson’s keypad

• Clear indicator of current input mode on the screen• Pinyin, 拼音，笔画， 123 ，

Confidential

3/152 83-LXE 110 0261 Uen Rev PA4 2008-04-22 35

Help

• Clear indicator or hint to guide user to get started if it’s not consistent with normal design.• E.g. “*” as an indicator for choosing candidate

Pinyin is perceived good.• Samsung has no indicator for user to choose

candidates by pressing “1” key, which is quite hidden for novice users.

• SonyEricsson didn’t has any indicator for users to long press related number to choose the corresponding candidate. (similar with toggling input mode on hash key)

• More advanced features could be introduced in help information.• No user actually used long press to enter

number in Chinese text input mode.

Confidential

3/152 83-LXE 110 0261 Uen Rev PA4 2008-04-22 36

Recap

Confidential

3/152 83-LXE 110 0261 Uen Rev PA4 2008-04-22 37

Why Chinese text input is so important?

• Statistical Significance between ZTA4, Nokia and SonyEricsson.• Users are willing to send more me

ssages via ZTA4 and Nokia phones, and significant less willingness to send SMS via Sony Ericsson.

• Important to operators to increase revenue.

• Important to Chinese market to increase market share.

“I think I would like to use this input method to send SMS frequently.”

- SUS item 1 analysis

Confidential

3/152 83-LXE 110 0261 Uen Rev PA4 2008-04-22 38

The rode to be No.1 in Chinese text input - 3 Most Important:

• Quick System Response • No lag of showing Pinyin• Support phrase input very well, either for phrase input or s

entence based input

• Significant UI• Consistence in key’s functionalities • Consistency with user’s former experience

• Advanced Features• Phrase input, including non-dictionary phrase input• Fast frequency adjustment

Confidential

3/152 83-LXE 110 0261 Uen Rev PA4 2008-04-22 39

THANKS

Please direct all your questions to John, Chen john.chen@sonyericsson.com