Critical Review: Chun, C. W. (2006). An Analysis of a Language Test for Employment: The Authenticity of the PhonePass Test

Tianfei (Faye) Jiang & Peipei (Christina) DongCTL 3799 Communication and L2 Learning in WorkplaceProf. Julie KerekesFebruary 11, 2014

OUTLINE

• Background & Context• Characteristics of the PhonePass test– Procedure– Setting

• Frameworks• Analysis of the PhonePass test• Conclusion• Critique & Implications• Questions & Comments

BACKGROUND

• Global technology has quickly progressed to provide more accessible business modes of telecommunication, like telephone, video phone, emails and other internet/computer-mediated system in workplace.

• Tests over telecommunication networks are usually highly commercialized..

CONTEXT

PhonePass Spoken English Test-10 (SET-10)• WHAT IS IT? An expedient assessment tool for screening job candidates’ ability

in spoken English• WHAT IS IT FOR? Evaluating employees for promotions and measuring service

employees with an accurate and objective test of their spoken English spoken capabilities.

• HOW DOES IT WORK? This test is a 10 minutes spoken English test for non-native English speakers, which is administered over a language-line telephone and immediately scored by computer. It consists of 5 sections, based on four diagnostic subscores which includes sentence mastery, vocabulary, fluency, and pronunciation. The cost ranges from $25 to $40.

• WHEN & WHERE IS IT? Anytime and Anywhere with a land-line phone system available.

• HOW IS IT SCORED? The first four sections are recorded and scored by computers, while the last section is not scored, which is open for review.

CONTEXT

Purpose:to see if the characteristics of the

domain of the test tasks found in the PhonePass SET 10 test correspond to real life domains of nontest tasks.

• Key Words: telecommunication; spoken language

test; authenticity;

CHARATERISTICS IN PHONEPASS TEST

Access to an

Internet-connected Computer

Download and Print out a Test

Paper

Access to a Land-line

Telephone

Follow Further

Instruction in the Phone

Testing

• PROCEDURE (requires employee’s computer technology & telephone technology capabilities)

CHARATERISTICS OF PHONEPASS TEST• SETTING

• “Please read No. 3.”• “The winter in Toronto is freezing.”

A: Reading Aloud

• “War broke out.”• “There are three basic ways in which a story

might be told to someone.”

B: Repeating Sentence

• “What season comes before spring?”• “Does a blanket go on the bed or the wall?”

C: Giving Short Answers to Question

• “in/ bed/ stay”• “we wondered/ would fit in here/ whether the

new piano”

D: Building Sentences

• “Which would you prefer to live, in the big city or a small town? And why?”

E: Responding to Open Questions

CHARATERISTICS OF PHONEPASS TEST

• Any location & time • High degree of speed• 10 minutes

• visual and aural channels

• Grammatical knowledge

• Textual cohesion

• Sentence mastery• Vocabulary• Fluency• Pronunciation• No reciprocal tasks

Flexibility & Short Length Input

ExpectationsTasks

FRAMEWORKAuthenticity• “the degree of correspondence of the characteristics of a given

language test task to the features of a target language use task.” (Bachman & Palmer, 1996, p.23)

• “the developers need to show that the content of the test tasks is representative of the demands of the corresponding task outside the test situation, and that the scoring reflects this.” (Luoma, 2004)

• “authenticity of input, purpose, and outcome” (Lewkowicz, 2000)

Test Fairness Framework “access, administration, validity, absence of bias, and social

consequence.” (Kunnan, 2004)

ANALYSIS: Authenticity

Rationale PhonePass Test (PART A)

Created real-life discourse Structured speaking task

Grammatical Knowledge Without correct sentence production and comprehension

Textual Knowledge Read in random order of items

Functional and Sociolinguistic Knowledge

Standard pronunciation

ANALYSIS: Authenticity

Rationale PhonePass Test (PART B)

Proof of comprehension Sounds reorganization and repetition & Memory

Speech production Without discourse creation and interpretation

ANALYSIS: Authenticity

Rationale PhonePass Test (PART C)

Vocabulary of professional employment level

low level of lexical knowledge

Extended production responses Limited production responses

Sociolinguistic knowledge N/A

ANALYSIS: Authenticity

Rationale PhonePass Test (PART D)

Fluency (constructing, reading and repeating the sentences)

Inadequate constructing

Pragmatic knowledge N/A

Sentence mastery One in six chance of guessing word groups

ANALYSIS: Authenticity

Rationale PhonePass Test (PART E)

Authenticity Extended production responsesNot scored

ANALYSIS: Fairness

Rationale PhonePass Test (PART A)Minimal bias No biased content or language

Educational access (speaking & listening)

Advantages for test takers who have had more opportunities to learn and become familiar with the tasks on the test

Telecommunication access (land-line telephone, printer & internet connected computer)

The absence of supporting telecommunication in underdeveloped countries

Appropriate accommodations Uncertain physical (e.g. sight and hearing) impairment

Rescoring and reevaluation N/A

CONCLUSIONS

• The performance on these tasks does not necessarily predict or reflect the speaking ability of the test taker to function in real-life domain of workplace.

• PhonePass SET 10 is considered relatively less authentic.

• The test tasks are relatively less specific to a particular domain and are not interesting and engaging.

CRITIQUE

Strengths: The author used strong theoretical basis to support his

analysis. It provides a valuable evaluation as a reference, emphasizing

the importance of authenticity, to language assessment in workplace.

Weakness: This article does not offer further practical implication in

telecommunication in workplace. It would be better if the author provides more constructive

suggestions to the phone language test.

IMPLICATIONS

• Since telecommunication lacks non-verbal compensation, such as body languages and facial gestures, the requirement of language is much higher than daily face-to-face communication.

• This analysis brings readers to have more deep provoking thoughts about telecommunication networks, for example, how can we communicate effectively without these compensation, and how can we minimize the misunderstandings in telecommunication.

• For employers, phone interview or language assessment via telephone should focus more on the authentic communicative competence than on linguistic performance.

• Communication effectiveness is a critically important criteria in assessing individual's language ability in workplace. Standards for communicative language capability in workplace is needed to be further explored.

Questions AND/OR Comments?

----- Tianfei Jiang & Peipei DongFebruary 11, 2014