Joan Jamieson’s Putting Principles

into Practice an article summarized by:

SEFUL BAHRI16705261003

Yogyakata State University student

This article is written by Joan Jamieson, Carol A Chappele, and Sherry Preiss

Purpose of the studyTo illustrate how a set of principles informed a series of processes in the evaluation of Longman English Online, a commercial CALL product for teaching intermediate English as a second or foreign language (ESL/EFL).

CALL Product to Evaluate: Longman English Online 3 (LEO 3). it’s the

intermediate level of the video-based multimedia, integrated language skills program.

The units not only contain listening comprehension practice, but also include practice with grammar, vocabulary, pronunciation, speaking, reading, research on the web, and writing.

Quizz in every unit, longer tests at the end of every four units (“module test”), and one at the end of all twelve units (“the end –of-level test).

Nature of the evaluation : The evaluation was focused on the materials

rather than on the tasks that the leaners actually carried out as they were using the materials

It was judgmental evaluation (: based on the logical analysis of a CALL activity), NOT empirical evaluation (: based on the quantitative or qualitative analysis of a CALL activity through observed data which are summarized by the evaluator)

CALL Evaluation Principles and their associated criteria & vriables

Principles Criteria Variables Examples

1. Language Learning potential should be the central concern when developing & evaluating

* to indicate CALL quality. (Chapele. 2001)

Enhanced input

Input saliece, modification, elaboration

Text highlighted on the screen

interaction Between people, between a person & computer, within a person

Learner-learner communication tasks, grammar help available

production Planning, correcting production, using help

Taking notes suggested, model for constructed responses

Principle Criteria Variables Examples of Operationalized variables

2. CALL should be evaluated in terms of the quality assessment in addition to instruction

*to indicate the usefulness of CALL test. (Bachmer & Palmer)

Authenticity Domain Sampling

Correspondence between tasks on the test and in courseware

Construct Validity

Score interpretation

Meaningfulness of scores

Principle Criteria Variables Examples of Operationalized variables

3. The criteria to conduct the analysis should be drawn from theory and practice in SLA & Lang. Assessment4. Judgemental evaluation through defined variables is valable for examining materials in a way that can improve future design

All of the previously mentioned criteria were examined, to find whether or not they had been operationalized in LEO 3

These criteria were investigated as they have a relatively strong theoretical, pedagogical, and empirical base:

Theory and research on SLA converge to suggest that the likelihood of learners acquiring linguistic input increases if their attention is drawn to linguistic features, using techniques such as marking salience, modification, or elaboration. (Schmidt, 1990; Sharwood Smith, 1993; Robinson, 1995; Skehan, 1998).

Benefits and types of interaction for langugage learning (Ellis, 1999; Kol & Schcolnik, 2000; Lantolf & Appel, 1994, Long, 1996, Pica, 1994)

Swain (1985): From the cognitive perspective, the benefits of producing language may be enhanced when learners have the opportunity to plan before speaking or writing and to correct linguistic output, which can be prompted by feedback from others or from self-evaluation.

From sociocognitive perspective, Swain (1998) stated that the help from the interlocutor allows for production beyond what the learner can accomplish alone.

Authenticity is explained as the degree of correspondence between the characteristics of a test-task and the features of a task in “target language use” (Bachman & Palmer, 1996).

Achievement test is considered authentic in terms of the degree to which its tasks & directions mirror the content & skills in an insructional unit (:domain sampling)

Construct Validity dals with the appropriateness and meaningfulness of the interpretations which are made on the basis of a test score.

Summary of Judgemental Evaluation of LEO 3

Area Criteria Variables Operationaizations

Evidence Judgement

Language Larning Potential

Enanced input

salience Font size/type/color, highlighting, repetition

Font size/highlighting graphics, animation, audio repetition

Good

modification Simplification, repetition, clarification, L1 transl., L2 definitions, image/video

Images/video, transcripts, L2 definitions

Good

elaboration Adding grammatical phrases or clauses to texts

Adding grammatical phrases

Fair

Language Learning Potential

Interaction

between people

Amount/types of co-constructed and attention to form

Meaningful tasks

Good

People and computer

Potential of providing enhanced input, obtaining help using the language

Program correction of learner’s explanations, culture notes

Good

an inner voice

choices  

Language Learning Potential

Production

planning Instructions to plan, directed activities, knowledge of expectations

Directions to plan, pre-writing exercises

Good

correcting Amount of self-correction, amount/type of signals from computer or another person

Self-correction of speaking/ pronunciation teacher

Fair

help Accessing dictionary, grammar, models

Glossaries, transcripts, grammar help, models of speaking and writing

Good

CALL assessmentAthenticityConstruct validity

domain sampling

Correspondence between tasks on tet & in courseware

Table of specifications, directions, texts, item types

Fair

score interpretation

Appropriate, meaningful

Reporting scheme

Good

CONCLUSIONbased on judgments on  the criteria of

language learning potential as well as authenticity and construct validity of assessment, LEO 3 received a good evaluation.

This evaluation also points to areas in which LEO 3 might be improved in the future and it suggests that LEO 3 might be worth pursuing further through the addition of an empirical evaluation.

=== end of the slides ===