An Investigation of Cognitive Operations on L2 Listening Comprehension Performance

Speaker: Dr. Hui-Fang Shang ( 尚惠芳博士 )

Outline:

I. IntroductionII. Literature ReviewIII. Purpose of StudyIV. MethodologyV. ResultsVI. Discussion and Implications

1

I. Introduction

* Hard to evaluate EFL learners’ listening comprehension performance due to lack of a valid listening test

* To develop a valid listening comprehension test

Need to understand the listening processes and

how listening functions

* Hard to evaluate EFL learners’ listening comprehension performance due to lack of a valid listening test

* To develop a valid listening comprehension test

Need to understand the listening processes and

how listening functions

II. Literature Review

* Listening comprehension processes:

-- Retain information in the memory: Deposit raw speech in short-term memory; predict information; recall background information to interpret message; hold information in long- term memory (Duzer, 1997)

-- Interact between background knowledge and text: Bottom- up and top-down processing: (Duzer, 1997)

-- Use local (links between clauses & sentences) and global (overall coherence) strategies (Van Dijk & Kintsch, 1983)

-- Use trivial (numerical details), local, and global strategies (Shohamy, 1991)

2

3

* How L2 listeners deal with text processing:

-- Efficient listeners use background knowledge; low listeners use mostly local details (Hildyard & Olson, 1982)

-- Efficient L2 listeners activate more L1 knowledge in the form of concept-driven schemata (Wolff, 1987)

-- Effective L2 listeners use both top-down and bottom-up strategies; ineffective listeners determine the meanings of individual words (O’Malley, Chamot, & Kupper, 1989)

* How L2 listeners deal with text processing:

-- Efficient listeners use background knowledge; low listeners use mostly local details (Hildyard & Olson, 1982)

-- Efficient L2 listeners activate more L1 knowledge in the form of concept-driven schemata (Wolff, 1987)

-- Effective L2 listeners use both top-down and bottom-up strategies; ineffective listeners determine the meanings of individual words (O’Malley, Chamot, & Kupper, 1989)

II. Literature Review (cont.)

4

II. Literature Review (cont.)

-- Low listeners perform better on local items than on global items (Shohamy, 1991)

-- Low listeners do not use appropriate content schemata (Jensen & Hansen, 1995)

-- 6 listening functions to comprehend a message: Identification, orientation, main idea comprehension, detail comprehension, full comprehension, and replication

(Lund’s Taxonomy, 1990)

-- Low listeners perform better on local items than on global items (Shohamy, 1991)

-- Low listeners do not use appropriate content schemata (Jensen & Hansen, 1995)

-- 6 listening functions to comprehend a message: Identification, orientation, main idea comprehension, detail comprehension, full comprehension, and replication

(Lund’s Taxonomy, 1990)

-- 4 similar categories to comprehend a text: (Dunkel, Henning, & Chaudron, 1993)

◎ Orientation meanings: Understand persons & their relationships ◎Detail meanings: Simple lexical meanings ◎Main ideas: Major points ◎ Implications: Applications of background knowledge

-- Those 4 categories of understanding the text meaning: “Cognitive tasks”

-- The ability to “identify” and “interpret” the cognitive tasks: “Cognitive operations”

II. Literature Review (cont.)

5

6

III. Purpose of Study

* Use Dunkel, Henning, & Chaudron’s (1993) theory

of cognitive operations to measure EFL listeners’

comprehension competence

* Select 3 components of cognitive operations:

(1) Interpreting main ideas (local)

(2) Identifying details (trivial)

(3) Interpreting implications (global)

* Use Dunkel, Henning, & Chaudron’s (1993) theory

of cognitive operations to measure EFL listeners’

comprehension competence

* Select 3 components of cognitive operations:

(1) Interpreting main ideas (local)

(2) Identifying details (trivial)

(3) Interpreting implications (global)

III. Purpose of Study (cont.)

* Explore 3 main research questions:

(1) What is the most difficult cognitive operation in terms of those 3 components of cognitive operations?

(2) For different proficiency listeners, what is their listening performance?

(3) How are listeners’ perceptions regarding their listening performance consistent with the quantitative research results?

7

IV. Methodology

8

* Subjects: -- Include 63 sophomores of Applied English Department at I-Shou University

-- Divide subjects into 3 proficiency groups: Low, intermediate, high groups based on the scores of simulated TOEFL listening test (from 6 to 13 out of 13, Mean=10.48, Median=10)

* Subjects: -- Include 63 sophomores of Applied English Department at I-Shou University

-- Divide subjects into 3 proficiency groups: Low, intermediate, high groups based on the scores of simulated TOEFL listening test (from 6 to 13 out of 13, Mean=10.48, Median=10)

Table 1 Number of Subjects and Scores of Each Proficiency Group

Low Intermediate High

Scores 6~9 10 11~13

Number(% of total sample)

13(21%)

20(32%)

30(47%)

Total 63

IV. Methodology (cont.)

* Materials:

-- Select 3 extended conversations with the similar topic of campus events from the textbook of “The Heinle & Heinle TOEFL Test Assistant: Listening (Broukal, 1995) -- Choose 3 listening texts containing: ◎ Paraphrasing : Understand main ideas (local strategy) ◎ Listening for details: Answer detail information (trivial strategy) ◎Making inferences: Draw conclusions (global strategy)

9

10

* Procedures: Construct 12 items in total and listen to the texts twice

* Measurement Instruments & data analyses:

-- Use 2 scoring methods: binary (correct/incorrect) of item scores and rating scale (a self-perceived survey)

-- Investigate mean differences among 3 cognitive operations on 3 proficiency groups by computing a one-way ANOVA analysis

-- Examine which cognitive operation yields higher listening performance by employing a Post Hoc test

-- Administer a self-rating questionnaire to probe subjects’ perceptions of the difficulty level of each cognitive operation

* Procedures: Construct 12 items in total and listen to the texts twice

* Measurement Instruments & data analyses:

-- Use 2 scoring methods: binary (correct/incorrect) of item scores and rating scale (a self-perceived survey)

-- Investigate mean differences among 3 cognitive operations on 3 proficiency groups by computing a one-way ANOVA analysis

-- Examine which cognitive operation yields higher listening performance by employing a Post Hoc test

-- Administer a self-rating questionnaire to probe subjects’ perceptions of the difficulty level of each cognitive operation

IV. Methodology (cont.)

11

IV. Methodology (cont.)

* 5 hypotheses:

(1) Interpreting implications is the most difficult question type.

(2) High proficiency listeners can successfully use both bottom-up and top-down processing to determine major,

literal, and implied meanings than the other two groups.

(3) The intermediate listeners’ performance regarding those three cognitive operations will be better than the low listeners’.

(4) The low proficiency listeners will perform better on items referring to detail questions than on items referring to main

idea and inference questions.

(5) Listeners’ perceptions will be consistent with the experimental research results.

* 5 hypotheses:

(1) Interpreting implications is the most difficult question type.

(2) High proficiency listeners can successfully use both bottom-up and top-down processing to determine major,

literal, and implied meanings than the other two groups.

(3) The intermediate listeners’ performance regarding those three cognitive operations will be better than the low listeners’.

(4) The low proficiency listeners will perform better on items referring to detail questions than on items referring to main

idea and inference questions.

(5) Listeners’ perceptions will be consistent with the experimental research results.

12

Table 2 Classification of Scores Obtained according to Three Question Types for Three Proficiency Groups

Table 2 Classification of Scores Obtained according to Three Question Types for Three Proficiency Groups

V. Results

Low

Mean (%)

Intermediate

Mean (%)

High

Mean (%)

Main idea (local) 67.3 73.8 80.8

Detail (trivial) 51.3 46.7 63.3

Inference (global) 78.5 70.0 79.3

* Each group performs worst on the detail questions ~ Does not support Hypotheses 1 & 4* High group performs best than the other two groups ~ Support Hypothesis 2* Intermediate group performs worse than low group except main idea questions ~ Broadly not support Hypothesis 3

13

Table 3 A T-Test Analysis between Local (Main Idea) and Global (Inference) Questions among Three Groups

Table 3 A T-Test Analysis between Local (Main Idea) and Global (Inference) Questions among Three Groups

V. Results (cont.)

Mean N SD SE Mean

t df Sig.

Low Main idea Inference

67.378.5

1313

.751

.641.208.178

-4.382 12 .001*

Intermediate Main idea Inference

73.870.0

2020

.686

.946.154.212

-2.463 19 .024 *

High Main idea Inference

80.879.3

3030

.898

.890.164.163

-4.097 29 .000 *

* The means for the main idea questions are higher than the inference ones except for the low group ~ broadly support Hypothesis 1

14

V. Results (cont.)

Table 4 A One-way ANOVA Analysis of Those Three Proficiency Groups

Table 4 A One-way ANOVA Analysis of Those Three Proficiency Groups

Score SS df MS F Sig.

Between Groups

Within Groups

Total

148.265

141.651

289.915

2

186

188

74.132

.762

97.342 .000

* There is a significant difference at the .001 level for those three cognitive operations on those three proficiency groups.

15

Table 5a A Post Hoc Test among Low, Intermediate, and High Groups in Three Cognitive Tasks

Table 5a A Post Hoc Test among Low, Intermediate, and High Groups in Three Cognitive Tasks

V. Results (cont.)

Group N Mean SDMain Idea Low Intermediate High

132030

2.6922.9503.233

.751

.686.898

Detail Low Intermediate High

132030

1.5391.4001.900

.776

.940.923

Inference Low Intermediate High

132030

3.9233.5003.967

.641

.946.890

* The high group outperforms the intermediate and low groups in those 3 cognitive tasks ~ Support Hypothesis 2* The low group outperforms the intermediate one in detail and reference sections ~ Broadly not support Hypothesis 3

16

Table 6a Self-Report of Difficulty Level for Main Idea QuestionsTable 6a Self-Report of Difficulty Level for Main Idea Questions

V. Results (cont.)

Very easy

Easy OK Difficult Very difficult

Low 0 0 8 (66.7%) 3 (25%) 1 (8.3%)

Intermediate 0 0 16 (84.2%) 3 (15.8%) 0

High 2 (4%) 2 (4%) 36 (72%) 10 (20%) 0

* The low group considers the main idea questions as the most difficult ones, followed by the high group, followed by the intermediate group ~ Partially consistent with Table 5a (mean scores: low＜ intermediate＜ high)

17

Table 6b Self-Report of Difficulty Level for Detail Questions Table 6b Self-Report of Difficulty Level for Detail Questions

V. Results (cont.)

Very easy

Easy OK Difficult Very difficult

Low 0 0 7 (53.8%) 5 (38.5%) 1 (7.7%)

Intermediate 0 0 12 (66.7%)

6 (33.3%) 0

High 0 1 (2%)

31 (63.3%)

17 (34.7%)

0

* The low group considers the detail questions as the most difficult ones, followed by the high group, followed by the intermediate group ~ Not consistent with Table 5a (mean scores: intermediate＜ low＜ high)

18

Table 6c Self-Report of Difficulty Level for Inference Questions Table 6c Self-Report of Difficulty Level for Inference Questions

V. Results (cont.)

Very easy

Easy OK Difficult Very difficult

Low 0 2 (15.4%)

6 (46.2%) 4 (30.8%) 1 (7.7%)

Intermediate 0 0 16 (84.2%)

3 (15.8%) 0

High 1 (2%)

3 (6.1%) 32 (65.3%)

13 (26.5%)

0* The low group considers the inference questions as the most difficult ones, followed by the high group, followed by the intermediate group ~ Not consistent with Table 5a (mean scores: intermediate＜ low＜ high)

19

Table 7a Frequency of Three Question Types for Low Proficiency Group

Table 7a Frequency of Three Question Types for Low Proficiency Group

V. Results (cont.)

Very easy

Easy OK Difficult Very difficult

Main idea 0 0 8 (66.7%) 3 (25%) 1 (8.3%)

Detail 0 0 7 (53.8%) 5 (38.5%) 1 (7.7%)

Inference 0 0 7 (53.8%) 5 (38.5%) 1 (7.7%)

* The main idea questions are the least difficult ones. ~ Not consistent with Table 2 (The inference questions are the least difficult ones.)

20

Table 7b Frequency of Three Question Types for Intermediate Proficiency Group

Table 7b Frequency of Three Question Types for Intermediate Proficiency Group

V. Results (cont.)

Very easy

Easy OK Difficult Very difficult

Main idea 0 0 16 (84.2%) 3 (15.8%) 0

Detail 0 0 12 (66.7%) 6 (33.3%) 0

Inference 0 0 12 (66.7%) 6 (33.3%) 0

* The detail and inference questions are (very) difficult for intermediate group. ~ Partly supports Hypothesis 1 that interpreting implications is the most difficult question type.

21

Table 7c Frequency of Three Question Types for High Proficiency Group

Table 7c Frequency of Three Question Types for High Proficiency Group

V. Results (cont.)

Very easy

Easy OK Difficult Very difficult

Main idea 2 (4%) 2 (4%) 36 (72%) 10 (20%) 0

Detail 0 1 (2%) 31 (63.3%)

17 (34.7%)

0

Inference 0 1 (2%) 31 (63.3%)

17 (34.7%)

0

* The detail and inference questions are (very) difficult for high group. ~ Partly supports Hypothesis 1 that interpreting implications is the most difficult question type.

22

VI. Discussion and Implications

1. Each group performs worst on the detail questions (see Table 2) because it’s difficult to memorize all the information from the listening texts.

=> Irrelevant recall of names of numerical data may distract listeners’ attention on the whole comprehension.

=> Since such cognitive operation serves no meaningful purpose as evaluating listening comprehension, it’s recommended to avoid the insignificant numerical details on listening comprehension tests.

23

VI. Discussion and Implications (cont.)

2. Except for low group, subjects perform better on local (main idea) questions than on global (inference) ones.

=> Comprehension of local information is more attainable than that of macro information (Shohamy, 1991).

=> It’s more difficult to employ the concept to generalize, infer, and synthesize the information from the cognitive processing than to get the main idea information only (Van Dijk & Kintsch, 1983).

24

VI. Discussion and Implications (cont.)

3. There is an inconsistency for the low group regarding the performance of global and local questions from the quantitative and qualitative research results.

=> The low listeners may guess the answers mostly since it’s a multiple-choice test.

=> It’s appropriate to use more test types to certainly reflect listening comprehension competence.

25

VI. Discussion and Implications (cont.)

4. The low group outperforms the intermediate group in the detail and inference sections.

=> The listening texts which were used to affect the degree of listening comprehension are relatively difficult to the immediate group.

=> Other factors, such as familiarity with the topic, background knowledge, text density, etc. may affect listeners’ performance.

=> Future research should look at the influence of those factors on the listening comprehension performance.

26

VI. Discussion and Implications (cont.)

5. To result in more construct valid listening

comprehension tests, teachers should carefully

select a variety of testing instrument with various

test types and question types, to better

reflect the trait of L2 listening comprehension.