Helen Shen and Chen-hui Tsai The University of Iowa Helen Shen and Chen-hui Tsai The University of Iowa Evaluating Reading Proficiency Gain and Program

Helen Shen and Chen-hui TsaiHelen Shen and Chen-hui TsaiThe University of IowaThe University of Iowa

Helen Shen and Chen-hui TsaiHelen Shen and Chen-hui TsaiThe University of IowaThe University of Iowa

Evaluating Reading Proficiency Gain and Program Effectiveness: A Three-tiered Web-Based Reading Assessment Approach

1

IntroductionIntroduction

This presentation reports the evaluation system for “The comprehensive web-based Chinese Reading Program” supported by a DOE grant.

• The program • The evaluation

2

The Program: PurposeThe Program: Purpose

• To remedy the acute dearth of appropriate Chinese reading materials (beyond the class textbook)

• To furnish a superior Chinese reading environment in which reading can take place anytime, anywhere

• To improve users’ vocabulary knowledge, reading skill, and cultural knowledge through extensive reading

3

The Program: The Targeted The Program: The Targeted UsersUsers• College students who are taking

Chinese courses at beginning, intermediate, and advanced levels

• High-school students who are taking Chinese AP courses

• Individuals who work in government agencies or private organizations and who wish to improve their reading skills in a systematic way

4

The Program: Content The Program: Content

• Three levels of reading materials (900 Lessons)

• 90 achievement tests• 15 proficiency tests

5

The Program: Format

• Flash-embedded HTML files• Image files

• NJStar files

• Audio HTML files

6

The Program: Academic The Program: Academic strengthstrength• Proficiency-guided definable reading levels• Cultural theme-based reading unit

organization• Fluency-oriented progressive reading skill

building• Emphasizing Intentional vocabulary gain• Supporting Constructive responses• Three-tiered web-based assessment

system

7

The Program: The DesignThe Program: The Design

8

The Evaluation The Evaluation

A Three-tiered web-based assessment system is developed to assess both individual reading achievement and the overall program effectiveness

• The flash-based built-in reading outcome assessment for individual users

• The FileMaker Pro reading database• The Quia web survey tool

9

The Flash-based Reading Outcome The Flash-based Reading Outcome Assessment Assessment for Individual Reading Proficiency Gain for Individual Reading Proficiency Gain • Vocabulary pre- and post tests

• Multiple choice and open-ended Reading comprehension questions

• Reading speed and comprehension rate (REI) report

• Achievement tests

• Proficiency tests

10

The FileMaker Pro The FileMaker Pro DatabaseDatabase This reading database allows us to record:• User’ contact information• User’s reading activities (reading time,

reading comprehension score, reading lessons completed, vocabulary tests score.)

• Built-in on-line survey

11

The Evaluation: Quia Web The Evaluation: Quia Web SurveySurvey

• The Quia survey data allow us to collect users’ additional quantitative and qualitative feedback on certain arising issues during the use of the program.http://www.quia.com/sv/246903.html

12

The Evaluation: Data The Evaluation: Data Analysis Analysis Data from FileMaker Pro • Distribution of Users in the world• Distribution of Reading levels and topics

among users

13

The Evaluation: Data The Evaluation: Data Analysis Analysis

14

Africa America Asia Europe Oceania

South Africa 8

USA1171

China204

United Kingdom84

Australia78

Argentina4

Canada77

Thailand48

Germany46

New Zealand11

Algeria2

Brazil25

Singapore33

France42

Fiji1

Distribution of Users in the world (78 countries; 1771 valid entries)

The Evaluation: Data The Evaluation: Data Analysis Analysis

15

Reading Level L1 Beginning L2 Intermediate L3 Advanced

Frequency of Read lessons

1712 840 189

Popular units U1 / 720打招呼和交往礼节

U11/ 493中国风俗

U 10/ 125中国各民族

U5/ 266做饭和吃饭

U1/ 116中国家庭

U20/ 14中国法制和司法

U13/ 114旅游

U3/ 42中国音乐和音乐家

U1/ 12中国现代城乡生活

Units not read U8/18/22 U6/8/9/16-30 U2-9/12-16/18/24/26/28/30

Distribution of Reading levels and topics among US users (2741)

The Evaluation: Data The Evaluation: Data analysis analysis Data from The Quia web survey: One

month reading activity and survey (February 2009)

• Distribution of users and age groups in the world

• Qualitative assessment of program effectiveness

• Quantitative assessment of program effectiveness

16

The Evaluation: Data The Evaluation: Data analysis analysis

17

Africa America Asia Europe Oceania

Nationality 1 37 35 11 3

Age Range

Age group ranking

21-30 Under 20-above 50

21-30Under 2031-40

Under 20 –above 50

21-3031-40Under 20

Under 20-above 5021-30

Under 20-40

Distribution of users’ nationality and age groups (valid entries: 87), Feb. 09

The Evaluation: The Evaluation: Qualitative Qualitative Qualitative data

18

Purpose Of helpfulness The part like most Problem Suggestion

TeacherUse

Improve reading skills

Articles in various topics

The sound file does not work for me

Include pinyin in the reading text

Student use

Improve listening skills

Articles are interesting; very close to real life

more “beginning” level topics

Self -study

Expand vocabulary Audio files Make sound files downloadable

Pleasure reading

Improve reading speed

Njstar files

Improve cultural knowledge

The interface is appealing and easy to use

The Evaluation: The Evaluation: Quantitative Quantitative

19

Questions Average 12 The reading topics presented are interesting and useful for me to learn Chinese

culture. 5.40

13 The structures of the reading materials are clear.

5.80

14 The difficulty level of the reading materials is arranged in a progressive order. 5.43 15 The amount of reading comprehension questions for each reading lesson are

suitable. 5.54

16 The pre- and post-reading vocabulary tests are helpful in vocabulary building. 5.66 17 I am making progress in reading Chinese by using this website. 5.37 18 The pictures presented in the reading text are clear and helpful. 5.34 19 The achievement tests are useful in helping me reviewing reading units. 5.29 20 The proficiency tests are useful in helping me find my reading strength and

weakness. 5.34

21 The sound quality of the audio file is good and I can hear the lesson without difficulty.

5.17

22 The 'help desk' is helpful for me to learn how to use this website. 5.09 23 The website is easy to navigate and imagines are clear.

5.77

Asia: from a 7-point scale ; 35 users

images are clear.


20

Questions Average 12 The reading topics presented are interesting and useful for me to learn Chinese

culture. 5.59

13 The structures of the reading materials are clear.

5.95

14 The difficulty level of the reading materials is arranged in a progressive order. 5.49

15 The amount of reading comprehension questions for each reading lesson are suitable.

5.70

16 The pre- and post-reading vocabulary tests are helpful in vocabulary building. 5.73 17 I am making progress in reading Chinese by using this website. 5.51 18 The pictures presented in the reading text are clear and helpful. 5.08 19 The achievement tests are useful in helping me reviewing reading units. 5.32 20 The proficiency tests are useful in helping me find my reading strength and

weakness. 5.41

21 The sound quality of the audio file is good and I can hear the lesson without difficulty.

5.30

22 The 'help desk' is helpful for me to learn how to use this website. 5.49 23 The website is easy to navigate and imagines are clear.

5.97

images are clear.

America: From a 7-point scale; 37 users


21

Population Item number

M SD t Sig.

AsiaUser

12 5.43 .22 -1.13 .272

USAUser

12 5.54 .26

T-test on the comparison of the ratings between Asia and US

Conclusion:Conclusion:

The three-tiered web-based assessment system is an excellent approach conducting the formative assessment for the program, which helps us to know users needs, expectations, and problems they are encountering so that we can adjust the program based on users’ direct and indirect feedback. This system contributes to maintain a long-lasting high quality program.

22

The reading program is free to all The reading program is free to all users!!!users!!!

http://chinesereadingworld.org

http://collections.uiowa.edu/chinese

Contact information: Contact information:

[email protected]@uiowa.edu

[email protected]@uiowa.edu

23

The EndThe End

HTML fileHTML fileBack

NJStarNJStarBack

Audio filesAudio filesBack

FileMakerFileMakerBack

PretestPretestGo to Post-test

PosttestPosttestBack

multiplemultipleGo to Open Q

Open-ended QuestionsOpen-ended QuestionsBack

Final ReportFinal ReportBack

Achievement TestAchievement Test

Back

Proficiency TestProficiency Test

Back

FileMakerFileMakerBack

Built-in SurveyBuilt-in SurveyBack

Quia SurveyQuia Survey

Documents

Helen Shen and Chen-hui Tsai The University of Iowa Helen Shen and Chen-hui Tsai The University of Iowa Evaluating Reading Proficiency Gain and Program