Embed Size (px)
DESCRIPTION
Helen Shen and Chen-hui Tsai The University of Iowa. Evaluating Reading Proficiency Gain and Program Effectiveness: A Three-tiered Web-Based Reading Assessment Approach. Introduction. - PowerPoint PPT Presentation
Citation preview
Helen Shen and Chen-hui TsaiHelen Shen and Chen-hui TsaiThe University of IowaThe University of Iowa
Evaluating Reading Proficiency Gain and Program Effectiveness: A Three-tiered Web-Based Reading Assessment Approach
1
IntroductionIntroduction This presentation reports the evaluation
system for “The comprehensive web-based Chinese Reading Program” supported by a DOE grant.
• The program • The evaluation
2
The Program: PurposeThe Program: Purpose• To remedy the acute dearth of appropriate
Chinese reading materials (beyond the class textbook)
• To furnish a superior Chinese reading environment in which reading can take place anytime, anywhere
• To improve users’ vocabulary knowledge, reading skill, and cultural knowledge through extensive reading
3
The Program: The Targeted The Program: The Targeted UsersUsers• College students who are taking
Chinese courses at beginning, intermediate, and advanced levels
• High-school students who are taking Chinese AP courses
• Individuals who work in government agencies or private organizations and who wish to improve their reading skills in a systematic way
4
The Program: Content The Program: Content
• Three levels of reading materials (900 Lessons)
• 90 achievement tests• 15 proficiency tests
5
The Program: Format • Flash-embedded HTML files• Image files
• NJStar files
• Audio HTML files
6
The Program: Academic The Program: Academic strengthstrength• Proficiency-guided definable reading levels• Cultural theme-based reading unit
organization• Fluency-oriented progressive reading skill
building• Emphasizing Intentional vocabulary gain• Supporting Constructive responses• Three-tiered web-based assessment
system
7
The Program: The DesignThe Program: The Design
8
The Evaluation The Evaluation A Three-tiered web-based assessment system
is developed to assess both individual reading achievement and the overall program effectiveness
• The flash-based built-in reading outcome assessment for individual users
• The FileMaker Pro reading database• The Quia web survey tool
9
The Flash-based Reading Outcome The Flash-based Reading Outcome Assessment Assessment for Individual Reading Proficiency Gain for Individual Reading Proficiency Gain • Vocabulary pre- and post tests
• Multiple choice and open-ended Reading comprehension questions
• Reading speed and comprehension rate (REI) report
• Achievement tests
• Proficiency tests
10
The FileMaker Pro The FileMaker Pro DatabaseDatabase This reading database allows us to record:• User’ contact information• User’s reading activities (reading time,
reading comprehension score, reading lessons completed, vocabulary tests score.)
• Built-in on-line survey
11
The Evaluation: Quia Web The Evaluation: Quia Web SurveySurvey
• The Quia survey data allow us to collect users’ additional quantitative and qualitative feedback on certain arising issues during the use of the program.http://www.quia.com/sv/246903.html
12
The Evaluation: Data The Evaluation: Data Analysis Analysis Data from FileMaker Pro • Distribution of Users in the world• Distribution of Reading levels and topics
among users
13
The Evaluation: Data The Evaluation: Data Analysis Analysis
14
Africa America Asia Europe Oceania
South Africa 8
USA1171
China204
United Kingdom84
Australia78
Argentina4
Canada77
Thailand48
Germany46
New Zealand11
Algeria2
Brazil25
Singapore33
France42
Fiji1
Distribution of Users in the world (78 countries; 1771 valid entries)
The Evaluation: Data The Evaluation: Data Analysis Analysis
15
Reading Level L1 Beginning L2 Intermediate L3 Advanced
Frequency of Read lessons
1712 840 189
Popular units U1 / 720打招呼和交往礼节 U11/ 493中国风俗 U 10/ 125中国各民族U5/ 266做饭和吃饭 U1/ 116中国家庭 U20/ 14中国法制和司法U13/ 114旅游 U3/ 42中国音乐和音乐家 U1/ 12中国现代城乡生活
Units not read U8/18/22 U6/8/9/16-30 U2-9/12-16/18/24/26/28/30
Distribution of Reading levels and topics among US users (2741)
The Evaluation: Data The Evaluation: Data analysis analysis Data from The Quia web survey: One
month reading activity and survey (February 2009)
• Distribution of users and age groups in the world
• Qualitative assessment of program effectiveness
• Quantitative assessment of program effectiveness
16
The Evaluation: Data The Evaluation: Data analysis analysis
17
Africa America Asia Europe OceaniaNationality 1 37 35 11 3
Age Range
Age group ranking
21-30 Under 20-above 50
21-30Under 2031-40
Under 20 –above 50
21-3031-40Under 20
Under 20-above 5021-30
Under 20-40
Distribution of users’ nationality and age groups (valid entries: 87), Feb. 09
The Evaluation: The Evaluation: Qualitative Qualitative Qualitative data
18
Purpose Of helpfulness The part like most Problem SuggestionTeacherUse
Improve reading skills
Articles in various topics
The sound file does not work for me
Include pinyin in the reading text
Student use
Improve listening skills
Articles are interesting; very close to real life
more “beginning” level topics
Self -study
Expand vocabulary Audio files Make sound files downloadable
Pleasure reading
Improve reading speed
Njstar files
Improve cultural knowledge
The interface is appealing and easy to use
The Evaluation: The Evaluation: Quantitative Quantitative
19
Questions Average 12 The reading topics presented are interesting and useful for me to learn Chinese
culture. 5.40
13 The structures of the reading materials are clear.
5.80
14 The difficulty level of the reading materials is arranged in a progressive order. 5.43 15 The amount of reading comprehension questions for each reading lesson are
suitable. 5.54
16 The pre- and post-reading vocabulary tests are helpful in vocabulary building. 5.66 17 I am making progress in reading Chinese by using this website. 5.37 18 The pictures presented in the reading text are clear and helpful. 5.34 19 The achievement tests are useful in helping me reviewing reading units. 5.29 20 The proficiency tests are useful in helping me find my reading strength and
weakness. 5.34
21 The sound quality of the audio file is good and I can hear the lesson without difficulty.
5.17
22 The 'help desk' is helpful for me to learn how to use this website. 5.09 23 The website is easy to navigate and imagines are clear.
5.77
Asia: from a 7-point scale ; 35 users
images are clear.
The Evaluation: The Evaluation: Quantitative Quantitative
20
Questions Average 12 The reading topics presented are interesting and useful for me to learn Chinese
culture. 5.59
13 The structures of the reading materials are clear.
5.95
14 The difficulty level of the reading materials is arranged in a progressive order. 5.49
15 The amount of reading comprehension questions for each reading lesson are suitable.
5.70
16 The pre- and post-reading vocabulary tests are helpful in vocabulary building. 5.73 17 I am making progress in reading Chinese by using this website. 5.51 18 The pictures presented in the reading text are clear and helpful. 5.08 19 The achievement tests are useful in helping me reviewing reading units. 5.32 20 The proficiency tests are useful in helping me find my reading strength and
weakness. 5.41
21 The sound quality of the audio file is good and I can hear the lesson without difficulty.
5.30
22 The 'help desk' is helpful for me to learn how to use this website. 5.49 23 The website is easy to navigate and imagines are clear.
5.97
images are clear.
America: From a 7-point scale; 37 users
The Evaluation: The Evaluation: Quantitative Quantitative
21
Population Item number
M SD t Sig.
AsiaUser
12 5.43 .22 -1.13 .272
USAUser
12 5.54 .26
T-test on the comparison of the ratings between Asia and US
Conclusion:Conclusion: The three-tiered web-based assessment
system is an excellent approach conducting the formative assessment for the program, which helps us to know users needs, expectations, and problems they are encountering so that we can adjust the program based on users’ direct and indirect feedback. This system contributes to maintain a long-lasting high quality program.
22
The reading program is free to all The reading program is free to all users!!!users!!!
http://chinesereadingworld.orghttp://collections.uiowa.edu/chinese
Contact information: Contact information: [email protected]@[email protected]@uiowa.edu
23
The EndThe End
HTML fileHTML fileBack
NJStarNJStarBack
Audio filesAudio filesBack
FileMakerFileMakerBack
PretestPretestGo to Post-test
PosttestPosttestBack
multiplemultipleGo to Open Q
Open-ended QuestionsOpen-ended QuestionsBack
Final ReportFinal ReportBack
Achievement TestAchievement Test
Back
Proficiency TestProficiency Test
Back
FileMakerFileMakerBack
Built-in SurveyBuilt-in SurveyBack
Quia SurveyQuia Survey
![Coarse to Fine Grained Sense Disambiguation in Wikipedia Hui Shen [Ohio University] Razvan Bunescu [Ohio University] Rada Mihalcea [University of North](https://img.pdfslide.net/doc/110x75/56649ddc5503460f94ad44ad/coarse-to-fine-grained-sense-disambiguation-in-wikipedia-hui-shen-ohio-university.jpg)
