Upload
hugh-haynes
View
218
Download
0
Tags:
Embed Size (px)
Citation preview
Measuring Student Learning at the Classroom Level: A Longitudinal Study
Aleidine MoellerLeyla MasmaliyevaDallas Malhiwsky
University of Nebraska-Lincoln
Research Focus
A longitudinal study at the classroom level that attempts to more accurately depict students’ acquisition of language proficiency that ultimately could contribute to a more accurate and smooth articulation tool for documenting language progress.
The Need: Clearly Articulated Descriptors
• What students know and are able to do at the end of each year of classroom language study
• Accurate and smooth articulation requires empirically tested
common yardstick at the classroom level that defines language progression from one level to the next in a cohesive manner
Smooth Articulation
• Accurate and smooth articulation requires an empirically tested common yardstick at the classroom level that defines language progression from one level to the next in a cohesive manner.
ACTFLconstructed a chart approximating what students should know and be able to do given specific years of language study, however, this chart does not reflect classroom based research, but rather is based on proficiency interviews with individual language students and “on information gathered from foreign language professionals representing a variety of program models and articulation sequences” (ACTFL Performance Guidelines for K-12 Learners, 1998).
Classroom Based Evidence• There is a dearth of classroom
tested studies that have investigated the level and rate at which language learners achieve proficiency after a sequenced course of language study.
• Classroom-based evidence has been limited to a few studies restricted to one or two classrooms that do not allow for generalizable results.
Subjects
7-12 Teachers of Spanish2 week Spanish Immersion funded by Teacher Quality Grants– Language, literature, culture
pedagogy, technology, LinguaFolio
– Teacher participants implemented LF (120 teachers)
– Selected 17 – Followed teachers & students for
5 years
Context for the Study: LinguaFolio• Modeled after European
Language Portfolio (ELP) of the Council of Europe (COE)
• Adapted by National Council of State Supervisors of Foreign Languages (NCSSFL)
• Promote reflective, autonomous language learning and cultural interactions through a self-assessment tool
• http://linguafolio.unl.edu
LinguaFolio Nebraska• seeks to inform students about
themselves and their own learning, showing them exactly where they are in relation to learning goals
• with the help of their teachers, knowing which steps to take to achieve these goals.
• through systematic self-assessment, goal setting, feedback and reflective learning tools, the LinguaFolio guides students to monitor and map their own learning.
Students must make what they learn part of themselves (Chickering & Gamson, 1987, p. 3)
• Talk about what they are learning, write about it, relate to past experiences, apply in their daily lives
• Too often we see students as subjects of assessment, not users of assessments. The student must be considered the primary user of assessment information.
Promote Self-regulation
• How do we position students to think about their own learning and make connections inside and outside the classroom to monitor their learning development?
Pilot Test: Lessons Learned
10 teachers of Spanish 7-12Tested & revised instruments
teacher observation formteacher beliefs & attitudes
Goals: needed trainingDevelopment of Teachers HandbookOnline LinguaFolio
Why goal setting?
“Goal setting influences learning and motivation by providing a target and information about how well one is doing.”
(Alderman, 2005, pg. 105)
Goal setting → increased self-efficacy→ increased intrinsic motivation
Goal setting:Monitor progressIncreased self-efficacyprovides expectation for
success and increases the likelihood of engaging in future tasks without external incentives
Increased intrinsic motivation
Characteristics of goals
Performance vs Learning based Goals
Performance- focus on comparing oneself to othersLearning- focus on gaining newknowledge and skills
Characteristics of goalsShort vs long term
• Near future versus distant future
• Both needed
Characteristics of goalsEasy vs. difficult
Analysis of 110 studies published from 1969 to 1980 demonstrated that 99 studies showed that difficult learning goals produced better performance than medium, easy, do-best, or no goals at all (Griffee and Templi,1997).
Characteristics of goals
Specific vs. Broad Goals• Students setting too specific goals may
focus their attention to such a degree that they will ignore “information not specifically related to their goal”
• (Marzano et al, 2001, p. 94).
• Broad goals are vague and have a nonspecific outcome
Characteristics of goals
Assigned versus Self-chosen Goals• Assigned goals foster motivation because
they imply that the individual is capable of achieving the goal or task
• Self-chosen- power of ownership• Crucial factor in assigned goals Is acceptance- once involved in a goal, the goal itself becomes more important than how it was set or whether it was imposed
Simply goal setting is enough?
Many educators overestimate their students’ ability to set high quality goals (Boekaerts, 2002).
“A perceived discrepancy between performance and the goal may create dissatisfaction” (Schunk, 2003, p. 163).
Examples of students’ poor goals:
• “speak fluently”
• “understand everything”
• “get an A (grade)”
For goals to enhance student achievement goals should meet specific criteria (Schunk, 2003).
SMART Goals
• SSpecificpecific
• MMeasurableeasurable
• AAgreed upongreed upon
• RRealistic (challenging yet realistic)ealistic (challenging yet realistic)
• TTime-basedime-based
SMART+ER =SMARTER
Return to Your Goals• What evidence do you have that indicates
whether you met your goals?• What could you have done to better meet your
goals?• How could you have restated your goals in
order for them to be more appropriate?• Are you satisfied with your progress today? Did
you put forth your “best effort” to achieve your goals?
Formula for Success:
Setting goals +documenting progress +self-assessing learning=student achievement
Results of Goal Setting on Student Achievement
Correlation between goal setting and student achievement at .01 level of significance in reading, speaking and writing
Correlations
LinguaFolio Goal Sc ore
LinguaFolio Action Plan
Score
LinguaFolio Reflection
Score
STAMP Readi ng
score STAMP
Writing score
STAMP Speaking
Score Pearson Correlation 1 .363(**) .585(**) .329(**) .317(**) .331(**) Sig. (2 -tailed) .000 .000 .000 .000 .000
LinguaFolio Goal Score
N 201 175 201 194 190 193 Pearson Correlation .363(**) 1 .158(*) .218(**) .222(**) .260(**) Sig. (2-tailed) .000 .036 .004 .004 .001
LinguaFolio Action Plan Score
N 175 175 175 169 165 168 Pearson Correlation .585(**) .158(*) 1 .215(**) .176(*) .311(**) Sig. (2 -tailed) .000 .036 .003 .015 .000
LinguaFolio Reflection Score
N 201 175 201 194 190 193
Pearson Correlation .329(**) .218(**) .215(**) 1 .756(**) .661(**) Sig. (2 -tailed) .000 .004 .003 .000 .000
STAMP Reading score
N 194 169 194 529 525 526 Pearson Correlation .317(**) .222(**) .176(*) .756(**) 1 .652(**) Sig. (2 -tailed) .000 .004 .015 .000 .000
STAMP Writing score
N 190 165 190 525 525 522 Pearson Correlation .331(**) .260(**) .311(**) .661(**) .652(**) 1 Sig. (2 -tailed) .000 .001 .000 .000 .000
STAMP Speak ing Score
N 193 168 193 526 522 527
** Correla tion is si gnificant at the 0.01 level (2-tailed). * Correla tion is s ignificant at the 0.05 level (2-tailed).
LinguaFolio Goal Score
LinguaFolio Action Plan Score
LinguaFolio Reflection Score
STAMP Reading score
Pearson Correlation
.315** .292** .253**
N 673 644 672
STAMP Writing score
Pearson Correlation
.436** .507** .256**
N 664 635 663
STAMP Speaking Score
Pearson Correlation
.404** .454** .236**
N 641 617 640
Assessment for Learning
• Students engage in assessment for learning when they use assessment information to learn how to judge the quality of their own work and set goals for their own improvement.
This interactive LF NE website • students log in and establish
base line knowledge and skills from which to build long term goals
• Self-assessment of competency chart: identify what they are able to do in their learning process while also reflecting upon the level of language competency they would like to attain
I can ask for and give directions with the help of a map or other visual aid. I can ask for directions to a place. I can tell someone how to get from one place to another (e.g. go straight, turn left, turn right). I can tell someone where something is located (e.g. next to, across from, in the middle of). I can exchange information with other people about what to do, where to go, and when to meet. I can make simple plans to do something with someone. I can exchange information about where or when to go or meet (e.g. store, movie, concert, restaurant). I can invite someone to do something or go somewhere. I can accept or refuse an invitation to do something or go somewhere.
6. Write
reflection based on work
(evidence) and goals
5. End of
chapter: review goals. Choose
work to represent goal
attainment.
4. Save goals, tasks and all
completed class work in a folder
3. Students determine tasks for achieving chapter goals.
2. Visit Chapter Content:
(Students) write personal goals
based on chapter content.
1. Review
Self-Assessment of Competency Level Rubric
(LF NE!)
Dossier of Evidence Student Driven Cycle:
Once Per Chapter
Measuring Student Learning
• STAMP– Archived student samples– Proficiency based– Accessible – Provided a control group– Measures speaking, writing
and reading
STAMP
• Teacher independent proficiency assessment that archives language skills over time
• Access to all reading, writing, speaking scores and individual student samples over time
STAMP• A summative assessment
tool, measuring student proficiency levels in foreign languages in four skills
• Test items are based on benchmarks linked to ACFTL guidelines
• Securely and efficiently delivered through the internet
• Computer adaptive online assessment
STAMP
Report can be displayed in terms of a class, group or individual
Is criterion-referenced (item evaluation is based on how they perform as students approach and then cross a mastery level)
STAMP• Assessment is made on basis of what
students can actually perform in real-world context
• Provides detailed and user-friendly report that aims to demonstrate that the best measure of educational success is progress in student performance.
• The ultimate purpose is to improve language teaching and learning
• Gives the teachers concrete feedback on students overall proficiency and breakdowns of performance according to topic and task
Individual Report for É ÉÉÉ. 5/1/2007 10:19:27 AM
Interpretive Presentational Reading 3
Topics Needing Further Study:
Community, Holidays/Celebrations, Leisure/Activities, Occupations/Professions, Shopping/Stores, Travel/Vacations, Weather/Seasons
Writing 3
Writing Sample Breakdown:
Sample 1 Does Not Meet Standards
Sample 2 Meets Standards
Sample 3 Meets Standards
Speaking 3
Speaking Sample Breakdown:
Sample 1
Does Not Meet Standards
Sample 2
Meets Standards
Sample 3
Meets Standards
When reading about introductory topics such as:
community daily routines shopping/stores food/beverage
When writing about introductory topics such as:
community daily routines shopping/stores food/beverage
When speaking about introductory topics such as:
community daily routines shopping/stores food/beverage
STAMP Reading
Minimum 1.00
Maximum 4.00
Yr 1 Mean N 274 1.3885
Yr 2 Mean N 683 1.6676
Yr 3 Mean N 204 2.2745
Yr 4 Mean N 17 2.6471
STAMP Writing
Minimum 1.00
Maximum 4.00
N 562 1.4217
N 673 1.9173
N 204 3.0882
N 17 3.2529
STAMP Speaking
Minimum 1.00
Maximum 5.00
N 525 1.3619
N 626 1.7923
N 197 2.8477
N 17 3.2353
Mean Year One Year 2
STAMP Reading score 1.3875 1.7488 2.2400 STAMP Writing score 1.3010 1.6408 2.4600 STAMP Speaking Score 1.1708 1.5670 2.4898 Pronunciation score 1 1.7670 2.0111 2.3061 Pronunciation score 2 1.7903 1.9865 2.3469 Pronunciation score 3 1.7908 1.9859 2.3404 Grammar score 1 1.6878 1.9224 2.3673 Grammar score 2 1.6874 1.9216 2.5102 Grammar score 3 1.6964 1.9181 2.4681 Vocab score 1 1.6222 1.8726 2.3673 Vocab score 2 1.6244 1.8703 2.5102 Vocab score 3 1.6752 1.9153 2.4681 Fluency score 1 1.6433 1.8753 2.3061 Fluency score 2 1.6521 1.8804 2.3673 Fluency score 3 1.6675 1.8955 2.3830
Context of the project
• Doc seminar• Sat. mornings• 10 students (6 Spanish
language experts; 4 researchers)
• 6 Spanish language (3 native speakers/3 non-native speakers)
The Process with Speaking• Started looking at student
speaking samples from 1st year, 2nd year and 3rd year
• From this initial analysis—the team of 6 lang. experts determined what could students do at the end of each year.
• The team of researchers did a lit. review on rubrics and how to create an effective one
• A rubric emerged based on student production
The Process (Cont.)
• Next came the transcription of the speaking samples
• Almost 1000 speaking samples transcribed (954)
• Average student response time was roughly 2 min.
Creating a Rubric
1. Gather samples of student performance
2. Describe samples3. Sort by levels of quality4. Cluster into groups
Creating a Rubric5. Identify levels—determine what
major categories, dimensions or traits seem to be emerging from the sorting process, eliminating redundancies and including descriptors from the list that best describe the traits at each achievement level
6. Samples that identify each ability level (exemplars)—will assist in maintaining consistency with each as you score across time, students and assignments. What each achievement level looks like—several samples at each level
Speaking Rubric
1st year N=105 students/N=31
5 samplesExceeds
StandardsMeets
StandardsDoes Not Meet
Standards Not Ratable
Pronunciation rating 3 180 66 62
Grammar rating 17 105 127 62
Vocabulary rating 22 87 140 62
Fluency rating 20 93 136 62
2nd year N=104
students/N=312 samples
Exceeds Standards
Meets Standards
Does Not Meet Standards Not Ratable
Pronunciation rating 9 232 19 43
Grammar rating 49 146 65 43
Vocabulary rating 64 117 79 43
Fluency rating 62 123 121 43
3rd year N=109
students/N=327 samples
Exceeds Standards
Meets Standards
Does Not Meet Standards Not Ratable
Pronunciation rating 47 169 47 53
Grammar rating 45 167 62 53
Vocabulary rating 52 154 68 53
Fluency rating 41 174 59 53
The Process with Writing
• The methodology was the same.
• The rubric was based on student production and created before analysis of samples.
• No transcription was necessary since they were writing samples
Writing Rubric
The Experience
• Interactive activity• Directions: You all have a
copy of the writing rubric. Please now look at the following three samples and use the writing rubric to assess these actual student samples.
Sample #1
Prompt Transcribed Sample
You are with your new host mother, and she would like to know more about your family. Describe at least 2 members of your family. Be sure to include for each one: • Name• Relationship to you• Age• His or her favorite food or activity
Esta es mi hermano Kane. Tiene diez y siete años. Le gusta mucho la lucha. Este es mi madre Marsha. Tiene cuarenta años. Le gusta preparar la cena. Este es mi padre Brad. Tiene cuarenta y dos años. Le gusta jugar deportes.
Sample #2Prompt Transcribed Sample
Your friend Alejandro is visiting from Honduras and is going to attend school with you for a few weeks. Tomorrow will be his first day and he wants to know what it will be like. Give as detailed a description as possible about a typical day, including: • what time you get up• what you do before school • how you get to school • how many classes you have • which classes you like or don’t like and why • what you do after school.
Mevanto a las seis y media por la manana.
Sample #3Prompt Transcribed Sample
Select one of the rooms pictured below (bedroom/kitchen), in Spanish: • name the room• describe at least 2 of the items that you see in the room including: • the name• color• location in relation to other items in the room
El tazón rojo está al lado de la cuchara. La cuchara está al lado del cuchillo. La ventana está cerca del tazón rojo.
Results
• Sample #1---Exceeds Expectations
• Sample #2---In Progress• Sample #3---Meets
Expectations
Interrater Reliability
• The 6 language experts rated samples to check for interrater reliability
• At first the percentage was low but after negotiation of sample ratings the interrater reliability was 83%
Influential Factors• The 4 researchers reviewed the
factors which influenced the study and did a lit. review on these:– Effect of time limits on testing– Teacher variance– STAMP prompts (listing activities)– Teacher attitudes– Degree of commitment by teachers– Time spent on goal setting– Degree of feedback students
received– Students at different levels– Reluctant learners– Challenges of 1st year language
learning
Related Questions that Emerged during the study• Teacher Beliefs &
instructional practices• Teacher
competency/content knowledge & student achievement
• Effect on student achievement across content areas
Teacher beliefs/change
• Multiple case study (4 teacher participants)
• Interviews, observations, field notes, document analysis
Themes
• Instruction more interactive
• More discovery learning• Students own their
learning• More facilitator of learning• Less textbook driven• Learning more of a
partnership• More accountability
• My instruction is “more interactive, more them doing the work, and them discovering”
• “more accountable to what I am going to be teaching them”
• ‘I can show their parents what they’ve learned, their best work”
• “not so much oriented by the textbook”
• “more purpose in mind of what I am trying to achieve”
• “more as a facilitator of knowledge”
• “much more interactive, I have the students create things themselves, not just looking at the texts”
• “more of a partnership in learning than what we had before”
• “they own what they learn”
• “a lot more self-discovery”
Questions