View
217
Download
0
Category
Tags:
Preview:
Citation preview
A Performance-Based Evaluation Model for Rewarding Merit in Italian Schools
Donatella Poliandri, INVALSI
Paola Muzzioli, INVALSI
Isabella Quadrelli, INVALSI
Sara Romiti, INVALSI
Rome, 3-5 October 2012
Evaluation and Development of Quality in schools (VSQ)
VSQ is a pilot study carried out by the Italian Ministry of Education (MIUR), INVALSI and Sapienza University of Rome
Its objectives are: to design a model for school evaluation; to acknowledge excellence and promote support for school
improvement.
The project integrates two perspectives:1. Merit rewarding. Top performing schools receive financial
rewards (competition mechanism). 2. School improvement. low performing schools are
supported to start an improvement plan (equal opportunity approach).
Aim of this presentation
The theoretical framework
Sen’s concepts of merit and meritocracy, in Merit and Justice (2000)
Merit and Justice
According to Sen, we can scarcely dispense with rewarding actions which produce good results, at least for their incentive effect.
Does rewarding merit necessarily produce inequality?
As merit is a derivative and contingent concept, equality or equal opportunities could be the leading criteria to define merit.
“The art of developing an incentive system lies in delineating the content of merit in such a way that it helps to generate valued consequences” (Sen
2000. p. 9)
“The art of developing an incentive system lies in delineating the content of merit in such a way that it helps to generate valued consequences” (Sen
2000. p. 9)
Research questions Starting from the structure of the VSQ project,
based on economic incentives and improvement actions, we explored the following research questions: Are students’ achievements at SAT sufficient to define the
quality of a school? Which criteria determine whether a school deserve a
reward or not?
In the light of the foregoing considerations, we have investigated the possible ways of defining a concept of quality for schools.
In the light of the foregoing considerations, we have investigated the possible ways of defining a concept of quality for schools.
The operationalization of quality
A multi-criteria concept of quality
Measurement of students’ achievement
standardized instruments were used to assess students’ learning such as reading understanding, grammar and math tests administered by INVALSI
Results from the INVALSI tests were used to construct a measure of Value-added (developed by Ricci & Falzetti from INVALSI) .
Value-added models allows to compare schools from the same starting point, that is not considering the differences in students’ achievement due to individual, structural and contextual variables. In Value-added models the variables which are not under the direct
control of schools are not considered.
The Value-added represents the “measure of quality in terms of the extent to which the educational experience enhances the knowledge, abilities and
skills of students” (Harvey and Green, 1993).
The Value-added represents the “measure of quality in terms of the extent to which the educational experience enhances the knowledge, abilities and
skills of students” (Harvey and Green, 1993).
Evaluation of positive actions
Evaluation Teams visited each school and assessed the quality level in several areas
InclusionInclusionEnrichment & Remediation programmes
Enrichment & Remediation programmes
School choice guidance
School choice guidance EvaluationEvaluation
Scoring Rubrics The instrument used to evaluate schools’ positive
actions is the Scoring rubric A rubric contains evaluation criteria and a
description of the good and less good performances related to different levels of quality (inadequate, acceptable, good, excellent).
Each quality criteria is articulated through a “multi-focus” perspective: that is the same quality criteria is assessed considering several dimensions
For each Scoring rubric it was provided a checklist that contains a list of indicators that were considered relevant and that needed to be observed for reaching a good level of quality
Multi-focus perspective In the Rubric assessing Remediation programmes
four dimensions are identified: needs analysis: in this dimension are considered how
schools collect information about, or monitor educational and training needs of students, in order to implement specific actions;
organizational structure: this dimension is related to organizational aspects of the intervention (e.g. Is there a teacher leader for remediation programmes? How students access remediation classes?);
teaching: refers to the specific teaching strategies and actions activated (e.g. grouping strategies, etc.)
satisfaction / effectiveness: considers students’ opinion about remediation classes and the impact of remediation classes attendance on students’ results.
Scoring Rubrics and Mean Rubric Score
Inclusion of students with disabilities
Inclusion of foreign students
Enrichment programmes
Remediation programmes
School choice guidance
Internal evaluation / Self-evaluation
Teachers’ educational planning and evaluation of students
Mean RubricScore
Mean RubricScore
Inclusion
Enrichment & Remediation
Evaluation
School choice guidance
Rubric Score
Rubric Score
Rubric Score
Rubric Score
Rubric Score
Rubric Score
Rubric Score
Rubric Score
Rubric Score
Rubric Score
Rubric Score
Rubric Score
Rubric Score
Rubric Score
The Final Score
The weight of the Mean rubric score is 40%
The weight of the Value-added score is 60%:
- 35% for Italian - 25% for Math
The final score – that defined each school’s position in the final ranking – was made up from the Value-added score and the Mean rubric score
The higher weight given to Value-added in Italian arises
from the relation between competences in Italian and
competences in other subjects (e.g. the link between scientific processes comprehension and
Italian literacy)
Results: quantitative and qualitative tools explore different aspects
77 primary/middle schools from 4 different provinces (Pavia, Mantova, Arezzo, and Siracusa) in Italy have participated to this study.
Correlation between Rubric scores and Value-added in Italian and Maths confirm that both quantitative and qualitative instruments contributed to the evaluation of the educational process through the exploration of different aspects.
Pearson correlation between Value-added, Mean rubric score and Final score
MEAN rubric score
Value-added
ITALIAN
Value-added
MATH
Final score
MEAN rubrics score 1
Value-added ITALIAN
,10 1
Value-added MATH ,16 ,61** 1
Final score ,51** ,85** ,81** 1
**. Correlation is significant at the 0,01 level (2-tailed)
Scoring Rubrics’ validity
In Scoring Rubrics there is a high correlation between the number of checks filled by evaluation Teams (that is the number of indicators observed by the Team) and the total quality reached by the school in a specific area.
The higher the number of indicators included in the checklist, the higher the level of quality reached by a school in a specific area.
Pearson correlation between Rubric score and percentage of checklist filling out
Rubric score
Inclusion students with disabilities ,796**
Inclusion foreign students ,770**
Internal evaluation/self-evaluation ,734**
Planning and student evaluation ,752**
Enrichment programmes ,775**
Remediation programmes ,753**
School choice guidance ,802**
Distribution of schools by Value-added levels
• Almost 50% of schools have an intermediate level of Value-added both in Italian and Maths. This is consistent with the results of other research using Value-added in Italy
• The number of schools with lower levels of Value-added is higher than schools with higher ones.
Distribution of schools by value added levels
Added value levels Added value
ITALIAN %
Added value
MATH %
Low 19 24,66 23 29,87 Medium-low 2 2,60 3 3,90 Medium 38 49,35 35 45,45 Medium-high 4 5,19 3 3,90 High 14 18,18 13 16,88 Total schools 77 100 77 100
Good and less good school performances (Rubric Score)
Overall Mean rubric score is 2.63 with a standard deviation of 0.60, this confirm that most schools have received acceptable and good scores.
The lowest performance was on the Rubric “Internal evaluation / self evaluation”: 30% of schools were assessed as inadequate, less than 10% obtained an excellent score.
The best performance was on the Rubric “Inclusion of students with disabilities”: only 1% of schools received an inadequate score.
0
5
10
15
20
25
30
35
40
45
Inadequate Acceptable Good Excellent
rubric score
per
cen
tag
e o
f sc
ho
ols
Inclusion students with disabilities Inclusion foreign students
Internal evaluation / self evaluation Planning and students' evaluation
Enrichment actions Remediation actions
Orientation
Correlation between Rubric scores and Value-added scores
Correlations show a significant positive association between students’ evaluation practices and the value added of Italian and Math.
There is a positive significant relation between Value-added scores in Math and effective Enrichment programmes
There is a significant association between student evaluation practices and enrichment and remediation programmes. These results show the importance of a good student evaluation system in order to plan effective individualisation interventions (enrichment or remediation programmes).
**. Correlation is significant at the 0,01 level (2-tailed).*. Correlation is significant at the 0,05 level (2-tailed).
VSQ: next steps
Regional ranking have been produced and 25% of schools placed in the top distribution of the ranking have received a money token
All participating schools have received an individual Evaluation report that highlights strengths and difficulties in the areas under investigation and give suggestion for improvement
Schools also have received financial support for implementing improvement plans: the amount of such support is higher for schools that were in the bottom distribution of the ranking
In 2013, participating schools will be evaluated again, using the same instruments, in order to assess the effect of the competitive mechanism and of the improvement plan on school performance, and to distribute the final part of the money incentive.
Thanks for your attention!
Contacts: donatella.poliandri@invalsi.it
Recommended