Release of 2011-12 Preliminary Value-

Added Data Webinar

August 13, 2012

Florida Department of Education

Topics to Discuss Review of state-level results of the model Review of data files received on August 6,

2012 Discussion of the data provided for non-

FCAT subject area teachers and guidelines for the use of these data

Question and Answer session

Florida Department of Education 2

Review of State-Level ResultsThe next series of slides will review: The model fit in 2012 compared to 2011 Comparison of the Reading and Mathematics

VAM scores by Grade Impact analyses showing relationship of VAM

score with various classroom characteristics

Florida Department of Education 3

Data analyses presented on slides 4-15 provided by American Institutes for Research (AIR)

Model Fit: Comparison of R-Square Values for Reading

The R-Square values – which are an indicator of the how well the model fits the data – are high and virtually equivalent to the values observed with the 2011 model

The model behaves the same using FCAT 2.0 data, as it did using FCAT data

Model Fit: Comparison of R-Square Values for Math

The R-Square values – which are an indicator of the how well the model fits the data – are high and virtually equivalent to the values observed with the 2011 model

The model behaves the same using FCAT 2.0 data, as it did using FCAT data

Teacher VAM Score by Grade (Reading)

VAM scores by grade show no particular advantage for any given grade

Teacher VAM Score by Grade (Math)

VAM scores by grade show no particular advantage for any given grade

Impact Analyses for VAM Scores

The following slides show the relationship of the teacher VAM score with various classroom characteristics

The observed score correlations for each characteristic are reported after the scatter plots

In all cases, the correlations are negligible indicating no advantages or disadvantages for any group of students

Correlation of Teacher VAM Score and Percent Students with Disabilities

Correlation of Teacher VAM Score and Percent English Language Learners

Correlation of Teacher VAM Score and Percent Economically Disadvantaged

Correlation of Teacher VAM Score and Percent Gifted

Correlation of Teacher VAM Score and Percent Non-White Students

Correlation of Teacher VAM Score and Mean Prior Achievement

Observed Correlations with Teacher VAM Score

Review of Data Files Received on August 6Files for FCAT Subject Area Teachers (Reading and Mathematics)State-level fileDistrict-level filesSchool-level filesTeacher-level filesStudent-level filesStudent-Teacher link files

Florida Department of Education 16

Review of Data Files Received on August 6 State-, district-, and school-level files provide

summary statistics that many districts have used as the basis for their classification decisions

Teacher-level files provide the performance data for reading/language arts and mathematics teachers Teacher_VAM_estimate is the value-added score Teacher_VAM_estimate_SE is the standard error

associated with the value-added score Pct_Meet_Expectations is the percentage of the teacher’s

students who met or exceeded their expectation as determined by the value-added model

Florida Department of Education 17

Review of Data Files Received on August 6 Student-Level files provide data on the students

included in the analysis, including the predicted (_1112_predictedscore) and actual scores (_1112_scalescore)

Student-Teacher Link files provide the students who were attributed to each teacher in the calculation of his/her value-added score This file should be used to ensure that the appropriate

students are linked to the appropriate teachers Only students for which necessary data (e.g., at least two

years of assessment data) are available are included

Florida Department of Education 18

Review of Data Files Received on August 6Aggregate school- and teacher-level files were also providedThese files combine value-added data (which is run at the subject and grade level) across grades, subjects, and years using the common metric of a proportion of an average year’s growth

The values for an average year’s growth were also provided to districts, as well, on August 6

The value-added scores are converted into proportions and then combinedA request has been made to provide an aggregate file for only 2011-12 (i.e., does not combine across years). That file should be delivered to districts early this week

Florida Department of Education 19

Review of Data Files Received on August 6In addition to files relating to FCAT subject area teachers, files were provided attributing the Reading and Mathematics value-added results to non-FCAT subject area teachersTeacher-level files for non-FCAT subject area teachers were providedStudent-Teacher link file for non-FCAT subject area teachers were provided

Florida Department of Education 20

Data for Non-FCAT Subject Area Teachers and Guidelines for UseSection 1012.34(7)(e), Florida Statutes, requires that for classroom teachers of courses for which the district has not implemented appropriate assessments or for which the school district has not adopted an equally appropriate measure of student learning growth, student learning growth must be measured by the growth in learning of the classroom teacher’s students on statewide assessments

Florida Department of Education 21

Data for Non-FCAT Subject Area Teachers and Guidelines for Use To meet this requirement, student level Reading

and Mathematics value-added results were attributed to non-FCAT subject area teachers serving students who took statewide assessments to calculate two measures that can be used within the evaluation of these teachers The percentage of the teacher’s students

meeting/exceeding expectations as determined by the value-added model

The average of the teacher’s students’ residuals (i.e., differences between the students’ actual and expected scores as determined by the value-added model)

Florida Department of Education 22

Data for Non-FCAT Subject Area Teachers and Guidelines for Use The average of student residuals is

conceptually similar to a teacher’s value-added score

It is a measure reflecting, on average, the degree to which teacher’s students exceeded or missed their expected score as determined by the value-added model

This data can be used in a similar fashion as the value-added score to classify teacher performance

Florida Department of Education 23

Data for Non-FCAT Subject Area Teachers and Guidelines for UseSuggested guidelines for the use of the “average of residuals” scoreIf using state- or district-level summary statistics (e.g., statewide mean) as a basis for classification, rely upon statewide (or district-wide) summary statistics for this group of teachers

Statewide and district-wide summary statistics will be provided to districts this week for this group of teachers

Use the standard error, as your local plans outline, to assist in classifying teacher performance

The August 6 files contained the standard deviation of the average of the residuals; the standard error can be calculated, as well, and will be provided to districts in updated data this week

Florida Department of Education 24

Other Topics Courses used in the calculation of the FCAT

Reading and Mathematics value-added model Two elementary reading/language arts courses

that some school districts began reporting in 2011-12 were not included in the preliminary value-added analysis Course 5010045 Course 5010046

These courses will be added during the reanalysis phase

Florida Department of Education 25

Other Topics Courses used in the calculation of the FCAT

Reading and Mathematics value-added model AP and IB language arts and mathematics courses

have not been included in the FCAT value-added analyses since they are courses with their own assessments measuring the college-level/international standards of those courses

There has been a request to add these courses to the analysis in order to produce an FCAT value-added score for AP/IB teachers who teach students in language arts or mathematics taking FCAT

These courses will be added during the reanalysis phase

Florida Department of Education 26

Other Topics Since the predictions derived from value-added

models are based on how students and classrooms perform statewide, given characteristics controlled for in the model, changes to the underlying data submitted through the review period necessitates a re-calculation of the model

Important, this will result in a new set of teacher value-added scores for all, based on the verified data

Given this fact, we advise that districts await final value-added results, following the review period, before finalizing evaluations

Florida Department of Education 27

Wrap-Up Reminder: Data review period is currently

scheduled to close on Thursday, September 6 at 5:00 EDT

Final value-added data should be delivered by the end of September

We will likely extend the October 19 reporting deadline for evaluation results to the Department

Additional files to be delivered this week are: One-year aggregate FCAT VAM file (2011-12) for teachers

and schools Updated non-FCAT teacher file with the standard error

statistic for the “average of residuals” included State- and district-level summary statistics for non-FCAT

teachers on the “average of residuals” measure

Florida Department of Education 28

Contact Information Juan Copa juan.copa@fldoe.org (850) 245-0744

Questions?

Florida Department of Education 29