The Indiana Acuity Efficacy Study: Year 2 Results and Implications Terry Spradlin Bridget Schleich June 21, 2010 CCSSO National Conference on Student Assessment

The Indiana Acuity Efficacy Study: Year 2 Results and Implications

Terry SpradlinBridget SchleichJune 21, 2010

CCSSO National Conference on Student Assessment

2

About the Center for Evaluation & Education Policy

•The Center for Evaluation & Education Policy (CEEP) is a client-focused, self-funded research center associated with the School of Education at Indiana University

•CEEP promotes and supports rigorous program evaluation and nonpartisan policy research primarily, but not exclusively, for education, human service and non-profit organizations

•In the area of K-12 education policy, CEEP’s mission is to help inform, influence and shape sound policy through effective, nonpartisan research and analysiswww.ceep.indiana.edu

http://www.ceep.indiana.edu/

Contents

I. Indiana’s Comprehensive Assessment Plan

II. 2009-10 Testing Schedule (formative and summative)

III. Objectives of Efficacy Study in Indiana

IV. Study Design and Methods

V. Findings for Each Qualitative Analysis

VI. Overall Summary of Qualitative Findings

3

I. Indiana’s Comprehensive Assessment Plan

• Adopted by the Indiana State Board of Education on November 1, 2006

• Plan called for moving the Indiana Statewide Testing for Educational Progress-Plus (ISTEP+) from Fall to Spring and the implementation of formative/diagnostic assessments

• Features implemented during the 2008-09 school year:

o Wireless Generation’s mClass Reading 3D and Math (Grades K-2 formative)o CTB/McGraw-Hill’s Acuity Assessment Program (Grades 3-8 formative)o Phase-out of the Graduation Qualifying Exam (GQE)

• Class of 2011 last to be required to pass GQE• To be replaced with end-of-course assessments in core subject areas

o Moved ISTEP+ from Fall to Spring• (Students in grades 3-10 were tested twice during the 2008-09 school

year)

4

II. 2009-10 Fall Testing Schedule (Formative and Summative)

5

Assessment Testing Window Grade

mCLASS: Reading 3D 8/24-9/18 K-2

mCLASS: Math 9/21-10/16 K-2

Acuity Predictive A ELA/Math 9/28-10/9 3-8

Acuity Diagnostic 1 10/19-10/30 3-8

Acuity Predictive B ELA/Math 11/30-12/11 3-8

Acuity Predictive B Science 12/7-12/18 4, 6

Acuity Predictive B Social Studies 12/7-12/18 5, 7

II. 2009-10 Spring Testing Schedule (Formative and Summative)

6

Assessment Testing Window Grade




Acuity Predictive C Science 2/8-2/19 4, 6

Acuity Predictive C Social Studies 2/8-2/19 5, 7

Acuity Predictive C ELA/Math 2/11-2/26 3-8

ISTEP+ Applied Skills 3/1-3/10 3-8


ISTEP+ Multiple Choice 4/26-5/5 3-8




III. Objectives of Efficacy Study in Indiana

Objectives of CEEP Study:

• Evaluate the effects of CTB/McGraw-Hill’s Acuity Assessment Program, a formative assessment system, on instructional practice and student achievement, particularly ISTEP+, for Acuity schools

• Information intended to inform CTB and the IDOE about the kind of support needed to make the implementation of Acuity most effective during subsequent school years

7

IV. Study Design and Methods

Qualitative and Quantitative Methods

1. Spring Statewide Online Survey of Acuity Schools2. Intensive Case Study3. Completion of a Comparison-Group Study

8

Study Design and Methods

Spring Statewide Online Survey of Acuity Schools

• Measure attitudes and perceptions of teachers and administrators on the benefits and outcomes of their participation in the Acuity Assessment Program during the 2009-10 school year

• Survey window was open from April 12 to April 30, 2010

• 1029 respondents included 655 (64%) teachers, 198 (19%) principals, 70 (7%) testing coordinators, and 106 (10%) other school personnel

9


Spring Statewide Survey (cont.)

Questions probed:

• Educator opinions regarding Acuity Assessment Program content, technology/user experience, professional development, and customer support after use of the system for a full school year or more were assessed

• In addition, a primary objective of the survey to obtain suggestions for improvement of the program and to gauge views regarding the impact of the program on classroom instruction, general student achievement, and student achievement on ISTEP+

10


11

Spring Statewide Survey (cont.)

2 Surveys were available :Schools in Year 1 : 16-question surveySchools in Year 2+: 18-question survey

Questions were identical on both surveys; however, Year 2+ schools were additionally asked:

1) The frequency with which they are using the Acuity tools (e.g. not using the tools; beginning to use the tools; using the tools extensively, etc.)

2) The extent to which they are using the Acuity data (e.g. not using the data; using Roster Reports; using Assessment Reports, etc.) to alter or inform instruction


Qualitative Methods

Intensive case study• Determine what factors make a difference in the effective

implementation of Acuity and use of Acuity data• Extent to which schools have implemented the Acuity Assessment

Program and identify obstacles encountered• Extent to which Acuity has altered or informed classroom instruction,

and impacted general student achievement as well as ISTEP+ performance

12

Study Design and Methods Intensive Case Study (cont.)

• The Project Team identified 11 schools (9 elementary schools; 6 middle schools) for the case study that were reflective of Indiana’s student population based on prior ISTEP+ performance, free and reduced-price meal program eligibility data, school size, and local type (urban, suburban, and rural)

• Additionally, the Project Team identified four schools with significant scale score improvement from fall to spring ISTEP+ during the 2008-09 school year

• Qualitative procedures included one-on-one, face-to-face interviews with 34 principals, testing coordinators, and Acuity trainers as well as focus groups with six to 10 teachers in all 15 schools (109 teachers total); conducted 2/03/10 through 4/06/10

13

School Location Scale Score Change

Forest Glen Elementary School Indianapolis, IN 71.5

Dupont Elementary School Dupont, IN 37

Blackhawk Middle School Fort Wayne, IN 30

Blair Pointe Upper Elementary School Peru, IN 24.5

Study Design and MethodsQuantitative MethodsCompletion of a Comparison-Group Study • Using matching, regression and other statistical techniques to analyze quantitative

data collected from the Acuity Predictive and Diagnostic assessments as well as from Indiana’s ISTEP+ summative assessmento These analyses will be used to assess the degree to which use of the

predictive and diagnostic assessments are associated with increased achievement on the state-required summative assessments in mathematics and English/language arts

o Comparison schools will be matched with Acuity schools using prior ISTEP+ performance, free and reduced-price meal program data, race/ethnicity, school size, and local type (urban, suburban, and rural)

o CEEP will subsequently conduct a longitudinal study adding 2009-10 data to the aforementioned analyses to monitor performance differences from one school year to the next.

14

V. Findings for Each Qualitative Analysis

A. Spring survey

B. Case Study Site Visitso Positive Educator Feedbacko Pressing Issueso Educator Recommendations and Considerations

15

A. Spring Survey Results: Findings, Opinions, and Suggestions

16

Frequency of Formative Assessment Use

• Respondents most commonly used the program only during administration windows.

17

Percent of Respondents from Year 1

Schools

Percent of Respondents from Year 2+

Schools

Once/week14.9 13.5

Multiple times/week15.9 15.3

Once/month 20.0 20.0

Only during administration windows

36.5 39.8

Other (Varied)13.5 11.3

Total Number of Respondents 416 600

Other Frequency of Use Issues• Most Acuity schools (78% for Year 1 schools; 74%, Year 2+) used the

online tools exclusively in their administration of the Acuity Assessment Program – 21% of Year 1 and 20% of Year 2+respondents use both online and

paper/pencil tests.

• Educators are not using all components of Acuity as often as necessary to maximize the use and benefits of the system – e.g., only 45% of Year 1 respondents and 47% of Year 2+ respondents

indicated using the Instructional Resources; only 18% of Year 1 and 26% of Year 2+ respondents indicated using the Item Bank

18

Perceived Impact on Classroom Instruction

• A majority of respondents (63.5% in Year 1 schools; 65.6%, Year 2+) indicated that participation in the Acuity Assessment Program helped to somewhat improve classroom instruction

• 19.4%/22.7% felt participation had no impact on instruction• 6.3%/5.5% indicated that participation led to a decreased quality of instruction (due to the

multiple assessments scheduled during the school year, both formative and summative, limiting instructional time)

• When comparing last year’s spring survey findings to schools in Year 2+ this school year, the percent of respondents who felt Acuity had at least somewhat improved classroom instruction increased from 51% to 66%

19

Percent of Respondents from Year 1 Schools

Percent of Respondents from Year 2+ Schools

Greatly decreased quality of instruction 1.0 1.3Somewhat decreased quality of instruction 5.3 4.2

No impact on instruction 19.4 22.7Somewhat improved quality of instruction 63.5 65.6

Greatly improved quality of instruction 10.8 6.2


Perceived Impact on Student Achievement

• The majority (63.5% in Year 1 schools; 61.9%, Year 2+) of respondents indicated that participation in the Acuity Assessment Program during the 2009-10 school year led to somewhat improved student achievement outcomes

• Opinions of the impact of Acuity on student achievement were very similar in schools using Acuity for the first time and schools in year 2+

• From last spring’s statewide survey of Acuity during the 2008-09 school year, 45% of respondents felt Acuity somewhat improved (43%) or greatly improved (2%) student achievement. Over the course of one year, respondents’ opinions improved 20 percentage points overall regarding the influence of Acuity on student achievement.

20



Greatly decreased achievement 0.5 0.5

Somewhat decreased achievement 2.2 4.2

No impact on achievement 29.1 30.1Somewhat improved achievement 63.5 61.9

Greatly improved achievement 4.4 3.3


Perceived Impact on Student Performance on ISTEP+

• A majority (70.2% in Year 1 schools; 58.6%, Year 2+) of respondents indicated that they anticipated participation in Acuity will lead to somewhat improved student performance on the Spring 2010 ISTEP+

• 23.6%/36.0% stated that participation would have no impact on student ISTEP+ scores• Year 1 schools were more optimistic about Acuity’s impact on ISTEP+ scores• In the 2008-09 Spring Survey, 49% of respondents believed Acuity somewhat improved

ISTEP+ performance as compared to 59% of Year 2+ respondents this 2009-10 school year. This indicates an increase of 10 percentage points in respondents positive perception regarding the impact of Acuity on student ISTEP+ performance.

21



Greatly decreased performance 0 0.5Somewhat decreased performance 1.2 3.0No impact on performance 23.6 36.0Somewhat improved performance 70.2 58.6Greatly improved performance 4.9 1.9


Impact on Instruction and/or Student Achievement: Year 1 Group

(Q 11) Why do you think classroom instruction and/or student achievement declined, improved, or did not change? Of the 296 written responses:

• For those who believed Acuity led to improvement in classroom instruction and/or student achievement, the largest number of respondents (44) cited that Acuity shows student strengths, weaknesses, and/or areas for improvement.

• For those who believed Acuity led to no change in classroom instruction and/or student achievement, the largest number of respondents (27) cited that the test’s impact could not yet be determined.

• For those who believed Acuity led to declined classroom instruction and/or student achievement, the largest number of respondents (25) indicated students were being tested on material they didn’t know yet.

22

23

(Q 13) Why do you think classroom instruction and/or student achievement declined, improved, or did not change? Of the 424 written responses:

• For those who believed Acuity led to improvement in classroom instruction and/or student achievement, the largest number of respondents (93) cited that teachers are better able to target teaching and instructional materials based on demonstrated student needs.

• For those who believed Acuity led to no change in classroom instruction and/or student achievement, the largest number of respondents (64) reported educators are not using all of the Acuity resources available.

• For those who believed Acuity led to declined classroom instruction and/or student achievement, the largest number of respondents (37) indicated students were not taking Acuity seriously because of boredom, frustration, or because they knew they were not being graded.

Impact on Instruction and/or Student Achievement: Year 2 Group

(Q 13) What was the most helpful component of the Acuity Assessment Program? (Year 1)

• Nearly half of respondents from Year 1 schools indicated Data and Instructional Resources were the most helpful component

• Teachers especially appreciated the Matrix Reports and having data available immediately

24

Most Helpful Program Components

Frequency PercentData and Instructional Resources 129 48.9

User friendliness for teachers 95 36.0

Reports 22 8.3Custom tests 14 5.3Predictive assessment 4 1.5

Total Number of Respondents 264

(Q 15) What was the most helpful component of the Acuity Assessment Program? (Year 2+)

• Acuity reports were frequently cited as the most helpful component (34.8%)

• 71 of 77 respondents who cited data mentioned using data to drive instruction

25

Most Helpful Program Components

Frequency Percent

Reports 139 34.8

Data 77 19.3

Instructional Resources 50 12.5User-friendly/aided teacher instruction 45 11.3

Preparation for ISTEP+ 37 9.3

Assessments 14 3.5

Alignment with Standards 9 2.3Breaking questions into standards; Blueprints 6 1.5

Measuring Student Progress

3

1.0

Total Number of Respondents 399

(Q 14) How can CTB/McGraw-Hill enhance the Acuity Assessment Program for future assessment

administrations? (Year 1)• The largest number of

respondents (34.3%) cited the need for further alignment of Acuity, ISTEP+ and curriculum pacing

• Content concerns included need for more questions per standard (especially in science and social studies) and less repetition of material

26

Suggested Program Improvements

Frequency Percent

Improve Instructional Resources 83 30.6

Improve content 42 15.5

Alignment with ISTEP+ and instruction 93 34.3

Better/more training and Professional Development 30 11.1

Make Acuity more student-friendly 18 6.6

Make Acuity more teacher-friendly 5 1.8

Total Number of Respondents

271

(Q 16) How can CTB/McGraw-Hill enhance the Acuity Assessment Program for future assessment

administrations? (Year 2+)• The largest number of

respondents (93 out of 426) indicated they would like improvements to Instructional Resources, especially ease of assigning and automatic assignment

• 15.3% of respondents were concerned with the difficulty of use for students, including technological concerns like scrolling in a window, aesthetics, and special needs accommodations

27

Suggested Program Improvements

Frequency PercentImprove Instructional Resources 93 21.8More student-friendly 65 15.3Content 44 10.3More teacher-friendly 37 8.7Improve reporting features 34 8.0Align Acuity with instruction and state standards 30 7.0

Improve the Instructional Resources 27 6.5

More/better training and Professional Development

26 6.1

Scheduling 26 6.1Alignment with ISTEP+ 18 4.2Improve Custom Tests 15 3.5Overload of testing 5 1.2General positive feedback 5 1.2No comment 23 5.4

Total Number of Respondents

426

B. Case Study Results: Site Visit Findings, Suggestions, and

Implications

28

Positive Educator Feedback

The program helps teachers identify skills and standards with which students struggle; many use it to help guide instruction

Teachers found the step-by-step approach for student use of the Instructional Resources extremely helpful

Teachers liked Acuity’s relation to ISTEP+, including comparability of questions and providing clear information to the student and teacher alike if the child is on target for passing ISTEP+

Teachers using the Diagnostic test appreciate aligning instruction with the test

Students are navigating the Predictive assessment system more easily now that they are used to it

29

Positive Educator Feedback (cont.)

30

Teachers reported that giving the Predictive assessment is fairly straightforward

Teachers who understand how to create and utilize reports are doing so, particularly batch reports, class rosters and Matrix reports

Educators are pleased with the response from the Help Desk

Teachers and students liked the clickers and said their use decreases cheating

The majority would continue using Acuity if the decision was theirs alone

Pressing IssuesTiming and Scheduling Acuity Predictive C is too close to the ISTEP+ window to be useful Time constraints impede feasibility of accessing reports and assigning Instructional

Resources, lab time, etc. Material students are being tested on is not aligned with the curriculum causing

frustration for both teachers and students

Instructional Resources Instructional Resources are complicated to assign and monitor Instructional Resources need more questions for each standard and variety Assigning Instructional Resources is time consuming to the point that negative

experiences are diminishing teacher use of the system

Professional Development Teachers would appreciate hands-on, differentiated training

31

Pressing Issues (cont.)

Student Experiences Acuity is not engaging for all students Vocabulary used is too difficult Constructed Response is too time consuming and seems pointless to

many since the score is not calculated into the final grade Acuity is not adaptable or appropriate for lower-ability students

Reports and Data Parent reports are needed that are easy to understand and readily

accessible

32

Educator Recommendations & Considerations

• Improving Student Experiences with Acuityo Teachers recommended that CTB/McGraw-Hill makes Acuity more visually appealing when students

receive their results; this may be a gateway to providing motivation to improve.o Students should be required to review items they answered incorrectly, and retests should be an

alternate form which tests the same content.o One suggestion for dealing with difficult vocabulary was to provide an easy definition of a word

which appears when a cursor scrolls over it.• Teaching Strategies

o Teachers would like to be able to personally arrange questions to more closely align with their schools’ curriculum pacing.

o The vast majority of teachers would like an auto assignment feature added to the system to assign Instructional Resources to students below a certain threshold on either the Diagnostic or Predictive assessments.

• Tips From Educatorso Educators would like an easily accessible parent report that can be pulled up during meetings. The

report should indicate areas in need of attention; should be easy for all parents to understand; and should help teachers indicate whether a child is expected to pass the ISTEP+.

33

V. Overall Summary of Qualitative Findings

The program helps teachers identify skills and standards with which students struggle allowing for instruction to better meet the needs of students

Teachers desire more training on using student and class data to inform and improve instruction

Educators expressed high levels of satisfaction with customer service and attentiveness by CTB

Teachers and students alike enjoy the clickers (student response devices)

Educators offered many suggestions for modification to Acuity; CTB appears to be listening

34

Summary of Qualitative Findings (cont.)

Some teachers expressed concern regarding the alignment between Acuity and the curriculum

Educators would like to see the initial assignment and reassignment of Instructional Resources made easier. They would also like to see an automatic tracking component of assigned Instructional Resources

Questions on the test as well as in the Instructional Resources need to be improved. This can be done by limiting repetition of material and ensuring adequate content is available for addressing each standard

The full impact of Acuity is yet to be determined, but more educators are becoming optimistic about the impact of Acuity on classroom instruction, student achievement in general, and on ISTEP+ in particular.

35

CEEP Contact Information

Terry E. Spradlin, MPAAssociate Director for Education [email protected]

Bridget SchleichGraduate Research [email protected]

1900 East Tenth StreetBloomington, Indiana 47406-7512812-855-4438Fax: 812-856-5890http://ceep.indiana.edu

36

mailto:[email protected]

mailto:[email protected]

Documents

The Indiana Acuity Efficacy Study: Year 2 Results and Implications Terry Spradlin Bridget Schleich June 21, 2010 CCSSO National Conference on Student Assessment