24
A School-Based Reading Program Evaluation Michael F. Lewis, Ph.D. Niagara Falls City School District

A School-Based Reading Program Evaluation Michael F. Lewis, Ph.D. Niagara Falls City School District

Embed Size (px)

Citation preview

Page 1: A School-Based Reading Program Evaluation Michael F. Lewis, Ph.D. Niagara Falls City School District

A School-Based Reading Program

Evaluation

Michael F. Lewis, Ph.D.

Niagara Falls City School District

Page 2: A School-Based Reading Program Evaluation Michael F. Lewis, Ph.D. Niagara Falls City School District

Changing Education Funding

Current changes in educational funding– New federal grant opportunities– Private grant opportunities

This funding often requires increased accountability

Page 3: A School-Based Reading Program Evaluation Michael F. Lewis, Ph.D. Niagara Falls City School District

Accountability

Landscape in the U.S. has shifted to mandated levels of accountability in many areas due to recent events– Deceptive accounting in business

ENRON

– Abuse of school district budgets Roslyn School District (New York)

Page 4: A School-Based Reading Program Evaluation Michael F. Lewis, Ph.D. Niagara Falls City School District

Accountability

Recently, when funding is awarded accountability is achieved through program evaluation– program evaluation required by independent

evaluators Mandated compensation (7%)

– Strict accounting of expenditures The budget is fixed

Page 5: A School-Based Reading Program Evaluation Michael F. Lewis, Ph.D. Niagara Falls City School District

What is Program Evaluation?

Definition:– a formalized approach to studying the goals,

processes, and impacts of projects, policies and programs

– can involve quantitative or qualitative methods of social research (or both)

– People who do program evaluation come from many different backgrounds:

sociology, psychology, economics, social work

Page 6: A School-Based Reading Program Evaluation Michael F. Lewis, Ph.D. Niagara Falls City School District

Types of Program Evaluation

A needs assessment examines the nature of the problem that the program is meant to address

The program theory is the formal description of the program's concept and design

Process analysis evaluates how the program is being implemented

The impact evaluation determines the causal effects of the program

Cost-effectiveness analysis assesses the efficiency of a program.

Page 7: A School-Based Reading Program Evaluation Michael F. Lewis, Ph.D. Niagara Falls City School District

Impact Evaluation

Impact evaluation is the most common form of program evaluation– Determines (as best as possible) the effects of a

particular program along some criteria DARE (on decreasing drug use) PBIS (on reducing negative behaviors in school) RTI (on reducing # of students identified as LD)

Page 8: A School-Based Reading Program Evaluation Michael F. Lewis, Ph.D. Niagara Falls City School District

Evaluation as ResearchAs conceptualized by: Stephen Truscott, Psy.D. (Georgia State University)

Big ‘R’ research– University Run– For Publication– Large Scale

Little ‘r’ research– Individually Run– For Information– Small Scale

Program Evaluation can be both or either

Page 9: A School-Based Reading Program Evaluation Michael F. Lewis, Ph.D. Niagara Falls City School District

The role of the School Psychologist

We do EVERYTHING!!!– SP has evolved:

We are now Jack (and mostly Jill) of all trades

– We were special education evaluators– We are now increasingly responsible for

many activities related to General EducationIncluding evaluation of programs

Page 10: A School-Based Reading Program Evaluation Michael F. Lewis, Ph.D. Niagara Falls City School District

Program Evaluation In Action

Niagara Falls School District evaluation of: Fast ForWord (FFWD)

– Computer-based– Auditory Processing and Literacy Skills– Timed ‘protocols’ for going through program

50, 90, and 120 minute protocols

The Problem: FFWD has never appeared in peer-reviewed literature

Page 11: A School-Based Reading Program Evaluation Michael F. Lewis, Ph.D. Niagara Falls City School District

The Problem (continued)

Fast ForWord– Proprietary program– No real scientific evaluation of program– How do we know it is really effective?

– We do a program evaluation on student’s who use FFWD…

Page 12: A School-Based Reading Program Evaluation Michael F. Lewis, Ph.D. Niagara Falls City School District

FFWD Evaluation

The Subject– Administration of FFWD to entire 2nd grade– 50 minute protocol (run every day)

The Evaluation– Pre-test/Post-test design– Evaluate every 2nd grader in reading– Analyze findings for increase in reading scores

Page 13: A School-Based Reading Program Evaluation Michael F. Lewis, Ph.D. Niagara Falls City School District

FFWD Evaluation

To evaluate you must have a measure The Measure: GRADE

– Group– Reading– Assessment (and)– Diagnostic– Evaluation

Standardized, norm-referenced

Page 14: A School-Based Reading Program Evaluation Michael F. Lewis, Ph.D. Niagara Falls City School District

FFWD Evaluation

To evaluate you must have a measure The other Measure: DRA

– Diagnostic– Reading – Assessment

(This is a tool we use locally for Reading level)

Page 15: A School-Based Reading Program Evaluation Michael F. Lewis, Ph.D. Niagara Falls City School District

FFWD Evaluation

The Procedure:

Every 2nd grader participated– Daily 50-minute FFWD protocol– Ran for 20 weeks– Took GRADE before and after FFWD

– Classroom instruction did not change

Page 16: A School-Based Reading Program Evaluation Michael F. Lewis, Ph.D. Niagara Falls City School District

FFWD Evaluation

The Procedure (a summary)

1) GRADE pre-test

2) 20 weeks of FFWD (50 minute protocol)

3) GRADE post-test

4) Score and analyze GRADE results

5) Determine effects of FFWD on reading

Page 17: A School-Based Reading Program Evaluation Michael F. Lewis, Ph.D. Niagara Falls City School District

FFWD Evaluation

Statistics:– Imported GRADE data into SPSS

SPSS: Statistical Package for Social Sciences

– Computed Paired Samples t-tests on all students with pre/post GRADE data

– Same computation for DRA levels This determined statistical significance

– Not the whole story…

Page 18: A School-Based Reading Program Evaluation Michael F. Lewis, Ph.D. Niagara Falls City School District

FFWD Evaluation

Statistics:– Statistical Significance vs. Effect Size

Effect size determines a measurable degree of the statistical significance

– Effect size reported in standard deviation form

Evaluation can have statistical significance but a small Effect Size

Page 19: A School-Based Reading Program Evaluation Michael F. Lewis, Ph.D. Niagara Falls City School District

FFWD Findings Paired Sample t-Test

Paired Samples Statistics

99.74 388 15.085 .766

101.84 388 14.367 .729

95.10 386 13.864 .706

99.23 386 15.094 .768

96.67 381 13.986 .717

100.64 381 15.367 .787

Vocabulary CompositeStandard Score-PreTest

Vocabulary CompositeStandard Score-PostTest

Pair1

ComprehensionComposite StandardScore-PreTest

ComprehensionComposite StandardScore-PostTest

Pair2

Total Test StandardScore-PreTest

Total Test StandardScore-PostTest

Pair3

Mean N Std. DeviationStd. Error

Mean

Page 20: A School-Based Reading Program Evaluation Michael F. Lewis, Ph.D. Niagara Falls City School District

FFWD Findings Paired Sample t-Test

Paired Samples Test

-2.11 10.980 .557 -3.20 -1.01 -3.778 387 .000

-4.13 9.575 .487 -5.09 -3.17 -8.473 385 .000

-3.97 8.705 .446 -4.84 -3.09 -8.892 380 .000

Vocabulary CompositeStandard Score-PreTest -Vocabulary CompositeStandard Score-PostTest

Pair1

ComprehensionComposite StandardScore-PreTest -ComprehensionComposite StandardScore-PostTest

Pair2

Total Test StandardScore-PreTest - Total TestStandard Score-PostTest

Pair3

Mean Std. DeviationStd. Error

Mean Lower Upper

95% ConfidenceInterval of the

Difference

Paired Differences

t df Sig. (2-tailed)

Page 21: A School-Based Reading Program Evaluation Michael F. Lewis, Ph.D. Niagara Falls City School District

FFWD Findings

Statistical Significance vs. Effect size

– With a large sample it is highly likely that even a small change will indicate a statistical significance

≈380 students is a large sample size

Page 22: A School-Based Reading Program Evaluation Michael F. Lewis, Ph.D. Niagara Falls City School District

FFWD Findings

Effect Size takes more realistic look at actual increase of significant findings– Hedge’s G: one way to calculate effect size

g = t √(n1 + n2) / √(n1n2) or g = 2t / √ N

Effect Size findings:– Vocabulary: g = .27– Comprehension: g = .35– Total Test: g = .33

These are considered small effect sizes

Page 23: A School-Based Reading Program Evaluation Michael F. Lewis, Ph.D. Niagara Falls City School District

Errors Inefficiency– No need for testing of all students

Maintain statistical meaning w/ smaller random sample – this was ignored by district administration

– Administration Testing all students reduces control of standard

administration– this was ignored by district administration

Evaluation Design– No Control Group

No way of determining if FFWD caused increased GRADE scores w/out control group

Page 24: A School-Based Reading Program Evaluation Michael F. Lewis, Ph.D. Niagara Falls City School District

???Questions???

Program Evaluation Reference:

– Posavac, E. & Carey, R. (2006). Program Evaluation: Methods and Case Studies (7th Ed.). Prentice Hall, New York, NY.

Contact:– [email protected]