Lincolnwood School District 74 MeasuredEffects

Lincolnwood School District 74MeasuredEffects.comBen Ditkowsky, Ph.D.Advanced Data Based Decision MakingIllinois State UniversityGaryCates.netGary Cates, Ph.D.

A Model of What RtI Might Look Like In Your School

Language!Corrective ReadingReading Mastery I - VHorizons A, BAMPHorizons A/B, C/DJolly PhonicsPrentice HallHoughton MifflinFountas and PinnellMaking MeaningJolly PhonicsUniversal ScreeningTeacher JudgmentCommon AssessmentsTri-annual screening / ODRReview DataAre scores typical?Typical Instruction with differentiationTypical scores (above 25%ile)Student on watch listretestTeacher disagrees

NOLow scores (

BIG IDEA #1Service is determined by needIntensity of service is determined by what it takes to be successful. Special Education Eligibility comes after we determine the intensity, and if that service can not be maintained in General education. Example RtI Manual: http://sd74.org/home/files/RTI/RtI_Manual_2009.pdf

Alternatively, RtI Might Look A Little Different In Your School

Lincolnwood School District 74Ben Ditkowsky, Ph.D.Using Data For Grouping Students into Interventions

DOESCANCANTDOESNTMotivateNot FluentFluentPracticeTEACH?respecifyImplicationsWith Whom Do We Intervene?Adapted from Gresham, Sugai and Horner, 2001

When Grouping Students We Should Consider The SourceWe need to intervene with students who do not have the skills to be successful, or those who have the skills but have not sufficiently mastered the skills to be successful independently.

What Data Do You Have or Collect?MAPScantron PerformanceCBM (DIBELS / Aimsweb)Common Grade Level Assessment (s)ISATEXPLORE, PLAN________________________________________________________________________________________________

Thoughts from the GroupGood PracticesQuestions

Two Factors To Consider When Using Multiple Points of AssessmentConvergence: Agreement or Overlap, a representation of common ground between theories i.e., Do most assessments result in similar conclusions?Magnitude: The property of relative size or extent (whether large or small) i.e., How large is the need?

ConvergenceThe question of convergence is

To what degree does all of the assessment data indicate a need for the decision you are considering?

If more than half of the assessments examined indicate a need for something different than what is typical, then we should at least consider something different.

If There Is Not ConvergenceThen we can discount a single indicator of magnitude becauseIt is likely that the poor test score is the result of a bad day

MagnitudeThe question of magnitude is

How Much of a Difference in Programming is Required?

If there are more indications that the difference between the student and typical peers is large then small, then more intensive service should be considered.

How Can We Use Convergence and Magnitude to Make Intervention Groups?Step 1. What data will you use?Step 2. What are the cut scores indicating a difference requiring additional assistance? What is the cut score indicating a need for substantial assistance?Step 3. Classify each score as Risk versus No RiskStep 4. Determine convergence, list scores with >50% convergenceStep 5. Sort by MagnitudeStep 6. Meet with teachers to validate decisions

Step 1. What data will you use?Previous State Test ScoresEPAS ScoresCBM ScoresGrade Level Common Assessment ScoresVocabulary Matching

Step 2. Cut Scores

Cutting Score (range)Asmt #Tier 3Tier 2# 1ISAT Reading120 - 190191 - 240# 2EXPLORE Reading813# 3CBM Reading> 100 WRC101 150 WRC# 4Grade Level Asmt >10th %ile>20th %ile# 5Vocabulary Matching>10th %ile>20th %ile

Step 3. Classify Each Score

Step 4. Determine Convergence

Step 5. Sort by Magnitude

Step 6. Meet with Teachers to Validate Decisions

Sometimes there is additional data that are relevant to groupings

Your TurnDetermine Convergence for the following

Your TurnDetermine Magnitude for the following

BIG IDEA #2Two factors to consider when grouping students for intervention are convergence of data and the magnitude of the problem. Placement in intervention is ideally based on multiple converging sources of data. The intensity of service that a student receives is based on the magnitude of the problem not whether the students has unsuccessfully jumped through hoops.

Problem Specification

Problem Identification

To Solve A Problem, First We Need To Define It

If The Student Can Not Do What Is ExpectedWe need to determine a starting point for intervention?Does this call for a quick fix or a comprehensive intervention?If the students performance is significantly below peers, we need to consider if a powerful Tier 3 Intervention has been implemented with Fidelity before consideration of Special Education.

If The Student Can Do What Is Expected But Not Very Good At It...Whether the task is academic or behavioral we all tend to avoid doing things we are not good at doing

The intervention should focus on building automaticity with the task

If The Expectation Is Too Easy, Idle hand are the devils playthings

Motivation Sprick, Garrison & HowardMotivation = Expectation x ValueMotivation = Expectation x Value

Would The Student Say Its Easy?If the teacher thinks the student can do it, but the student does not, then the intervention should focus on getting the student to see that they CAN do it.

Remember, this is from the students perspective.

Does The Student Value Doing It?When internal motivation is lacking, sometimes we need something else.

If Other Students Encourage The ProblemThe intervention should focus on teaching the student about more appropriate ways to get peer attentionGetting Peers to ignore problem behaviorGetting peers to reinforce appropriate behavior

Does The Student Want Adult Attention?The intervention should focus on teaching more appropriate ways to get adult attentionBe Clear about expectationsProvide Specific PraiseIgnore minor infractions

Is The Student Just Overwhelmed?If the student is overwhelmed with the quantity of work, the intervention might focus on first improving quality in exchange for quantity, then gradually increase the amount of work

YesNoQuestion 1Are there Medical ImpairmentsContact parents resolve medical (e.g., vision, hearing) issues.Continue to Question 2Question 2Does the student understand the expectations?Continue to Question 3Clarify expectations for the studentQuestion 3Can the student do what is expected?Continue to Question 4Teach the skills necessary to meet the expectation.Question 4Is the student good at doing what is expected?Continue to Question 5Provide opportunities for practice.Question 5Is the expectation too easy?Provide more challengeContinue to Question 6The student requires motivation (i.e., this is a behavior problem).Question 6If the student were to tell us how likely it is that s/he would be successful in completing the expectations, the student would say the task could easily be completed.Continue to Question 7Increase student self confidence (refer to social worker)Question 7If the student were to tell us how valuable it would is to successfully meet the expectations, the student would say the value of doing what the teacher wants is high.Review / revise responses to Question 3, 4, and 6Increase the value of meeting expectations (continue to Question 8)Question 8Do peers encourage the student to be disruptive?1. Among other things, this is a classroom management issue. The plan should address peers as well as the student.Continue to Question 92. The plan should include appropriate ways to gain peer attention.Question 9Does the student respond to direct teacher attention?1. Give explicit directionsContinue to Question 102. Provide specific praise each time the student complies3. Ignore minor infractionsQuestion 10Does the student appear to be overwhelmed when assignments are long or detailed?Consider providing opportunities for the student to decrease quantity of work in exchange for quality of work.Continue to Question 11Question 11Does the problem results in little to No work being done?Consider a behavior contract in which the student can earn something for meeting the expectation.Re-examine the task and the expectations

If The Problem Is Simply Not Enough Work, Then We Are Back To Contracting, or else

BIG IDEA #3Solving Problems is based on identifying a focal point for intervention.

Module D. Progress Monitoring

rogress Monitoring Progress Monitoring is used to understand student performance and to adjust instruction

How Often We Monitor Progress Depends On How Quickly We Want To Be Able To Make A Decision

TierFrequencyNotes13 x per yearWords Read Correctly (WRC) is the median (middle score) of the three passages. Recorded errors is the median of the three passages1(watch* and transition)WeeklyReview progress after 4 weeks (on the 5th week). Students who do not demonstrate grade level performance enter Tier 2 intervention. 2Every other weekStudents may be monitored more frequently at the discretion of the teacher. Students not demonstrating progress should be monitored more frequently. If tweaks in Tier 2 intervention are not successful in improving performance, a student may move to more intensive Tier 3 intervention. 3WeeklyIndividual progress should be reviewed by at least two staff members every eight weeks. Interventions for students not demonstrating progress should be adjusted in terms of time allocated for intervention, teaching and motivational strategies, grouping arrangements before exchanging materials.

Goal Setting For Progress MonitoringSetting goals is a value driven activityThree MethodsStandards Based GoalsThis type of goal would indicate that a student who meets the expectation is likely to meet standards (either grade level or state test).Norm Referenced Goals This type of goal would indicate that a student who meets the expectation is likely to be successful working with in a group with typical students. Growth Referenced GoalsThis type of goal would indicate that a student who meets the expectation is making progress relative to their own level of achievement, even though it may still be substantially below either normative information or standards.

Type of GoalStrengthsWeaknessesExamplesStandards Baseddesignates what is needed for a student to be successful regardless of what the student can currently doCan be unrealistic for some students with special needsDIBELS Benchmark Targets; Cut scores to the ISAT, No more than 1 ODR per semester can be very powerful when a staff decides that a student has to be proficient and is willing to sacrifice typical practices, schedules, pedagogy and materials to achieve the goalNorm Referencedthat once typical is defined, a target is easy to setCan be unrealistic for some students with special needsRecommendation for using Norm Referenced Goals is to set the goal for the lower bound of average (i.e., the 25th percentile); Classroom comparisoncan be used to maximize compliance with the rules of Least Restrictive EnvironmentGrowth Referencedthat this type of goal is always individualizedwithout using a standard for expected growth, goals can be written to be either too difficult or too easy to attainrequire that the same measure is used during baseline and evaluation phases of the goal

Goal Examples From a High Performing DistrictGrowth Referenced reasonable rate ~ 1.2Normative 25th percentile = 125Standards Based = 115

Goal Examples From a High Performing DistrictNormative 25thStandards Based Growth Referenced

The Decision You Make Regarding Progress Depends on The Goal We SetNormative 25thStandards Based Growth Referenced

BIG IDEA #4The type of goal that we set (i.e., standard, normative, or growth referenced) have different strengths, weaknesses and implications.

Once We Know What We WantWe need to determine how we will monitor (i.e., graph) the dataWe need to set a goalWe need to do monitor the goal

Progress Monitoring with dibels.uoregon.eduDIBELS uses a standards based approach

Progress Monitoring with aimsweb.comAimsweb allows you to decide on an approach

Monitoring Progress with Confidence

Available free from MeasuredEffects.comMeasuredEffects templates are all in Excel and allows you to decide on an approach

There are important implications for the method of evaluation we use to judge the effectiveness of an intervention.

I advocate using confidence intervals when monitoring progress.

SEM and Progress MonitoringStandard Error of Measurement is a real issue with progress monitoring and not just with CBM.The degree of confidence that we have in our decisions is important How confident should we be?Lets look at a progress monitoring graph of a real student .

Upper bound of the 95% Confidence Interval of the slope

Anyone ready to make an instructional decision?Upper bound of the 95% Confidence Interval of the slope

Using most decision rules this would be considered an inadequate response to interventionWhat do you think?Are we ready to make a decision yet?

Beginning with about eight data points we can begin to be confident

Real data have real ups and downs

Different People Advocate For Different Methods of Evaluating ProgressThree (or x) data points (e.g., DIBELS)Slope or trend (Aimsweb)Confidence Intervals (MeasuredEffects)Visual Analysis

The method of evaluating progress that we choose to use will sometimes result in different conclusions. Each method has pros and cons. What is most important is that the decision for how to evaluate progress is decided upon prior to beginning progress monitoring.

Your Turn: Lets Practice Evaluating Progress Monitoring Charts

Do you see progress? You call em

You Decide You should be able to determine how confident you want to be in the effectiveness of your intervention

95% Confidence interval(12.2 points)

90% Confidence interval(10 points)

Being Confident With Progress Monitoringis more than just projecting out the trend linetypically requires far more than 3 data pointsrequires informed decision-making

You can download a template for progress monitoring with confidence from MeasuredEffects.comFile name: DBG_SL_free_V1.02.xls

Progress Monitoring: Math Concepts and Applications

For Secondary Level Students We Can Monitor Mastery of Key VocabularyEspin, C. A., Busch, T., Shin, J., & Kruschwitz, R. (2001). Curriculum-based measures in the content areas: Validity of vocabulary-matching measures as indicators of performance in social studies. Learning Disabilities Research & Practice, 16, 142151.Espin, C. A., & Deno, S. L. (1993a,b). Content specific and general reading disabilities of secondary-level students: Identification and educational relevance. The Journal of Special Education, 27, 321337.Espin, C. A., & Tindal, G. (1998). Curriculum-based measurement for secondary students. In M. R. Shinn (Ed.), Advanced applications of curriculum-based measurement (pp. 214253). New York: Guilford Press.

Espin, C.A., & Foegen, A. (1996). Validity of three general outcome measures for predicting secondary students' performance on content-area tasks. Exceptional Children, 62, 497-514.Espin, C.A., Scierka, B.J., Skare, S., & Halverson, N. (1999). Criterion-related validity of curriculum-based measures in writing for secondary students. Reading and Writing Quarterly, 15, 5-27.Espin, C. A., Shin, J., & Busch, T. W. (2005). Curriculum-based measurement in the content areas: Vocabulary matching as an indicator of progress in social studies learning. Journal of Learning Disabilities, 38, 353-363.Espin, C. A., Busch, T., Shin, J., & Kruschwitz, R. (2001). Curriculum-based measures in the content areas: Validity vocabulary-matching measures as indicators of performance in social studies. Learning Disabilities Research & Practice, 16, 142151.

[VM Example] The Set UpHow many people in this room have played basketball?How do you get better at playing?How many people cheat or know someone who cheats while practicing basketball?For this whole semester, we are going to practice with vocabulary it is practice, you will score your own work

What Does It Look Like? Vocabulary Matching185142017101131722138626122141625

Mastery Monitoring Progress in Vocabulary For Instructional Planning For Groups

Using Group Data For Instructional Decision MakingAlthough all words had not been taught, if a word had been taught and assessed AND a large proportion of students were unable to identify the wordThe teacher revisited, and when necessary retaught the word and surrounding concepts

Mastery Monitoring Progress in Vocabulary For Group ProgressPeriods 1,2,3,6,7,8Periods 1,2,3 only

Mastery Monitoring Progress in Vocabulary For Individual Progress (A Student Included in the General Education Setting)

Mastery Testing / Monitoring For Instructional Planning (Grade 9 INFER)Students goal was to complete 9th grade passages at 80% or better accuracy.Every time a student scored 80% or better, they moved to a higher level.Students graphed their progress through the levels.However, students were encouraged to continue to 10th grade passages if they had time.

Mastery Testing / Monitoring For Instructional Planning (Grade 9 INFER)

Effects of Intervention

Implications of The Source of Criteria We Use

Allocating ResourcesWe know what it should look likeAllocating Resources But what happens whenor?

RANGE of scoresBased on Criterion ReferenceCut scores most (42%) students need more than what is available in the CORE programNumber of studentsIn many schools, it is reasonable to provide Tier 2 (supplemental) instruction to 14% of students.In most schools, it is NOT reasonable to provide Tier 3 (intensive) instruction to 28% of students.The solution: Use Criterion Reference for program evaluation purposes. In this case, it is the CORE program that needs intensification. Big Idea Use A Criterion Reference for Program Evaluation

Implication What would be the implication of using a Cut Scores in this situation?

(i.e., Tier 1 is working for 57% of students ; Tier 2 = 14%, Tier 3 = 28%)

What Would Be the Implication of Using Cut Scores in This Situation? (i.e., Tier 2 = 14%, Tier 3 = 28%)

ThinkThinkThinkThinkThinkThinkThinkThinkPairPairPairPairPairPairShareShareShareShareShare

All students receive CORE instruction, The lowest 10% of students receive intensive interventionsThe next 15% of students receive supplemental interventions

Implication What would be the implication of using a Normative Scores in this situation?

(i.e., Tier 1 is working for 57% of students; Tier 2 = 14%, Tier 3 = 10%)

What Would Be the Implication of Using Cut Scores in This Situation? (i.e., Tier 2 = 15%, Tier 3 = 10%)

ThinkThinkThinkThinkThinkThinkThinkThinkPairPairPairPairPairPairShareShareShareShareShare

All students receive CORE instruction, The lowest 10% of students receive intensive interventionsThe next 15% of students receive supplemental interventionsBUT the CORE needs to be intensified

BIG IDEA #6Use local norms for resource allocation to match student needs with available resources.

The Method of Calculation of Percentile Ranks has Implications

Calculate Percentile RanksPercentiles indicate how many students score At or Below a particular scoreRank = f (below) + n (at)Score = 50 Rank = f (below) + n (at)Rank = 9 + (2)Rank = 10th percentile

50

Calculate NormsYou can use the percentrank method in Excel, but it is not identical to the mid-interval method.

How Are Criterion Reference Cut Scores Determined?

How many words did they read in one minute?Sam read 22Mary read 44Juan read 65Dorothy read 94

Sam read 22Mary read 44Juan read 65Dorothy read 94

Exceed StandardsMeets StandardsBelow StandardsAcademic Warning

in Fall, Dorothy read 94 correct words in a minute She obtained a score of 169 on the state test in the spring

Medical Decision - Making Educational Decision - Making From:To: In medicine indices of diagnostic accuracy help doctors decide who is high-risk and who is not likely at risk for developing a diseaseCan we borrow this technology for tracking adequate growth and educational decision-making

Diagnostic IndicesSensitivity the fraction of those who fail to meet standards who were predicted to fail to meet standardsSpecificity the fraction of students who meet standards who were predicted to meetPositive Predictive Powerthe fraction of students who were predicted not to meet who failed to meet standards Negative Predictive Power the fraction of students who were predicted to meet who met standardsCorrect Classificationthe fraction of students for whom predictions of meeting or not meeting were correct

ISAT Gr. 3Sensitivity considers only students who did not meet standards. As WRC increases sensitivity increases

Specificity considers only students who meet standards. As WRC increases specificity decreasesISAT Gr. 3

Positive Predictive Power considers the fraction of students who scored Less than a particular cutwho did not meet standards. As WRC increases PPV decreases

Negative Predictive Power considers the fraction of students who scored more than a particular cutwho met standards. As WRC increases PPV decreases

Decisions, decisionsHow should we determine where to draw the line?

At 60 we classify 77% of students correctly

Decisions, DecisionsWhy do we have to draw just one line?Maximize Correct ClassificationAdmit the limitations of the tool

Statistical Methods For Group DeterminationLogistic RegressionMaximum likelihood method for predicting the odds of group membership Appears to maximize specificity in cut-score selectionLinear Discriminant Function AnalysisLeast Squares method for predicting the linear relation between variables that best discriminates between groupsAppears to maximize sensitivity in cut score selectionReceiver Operator Characteristic CurvesThe ROC Curve is a plot of Sensitivity Versus 1 Specificity (i.e., False Positives), providing the Area Under the Curve StatisticMinimizes the difference between false positives and false negatives

NPV = 83%

PPV = 77%SENS = 81%Spec = 93%, UNCLASSIFIED(Cut L92 - H112) classifies 78% of students in the data set. Of these students who were classified, 86% were classified correctly, with a rate of 14% error in classification. Note that the 86% classified correctly in this model is 78% of the total group.

The reduction of error in identification comes at a cost of failing to identify risk status for 127 of 565 students.

What to do with the Unclassified student?R-CBM does not attempt to tell us everything about a students reading, it is a strong indicator.Use of convergent data may be able to provide us with a more fine-grained prediction

At 80 NPV = 83%At 80 SENS = 80%At 59 Spec = 94%, UNCLASSIFIED

At 59 PPV = 81%(Cut L59 - H80) classifies 80% of students in the data set. Of these students who were classified, 87% were classified correctly, with a rate of 13% error in classification. The 87% classified correctly in this model is 80% of the total group. The reduction of error in identification comes at a cost of failing to identify risk status for 114 of 565 students.

At 101 NPV = 89%At 101 SENS = 80%At 79 Spec = 93%, UNCLASSIFIED

At 79 PPV = 77%(Cut L79 - H101) classifies 75% of students in the data set. Of these students who were classified, 86% were classified correctly, with a rate of 14% error in classification. Note that the 86% classified correctly in this model is 75% of the total group. The reduction of error in identification comes at a cost of failing to identify risk status for 143 of 565 students.

(Cut L95 - H131) serves to classify 71% of students in the data set. Of these students who were classified, 92% were classified correctly, with a rate of 8% error in classification. The 92% classified correctly in this model is 71% of the total group. The reduction of error in identification comes at a cost of failing to identify risk status for (166 of 565 students).

(Cut L122 - H144) serves to classify 79% of students in the data set. Of these students who were classified, 87% were classified correctly, with a rate of 13% error in classification. Note that the 87% classified correctly in this model is 79% of the total group. The reduction of error in identification comes at a cost of failing to identify risk status for (118 of 565 students).

(Cut L137 - H157) serves to classify 80% of students in the data set. Of these students who were classified, 86% were classified correctly, with a rate of 14% error in classification. The 86% classified correctly in this model is 80% of the total group. The reduction of error in identification comes at a cost of failing to identify risk status for 111 of 565 students.r = .62, R2 = .38

Program EvaluationWhen determining Cut Scores maximizing correct decisions may require admitting the limitations of our assessment technology.

Adult ReadersExamination of aggregated trajectories

BIG IDEA: There are implications for the decisions we make1. With Response to Intervention, service is determined by need, intensity of service is determined by what it takes to be successful. -Special Education Eligibility comes after we determine the intensity, and if that service can not be maintained in the general education setting. 2. Grouping Students. Two factors to consider when grouping students for intervention are convergence of data and the magnitude of the problem. 3. Problem Solving. Solving Problems is based on identifying a focal point for intervention.

BIG IDEA: There are implications for the decisions we make4. Goal Setting. The type of goal that we set (i.e., standard, normative, or growth referenced) have different strengths, weaknesses and implications.5. Progress Monitoring. The method of evaluating progress that we choose to use will sometimes result in different conclusions. Each method has pros and cons. What is most important is that the decision for how to evaluate progress is decided upon prior to beginning progress monitoring.6. Resource Allocation. Use local norms for resource allocation to match student needs with available resources.7. Program Evaluation. When determining Cut Scores maximizing correct decisions may require admitting the limitations of our assessment technology.

**

Documents

Lincolnwood School District 74 MeasuredEffects