Upload
others
View
2
Download
0
Embed Size (px)
Citation preview
>www.dpw.state.pa.us >www.dhs.pa.gov
Getting the Most out of Data Collection and Analysis:
Part 3- Analyzing Graphed Data to Inform Next Steps
Amy Alford, M.Ed., BCBAClinical ConsultantDecember 14, 2017
>www.dpw.state.pa.us >www.dhs.pa.gov
• Brief recap from parts 1 & 2 of this series• The 5W’s of data analysis• Visual instead of statistical analysis• Using the data during team meetings• Using the data to make informed, instructional decisions
Agenda
>www.dpw.state.pa.us >www.dhs.pa.gov
PART 1 RECAP
>www.dpw.state.pa.us >www.dhs.pa.gov
• The goal for all data collection is to capture quality evidence that allows analysis to lead to the formulation of convincing and credible answers to the questions that have been posed.
• Data collection is more than just a frequency count• Needs > Goals/Objectives > Data Collection Tool• Refine and individualize your data sheet• Training others to correctly use the data sheet will yield
better data• As a professional in this field, you have a responsibility to
collect data- No more excuses!
Recap
>www.dpw.state.pa.us >www.dhs.pa.gov
PART 2 RECAP
>www.dpw.state.pa.us >www.dhs.pa.gov
• There are 2 parts to graphing in Excel- 1) setting up the spreadsheet and 2) Data Entry
• Let the data sheet drive Excel formatting (and vice versa)• Choosing to NOT graph and analyze data is NOT acceptable• Use what you know, learn what you don’t Stay relevant and current with changing technologies
• Excel is only ONE program out there- don’t limit yourself if you have access to other programs
• Data collected (and analyzed) should ALWAYS ask more questions
Recap
>www.dpw.state.pa.us >www.dhs.pa.gov
THE 5 W’S OF DATA ANALYSIS
>www.dpw.state.pa.us >www.dhs.pa.gov
• Who (should do the analysis)
The person who designed the data collection process (may or may not be those who actually collected the data) OR someone who has a skill set to analyze (may depend on intent)
• What (does data analysis mean)
Count, Mean/Median/Mode, Slope/Level/Variability, rate, identify patterns over time, comparing results to previously determined benchmarks, correlations, statistical significance
• Where (does data analysis take place)
Excel (most common) but many other tools to utilize exist (statistical software, apps on phone/iPad/computer, paper/pen), during team meetings, written in reports
The 5 W’s of Data Analysis
>www.dpw.state.pa.us >www.dhs.pa.gov
• When/Why (do we analyze data)
Ideally, analysis should begin early and continue often (begin with the end in mind, frame questions to answer as you go along)
Prior to implementing a new program (e.g. to determine baseline, develop hypotheses)
During implementation (to make tweaks to instruction as you go along)
After implementation (to ensure what you did continues to work as intended)
• How (do we analyze data)
Visually Statistically
The 5 W’s of Data Analysis
>www.dpw.state.pa.us >www.dhs.pa.gov
VISUAL INSTEAD OF STATISTICAL ANALYSIS
>www.dpw.state.pa.us >www.dhs.pa.gov
• It is relatively easy to learn• Most common program (Excel) is readily available • Easy to disseminate final products • Helps show the patterns without needing to read a novel• Makes data manipulation more efficient to get to the
intended goal• Provides a clear communication tool (for other staff,
participants, families)
Why do we use visual analysis?
11
>www.dpw.state.pa.us >www.dhs.pa.gov
• Technically, yes…BUT• Beyond formal research studies or analyzing “big data” it is
not something necessary to make informed decisions and plan for next steps
• Stats involves an advanced skill set (and many sleepless nights) and are tricky to interpret
• Advanced computer software often necessary• In our daily work, statistical analysis is not necessary and not
a good use of our time on a daily basis.
Wouldn’t statistical analysis make for a stronger argument?
Correlation does not equal causation!
>www.dpw.state.pa.us >www.dhs.pa.gov
• Three characteristics of graphed data used to analyze and summarize information:
a) Level
b) Slope
c) Variability
Guidelines For Visual Analysis
14
>www.dpw.state.pa.us >www.dhs.pa.gov
• Refers to the average level of performance across a period of time.
• Often use the mean or arithmetic average to reflect this value, but we can also use median performance.
– Median is especially important when there is an unusual outlier in your data (for example, an individual has a significant life event that greatly impacts their performance on a single day).
• Important to review any changes that occur immediately after a program or strategy is implemented or modified
– Does it produce an immediate step (up or down) in behavior or skill?
Level
15
>www.dpw.state.pa.us >www.dhs.pa.gov 160 2 4 6 8 12 14 16 181020
30
40
50
60
70
80
90
100
Successive Days
Example: Level
The difference betweenthese two lines is thechange in level.
16
>www.dpw.state.pa.us >www.dhs.pa.gov
• Refers to trend over time (rate of change)• Many ways to estimate and draw trend from a data set.
– it takes at least 3 and usually 6 data points to establish a trend that is meaningful.
– Not necessary to get too concerned about single days of performance that are unusual from other data points.
– In Excel it’s as simple and clicking a button to “add trend line”
• How fast is performance changing over time?• Look at the graph and judge the general direction in which a
participant’s performance is: – Increasing (accelerating), Decreasing (decelerating) or Staying the same
Slope
17
>www.dpw.state.pa.us >www.dhs.pa.gov 18
Example: Slope/Trend
0
2
4
6
8
10
12
14
16
18
1 2 3 4 5 6
Sessions
# of
exc
essi
ve v
erba
lizat
ions
18
>www.dpw.state.pa.us >www.dhs.pa.gov
• This refers to the “bounce” (up and down movement) in the data to reveal stability or consistency). Too much bounce may be a concern
• Causes of bounce may include – personal characteristics of the individual (i.e., the “bad day”), changes in difficulty in the
activities, changes in how the data were collected (I.e., yesterday the staff collected data during the small group activity, today individually), and a host of other unknown factors.
– We expect a reasonable amount of “bounce” in the data but if the variability becomes too great, we need to try to understand WHY.
– We might need to examine environmental factors, and other variables as to what is causing the problem. Regardless, once we determine there is too much bounce in the data we need to put things in place that can control the bounce.
• Scan the data and make judgment about the degree of variability, ranging from high to low.
Variability
20
21
0 2 4 6 8 12 14 1610
10
20
3040
50
Successive Days
Occ
urre
nce
Example: Variability
60
70
80
90
>www.dpw.state.pa.us >www.dhs.pa.gov
Beyond Level, Slope, and Variability
• Average (across time• Count (whole number or percentage)• Themes, patterns across qualitative data• Compare against previous graphs on the same variables• Drill down the data further; Ask others for insight • Geographical Setting/Location patterns
>www.dpw.state.pa.us >www.dhs.pa.gov
EXAMPLES
0
2
4
6
8
10
12
14
16
18
20
22
6-May 8-May 10-May 13-May 15-May 3-Jun 5-Jun 7-Jun 12-Jun 14-Jun 21-Jun 26-Jun 28-Jun
# ti
mes
eac
h be
havi
or o
ccur
red
Dates Behaviors Occurred at Work
Jon- Hitting and Yelling Behaviors at Work- June, 2013Hitting Yelling
Baseline BSP Implementation-
0
10
20
30
40
50
6-May 8-May 10-May 13-May 15-May 3-Jun 5-Jun 7-Jun 12-Jun 14-Jun 21-Jun 26-Jun 28-Jun
Dur
atio
n of
beh
avio
ral o
ccur
renc
es (m
inut
es)
Dates Behaviors Occurred
Jon- Duration of Behavioral Occurrences at Work- June, 2013
Baseline BSP Implementation
0
2
4
6
8
10
12
14
16
18
# ti
mes
eac
h be
havi
or o
ccur
red
Dates Behaviors Occurred at Work
Jon- Hitting and Yelling Behaviors at Work- July and August, 2013Hitting Yelling
June July August
27
>www.dpw.state.pa.us >www.dhs.pa.gov 28
0
5
10
15
20
25
30
35
Antecedents Behaviors Consequences
Freq
uenc
y of
Occ
urre
nce
George- Observation Summary of Data Collection at Day Program
>www.dpw.state.pa.us >www.dhs.pa.gov
USING DATA DURING TEAM MEETINGS
>www.dpw.state.pa.us >www.dhs.pa.gov
• Take time to learn baseline: You can’t decide where to go next if you don’t know
where you first came from. It may not always be clear how someone is doing
right now until the data is presented Behaviors occur “all the time” should be a red flag to
dig deep
• Share results (or partial results) as soon as possible• Ask open-ended questions• Don’t forget about the indirect data!
Sharing the Results
31
>www.dpw.state.pa.us >www.dhs.pa.gov
• Revisit the intent/purpose- Why are you here?• What is important to the behavior?• Are you capturing setting events or other factors that play a
role in that person’s day (e.g. sleep, medication refusals, eating habits, daily schedule, other people)
• Can simple coding be done to eliminate writing and not take up as much space
• What are others already collecting (e.g. sleep patterns, incidents, call-logs with family, medications)
• What do you think happened here? Why? Is that typical?
Questions to Engage the Team
32
>www.dpw.state.pa.us >www.dhs.pa.gov
• Share small pieces of data to evoke conversation amongst team members, without YOU telling the whole story
• Plant the seed, then take notes of the dialogue• Engage team ongoing, not at the end• “Slice the data” in real time (advanced skill- can be done
using pivot tables)• Link areas of need to goals (current or new)
Additional Strategies to Encourage Team Collaboration
>www.dpw.state.pa.us >www.dhs.pa.gov
EXAMPLES
What questions would you ask?
35
What about now?
36
What if this was the first graph shared with the team?
37
Would this be helpful?
38
0
2
4
6
8
10
12
14
16
18
20
Person 1 Person 2 Person 3 Person 4 Person 5 Average
MAS
Fre
quen
cy S
core
sGeorge- Negative Verbalization Behavior at Day Program-
Motivational Assessment Summary data across staff
Sensory Escape Attention Tangible
>www.dpw.state.pa.us >www.dhs.pa.gov
USING DATA TO MAKE INFORMED, INSTRUCTIONAL DECISIONS
Data Analysis Decision
Progress-Trend is going up
Make no changes. Continue instruction.If trend is very steep then consider making instruction more difficult to facilitate greater learning.
Mastery-Met criterion
Develop a new plan to maintain and extend performance (e.g., fluency, generalization)
Minimal Progress-Shallow trend-Error patterns
Change Instruction-THINK!Change prompting strategy-Address motivation-Simplify or modify instruction for problem steps/skills-Teach a more simple version of the skill
No Progress-No improvement since baseline-No independent correct responses (may vary depending upon prompting strategy)-No improvement since the last data decision
Change Instruction-THINK!-Simplify skill-Change prompting strategyContinue Instruction and Observe-When progress in other aspects of instruction is noted (e.g., progress within the prompting system).
Regression-Trend is going down
Change Instruction-THINK!What could be the problem?-motivation (e.g., hate you, hate skill, skill not functional or meaningful )
Progress
Progress is being made as the time engaged in activity is going up.
Instructional decision: continue instruction.
Mastery Fluency – speed or ease with which skill is performedSome strategies for promoting fluency:- manipulate reinforcement (e.g., reinforce when dishwasher is loaded versus reinforcing each correct step of the task analysis)-load dishwasher after every meal instead of just after dinner
Generalization- being able to perform the skill in situations other than the “training” situationSome examples:-load dishwasher at volunteer site (senior center)-load dishwasher on Sundays after having lunch with grandparents
Minimal Progress Instructional Decisions:
• Change prompting strategy to include a direct verbal prompt since progress is not being noted using an indirect verbal prompt.
• Errors consistently occurring on steps 6, 7, and 9
• Consider if motivation is a factor and where it can be increased
No ProgressInstructional Decisions:
• Select different activities if lack of interest/motivation is problematic
• Chose a simpler activity if it is too difficult
• Change person who does the activity with participant
RegressionInstructional Decisions:
• Revise the task analysis to simplify, currently too many steps. Decreased steps from 15 to 10.
• Participant expressed dislike of meals currently making, increase motivation by allowing participant to choose what meals to make
>www.dpw.state.pa.us >www.dhs.pa.gov
• Where should supports be targeted?• When is the best time to teach the skill? And, under what
conditions?• Does there need to be a staffing change?• How can we arrange the environment based upon what is
working (or not) right now• Are there medical underpinnings that first need to be
addressed?• If unclear, is there a need for more data? Are we missing the
perspective of a person who may have better insight?
Additional Informed Decisions To Consider
>www.dpw.state.pa.us >www.dhs.pa.gov
FINAL THOUGHTS
>www.dpw.state.pa.us >www.dhs.pa.gov
• Collecting data is good, but doing something with it is necessary
• You don’t need to be a statistician to analyze data (but you should have a few tricks up your sleeve to dig deep into the data)
• Don’t get discouraged by inconclusive results- go back and collect more until the picture is clear
• Sharing analyzed data with teams and opening up the discussion will yield powerful insight you may have otherwise missed
• Thinking about the analysis from the beginning is key
Final Thoughts
We’ve only just begun…
>www.dpw.state.pa.us >www.dhs.pa.gov
Thank you for joining the Data Series!