Upload
melinda-cox
View
219
Download
1
Tags:
Embed Size (px)
Citation preview
Data Burger:Response Rate Improvement
andBasic Quantitative Data Analysis
Nikki DettmarNational Network of Libraries of Medicine
Outreach Evaluation Resource CenterJuly 22, 2014
This webinar is adapted from classes created by Cynthia Olney, PhD, NN/LM Outreach Evaluation Resource Center
Today’s Topics
• Response rate overview:
– What is a ‘good’ response rate?
– Maximizing response rate
– Assessing response rate
• Basic quantitative data analysis– Descriptive statistics
– Levels of data
– Graphs and charts
(Adapted from a class created by Cynthia Olney, PhD,
Outreach Evaluation Resource Center)
Part One:
Response Rate Overview
Why use a questionnaire?
• Learn about characteristics of a group
• Evaluate quality of programs and services
• Document results of programs and services
OERC Resources Questionnaire Design
• Details about questionnaire design:http://nnlm.gov/evaluation/booklets508/bookletThree508.html#st1
• Webinar recording:https://webmeeting.nih.gov/p90742547/
• “Writing Good Questions” handout:http://nnlm.gov/pnr/training/Writing_good_questions.pdf
Questionnaires primarily gather quantitative data
• Exception:– Open-ended questions gather qualitative data
– Information about basic qualitative data analysis is here:
http://nnlm.gov/evaluation/booklets508/bookletThree508.html#qual
Three goals when using questionnaires
Collect just the right amount of data
Write questions that get the information you want
Get a high rate of participation from respondents (response rate)
GOAL 3: GET A HIGH RATE OF PARTICIPATION FROM RESPONDENTS (RESPONSE RATE)
Why?
“A low cooperation or response rate does more damage in rendering a survey's results questionable than a small sample, because there may be no valid way scientifically of inferring the characteristics of the population represented by non-respondents.”
American Association of Public Opinion Research, 2002,Standards and Best Practices
Equation
# of completed and partially completed questionnaires
# of eligible participants in your sample
Response rate defined
What is a “good” response rate?
A rule of thumb from a standard textbook:
• 50% is adequate• 60% is good• 75% is very good
The Practice of Social Research. Earl R. Babbie. Belmont, Calif : Wadsworth Cengage, 2007.
Standard• 5-day • 1997: 36% • 2003: 25%
Rigorous• 21 weeks• 1997: 61%• 2003: 50%
Pew Research Center Experiment*
*Telephone Survey
Response rates have been declining
http://poq.oxfordjournals.org/cgi/content/full/70/5/759
Dillman has found the following principles increase response rates
Dillman, et al., Internet, Mail, and Mixed-Mode Surveys: The Tailored Design Method, Third Edition. Hoboken: Wiley, 2009.
Perceived
CostPerceived reward
Trust Response rate
Here’s how to… Decrease cost
Print questionnaires:
• Make questionnaires as short as possible (1-2 pages)
Online questionnaires
•Make first questions easy to complete•Describe computer actions at point of usage•Use scrolling rather than screen-to-screen format•Show progress bars•Don’t require responses to items
Here’s how to… Increase trust
All questionnaires Use a credible sponsor Have sponsor send out preliminary mailing of endorsement Use a “special” follow-up Use first-class postageOnline questionnaires Use motivational welcome screen (instead of cover letter)
Improving response rates
• Short surveys (1-2 pages)
• Special third contact (e.g., certified mail, telephone call)
• Content meaningful to respondents
• Government sponsorship (as opposed to corporate or marketing firm)
• Survey population (employee, school, military)
• Pre-paid incentives (cash works best)
Improving response rates
http://nnlm.gov/evaluation/booklets508/bookletThree508.html#st2
Effective Mail Procedure
Timing (approx) Activity
Day 1 Pre-notification
Day 5 USPS questionnaire w/stamped return envelope + $2
Day 18 USPS replacement to nonrespondents (thank-you postcard to respondents)
Day 32 FedEx final replacement to nonrespondents(thank-you postcard to respondents)
Adapted from:Don A. Dillman,Washington State UniversityJanuary, 2010
Surface mail more effective than email for healthcare providers
Klabunde, CN, et al. “Facilitators and Barriers to Survey Participation by Physicians: A Call to Action for Researchers.” Evaluation & the Health Professions 2013 36(3):279-295.
http://nnlm.gov/evaluation/booklets508/bookletThreeFigures508.html#F5
Timing (approx) Activity
Day 1 Pre-notification
Day 3 Email with link to questionnaire
Day 9 Email reminder to non-respondents with URL link
Day 13 Email and USPS mail questionnaire to non-respondents
Day 17 Email reminder to non-respondents
Adapted from:Don A. Dillman,Washington State UniversityJanuary, 2010
Example of email/online administration
http://nnlm.gov/evaluation/booklets508/bookletThree508.html#st2
Improving response rates
Pre-notification letter/email:
• Signature that respondents know and trust. Also, for electronic questionnaires, use that person’s name and email address in the “FROM” field if possible.
• Briefly describe the project and why it is important that they respond. (Emphasize how it is important to them as well as how their responses are important to you and your organization.)
• Explain when and how they can expect the questionnaire and who will send it to them.
• Thank them in advance for their help.
Tips for pre-notification letters
http://nnlm.gov/evaluation/booklets508/bookletThreeFigures508.html#F6
Cover letter/email:
• Briefly describe your project and why you want respondents to reply to your questionnaire.
• Include a motivational appeal and explain how to complete the questionnaire.
• Describe who will see their individual responses and how you will maintain confidentiality and any risks to the respondent if he or she chooses to respond.
• Describe incentives.
Tips for cover letters
http://nnlm.gov/evaluation/booklets508/bookletThreeFigures508.html#F6
Checklist for reminder letters/emails:
• State that you are sending a reminder for them to complete the questionnaire.
• Thank those who have already responded.
• Request those who have not responded to do so by a given date.
• For electronic questionnaires, include a link to the questionnaire in every follow-up.
Tips for reminder letters
http://nnlm.gov/evaluation/booklets508/bookletThreeFigures508.html#F6
Incentives work better when sent before or with the questionnaire
Obligation is cheaper than motivation
Cost ofMotivation
Cost ofObligation
• Prepaid cash had strongest effect
• Contributions to charity had little effect
• Lotteries had no effect
Warriner K, Goyder J, Gjertsen PH, McSpurren, K. Charities, no; lotteries, no; cash, yes: Main effects and interactions in a Canadian incentives survey. Public Opinion Quarterly, 1996, 60(4), 452-563 3
Incentives: cash is best
Bigger incentives are not necessarily better
Adapted from:Don A. Dillman, Washington State University, January, 2010
• Use time-tested methods to boost response rates
• Check for bias regardless of response rates
• Use more than one source of information to assess the accuracy of your findings
Summary: Maximizing response rates
http://nnlm.gov/evaluation/booklets508/bookletThree508.html#st2
Use some key tools to assess response rates
Strategy 1 Compare sample profile to population
Strategy 2 Compare early and late responses
Strategy 3 Contact a small percentage of non-respondents with key questions for comparison
• Purposeful sampling: quota sampling
• Triangulate (corroborate) - collect information from multiple sources and look for consistency
More tools to assess response rates
(Adapted from a class created by Cynthia Olney, PhD,
Outreach Evaluation Resource Center)
Part Two:
Basic Quantitative Data Analysis
It helps to approach data analysis in an organized way.
• Gather data (use credible procedures)
• Summarize (reduce and display data)
• Apply findings (pass judgment & make decisions)
http://nnlm.gov/evaluation/booklets508/bookletThree508.html#st3
There is a process that will help you analyze and use data
• Learn the logic of evaluation
• Use data to describe what happened
• Judge value
• Make decisions
Evaluation is a logical process
1. Establish criteria
3. Describe what happened
4. Compare with standards
2. Construct standards
5. Make judgment
6. Make recommendations
Evaluations usually require analysis of quantitative and qualitative data
Quantitative methods for numbers
Qualitative methods for text
Today’s focus
http://nnlm.gov/evaluation/booklets508/bookletThree508.html#ste3
Handy Excel tip: "Format Cells" feature helps you format data
Use Excel Functions to
perform calculations for you. For this, I
used the COUNT function:=COUNTIF(A2:A1675,“x")
Handy Excel tip 2: "Format Cells" feature helps you format the data
Using your mouse,
“right-click” to display this
menu to format a
number as a percentage.
Descriptive statistics summarize distributions of numbers
• Frequencies• Percentages• Charts• Measures of Central Tendency (mean, median, mode)
Frequency distributions show patterns of response
Yes No Don’t Know
Blank
Frequency 52 35 8 5
Percent 52/100= 52%
35/100=35%
8/100=8%
5/100 = 5%
Valid Percent
52/95=55%
35/95 =37%
8/95 = 8%
*
Would you attend an instructional session on Survey Monkey? (N=100)
Frequencies help to "clean" data and check for vulnerabilities
• Outlier responses• Socially desirable
responses• Low or high response rate
to response options • Low response rate for
certain questions
Bar charts display comparisons across categories
Would you attend an instructional session on Survey Monkey?
Bar charts can compare subgroups
Do you need easy-to-read patient informationIn the following languages?
English Spanish Other0%
20%
40%
60%
80%
100%
Clinic 1Clinic 2Clinic 3
Line charts also can show changes over time for multiple factors
0
50
100
150
200
Winter Spring Summer Fall
People Trained in Year 1
Consumers
Health professionals
Pie charts show sub-group breakdown of a larger group
(demographics)
In which type of library do you work? (check one)
23%
20%
16%
17%
22%Hospital
HSC
Public
Academic
Other
Visual comparisons in pie charts are less obvious than in bar charts
In which type of library do you work? (check one)
Hospital
HSC
Public
Academic
Other Hospital HSC Public Academic Other0%
5%
10%
15%
20%
25%
30%
Hospital
HSC
Public
Academic
Other
Measures of central tendency are the "most representative" number in a data
set
• Mean (average)• Median (middle)• Mode (most frequent)
The mean (average) is the sum of all scores divided by total number of
responses
Years Employed at Library X
Person 1 8
Person 2 12
Person 3 11
Person 4 9
Mean (average) years of service to Library X
10
Computation: 8 + 12 + 11 + 9 = 40 40 years total/4 people = 10
The median (middle) is the number that divides a distribution equally
Distribution 1
Distribution 2
Distribution 3
4 5 4
5 6 4
5 7 5
5 7 6
6 8 8
7 24 10
7
5 7 5.5 =(5+6)/2Median
The mode (most frequent) can be found on any frequency table
In what state do you work? (N=200)
DE NJ NY PA Other
Frequency 24 28 24 54 70
Percent 12% 14% 12% 27% 35%
Handy Excel tip: Calculate measures of central tendency in a column or a row
• Mean (average)=AVERAGE(A2:A7)
• Median (middle) =MEDIAN(A2:A7)
• Mode (most frequent)=MODE(A2:A7)
The type (level) of data determines how you summarize it
Nominal:(Label)
Ordinal:(Ranking)
Interval/Ratio:(Equal intervals)
Nominal Data (Labels)Nominal Data:
Labels
"Type of institution" is an example of nominal data (labels)
Hospital Clinic Library Other
# Participants 18 42 19 5
% 21% 50% 23% 6%
Mode
Bar graphs usually provide the best visual representation of nominal data
Hospital Clinic Library Other 0%
10%
20%
30%
40%
50%
60%
21%
50%
23%
6%
Participants’ Work Settings
These statistics and charts are most typically used for
nominal data analysis
Nominal dataFrequenciesPercentagesModeBar charts
Rank the following features for their influence over your decision to attending [Your Profession’s Important Conference], with “1” a the most influential and “5” as the least influential.
__ Registration cost__ Location__ Hotel accommodations__ CE offerings__ Networking opportunities
Rankings are “relative” to the list of items to be ranked
Rank the following features for their influence over your decision to attend [your profession’s national conference]
Resp’t1
rank
Resp’t2
rank
Resp’t3
rank
Resp’t4
rank
Resp’t5
rank
MedianRank
Registration fee 2 5 1 5 4 4
Location 1 3 2 1 2 2
Hotel 5 1 4 4 5 4
CE offerings 3 2 5 3 3 3
Networking 4 4 3 2 1 3
Median rankings summarize ranked data
Rank the following features for their influence over your decision to attend [your profession’s national conference]
Resp’t1
Resp’t2
Resp’t3
Resp’t4
Resp’t5
% with“Top 3”
Registration fee 2 5 1 5 4 40%
Location 1 3 2 1 2 100%
Hotel 5 1 4 4 5 20%
CE offerings 3 2 5 3 3 80%
Networking 4 4 3 2 1 60%
*Percentages for demonstration purposes only. Do not report percentages on groups of less than 50.
You can also report percentage of “Top 3”
“Top Rankings” percentages can be charted
Registration fee
Location Hotel CE Offerings Networking0%
20%
40%
60%
80%
100%
120%
40%
100%
20%
80%
60%
Percentage of Respondents Giving “Top 3” Rank for Each Item
Report the following statistics for ordinal data (rankings)
Median ranks per item Percent of "Top 3" Tables of items in
descending order of importance
Interval/ratio data (equal distances) allow comparisons and computation
Health Fair Location Visitors
Community Center 300
High School 150
YWCA 150
Mean = 600/3 = 200
You can chart interval-level data
Community Center High School YWCA 0
50
100
150
200
250
300
350
300
150 150
Visitors to Health Fair LocationsF
req
ue
nc
ies
Number of visits to a web site is a typical example of interval data
(equal distances)
Visits to NN/LM WebsiteMonth # VisitsAugust 2008 140,027July 2008 167,445June 2008 180,605May 2008 241,052April 2008 203,084March 2008 203,390
Mean = 189,267
Line charts are appropriate for monthly statistics.
Mar-08 Apr-08 May-08 Jun-08 Jul-08 Aug-080
50,000
100,000
150,000
200,000
250,000
300,000
# Visits to the NN/LM Web Site by Month
# Visits
Rate the following features for their influence over your decision to attend [your profession’s national conference]
No Influence
MuchInfluence
Registration fee 1 2 3 4 5
Location 1 2 3 4 5
Hotel 1 2 3 4 5
CE offerings 1 2 3 4 5
Networking 1 2 3 4 5
How do ratings compare to rankings?
Which is the most representative: mean or median?
How would you rate this instructor? (N=20)
High=4 3 2 Low=1
Fr. Count 11 7 1 1
Mean = 3.4
Median = 4
Web Site Visits
Month # Visits
August 2008 275
July 2008 167,445
June 2008 180,605
May 2008 241,052
April 2008 203,084
Mean 158,492Median 180,605
When in doubt, use median or report mean and median
"Outliers" affect means but not medians
Report the following statistics to describe interval data (equal distances)
Frequencies and percentages
Mean or medianBar charts of ranges
ORLine charts for means
at different points in time
Qualitative analysis is beyond the scope of this webinar
It's like déjà vu all over again-- Yogi Berra
Review:Evaluation is a logical process
1. Establish criteria
3. Describe what happened
4. Compare with standards
2. Construct standards
5. Make judgment
6. Make recommendations
Handy reporting tip: Stephanie Evergreen
http://stephanieevergreen.com
For More Information:
Planning and Evaluating Health Information Outreach – Series of 3 booklets:
http://nnlm.gov/evaluation/guides.html#A2
1. Getting Started with Community-Based Outreach
2. Including Evaluation in Outreach Project Planning
3. Collecting and Analyzing Evaluation Data
Contact Information
Nikki Dettmar
Evaluation Librarian
206- 543-3409
Cindy Olney
Acting Assistant Director
678-682-3864
NN/LM OUTREACH EVALUATION
RESOURCE CENTER
OERC website:http://nnlm.gov/evaluation/
Booklets website: http://nnlm.gov/evaluation/guides.html
OERC tools:http://guides.nnlm.gov/oerc/tools