Upload
jess
View
28
Download
0
Embed Size (px)
DESCRIPTION
Survey Dilemmas: Determining Sample Size Improving Response Rate Measuring Satisfaction. Daniel R. Judd, Ph.D., M.P.A. Judd Research. 2009 NACEP Member Survey. Why do Surveys?. Collect data from CEP stakeholders to Measure for regulatory requirements, e.g., NACEP accreditation standards - PowerPoint PPT Presentation
Citation preview
Survey Dilemmas:Determining Sample Size Improving Response Rate Measuring Satisfaction
Daniel R. Judd, Ph.D., M.P.A.Judd Research
2009 NACEP Member Survey
% Very Important
1-Developing strong school partnerships 74%2-How concurrent enrollment compares to other accelerated learning options 71%3-Finding the resources needed to build a strong program 58%
4-How to conduct a student survey; how to raise the response rate to surveys. 56%5-The process of becoming accredited—what
to expect 54%
Why do Surveys?
Collect data from CEP stakeholders to
Measure for regulatory requirements, e.g., NACEP accreditation standards
Monitor quality outcomes for improvement
Example: Retention of CEP students
CEPs are being asked to Quantify their
Contribution to the Sponsoring Institution
Does greater involvement in CE increase the likelihood that students will attend the Sponsor Institution?
Credits earned n USU
1-6 credits 60 20%
7-13 credits 52 29%
14-24 credits 41 46%
25-63 credits 47 49%
Comparison Current Status
Study n Work
Vol. Svc
2 yr Coll.
Other4 yr. USU
5-yr. 196 7% 0% 10% 29% 55%
1-yr. 200 7% 16% 13% 30% 35%
How do I Conduct a Survey? Plan in established ethical context – OIR/IRB
Formulate research questions - latent variablesexample: Does CEP affect student development?
Simultaneously, develop instrument & get list Compose tracking strategy – submit to IRB Pilot test instrument and tracking strategy Send out a request for responses – follow-up... Collect and process data Perform analysis – develop a report
AAPOR PROFESSIONAL ETHICS We shall not knowingly make interpretations
of research results that are inconsistent with the data available, nor shall we tacitly permit such interpretations.
We shall not knowingly imply that interpretations should be accorded greater confidence than the data actually warrant.
American Association for Public Opinion Research, Code of Professional Ethics and Practices.
How Many Respondents Do You Really Need?
First Dilemma:
Determining sample size is a
very important issue:Samples that are too large may
waste time, resources, and money, while samples that are
too small may lead to inaccurate results.
Sample Size Formula
Sample size = z 2 * (p) * (1-p) c 2
Where:z = z value (e.g. 1.96 for 95% confidence level) p = percentage picking a choice, expressed as decimal (.5 used for max. )c = confidence interval, expressed as decimal
(e.g., .04 = ±4)
Reduced n because of homogeneity:p = 30%
n=1.96² x .3(1-.3) .05²
n=3.8416 x .21.0025
n =.8068 .0025
n =322.72 ~ 323
p = 80%
n=1.96² x .8(1-.8) .05²
n =3.8416 x .16.0025
n =.61465 .0025
n =245.86 ~ 246
Count Valid Percent
NA 28 2%
Strongly Disagree 13 1%
Disagree 78 6%
Agree 798 57%
Strongly Agree 488 35%
Total Agree 1,286 92%
Total 1,406 100%
I was better prepared
Calculating a Minimum Sample Determine confidence level and interval
95% confidence level confidence interval is the plus-or-minus range, if
question asked of entire population—usually ±5%
Use Internet: sample size calculator surveysystem.com researchinfo.com
Second Dilemma
How Important is the Response Rate?
Dilemma Response rate: the number of eligible sample
units that cooperate in a survey -- has historically been central to survey research in the United States. . . . but the relationship between response rates and survey quality has become much less clear.
Results that show the least bias have turned out, in some cases, to come from surveys with less than optimal response rates. http://www.aapor.org/Response_Rates_An_Overview.htm
NACEP Member Survey - March 2009 Note about Methodology: A notice sent to
all 255 members of the Partnership Listserv at the end of February 2009, directed people to a survey in SurveyMonkey. Reminders were sent 1 week, 2 weeks, and 3 weeks . . .
Response rate: 183 of the 255 members responded, for a 72% response rate.
What works? Grades!! UU grants early access to grades
for completion of online survey = 90% rr
Follow-up. . . by telephone and/or postcard
Parents? (next)
What have you found that works?
Can Parents Answer for Students?
Ranking of Decision Factors - Identical Results
Students Parents
1 Education for better job 1
2 Affordable tuition 2
3 Quality program for chosen career 3
Concurrent Enrollment as a Vehicle for Recruitment & Retention, 2008
Third Dilemma
What is satisfaction? The gap between expectation and experience
Why measure satisfaction? Word of mouth recommendation is powerful.
Who is it best to ask? Students, parents, faculty, HS counselors
Q8. How do you rate the value of NACEP?
Answer Options Count PercentNot very valuable 2 1%
Somewhat valuable 27 15%
Valuable 61 34%
Very valuable 92 51%
Total 182 100%
Comparison Credit and Satisfaction 2006-07 1-Yr. Follow-up StudyUSU credits earned n Excellent Good Fair Poor
1-6 credits60
55% 35%8%
2%
7-13 credits52
52% 40% 2% 6%
14-24 credits41 71% 29% 0 0
25-62 credits47 83% 17% 0 0
Thanks
Complete the Evaluation!