8/18/2019 How Many is Enough

1/6

September 16, 2009 at 3:31 pm (https://kkbiersdorff.wordpress.com/2009/09/16/how‑many‑is‑enough/)   9 comments (https://kkbiersdorff.wordpress.com/2009/09/16/how‑many‑is‑

enough/#comments)

How many people do I need to survey? That’s the most common question I get as a researcher.The answer, unfortunately, is not simple.

The goal of a survey is to give an accurate picture of how people feel about a topic and predictwhat they would do (e.g., buy, vote). In public opinion surveys, we often want to get ananswer that is within a few percentage points of “truth.” A statistic called the standard error of the mean measures this wiggle room and is inversely related to the number of respondents. So

from this perspective, the answer to “How many is enough” depends on how far off we’rewilling to be.

Why response rate matters

In general, there are four sources of error in survey research and three of them relate to thepeople surveyed. (The fourth has to do with problems in the way the questions are designed.)Mistakes can be made in defining the population (e.g., people in special needs schools andinstitutions were not included in the original IQ test standardization). The sampling processmay result in different demographics from the population (e.g., picking numbers at random

from the White Pages eliminates the ~30% with an unlisted number). Or the responders maydiffer from the non‑responders in some important way.

To get an accurate picture, we need to survey a representative sample of the population we areinterested in. If the population of interest is small, we may send surveys to everyone. When wewant to know what the general population thinks, we use random sampling or stratifiedrandom sampling, which sets quotas for the number of people sampled from categories of age,income, gender and/or other demographics. If the sampling process is sound and response rateis acceptably high, we can trust that the people who opt out do not differ in critical ways fromthe people who complete the survey. When the response rate is low, we may question therepresentativeness of the responders and validity of the results.

Measuring response rate

Although response rate is defined as the number of people we surveyed compared to the 

https://kkbiersdorff.wordpress.com/2009/09/16/how-many-is-enough/https://kkbiersdorff.wordpress.com/https://kkbiersdorff.wordpress.com/2009/09/16/how-many-is-enough/https://kkbiersdorff.wordpress.com/

8/18/2019 How Many is Enough

2/6

 number of people we tried to survey, there is more than one way to measure it. The number ofpeople surveyed may include surveys with some answers left blank or only those where everyquestion was answered. The number we tried to survey may include only those known to fitour criterion for eligibility or also those on the mailing or phone list who are not eligible.

In actual fact, what some people report as response rate is actually their cooperation rate (i.e., thenumber of completed surveys compared to the number of people reached). The American

Association for Public Opinion Research (AAPOR), an authority on all things survey‑related,accepts several different formulas for response rate and asks that researchers disclose how theymeasured it in reports.

Survey mode makes a difference

The survey method has an impact on response rate. Generally, e‑mail surveys have a lowerresponse rate than mail surveys, even when access to the Internet is not an issue. For example,in a 2004 survey of university undergrads with e‑mail access, about 21% responded to an e‑mail survey while 31% responded to a mail survey (Kaplowitz et al., 2004 in Public Opinion

Quarterly pp. 94‑101). Face‑to‑face surveys achieve the highest response rates, with the best I’veseen being a whopping 92%. Some studies report that telephone surveys have a higherresponse rate than mail surveys, while others report the reverse. Sending reminders boostsresponse rates. Oddly enough, studies have shown that sending a $2 incentive boosts bothresponse rate and representativeness.

When I said that there is no simple answer to the question of how many is enough, this doesnot mean that people have been unwilling to go on record with a numerical answer. Here aresome expert opinions as to what is considered good or adequate as a mail survey response rate:

25% – Dr. Norman Hertz when asked by the Supreme Court of Arizona30% – R. Allen Reese, manager of the Graduate Research Institute of Hull U. in the UnitedKingdom36% – H. W. Vanderleest (1996) response rate achieved after a reminder38% – in Slovenia where surveys are uncommon50% – Babbie (1990, 1998)60% – Kiess & Bloomquist (1985) to avoid bias by the most happy/unhappy respondentsonly60% – AAPOR study looking at minimum standards for publishability in key journals

70% – Don A. Dillman (1974, 2000)75% – Bailey (1987) cited in Hager et al. (2003 in Nonprofit and Voluntary Sector Quarterly , pp.252‑267)

In addition, various studies described their response rate as “acceptable” at 10%, 54%, and65%, while others on the American Psychological Association website reported caveatsregarding non‑responder differences for studies with 38.9%, 40% and 42% response rates.

It’s enough to make one’s head spin. And population surveys of employees or group membershave different standards for acceptable response rates than the general population. In smallerpopulations such as these, it becomes difficult to compare subsets of the overall group, evenwith a high response rate, because the statistics lack the necessary power.

 

8/18/2019 How Many is Enough

3/6

To make matters worse, there is good evidence that response rates for telephone surveys aredeclining. Reasons for this include increasing concerns about invasion of privacy and misuse ofpersonal information; increased use of call display to filter calls; cynicism (especially amongGen Xers); reduced civic participation in general; and increased requests for surveyparticipation—particularly in satisfaction surveys and program evaluations. (For example, mylast car purchase resulted in three surveys from the dealership and Honda Canada.)

It’s about representativeness

Response rate is not the best way to judge the accuracy of survey results, but representativenessof respondents is. Not all demographic characteristics make a difference. Gary Langer of theABC News Polling Unit described an instance in which repeated surveys gave the samepattern of answers, even though the respondents differed significantly in demographiccharacteristics. The main advantage of a high rate of response is in reducing the possibility of anon‑representative sample.

Entry filed under: Management (https://kkbiersdorff.wordpress.com/category/management/) ,

Research (https://kkbiersdorff.wordpress.com/category/research/). Tags: representativeness(https://kkbiersdorff.wordpress.com/tag/representativeness/), Research(https://kkbiersdorff.wordpress.com/tag/research/), response rate(https://kkbiersdorff.wordpress.com/tag/response‑rate/), survey(https://kkbiersdorff.wordpress.com/tag/survey/).

Social media: Time waster or valuable business tool Finding the “Truth”: Using surveys tofind out what people really think

1. Tina Dole | September 4, 2012 at 11:31 am

I am thinking about participating in paid surveys. I have a few friends that do them andmake pretty good money. It seems like an easy, worthwhile job.

Reply2. kkbiersdorff | September 4, 2012 at 11:55 am

Hi Tina. I respond to surveys for frequent flier points and I can tell you that you will not“make pretty good money.” So if your comment here is just for advertising, you maywant to withdraw it now. While most of the ones I respond to are legitimate and well‑thought out, a few have glaring glitches, while others collect more information than theyneed to be sure you “qualify.”

These surveys use a “stratified random sampling process,” which means that they usethe breakdown of population by gender, age, income and education (at the very least) toensure that the sample characteristics match the population characteristics. So, if theirsample already has enough responses from your gender, age group, etc., you will “not

 

http://-/?-http://www.kkbiersdorff.ca/https://kkbiersdorff.wordpress.com/2009/09/16/how-many-is-enough/?replytocom=54#respondhttp://www.opinionoutpost.com/https://kkbiersdorff.wordpress.com/2009/12/16/surveytruth/https://kkbiersdorff.wordpress.com/2009/05/26/social-media/https://kkbiersdorff.wordpress.com/tag/survey/https://kkbiersdorff.wordpress.com/tag/response-rate/https://kkbiersdorff.wordpress.com/tag/research/https://kkbiersdorff.wordpress.com/tag/representativeness/https://kkbiersdorff.wordpress.com/category/research/https://kkbiersdorff.wordpress.com/category/management/

8/18/2019 How Many is Enough

4/6

qualify” for the remainder of the survey and, as a result, get a minimal payout. On theother hand, you will have expended minimal time, and, as they say, time is money.These surveys have potential flaws. For instance, they are only completed by peoplewho use the Internet and who are willing to do Internet market surveys. If thepopulation whose behaviour you want to predict are Internet users (and Internet surveycompleters), then you are fine. If you are hoping to sell to a market that includes peoplenot addicted to the Internet, then your sample will give you unhelpful information.

They also eliminate people who work for PR and marketing companies and, sometimes,people in those industries who are independent. I understand the reasons, but it doeslimit their sample characteristics.

In short (and when am I ever), while stratified random sampling reduces the number ofrespondents needed, anything that skews the sample from the general population isgoing to have a negative impact on representativeness.

Reply3. Paula J. MacLean | February 11, 2014 at 6:26 pm

Kathleen, do you have a comment on adequate response rates for employers doingemployee satisfaction surveys? The issue I encounter is: at what percentage can wereasonably assume that the results are representative of the majority of employees?

Reply4. kkbiersdorff | February 24, 2014 at 4:23 pm

Again, there is no easy answer, Paula. One question I would ask is how big and diverseis the organization. In a small organization–say 20 employees–you need a really highresponse rate because one person’s response makes a big difference to the total percent.For example, if 10 employees respond and 1 changes his/her mind, the result is a changeof 10% in level of agreement or satisfaction. With 20 respondents, the single individualaffects the overall level of agreement/satisfaction by 5%. Therefore, a small organizationis better off not using survey technologies to gauge employee satisfaction. Focus groups(without supervisors) give better information and increase employee engagement at thesame time (or can, if the organization demonstrates that it has listened to what theysaid).With a larger organization, my answer would depend on its diversity and whether the

leaders want to understand the differences between the different units or departments.Any time you break the large organization into smaller units, you run into the sameissue of small numbers having a big impact on the percentages.If you have a large, non‑diverse organization, you do not need as high a response rate inorder to have interpretable data. For example, if 100 employees respond out of 500 staff(a 20% response rate), I would be just as confident about the results as if 100 employeesrespond out of 1000 staff (a 10% response rate), assuming that both sets of respondentsare representative of the organization’s employees. If, however, managers want to beable to compare results from different departments, then the 100 are divided into

smaller chunks and my analyses start to lose power and each subset is less likely to berepresentative of its department or unit.In general, the people who are most likely to take the time to respond to a satisfactionsurvey are those who are very happy and want you to know it, and those who are veryunhappy and want you to know it. The ones in between may respond if you provide an

 

http://www.kkbiersdorff.ca/https://kkbiersdorff.wordpress.com/2009/09/16/how-many-is-enough/?replytocom=108#respondhttp://www.silvercreekpress.ca/https://kkbiersdorff.wordpress.com/2009/09/16/how-many-is-enough/?replytocom=55#respond

8/18/2019 How Many is Enough

5/6

unrelated incentive to do so, such as the opportunity to win one of your fabulouslyuseful books, Paula. Valued incentives increase both response rate andrepresentativeness.From a statistics standpoint, a sample size (number of respondents) of 30 approximatesa normal distribution. I am typically uncomfortable with fewer respondents unless theorganization is small. Otherwise, the percentage that I am comfortable with decreases asthe size of the organization increases. Ultimately, the only way to ensure

representativeness is through an analysis of the demographics of the organization andyour respondents. There, I would say that age, gender and role or unit in theorganization would be the most important factors to measure. However, asking for jobtitle or unit may make some people hesitate to respond for fear of losing anonymity. It’sa balancing act.

Reply5. Welcome to the National Social Norms Institute. – What is an acceptable survey responserate? | November 12, 2014 at 10:29 am

[…] is from a 2009 web paper by Kathy Biersdorff who is a business consultant in theCalgary […]

Reply6. kkbiersdorff | November 24, 2014 at 6:50 pm

An interesting and updated article for those wanting more information (but notdefinitive answers, of course) on this same topic.

Reply7. Patricia Adams | January 28, 2015 at 7:42 am

I think online surveys are a better way to gain more responses. I have published surveys onsocial media and have received good no.of responses

Reply8. kkbiersdorff | January 28, 2015 at 5:18 pm

The research data say that online surveys have the lowest response rates of any type of

feedback gathering process. If you put an online survey out there on social media, youwill (with luck) get a lot of responses, but the percentage of responses (that is number ofresponses divided by the number of people who had access to the survey times 100) will

 be low because the population using social media is quite high. I am assuming, ofcourse, that the social media you are using is not a closed group but, for instance, allTwitter users. The important consideration is not how many responses you get per se,

 but how representative they are of the population or sub‑population whose behaviouror attitudes you wish to predict. Now let me move from professional to rude andobnoxious. You started by saying “I think” which suggests that you are giving anopinion based on personal experience. That is fine for issues where data have not been

collected extensively. But where sound data exist, it is possible to KNOW, at whichpoint to think something contrary to the data is not smart. I apologize if I was readingmore into you choice of words.

https://kkbiersdorff.wordpress.com/2009/09/16/how-many-is-enough/?replytocom=360#respondhttp://www.kkbiersdorff.ca/https://kkbiersdorff.wordpress.com/2009/09/16/how-many-is-enough/?replytocom=359#respondhttp://www.sogosurvey.com/https://kkbiersdorff.wordpress.com/2009/09/16/how-many-is-enough/?replytocom=331#respondhttp://www.kkbiersdorff.ca/https://kkbiersdorff.wordpress.com/2009/09/16/how-many-is-enough/?replytocom=328#respondhttp://socialnorms.org/what-is-an-acceptable-survey-response-rate/https://kkbiersdorff.wordpress.com/2009/09/16/how-many-is-enough/?replytocom=118#respond

8/18/2019 How Many is Enough

6/6

Reply9. Ronnie | December 29, 2015 at 8:13 am

Nice piece, I have learned much.

Reply

Trackback this post | Subscribe to the comments via RSS Feed

Are you really ready to embrace diversity?Telling your organization’s storyA Picture’s Worth 1000 Words...and a Few StatisticsStatistics You Need to UnderstandConsultation and collaboration in change management

Blog at WordPress.com. The Blix Theme.

https://wordpress.com/themes/blix/https://wordpress.com/?ref=footer_bloghttps://kkbiersdorff.wordpress.com/2013/06/18/consultation-and-collaboration-in-change-management/https://kkbiersdorff.wordpress.com/2013/09/20/statistics-you-need-to-understand/https://kkbiersdorff.wordpress.com/2013/12/20/a-pictures-worth-1000-words-and-a-few-statistics/https://kkbiersdorff.wordpress.com/2014/03/13/telling-your-organizations-story/https://kkbiersdorff.wordpress.com/2014/06/27/are-you-really-ready-to-embrace-diversity/https://kkbiersdorff.wordpress.com/2009/09/16/how-many-is-enough/feed/https://kkbiersdorff.wordpress.com/2009/09/16/how-many-is-enough/trackback/https://kkbiersdorff.wordpress.com/2009/09/16/how-many-is-enough/?replytocom=483#respondhttps://kkbiersdorff.wordpress.com/2009/09/16/how-many-is-enough/?replytocom=360#respond