MichiganScience No. 6 � �

Understanding Public Opinion

Surveys

w w w . M i c h i g a n S c i e n c e O n l i n e . o r g � �

the first known opinion poll was apresidential“strawvote”conductedbytheHarrisburgPennsylvanianin�8��. The newspaper simply sur-

veyed500orsoresidentsofWilmington,Del., without regard to demographics.TheresultsshowedAndrewJacksonlead-ingJohnQuincyAdamsby66percentto��percent.�(WilliamCrawford,secretaryof thetreasury,andHenryClay, speakeroftheU.S.HouseofRepresentatives,alsowerecandidates.)

The poll accurately predicted theoutcome of the popular vote, but noneofthecandidatesactuallywonamajority

of the electoral vote. Consequently, theelection was decided by the House ofRepresentatives,whichchoseAdams.�

Election polling was popularized inthe �0th century by the Literary Digest,a weekly magazine published by Funkand Wagnalls. Beginning with the re-electionofWoodrowWilsonin�9�6,theDigest correctly predicted the result ofevery presidential campaign until �9�6.Its polling consisted of mailing millionsof postcard “ballots” nationwide (alongwithasubscriptionform)andtallyingtheresponses. Many considered the Digestpolltobeunassailable.

Soitseemeduntil�9�6,whentheDi-gest forecast a landslide win by AlfredLandonoverFranklinRoosevelt.�Incon-trast,ayoungGeorgeGalluppredictedaRooseveltvictorybasedona“representa-tivesample”of50,000people—thatis,agroupthatresembledthepopulationatlarge.Yet hisprediction waswidely ridi-culedasnaive.�

Inhindsight, theDigest’serror iseasytospot:Itsmailinglistwascomprisedofhouseholds with telephones, cars andmagazine subscriptions. In the midst oftheGreatDepression,thosewithmoneytospareforsuchrelativeluxurieshardlyrepresentedthevoterswhowouldfavorRooseveltandhisNewDeal.

The Literary Digest folded soon afterRoosevelt’selection.Whatwastruethenis true today: A flawed sample cannotbe corrected simply by increasing thenumberofrespondents.

The Gallup organization suffered itsown embarrassment in �9�8, when it

ByDianeS.Katz,NathanielEhrlich,Ph.D.,andLarryHembroff,Ph.D.

Election season is upon us, and

there’s no shortage of polls telling us what we think. But judging the accuracy of all these numbers requires a basic understanding of the science of polling.

Understanding Public Opinion

SurveysYes

No

Undecided Did not vote

MichiganScience No. 6 � �

predictedavictorybyThomasDeweyoverHarryTruman. Gallup claimed the error resulted fromending his polling three weeks before ElectionDay.Othersfaultedhisuseof“quotasampling,”bywhichrespondentsweretargetedbasedongen-der,age,raceandincome.Thistypeofsamplingcan be useful when comparing differences ofopinion between specific demographic groups;it is not accurate, however, for estimating howmanypeoplewithinanentirepopulationholdaparticularviewpoint.

The use of quota sampling for elections waslargelyreplacedafter�9�8by“randomsampling,”in which all segments of the population have areasonablyequalchanceofbeingsurveyed.

SAMPLE METHODS Whereas the Literary Digest used postcards

and Gallup posed questions face-to-face, mostpolls today are conducted by telephone. Callsaretypicallymadeby“randomdigitdialing,” theprocess by which computers generate wirelinetelephone numbers. This method improves thechances of reaching households with unlistednumbers or new service. But the substitutionof cell phones for traditional land lines amongsome segments of the population has latelyraisedquestionsabouttheabilityofrandomdigitdialingtoproduceanunbiasedsample.

The term “sampling error” refers to instancesinwhichthepollresultsfromaparticularsamplediffer from the relevant population just bychance. For example, a majority of respondentsin a particular sample might have supportedAlfred Landon even if a majority of the overallpopulationsupportedFranklinRoosevelt.

To achieve more accurate findings, pollstersadjust their results to more closely reflect keydemographiccharacteristicsoftheoverallpop-ulation. For example, if a survey is written tomeasurepoliticalviews,and if�0percentofallregistered voters at the time of the survey areDemocrats,theresultswillgenerallybemorere-flectiveoftheentirepopulationifthesampleis“weighted” so that Democrats effectively makeup �0 percent of the responses to any givenquestion.

In the following table, the “unweighted”column contains the number of respondents inthe sample according to their party affiliation.

The“weighted”column,incontrast,containsthenumberofrespondentsafterweightsareappliedtomoreaccuratelyreflecttheactualpopulation.

Unweighted Weighted

Total Republicans 358 328

Total Democrats 366 388

Total Independents 412 420

Total Respondents 1,136 1,136

Surveyresultsbasedonsamplesthathavebeenheavily weighted are generally considered lessreliable.Consequently, it ishelpfultoknowhowasamplehasbeenweightedinordertojudgetheprecisionofthepoll.

Most polls are reported with their“margin oferror.”This is an estimate of the extent to whichthe survey results would vary if the same pollwererepeatedmultipletimes.Ifapollsterreportsthat 75 percent of voters are dissatisfied withCongress, a ±� percent margin of error usuallymeansthatthereisa95percentchancethatthetruemeasureofvoterdissatisfactionfallsbetween7�percentand78percent.

The margin of error accounts only for apotential error in the random sampling, not forsurvey bias or miscalculations. The margin oferroristypicallyformulatedbasedononeofthree“levelsof confidence:”99 percent,95 percentor90percent.A99percentlevelofconfidence(themost conservative) indicates that the surveyresultswouldbe“true”withinthemarginoferror99percentofthetime.

Incomparingtheresultsoftwoormoresurveys,it is important to recognize that the margin oferror that would apply to both sets of resultstakentogetherwouldbegreaterthanthemarginoferrorforeitherofthesurveysalone.

SURvEy WORDINgThewordingofquestionsiscrucialtointerpret-

ing poll results. Individual responses often de-pend on the way in which survey questions areposed.Thethreequestionsbelow,developedbyresearchersatMichiganStateUniversity,demon-stratehowwordingcanaffectpollresults.5

�.DoesMichiganspendtoomuch,toolittleortherightamountoncorrections?

�.Is$�.�billionspentoncorrectionsannuallytoomuch,toolittle,ortherightamount?

w w w . M i c h i g a n S c i e n c e O n l i n e . o r g � 5

�.Isspending$��,700perprisonerannuallytoomuch,toolittleortherightamount?

From a budgetary standpoint, expendituresof $��,700 per prisoner and $�.� billion in totalare equivalent. Yet public opinion about theappropriatenessofeachfigurevariesremarkably,asdoestheresultwhennofigureiscited.

Similarly, it is helpful to know the range ofresponses to understand fully the poll results.For example, in the table of hypothetical databelow, it could be accurately reported that�5 percent of Americans have an unfavorableimpressionoftheFederalEmergencyManage-ment Agency. In the absence of informationabout the range of responses, readers couldeasily conclude that 55 percent of Americanshave a favorable opinion. An entirely differentimpressionwouldbeconveyedwereitreportedthatlessthanoneinfivesaytheirimpressionofFEMAisfavorable.

Very Favorable ........................... 2%Somewhat Favorable .............17%Somewhat unfavorable ........20%Very unfavorable ....................25%haven’t heard enough...........31%don’t Know/not applicable ... 5%

Ideally,allofasurvey’swordingandresponseswill be made available to the public when thesurvey results are published. When survey lan-guageisnotmadepublic,however,itisusefultoknowwhosponsoredthepolltohelpjudgethesurvey’sreliability.

RESPONSE BIASSurvey results may also be affected by “re-

sponsebias,” inwhich respondents’answersdo

different Wording elicits Varying responses

No Figure $1.3 Billion $23,700

80%

60

40

20

notreflecttheiractualbeliefs.Thiscanoccurforavarietyofreasons:• The“BandwagonEffect,”inwhichrespondents

basetheiranswers—intentionallyorotherwise—onadesiretobeassociatedwiththeleadingcandidateorcause.

• The“UnderdogEffect,”inwhichrespondentsformulatetheiropinionsoutofsympathyforthecandidateorthecausethatappearstobetrailinginsupport.

• The“SpiralofSilence,”inwhichrespondentsfeelundersocialpressuretogiveonlyapoliticallycorrectanswerratherthantheiractualopinion.

CONCLUSIONWhenconductedproperlyandunderstoodcor-

rectly,pollsprovideausefulsnapshotofpopularopinion. If manipulated or misinterpreted, pollscandistortreality.Voterswoulddowelltoevalu-atepollsbasedonthefollowing�0questions.6

�. Whopaidforthepoll?�. Howmanypeoplewereinterviewedforthe

survey?�. Howwerethosepeoplechosen?�. Aretheresultsbasedontheanswersofall

thepeopleinterviewed?5. Whenwasthepolldone?6. Howweretheinterviewsconducted?7. Whatisthemarginoferrorforthepollresults?8. Whatisthelevelofconfidenceforthepoll

results?9. Whatquestionswereasked?�0.Whatotherpollshavebeendoneonthis

topic?Dotheysaythesamething?Iftheyaredifferent,whyaretheydifferent?

1 Terry Madonna and Michael Young, “Politically Uncorrected: The First Political Poll,” Franklin & Marshall College, Lancaster, Pa. Available on World Wide Web: http://www.fandm.edu/x3905.xml. 2 Ibid. 3 The U.S. Survey Course on the Web, “Landon in a Landslide: The Poll That Changed Polling,” City University of New York and George Mason University. Available on World Wide Web: http://historymatters.gmu.edu/d/5168. 4 Sally Sievers, “The Infamous Literary Digest Poll, and the Election of 1936,” Wells College, Aurora, N.Y. Available on World Wide Web: http://aurora.wells.edu/~srs/Math151-Fall02/Litdigest.htm. 5 Institute for Public Policy and Social Research, 1997. State of the State Survey, “Attitudes Toward Crime and Criminal Justice: What You Find Depends on What You Ask,” No. 97-20, Michigan State University, East Lansing, Mich. Available on World Wide Web: http://ippsr.msu.edu/Publications/bp9720.pdf. 6 Adapted from Sheldon R. Gawiser and G. Evans Witt, “20 Questions a Journalist Should Ask About Poll Results,” National Council on Public Polls. Available on World Wide Web: http://www.ncpp.org/?q=node/4.

Too Much Too Little Right Amount