Evaluation: Practical Guidelines - UK Research and Innovation...4.2 Quantitative research 15 4.3 Qualitative research 18 4.4 Other options 20 4.5 Confidentiality 21. 5 Data Handling

Evaluation: PracticalGuidelines

A guide for evaluating public engagement activities

Evaluation: Practical Guidelines i

Contents

1 Introduction 1

1.1 Usingthisguide 1

2 WhatisEvaluation? 2

2.1 Evaluation 2

2.2 Monitoring 2

2.3 Thedifferentaspectsofevaluation 2

3 BuildinganEvaluationStrategy 4

3.1 Settingaims 4

3.2 Settingobjectives 5

3.3 Choosingyourevaluationtools 7

3.4 Howmuchevaluation? 12

3.5 In-houseorindependent? 13

3.6 Ifyourunintotrouble 13

4 GatheringData–ToolsandTechniques 15

4.1 Introduction 15

4.2 Quantitativeresearch 15

4.3 Qualitativeresearch 18

4.4 Otheroptions 20

4.5 Confidentiality 21

5 DataHandling–ToolsandTechniques 23

5.1 Quantitativedata 23

5.2 Qualitativedata 24

5.3 Observationaldata 25

6 Reporting 27

6.1 Reporting 27

6.2 Reportstructure 27

ii Evaluation: Practical Guidelines

Annex1EvaluationQuestionnairesandQuestions 30

Questionnairedesign 30

Questiondesign 32

Annex2Questionnairemodules 36

Annex3SummativeEvaluationSchema 40

MonitoringData 40

EvaluationData 40

Annex4Glossary 41

Annex5Furtherreading 44

Findinginformationonevaluation 44

EvaluationMethods 44

WebResources 44

Evaluation: Practical Guidelines 1

1Introduction

Thisguideisforresearchersseekingtoengagegeneralaudienceswiththeirsubjecttoevaluatepublicengagementactivities,regardlessofpriorexperienceofeitherpublicengagementorevaluation.Itisintendedtohelpresearchersfromanydiscipline.Foramoregeneralguidetoundertakingpublicengagementactivities,see:http://www.vitae.ac.uk/CMS/files/upload/The_engaging_researcher_2010.pdf

Evaluationtechniquesarebasedonsocialandmarketresearchmethods,soeventhoughsomematerialwillbefamiliartosomereaders,thequestionsposedandthethoughtprocessessuggestedwillberelevanttoallthoserunningandevaluatingpublicengagementactivities.

Theevaluationapproachescoveredinthisguidearesuitableforarangeofactivitiesincluding,butbynomeanslimitedto:

• Publiclectures,talksanddebates(bothliveandon-lineusingnewmedia).

• Shows.

• Festivals.

• Activitiesforschoolstudentsand/orteachers(realorvirtual).

• Websites,CD-ROMsandothervirtualactivities.

• Exhibitionsandhands-onevents,includingworkingwithmuseumsandgalleries.

• Opendays.

• Activitiesthatinvolvemembersofthepublicinresearch.

Evaluationstrategiesshouldbeintegraltotheactivitydesignprocess.Iftheactivityinvolvessubmittinggrantapplicationstofundit,youmayfindithelpfultotalktopotentialfundersabouttheirevaluationrequirements,whilstpreparingyourapplication.

Thereisnomagicformulaforevaluation.Toconstructanevaluationstrategy,youneedtothinkaboutyourobjectives,thedatayoucancollectandthereportsyouhavetomake.Donewell,evaluationwillimproveyouractivityandgiveyouasenseofachievementasyouwillunderstandmoreabouttheimpactofwhatyouhavedone.

1.1UsingthisguideTheremainderofthisguidehasfivemainsectionsandaseriesofannexes.

Section2Introducesevaluationanddifferentiatesitfrommonitoring.

Section3Looksatbuildinganevaluationstrategy.Developingyourevaluationstrategyasanembeddedpartoftheactivitywillhelpyoutogetabetterunderstandingofwhattheactivitycanachieveandshouldhelpyouinmakingapplicationsforfunding.Itisthisstagewhichtendstocausethemostdifficultyasyoureconcileyourenthusiasmwithresourceconstraintsandestablishrealisticobjectives.

Section4Focusesondatacollection.Thissectiondescribesdifferenttechniques,whattheycandoandhowyoucanmakethemworktogether.

Section5Givesguidanceonhowtoanalysethedatayouhavecollected.

Section6Looksathowtodraftyourreport.

Annex1Providessomedetailedadviceonconstructingquestionnairesandphrasingquestions.

Annex2Providessomequestionnairemoduleswithspecificquestionsyoucoulduseinyourevaluation.

Annex3Providesanoverviewofwhichtechniquestousedependingontheinformationyouwant.

Annex4Isaglossaryofterms.

Annex5Suggestssomefurtherresourcesontheweb.

http://www.vitae.ac.uk/CMS/files/upload/The_engaging_researcher_2010.pdf


2 Evaluation: Practical Guidelines

WhatisEvaluation? 22.1EvaluationEvaluationissaidtobe“veryimportant”,yetprojectmanagerstendtobefrightenedofevaluationbecausetheyseeitasatestandathreat.Evaluationisaprocessthattakesplacebefore,duringandafteranactivity.Itincludeslookingatthequalityofthecontent,thedeliveryprocessandtheimpactoftheactivityorprogrammeontheaudience(s)orparticipants.

Takingtheopportunitytounderstandwhetheryouachievedwhatyousetoutto,howwellyoudidit,whatimpactyouractivityhashadandtoreflectcriticallyonboththeactivitiesandprocesseswillbenefityouandyouraudiences.Thisknowledgecanbeusedinternallybyyourteamtodriveimprovementandexternallytodemonstrateachievements.

2.2MonitoringThesamebasictoolsforgatheringandanalysingdatacanbeusedforcollectingbothevaluationandmonitoringinformation.Asaresult,thereisoftenconfusionbetweenmonitoringandevaluation.

Monitoringisaboutcountingthingsandensuringyouractivityisontrack.Thingslikenumbersofevents,audiencenumbersandnumbersofCDROMsdistributed,etc.aremonitoringdata,notevaluationdata.

2.3ThedifferentaspectsofevaluationTherearethreeaimsofevaluation:

• Tosupportthedevelopmentofyouractivity(formativeevaluation).

• Toensureyoumanageitbetternexttime(evaluationofyourprocesses).

• Toassessthefinalimpactofyouractivity(summativeevaluation).

2.3.1Formativeevaluation

Youshoulduseformativeevaluationduringthedevelopmentoftheactivitytotestideas,concepts,venues,timingsandprototypesonrepresentativesoftheaudience.Thiswillhelpyouassesswhatsort

of‘productanddeliverychannel’isgoingtobemosteffectiveatreachingandengagingyourtargetaudiences.Youmayneedtotestthingsoutwithasampleaudienceseveraltimesduringyouractivity’sdevelopment;thissortofresearchisacentralpartofanevaluationstrategy.

Theemphasisatthisstageislikelytobeondiscussion-basedtools(qualitative research)thatwillgiveyouanin-depthunderstandingofyouraudience(s).Ifyouaredesigninganexhibitorawebsite,watchingpeopleinteractingwithaprototypecanprovideusefulinputforfinalisingit.Qualitativeinputatthisstagecanbecrucialtounderstandinghowtochangeyouractivitytoimproveitsappeal.Tomakeimprovements,it’snogoodsimplyknowingpeopledidn’tlikeitunlessyouknowwhattheydidn’tlikeandunderstandwhytheydidn’tlikeit.

However,don’tover-eggthepudding.Youneedtothinkabouthowmuch,ifany,formativeevaluationisreallyneededasitwilladdtimeandcosttoyouractivity.Ifyouareusingatriedandtestedformula,suchasadebateformat,youmaynotneedtoincludeformativeevaluationinyouractivityplan.Ifyouhaveanydoubtabouthowsomethingwillbereceivedbyyouraudience,testit.

2.3.2Evaluationofprocesses

Youshouldevaluatetheprocessofmanaginganddeliveringyouractivity.Thiswillhelpyoutodoitbetternexttime.Thisinformationisusefultopeersandcolleagues,andwherepossibleshouldbesharedwiththewiderpublicengagementcommunitysothattheycanlearnfromyourexperiences.

Formanyactivities,thistypeofevaluationcanbehandledentirelybytheprojectteamandissimilartoconductingaresearchproject,oryoumaywantexternalhelp,seesection 3.5.Ifthereareanumberofpeopleinvolved,makesureyouhavescheduledintotheworkprogrammeopportunitiestotalkthroughprogressandanydifficultiesorissuesthatarearising.Don’tforgettoincludethistimewhenyouarecostinganactivity.Itwillbetimewellspentandwhenpresentedaspartofyouroverallevaluationstrategy,willshow


2

Remember,summativeevaluationismainlyaboutimpacts.Thismeansthatthecountingofoutputs,suchasnumberofpeopleatanevent,numberofhitsonawebsite,etc.isnotpartofyourevaluation.Thisdataisimportantandyoushouldbegatheringit,butstrictlyspeakingthisismonitoringdata,notevaluationdata.

2.3.4Benchmarkingandbaselines

Itsoundsobvious,butifyou’retryingtochangesomething,thenyouneedtoknowthestateofaffairsbeforepeopleinteractedwithyouractivity,sothatyoucanseeiftherehasbeenachangeafterwards.

peoplemakingfundingdecisionsthatyouknowwhatisinvolvedinrunninganactivity.

2.3.3Summativeevaluation

Summativeevaluationisthetypeofevaluationwithwhichpeoplearemostfamiliar.Thislooksatoutcomesandmeasureswhetherornotyouhavemetyouraimsandhadanyimpactontheaudience.Youshouldaskquestionssuchas:

• Howmuchdidtheaudienceenjoyyouractivity?

• Diditchangepeople’sunderstanding/knowledgeorattitudes?

• Hasitinfluencedtheiractions/behaviour?

• Howbiganimpactdidithaveonthosewhoengaged?

Sotheemphasisislikelytobeonnumericaldata,butdepthofunderstandingcanbeimportant,especiallyifyou’reinterestedinwhatpeoplelearntorwhatmessagetheytookaway.Qualitativedatacanalsobecrucialinexplainingwhatliesbehindyourquantitative data.Thereisnoreasonwhyquestionnairescannotaskquestionsthatwillhelpyoutounderstand‘why’peoplegivecertainresponses.Thisisespeciallytrueifyouhavequalitativeresearch,perhapsfromyourformativeevaluation,onwhichtobaseyourquestions.

Youneedtothinkverycarefullyabouttheimpactsyou’reaimingtoachieve,becauseinordertoevaluateimpactstheymustbemeasurable.Youneedtothinkabouttherealisticlevelofimpactthatyoucanmakeandthepracticalitiesofidentifyingthatimpact.


3BuildinganEvaluationStrategy

Thissectionwillhelpyouplanyourevaluationandsupportanyapplicationsforfunding.

Table3.1Thissectioncovers:

Buildinganevaluationstrategyrequiresmovingthroughhighlevelaims,toobjectivesetting,identifyingindicatorsandmeasures,andselectionofappropriatetoolstogatherdataagainsttheindicatorsandmeasures.Thischapterofferssometoolsandwaysofthinkingthatwillsupportyouinthisprocess.

3.1SettingaimsThefirstthingyouneedtodoistoclarifyyouraim(s).Whatdoyouwanttoachieve?Whatisthepurposeofyouractivity?Thisisbigpicturestuff.Thereareseveralreasonswhypeopleandorganisationswanttodevelopactivitiestocommunicateresearchtoawideraudience.TheScienceforAll1reportpublishedbytheDepartmentforBusiness,InnovationandSkillsinFebruary2010,mapsthemyriadofmotivationsresearchershaveforwantingtoengagewiththepublicandsummarisesthemasfollows,to:

• winsupportfor‘science’,whichincludesallfieldsofacademicresearch,

• maketheworldabetterplace,

• developskillsandinspirelearning,

• enhancemycareer,

• createamoreefficient,dynamicandsustainableeconomy,

• increasethequalityandimpactofmyresearch,

• enhancesocialcohesionanddemocraticparticipation,

• beethical,accountableandtransparent.

TheResearchCouncilsincreasinglyexpectsomelevelofpublicengagementbytheresearcherstheyfund,seethe“ConcordatforEngagingthePublicwithResearch”(http://www.rcuk.ac.uk/per/Pages/Concordat.aspx).YoumightwanttocheckwhetheryourfunderhassignedtheConcordat.

TheResearchCouncilsviewpublicengagementwithresearchasapossiblewayofenabling,and/ordemonstratingthatyouareconsideringtheimpactofyourresearch,seePathwaystoImpact(http://www.rcuk.ac.uk/kei/impacts/Pages/home.aspx).Otherresearchfundersmayalsorequire,encourage,orofferfinancialsupportfor,publicengagementwork.

YourinstitutionmaybecommittedtopublicengagementandbeasignatorytotheNationalCo-ordinatingCentreforPublicEngagement’s“ManifestoforPublicEngagement”.http://www.publicengagement.ac.uk/why-does-it-matter/manifesto

TheHigherEducationFundingCouncilforEnglandisconsideringhowtoincludetheimpactofresearchbeyondacademicimpactintotheResearchExcellenceFramework.Foranup-dateonhowthisisprogressingseehttp://www.hefce.ac.uk/research/ref/

Whateveryourdrivingforce,havingaclearaimwillhelpyoutostarttheprocessofdevelopingboththeactivityandanevaluationstrategy.

1Thedefinitionof‘science’usedintheBISScienceandSocietystrategyis“deliberatelywide,encompassingphysical,biological,engineering,medical,naturalandsocialdisciplines,includingresearchintheartsandhumanities”.http://interactive.bis.gov.uk/scienceandsociety/site/defining-science/

Sub-section Contents

Settingaims

Settingobjectives

Beingclearaboutthedifferencebetweenaimsandobjectivesandensuringthateacharerealistic.

Choosingyourevaluationtools

Identifyingthedatathatyouwillneedtocollect,andthetoolswithwhichtocollectit,todemonstratewhetheryouhaveachievedyourobjectives.

Howmuchevaluation Beingclearaboutwhatyoucanevaluate.

In-houseorindependent

Canyoumanagetheevaluationyouthinkyouractivityneedsandyourfunderswant?

Ifyourunintotrouble Whattodoifyouractivitystartstogowrong.

http://www.publicengagement.ac.uk/why-does-it-matter/manifesto

http://www.publicengagement.ac.uk/why-does-it-matter/manifesto

http://interactive.bis.gov.uk/scienceandsociety/site/defining-science/

http://interactive.bis.gov.uk/scienceandsociety/site/defining-science/


33.2SettingobjectivesNextyouneedtosetobjectives–thesearethethingsyouneedtodotoachieveyouraim.Eachofthemotivatingfactorslistedabovewillyielddifferenttypesofobjectives.Forexample,ifyouwanttoenhancedemocraticparticipationyouwillsetdifferentobjectivesfromsomeonewhowantstoenhancetheircareer(and/orthatoftheircolleagues).Anactivitywiththefirstaimwillbefocusedonmakinganimpactontheexternalaudience,whilstanactivitywiththesecondaimmayfocusonmakinganimpactonpeoplewithinyourownorganisation.However,havingaimsthatrelatetochangingthingswithinyourownorganisationanddevelopingtheskillsofyouandyourcolleaguesarevalidanditisimportanttobearthisinmindasyoustarttosetobjectives.

Settingobjectivesisanart.Attheoutset,funders(whetherinternalorexternalprovidersofresources)willwanttoseethatyou’regoingtogivethemvaluefortheirmoney.However,youractivitywillbeevaluatedagainstyourobjectives,soitisimportanttoberealisticwhensettingthem.Trytokeepobjectivessimple,ideallyasinglekeyresultwillshowwhetheryouhavesucceeded.Ifanobjectiveiscomplicated,askyourselfwhetheritneedstobebrokendownintotwoormoresimplerobjectives.

Figure3.1Linkingyourobjectivestoyouraimandanyotherrequirements

Goodobjectivesettinghelpsyouthinkthroughnotjusttheevaluationstrategy,butthewholeprocessofrunningtheactivity.

MakeyourobjectivesSMART:

• Specific

• Measurable

• Achievable

• Relevant

• Time-bound

SMARThasbecomeawellknownacronym,butworkingthroughthelistinthisordermaynotbethebestwaytosetaboutdevelopingyourobjectives.Whenconstructingevaluationframeworks,trySRATMinstead.

ThetablebelowsetsoutsomeofthequestionsyouneedtothinkaboutwhenconstructingSMARTobjectivesusingtheSRATMorder.

Table3.2SMARTquestions

Your aim

Funder Objective

Stakeholder Objective

Activity Objective 2



SpecificWhatexactlydoyouwanttodo,withor

forwhom?

Relevant

Willachievingthisobjectivecontributeto

thedeliveryofyouroverallaimand

supportyour/yourfunders’/yourinstitution’s

goals?

Achievable

Istheobjectiveachievable?Inparticularcan

yougetitdoneinthetimeyouhave

available,withinyourbudgetandwithinthe

prevailingpolitical/institutionalclimate?

Time-bound

Whendoyouwanttoachievethisobjective

and/orwhendoyouthinkyouwillbe

abletoachievethisobjective?

MeasurableCanyoumeasurewhetherornotyouhave

achievedtheobjective?


33.2.1Specific

Aspecificobjectivehasamuchgreaterchanceofbeingaccomplishedthanageneralgoal.Tosetaspecificobjectiveyouneedtoanswerthesix“W”questionsbelow:

• Why:Specificreasons,purposeorbenefitsofaccomplishingtheobjective.

• Who:Whoneedstobeinvolved?

• Where:Identifyalocation,whichmaybevirtual.

• What:WhatarethetasksIneedtoaccomplish?

• Which:Identifyrequirementsandconstraints.

• When:Establishatimeframe.

Anobjectivethatisspecificwillusuallyhaveasingleresult,whichmeanstherewillbeanobservableaction,behaviourorachievementthatcanbedescribed.

3.2.2Relevant

Toberelevanttheobjectivemustbesomethingyoucanactuallydothatwillhelpyoutoachieveyouraim,takingintoaccounttheresourcesyouhaveforyouractivityincludingfinancial,personnelandtimeresources.Atthispointre-considerwhetherornotthespecificsyouhaveidentifiedaregenuinelyrelevanttoyouroverallaim.Iftheyare,fine,ifnotgobackandrefinethemsothattheyare.

So,ifyouraimisto“win support for my research field”thenall ofyourobjectivesmusthelpyoutoachievethisaim.

3.2.3Achievable

Itisnogoodhavingobjectivesthatyoucan’tachievegiventheresourcesyouhaveforyouractivity.Makingsureyourobjectivesareachievablemakesyouthinkabouthowmuchmoneyyouneed,whatskillsyouandyourteamneedandhowlongitwilltaketoplanandruntheactivity.Peopletendtoforgetaboutthetimeinvolvedandresearchhasshownthatresearcherssaytheygrosslyunderestimatedthetimeinvolved2.Youmaywanttolookforapartnerwhocanprovidefinancialor‘inkind’support,forexample,freeroomhireorsomeonewhohasaskillyouoryourteamlacks.

Inconsideringwhethereachobjectiveisachievableyoushouldthinkaboutthebarriersthatstandbetweenyouandachievingyourobjective.Thiswillhelpwithcontingencyplanning.Askyourself:‘whatif?’Onebarriermaybetheamountoftimethatyouhavetodelivertheproject.Aone-offengagementmayhavelessimpactthanasustainedseriesofcontacts,althoughonetrulyinspiringsessionislikelytohaveamorepositiveeffectthanaseriesofratherdullones.

Mostimportantly,don’tsetyourselfuptofail.Objectivesshouldbeambitious,buttheyshouldalsomotivatetheteamandshouldthereforebeachievable.Also,anyfunderwilljudgeyourcapabilitytoorganisetheactivitybywhetherornotyourobjectivesarefeasible.

3.2.4Time

Foreachobjectivetherehastobeatimescaleforachievingit.However,juststatingadateisnotenough,youneedtoaskyourself:

• Haveyouleftenoughtimetogeteverythingdone?

• Willothercompetingdemandsonyourtimeorthatofothersinvolved,causedelay?

Addressingthesequestionshelpstoshowthefundamentallinkbetweentimeandachievability.

3.2.5Measurable

Youneedtoestablishhowyouwillmeasureprogresstowardtheattainmentofeachobjectiveyouset.Youneedtothinkabouthowyouwillknowwhethertheobjectivehasbeen,orisbeing,achieved.Whataretheindicators?Thenyouneedtothinkabouthowyouwillmeasureeachindicator.Todetermineifyourobjectiveismeasurable,askquestionssuchasHowmuch?Howmany?Rememberthough,whatgetsmeasured,getsdone.So,becarefulthatthecapacitytomeasuredoesnotstarttodominateattheexpenseofrelevance.Seesection 3.4.

2Partnership for Public Awareness – Good practice guide,PeopleScience&PolicyLtd,EngineeringandPhysicalSciencesResearchCouncil,June2003www.epsrc.ac.uk/funding/grants/pe/ppe/Pages/goodpractice.aspx

www.epsrc.ac.uk/funding/grants/pe/ppe/Pages/goodpractice.aspx

www.epsrc.ac.uk/funding/grants/pe/ppe/Pages/goodpractice.aspx


3Youmaydecidethatinordertoachieveyouraimthereareintermediarystepsyouneedtoundertake,suchasunderstandingwhatpeoplecurrentlythinkorundertakingsimpleawarenessraisingpriortodevelopingsupportforyourresearch.Thefollowingfourobjectivescouldallsupporttheachievementofthisaim:

• Todevelopanunderstandingofwhatpeoplethinkaboutmyresearchwithinthenextyear.

• Togenerateawarenessofmyresearchinthelocalcommunitywithinthenextyear.

• That75%ofallparticipantsintheactivitiesruninthenextyearreportthattheyenjoyedtheactivities.

• Toincreasethenumberofstudentparticipantsconsideringacareerinresearchattheendoftheproject.

Thenextsectionusesthesefourobjectivestoshowhowthenatureofyourobjectivescanhelpyoutoidentifyappropriatesourcesofevaluationdata.

3.3ChoosingyourevaluationtoolsSection 4ofthisguidedescribesdifferentevaluationtoolsinsomedetail,butitisimportanttogiveearlythoughttothenatureofthetoolsthatyouarelikelytoneed.Carefulsettingofobjectiveswillhavegivenyouideasaboutwhichtoolsyouwillneedtouseasyouhavethoughtabouthowtodemonstratewhetherornotyouachieveeachofyourobjectives.Theexamplesbelowgivearangeofdifferentdatasourcesforsomeindicators–youdon’thavetousethemall.

3.3.1Evaluationtable

Anevaluationtablesupportsyouinbuildingonthethoughtprocessesusedwhenyoudefinedyourobjectivestoidentifythetypesofdatasources,andthustheevaluationtools,thatyouwillneedtousetoassesswhetherornotyouhavebeenabletomeetyourobjectives.Overleafisanevaluationtableforeachoftheobjectivessetoutin3.2.6above.

Onceyouhaveclearobjectives,youhavethemeanstofocusyourefforts,sotimespentatthisstagewillnotonlyhelpyourevaluation,butalsoyourprojectmanagement,byprovidingameanstoprioritiseyourresources.

3.2.6Fromaimtoobjectives–anexample

Fromthevariousaimsinsection3.1wehaveselectedthefollowingasastartingpoint:

Aim:toinspirepeopleaboutmyresearchfield.

Thissub-sectiondevelopssomepossibleobjectivesforthisaim.Indevelopingobjectivesyouneedtothinkabouthowyouwillrecognisethatpeoplehavebeeninspired.Thismaybedemonstratedbywiderpublicappreciationthatyouaredoingsomethingworthwhile(orlessoppositiontoyourresearch)orbyincreasednumbersofstudentsstudyingyoursubject;itmighteveninfluencetheprioritiesofresearchfunders.Withinthisoneaimyoucouldhavemanyactivitiesforverydifferentsetsofpeople,soindevelopingyourSMARTobjectivesyouneedtothinkabout:

• Targetgroup(s).

• Theresourcesyouhaveavailablefortheactivity.

• Yourtimescale.

• Thecapacityyouhavefordatacollectionwithinanevaluation.


3assetoutintheexamplebelow,butcouldalsobegroupsdefinedbygeography(wheretheylive),socio-demographicfeaturessuchasage,genderorethnicity,orexistingattitudesforexampleproresearch,scepticalofauthorityetc.

Todevelopanunderstandingofwhatpeoplethinkaboutmyresearchwithinthenextyear.

Youmaybeinterestedintheviewsofthegeneralpublicasawhole,orparticularsub-setsofthepublic.Thesub-setsmaybeprofessionalorstakeholdergroups

Table3.3(a)Fromobjectivestodatasources

Objectives Indicators Measures DataSources

Todevelopanunderstandingofwhatpeoplethinkaboutmyresearchwithinthenextyear

Publicperspectives

Viewsofspecificsetsofpeoplee.g.professionalgroupA

Viewsofspecificsetsofpeoplee.g.patientgroupB

Viewsofspecificsetsofpeoplee.g.historyteachers

Percentageawareofmyresearchfield.

Percentageunderstandingmyfield.

Percentagewithpositive/negativeviewsaboutmyfield.

Factorsunderpinningtheperspectives.




PercentageofgroupAthatareawareof/understand/positivetowards/negativetowardsmyresearch.

PercentageofgroupBthatareawareof/understand/positivetowards/negativetowardsmyresearch.

Percentageofhistoryteacherswhoareawareof/understand/positivetowards/negativetowardsmyresearch.

Nationallyrepresentativesurvey.

RepresentativesurveyofprofessionalgroupA.

RepresentativesurveyofpatientgroupB.

Surveyofhistoryteachers.

Groupdiscussions.

Groupdiscussions.

Groupdiscussions.

Groupdiscussions.

In-depthinterviews.

Interviews.

Interviews.

Interviews.

Socialmedia/discussionlistsusedbyprofessionalgroupA.

Socialmedia/discussionlistsusedbypatientgroupB.

Socialmedia/discussionlistsusedbypatientgroupB.


3

Astheabovetableshows,ifyousimplywanttoknowwhetherthereisanyawarenessandwhetheryouractivitieshavecontributedtothisawareness,youcanuseeitherqualitativeorquantitativetechniques.If

however,youwanttoknowhowmuchawarenessthereisyouneedrepresentativequantitativedataandifyouwanttoknowaboutchangesovertimeyouneedrepeatedcollectionsofquantitativedata.

evensomeone),thelessyoulikeit(them).However,youcannotinspireanyoneiftheyarenotawareofyourresearch.

Togenerateawarenessofmyresearchinthelocalcommunitywithinthenextyear.

Buildingawarenessdoesnotnecessarilymean‘inspiring’.Sometimesthemoreyouknowaboutsomething(or

Table3.3(b)Fromobjectivestodatasources


Togenerateawarenessofmyresearchinlocalcommunitywithinthenextyear

Awarenessofresearch

Awarenessofresearchprocesses

Roleofyouractivityingeneratingawareness

Percentageofpeople,withinthelocalcommunity,awareofyourresearch.

Percentageincreaseinpeople,withinthelocalcommunity,whoareawareofyourresearch.

Percentageincreaseinpeople,withinthelocalcommunity,whounderstandyourresearchobjectivesandprocesses.

People,withinthelocalcommunity,becameawareofyourresearchthroughwiderdisseminationofyouractivity.

Percentageofpeople,withinthelocalcommunity,whounderstandyourresearchobjectivesandprocesses.

People,withinthelocalcommunity,becameawareofyourresearchthroughdirectinteractionwithyouractivity.

Representativesurveyoflocalcommunity.

Representativesurveyoflocalcommunity.

Surveyoflocalcommunity.

Surveyoflocalcommunity.

Representativesurveysoflocalcommunity(timeseries).

Representativesurveysoflocalcommunity(timeseries).

Groupdiscussions.

Groupdiscussions.

Socialmedia.

Socialmedia.


3

Youcanmeasureperformanceagainstthisobjectiveonlybyusingquantitative data.However,youmayalsowishtofindoutwhypeopledidordidnotenjoyyouractivitiessothatyoucanimprovethisaspectofyourperformance.Thiscanbecoveredinsurveys,butyoucouldalsouseobservationalorqualitativetoolstobuild

adeeperunderstanding.Thisisthesortofareawhereyoucanmakeyourevaluationworkontwolevels,bothformativeandsummative.

3http://www.epsrc.ac.uk/SiteCollectionDocuments/Publications/reports/EaBWreport.pdf http://www.ytouring.org.uk/productions/archive/mtg2005/evaluation.html

enjoymentalsoactstodriveotherimpacts.Evaluationshavebeenundertakenwhereenjoyment(orthelackofit)stronglycorrelatedwiththedevelopmentofmorepositive(ornegative)attitudesandenjoymenthasbeenlinkedtothedegreetowhichknowledgeisacquired,aswellasthedegreetowhichattitudesareinfluenced3.

Youmayneedtothinkabouthow‘enjoyment’mightmanifestitselfandaskaboutthataswellasadirectquestionaboutlevelofenjoyment.Forexample,enjoymentmightbemanifestedasadesiretorepeattheexperienceortorecommendtheactivitytoafriend.

75%ofallparticipantsintheactivitiesruninthenextyearreportthattheyenjoyedtheactivities.

Enjoymentofactivitiescanbeveryimportant.Ifyou(andyourcolleagues)haveinvestedtimeandquitepossiblyadditionalmoneyinanactivityandtheaudience/participantshavegivenuptheirtimetotakepart,ensuringthatallthesecontributorsenjoytheactivityisnobadthing.Inadditiontothefairlyobviousfactthatenjoymentislikelytoleadtofurtherparticipation(anddon’tforgetthatthisappliesasmuchtoyouandyourcolleaguesastoanyoneelse)

Table3.3(c)Fromobjectivestodatasources


That75%ofallparticipantsintheactivitiesruninthenextyearreportthattheyenjoyedtheactivities.

Viewsof“lay”participants

Viewsof“expert”orstakeholderparticipants

Viewsofyourteam

Proportionoflayparticipantsreportingenjoyment.

Proportionofexpertorstakeholderparticipantsreportingenjoyment.

Proportionofyourteamwhoenjoyedtakingpart.

Numberandnatureoffactorsaffectingenjoyment.



Representativesurveyofparticipants.



Observation.

Observation.

Observation.

Focusgroups.

Focusgroups.

Focusgroups.

Interviews.

Interviews.

Interviews.

http://www.epsrc.ac.uk/SiteCollectionDocuments/Publications/reports/EaBWreport.pdf

http://www.epsrc.ac.uk/SiteCollectionDocuments/Publications/reports/EaBWreport.pdf

http://www.ytouring.org.uk/productions/archive/mtg2005/evaluation.html

http://www.ytouring.org.uk/productions/archive/mtg2005/evaluation.html


3along-termcommitmenttotheevaluation(andquitepossiblytotheactivityaswell).Ifyouaresettinganobjectiveofthistype,youneedtobesurethatitispracticaltoassesswhetherornotyouhaveachievedit,aswellaswhetheritisachievableinitsownright.Oftenthepracticaloptionistouseintentiontostudyorthechoicesthatstudentsmakeearlierintheirstudiesasshorter-termproxyindicators,asinthisexample.

Toincreasethenumberofstudentparticipantsconsideringacareerinresearchattheendoftheproject.Thiscouldbeeitheracrossthegeneralpopulationoramongstspecificgroups.

Anactivitywithaspecificobjectiveassociatedwithrecruitmentwouldneedtohavethescopetotrackparticipatingyoungpeopleovertimeinordertoassesswhetherornotthisobjectiveisachieved.Thisimplies

Table3.3(d)Fromobjectivestodatasources


Toincreasethenumberofstudentparticipantsconsideringacareerinresearchattheendoftheproject.

Intentions

Choices

Numberofstudentsstatingintentionstostudyrelevantsubjects.

Numberofstudentschoosingtostudyrelevantsubjectsatage14.

Factorsaffectingchoicesatage14and16.

Numberofstudentschoosingtostudyrelevantsubjectsatage16.

Factorsaffectingchoicesatage14and16.

Baselinesurveysofparticipants.

Follow-upsurveysofparticipants.



Managementdatafromschools.

Managementdatafromschools.

Focusgroups.

Focusgroups.




3

Figure3.2providesasimilaroutputtotable3.3(c),butthedifferenceistheprocessofdevelopingtheoutput.Table3.3(c)wasdevelopedusingadirectionalcausalchain,whereasindevelopingthemindmap,thelinksaremorelikelytobeputinplaceafterthevariouscomponentshavebeenidentified.

3.4Howmuchevaluation?Theresourcesavailablewilldeterminewhatsortofevaluationyoucando.Generallyevaluationisonelinewithinanoverallactivitybudget,eachcomponentofwhichyouarelikelytobeunderpressuretoreduce.

tosettingobjectives,identifyingindicatorsandmeasuresandworkingoutthetoolsyouneedtousetocollecttherelevantinformationmaybemoreappropriateforyou.

3.3.2 Mindmapping

Someresearcherswillbemorecomfortablewithamorefluidapproach.Takingamind-mappingapproach

Figure3.2Fromobjectivestodatasources

AimWinsupport

formyresearch

That75%ofallparticipants

intheactivitiesruninthenextyearreportthattheyenjoyedthe

activities.

Feedbackfrom“expert”orstakeholder

participants

Feedbackyourteam

IndicatorA

IndicatorB

IndicatorC

IndicatorD

IndicatorE

IndicatorF

Feedbackfrom“lay”participants

Objective4

Objective3

Objective2

Measure1

FocusGroups

Observations

Survey

Interviews

Measure2

Measure1 Measure2

Factors

Proportion


3Evaluationisnotalwaysseenascontributingdirectlytotheactivityandcanthereforeberegardedasalowerpriority.Todefenditsplace,therehavebeencallstosetaspecificportionofbudgetsasideforevaluation,10%oftheactivitybudgetisoftenquoted.Thisisrarelysensible.Inthecaseofsmallactivitiesthiswouldmeanthatyoucandoalmostnothing,yetformuchlargeractivitiesasmallerfractionmayprovideampleresources.

Ratherthansetanarbitraryproportionofthebudget,itismuchbettertothinkaboutwhatinformationyouneed,howitcanbecollectedandanalysedandthentoconsiderwhethertheeffortandcostofthisworkisproportionatetotheactivity.Ifitisnot,areyoubeingeithertooambitiousornotambitiousenoughwithyourevaluation?

Also,familiarityisafactor.Ifthisisthefirsttimeyou,orindeedanyone,hasdonesomethingitmightbeworthinvestingmoreingoodevaluationsothatyoucanimproveyouractivityinthefuture,andthereforerunsmallerevaluations.Equally,ifyouarerunningafairlystandardactivitytheremaybenoneedtoevaluateitatall,especiallyifthebudgetislimited.

Don’tbeafraidtospeakupifitisnotpossibleorpracticaltoevaluatesomeorallofyouractivities.However,youshouldn’tusethisargumenttoavoidundertakingworkthatiseasilyachievableandthatwill,ifdoneproperly,provideyouandyourstakeholderswithmuchusefulinformation.

3.5In-houseorindependent?Afrequentlyaskedquestionis‘DoIneedanindependentevaluation?’Maybe-thereareprosandconstobothoptions.Somepeoplebelievethatifevaluationisatoolbywhichtoimprove,thenthereareclearbenefitstoself-evaluation.Thefeedbackisdirectandyouwillhavearealdepthofunderstandingofyouraudience,althoughyoumayhavetorecognisethattherearelimitationstothedatagatheringandanalysisskillstowhichyouhaveaccess.Youmayneedtobringinanoutsiderwithspecialistskillsifyouwanttoanswer

moredifficultquestionsaboutimpact.Theymightbefromanotherdepartmentwithinyourinstitutionoranexternalorganisation.

Ontheotherhand,somepeoplebelievethatifyouevaluateyourownactivityyouwillcheattomakeitlookgood,maybeevenwithoutrealisingit.Also,youshouldconsiderwhetherpeoplemightgiveyoumorefavourableanswersbecausetheydon’twanttohurtyourfeelingsorbecausetheybelievethattheremaybeanegativeimpactonthemiftheyarecritical.

Ifitiscrucialthattheevaluationisseentobeunbiased,thenanindependentevaluationmaybetheonlyoption.Indeed,fundersmaywellstipulatethatanindependentevaluationisrequired,perhapsbecauseofthelargesumsofmoneyinvolvedorbecausetheyaretryingtocomparethemeritsofdifferentapproaches.Thisiswherehavingclearobjectiveswillreallyhelp.Ifyouhaven’tgotany,externalevaluatorswilltrytosetthemforyouandbringtheirownvaluejudgementstobear.

Theneedforconfidentialitymayalsoinfluenceyourchoiceofinternalorexternalevaluation.

Ifyoufeelyouareill-equippedtoundertakeyourownevaluation,youmaylooktopeoplewithinyourowninstitution.Ifyouhaveapublicengagementoroutreachoffice,theyshouldbeablehelpwithyourevaluation.Otherwise,youmightconsiderapproachingsomeoneinyourresearchofficeorcolleagueswithexpertiseinsocialresearchmethodsordatacollectiontools.However,ifyou’renotasocialscientist,youshouldnotethatnotallsocialscientistsarefamiliarwithevaluationorthedatacollectiontoolsusedinevaluation.

3.6IfyourunintotroubleSomefunderswereinterviewedduringproductionofthefirsteditionofthisguide.Ingeneralfunderswanttheactivitiesthey’resupportingtobesuccessful.Ifyou’renotreachingyourobjectives,butyouron-goingevaluationmeansthatyouknowwhy,andtheevaluationenablesyoutosetoutaplanthatwillhelp


3yougetbackontrack,mostfunderswillgiveyouasympathetichearing.Youmightwanttorevisetheobjectives.Theearlierintheactivityyoutellfunderswhatishappening,themoresympatheticthey’relikelytobe.Remember,inmostcasesthefunder’sprogrammemanageristheretohelpyou.

WhensettingobjectivestheSMARTcriteriawilldriveyoutobespecific,butifthisisyourfirstexperienceyoumaynotknowwhatrepresentsanattainableandrealistictarget.Youwillhavetouseyourjudgementandmakesurethatyourformativeandprocessevaluationsgiveyoudatatoexplainwhyyouareoverorunder-achieving,andinthecaseofthelatterwhatyoucandoaboutit.Asevaluationdatabanks4grow,fundersmaybeabletoprovideadvicethatwillgivebenchmarks.Inthefutureyourdatawillhelpotherswiththisconundrum.

4Seeforexamplehttp://www.britishscienceassociation.org/forms/scicomm/evaluation/access/

http://www.britishscienceassociation.org/forms/scicomm/evaluation/access/



ToolsandTechniquesGatheringData 44.1IntroductionWorkingthroughtheprocessofdevelopingSMARTobjectivesshouldhavestartedtheprocessofthinkingaboutdatacollectiontools.Thebasictoolsandtechniquesarebasedinsocialandmarketresearchmethods.Thosewhoworkinthesefieldsusecertaintermswithacommonunderstandingofwhattheymean.Thislanguageisexplainedinthissectionandinannex 4.

Thissectionfocusesonthetoolsthatresearchersmightusetosupportin-houseevaluations,ratherthanthemorecomplex,andexpensive,optionsthatmightbeusedbyanexternalcontractor.

Ifyouarecommissioningotherstodoyourevaluation,youshouldfocusontheintelligencethatyourequire(theobjectivesoftheevaluation)ratherthantryingtospecifythenitty-grittyoftools.However,youwillbeabettercustomerifyouunderstandthestrengthsandlimitationsofdifferenttechniquesandunderstandthelanguagebeingusedbyyoursub-contractors.

Broadlyspeakingtherearetwomaintypesofdata,quantitativeandqualitative.Forbothtypesofdatatherearewaysofgatheringdatathatwillensuretheyarereliable.

4.2QuantitativeresearchQuantitativeresearchisbestsuitedtoansweringquestionsabouthowmanypeopledidorthoughtsomething.Youcanalsoaskthemhowmuch,towhatextentandother‘measure’typequestions.

Therearetwounderlyingprinciplestoquantitativeresearch:

• everyrespondentshouldbeaskedthesamequestionsinthesameway,sothattheanswerscanbeaddedtogether;and

• theinformationyoucollectisrepresentativeofallthepeoplethattookpartin,orused,youractivity.

Thequestionscanbeaskedbyaninterviewerface-to-faceoroverthetelephone,orpeoplecancompleteaquestionnairethemselveseitheronpaperorviatheInternet.Whateverthemethod,thequestionsarehighlystructuredtoensureconsistency.Youcanusethisstructuredquestionnaireformattocollectfactualandattitudinaldataandtoexplorethereasonsbehindpeople’sinitialanswers.Oftenpeoplecandothesamething,butfordifferentreasonsandbeingabletocomparethosewithdifferentrationalescanbeimportantinmakingdecisionsonhowtoimproveanactivity.

4.2.1Quantitativesampling

Strictlyspeaking,whendrawingasampletoberepresentativeofyour‘users’everyoneyoureachbyyouractivityshouldhaveanequalchanceofbeingaskedtorespond,althoughthereisacost-qualitytrade-offandsomesamplingmethodsarebetteratthisthanothers.Samplingisaboutavoidingbiasandonlygettingresponsesfromcertaintypesofpeopleorpeoplewholikedyouractivity.Itisimportanttonotethatjusthavingalotofrespondentsisnotgoodenoughiftheyarenotrepresentativeofyouraudience.

Thetypesofsamplingtechniquesyou’llmostlikelyusetocollectquantitativedatathatarerepresentativeofyourusersare:

Census–collectinginformationfromeveryonewhoengagedwiththeactivity.

Systematicsampling–takingevery‘nth’personwhopassesaparticularspotoraccessesawebsite,requestsapacketc.

Quotasampling–ifyouknowthat50%ofyouraudiencewillbegirls(perhapsaschoolhastoldyou)thenyousetaquotaof50%ofyoursampletobegirls.Whichgirlsyouaskshouldthenbe‘random’,sothatyouhavenoreasontobelievethatthoseyouinterviewareinanywaydifferentfromthoseyoudonot.


4Youcanalsouseverysimplemethodstoselectasample,likethosebornonacertaindateinthemonth.Forexample,takingeveryonebornonthefifth,fifteenthandtwenty-fifthofeverymonthoftheyearhasbeenfoundtoconsistentlyyielda10%sample.

Samplingmethodsareallverywell,butyouwillfindthatnoteveryoneyouasktotakepartwilldoso.Themainissuewillbethatthosewhoenjoyedyouractivitywillbemorelikelytorespondthanthosewhodidnot.Thisiswhygettingahighresponserateisimportant.Thehighertheproportionofthoseyouselectforinterviewwhorespond,themoreconfidentyoucanbethatyourresultsarerepresentativeofyouraudience.Usinginterviewersmeansyoucangetamorerepresentativesamplethanrelyingonself-completionasthereisabitofpressureonpeopletotakepartbecausetheywon’t(generally)wanttoberude.However,ifyoucan’tbesurethatthosewhodidn’trespondarenodifferentfromthosewhodid,makesureyouincludethelimitationsofyourdatainyourreport.Also,youneedtoaskatleast100peoplebeforeyoucanstartstatingpercentages,evenifyouaskeveryonewhotookpart.

SamplingandtheInternet

Withtheadventofeasytouseandfree/cheapInternetsurveytools,suchasSurveymonkey,ZoomerangandSurveyGizmoyoucandesignyourownsurveyandcirculatethelinktothequestionnairebye-mail.

However,simplybeingabletorunasurveydoesnotmeanthatthedatacollectedwillberepresentativeofanyparticulargrouporaudience.Manypeoplethinkthatcirculatingthelinktoeveryonetheyknow,orcanreachviae-maillists,willprovidealargenumberofresponsesthatcanbereportedonquantitatively.However,thisisnotthecase.

Ifthepeoplereachedbyasurveyarenotsystematicallyselectedtoberepresentativeofaparticularaudienceandarelikelytohaveparticularviewsbecause,forexample,theyallbelongtothesamee-maillist,allyouhaveundertakenisaverylargequalitativeresearchproject;evenifyouhaveaveryhighnumberofresponses.

Anobviousanalogywouldbeasurveyrunbyanewspaperthroughitswebsite.No-onewouldsuggestthatthefindingsrepresenttheviewsofthepopulationasawhole;indeedtheymaynotevenrepresenttheviewsofthenewspaperreadershipasawhole.

4.2.2Quantitativedatacollectiontechniques

Havingconstructedyoursample,ordecidedonacensusapproach,youneedtothinkaboutwhichdatacollectiontechniquetouse.Therearefourbasicquantitativedatacollectiontechniques,eachofwhichhasstrengthsandweaknesses.Thefouroptionsare:

• Face-to-faceinterviews.

• Telephoneinterviews.

• Self-completiononpaper.

• Self-completionon-line(e-mailorInternet).

Forsmall,liveevents,themostlikelymethodwillbeself-completiononpaper,wherequestionnairesaredistributedandattendeesareencouragedtocompleteandreturnthemattheendoftheeventorpostthembacklater.Forsomeactivities,face-to-faceinterviewswithasampleofattendeesisanotheroption.Thesearetoolsforgatheringinstantfeedback.

Youmaywantaconsideredresponse,ortogaugeresponsesafterparticipantshavehadtimetoreflectontheirexperienceortakesomeactions.Inthiscasesendingoutself-completionquestionnairesbypostore-mailorgettinganinterviewertocontactpeoplebytelephoneortovisitpeople,areallpossibleoptions.Thetablebelowsetsoutbasicstrengthsandweaknessesofthedifferentoptions.


4Surveystyle Strengths Weaknesses

FacetoFace

Telephone

Self-completiononpaper

On-line

Youknowtherightpeoplehaveresponded.

Youcanbemoreconfidentthatintervieweeshaveunderstoodthequestion.

Youcanrepeatquestionsthatintervieweeshavenotunderstood.

Interviewerswithdifferentlanguageskillscanaccessdifferentcommunities.

Highqualitydata.

Youcanrepeatquestionsthatintervieweeshavenotunderstood.

Youhaveagoodchanceofmakingsuretherightpeoplehaveresponded.

Interviewerswithdifferentlanguageskillscanaccessdifferentcommunities.

Moderatelyhighqualitydata.

Relativelycheaptoundertake(butrememberprintingandpostalcostsifapplicable).

Youcanuselongerquestionnairesbutyouwillgetabetterresponseiftheyareshort.

TranslationcanhelpyoutoaccesscommunitieswhosefirstlanguageisnotEnglish.

Moderatequalitydata.

Ifyouhaveadefinedsampleande-mailaddresses,youhaveagoodchanceofmakingsuretherightpeoplehaveresponded.

Pop-upsworkonaskingeverynthpersonwhovisitsthewebsitetocompleteaquestionnaire.

Youcanuseadditionalmaterialsuchaspicturestohelppeoplerespond.

Relativelycheaptoundertake.

Moderate/highqualitydata.

Resourceintensivesomostexpensive.

Thepersondoingtheinterviewcaninadvertentlybiastheresponse.Seeannex1,sectiononavoidingbias.

Quiteresourceintensivesoexpensive,butcheaperthanface-to-face.

Youneedtelephonenumbersforyoursample.

Youcannotbeconfidentthatintervieweeshaveunderstoodthequestion.

Peoplecanlookaheadatthequestions,whichmightbiastheiranswerstosomequestions.

Youcannotbesurethattherightpeoplehaveresponded.

Youcannotbeconfidentthatintervieweeshaveunderstoodthequestion.

Dependingonhowyouset-upthequestionnairepeoplemaybeabletolookaheadatthequestions,whichmightbiastheiranswerstosomequestions.

Ifyoudonothaveadefinedsample,youdonothavequantitativedata.

Youcannotbesurethattherightpeoplehaveresponded.

OnlyrespondentswithInternetaccesscantakepart.


4It’sagoodideatothinkaboutdoingexitsurveysaspeopleleaveeventsorfollow-upsurveysbytelephone,postore-mail–dependingonwhatinformationyouhaveaboutpeople.Ifyou’vedevelopedawebsiteyoucanimplantwhatarecalled‘pop-up’questionnairestocollectdatafromevery‘nth’visitororcollecte-mailaddressestosendaquestionnairelater.

Experienceshowsthate-mailssenttospecificindividualsyieldhigherresponseratesthanpop-upquestionnairesandassection4.1.1emphasises,alowerresponseratereducestheconfidencethatyouhavecollecteddatathatrepresentstheviewsofthewholepopulationinwhichyouareinterested.

4.2.3Constructingaquestionnaire

Therearemanyfactorstoconsiderwhendesigningaquestionnaire.Thefirstis‘whatdoyouactuallywanttoknow?’whichtakesyoubacktotheobjectivesofyouractivity.Donotwasteyourtimeandthoseofthepeoplerespondingbyaskingquestionssimplybecauseyoucan,everyquestionshouldjustifyitsinclusion.

Onceyouhaveidentifiedtheinformationyouneed,thelength,structureandlayout(forself-completionquestionnaires)willimpactontheresponserate,whichyouwanttomaximise.Rememberthatthequestionsyouaskwillinfluencepeople’sanswerstolaterquestions,soyouneedtothinkabouttheorderinwhichyouaskquestions.

Questionscanbe‘pre-coded’wheretherespondentselectsananswerfromalistor‘open-ended’whererespondentscanwriteintheirowncomments.Somepre-codedquestionsincludean‘other,specify’categorywherethosewhohavenotfoundapre-codetotickcanwrite-intheirviews.Remember,ifyouuseopen-endedquestions,someonewillhavetoreadalltheresponses.

Annex 1looksatthefactorstoconsiderandquestioningtechniques,suchasattitudestatementsandscales,thatwillgiveyouappropriatedata.AlthoughAnnex1iswrittenforself-completionpaperquestionnaires,thesamebasicprinciplesapplyforall

datacollectionmethods. Annex 2givesyouexamplequestionsandquestionnairemodules.

Itisalwaysagoodideatopilotaquestionnaireonthetargetaudience.Ideallythispilotingshouldincludeconversationsabouttherespondent’sunderstandingofthemeaningofboththequestionsandtheavailableresponses,sothatyouaresurethatthequestionsareelicitingtheinformationyouwant.Youcanusefriends,familyorcolleaguesforthistokeepcostsdown,iftheyareareasonableapproximationofamemberofthetargetaudienceandnotasimmersedintheactivityasyou.

4.3QualitativeresearchQualitativemethodsenableyoutoaddressthedeeperquestions,suchaswhypeopledidordidnotlikeanactivity,whytheyfeltitwasgoodorbad,andwhatyoucouldchangetomakeitbetter.Thenumbersofferedbyquantitativetechniquescanofferaseductivepromiseofcertainty,buttherearemanyinstanceswherethedepthofinsightfromqualitativeworkwillofferthebestvalueformoney.Thisisparticularlytrueforformativeevaluation.

Socialandmarketresearchersusetheterm‘qualitativeresearch’torefertoindividual,one-to-onein-depthinterviewsandgroupdiscussions/focusgroupsconductedbysomeonewhohasbeeninvolvedinthewholeprocessoftheevaluationandwhothereforehasadeepunderstandingoftheobjectivesoftheactivity.Thisislikelytobeeitheryouoroneofyourteamandbecausequalitativemethodsallowyoutointeractdirectlywith‘users’youcantestoutideasthatyouformduringtheevaluationprocess.


4

Bothin-depthinterviewsandgroupdiscussionscanbeone-offeventsorre-convenedtoallowtheinterviewees/participantstoofferfurtherviewsaftertimeforreflection.

4.3.1Qualitativesampling

Qualitativeresearchisaboutdepthofunderstanding,sosamplestendtobesmall.Youwillfindthatyoudon’tneedtotalktolargenumbersofpeoplebeforeyoustopgettingnewinformation.Peopletendtobeselectedtogiveyouacross-sectionofyouraudienceratherthanarepresentativesample.Ratherthanusingthestructuredapproachestoselectpeoplethatyouneedforquantitativesurveys,youpickindividualswhomeetyourcriteriaforinclusion.Ifpossible,trynottoincludepeoplewhoknoweachother.Theirviewsmay

besimilarandtheremaybespecificdynamicswithintheirrelationshipsthataffectthoseofthegroupmorewidely.

Thesmallsizeofyoursampleandthewayithasbeenselectedmeansthattheresultsarenotstatisticallyrepresentativeofthegroupyourparticipants‘represent’.Notonlymightyouhavemissedsomemarginalviews,becauseyoursampleissmall,youhavenoideahowprevalenttheirviewsareinthewiderpopulationbecauseofthewayinwhichtheyhavebeenselected.Foryourfindingstobestatisticallyrepresentativeyouneedtouseoneofthemethodsdescribedinthesectiononquantitativemethods.

Toensurethatyouincludetherightsortsofpeople,youshouldthinkaboutthemainvariablesor

Technique Strengths Weaknesses

In-depthinterviews

Groupdiscussionsorfocusgroups

Usefulfortalkingtothosewithbusydiaries.

Goodforsituationswhererepliesmaybesensitive,wherepeoplemightbereluctanttosaythingsinfrontofothers.

Idealwhenyouwanttocollectdetailsthatarelikelytobeveryindividual,suchashistoriesofindividualinvolvementinanactivity.

Canbeface-to-faceorbytelephone.

Interactionbetweenparticipantscanstimulateideas.

Goodforformativeevaluationasthistypeofresearchcangiveyouquickfeedbackonhowpotentialaudiencesviewyouremergingideas.

Flexible-Youmightputsimilarpeopletogethertoencouragethemtoexploretheissuesindepthfromsimilarperspectives.Alternatively,youmightcreatemixedgroupstogivepeopleexposuretodifferentviewpoints.

Canbeface-to-faceoron-line.

Relativelyexpensiveasyouonlyinvolveonepersonatatime.

Stimulusonlycomesfromtheinterviewer,sotheremaybelittlechallengetoviewsandlimitedcapacitytoprovokereflection.

Notgoodforsituationswhereyoumightwanttoexplorepersonalexperiences.

Audioorvideorecordingisrequiredifyouwanttododetailedanalysisafterthefact(youneedtogetsignedpermissionfromparticipantsforaudioandvideorecording).

Strongpersonalitiescandominatesoyouneedaskilledfacilitator.

Mencandominatemixed-genderdiscussions,soyoushouldthinkcarefullyaboutseparatingmenfromwomenindiscussiongroups.

Youngeradultstendtospeaklesswhengroupedwitholderpeoplesoyoushouldthinkcarefullyaboutagemixes.


4characteristicsthatyouthinkwillbeimportant,givenyouractivity.Forexample,aremenandwomenlikelytoresponddifferently?Areyoungerpeoplelikelytobedifferentfromolderpeople?Arethoseinurbanareaslikelytobedifferentfromthoseinruralareas?Iftheanswertothesetypesofquestionsisyes,thenyouneedtomakesureyouincludemenandwomen,youngerandolderpeopleandurbanandruraldwellersinyoursample.AndalthoughmorepeopleinBritainliveinurbanareasthanliveinruralareas,youmightincludethesamenumberofeachgroupinyourresearch.However,youdon’tneedtocovereverysegmentyouidentifyexplicitly–soyoudon’tneedoldermeninruralareas,olderwomeninruralareas,andsoon.Youcouldjusthaveoldermeninruralareasandyoungerwomeninruralareaswithyoungermeninurbanareasandolderwomeninurbanareas.Runningfocusgroupscanbeexpensive,especiallyifyouuseanoutsidecontractor,soyoumayneedtoprioritisewhichgroupstoinclude.

Anoptionthatisbecomingincreasinglyavailableison-linediscussionseitherusingvideo/audiostreamsortextualconversations.Forthesediscussionstobeclassedasqualitativeresearchyoumustapplythesamedepthofrigourtoselectionoftheparticipantsandcoverageofallsalientpointsasyouwouldforface-to-facework.Lessstructuredapproachesusingnewmedia,andparticularlysocialnetworkingtoolsarediscussedinsection4.4.

4.3.2Anecdote

Qualitativeresearchisnotanecdotal.Thesampleandthediscussionarestructuredbyresearchersagainstclearobjectivesfortheevaluation.

However,giventhebudgetsthataresometimesavailableforengagementactivities,youshouldthinkofconversationsinthemarginsofaneventasprovidingusefulfeedback.Whileanyfeedbackisbetterthannone,youneedtobecarefulabouthowyouinterpretsuchad-hoccomments.Forexample,theoneparticipantwhomadeacommentmaybeatypicalandhavesought

yououtsimplytomaketheirpoint.Itisusefultoknowwhatsuchanindividualthinksastheremaybeotherswhofeelthesame,butequallyanysingleindividualmayhavetheirownagenda.

Thinkingofanecdotalevidenceasasourceofideastobeinvestigatedmorethoroughlyusingmorerigoroustechniquesisahelpfulwayofattachingtherightweighttothissortoffeedback.

4.4Otheroptions4.4.1Observationalresearch

Observationinvolvestheplannedwatching,recording,andanalysisofbehaviourasitoccursina‘natural’setting.Inmostcasesyouwillbeobservingpeopleinteractingwithyouractivity.Itisparticularlyusefulforunderstandinghowpeopleusewebsitesorflowthroughanexhibition,aswellastoexplorehowtogetmorepeopletoactivelyengagewithtalksanddiscussions.

Observationenablesevaluatorsto:

• understandindividuals’engagementwithspecifictasksandprocesses,

• understandindividuals’attitudesandrelationshipsincontext,

• definekeyissuesthatmaybefollowed-upininterviewsandsurveys,

• formrelationshipswiththeparticipants,whichwillhelpwithanyfollow-upinterviewing,and

• iteliminatesthebiasofself-reportingthatmayoccurininterviewsandsurveys.

Youcanobserveandmakenotesonhowindividualsinteractwithyouractivity,butyoucanalsouseobservationalmethodstocomparegroupdynamicsacrossevents.Thisisdonebyhavingastructureagainstwhichyoucanrecorddetailsyouareinterestedin,forexample:theorderinwhichwebpagesareaccessed,thenumberandtypeofparticipantsatanevent,levelofinputtodiscussions,typesofpeoplewhoactivelyparticipate,mainsubjectsofconcernandsoon.


4Thereducingprice,andincreasingquality,ofdigitalphotographyandvideomakevisualrecordingsaviableoption,butrememberthatyoushouldseekthesignedconsentofparticipantstoanyrecordingandparentalconsentforrecordingsofyoungpeople.

Sampling

Observationalresearchismainlyqualitativewhenconductedaspartofformativeevaluation,see4.3.1forguidanceonsampling.

Whenobservingfinalactivitiessuchasexhibitions,youneedtothinkabouttimesofdayanddaysofweek,asdifferentpeoplearelikelytoengageatdifferenttimesandyouneedtoensurethatyoursampleisasrepresentativeaspossibleofallthosewhoengagewithyouractivity.

4.4.2Visitors’book

Avisitors’book(orsimilartoolsuchasa“PostIt”board)isagoodwayofcapturingthethoughtsofvisitorsandgettingfeedback.However,onlythosewhoarehighlymotivatedwillgivecomments.Sothecomments,whilehelpfulinthedevelopmentofyourprojectandsimilar,futureprojects,willnotberepresentativeofyourachievedaudience,oryourinitialtargetaudience.

4.4.3Socialmedia

Socialmediareferstoweb-basedandmobiletechnologiesthatenableinteractionbetweenusers.Thereareliterallyhundredsofsocialmediatools.AmongthebestknownareFacebookandTwitter,whichareillustrativeofthedifferenttypesoftools.

Thesetoolsallenableyoutointeractwiththepeoplewhoengagewithyouractivity.Theyalsoallowthe‘users’tointeractwitheachother.However,feedbackthroughsocialmediaisbestseenasanecdotalevidence,becauseonlythosewhoareverymotivatedarelikelytotakepart.Youshouldregardtheissuesthatarisethroughthesediscussionsastopicstobeexploredwithmoresystematicqualitativeorquantitativetechniquesratherthandefinitiveevidence.

TheResearchInformationNetworkhasproducedausefulguideforsocialresearchersthatdocumentsthedifferenttypesofsocialmediaandhowtheycanbeusedinsocialresearchandmuchofthiscarriesovertoevaluation.See:http://www.rin.ac.uk/our-work/communicating-and-disseminating-research/social-media-guide-researchers

4.4.4Recordkeeping–managementdata

Recordkeepingcanstimulateself-reflectionandmightbethoughtofasself-observation.Atitssimplestyoucouldkeepadiary,whichrecordsyourthoughtsandfeelingsthroughouttheprocessaswellasyourreflectionsontheprocessitself.Thisformsarecordofwhathappenedandwhen,andisausefulresourcewhenlookingathowyoucoulddothingsbetterinthefuture.Youcouldalsoaskusersofyouractivitytokeeprecordsoftheirinteractionsandtheirthoughtsoveraperiodoftime.Thewidespreadavailabilityofdigitalphotographymeansthatkeepingpictorialrecordsisanoptionthatisreadilyavailable.

4.4.5Mediaimpact

Ifyouwanttoraiseawarenessyoumightsetanobjectiveaboutpressormediacoverage.Measuringtheimpactofthiscanbeverydifficult.Somepeoplemeasurecolumninchesandusethesalesorreadershipfiguresofthepublicationtoestimatethenumbersreached.However,noteveryonereadseverypageofanewspaperormagazineandtheimpactonreadersisgenerallyunknown.

Whenitcomestotelevisionandradioprogrammes,viewingandlisteningfiguresmaybeavailable.Nevertheless,evenintelevision,wherethereisaconsiderableamountofprogrammeresearch,dataonimpactratherthanenjoyment,isunlikelytobeavailable.

4.5ConfidentialityMarketandsocialresearchoperatesfromthepremisethatinformationgivenbyrespondentsinresearchprojectsisconfidential.Questionnairesusuallyreassurepeoplethattheywillnotreceivemarketinginformationorsalescallsasaresultoftakingpartandthatnoone

http://www.rin.ac.uk/our-work/communicating-and-disseminating-research/social-media-guide-researchers




4willknowwhattheypersonallyhavesaid,otherthanthepeopleprocessingthedata.

Peoplemaybehappytoputtheirnametotheirviewsandrespondingtoanevaluationofapublicengagementactivityisunlikelytocauseproblemsforrespondents.Nevertheless,theremaybecaseswherepeoplefeelthatyouwillpassontheirviewstoothersandthatthismaybedetrimentaltotheminsomeway.Offeringconfidentialitydoesn’tusuallycauseproblemsinreportingandcanbebeneficial,butitmaymeanyoucan’tconducttheevaluationin-house.Somepeoplemaybemoreforthcomingtosomeoneunrelatedtotheactivity–anothercaseforanindependentevaluator.

Youshouldalsonotethataskingpeopletheirviewsinfrontofothersmayrestrictwhattheyarepreparedtosay,dependingonthetopic.

YoualsoneedtobeawareoftheDataProtectionActsee:www.informationcommissioner.gov.uk


ToolsandTechniquesDataHandling 5Theprevioussectionlookedatthevariousmethodsforcollectingdata.Thischapterlooksathowyouanalyseit.Hereitisassumedthatdatacollectionandanalysisisbeinghandledin-houseratherthanthroughspecialistsub-contractors.Ifyouareusingsub-contractors,buttheyarenotprovidingyouwithafinalreport,thenyouwillneedtospecifytheanalysisyouwantandtheformatinwhichyouwantthedata.

5.1Quantitativedata5.1.1Coding

Thefirstthingtodowithyourquestionnairesistocodeanyquestionswhererespondentshaveenteredtheirownanswersratherthantickingabox.Readthrougheachquestiononeatatime,thatis,lookatalltheresponsestoQ1together,allthosetoQ2togetherandsoon.Youshouldbelookingforsimilarresponsessothatyoucandraw-upa‘codeframe’forthequestion.Thisallowsyoutoaddtogethersimilarresponsesfromdifferentpeople.Onceyouhaveyourcodeframeyouwillgiveeachcodeanumber.Thenyouneedtoreadeachquestionnaireandputtheappropriatecodeorcodes(peoplemayhavesaidmorethanonething)bythesideofthequestion.Itisthisnumberthatyouwillenterintoyourdataset,nottheverbatimcomments,theseshouldbekeptseparately.

5.1.2Dataentry

Ifyouareusingpaperquestionnairesyouwillhavetoinputyourdata.Dataentryisatimeconsumingandrelativelyspecialisttask.Youhavetomakeverysureyoudon’tmakemistakes.Forsmallamountsofdatayoumightdoitin-house,forlargeramountsconsiderusingaspecialistdataentryfirm,thespeedandqualityofentryislikelytoyielddividendsandthecostisonlygoingtobeafewhundredpoundsforthesizeofjobyou’relikelytohave.Astrengthofon-linesurveysisthatthedataisautomaticallycollated.Similarlyifyouhavecommissioneddatacollectionitwillbeprovidedintheelectronicformatthatyouspecify.

5.1.3Analysis

Ifyouonlyhaveasmallnumberofrespondents,perhapsfewerthan50,youcoulddoyouranalysisbyhandbyjustcountingthroughthequestionnaires.However,ifyouwanttodoanyanalysisbeyondtotalcountsofhowmanypeoplegaveeachanswer,oryouhavemorerespondents,thenthesimplestwaytoanalysesmalldatasetsistousespreadsheets.Itreallyisworththetimetoenterthedata.

Figure5.1overleafshowsanexamplespreadsheetwithrawdataentered.Eachcolumnrepresentsasinglerespondentandeachrowaquestion.Eachcellrepresentstheanswertoaquestion.So,forexample,forQ1thepossibleanswersareWedandSat.‘1’indicatestheansweris“yes”and‘0’thatitis“no”.Sothepersonwhoseanswersareenteredincolumn‘C’said‘yes’to‘Wed’and‘no’to‘Sat’.Itmayseem,andindeedis,timeconsumingtobreakdatadowntobinarycomponents,butthisisvitalifyouwanttobeabletodomoresophisticatedanalysisofyourdatathansimplecountsofthewholepopulation.

Summingacrosstherowsgivesyouthetotalnumberofresponsesforthatansweracrossthesample.However,you’requitelikelytowanttoanalysethedataagainstkeyvariables,suchasageorgender.Thesevariablesareoftenknownascross-breaksandifyouarecommissioningdatacollectionandpreparationyouwillbeaskedtospecifythecross-breaksthatyouwouldliketohaveincludedinthedatasetthatisprepared.

Itisfairlystraightforwardtosetupsimplecross-breaksinaspreadsheetpackageusingtheSUMIFfunction.Thislogicalfunctionallowsyoutosumtheresponsesinonerow(orcolumn)wheretheresponsesinanothercolumnmatchaparticularcriterion.So,intheexampleabovethefunctioncouldbesettosumrow4(verygood)forallthecaseswhererow3equals1i.e.peoplewhoattendedonSaturday.IfpeoplewhoattendedonSaturdayaremorelikelythanthosewhoattendedonWednesdaytohavethoughtthattheeventwasverygood,thenyouhaveanimportantfinding.


5

Thecapacitythatspreadsheetsofferisstraightforward,butimportantanalysisshowswhyitisvitaltobreakdatadownintothebinarydata.Thissimpleformulawouldnotworkifyouhavetwosinglerowsforquestions1and2withSaturdayorWednesdayenteredinthefirstrowandVGood,Good,OK,Poor,VPoor,NoopinionorDKinthesecond.Assoonasadatasetbecomeslargeenoughthatyoumightwanttointerrogatesub-samplestheneedforproperdatapreparationbecomesparamount.

Forlargedatasetsandcomplexsurveysitisbettertousebespokepackagesfordataanalysis.TheindustrystandardisSPSS(StatisticalPackagefortheSocialSciences),whichoffersfargreaterprocessingpower

thanspreadsheetsandaccesstoawiderrangeofstatisticaltestsandmethods.

5.2QualitativedataQualitativedataisgatheredbyrecordingthediscussions.Recordingmaybeliteralthroughaudioorvideo(ineithercasepermissionshouldbesoughtfromtherespondentsbeforerecordingstarts)orvianote-takingtorecordkeypoints.Bearinmindthatifyouchoosetotakenotesratherthantaperecord,youwillloosesomeoftherichnessofthedataandyouwillneverbeabletorecaptureit.Mostsocialandmarketresearchersrecordfocusgroupsandone-to-oneinterviewssotheycanconcentrateonrespondingto,andobserving,theinterviewee(s).

Figure5.1RawData


5Youcanuseflipchartsinagroupsituationandthisallowsparticipantstoconfirmthatyouhaveaccuratelyrecordedwhattheymeant.Thisapproachalsomeansthatsomeanalysisisbeingundertakeninsituaskeypointsareidentifiedandrecordedbythegroupsupportedtoagreaterorlesserextentbythefacilitator.

Analysisofrecordedconversationscanbeundertakenbymakingtranscriptsorbylisteningbacktothetapes,makingnotesandrecordingquotes.

Whattolookfor:

• Mainandsub-themesandissues(acrossdifferentgroups/individuals)thatemergefromthediscussion.

• Ideasfromparticipantsthatwillsupportthedevelopmentofyouractivity.

• Trackingindividualviewsthroughthediscussion,exploringhowandwhyviewschange(iftheydo)andanypreconceivedorhyperbolicviews.

• Thecontext,andthustheinterpretation,ofcomments.

• Illustrativequotesforuseinreports.

• Thelanguageused–thiswillhelpwiththedesignofquantitativequestionnaires.

Itisunlikelythatyouwillbeusingqualitativedatatoproveordisproveahypothesis,ratheryouwilllookatdatatoseewhatissuesemergefromthem.Sotheapproachisnot“Was the event boring because the speaker was no good?”ratheritis“How enjoyable was the event?”,“Why was it enjoyable/not enjoyable?”

Inessence,interviewdatacanbetreatedintwoways.Somepeopletakecommentsatfacevalueandcategorisethetextintothemes.Itisimportanttorememberthough,thatqualitativeresearchisaboutmorethanjustwhatpeoplesay.Peopledonotalwaysexpressthemselvesclearly,maycontradictthemselvesandtheirbodylanguagewilladdtoyourunderstandingofwhattheymean.Yourunderstandingofwhattheymeanisimportant,butyouneedtorecognisethatitisyour understanding.

Oneofthesimplestwaystoanalysequalitativedata,thatallowsyoutoincorporatethecontextofthe

discussion,is‘charting’.Listeningbacktothetapesorworkingthroughtranscripts,youidentifythemainissuesorthemesraisedinthediscussions.Youplotwhomadeeach(relevant)comment,leadingyoutobeabletoidentifythetypeofpersonwhoraisedeachissueandthereforeforwhomthiswasanimportantpoint.However,asyouworkthrougheachdiscussionchartingit,youtakeintoaccountcontextandintendedmeaningaswellasthepuretext.

TherearespecialistsoftwarepackagestosupporttheanalysisofqualitativedatatheseincludeNVivo,ATLAS.tiandFrameworkamongstothers.Suchpackagesallowqualitativedatafromdifferentsourcestobelinkedtogetherinasingleworkunit,sothatelectronicannotationscanbeusedtolinkthemes,dataandissuesacrossthedifferentsources,whichcanincludenotesandtranscriptsofdiscussions,pictures,audioandvideofiles.Importantly,usingthesetoolsdoesnotreplacethein-depthreadingoftranscripts/notes.Itonlyprovidesanelectronicmeansofcataloguingandlinkingsourcestosimplifytheprocessofdrawingtogetherevidencefromdifferentinterviewsorsources.

5.3ObservationaldataObservationaldataprovidescontextualinformationagainstwhichtounderstandtheresultsthatemergefromsurveysandfocusgroupsaswellasprovidingdataonhowpeopleengagewithyourexhibitionorwebsite.

Socialresearchersareawarethatwhatpeopledoisnotwhatnecessarilywhattheysaytheydo,andobservationaldata,takenwithotherinformation,canhighlighttheseinconsistencies.Forexample,afacilitator’sobservationsofgroupmembers’behaviourmayrevealwhetheraparticularindividualwhohadnotbeencontributingtothediscussiondidsobecausetheyfeltalienatedbytheprocessorsimplybecausetheywereshy.

Asmentionedinsection4.4,observationaldatacantakeanumberofforms.Hand-writtennotesorfielddiariestakenbytheevaluatorthroughouttheprocess,whichmaybeaugmentedbystillphotographsorvideo


5footage,canprovidearecordthatcanbesubjectedtoanalysisinthesamewayasqualitativedatafrominterviewsandfocusgroups.

Atthesametime,systematisedobservationscanrangefromquitesimpletalliesofattendanceataneventandbasicdemographicinformation,tofairlycomplexcategorisationsandcodingofbehaviour.Thesedatacanbefairlystraightforwardlyanalysedinastandardspreadsheetpackage.Astheamountofdatagrowsandtheformsinwhichtheyareheldmultiply,itislikelytobecomemoreappropriatetouseoneofthespecialistanalyticaltoolsforqualitativeresearch.


Reporting 66.1ReportingGatheringandanalysingdataisallverywell,butreporting,orsharingdata,iswherethebenefitsofgoodevaluationstarttoberealised.Therearefourmainaudiencesforevaluation:

• Youandyourteam.

• Yourfunder.

• Otherstakeholders.

• Yourpeers.

6.1.1Youandyourteam

Evaluationplaysanimportantroleinhelpingyoutoimproveyourengagementactivities.So,theprimaryaudienceformuchoftheevaluationisyourteam.Whathaveyoulearnt?Howwillyouapplythisinthefuture?Aretherethingsthatyou’velearntthatcarryacross,notjusttootherpublicengagementactivities,butyourotherworkoryourinteractionswithfundersandusers?

Choosethemostsuitableformatforyourreport.Itmaybeinwriting,butitmaybethatapresentationenablestheteamtoreflectbetterontheirexperiences.

6.1.2Yourfunder

Whoeverhasprovidedtheresourcesforyouractivitywillprobablywanttoknowwhatthey’vegotfortheirmoney,sothefirstaimofyourevaluationreportisdemonstratingtheoutcomesandimpact.However,fundersareinterestedinsharinggoodpractice,soifthereareimportantpiecesoflearning,sharethem.

Yourfundermayhavegivenyouastandardformatforyourreport.Youshoulduseit,evenifitisconstrictive.Theyprobablyusethestandardformattoallowthemtosumdataorcollateinformationfromacrossdifferentactivities,sotheycanreporttotheirseniorsandaccountfortheirbudgets.Ifthereareotherthingsthatyouwanttosay,sendanadditionalnoteorreport.Fewfundersarelikelytocomplainaboutgettingtoomuchfeedback.Youneverknow,youmightwanttoapproachthemagain,inwhichcaseyoushouldshowhoweffectivelyyoumanagedtheactivityandhowthoroughlyyouevaluateditandreportedonit.

6.1.3Otherstakeholders

Youmayhavehadpartnerswhocontributedresources,peopleorexpertisetotheevent.Makesurethatyoushareyourevaluationresultswiththem.Itwillhelpthemtoassessthevalueoftheircontributionsandmayinfluencetheirfuturedecisionsongettinginvolvedinfutureactivities,eitherwithyouorothers.

6.1.4Yourpeers

Finally,thereareyourpeerswhoaregettinginvolvedinpublicengagementwork.Youmayknowsomedirectly,butalsotalktoyourfunderaboutanynetworkstheyhaveorareawareof,forsharinggoodpractice.Yourevaluationcouldprovidetheanswersthatsomeoneelsehasbeenlookingfor.Itmightbethatyou’vecrackedaproblem,orthatyoustumbledintoatrapthatyoucanhelpotherstoavoid.Ittakessomecouragetoadmitthelatter,butpeoplewillbeverygratefulandinthefuturetheymayrepayyouwithvaluableadvice.Youmightwanttopostyourevaluationathttp://www.britishscienceassociation.org/forms/scicomm/evaluation/access/ sootherscanlearnfromyourexperiences.

6.2ReportstructureYourreportshouldbestructuredaroundthequestions/objectivesyourevaluationsetouttoaddress.Theseshouldstemfromtheobjectivesyousetforyouractivity.Monitoringinformationisalsolikelytohavearoleinyourfinalreport.Forclarity,youshouldhave:(seetableoverleaf)





6

Forthesectionsabouttheobjectives,trytoturntheobjectiveintoaquestionorseveralquestions.Thenassessalltheinformationyouhavefrombothmonitoringandevaluationandallocateeachpieceofinformationtoasectionofthereport.Ifyouhaveinformationthatdoesn’taddanything,don’tincludeit.

Don’tfeelyouhavetoreporteverypieceofdetail.Itcanbeusefultohaveshort‘conclusions’attheendofeachmainsectionthatsumsupthemainpointsfromthesection.

Theuseofchartstoillustratenumericaldata,whichcaneasilybederivedifyouhaveyourdatainaspreadsheet

Section Contents

Executivesummary

Introduction

Annexes

Lessonslearnt

Conclusions

Objective2,3,etc.

Objective1

Unexpectedoutcomes

Somepeople,especiallyseniorpeopleinfundingorganisationsorthoseonassessmentpanels,willonlyreadthissection.Itshouldthereforesetouttheactivity’sobjectivesandgiveashortdescriptionoftheactivity,butimportantly,itmustpulloutthekeypoints.

Therearetwomainstylesforsummaries.Anexecutivesummaryshouldmirrorthestructureofthefullreportsothatanyonewhowantsmoreinformationonacertainsectioncaneasilyfindit.

Ageneralsummaryisshorterandmorenarrativeinstyle.

Thismeansyoushouldthinkaboutwhichstyleisrequiredandwritethissectionlast.

Setsout:• thebackgroundtoyouractivity,• whyyouwantedtodotheactivity,

• whatyouhopedtoachieveandwhy,

• theaimsandobjectivesoftheactivity,

• theaimsandobjectivesoftheevaluation,

• thestructureoftheremainderofthereport.

Include:

• fulldetailsofyourmethodology,

• howyouselectedyoursample,

• copiesofquestionnairesandtopicguides,

• someinformationabouthowyouanalysedyourdata.

Whatyouwoulddodifferentlynexttimeandwhy.

Keylearningpointsforothers.

Adiscussionofunexpectedoutcomesandhowtoensuretheyeitheroccurornotagain,asappropriate.

Asummingupofthekeyachievementsofyouractivity,itsstrengthsandweaknesses.

Asabove.

Theobjectiveanddatarelatingtowhetheritwasmetwithsomediscussionastowhytheactualoutcomeoccurred.

Describeanyunexpectedoutcomesandwhethertheyaregoodorbad.


6suchasExcel,willhelpyouandthereadertoidentifyhighsandlowsandtrends.Usebulletpointsundertables,chartsandgraphstohighlightthemainpoints.Quotesfrominterviewsandfocusgroupswillservetobringtolifethespiritoftheactivity.Thinkabouthowyoumightuseimagestoillustrateyourresearchfindings.Somepeopleliketohavethissortofdetailinannexestokeepthemainreportshort.You’llfindthatdifferentpeoplehavedifferentpreferencesabouthowdataispresented,soitissensibletocheckwithkeyreaders,suchasyourfunder,toseewhattheyprefer.

Theconclusionssectionshouldpulltogetherallthedataandgettothepointof‘whatdoesitallmean’?‘Sowhat?’Whatdotheprecedingsectionstellyouinanutshell?Somepeoplewillonlyreadthissection,especiallyifthereisnotanexecutivesummary.

YoushouldusePlainEnglish.Yourreportshouldbeaccessibletoasmanypeopleaspossible.Themainaudienceswillbeotherpeoplelikeyouandyourteamandyourfunder.Remember,manyfunderswillnotbeevaluationexpertseither.

Donotbetemptedtousejargon.Thereareoccasionshowever,whenyou’llwanttouseverypreciseterms.Ifyoudo,makesureyoudefinethemsomewheresoall

yourreadersknowwhatyoumean.Forexample,thisguidehasaglossaryofterms.

Don’tbeafraidtomakevaluejudgementsorgiveyourviews,especiallywhenyou’reconsideringhowyoumightdothingsdifferentlynexttime.Remember,youhavelearntalotfromconductingyouractivityandcanspeakwithsomeauthority,especiallyifyou’vebackeditupwithgoodevaluationdata.


EvaluationQuestionnairesandQuestions

Annex1

QuestionnairedesignThissectionfocusesondesigningpaperquestionnairesforselfcompletionsurveys.Theprocessofdesigningon-linequestionnaires,whichwillalsobecompletedbytherespondent,isverysimilar.Youcan‘route’peoplesotheyarenotaskedquestionsthatarenotrelevanttothembasedontheirpreviousanswersinpaperquestionnaires,butwithon-linequestionnairesyoucanprogramtheroutingandsoitcanbemuchmorecomplex,forexample,combiningtheanswerstomorethanonequestion.

Maketherespondent’sexperiencepositive

Makesurethattherespondentfindstheexperiencestraightforward.Theymayevengainsomethingfromtheprocess.

Makesurethatyourlanguageisappropriatetoyouraudience.Thisisespeciallyimportantwithyoungeraudiences,butalsobearinmindthatgeneralliteracylevelsintheUKarenotthesameasthoseofgraduatesandthatforsomepeopleEnglishmaynotbetheirfirstlanguage.

Youcanusecolour,pictures,cartoon“smileyfaces”orotherlighterapproaches,butmakesurethesematchthemoodofyourevent.Youwanttoengagerespondents,butyoudonotwantthemtofeelthatyouarebeingcondescending.

Makesureyourrespondenthasthechancetosaywhatisontheirminde.g.byusingageneralopen-endedquestionattheend.

Ifpossible,pilotthequestionnaireonafewpeoplebeforecirculatingitwidely.Thiswillhelpyouidentifyanydifficultieswithwordingorconcepts.Evenpilotingitonyourcolleagues,friendsorfamilywillprovidesome

usefulfeedbackonhowtoclarifythequestionsandthewayyouaskthem.

Reluctancetogivefeedback

Thiscanbethecurseoffeedbackquestionnaires–peopleeitherdon’twanttohurtyourfeelings,sotonedowntheircomments;orforgetthemselvesandlaunchscathingattacksthatdon’treallyhelpyoutoimprove.Thekeyistoensurethatpeopleunderstandtheirfeedbackisimportantandcanhelpyou.Emphasisethatcriticalfeedbackwillhelpyougetbetterandthatitwon’thurtyourfeelings(eventhoughitmight)andthatpositivefeedbackalsohelps,becauseitshowswhatworkswell.

Theproblemwithleavingquestionnairesfortheaudiencetocompleteonleavinganeventisthatthosewhohadagreattimearemostlikelytofillitin.ThosewhohateditarealsomorelikelytofillitinthanthosewhohadanOKtime.Also,somepeoplewilljustwanttoleaveandnothavethetimeortheinclinationtocompletethequestionnaire,althoughleavingafewminutesattheendoftheactivityforpeopletofillinaquestionnairemayhelp.However,unlessyouhavesomewayofensuringthatahighproportionofthoseyoutargetcompletethequestionnaire,theresultswillnotnecessarilyberepresentativeofyourwholeaudienceandyouwillnotbeabletotellinwhatwaysortowhatextenttheresultsarebiased.

Thisisnottosaythatthedataisuseless,butitneedscarefulinterpretationandifyoucancalculateorestimatetheproportionthatcompletedaquestionnaire,thiswillhelpyouassesstheaccuracyofyourresults.Thehighertheresponserate,themorerepresentativetheresultswillbeandanythingover65%isverygood.Ifeveryonewhoengagedwithyouractivityresponded,thedataisrobust,eveniftherewereonlysixpeople.Justdon’ttrypercentagingonfewerthanabout100people.


Maximisingtheresponserate

Ifyouractivityisanevent,distributethequestionnaireatthestartandaskpeopletocompleteitbeforetheyleave.

• Makethequestionnaireshort,simpleandrelevant.

• Usepre-paidenvelopestoincreasetheresponseratewhereyouhavedistributedquestionnairesbypostorwhereyouthinkpeoplewillwanttopostbackthequestionnaire.

Askingpeopletocompletetheirquestionnairesattheendofengagingwithyouractivitywillincreaseyourresponserateandmayimprovethequalityoftheinformationyougather.However,itwillmeanthatyoucanonlyaskrelativelyfewquestionsbecausepeoplewillonlybepreparedtogivealimitedamountoftime.Youalsoneedtothinkaboutthetimepeoplehaveforreflection.Youmaygetamoreconsideredresponsewhenpeoplehavehadachancetothinkabouttheactivitylater.

Remember,trytomakeitfun.Inresponsetoaconsultationonup-datingthisguide,onerespondentsaidthatatatalkaboutaerodynamicstheaudiencewasaskedtomaketheirfeedbackquestionnairesintopaperaeroplanesandtherewasaprizeforthosewhocouldgettheirplaneintotheboxatthefront!

Confidentiality/dataprotection

Youmusttakeallreasonablestepstomakesurethattherespondentsarenotadverselyaffectedbytakingpartintheevaluation.Youmustkeeptheirresponsesconfidential,unlessyouhavetheirpermissiontodootherwise,andyoumustnotdoanythingwiththeirresponsesthatyouhavenotinformedthemabout.So,unlessyoumadeitclearwhenyougavethemthequestionnaireoronthequestionnaireitself,youcannotusetheresultstobuildadatabaseformarketing,forexample.

Therearetwousefulsourcesofinformation:theInformationCommissioner’sOfficewebsite,andtheMarketResearchSocietywebsite,whichhasvariouscodesofconductrelatingtodataprotectionandconfidentialityissues.

FortheInformationCommissioner’sOffice:www.dataprotection.gov.uk

Formarketresearchguidelines: http://www.mrs.org.uk

Ifyouareworkingwithchildrenorvulnerableadults,suchaspeoplewithlearningdifficultiesortheelderly,youandyourteamshouldbecheckedbytheCriminalRecordsBureau.Seehttp://www.direct.gov.uk/en/Employment/Startinganewjob/index.htm?CID=EMP&PLA=url_mon&CRE=crb Thisisveryimportantforanycontactaswellaspartoftheevaluationasforcontactduringyourproject.

Length

Keepitfocused,simpletocomplete,andasshortaspossible–onetotwopages.Thiswillmaximisethenumberofresponsesandminimisethetimeitwilltakeyoutoanalyseit.

Donotaskforinformationthatyoudonotplantouse:itwasteseveryone’stime.However,donotworryifyouaskquestionswhichlaterprovenottobeuseful;justdon’taskthemnexttime.

Structure

Designthequestionnaireasafunnel,movingfromsimple,unthreateningandnon-sensitivequestions,tothosethatrequiremorethoughtandmaybemorepersonalinformation.

Mostquestionnaireswillbenefitfromamixof‘closed’or‘pre-coded’and‘open’questions,wherepeopleentertheirresponseintheirownwords,asthishelpstokeeppeopleinterested.

Itisusuallybesttoplacesensitivequestionstowardstheendandalwaysgiveareasonwhyyouwantpersonalinformationsuchasdemographicdata(e.g.age,sex,education,andethnicity).

Avoidlongbatteriesofagree/disagreequestions,asrespondentswilldriftintogivingautomaticanswers-breakupquestionsvisuallyifthequestionnaireislong.

www.dataprotection.gov.uk

www.dataprotection.gov.uk

http://www.mrs.org.uk

http://www.direct.gov.uk/en/Employment/Startinganewjob/index.htm?CID=EMP&PLA=url_mon&CRE=crb




Avoidingbias

Therearemanydifferentsourcesofpotentialbiasinresearch.Theseinclude:

• Questionnairebias-leadingquestions.

• Methodologicalbias-forexample,on-linesurveysexcludepeoplewhodonothaveinternetaccess.

• Samplingbias-forexample,askingforfeedbackonlyfromthosewhoaskedquestionsataneventmaybeaverypoorindicationofhowtheaudienceasawholefelt.

• Responsebias-thosewhocompletequestionnairesmaybeverydifferentfromthosewhodon’t.Thisisamajorissueforallsurveyresearch.

• Questionbias-mostpeopleliketoputthemselvesandtheirbehaviourinagoodlight.Ifyouaskaquestionwhichembarrassestherespondentormakeshim/herfeelbad,theymaymassagethetruth.So,forexample,asking“How often do you eat chips?”ratherthan“Do you eat chips?”givespeoplepermissiontosaythattheydowithoutfeelingthattheyshouldn’t,andthosethatdon’t,justsaynever!

• Interviewerbias–thenatureorbehaviouroftheinterviewermeansthatpeoplearereluctanttogivesomepossibleanswers.Thewaythatyouframediscussionsandwhoispresent,aswellasthewayyoulookandspeak–canhaveasignificanteffectonresponses.It’simpossibletoentirelyneutralisethesethings,butyoucanatleastbeawareoftheeffectyoumayhave.

• Biasedscales–thathave,forexample,lotsofoptionsfor‘good’,butonlyonefor‘bad’;forexample,excellent,verygood,fairlygood,notverygood.

QuestiondesignThemostimportantthingsinquestiondesignare:

• avoidleadingquestions.

• avoidbiasedscales.

• neverasktwoquestionsinone,youdon’tknowwhichpartpeoplehaveanswered;forexample,‘wouldyousaythatyouunderstoodthespeakerandthediscussant?’

• neveraskhypotheticalquestions,forexample,‘willyougobacktoworkafteryourbabyisborn?’Youcanask:‘doyouplantogobacktoworkafteryourbabyisborn?’

• makesuremeasurementbandsdon’toverlap.Sodon’task‘howoldareyou?’withasetoftickboxanswersthatrun15-40,40-60,60andover.Theanswersmustrun:15-39,40-59,60andover,otherwisethoseaged40and60don’tknowwhichboxtotick.

Thereareanumberofdifferenttypesofquestionsyoucanuseandbelowisanintroductiontoeach.

Pre-codedquestions

Thesearetheeasiestquestionsforpeopletoanswerandtheeasiesttoanalyse,buttheycanbedifficulttodraft.Pre-codedquestionstaketheformofashortquestionthatencapsulatesonethought,followedbyalistofpossibleanswers.Peopleareaskedtotickoneormoreoftheanswers.Atypicalquestionwouldbegender–peoplejustticktheirgenderfromthe‘male’,‘female’options.Anothertypicalquestionwouldbe:‘Whichsectionsoftheexhibitiondidyouenjoy?Pleasetickallthosethatapply.’Thequestionwouldbefollowedbyalistoftheexhibitionsectionsandpeoplewouldtickallthesectionsthattheyenjoyed.

Thesetypesofquestionsareeasytoanalysebecauseyoujustadd-upthenumberofpeoplegivingeachresponse.

Theycanbedifficulttodraftbecausetheyneedtobeveryclearandtheanswerlistmustberelevanttothequestionandunderstoodbytherespondent.

Ifyouareusing‘pre-coded’questions,youneedtobeconfidentthatthecategorieschosenreflectthespectrumofactualexperience.Ifyouaskpeoplewhattheirfavouritesubjectswereatschoolandoffer“maths”,“science”,“designandtechnology”and“other”asoptions,butyouractivitywasbasedinanartgallery,you’renotlikelytogetaveryaccuratepictureofpeople’srealfavourites.Youcanalwaysusean“other,writein”categorytocaptureanythingyou’vemissed,


butrememberyou’llhavetoreaditbeforeyoucananalyseit.Oryoucouldjusthave‘other’anddon’tforgettoadda‘don’tknow’option.However,ifthelististoolongsomepeoplewillonlytickthetopfewandnotreadthefulllist.Pilotingthequestionnairewillhelpyoutoensurethatyouhaveasfullalistaspossible.

Usingscales

The1-5Likertscaleisthemostcommonlyusedformofrating.Itissimpletounderstandandrelativelydiscriminating.Thescaleiscommonlyanchoreddescriptivelye.g.5=Agreestrongly,4=agree,3=neitheragreenordisagree,2=disagree,1=disagreestrongly.Youshouldalsoadda“don’tknow”category.Otherscalesthatareusedincludescoringonalineofonetotenorapercentagescore.

Anotherwayofdifferentiatingbetweenpeople’sviewistopresentthemwithstatementsthattherespondentchoosesbetween.Theseareoftenorderedonanimplicitscale,butyouareaskingtherespondenttoticktheonethatbestfitstheirviewwheninfacttheymaynotagreewithanyofthosepresented.Anexampleofthistypeofquestionis:

Whichofthefollowingstatementsbestreflectsyourfeelingsaboutsciencetoday?

a) It’scontinuallymakingourlivessaferandbetter.

b) It’schangedmanythingsforgood,butIwonderhowmuchmorethereisthatcanbeachieve.

c) It’sproducinglotsofnewthings,butI’mnotsureweneedthemall.

d) It’soutofcontrolanddamagingourlivesandenvironment.

Whenitcomestotheanalysisallyoucanreallydoispresentthepercentageofrespondentswhoagreewitheachstatement.

Ranking

Rankorderingisbestavoided–manyrespondentswon’tdoitproperly,unlessyousticktoaskingforfirst,secondandthirdchoices.Otherwisepeoplegetconfusedandgetpushedintodeclaringpreferences

theydon’treallyhaveortheymayjustgiveup.Youcangetroundthisbyaskinghowimportantanissueis,usingaLikertscale.

Attitudinalquestions

AttitudinalquestionsareusuallyaskedusingLikertscalesanditisusualtohaveasetofattitudinalquestionsthattogethergiveyouanoverallimpressionoftheattitudesoftherespondents.

Ifyouhaveasetofattitudestatements,someshouldbepositiveaboutyouractivityandanequalnumbershouldbenegative.Ingeneralpeoplearemorelikelytoagreewithstatementsthantodisagreeandyouneedtobeawareofthisinyouranalysis.Becarefulnottohavestatementssuchas“There is no point in studying history”.It’sverydifficultforpeopletoknowwhethertheyshouldagreeordisagree.Peoplewhoagreemightsay,‘no,thereisn’t’,ratherthan‘yes,Iagree’.Ifyou’reusingnegativestatementskeepitshort–forexample,“Studying history is pointless”.Thenpeoplecansay‘yes,itis’.

True/falsestatements

Agoodwaytotestwhetherpeoplehavelearntanything,orwhethertheyknewanythinginthefirstplace,istoasktrue/falsestatements,butyouneedtobesurethatthereiscertaintyontheissue.

Open-endedquestions

Open-endedquestionsallowrespondentstoanswerquestionsintheirownwords,butyouhavetoreadthemall.Also,somepeoplewillbeverybrieforsaynothing,soyoumaynotbeveryenlightenedbytheirresponses.However,anopen-endedquestionattheendofaquestionnaireactsasacatch-all.Ifyouwanttouseopen-endedquestionsherearesomeguidelines.

• Usebroadopeners:Who,what,where,when,and(especially)why,toencouragepeopletoexpressthemselves.

• Balancedopenquestions(whydidyoulike[it],whydidyoudislike[it])helptherespondentstructureananswerwithoutfeelingpressuredtogiveaparticularreply.


• Inface-to-faceinterviewsconductedbyaninterviewer,usingopenphrasessuchas‘Tellmeabout’,‘Tellmemore,’or‘Whydoyousaythat?’encouragespeopletotalk.

• Avoidquestionsthatcanbeansweredwithasimpleyesorno.

• Avoidaskingmorethanonequestionatthesametime.

Analysis

Thinkabouthowyouwillusethedatacollectedwhendraftingyourquestionnaire.

Ifyouwanttocalculatetheaverageageofvisitors,usingbands16-25,26-40,41-60,over60willnotallowyoutodoso.Youwillneedtoaskactualage.

Questionnairechecklist

Belowisachecklistthatwillhelptoensureyouhaveaclear,easytocompleteandanalysequestionnaire.Youcanjustinitialtherowwhenyou’vedonethattaskandaddthedatetokeeptrackofwhereyouare.Itmightbehelpfultoalsokeepnotesofwhyyoumadecertaindecisionsaboutthequestionnaire.

Section Initials Date Notes

Questionformat&wording

Doallquestionshaveoneunambiguousfocus(i.e.thereareno‘twoinone’questions)?

Doquestionshave‘don’tknow’codes(formostquestionsthiswillbeavalidresponse)?

Domultipleresponselistshave‘noneofthese’options(wherethiscouldbeavalidresponse)?

Dosingle-responsequestionshaveaninstructionaskingtherespondentto‘tickoneonly’?

Domultiple-responsequestionshaveaninstructionaskingtherespondentto‘tickallthatapply’?

Whereanswersaresoughtasnumericvalues,arethereinstructionstellingrespondentswhatformattoentertheirdatain(e.g.forfivepleasewrite/typeas5)andifnecessary,whatunit(e.g.UKpounds).Thisismoreimportantforon-linesurveysbutwillhelpwithdataentryofpaperquestionnaires.

Topandtail

Doestheintroductioncontainallnecessaryinformationaboutconfidentiality,purpose,lengthandcontactsforfurtherinformation?

Hasare-contactquestionbeenasked?(Follow-upresearchwillbedifficultwithoutthisastheInformationCommissioner’sOfficedoesn’treallylikeitanditcontravenestheMarketResearchSocietyCodeofConduct).


Section Initials Date Notes

Overview

Hasthequestionnairebeenpilotedorcheckedforsensebycolleagues?

Havequestionsbeenmappedagainsttheinformationyouneed?

Isthenumberofopenquestionsappropriateorisitlikelytocauseanunreasonableamountofcoding?

Isthenumberofopenquestionsappropriateorisitlikelytoputanunreasonableburdenontherespondents?


QuestionnairemodulesAnnex2

Thissectionprovidessomequestionnairemodulesthatcanbecopiedintoyourquestionnaire.Thesemoduleswillalsoenableyoutodesignsimilarquestionsonotherissuesthatyouwanttoaskabout.

Whowasthere?

Togetbasicsocio-demographicinformationusesimpletickboxes.

Thereareatremendousnumberofcharacteristicsthatcanbeexplored.Youneedtofocusonwhatmattersinrelationtoyourobjectives.Isitpeoplefromageographiclocation,ofaparticularageoracertainmindsetthatwereimportanttoyou?Doyouwantamixofgenderand/orethnicgroupsorareyouspecificallytargetingonegroup?

Peoplecanbeverysensitiveaboutgivingthistypeofpersonaldata,soyoushouldexplainwhyyouwantit.That’swhyitisusualtoleavethesetypesofquestionsuntiltheendofthequestionnaire.Ifpeoplereallydon’twanttogiveyouthisinformation,theywillstillhavegivenyoufeedback.Ifyouputthisfirstandtheydon’twanttocompleteit,theywon’tanswertheotherquestionseither.

Q Sothatwecanseethetypesofpeoplewhocametoourevent,foreachofthequestionsbelow,pleaseticktheboxthatbestfitsyou.

OR

Q Sowecananalysethefindingsfromthissurveyfordifferentgroups,foreachofthequestionsbelow,pleaseticktheboxthatbestfitsyou.

Areyou

Male

Female

Areyou

Infull-timeeducation

Employed–full-time

Employed–part-time

Notcurrentlyemployedbutlookingforwork

Retired

Other

Whichofthegroupslistedbelowdoyouconsideryourselftobelongto

White

Black-African

Black-Caribbean

Black-Other

Pakistani

Indian

Bangladeshi

Chinese

Other(pleasewritein)

Whatwasyouragelastbirthday?

Lessthan16

16-30

31-45

46-50

51-65

Over65

Approximatelyhowfarfromheredoyoulive?

Lessthan1mile

1-5miles

6-15miles

Morethan15miles

Don’tknow

Ifyouwanttocalculatetheaverageageofvisitors,youwillneedtoaskpeopletowriteintheiractualage.


Whatweretheirattitudes?

ForattitudinalinformationitisprobablymostappropriatetouseLikertscales.AgoodsourceofattitudequestionsisthePublicAttitudestoScienceseries.TheDepartmentforBusiness,InnovationandSkills(BIS)defines‘science’asfollows:

By ‘science’ we mean all-encompassing knowledge based on scholarship and research undertaken in the physical, biological, engineering, medical, natural and social disciplines, including the arts and humanities, which is underpinned by methodologies that build up and test understanding about our world and beyond. http://www.bis.gov.uk/policies/science/science-and-society

Themostrecentreportandquestionnairecanbeaccessedhttp://interactive.bis.gov.uk/scienceandsociety/site/

Theabovereportalsogivesthefiguresfortheresponsesofanationallyrepresentativesampleofadultstothesestatements.Thismeansthatyoucandevelopapictureofhowtypicalyouraudienceisofthewiderpublic.

Belowisaseriesofstatementsthatpeoplehavesaidaboutscience.Foreachpleasetickwhetheryouagreestrongly,agree,neitheragreenordisagree,disagree,disagreestronglyordonotknow.

Statement Agree Agree Neither Disagree Disagree Don’tKnow Strongly Strongly

ThespeedofdevelopmentinscienceandtechnologymeansthatitcannotbeproperlycontrolledbyGovernment.

Scienceisgettingoutofcontrolandthereisnothingwecandotostopit.

Itisimportantthatyoungpeoplehaveagraspofscienceandtechnology.

Thebenefitsofsciencearegreaterthantheharmfuleffects.

Scienceandtechnologyaremakingourliveshealthier,easierandmorecomfortable.

http://www.bis.gov.uk/policies/science/science-and-society

http://www.bis.gov.uk/policies/science/science-and-society

http://interactive.bis.gov.uk/scienceandsociety/site/

http://interactive.bis.gov.uk/scienceandsociety/site/


UKlawstatesthatallmedicinesmustbetestedonanimalsbeforebeingusedonpeople

PleaseputXintheappropriatebox

True

False

Don’tknow

AnyresearcherintheUKcancarryoutexperimentsthatinvolvepeoplewithoutalicence


True

False

Don’tknow

ThetestingofnewdrugsonpeopleisnotregulatedintheUK


True

False

Don’tknow

Youneedalicencebeforeyoucanplantgenetically-modified(GM)cropsintheUK


True

False

Don’tknow

TheGovernmentdoesnotregulatethedisposalofradioactivewasteintheUK


True

False

Don’tknow

Therearestrictlawsgoverningtheuseinresearchofgovernmentpapers


True

False

Don’tknow

Didtheeventwork?

Atonelevelyoumightwanttoknowifpeoplesimplyenjoyedengagingwiththeactivity.Givethemthechancetotellyou,butyoucangetmorevaluebyfollowing-upthequestionandaskingthemtosaywhy.Youcandothisbyhavingalistofpossiblereasonsandaskingthemtotickthosethatapplytothem.Oryoucanaskthemtowrite-inusingtheirownwords.This“writein”approachworksbestforsmallnumbersofrespondents,becauseyou’llneedtoreadwhatpeoplehavewritten.Forlargernumbersofpeople“pre-codes”willbemoremanageable,butleaveopenthe“other”optionsothatyoucancaptureanswersyouhadn’tthoughtof.

Youmightbelookingformoresophisticatedfeedback.Iftheprimaryfunctionoftheactivitywastogiveparticipantsthechancetocontributetheirviewsandcomments,itisimportanttoseewhetherthishasbeenachievedandwhatfactorshaveenabled,orhindered,effectiveparticipation.

Ifinformationprovisionwaspartoftheprocesswasitaccessibleanduseful?Similarly,ifyouwereusing“experts”howwastheircontributionrated?

Whatdidtheylearn?

Herearesomeexamplesoftrue/falsequestions.Youwillbeabletodevelopyourown,dependingonyourfield.

Foreachofthestatementsbelow,pleasetickwhetheryouthinkitistrueorfalseorwhetheryoudon’tknow.


Youmightalsothinkaboutwhattheexpertsexperienced.Didtheyenjoytheprocess,whathavetheylearned,havetheirattitudeschanged?

Theexamplesbelowshowsomeofthequestionsthatyoumightconsiderasking.Forthese“diditworkforyou?”questions,themostvaluablebitoffeedbackcanbethewhyorwhynotthatunderpinstheyesornoanswer,soitisalwaysworthleavingsomespaceforthis.

1.Ienjoyedtheevent

Pleaseticktheboxthatbestdescribesyourview

StronglyAgree Agree

Disagree StronglyDisagree

2.Pleasewriteinwhichpartoftheeventyouenjoyedthemost

3.Pleasewriteinwhichpartoftheeventyouenjoyedtheleast

4.WhatdoyouthinkIshouldchangeabouttheevent?

Wereyouabletoexpressyourviewsfreelyandopenly?


Yescompletely

Yes,butsometimesIfeltnervous

NotasmuchasIwouldhaveliked(ifyoutickthispleasesaywhyintheboxbelow)

Notatall(ifyoutickthispleasesaywhyintheboxbelow)

Whywasthis?Pleasewrite-in

Don’tKnow

Didyouunderstandthetalk?


Yes,easily

Yes,butonlyafterthediscussion

Notverywell(ifyoutickthispleasesaywhatmighthavehelpedintheboxbelow)

Notatall(ifyoutickthispleasesaywhatmighthavehelpedintheboxbelow)

Whatmighthavehelpedyouunderstandthesciencemoreeasily?

Pleasewrite-in

Don’tKnow

Didyoufindtheexperts:

PleaseputXinanyboxyouagreewith

Helpful?

Confusing?

Abletoansweryourquestions?

Selfimportant?

Didnotwanttolistentoyouropinions?

Abletoexplainthemselvesclearly?

Eagertolisten?

Ifyou’dliketosayanythingelseabouttheexperts,pleasewriteitintheboxbelow


Annex3SummativeEvaluationSchema

Belo

wis

as

chem

ato

hel

pyo

uth

ink

abou

tth

ety

peo

finf

orm

atio

nyo

uw

ant

and

how

you

mig

htc

olle

ctit

,dep

endi

ngo

nth

ena

ture

ofy

our

activ

itya

ndy

our

obje

ctiv

es.R

emem

ber:

Tom

easu

rec

hang

eyo

une

edt

oha

vea

bas

elin

efro

mb

efor

eth

eau

dien

cee

ngag

edw

ithy

our

activ

itya

nda

noth

ers

eto

fdat

ata

ken

afte

rth

eyt

ook

part

.You

will

need

to

ask

exac

tlyt

hes

ame

ques

tions

bef

ore

and

afte

r.

Nat

ure

ofd

ata

Dis

cuss

ion/

M

eeti

ng/T

alk

Web

site

Pro

duct

se.

g.p

ost

er/

CD

RO

M/v

ideo

Exh

ibit

ion/

Ope

nda

yS

how

/Pla

yC

om

peti

tio

n

Num

ber

ofp

eopl

e.C

ount

peo

ple

on

entr

y.C

ount

hits

.N

umbe

rdi

strib

uted

.C

ount

peo

ple

one

ntry

.C

ount

aud

ienc

eon

en

try.

Cou

nte

ntrie

s.

Type

sof

peo

ple.

Cat

egor

isep

eopl

eat

reg

istra

tion

or

byo

bser

vatio

nor

qu

estio

nnai

re

Pop-

upq

uest

ionn

aire

son

the

site

or

regi

stra

tion

proc

edur

es.

Use

ofo

rder

/req

uest

fo

rms

and

ques

tionn

aire

s.C

ateg

orise

peo

ple

one

ntry

by

regi

stra

tion

orq

uest

ionn

aire

.U

set

icke

tsa

les

or

book

ing

mec

hani

sms

to

gath

erin

form

atio

n.

Cou

nte

ntrie

s.

Mo

nito

ring

Dat

a

Eva

luat

ion

Dat

a

Nat

ure

ofd

ata

Dis

cuss

ion/

M

eeti

ng/T

alk

Web

site

Pro

duct

se.

g.p

ost

er/

CD

RO

M/v

ideo

Exh

ibit

ion/

Ope

nda

yS

how

/Pla

yC

om

peti

tio

n

Cha

nge

Cha

nge

view

s/at

titud

es.

Cha

nge

beha

viou

r.In

crea

sein

tere

st.

Incr

ease

kno

wle

dge.

Ask

peo

ple

for

base

line

view

son

a

pape

rqu

estio

nnai

re

whi

let

hey

wai

tfo

rth

eev

ent

tos

tart

or

whe

nth

eyr

egist

ert

oco

me.

Regi

stra

tion

ques

tionn

aire

on

the

site

tog

athe

rin

form

atio

n.

Dist

ribut

ion

met

hods

w

illaf

fect

the

abi

lity

to

colle

ctin

itial

dat

a.U

sing

ano

rder

ing

mec

hani

sm

allo

ws

data

to

be

gath

ered

.

Ask

for

base

line

view

son

a

pape

ror

e-m

ailq

uest

ionn

aire

w

hen

peop

ler

egist

ert

oco

me

orb

uyt

icke

ts.

Ask

for

base

line

view

son

ap

aper

or

e-m

ail

ques

tionn

aire

whe

npe

ople

boo

kor

buy

tic

kets

.

Build

ing

ina

nin

itial

da

tag

athe

ring

exer

cise

to

the

com

petit

ion

proc

ess

will

allo

w

base

line

data

to

be

gath

ered

.

Qua

lity/

fit

for

purp

ose

Stre

ngth

s.W

eakn

esse

s.

Obs

erve

the

ev

ent.

Use

exi

tqu

estio

nnai

res

and/

orfo

llow

-up

focu

sgr

oups

or

ques

tionn

aire

s.

Incl

ude

ques

tions

on

this

ina

que

stio

nnai

re

host

edo

nth

esit

e.

Reco

rdd

wel

ltim

epe

rpa

gea

ndp

age

requ

ests

.

Follo

w-u

pqu

estio

nnai

res

and

focu

sgr

oups

.Ex

itor

follo

w-u

pqu

estio

nnai

res.

Shor

tfa

ce-

to-fa

cein

terv

iew

sdu

ring

the

even

t.O

bser

vatio

n.

Follo

w-u

pqu

estio

nnai

res.

Gro

up

disc

ussio

ns.

Use

ent

rym

echa

nism

to

gat

her

feed

back

.

Inte

ract

ion

wit

hac

tivi

ty.

Obs

erva

tion

of

dyna

mic

sw

illhe

lpy

ou

plan

bet

ter

even

tsin

th

efu

ture

.

Reco

rdt

heo

rder

in

whi

chp

ages

are

ac

cess

eda

ndd

wel

ltim

epe

rpa

ge.

Obs

erva

tion

ofu

sers

and

qu

estio

nnai

res.

Obs

erva

tion.

In-d

epth

inte

rvie

ws

orfo

cus

grou

psa

ndq

uest

ionn

aire

s.Fe

edba

ckfr

oms

taff/

colle

ague

s.

Obs

erva

tion.

Q

uest

ionn

aire

sIm

plic

itin

tak

ing

part

,us

een

try

num

bers

as

am

easu

re.

Lev

elo

fdi

scus

sio

nO

btai

nvi

ews

onis

sue.

List

enin

gto

the

co

nver

satio

ns,r

ecor

dke

ypo

ints

.

An

inte

ract

ive

e-m

ail

faci

lity

will

allo

wt

his.

Not

ag

ood

med

ium

for

gett

ing

peop

le’s

view

s.C

an

use

thes

eas

as

timul

usa

nd

then

use

gro

upd

iscus

sions

an

dqu

estio

nnai

res

Com

men

tbo

oks

and

exit

ques

tionn

aire

s.Bu

ildin

op

port

uniti

esfo

rst

aff/

colle

ague

sto

eng

age

with

vi

sitor

s.

Not

usu

ally

des

igne

dfo

rgi

ving

feed

back

.C

anu

sed

ebat

eaf

ter

the

perfo

rman

ce.

Can

bui

ldt

his

int

oen

try

proc

ess,

but

not

ano

rmal

mec

hani

sm

for

gett

ing

peop

le’s

view

s.


GlossaryAnnex4

ActivityUnlessaspecificactivityorapproachisbeingdiscussed,thisguideusesthetermactivityasanall-encompassingphrasefortalks,shows,teachers’packs,hands-onevents,websitesandthemanyotherwaysthatresearchersareusingtoengagegeneralaudiences.

AimTheaimofyouractivityiswhatyouultimatelywanttoachieve.Theaimissupportedbyanumberofobjectivesthatwillhelpyoutorealisetheoverallgoal.

AudienceTheaudienceisthepeoplewithwhomyouaretryingtoengage.

BaselineAmeasureatthebeginningofanactivitythatenablesdeterminationofchange,ifany.

CensusAsurveythatcollectsdatafromeveryone.

ChartingAmethodforanalysingqualitativeresearchdata.

DataInformationcollectedthroughmonitoringandresearch.

EvaluationEvaluationhelpsyoutoseewhetherornotyouhaveachievedyourobjectivesandtoidentifywaystoimprovewhatyoudoduringandafteryouractivity.

EvaluationstrategyTheplanthroughwhichyouwilldeterminewhetherornotyouhaveachievedyourobjectives.

ExitsurveyAsurveyofpeopleundertakenastheyleaveanevent,exhibition,etc.Usuallyconductedbyaninterviewerratherthanaresearcher.

Face-to-faceinterviewsUsedinmarketandsocialresearchtomeanstructuredquantitativesurveysconductedface-to-face.

FocusgroupAresearchmethodthatinvolvesagroupofusually6-8peopleconvenedtodiscussaparticulartopic.

FormativeevaluationResearchthattakesplaceduringthedevelopmentofanactivitytoensureitmeetstheaudiences’needs.

FundersThefundersarethepeoplewhoprovidetheresourcesthatallowyoutoundertakeyouractivity.

In-depthinterviewAninterviewconductedbyaresearcherusingatopicguide,whichallowsrespondentstoexpressthemselvesintheirownwayandraiseissuestheresearcherhasnotconsidered.

InterviewerApersonwhoconductsinterviewsfollowingapredeterminedquestionnairedesignedbyaresearcher.

MilestonesMilestonesareinterimmeasuresthatallowyoutomonitorwhetherornotyouareontracktomeetyourobjectives.

ObservationFormalisedobservationofbehaviour,eitherdirectlyorfromarecording.

ObjectivesObjectivesarethetangiblethingsthroughwhichyouwillachieveyouroverallaim.

OutcomesOutcomesaremeasuresoftheimpactyouhavehadonpeople.


OutputsOutputsarethethingsyouproduceaspartoftheactivitye.g.awebsite,aleaflet.

Pop-upquestionnaireAquestionnairethatliterally‘pops-up’onenteringawebsitetocollectinformationaboutusersandusageofthesite.

ProgrammeThereisnoblueprintforaprogramme,itislikelytocontainoneormoreofthefollowingfeatures:

• Afundingmechanismtowhichotherpeopleororganisationscanapply.

• Abudgettocommissionspecificpiecesofwork.

• Resourcestoundertakein-houseactivities.

• Areportingprocess,throughwhichtheprogrammemanagerbidsforresourcesandaccountsfortheiruse.

ProgrammemanagerSomeonewhohasoverallresponsibilityfordeliveringagainstasetofpre-determinedobjectives,andusesavarietyofactivitiesandactionstoachievetheseobjectives.Oftenworksforafunder.

ProjectmanagementProjectmanagementinthiscontextissimplytheproceduresthroughwhichyouensureyoudeliveryouractivity.

ProjectmanagerTheprojectmanageristhepersonultimatelyresponsiblefortheactivity.

QualitativeresearchTechniquesthatallowpeopletoexpressthemselvesintheirownwordsandtoraisetheirconcerns,usuallyviain-depthinterviewsandfocusgroupsconductedbyresearchers,helpsyoutounderstandwhypeopledoorsaywhattheydoorsay.

QuantitativeresearchTechniquesthataskpeoplethesamequestionsinsuchawayastoenabletheanswerstobeaddedtogetherforasamplethatisrepresentativeofthetargetgroup,thusprovidingnumericaldataonthepercentageofpeoplewithparticularviewsorbehaviour.

QuestionnaireAstructuredsetofquestionscallingforapreciseresponsethatallowsanswersfromallthosewhocompleteittobeaddedtogether.

QuotasamplingSetting‘quotas’toensureasamplehasthesamepercentageofpeoplewithspecificcharacteristicsasthepopulationofinterest.Requiresotherdatagivingtheinformationonthepopulation.

ResearcherApersonwhoisinvolvedindesigningandoverseeingaresearchproject.

SamplingAwayofselectingpeopletotakepartinresearchthatensurestheyarechosentoberepresentativeofthepopulationofinterest,althoughnotalwaysinastatisticalsense.

SMARTAllobjectivesshouldbeSMART,whichstandsfor:

• Specific;

• Measurable;

• Achievable;

• Relevant;and

• Time-bound.

StakeholdersThosewhohavealegitimateinterestinyouractivity,e.g.audiencesandfunders.


Sub-contractorsSub-contractorsarepeopleororganisationsemployedbytheprojectmanagertodeliverspecificallydefinedproductsorservices.

SummativeevaluationEvaluationattheendofanactivitythatdetermineswhetherornottheobjectiveshavebeenmet.

SystematicsamplingTakingevery‘nth’personwhoengageswithanactivity.Producesastatisticallyrepresentativesample.

TopicguideAlistofquestionsandissuesaresearcherwantstocoverduringanin-depthintervieworfocusgroup.

UserForthepurposesofthisguide,someonewhoengageswithanactivity.


FurtherreadingAnnex5

FindinginformationonevaluationTheterm‘evaluation’isusedwidelyineducation,socialpolicy,andtraininganditisintheseareasthatyoucanfindpapersandbooksdevotedtoevaluationasadistincttradition.Elsewhere,toolsandtechniquesusedinevaluationaresimplytheapplicationofresearchmethodologies.Thereislittlewrittenthatisspecifictopublicengagementorcommunicatingaboutresearch–itisoftenacaseofborrowingandadaptingmethodsthathavebeenusedsuccessfullyinotherfields.

EvaluationMethodsBreakwell, G. and Millward, L, (1995) Basic evaluation methods BPS Books, Leicester. 145 pp

Agoodgeneralintroductiontoevaluation,whichcanbeappliedinarangeofsettings.Containscasestudiesandexamplesthatarerelevanttocommunicatingresearchfindings,e.g.evaluationofamuseumexhibition,andcoversawiderangeofresearchmethods,includingquestionnaireconstructionandtimeseries.Containsaninterestingsectiononthepoliticsofcommunicatingevaluationresultstoaudiences.

WebResourcesThemainwebresourcesthatwillhelpyouwithyourevaluationarethewebsitesofvariousevaluationsocieties(usuallyspecialisingineducationalorsocialpolicyresearch).Therearealsosomeuseful‘hintsandtips’sites,mostlyUS-based,writtenbyacademicsandgeneralenthusiasts,allyouneedtodoisentersearchtermsinasearchengine.Thereisanincreasingsetofworkthataimstohelpthosewhowanttoevaluatetheirpublicengagementactivities.

GeneralEvaluationResources

http://www.evaluation.org.uk/resources/online-resources.aspxWebsiteoftheUKEvaluationsociety.

www.eval.orgTheAmericanEvaluationAssociation.

http://www.reviewing.co.uk/evaluation/index.htmGeneralarticlesongettingthebestoutofcourseevaluation,withtipsandlinks.

http://gsociology.icaap.org/methods/SetoflinkstoUS-basedevaluationresources,withabasicguideforthenon-specialist.

http://www.mapnp.org/library/evaluatn/fnl_eval.htmAUS-basedbasicguidetoprogramevaluationincludingoutcomesevaluation.Usefuloverviewofmethodsandissues.

www.mrs.org.ukTheUKMarketResearchSociety’swebsitehasup-to-dateguidanceonethicsandconfidentialityininterviewingthepublic.

www.dfes.gov.ukDepartmentforEducationhasdataonindividualschoolexaminationresultsatvariousKeyStages,GCSEand‘A’level.

http://www.cdc.gov/eval/resources.htmDigitalCreativityspecialissueonevaluation,2009.Widerangeofprofessionalon-lineevaluationresources.Comprehensivelistingofevaluationresources,materials,guidelinesetc.(Americanbased).

http://www.evaluation.org.uk/resources/online-resources.aspx

http://www.evaluation.org.uk/resources/online-resources.aspx

www.eval.org

http://gsociology.icaap.org/methods/

http://www.mapnp.org/library/evaluatn/fnl_eval.htm

http://www.mapnp.org/library/evaluatn/fnl_eval.htm

www.mrs.org.uk

www.dfes.gov.uk

http://www.cdc.gov/eval/resources.htm


Evaluationresourcesaimedatthoseengagingthepublicintheirresearch

http://www.raeng.org.uk/societygov/public_engagement/ingenious/evaluation.htmTheIngeniousEvaluationFAQsgiveexamplesofgoodpracticeinevaluationandideasonusingcreativeapproaches.

http://caise.insci.org/resourcesExcellentoverviewofAmerican(NSFfunded)evaluationresources,includingtoolkitsplusworkacrossvariousinformalenvironmentsincludingzoos&aquaria.

http://caise.insci.org/resources/Eval_Framework.pdfNSFevaluationframework.

http://practice.ie/blogs/heather/ideas-and-examples-creative-evaluation-kidsCAISEFrameworkforEvaluatingImpactsofInformalScienceEducationProjects.

http://www.scidev.net/en/practical-guides/evaluating-science-communication-projects-1.htmlJoubert,M.(2007)Evaluatingsciencecommunicationprojects.SciDevNet,8thJanuary2007.

http://www.impact.arts.gla.ac.uk/TheImpactDatabaseisabibliographicaldatabaserelatingtoresearchonthesocialandeconomiceffectsofarts,cultureandmajorevents.

http://www.publicengagement.ac.uk/TheNationalCo-ordinatingCentreforPublicEngagementisfundedbytheUKHigherEducationFundingCouncils,ResearchCouncilsUKandtheWellcomeTrusttohelpacademicsengagewiththepublicintheirresearch.Thewebsitehasahostofresourcestosupportuniversitystaffandstudentstodeveloppublicengagementactivities.

http://www.ucl.ac.uk/public-engagement/research/toolkits/MethodsAseriesofdocumentscollatedbyUniversityCollegeLondontosupportacademicsintheevaluationoftheirpublicengagementactivities.

http://www.ucl.ac.uk/public-engagement/research/frameworkAframeworkforevaluationputtogetherbytheUCLBeaconforPublicEngagementforacademicswhowanttoevaluatetheirpublicengagementactivities.

http://www.manchesterbeacon.org/files/manchester-beacon-pe-evaluation-guide.pdfAguidetoevaluationputtogetherbytheManchesterBeaconforPublicEngagementforacademicswhowanttoevaluatetheirpublicengagementactivities.

http://www.britishscienceassociation.org/forms/scicomm/evaluation/access/ThisdatabaseismanagedbytheBritishScienceAssociationandisaresourceforthoseinterestedinengagingpeoplewithscience&engineering.Theobjectiveofthedatabaseistosharelearning.Itisfreetouploadyourevaluationandtoaccessthedatabase.

Ifnoneofthesesuit,tryenteringkeywordsintoasearchengine.

Forhelponplanningyouractivitysee:http://www.vitae.ac.uk/CMS/files/upload/The_engaging_researcher_2010.pdf

http://www.raeng.org.uk/societygov/public_engagement/ingenious/evaluation.htm

http://www.raeng.org.uk/societygov/public_engagement/ingenious/evaluation.htm

http://caise.insci.org/resources

http://caise.insci.org/resources/Eval_Framework.pdf

http://caise.insci.org/resources/Eval_Framework.pdf

http://practice.ie/blogs/heather/ideas-and-examples-creative-evaluation-kids

http://practice.ie/blogs/heather/ideas-and-examples-creative-evaluation-kids

http://www.scidev.net/en/practical-guides/evaluating-science-communication-projects-1.html



http://www.publicengagement.ac.uk/

http://www.ucl.ac.uk/public-engagement/research/toolkits/Methods

http://www.ucl.ac.uk/public-engagement/research/toolkits/Methods

http://www.ucl.ac.uk/public-engagement/research/framework

http://www.ucl.ac.uk/public-engagement/research/framework

http://www.manchesterbeacon.org/files/manchester-beacon-pe-evaluation-guide.pdf

http://www.manchesterbeacon.org/files/manchester-beacon-pe-evaluation-guide.pdf






Research Councils UK Polaris House, North Star Avenue Swindon Wiltshire SN2 1ET United Kingdom

www.rcuk.ac.uk

April 2011. All cartoons © Noel Ford

Produced by RCUK’s in-house service provider