ONS Census Transformation Programme

Summary of feedback on the 2015 administrative data research outputs

May 2016

Summary of feedback on the 2015 administrative data research outputs

1 Office for National Statistics

Table of contents

1. Introduction 2

2. Summaryoffeedback 3

2.1 Levelandqualityofinformationpresentedintheresearchreport 3

2.2 Improvementstothemethodologyusedtoproduceadministrativedata populationestimates 3

2.3 Otherpotentialadministrativedatasources 5

2.4 PossibleexplanationsforthedifferencesbetweentheResearchOutputsand theofficialestimates 5

2.5 Usefulnessoftheinteractivecontent 6

2.6 GeneralcommentsonhowtoimprovetheResearchOutputs 7

3. PlansforAutumn2016ResearchOutputsRelease 8

AnnexA SummaryoftheactionsthatONSplanstotakeinresponsetofeedback 9

Summary of feedback on the 2015 administrative data research outputs

2 Office for National Statistics

1. Introduction

ONSwouldliketothankeverybodywhoprovidedfeedbackontheAdministrativeDataResearchOutputsthatwerepublishedinOctober2015.

Followingpublication,userswereinvitedtoprovidefeedbackon:

• thelevelandqualityofinformationcontainedinthemainreport

• howthemethodologycouldbeimproved,includingsuggestionsofadditional administrativedatasourcestoachievethis

• possibleexplanationsforthedifferencesbetweentheResearchOutputsandthe officialestimates

• theusefulnessoftheinteractivecontent

• anygeneralcomments

Thepurposeofthisreportistoprovideanoverviewofthefeedbackreceivedincludingplanstomakeimprovementsasaresult.Section2focusesonthefeedbackthatwasprovided,Section3summarisestheactionsthatONSplanstotakeinresponsetothisfeedbackandSection4outlinestheplansforthenextreleaseofResearchOutputsduetobepublishedinautumn2016.

Summary of feedback on the 2015 administrative data research outputs

3 Office for National Statistics

2 Information from administrative data sources about when individuals have interacted with systems or services, such as the National Insurance, tax or benefits systems, or a hospital visit through the NHS system.

1 Once individual records have been matched across data sources using the approach outlined in Estimating the Population: In Practice, the information is pulled together into a single, coherent dataset that forms the basis for estimating the population. This is called a Statistical Population Dataset (SPD).

2. Summary of feedback

FeedbackwassoughtthroughasetofstructuredquestionsviaafeedbackformthatwaspublishedalongsidetheResearchOutputs.Thissectionprovidesasummaryofthecommentsreceivedoneachofthetopicsthatrespondentswereaskedtoconsider,includingactionsthatONSplanstotakeinresponsetothefeedback.FeedbackwasreceivedfromanumberoflocalauthoritiesandGovernmentDepartments.

2.1 Levelandqualityofinformationpresentedintheresearchreport

AthemethroughoutthemajorityofresponseswasthatthemainreportwascomprehensiveandgenerallyprovidedaclearaccountofhowtheStatisticalPopulationDataset 1(SPD)hadbeenconstructed.Themethodologyusedwaswellcoveredandthecasestudiesprovidedusefulexplanationsofthelimitationsofthecurrentmethod.Somerespondentscommentedthatitwouldbemorebeneficialtoselectgenericissuesthatwerecommonacrossareassimilartotheirownforfuturecasestudies.ThiswouldhelptobetterunderstandthediscrepanciesbetweentheResearchOutputsandtheofficialestimates.

Respondentssuggestedthatamoreconcisemethodologysectionwouldbeapositivestepinthefutureasitwouldreducethesizeofthereport.Howeveritwasrecognisedthatthislevelofdetailwasnecessaryforthefirstreleasetoexplainarathercomplexnewapproachtoestimatingthepopulation.

Action1:ONSiscurrentlyreviewingtheapproachfordisseminatingfutureResearchOutputs.Optionsunderconsiderationare:

• reducingthesizeofthereport

• enablinguserstonavigatethroughthereportmoreeasily

• makinguseofblogstocommunicatekeyqualityissuesorpresentcasestudy-typeanalyses,and

• makingmoreuseofvisualapplications

2.2 Improvementstothemethodologyusedtoproduceadministrativedata populationestimates

ThefeedbackhighlightedtheneedforaPopulationCoverageSurveytoassessandadjustforcoverageerrorintheadministrativedata.Referencewasmadetotheneedfor‘activitydata’ 2 tostrengthentheSPDexclusionrulesandreducetheover-coverageontheSPD,especiallyforworkingagemales.Itwasalsosuggestedthatsuchdatamighthelptoreducetheproportionofhalf-weights(whererecordsareweighted0.5atconflictingadministrativedatasourcelocations).RespondentsalsoencouragedfurtherresearchintotherulesthatareusedtodeterminewhichrecordsontheadministrativesourcesshouldbeincludedintheSPD.

Summary of feedback on the 2015 administrative data research outputs

4 Office for National Statistics

Action2:ONSiscurrentlyexploringhowactivitydatacanbeusedtoimproveadministrativedatapopulationestimates(usingdatafrominteractionswithaDWP/HMRCadministrativesystemandfrompatientsinteractingwithNHSsystemsthroughnewregistrationsortoupdatetheiraddressandotherdetails).

Action3:ONSisconsideringatestofthePCSin2018.Datacollectedthroughthisexercise,datacollectedinthe2017Censustestand,wheresuitableandpossible,existingsurveydatawillbeusedtodemonstratehowadministrativeandsurveydatacanbecombinedtoproducehighqualityestimatesaboutthesize(andcharacteristics)ofthepopulation.

AnothersuggestionwastopublishmeasuresorlevelsofuncertaintyaroundtheResearchOutputestimates,whichwouldenableanindependentassessmentofthequalityoftheSPDs.

Action4:ONSplanstoreviewthecurrentqualitystandardsusedtoassessthequalityoftheSPDs.Thisworkincludesexploringhowanindependentqualitymeasure(suchastheproductionofconfidenceintervalsaroundtheSPD)mightbeproduced.Inthelongerterm,oneuseforthePCSwillbetoprovideconfidenceintervalsaroundtheestimates.

Somerespondentsexpressedconcernsaroundthestabilityofadministrativedataovertimeaschangestotheadministrativedatacouldpotentiallyaffectthemethodologyandtheestimates.Similarly,changestopolicyorprocessesforcollectingthedatacouldhaveanimpactonthecontinuedqualityandaccuracyofthedata.TheSPDshavebeendesignedtomakeuseofarangeofdatasources,bothtomaximisequalityandtoavoidrelianceonasinglesourceofdata.

TheGovernmentrecentlyconsultedonarangeofproposalsfornewlegislationtoenableBetterUseofDatainGovernment,includingforONS’sroleinproducingNationalandOfficialStatisticsandstatisticalresearch.Proposednewlegislationwouldinclude:

“An obligation to consult the National Statistician before changes to data collection are made in order to protect the security of data supply, and the accuracy and reliability of statistical outputs derived from these data”.Thiswouldhelptoimprovetheresilienceoftheadministrativedata,andthereforetheSPDs,overtime.ProposednewlegislationisdiscussedinmoredetailinSection6.1ofthe‘AnnualAssessmentofONS’sprogresstowardsanAdministrativeDataCensuspost-2021’publishedalongsidethispaper.

Action5:ONSwillcontinuetoworkcloselywithdatasupplierstounderstandandimprovestatisticalqualityissuesidentifiedinthedata.ADataSuppliersGrouphasbeensetupwhichincludesmembersfromkeyGovernmentDepartmentsacrossWhitehall.Thisgroupisaforumfordiscussingissuesrelatingtodatasharing,sharingresearchandbuildingstrongerrelationships.Ifnewdatasharinglegislationgoesahead,thisgroupwillbevitaltothesuccessofunlockingdataacrossGovernmentasworkwillneedtobedonetoputinplacepracticalarrangementsfor:

• datasupply• protectingthatsupplyinthefuture,and• improvequality

Summary of feedback on the 2015 administrative data research outputs

5 Office for National Statistics

2.3 Otherpotentialadministrativedatasources

Variousadministrativedatasourcesweresuggestedaspotentialadditionstoimprovethecurrentmethodology.Themainonesarelistedbelow:

• CouncilTaxdatacouldbeusedfortypeofaccommodation,numberofrooms,vacant properties,HMO(HousesinMultipleOccupation)flagandspecialtypesofoccupier(for exampleallstudenthouseholds)

• LifetimeLabourMarketData(L2)activeNationalInsurancepaymentscouldbeusedas ‘activitydata’toindicatewhetheranindividualispartoftheusuallyresidentpopulation

• IndividualElectoralRegister(IER)couldbeusedtoprovideinformationtoindicateHMOsand largehouseholds,nationalityandtoimprovecoverageforolderages

• ChildBenefitsdatacouldimprovethecoverageof0-4yearolds

• DVLAcouldprovide‘activitydata’1onapplicationsforvehicleregistrationandroadtaxto helpimprovethequalityofpopulationestimatesandinformationaboutthenumberofcars andvans

• Educationdatacouldincludeindependentschoolsandboardingschoolsnotcoveredbythe SchoolCensusdata,andalsoprovideinformationaboutqualifications

• SchoolCensusdata(EnglandandWales)wassuggestedtoimprovethecoverageforschool agechildrenintheSPD

• UniversalCreditdatacouldbeusedas‘activitydata’ 3

SuggestionswerealsomadeaboutsourcesthatmayhelptoimproveestimatesforspecialpopulationgroupssuchasinformationaboutarmedforcesfromtheMinistryofDefence,andprisondatafromtheMinistryofJustice.

AnnexAoftheAnnualAssessmentreportprovidesanupdateonthedatathatONShasobtainedaccessto,anddatathatwearefocusingonnext.

Action6:ONSaimtoshowdevelopmentstothemethodsforproducingestimatesaboutthesizeofthepopulation(includingtheuseofSchoolCensusandDWP/HMRCactivitydata)inthenextreleaseofResearchOutputsinautumn2016.

2.4 PossibleexplanationsforthedifferencesbetweentheResearchOutputsand theofficialestimates

Anumberofpossiblefactorsweresuggestedbyrespondentstoexplainthedifferencesbetweentheadministrativedatabasedestimatesandtheofficialestimatesfortheirparticulararea,including:

• areaswithhighpopulationchurn(inparticular,foryoungadults)

• areaswithhighvolumesofinternationalmigration

• studentpopulations

3 Using ‘activity data’ (for example, when individuals have interacted with systems or services, such as the National Insurance, tax or benefits systems, or a hospital visit through the NHS system) could help reduce levels of over-coverage that are seen for particular age groups.

Summary of feedback on the 2015 administrative data research outputs

6 Office for National Statistics

• prevalenceofarmedforcesincludingforeignarmedforcesandtheirdependents

• highproportionsofsecondhomeownership

• highproportionsofpart-timeresidency

• areaswhereprivateeducationanduptakeofprivatehealthcareismoreprevalent(asthese servicesarenotcoveredontheadministrativedatausedinthecurrentmethodology)

• communitiesthatarereluctanttointeractwithservicesandpublicbodies

Someoftheseexplanationswerecoveredinthemainreportandareknownissueswiththemethodology.

Akeyconcernforrespondentswashowlagsonadministrativedata(forexample,throughdelaysinregisteringforservicesorupdatingaddressinformation)mightaffectthequalityoftheSPDsandtheirabilitytospotchangeovertime.Thiscanvaryacrosspopulationgroupsandareas.Forexample,youngermales(whoaregenerallyrecognisedasbeingslowtoupdatetheirrecordswhentheychangeaddress),orareasthathavehighlevelsofpopulationchurn.Similarly,areaswithhighproportionsofspecialpopulationgroups,suchasprisonersorarmedforcespersonnelmaybeimpactedasthesegroupsmayinteractdifferentlywithadministrativeservices.

Action7:ONSwillpublishadministrativedatapopulationestimatesdowntoLowerlayerSuperOutputArea(LSOA)andbysingleyearofage(atLAlevel)inautumn2016.Analysingdataattheselevelswillhelpidentifyagegroupsandareasthathavethebiggestdiscrepancyfromofficialpopulationestimates,andwillhelptofocusfurtherrefinementofthemethodologyandthedatasourcesused.

Action8:ONSisworkingcloselywiththreegroupsoflocalauthoritiestobetterunderstanddiscrepanciesbetweentheSPDsandofficialpopulationestimates.Wherepossible,acollaborativeapproachwillbetakenonfuturecasestudies.

2.5 Usefulnessoftheinteractivecontent

Theinteractivecontentwasgenerallywell-receivedbyusers.Feedbackmentionedthatthedatavisualisationtoolwaseasytouseandhelpfulinundertakingquickanalysisofanareaagainstthecorrespondingofficialestimateforthereferenceyears.Someenhancementsweresuggestedasfollows:

• includetotalpopulationsalongsidemalesandfemales

• includeafacilitytoincludecomparisonsbyneighbouringareaorbylargerarea(forexample, London,CountiesandDistricts)andnationally

• includeadditionalqualitymeasuresasoutlinedinthemainreportandusedinthecase studies,forexample,matchratesandFP69flags

Action9:ONSwillmakeimprovementstothedatavisualisationtoolforthenextrelease,notingthecommentsreceived.

Summary of feedback on the 2015 administrative data research outputs

7 Office for National Statistics

2.6 GeneralcommentsonhowtoimprovetheResearchOutputs

ConcernswereexpressedthatthecurrentmethodologyfortheResearchOutputsisstockbasedandreferstoasnapshotofthepopulationatapointintime.ThisisdifferentfromtheflowsbasedmethodologyfortheMYEswhichincorporatesinternalmigrationfromGPpatientregistertotrackmovementbetweenyears.

Action10:ONSplanstoinvestigatehowtheSPDmethodologycanbeusedtoidentifypopulationflows,includingmigration.

RespondentswerealsoverykeentoseetheSPDestimatesonthesizeofthepopulationextendedtosmallarealevel.Action7setsoutONS’splansforthis.

RespondentsflaggedaconcernedthatthemethodologyfocusesonproducingestimatesforEnglandandWales,anddoesnotyetextendtoproducingUKestimates.ScotlandandNorthernIrelandareawareofthecurrentresearchbeingconductedbyONSandworkwillbedonetounderstandhowcoherentUKestimatescanbeproduced.

Action11:ONSwillcontinuetoworkcloselywithNRS(NationalRecordsofScotland)andNISRA(NorthernIrelandStatisticsandResearchAgency)toproduceUKstatisticsfromthecensus.

Currentlypopulationestimatesareusedtoprovidepopulationprojections,populationestimatesoftheveryold,andpopulationestimatesforsmallgeographicalareas.Theyarealsousedforweightingsurveyestimatesandothersocialsurveystoensurethattheyarerepresentativeofthetotalpopulation.Concernshavebeenraisedaboutwhetherthenewmethodologywillbesufficientlyrobusttoprovidethisrole.

Action12:AnassessmentofONS’sabilitytomovetowardsanAdministrativeDataCensuswillbepublishedeachyear.Thefirstoftheseassessmentsispublishedalongsidethispaper.Thisassessmentwillbebasedonarangeofcriteria,includingwhetherthemethodologyhastheabilitytomeettheinformationneedsofkeyusers.

Othercommentsrelatingspecificallytocensusmethodologyarenotcommentedonhere.ONSwillsetoutdetailedmethodologyforthe2021Censusinduecourse.

Summary of feedback on the 2015 administrative data research outputs

8 Office for National Statistics

3. Plansforautumn2016ResearchOutputsRelease

ONSplanstopublishthenextsetofResearchOutputsinautumn2016.Theaimistoexpandtheaccuracyand/ordetailand/orbreadth(rangeoftopics)oftheoutputseachyear.The2016releasewillcontinuewiththeSPDtimeseriesreleasedlastyear.ONSisalsoaimingtoproduce:

• populationestimatesdowntoLowerlayerSuperOutputArea(LSOA)andbysingle yearofage

• improvementstothemethodsusedtoproduceadministrativedatapopulationestimates

• outputsonthenumberofhouseholds

• researchonincomefromcombinedPAYEandbenefitsdata

Furtherdetailswillbeannouncedinduecourse.Userswillcontinuetohavetheopportunitytoprovidefeedbackontheseoutputs.

Summary of feedback on the 2015 administrative data research outputs

9 Office for National Statistics

AnnexA TablesummarisingtheactionsthatONSplanstotake inresponsetothefeedback

Actionnumber Action

1

2

3

4

6

5

ONSiscurrentlyreviewingtheapproachfordisseminatingfutureResearchOutputs.Optionsunderconsiderationare:

• reducingthesizeofthereport• enablinguserstonavigatethroughthereportmoreeasily• makinguseofblogstocommunicatekeyqualityissuesorpresentcasestudy- typeanalyses,and• makingmoreuseofvisualapplications

ONSiscurrentlyexploringhowactivitydatacanbeusedtoimproveadministrativedatapopulationestimates(usingdatafrominteractionswithaDWP/HMRCadministrativesystemandfrompatientsinteractingwithNHSsystemsthroughnewregistrationsortoupdatetheiraddressandotherdetails).

ONSisconsideringatestofthePCSin2018.Datacollectedthroughthisexercise,datacollectedinthe2017Censustestand,wheresuitableandpossible,existingsurveydatawillbeusedtodemonstratehowadministrativeandsurveydatacanbecombinedtoproducehighqualityestimatesaboutthesize(andcharacteristics)ofthepopulation.

ONSplanstoreviewthecurrentqualitystandardsusedtoassessthequalityoftheSPDs.Thisworkincludesexploringhowanindependentqualitymeasure(suchastheproductionofconfidenceintervalsaroundtheSPD)mightbeproduced.Inthelongerterm,oneuseforthePCSwillbetoprovideconfidenceintervalsaroundtheestimates.

ONSaimstoshowdevelopmentstothemethodsforproducingestimatesaboutthesizeofthepopulation(includingtheuseofSchoolCensusandDWP/HMRCactivitydata)inthenextreleaseofResearchOutputsinautumn2016.

ONSwillcontinuetoworkcloselywithdatasupplierstounderstandandimprovestatisticalqualityissuesidentifiedinthedata.ADataSuppliersGrouphasbeensetupwhichincludesmembersfromkeyGovernmentDepartmentsacrossWhitehall.Thisgroupisaforumfordiscussingissuesrelatingtodatasharing,sharingresearchandbuildingstrongerrelationships.Ifnewdatasharinglegislationgoesahead,thisgroupwillbevitaltothesuccessofunlockingdataacrossGovernmentasworkwillneedtobedonetoputinplacepracticalarrangementsfor:

• datasupply

• protectingthatsupplyinthefuture,and

• improvequality

Summary of feedback on the 2015 administrative data research outputs

10 Office for National Statistics

Actionnumber Action

8

9

10

11

12

ONSisworkingcloselywiththreegroupsoflocalauthoritiestobetterunderstanddiscrepanciesbetweentheSPDsandofficialpopulationestimates.Wherepossible,acollaborativeapproachwillbetakenonfuturecasestudies.

ONSwillmakeimprovementstothedatavisualisationtoolforthenextrelease,notingthecommentsreceived.

ONSplanstoinvestigatehowtheSPDmethodologycanbeusedtoidentifypopulationflows,includingmigration.

ONSwillcontinuetoworkcloselywithNRS(NationalRecordsofScotland)andNISRA(NorthernIrelandStatisticsandResearchAgency)toproduceUKstatisticsfromthecensus.

AnassessmentofONS’sabilitytomovetowardsanAdministrativeDataCensuswillbepublishedeachyear.Thefirstoftheseassessmentsispublishedalongsidethispaper.Thisassessmentwillbebasedonarangeofcriteria,includingwhetherthemethodologyhastheabilitytomeettheinformationneedsofkeyusers.

7 ONSwillpublishadministrativedatapopulationestimatesdowntoLowerlayerSuperOutputArea(LSOA)andbysingleyearofage(atLAlevel)inautumn2016.Analysingdataattheselevelswillhelpidentifyagegroupsandareasthathavethebiggestdiscrepancyfromofficialpopulationestimates,andwillhelptofocusfurtherrefinementofthemethodologyandthedatasourcesused.

RPT 02 CAD