Modernising Official Statistics

11 February 2015

Central Statistics Office Ireland

Central Statistics Office, Ireland 2

Introduction

 How the CSO is working to Modernise Official Statistics

 How ICHEC is helping with that

 The international impact of what we are doing

Central Statistics Office, Ireland 3

What do we do?

 The Central Statistics Office produces Official Statistics for Ireland

 Economic Statistics, Social Statistics and others

 Used by policy makers

Central Statistics Office, Ireland 4

How do we do it now?

 Surveys

 Administrative Data

 Data in databases

 Data is small enough

Central Statistics Office, Ireland 5

Looking to the Future

 Established in 1949 so always modernising

 Non-Traditional Data Sources

 Mobile phone data, satellite imagery, scanner data

 Big Data

Central Statistics Office, Ireland 6

Why Big Data?

 More and more requirements from our customers and International Organisations

 Also want to reduce respondent burden

 Complement existing outputs in new ways and produce brand new outputs

Central Statistics Office, Ireland 7

Big Data Challenges

 Data Quality

 Confidentiality

 Methodology

 Technology

Central Statistics Office, Ireland 8

Technology

 Unused to working with Big Data levels of data

  ICHEC provided R training as an alternative to our current tools

 Training with a focus on how we would use R

Central Statistics Office, Ireland 9

We are not alone

 Lots of NSI’s and International Organisations looking at ways to use Big Data for Official Statistics

 UNECE High Level Group for the Modernisation of Statistical Production and Services

 Looking at various modernisation measures including Big Data

Central Statistics Office, Ireland 10

Work together

 Big Data allows a different approach

 No legacy systems

 Starting with a blank canvas

 Start with collaboration and standardisation

 Looking for a shared environment to do this together

Central Statistics Office, Ireland 11

The Sandbox is born

 The CSO and ICHEC collaborated to provide a shared system for the UNECE group called the Sandbox

  It is a shared remotely accessible High Performance Computing environment

  ICHEC host the system and provided the High Performance Computing expertise necessary to make this happen

 System launched in April 2014 in Dublin

Central Statistics Office, Ireland 12

Aims of the Sandbox

 Test feasibility of remote access and processing

 Test Statistical models and methods

 Test Big Data tools

 Learning by Doing

 And do it together

Central Statistics Office, Ireland 13

Projects ongoing

 Sentiment analysis using twitter data

 Looking at mobile phone records for tourism and other statistics

 Satellite imagery and aerial photography

 Data from road traffic sensors

Central Statistics Office, Ireland 14

More projects

 Price data sourced using web scraping

  Job vacancy data sourced using web scraping

 Scanner data and other price information sourced directly from large vendors

 At present approximately 40 statisticians and data scientists from 25 different organisations are working with the Sandbox

Central Statistics Office, Ireland 15

Who are the people involved?

Central Statistics Office, Ireland 16

Advantages of the Sandbox

  International Collaboration on methods and techniques

 Learning by doing

 Gives users the chance to use Big Data techniques and hardware without having the equipment or technical expertise in-house

 Allows for experimentation

Central Statistics Office, Ireland 17

What it means for the future

 Discussions ongoing about the infrastructure required for Big Data for National Statistical Institutes

 Providing a template for how Big Data work could be done in a centralised way

 Could possibly become a production system used internationally

Central Statistics Office, Ireland 18

Has it worked?

 Sandbox has been a huge success and is still ongoing

 New idea in Official Statistics that is attracting interest across the world

 Raises Ireland’s profile not just in the area of Big Data for Official Statistics but also in the area of High Performance Computing

Central Statistics Office, Ireland 19

Conclusions

 We continue moving towards modernising our Statistical production

  ICHEC is helping us with relevant training to keep our staff skills up to date

  ICHEC is also helping us with the hardware and technical expertise that was crucial to the success of the Sandbox

 Thank you and we look forward to continuing to work together

Modernising Official Statistics

11 February 2015

Thank You

Questions ?