Web based tools for Raw Data collection Bertrand Loison, Chief Information Officer (CIO) Eurostat /...

Web based tools for Raw Data collection

Bertrand Loison, Chief Information Officer (CIO)

Eurostat / IT Directors Group / 25.10.2005 / Luxembourg

Swiss Federal Statistical Office

Eurostat / IT Directors Group

TABLE OF CONTENT

Raw Data Collection on the Web with eSurvey

– Purpose and Scope– Architecture– Development– Business View– Benefits– Conclusion

Eurostat / IT Directors Group

Purpose and ScopeIT STRATEGY

Historical: individual application

Vision: horizontal application

Sta

tistic

4

Sta

tistic

3

Sta

tistic

2

Sta

tistic

1

Sta

tistic

5

As is: SW-Modules

Statistical Web-Portal

Stat. Datawarehouse

Transformation

Analysis

Paper sur. Elec. Sur. ...

4321 5

Statistical Web-Portal

Stat. Datawarehouse

Analysis

Pro

duct

ion

proc

ess

Statistical products

2000 2005 2010

E & E Papier-kanal

E & E Datei-kanal

E & E Web-kanal

E & E PDA

E & E Telefon

E & E per-

sönlich

Eurostat / IT Directors Group

TABLE OF CONTENT

Raw Data Collection on the Web with eSurvey

– Purpose and Scope– Architecture– Development– Business View– Benefits– Conclusion

Eurostat / IT Directors Group

ArchitectureAPPLICATION LAYER

Distribution

Admin/Auth. DB

Admin Server, Transfer Server

eSurvey Metadata,Respondent Data

eSurvey Response

DMZ

WebDB

Webserver

eSurvey Metadata,Respondent Data

eSurvey Response

Respondents

1. import

6. export

2. publish

5. replicate

4. fill out

3. enable logonIntranet

Internet

SFSO

Prod.DB

Production

eSurvey

MetaDB

Statistician Statistician

Eurostat / IT Directors Group

ArchitectureTECHNOLOGICAL LAYER

Eurostat / IT Directors Group

TABLE OF CONTENT

Raw Data Collection on the Web with eSurvey

– Purpose and Scope– Architecture– Development– Business View– Benefits– Conclusion

Eurostat / IT Directors Group

Development

– Putting together experienced business and IT people with experience in the domain of Internet surveys together in one small team.

– Flexible and extensible meta modelling with UML.– Prototypical approach to master complexity from the

very beginning.– Initial focus on specific business surveys with specific

statistical projects as “clients” helped to establish concrete real world requirements.

– Consequent incremental and iterative development in which the solution grew continually through small steps and approximations to a first productive version (RUP).

KEY ELEMENTS

Eurostat / IT Directors Group

TABLE OF CONTENT

Raw Data Collection on the Web with eSurvey

– Purpose and Scope– Architecture– Development– Business View– Benefits– Conclusion

Eurostat / IT Directors Group

Business ViewSURVEY PROFILES Parameter Selection Explanation Data provider Business Surveys Businesses are well

equipped with internet technology

Media Paper based surveys Economy measures (compared to scan/reco expenditures)

Periodicity Periodical surveys (rotation, panels)

Economy measures (reusability)

Questionnaire Type

Standard or personalized

Moderate expenses

Number of Respondents

More than 1'000 respondents

Economy measures (cost/benefit ratio)

Process (data requirements)

In house SFSO Both business know how and IT support available.

Security Requirements

Low to middle The big majority of online surveys do not require high level IT security (e.g. strong authentication).

Eurostat / IT Directors Group

Business ViewEXPERIENCES #1

Economy / Business Data provider Business Surveys Businesses are well

equipped with internet technology

Media Paper based surveys Economy measures (compared to scan/reco expenditures)

Periodicity Periodical surveys (rotation,panels)

Economy measures (reusability)

Questionnaire Type

Standard or personalized

Moderate expenses

Process (data requirements)

In house SFSO Both business know how and IT support available.

Security Requirements

Low to middle The big majority of online surveys do not require high level IT security (e.g. strong authentication).

Eurostat / IT Directors Group

Business View

Population / Household

eUGH Survey preparing the next census (2010) collecting metadata from municipalities.

Yearly, 3000 respondents

eSILC Non response survey (Swiss income and living statistic)

once

ePSM Non response survey (Swiss household panel)

once

Tourism

eHESTA Survey of hotel room occupation Monthly

EXPERIENCES #2

Social Security eASYL Survey of social aid for asylum-seekers Twice a year.

About 3000 respondents.

Eurostat / IT Directors Group

TABLE OF CONTENT

Raw Data Collection on the Web with eSurvey

– Purpose and Scope– Architecture– Development– Business View– Benefits– Conclusion

Eurostat / IT Directors Group

BenefitsINPUT DATA CHANNELS 2001-2003

Eurostat / IT Directors Group

TABLE OF CONTENT

Raw Data Collection on the Web with eSurvey

– Purpose and Scope– Architecture– Development– Business View– Benefits– Conclusion

Eurostat / IT Directors Group

Conclusion

– The creation of a small core team of IT and business people working together to develop an application users the example of real world web based surveys.

– The definition of conceptual models which can be defined and refined, which are used to develop a modular software tool.

– The use of prototypes which can be used and tested by end users to provide input for refinements and improvements.

– A well defined selection process for “suitable” surveys.

KEY ELEMENTS