Formats and Tools for Data Trasmission
Vincenzo PATRUNO
Pristina, Kosovo7, 8 Dec 2009
EDAMIS Web Application (EWA)
EDAMIS Web Portal (EWP)
EDAMIS Web Form (EWF)
Formats for data Transmission
Gesmes
Data Repository (Warehousing) Architecture
NSI
EurostatPull Requestor
eDAMIS
Data Input
SDMX Registry
Intermediatestorage
Verification /ConversionTo SDMX
Receiveddata in
SDMX-MLLoader
register
Warehousestorage
Eurobase
query
Dissemination
XSL forSDMX-ML
PULL
PUSH
Francesco Rizzo - Eurostat
The Role of TCO
PUSH & PULL
We have normally two different approach to exchange data: PUSH and PULL
PUSH & PULL
PUSH mode means that the data provider takes action to send the data to the party collecting the data.
PULL mode implies that the data provider makes the data available via the Internet. The data consumer then fetches the data on his own initiative.
Exchange and Sharing of statistical informationExchange and Sharing of statistical information
Emphasis on macro-data (aggregated statistics)Emphasis on macro-data (aggregated statistics)
Promotes a “data sharing” model Promotes a “data sharing” model
Structural metadata
Reference metadata
Statistical data Statistical metadata
low-cost
high-quality of transmitted data
interoperability between (otherwise) incompatible systems
Describe data structure
Describe the contents and the quality of the statistical data
SDMX deals with…
Marco Pellegrino - Eurostat
SDMX
SDMX promotes a “data sharing” model to facilitate low-cost, high-quality statistical data and metadata exchange.
Data Providers publishes the availability of data/metadata to Data Consumers and the latter are responsible for fetching the data/metadata at will.
.
Query SDMX
DataSDMX-
ML
RSS
WSNSI
Data Sharing Model
Data Hub Architecture
NSI
NSI
NSI
NSI
SDMX RegistryRSS / data registration
Dissemination
XSL forSDMX-ML
Data Portal
Query
Data queryResponse
Retrievedataset
cache
Francesco Rizzo - Eurostat
http://91.194.86.67:8080/CensusHub2
The Census Hub architecture
based on the concept of data sharing, where a group of partners agree on providing access to their data according to standard processes formats and technologies
The hub is based on agreed hypercubes, but here the hypercubes are not sent to the central system, but data are fetched directly from the data producer databases when a user requests them
SDMX formats and architecture are used
Francesco Rizzo - Eurostat
http://bms.istat.it/sodidownload/download.aspxhttp://bms.istat.it/sodidownload/download.aspx
ISTAT SDMX Framework
ISTAT SDMX Framework It is based on the SDMX Information Model It covers all the dissemination business process It can be used entirely
or Only some building bocks can be used, in order to integrate them in an already existing information system
Available APIs SDMX Query Parser
It allows to set up an SDMX architecture in a short time It can be used for didactic purpose It will be available as Open Source
The NSI perspective on SDMX: benefits
Reduce reporting burden to National, European and International institutions
Can improve harmonisation, standardisation and integration processes inside a NSI
Be part of an international “community” where NSIs can:– share experiences and best practices– share freely software and tools
Collective intelligence is a shared or group intelligence that emerges from the collaboration and competition of many individuals
The Collective Intelligence