Data Transmission Courier - CIRCABC - Welcome · PDF fileFor a summary of the monitoring of dataset occurrences with a tabular representation: ... EUROSTAT - B3 Data Transmission Courier

UNIT B3 Help Desk: [email protected]

DDAATTAA TTRRAANNSSMMIISSSSIIOONN

CCOOUURRIIEERR

Issue 67

February 2008

1

Malta: change of Local Coordinator

Mr. Stefan FARRUGIA National Statistics Office email: [email protected]

has replaced Mrs. Maria KIOMALL as Local Coordinator for Malta.

eDAMIS: eWA and eWP progress

eDAMIS compulsory as of 1 July 2008:

The Eurostat Directors' Meeting decided on Tuesday 12 February that eDAMIS will become compulsory for the transmission of all regular datasets as of 1 July 2008. This statement was supported two days later by the Statistical Program Committee (SPC) which regroups the Directors General of the NSIs.

eWP 2.6 in production on Wednesday 27th February:

eDAMIS Web Portal 2.6 is a major new version. Two days have therefore been planned for the installation and correct function tests. Installation will start on Monday 25 February and eWP should be available again on Wednesday 27. A message has been put on the home page of eWP to inform about these 2 days when the application will be off-line. To avoid misunderstandings, this message also states that eDAMIS Web Applications (mostly used in NSIs) will be fully operational during these 2 days.

Performance issues are the priority now:

Apart from an improvement in the reports and the functionality of Web Forms which are now compliant with SDMX-ML 2.0, the main added values of eWP 2.6 are the user-friendliness and robustness of the application. Nevertheless, performance issues remain. They are being treated now and improvements should be noticed throughout the coming year.

https://webgate.cec.eu.int/edamis is dead:

Until the beginning of 2007, access to eDAMIS was via the internet address https://webgate.cec.eu.int/edamis. One year ago, the new internet address https://webgate.ec.europa.eu/edamis became operational. The old internet address is now no longer valid. If not already changed, please replace in your favourites https://webgate.cec.eu.int/edamis by https://webgate.ec.europa.eu/edamis.

eDAMIS Web Forms for Energy

Introduction of eDAMIS Web Forms in Member States in 2008 for the Energy domain

A visit to Statistics Slovenia is planned for the beginning of March. The eDAMIS Web Forms component will be introduced to statisticians working for different domains, among them energy and agriculture.

No other missions are currently planned for the energy domain.

https://webgate.cec.eu.int/edamis


https://webgate.ec.europa.eu/edamis



EUROSTAT - B3 Data Transmission Courier February 2008

2

20th GLC meeting 10-11 March 2008

Tour de table:

During the next meeting of the Group of Local Coordinators (GLC20 / 10-11 March 2008) all Local Coordinators will be asked to answer two specific questions relating to data transmission to Eurostat:

− Are there specific statistical domains needing more work/attention from the Eurostat side?

− Are there any plans in your country to change the data management system(s) used to prepare data for sending to Eurostat (for example, introduction of a data warehouse)?

Interventions will be limited to 3 minutes.

eDAMIS Inventory changes

Domain EMPLOY

The dataset EMPLOY_Q_Q has been deleted.

Domain PRAG

The dataset PRAG_INDICE1_A has been deleted.

The dataset PRAG_INDICE2_A has been renamed PRAG_INDICES_A.

Domain CROPROD

The datasets CROPROD_EARLY_M and CROPROD_FVEARLY_M has been changed to CROPROD_EARLY_A and CROPROD_FVEARLY_A.

Domain COSA

The domain COSA has been deleted (it contained only two datasets : COSA_ALI1_A and COSA_ALI2_A).

Domain VITIS

The dataset VITIS_FORECST_5 has been changed to. VITIS_FORECST_A.

Domain ANI

The datasets ANI_PRODMT1_M, ANI_PRODMT2_M and ANI_PRODMT3_M have been deleted.

STS domains

The datasets STSIND_PRIC_Q, STSSERV_PRIC_M and STSRTD_TURN_Q have been deleted.


3

How a Transmission Coordinator could manage other National Organisations

At GLC 18 (document LC/18/005), it was stated that a Local Coordinator may wish to manage other Organisations than his/her NSI in his/her country in order to:

• Allocate rights to users of other Organisations, • Manage the flag “Allow Eurostat to grant rights for users of the

Organisation”, • Transmit data on behalf of users of the other Organisation(s).

eDAMIS permits this, but Eurostat will ask for evidence of the agreement of the other organisation(s) before implementing it.

Eurostat is finalising a procedure based on a letter or an email that would come from the (non NSI) National Organisation and request the nomination of a Transmission Coordinator (or several). The National Organisation would receive a note explaining the role of a Transmission Coordinator as well as some major recommendations of Eurostat policy (use of individual user-ids, eWA installed mostly in NSI, etc.).

It is again reminded that in order to have information concerning all Organisations of the country, it is not necessary to be Transmission Coordinators of all the Organisations. eDAMIS displays reports on traffic as well as the dataset inventory and users for all Organisations (see DTC 66 – January 2008). If wished, a Transmission Coordinator may also ask to be notified by email of all changes for all Organisations in the country related to the dataset inventory and/or the user management (through user preferences).


4

eDAMIS: Traffic Monitoring:

Reports "Timetable by occurrences"

The Management Information System (MIS) in the eDAMIS Web Portal groups all reports about traffic monitoring, the inventory (themes, modules, domains, datasets) and user rights.

In total four families of reports exist to support the monitoring of traffic:

1. Data file traffic For monitoring the data files received by Eurostat. Indicates whether they arrived before of after the indicative deadline.

2. Timetable between dates (detailed and summary) For monitoring the dataset occurrences in detail (with one row per dataset occurrence). The first and last reception dates are indicated as well as the number of data files (versions) sent for each dataset occurrence.

3. Timetable by occurrences (detailed and summary) For a summary of the monitoring of dataset occurrences with a tabular representation: the datasets are in columns and the countries are in rows.

4. Volume and number of files transmitted For monitoring the volume and number of data files received by Eurostat and aggregated by Theme, Domain, Dataset, Eurostat Unit, and Country.

This article is the third in a series of articles on this topic and the focus will be now on the timetable by occurrences (detailed and summary). The fourth and final report will be in the next Data Transmission Courier.

The two reports "timetable by occurrences" (detailed and summary) allow taking into account in the report all dataset occurrences which reference year/period is between "year1" and "year2". The user simply asks for a "from" and "to" year. There is no need to specify whether the expected or received dataset occurrences should be displayed and no need to worry about the transmission period ("from" and "to" dates of the other reports). Only the year of the reference period (period for the data) cares.

Timetable by occurrences (detailed)

The timetable by occurrences (detailed) shows, for a given period of time and filtered on specific objects (countries, organisation, Eurostat unit, themes, domains and datasets), the datasets which had at least an occurrence between the years selected. A dataset occurrence represents a dataset linked to a country/organisation for a specific year and period. The first and last reception dates are indicated as well as the number of data files (versions) which passed through the Eurostat Single Entry Point (via eWA or eWP). Incoming datasets


5

(received by Eurostat) and outgoing datasets (sent by Eurostat) are visible in the report. Each row corresponds therefore to a dataset occurrence.

The selection in the upper part of the report is made from the country group to the dataset. By default, the country group, country and organisation are pre-selected according the user’s organisation. The scope is not strict, meaning that the pre-selection can be changed. It is advised to filter the report enough in order to avoid response time problems. For example, it could be difficult to consult the dataset occurrence traffic for a complete year and for all countries and domains. It would take a very long time before getting the rows displayed and, in worst cases, timeout problems could occur. The narrower the range of dates is and the more the selection is targeted, the more the report is displayed quickly.

The Period from..to selection allows to filter dataset occurrences which reference year/period is between the years provided.


6

The selection criteria are dependant on other selection criteria by following these rules:

• The Country list box depends on the selected country group. • The theme list box depends on the Eurostat Unit • The domain list box depends on the selected theme. • The dataset list box depends on the selected domain.

Each time a selection is done, the number of items present in each selection list as well as the items selected are displayed in bold on the right top corner of the box.

When the selection criteria are specified, the user clicks on the “View” button to see in the bottom part of the report the list of dataset occurrences received by the Single Entry Point.

As the number of columns in the report is fairly high, the column headers need to be shortened. By keeping the mouse pointer over the header, a tool tip appears and displays the header complete definition.

Data occurrences are uniquely identified by following fields: “Dataset”, “From”, “Year”, and “Per.” (period).

Column From shows the Country responsible of the sending of the dataset occurrence.

Columns Year and Per. both indicate which reference period the dataset occurrence corresponds to.

Column Per. for trans. indicates the periodicity for transmission of the dataset (e.g. Monthly, Quarterly…).

Field Status can be:

• “Received” when at least one version of the dataset occurrence has been received at the date of the report. It will be displayed in red when the first version was received after the deadline (and it was expected to send without derogation for the country/period).

• “Not Received” when no version of file has been received at the date of production of the report and the indicative deadline has been passed.

• “Expected” when no version has been received at the date of production of the report, but the deadline has not been passed.

Column Indic. deadline displays the indicative deadline of dataset occurrence transmissions to the Single Entry Point. The indicative deadline is automatically calculated by eDAMIS taking as basis the timeliness information defined by Eurostat in the dataset inventory. The columns in the report shows:

• Nothing if Eurostat did not provide enough information in the timeliness part of a dataset,

• Else, the date of the deadline foreseen for the dataset occurrence.

When an indicative deadline is calculated for a dataset, a colour scheme is applied in the report:

• Delay (days) are displayed in red when the first version of the dataset occurrence was received late with regards to the indicative deadline.

• Delay (days) are displayed in green when the first version of the dataset occurrence was received before or on the day of the indicative deadline.

Field Delay (days) is calculated by eDAMIS and indicates the number of days the


7

first version has been received from the indicative deadline.

Field Max. Delay (maximum delay) indicates the number of months and/or days between the end of reference period and the indicative deadline. This parameter is defined in the dataset inventory by Eurostat and corresponds to the length of the period during which Eurostat expects to receive the dataset occurrences.

Field Number of files indicates the number of versions received for the dataset occurrence.

Column Total volume indicates the sum of the sizes of the received files in bytes.

Fields Min Vol., Max Vol. and Av. Vol. indicate respectively the lowest, average and greatest size of the received files in bytes.

Timetable by occurrences summary

The report provides with the same information as the detailed one, but presents the results in a pivot table in which each row is a country and each column is a dataset occurrence. Therefore, cells of the table display the number of dataset occurrences which have the status: not received, received and expected.


8

Statistics – coverage indicator

Following -1- the publication in the last DTC of statistics covering the whole year 2007 and presenting the coverage indicator and -2- the presentation of the coverage indicator to the Statistical Program Committee, several requests were received from countries in order to clarify the method of calculation of the coverage indicator which could appear to be under-estimated.

The detailed method to calculate the coverage indicator has been presented in Annex-1 of document LC/16/011 (GLC 16 March 2006). This method is quick and easy (to calculate and to understand) but only gives a rough estimation. It considers that, on average, 1340 new dataset occurrences are expected from one EU country in one year. So the coverage indicator for one country in one year corresponds to the number of new dataset occurrences received divided by 1340. Over 3 months, is it divided by 335 (1340/4). Please note that what we consider as a new dataset occurrence is the first version of dataset occurrences independent of the action code used in eWA (for instance if in eWA dataset AIR_A1_A for the U.K., year 2007, was sent twice with action code "New", it is counted only once). The estimation of 1340 expected dataset occurrences was made after deep studies based on the eDAMIS dataset inventory, so it should not be too far from the reality. You could also by another way find similar results by looking to the file "1- New dataset occurrences received in 2007 by country - subset datasets.xls" which can be found on CIRCA at: http://circa.europa.eu/Members/irc/dsis/loccord/library?l=/statistics/coverage_indicator This table takes into account the new occurrences received for each dataset that have seen at least one transmission from one country in 2007 (only 505 datasets from among the 805 listed in eDAMIS). It also considers that all EU countries have to send all datasets. Even if not precise, the current method is nevertheless very useful to compare between countries and to see the evolution over time. It suffers from two main drawbacks, the first one has a tendency to lower the coverage indicator and the second one, to increase it, so both could potentially compensate: -1- it does not take into account the "expected to send flag" which was up till now not reliable in eDAMIS: it considers that for each datasets, all EU countries are expected to send. -2- it does not count datasets which have not been transmitted at all in eDAMIS. For example, we know this is the case with the Education domain and several datasets related to agriculture and energy (on that point the situation should improve soon). At the end of this DTC, the results of the calculation of the coverage indicator based on other methods are presented. Globally, for the total EU27 countries It goes from 38% to 62%.

The last column (in yellow) gives the results of DTC66. The 3 first columns (in green and blue) calculate the coverage indicator with different methods on a subset of 376 incoming datasets which have the following attributes: -1- periodic, -2- active, -3- created before 1/1/2007, -4- at least one transmission of a new occurrence from an EU27 country in 2007.

http://circa.europa.eu/Members/irc/dsis/loccord/library?l=/statistics/coverage_indicator



9

The first method (raw in green) gives the calculation of the coverage indicator for each country only for the datasets which have the "expected to send" flag set to "yes". That means that for one country: * only the new occurrences transmitted for datasets with the expected to send flag sent to "yes" are counted, * the total of the new occurrences received is divided by the expected occurrences of the datasets which have this flag set to yes as well. This method could be more used in the future. It should improve when the "expected to send" flag will be more reliable in eDAMIS. For the moment it is probable that this way to calculate the coverage indicator is still a bit optimistic because many low coverage monthly datasets such as for agriculture or energy are either not selected in the 376 or over evaluated because their flag "expected to send" is set to "no" for most countries whereas it should be set to "yes". Please be aware that the monthly datasets represent 11% of the total number of datasets, but count for 50% in the calculation of the coverage indicator and that quarterly datasets represent 18% of all datasets but count for 25% in the calculation of the coverage indicator. So to improve quickly your coverage indicator, you could address first these datasets for which many occurrences are expected each year.

Last element: the coverage indicator is calculated for the country as a whole, not for the NSI only, so if you are in a very decentralised country with many datasets managed by so called "small data providers", then it is more difficult for you to increase your coverage indicator. In average, we estimate that the NSI are in charge of 50% of the datasets transmitted to Eurostat.

If you want more information, you can consult the file "2- Detailed study coverage indicator - expected received NSI for 2007.zip" on CIRCA at:


This file presents a detailed study on what is sent by which country and gives the details of the different ways to calculate the coverage indicator. We can give you more information on it if needed.

The file could in the future be updated on a regular basis and later on integrated as a report available in eDAMIS.

eDAMIS https://webgate.ec.europa.eu/edamis

eDAMIS Support [email protected]

eDAMIS Help Centre https://webgate.ec.europa.eu/edamis + "eDAMIS Help Centre"




mailto:[email protected]



10

Statistics for last three months

New dataset occurrences by client type group

From 01/10/2007 To 31/12/2007

E-MAIL to SEP 227 5.9%

STATEL 58 1.5%

VB 6 0.2%

eWA 3312 86.0%

eWF 109 2.8%

eWP 139 3.6%

Total 3851 100.0%

Number of new dataset occurrences by country October-December 2007 versus October - December 2006

Country 2007 2006 Changes Coverage BE 112 87 29% 34% BG 119 61 95% 36% CZ 141 138 2% 42% DK 101 82 23% 30% DE 147 136 8% 44% EE 137 124 10% 41% IE 97 84 15% 29% EL 97 108 -10% 29% ES 137 97 41% 41% FR 126 128 -2% 38% IT 121 109 11% 36% CY 163 88 85% 49% LV 116 101 15% 35% LT 132 105 26% 40% LU 105 105 0% 31% HU 118 79 49% 35% MT 90 117 -23% 27% NL 183 205 -11% 55% AT 112 114 -2% 34% PL 111 100 11% 33% PT 143 110 30% 43% RO 155 120 29% 46% SI 260 176 48% 78% SK 109 81 35% 33% FI 133 118 13% 40% SE 129 110 17% 39% UK 142 113 26% 43%

HR 45 53 -15% MK 15 7 114% TR 33 15 120%

IS 15 13 15% LI 10 4 150% NO 91 71 28% CH 72 92 -22%

other 34 9 278%

Totals 3851 3260 18%

New dataset occurrences by file type group From 01/10/2007 To 31/12/2007

Compressed 340 8.8%

Encrypted 123 3.2%

GESMES 1770 46.0%

Non-Proprietary 795 20.6%

Proprietary 821 21.3%

XML 2 0.1%

Total 3851 100.0%

Countries that passed the 50% threshold as well as changes above 50% are highlighted.

EUROSTAT - B3 Data Transmission Courier

11

eDAMIS First Transmissions of "Dataset Occurences" November 2007 - January 2008

0 50 100 150 200 250 300

other

CH

NO

LI

IS

TR

MK

HR

UK

SE

FI

SK

SI

RO

PT

PL

AT

NL

MT

HU

LU

LT

LV

CY

IT

FR

ES

EL

IE

EE

DE

DK

CZ

BG

BE

Cou

ntry

Number

November 2006 - January 2007

November 2007 - January 2008

Average EU27: 131

Total November 2006 - January 2007 3260 Total November 2007 - January 2008 3851 Increase on 3 months +18% Coverage SEP for EU27 on 3 months 39%

The threshold of 167 (50% coverage of SEP) is to be considered as an estimated average for EU27 countries. It may vary according to the obligations of each country. It is lower for EFTA countries who have less obligations.

Number 50% coverage SEP: 167


12

eDAMIS - First transmission of "Dataset Occurences" by Client November 2007 - January 2008

eWF2.8%

eWP3.6% STATEL

1.5%

VB0.2%

eWA86.0%

E-MAIL to SEP5.9%

eDAMIS First transmission of "Dataset Occurrences" by Format November 2007 - January 2008

GESMES46.0%

Non-Proprietary20.6%

Proprietary21.3% Encrypted

3.2%

XML0.1% Compressed

8.8%


13

Calculation of the coverage indicator for 2007 according to different methods

Coverage

calculation Filtered flag expected (1)

Filtered flag expected (2)

Filtered 27 expected DTC 66

BE 52% 58% 42% 34%

BG 52% 51% 37% 29%

CZ 69% 74% 53% 46%

DK 44% 48% 35% 29%

DE 65% 74% 54% 44%

EE 50% 56% 41% 33%

IE 44% 46% 34% 29%

EL 51% 55% 40% 31%

ES 47% 60% 44% 39%

FR 62% 68% 49% 39%

IT 59% 64% 46% 36%

CY 65% 67% 49% 38%

LV 53% 54% 40% 34%

LT 60% 67% 49% 42%

LU 50% 56% 41% 33%

HU 49% 56% 41% 33%

MT 34% 39% 28% 27%

NL 77% 88% 64% 57%

AT 55% 64% 47% 38%

PL 51% 60% 44% 37%

PT 51% 66% 48% 44%

RO 54% 70% 50% 43%

SI 72% 92% 67% 55%

SK 55% 62% 45% 38%

FI 58% 67% 49% 40%

SE 51% 59% 43% 39%

UK 42% 49% 36% 30%

EU27 54.6% 61.9% 44.9% 37.8%

Documents

Data Transmission Courier - CIRCABC - Welcome · PDF fileFor a summary of the monitoring of dataset occurrences with a tabular representation: ... EUROSTAT - B3 Data Transmission Courier