Data structure definition for the production testing

SIMSTAT-Pilot ESSnet: WP1 DocMet400-SIMSTAT-ver. 1.5

GRANT AGREEMENT FOR AN ACTION WITH MULTIPLE

BENEFICIARIES “PREPARATION FOR EXCHANGE OF

MICRO-DATA ON INTRA-EU TRADE BETWEEN MEMBER

STATES (SIMSTAT-PILOT ESSNET)”

AGREEMENT NUMBER 06151.2013.002-2013.435:

TASKS 1.1 AND 1.2

Data structure definition for the production

testing environment

“DOC MET 400 SIMSTAT”

(WP1)

Ref. Ares(2014)3287933 - 06/10/2014


Page 2 / 16

Content

1. INTRODUCTION ....................................................................................................... 3

1.1 Purpose of the document ................................................................................... 3

1.2 General comments ............................................................................................. 3

2. STRUCTURE OF THE SIMSTAT DATA SET ........................................................ 4

3. TECHNICAL SPECIFICS OF THE CHARACTER SEPARATED VALUES

SIMSTAT FILE ........................................................................................................ 14


Page 3 / 16

1. INTRODUCTION

1.1 Purpose of the document

The purpose of this document is to provide a detailed specification of the structure of the

dataset to be used for the SIMSTAT micro-data exchange of Intrastat dispatches data

collected in Member States (MSs) for preparation of intra-EU trade in goods statistics

(ITGS).

The specified dataset will be used both for:

- The ESSnet data exchange outside the SIMSTAT system (the Hub) in May 2014;

- The trial SIMSTAT data exchange in 2015.

1.2 General comments

The specified dataset will be used for the exchange between MSs of data files containing

Intrastat collected dispatches data in these MSs and enhanced with additional data elements and

metadata (status and confidentiality flags) to serve the objectives of the SIMSTAT data exchange

process.

The Trial test exchange of data: It is agreed that all data transfers (except for the ESSnet

exchange of data in May 2014) will be effected via the SIMSTAT Hub, situated in, and operated

the Commission (Eurostat) in Luxembourg. The physical interconnection between the

participating MSs and Eurostat will be done on the basis of the Common Communication

Network (CCN) operated by DG TAXUD of EC. All MSs as well as the Hub will be connected

to the CCN via local gateways.

The ESSnet exchange of data: For the physical exchange of data the character-separated values

type of text files (commonly referred to as “Comma Separated Values” or CSV) will be used. In

order to avoid possible errors due to incorrect usage of dot as a decimal sign or separator, the

semi-colon (;) will be applied as a field separator in the text file. Filename should contain

sending MS, receiving MS, timestamp of production of the file (format YYYYMMDD) and

number (2-numeric, that because file splitting will be possible), e.g. Finnish filename to Austria

would be like “FIAT2014042801.csv”). The characters to be used in line breaks are carriage

return + line feed (records delimited by "\r\n).

The structure of the dataset: The structure of the dataset is detailed below in the form of

“sections” (one for each data element). For each section the following information is provided:

- Mandatory or Optional data element – whether a code or value must be entered or not;

Please note, that some of the fields might be specified in different MSs as the one or

the other depending on specific circumstances which would be stipulated in the

description. For each of the sections a symbol on the left will stand for:

■ mandatory ◘ mandatory or optional or

blank depending on specific

circumstances □ optional

- Type of the field – text (string), numeric, etc., and specific requirements if any;


Page 4 / 16

- Attached code list to be applied. Please note, that if a MS would not be in a position to

refer to specific codes, the field would be “optional” and should be left blank;

- Other specifics, if necessary.

Deadline: The files should be transmitted as defined in SIMSTAT ESSnet WP4 Timetable.

Checklist: The data should be validated as defined in SIMSTAT ESSnet WP3 Validation rules.

Confidentiality: The Member States may make part of the trade results confidential for

commercial or military reasons. There are two main types of confidentiality:

– Product confidentiality

– Partner country confidentiality

Confidentiality will be exchanged as separate metadata file that will be defined

later. Confidentiality flag has no relevance for the trials, because the resulting

statistics will not be disseminated. All records are confidential during

SIMSTAT ESSnet and trial tests.

Empty sections If Section 8 Record Action is “D” = Delete then the Sections 5-6 and 9-31, not

relevant to the delete record, could be left blank. For records to be deleted, it is

sufficient to fill-in in addition to section 8 only Sections 1, 2, 3, 4 and 7.

2. STRUCTURE OF THE SIMSTAT MICRO-DATA SET

The following 31 data elements comprise the structure of the SIMSTAT micro-data set:

Section 1 Unique Record Identifier (URI)

■ URI is a Mandatory element.

The URI is a variable string of 30-alphanumeric characters which for

each record of files from a MS would be created by that MS and

inserted into the file before it was sent to the Hub. So far it has been

proposed as a minimum requirement that the first two digits would be

the country code (from GEONOMENCLATURE) and 3rd – 4th digits

would be the year (e.g “14” for year 2014), so e.g. Finnish URI would

be like “FI14xxxxxxxxxxxxxxxxzzzzzzzzzz” (30-alphanumeric

character string). The URI should be preserved through all the data

exchange steps and finally delivered to the recipient.

To avoid problems with cross-country processing, storage and

presentation of alphabetical characters in the URIs, only the small and

capital letters of the Latin alphabet will be used.

- Full uniqueness of a record must be guaranteed across all MSs. When

the structure of the URI string is finalized, the ways of achieving and

using uniqueness of records across all MSs and the Hub will be

explained.

It is the obligation of MSs to provide for the connection between the

URI and the particular data record in their national databases (e.g based

on a national URI, cross-reference tables, etc.) by creating and


Page 5 / 16

maintaining index (cross-reference) files or other techniques, and for

the preservation of modified records for historical reasons.

Section 2 Time of production

■ Time of production is a Mandatory element.

The time of production of the record will be presented by a full

timestamp (18-alphanumeric character string, not ISO because of

minimum length) with the following structure:

YYYYMMDD”T”hhmmss”TimeZone”, e.g. 20140305T232219+02

Depending on national practices for preparation of the SIMSTAT

dataset during the ESSnet1, the time of production could be:

- the same for all records, which would be equal to the time of

production of the dataset itself, or

- could be different for different records stipulating the exact time of

finalization of the data in the record.

“The time of production” during the trial tests will be the time of

finalization of the data into national production database. Examples

how time will be defined when Section 8 “Record action” is “I” Insert:

1. “The time of production” is the time when the data (line) has

been uploaded to national statistical database.

2. if correction has been made to the data (line) after uploading

that to national statistical database then “The time of

production” is the latest time when correction or corrections has

been made.

Section 3 Sending MS (Reporting MS)

■ Sending MS (Reporting country) is a Mandatory element.

The code list, attached to this coded variable, contains the 2-alphabetic

character codes for the list of 28 EU MSs according to the

GEONOMENCLATURE.

Section 4 Receiving MS (MS of Destination)

■ Receiving MS (MS of Destination) is a Mandatory element.

The code list, attached to this coded variable, contains the 2-alphabetic

character codes for the list of 28 EU MSs according to the

GEONOMENCLATURE.

Section 5 Data status flag for receiving MS

■ Data status flag for receiving MS is a Mandatory element.

The Data status flag for receiving MS is a 1-numeric character string.

The following code list is used:

0 – originally collected variable, not flagged as an outlier

1 – estimated


Page 6 / 16

2 – originally collected variable, flagged as an outlier and corrected

after contacting with the PSI

3 – originally collected variable, flagged as an outlier and auto-

corrected

4 – originally collected variable flagged as an outlier, but not corrected

5 – originally collected variable, flagged as an outlier, but confirmed


6 – other methods or dummy value

Section 6 Reference period

■ Reference period is a Mandatory element.

The reference period is the calendar month of dispatch of the

commodity according to the applied national reporting practice.

The reference period is a custom 6-numeric character string: the first

four digits = the year; the last two digits = the month, e.g. 201401 for

January 2014.

Section 7 Flow

■ Flow is a Mandatory element.

The code for this variable is a 1-numeric character string “2” for

“Dispatches” (as used in DocMet 400 for transmission of ITGS data to

Eurostat).

Section 8 Record action

■ Record action is a Mandatory element.

Record action is a 1-alphabetic character string. The variable indicates

the action that should be undertaken with each received record from a

SIMSTAT dataset.

A code list containing 2 codes is attached to this variable:

- “I” = “Insert”: The record has to be added to the existing data for

the particular reference period;

- “D” = “Delete”: The record, which should exist in the database of

the receiving MS, has to be deleted.

In case an old record has to be updated, a sequence of two logically

interconnected records has to be entered in the exchange file:

- the first one with Record action code = “D” indicating the old record,

which should be deleted;

- the second one with Record action code = “I” indicating the new

record to be inserted on the place of the deleted one.

It is apparent, that every completely new or corrected/updated record,

which is created to be “inserted” in the data file, has its own unique

combination of URI, time of production, sending/receiving MS codes

and trade flow indicator. BUT FOR THE PURPOSE OF DELETING

AN EXISTING RECORD, THE PARTICULAR LINE IS JUST A

COMMAND LINE AND NOT A DATA RECORD, that is why it

should carry the timestamp of the record to be deleted so that the

receiving DB could check the correctness of the command itself that it


Page 7 / 16

refers to the latest (of a possible series of corrected records with same

URI) existing record. It should not be in principle allowed to have more

than one occurrence of a record with the same URI, sending/receiving

MS and flow in a final (operational) database – older deleted/updated

records (with older timestamps) should be kept separately in other

tables for historical purposes, but this is not part of the data exchange

process and is not the subject of this document.

Concerning the data exchange file itself, it is even possible to have

several consecutive occurrences of couples of “Delete+Insert” records

to modify (update) an already reported record, and at the end to even

delete it. BUT IT IS OBLIGATORY that every consecutive “Insert”

record has a more recent (“bigger”) Time of Production timestamp in

order to guarantee the correct information flow.

Section 9 ID number of PSI

□ ID number of PSI is an Optional element.

The ID number of PSI is a variable string of maximum 14-

alphanumeric characters and for those MSs that would exchange this

data element, will be the real VAT Identification Number (VIN)

allocated to the trader in the sending MS in accordance with Article 214

of Directive 2006/112/EC according to the specification of national

VAT numbers given in DG TAXUD’s document “VIES – VAT

Number Construction Rules Functional Description”, version 16.1 from

year 2012.

VIN has a prefix (country code, 2-characters). “ID number of PSI” have

to have a prefix.

Section 10 ID Number of Partner

■ ID Number of Partner is a Mandatory element.

ID Number of Partner is a variable string of maximum 14-

alphanumeric characters and will be the real or simulated VAT

Identification Number (VIN) allocated to the partner trader in receiving

MS accordance with Article 214 of Directive 2006/112/EC. VIN is

composed of the 2-alphabetic character codes for the list of 28 EU MSs

according to the GEONOMENCLATURE - with the exception of

Greece, the code for which is “EL” - and the national VAT number

according to the specification of national VAT numbers given in DG

TAXUD’s document “VIES – VAT Number Construction Rules

Functional Description”, version 16.1 from year 2012.

If partner ID is not known and cannot be simulated, a dummy

“QV999999999999” may be used.

Section 11 Data status flag for ID Number of Partner

■ Data status flag for ID Number of Partner is a Mandatory element.

The data status flag for ID Number of Partner is a 1-alphanumeric

character string and indicates the way in which the VIN (VAT

Identification Number) of the partner (receiving) trader has been

determined. The following code list for this variable is used:


Page 8 / 16

0 – Originally collected variable

1 – estimated with 1:1 relation

2 – error/outlier corrected by PSI

3 – error/outlier auto-corrected

4 – error/outlier not corrected

5 – dummy ID, not intra-EU supply

X – estimated with 1:N relation

Y – dummy ID, no correspondence in VIES

Section 12 Commodity code

■ Commodity code is a Mandatory element.

Commodities are reported in accordance to the Combined

Nomenclature including codes from chapter 98 and code 9950 00 00.

The codes used are 8-numeric character strings. For Intra-EU estimates,

reporting is allowed also at HS6, HS4 and HS2 level (8, 4 or 2-numeric

character strings).

A special attention should be paid to the commodity codes initiated

with zero, which must be kept.

Section 13 Data status flag for commodity code

■ Data status flag for commodity code is a Mandatory element and is a

1-numeric character string. The following code list is used:


1 – estimated




corrected





Section 14 Country of origin

◘ Country of origin is an Optional element if collected and known, or

should be left Blank if unknown.

The 2-alphabetic character codes according to the

GEONOMENCLATURE will be applied when data is known and

reported, otherwise the field must be left blank.

Section 15 Data status flag for country of origin

◘ Data status flag for country of origin is a Mandatory element when

the Country of origin element is reported, or is left Blank when the

Country of origin element is blank.

The Data status flag for country of origin is a 1-numeric character

string. The following code list is used:


Page 9 / 16

0 – collected

1 - estimated

Section 16 Nature of transaction

◘ Nature of transaction is a Mandatory element presented by a 2-

numeric character string. If simplification threshold and/or individual

transaction threshold is applied the section should be left Blank if not

collected.

The code table for “List of Nature of transaction codes” – columns

A+B (Regulation (EC) 1982/2004, Annex III) is attached to this

variable. A “0” (zero) is added for column B for column-A codes 6, 7

and 8, or to all column-A codes if data is collected at national level only

at 1-digit level (column A).

Section 17 Data status flag for nature of transaction

◘ Data status flag for nature of transaction is a Mandatory element.

The Data status flag for nature of transaction is a 1-numeric character



1 – estimated




corrected





Section 18 Delivery terms

◘ Delivery terms is a Mandatory element if collected and known, or


The code table for “Coding of delivery terms” - Regulation (EC)

1982/2004, Annex IV – Incoterm codes” with the addition of the two

new delivery terms from ICC Incoterms2010 “DAP” – Delivered at

Place, and “DAT” – Delivered at Terminal, is attached to this variable

and will be applied when data is collected and reported, otherwise the

field must be left blank.

Specific terms of delivery, used for national purposes, are not allowed.

Section 19 Data status flag for Delivery terms

◘ Data status flag for country of origin is a Mandatory element when

the Delivery terms element is reported, or is left Blank when the

Delivery terms element is blank.

The Data status flag for Delivery terms is a 1-numeric character string.


Page 10 / 16



1 – estimated




corrected





Section 20 Mode of transport

◘ Mode of transport is a Mandatory element if collected and known, or


The Mode of transport is a 1-numeric character string. The code table

for “Coding of mode of transport” - Regulation (EC) No 1982/2004,

Annex V, is attached to this variable and will be applied when data is

collected, otherwise the field must be left blank.

Section 21 Data status flag for mode of transport

◘ Data status flag for mode of transport is a Mandatory element when

the mode of transport element is reported, or is left Blank when the

mode of transport element is blank.

The Data status flag for mode of transport is a 1-numeric character



1 – estimated




corrected





Section 22 Quantity in net mass

◘ Quantity in net mass is a Mandatory element except for the

enumerated below exceptions.

Quantity in net mass is a positive numeric field expressed in kg with 3

decimals (up to thousandths) with the total length of 15 characters

which includes Dot as decimal separator and 3 characters for decimals.


Page 11 / 16

The following exceptions apply for reporting net mass:

- For sea-going vessels (CN8 codes 8901 10 10, 8901 20 10, 8901

30 10, 8901 90 10, 8902 00 10, 8903 91 10, 8903 92 10, 8904

00 10, 8904 00 91, 8905 10 10, 8905 90 10, 8906 10 00, 8906

90 10) the quantity in net mass must be “0” (zero) and only

Quantity in supplementary unit (number of pieces) will be

reported;;

- For electricity (CN8 code 2716 00 00) the quantity in net mass

must be “0” (zero) and only Quantity in supplementary unit will

be reported;

- For the following commodities reporting of net mass is optional:

Component parts of industrial plant - CN8 codes beginning with

9880; Goods delivered to vessels and aircraft – CN8 codes 9930

24 00 and 9930 99 00; Goods delivered to and from offshore

installations – CN8 codes 9931 24 00 and 9931 99 00; Residual

products for simplification thresholds – CN8 code 9950 00 00;

Military goods. If not reported, field must be “0” (zero).

- For code 9950 00 00 reported when individual transaction

threshold is applied, the quantity in net mass must be “0” (zero).

Reporting whole real numbers (without decimal part) is allowed.

Section 23 Data status flag for Quantity in net mass

◘ Data status flag for Quantity in net mass is a Mandatory element

when the Quantity in net mass element is reported, or is left Blank

when the Quantity in net mass element is blank.

The Data status flag for Quantity in net mass is a 1-numeric character



1 – estimated




corrected





Section 24 Quantity in supplementary unit

◘ Quantity in supplementary unit is a Mandatory element only when

provided for in the Combined Nomenclature.

The Quantity by supplementary unit will be expressed in the particular

unit with 3 decimals (up to thousandths) with the total length of 15

characters which includes Dot as decimal separator and 3 characters for

decimals.


Page 12 / 16

The following exceptions apply for reporting quantity in supplementary

units:

- For the following commodities reporting of quantity in

supplementary unit is optional: Component parts of industrial

plant - CN8 codes beginning with 9880; Goods delivered to

vessels and aircraft – CN8 codes 9930 24 00, 9930 27 00 and

9930 99 00; Goods delivered to and from offshore installations

– CN8 codes 9931 24 00, 9931 27 00 and 9931 99 00; Residual

products for simplification thresholds – CN8 code 9950 00 00;

Military goods. If not reported, field must be “0” (zero).

- For code 9950 00 00 reported when individual transaction

threshold is applied, the quantity in supplementary unit must be

“0” (zero).

Reporting whole real numbers (without decimal part) is allowed. In the

absence of supplementary unit the field must be “0” (zero).

Section 25 Data status flag for Quantity in supplementary unit

◘ Data status flag for Quantity in supplementary unit is a Mandatory

element when the Quantity in supplementary unit element is reported,

or is left Blank when the Quantity in supplementary unit element is

blank.

The Data status flag for Quantity in supplementary unit is a 1-numeric

character string. The following code list is used:


1 – estimated




corrected





Section 26 Taxable amount (value)

■ Taxable amount (value) is a Mandatory element.

Taxable amount (value) is a positive numeric value expressed the

national currency of the sending MS with 2 decimals (up to hundredths)

and the total length of 14 characters which includes Dot as decimal

separator and 2 characters for decimals.

- in Euros for the Member States of the euro area

- in national currency units for the others.



Page 13 / 16

Section 27 Converted taxable amount (converted value)

◘ Converted taxable amount (converted value) is not provided and

left blank by the sending MS, but is obligatory inserted by the Hub

when data is sent to the receiving MS.

The Converted taxable amount (converted value) is a positive numeric

field and gives the reported values in national currency by the sending

MS converted to the national currency of the receiving MS by agreed

conversion coefficients stored in the Hub, with the total length of 14

characters which includes Dot as decimal separator and 2 characters for

decimals.


The following conversions are done by the Hub:

- from non-Euro to Euro;

- from Euro to non-Euro;

- from non-Euro to non-Euro.

If both sending and receiving MSs are members of the euro area, no

conversion is done by the Hub – the values are copied one-to-one.

Section 28 Data status flag for Taxable amount (value)

■ Data status flag for Taxable amount (value) is a Mandatory

element.

The Data status flag for Taxable amount (value) is a 1-numeric

character string. The following code list is used:


1 – estimated




corrected





Section 29 Statistical value

■ Statistical value is a Mandatory element.

Statistical value is a positive numeric value expressed the national

currency of the sending MS with 2 decimals (up to hundredths) and the

total length of 14 characters which includes Dot as decimal separator

and 2 characters for decimals.


- in Euros for the Member States of the euro area


Page 14 / 16

- in national currency units for the others.

Section 30 Converted statistical value

◘ Converted statistical value is not provided and left blank by the

sending MS, but is obligatory inserted by the Hub when data is sent

to the receiving MS.

The Converted statistical value is a positive numeric field and gives the

reported values in national currency by the sending MS converted to the

national currency of the receiving MS by agreed conversion

coefficients stored in the Hub, with the total length of 14 characters

which includes Dot as decimal separator and 2 characters for decimals.


The following conversions are done by the Hub:

- from non-Euro to Euro;

- from Euro to non-Euro;

- from non-Euro to non-Euro.

If both sending and receiving MSs are members of the euro area, no

conversion is done by the Hub – the values are copied one-to-one.

Section 31 Data status flag for Statistical value

■ Data status flag for Statistical value is a Mandatory element.

The Data status flag for Statistical value is a 1-numeric character string.



1 – estimated




corrected





Notes:

All records are confidential during SIMSTAT ESSnet and trial tests. Confidentiality will

be changed as metadata file that will be defined later. Confidentiality is generally not

defined at the micro data level.

3. TECHNICAL SPECIFICS OF THE CHARACTER SEPARATED VALUES

SIMSTAT FILE

The following characteristics for the physical file for the SIMSTAT data exchange are generally

valid:


Page 15 / 16

- One line of the file represents one data record;

- each record of the data file contains all data fields defined as sections of the dataset

structure, populated with allowable values – codes, string or numeric values, etc., or left

blank;

- values of the data fields in each record are separated by a special character determined for

that purpose. For this dataset the character separator is the semi-colon “;”. The separator

is inserted after a data field even it is left blank;

- numeric values are entered without leading or trailing blanks or zeros; for the numeric

data elements, when reported value is <1, the leading zero before the Dot (the decimal

separator) has to be entered. For example if Section 22 “Quantity in net mass” would be

half a kilogram then the value could be entered as “0.5” or “0.50” or “0.500”, but not

as“.5” or “.50” or “.500”.

An example of sequence of record and files

An example of sequence of records within two different files from a sending MS and the final

result on the side of the receiving MS:

a. Correct sequence

Sequence

of actions

sending

MS receiving

MS URI

record

action Result

1 AT FI AT1401 I

New record with URI AT1401 is inserted

and is active

2 AT FI AT1401 D

Record with URI AT1401 is deleted. No

record is active

3 AT FI AT1401 I


and is active

b. False sequence - if MS/CCN/HUB do not pay attention to the sequence

Sequence

of actions

sending

MS receiving

MS URI

record

action Result

1 AT FI AT1401 I


and is active

2 AT FI AT1401 I

Duplicated record with URI AT1401 is

discarded due to error

3 AT FI AT1401 D

Record with URI AT14011 is deleted. No

record is active

About “URI” data element


Page 16 / 16

If you create a new record only for correction of the statistical data in an already exchanged

record a new URI has to be always used if the year of the reference period changes as it is part of

the URI.

Defining common rules for structure of “URI” would bring important operational advantages.

Uniqueness of a record is practically guaranteed only on the basis of all the five data elements –

“URI”, “Time of Production”, ”Sending MS”, “Receiving MS” and “Flow”.

Documents

Data structure definition for the production testing