Mapping Tool User Manual 2012

8/17/2019 Mapping Tool User Manual 2012

1/19

Supporting Publications 2012:EN-XXX

© European Food Safety Authority, 2012

TECHNICAL REPORT

User Manual for Reporting Officers and Reportersfor Mapping Member State Standard Terminology to EFSA

Standard Terminology

European Food Safety Authority1

European Food Safety Authority (EFSA), Parma, Italy

ABSTRACTEuropean Food Safety Authority (EFSA) is charged with coordinating the reporting of

zoonoses, zoonotic agents, animal population, antimicrobial resistance and foodborne

outbreaks in the European Union under the Directive 2003/99/EC as well as analysing

and summarising the data collected. For data collection purposes EFSA has created a

sample Excel based mapping tool to allow Member States to manage the mapping of

their standard terminology to that published by EFSA. This technical report is a user

manual document that describes how to map terms and generate XML for submission of

2012 data, via the published BIOMO data models.

© European Food Safety Authority, 2013.

KEY WORDSZoonoses, XML, BIOMO Data Models ,XSD, DCF, Excel.

1 Correspondence: [email protected]

mailto:[email protected]:[email protected]:[email protected]:[email protected]


2/19

User Manual for the 2012 Zoonoses Mapping Tool

Supporting publications 2012:EN-XXX 2

SUMMARY

This user manual document provides guidance on use of the BIOMO mapping tool for the

submission of 2012 data by the Member States and other reporting countries.

The manual provides a general introduction to the mapping tool. The report covers allaspects related to the management and use of the mapping tool. In particular detailed

guidelines are provided for the mapping of standard terminology and the creating of a

locally validated XML file for submission of data to EFSA via the DCF (Data CollectionFramework)

Instructions are also given on the updating of EFSA standard terminology dictionaries,and how to incorporate these updates in the Mapping tool.

This manual is specifically aimed to guide the mapping of member specific standard

terminology to those published by EFSA, thus creating a much needed documented and

centralized mapping of controlled terminology.


3/19



TABLE OF CONTENTS

ABSTRACT 1

Summary 2

1

Introducing the Mapping Tool 4

1.1

Getting Started by Profiling Your Data ...................................................................6

1.2 Mapping Your Profiled Terms ...................................................................................9 2

The coding of your data 11

2.1.1 The LookUpDicTerm Function............................................................11

3

Generating your XML 13

3.1 Copy, pasting and preparing the data for XML export....................................... 13 3.2 Activating the XML Developer tab ......................................................................... 15 3.3 Adding your data model XSD as an Excel XML Map ........................................... 16 3.4 Mapping the XML Map to the data in the Coded_XML worksheet .................... 17

3.5 Exporting our XML Map table to an XML File ....................................................... 18


4/19



1 Introducing the Mapping Tool

The Mapping tools main goal is to provide a simple yet useable platform for member

states to map their country specific standard terminology to those published by EFSA,

producing an XML file for the submission of sample or aggregated based zoonoses

monitoring data via the DCF.

This User Manual introduces the basic functionalities available in the Mapping Tool, such

as:

Mapping terms explicitly

Providing lookup and encoding of explicitly mapped terms

Generating and validating XML via published EFSA data mode XSD

This manual will include some suggestions on how to proceed when you want to profile

your data, these are simply suggestions on our behalf and do not need to be followed to

the letter.

The mapping tool is essentially an Excel workbook which has three default worksheets,

which are colour coded. In the diagram below you will see these default worksheets, the

red coloured tab worksheet (number 1 in diagram) which contains the text

ZOO_FACT_(MODEL_NAME) is where you will put your own data, this is the first step,

whereby you must fit your current data into the structure of the data model in question.

This worksheet is the format in which Member States must organise their data for

successful mapping and submission of data to EFSA.

The next worksheet of importance is coloured green and is called CODED (number 2 in

diagram), this worksheet reads the data inserted into the red ZOO_FACT worksheet, and

then searches for the correct mapped term in the various pick list worksheets included in

the Excel Workbook (more on this later).

The last coloured worksheet (number 3 in diagram) is called DCF_FORMATTED_DATA

(XML in the old mapping tool), and this is where you will eventually paste your finisheddata, either to export as XML via the published BIOMO XSD schema or to leave as is in

Excel format, remember the DCF supports both Excel and XML format. It is very

important that the Excel workbook is saved in Excel 97-2003 Workbook (*.xls) format as

the DCF only supports Excel files of this type, also when submitted to the DCF if the

workbook contains more than one worksheet it is only the first worksheet in the

workbook that is read, this is the reason why DCF_FORMATTED_DATA is the f ist sheet in

the series.


5/19



Figure 1

The CODED worksheet contains the LookUpDicTerm function that will query the

mapping you will create in each of the pick list worksheets. The function will return a

mapped EFSA code, it is important to make sure that the mappings you have created

are correct. In figure 2 below, highlighted by number two, there is a cell under the

ZOO_FACT_AMR_ISOLATE_AST data model element “zoonosis” selected, as you cansee in number three this cell contains a formula called LookUpDicTerm.

The LookUpDicTerm excepts as its first value the name of a pick list to search, in this

case “ZOO_CAT_PARAM_AMR ” and a value, the value will be selected from the red

worksheet ZOO_FACT_AMR_ISOLATE_AST, this value must have been mapped to a

pick list term in the ZOO_CAT_PARAM_AMR pick list, if not the function will return

NOT MAPPED, which mean the value you are trying to use has no valid mapping. All

NOT MAPPED values must be fixed before a valid submission can be made.

Number four in figure 2 is highlighting the fact that ZOO_FACT_(MODEL_NAME)

elements (like repCountry, lang, zoonosis .etc) that require a EFSA standard term(pick

list), are linked, you need to simply click on the link and you will be moved to the

correct pick list/catalogue in the workbook, where you can easily create your mappings.By default the CODED worksheet only has 200 pre-filled cells with formula’ s, if you

have more than 200 data entries (rows) in the red worksheet

ZOO_FACT_(MODEL_NAME), you will need to extend the number of rows with

formula’s to match. For example I have 560 rows in ZOO_FACT_(MODEL_NAME),

only the first 200 will be visible in the CODED worksheet, you will need to select row

number 200 in the CODED worksheet and drag it to the corresponding row number in

ZOO_FACT_(MODEL_NAME), which is 560, this way all values will be copied across.

Figure 2Each ZOO_FACT_(MODEL_NAME) data model contains an up to date list of BIOMO

standard terms; these are the pick lists that are also available on the online Zoonoses

Web reporting tool. Each model will contain only the pick lists that it needs to reference .

General Points on Excel worksheets

It is worth mentioning here that because each cell in the CODED worksheet contains a

formula, the more data you put in, the slower the use of the Excel worksheet will

become. I personally would recommend that you profile all your data, mapping all the

terms existent, and then create multiple copies of the excel workbook. You can then split

the data into lots of say five to ten thousand rows, thus keeping the excel worksheets

lighter and quicker to use, of course this is just a recommendation.


6/19



1.1 Getting Started by Profiling Your Data

Figure 3

When you open any of the data models ZOO_FACT Files you must “ enable”

Marcros so that the custom function LookUpDicTerm can run. The figurebelow shows you the sequence of steps to follow in doing this.


7/19



If you do not already have a list of all the terms that need to be mapped per data model

to EFSA terms, one simple way to profile your data is to use an Excel pivot table. To

profile quickly your data follow the following steps:

1.

Before inserting a Pivot table into a new worksheet, first select all data in the redZOO_FACT_(MODEL_NAME) worksheet by c licking the select all table feature

indicated in the figure below.

Figure 4

2.

Then select the Insert Tab from the control Ribbon, and choose Pivot table.

Figure 5

3. The table range is already set by virtue of the “select all table feature “we

performed in step 1, now we want to create a pivot table in a new sheet.


8/19



Figure 6

4. We now have a pivot table that has all the data values that are in the

ZOO_FACT_(MODEL_NAME) red worksheet, so we can now start profiling the

elements like zoonosis and matrix which need to have there terms matched with

the BIOMO standard term pick lists ZOO_CAT_PARAM_ZOO and

ZOO_CAT_MATRIX. The below figure shows the pivot table that we created,

where we have selected Zoonosis, as you can see on the left of the figure, all the

unique terms in the Zoonosis column are displayed. These are the terms that we

need to insert into the ZOO_CAT_PARAM_ZOO pick list worksheet . We now

copy the unique terms under the zoonosis column on displayed on the left, and

paste them somewhere on in the profiling worksheet to the right of the pivot table

so that we can reference them later.

Figure 7We repeat the above step for all data model elements that need to be mapped to

the EFSA standard terms pick lists, copy and pasting the resultant unique terms

into columns to the right the pivot table.


9/19



Figure 8

5. Eventually the list to the right of the pivot table will have all the unique terms that

need to be mapped like in the figure below.

Figure 9

1.2

Mapping Your Profiled TermsAt this point you have profiled and saved the l ists of unique terms that need to be

mapped from your data set to those of EFSA’s. Step five in “profiling your data” lists the

profiled terms that we will use in our example mapping. The following sequence of steps

will map a these values to their corresponding EFSA standard term.

The figure below highlights the sect ions where mapping is performed in the

ZOO_CAT_PARAM_AMR pick list; all pick lists have these mapping sections. A mapping

section is linked to a specific standard term pick list. Area one is where your Member

State specific terms are inserted and area two where the BIOMO standard term that

matches the Member State term is selected from the dropdown box highlighted in area

figure 10 below.


10/19



Figure 10

The following steps describe the mapping process.

1. Open the CODED worksheet in the workbook. As we know from figure 2 on page

5, the Zoonosis element name is linked to the ZOO_CAT_PARAM standard term

pick list. By using the hyperlinked index in the element “zoonosis” we can move to

the ZOO_CAT_PARAM _AMR pick list.

Once we have selected the ZOO_CAT_PARAM _AMR standards terms pick list we

paste in our profiled terms under column A (MemberState_TERMS.

Figure 11

2. Once you have pasted your profiled terms into the A (MemberState_TERMS

column, it’s time to start looking for their matching term in the Zoonoses data

models standard terms pick list. In column B, EFSA_TERM, there is a dropdownlist of all the standard terms in the ZOO_CAT_PARAM _AMR pick list for the

element Zoonosis, you have to now search through this list until you find a


11/19



standard term that matches your Member State specific standard term. When you

find a match, select it, and now your standard term is physically matched

to EFSA’ s term. The LookUpDicTerm function(figure 2, page 5) in the

CODED worksheet will now return for every occurrence of your standard term, in

this example “Salmonella - S. Abortusovis ” the code value“RF-00000372-

MCG”.

Figure 12

3. We repeat the above step until all your terms are matched for all you profiled

elements

2 The coding of your data

The CODED worksheet is an exact copy of the ZOO_FACT_(MODEL_NAME)

red worksheet, every cell in the CODED worksheet references its opposite cell

in the ZOO_FACT_(MODEL_NAME) red worksheet. The default CODED

worksheets for all the published BIOMO data models have two hundred pre-

filled rows containing all the correct formula references.

Only data model elements that need to be mapped to BIOMO standard term

pick list need to have the LookUpDicTerm funct ion reference, all the rest of

the elements have a simple formula that select s its cell counter parts value inthe ZOO_FACT_(MODEL_NAME) red worksheet and copies it. When an empty

string or value is found, the formula replaces this with a string NULL. The NULL

string values will be replaced by a true null value when we generate XML via an

XSD.

2.1.1

The LookUpDicTerm Function

The LookUpTerm Function is the glue between the CODED worksheet and the

EFSA standard terminology pick list worksheets. In figure 13 below, you can see

that the zoonosis element in the CODED worksheet has a LookUpDicTerm

function reference; this is because all zoonosis values must come from BIOMO’s

standard term pick list ZOO_CAT_PARAM.


12/19



The interactions one through to four in figure 13 outline how the LookUpDicTerm

interacts with the pick lists linked with data model elements, in this case the zoonosis

element in the ZOO_FACT_AMR_ISOLATE_AST data model, to return a mapped term

code.

The LookUpDicTerm excepts as its first value the name of a pick list to search, in this

case “ZOO_CAT_PARAM_AMR ” and a value, step one in figure 13, the value will beselected from the red worksheet ZOO_FACT_AMR_ISOLATE_AST, step two in figure

13. If you look again at step one, you will see that under the element zoonosis “Not

Mapped” is displayed. This means that the value you are trying to use has no valid

mapping. The value we are looking for in ZOO_FACT_AMR_ISOLATE_AST can be seen

in step two, which is “Salmonella - S. Abortusovis” .

In step 3 you can see that in the pick list ZOO_CAT_PARAM_AMR we mapped the term

“Salmonella - S. Abortusovis” under column A (MemberState_TERMS) to column B

(EFSA_TERM) “RF-00000372-MCG | Salmonella - S. Abortusovis” .

The LookUpDicTerm function can now use the value found in

ZOO_FACT_AMR_ISOLATE_AST!E2 ( “Salmonella - S. Abortusovis” ) to search in

column A (MemberState_TERMS) for a text match for “Salmonella - S. Abortusovis” .When a match is found, and it will be because we just mapped it, the function will ret urn

the code “RF-00000372-MCG” from the “RF-00000372-MCG | Salmonella - S.

Abortusovis” ZOO_CAT_PARAM_AMR pick list.

In step four you can now see that under the zoonosis element the correct coded term is

now displayed for the member state term “Salmonella - S. Abortusovis” . All these step

need to be repeated for all data model elements that require a mapping in an EFSA

standard control terminology pick list.

Figure 13


13/19



3 Generating your XML

With the release of Office Excel 2003, the importing and exporting of XML via a

valid XSD has been supported. Under the Developer tab, an XML tool exists forthe creating of what Excel calls XML Maps. When BIOMO publishes its data

models, it also publishes an XML schema or XSD, which supports the strict

structure and validation of any data model, and as such can be used to generate

from our CODED worksheet a valid XML file containing all our coded values

The following sequence of steps will produce and exportable validated data set

in XML.

3.1 Copy, pasting and preparing the data for XML/Excel export

Copy all the data from the CODED worksheet. The quick way to copy all cells

containing data is to use the hot key combination of Control + Shift +Space as

indicated in the figure below, or select the select all square in the top left had

corner of the worksheet, as indicated in figure 4 page 7.

Figure 14

1. In step two we want to paste all the values we copied from the CODED worksheet

in step one, into the Excel worksheet DCF_FORMATTED_DATA. The worksheet

CODED uses cell formulas to generate its cell contents, by referencing data in the

ZOO_FACT_AMR_ISOLATE_AST red worksheet. In the DCF_FORMATTED_DATA

worksheet we only want to paste the actual cell values and not the formulas contained in the CODED worksheet. To do this, first we select all the cells in the

active DCF_FORMATTED_DATA worksheet, and then under Paste we choose

“Paste Values” , you can also do the same under Paste Special, selecting the

values option in the dialog box.


14/19



Figure 15

2. This next step will replace all occurrences of the string “NULL” in cells, with an

actual true value of null, this is to prevent errors in generating and validating XML

via the XSD. Empty string values in Excel cells cause problems for the XML parser

used by Excel, that is why we replace empty strings first with a string “NULL” and

later with a real null value. To replace the string “NULL” , choose the Find and

Select tab, then select Replace, when the Find and replace dialog appears type

NULL into the Find what: search box, leave the Replace with: text box empty

(indicates null to Excel). Finally select the Replace All button which will now

update all instances of NULL with a true null value.

Figure 16

If you do not want to submit XML validated data you can stop here at this point, you

must only insure that the DCF_FORMATTED_DATA worksheet is the first worksheet in

the workbook. It is very important that the Excel workbook is saved in Excel 97-2003

Workbook (*.xls) format as the DCF only supports Excel files of this type, also whensubmitted to the DCF if the workbook contains more than one worksheet, it is only the


15/19



first worksheet in the workbook that is used, this is the reason why

DCF_FORMATTED_DAT A has the fist sheet in the series.

3.2 Activating the XML Developer tab

The Developer tab allows you to import and create XML files, normally it is notimmediately available as a default option and so needs to be activated direct ly by

you. The following figure indicates the sequence of selections you need to undergo

to activate the Developer tab.

Figure 17


16/19



3.3 Adding your data model XSD as an Excel XML Map

To import the XSD that you want to use to validate the data model you must first go

to the Developer tab and select the Source icon. The Source icon will activate the

XML Source window, at the bottom of this window there is a XML Maps button

which you need to select. Once the XML Maps button is selected the XML Maps

dialog box will appear, you now need to select the Add button, which will activate a

File d ialog. In the file dialog you now navigate to the location of the data model XSD

that you wish to import as an Excel XML Map.

Figure 18

Once you have selected the XSD, the Multiple Roots dialog will appear, please

select dataset as indicated in the figure below and then select OK.

Figure 19


17/19



3.4 Mapping the XML Map to the data in the Coded_XMLworksheet

1. After we have imported our data models XSD file in as an XML Map, you can now

see the structure of the XSDs elements in the XML Source window. The section ofthe XML Map that we are interested in is under the dataset branch. To map the

XML Map to your data you will need to left click on your mouse button on the

Result branch, with the mouse button still held down you will need to drag the

Result branch across to the Top Left most cell (in this case resultCode), containing

the first element column name in the Coded_XML worksheet. Let go now of the

mouse button when the Top Left cell becomes active. Excel will now create a

Table with our XML Map as its basis. We have now successfully mapped our XML

Map to the elements in our data Model.

Figure 20


18/19



2. Setting the Validate against Schema for Export Feature. To activate this feature

we need to first select the Developer tab followed by the Map Properties

button. The XML Map Properties dialog box will now be displayed, under XML

Schema va lidation, select the Validated data against Schema option followed

by the Ok button. Now when we export the data, the XML file c reated will

automatically be validated by the data model XSD that we imported.

Figure 21

3.5 Exporting our XML Map table to an XML File

To Export our data now we first select the Developer tab followed by the Export

button. The Export button will bring up the Export XML File dialog box. Here you

can name and navigate to where you want to store your XML file. After naming your

file simply select the export button to initiate the export procedure.

Figure 22


19/19


Supporting publications 2012:EN XXX 19

Documents

Mapping Tool User Manual 2012