Dictionaries Tutorial Overview – Public Sector NIEM Team, November 2012 CAM Test Model Data Deploy Requirements Build Exchange Generate Dictionary XML

<Insert Picture Here>

Dictionaries Tutorial

Overview – Public Sector NIEM Team, November 2012

CAMTest Model Data

Deploy Requirements

Build Exchange

GenerateDictionary

XML Exchange Development

Content

Assembly

Mechanism

Copyright ©2011/12, Oracle. All Rights Reserved.

The following is intended to outline Oracle general product direction. It is intended for information purposes only, and may not be incorporated into any contract. It is not a commitment to deliver any material, code, or functionality, and should not be relied upon in making purchasing decisions.

The development, release, and timing of any features or functionality described for Oracle’s products remains at the sole discretion of Oracle.

Disclaimer Notice

Slide 2


Tutorial Overview

• Dictionary Concepts

• Dictionary Services and Facilities

• Examples Walkthrough

• Summary and Review


DICTIONARY CONCEPTS

Understanding Dictionaries Usage


The Dictionary Role and Content

Provide a set of common, controlled, and managed information data structures and component definitions for a

particular purpose and community

Data elements are organized into core and domain-specific components

Core components are used across information exchanges and can be described by structure,

semantics, and definition with global context

Context-specific components are

particular to a limited number of information exchanges and have a

limited scope and applicability

Naming and Design Rules (NDR) specify how

each of these components are defined

and utilized


The 8 “D”s of software development

• Design• Develop• Document• Dictionaries• Discovery• Differentiate• Deploy• Diagnose

Predictable, Repeatable, Reusable Process

Solution Delivery Process

LIFECYCLE


XML Samples

DictionariesDiscovery

Exchange Delivery Lifecycle

Diagnose

Differentiate

Design

Develop

Validated

Templates /

Schema

XSD Schema

Evaluation Reports

XMI / UML Models

Production

Results

Requirements

Updates

Documentation

Drag and Drop Visual

Designer

DocumentDeploy

Exchange

Templates

CAM editor Toolkit

*CAM – Content Assembly Mechanism – OASIS standard


CAM Dictionary Framework

CAM Editor

Drag and Drop

Designer

Collection Manager

CAM RulesWizards

Model Generation

Exchange Structure Builder

Dictionary

Dictionary

Dictionary

Codelist

Examples and instructions:http://www.cameditor.org/#Dictionaries

Codelist


How to create dictionaries

• Top down - directly from information models

• Enterprise modelling tools

• SQL database physical table definitions

• Bottom up

• Harvesting from existing data assets

• Harvesting from XML exchange structures and schema

• Manual entry through Excel spreadsheets

• Using neutral abstract standards based representations


Dictionary Content Model

Child(BBIE)Item

Child(BBIE)Item

Parent (ABIE)Item

Parent (ABIE)Item

Follows

Naming and Design Rule (NDR) principles and guidelines

Canonical Components

DictionaryXML

Canonical Components

DictionaryXML

ebXML CCTS terms (ABIE, BBIE, ASBIE)

Parent = Aggregate Business Information Entity

Child = Basic Business Information Entity

Attribute = Association Business Information Entity

Parent (ABIE)Item

Parent (ABIE)Item

Parent (ABIE)Item

Parent (ABIE)Item

Parent (ABIE)Item

Parent (ABIE)Item . . . . .

Child(BBIE)Item

Child(BBIE)Item

Child(BBIE)Item

Child(BBIE)Item

Child(BBIE)Item

Child(BBIE)Item

Attribute(ASBIE)

Attribute(ASBIE)

Attribute(ASBIE)

Attribute(ASBIE)

* CCTS – Core Components Technical Specification

Attribute(ASBIE)

Attribute(ASBIE)

Attribute(ASBIE)

Attribute(ASBIE)

Each compound component

Each atomic component

Optional attributes of component


Example – Person Name

• Person Name (ABIE)• Verified Details? (ASBIE)

• First Name (BBIE)• Middle Name (BBIE)• Last Name (BBIE)

• Previous Name? (ASBIE)• Language Code (BBIE)

Language Code may exist independently of Person Name

Verified Details and Previous Name are flags that denote additional information about the entity they are associated with

There are three component items aspects:

structure relationships; content rules; definitions


Semantics Approach

• Neutral formatted XML dictionaries

• Contains core information on XML

components

• Representation subset of UN/CEFACT Core

Components Technical Specification (CCTS)

• Can generate XSD, OWL, XMI and more as

desired (contains the vocabulary terms,

definitions, purpose and content models)

• Renamer tool aligns with NIEM NDR


Value Proposition

• Allow business data analysts to focus on information needs and build data exchanges

• Aligns information with industry standards and enterprise information stores

• Ensures consistent usage and definitions using profile of syntax-neutral terms and constructs

• Components can be derived from existing domain schema and data structures

• Supports providing sharing and collaboration services

• Allows development of supporting analytics tools


Sources of Your Dictionaries

• Ingest existing application semantic artifacts – e.g. XML Schema, Database schema, Excel spreadsheets, Enterprise Data Models (EDM)

• Starting from scratch:• Excel spreadsheet, or• SQL table entry, or• XML components entry

• Then generate dictionaries of components• Download existing domain dictionary collection


Renaming Tool

• Expands abbreviations (has built-in lists with context)• Corrects common typos and spelling errors (200+)• Adjusts names to camel case with no separators (_)• Retains acronyms• Adds name suffix according to NIEM representation terms

and type of content, e.g. date, number, code, etc• Optionally refactor rules by their type for interoperability;

make nillable optional; relax DB length restrictions• Can optionally insert namespace prefix• Can be configured by editing the lists in: word-utils.xsl


Analytics Tools

• Compare to Dictionary tool – provides reuse scoring statistics for information exchange structures

• Template Evaluation Tool – • Provides Naming and Design Rule (NDR)

analysis• Spelling Checking• Information exchange interoperability analysis• Data modelling errors and warnings• Naming / use inconsistencies / omissions


CAM: Top Down Exchange Assembly

Canonical XML Components DictionaryCanonical XML Components Dictionary11

Component Associations and CouplingsComponent Associations and Couplings22

Exchange Templates and RulesExchange Templates and Rules33

W3C Schema and Model RepresentationsW3C Schema and Model Representations44

Delivery Control, Messaging, SecurityDelivery Control, Messaging, Security55

Presentation

Collaboration S

ervices

Artefact relationships

Implementation Artifacts and ExamplesImplementation Artifacts and Examples66

Navigation and Query

Tools

Domains Reuse Library

Components

Automatic Rendering Tools

Deployment Environments and Middleware

XML

User Inserts

Relationship Lookups

TemplateCatalog

Testing Workbench, Rules Engine, Data Samples, Integration ETL

SchemaXML

Models Components

Canonical Dictionary Collections

*CAM – Content Assembly Mechanism – OASIS standard


DICTIONARY SERVICES AND FACILITIES

Tools Architecture


Dictionary Management / Acquisition


CAM Dictionary From Existing XML schema

OASIS CAM template

OASIS CAM template

Source schema in

XSD syntax

Source schema in

XSD syntax

NDR Evaluation,Refactor,

RenamingTools

NDR Evaluation,Refactor,

RenamingTools

33

GenerateStandard

Components Dictionary

XML

GenerateStandard


XML

44

Canonical XML Components

DictionaryXMLXML

Import

11 22

Apply

Build


CAM Dictionary From ERwin models

EDM

EleEle

DefDef

DDLDDL

11

Export Components in XSD syntax

Collection of objects from model

From Enterprise Data ModelImport XSD and refactor for use with CAM editor

OASIS CAM

template

OASIS CAM

template

Model Components XSD schema

Model Components XSD schema

NDR Evaluation,Refactor,Renaming

Tool

NDR Evaluation,Refactor,Renaming

Tool

Apply Naming and Design Rule (NDR) checks and edits

44

GenerateStandard


XML

GenerateStandard


XML

55

Dictionary

of exchange components

XMLXML

Import 33Analyst Review

ebXML CCTS compatible

(ABIE, BBIE, ASBIE)

22


CAM Dictionary From SQL tables

11

SQL tablesDrag and

Drop

Designer

from SQL to

XML

22

33

Configure renamer drag and drop

XML mapping

44 Generate Dictionary

from XML components


CAM Dictionary From Excel Spreadsheet

11

Dictionary Generator Spreadsheet

22

33

XML data mappingExport to XML

results

Download Excel to Dictionary pack

http://sourceforge.net/projects/camprocessor/files/Dictionaries/Tools/

Enter components

README

has complete instructions


Renamer Tool – Align to XML NDR-renamer added by default – otherwise edit name as desired

Leave blank or pick namespace to process

NIEM mode (with term representations); or CamelCase (Java style)

Leave as normal

Optionally insert prefix to all items without prefixSet to false to retain all rules “as is”

Note: Produces renamer report XML file of old / new names.

Can be reviewed as spreadsheet.


Dictionary Collection Manager

11

dictionary collection control file

22

33

Collection name

Component dictionaries

Download Excel to Dictionary pack

44

Editor preference


Available XML Dictionaries

• NIEM 2.1 dictionaries• CBRN dictionary• Emergency dictionary• Family dictionary• Immigration dictionary• Infrastructure dictionary• Intelligence dictionary• Justice dictionary• Maritime dictionary• Screening dictionary• Trade dictionary• Immigration blueprint• NIEM core dictionary

Available from download site

direct link:http://sourceforge.net/projects/camprocessor/files

XMLXML XMLXML XMLXML XMLXML XMLXML XMLXML

+ includes spreadsheets and sample models

Note: Those marked in bold are model style dictionaries with recursive components.


EXAMPLES WALKTHROUGH

Understanding capabilities


Visual Designer with Dictionary Collection

Drag n’ Drop Tool

Drag n’ Drop Tool

Industry dictionaryIndustry dictionaryDomain dictionaryDomain dictionary

Component Definitions

44

Component Definitions

Search

Tools

Search

Tools

22

Exchange

Designer

Exchange

Designer

11

Insert

Dictionary

Parent

Components

Insert

Dictionary

Parent

Components

33

Completed Exchange Template

Completed Exchange Template

55

Collection


Developing domain dictionaries (EIEM)

• Allows domains to manage their components libraries• Provides consistency for project development teams

• Sets of NIEM consistent XML exchange components• Aligned to enterprise data stores• Optimized for reuse and interoperability

• Save time and effort across the enterprise• Perennial question for developers – when should I use NIEM

components, and when our own local ones?• Are there components already available for that purpose?

• Provide formal mechanisms and procedures to share components and collaborate across SDLC process

• Provide external parties consistent data views

*EIEM – Enterprise Information Exchange ModelCAM Toolkit for EIEM generation


Example: N-DEx Dictionary and Model

(Harvested from N-DEx schema – ndexia.xsd and ndexibp2.xsd)

Freemind Interactive ModelDictionary viewer with N-Dex components


Example - Suspicious Activity Report V2.0

dictionariesdictionaries

XMLXML

XMLXML

XMLXML

• SAR v1.5 components

• NIEM core dictionary

• LEXS 3.1.4 dictionary

• SAR v1.5 components

• NIEM core dictionary

• LEXS 3.1.4 dictionary LEXS

components

referenced

New structure components

based on NIEM + SAR + new

SAR components

Definitions stored as syntax

neutral canonical XML

NIEM core components

Dictionary Collection

Namespaces of

dictionary components

DRAFT


SUMMARY AND REVIEW

Reviewing Dictionary Technology


Opportunities / Challenges

• Paradigm shift

• Teams used to manually crafting XML Schema and Java code

in developer XML IDE tools

• Collaboration between developers and data analysts

• Lowering learning curve for application development teams

• Integration into SDLC and NIEM IEPD processes

• Unit test XML instance generation

• XML test framework and validation services


Work in Progress

• NIEM ‘lite’ – easier simpler use by adding context

aware mechanisms to automatically filter dictionary

components based on domain use case

• Expose context lists to allow click and choose

selections

• Dictionary Analytics tools – find out what is going on

inside your dictionaries – consistency, omissions,

content/name conflicts, orphans

• Store and Share Collaboration - API with SOA Suite

Repository integration


Value Proposition

• Contextual mechanisms allow linear number of true

core components – not exponential growth

• Easier for practitioners to leverage with more

predictable and repeatable results

• Enable domain use horizontally across government

• ETL mapping automation support

• Continuous collaboration environment


CAMeditor.ORG Project Statistics

‹#›

SNAPSHOT OF PROJECT

ACTIVITIES

50,000 CAMeditor.org page

visits in one year

140 countries have downloaded tools; 25% of visitors are

from U.S.;600+ downloads

weekly

50 views of online video training

resources weekly

6 languages now available

Documents

Dictionaries Tutorial Overview – Public Sector NIEM Team, November 2012 CAM Test Model Data Deploy Requirements Build Exchange Generate Dictionary XML