Setting Up a New Project

7/23/2019 Setting Up a New Project

http://slidepdf.com/reader/full/setting-up-a-new-project 1/45

<Insert Picture Here>

Oracle Data Integrator

Technical Session 3-4:

Defining The Logical Architecture. Setting up a new project

Designing Interfaces and Packages




Agenda

• ODI Repositories

• ODI Data environments

• Reverse and enhance metadata

• Design your transformations

• Create packages workflow




ODI Repositories:

scalable metadata



Example of a Repository Set-Up

Models

Projects

Execution

Work Repository(Development)

Models

Projects

Execution

Work Repository

(Test & QA)

Execution

Execution Repository

(Production)

Security

Topology

Versioning

Master

Repository

Create and archiveversions of models,projects andscenarios Import released

versions of models,projects andscenarios for testing

Import released andtested versions ofscenarios forproduction

Development – Test – Production Cycle




ODI Data environments:

scale from development to production



Logical vs. Physical Architecture

• Physical Architecture

• Defines physical resources• Describes the real locations of servers and schemas

• How to access databases (Pack. Appl, etc.)

• Which data area to use, also as physical staging area

• Logical Architecture• Abstract view of the resources

• The accounting application schema is defined only once

here. It is one Logical Schema implemented in different

Contexts.



Mapping Logical and Physical

Resources

MS SQL Server

db_dwh

Windows

Development site: New York

Windows

MS SQL Server

db_dwh

db_purchase

Production site: Boston

Windows

MS SQL Server A

dwh

Production site: Tokyo

Datawarehouse(Logical Schema)

D e v

e l o p m e n

t

B o s t o

n

T o k

y o

Logical Architecture

Contexts

Physical Architecture



Our Example Infrastructure With Agents

Oracle

ACCOUNTING

SALES

MS SQL Server

db_dwh

db_purchase

Windows Windows

MS SQL Server B

purchase

Windows Linux

Oracle

ACCT

SAL

Linux

Oracle 9

ACCOUNTING

Oracle 10

SALES

Windows

MS SQL Server A

dwh

MS SQL Server

db_dwh

db_purchase

Production site: Tokyo

Production site: BostonDevelopment site:

New York




ODI still on step 1) going to production:

Reverse and enhance metadata



MS SQL Server

ODI Repository

Standard vs. Customized Reverse-

Engineering

Data Model

System tables

Model (Metadata)

Oracle Data Integrator

CustomizedReverse-engineering

JDBC Driver

StandardReverse-engineering

Delimited format

Fixed formatCOBOL copybooks

File-specificreverse-engineering



MDM Enabled: Canonical Format Design

• Use in conjunction with

packaged MDM solution

• Design and Populate CanonicalFormat

1. Use existing metadata artifacts

to design MDM application

(entities, fields, relationships)

2. Generate and maintain MasterData structure

3. Generate and deploy

transformations using metadata

artifacts

CRM

Enterprise Service Bus

SCM Legacy ERP

Master Data



1. Select the Constraints nodeunder the datastore.

2. Right-click, select Insert Key .

3. Fill in the Name.

4. Select the Key or Index Type

5. Go to the Columns tab6. Add/remove columns from the

key.

Adding primary Keys



1. Select the Constraints node underthe datastore

2. Right-click, select Insert Reference

3. Fill in the Name

4. Select the reference type• User Reference

• Complex Reference5. Select a Parent Model and Table

• Set the model and table to<undefined> to manually enter thecatalog, schema and table name.

Creating a Reference



Creating Simple or Complex Reference



1. Right-click Constraints node,selectInsert Condition

2. Fill in the Name.

3. Select ODI Condition type.

4. Edit the condition clause• Use the Expression Editor

5. Type in the error message for thecondition.

Creating a Condition





Design your transformations



What Is an Interface?

An Interface is an ODI object that defines the

rules to load one target datastore with datafrom one or more source datastores

A declarative design is used to implement thetransformations as mappings.



Using the Expression Editor

1. Click the expression

editor button ( ) in the

mapping window

2. Build your SQL

expressions from the

SQL help at the bottom,and from the Columns at

the left



The Scenario

Source Sybase

ORDERS

LINES

CORRECTIONS

File

Target Oracle

SALES

Mapping

- SALES = SUM(LINES.AMOUNT)+ CORRECTION.VALUE.- SALES_REP =ORDERS.SALES_REP_ID…

Filter

- ORDERS.STATUS=‘CLOSED’…

Join

- ORDERS.ORDER_ID =LINES.ORDER_ID…

About the interface



What Defines the Flow?

Three factors:

• Where the staging area is located• On the target, on a source or on a third server

• How mappings, filters and joins are set up• Execution location: Source, target or staging area

• Whether transformations are “active”• Choice of Knowledge Modules

• LKM: Loading Knowledge Module

• IKM: Integration Knowledge Module



The Basic Process

Source: Sybase

ORDERS

LINES

CORRECTIONSFile

Target: Oracle

SALES

Sequence of operations

C$_01

1

Extract/Join/Transform

C$_12

2

Extract/Transform

I$_SALES

33Join/Transform

55

Transform & Integrate



What Is the Staging Area?

Staging Area – A separate, dedicated area inan RDBMS where Sunopsis creates itstemporary objects and executes some of your

transformation rules.You usually put the staging area on the targetdata server.



Target (Oracle)

Staging Area

Case Study: Flow #1

Source (Sybase)

ORDERS

LINES

CORRECTIONSFile

SALES

The Staging Area on the target server

C$_0

C$_1

I$_SALES

11

22

33

55


Extract/Transform

Join/Transform




Case Study: Flow #1 in Sunopsis

Staging Area in the Target

Source Sets

Staging Area + Target



DB2 UDB, Sunopsis Engine, etc.

Staging Area

Case Study: Flow #2

Source (Sybase)

ORDERS

LINES

CORRECTIONSFile

Target (Oracle)

SALES

The Staging Area on a third server

C$_0

C$_1

I$_SALES

11

22

33

55


Extract/Transform

Join/Transform





Staging Area is the Sunopsis Memory Engine

Source Sets

Target

Staging Area



Source (Sybase)

Staging Area

Case Study: Flow #3

ORDERS

LINES

CORRECTIONSFile

Target (Oracle)

SALES

The Staging Area on the Sybase source server

C$_0

C$_1

I$_SALES

11

22

33

55


Extract/Transform

Join/Transform





Staging Area in the Source

Source Sets

Target

Staging Area



KM Description

• A Knowledge Module is

made of steps.

• Each step has a nameand a template for the

code to be generated.

• These steps are listed in

the Details tab.• The code that will be

generated by ODI will list

the same step names



Details of the Steps

• Details of the steps are

generic: the source and target

tables are not known, only the

technologies are known

• Substitution Methods are used

as placeholders for the tablenames and column names

• Parameters of the substitution

methods let you select which

tables or columns are used in

the KM



Staging areaSource TargetIntegration phaseLoading phase

Which KMs for What Flow?

• Four possible arrangements:

Multi-tech LKM Multi-tech IKM

Staging area on targetMulti-tech LKM

Single-tech IKM

Staging area on sourceMulti-tech IKM

(No LKM needed)

Source, staging area and target in same location

Single-tech IKM(No LKM needed)



Target (Oracle)

Staging Area

Flow #1

Source (Sybase)

ORDERS

LINES

CORRECTIONSFile

SALES

Using the Target as the Staging Area

C$_0

C$_1

I$_SALES

LKM_1LKM_1

LKM SQL to Oracle

LKM_2LKM_2

LKM File to Oracle (SQLLDR)

IKM_1IKM_1

IKM Oracle Incremental Update

IKM_1IKM_1

IKM Oracle Incremental Update



Sunopsis Memory Engine

Staging Area

Flow #2

Source (Sybase)

ORDERS

LINES

CORRECTIONSFile

Target (Oracle)

SALES

Using a third server as the Staging Area

C$_0

C$_1

I$_SALES

LKM_1LKM_1

LKM SQL to SQL

LKM_2LKM_2

LKM File to SQL

IKM_1IKM_1

IKM SQL to SQL Append

IKM_1IKM_1




Source (Sybase)

Staging Area

Flow #3

ORDERS

LINES

CORRECTIONSFile

Target (Oracle)

SALES

Using the Source as the Staging Area

C$_0

C$_1

I$_SALES

IKM_1IKM_1

LKM_1LKM_1

IKM_1IKM_1

IKM_1IKM_1

LKM File to SQL






Real-time enabled: Message Oriented

Integration

• Connect to Publish andSubscribe JMS MessageProviders

• Ensure messages delivery withtransaction integrity

• High-volume bulktransformations1. Design complex bulk

transformations mixing Queues,Databases and Applications

2. Use JMS Queues and topics assources or targets

JMS Provider (MOM, ESB)

CDC

Subscribe

Publish

ODI and XML files



ODI and XML files

• ODI XML driver can handle also very huge XML files, using Oracle and non Oracle RDBMS asworkspace for the schema.

• ODI and OWB can use also the native XML management functions in Oracle RDBMS (xml_load_f)

Source

Or Target

XMLAgent

Workspace for XML

(OME or RDBMS)

Parser JDBC JDBCJDBC

Loading

Driver

Synchronization

or Connection

Integration

XML driver management XML driver transforms the fileobject in a relational model





Creating packages workflow



The Package Diagram

Interface step(selected)

ODI tool step

Diagram

Toolbox for ODI

tools

Toolbar

Properties ofselected step



How to Create an ODI Tool Step

1. In the Toolbox , expand the

group containing the tool you

want to add.

2. Click the tool.3. Click the diagram.

A step named after

the tool appears.

4. Change the Step Name

in the Properties panel.

5. Set the tool’s Properties.

6. Click Apply to save.



Defining a Variable and using it in a package

• Declare Variable step type:• Set Variable step type:

• Refresh Variable step type:• Evaluate Variable step type:

C t N P d



Create a New Procedure

1. Right-click the Procedures nodeunder a project.

2. Select Insert Procedure

3. Fill in the• Name

• Description

4. Optionally, define the default:• Source Technology

• Target Technology

5. Click Apply to save.



How to Create Model, Sub-Model and Datastore Steps

1. Select the model, sub-model or datastorewhich you want to work

with in the Models view.2. Drag and drop it into the package.

3. Select the type ofoperation to perform.

4. Set the Optionsfor the chosenoperation



A Simple Package

Step on failure

Step onsuccess

First step

• This package executes two interfaces then archives some files.

• If one of the three steps fails, an email is sent to the administrator.



How to Create a Loop

• Loops need a counter variable

1. Set the counter to an initial value

2. Execute the step or steps to be repeated

3. Increment the counter

4. Evaluate counter value and loop to step 2 if the goal has beenreached

Package Loop

1. 2.

3.

4.



Executing a Package

1. Click the Execute buttonin the package window

2. Open Operator

• The package isexecuted as a session

• Each package step is a step

• Tool steps appearwith a single task

• Interface steps show eachcommand as a separate task

Documents

Setting Up a New Project