ODI Training

Copyright 2007

Oracle Data Integrator

Hands-On Workshop

Version 2.2

Nguyen Tuan Khang [email protected] (0913 008994) Senior Solutions Consultant Oracle Fusion Middleware Oracle Vietnam

Confidential

Page 2 of 79

Table of Contents

LAB 1: SOFTWARE INSTALLATION AND SETUP ................................................ 4

Download the Software (Pre-Requisite).......................................................................... 4

Install Oracle Data Integrator........................................................................................ 4

Install Oracle XE ........................................................................................................... 7

Setup the ODI Metadata Repositories............................................................................. 8

DATA INTEGRATION SCENARIO: PERMIT PROCESSING.............................. 19

LAB 2: CAPTURING METADATA........................................................................... 22

Configure Data Servers in the ODI Topology Manager................................................ 22

Define Data Models in the ODI Designer..................................................................... 27

LAB 3: SIMPLE DATA INTEGRATION .................................................................. 34

Create an Integration Project in the ODI Designer ...................................................... 34

Import the Knowledge Modules .................................................................................... 35

Create the Interface to Load the Applicant Data........................................................... 36

Define Basic Transformations ...................................................................................... 38

Define a Filter.............................................................................................................. 39

Execute the Interface.................................................................................................... 40

Monitor the Execution .................................................................................................. 41

LAB 4: HETEROGENEOUS DATA INTEGRATION AND VALIDATION .......... 45

Create the Interface to Load the Permit Data ............................................................... 45

Integrate Heterogeneous Sources ................................................................................. 47

Create Data Validation Rules....................................................................................... 50

Execute the Interface.................................................................................................... 53

Monitor the Execution .................................................................................................. 53

LAB 5: CHANGE DATA CAPTURE ......................................................................... 56

Import the Journalization Knowledge Module .............................................................. 56

Enable Journalization .................................................................................................. 56

Using Journalized Data in an Interface ........................................................................ 58

LAB 6: ORCHESTRATE INTERFACES INTO PACKAGES................................. 61

Create the Permit Integration Package......................................................................... 61

Archive the Input File................................................................................................... 63

Execute the Package..................................................................................................... 65

LAB 7: SETUP AGENT .............................................................................................. 66

Setup ODI Parameters for Agent .................................................................................. 66

Create a Physical Agent ............................................................................................... 67

Create a Logical Agent................................................................................................. 69

Execute the Agent (use command line) ......................................................................... 70

LAB 8: CREATE AND SCHEDULE AN ODI SCENARIO...................................... 71

Create an ODI Scenario............................................................................................... 71

Schedule a New Scenario within Oracle Data Integrator.............................................. 72

APPENDIX A: IMPORTING THE ODI REPOSITORY SCHEMAS...................... 77

APPENDIX B: RESETTING THE PERMIT APPLICATION SCENARIO............ 78

Confidential

Page 3 of 79

Before you begin

General notes:

1. All functions or actions performed by the student (for example, text to be entered or buttons to click) are written in bold print.

2. When you see a reference to , enter the name of the host for your assigned system (for example, edrsrsp1). For this workshop, using localhost should work

correctly.

3. All references to the installation home directory of the Oracle Data Integrator are:

. (change this as per your install directory)

4. If a Welcome page appears as the first page of a wizard, then just click Next to advance to the first step of the wizard.

5. The data used in this workshop are all fictional. Applicant names, SSNs, telephone numbers, addresses and email addresses were all randomly created for training purposes.

Confidential

Page 4 of 79

Lab 1: Software Installation and Setup

Time estimate: 30 minutes (not including software download)

Download the Software (Pre-Requisite)

1. Go to the following web site: http://www.oracle.com/technology/products/oracle-data-integrator/index.html

2. Follow the link to download: Oracle Data Integrator (10.1.3.4.0) (odi_windows_x86_101340.zip)

3. We will use the Oracle XE database for the workshop. Go to the following URL: http://www.oracle.com/technology/software/products/database/xe/index.html

4. Following the link to download: Oracle Database 10g Express Edition (Western European) (OracleXE.exe)

Install Oracle Data Integrator

To install the Oracle Data Integrator, perform the following steps:

1. Unzip the odi_windows_x86_101340.zip file to some temporary directory .

2. From the directory where you unzipped the files, navigate to the sub-directory \setup\windows and execute the setup.bat program to start the installer.

3. From the installation wizard, select the following options for each step:

Select a Product to Install: Oracle Data Integrator 10.1.3.4

Confidential

Page 5 of 79

Installation Type: Complete

Oracle Home: ODIHome

Path: c:\oracle\product\10.1.3.4\oracledi

Confidential

Page 6 of 79

4. From the Summary Screen, Click Install.

5. From the End of Installation Screen, Click Exit.

Confidential

Page 7 of 79

Install Oracle XE

The Oracle Database 10g Express Edition (Oracle XE) is an entry-level, small-footprint

database based on the Oracle Database 10g Release 2 code base that's free to develop, deploy,

and distribute; fast to download; and simple to administer.

For this workshop, we will use Oracle XE as the database engine for ODI. To install Oracle

XE, perform the following steps:

1. Double-click the OracleXE.exe that was downloaded earlier to launch the installer.

2. Click Next.

3. Review and Accept the Licensing Agreement. Click Next.

4. Accept the Default Destination Location (c:\OracleXE). Click Next.

5. Enter a password for the SYS and SYSTEM accounts. Recommended to just use oracle for the purposes of this workshop. Click Next.

6. From the Summary Screen, click Install.

7. From the InstallShield Wizard Complete window, check the Launch the Database homepage. Then click Finish.

8. You should see the Oracle XE Database Login page from your browser. Do not close the browser at this point.

Confidential

Page 8 of 79

Setup the ODI Metadata Repositories

Oracle Data Integrator uses 2 types of repositories:

The Master Repository is where all the information on the topology of resources, security and version management of projects and data models is stored. Only one

master repository is required for typical Oracle Data Integrator installations.

The Work Repository is where the information on data models, projects, and their use is stored. A work repository can be linked with only one master repository for version

management purposes.

These repositories are stored in a relational database accessible in client/server mode from the

different Oracle Data Integrator modules. For this workshop, we will use the Oracle XE

database to store these repositories.

Setup Oracle Users

First, we need to create Oracle database users for the Master and Work Repositories.

1. From the Oracle XE Database Home Page, login using the system account with the password you specified.

Note: For this workshop, you may skip the creation of the master and work repositories

by running a script that will import the repository schemas into Oracle XE. If you wish

to do so, please refer to Appendix A: Importing the ODI Repository Schemas.

Confidential

Page 9 of 79

2. From the Database Home Page, select the Administration module, and from the menu select Database Users > Create User.

3. From the Create Data User page, create a user called master and set the passwords to master. Check the DBA role under User Privileges.

Confidential

Page 10 of 79

4. From the Manage Database Users page, click the Create button to create another user called workrep1 and set the passwords to workrep1. Again, check the DBA role under

User Privileges.

5. Verify that the master and workrep1 users have been created as shown:

Confidential

Page 11 of 79

Install the Master Repository

To install the Master Repository on the Oracle XE database, perform the following steps:

1. From the Start Menu, go to Oracle > Oracle Data Integrator > Repository Management > Master Repository Creation.

2. This will launch the Master Data Creation Wizard.

3. For the Driver, click the browse button and select the Oracle JDBC Driver from the list. This should populate the text box with oracle.jdbc.driver.OracleDriver.

4. For the URL, click the browse button and select the jdbc:oracle:thin:@:: URL from the list. In the populated text box,

replace with localhost, with 1521 and sid with XE. The resulting URL

should be:

jdbc:oracle:thin:@localhost:1521:XE

5. Enter master for the username and password.

6. In the Technology list, select Oracle.

7. Click the Test Connection button to verify that the database connection information is correct.

Confidential

Page 12 of 79

8. Click the OK and OK again on the wizard to start the repository creation.

9. The Command Window will display log messages during the creation of the repository. After successful repository creation, you should see the following:

10. Click OK to exit.

Confidential

Page 13 of 79

Access the Master Repository from the Topology Manager

1. Start the Oracle Data Integrator Topology Manager. From the Start Menu, go to Oracle > Oracle Data Integrator > Topology Manager

2. The Security Repository Connection dialog box will prompt for login information. Click on the New Connection icon to configure the Topology Manager to access the newly

created Master Repository.

3. Configure the Repository Connection as follows:

For the Oracle Data Integrator Connection use the default username SUPERVISOR and password SUNOPSIS. This is the default admin user for the

master repository.

For the Database Connection use information you entered earlier for the master repository connection (Username/Password: master).

Check the Default Connection option to make this the default.

Click Test to make sure the connection is valid and Click OK.

Confidential

Page 14 of 79

4. Back in the Security Repository Connection, use the ODI Master Repository and login with the user SUPERVISOR and password SUNOPSIS. This will launch the

Oracle Data Integrator Topology Manager connected to your Master Repository in the

Oracle XE database.

Install the Work Repository

To install the Work Repository on the Oracle XE database, perform the following steps:

1. Login to the ODI Topology Manager and click the Repositories tab (the 5th tab) from the Topology Manager as shown:

2. Right-click on Work Repositories and select Insert Work Repository.

Confidential

Page 15 of 79

3. On the Data Server Definition tab, enter the name WORKREP1 and select Oracle from the Technology list. Under the connection, enter the username workrep1 and

password workrep1.

4. Click on the JDBC tab, enter oracle.jdbc.driver.OracleDriver for the JDBC Driver and enter jdbc:oracle:thin:@localhost:1521:XE for the JDBC URL. You can also

use the browse button to select the values.

5. Click Test to make sure the connection is valid and Click OK. When prompted for an Agent, select Local (No Agent).

6. Click OK.

7. The Work Repository dialog box will appear. For the ID, enter 0 and for the name, enter WORKREP1 and click OK.

Confidential

Page 16 of 79

Confidential

Page 17 of 79

Access the Work Repository from the ODI Designer

To login to the Work Repository, perform the following steps:

1. Start the Oracle Data Integrator Designer. From the Start Menu, go to Oracle > Oracle Data Integrator > Designer.

2. The Oracle Data Integrator Login box will prompt for login information. Click on the New Connection icon to configure the Designer to access the newly created Work

Repository.

3. Configure the Work Repository Connection as follows:

1. For the Oracle Data Integrator Connection use Login Name WORKREP1 with

Confidential

Page 18 of 79

the default username SUPERVISOR and password SUNOPSIS. This is the

default admin user for the master repository.

2. For the Database Connection use information you entered earlier for the master repository connection (Username/Password: master). You can also browse for the

JDBC driver and URL by clicking the browse icon on the right.

3. For the Work Repository Name, enter WORKREP1 (or you can browse for by clicking the browse icon on the right.)

4. Check the Default Connection option to make this the default.

5. Click Test to make sure the connection is valid and Click OK.

4. Back in the Oracle Data Integrator Login, select WORKREP1 and login with the user SUPERVISOR and password SUNOPSIS. This will launch the Oracle Data

Integrator Designer connected to your Work Repository.

You are now ready to use the Work Repository called WORKREP1 for the other sections of

this Workshop.

Confidential

Page 19 of 79

Data Integration Scenario: Permit Processing

Time estimate: 10 minutes

For this workshop, we will work on the integration requirements of a Permit Processing

application. The Permit Processing application is used for receiving and handling permit

requests. Aside from real-time permit processing, the Permit Processing application needs to

process permit requests in batches coming from various groups.

Multiple permit requests that include permit and applicant information are sent in a flat file.

The file typically contains more than 10000 requests including applicant information. This file

has to be extracted, transformed and loaded into the Permit Application tables. Background

check data coming from another relational database system will need to be merged to each

permit request as well.

Before the data gets loaded to the Permit Application, we need to make sure that the data is

valid (based on certain business validation rules). We also need to make sure that we only

process data that has changed from the background check system and that we archive the

permit files after processing it successfully. Lastly, there is a limited-time window involved

for processing, so we need to make sure that the data processing is done in the most efficient

manner.


Metadata Repository

BGCHECK

Permit Application

RDBMS

Load

Load

Background

Check

Data

CDC

Transformation

Permit RequestFlat File

Data Validation

Integration

----

----

----------------

----------------

----------------

----------------

----------------

----------------


Metadata Repository

BGCHECK

Permit Application

RDBMS

Load

Load

Background

Check

Data

CDC

Transformation


Data Validation

Integration

----

----

----------------

----------------

----------------

----------------

----------------

----------------

BGCHECK

Permit Application

RDBMS

Load

Load

Background

Check

Data

CDC

Transformation


Data Validation

Integration

----

----

----------------

----------------

----------------

----------------

----------------

----------------

----

----

----------------

----------------

----------------

----------------

----------------

----------------

In this workshop, you will focus on building out this data integration scenario. You will use

the Oracle Data Integrator to do the following:

1. Capture the metadata information of these systems into ODIs metadata repository:

Permit Request flat-file

Permit Application RDBMS

Background Check RDBMS

2. Filter, transform and load applicant data into the Permit Application.

3. Merge permit requests with the background check data.

4. Create data validation rules and capture validation errors.

Confidential

Page 20 of 79

5. Use change-data-capture (CDC) to process changed data from the background check system

6. Arrange these tasks in a data integration package with error notifications and file archival after successful completion.

Before continuing, please perform the following tasks:

1. Open up the ODIWorkshop-MyFiles.zip file from the workshop DVD.

2. Unzip the file to your C:\ drive. This will create a directory called C:\MyFiles. This includes the Permit Request flat file (permit_batch.csv in the CSV subdirectory) and

database scripts that will be utilized during the workshop.

3. Start a command prompt window. Go to Windows Start > Run and enter cmd.

4. From the command prompt, navigate to c:\MyFiles and enter the following commands:

sqlplus system/oracle

(Note: replace oracle with the password you specified during installation of

OracleXE.)

5. At the SQL Prompt, enter the following:

@workshopschema

This script creates the Permit Application and Background Check RDBMS schemas.

6. From the Command Prompt, enter the following:

LoadBGData

Confidential

Page 21 of 79

This command populates the background check RDBMS with data.

Confidential

Page 22 of 79

Lab 2: Capturing Metadata


Before we can implement our data integration interfaces, it is essential to capture the various

sources and targets into a unified data integration platform. This lab shows how ODI can be

used to capture the metadata information for the Permit Application and Background Check

database schemas and the Permit Request Flat File into a central metadata repository.

Configure Data Servers in the ODI Topology Manager

The ODI Topology Manager is what we use to manage the physical and logical architecture of

the data sources. Each data source is configured as a data server belonging to a specific

technology. In this workshop, we will use Oracle database tables and a flat-file as our data

sources, so we need to create the data servers for these specific technologies.

1. Start the Oracle Data Integrator Topology Manager. From the Start Menu, go to Oracle > Oracle Data Integrator > Topology Manager

2. Login using User SUPERVISOR and Password SUNOPSIS.

3. To define technologies, click the Physical Architecture tab (the 1st tab) from the Topology Manager as shown:

Confidential

Page 23 of 79

Lets first capture the Permit Applications Oracle database that will be used as the target.

4. From the list of Technologies, right-click on the Oracle technology and select Insert Data Server.

5. From the Data Server Definition Tab, enter the name PERMIT_APP. On the connection, enter the username PERMIT_APP and password PERMIT_APP.

Confidential

Page 24 of 79

6. Click on the JDBC tab. For the Driver, click the browse button and select the Oracle JDBC Driver from the list. This should populate the text box with

oracle.jdbc.driver.OracleDriver.

7. For the URL, enter jdbc:oracle:thin:@localhost:1521:XE.

8. Click OK.

9. In the Physical Schema dialog box, select PERMIT_APP for the Directory (Schema) and PERMIT_APP_WORK for the Directory (Work Schema) as shown:

The Schema is where the actual tables are located, while the Work Schema is where ODI

will create temporary objects such as staging and integration tables.

Confidential

Page 25 of 79

10. Click on the Context tab. Click the new context button and type in PERMIT_APP_TARGET as the Logical Schema.

This defines the schema in the Logical Architecture.

11. Click on OK.

Next, lets do the same steps for the Oracle database that contains background check

information. This will be used as a source.

12. Repeat steps 4-11 for the Oracle schema BGCHECK.

Name: BGCHECK

Connection User: BGCHECK

Connection Password: BGCHECK

JDBC Driver: oracle.jdbc.driver.OracleDriver

JDBC URL jdbc:oracle:thin:@localhost:1521:XE

Directory (Schema): BGCHECK

Directory (Work Schema): BGCHECK_WORK

Logical Schema BGCHECK_SRC

Important Note: The BGCHECK schema actually resides on the same database instance as

the PERMIT_APP. For this workshop, we are simulating that the BGCHECK is a totally

separate database server, which is why we are configuring it as a separate data server. In

real-life scenarios, if your database schemas belong to the same instance, the best practice is

to just use the same data server for these schemas.

Next, lets configure the Permit File source:

13. From the list of Technologies, right-click on the File and select Insert Data Server

Confidential

Page 26 of 79

14. From the Data Server Definition Tab, enter the name PERMIT_FILE.

15. Click on the JDBC tab. For the driver, click the icon and select Sunopsis File JDBC Driver. For the URL, enter jdbc:snps:dbfile, as shown below:

16. Click OK.

17. In the Physical Schema dialog box, type c:\MyFiles\CSV for both Directory (Schema) and Directory (Work Schema) as shown:

Confidential

Page 27 of 79

18. Click on the Context tab. Click the new context button and type in PERMIT_SRC as the Logical Schema.

19. Click OK.

Define Data Models in the ODI Designer

After we configure the data servers, we can now define data models in ODI Designer. The

ODI Designer is what we use to define data models and integration projects that use the data

models. Before we can create integration projects, we need to be able to capture metadata

information from sources and targets and use this for defining the data models. This is the

process called Reverse Engineering, which we perform through the ODI Designer.

1. Login to the ODI Designer. Start the Oracle Data Integrator Designer. From the Start Menu, go to Oracle > Oracle Data Integrator > Designer.

2. Login using User SUPERVISOR and Password SUNOPSIS.

Reverse Engineer the Permit Application Database

Lets start with Reverse Engineering the Permit Application target.

1. To define models, click the Models tab (the 2nd tab) from Designer as shown:

2. Click the new Model icon.

3. Enter PERMIT_APP_MODEL for the model name. Select Oracle as the Technology and PERMIT_APP_TARGET as the Logical Schema.

Confidential

Page 28 of 79

4. Click the Reverse tab and select the context called Global.

5. Click the Selective Reverse tab and check the check boxes for Selective Reverse and Objects to Reverse. Youll see the APPLICANTS, PERMITS and PERMIT_TYPES

tables from the list. Click the Reverse button to start the Reverse-Engineering Process.

Confidential

Page 29 of 79

6. You should now see the APPLICANTS, PERMITS and PERMIT_TYPES under your data models with their corresponding columns and constraints.

Confidential

Page 30 of 79

Reverse Engineer the Background Check Database

Next, lets reverse engineer the Background Check database.


2. Enter BGCHECK_MODEL for the model name. Select Oracle as the Technology and BGCHECK_SRC as the Logical Schema.

3. Repeat the Reverse-Engineering process done earlier to capture the BGCHECK table.

Reverse Engineer the Permit Request Flat File

Lastly, lets reverse engineer the Permit Request Flat File. The Permit Request Flat File,

located in c:\MyFiles\CSV\PermitBatch.csv, is a comma-delimited file that contains around

10,000 permit requests.


2. Enter PERMIT_SRC_MODEL for the model name. Select File as the Technology and PERMIT_SRC as the Logical Schema.

Confidential

Page 31 of 79

Note: If you do not see the PERMIT_SRC in the Logical Schema combo box (just see

FILE_GENERIC_SRC), check the Topology again that you put the PERMIT_SRC in the File

technology

3. Click the Reverse tab and select the context called Global.

4. Click OK. (For Files, we do not perform Selective Reverse since each file is reverse-engineered individually.)

5. From the Models window, right-click on PERMIT_SRC_MODEL and select Insert Data Store.

Confidential

Page 32 of 79

6. Enter PERMIT_CSV for the name and click the Browse icon to locate the permit_batch.csv file.

7. Click the Files tab and select Delimited for the File Format, 1 for the Heading and comma (,) for the Other Field Separator:

Confidential

Page 33 of 79

8. Click the Columns tab and click the Reverse button to generate the column names based on the file header as shown:

9. Click OK.

You have successfully captured the sources and target metadata for the data integration

scenarios in the next labs.

Confidential

Page 34 of 79

Lab 3: Simple Data Integration


Now that the Permit Application, Background Check database and Permit Request File data

models have been defined, we can use them as sources or targets in our interfaces. For this

lab, we will create a simple data integration interface that reads applicant data from the Permit

Request file, performs basic transformation, and then stores it in the Permit Applications

applicant table.

Create an Integration Project in the ODI Designer

As mentioned earlier, the ODI Designer is what we use to define data models and integration

projects that use the data models. Now that data models are defined, we can proceed in using

the data models for interfaces in data integration projects. Lets start by creating a new

Project for this workshop.

1. Login to the ODI Designer.

2. To create a project, click the Projects tab (the 1st tab) from Designer as shown:

3. Click the new Project icon.

4. Enter Integration Project 1 for the project name.

Confidential

Page 35 of 79

5. You will now see an Integration Project 1 project in your project list. Click the + sign on the left to expand and you will see a default folder called First Folder. Double-Click

the First Folder and rename this to Permit Integration as shown:

Import the Knowledge Modules

We will now import the knowledge modules that will be used for loading the Permit File

Source to the Permit Application.

We will use the following knowledge modules:

1. Loading Knowledge Module LKM File to SQL 2. Integration Knowledge Module IKM SQL Incremental Update 3. Check Knowledge Module CKM Oracle

To import the knowledge module:

1. Expand the Knowledge Modules folder under Integration Project 1 and you will see the various types of knowledge modules.

2. To Import the Loading Knowledge Module called IKM File to SQL, Right Click on the Loading (LKM) folder and select Import Knowledge Module.

3. From the Import Knowledge Modules File Import Directory, enter or browse for the following path:

Knowledge Modules are customizable integration strategy templates that can be

re-used to perform a specific set of tasks against a specific technology or set of

technologies. ODI comes out of the box with more than 100 Knowledge Modules

for the major database engines and application packages of the market.

Confidential

Page 36 of 79

c:\oracle\product\10.1.3.4\oracledi\impexp (Replace with your ODI installation home directory if you used a different one.)

4. From the list of files(s) to import, select LKM File to SQL, as shown.

5. Repeat the same steps for the other knowledge modules listed above. You should see the imported knowledge modules as shown:

Create the Interface to Load the Applicant Data

Now that the knowledge modules have been imported, we can use ODIs Declarative Design

framework in building the interface. With Declarative Design, you focus on what the rules

for your integration instead of the underlying technical aspect of the process. The imported

knowledge modules will take care of that for you.

Lets create the interface to load the applicant data from the Permit Data file:

1. Expand the Permit Integration folder, right-click on the Interfaces folder and select Insert Interface.

2. For the Interface name, enter Load Applicant Data.

Confidential

Page 37 of 79

3. Click the Diagram tab.

4. You drag and drop data stores from models as sources and the target in this diagram. To do this, select the Models tab in the left-navigation window (2

nd tab from the bottom of

the left navigation window) to display the list of Data Models.

5. Drag the PERMIT_CSV data store from the PERMIT_SRC_MODEL and drop it on the Source section.

6. Drag the APPLICANTS data store from the PERMIT_APP_MODEL and drop it on the Target section.

Confidential

Page 38 of 79

7. You will be asked if you want to perform an automatic mapping. Select Yes.

8. The mapping is based on exact field names so all the fields except the APPLICANT_ID will be automatically mapped.

(Note that you will see a warning for the Phone and SSN mappings, indicating that the

target column is smaller than the source column. This can be ignored.)

9. To manually map the APPLICANT_ID, click the empty mapping field of the APPLICANT_ID column from the Target Data Store. This will bring up the

APPLICANT_IDs mapping implementation below (which is empty). Drag the Appl_Id

column from the PERMIT_CSV source node to the mapping implementation text area as

shown:

Define Basic Transformations

Now lets add some basic transformations. Well add a transformation to the first name and

last name so that the first letter is capitalized.

Confidential

Page 39 of 79

1. Click the mapping field of the APPLICANTS FIRST column from the Target Data Store. This will bring up the mapping implementation below, which shows the

MAPPING as PER.First.

2. ODI is based on an E-L-T architecture that utilizes the database engines transformation capabilities. For this transformation, we will use Oracles INITCAP function. To do this,

we need to specify to execute the transformation in the Staging Area rather than the

source. Select the option Execute on Staging Area below the mapping implementation.

3. Enter the mapping implementation as: INITCAP(PER.First) as shown below.

Optionally, you can click on the icon on the right to launch the expression editor and

select the INITCAP function under the list of String functions.

4. Repeat the same transformation for the last name column (APPLICANTS.LAST).

Define a Filter

Now lets add a filter to get the permits with permit ID greater than 4008000.

1. From the sources window, drag the Permit_ID column outside the PERMIT_CSV node to create a filter.

2. Click on the filter icon to show the filter implementation expression below and enter the expression PER.Permit_ID > 4008000.

Confidential

Page 40 of 79

Execute the Interface

Now were ready to fire off this interface.

1. Click on the Flow tab. The diagram shows you how the execution takes place.

2. Click on the source node and the target node to see the available options provided by the knowledge modules.

Notice that only the source and target nodes are show. There is no separate

transformation engine. All the transformation and filtering is done at the target

database where the data is processed and loaded using a set-based operation rather

than processing it row-by-row in a separate engine. This approach yields much

higher performance and shows the benefits of the E-L-T Architecture.

Confidential

Page 41 of 79

3. Click on the Execute button below.

4. Click OK in the execution dialog box. You will be informed that the session has started. Click OK again.

Monitor the Execution

The ODI Operator is what we use to monitor the execution of sessions.

1. Start the ODI Operator by clicking the Operator icon from top menu in Designer.

You can also use the Start Menu: Oracle > Oracle Data Integrator > Operator.

2. Drill down on All Executions and find the Load Application Data session. Click and expand the second Load Application Data step. You will see the loading, integration and

control steps implemented by the knowledge modules executed by ODI.

Notice that all we had to for this interface was drag the source to the target and specify

the transformations and filters, which are all part of the integration rules. The

knowledge modules take care of all the underlying technical details such as creating the

staging and error tables, committing, cleaning up temporary objects, and performing

flow controls, among others. These are the benefits of Declarative Design.

Confidential

Page 42 of 79

3. On the left hand window, scroll to the right to view the various statistics for each step. Youll see that 18000 rows have been staged and 10000 rows have been inserted into the

target table.

4. Double-click Step 7: Integration Load Applicant Data - Insert flow into I$ table. This is the step that integrates the source data in the staging area.

5. Click on the Description tab and examine the SQL generated with the transformations and filters that you defined.

Confidential

Page 43 of 79

6. Lastly, go back to ODI Designer, click on the Diagram tab for the Load Applicant Data interface and right click the Target Data Store - APPLICANTS header as shown:

7. Select Data to view the data that has been inserted:

Confidential

Page 44 of 79

Congratulations! You have successfully implemented an interface that loaded 10000

applicants from a flat-file into a target database.

Confidential

Page 45 of 79

Lab 4: Heterogeneous Data Integration and Validation


We will now create the interface that populates the PERMIT table. The data will still come

from the permit file, but this time, we will need to do a heterogeneous join between the file

and the background check database table to get the results of the background check for the

applicant. We will also apply data validation rules to the data models so that only valid data

is stored in the Permit Application. Invalid data will be captured in an error table.

Create the Interface to Load the Permit Data

Lets create the interface to load the permit data from the Permit Data file:

1. Start and login to the ODI Designer, if it is not already running.

2. Expand the Permit Integration folder, right-click on the Interfaces folder and select Insert Interface.

3. For the Interface name, enter Load Permit Data.

4. Click the Diagram tab.

5. Now once again, well drag the models as sources and targets. Select the Models tab in the left-navigation window (2

nd tab from the bottom of the left navigation window) to

display the list of Data Models.

Confidential

Page 46 of 79

6. Drag the PERMIT_CSV data store from the PERMIT_SRC_MODEL and drop it on the Source section.

7. Drag the PERMITS data store from the PERMIT_APP_MODEL and drop it on the Target section.

8. You will be asked if you want to perform an automatic mapping. Select Yes.

9. Like earlier, manually map the Appl_Id column from the PERMIT_CSV source to the APPLICANT_ID target column.

10. Add the TO_DATE conversion function to the PER.Application_Date and PER.Request_Date target columns to convert the string data into a date.

Make sure you specify to execute this transformation in the Staging Area.

11. For the LOAD_DATE column, use the Oracle SYSDATE function to record the current date.

Confidential

Page 47 of 79

12. Like earlier, create a filter for Permit_ID using the expression PER.Permit_ID > 4008000.

13. You should now have something like:

Integrate Heterogeneous Sources

Now we need to get the background check data from another relational system to join to the

Permit File using the Social Security Number.

1. Since we are now using a database as a source, we need to import the appropriate knowledge module. Go to the Projects tab, expand the Knowledge Modules folder and

right-click on the Loading (LKM) folder to import the knowledge module called: LKM

SQL to Oracle.

2. Switch back to the models tab and drag the BGCHECK data store from the BGCHECK_MODEL and drop it on the Source section.

Confidential

Page 48 of 79

3. Drag the SSN column from PERMIT_CSV to the SSN column in BGCHECK to create the join.

4. Map the BGCHECKs RESULT column from the source to the PERMITs BACKGROUND_CHECK column in the target.

5. Map the BGCHECKs REASON column from the source to the PERMITs BACKGROUND_CHECK_MSG column in the target.

6. We need to only accept background check data only for the current year. Create a filter on the BGCHECKs DATE_CHECKED column using the expression:

BGCHECK.DATE_CHECKED > TO_DATE('01-JAN-07')

We also need to lookup the correct permit type codes from the Permit Applications

PERMIT_TYPES table because the permit data file only contains the permit type name. To

do this, we need to include the PERMIT_TYPES table as a source.

Confidential

Page 49 of 79

7. Drag the PERMIT_TYPES data store from the PERMIT_APP_MODEL and drop it on the Source section.

8. You will be asked if you want to perform an automatic mapping. Select Yes. This will automatically create the mapping for the PERMIT_TYPE_CODE.

9. Drag the Permit_Type column from PERMIT_CSV to the PERMIT_TYPE_NAME column in PERMIT_TYPES to create the join.

10. You should now have the following:

Note:

- Because Technology File does not support all functions like RDBMS, make sure that you set the Execute on for the following fields on Staging Area instead of

Source: APPLICATION_DATE; REQUEST_DATE; APPLICANT_ID; LOAD_DATE

- Make sure that you have used TO_DATE for APPLICATION_DATE, REQUEST_DATE - You should use: PERMIT_ID = TO_CHAR(PER.Permit_ID) if you receive an mis-datatype

error

11. Click on the Flow tab and observe how and where the transformations and merges take place:

Notice that the joins are done in the staging area. The sources get staged first and then joined in the staging area for best performance.

Confidential

Page 50 of 79

Notice that PERMIT_TYPES, although we brought it in as a source, is not staged like the other sources. ODI detects that PERMIT_TYPES belongs to the same data server

as the PERMITS target and therefore it integrates this directly in the staging area.

Notice that the filter for BGCHECK occurs on the source since BGCHECK is a relational source that can perform filtering logic. It will be more optimal to perform

filtering on the source when possible to minimize the data coming in. Since the

PERMIT_CSV is a file, it does not have an engine to perform filtering and so it

utilizes the staging area for applying the filters.

12. Click Apply to save the interface.

Create Data Validation Rules

Before we execute, lets add some validation rules on the Permit data model. These rules will

be enforced during execution time. Data that do not meet the rules will be rejected and

captured in an error table, where it can then be corrected and reprocessed.

1. Go to the Models tab and drill down on PERMITS and view its Constraints. There are 2 constraints there, the primary key and the foreign key to the APPLICANTS table. These

were captured during the reverse-engineering process.

We will now add 2 additional constraints that will contain our validation rules. Lets create a

rule for making sure that each permit application has a valid reason.

Confidential

Page 51 of 79

2. Right-click the Constraints folder and select Insert Condition.

3. Enter the following information:

Name: Reason is Required

Type: Oracle Data Integrator Constraint

Where: PERMITS.REQUEST_REASON IS NOT NULL

Optionally, use the Expression Editor to build the expression.

Hint: Use the Test Query icon to validate your expression.

Message: There has to be a reason

Note you can also create "NOT NULL" rules at the column level by checking the

"mandatory" check box. The advantage of utilizing conditions as shown above is that error

messages can be customized.

4. Click Ok.

Now lets create a rule to make sure only valid and completed background check results are

accepted.

5. Right-click the Constraints folder and select Insert Condition.

6. Enter the following information:

Name: Background Check Validation

Type: Oracle Data Integrator Constraint

Where: PERMITS.BACKGROUND_CHECK IN ('Pass','Fail')

Optionally, use the Expression Editor to build the expression.

Hint: Use the Test Query icon to validate your expression.

Message: Background check needs to be Pass or Fail

Confidential

Page 52 of 79

7. Click Ok.

8. Go back to the Load Permit Data Interface and click the Controls tab. Verify that your validation rules are listed under constraints and that they are enabled.

Confidential

Page 53 of 79

Execute the Interface

1. Click on the Execute button below.


Monitor the Execution

1. Go to the ODI Operator by clicking the Operator icon from top menu in Designer.

2. Drill down on All Executions and find the Load Permit Data session. Click and expand the second Load Permit Data step.

3. On the left hand window, scroll to the right to view the various statistics for each step. Notice that 8066 rows loaded with errors caught for the PK and CK constraints that you

created.

4. Optional: Try to locate the step that contains the SQL logic for integrating the 3 data sources.

5. Go back to ODI Designer. From the Models tab on the left, right-click on the PERMITS data store from the PERMIT_APP_MODEL and select View Data.

Confidential

Page 54 of 79

6. You seed the data that was inserted as shown below:

7. To view the captured errors, right-click on the PERMITS data store from the PERMIT_APP_MODEL and select Controls > Errors.

8. You should see the errors that have been captured and the corresponding error messages as shown below:

Confidential

Page 55 of 79

You have successfully implemented an ODI interface that performs heterogonous data

integration and validation.

Confidential

Page 56 of 79

Lab 5: Change Data Capture


ODI provides a Change Data Capture (CDC) framework to be able to identify and capture

changed data from the sources (inserted, updated, or deleted data) and use these changes in

integration processes. Changed Data Capture is performed by enabling journalization on data

models. Journalizing a model consists of setting up the infrastructure to capture the changes

(inserts, updates and deletes) made to the records of this model's data stores. Behind the

scenes, the CDC framework utilizes database triggers or log mining capabilities of the

database. These are implemented by the Journalization Knowledge Modules.

For this workshop, we will enable journalization on the RDBMS source, which is the

BGCHECK database table. This will enable us to only new or updated background check

information.

Import the Journalization Knowledge Module

1. Start and login to the ODI Designer, if it is not already running.

2. Go to the Projects tab, expand the Knowledge Modules folder and right-click on the Journalization (JKM) folder to import the knowledge module called: JKM Oracle

Simple.

Enable Journalization

To enable journalization on the BGCHECK data store, perform the following steps:

1. Go to the Models tab and double-Click the BGCHECK_MODEL and click on the Journalization tab.

2. For the Journalizing JKM, select JKM Oracle Simple.Integration Project 1

Confidential

Page 57 of 79

3. Expand the BGCHECK_MODEL, right-click on the BGCHECK data store and select Change Data Capture > Add to CDC.

4. Next, right-click on the BGCHECK data store and select Change Data Capture > Subscriber > Subscribe.

5. When asked to Select your Subscriber(s), enter SUNOPSIS and click the icon to add the ODI subscriber to the list as shown. Click OK.


7. Right-click on the BGCHECK data store and select Change Data Capture > Start

Confidential

Page 58 of 79

Journal.

8. When asked to Select your Subscriber(s), select SUNOPSIS from the list and click OK.


10. Go to the ODI Operator and verify that the 2 jobs for BGCHECK completed successfully. Optionally, examine the steps under each job. These are the journalization setup work

performed by the Journalizing Knowledge Module (JKM).

Using Journalized Data in an Interface

1. Open the Load Permit Data Interface that you create earlier and click the Diagram tab.

2. You can preview the source data that will be loaded by right clicking on the BGCHECK source node and selecting Data.

Confidential

Page 59 of 79

3. Notice the BGCHECK table contains several rows that have already been processed earlier. This is because the interface is not set up to use the journalized data.

4. To use the journalized data in the interface, click on the BGCHECK source node header. From the source configuration below the diagram, mark the Journalized Data only

option as shown below.

5. Notice that a new filter has been created on the BGCHECK source. Click the filter icon to see the filter expression.

6. Try right clicking on the BGCHECK source node again and select Data. There are no new rows to process.

7. Lets try adding some new rows. Start a command prompt and enter the following command:

sqlplus BGDEMO/BGDEMO

8. At the SQL Prompt, enter the following:

Confidential

Page 60 of 79

@c:\myfiles\newBGData.sql

9. This will insert 10 new background check results into the background check table.

10. Go back to Load Permit Data interface in ODI Designer and view the BGCHECK source data again. This time you will see 10 new rows waiting to be processed.

11. Execute the interface and verify through the ODI Operator that only10 new rows have been added and not the entire data set.

Confidential

Page 61 of 79

Lab 6: Orchestrate Interfaces into Packages


Now, let's put together these 2 interfaces in a data integration package. A package is a

sequence of steps organized in an execution diagram. The steps can be interface executions,

variable assignments, procedure calls, file operations, web service calls, OS commands,

notifications and many other utilities. Packages can then be compiled as scenarios, which can

be invoked by external applications such as web service clients.

Create the Permit Integration Package

1. Go to ODI Designer and click on the Projects tab.

2. Expand the Permit Integration folder, right-click on the Packages folder and select Insert Package.

3. For the Package name, enter Run Permit Integration.

4. Click the Diagram tab. You can drag and drop interfaces that you want to orchestrate as part of the package in this diagram.

5. Drag the Load Applicant Data interface into the diagram.

6. Drag the Load Permit Data interface into the diagram.

7. Click the OK connector icon on the package toolbar.

8. Drag the mouse from the Load Applicant Data step to the Load Permit Data step to create the connection between the 2. This will make Load Permit Data run after Load

Applicant Data completes successfully.

Confidential

Page 62 of 79

9. Click on the Internet folder from the toolbar.

10. Click on the OdiSendMail step and then click on the diagram.

11. Click on the step and rename it to Error Notification from the properties section below the diagram. Dont worry about the mail server configuration for now, well just make this

error out.

12. Click the KO connector icon on the package toolbar.

13. Drag the mouse from the Load Applicant Data step to the Error Notification step to create an exception path. This will make the Error Notification step run if the Load

Applicant Data throws an exception.

14. Create the same exception path for Load Permit Data.

Confidential

Page 63 of 79

Archive the Input File

As the last step in the package after successfully loading the permit data, we need to move the

source file into an archive directory with the current timestamp appended to the filename.

1. First, lets create a variable for the current timestamp. From your Integration Project, right click on Variables and select Insert Variable. For the name, use

CurrentTimestamp

2. Click the Refreshing tab. Select PERMIT_APP_TARGET for the Schema and enter the following SQL for the Select Query enter:

SELECT TO_CHAR(SYSDATE,'YYYYMMDDHH24MISS') FROM DUAL

Confidential

Page 64 of 79

3. Drag the variable to the package diagram and create an OK connection from Load Permit Data to CurrentTimestamp.

4. Click on the Files folder from the toolbar.

5. Click the OdiFileMove step and bring it to the diagram and create an OK connection from CurrentTimestamp.

6. Click on the step to display the properties section below the diagram. Enter the following:

Name: Archive File

Filename: C:\myfiles\CSV\permit_batch.csv

Target File: C:\myfiles\archive\permit_batch_#CurrentTimestamp.csv

Note: #CurrentTimestamp will get the value from the variable at

runtime to append it to the file name.

7. You should see the following:

Confidential

Page 65 of 79

Execute the Package

1. Execute the interface and verify through the ODI Operator it completed successfully.

2. Verify that the archived file exists in C:\myfiles\archive.

Confidential

Page 66 of 79

Lab 7: Setup Agent


Once ODI Scenarios are created they can be scheduled and orchestrated using an ODI Agent.

The ODI agent is a lightweight java process that orchestrates the execution of ODI Scenarios.

The ODI Agent can be installed as a service to allow for lights out processing of ODI

Scenarios.

Setup ODI Parameters for Agent

ODI Agent will use the repository connection information stored in the file \bin\odiparams.bat. For security, the password information is stored in this file

as an encrypted

1. Generate password to access Master Repository from command line or service

\bin\Agent encode master

where master is the password to access work repository in this lab document

When the password is generated, copy the generated encrypted password for later usage

Note that, the ODI_ENCODED_PASS is set to the encoded password for SUNOPSIS by

default; this does not need to be modified unless the SUPERVISOR password has been

changed.

2. Open the odiparams.bat with any text editor, and edit the following parameters:

set ODI_SECU_DRIVER=oracle.jdbc.driver.OracleDriver set ODI_SECU_URL=jdbc:oracle:thin:@localhost:1521:xe set ODI_SECU_USER=master set ODI_SECU_ENCODED_PASS= set ODI_SECU_WORK_REP=WORKREP1 set ODI_USER=SUPERVISOR

set ODI_ENCODED_PASS=LELKIELGLJMDLKMGHEHJDBGBGFDGGH

Note: You can copy values from Designer Connection Dialog, except the encrypted passwords

3. Save this file

With this information, the agent could be executed directly from the command line or

installed as the service to run in the background

Confidential

Page 67 of 79

Create a Physical Agent

1. Go to ODI Topology Designer and click on the Physical Architecture tab.

2. Right click Agents

3. Select Insert Agent and the following screen apprears

Set values as following

Name: Localagent

Host: Localhost

Port: 20910

4. Launch the agent: Start menu > Programs > Oracle > Oracle Data Integrator > Agent

The following screen should apprear

Confidential

Page 68 of 79

5. Click button Test and the following screen should apprear

6. Click Yes and the following screen apprears

7. Click Yes and the following screen apprears

If the test is not successful, please make sure the agent service is started or fix any

OIDPARAMS.bat errors

8. Close the Agent window that you ran in step 4)

The Physical Agent is now setup successfully

Confidential

Page 69 of 79

Create a Logical Agent

1. Click the Logical Architecture tab

2. Right click Agents and click Insert Logical Agent

3. The following screen should apprear

Set the Name: Localagent

Set the Physical Agents: Localagent

Confidential

Page 70 of 79

4. Click OK

The Physical and Logical Agent are now setup successfully. The ODI Agent can now be used

to execute ODI objects

Execute the Agent (use command line)

Make sure that you already have at least 1 scenario to execute

1. Close any Agent windows if you have executed

2. Run the Agent Scheduler in the \bin (case sensitive)

Agentscheduler name=Localagent

3. The following screen should appear

4. Check the numbers of scheduled executions is equal to your scheduled scenarios. If you do not run the lab 8 below, the number should be zero (0).

You have now successfully completed the lab, which walked through the steps that are needed

to setup the ODI Agent. The lab also walked through the steps to execute an ODI object on

the newly setup ODI Agent.

Confidential

Page 71 of 79

Lab 8: Create and Schedule an ODI Scenario


When a set of objects are complete and tested it is a good practice is to create ODI Scenarios

for each object. Once a scenario is created it cannot be edited and the code for the object is

regenerated and stored in the scenario. Once a scenario is created it can also be scheduled to

be run on virtually any interval within ODI.

Create an ODI Scenario

1. Go to ODI Designer and click on the Projects tab.

2. Expand the Permit Integration folder, right-click on the Packages folder and select the package Run Permit Integration

3. Click Generate Scenario

4. Name the scenario Run Permit Integration and set the version to 001

5. Expand the scenario

Confidential

Page 72 of 79

6. Right click the scenario to view options

The scenario has now been successfully created, you can now execute the scenario directly,

use the scenario within a package or schedule the package within ODI.

Schedule a New Scenario within Oracle Data Integrator

1. To schedule the package expand the scenario Run Permit Integration

2. Click Insert Scheduling

If an agent is not setup within the ODI Topology Manager, the following message will appear

and the scenario cannot be scheduled.

To schedule a scenario an Agent must be setup. To setup an ODI Agent, see the lab 7 Setup

an ODI Agent

Confidential

Page 73 of 79

3. Click OK to see scheduling options

Once an ODI agent is setup correctly the scenario can be scheduled. Make sure to start the

agent using (case sensitive), you must run only 1 screen

Agentscheduler.bat NAME=Localagent

to start agent as a scheduler

4. Select tab [Definition] and set the values as following

Context : Global

Agent : Localagent

Log Level : 5

Executtion : Simple

Confidential

Page 74 of 79

5. Select tab Execution Cycle to set the execution frequency and contraints when failure

Set Repetition:

Many times: Interval between Repetitions : 1 minutes

Constraints:

Number of Attempts on Failure : 10

Stop Executive After : 5 minutes

6. Click OK. An entry will now be inserted under Scheduling

7. Start the Topology Manager to review scheduling of the agent

8. Right Click and click Edit

Confidential

Page 75 of 79

9. Click Update Scheduling Information. The following screen appears

10. Click OK

11. Click Scheduling Information. The following screen appears

12. Check the execution of the job in the ODI Operator

Confidential

Page 76 of 79

13. Refresh Operator and job should be executed at appropriate time.

You have now successfully scheduled a scenario with an agent scheduler, and the scenario

was executed by the agent successfully.

Congratulations! You have completed the Oracle Data Integrator Hands-On Workshop

for Public Sector. For more learning material on ODI go to:

http://www.oracle.com/technology/products/oracle-data-integrator/index.html

Confidential

Page 77 of 79

Appendix A: Importing the ODI Repository Schemas

If you wish to import the ODI Metadata repository, or if you wish to go back to a fresh copy

of the repository, please follow the instructions below.

Note that running this script will drop the MASTER and WORKREP1 database

schemas.

1. If you have not unzipped the workshop files unzipped to c:\MyFiles, unzip the ODIWorkshop-MyFiles.zip file from the workshop DVD into your c: drive. This will

create the c:\MyFiles directory containing the workshop files.

Important: Shutdown all the ODI Applications (Topology Manager, Designer and

Operator) before you begin importing.



ImportODIRepositories.cmd

Enter the system password used to install Oracle XE.

This command drops and recreates a new version of the ODI Master and Work Repositories.

Confidential

Page 78 of 79

Appendix B: Resetting the Permit Application Scenario

If you wish to reset the Permit Application target database, the Background Check and the

Permit Request File, please follow the instructions below.

1. Make sure journalization is disabled for the BGCHECK data store.

a. Right-click on the BGCHECK data store and from the Models and select Change Data Capture > Drop Journal.

a. Right-click on the BGCHECK data store and select Change Data Capture > Remove from CDC

2. Make sure journalization is disabled for the Load Permit Data interface.

a. Open the Load Permit Data interface from the Projects and go to the diagram.

b. Click on the BGCHECK source node header. From the source configuration below the diagram, unmark the Journalized Data only option as shown below.

Confidential

Page 79 of 79



resetPermitScenario.cmd

This command truncates the Permit Application target tables, removes the new data inserted

into the Background Check database during the CDC lab, clears the ODI Error tables and

moves the Permit Request File back to its original location.

Documents

ODI Training