Data warehouse migration to oracle data integrator 11g

1 Copyright © 2011, Oracle and/or its affiliates. All rights reserved.

Insert Information Protection Policy Classification from Slide 8

Data Warehouse Migration to Oracle Data Integrator 11gMichael RaineyData Warehouse Developer, Business Information SystemsEmail: [email protected]

LinkedIn: http://www.linkedin.com/in/rainey

Web: http://www.pnnl.gov

mailto:[email protected]

http://www.linkedin.com/in/rainey

http://www.pnnl.gov/



Pacific Northwest National Laboratory• Department of Energy research laboratory,

operated by Battelle– $1.1 billion in business volume in fiscal year 2010

• Mission Outcomes– Strengthen U.S. scientific foundations for innovation– Increase U.S. energy capacity and reduce dependence on

imported oil– Prevent and counter terrorism and proliferation of weapons of

mass destruction– Reduce environmental effects of human activity and create

sustainable systems



PNNL Data Warehouse

• Industry standard hub & spoke architecture• Millions of rows processed daily

– 40+ source applications– Standardization & transformations performed within the DW– Final tables replicated to 70+ targets for operational/analytical use

• Legacy software– Visual Basic 6– SQL Server 2000 Data Transformation Services (DTS)

Legacy System



Migration to Oracle Data Integrator

• Minimize implementation risk– Data warehouse was mature, well designed– High level of integration within data transformations

• Typical migration of this size estimated to take 2-3 years– 4900+ DTS packages to migrate– Interface building & testing estimated based on SQL code

complexity and inexperience with ODI

The Challenge



Migration to ODI – Conversion EstimateThe Algorithm

Complexity Level

Point Analysis

Worst Case - Hours To Convert

Best Case - Hours To Convert

Transform Gathers and Distributions

Non-Standard

Total Processes

Worst Case - Est. lbr hours

Best Case - Est. lbr hours

Level 1 1-6 1 0.5 285 4187 0 4472 4472 2236

Level 2 7-11 3 2 137 99 0 236 708 472

Level 3 12-17 5 5 83 32 0 115 575 575

Level 4 18 - 28 10 8 56 16 35 107 1070 856

Level 5 Above 28 15 15 42 7 0 49 735 735

Total 7560 4874

Tight schedule & limited resources led to use of ODI SDK



Migration to ODI – The SolutionOverview

SQL parse toolODI SDK - Java

Build ODI objects



Migration to ODI – The SolutionMetadata Stored as SQL

SELECT ...FROM LOCATION_HIERARCHY LH LEFT OUTER JOIN(

SELECT F.FACILITY_ID, F.FACILITY_ID AS VALUE, ‘ID' AS SOURCE FROM FACILITY_S1 F WHERE F.DW_CURRENT_FLAG = 'Y'UNIONSELECT G.FACILITY_ID, G.ATTRIBUTE_VALUE AS VALUE, 'ALIAS' AS SOURCE FROM FACILITY_ATTRIBUTE G WHERE G.ATTRIBUTE_ID = 'BUILDING'

AND G.FACILITY_ID <> G.ATTRIBUTE_VALUE AND G.DW_CURRENT_FLAG = 'Y'UNIONSELECT H.FACILITY_ID, H.ATTRIBUTE_VALUE AS VALUE, ‘CORP' AS SOURCE FROM FACILITY_ATTRIBUTE H WHERE H.ATTRIBUTE_ID = ‘BUILDINGA'

AND H.FACILITY_ID <> H.ATTRIBUTE_VALUE AND H.DW_CURRENT_FLAG = 'Y'UNIONSELECT J.FACILITY_ID, J.LOCATION AS VALUE, ‘MAP' AS SOURCE FROM LOC_TO_FAC_MAPPING J WHERE J.DW_CURRENT_FLAG = 'Y'

AND J.FACILITY_ID <> J.LOCATION) LOCFIS ON LH.LOCATION = LOCFIS.VALUE LEFT OUTER JOIN( SELECT K.FACILITY_ID, K.FACILITY_ID AS VALUE, ‘ID' AS SOURCE

FROM FACILITY_S1 K WHERE K.DW_CURRENT_FLAG = 'Y'UNION

...



Migration to ODI – The SolutionInterface Built in ODI



Migration to ODI – Success!

• ODI SDK was a key component to a successful migration• Large volume of ODI objects built in a short period of time• Risk of human error minimized by automating Interface

creation • Continued use of ODI SDK for day-to-day operations



Thank you!Michael RaineyData Warehouse Developer, Business Information SystemsEmail: [email protected]

LinkedIn: http://www.linkedin.com/in/rainey

Web: http://www.pnnl.gov

mailto:[email protected]

http://www.linkedin.com/in/rainey

http://www.pnnl.gov/

Technology

Data warehouse migration to oracle data integrator 11g