Oracle OpenWorld 2010 - Consolidating Microsoft SQL Server Databases into an Oracle 11g Cluster

Consolidating Microsoft SQL Server Databases into an Oracle 11g ClusterDylan KuceraDirector – Data ArchitectureOntario Teachers’ Pension Plan

Oracle OpenWorld 2010– S313546Sunday September 19, 20103:00pm-4:00pm

Agenda

• Gaining buy-in for a DW migration• Adoption of a new DW• Oracle Migration Workbench• Views over Transparent Gateway• Stored Procedure stubs over TG• Oracle Streams

Gaining buy-in for a DW migration

• Patience isn’t just a virtue, it is a

requirement

• Start thinking about metrics before

anything else

• Tie every message about the DW to

business requirements


• Scalability• Don’t just focus on RAC• Locking Model

• Writers do not block Readers• No Lock Escalation

• Workload Balancing

• Availability• RAC• Extprocs and links to foreign DB’s run out of

process• Flashback


• Environment Capability• PL/SQL capabilities may simplify your

deployments (Language constructs,

packages)• Advanced DW features eg. Materialized Views

• Maintainability• Oracle Enterprise Manager• Less replication = lower support costs

• Fit with strategic deployment• Best, if you are lucky enough!

A new Data Warehouse – If we build it, will they come?

• Promotion• Skills Development• Mentoring• Standards• Architectural Reviews• Controls

Oracle Migration (Workbench)

• OMW can help fast-track code

migration• Result may not conform to your standards

• Can be used for tables as well, but• Do you want to adjust model?• What about the legacy DW?

OMW – Capturing a SQL Server model

OMW – Captured T-SQL Procedure

OMW – Converted PL/SQL Procedure

OMW – But what about the legacy?

• Big bang – not possible or risky• Migrating consumers will take time

• Take a staged approach• Transparent Gateway• Oracle Streams

Views employing Transparent Gateway

CREATE OR REPLACE VIEW PLAY.VALUE_TABLE_SAMPLE ASSELECT "IDENTIFIER" AS ID_, "VALUE" AS VALUE_, FILE_DATE AS FILE_DATEFROM SampleLegacyTable@MSSQL

Pitfalls of Transparent Gateway - Views

DECLARE tDate DATE := '2008-12-31';BEGIN INSERT INTO PLAY.TEMP_SAMPLE_7445 (ID_, NAME_, PREV_VALUE, CURR_VALUE, VALUE_SUPPLIER, DATE_VALUE_CHANGED) SELECT ID_, '', '', '', 'SAMPLE', MAX(FILE_DATE) FROM PLAY.VALUE_TABLE_SAMPLE WHERE FILE_DATE <= tDate GROUP BY ID_;END;

Results in ORA-03113: end-of-file on communication channel

Alert log says ORA-07445: exception encountered: core dump [intel_fast_memcpy.A()+18] [ACCESS_VIOLATION] [ADDR:0x115354414B] [PC:0x52A9DFE] [UNABLE_TO_READ] []

Fixed in 11.1.0.6 Patch 10 and 11.1.0.7 Patch 7

CREATE PROCEDURE dbo.GetNHLTeamStats

@inDate DATETIME, @inTeam VARCHAR(8)

AS

SELECT SWEATER_NO, NAME, POINTS, FACE_OFF_PCTFROM NorthWind..NHL_PLAYER_STATSWHERE DATE = @inDate AND TEAM = @inTeam

Procs employing Transparent Gateway

CREATE OR REPLACE PROCEDURE PLAY.RPT_NHL_TEAM_STAT ( inDate DATE, inTeam VARCHAR2, RC1 IN OUT SYS_REFCURSOR) IS

tRC1_MS SYS_REFCURSOR;tSweater_No NUMBER;tName VARCHAR2(8);tPoints NUMBER;tFace_Off_Pct NUMBER;

BEGIN DELETE FROM PLAY.TEMP_NHL_TEAM_STAT; dbo.GetNHLTeamStats@MSSQL(inDate, inTeam, tRC1_MS);

LOOP FETCH tRC1_MS INTO tSweater_No, tName, tPoints, tFace_Off_Pct;

EXIT WHEN tRC1_MS%NOTFOUND; BEGIN INSERT INTO PLAY.TEMP_NHL_TEAM_STAT (SWEATER_NO, NAME_, POINTS, FACE_OFF_PCT) VALUES(tSweater_No, tName, tPoints, tFace_Off_Pct); END; END LOOP; CLOSE tRC1_MS; OPEN RC1 FOR SELECT SWEATER_NO, NAME_, POINTS, FACE_OFF_PCT FROM PLAY.TEMP_NHL_TEAM_STAT ORDER BY SWEATER_NO;

END RPT_NHL_TEAM_STAT ;

Procs employing Transparent Gateway

RECIPE:

1) Declare Variables for all MSSQL Result set columns

2) Call MSSQL Procedure

3) Fetch Result one row at a time

4) Insert row to Oracle Temporary Table

5) Open Ref Cursor result set

{CALL PLAY.RPT_NHL_TEAM_STATS ('2010-08-31', 'SJS')}

Results in ORA-06504: PL/SQL: Return types of Result Set variables or query do not match

Fixed in 11.1.0.7 Patch 7

Pitfalls of Transparent Gateway – Procs and Result Sets

Oracle Streams as an enabler of migration

Oracle Streams

ETL

Heterogeneous Streams – General Architecture

Diagram Adapted from “Oracle Database 11g: Oracle Streams Replication, An Oracle White Paper, July 2007”

Heterogeneous Streams – Example

Legacy Microsoft SQL Server Data Warehouse:

TABLE NorthWind.dbo.NHL_PLAYER_STATS ( DATE DATETIME, TEAM VARCHAR(8), SWEATER_NO INT, NAME VARCHAR(128), BIRTH_DATE DATETIME, POINTS INT, FACE_OFF_PCT FLOAT )

New Oracle Data Warehouse:TABLE PLAY. NHL_PLAYER_STAT ( DATE_ DATE, TEAM VARCHAR2(8), SWEATER_NO NUMBER, NAME_ VARCHAR2(128), BIRTH_DATE DATE, POINTS NUMBER, FACE_OFF_PCT NUMBER );

BEGIN DBMS_APPLY_ADM.CREATE_APPLY( queue_name => 'SAMPLE_STREAM_Q', apply_name => 'SAMPLE_APPLY_NORTHWIND', apply_captured => TRUE, apply_database_link => 'MSSQL_STREAMS_NORTHWIND'); END; /BEGINDBMS_STREAMS_ADM.ADD_TABLE_RULES( table_name => 'PLAY.NHL_PLAYER_STAT', streams_type => 'APPLY', streams_name => 'SAMPLE_APPLY_NORTHWIND', queue_name => 'SAMPLE_STREAM_Q', include_dml => true, include_ddl => false);END;/

Heterogeneous Streams – Heterogeneous Apply

Heterogeneous Streams – Heterogeneous Apply Transforms

BEGINDBMS_STREAMS_ADM.RENAME_TABLE( rule_name => 'NHL_PLAYER_STAT2283', from_table_name => 'PLAY.NHL_PLAYER_STAT', to_table_name => '"dbo".NHL_PLAYER_STATS', step_number => 0, operation =>'ADD');END;/BEGIN DBMS_STREAMS_ADM.RENAME_COLUMN( rule_name => 'NHL_PLAYER_STAT2283', table_name => 'PLAY.NHL_PLAYER_STAT', from_column_name => '"DATE_"', to_column_name => '"DATE"', value_type => '*', step_number => 0, operation => 'ADD');END;/

Heterogeneous Streams – Before inserts to Captured Oracle table

Heterogeneous Streams – Insert some rows into Captured Oracle table

SQL> INSERT INTO PLAY.NHL_PLAYER_STAT (DATE_, TEAM, SWEATER_NO, NAME_, BIRTH_DATE, POINTS, FACE_OFF_PCT) 2 VALUES ('2010-08-31', 'VAN', 33, 'Henrik Sedin', '1980-09-26', 112, 49.5); 1 row insertedSQL> INSERT INTO PLAY.NHL_PLAYER_STAT (DATE_, TEAM, SWEATER_NO, NAME_, BIRTH_DATE, POINTS, FACE_OFF_PCT) 2 VALUES ('2010-08-31', 'PIT', 87, 'Sidney Crosby', '1987-08-07', 109, 55.9); 1 row insertedSQL> INSERT INTO PLAY.NHL_PLAYER_STAT (DATE_, TEAM, SWEATER_NO, NAME_, BIRTH_DATE, POINTS, FACE_OFF_PCT) 2 VALUES ('2010-08-31', 'WSH', 8, 'Alex Ovechkin', '1985-09-17', 109, 45.4); 1 row insertedSQL> INSERT INTO PLAY.NHL_PLAYER_STAT (DATE_, TEAM, SWEATER_NO, NAME_, BIRTH_DATE, POINTS, FACE_OFF_PCT) 2 VALUES ('2010-08-31', 'SJS', 19, 'Joe Thornton', '1979-07-02', 89, 53.9); 1 row insertedSQL> INSERT INTO PLAY.NHL_PLAYER_STAT (DATE_, TEAM, SWEATER_NO, NAME_, BIRTH_DATE, POINTS, FACE_OFF_PCT) 2 VALUES ('2010-08-31', 'OTT', 11, 'Daniel Alfredsson', '1972-12-11', 71, 35.0); 1 row insertedSQL> INSERT INTO PLAY.NHL_PLAYER_STAT (DATE_, TEAM, SWEATER_NO, NAME_, BIRTH_DATE, POINTS, FACE_OFF_PCT) 2 VALUES ('2010-08-31', 'CGY', 12, 'Jarome Iginla', '1977-07-01', 69, 47.0 ); 1 row insertedSQL> INSERT INTO PLAY.NHL_PLAYER_STAT (DATE_, TEAM, SWEATER_NO, NAME_, BIRTH_DATE, POINTS, FACE_OFF_PCT) 2 VALUES ('2010-08-31', 'TOR', 15, 'Tomas Kaberle', '1978-03-02', 49, NULL); 1 row inserted SQL> COMMIT; Commit complete

Heterogeneous Streams – After inserts to Captured Oracle table

Streams – NULL vs. Empty String

• Oracle treats empty string = NULL• Convert empty strings on SQL Server

target to NULL before streaming

• Could require business analysis and

application change

Heterogeneous Streams – Syncing tables containing Floats



Wrap-up

• Gaining buy-in for a DW migration• Oracle Migration Workbench• Views over Transparent Gateway• Stored Procedure stubs over TG• Oracle Streams

• Standards, metadata, education,

mentoring

Questions?

Dylan KuceraDirector – Data ArchitectureOntario Teachers’ Pension Plan

Oracle OpenWorld 2010– S313546Sunday September 19, 20103:00pm-4:00pm

Thank You! - I value your feedback!

Documents

Oracle OpenWorld 2010 - Consolidating Microsoft SQL Server Databases into an Oracle 11g Cluster