Multiplicity – Progress Data Replication Methodologies

Webster says…• mul·ti·plic·i·ty• noun \ˌməl-tə-ˈpli-sə-tē\• plural mul·ti·plic·i·ties• Definition of MULTIPLICITY• 1• a : the quality or state of being multiple or various • b : the number of components in a system (as a multiplet or a group of

energy levels) • 2• : a great number • 3• : the number of times a root of an equation or zero of a function occurs

when there is more than one root or zero <the multiplicity of x = 2 for the equation (x − 2)3 = 0 is 3>

What we want out of Replication

Michael Keaton 1996

What we end up with…

Current Choices• Choice 1: ODBC Direct Reporting - Use ODBC and run

reports on live OLTP database • Choice 2: Nightly do a complete rebuild - Dump

entire db and build a new data warehouse every night.

• Choice 3: Nightly Incremental Load - Only dump the changed records and add to the data warehouse nightly

• Choice 4: ODBC Incremental Data Pull• Choice 5: Build your own replication engine• Choice 6: Pro2Enterprise (Progress, MS SQL, Oracle)

Choice 1: ODBC Direct ReportingUse ODBC and run reports on live OLTP database

Pros:• No extra database needed• Live reportingCons:• ODBC setup• ODBC Security• ODBC Performance• Impact of live reporting on OLTP Database• This is not a data warehouse.

Choice 2: Nightly Complete RebuildDump entire db and build a new data warehouse every night.

Pros:• Least Complicated• Scriptable• Target can be a data warehouse (you can transform)• You can use a different databaseCons:• Huge load on OLTP server and network• Can it complete in time???• Requires a lot of storage space• Accessibility of data during dump and load (dirty data)• Code Page Compatibilities (ASCII)• Both source and target databases unavailable.

Choice 3: Nightly Incremental LoadOnly dump the changed records and add to the data warehouse nightly

Pros:• Less Space/Network/CPU required• Scriptable• Target can be a data warehouse• Less downtime of Data Warehouse• You can use a different databaseCons:• Identifying changed/added records? (deltas)• How do you know what records were deleted??• Can it complete in time???

Choice 4: ODBC Incremental Data Pull

Pros:• Target can be a data warehouse (you can transform)• You can use a different databaseCons:• More complex to configure and code• ODBC setup• ODBC Security• ODBC Performance• Data Type Incompatibilities• Identifying changed/added records? (deltas)• How do you know what records were deleted??• Can it complete in time???

Choice 5: Build own replication engine

Pros:• Infinitely customizable• OpenEdge supports ABL and Java Replication triggers• Target can be a data warehouse Cons:• Very complicated to code• Limited development time and resource• Unproven stability and accuracy• Risk of loss of tribal knowledge• Unsupported

Being Aware of Technology.

Choice 6: Pro2Enterprise Pro2SQL, Pro2Oracle, Pro2Pro

Pros:• Proven application• Stable and tested in multiple environments• Near Real-time data replication and availability• Full end user administration suite• Continual product improvements• Maintenance and ongoing support• Large install base• Automated source to target schema creation and mapping.• Multithreaded, Native Progress Technology• Customizable thru Progress 4GL transformation layer• WAN and LAN supportCons:• Upfront investment• Ongoing Maintenance costs• Requires Progress DataServer Components• Only support replication to Progress, MS SQL, and Oracle (No MySQL)• Target cannot be used as part of disaster recovery solution• Only supported on Unix, Linux, and Windows

Pro2Enterprice Product Suite

• Near real-time replication from OpenEdge to MS-SQL, Oracle, and Progress

• Supports Progress 8.3 – OE10.3• Mature Product

– Version 1 released in 2006– Version 4 released May 2011– Version 5 to be released Summer 2013

• Large install base– > 350-400 implementations– >150+ customers world wide

Pro2 Features

• Extremely low-impact, scalable architecture– Replication speeds of 50,000-100,000 rows/minute are

typical (LAN environment)– Replication speeds of 20,000-50,000 rows/minute are

typical (WAN environment)

• Captures both ABL and SQL changes• Customizable replication

– by table/row/column– Custom transformations using ABL supported– Datatype and name transformation

Pro2 Features

• Management of Datatype issues– MS-SQL Dates– Column Widths– Null (unknown value) issues

• Administrative Console• Monitoring

– Email/SMS alerting

Typical Pro2 Uses

• Data Warehousing• Reporting Databases• B2B/B2C Support• Consolidation of divisions• Roll-up of financial data• End-of-Month reporting• etc

Pro2 Release 4

WAN replication module availableLeverages OE compression features

Multiple replication threadsMultiple replication targetsHttp monitorUsability enhancementsTransaction batching for performance improvement with Oracle

Pro2 OverviewPro2 Overview

Schema Holder

Schema Holder

Progress DataServers

Replication(Progress)

6 Tables

Replication Layer

ORACLE

SQL Server

ORACLE

SQL Server

StandardProgress DB’s

A Pro2SQL SampleA Pro2SQL Sample

MFG/Pro(Progress)773 Tables

Sports2000(Progress)

42 tables

Replication(Progress)

6 Tables

MFGHolder(Progress)?? Tables

SportsHolder(Progress)?? Tables

MFGSchema(MSSQL)

773 Tables

MySport(MSSQL)42 Tables

Progress DataServer

StandardProgress DB’s

MSSQL 2005 DB’s

ReplicationLayer

Pro2 Http Monitor

Pro2 Http Properties

Pro2 Http Tools

Pro2 Http DB Mapping

Questions?