built by

Bill HaydukCEO/President

RTTS

Data Migrations: How to avoid the #1 project pitfall

Christopher ThompsonSenior Domain Expert

RTTS

Laura PoggiMarketing Manager

RTTS

Webinar

Today’s Agenda

• Data Migration o Business Reasons o Riskso Costso Scary Metricso Common Defects

• #1 Project Pitfall

• QuerySurge Solution

• Demo

built by

AGENDA

Data Migration

o Business Reasons o Riskso Costso Scary Metricso Common Defects

#1 Project Pitfall

QuerySurge Solution

Demo

About

RTTS is the leading provider of software quality for critical business systems

FACTSFounded: 1996

Primary Focus:

consulting services, software

Locations: New York, Atlanta, Philly, Phoenix

Geographic region:North America

Customer profile:Fortune 1000, > 600 clients

Software:

QuerySurge

built by

80 percent of data migration projects fail to meet expectations, running over time and budget.

- Gartner

83 percent of migrations experience problems- IBM survey

80% of organizations… will underestimate the costs related to the data acquisition tasks by an average of 50 percent

- Gartner

Data Migrations

Data Migration – Top Reasons

built by

Top reasons for Data Migration• Decommissioning legacy systems

• Adding new line of business

• New deployment (i.e. ERP or CRM system)

• Consolidation of systems

• Relocation

• the average cost for a Data Migration = $875,000

• 34% of migrations have missing or lost data

• 38% have some form of data corruption

• 64% of migration projects have unexpected outage/downtime

• typical cost of downtime:

o $6.5 million/hour - Brokerage industry

o $2.8 million/hour - Energy Industry

• 72% of organizations deferred because data migration is too risky

Data Migration – Scary Metrics

built by

built by

• Unexpected downtime

• Budget overruns

• Customer or Brand Impact

• Data corruption

• Application performance issues

• Data loss

Data Migration – Key Risks

Key Risks

• Remediation activities

• Server outage

• Application outage

• Labor costs

• Software Tools

• System Maintenance

Data Migration – Cost Factors

built by

Key Cost Factors

Data Migration – Key Pitfalls

• Failing to engage the lines of business at the outset

• Absence of data governance policies and organizational structure

• Poor data quality in a legacy system

• Neglecting to validate and redefine business rules

• Failure to validate and test the data-migration process

Which is the #1 Pitfall?built by

Key Pitfalls of Migration Projects

• Failure to validate and test the data-migration process

"You really need to make sure that you're validating and testing throughout the process”.

“Unfortunately companies often don't spend enough time aligning the data testing, validation, and migration cycles to the project timeline”. - InformationWeek

Question:

How are you going to test the data?

Failure to validate and test the process

Data Migration – Popular Test Strategy

Post-Migration Testing• Row Counts• Comparing migrated records to

source using sampling

Question:

Is there a better way?

built by

12

Our Solution: Automated Testing of the

Migration Process

built by

QuerySurge is the

premier test tool built

to automate Data Testing

and the ETL Testing Process

What is QuerySurge?

built by

What does

QuerySurge ™do?

built by

QuerySurge finds bad data

• Most firms test < 1% of their data

• Databases have at best, untested data & at worst, bad data

• Bad data will cost companies between $8 - $100 million annually (Gartner)

• QuerySurge tests up to 100% of your data quickly & finds bad data

QuerySurge™ Architecture

built by

Target

Sources

Target Database

built by

Data Migration: Data Flow

Source Data Migration Process

ETL & EL

SQL

SQL

SQL

SQL

SQL

SQL

built by

Design Library Create Query Pairs (source & target queries)

17

QuerySurge™ Modules

Scheduling Build groups of Query Pairs Schedule Test Runs

Deep-Dive Reporting Examine and automatically

email test results

Run Dashboard View real-time execution Analyze real-time results

QuerySurge™ Modules

built by

automates the testing effort the kickoff, the tests, the comparison, emailing the results

speeds up testing up to 1,000 times faster than manual testing

tests across different platformsany JDBC-compliant db, DWH, DMart, flat file, XML, Hadoop

The QuerySurge solution…

built by

verifies more data verifies upwards of 100% of all data quickly

Automated Data Migration Strategy

built by

• Automated Row Counts• Automated Table Compares (no transformations)

#1: Automated testing with Data Wizards (no programming)

#2: Automated testing with custom QueryPairs (custom SQL)

• All tables with transformations are compared • Upwards of 100% coverage of all data quickly

SQL

SQLSQL

SQL

SQL

SQL SQL

SQL

2 Steps:

21

Fast and Easy. No programming needed.

built by

• Perfect for data migrations & upgrades

• Great for verifying that all rows have come over without data issues

• Superb at verifying table-to-table transfers

• Ability to graphically create SQL stubs for advanced scripting

Step #1:

Uses: Verify that the amount of rows match

the amount that were expected Great for smoke testing

built by

Result:

Quickly compares rows of all tables

SQL Skill set / Value added:

beginner: no coding needed

experienced user: saves time

Uses: - Verifies tables from source

and target match

built by

Result: Can perform 100s of tables in

a few minutes.

SQL Skill set / Value added: beginner: no coding neededexperienced user: saves

time

built by

Custom QueryPairs • Write Query against the source and against

the target• QuerySurge quickly compares tables after

transformation

Step #2:

QuerySurge Value-Add

QuerySurge provides value by either:

in testing data coverage from < 1% to upwards of 100%

in testing time by as much as 1,000 x 

combination of in test coverage while in testing time

25built by

Return on Investment (ROI)

redeployment of head count because of an increase in coverage and decrease in need for testers

an increase in better data due to shorter / more thorough testing cycle, possibly saving $ millions by preventing bad data. 

26built by

Ensuring Data Warehouse Quality

Demonstration

Christopher ThompsonSenior Domain Expert

RTTS

28

• SAP Data Migration: Turning Risk into Opportunity, Jerome Delune, EIM Global Practice Director, SAP Consulting

• Data Migration Best Practices and Nondisruptive Migration Service Capability for Enterprise Storage, Hitachi Data Systems, February 2012

• Risks and Challenges in Data Migrations and Conversions, Gartner, February 2009

• Data Migration – Key Considerations, Glen Bell, SearchDatamanagement.com, May, 2011

• Data Migration in the Global 2000, Bloor Research, September 2007

• Zero-Downtime Upgrades and Migrations with Oracle GoldenGate 11g, Oracle, 2009

• 10 Big Data Migration Mistakes, InformationWeek, August 9, 2012

• How to Implement an Effective Data Migration Testing Strategy, DataMigration Pro magazine, David Katzoff, Managing Director of Product Development, Valiance Partners

• The Hidden Costs of Data Migration, IBM, 2007

References

built by