Data Virtualization

Composite Data Integration

© 2008 Composite Software

Peter Tran

April 15, 2008

Data Virtualization

2© 2008 Composite Software, Inc.

Data Virtualization – The Unstoppable Force

The Virtualization TrendWhat is Data VirtualizationHow it WorksWhy VirtualizeWhen to Virtualize, When to ConsolidateData Virtualization and SOAGetting StartedQuestions and Answers


Next-Generation Virtualization

Disk 1 Disk 2 Disk 3 Disk 4

Application 1 Application 2 Server 1 Server 2

Data Warehouses

RDBMS Flat FilesPackaged Apps

Web Services

Storage Virtualization Abstracts logical storage from physical storage

Data Virtualization Abstracts data location and complexity

Abstracts logical apps & servers from physical apps & servers

Application / Server Virtualization


Common Thread – Real Value to Business & IT

Overcome (hide) ComplexityHardwareSoftware

Improve AgilityNew solutions Existing solutions

Reduce CostsOperatingNew development

Storage Virtualization

Data Virtualization

Application / Server Virtualization





Data Warehouses


Web Services

Data Virtualization - The New Frontier of Data Integration

ExistingData

ConstantChange

Siloed&

Complex

DataIntegrationChallenge

BusinessSolutions

ReportingDashboards Applications

Code within the Application? File Extracts / Database Replications?ETL with Warehouses, Marts, Stores?

Virtual Data Federation / EII?

DIOptions


But This Sounds A Lot Like…..

Enterprise Information Integration (EII)High Performance Query / Data Federation SOA Data Services Information As A Service (IaaS)…

Terms slightly different, but core concepts are the same.

Virtualize data silos Abstract away complexityFederate heterogeneous data


Customers

Orders

Invoices

Trades

Positions

Vendors

Employees

POs

GLs

Data Warehouses


Web Services

Three Core Concepts At A Glance

ExistingData

Real-timeInformation

Siloed&

Complex

VirtualizeVirtualDataLayer

Abstract

BusinessSolutions


Federate


Data Virtualization – From an Analyst’s Perpsective

Challenge:“Enterprises are facing growing challenges in using disparate sources of data managed by different applications, including problems with integration, security, performance, availability, and quality.

Requirements:Business users need fast, real-time, and reliable information to make business decisions, while IT wants to lower costs, minimize complexity, and improve operational efficiency.

Solution: New technology is emerging that Forrester has dubbed ‘information fabric,’ defined as a virtualized data layer that integrates heterogeneous data and content repositories in real time.”

"Information Fabric 2.0: Enterprise Information Virtualization Gets Real" by Noel Yuhanna, Mike Gilpin

Publication Date: April 9, 2007 © 2007 Forrester Research Inc





Project Level Data Virtualization – Global Pharmaceutical

Research Scientist Workbench

Situation and Requirements:Researchers require on-demand access from many sources to speed drug discovery projectsRapid applications development approachSOA compliance

Solution: • Composite combines disparate data for

Research Scientist WorkbenchResults:

50% reduction in development time for complex data integration projectsSOA-compliance, with data services in the format required by data usersData quality improved through rapid prototypingReduced cost of operational infrastructure

XML data

Packaged Apps

HTTP files

Historical Data

Virtual Data Mart

Research Scientists WorkbenchDrug Discovery

SOA to the Rescue, When Drug Discovery Needs Data Fast! Dan Eng, Pfizer, SOA World Magazine

SOA to the Rescue, When Drug Discovery Needs Data Fast! Dan Eng, Pfizer, SOA World Magazine


Project Level Data Virtualization – Top 5 Investment Bank

Mortgage Loan Management

Situation and Requirements:10,000 Loans in process Management seeks visibility and controlOperations needs to complete funding stepsFive source systems, cannot replicate data

Solution:Virtual data mart accessing and delivering data via Web data servicesProvide real-time data to Management Portal and Workflow System

Results:Management get visibility on funding needs, exposure, operating metricsLoan officers process loans fasterSensitive data is protected

ManagementReporting

Virtual Data Mart

Loan ProcessingWorkflow Mgmt

LoanOrigination

RiskAnalysis

Loan Approval

LoanFunding


Enterprise Data Virtualization - Global Pharmaceutical

Situation and Requirements:More up-to-date information needed for

HR Operations ManagementSales Operations ManagementDrug Discovery

Share data across the companyStructured and semi-structured dataMultiple consuming front-ends

Solution: Virtual data layer using Composite Information ServerReusable views and data services for sourcing and delivering data

Results:Simplified data sharing across the companyBuild business applications faster to keep up with changing businessCost savings in reusing data services

SalesforceCompensation

GlobalPharma

Prescription

GeographicCore

Master

Call ClusterPlanningFinancial

Warehouse

HR OperationsManagement

Sales OperationsManagement

Drug Discovery

Virtual Data Layer


Enterprise Data Virtualization - Top 5 Investment Bank

Situation & Requirements:Up to the minute information needed for Risk management, trade order management, and Debt/equity market researchShare data across the bankSimplified access to master & reference data stored in Teradata and other sourcesServe data to .NET & other front ends

Solution: Implement data virtualization layer using Composite Information Server Virtualize data for delivery in format needed by consuming applications25 Applications, 200+ Sources

Results:Simplified data sharing across bankFaster access to reference & transaction dataCost savings in reusing data services

Virtual Data Layer

Sybase -Product data

SQL Server -Coverage

Holdings &Offerings

Teradata -Cache

Trade OrderManagement

RiskManagement

Market Research


Data Warehouses


Web Services

Design Time - Easy to Use Modeling & Code Generation

ExistingData

DesignTime

Functions

VirtualData Layer

BusinessSolutions


• Drag and Drop Designer• Metadata Integration & Mgmt • Standard Drivers • Pre-built Enterprise Apps

Data Services

• Automated CodeGeneration

• XML/SQLTransformations

• Scripting Optional


Data Warehouses


Web Services

Run Time - High Performance Query & Reliable Delivery

ExistingData

Run TimeFunctions

VirtualData Layer

BusinessSolutions


• Access / Connectivity• Query Optimization• Federation• Abstraction• Delivery via Service or View

• Data Security• Caching• Clustering• Flexible Management• Technology Independent





Virtualize Data To Increase Agility

Deliver Solutions2-4x Faster

Time AdvantagesConfigure, not codeNo need to build & test martQuicker iterationsLeverage abstractionsReuse prior work

Virtual DataMarts

File Extracts

Time toSolution

The VirtualDifference


Virtualize Data To Reduce Costs

Reduce DI Costs by 25-50%

Marts, Extracts,

Other Costs

VirtualData Integration

The VirtualDifference

Cost AdvantagesFewer data martsFewer data extractsBetter data controlBetter leverage of existing resourcesMore flexible architecture

$

Time





Virtualize, Consolidate, Replicate?

Need to considerPeople…..Skills, ICC, Politics…Process….SDLC, Architecture Review, PPM…Technology….Virtualize, Consolidate, Replicate, or maybe even a Combination…

Factor-based AnalysisBusiness Consideration…Data Source Considerations…Data Consumer Considerations


A Simple Tool to Help You Decide


Combined Approaches Sometimes Provide Best of Both Worlds

European TelecomNear real-time sales & customer management reporting in high transaction volume environment

Consolidation / ETL Reporting data mart refreshed hourlyReduce reporting load on operational (Siebel CRM) system

Data Virtualization / EIIOne face to the reporting portal system (Cognos)Combine real-time and historical data

Data Virtualization with EII

Data Consolidationwith ETL

CRMApplications

ReportingData Mart

NetworkApplications

FinancialWarehouse

Reporting Portal





Business Solutions

Transaction Services

App Server

1) Data services stand alone2) Transaction services stand

alone3) Data services integrated

with transaction services4) Data services with

messaging using an ESB5) Data Services with a BPM6) Data services WSDL stored

in a UDDI registryData Services (Info Server)Data Services

Info ServerDATADATA

DATADATA

DATADATA

DATADATA

DATADATA DATA

DATA

SOA Components Working Together

UDDIRegistry

“Over 60% of SOA services will beData Services”

Source: IDC

“Over 60% of SOA services will beData Services”

Source: IDC Data Warehouses


Web Services

ESB – Messaging/Comm System

BPM





Next Steps - Guidance

Keep an open mind Data Virtualization is a NOW a proven option with significant time and cost saving benefits It’s not always either/or…sometimes it’s both

Leverage what you already knowModeling and SQL are common denominatorsIntegration Competency Centers

Get startedPick a project and a SWAT teamSelf-fund expansion with savings

Take advantage of enterprise synergiesBusiness Intelligence, Portals, SOA, CDI-MDM…Success of the “Other” virtualizations


Thank You

Peter TranComposite Software

[email protected]