InfoSphere: Leading from the Front - Accelerating Data Integration through Metadata

Leading from the FrontAccelerating Data Integration through MetadataScott AbbottCertified IT Architect, InfoSphere Software

Make change work for youIBM Insight Forum 09®

C t tContext

22IBM Insight Forum 09®

Make change work for you

Are you e youconstantly disappointeddisappointed by your Data I t tiIntegration projects?

Often it’s because we rush in without thinkingthinking what we are d idoing

Typical Data Integration Project REPORTSTypical Data Integration ProjectOLAP

4WAREHOUSE

DATA INTEGRATIONDATAMARTS

LEGACYSOURCES

REFERENCE DATA “if we build it they will come”

MASTER DATA

“The custom data model”

“of course our data is good”

“we’ll work it out in the testing”

Th I f S h S ft E l tiThe InfoSphere Software Evolution

Ch D tDataMirror

LAS Global Name

Change Data Capture

DWLOperational Master Data

Management

Global Name Enrichment

Unicorn

TrigoSRD

Ascential

Transformation, Cleansing, Profiling and metadata integration

Entity Resolution and

Metadata Management

Product Information Management

Entity Resolution and Analysis

InfoSphere Information Server

4WAREHOUSE

LEGACYSOURCES

REFERENCE DATA

MASTER DATA

METADATA

Pitf ll #1Pitfall #1

“Th C t M d l”“The Custom Model”

DI Pitfall #1WAREHOUSE

“The custom data model”

“ h k i d

data model

NZ Customer Experience

“who knows our industry better than us”

• Project duration 24-36 mths• Model never fully deployed• Complex ETL feeds d t bili d ti BI t“it will only take a couple of

months”

destabilized entire BI system• Users bypass to get required information

DI Pitfall #1 AcceleratorAccelerator

80:20 rule (20% customization)80:20 rule (20% customization) Months not years

Fully attributed data models across six industries

C l t b i t l t fComplete business templates for industry KPIs

Ke accelerators for migration &Key accelerators for migration & integration projects

A t l ti t l t ithiAct as acceleration templates within Information Server & Cognos 8 BI

4WAREHOUSE

industry

LEGACYSOURCES

industry models

REFERENCE DATA

MASTER DATA

Target state

METADATA

if b ild itif we build itthey will come..y

14DI Pitfall #2

REPORTS

“if we build it they will come”

“it is what the business

they will come”

asked for” • Multiple examples of BI solutions not meeting initial business driversU i BI“the users will understand

the new system”• Users perceive new BI initiatives as burdens rather than assets

15Missing the PointC t Chi WhiCorporate Chinese Whispers

Identify High Value Customers to support

Call Centre & Web

Monthly Report on Customers Revenue

breakdownCall Centre & Web Personalization

breakdown

DBAsArchitectsSubject Matter Experts

Business Users

DevelopersDataAnalysts

IBM Insight Forum 09®

16Bridging the Gapl ti th t th ldrelating the new to the old

“item”

“component” “part”?

U d t di Y D tUnderstanding Your Data

InfoSphere Business Glossary

Captures Business TaxonomiesCaptures and defines shared searchable business glossaryAssigns stewardship to key business termsLinks business terms to technical assets

InfoSphere Business GlossaryInfoSphere Business GlossaryWeb-based authoring, managing and sharing of business metadataAligns the efforts of IT with the goals of the business Provides business context to

Subject Matter Experts

I f S h B i Gl

Business Users

information technology assetsEstablishes responsibility and accountability

Create and manage business vocabulary and relationships, while

linking to physical sources

InfoSphere Business Glossary

y linking to physical sources

GL Account Database = DB2Number

The ten digit account number. Sometimes referred to as th t ID

Schema = NAACCT

Table = DLYTRANS

C l Technical Business

Business View

the account ID. This value is of the form L-FIIIIVVVV.

Column = ACCT_NO

data type = char(11)

Technical

Business Glossary Anywhere ANYBusiness Glossary AnywhereReal-time access to business glossary from any desktop application

ANY User

FeaturesFrom any desktop application, click on a term & view its business definition in a pop-up window without any loss of context or focusI t lli t t hi t b t did t i

From Any Application..

Intelligent matching returns best candidates in a single searchSearch engine for terms and categoriesAccess steward contact information directlySecurity enforced via the Information Server common security layer

BenefitsIncreased trust and acceptance of information by delivering definitions in contextExpanded adoption of enterprise glossary outside ofExpanded adoption of enterprise glossary outside of Information Platform technologiesImproved information availability with multiple access mechanisms for electronically stored information (ESI)

Pop the Definition!

4WAREHOUSE

LEGACYSOURCES Correct

REFERENCE DATA

Data Steward

Understood

MASTER DATA

TermsTerms

Target state

METADATA

d t litdata quality

DI Pitfall #3

LEGACYSOURCES

“of course our data is good”

“ h b i h

“the business owner says the information we need is in there”

• ETL Proof of Concept• Client assured data quality sufficient so

excluded data cleansing from scope• At end of 2wk pilot, project halted due to

unsolvable data quality issues

“the schema’s show they have the same keys”

• Many 15-20 year old systems still in operation in NZ market

InfoSphere Information AnalyzerInfoSphere Information Analyzer

Data-centric analysis of application, database and file-based sources Data

AnalystsSubject Matter

Experts

Secure, detailed profiling of fields, across fields, and across sources

Analyse source data structures, and monitor adherence to integration and

InfoSphere Information Analyzer

Creation of metadata from profiling results

Results instantly promotable across

quality rules

Results instantly promotable across IBM InfoSphere Information Server

Physical View

4WAREHOUSE

LEGACYSOURCES

Correct

REFERENCE DATA

Data Steward

Understood

MASTER DATA

TermsTerms

Target state

Target stateSource

StateSource State

ETLHints

METADATA

It tiIterative Developmentp

DI Pitfall #4

DATA INTEGRATION3

“we’ll work it out in the testing”

• ETL development >75% total project $$P j t t ki 2 3 l th l d• Projects taking 2-3x longer than planned

• Some clients taking 70+% of dev.time doing impact analysis• Impact analysis methods very basic• Largely iterative development method• Unreliable forecast completion dates• Low levels of trust by business in IT ability to achieve BI

outcomes• Substantial cost overruns• Expensive BI maintenance costs

H d I Fi d O tWhere does the

data for thisHow do I Find Out …Data Analyst

data for this report come

…where this data comes from?

… when the job had been running last time?

… the details for these assets?

D l tDevelopment(Impact Analysis)( p y )

What is the InfoSphere Metadata Workbench?What is the InfoSphere Metadata Workbench? Web-based exploration of Information Assets generated and

Data I t ti Developers

gused by Information Server applicationsOut of the box reporting on data

Integration Managers

Developers

Provides IT professionals with a tool for

InfoSphere Metadata Workbench®

p gmovement, data lineage, business meaning, impact of changes and dependencies

Provides IT professionals with a tool for exploring and understanding the assets generated and used by the Information Server suite.

Tracing the data lineage of Business Intelligence Reports to provide basis for compliance with

Slegislation such as Sarbanes-Oxley and Basel II

4WAREHOUSE

LEGACYSOURCES

Correct

REFERENCE DATA

Data Steward

Understood

MASTER DATA

TermsTermsImpact AnalysisImpact

Analysis

Target state

Target stateSource

StateSource State

ETLHints

METADATA

D l tDevelopment(Iterative cycles)( y )

4WAREHOUSE

LEGACYSOURCES

Correct

REFERENCE DATA

Data Steward

UnderstoodRequirements

ETL Code GenerationETL Code

Generation

MASTER DATA

Analysis

Target state

Target stateSource

StateSource State

ETLHints

METADATA

InfoSphere FastTrack

Business analysts and IT

InfoSphere FastTrackTo reduce costs of integration projects through automation

Business analysts and IT collaborate in context to create project specification

Leverages source analysis

Specification

Leverages source analysis, target models, and metadata to facilitate mapping process

Auto-generation of data transformation jobs and reportsj p

Auto-generates DataStage jobs

Flexible Reporting

4WAREHOUSE

LEGACYSOURCES

Correct

REFERENCE DATA

Data Steward

UnderstoodRequirements

ETL Code GenerationETL Code

Generation

MASTER DATA

Analysis

Target state

Target stateSource

StateSource State

ETLHints

METADATA

93Information ServerO ti i i A li ti D l tOptimizing Application Development

IBM InfoSphere Information Server94

IBM InfoSphere Information ServerDelivering information you can trust

I f ti SInformation Server

Information Services DirectorInfoSphere

Data Architect

Information AnalyzerInfoSphere

Business GlossaryInfoSphereQualityStageInfoSphere DataStageInfoSphere

Federation ServerInfoSphere

Replication Server / EVPInfoSphereInfoSphere

FastTrackInfoSphere Change Data CaptureInfoSphere

Metadata ServerInfoSphere

Metadata WorkbenchInfoSphere Metadata WorkbenchInfoSphere

95Bringing It All Togetherg g g

DevelopersSubject Matter Experts

DataAnalysts

Business Users

Architects DBAs

Simplify Integration Increase trust and confidence in informationI li tF ilit t h

Information Server – Common Framework

Increase compliance to standards

Facilitate change management & reuseDesign Operational

Leading from the FrontGreater Preparation will yield dramatically lowerGreater Preparation will yield dramatically lower project costs/times

Typical Work Effort for Migration Activities

15-30% of total project budget will be spent on Migration Activities15-30% of total project budget will be spent on Migration Activities15 30% of total project budget will be spent on Migration Activitiesp j g p g

30%Understanding

40%Cleaning, Standardising

30%Conversion, Loading,

DeliverDiscover Prepare

Largely manual effort on small percentage of data. Some manual

This effort is the most unpredictable. The work can vary greatly depending on condition of data, however it is always the largest piece of work in the data initiative.

Largely manual effort on 100% of data. This can mean d f l i t ll t

Coding transformations and loads. Traditionally this effort is plagued with problems related to data quality and it

can easily be pulled by necessity into the

75% Business 50% Business 25% Business

Source Data Harmonizing, Management Interfaces, Connectivity

percentage of data. Some manual coding can review all data . dozens of persons cleaning source systems manually to

correct and augment data and manually aligning records to MRD. Some manual coding can reduce the manual

effort.

can easily be pulled by necessity into the Cleaning, Standardising and Harmonising

area causing timing and budget problems.

75% IT50% IT25% IT

Th kThank you

Questions?Questions?

InfoSphere: Leading from the Front - Accelerating Data Integration through Metadata

Technology

InfoSphere DS8

What's new in InfoSphere Information Server 8 - DataStagedsxchange.net/uploads/DataStage_8.7.pdf · 4 What’s new in IBM InfoSphere Information Server 8.7 Joint InfoSphere DataStage

FÆ A4 FÇ DatabaseGearInformatica PowerCenter, IBM InfoSphere DataStage, Oracle Data Integrator FúFù Flat File Structures XSD RDBMS Schema/DDL OLAP metadata Forward Reverse Migration

IBM InfoSphere Guardium overview

Click to add text © 2012 IBM Corporation 1 InfoSphere Streams Streams Console Applications InfoSphere Streams Version 3.0 Warren Acker InfoSphere Streams

The value of IBM InfoSphere BigInsights - · PDF fileThe value of IBM InfoSphere BigInsights. Enterprise software integration . Integrating InfoSphere BigInsights—and its analysis

IBM InfoSphere Streams

IBM InfoSphere Master Data Management...Mar 12, 2014 · IBM InfoSphere Master Data Management IBM InfoSphere MDM Inspector ΓU 11 3 GI13-2653-01

InfoSphere DataStage Grid Solution - dsxchangedsxchange.net/uploads/2011IISGridSolution.pdf · InfoSphere DataStage Grid Solution ... • Available for DataStage, ... integration

15 The OIOSAML Toolkits - Accelerating a common eGov ... · Identity Provider Service Provider ok ... SAML 2.0 MetaData ... 15 The OIOSAML Toolkits - Accelerating a common eGov infrastructure

Making the Most of InfoSphere Guardium Vulnerability ... · PDF fileMaking the Most of InfoSphere Guardium Vulnerability Assessment ... InfoSphere Guardium Vulnerability Assessment

6 System requirements for InfoSphere CDC for InfoSphere

Version 1 Release 2 IBM InfoSphere Information Server on Cloud · 2017-11-09 · interchange agent InfoSphere Metadata Asset Manager InfoSphere Information Server istool command-line

IBM InfoSphere MDM Inspector · 2020. 10. 30. · IBM InfoSphere Master Data Management V11.5 IBM InfoSphere MDM Inspector C'8O SC43-1914-01

IBM InfoSphere Change Data Capturedownload2.boulder.ibm.com/sar/CMA/IMA/00khx/0/ISCDC_zOS_6_2_1.… · IBM InfoSphere Change Data Capture InfoSphere Change Data Capture for z/OS,

InfoSphere MDM Collaboration Server: Installation Guide · iv InfoSphere MDM Collaboration Server: Installation Guide Preface IBM ®InfoSphere Master Data Management Collaboration

Survey’of’Streaming’Applicaons’ using’Streams’ · Streams’Programming’Language ... Social Media Consumer Profiles Customer Models InfoSphere Streams InfoSphere BigInsights

IBM InfoSphere Roadshow – WELCOME InfoSphere Roadshow – WELCOME October 2010. 2 AGENDA •Introduction / Welcome •InfoSphere Positioning within Information Management •InfoSphere

IBM InfoSphere Information Analyzer

Infosphere Newsletter-Winter 2008