Practical Applications for Data Warehousing, Analytics, BI, and Meta-Integration Technologies

TITLE

PRODUCED BYDATA BLUEPRINT 10124-C W. BROAD ST, GLEN ALLEN, VA 23060

CLASSIFICATION

EDUCATIONDATE SLIDE

7/10/201207/06/12 © Copyright this and previous years by Data Blueprint - all rights reserved!

Welcome!

Date: July 10, 2012Time: 2:00 PM ETPresented by: Dr. Peter Aiken

1

Practical Applications for Data Warehousing, Analytics, BI, and Meta-Integration Technologies

TITLE


CLASSIFICATION

EDUCATIONDATE SLIDE


2

Meta-integration is considered data warehousing by some, while others describe it as data virtualization. This presentation provides an overview of meta-integration starting with organizational requirements. We will discuss how meta-models can be used to jump-start organizational efforts. Participants will understand the strengths and weaknesses of various technological capabilities, and the key role of data quality in all of them. Turns out that proper analysis at this stage makes actual technology selection far more accurate.

Abstract: DW, Analytics, BI, Meta-Integration Technologies

TITLE


CLASSIFICATION

EDUCATIONDATE SLIDE


Live Twitter Feed & Facebook Updates

Join the conversation on Twitter!

Follow us @datablueprint and @paiken

Ask questions and submit your comments: #dataed

3

www.facebook.com/datablueprint

Post questions and comments

Find industry news, insightful content

and event updates

TITLE


CLASSIFICATION

EDUCATIONDATE SLIDE


LinkedIn Group: Join the Discussion

New Group:Data Management & Business Intelligence

4

TITLE


CLASSIFICATION

EDUCATIONDATE SLIDE


Meet Your Presenter: Dr. Peter Aiken

5

• Internationally recognized thought-leader in the data management field with more than 30 years of experience

• Recipient of the 2010 International Stevens Award

• Founding Director of Data Blueprint (http://datablueprint.com)

• Associate Professor of Information Systems at Virginia Commonwealth University (http://vcu.edu)

• President of DAMA International (http://dama.org)• DoD Computer Scientist, Reverse Engineering Program Manager/

Office of the Chief Information Officer • Visiting Scientist, Software Engineering Institute/Carnegie Mellon

University• 7 books and dozens of articles• Experienced w/ 500+ data management practices in 20 countries

#dataed

7/10/2012

Data Warehousing, Analytics, BI,

Meta-Integration Technologies

Data Warehousing, Analytics, BI, Meta-Integration Technologiesn/a

TITLE


CLASSIFICATION

EDUCATIONDATE SLIDE


Outline1. Data management overview2. What are DW, analytics, BI and meta-

integration technologies and why are they important?

3. Top 10 causes of data warehouse failures

4. DW & architecture focus5. Business intelligence focus6. The use of meta models 7. DW, analytics & BI building blocks8. Guiding principles & best practices9. Take aways, references and Q&A

7

Tweeting now: #dataed

TITLE


CLASSIFICATION

EDUCATIONDATE SLIDE


The DAMA Guide to the Data Management Body of Knowledge

8

Data Management

Functions

Published by DAMA International• The professional

association for Data Managers (40 chapters worldwide)

DMBoK organized around • Primary data

management functions focused around data delivery to the organization

• Organized around several environmental elements

TITLE


CLASSIFICATION

EDUCATIONDATE SLIDE


The DAMA Guide to the Data Management Body of Knowledge

9

Environmental Elements

Amazon:http://www.amazon.com/DAMA-Guide-Management-Knowledge-DAMA-DMBOK/dp/0977140083Or enter the terms "dama dm bok" at the Amazon search engine

TITLE


CLASSIFICATION

EDUCATIONDATE SLIDE


Data Management

10

TITLE


CLASSIFICATION

EDUCATIONDATE SLIDE


Data ManagementManage data coherently.

Share data across boundaries.

Assign responsibilities for data.Engineer data delivery systems.

Maintain data availability.

11

Data Program Coordina;on

Organiza;onal Data Integra;on

Data Stewardship

Data Development

Data Support Opera;ons

TITLE


CLASSIFICATION

EDUCATIONDATE SLIDE


Data Management

12

TITLE


CLASSIFICATION

EDUCATIONDATE SLIDE


Summary: Data Warehousing & Business Intelligence Management

13

TITLE


CLASSIFICATION

EDUCATIONDATE SLIDE






14


TITLE


CLASSIFICATION

EDUCATIONDATE SLIDE


DW, Analytics, BI, Meta-Integration TechnologiesDefinitions• Beyond the nuts and bolts of

data management• Analysis of information that had

not been integrated previously

Business Intelligence• Dates at least to 1958• Support better business

decision making• Technologies, applications and

practices for the collection, integration, analysis, and presentation of business information

• Also described as decision support

15

from The DAMA Guide to the Data Management Body of Knowledge © 2009 by DAMA International

Data Warehousing• Operational extract, cleansing,

transformation, load, and associated control processes for integrating disparate data into a single conceptual database

TITLE


CLASSIFICATION

EDUCATIONDATE SLIDE


16

Definitions, cont’d• Study of data to discover and

understand historical patterns to improve future performance

• Use of mathematics in business

• Analytics closely resembles statistical analysis and data mining

– based on modeling involving extensive computation.

• Some fields within the area of analytics are

– enterprise decision management, marketing analytics, predictive science, strategy science, credit risk analysis and fraud analytics.

TITLE


CLASSIFICATION

EDUCATIONDATE SLIDE


• Inmon:– "A subject oriented, integrated, time variant, and

non-volatile collection of summary and detailed historical data used to support the strategic decision-making processes of the organization."

• Kimball:– "A copy of transaction data specifically structured

for query and analysis."• Key concepts focus on:

– Subjects– Transactions– Non-volatility– Restructuring

Warehousing Definitions

17

TITLE


CLASSIFICATION

EDUCATIONDATE SLIDE


• Bank accounts are of varying value and risk

• Cube by – Social status– Geographical location– Net value, etc.

• Balance return on the loan with risk of default

18

• How to evaluate the portfolio as a whole?– Least risk loan may be to the very wealthy, but there are a very

limited number – Many poor customers, but greater risk

• Solution may combine types of analyses– When to lend, interest rate charged

Example: Portfolio Analysis

TITLE


CLASSIFICATION

EDUCATIONDATE SLIDE


19

from MicroStrategy, Better Business Decisions Every Day: Integrating Business Reporting & Analysis

Example: Set Analysis

TITLE


CLASSIFICATION

EDUCATIONDATE SLIDE

7/10/201207/06/12 © Copyright this and previous years by Data Blueprint - all rights reserved! 20

15 years ago, CarMax started as a way to make the car buying experience simple, fair, and fun. Today CarMax is a FORTUNE 500 retailer and one of FORTUNE’s “100 Best Companies to Work For.” And we are hiring talented individuals who are interested in:--solving original, wide-ranging, and open-ended business problems--not only discovering new insights, but successfully implementing them--making a significant mark on a growing company--developing the fundamental skills for a rewarding business career

If that sounds like you, the Strategy Analyst position is the unique opportunity you’ve been looking for. The strategy team at CarMax currently consists of over 40 analysts, many of whom are recent college graduates from top schools with a variety of academic backgrounds (computer science, economics, English, engineering, journalism, math, political science). These analysts lead advances and decisions in several key business areas:-Inventory and pricing—what is the optimal selection of inventory, how do we acquire it, what should we pay for it, what should we price it for?-Expansion planning—which markets should we enter and how do we store those markets? Will each $10-30 million store investment generate a sufficient economic return?-Credit strategy—how can our bank (CarMax Auto Finance) approve more customers for loans and convert more approvals to sales?-Marketing and consumer insight—how do we reach our customers, increase traffic to our stores, and best use the internet to drive sales and build our brand-Industry and competitive research—what middle- and long-term risks are we exposed to, and how best do we prepare to respond? -Production—how do we increase vehicle reconditioning quality while reducing cost and production time?-Sales process and workforce—what is the best way to serve customers in our stores, and how do we manage, motivate and compensate our sales team?

Even early in your career at CarMax, you will have the responsibility to own an area of the business and will be expected to improve it. For example, one undergraduate recruit used data analysis to reformulate our retail pricing strategy, pitched and sold his idea to the senior executive team, and implemented a new system nationwide in his first 6 months with the company. That is the kind of impact you can make at CarMax. And as you do this, you will work closely with the senior executives and analytical managers to develop the fundamental and advanced skills that underpin a successful career in business. In fact, most of our managers in the strategy group started at CarMax as analysts, and our VP of Strategy and Analysis started his career here through our undergraduate recruiting program. While an MBA is not required to advance or contribute at CarMax, analysts who have chosen to pursue a business degree have enjoyed superior acceptance rates at their first choice schools, including Harvard, Chicago, UVa, Columbia, and Duke.

Your opportunities to develop, contribute, and lead as an analyst at CarMax are as great as the company’s opportunity to grow. While CarMax is already the largest used car retailer in the country (with over $8 billion in sales and over 90 superstores across the country), we have only 2% of the 1 to 6-year-old used car market, which, at $280 billion annually, is bigger than the home improvement or consumer electronics industries. CarMax is already growing at 15% a year, and over the next 10 years plans to have 250-300 stores and achieve $25+ billion in annual sales. As an analyst, you can be an integral part of that growth, all while enjoying a casual and friendly environment, a diverse group of talented associates, a healthy work-life balance, and excellent compensation and benefits.

An ideal candidate will have--Demonstrated top caliber analytic and problem solving skills --History of achievement demonstrated by top 15% GPA, with a quantitative major(s), and/or other recognition such as scholarships, awards, honor societies -- Passion for business and desire to develop into a strong business leader

We encourage you to apply. For more information, please visit us at the career fair, on our website (www.carmax.com/collegerecruiting), or email us at [email protected]://www.seas.virginia.edu/careerdevelopment/index.php?option=com_careerfairstudent&task=detailView&employerId=216&eventId=3

- datablueprint.com 8/2/2010 © Copyright this and previous years by Data Blueprint - all rights reserved!

CarMax Example Job Posting

24

own an area of the business and will be expected to improve it

--solving original, wide-ranging, and open-ended business problems--not only discovering new insights, but successfully implementing them--making a significant mark on a growing company--developing the fundamental skills for a rewarding business career

TITLE


CLASSIFICATION

EDUCATIONDATE SLIDE


Operations Research

• Interdisciplinary branch of applied mathematics and formal science • Uses methods such as mathematical modeling, statistics, and

algorithms to arrive at optimal or near optimal solutions • Typically concerned with optimizing the maxima (profit, assembly

line performance, crop yield, bandwidth, etc) or minima (loss, risk, etc.) of some objective function

• Operations research helps management achieve its goals using scientific methods http://en.wikipedia.org/wiki/Operations_research

21

TITLE


CLASSIFICATION

EDUCATIONDATE SLIDE






22


TITLE


CLASSIFICATION

EDUCATIONDATE SLIDE


Indiana Jones: Raiders Of The Lost Ark

23

TITLE


CLASSIFICATION

EDUCATIONDATE SLIDE


Top Causes of Data Warehouse Failure• Poor Quality Data

– Many more values of gender code than (M/F)

• Incorrectly Structured Data

– Providing the correct answer to the wrong question

• Bad Warehouse Design

– Overly complex

24


TITLE


CLASSIFICATION

EDUCATIONDATE SLIDE


Top 10 Data Warehouse Failure Causes1. The project is over budget2. Slipped schedule3. Functions and

capabilities not implemented

4. Unhappy users5. Unacceptable performance6. Poor availability7. Inability to expand 8. Poor quality data/reports9. Too complicated for users10. Project not cost justified

25

from The Data Administration Newsletter, www.tdan.com

TITLE


CLASSIFICATION

EDUCATIONDATE SLIDE






26


TITLE


CLASSIFICATION

EDUCATIONDATE SLIDE


Health Care Provider Data Warehouse• 1.8 million members• 1.4 million providers• 800,000 providers no key• 2.2% prov_number = 9 digits (required)• 29% prov_ssn ≠ 9 digits• 1 User

27

"I can take a roomful of MBAs and accomplish this analysis faster!"

TITLE


CLASSIFICATION

EDUCATIONDATE SLIDE


28

Basic Data Warehouse Analysis

• Emphasis on the cube

• Permits different users to "slice and dice" subsets of data

• Viewing from different perspectives


TITLE


CLASSIFICATION

EDUCATIONDATE SLIDE


29

Warehouse Analysis

• Users can "drill" anywhere

• Entire collection is accessible

• Summaries to transaction-level detail


TITLE


CLASSIFICATION

EDUCATIONDATE SLIDE


Oracle

30


TITLE


CLASSIFICATION

EDUCATIONDATE SLIDE


Corporate Information Factory Architecture

31


TITLE


CLASSIFICATION

EDUCATIONDATE SLIDE



32


TITLE


CLASSIFICATION

EDUCATIONDATE SLIDE


33



TITLE


CLASSIFICATION

EDUCATIONDATE SLIDE


34



TITLE


CLASSIFICATION

EDUCATIONDATE SLIDE


Kimball's DW Chess Pieces

35



MetaMatrix Integration Example

36

• EII Enterprise Information Integration– between ETL and EAI -

delivers tailored views of information to users at the time that it is required


Linked Data

37

Linked Data is about using the Web to connect related data that wasn't previously linked, or using the Web to lower the barriers to linking data currently linked using other methods. More specifically, Wikipedia defines Linked Data as "a term used to describe a recommended best practice for exposing, sharing, and connecting pieces of data, information, and knowledge on the Semantic Web using URIs and RDF."

linkeddata.org

TITLE


CLASSIFICATION

EDUCATIONDATE SLIDE


R& D Applica.ons(researcher supported, no documenta.on)

Finance Applica.on(3rd GL, batch system, no source)

Payroll Applica.on(3rd GL)

Payroll Data(database)

FinanceData

(indexed)

Personnel Data(database)

R & DData(raw)

Mfg. Data(home growndatabase) Mfg. Applica.ons

(contractor supported)

Marke.ng Applica.on(4rd GL, query facili.es, no repor.ng, very large)

Marke.ng Data(external database)

Personnel App.(20 years old,

un-‐normalized data)

38

Multiple Sources of (for example) Customer Data

TITLE


CLASSIFICATION

EDUCATIONDATE SLIDE






39


TITLE


CLASSIFICATION

EDUCATIONDATE SLIDE


40

Styles of Business Intelligence


TITLE


CLASSIFICATION

EDUCATIONDATE SLIDE


41

Business Intelligence Features

Problema)c Data Quality

TITLE


CLASSIFICATION

EDUCATIONDATE SLIDE


5 Key Business Intelligence Trends1. There's so much data, but too little

insight. More data translates to a greater need to manage it and make it actionable.

2. Market consolidation means fewer choices for business intelligence users.

3. Business Intelligence expands from the Board Room to the front lines. Increasingly, business intelligence tools will be available at all levels of the corporation

4. The convergence of structured and unstructured data Will create better business intelligence.

5. Applications will provide new views of business intelligence data. The next generation of business intelligence applications is moving beyond the pie charts and bar charts into more visual depictions of data and trends.

42

hOp://www.cio.com/ar.cle/150450/Five_Key_Business_Intelligence_Trends_You_Need_to_Know?page=2&taxonomyId=3002

TITLE


CLASSIFICATION

EDUCATIONDATE SLIDE






43


TITLE


CLASSIFICATION

EDUCATIONDATE SLIDE


Source:http://dmreview.com/article_sub.cfm?articleID=1000941 used with permission

Meta Data Models

44

TITLE


CLASSIFICATION

EDUCATIONDATE SLIDE


WarehouseProcess

WarehouseOpera.on

Transforma.on

XMLRecord-‐Oriented

Mul.DimensionalRela.onal

BusinessInforma.on

So`wareDeployment

ObjectModel(Core, Behavioral, Rela.onships, Instance)

WarehouseManagement

Resources

Analysis

Object-‐Oriented

(ObjectModel)

Foundation

OLAP Data Mining

Informa.onVisualiza.on

BusinessNomenclature

DataTypes Expressions Keys

IndexType

Mapping

Overview of CWM Metamodel

http://www.omg.org/technology/documents/modeling_spec_catalog.htm

45

TITLE


CLASSIFICATION

EDUCATIONDATE SLIDE






46


TITLE


CLASSIFICATION

EDUCATIONDATE SLIDE


Data Warehousing, Analytics, BI, Meta-Integration Technologies

47

üü ü üü ü üüü ü üü ü üüü ü üü ü üüü ü üü ü üüü

üü

üü

üü

üü

üü

üü


TITLE


CLASSIFICATION

EDUCATIONDATE SLIDE



Goals and Principles1. To support and enable

effective business analysis and decision making by knowledgeable workers

2. To build and maintain the environment/infrastructure to support business intelligence activities, specifically leveraging all the other data management functions to cost effectively deliver consistent integrated data for all BI activities

48

TITLE


CLASSIFICATION

EDUCATIONDATE SLIDE


• Understand BI information needs

• Define and maintain the DW/BI architecture

• Process data for BI

• Implement data warehouse/data marts

• Implement BI tools and user interfaces

• Monitor and tune DW processes

• Monitor and tune BI activities and performance

49


Activities

TITLE


CLASSIFICATION

EDUCATIONDATE SLIDE


Primary Deliverables • DW/BI Architecture

• Data warehouses, marts, cubes etc.

• Dashboards-scorecards

• Analytic applications

• Files extracts (for data mining, etc.)

• BI tools and user environments

• Data quality feedback mechanism/loop

50


TITLE


CLASSIFICATION

EDUCATIONDATE SLIDE


Roles and ResponsibilitiesSuppliers:• Executives/managers• Subject Matter Experts• Data governance council• Information consumers• Data producers• Data architects/analysts

Consumers:• Application Users• BI and Reporting

Users• Application

Developers and Architects

• Data integration Developers and Architects

• BI Vendors and Architects

• Vendors, Customers and Partners


Participants:• Executives/managers• Data Stewards• Subject Matter Experts• Data Architects• Data Analysts• Application Architects• Data Governance Council• Data Providers• Other BI Professionals

51

TITLE


CLASSIFICATION

EDUCATIONDATE SLIDE


Technology • ETL• Change Management Tools • Data Modeling Tools• Data Profiling Tools• Data Cleansing Tools• Data Integration Tools• Reference Data Management Applications• Master Data Management Applications• Process Modeling Tools• Meta-data Repositories• Business Process and Rule Engines


52

TITLE


CLASSIFICATION

EDUCATIONDATE SLIDE






53


TITLE


CLASSIFICATION

EDUCATIONDATE SLIDE


Guiding Principles1. Obtain executive commitment and

support. 2. Secure business SMEs. 3. Be business focused and driven. Let

the business drive the prioritization.4. Demonstrate data quality is

essential.5. Provide incremental value.

54


6. Transparency and self service. 7. One size does not fit all: Find the right tools and products for each

of your segments.8. Think and architect globally, act and build locally.9. Collaborate with and integrate all other data initiatives, especially

those for data governance, data quality and metadata.10. Start with the end in mind. 11. Summarize and optimize last, not first.

TITLE


CLASSIFICATION

EDUCATIONDATE SLIDE


6 Best Practices for Data Warehousing

55

1. Do some initial architecture envisioning.

2. Model the details just in time (JIT).

3. Prove the architecture early.

4. Focus on usage.

5. Organize your work by requirements.

6. Active stakeholder participation.

hEp://www.agiledata.org/essays/dataWarehousingBestPrac;ces.html

TITLE


CLASSIFICATION

EDUCATIONDATE SLIDE






56


TITLE


CLASSIFICATION

EDUCATIONDATE SLIDE


Summary: Data Warehousing & Business Intelligence Management

57


TITLE


CLASSIFICATION

EDUCATIONDATE SLIDE


References

58

TITLE


CLASSIFICATION

EDUCATIONDATE SLIDE


References

59

TITLE


CLASSIFICATION

EDUCATIONDATE SLIDE


Additional References• http://www.information-management.com/infodirect/20050909/1036703-1.html • http://www.agiledata.org/essays/dataWarehousingBestPractices.html • http://www.cio.com/article/150450/

Five_Key_Business_Intelligence_Trends_You_Need_to_Know?page=2&taxonomyId=3002

• http://www.computerworld.com/s/article/9228736/Business_Intelligence_and_analytics_Conquering_Big_Data?taxonomyId=9

• http://www.enterpriseirregulars.com/5706/the-top-10-trends-for-2010-in-analytics-business-intelligence-and-performance-management/

• http://www.itbusinessedge.com/cm/blogs/vizard/taking-the-analytics-pressure-off-the-data-warehouse/?cs=50698

• http://www.informationweek.com/news/software/bi/240001922

60

TITLE


CLASSIFICATION

EDUCATIONDATE SLIDE


Questions?

61

It’s your turn! Use the chat feature or Twitter (#dataed) to submit

your questions to Peter now.

+ =

TITLE


CLASSIFICATION

EDUCATIONDATE SLIDE


Upcoming Events

62

August Webinar:Your Documents and Other Content: Managing Unstructured DataAugust 14, 2012 @ 2:00 PM – 3:30 PM ET(11:00 AM-12:30 PM PT)

September Webinar:Let’s Talk Metadata: Strategies and SuccessesSeptember 11, 2012 @ 2:00 PM – 3:30 PM ET(11:00 AM-12:30 PM PT)

Sign up here:• www.datablueprint.com/webinar-schedule • www.Dataversity.net

Brought to you by: