An Introduction to Data Warehousing Concept and Technology Mort Anvari

Data Warehousing ConceptData Warehousing Concept

Data Access TechnologyData Access Technology

Enterprise Real-Time Knowledge Enterprise Real-Time Knowledge Architecture for Data WarehousingArchitecture for Data Warehousing

Data Collection and Delivery Data Collection and Delivery

Data Warehousing ConceptData Warehousing Concept

Data Access TechnologyData Access Technology

Enterprise Real-Time Knowledge Enterprise Real-Time Knowledge Architecture for Data WarehousingArchitecture for Data Warehousing

Data Collection and Delivery Data Collection and Delivery

TopicsTopics

M. Anvari Page 3

Benson & Parker’s “Square Wheel”Benson & Parker’s “Square Wheel”Benson & Parker’s “Square Wheel”Benson & Parker’s “Square Wheel”

BusinessBusinessEnvironmentEnvironment

TechnologyTechnologyEnvironmentEnvironment

BusinessBusinessPlanningPlanning

BusinessBusinessOperationsOperations

M. Anvari Page 4






TechnologyTechnologyPlanningPlanning

TechnologyTechnologyOperationsOperations

M. Anvari Page 5








AlignmentAlignment

ImpactImpact

OrganizationOrganization OpportunityOpportunity

M. Anvari Page 6








AlignmentAlignment

ImpactImpact

OrganizationOrganization OpportunityOpportunityInformation Technology has to do more than Information Technology has to do more than just align itself with the business, it has to helpjust align itself with the business, it has to helpthe business have the maximum impact in the the business have the maximum impact in the marketplace.marketplace.

Data Access Data Access and

Delivery System

Data Access Data Access and

Delivery System

M. Anvari Page 8

Technology EvolutionTechnology EvolutionTechnology EvolutionTechnology Evolution

New classes of computersNew classes of computers

New classes of communicationsNew classes of communications

New classes of technology (image, sound, video, New classes of technology (image, sound, video, multimedia)multimedia)

New classes of softwareNew classes of software

Much more complex technical environmentMuch more complex technical environment

Cooperative Processing/Client-ServerCooperative Processing/Client-Server Distributed Data BasesDistributed Data Bases LANs, WANs, etc.LANs, WANs, etc.

Obsolescence Problem Multiple Legacy Systems

New classes of computersNew classes of computers

New classes of communicationsNew classes of communications

New classes of technology (image, sound, video, New classes of technology (image, sound, video, multimedia)multimedia)

New classes of softwareNew classes of software

Much more complex technical environmentMuch more complex technical environment

Cooperative Processing/Client-ServerCooperative Processing/Client-Server Distributed Data BasesDistributed Data Bases LANs, WANs, etc.LANs, WANs, etc.

Obsolescence Problem Multiple Legacy Systems

M. Anvari Page 9

IT Impact on BusinessIT Impact on BusinessIT Impact on BusinessIT Impact on Business

HP

IBMDEC

Compaq

Enterprise Network Computing and Client/Server Technology areEnterprise Network Computing and Client/Server Technology arechanging the way organizations look at all of their information systemschanging the way organizations look at all of their information systems

Data Jail

Obsolescence

IT Wastes

M. Anvari Page 10

The Existing EnterpriseThe Existing EnterpriseThe Existing EnterpriseThe Existing Enterprise

Support Existing ProductsSupport Existing Products

Support Existing CustomersSupport Existing Customers

Support Existing OrganizationSupport Existing Organization

Support Existing WorkforceSupport Existing Workforce

Support Existing TechnologySupport Existing Technology

Support Existing ProductsSupport Existing Products

Support Existing CustomersSupport Existing Customers

Support Existing OrganizationSupport Existing Organization

Support Existing WorkforceSupport Existing Workforce

Support Existing TechnologySupport Existing Technology

M. Anvari Page 11

Controlling the (Global)Controlling the (Global)Real-time OrganizationReal-time OrganizationControlling the (Global)Controlling the (Global)Real-time OrganizationReal-time Organization

RTO = 24 x 7 x ERTO = 24 x 7 x ERTO = 24 x 7 x ERTO = 24 x 7 x E

(Where E means every major market)(Where E means every major market)

M. Anvari Page 12

Information and the EnterpriseInformation and the EnterpriseInformation and the EnterpriseInformation and the Enterprise

Organizational needs for dataOrganizational needs for data

Organizational needs for informationOrganizational needs for information

Organizational needs for knowledgeOrganizational needs for knowledge

Organizational needs for dataOrganizational needs for data

Organizational needs for informationOrganizational needs for information

Organizational needs for knowledgeOrganizational needs for knowledge

M. Anvari Page 14

Needs for DataNeeds for DataNeeds for DataNeeds for Data

Data = Values (Measurements)Data = Values (Measurements)

Data to operateData to operate

Data to controlData to control

Data to planData to plan

Data = Values (Measurements)Data = Values (Measurements)

Data to operateData to operate

Data to controlData to control

Data to planData to plan

M. Anvari Page 15

Needs for InformationNeeds for InformationNeeds for InformationNeeds for Information

Information = Content + Structure (Relationships)Information = Content + Structure (Relationships)

Structure of the Real-worldStructure of the Real-world

Relating data to the businessRelating data to the business

Cross functional processesCross functional processes

Relating data to the real worldRelating data to the real world

External DBExternal DB

External Data Feeds (D&B, Reuters, etc.)External Data Feeds (D&B, Reuters, etc.)

Text, Image, Voice, Video, etc.Text, Image, Voice, Video, etc.

Statistical StudiesStatistical Studies

Information = Content + Structure (Relationships)Information = Content + Structure (Relationships)

Structure of the Real-worldStructure of the Real-world

Relating data to the businessRelating data to the business

Cross functional processesCross functional processes

Relating data to the real worldRelating data to the real world

External DBExternal DB

External Data Feeds (D&B, Reuters, etc.)External Data Feeds (D&B, Reuters, etc.)

Text, Image, Voice, Video, etc.Text, Image, Voice, Video, etc.

Statistical StudiesStatistical Studies

M. Anvari Page 16

Needs for KnowledgeNeeds for KnowledgeNeeds for KnowledgeNeeds for Knowledge

Knowledge = Goals + Actions + LearningKnowledge = Goals + Actions + Learning

Learning more about our businessLearning more about our business

Learning more about our marketLearning more about our market

Learning more about the business environmentLearning more about the business environment

Knowledge is the area in which Data Warehousing and Knowledge is the area in which Data Warehousing and Data Mining are potentially critical technologiesData Mining are potentially critical technologies

Knowledge = Goals + Actions + LearningKnowledge = Goals + Actions + Learning

Learning more about our businessLearning more about our business

Learning more about our marketLearning more about our market

Learning more about the business environmentLearning more about the business environment

Knowledge is the area in which Data Warehousing and Knowledge is the area in which Data Warehousing and Data Mining are potentially critical technologiesData Mining are potentially critical technologies

M. Anvari Page 17

Data, Information and KnowledgeData, Information and KnowledgeData, Information and KnowledgeData, Information and Knowledge

Data Centers Data Centers

Information CentersInformation Centers

Knowledge CentersKnowledge Centers

Data Centers Data Centers


Knowledge CentersKnowledge Centers

Data BasesData Bases

Information BasesInformation Bases

Knowledge BasesKnowledge Bases

Data BasesData Bases

Information BasesInformation Bases

Knowledge BasesKnowledge Bases

M. Anvari Page 18

Old Data Never DiesOld Data Never DiesOld Data Never DiesOld Data Never Dies

Note that none of the early computing styles have Note that none of the early computing styles have ever gone away!!!ever gone away!!!

Note that none of the early computing styles have Note that none of the early computing styles have ever gone away!!!ever gone away!!!

Batch

On-line

Minis

PCs

Networking

Enterprise Computing (Peer to Peer, Network to Network)

60s 70s 80s 90s

M. Anvari Page 19

Operational vs. InformationalOperational vs. InformationalSystemsSystemsOperational vs. InformationalOperational vs. InformationalSystemsSystems

Information Access TodayInformation Access Today

M. Anvari Page 20



OperationalOperationalSystemsSystems

Mafg.Mafg. Ord.Ord.EntryEntry

M. Anvari Page 21




InformationalInformationalSystemsSystems

M. Anvari Page 22





EstimatingEstimating & Analysis& Analysis

MarketingMarketingSystemsSystems

ProductProductPlanningPlanning

M. Anvari Page 23





InformationInformationDelivery SystemDelivery System

M. Anvari Page 24





InformationInformationDelivery SystemDelivery SystemData Warehousing is fundamentallyData Warehousing is fundamentally

an issue of Enterprise Data Architecturean issue of Enterprise Data Architecture

M. Anvari Page 25





M. Anvari Page 26




InformationInformationDelivery SystemDelivery SystemDataDataWarehouseWarehouse

M. Anvari Page 27



InformationInformationDelivery SystemDelivery SystemDataDataWarehouseWarehouse


Data Data MartsMarts

M. Anvari Page 28





Data Data WarehouseWarehouse

ExternalExternalDataData

DataDataGaragesGarages

M. Anvari Page 29





DataDataWarehouseWarehouse

ExternalExternalDataData

ExternalExternalUsersUsers

M. Anvari Page 30

End User EvolutionEnd User EvolutionEnd User EvolutionEnd User Evolution

Data Base Management Systems usersData Base Management Systems users

Ad Hoc Reports usersAd Hoc Reports users

Today’s Customer Demands Automated Real-Time Today’s Customer Demands Automated Real-Time Response.Response.

End User SystemsEnd User Systems

Decision Support SystemsDecision Support Systems

Executive Information SystemsExecutive Information Systems


Data Base Management Systems usersData Base Management Systems users

Ad Hoc Reports usersAd Hoc Reports users

Today’s Customer Demands Automated Real-Time Today’s Customer Demands Automated Real-Time Response.Response.

End User SystemsEnd User Systems

Decision Support SystemsDecision Support Systems

Executive Information SystemsExecutive Information Systems


M. Anvari Page 31

Ways to Organize DataWays to Organize DataWays to Organize DataWays to Organize Data

TablesTables Flexible, SimpleFlexible, Simple

HierarchiesHierarchies Speed, Natural ReportingSpeed, Natural Reporting

NetworksNetworks Multiple Directions, Complex StructureMultiple Directions, Complex Structure

ListsLists Updating Complex StructureUpdating Complex Structure

Matrices / ArrayMatrices / Array Manipulate Multiple Dimensions Manipulate Multiple Dimensions

Inverted FilesInverted Files Unplanned queries, text retrievalUnplanned queries, text retrieval

ObjectsObjects Complex structures, hide structureComplex structures, hide structure

Multidimensional Data Bases (Data Warehousing)Multidimensional Data Bases (Data Warehousing)

TablesTables Flexible, SimpleFlexible, Simple

HierarchiesHierarchies Speed, Natural ReportingSpeed, Natural Reporting

NetworksNetworks Multiple Directions, Complex StructureMultiple Directions, Complex Structure

ListsLists Updating Complex StructureUpdating Complex Structure

Matrices / ArrayMatrices / Array Manipulate Multiple Dimensions Manipulate Multiple Dimensions

Inverted FilesInverted Files Unplanned queries, text retrievalUnplanned queries, text retrieval

ObjectsObjects Complex structures, hide structureComplex structures, hide structure

Multidimensional Data Bases (Data Warehousing)Multidimensional Data Bases (Data Warehousing)

M. Anvari Page 32

End User Computing EvolutionEnd User Computing EvolutionEnd User Computing EvolutionEnd User Computing Evolution

Tool or Technique StrengthsFile Access Systems Physical access to dataNetwork DB Support for complex data interrelationsHierarchical DB Support for hierarchical views of dataInverted File DB Support for unplanned inquiry, esp. textRelational DB Flexibility and ease of updatingReport Generator Support for simple ad hoc reportingQuery Language Support for simplead hoc inquiries4GL Ability to develop simple systems easilyDecision Support System Ability to support financial and statistical

data analysisExecutive Information System Ability to present information to

executivesInformation Center Support for end users trying to access

enterprise information

M. Anvari Page 33

Data WarehousingData WarehousingData WarehousingData Warehousing

Data Warehouse can be thought of as an automated version of the

Information Center that was widely popular in the mid-1980s or

even ultimately as the automation of Information Resource

Management. And while technologies such as client-server have

begun to put enormous computing and graphics power in the

hands of individuals, however, these technologies have not, in

general, provided the link to the operational data that end users

need to make critical business decisions.

Data Warehouse can be thought of as an automated version of the

Information Center that was widely popular in the mid-1980s or

even ultimately as the automation of Information Resource

Management. And while technologies such as client-server have

begun to put enormous computing and graphics power in the

hands of individuals, however, these technologies have not, in

general, provided the link to the operational data that end users

need to make critical business decisions.

M. Anvari Page 34

Data Warehouse RequirementsData Warehouse RequirementsData Warehouse RequirementsData Warehouse Requirements

Support for Universal Access to Multi-platform Data BasesSupport for Universal Access to Multi-platform Data Bases

Support for Multiple User Types Support for Multiple User Types

Separation of Operational and Informational ConcernsSeparation of Operational and Informational Concerns

Support for Networked DataSupport for Networked Data

Support for Directories, Repositories and Information Models, Support for Directories, Repositories and Information Models,

Support for Advanced End User InterfacesSupport for Advanced End User Interfaces

Support for Universal Access to Multi-platform Data BasesSupport for Universal Access to Multi-platform Data Bases

Support for Multiple User Types Support for Multiple User Types

Separation of Operational and Informational ConcernsSeparation of Operational and Informational Concerns

Support for Networked DataSupport for Networked Data

Support for Directories, Repositories and Information Models, Support for Directories, Repositories and Information Models,

Support for Advanced End User InterfacesSupport for Advanced End User Interfaces

M. Anvari Page 35

Access to Heterogeneous DataAccess to Heterogeneous Data

HP

IBMDEC

Compaq

M. Anvari Page 36

Multiple User Types Multiple User Types (Knowledge workers)Multiple User Types Multiple User Types (Knowledge workers)

Top ExecutivesTop Executives ManagersManagers AnalystsAnalysts PlannersPlanners Product DevelopersProduct Developers ConsultantsConsultants LawyersLawyers etc.etc.

Top ExecutivesTop Executives ManagersManagers AnalystsAnalysts PlannersPlanners Product DevelopersProduct Developers ConsultantsConsultants LawyersLawyers etc.etc.

M. Anvari Page 37

Separation of Operational and Separation of Operational and Informational ConcernsInformational Concerns

Separation of Operational and Separation of Operational and Informational ConcernsInformational Concerns

Operational SystemsOperational Systems Response TimeResponse Time

ReliabilityReliability

SecuritySecurity

RecoverabilityRecoverability

Informational SystemsInformational Systems Flexibility, Performance, Ease of NavigationFlexibility, Performance, Ease of Navigation

Large numbers of different viewsLarge numbers of different views

Manage Huge Amounts of Data (VLDBs)Manage Huge Amounts of Data (VLDBs)

Need to drill down/drill thru into data Need to drill down/drill thru into data

Need to draw on data from many sourcesNeed to draw on data from many sources

Operational SystemsOperational Systems Response TimeResponse Time

ReliabilityReliability

SecuritySecurity

RecoverabilityRecoverability

Informational SystemsInformational Systems Flexibility, Performance, Ease of NavigationFlexibility, Performance, Ease of Navigation

Large numbers of different viewsLarge numbers of different views

Manage Huge Amounts of Data (VLDBs)Manage Huge Amounts of Data (VLDBs)

Need to drill down/drill thru into data Need to drill down/drill thru into data

Need to draw on data from many sourcesNeed to draw on data from many sources

M. Anvari Page 38

Support for Networked DataSupport for Networked DataSupport for Networked DataSupport for Networked Data

All the data that is required to support informational needs is often not on the same operational data base. The need for Labor Negotiations, for example, may come from a variety of operational data bases, such as Manufacturing, Personnel, and Accounting.

Distributed Systems

All the data that is required to support informational needs is often not on the same operational data base. The need for Labor Negotiations, for example, may come from a variety of operational data bases, such as Manufacturing, Personnel, and Accounting.

Distributed Systems

M. Anvari Page 39

Support for Advanced End User Support for Advanced End User InterfacesInterfacesSupport for Advanced End User Support for Advanced End User InterfacesInterfaces

M. Anvari Page 40

Dimensions of Data WarehousingDimensions of Data WarehousingDimensions of Data WarehousingDimensions of Data Warehousing

PerformancePerformance

FlexibilityFlexibility

ScalabilityScalability

Ease ofEase ofUseUse

QualityQuality

Connection to Connection to the Operational Datathe Operational Data

Distributed DataDistributed Data

SecuritySecurity

M. Anvari Page 41

Enterprise Knowledge ArchitectureEnterprise Knowledge Architecture

for for

Data WarehousingData Warehousing

Enterprise Knowledge ArchitectureEnterprise Knowledge Architecture

for for

Data WarehousingData Warehousing

M. Anvari Page 42





M. Anvari Page 43


M. Anvari Page 44

Enterprise Network Computer Enterprise Network Computer ArchitectureArchitectureEnterprise Network Computer Enterprise Network Computer ArchitectureArchitecture

Data Directory(Repository)

InformationAccess

Data Warehouse

DataStaging

Application Messaging

OperationalDBs

Lake Er ie

L a ke O n ta ri o

PennsylvaniaCT

MA

NJ

DE

RI

MD

Maine

North

NHVT

New York

Virginia W

est

Virg

inia

Process Management

Data Directory Functions

External DBs

DataAccess

DataDataMartMart

M. Anvari Page 45

Freeing the “Data in Jail”Freeing the “Data in Jail”Freeing the “Data in Jail”Freeing the “Data in Jail”


InformationAccess

Data Warehouse

DataStaging


OperationalDBs

Lake Er ie

L a ke O n ta ri o

PennsylvaniaCT

MA

NJ

DE

RI

MD

Maine

North

NHVT

New York

Virginia W

est

Virg

inia

Process Management


External DBs

DataAccess

M. Anvari Page 46

The Information Access LayerThe Information Access LayerThe Information Access LayerThe Information Access Layer


InformationAccess

Data Warehouse

DataStaging


OperationalDBs

Lake Er ie

L a ke O n ta ri o

PennsylvaniaCT

MA

NJ

DE

RI

MD

Maine

North

NHVT

New York

Virginia W

est

Virg

inia

Process Management


External DBs

DataAccess

M. Anvari Page 47

The Legacy Data LayerThe Legacy Data LayerThe Legacy Data LayerThe Legacy Data Layer


InformationAccess

Data Warehouse

DataStaging


OperationalDBs

Lake Er ie

L a ke O n ta ri o

PennsylvaniaCT

MA

NJ

DE

RI

MD

Maine

North

NHVT

New York

Virginia W

est

Virg

inia

Process Management


External DBs

DataAccess

M. Anvari Page 48

The External Data LayerThe External Data LayerThe External Data LayerThe External Data Layer


InformationAccess

Data Warehouse

DataStaging


OperationalDBs

Lake Er ie

L a ke O n ta ri o

PennsylvaniaCT

MA

NJ

DE

RI

MD

Maine

North

NHVT

New York

Virginia W

est

Virg

inia

Process Management


External DBs

DataAccess

M. Anvari Page 49

The Data Access LayerThe Data Access LayerThe Data Access LayerThe Data Access Layer


InformationAccess

Data Warehouse

DataStaging


OperationalDBs

Lake Er ie

L a ke O n ta ri o

PennsylvaniaCT

MA

NJ

DE

RI

MD

Maine

North

NHVT

New York

Virginia W

est

Virg

inia

Process Management


External DBs

DataAccess

M. Anvari Page 50



InformationAccess

Data Warehouse

DataStaging


OperationalDBs

Lake Er ie

L a ke O n ta ri o

PennsylvaniaCT

MA

NJ

DE

RI

MD

Maine

North

NHVT

New York

Virginia W

est

Virg

inia

Process Management


External DBs

DataAccess

Data AccessData AccessFilterFilter

M. Anvari Page 51



InformationAccess

Data Warehouse

DataStaging


OperationalDBs

Lake Er ie

L a ke O n ta ri o

PennsylvaniaCT

MA

NJ

DE

RI

MD

Maine

North

NHVT

New York

Virginia W

est

Virg

inia

Process Management


External DBs

DataAccess

SQL QueriesSQL Queries

M. Anvari Page 52



InformationAccess

Data Warehouse

DataStaging


OperationalDBs

Lake Er ie

L a ke O n ta ri o

PennsylvaniaCT

MA

NJ

DE

RI

MD

Maine

North

NHVT

New York

Virginia W

est

Virg

inia

Process Management


External DBs

DataAccess

SQL QueriesSQL Queries

SQL AnswersSQL Answers

M. Anvari Page 53

Application MessagingApplication MessagingApplication MessagingApplication Messaging


InformationAccess

Data Warehouse

DataStaging


OperationalDBs

Lake Er ie

L a ke O n ta ri o

PennsylvaniaCT

MA

NJ

DE

RI

MD

Maine

North

NHVT

New York

Virginia W

est

Virg

inia

Process Management


External DBs

DataAccess

M. Anvari Page 54

The Meta-Data Repository LayerThe Meta-Data Repository LayerThe Meta-Data Repository LayerThe Meta-Data Repository Layer


InformationAccess

Data Warehouse

DataStaging


OperationalDBs

Lake Er ie

L a ke O n ta ri o

PennsylvaniaCT

MA

NJ

DE

RI

MD

Maine

North

NHVT

New York

Virginia W

est

Virg

inia

Process Management


External DBs

DataAccess

M. Anvari Page 55

The Process Management LayerThe Process Management LayerThe Process Management LayerThe Process Management Layer


InformationAccess

Data Warehouse

DataStaging


OperationalDBs

Lake Er ie

L a ke O n ta ri o

PennsylvaniaCT

MA

NJ

DE

RI

MD

Maine

North

NHVT

New York

Virginia W

est

Virg

inia

Process Management


External DBs

DataAccess

M. Anvari Page 56

The Core Data WarehouseThe Core Data WarehouseThe Core Data WarehouseThe Core Data Warehouse


InformationAccess

Data Warehouse

DataStaging


OperationalDBs

Lake Er ie

L a ke O n ta ri o

PennsylvaniaCT

MA

NJ

DE

RI

MD

Maine

North

NHVT

New York

Virginia W

est

Virg

inia

Process Management


External DBs

DataAccess

M. Anvari Page 57

Data Staging and QualityData Staging and QualityData Staging and QualityData Staging and Quality


InformationAccess

Data Warehouse

DataStaging


OperationalDBs

Lake Er ie

L a ke O n ta ri o

PennsylvaniaCT

MA

NJ

DE

RI

MD

Maine

North

NHVT

New York

Virginia W

est

Virg

inia

Process Management


External DBs

DataAccess

M. Anvari Page 58

Data Mart (Post-process/Indexing)Data Mart (Post-process/Indexing)Data Mart (Post-process/Indexing)Data Mart (Post-process/Indexing)


InformationAccess

Data Warehouse

DataStaging


OperationalDBs

Lake Er ie

L a ke O n ta ri o

PennsylvaniaCT

MA

NJ

DE

RI

MD

Maine

North

NHVT

New York

Virginia W

est

Virg

inia

Process Management


External DBs

DataAccess

Post-Proc.&Indexing

M. Anvari Page 59

Goals of WarehouseGoals of WarehouseGoals of WarehouseGoals of Warehouse

1. Performance (Canned queries, MD Analysis, Ad hoc, 1. Performance (Canned queries, MD Analysis, Ad hoc, Impact on Operational System)Impact on Operational System)

2. Flexibility (MD Flex, Ad hoc, Change data structure)2. Flexibility (MD Flex, Ad hoc, Change data structure)

3. Scalability (No. of Users, Volume of Data)3. Scalability (No. of Users, Volume of Data)

4. Ease of Use (Location, Formulation, Navigation, 4. Ease of Use (Location, Formulation, Navigation, Manipulation)Manipulation)

5. Data Quality (Consistent, Correct, Timely, Integrated)5. Data Quality (Consistent, Correct, Timely, Integrated)

6. Connection to the Detail Business Transactions 6. Connection to the Detail Business Transactions

1. Performance (Canned queries, MD Analysis, Ad hoc, 1. Performance (Canned queries, MD Analysis, Ad hoc, Impact on Operational System)Impact on Operational System)

2. Flexibility (MD Flex, Ad hoc, Change data structure)2. Flexibility (MD Flex, Ad hoc, Change data structure)

3. Scalability (No. of Users, Volume of Data)3. Scalability (No. of Users, Volume of Data)

4. Ease of Use (Location, Formulation, Navigation, 4. Ease of Use (Location, Formulation, Navigation, Manipulation)Manipulation)

5. Data Quality (Consistent, Correct, Timely, Integrated)5. Data Quality (Consistent, Correct, Timely, Integrated)

6. Connection to the Detail Business Transactions 6. Connection to the Detail Business Transactions

M. Anvari Page 60

Virtual WarehouseVirtual WarehouseVirtual WarehouseVirtual Warehouse


InformationAccess

Data Warehouse

DataStaging


OperationalDBs

Lake Er ie

L a ke O n ta ri o

PennsylvaniaCT

MA

NJ

DE

RI

MD

Maine

North

NHVT

New York

Virginia W

est

Virg

inia

Process Management


External DBs

DataAccess

M. Anvari Page 61



InformationAccess

Data Warehouse

DataStaging


OperationalDBs

Lake Er ie

L a ke O n ta ri o

PennsylvaniaCT

MA

NJ

DE

RI

MD

Maine

North

NHVT

New York

Virginia W

est

Virg

inia

Process Management


External DBs

DataAccess

M. Anvari Page 62



InformationAccess

Data Warehouse

DataStaging


OperationalDBs

Lake Er ie

L a ke O n ta ri o

PennsylvaniaCT

MA

NJ

DE

RI

MD

Maine

North

NHVT

New York

Virginia W

est

Virg

inia

Process Management


External DBs

DataAccess

A Virtual Data WarehouseA Virtual Data Warehouseapproach is often chosenapproach is often chosenwhen there are infrequent when there are infrequent demands for data and demands for data and management wants to management wants to determine if/how users will determine if/how users will use operational data.use operational data.

M. Anvari Page 63



InformationAccess

Data Warehouse

DataStaging


OperationalDBs

Lake Er ie

L a ke O n ta ri o

PennsylvaniaCT

MA

NJ

DE

RI

MD

Maine

North

NHVT

New York

Virginia W

est

Virg

inia

Process Management


External DBs

DataAccess

One of the weaknesses of One of the weaknesses of a Virtual Data Warehousea Virtual Data Warehouseapproach is that user approach is that user queries are made against queries are made against operational DBs.operational DBs.

One way to minimize this One way to minimize this problem is to build a problem is to build a “Query Monitor” to check “Query Monitor” to check the performance the performance characteristics of a query characteristics of a query before executing it.before executing it.

M. Anvari Page 64

Distributed Data WarehouseDistributed Data WarehouseDistributed Data WarehouseDistributed Data Warehouse


InformationAccess

Data Warehouse

DataStaging


OperationalDBs

Lake Er ie

L a ke O n ta ri o

PennsylvaniaCT

MA

NJ

DE

RI

MD

Maine

North

NHVT

New York

Virginia W

est

Virg

inia

Process Management


External DBs

DataAccess

M. Anvari Page 65

Distributed Data WarehouseDistributed Data WarehouseDistributed Data WarehouseDistributed Data Warehouse


InformationAccess

Data Warehouse

DataStaging


OperationalDBs

Lake Er ie

L a ke O n ta ri o

PennsylvaniaCT

MA

NJ

DE

RI

MD

Maine

North

NHVT

New York

Virginia W

est

Virg

inia

Process Management


External DBs

DataAccess

A Distributed Data A Distributed Data Warehouse is similar in most Warehouse is similar in most respects to a Central Data respects to a Central Data Warehouse, except that the Warehouse, except that the data is distributed to data is distributed to separate mini-Data separate mini-Data Warehouses (Data Marts )Warehouses (Data Marts )on local or specialized on local or specialized serversservers

M. Anvari Page 66

Information Access ToolsInformation Access ToolsInformation Access ToolsInformation Access Tools

Desktop DBsDesktop DBs

SpreadsheetsSpreadsheets

4GL/Desktop Query Tools4GL/Desktop Query Tools

Decision Support Systems (DSS)Decision Support Systems (DSS)

Multi-dimensional DBs (MDDs)Multi-dimensional DBs (MDDs)

OLAP (On-line Analytical ProcessingOLAP (On-line Analytical Processing

Executive Information Systems (EIS)Executive Information Systems (EIS)

Data Visualization Tools Data Visualization Tools

Data Mining ToolsData Mining Tools

Business Modeling and Simulation ToolsBusiness Modeling and Simulation Tools

Desktop DBsDesktop DBs

SpreadsheetsSpreadsheets

4GL/Desktop Query Tools4GL/Desktop Query Tools

Decision Support Systems (DSS)Decision Support Systems (DSS)

Multi-dimensional DBs (MDDs)Multi-dimensional DBs (MDDs)

OLAP (On-line Analytical ProcessingOLAP (On-line Analytical Processing

Executive Information Systems (EIS)Executive Information Systems (EIS)

Data Visualization Tools Data Visualization Tools

Data Mining ToolsData Mining Tools

Business Modeling and Simulation ToolsBusiness Modeling and Simulation Tools

M. Anvari Page 67


InformationAccess

Data Warehouse

DataStaging


OperationalDBs

Lake Er ie

L a ke O n ta ri o

PennsylvaniaCT

MA

NJ

DE

RI

MD

Maine

North

NHVT

New York

Virginia W

est

Virg

inia

Process Management


External DBs

DataAccess

Data Warehousing Tools and Data Warehousing Tools and TechnologyTechnologyData Warehousing Tools and Data Warehousing Tools and TechnologyTechnology

Desktop Data Bases:Desktop Data Bases:

•Structured for Database ManipulationStructured for Database Manipulation•Provides facility for selecting, andProvides facility for selecting, and loading of Desktop DBs from loading of Desktop DBs from Informational DBs Informational DBs•Provides ability to Create HighlyProvides ability to Create Highly “Personalized” Informational Systems “Personalized” Informational Systems

ExamplesExamples•AccessAccess•ParadoxParadox•dBase/FoxPro/ClipperdBase/FoxPro/Clipper

M. Anvari Page 68


InformationAccess

Data Warehouse

DataStaging


OperationalDBs

Lake Er ie

L a ke O n ta ri o

PennsylvaniaCT

MA

NJ

DE

RI

MD

Maine

North

NHVT

New York

Virginia W

est

Virg

inia

Process Management


External DBs

DataAccess


Spreadsheets:Spreadsheets:

•Structured to get any subset of Structured to get any subset of Information Information •Ability to Interface with standardAbility to Interface with standard Spreadsheet tools ( Spreadsheet tools (

ExamplesExamples• ExcelExcel• 1-2-31-2-3• Quatro ProQuatro Pro

M. Anvari Page 69


InformationAccess

Data Warehouse

DataStaging


OperationalDBs

Lake Er ie

L a ke O n ta ri o

PennsylvaniaCT

MA

NJ

DE

RI

MD

Maine

North

NHVT

New York

Virginia W

est

Virg

inia

Process Management


External DBs

DataAccess


Ad Hoc Query Systems:Ad Hoc Query Systems:

•Tailored for Flexible ReportingTailored for Flexible Reporting•Ability to do Sophisticated Analysis Ability to do Sophisticated Analysis Functions Functions•Aimed a a variety of users from casual toAimed a a variety of users from casual to the power user the power user

ExamplesExamples•Focus for Windows (IBI)Focus for Windows (IBI)•SASSAS•Business ObjectsBusiness Objects•GQL (Anadyne)GQL (Anadyne)•Esperant (Software AG)Esperant (Software AG)•Forrest & Trees (Platinum)Forrest & Trees (Platinum)•Visualizer (IBM)Visualizer (IBM)•Impromptu (Cognos)Impromptu (Cognos)•Beacon (Prodea)Beacon (Prodea)

M. Anvari Page 70


InformationAccess

Data Warehouse

DataStaging


OperationalDBs

Lake Er ie

L a ke O n ta ri o

PennsylvaniaCT

MA

NJ

DE

RI

MD

Maine

North

NHVT

New York

Virginia W

est

Virg

inia

Process Management


External DBs

DataAccess


Multi-dimensional Databases (MDDB)Multi-dimensional Databases (MDDB)OLAP (On-line analytical processing):OLAP (On-line analytical processing):

•Highly Structured DataHighly Structured Data•Tailored for Financial ModelingTailored for Financial Modeling•Tailored for “Power Users”Tailored for “Power Users”•Ability to do Sophisticated Ability to do Sophisticated Financial “What-if” Analysis Financial “What-if” Analysis•Ability to “drill-down” from high-level toAbility to “drill-down” from high-level to Detail Data Detail Data

ExamplesExamples• Acumate (Kenan Tech.)Acumate (Kenan Tech.)• Beacon (Prodea)Beacon (Prodea)• CrossTarget (Dimensional Insight) CrossTarget (Dimensional Insight) • eSSbase (Arbor)eSSbase (Arbor)• Oracle Express (Oracle)Oracle Express (Oracle)

M. Anvari Page 71


InformationAccess

Data Warehouse

DataStaging


OperationalDBs

Lake Er ie

L a ke O n ta ri o

PennsylvaniaCT

MA

NJ

DE

RI

MD

Maine

North

NHVT

New York

Virginia W

est

Virg

inia

Process Management


External DBs

DataAccess


Executive Information Systems (EIS):Executive Information Systems (EIS):

•Highly Structured DataHighly Structured Data•Tailored for Non-technical UsersTailored for Non-technical Users•Ability to “slice and dice” dataAbility to “slice and dice” data•Ability to “drill-down”Ability to “drill-down”

ExamplesExamples• Commander OLAP ServerCommander OLAP Server• Pilot (Lightship)Pilot (Lightship)• VBVB• PowerbuilderPowerbuilder

M. Anvari Page 72


InformationAccess

Data Warehouse

DataStaging


OperationalDBs

Lake Er ie

L a ke O n ta ri o

PennsylvaniaCT

MA

NJ

DE

RI

MD

Maine

North

NHVT

New York

Virginia W

est

Virg

inia

Process Management


External DBs

DataAccess


Data Visualization:Data Visualization:

• Automatic Categorization Automatic Categorization • Visualization of Multi-dimensional dataVisualization of Multi-dimensional data• Automatic Analysis and/or IndexingAutomatic Analysis and/or Indexing

ExamplesExamples• WinViz (IBI)WinViz (IBI)• dbExpress (Computer Concepts)dbExpress (Computer Concepts)• Data Explorer (IBM)Data Explorer (IBM)• ARC Info/ARC ViewARC Info/ARC View• Strategic MappingStrategic Mapping

M. Anvari Page 73


InformationAccess

Data Warehouse

DataStaging


OperationalDBs

Lake Er ie

L a ke O n ta ri o

PennsylvaniaCT

MA

NJ

DE

RI

MD

Maine

North

NHVT

New York

Virginia W

est

Virg

inia

Process Management


External DBs

DataAccess


Data Mining:Data Mining:

•High Speed Analysis of Detail DataHigh Speed Analysis of Detail Data•Constructs Business PatternsConstructs Business Patterns•Provides Statistical SupportProvides Statistical Support

ExamplesExamples• IBM beta-testIBM beta-test• Information HarvesterInformation Harvester• IDISIDIS• d.b.Expressd.b.Express• DataMindDataMind

M. Anvari Page 74


InformationAccess

Data Warehouse

DataStaging


OperationalDBs

Lake Er ie

L a ke O n ta ri o

PennsylvaniaCT

MA

NJ

DE

RI

MD

Maine

North

NHVT

New York

Virginia W

est

Virg

inia

Process Management


External DBs

DataAccess


Business Modeling and Simulation:Business Modeling and Simulation:

•Business Feedback ModelBusiness Feedback Model•Direct ManipulationDirect Manipulation•Business GamingBusiness Gaming•Management/Operations TrainingManagement/Operations Training

ExamplesExamples• SimRefinerySimRefinery• SimTelephoneSimTelephone• iThinkiThink• MicroworldsMicroworlds

M. Anvari Page 75

3. Meta-data Repository Layer3. Meta-data Repository Layer3. Meta-data Repository Layer3. Meta-data Repository Layer


InformationAccess

Data Warehouse

DataStaging


OperationalDBs

Lake Er ie

L a ke O n ta ri o

PennsylvaniaCT

MA

NJ

DE

RI

MD

Maine

North

NHVT

New York

Virginia W

est

Virg

inia

Process Management


External DBs

DataAccess

Data Dictionary/Data Dictionary/RepositoryRepository

• Meta-data ModelingMeta-data Modeling• Meta-data UpdatingMeta-data Updating• Meta-dataMeta-data

ExamplesExamples o Platinumo Platinum o Rochadeo Rochade o MSPo MSP o Data Atlas (IBM)o Data Atlas (IBM) o MS/TIo MS/TI

M. Anvari Page 76

3. Process (Systems) Management3. Process (Systems) Management3. Process (Systems) Management3. Process (Systems) Management


InformationAccess

Data Warehouse

DataStaging


OperationalDBs

Lake Er ie

L a ke O n ta ri o

PennsylvaniaCT

MA

NJ

DE

RI

MD

Maine

North

NHVT

New York

Virginia W

est

Virg

inia

Process Management


External DBs

DataAccess

Process Process ManagementManagement

• SchedulingScheduling• ExecutionExecution• SubscriptionSubscription

ExamplesExamples o Data Harvestero Data Harvester o Data Hubo Data Hub o Detect and Alerto Detect and Alert(Comshare)(Comshare)

M. Anvari Page 77

3. Post-processing/Indexing Layer3. Post-processing/Indexing Layer3. Post-processing/Indexing Layer3. Post-processing/Indexing Layer


InformationAccess

Data Warehouse

DataStaging


OperationalDBs

Lake Er ie

L a ke O n ta ri o

PennsylvaniaCT

MA

NJ

DE

RI

MD

Maine

North

NHVT

New York

Virginia W

est

Virg

inia

Process Management


External DBs

DataAccess

Post-processing/Post-processing/IndexingIndexing

ExamplesExamples•Sybase IQ AcceleratorSybase IQ Accelerator•OMNIdexOMNIdex•Oracle 7.3Oracle 7.3•eSSbaseeSSbase•IRI ExpressIRI Express

Documents

An Introduction to Data Warehousing Concept and Technology Mort Anvari