Data Vault Modeling - NYOUGnyoug.org/Presentations/2010/December/Govindarajan_Data_Vault.pdf ·...

Data Vault ModelingThe Next Generation DW Approach

http://www.globytes.comhttp://www.globytes.comhttp://www.globytes.comhttp://www.globytes.com

siva02@globytes.comsiva02@globytes.comsiva02@globytes.comsiva02@globytes.com

Presented by: Siva Govindarajan

� Introduction

� Data Vault Place in Evolution

� Data Warehousing Architecture

� Data Vault Components

� Case Study� Typical 3NF / Star Schema

� Data Vault Approach

� Temporal Data + Auditability

� Implementation of Data Vault

� Conclusion

Data Vault Modeling

Agenda

� Data Vault concept originally conceived

by Dan Linstedt

� Enterprise Data Warehouse Modeling

approach.

� Hybrid approach - 3NF & Star Schema

� Adaptable to changes

� Auditable

� Timed right with Technology

Introduction

� 1960s - Codd, Date et. al Normal Forms

� 1980s - Normal Forms adapted to DWs

� 1985+ - Star Schema for OLAP

� 1990s - Data Vault concept developed

Dan Linstedt

� 2000+ - Data Vault concept published by

Dan Linstedt

Data Vault Place in Evolution

Data Warehousing Architecture

Bill Inmon’s Data Warehouse Architecture

Data Mart

Data Marts

Data Mart

Data MartsEnterprise

Staging(optional)

SystemsStaging

Adaptive

Kimball’s Data Warehouse Bus Architecture

Data Mart

Systems Staging

Data Mart

� Hub Entities

� Candidate Keys + Load Time + Source

� Link Entities

� FKs from Hub + Load Time + Source

� Satellite Entities

� Descriptive Data + Load Time + Source + End Time

� Dimension = Hub+ Satellite

� Fact = Satellite + Link [+ Hub ]

Data Vault Components

� Modeling Approaches

� Typical 3NF / Star Schema

� Data Vault approach

� Scenario

� Typical Fact and Dimension

� New Dimension data

� Reference Data change - 1:M to M:M

� Additional attributes to Dimension

Case Study

Typical 3 F / Star Schema

TR ORDER

tr_order_id

dim_product_id

system order id

system root order id

global root order id

order quantity

total executed qty

event timestamp

DIM PRODUCT

dim_product_id

product_id

product_code

product desc

product_name

leng thleng thleng thleng th

1. Typical Fact and Dimension

DIM PRODUCT

dim_product_id

product_id

product_code

product desc

product_name

product_ca tego ry_idp roduct_ca tego ry_idp roduct_ca tego ry_idp roduct_ca tego ry_id

p roduct_ca tego ry_nameproduct_ca tego ry_nameproduct_ca tego ry_nameproduct_ca tego ry_name

product_ca tego ry_codeproduct_ca tego ry_codeproduct_ca tego ry_codeproduct_ca tego ry_code

product_ca tego ry_descp roduct_ca tego ry_descp roduct_ca tego ry_descp roduct_ca tego ry_desc

is_stra tegyis_stra tegyis_stra tegyis_stra tegy

TR ORDER

tr_order_id

dim_product_id

system order id

order quantity

total executed qty

event timestamp

REF PRODUCT CATEGORYREF PRODUCT CATEGORYREF PRODUCT CATEGORYREF PRODUCT CATEGORY

product_ca tego ry_codeproduct_ca tego ry_codeproduct_ca tego ry_codeproduct_ca tego ry_code

product_ca tego ry_descp roduct_ca tego ry_descp roduct_ca tego ry_descp roduct_ca tego ry_desc

2. ew Dimension Data

DIM PRODUCT

dim_product_id

product_id

product_code

product desc

product_name

TR ORDER

tr_order_id

dim_product_id

system order id

order quantity

total executed qty

event timestamp

d im_product_ca tego ry_idd im_product_ca tego ry_idd im_product_ca tego ry_idd im_product_ca tego ry_id

product_ca tego ry_codep roduct_ca tego ry_codep roduct_ca tego ry_codep roduct_ca tego ry_code

p roduct_ca tego ry_descp roduct_ca tego ry_descp roduct_ca tego ry_descp roduct_ca tego ry_desc

DIM PRODUCT CATEGORYDIM PRODUCT CATEGORYDIM PRODUCT CATEGORYDIM PRODUCT CATEGORY

product_category_code

product_category_name

product_category_desc

is_strategy

namespace

DIM PRODUCT

dim_product_id

product_id

product_code

product desc

product_name

TR ORDER

tr_order_id

dim_product_id

system order id

order quantity

total executed qty

event timestamp

order owner firm id

product_ca tego ry_codep roduct_ca tego ry_codep roduct_ca tego ry_codep roduct_ca tego ry_code

p roduct_ca tego ry_descp roduct_ca tego ry_descp roduct_ca tego ry_descp roduct_ca tego ry_desc

DIM PRODUCT CATEGORYDIM PRODUCT CATEGORYDIM PRODUCT CATEGORYDIM PRODUCT CATEGORY

is_strategy

namespace

3. Reference 1:M to M:M Change

DIM PRODUCT

dim_product_id

product_id

product_code

product desc

product_name

he ighthe ighthe ighthe ight

we ightwe ightwe ightwe ight

packag ingpackag ingpackag ingpackag ing

co lo rco lo rco lo rco lo r

s izes izes izes ize

TR ORDER

tr_order_id

dim_product_id

system order id

order quantity

total executed qty

event timestamp

d im_p roduct_ca tegory_idd im_p roduct_ca tegory_idd im_p roduct_ca tegory_idd im_p roduct_ca tegory_id

order owner firm id

4. Additional attributes to Dimension

Data Vault Approach

1. Typical Hub, Satellite and Link(Replacing Fact/Dimension)

HUB_PRODUCT

hub_p roduc t_idhub_p roduc t_idhub_p roduc t_idhub_p roduc t_id

hub_reco rd_sourcehub_reco rd_sourcehub_reco rd_sourcehub_reco rd_source

hub_load_da tehub_load_da tehub_load_da tehub_load_da te

product_id

product_code

SAT_PRODUCT01SAT_PRODUCT01SAT_PRODUCT01SAT_PRODUCT01

sa t_produc t01_idsa t_produc t01_idsa t_produc t01_idsa t_produc t01_id

hub_p roduct_idhub_p roduct_idhub_p roduct_idhub_p roduct_id

sa t_load_da tesa t_load_da tesa t_load_da tesa t_load_da te

sa t_end_da tesa t_end_da tesa t_end_da tesa t_end_da te

sa t_reco rd_sourcesa t_reco rd_sourcesa t_reco rd_sourcesa t_reco rd_source

product desc

product_name

symbol

SAT ORDER01SAT ORDER01SAT ORDER01SAT ORDER01

sa t_o rder01_idsa t_o rder01_idsa t_o rder01_idsa t_o rder01_id

hub_o rde r_idhub_o rde r_idhub_o rde r_idhub_o rde r_id

order quantity

total executed qty

event timestamp

LNK_ORDER

lnk_o rde r_idlnk_o rde r_idlnk_o rde r_idlnk_o rde r_id

lnk_load_da telnk_load_da telnk_load_da telnk_load_da te

lnk_reco rd_sourcelnk_reco rd_sourcelnk_reco rd_sourcelnk_reco rd_source

hub_p roduc t_idhub_p roduc t_idhub_p roduc t_idhub_p roduc t_id

HUB_ORDER

tr_order_id

system order id

Data Vault Approach

HUB_PRODUCT

hub_product_idhub_product_idhub_product_idhub_product_id

product_id

product_code

SAT_PRODUCT01

sa t_p roduct01_idsa t_p roduct01_idsa t_p roduct01_idsa t_p roduct01_id

product desc

product_name

symbol

SAT ORDER01

sa t_o rde r01_idsa t_o rde r01_idsa t_o rde r01_idsa t_o rde r01_id

order quantity

total executed qty

event timestamp

LNK_ORDER

lnk_orde r_idlnk_o rde r_idlnk_o rde r_idlnk_o rde r_id

hub_orde r_idhub_orde r_idhub_orde r_idhub_orde r_id

HUB_ORDER

tr_order_id

system order id

LNK_PRODUCT_X_CATEGORYLNK_PRODUCT_X_CATEGORYLNK_PRODUCT_X_CATEGORYLNK_PRODUCT_X_CATEGORY

lnk_p roduct_x_ca tego ry_idlnk_p roduct_x_ca tego ry_idlnk_p roduct_x_ca tego ry_idlnk_p roduct_x_ca tego ry_id

hub_ca tego ry_idhub_ca tego ry_idhub_ca tego ry_idhub_ca tego ry_id

SAT_CATEGORY01SAT_CATEGORY01SAT_CATEGORY01SAT_CATEGORY01

sa t_ca tego ry01_idsa t_ca tego ry01_idsa t_ca tego ry01_idsa t_ca tego ry01_id

is_strategy

HUB_CATEGORYHUB_CATEGORYHUB_CATEGORYHUB_CATEGORY

product_category_id

Data Vault Approach

3. Reference 1:M to M:M Change

HUB_PRODUCT

product_id

product_code

SAT_PRODUCT01

product desc

product_name

symbol

SAT ORDER01

order quantity

total executed qty

event timestamp

LNK_ORDER

lnk_orde r_idlnk_o rde r_idlnk_o rde r_idlnk_o rde r_id

HUB_ORDER

tr_order_id

system order id

LNK_PRODUCT_X_CATEGORYLNK_PRODUCT_X_CATEGORYLNK_PRODUCT_X_CATEGORYLNK_PRODUCT_X_CATEGORY

lnk_p roduct_x_ca tego ry_idlnk_p roduct_x_ca tego ry_idlnk_p roduct_x_ca tego ry_idlnk_p roduct_x_ca tego ry_id

is_strategy

HUB_CATEGORYHUB_CATEGORYHUB_CATEGORYHUB_CATEGORY

product_category_id

Data Vault Approach

4. Additional attributes to Dimension

HUB_PRODUCT

product_id

product_code

SAT_PRODUCT01

product desc

product_name

length

height

weight

packaging

Temporal Data + Auditability

HUB_PRODUCT

product_id

product_code

product desc

product_name

LNK_PRODUCT_X_CATEGORY

lnk_product_x_ca tego ry_idlnk_product_x_ca tego ry_idlnk_product_x_ca tego ry_idlnk_product_x_ca tego ry_id

hub_produc t_idhub_produc t_idhub_produc t_idhub_produc t_id

HUB_CATEGORY

hub_ca tegory_idhub_ca tegory_idhub_ca tegory_idhub_ca tegory_id

product_category_id

is_strategy

length

height

weight

packaging

SAT ORDER01SAT ORDER01SAT ORDER01SAT ORDER01

order quantity

total executed qty

event timestamp

source_system

LNK_ORDER

lnk_o rde r_idlnk_o rde r_idlnk_o rde r_idlnk_o rde r_id

HUB_ORDER

tr_order_id

system order id

� Make it Simple

� View for Dimension

� View For Fact

� Data Loading

� Take advantage of Technology

� DW Appliances

� High-throughput storage devices

� RDBMS Features

Implementation of Data Vault

� Create views for Dimension

From our Demo model, Join Product related Hubs and

Satellites to build View for Product Dimension as shown

below :

�HUB_PRODUCT

�SAT_PRODUCT01

�SAT_PRODUCT02

Make it Simple

� Create Views For Fact

Join Order related Hubs, Satellites and Links to build View for ORDER Fact using ORDER related Hubs/ Satellites and Links.

�HUB_ORDER

�SAT_ORDER01

�LNK_ORDER

Make it Simple

Typically data loading for a Data Vault is in the following sequence.

� Hubs for Dimensions

� Links for Dimensions

� Satellites for Dimensions

� Hubs for Fact (if any )

� Links for Fact

� Satellites for Fact

Refer to Data Vault Series article 5 of Linstedt for further details

Data Loading

Next wave of IT Data Warehouse hardware solutions are

the Data Warehouse Appliances. Take advantage of the

technology where possible. Few DW Appliances in the

market :

� Oracle Exadata

� Netezza

� Teradata

� RedBrick

� GreenPlum

DW Appliances

Utilize Storage devices designed for DW / OLAP

applications. Following are some examples only and

would change over time. Please do your research based

on your sizing requirements.

� EMC CLARiiON Storage family

� Texas Memory systems RamSan – Solid State

devices

� Hitachi USP / VSP Storage family

� Other Hybrid solutions with High performance

storage and Solid State devices

High-throughput storage devices

Some Oracle Features catering DW / OLAP

applications:

� Exadata Smart Scans

� OLAP Based Materialized views

� Partitioning Reference partition

� Advanced data compression

� Automatic degree of parallelism (ADOP).

� Star query optimization

� Oracle OLAP/DWH Features.

RDBMS Features

ConclusionIn this presentation we have addressed high level overview of Data

Vault Architecture. Topics discussed are

�Data Vault concept and architecture

�Data Vault Components such as Hubs, Satellites and Link

tables

�Typical modeling challenges with traditional modeling

approaches

�How those challenges could be handled using Data Vault

Modeling Approach.

�Auditing and Temporal data capture using DV Approach.

�And finally, some implementation details

If interested in learning more, try

Dan Linstedt's special coaching area at:

http://danLinstedt.com/my-coach.

Questions ?

References

� Data Vault Series articles by Dan E. Linstedt --

http://www.tdan.com

� Referred few articles from Genesee Academy --

http://geneseeacademy.com

� Articles and books by Ralph Kimball and Bill Inmon

from multiple sources.

� Technical Documents from

http://www.oracle.com/technetwork

Special Thanks to

� Dan Linstedt for review and corrections to the article.

� My colleague Mark Bruscke for his assistance in preparing

the article.

The End

Contact Email :

siva02@globytes.com

Data Vault Modeling - NYOUGnyoug.org/Presentations/2010/December/Govindarajan_Data_Vault.pdf ·...

Documents

Data Warehouse Modeling Data Vaultdatabaser.net/moniwiki/pds/DataVaultModeling/QOSQO_Data_Vault... · What is Data Vault? Definitions: “The Data Vault is a detail oriented, historical

Introduction to Data Vault Modeling - · PDF fileData Vault Modeling is a specific data modeling technique for designing highly flexible, scalable, and adaptable data structures for

Data Sheet | Model: Armorgard StrimmerSafe Vault SSV6 Vault Dat… · Data Sheet | Model: Armorgard StrimmerSafe Vault SSV6 . Product Armorgard StrimmerSafe Vault SSV6 Internal dimensions

Data vault modeling et retour d'expérience

Data Vault Ein Leben zwischen 3NF und Star - oracledwh.de · 3 Data Vault - Modellierungsbeispiele 2 Data Vault - Grundlagen in der Modellierung 5 5 Vorteile von Data Vault 4 Schnellere

Data Vault Modeling & Methodology - 1105 Media: Home ...download.101com.com/pub/tdwi/files/DV_Presentation_TDWI Boston_Data... · Staging Area EDW – Data Vault Stage Load STAGING

WhereScape and Data Vault 2 · Data Sheet Data Vault modeling provides long-term historical storage of data coming in from multiple operational

DATA VAULT MODELING GUIDE - The Hans Blog | DATA ... · PDF fileDATA VAULT MODELING GUIDE ... and changes to operational latency requirements. For this reason, the EDW itself is not

Metadaten und Data Vault (Meta Vault)

DATA VAULT MODELING GUIDE - The Hans Blog · Data Vault modeling is most compelling when applied to an enterprise data warehouse program (EDW). Several key decisions concerning the

DATA VAULT - TUMI...DATA VAULT MICRO DISQUE DUR DISCO DURO DATA VAULT DATA VAULT FESTpLATTE トゥミ・データ・ボールト数据保险库 Tumi is the leading international business,

Data Vault Consortium A Mathematical Perspective of Data Vault

Govindarajan Data Vault

DATA VAULT MODELING GUIDE - DATA WAREHOUSING, DATA …€¦ · 4/21/2015. The EDW Program . The Enterprise Data Warehousing ( EDW) Program represents the ongoing data warehousing

Data Vault Overview

Data Vault and Anchor Modeling - Percona

Data Vault and Data Virtualization: Double Agility · The SuperNova Modeling Technique – Applying the data modeling technique called SuperNova results in a set of views defined

Data Warehousing Data Warehouse Design & Multi-dimensional ... · Data Vault Modeling The Data Vault technique has been introduced in the 1990s Today it is used in many DWH projects

(OTW13) Agile Data Warehousing: Introduction to Data Vault Modeling

Why Data Vault?