Upload
erik-fransen
View
1.281
Download
0
Tags:
Embed Size (px)
DESCRIPTION
Citation preview
1
CDM
Centennium Datawarehouse MethodologyStructure – Model – Generate – Knowledge PartnershipErik FransenAntoine StelmaOctober 6, 2011
2
Agenda
• Centennium BI expertisehuis• CDM
– Structure Reference Architecture– Model Data Vault– Generate Template Based
Development– Knowledge
Partnership Knowledge transfer, coaching
• CDM process
3
Centennium BI expertisehuis• Independent • We provide our customers with the right skills, at the
right time and the right way• Consultancy, ad-interim support, project
implementation and training services, but primarily acts as a BI/DWH knowledge partner for its clients
• We take or share responsibility for the execution and management of BI and DWH projects and provide second or third line support for existing implementations
• We offer an extensive portfolio of courses and training services
• We provide our customers with the knowledge and practical insights required to be(come) self-sufficient in maintaining and expanding their BI-environments
www.centennium.nl
4
Centennium Datawarehouse Methodology
• A methodology, including DWH-tools• Build or migrate datawarehouses fast, with high quality
and low cost• Automatic generation of datawarehouse based on
descriptive metadata• CDM includes:
– Template Based Development– Best practices– Quality control mechanism– Knowledge partnership
• Best Practices: Data Vault, Kimball, Reference Architecture• Quality control mechanism: extensive checklists and
documentation• Knowledge transfer by training, certification and learning
on-the-job
5 ModellingStructuring Generating
Knowledge Partnership
Centennium Datawarehouse Methodology
6
Knowledge partnership
• Goal: Self-supporting, CDM-aware customers• Training options
– BI and DWH fundamentals– CDM designer / TBD developer training– Data Vault fundamentals– Data Vault certification (Genesee Academy)– Dimensional modelling– More info on www.bi-opleidingen.nl
• On-the-job coaching, learning by doing• Centennium supports it’s customers by
(management) consultancy, assessments, projects, training and sourcing
7
Reference Architecture: Structuring
8
Template Based Development:Generating
• TBD generates all ETL and objects for– Registration layer– Presentation/Publication layer
• Repository and scripts are free of charge
• Creating the staging layer is not part of TBD but can be automated per customer
9
TBD databases
Central Data Warehouse
(CDW)
Staging (STA)
Publication
(PUB)
Repository (REP)
10
TBD repository
Central Data Warehouse
(CDW)
Staging (STA)
Publication
(PUB)
Repository (REP)
11
Functionality TBD repositorylog tables (1)
• Functionality: – Registration of all ETL processes– Insight in data logistics– Historic profile of loading processes
• Goal:– Traceability – Maintenance: localisation and identification
of all data/transactions (wrong and correct)– Linkage to source systems
PUBCDWSTA
REP
12
Functionality TBD repositorytemplate tables
• Functionality: – Central storage of template scripts for
generating tables and ETL procedures– Easy promotion to other environments
(OTAP)– Transparant
• Goal: – Version management of templates– Object (re-)creation and maintenance
PUBCDWSTA
REP
13
Functionality TBD repositorymeta data tables
• Functionality: – Defines registration and publication layer– Contains object and ETL definitions– Contains data logistics
• Goal:– Generating:
• Registration layer (Data Vault tables)• Publication layer (Star schemes) • ETL processes via ‘stored procedures’
– Object creation and maintenance
PUBCDWSTA
REP
14
Functionality TBD repositorymeta data tables
• Table ‘Repos_Table’: – Contains object names for
• Staging • Registration• Publication
– Table is filled by an application or Excel sheet
• Table ‘Repos_Mapping’: – Contains mapping of staging, registration and
presentation– Table is filled by an application or Excel sheet
PUBCDWSTA
REP
15
Registration layer
Central Data Warehouse
(CDW)
Staging (STA)
Publication
(PUB)
Repository (REP)
16
Registration layer
• Script ‘CDW 01a table hub’– Generates all hub tables in the CDW, as
defined in repository– Every hub contains the columns:
• id (primary key), <afkorting>_key, meta_laad_dts, meta_bron en meta_audit_id
– Variables from repository are applied– Once the hubs are generated, the
repository is updated– Hubs generated according to Data Vault
Standards!
PUBCDWSTA
REP
17
Registration layer
• Script ‘CDW 01a tabel hub’– Generates all hub tables in the CDW, as
defined in repository– Every hub contains the columns:
• id (primary key), <afkorting>_key, meta_laad_dts, meta_bron en meta_audit_id
– Variables from repository are applied– Once the hubs are generated, the
repository is updated– Hubs generated according to Data Vault
Standards!
PUBCDWSTA
REP
18
Registration layer
• Script ‘CDW 02a table sat’– Generates all satellite tables in the CDW, as
defined in repository– Every satellite contains the the columns:
• h_id (foreign key), meta_laad_dts, meta_laad_eind_dts, meta_bron en meta_audit_id
– Variables from repository are applied– Once the hubs are generated, the
repository is updated– Sats generated according to Data Vault
Standards!
PUBCDWSTA
REP
19
Registration layer
• Script ‘CDW 02a tabel sat’– Generates all satellite tables in the CDW, as
defined in repository– Every satellite contains the the columns:
• h_id (foreign key), meta_laad_dts, meta_laad_eind_dts, meta_bron en meta_audit_id
– Variables from repository are applied– Once the hubs are generated, the
repository is updated– Sats generated according to Data Vault
Standards!
PUBCDWSTA
REP
20
Registration layer
• Script ‘CDW 03a table link’– Generates all link tables in the CDW, as
defined in repository– Every link is connected to two or more hubs– Variables from repository are applied– Once the links are generated, the
repository is updated– Links generated according to Data Vault
Standards!
PUBCDWSTA
REP
21
Registration layer
• Script ‘CDW 03a table link’– Generates all link tables in the CDW, as
defined in repository– Every link is connected to two or more hubs– Variables from repository are applied– Once the links are generated, the
repository is updated– Links generated according to Data Vault
Standards!
PUBCDWSTA
REP
22
Registration layer
• Script ‘CDW 04a table link satellite’– Generates all link satellite tables in the
CDW, as defined in repository– Variables from repository are applied– Once the link satellites are generated, the
repository is updated– Link satellites are generated according to
Data Vault Standards!
PUBCDWSTA
REP
23
Registration layer
• Script ‘CDW 04a table link satellite’– Generates all link satellite tables in the
CDW, as defined in repository– Variables from repository are applied– Once the link satellites are generated, the
repository is updated– Link satellites are generated according to
Data Vault Standards!
PUBCDWSTA
REP
24
Publication layer
Central Data Warehouse
(CDW)
Staging (STA)
Publication
(PUB)
Repository (REP)
25
Publication layerdimensions and facts
• Fed only and directly from CDW• Conforms to Kimball’s star scheme
standards– Or any other format
• Business rules can be applied– Currently using views– Business rule editor in next release
• Type SCD I, II, etc.• Conformed dimensions when needed
PUBCDWSTA
REP
26
CDM: process• Incremental approach• Timeboxes of 4-6 weeks
100% Centennium90% Centennium 30% Customer
70% Customer40% Centennium
100% Customer10% Centennium
Phase 1 Phase 2 Phase 3 Phase 4
Increment
1-8-2011 31-12-2011
CDM: typical increment
CDM-trainin
g
Data Vault
Certification
Training and Coaching on-the-job
Supporting customer
• Typical increment ranges from 2 to 6 months• Centennium role changes from LEAD to FOLLOW• Customer is fully CDM-aware at the end of the increment• Centennium continues supporting customers through
knowledge partnership
Q & A
28
29
ABOUT CENTENNIUM
Centennium BI expertisehuis houses all the experts under one roof, hereby offering all knowledge and expertise to address the complex business intelligence issues facing our clients today
Services overview:• Consultancy• Projects • Resourcing • Education
Some of our clients: Woonbron, Albron, NZa, CAK, OBR, Vopak, several Dutch Municipalities, Aegon, Nutreco, TNO, Genzyme, Tata Steel, KPN, DELTA, IKEA, Accell, TomTom, KAS BANK, LeasePlan, Brabant Water
Facts and figures:• Founded: 1998• 45+ business intelligence
consultants
Core values:• Human Capital• In close collaboration• Objective and Independent
Expertise:• Business intelligence• Strategic, tactic and operational• Vision based on “effective BI”
Certification and Industry Recognition
All Centennium consultants are Certified BI Professionals (CBIP) awarded by TDWI.
CBIP, backed by TDWI's reputation as the leader in BI education and research, makes a strong statement that we are among the leaders in the industry—up to date with BI technologies, knowledgeable about best practices and state-of-the-art solutions, and informed of emerging trends
All Centennium Data Vault specialists and architects are certified on Data Vault modeling by Dan Linstedt, Data Warehouse industry thought leader and founder of the Data Vault modeling standard.Centennium is the business partner of Genesee Academy in organizing the Data Vault Modeling & Certification – Europe Course. We are accredited to perform the Genesee Academy Data Vault trainings.
Centennium’s customer Nutreco was awarded the world’s first Certified Data Vault Data Model in 2009 for their corporate financial and procurement data warehouse model.
32
Centennium BI expertisehuisLange Voorhout 432514 EC 's-Gravenhage Telefoon 070 31 20 370Fax 070 31 20 371URL www.centennium.nl
www.bi-opleidingen.nlwww.data-vault.nl