The Big Green Thingy – A Case Study in Data Warehousing

The Big Green Thingy – A Case Study in Data Warehousing

Allison Lobato, DBAEnterprise Data Warehouse

Department of Technology ServicesDenver Public SchoolsDenver, Colorado

Agenda

DPS and the CIF Current Environment Overview

Staffing Hardware Architecture Software Architecture

Design, Development and Deployment Architecture

Installation and Configuration Notes The Big Green Thingy Overview Conclusion Questions

My Bio

Allison LobatoDatabase Administrator – 21 years w/

DPS in IT Applications Development and DBA Support

Standard Survey

Who are you? DBAs Data Warehouse Designers Data Warehouse Architects Managers

Experience Warehouse Builder (OWB), Discoverer, Designer, Reports Oracle 9iAS or Portal Data Warehousing

Less than 1 yr?1-3 yrs?Over 3 years?

DPS, Data Warehousing and the CIF

DPS has no shortage of data – inconsistent, disjointed and disparate

DPS knows value and importance of getting our hands around the data

Using the Corporate Information Factory (CIF) conceptual architecture Developed by Bill Inmon and Claudia

Imhoff

DPS District Information Factory

DPS, Data Warehousing and the CIF

Current EnvironmentCIF is a long term architecture strategy for

DPS (a.k.a. District Information Factory-DIF)Pilot mission:

Prove the value to get funding!Deliver current student profile information

Using an Operational Data Store (ODS)Student profile subject area

Target audience 1 high school1 middle school1 elementary school1 administrative department

DPS, Data Warehousing and the CIF

Current Environment (continued)Enterprise Data Warehouse is an unfunded

projectUsing existing IT resources

Hardware – existing servers and disk spaceStaff

3 full-time (re-allocated)Data warehouse architect (Supervisor)DBAETL programmer

2 part-time (shared)DBA & iAS administrator

Hardware Architecture

Three-tiered approachEnd User Layer (tier 1)

Workstations

Middle Layer (tier 2)Web and application services

Database Layer (tier 3) Database and storage services

Hardware Architecture

The Workstations

For developers’ machines: robust PCs are a must. They need lots of memory (>512 Mb)Fast processors

End users machines: they are easierBrowser capable desktop running a

supported browser version Macs and PCs

Middle Tier

9iAS – Web and Application ServicesProduction Server

Dell PowerEdge 2650– 4 Gb Memory2 – 36 Gb Mirrored Hard Drives1 – 36 Gb Hot Spare Hard DriveWindows 2000 Advance Server with SP3

Development ServerDell PowerEdge 2500 – 4 Gb Memory2 – 18 Gb Mirrored Hard DrivesWindows 2000 Advance Server with SP3

Database Servers and Storage Devices

Database ServersHP’s RP8400 class server

Production ODS (64-bit)HP’s RP7410 class server

Production repositories & development ODSHP-UX 11.11

Storage DeviceEMC Symmetric 8430 Using less than 150GB currently

Estimated need over 400GB

Software Architecture

End-to-end Oracle solution (Oracle 9iDS , 9iAS & RDBMS) on all 3 tiers (workstation, web, database) RDBMS Designer Warehouse Builder (OWB) Discoverer Reports Portal Single Sign On (SSO) Oracle Internet Directory (OID) Enterprise Manager (OEM and WebOEM) Workflow (Job scheduling and monitoring)

Version compatibility was key (and constantly changing)

Oracle Software Tool Versions

Tool Name ClientVersion

RepositoryVersion

DatabaseVersion

Server Type

Designer 9.0.2.3 9.0.2.90.10 9.2.0.3 HP RP7410

Warehouse Builder 9.0.4.8.21 9.0.4.0.27 9.2.0.3 HP RP7410

Reports 9.0.2.0.1 n/a n/a Dell PE2650

Discoverer 9.0.2.53.09 9.0.2.53.09 9.2.0.3 HP RP7410

Enterprise Manager n/a * 9.0.1.3.1 Dell PE2650

Portal n/a 9.0.2.2.22 9.0.1.3.1 Dell PE2650

SSO n/a * 9.0.1.3.1 Dell PE2650

OID n/a * 9.0.1.3.1 Dell PE2650

Design Phase Tasks

Create logical and physical data models using DesignerStaging area for source dataOperational Data Store (ODS)

Create transformation routines using OWBGUI ETL (Extract,Transform,Load) tool

Diagram inputs, outputs, and transformation routines for moving data into ODS from source

Generates PL/SQL code

Development Phase Tasks

Create the databases Using OEM or SQL*Plus

Deploy the data modelsUsing DDL from Designer

Deploy and test the transformations Using OWB

Development Phase Tasks

Install and configure 9iAS and the Reports server

Create the access portal/interface Using Oracle Portal

Create static, parameter driven reports Using Oracle Reports1st cut was generated from Designer

Development Phase Tasks

Create the various portlet providers Using Oracle Portal

Create the Business Areas (End User Layer)Using Discoverer Administration

Create the dynamic business intelligence interface Using Discoverer End User Edition

(workstation) or Discoverer Plus (web)

Development Phase Tasks

Create the Discoverer public connections Within 9iAS environmentUsing Web OEM

Deploy Discoverer portlets (worksheet & workbook)Using Oracle Portal

Develop additional PL/SQL scripts For automating the data loads

Develop Workflow Process Flows Using OWB vs. Workflow Builder

Discoverer Portlets

Management Software

OEM (Oracle Enterprise Manager)Manage the database – storage, users,

collecting stats, etc.Web OEM

Manages the 9iAS componentsOracle Workflow

Schedules and monitors the ETL mappings and load routines

Oracle Tool Repositories

REPOPROD database (meta data)Oracle DesignerOracle Warehouse BuilderOracle Enterprise Manager

ODS databaseOracle Discoverer (EUL)OWB runtimeOWF runtime

Oracle Tool Repositories

iAS databaseOracle PortalWeb OEMSSO (Single Sign On)OID (Oracle Internet Directory)

Repository Schema Locations

Installation and Configuration

Workstation notesCaution – numerous Oracle homesMemory, memory and more memory

9iAS (all components on the same server)Required to do the install 3 times

InfrastructureApplications Tools (if using the Portal Developer’s Kit-PDK)

Installation and Configuration

9iAS notes (continued)Infrastructure installation

9i database (IASDB) automatically builtVersion 9.0.1.3.1

Application installationReports server configuration will fail 1st time

Series of services started after each installDocumentation is “shaky” for all 9iAS

installation proceduresBe prepared to delete and start overDPS submitted to IOUG a 9iAS Rel 2 Survival Guide

for Windows for publication in SELECT magazine.

Installation and Configuration

Database notesAlways plan carefully

Check for operating system patches firstTest all components (interfaces, db links,

backups, etc…)

If installing under the same OS user-id All other Oracle databases must be stopped

due to the shared java components

Get a test server (if you can!)

The Big Green Thingy – Design Phase Components

The Big Green Thingy -Development Phase Components

The Big Green Thingy -Deployment Phase Components

Finally –The Big Green Thingy

Conclusion

Complex Setup Lot of work, research, trial and errorLimited published documentation

ResultThe foundation of our architecture is up and

runningIntegrated, single vendor solution

Will support our efforts to build our District Information Factory

Hopefully this information will add some clarity and make life easier when building the components of your own data warehouse

Contact Information

Allison Lobato• Allison_Lobato@dpsk12.org