40
Optimizing Data Warehouse Ad-Hoc Queries against "Star Schemas" By Bert Scalzo, Ph.D. [email protected]

Optimizing Data Warehouse Ad-Hoc Queries against "Star Schemas " By Bert Scalzo, Ph.D. [email protected]

Embed Size (px)

Citation preview

Page 1: Optimizing Data Warehouse Ad-Hoc Queries against "Star Schemas " By Bert Scalzo, Ph.D. Bert.Scalzo@Quest.com

Optimizing Data WarehouseAd-Hoc Queries against

"Star Schemas"

By Bert Scalzo, [email protected]

Page 2: Optimizing Data Warehouse Ad-Hoc Queries against "Star Schemas " By Bert Scalzo, Ph.D. Bert.Scalzo@Quest.com

Learning Objectives

• Just one major objective – to make the majority of unknown/unplanned ad-hoc end-user queries (e.g. BI tool reports’ many “SELECT” statements) execute as fast as is possible – and very efficiently, thus using the least amount of database server resources

We cannot allow big databases to have queries that simply take too long …

Page 3: Optimizing Data Warehouse Ad-Hoc Queries against "Star Schemas " By Bert Scalzo, Ph.D. Bert.Scalzo@Quest.com

Speaker’s Qualifications• Oracle Solutions Product Architect for Quest Software• Chief architect for Quest’s popular “TOAD” product• Oracle DBA for 20+ years, versions 4 through 10g• Worked for Oracle Education & Consulting• Holds several Oracle Masters (DBA & CASE)• BS, MS, PhD in Computer Science and also an MBA• LOMA insurance industry designations: FLMI and ACS• Books

– The TOAD Handbook (Feb 2003)– Oracle DBA Guide to Data Warehousing and Star Schemas (Mar

2003)– The TOAD Pocket Reference 2nd edition (June 2005)

• Articles– Oracle Magazine– Oracle Technology Network (OTN)– Oracle Informant– PC Week (now E-Magazine)– Linux Journal– www.Linux.com

Page 5: Optimizing Data Warehouse Ad-Hoc Queries against "Star Schemas " By Bert Scalzo, Ph.D. Bert.Scalzo@Quest.com

About Quest Software• Quest Software (NASDAQ: QSFT)• Founded: 1987• More than 2000 employees in 40 offices: North

America, South America, Europe, Asia, Australia• Application management leader: 75% of Fortune

500• Develop, deploy, manage and maintain

enterprise applications without downtime or business interruption

• Best known in the Oracle community for TOAD, Spotlight, Quest Central, Shareplex, etc.

Page 6: Optimizing Data Warehouse Ad-Hoc Queries against "Star Schemas " By Bert Scalzo, Ph.D. Bert.Scalzo@Quest.com

Know Your Application …

What type of application are you building:

• On Line Transaction Processing (OLTP)

• Operational Data Store (ODS)

• On Line Analytical Processing (OLAP)

• Data Mart / Data Warehouse (DM/DW)

Page 7: Optimizing Data Warehouse Ad-Hoc Queries against "Star Schemas " By Bert Scalzo, Ph.D. Bert.Scalzo@Quest.com

OLTP ODS OLAP DM/DWBusiness Focus Operational Operational

TacticalTactical Tactical

Strategic

End User Tools Client Server Web

Client Server Web

Client Server Client Server Web

DB Technology Relational Relational Cubic Relational

Trans Count Large Medium Small Small

Trans Size Small Medium Medium Large

Trans Time Short Medium Long Long

Size in Gigs 10 – 200 50 – 400 50 – 400 400 - 4000

Normalization 3NF 3NF N/A 0NF

Data Modeling Traditional ER

Traditional ER N/A Dimensional

Application Natures…

Page 8: Optimizing Data Warehouse Ad-Hoc Queries against "Star Schemas " By Bert Scalzo, Ph.D. Bert.Scalzo@Quest.com

Embrace New Concepts

• “Teach Old Dog New Tricks”

• Throw out any OLTP baggage

• Forget OLTP “Golden Rules” X

Page 9: Optimizing Data Warehouse Ad-Hoc Queries against "Star Schemas " By Bert Scalzo, Ph.D. Bert.Scalzo@Quest.com

Star Schema Design

“Star schema” approach to dimensional data modeling was pioneered by Ralph Kimball

Dimensions: smaller, de-normalized tables containing business descriptive columns that end-users query on

Facts: very large tables with primary keys formed from the concatenation of related dimension table foreign key columns, and possessing numerically additive, non-key columns used for calculations during end-user queries

Page 10: Optimizing Data Warehouse Ad-Hoc Queries against "Star Schemas " By Bert Scalzo, Ph.D. Bert.Scalzo@Quest.com

Dimensions

Facts

Page 11: Optimizing Data Warehouse Ad-Hoc Queries against "Star Schemas " By Bert Scalzo, Ph.D. Bert.Scalzo@Quest.com

108th -1010th

103rd -105th

Page 12: Optimizing Data Warehouse Ad-Hoc Queries against "Star Schemas " By Bert Scalzo, Ph.D. Bert.Scalzo@Quest.com

The Ad-Hoc ChallengeThe Ad-Hoc Challenge

How much data would a data miner mine, if a data miner could mine data?

Dimensions: generally queried selectively to find lookup value matches that are used to query against the fact table

Facts: must be selectively queried, since they generally have hundreds of millions to billions of rows – even full table scans utilizing parallel are too big for most systems

Page 13: Optimizing Data Warehouse Ad-Hoc Queries against "Star Schemas " By Bert Scalzo, Ph.D. Bert.Scalzo@Quest.com

Hardware Not CompensateHardware Not Compensate

Often, people have expectation that using expensive hardware is only way to obtain optimal performance for a data warehouse

•CPU•SMP•MPP

•Disk•15,000 RPM•RAID (EMC)

•OS•UNIX•64-bit

•Oracle•OPS / PQO•64-bit

Page 14: Optimizing Data Warehouse Ad-Hoc Queries against "Star Schemas " By Bert Scalzo, Ph.D. Bert.Scalzo@Quest.com

DB Design ParamountDB Design Paramount

In reality, the database design is the key factor to optimal query performance for a data warehouse built as a “Star Schema”

There are certain minimum hardware and software requirements that once met, play a very subordinate role to tuning the database

Golden Rule: get the serial explain plan correct before doing partitioning or PQ

Page 15: Optimizing Data Warehouse Ad-Hoc Queries against "Star Schemas " By Bert Scalzo, Ph.D. Bert.Scalzo@Quest.com

Key Tuning RequirementsKey Tuning Requirements

1. Right version of Oracle

2. Properly tuned Init.Ora

3. Lots of Bitmap Indexes!!!

4. Cost Based Optimizer

5. Statistics and Histograms

6. “Star Transformation” explain plan

Page 16: Optimizing Data Warehouse Ad-Hoc Queries against "Star Schemas " By Bert Scalzo, Ph.D. Bert.Scalzo@Quest.com

Oracle Version ProblemsOracle Version Problems

Oracle 7.X lacks:•True Partitioning of Tables and Indexes•Reliable and Efficient Bitmap Indexes•Histograms for Cost-Based Optimization •“Star Transformation” explain plan

Oracle 8.X lacks:•List Partitioning•Scalable Bitmap Indexes (500+ million)•Scalable Histograms (500+ million)•Bitmap Join Index•Numerous new GORUP & OLAP Functions•ANSI Syntax Support (e.g. ANSI 99 Join)

Page 17: Optimizing Data Warehouse Ad-Hoc Queries against "Star Schemas " By Bert Scalzo, Ph.D. Bert.Scalzo@Quest.com

Oracle 8.X Init.OraOracle 8.X Init.Oraalways_anti_join=HASHalways_semi_join=HASHbitmap_merge_area_size=>16Mbuffer_pool_keep=xM (for bitmap indexes)buffer_pool_recycle=xM (for fact tables)compatible=8.1.7.3.0create_bitmap_area_size>=16Mdb_block_buffers= (1.7G / DB_BLOCK_SIZE) – shared_pool_sizedb_block_size>=16Kdb_file_direct_io_count= MAX_IO_SIZE / DB_BLOCK_SIZEdb_file_multiblock_read_count= MAX_IO_SIZE / DB_BLOCK_SIZEdb_writer_processes=10disk_asynch_io=TRUEhash_area_size>=16Mhash_join_enabled=TRUEhash_multiblock_io_count= MAX_IO_SIZE / DB_BLOCK_SIZEoptimizer_features_enable=8.1.7optimizer_percent_paralell=0 (favor best serial plan – explained later)parallel_automatic_tuning=TRUEparallel_broadcast_enabled=TRUEparallel_max_servers=4 * CPU_COUNT if you have fast disksparallel_min_percent=0query_rewrite_enabled=TRUEsort_area_size>=4Msort_multiblock_read_count= MAX_IO_SIZE / DB_BLOCK_SIZEstar_transformation_enabled=TEMP_DISABLE

Page 18: Optimizing Data Warehouse Ad-Hoc Queries against "Star Schemas " By Bert Scalzo, Ph.D. Bert.Scalzo@Quest.com

Oracle 9.X Init.OraOracle 9.X Init.Ora

compatible=9.2.0.6.0db_block_size>=16Kdb_cache_size=(1.7G / DB_BLOCK_SIZE) – shared_pool_sizedb_file_multiblock_read_count= MAX_IO_SIZE / DB_BLOCK_SIZEdb_keep_cache_size=xM (for bitmap indexes)db_recycle_cache_size=xM (for fact tables)db_writer_processes=20disk_asynch_io=TRUEfilesystemio_options=SETALL (directIO + asynch)hash_join_enabled=TRUElog_parallelism=8optimizer_features_enable=9.2.0parallel_automatic_tuning=TRUEparallel_max_servers=4 * CPU_COUNT if you have fast disksparallel_min_percent=0pga_aggregate_size>=64Mquery_rewrite_enabled=TRUEstar_transformation_enabled=TRUE or TEMP_DISABLEundo_management=AUTOworkarea_size_policy=AUTO

Page 19: Optimizing Data Warehouse Ad-Hoc Queries against "Star Schemas " By Bert Scalzo, Ph.D. Bert.Scalzo@Quest.com

Oracle 10.X Init.OraOracle 10.X Init.Ora

compatible=10.2.0.2.0db_block_size>=16Kdb_file_multiblock_read_count= MAX_IO_SIZE / DB_BLOCK_SIZEdb_keep_cache_size=xM (for bitmap indexes)db_recycle_cache_size=xM (for fact tables)db_writer_processes=20disk_asynch_io=TRUEfilesystemio_options=SETALL (directIO + asynch)optimizer_features_enable=10.2.0parallel_automatic_tuning=TRUEparallel_max_servers=4 * CPU_COUNT if you have fast disksparallel_min_percent=0pga_aggregate_size>=64Mquery_rewrite_enabled=TRUEsga_target_size=1.7G (for 32-bit systems)star_transformation_enabled=TRUE or TEMP_DISABLEundo_management=AUTOworkarea_size_policy=AUTO

Page 20: Optimizing Data Warehouse Ad-Hoc Queries against "Star Schemas " By Bert Scalzo, Ph.D. Bert.Scalzo@Quest.com

Bitmap IndexesBitmap Indexes

All fact table, foreign key columns must have individual bitmap indexes on them

If using Oracle 8.1, then ALTER TABLE MINIMIZE RECORDS_PER_BLOCK to optimize the mapping of bitmaps to rowids

All dimension table, non-key columns should have individual bitmap indexes

Page 21: Optimizing Data Warehouse Ad-Hoc Queries against "Star Schemas " By Bert Scalzo, Ph.D. Bert.Scalzo@Quest.com

Nighttime - 10 B-Tree Indexes

Page 22: Optimizing Data Warehouse Ad-Hoc Queries against "Star Schemas " By Bert Scalzo, Ph.D. Bert.Scalzo@Quest.com

Daytime - 48 Bitmap Indexes!!!

Page 23: Optimizing Data Warehouse Ad-Hoc Queries against "Star Schemas " By Bert Scalzo, Ph.D. Bert.Scalzo@Quest.com

Bit-map indexes– Contrary to widespread belief, can be effective when

there are many distinct column values

– Not suitable for OLTP however

0.01

0.1

1

10

100

1 10 100 1,000 10,000 100,000 1,000,000

Distinct values

Ela

psed tim

e (

s)

Bitmap index B*-Tree index Full table scan

Page 24: Optimizing Data Warehouse Ad-Hoc Queries against "Star Schemas " By Bert Scalzo, Ph.D. Bert.Scalzo@Quest.com

Fact Table IndexesFact Table Indexes

Table Index Type Uniq Column Null---------------- -------------------- ------ ---- -------------------- ----DW_ORDER DW_POS_DAY_B1 BITMAP N PERIOD_ID N

DW_POS_DAY_B2 BITMAP N LOCATION_ID N

DW_POS_DAY_B3 BITMAP N PRODUCT_ID N

DW_POS_DAY_B4 BITMAP N VENDOR_ID N

DW_POS_DAY_PK NORMAL Y PERIOD_ID N NORMAL Y LOCATION_ID N NORMAL Y PRODUCT_ID N NORMAL Y VENDOR_ID N NORMAL Y RECMD_FLAG N NORMAL Y NEW_ITEM_FLAG N

Page 25: Optimizing Data Warehouse Ad-Hoc Queries against "Star Schemas " By Bert Scalzo, Ph.D. Bert.Scalzo@Quest.com

Dimension Table IndexesDimension Table Indexes

Table Index Type Uniq Column Null---------------- -------------------- ------ ---- -------------------- ----DW_VENDOR DW_VENDOR_B02 BITMAP N LEVELX N

DW_VENDOR_B03 BITMAP N VENDOR_CODE N

DW_VENDOR_B05 BITMAP N CITY Y

DW_VENDOR_B06 BITMAP N STATE Y

DW_VENDOR_B07 BITMAP N ZIP N

DW_VENDOR_PK NORMAL Y VENDOR_ID N

DW_VENDOR_U1 NORMAL Y VENDOR_NAME N

Page 26: Optimizing Data Warehouse Ad-Hoc Queries against "Star Schemas " By Bert Scalzo, Ph.D. Bert.Scalzo@Quest.com

Cost Based OptimizerCost Based Optimizer

Statistics and Histograms:

1. First, analyze table delete statistics

2. For dimension tables, estimate statistics for table for all indexes

3. For fact tables, estimate statistics for table for all indexes for all indexed columns size 254 sample 20000 rows

Page 27: Optimizing Data Warehouse Ad-Hoc Queries against "Star Schemas " By Bert Scalzo, Ph.D. Bert.Scalzo@Quest.com

Ready to Rock and RollReady to Rock and Roll

With all our database tuning pre-requisites addressed, we can now rely on Oracle to formulate reasonable explain plans for “Star Schema” data warehouse queries

In fact if we’ve done everything correctly, then table/index partitioning and parallel query will merely eliminate partitions or perform parallel operations on our serial explain plans (i.e. 95%+ identical plans)

Page 28: Optimizing Data Warehouse Ad-Hoc Queries against "Star Schemas " By Bert Scalzo, Ph.D. Bert.Scalzo@Quest.com

Sample Tuning CasesSample Tuning Cases

Oracle 9.2.0.6 / HP-UX 11.11• HP-9000, V2200, 16 CPU, 16 GB RAM• Avg Fact table row count = 600 million

Oracle 8.1.7.3 / HP-UX 10.20• HP-9000, K460, 4 CPU, 4 GB RAM• Avg Fact table row count = 3 million

Oracle 10.1.0.5 / Windows 2003 Server• Quad Xeon, 4 GB RAM• Avg Fact table row count = 3 million

Oracle 10.2.0.2 / Redhat Linux AS 4.0• Quad Xeon, 4 GB RAM• Avg Fact table row count = 3 million

SameExplainPlans!!!

Page 29: Optimizing Data Warehouse Ad-Hoc Queries against "Star Schemas " By Bert Scalzo, Ph.D. Bert.Scalzo@Quest.com

Query: beer and coffee sales for November of 98 in Dallas

Page 30: Optimizing Data Warehouse Ad-Hoc Queries against "Star Schemas " By Bert Scalzo, Ph.D. Bert.Scalzo@Quest.com

Star Transformation

Page 31: Optimizing Data Warehouse Ad-Hoc Queries against "Star Schemas " By Bert Scalzo, Ph.D. Bert.Scalzo@Quest.com

Star Transformation

Page 32: Optimizing Data Warehouse Ad-Hoc Queries against "Star Schemas " By Bert Scalzo, Ph.D. Bert.Scalzo@Quest.com

Star Transformation

Page 33: Optimizing Data Warehouse Ad-Hoc Queries against "Star Schemas " By Bert Scalzo, Ph.D. Bert.Scalzo@Quest.com

Star join performance

0.06

3.43

59.86

0 10 20 30 40 50 60

Elapsed time (s)

STAR hint

Cost Based (no STAR)

Rule Based

3 orders of magnitude difference between best and worst plan

Page 34: Optimizing Data Warehouse Ad-Hoc Queries against "Star Schemas " By Bert Scalzo, Ph.D. Bert.Scalzo@Quest.com

-------------------------------------------------------------------------------------------------------| Operation | Name | Rows | Bytes| Cost | Pstart| Pstop |-------------------------------------------------------------------------------------------------------| SELECT STATEMENT | | 1 | 154 | 34 | | || SORT GROUP BY | | 1 | 154 | 34 | | || HASH JOIN | | 1 | 154 | 29 | | || TABLE ACCESS BY INDEX ROWID | DW_ORDER | 1 | 95 | 17 | | || BITMAP CONVERSION TO ROWIDS | | | | | | || BITMAP AND | | | | | | || BITMAP MERGE | | | | | | || BITMAP KEY ITERATION | | | | | | || TABLE ACCESS BY INDEX ROWID | DW_PERIOD | 1 | 51 | 2 | | || BITMAP CONVERSION TO ROWIDS | | | | | | || BITMAP AND | | | | | | || BITMAP INDEX SINGLE VALUE | DW_PERIOD_B03 | | | | | || BITMAP INDEX SINGLE VALUE | DW_PERIOD_B12 | | | | | || BITMAP INDEX RANGE SCAN | DW_ORDER_B1 | | | | | || BITMAP MERGE | | | | | | || BITMAP KEY ITERATION | | | | | | || TABLE ACCESS BY INDEX ROWID | DW_LOCATION | 1 | 46 | 2 | | || BITMAP CONVERSION TO ROWIDS | | | | | | || BITMAP AND | | | | | | || BITMAP INDEX SINGLE VALUE | DW_LOCATION_B03 | | | | | || BITMAP INDEX SINGLE VALUE | DW_LOCATION_B41 | | | | | || BITMAP INDEX RANGE SCAN | DW_ORDER_B2 | | | | | || BITMAP MERGE | | | | | | || BITMAP KEY ITERATION | | | | | | || TABLE ACCESS BY INDEX ROWID | DW_PRODUCT | 17 | 1K| 10 | | || BITMAP CONVERSION TO ROWIDS | | | | | | || BITMAP AND | | | | | | || BITMAP INDEX SINGLE VALUE | DW_PRODUCT_B03 | | | | | || BITMAP OR | | | | | | || BITMAP INDEX SINGLE VALUE | DW_PRODUCT_B14 | | | | | || BITMAP INDEX SINGLE VALUE | DW_PRODUCT_B14 | | | | | || BITMAP INDEX RANGE SCAN | DW_ORDER_B3 | | | | | || TABLE ACCESS BY INDEX ROWID | DW_PRODUCT | 17 | 1K| 10 | | || BITMAP CONVERSION TO ROWIDS | | | | | | || BITMAP AND | | | | | | || BITMAP INDEX SINGLE VALUE | DW_PRODUCT_B03 | | | | | || BITMAP OR | | | | | | || BITMAP INDEX SINGLE VALUE | DW_PRODUCT_B14 | | | | | || BITMAP INDEX SINGLE VALUE | DW_PRODUCT_B14 | | | | | |-------------------------------------------------------------------------------------------------------

UTLXPLS: Serial Plan, No Partitioning

Page 35: Optimizing Data Warehouse Ad-Hoc Queries against "Star Schemas " By Bert Scalzo, Ph.D. Bert.Scalzo@Quest.com

-------------------------------------------------------------------------------------------------------| Operation | Name | Rows | Bytes| Cost | Pstart| Pstop |-------------------------------------------------------------------------------------------------------| SELECT STATEMENT | | 1 | 154 | 35 | | || SORT GROUP BY | | 1 | 154 | 35 | | || SORT GROUP BY | | 1 | 154 | 35 | | || HASH JOIN | | 1 | 154 | 30 | | || PARTITION RANGE ALL | | | | | 1 | 10 || TABLE ACCESS BY LOCAL INDEX ROWID| DW_ORDER_PART | 0 | 20 | 18 | 1 | 10 || BITMAP CONVERSION TO ROWIDS | | | | | | || BITMAP AND | | | | | | || BITMAP MERGE | | | | | | || BITMAP KEY ITERATION | | | | | | || SORT BUFFER | | | | | | || TABLE ACCESS BY INDEX ROWID| DW_PERIOD | 1 | 51 | 2 | | || BITMAP CONVERSION TO ROWID| | | | | | || BITMAP AND | | | | | | || BITMAP INDEX SINGLE VALU| DW_PERIOD_B03 | | | | | || BITMAP INDEX SINGLE VALU| DW_PERIOD_B12 | | | | | || BITMAP INDEX RANGE SCAN | DW_ORDER_PART_B1 | | | | 1 | 10 || BITMAP MERGE | | | | | | || BITMAP KEY ITERATION | | | | | | || SORT BUFFER | | | | | | || TABLE ACCESS BY INDEX ROWID| DW_LOCATION | 1 | 46 | 2 | | || BITMAP CONVERSION TO ROWID| | | | | | || BITMAP AND | | | | | | || BITMAP INDEX SINGLE VALU| DW_LOCATION_B03 | | | | | || BITMAP INDEX SINGLE VALU| DW_LOCATION_B41 | | | | | || BITMAP INDEX RANGE SCAN | DW_ORDER_PART_B2 | | | | 1 | 10 || BITMAP MERGE | | | | | | || BITMAP KEY ITERATION | | | | | | || SORT BUFFER | | | | | | || TABLE ACCESS BY INDEX ROWID| DW_PRODUCT | 17 | 1K| 10 | | || BITMAP CONVERSION TO ROWID| | | | | | || BITMAP AND | | | | | | || BITMAP INDEX SINGLE VALU| DW_PRODUCT_B03 | | | | | || BITMAP OR | | | | | | || BITMAP INDEX SINGLE VAL| DW_PRODUCT_B14 | | | | | || BITMAP INDEX SINGLE VAL| DW_PRODUCT_B14 | | | | | || BITMAP INDEX RANGE SCAN | DW_ORDER_PART_B3 | | | | 1 | 10 || TABLE ACCESS BY INDEX ROWID | DW_PRODUCT | 17 | 1K| 10 | | || BITMAP CONVERSION TO ROWIDS | | | | | | || BITMAP AND | | | | | | || BITMAP INDEX SINGLE VALUE | DW_PRODUCT_B03 | | | | | || BITMAP OR | | | | | | || BITMAP INDEX SINGLE VALUE | DW_PRODUCT_B14 | | | | | || BITMAP INDEX SINGLE VALUE | DW_PRODUCT_B14 | | | | | |-------------------------------------------------------------------------------------------------------

UTLXPLS: Serial Plan, with Partitioning

Page 36: Optimizing Data Warehouse Ad-Hoc Queries against "Star Schemas " By Bert Scalzo, Ph.D. Bert.Scalzo@Quest.com

----------------------------------------------------------------------------------------------------------------------------------| Operation | Name | Rows | Bytes| Cost | TQ |IN-OUT| PQ Distrib | Pstart| Pstop |----------------------------------------------------------------------------------------------------------------------------------| SELECT STATEMENT | | 1 | 154 | 34 | | | | | || SORT GROUP BY | | 1 | 154 | 34 | 2,03 | P->S | QC (RANDOM)| | || SORT GROUP BY | | 1 | 154 | 34 | 2,02 | P->P | HASH | | || HASH JOIN | | 1 | 154 | 29 | 2,02 | PCWP | | | || TABLE ACCESS BY INDEX ROWID | DW_ORDER | 1 | 95 | 17 | 2,01 | P->P | HASH | | || BITMAP CONVERSION TO ROWIDS | | | | | | | | | || BITMAP AND | | | | | | | | | || BITMAP MERGE | | | | | | | | | || BITMAP KEY ITERATION | | | | | | | | | || TABLE ACCESS BY INDEX ROWID | DW_PERIOD | 1 | 51 | 2 | | | | | || BITMAP CONVERSION TO ROWIDS | | | | | | | | | || BITMAP AND | | | | | | | | | || BITMAP INDEX SINGLE VALUE | DW_PERIOD_B03 | | | | | | | | || BITMAP INDEX SINGLE VALUE | DW_PERIOD_B12 | | | | | | | | || BITMAP INDEX RANGE SCAN | DW_ORDER_B1 | | | | | | | | || BITMAP MERGE | | | | | | | | | || BITMAP KEY ITERATION | | | | | | | | | || TABLE ACCESS BY INDEX ROWID | DW_LOCATION | 1 | 46 | 2 | | | | | || BITMAP CONVERSION TO ROWIDS | | | | | | | | | || BITMAP AND | | | | | | | | | || BITMAP INDEX SINGLE VALUE | DW_LOCATION_B03 | | | | | | | | || BITMAP INDEX SINGLE VALUE | DW_LOCATION_B41 | | | | | | | | || BITMAP INDEX RANGE SCAN | DW_ORDER_B2 | | | | | | | | || BITMAP MERGE | | | | | | | | | || BITMAP KEY ITERATION | | | | | | | | | || TABLE ACCESS BY INDEX ROWID | DW_PRODUCT | 17 | 1K| 10 | | | | | || BITMAP CONVERSION TO ROWIDS | | | | | | | | | || BITMAP AND | | | | | | | | | || BITMAP INDEX SINGLE VALUE | DW_PRODUCT_B03 | | | | | | | | || BITMAP OR | | | | | | | | | || BITMAP INDEX SINGLE VALUE| DW_PRODUCT_B14 | | | | | | | | || BITMAP INDEX SINGLE VALUE| DW_PRODUCT_B14 | | | | | | | | || BITMAP INDEX RANGE SCAN | DW_ORDER_B3 | | | | | | | | || TABLE ACCESS BY INDEX ROWID | DW_PRODUCT | 17 | 1K| 10 | 2,00 | S->P | HASH | | || BITMAP CONVERSION TO ROWIDS | | | | | | | | | || BITMAP AND | | | | | | | | | || BITMAP INDEX SINGLE VALUE | DW_PRODUCT_B03 | | | | | | | | || BITMAP OR | | | | | | | | | || BITMAP INDEX SINGLE VALUE | DW_PRODUCT_B14 | | | | | | | | || BITMAP INDEX SINGLE VALUE | DW_PRODUCT_B14 | | | | | | | | |----------------------------------------------------------------------------------------------------------------------------------

UTLXPLP: Parallel Plan, No Partitioning

Page 37: Optimizing Data Warehouse Ad-Hoc Queries against "Star Schemas " By Bert Scalzo, Ph.D. Bert.Scalzo@Quest.com

----------------------------------------------------------------------------------------------------------------------------------| Operation | Name | Rows | Bytes| Cost | TQ |IN-OUT| PQ Distrib | Pstart| Pstop |----------------------------------------------------------------------------------------------------------------------------------| SELECT STATEMENT | | 1 | 154 | 34 | | | | | || SORT GROUP BY | | 1 | 154 | 34 | 5,03 | P->S | QC (RANDOM)| | || SORT GROUP BY | | 1 | 154 | 34 | 5,02 | P->P | HASH | | || HASH JOIN | | 1 | 154 | 29 | 5,02 | PCWP | | | || PARTITION RANGE ALL | | | | | 5,02 | PCWP | | 1 | 10 || TABLE ACCESS BY LOCAL INDEX ROWID| DW_ORDER_PART | 0 | 20 | 18 | 5,01 | P->P | HASH | 1 | 10 || BITMAP CONVERSION TO ROWIDS | | | | | | | | | || BITMAP AND | | | | | | | | | || BITMAP MERGE | | | | | | | | | || BITMAP KEY ITERATION | | | | | | | | | || SORT BUFFER | | | | | | | | | || TABLE ACCESS BY INDEX ROWID| DW_PERIOD | 1 | 51 | 2 | | | | | || BITMAP CONVERSION TO ROWID| | | | | | | | | || BITMAP AND | | | | | | | | | || BITMAP INDEX SINGLE VALU| DW_PERIOD_B03 | | | | | | | | || BITMAP INDEX SINGLE VALU| DW_PERIOD_B12 | | | | | | | | || BITMAP INDEX RANGE SCAN | DW_ORDER_PART_B1 | | | | | | | 1 | 10 || BITMAP MERGE | | | | | | | | | || BITMAP KEY ITERATION | | | | | | | | | || SORT BUFFER | | | | | | | | | || TABLE ACCESS BY INDEX ROWID| DW_LOCATION | 1 | 46 | 2 | | | | | || BITMAP CONVERSION TO ROWID| | | | | | | | | || BITMAP AND | | | | | | | | | || BITMAP INDEX SINGLE VALU| DW_LOCATION_B03 | | | | | | | | || BITMAP INDEX SINGLE VALU| DW_LOCATION_B41 | | | | | | | | || BITMAP INDEX RANGE SCAN | DW_ORDER_PART_B2 | | | | | | | 1 | 10 || BITMAP MERGE | | | | | | | | | || BITMAP KEY ITERATION | | | | | | | | | || SORT BUFFER | | | | | | | | | || TABLE ACCESS BY INDEX ROWID| DW_PRODUCT | 17 | 1K| 10 | | | | | || BITMAP CONVERSION TO ROWID| | | | | | | | | || BITMAP AND | | | | | | | | | || BITMAP INDEX SINGLE VALU| DW_PRODUCT_B03 | | | | | | | | || BITMAP OR | | | | | | | | | || BITMAP INDEX SINGLE VAL| DW_PRODUCT_B14 | | | | | | | | || BITMAP INDEX SINGLE VAL| DW_PRODUCT_B14 | | | | | | | | || BITMAP INDEX RANGE SCAN | DW_ORDER_PART_B3 | | | | | | | 1 | 10 || TABLE ACCESS BY INDEX ROWID | DW_PRODUCT | 17 | 1K| 10 | 5,00 | S->P | HASH | | || BITMAP CONVERSION TO ROWIDS | | | | | | | | | || BITMAP AND | | | | | | | | | || BITMAP INDEX SINGLE VALUE | DW_PRODUCT_B03 | | | | | | | | || BITMAP OR | | | | | | | | | || BITMAP INDEX SINGLE VALUE | DW_PRODUCT_B14 | | | | | | | | || BITMAP INDEX SINGLE VALUE | DW_PRODUCT_B14 | | | | | | | | |----------------------------------------------------------------------------------------------------------------------------------

UTLXPLP: Parallel Plan, with Partitioning

Page 38: Optimizing Data Warehouse Ad-Hoc Queries against "Star Schemas " By Bert Scalzo, Ph.D. Bert.Scalzo@Quest.com

SQL Plus Timing ResultsSQL Plus Timing Results

Fact Implementation UNIX 2003

Serial, No Partition 9,688 22,344

Serial, with Partition 5,578 11,625

Parallel, No Partition 7,750 17,875

Parallel, with Partition 11,140 25,454

NOTE: specific to my data and user queries

Page 39: Optimizing Data Warehouse Ad-Hoc Queries against "Star Schemas " By Bert Scalzo, Ph.D. Bert.Scalzo@Quest.com

Plethora Table Design OptionsPlethora Table Design Options

Fact Implementation UNIX

Regular “Heap” Table 9,293

Single Column Partition 4,747

Multi Column Partition 4,987

Composite Partition 6,319

Index Organized 12,508

Partition Index Organized 14,902

NOTE: specific to my data and user queries

Page 40: Optimizing Data Warehouse Ad-Hoc Queries against "Star Schemas " By Bert Scalzo, Ph.D. Bert.Scalzo@Quest.com

ConclusionsConclusionsRemember:

•should use Oracle latest Oracle version for best results (optimizer has matured for DW and specifically star queries)

•proper init.ora (i.e. spfile) parameters

•lots and lots of bitmap indexes

•must use statistics and histograms

•shoot for the “Star Transformation”

•Get non-parallel, non-partitioned explain plan right first, then improve upon that with others

•highly recommend SQL tuning tools – like “TOAD Xpert”