DB2 LUW V9.7 (Cobra) - Storage is Charmed

  • Published on

  • View

  • Download

Embed Size (px)




<ul><li> 1. Sept. 2009 Bill Minor IBM DB2 LUW V9.7 (Cobra) Storage is Charmed</li></ul> <p> 2. Highlights </p> <ul><li>This presentation will dive into storage management and administration in DB2 LUW. A detailed, holistic look into storage will be provided, including a preview of new exciting features and capabilities planned for DB2's next release.</li></ul> <ul><li>Topics of discussion will include:</li></ul> <ul><li><ul><li>Line dancing with LOB and XML data </li></ul></li></ul> <ul><li><ul><li>Compression to the max- Deep compression gets deeper with index, temp, xml and lob compression along with replication support </li></ul></li></ul> <ul><li><ul><li>Swimming in shallow water- life without the High Water Mark and space reclamation made easy </li></ul></li></ul> <ul><li><ul><li>Itty Bitty Reorg- one range partition at a time </li></ul></li></ul> <ul><li><ul><li>Mean and lean MDC </li></ul></li></ul> <p> 3. Compression To Date </p> <ul><li>DB2 V8.1 Database Backup Compression </li></ul> <ul><li>DB2 V8.1 NULL and Default Value Compression </li></ul> <ul><li>DB2 V8.1 MDC Tables </li></ul> <ul><li>DB2 9.1 Row (or Table) Compression </li></ul> <ul><li>DB2 9.5 Enhanced Row Compression support </li></ul> <ul><li>DB2 9.5 XML Inlining (compression of small XML docs) </li></ul> <p> 4. Roadmap: Where Are We Headed? </p> <ul><li>Further reduction in overall storage footprint </li></ul> <ul><li>But also, improved administration of storage as a whole (capacity, space reclamation, online support, monitoring, ) </li></ul> <ul><li>Technology Preview (TP) </li></ul> <p> 5. Agenda </p> <ul><li>LOB Inlining(XML Inlining DB2 V9.5) </li></ul> <ul><li>Compression Replication Support</li></ul> <ul><li>Index Compression </li></ul> <ul><li>Temp Table Compression </li></ul> <ul><li>Space Reclaim: Sparse MDC Tables </li></ul> <ul><li>Tablespace Extent Remapping (Tablespace HWM Management) </li></ul> <ul><li>Local or Partitioned Index Support for Range Partitioned Tables </li></ul> <ul><li>XDA (XML) Compression </li></ul> <p> 6. Inlining </p> <ul><li>If a table possesses LOB or XML data that can be inlined, there are considerable benefits with respect to performance and storage use </li></ul> <ul><li>Performance </li></ul> <ul><li><ul><li>Whenever LOBs are inserted/retrieved, a disk I/O cost is incurred each time since this data is not buffered (unlike with base table data) </li></ul></li></ul> <ul><li><ul><li>When inlined, this I/O cost is reduced since this data get buffered along with the base table data they are inlined with </li></ul></li></ul> <ul><li>Storage </li></ul> <ul><li><ul><li>Storage allocated to the storage object is reduced by inlining XML/LOB data in the base table (though base table storage increases) </li></ul></li></ul> <ul><li><ul><li>Inlining small XML/LOBs can result in a noticeable decrease in net total storage since the decrease in storage size is greater than the increase in base table storage size </li></ul></li></ul> <ul><li><ul><li>XML/LOBs inlined within the base table data can be compressed </li></ul></li></ul> <p> 7. XML Inlining and Compression(DB2 V9.5) 100 Concurrent Users 12-way AIX 8. LOB Inlining (TP) </p> <ul><li>Instead of strictly storing LOB data in the LOB storage object, the LOBs, if sufficiently sized, can be stored in the formatted rows of the base table </li></ul> <ul><li>Dependant on page size, the maximum length a LOB can be to qualify for inlining is 32 669 bytes </li></ul> <ul><li>LOB inlining is analogous to XML inlining for XML data </li></ul> <ul><li>Example: </li></ul> <ul><li>create table mytab1 (a int, b char(5), c clob inline length 1000 ) </li></ul> <p>a Base table, mytab1 LOB storage object b c 3 cat 9 dog rat 27 bbbbbbbbbbbb aaaaaaaaaaa cccccccccccc 9. How does it work? </p> <ul><li>When creating a table with LOB columns, by default, an inline length is set for each LOB column (based on the maximum length of the column) </li></ul> <ul><li>To explicitly specify an inline length, there are two methods: </li></ul> <ul><li><ul><li>During table creation </li></ul></li></ul> <ul><li><ul><li>By altering an existing tables LOB columns </li></ul></li></ul> <ul><li>To then inline the existing data, reorg table with LONGLOBDATA option </li></ul> <ul><li>Examples </li></ul> <ul><li>create table mytab1 (a int, b char(5), c clob inline length 1000) </li></ul> <ul><li>alter table mytab1 alter column c set inline length 2000 </li></ul> <ul><li>Note: </li></ul> <ul><li>When altering a LOB columns inline length, it cannot be smaller than the inline </li></ul> <ul><li>length set by a prior table create/alter </li></ul> <p> 10. Example: LOB Inlining </p> <ul><li>All LOBs in this table are smaller than the default inline length for the LOB columns </li></ul> <ul><li><ul><li>100% of the LOBs for this table, inlined in Cobra! </li></ul></li></ul> <p>15616 33152 46336 65536 Total (KBs) 10752 10880 10752 10880 Indexes (KBs) 128 128 30336 30336 LOB storage (KBs) 4736 22144 5248 24320 Base table (KBs) Cobra (LOB inlining, compression) Cobra (LOB inlining) V9.5 (compression) V9.5 11. LOB Inlining Notes: </p> <ul><li>If the clause INLINE LENGTH is not specified, the default initial inline length is equal to the size of the LOB descriptor (the size of the LOB descriptor varies depending on the LOB size of the column) </li></ul> <ul><li>Inlined LOB data is treated as if it were VARCHAR data </li></ul> <ul><li>If LOB data cannot be inlined it will be inserted in the LOB object (which is stored separately from the table rows) </li></ul> <p> 12. Inlining Administrative Table Functions </p> <ul><li>ADMIN_EST_INLINE_LENGTH() </li></ul> <ul><li><ul><li>Returns an estimate of the inline length required to inline the data, a negative value if data cannot be inlined, or if data already inlined, the actual inline length is returned </li></ul></li></ul> <ul><li>ADMIN_IS_INLINED () </li></ul> <ul><li><ul><li>Will report whether the XML documents in a column are inlined or not </li></ul></li></ul> <ul><li>Example: </li></ul> <ul><li>select admin_is_inlined(xml_doc1) as IS_INLINED, admin_est_inline_length(xml_doc1) as EST_INLINE_LENGTH from tab1 </li></ul> <ul><li>IS_INLINEDEST_INLINE_LENGTH </li></ul> <ul><li>--------------------------- </li></ul> <ul><li>1292 </li></ul> <ul><li>0450 </li></ul> <ul><li>0454 </li></ul> <p> 13. Table Compression in V9.1/9.5 </p> <ul><li>To compress a table the COMPRESS attribute must be enabled </li></ul> <ul><li>When compression is active, user data written to the log files is compressed (we do not compress log files but compress the data in them) </li></ul> <ul><li>Replication programs read the log files and therefore need to be able to decipher the contents of a log file that contains compressed data </li></ul> <ul><li>Only one of COMPRESS YES or DATA CAPTURE CHANGES can be enabled on a table in V9.1/9.5 </li></ul> <p> 14. Compression &amp; Replication (TP)</p> <ul><li>DB2 V9.5 -compress yesnot compatible withdata capture changesclause </li></ul> <ul><li>Capture requires uncompressed log data in log records </li></ul> <ul><li>Recall- log data from compressed records in compressed format </li></ul> <ul><li><ul><li>alter table rowcomp.rc1 compress yes </li></ul></li></ul> <ul><li><ul><li>DB20000IThe SQL command completed successfully. </li></ul></li></ul> <ul><li><ul><li>alter table rowcomp.rc1 data capture changes </li></ul></li></ul> <ul><li><ul><li>DB21034EThe command was processed as an SQL statement because it was not a </li></ul></li></ul> <ul><li><ul><li>valid Command Line Processor command.During SQL processing it returned: </li></ul></li></ul> <ul><li><ul><li>SQL0270NFunction not supported (Reason code = "83").SQLSTATE=42997 </li></ul></li></ul> <p>Example: Insert log record User record not compressed User record compressed Log Header TX 1235 Mary Smith Log Header 05 06 02 07 04 15. Compression and db2ReadLog </p> <ul><li>db2ReadLog API will now decompress log data before returning log records </li></ul> <p>Log db2ReadLog API Log db2ReadLog API Dictionary Compressed user data in logs Uncompressed user data in logs DB2V9.7with iFilterOption ONDB2V9.5with iFilterOption ON 16. Compression Replication Support </p> <ul><li>When enabled, a single historical dictionary will also be retained in the table data object </li></ul> <ul><li>This allows for latency in the log read </li></ul> <ul><li><ul><li>If the existing compression dictionary is reset, the prior dictionary is still available to the log reader in order to interpret log records that were written using that earlier version of the dictionary </li></ul></li></ul> <ul><li>ADMIN_GET_TAB_INFO can be used to get the sizes of dictionaries </li></ul> <p> 17. Index Compression (TP) </p> <ul><li>The approach used is different than row compression </li></ul> <ul><li><ul><li>Various algorithms are employed at the discretion of the database engine </li></ul></li></ul> <ul><li><ul><li><ul><li>RID List Compression </li></ul></li></ul></li></ul> <ul><li><ul><li><ul><li>Prefix Compression </li></ul></li></ul></li></ul> <ul><li><ul><li><ul><li>Index meta-data optimization </li></ul></li></ul></li></ul> <ul><li>Compression of index objects via</li></ul> <ul><li><ul><li>COMPRESS clause on CREATE INDEX statement </li></ul></li></ul> <ul><li><ul><li>COMPRESS clause on new ALTER INDEX statement </li></ul></li></ul> <ul><li><ul><li><ul><li>Index reorganization required to rebuild the index in compressed (or uncompressed) format </li></ul></li></ul></li></ul> <ul><li><ul><li>By default, new indexes on a row compressed table will be compressed (except for MDC block indexes and XML meta and paths indexes) </li></ul></li></ul> <p> 18. Index Compression (continued) </p> <ul><li>Two new columns added to SYSIBM.SYSINDEXES and SYSIBM.SYSINDEXPARTITIONS </li></ul> <ul><li><ul><li>COMPRESSION </li></ul></li></ul> <ul><li><ul><li>PCTPAGESSAVED </li></ul></li></ul> <ul><li>Two new table functions (see upcoming slides) </li></ul> <ul><li><ul><li>ADMIN_GET_INDEX_INFO </li></ul></li></ul> <ul><li><ul><li>ADMIN_GET_INDEX_COMPRESS_INFO </li></ul></li></ul> <ul><li><ul><li><ul><li>Index compression estimation only through this admin function, no INSPECT support like there is for tables </li></ul></li></ul></li></ul> <p> 19. ADMIN_GET_INDEX_INFO </p> <ul><li>SQL Admin Routine: </li></ul> <ul><li>ADMIN_GET_INDEX_INFO(objecttype, objectschema, objectname) </li></ul> <ul><li>Column name :</li></ul> <ul><li>INDSCHEMA</li></ul> <ul><li>INDNAME</li></ul> <ul><li>TABSCHEMA</li></ul> <ul><li>TABNAME</li></ul> <ul><li>DBPARTITIONNUM</li></ul> <ul><li>IID</li></ul> <ul><li>DATAPARTITIONID</li></ul> <ul><li>COMPRESS_ATTR</li></ul> <ul><li>INDEX_COMPRESSED</li></ul> <ul><li>INDEX_PARTITIONING</li></ul> <ul><li>INDEX_OBJECT_L_SIZE</li></ul> <ul><li>INDEX_OBJECT_P_SIZE</li></ul> <ul><li>INDEX_REQUIRES_REBUILD</li></ul> <ul><li>LARGE_RIDS</li></ul> <p>db2 select * from table(sysproc.admin_get_index_info(,BMINOR,STAFF)) as t 20. ADMIN_GET_INDEX_COMPRESS_INFO </p> <ul><li>ADMIN_GET_INDEX_COMPRESS_INFO(objecttype, objectschema,</li></ul> <ul><li>objectname, dbpartitionnum,</li></ul> <ul><li>datapartitionid) </li></ul> <ul><li>Column name : </li></ul> <ul><li>INDSCHEMA</li></ul> <ul><li>INDNAME</li></ul> <ul><li>TABSCHEMA</li></ul> <ul><li>TABNAME</li></ul> <ul><li>DBPARTITIONNUM</li></ul> <ul><li>IID</li></ul> <ul><li>DATAPARTITIONID</li></ul> <ul><li>COMPRESS_ATTR</li></ul> <ul><li>INDEX_COMPRESSED</li></ul> <ul><li>PCT_PAGES_SAVED</li></ul> <ul><li>NUM_LEAF_PAGES_SAVED</li></ul> <p>db2 select * from table(sysproc.admin_get_index_compress_info(,BMINOR,STAFF, 2 , 3)) as t 21. Compression Savings from SAP supplied tables Bytes saved = 2.4G -1.1G = 1.3G Percentage saved Number of Index Pages 55.2% 65919 RID list compression + Variable Slot Directory + Byte based prefix compression 30.6% 102104 Byte based prefix compression 32.2% 99766 RID list compression 0% 147152 Not compressed 22. Temporary Tables - Overview </p> <ul><li>Temporary tablespaces are composed of physical dedicated/reserved storage which is used to hold temporary tables </li></ul> <ul><li>Temporary tables are transient as they hold data from intermediate processing which is required to achieve an end result usually associated i.e. they are work tables </li></ul> <ul><li>with sort or join processing </li></ul> <ul><li>When such results cannot be contained within memory, we say processing has spilled to a temporary table and therefore tempspace is consumed </li></ul> <ul><li>System Temporary Tables: created by the database manager </li></ul> <ul><li><ul><li>Sort or join processing: ORDER BY, DISTINCT, GROUP BY, merge join, hash join, =ANY, ALL, INTERSECT or EXCEPT, UNION (without ALL), </li></ul></li></ul> <ul><li><ul><li>Utility processing </li></ul></li></ul> <ul><li><ul><li>Table Queues (TQ): internode/database partition processing </li></ul></li></ul> <ul><li>User Temporary Tables: user explicitly creates temporary tables </li></ul> <ul><li><ul><li>Declare Global Temporary Tables (DGTTs) </li></ul></li></ul> <ul><li><ul><li>(TP) Create Global Temporary Tables (CGTTs)</li></ul></li></ul> <p> 23. Temp Table Compression (TP) </p> <ul><li>Compression of temporary tables aims to: </li></ul> <ul><li><ul><li>Reduce the amount of temporary disk space required </li></ul></li></ul> <ul><li><ul><li>Have minimal performance impact as a result of the extra processing required for row compression.If there is an increase in query performance, that is an added side benefit </li></ul></li></ul> <ul><li><ul><li>Target up to 40% reduction </li></ul></li></ul> <ul><li>Applicable to User temporary tables and System temps</li></ul> <ul><li>If Deep Compression is licensed, then temporary tables will be compressed by default.There is no additional action required by the user in order to use it.DB2 will evaluate the query and apply compression where appropriate.</li></ul> <p> 24. Monitoring Temp Compression </p> <ul><li>db2pd db -tablespaces </li></ul> <ul><li><ul><li>Field called Max HWM which illustrates high water mark of temp tablespace consumption </li></ul></li></ul> <ul><li>db2pd db -temptables </li></ul> <ul><li><ul><li>Displays System and User Temporary tablespace usage along with bytes saved as a result of temp compression being activated </li></ul></li></ul> <p> 25. Explain Format </p> <ul><li>3 ) SORT: (Sort) </li></ul> <ul><li>Cumulative Total Cost:95095.5 </li></ul> <ul><li>Cumulative CPU Cost:2.06275e+09 </li></ul> <ul><li>Cumulative I/O Cost:3755 </li></ul> <ul><li>Cumulative Re-Total Cost:0 </li></ul> <ul><li>Cumulative Re-CPU Cost:0 </li></ul> <ul><li>Cumulative Re-I/O Cost:1347 </li></ul> <ul><li>Cumulative First Row Cost:95095.5 </li></ul> <ul><li>Estimated Bufferpool Buffers:3755 </li></ul> <ul><li>Arguments: </li></ul> <ul><li>--------- </li></ul> <ul><li>DUPLWARN: (Duplicates Warning flag) </li></ul> <ul><li>FALSE </li></ul> <ul><li>NUMROWS : (Estimated number of rows) </li></ul> <ul><li>286720 </li></ul> <ul><li>ROWWIDTH: (Estimated width of rows) </li></ul> <ul><li>40</li></ul> <ul><li>SORTKEY : (Sort Key column) </li></ul> <ul><li>1: Q1.ID(A) </li></ul> <ul><li>SPILLED : (Pages spilled to bufferpool or disk) </li></ul> <ul><li>1347 </li></ul> <ul><li>TEMPSIZE: (Temporary Table Page Size) </li></ul> <ul><li>8192 </li></ul> <ul><li>TMPCMPRS: (Temp table compression) </li></ul> <ul><li>YES </li></ul> <ul><li>UNIQUE: (Uniqueness required flag) </li></ul> <ul><li>FALSE </li></ul> <p> 26. Storage for MDC Tables</p> <ul><li>Storage in an MDC table is tracked through a block map: which extents have data and which dont ( block == extent) </li></ul> <ul><li>When a block is emptied, the storage remains with the table and is available for later reuse </li></ul> <ul><li>Currently, only Classic/Offline table reorg can free this storage </li></ul> <ul><li><ul><li>Does not allow write to the table </li></ul></li></ul> <ul><li><ul><li>Shadow copy approach, approx. double the storage for the duration of the processing </li></ul></li></ul> <ul><li><ul><li>Takes time as whole table is processed </li></ul></li></ul> <p> 27. Sparse MDC Space Reclaim (TP) </p> <ul><li>If MDC table has empty blocks, these can be reclaimed without performing a conventional table reorg </li></ul> <ul><li>New option on reorg table command to not reorg this table but reclaim these empty blocks/extents</li></ul> <ul><li>REORG TABLE----RECLAIM EXTENTS ONLY-- . . . </li></ul> <ul><li>This storage is freed back to the tablespace and hence is available for use by any object in the tablespace </li></ul> <ul><li>Storage is freed incrementally during the processing </li></ul> <ul><li>During its processing, this new capabilityallows concurrent write to the table and occurs with a minimum amount of time and resource </li></ul> <ul><li>Can be invoked at the data partition level </li></ul> <ul><li>Amount of space that can be reclaimed is reported through the column RECLAIMABLE_SPACE in the ADMIN_GET_TAB_INFO_Vxx table function </li></ul> <p> 28. The "High Water Mark" (HWM) Tables &amp; Tablespaces </p> <ul><li>Table: It is the highest occupied page in a table (SMS or DMS) </li></ul> <ul><li>Tablespace: It is the page number of the highest allocated page in aDMStable space </li></ul> <ul><li>HWM quite simply, is impacted by data manipulation: </li></ul> <ul><li><ul><li>- Offline' REORG of a tablewithinthe DMS table space that the table resides in (Tablespace) </li></ul></li></ul> <ul><li><ul><li>- Index REORG with either ALLOW READ ACCESS or ALLOW WRITE ACCESS (Tablespace) </li></ul></li></ul> <ul><li><ul><li>- SQL DELETE activity (Table) </li></ul></li></ul> <ul><li>HWM affects: </li></ul> <ul><li><ul><li>- Redirected Restore - redefinition of containers allowing table space to shrink in size; cannot be shrunk lower than HWM </li></ul></li></ul> <ul><li><ul><li>Dropping or reducing the size of container via ALTER TABLESPACE only affects extentsabovethe HWM </li></ul></li></ul> <ul><li><ul><li>Table shrink </li></ul></li></ul> <ul><li>Tablespace occupancy can be assessed via the output from LIST TABLESPACES SHOW DETAIL </li></ul> <ul><li><ul><li>-"Used pages", "Free pages", and "High water mark (pages)"</li></ul></li></ul> <ul><li><ul><li>Note that "Used pages" + "Free pages"is always equal to "Useable pages". </li></ul></li></ul> <p> 29. Classic (Offline) Table Reorg and the HWM DMS PERM TABLESPACE T1 T1 T1' db2 reorg table T1 T1 SHADOW Table Space HWM Table Space HWM Intermediate state Buildof shadow copy image Table Reorg is complete And HWM is moved up Reorganizing Table T1withinthe Tablespace causes new extents to be allocated within the tablespace T1 T1 T1' db2 reorg table T1 T1 SHADOW Table Space HWM Table Space HWM Table Reorg is complete And HWM is moved down 30. Space Reclamation Reducing DMS Tablespace HWM (V9.1/9.5) </p> <ul><li>If no free extents below the HWM then the only way to reduce the HWM is to drop the object holding it up </li></ul> <ul><li>db2dart/DHWM </li></ul> <ul><li><ul><li>- displays detailed table space information includingwhich extents are free, which are in use and what object is using them as well as information about the object holding up the HWM(if online, db2pdcfg flushbp first; V9.1FP5, V9.5FP1) </li></ul></li></ul> <ul><li>db2dart/LHWM </li></ul> <ul><li><ul><li>- provides guidance as to how the HWMmight potentially be lowered </li></ul></li></ul> <ul><li>If DMS table data object holding up HWM then 'offline' REORG of table within the DMS table space that the table resides can be used to lower the HWM if enough free extents exist below the HWM to contain the shadow copy </li></ul> <ul><li>If DMS index object holding up HWM, index reorg may be able to r