Sept. 2009
Bill MinorIBM
DB2 LUW V9.7 (Cobra) – Storage is Charmed
2
Highlights
• This presentation will dive into storage management and administration in DB2 LUW. A detailed, holistic look into storage will be provided, including a preview of new exciting features and capabilities planned for DB2's next release.
• Topics of discussion will include: • Line dancing with LOB and XML data • Compression to the max - Deep compression gets deeper
with index, temp, xml and lob compression along with replication support
• Swimming in shallow water - life without the High Water Mark and space reclamation made easy
• Itty Bitty Reorg - one range partition at a time• Mean and lean MDC
3
Compression To Date
• DB2 V8.1 Database Backup Compression• DB2 V8.1 NULL and Default Value Compression• DB2 V8.1 MDC Tables
• DB2 9.1 Row (or Table) Compression
• DB2 9.5 Enhanced Row Compression support• DB2 9.5 XML Inlining (compression of small XML docs)
4
Roadmap: Where Are We Headed?
• Further reduction in overall storage footprint
• But also, improved administration of storage as a whole (capacity, space reclamation, online support, monitoring, …)
• Technology Preview (TP)
5
Agenda
• LOB Inlining (XML Inlining DB2 V9.5)
• Compression Replication Support
• Index Compression
• Temp Table Compression
• Space Reclaim: Sparse MDC Tables
• Tablespace Extent Remapping (Tablespace HWM Management)
• Local or Partitioned Index Support for Range Partitioned Tables
• XDA (XML) Compression
6
Inlining• If a table possesses LOB or XML data that can be inlined, there are
considerable benefits with respect to performance and storage use
• Performance• Whenever LOBs are inserted/retrieved, a disk I/O cost is incurred each time
since this data is not buffered (unlike with base table data)• When inlined, this I/O cost is reduced since this data get buffered along with
the base table data they are inlined with
• Storage• Storage allocated to the storage object is reduced by inlining XML/LOB data
in the base table (though base table storage increases)• Inlining small XML/LOBs can result in a noticeable decrease in net total
storage since the decrease in storage size is greater than the increase in base table storage size
• XML/LOBs inlined within the base table data can be compressed
7
XML Inlining and Compression (DB2 V9.5)
TPoX throughput by workload type with inlining and compression
0
50,000
100,000
150,000
200,000
250,000
300,000
350,000
400,000
inserts queries mixed
Tra
nsa
ctio
ns/
min
ute V9
V9.5 VanillaV9.5 InlinedV9.5 Compressed
Page Type V9 Vanilla V9.5 Inlined V9.5Compressed V9.5
User Tables 21,178 21,178 628,518 259,706 XDA 955,600 632,448 250 250 User Indexes 88,388 88,233 88,122 88,312 Path/Region Indexes 17,961 15,458 6 6 Total 1,083,127 757,317 716,896 348,274 GB (= 1024**3) 16.5 11.6 10.9 5.3
TPoX Database Number of 16K pages required
100 Concurrent Users12-way AIX
8
LOB Inlining (TP)
• Instead of strictly storing LOB data in the LOB storage object, the LOBs, if sufficiently sized, can be stored in the formatted rows of the base table
• Dependant on page size, the maximum length a LOB can be to qualify for inlining is 32 669 bytes
• LOB inlining is analogous to XML inlining for XML data
Example:create table mytab1 (a int, b char(5), c clob inline length 1000)
a
Base table, mytab1 LOB storage object
b c
3 ‘cat’
9 ‘dog’
‘rat’27
‘bbbbbbbbbbbb…’
‘aaaaaaaaaaa…’
‘cccccccccccc…’
9
How does it work?
• When creating a table with LOB columns, by default, an inline length is set for each LOB column (based on the maximum length of the column)
• To explicitly specify an inline length, there are two methods:• During table creation• By altering an existing table’s LOB columns
• To then inline the existing data, reorg table with LONGLOBDATA option
Examplescreate table mytab1 (a int, b char(5), c clob inline length 1000)alter table mytab1 alter column c set inline length 2000
Note:When altering a LOB column’s inline length, it cannot be smaller than the inlinelength set by a prior table create/alter
10
Example: LOB Inlining
0
10000
20000
30000
40000
50000
60000
70000
V9.5 V9.5 (compression) Cobra (LOB inlining) Cobra (LOB inlining andcompression)
Tota
l phy
sica
l sto
rage
(KB
s)
V9.5 V9.5
(compression)
Cobra
(LOB inlining)
Cobra
(LOB inlining,
compression)
Base table (KBs)
24320 5248 22144 4736
LOB storage (KBs)
30336 30336 128 128
Indexes (KBs) 10880 10752 10880 10752
Total (KBs) 65536 46336 33152 15616
• All LOBs in this table are smaller than the default inline length for the LOB columns
• 100% of the LOBs for this table, inlined in Cobra!
11
LOB Inlining Notes:
• If the clause INLINE LENGTH is not specified, the default initial inline length is equal to the size of the LOB descriptor (the size of the LOB descriptor varies depending on the LOB size of the column)
• Inlined LOB data is treated as if it were VARCHAR data
• If LOB data cannot be inlined it will be inserted in the LOB object (which is stored separately from the table rows)
12
Inlining Administrative Table Functions• ADMIN_EST_INLINE_LENGTH(<column name>)
• Returns an estimate of the inline length required to inline the data, a negative value if data cannot be inlined, or if data already inlined, the actual inline length is returned
• ADMIN_IS_INLINED (<column name>)• Will report whether the XML documents in a column are
inlined or not
Example:select admin_is_inlined(xml_doc1) as IS_INLINED, admin_est_inline_length(xml_doc1) as
EST_INLINE_LENGTH from tab1
IS_INLINED EST_INLINE_LENGTH---------- -----------------1 2920 4500 454
13
Table Compression in V9.1/9.5
• To compress a table the COMPRESS attribute must be enabled
• When compression is active, user data written to the log files is compressed (we do not compress log files but compress the data in them)
• Replication programs read the log files and therefore need to be able to decipher the contents of a log file that contains compressed data
• Only one of COMPRESS YES or DATA CAPTURE CHANGES can be enabled on a table in V9.1/9.5
14
Compression & Replication (TP) • DB2 V9.5 - compress yes not compatible with data capture
changes clause
• Capture requires uncompressed log data in log records• Recall - log data from compressed records in compressed
format
alter table rowcomp.rc1 compress yesDB20000I The SQL command completed successfully.
alter table rowcomp.rc1 data capture changesDB21034E The command was processed as an SQL statement because it was not avalid Command Line Processor command. During SQL processing it returned:SQL0270N Function not supported (Reason code = "83"). SQLSTATE=42997
Log Header
Smith Mary 1235 TX
Log Header
05 06 02 07 04
Example:Insert log record
User record not compressed
User record compressed
15
Compression and db2ReadLog
• db2ReadLog API will now decompress log data before returning log records
Log db2ReadLog API
Log db2ReadLog API
Dictionary
Compressed user data in logs
Uncompressed user data in logs
DB2 V9.7 with iFilterOption ON
DB2 V9.5 with iFilterOption ON
16
Compression Replication Support
• When enabled, a single historical dictionary will also be retained in the table data object
• This allows for latency in the log read• If the existing compression dictionary is reset, the prior dictionary
is still available to the log reader in order to interpret log records that were written using that earlier version of the dictionary
• ADMIN_GET_TAB_INFO can be used to get the sizes of dictionaries
17
Index Compression (TP)
• The approach used is different than row compression• Various algorithms are employed at the discretion of the database
engine• RID List Compression• Prefix Compression• Index meta-data optimization
• Compression of index objects via • COMPRESS clause on CREATE INDEX statement• COMPRESS clause on new ALTER INDEX statement
• Index reorganization required to rebuild the index in compressed (or uncompressed) format
• By default, new indexes on a row compressed table will be compressed (except for MDC block indexes and XML meta and paths indexes)
18
Index Compression (continued)
• Two new columns added to SYSIBM.SYSINDEXES and SYSIBM.SYSINDEXPARTITIONS• COMPRESSION• PCTPAGESSAVED
• Two new table functions (see upcoming slides)• ADMIN_GET_INDEX_INFO• ADMIN_GET_INDEX_COMPRESS_INFO
• Index compression estimation only through this admin function, no INSPECT support like there is for tables
19
ADMIN_GET_INDEX_INFO
SQL Admin Routine:ADMIN_GET_INDEX_INFO(objecttype, objectschema, objectname)
Column name :
• INDSCHEMA • INDNAME • TABSCHEMA • TABNAME • DBPARTITIONNUM • IID • DATAPARTITIONID • COMPRESS_ATTR • INDEX_COMPRESSED • INDEX_PARTITIONING • INDEX_OBJECT_L_SIZE • INDEX_OBJECT_P_SIZE • INDEX_REQUIRES_REBUILD • LARGE_RIDS
db2 “select * from table(sysproc.admin_get_index_info(‘’,’BMINOR’,’STAFF)) as t”
20
ADMIN_GET_INDEX_COMPRESS_INFO
ADMIN_GET_INDEX_COMPRESS_INFO(objecttype, objectschema, objectname, dbpartitionnum, datapartitionid)Column name :
• INDSCHEMA • INDNAME • TABSCHEMA • TABNAME • DBPARTITIONNUM • IID • DATAPARTITIONID • COMPRESS_ATTR • INDEX_COMPRESSED • PCT_PAGES_SAVED • NUM_LEAF_PAGES_SAVED
db2 “select * from table(sysproc.admin_get_index_compress_info(‘’,’BMINOR’,”STAFF’, 2 , 3)) as t”
21
Compression Savings from SAP supplied tables
Number of Index Pages Percentage saved
Not compressed 147152 0%
RID list compression 99766 32.2%
Byte based prefix compression
102104 30.6%
RID list compression +
Variable Slot Directory +
Byte based prefix compression
65919 55.2%
Bytes saved = 2.4G -1.1G = 1.3G
22
Temporary Tables - Overview
• Temporary tablespaces are composed of physical dedicated/reserved storage which is used to hold temporary tables
• Temporary tables are transient as they hold data from intermediate processing which is required to achieve an end result usually associated i.e. they are work tables
• with sort or join processing• When such results cannot be contained within memory, we say processing has
spilled to a temporary table and therefore tempspace is consumed• System Temporary Tables: created by the database manager
• Sort or join processing: ORDER BY, DISTINCT, GROUP BY, merge join, hash join, ‘=ANY’, ‘<>ALL’, INTERSECT or EXCEPT, UNION (without ALL), …
• Utility processing• Table Queues (TQ): internode/database partition processing
• User Temporary Tables: user explicitly creates temporary tables• Declare Global Temporary Tables (DGTTs)• (TP) Create Global Temporary Tables (CGTTs)
23
Temp Table Compression (TP)
• Compression of temporary tables aims to:• Reduce the amount of temporary disk space required• Have minimal performance impact as a result of the extra
processing required for row compression. If there is an increase in query performance, that is an added side benefit
• Target up to 40% reduction
• Applicable to User temporary tables and System temps
• If Deep Compression is licensed, then temporary tables will be compressed by default. There is no additional action required by the user in order to use it. DB2 will evaluate the query and apply compression where appropriate.
24
Monitoring Temp Compression
• db2pd –db <dbname> -tablespaces• Field called “Max HWM” which illustrates high water mark of
temp tablespace consumption
• db2pd –db <dbname> -temptables• Displays System and User Temporary tablespace usage along
with bytes saved as a result of temp compression being activated
25
Explain Format 3) SORT : (Sort) Cumulative Total Cost: 95095.5 Cumulative CPU Cost: 2.06275e+09 Cumulative I/O Cost: 3755 Cumulative Re-Total Cost: 0 Cumulative Re-CPU Cost: 0 Cumulative Re-I/O Cost: 1347 Cumulative First Row Cost: 95095.5 Estimated Bufferpool Buffers: 3755
Arguments: --------- DUPLWARN: (Duplicates Warning flag) FALSE NUMROWS : (Estimated number of rows) 286720 ROWWIDTH: (Estimated width of rows) 40 SORTKEY : (Sort Key column) 1: Q1.ID(A) SPILLED : (Pages spilled to bufferpool or disk) 1347 TEMPSIZE: (Temporary Table Page Size) 8192 TMPCMPRS: (Temp table compression) YES UNIQUE : (Uniqueness required flag) FALSE
26
Storage for MDC Tables
• Storage in an MDC table is tracked through a ‘block map’: which extents have data and which don’t ( block == extent)
• When a block is emptied, the storage remains with the table and is available for later reuse
• Currently, only Classic/’Offline’ table reorg can free this storage• Does not allow write to the table• Shadow copy approach, approx. double the storage for the
duration of the processing• Takes time as whole table is processed
27
Sparse MDC Space Reclaim (TP)
• If MDC table has empty blocks, these can be reclaimed without performing a conventional table reorg
• New option on reorg table command to not reorg this table but reclaim these empty
blocks/extents
REORG TABLE--<mdc-table-name>--RECLAIM EXTENTS ONLY-- . . .
• This storage is freed back to the tablespace and hence is available for use by any object in the tablespace
• Storage is freed incrementally during the processing• During it’s processing, this new capability allows concurrent write to the table and occurs with
a minimum amount of time and resource• Can be invoked at the data partition level• Amount of space that can be reclaimed is reported through the column
RECLAIMABLE_SPACE in the ADMIN_GET_TAB_INFO_Vxx table function
28
The "High Water Mark" (HWM) – Tables & Tablespaces
Table: It is the highest occupied page in a table (SMS or DMS)
Tablespace: It is the page number of the highest allocated page in a DMS table space
HWM quite simply, is impacted by data manipulation:
- ‘Offline' REORG of a table within the DMS table space that the table resides in (Tablespace)
- Index REORG with either ALLOW READ ACCESS or ALLOW WRITE ACCESS (Tablespace)
- SQL DELETE activity (Table)
HWM affects:
- Redirected Restore - redefinition of containers allowing table space to shrink in size; cannot be
shrunk lower than HWM
- Dropping or reducing the size of container via ALTER TABLESPACE only affects extents
above the HWM
- Table shrink
Tablespace occupancy can be assessed via the output from LIST TABLESPACES SHOW DETAIL
-"Used pages", "Free pages", and "High water mark (pages)"
Note that "Used pages" + "Free pages" is always equal to "Useable pages".
29
Classic (‘Offline’) Table Reorg and the HWM
DMS PERMTABLESPACE
T1 T1
T1'
db2 reorg table T1
T1SHADOW
Table Space HWM
Table Space HWM
Intermediate state – Build of shadow copy image
Table Reorg is completeAnd HWM is moved up
Reorganizing Table T1 within the Tablespace causes new extents to be allocated within the tablespace
T1 T1
T1'
db2 reorg table T1
T1SHADOW
Table Space HWMTable Space HWM
Table Reorg is completeAnd HWM is moved down
30
Space Reclamation – Reducing DMS Tablespace HWM (V9.1/9.5)
If no free extents below the HWM then the only way to reduce the HWM is to drop the object
holding it up
db2dart /DHWM
- displays detailed table space information including which extents are free, which are in
use and what object is using them as well as information about the object holding up the
HWM (if online, “db2pdcfg –flushbp” first; V9.1FP5, V9.5FP1)
db2dart /LHWM
- provides guidance as to how the HWM might potentially be lowered
If DMS table data object holding up HWM then 'offline' REORG of table within the DMS table
space that the table resides can be used to lower the HWM if enough free extents exist below
the HWM to contain the shadow copy
If DMS index object holding up HWM, index reorg may be able to reduce HWM
db2dart /RHWM and ALTER TABLESPACE REDUCE
db2dart usability enhancements: IZ04647 V9.1 FP4; IZ07663 V9.1 FP5
31
Interpreting a DHWM ReportHighwater Mark: 122 pages, 61 extents (extents #0 - 60) [0000] 65534 0x0e [0001] 65534 0x0e [0002] 65535 0x00 [0003] 4 0x40* [0004] 4 0x00* [0005] 4 0x00 [0006] 4 0x00 [0007] 4 0x00 [0008] 4 0x00 [0009] 4 0x00 [0010] 4 0x00 [0011] 4 0x00 [0012] 4 0x00 [0013] 4 0x00 [0014] 4 0x00 [0015] 4 0x00 [0016] 4 0x00 [0017] 4 0x00 [0018] 4 0x00 [0019] 4 0x00 [0020] 4 0x00 [0021] 4 0x00 [0022] 4 0x00 [0023] 4 0x00 [0024] 4 0x00 [0025] 4 0x00 [0026] 4 0x00 [0027] 4 0x00 [0028] 4 0x00 [0029] 4 0x00 [0030] 4 0x00 [0031] 4 0x00 [0032] 4 0x00 [0033] 4 0x00 [0034] 4 0x00 [0035] 4 0x00 [0036] 4 0x00 [0037] 4 0x00 [0038] 4 0x00 [0039] 4 0x00 [0040] 4 0x00 [0041] 4 0x00 [0042] 4 0x00 [0043] 4 0x00 [0044] 4 0x00 [0045] 4 0x00 [0046] 4 0x00 [0047] 4 0x00 [0048] 4 0x00 [0049] 4 0x00 [0050] 4 0x00 [0051] 4 0x00 [0052] 4 0x00 [0053] 4 0x00 [0054] 4 0x00 [0055] 4 0x00 [0056] 5 0x40* [0057] 5 0x00* [0058] 5 0x41* [0059] 5 0x01* [0060] 5 0x01
Dump highwater mark processing - phase start.Number of free extents below highwater mark: 0Number of used extents below highwater mark: 61
Object holding highwater mark: Object ID: 5 Type: Index Extent
Possible extent types:
0x00 data
0x01 index
0x02 long field
0x03 lob
0x04 lob allocation
0x05 block map (mdc)
0x06 xda (xml storage)
0x0e smp
Object ID: 65534 Space Map Page (SMP)
65535 Object Table
* denotes extent anchor (where object begins)
Extent size for this tablespace is 2
Two tables have been created in this tablespace with IDs 4 and 5
An index was created on table with object ID 5
32
New to DB2 V9.5
• Reclaim space within a DMS tablespace
ALTER TABLESPACE <tablespaceName> REDUCE• Drops/reduces containers to the table space’s high-water mark• In V9.5 this will now first remove unused SMP extents – no need for
db2dart /RHWM• Remove unused SMP extents logic exists in other operations as
well • Occurs as part of ALTER TABLESPACE when using REDUCE,
RESIZE (to smaller), or DROP container operations• Also occurs during online backup and LIST TABLESPACES
SHOW DETAIL
33
Tablespace Extent Remapping : Remedy for “HWM Issues” (TP)
• High Water Marks will always exist • A HWM can become an ‘issue’ when there is free space below it
AND no further growth with the tablespace is expected• Proposal:
• All new DMS tablespaces created by default with a reclaimable storage attribute
• New ALTER TABLESPACE options to lower the HWM and reclaim free space below it by transparently ‘shuffling’ the contents of the tablespace to lower positions in the tablespace (no db2dart operations)
• Online to DML and DDL operations – one exception, Backup and Restore
34
Reclaimable Tablespace Storage- How?
• For Automatic Storage Tablespaces:
ALTER TABLESPACE <name> REDUCE --------------------------|
+--<size>--<units>--+
+--MAX--------------+
+--STOP-------------+
• Triggers asynchronous extent movement but iterative in the sense that space is gradually release back to the system
• If no option specified, abide by existing behaviour, viz. reclaim pending delete extents and reduce size of containers down to HWM
• MAX: reclaim as much storage as possible and reduce containers to new HWM
• <size>: to reclaim a user specified amount of space only
• STOP: to stop extent movement at the current point in time reclamation
35
Reclaimable Tablespace Storage - How? (continued)
• For non-automatic storage tablespaces:
ALTER TABLESPACE <name> LOWER HIGH WATER MARK -----------|
+-STOP-+• Triggers asynchronous extent movement without any container operations
• ALTER TABLESPACE REDUCE <cont-ops> is required after this in order to shrink the storage
• This lower HWM command will also work for AS tablespaces without any container operations happening
36
Reclaimable Tablespace Storage - Monitoring
Example: MON_GET_TABLESPACES
db2 "select substr(tbsp_name,1,15) as tbsp_name, tbsp_type, tbsp_total_pages, reclaimable_space_enabled from table(sysproc.mon_get_tablespaces(‘USERSPACE1’, 0))”
TBSP_NAME TBSP_TYPE TBSP_TOTAL_PAGES RECLAIMABLE_SPACE_ENABLED
--------------- ---------- ---------------- -------------------------
USERSPACE1 DMS 5000 1
Other columns of interest:
• TBSP_USED_PAGES TBSP_PENDING_FREE_PAGES
• TBSP_FREE_PAGES TBSP_PAGE_TOP
• TBSP_USABLE_PAGES TBSP_MAX_PAGE_TOP
37
Reclaimable Tablespace Storage - Monitoring
Example: MON_GET_EXTENT_MOVEMENT_STATUS
db2 "select substr(tbsp_name,1,15) as tbsp_name, current_extent, last_extent, num_extents_moved,num_extents_left,total_move_time from table(sysproc.mon_get_extent_movement_status('TBSP_DMS1',0))"
38
Partitioned (Local) Indexes (TP)
• Support for the ability to create local (partitioned) indexes• This will relieve issues associated with Roll-In processing, mainly the
global index maintenance and associated logging• Allows for reorg table at the range partition level• Improved ease of use with respect to range level compression
Example:CREATE INDEX pINX1 on SALES (sales_date, partID) PARTITIONED Partitioned index is the default
• Statistics –• RUNSTATS will collect at the table level as well as at partition data
and index level• SYSIBM.SYSDATAPARTITIONS is extended to include: CARD,
OVERFLOW,NPAGES, FPAGES, AVGROWSIZE, PCTROWSCOMPRESSED, …
39
XML (XDA) Compression (TP)
• In addition to XML Inlining, the (separate) XML storage object (XDA) can be compressed
• Approach will be very similar to that of Table Compression• Static dictionary approach• Classic/’Offline’ reorg table based
• Requires LONGLOBDATA clause• ADC applicable• No online dictionary support via DB2 INSPECT
• Enablement is via the table COMPRESS attribute i.e. piggyback data table setting – there is no specific compress setting for XML object alone
40
XML Compression (continued)
• Not supported on tables with XML columns created in prior releases (new XML storage format is required)
• Admin function to determine: • XML object size• XML storage type• XML dictionary existence and size
41
ADMIN_GET_TAB_INFO_Vxx
• TABSCHEMA • TABNAME • TABTYPE • DBPARTITIONNUM • DATA_PARTITION_ID • AVAILABLE • DATA_OBJECT_L_SIZE • DATA_OBJECT_P_SIZE • INDEX_OBJECT_L_SIZE • INDEX_OBJECT_P_SIZE • LONG_OBJECT_L_SIZE • LONG_OBJECT_P_SIZE • LOB_OBJECT_L_SIZE • LOB_OBJECT_P_SIZE • XML_OBJECT_L_SIZE XML_OBJECT_L_SIZE • XML_OBJECT_P_SIZEXML_OBJECT_P_SIZE
• INDEX_TYPE • REORG_PENDING • INPLACE_REORG_STATUS • LOAD_STATUS • READ_ACCESS_ONLY • NO_LOAD_RESTART • NUM_REORG_REC_ALTERS • INDEXES_REQUIRE_REBUILD • LARGE_RIDS • LARGE_SLOTS • DICTIONARY_SIZE • BLOCKS_PENDING_CLEANUP
• STATSTYPE • XML_RECORD_TYPE • RECLAIMABLE_SPACE • XML_DICTIONARY_SIZE
Recommended
