Index Advisor and Autonomic Indexes

i want stress-free IT.i want control.

i want an i.

IBM System i™

DB2 for i5/OSV5R4 Enhancements for Index Advisor and Autonomic Indexes

OrMake Your SQL Code Fly in V5R4

Satid SingkorapoomAdvisory Technical Sales Specialist, System i Sales, IBM ThailandMarch 2007

IBM System i

© 2007 IBM Corporationi want an i.

BackgroundThis presentation is based on the following articles:

• DB2 for i5/OS Redefines On Demand for Indexing by Scott Forstie and Jim Flanagan, IBM Systems Magazine – System i Edition, May 2006 at URL:http://www.ibmsystemsmagazine.com/ME2/Audiences/dirmod.asp?sid=&nm=&type=Publishing&mod=Publications%3A%3AArticle&mid=8F3A7027421841978F18BE895F87F791&tier=4&id=00D363297CDB4F54BAF9FA42E9286218&AudID=1E8FEE745A284521B6CFB3FD70B49099

• The Optimizer Takes Its Own Advice by Rob Downer and Shantan Kethireddy, IBM Systems Magazine – System i Edition, June 2006 at URL:http://www.ibmsystemsmagazine.com/ME2/Audiences/dirmod.asp?sid=&nm=&type=Publishing&mod=Publications%3A%3AArticle&mid=8F3A7027421841978F18BE895F87F791&tier=4&id=15C863E9852B45F6960150532D43F2A5&AudID=1E8FEE745A284521B6CFB3FD70B49099

• Simplify DB2 for i5/OS index advice by Scott Forstie, IBM developerWorks article, Jan 2007 at URL: http://www-128.ibm.com/developerworks/db2/library/techarticle/dm-0701forstie/

IBM System i


Background

SQL Performance Tuning facilities have been the strength of DB2 for i5/OS for many years. Such useful features have been appreciated by DB2 SQL programmers:

• Detailed query optimizer feedback messages in the job log• QAQQINI query control file• Summary and Detailed SQL performance monitor• System-wide SQL statement cache• Visual Explain with Index and Statistics Advisors• Automatic column statistics collection

This presentation describes three recent SQL performance enhancements to V5R4 of DB2 for i5/OS.

IBM System i


BackgroundAll V5R4 enhancements discussed in this presentation apply to the new SQEEngine. The enhanced index advisor and its condenser also support CQE.

SLIC

Optimizer

Query Dispatcher

CQE Optimizer SQE Optimizer

DB2 Data Storage & Management

SQE Optimizer

SQE Primitives

SQE Statistics Manager

CQE Database Engine

IBM System i


Agenda

• New Index Advisor features in V5R4

• Condensed Index Advisor for the creation of “perfect” indexes

• Benefits of Autonomic Indexes

IBM System i





IBM System i


Pre-V5R4 Index Advisor

Pre-V5R4, Index Advisor is accessed by browsing job log for query optimizer messages, Visual Explain (above), and Detailed DB monitor.

Invoking Index and statistics advisors dialog

Highlighting steps with advised Index and statistics

IBM System i


NotePre-V5R4, Index Advisor is accessed by browsing job log for query optimizer messages, Visual Explain (above), and Detailed DB monitor.

Having a good indexing strategy is imperative to achieving high query performance on any database platform. A comprehensive and efficient indexing strategy can mean the difference between queries that complete in seconds rather than minutes or hours. Implementing such a strategy is an important task. DB2* Universal Database* (UDB) for i5/OS* wants to lend a hand in the form of Index Advised, which is a feature that enumerates definitions for nonexistent indexes with a high potential for improving a query's performance. Not only does Index Advised help guide the creation of permanent indexes, but it's also the basis for creating a new flavor of temporary index in V5R4 called a Maintained Temporary Index (MTI). An understanding of how Index Advised works, how it's used to create MTIs and which tools are available to analyze this advice is essential for anyone looking to create a highly effective indexing strategy.

IBM System i


Enhanced Index Advisor in V5R4

• Index Advisor is built directly into the SQE optimizer. As the queries execute, DB2 deposits index advice entries into a new V5R4 catalog table named SYSIXADV in QSYS2 schema.

• Always On - Index Advisor automatically analyzes all SQL queries based on the rules outlined in the "Indexing and Statistics Strategy" whitepaper.(www.ibm.com/servers/enable/site/education/abstracts/indxng_abs.html)

• SQE uses advised indexes and Plan Cache as the basis for creating Maintained Temporary Index (MTI) – more on MTI later.

IBM System i


NoteIndex Advised itemizes the set of index definitions that the optimizer deems necessary to achieve better query performance. Because this feature is built directly into the optimizer, all SQL queries are automatically analyzed for new indexing opportunities based on the rules outlined in the "Indexing and Statistics Strategy" whitepaper (at http://www.ibm.com/servers/enable/site/education/abstracts/indxng_abs.html)

IBM System i


Advised indexes can be found:1. Pre-V5R4: within database monitor output under “Table Scan” (type 3000)

records. Only one advised index allowed per one record of this type.

2. V5R4 onwards: within database monitor output under the new type 3020 records. All advised indexes for the same table are now recorded.

3. Within Visual Explain under the Index and Statistics Advisor

4. From iSeries Navigator's Index Advisor facility

5. Via the SQL Plan Cache by filtering on "Queries with index advised“

In iSeries Navigator, advised indexes can be displayed:– At database instance level– Within a specific schema (library) of a specific database instance– For a specific table

Index Advisor

IBM System i


NoteWith V5R4, index advice can be found in four main areas:

1. Within database monitor output under 3020 records2. Within Visual Explain under the Index and Statistics Advisor3. From iSeries* Navigator's Index Advisor facility4. Via the SQL Plan Cache by filtering on "Queries with index advised"

The first two areas existed prior to V5R4. The latter two, introduced with V5R4, are the focus of thisarticle. First, we'll explain the new Index Advisor that's shipped with V5R4 and detail, through an example, how it can be used to implement an improved index strategy. Next, we'll show how to locate the advice and elaborate on what it means. Then we'll examine how this index advice information can be used in conjunction with the Index Evaluator to provide a more complete picture of your query environment.

Index Advisor Information in DBMON Output

Prior to V5R4, one way to extract information on advised indexes is from the DB Monitor output by looking for the “Table Access” records (type 3000). But each type-3000 record only contains a single advised index. If a query receives multiple advised indexes from the optimizer when scanning a table, there will be only one index advisor entry in the output. Moreover, if the query is implemented without accessing a table but an index is still advised, there will not be any entry recorded because there will not be a type 3000 record for that table. Starting in V5R4, all advised indexes are recorded in DB monitor output with the new record type 3020.

IBM System i


Invoking Index Advisor from iSeries Navigator

IBM System i


Index Advice Output ExampleDefault sorting

By default, the list is sorted by 3 columns: “Times Advised for Query Use” + “Most Expensive Query” + “Table Size”, all in descending order.

3 Levels of Index Advice

IBM System i


Complete Index Advice Attributes

Number of currently displayed entries against total number of entries

IBM System i


Note: Index Advice Attributes – Some Highlights

The cumulative number of times this index has been advised. This count should stop increasing when a matching permanent index is created. The row of advice will remain in this table until the user removes it.

Times Advised for Query Use

The timestamp representing the last time this index was advised for a query.Last Advised for Query Use

Binary Radix or Encoded Vector IndexIndex Type Advised

The keys at the beginning of the KEY_COLUMNS_ADVISED field that could be reordered and still satisfy the index being advised.

Leading Keys Order Independent

Column names for the advised index. The order of the column names is important. The names should be listed in the same order on the CREATE INDEX SQL statement, unless the leading, order independent key information indicates that the ordering can be changed.

Keys Advised

Schema or library name for the table.Schema

The optimizer is advising creation of a permanent index over this table. This is the long name format for the table. The advice was generated because the table was queried and no existing permanent index could be used to improve the performance of the query.

Table for Which Index was advised

DescriptionColumn Name

IBM System i


Note: Index Advice Attributes (continued)

Execution time in seconds of the longest-running query that generated this index advice.

Most Expensive QueryEstimate

Average execution time in seconds of all queries that generated this index advice.Average of Query Estimates (seconds)

The timestamp representing the last time this specific MTI was used by the optimizer to improve the performance of a query. The MTI Last Used field can be blank, which indicates that an MTI that exactly matches this advice has never been used by the queries that generated this index advice.

MTI LAST USED

The number of times that this specific MTI has been created by theoptimizer. MTIs do not persist across system IPLs.

MTI CREATED

The number of times that this specific MTI has been used by the optimizer. The optimizer will stop using a matching MTI when a permanent index is created.

MTI USED

Number of rows in the table for which the index is being advised, for the last time this index was advised.

Rows in Table when Advised

Coded reason why index was advised. Possible values: Row selection, Ordering/Grouping, Row selection and Ordering/Grouping.

Reason Advised

Estimated time required to create this index.Estimated Index Creation Time

DescriptionColumn Name

IBM System i


“Keys Advised” and “Leading Keys Order Independent”

• When generating advice, SQE first looks for local selection equal-predicates (e.g., table.column=constant).

– Columns against equal-predicates can be moved around in their position with other equal-predicate columns (Order Independent).

– Non-equal selection columns are generally order dependent.

• For table join, SQE adds join columns to the advised column listfollowing the columns from local selection predicates.

• For grouping or ordering, SQE may include the grouping and ordering columns after the local selection equal predicates.

– Only when all of the columns in the "group by“ or "order by" list are in the same table.

IBM System i


NoteIn general, Index Advised first looks for local selection equal predicates (e.g., table.column=constant). (Note: Index Advised can advise radix indexes or encoded vector indexes, or EVIs, to handle selection.)

For joining, the optimizer adds join columns to the advised column list following the columns from local selection equal predicates. If present, columns from in-list predicates or a column from a single inequality local selection predicate (e.g., >, <, != etc.) are appended to the list. These columns appear last because the number of records they select can vary widely. Also, whereas columns against equal predicates can be moved around in their position with other equal predicate columns, non-equal selection columns don't provide this flexibility.

For grouping or ordering, Index Advised may include the grouping and ordering columns after the local selection equal predicates, but this is only true when all of the columns in the "group by“ or "order by" list come from the same table. Additionally, the optimizer can recommend using a multiple-key column EVI to handle star-schema-type joins or simple grouping.

IBM System i


“Keys Advised” and “Leading Keys Order Independent”

• Although all entries in index advice are unique, some entries can present repetitive group of key columns.

– Example: when the same group of columns are advised in more than one entry but in a different order and the order of these keys is unimportant.

– This opportunity for reordering of the advised keys is indicated via the columns displayed in "Leading Keys Order Independent" column.

• These columns can be rearranged to create only one index that can still satisfy all instances of advice.

• This is when the new Condensed Advised Indexes function is useful – more on this in the next section.

IBM System i


NoteIn its raw form, index advice can be unnecessarily verbose. Although all entries in the table are unique, some entries can be redundant. One good example of this occurs when the same columns are advised but in a different order and the order of these keys is unimportant. In this case, the columns can be arranged to create one index that covers both instances of advice. This opportunity for reordering of the advised keys is indicated via the "Leading Keys Order Independent" column. The keys listed in this column are the leading keys which could be reordered while still satisfying the index being advised.

IBM System i


iSeries Navigator - User Actions on Index Advice

• Clear All Advised Indexes: as indexes are created or tables are dropped, the old advice entry isn't removed automatically by the system. It can be considered a historical table and as such will keep accumulating rows, so you may want to clear out the table occasionally.

• Condense Advised Indexes: provides an “intelligent” view for the creation of advised “perfect” index.

• Prune Advised Indexes: removes all index advice rows where the table that the advice was created for no longer exists.

• Create Index: provides a view of the recommended index to be created from the New Index dialog, including index type, key columns, column order, etc.

• Remove from List: delete the selected entries from SYSIXADV table.

• Show statements: display all the SQL statements that have advised index for the table referenced in the “Table Which Index Was Advised” column.

• Table functions of the one on which the entry is based.

IBM System i


iSeries Navigator - User Actions on Index Advice (cont.)

IBM System i


NoteMany actions can be taken on the entries in the list. First we'll explain the actions that work against the entire list. For example, the underlying datastore for the advice is an SQL table (QSYS2/SYSIXADV), which is populated by the database but not maintained. As indexes are created or tables are dropped, the old advice isn't removed automatically by the system. It can be considered an historical table and as such will keep accumulating rows, so you may want to clear out the table occasionally. You can use the Clear Advised Indexes menu option. For the database-wide and schema-wide views you have an additional option to "Prune Advised Indexes." This option removes all index advice rows where the table that the advice was created for no longer exists. There's also a menu option to remove the selected row of index advice from the list.

Let's describe the menu options for each of the indexes advised in the list:

The Create Index option, which you can use to display the advised index in the New Index dialog, provides a view of the recommended index as it would be if you created it from the New Index dialog, including index type, key columns, column order, etc. Additionally, on the New Index dialog you can use the Show SQL button to display the CREATE INDEX DDL statement in Run SQL Scripts. From there, you can copy and paste this SQL statement into your change management system. In our INDEXADVIC example, two rows of advice can be satisfied by creating an index with key columns SALES_PERSON ASC and SALES_DATE ASC. A third row is satisfied by creating an index with key columns SALES_DATE ASC, REGION ASC and SALES_PERSON ASC.

IBM System i


NoteShow Statements shows all statements in the Query Plan Cache (QPC) that referenced this table and generated index advice. With this information, determine which other query statements have generated index advice over the same table, and what additional positive impact may be gained by creating the index. (Note: This option only displays SQL statements that were executed using the SQE query engine.)

A cascading Table menu is available for each index advice that allows you to perform many of the table actions that you can perform for a particular table in the list - view the contents, definition, or description and several other options. The most intriguing of these options may be the Show Indexes menu option. If you select this option, you can launch the Show Index list to see all of the indexes currently defined over the table. By using the Index Advice list in conjunction with the Show Indexes list, you can work on your table's indexing strategy. For example, if one index-advice row shows that you should have an index with certain columns but you already have some of those columns in an existing index, try adding a column recommended by the Index Advisor to your existing index..

IBM System i





IBM System i


In Search of the “Perfect” Indexes

• A “perfect” index is an index created to satisfy as many queries as possible that run against the table on which this index is built.

– Less indexes implies less complex DB environment to manage

• In the past, DB2 for i5/OS provided Index Advisor to help SQL programmers in identifying the set of efficient candidate key columns

– Each query must be examined by an SQL programmer for the advised indexes in the job log or Visual Explain

– Practically, too many queries impose daunting amount of time andeffort for programmers to analyze and “collapse” many advised indexes to as few as possible – not an easy task!

• Here comes a new tool in V5R4 for this task

IBM System i


Condense Advised Indexes for “perfect” indexes • When you see columns displayed in Leading Keys Order Independent

of the Index Advisor, there is an opportunity for you to “collapse”multiple entries into one to facilitate creation of a “perfect” index.

• Use the new Condensed Advised Indexes function in iSeries Navigator to do this hard work for you!

– Requires PTFs: SI25391, SI25469, and SI25470 for 5722-SS1 at V5R4M0– Also the latest Service Pack for iSeries Access for Windows V5R4M0– Like Index Advisor, it is available at database instance, schema, and table

levels

• Condensed advice entries are materialized from a new V5R4 view named CondensedIndexAdvice provided in QSYS2 schema.

– CondensedIndexAdvice view uses a new V5R4 UDTF named Condense_Advice() to process QSYS2/SYSIXADV for the materialized result

– Example query for result: SELECT * FROM QSYS2.CONDENSEDINDEXADVICE WHERE table_schema = ….. ;

IBM System i


NoteThe condense index advice technology requires the following V5R4M0 5722-SS1 PTFs to be installed on the server: SI25391, SI25469, and SI25470.

When index advice is condensed, the raw index advice is placed into a summary form. Some fields within the raw advice are specific to the instance of advice and have no value in a summarized form. Other fields are quite important to preserve and weight the importance of the condensed advice. The condensed index advice permits the same iSeries Navigator actions as those found under the Index Advisor, with one exception. Since the condensed index advice does not exist in a permanent form, there is no "Remove from List" action.

IBM System i


CondensedIndexAdvice view definitionQSYS2.CONDENSEDINDEXADVICE ( TABLE_NAME FOR COLUMN TABNAME VARCHAR(258) CCSID 37 NOT NULL , TABLE_SCHEMA FOR COLUMN TABSCHEMA CHAR(10) CCSID 37 NOT NULL , SYSTEM_TABLE_NAME FOR COLUMN SYS_TNAME CHAR(10) CCSID 37 NOT NULL , PARTITION_NAME FOR COLUMN TABPART VARCHAR(128) CCSID 37 DEFAULT NULL , KEY_COLUMNS_ADVISED FOR COLUMN KEYSADV VARCHAR(16000) CCSID 37 DEFAULT NULL , INDEX_TYPE CHAR(14) CCSID 37 DEFAULT NULL , LAST_ADVISED FOR COLUMN LASTADV TIMESTAMP DEFAULT NULL , TIMES_ADVISED FOR COLUMN TIMESADV BIGINT DEFAULT NULL , ESTIMATED_CREATION_TIME FOR COLUMN ESTTIME INTEGER DEFAULT NULL , LOGICAL_PAGE_SIZE FOR COLUMN "PAGESIZE" INTEGER DEFAULT NULL , MOST_EXPENSIVE_QUERY FOR COLUMN QUERYCOST INTEGER DEFAULT NULL , AVERAGE_QUERY_ESTIMATE FOR COLUMN QUERYEST INTEGER DEFAULT NULL , TABLE_SIZE BIGINT DEFAULT NULL , NLSS_TABLE_NAME FOR COLUMN NLSSNAME CHAR(10) CCSID 37 DEFAULT NULL , NLSS_TABLE_SCHEMA FOR COLUMN NLSSSCHEMA CHAR(10) CCSID 37 DEFAULT NULL )

-- Example: Condensed index advice for a range of schemas, where the average

-- query estimate driving the index advice is greater than 10 seconds

SELECT * FROM QSYS2.CONDENSEDINDEXADVICE

WHERE table_schema LIKE 'Q%' AND average_query_estimate > 10 ;

IBM System i


NoteBesides providing the user with the condensed column key order, the condensed advice includes contextual information to aide in determining the importance of the advice. Columns, such as "Times Advised for Query Use" and "Average of Query Estimates", provide a sense of how beneficial the permanent index could be to this environment. While the "Estimated Index Creation Time" column, helps to determine whether the creation of the index needs to be a scheduled activity.

IBM System i


Condense_Advice() UDTF definitionCREATE FUNCTION QSYS2.Condense_Advice(TABLE_SCHEMA VARCHAR(128), TABLE_NAME VARCHAR(128) )

RETURNS TABLE(TABLE_PARTITION VARCHAR(128), KEY_COLUMNS_ADVISED VARCHAR(16000),

INDEX_TYPE CHAR(14), LAST_ADVISED TIMESTAMP, TIMES_ADVISED BIGINT,

ESTIMATED_CREATION_TIME INTEGER, LOGICAL_PAGE_SIZE INTEGER,

MOST_EXPENSIVE_QUERY INTEGER, AVERAGE_QUERY_ESTIMATE INTEGER,

TABLE_SIZE BIGINT, NLSS_TABLE_NAME CHAR(10),NLSS_TABLE_SCHEMA CHAR(10))LANGUAGE C NOT DETERMINISTIC READS SQL DATA CALLED ON NULL INPUT SCRATCHPAD 325064 DISALLOW PARALLEL FINAL CALL CARDINALITY 1 EXTERNAL NAME 'QSYS/QDBSSUDF2(CONDENSE_ADVICE)' PARAMETER STYLE DB2SQL;

-- Example: Query the condenser UDTF directly, providing selection criteria

-- and ordering the results of the 15 most important condensed entries

SELECT * FROM TABLE(QSYS2.CONDENSE_ADVICE('CRPDTA','B123456')) AS a

WHERE a.table_size >(1024*1024) AND MONTH(last_advised)= MONTH(CURRENT TIMESTAMP)

ORDER BY average_query_estimate DESC

FETCH FIRST 15 ROWS ONLY ;

IBM System i


NoteThe Condense_Advice UDTF used by the new CondensedIndexAdvice view can also be accessed directly by user queries.

IBM System i


C1C1, C3, C2C1, C2, C3C1, C2, C3

Leading Keys Order independentKeys Advised

How Condense Advised Indexes works – example 1

• In this example, the columns in the second entry cannot be reordered.

• To satisfy the index advice, a permanent index would need to have keys in the order { C1, C3, C2 }. However, the first entry of advice has greater flexibility. The “Leading Keys Order Independent” detail shows that they can be in any order, so you can condense the two entries of advice into a single permanent index by reordering { C1, C2, C3 } into { C1, C3, C2 }.

• CREATE INDEX … ON TABLE_A(C1, C3, C2) will cover both entries

Advised Indexes Entries

IBM System i


NoteIn this example, the second row of advice is not pliable. To satisfy the index advice, a permanent index would need to have keys in the order { C1, C3, C2 }. However, the first row of advice has great flexibility. The leading key order independent detail shows that the keys could be in any order, so you can condense the two rows of advice into a single permanent index by reordering { C1, C2, C3 } into { C1, C3, C2 }.

IBM System i


C1C1, C3, C2

C1, C2, C3, C4C1, C3, C2, C4

C1, C2, C3C1, C2, C3Leading Keys Order independentKeys Advised

• If you extend the previous example to include a third entry of advice as indicated above, you are still able to conclude that the addition of a single permanent index { C1, C3, C2, C4 } should provide SQE all it needs to permanently improve the performance of these three queries.

• CREATE INDEX … ON TABLE_A(C1, C3, C2, C4) will cover all three entries

How Condense Advised Indexes works – example 2

Advised Indexes Entries

IBM System i


NoteIf you extend the preceding example to include a third row of advice, you are still able to conclude that the addition of a single permanent index { C1, C3, C2, C4 } should provide the optimizer all it needs to permanently improve the performance of these queries.

IBM System i


So, from these sample Advised Indexes

YouSelectThisFunction

IBM System i


You get the following Condensed Advice

The Index Advisor Condenser produces this result for you at a mouse click!

The condensed advice includes contextual information to aide in determining the importance of the advice. Columns, such as "Times Advised for Query Use" and "Average of Query Estimates", provide a sense of how beneficial the permanent index could be to this environment. While the "Estimated Index Creation Time" column helps to determine whether the creation of the index needs to be a scheduled activity. The number of times advised is summed together for display, and the longest estimated creation time is used. So, not only are the key columns condensed, but other columns are processed as well.

IBM System i


• With Index Advisor and its Condenser, you are capable of creating a few “perfect” indexes that can satisfy as many queries as possible for a table

• What about other existing indexes of that table that have been around?

– Are there any of them that are rarely used or not used at all since they were created?

– Is there a tool to help identify these ? Yes, use Index Evaluator.

• Index Evaluator is invoked by right-clicking the table of your interest in iSeries Navigator and select option “Show Indexes”

– Available since V5R2

– Show all of the indexes currently defined over the table of your interest.

Managing Existing Indexes with Index Evaluator

IBM System i


Invoking Index Evaluator

IBM System i


• When both columns Query Used Count and Query Statistics Used Count of any index entry display a value of 0 (zero), this indicates a candidate index to be deleted due to lack of use.

– Except for Temporary Indexes. Leave them to the system.– Make sure you know how long these counts have been active.– If in doubt, you can restart the index usage counting by issuing the

following command against the index object to reset the counting to zero: CHGOBJD OBJ(schema/index) OBJTYPE(*FILE) USECOUNT(*RESET)). You then return to display the usage counting some times later.

• Also look at the Last Used Date column. This column is updated when native read/write operation is done to a keyed logical file or physical file. So, a date displayed in this column indicates how long ago the index was used by the native operation.

– This column information was newly added in V5R4.

Index Evaluator Provides Usage Information

IBM System i


NoteFor existing indexes, it might be difficult to gauge the effect of a change in index strategy because the usage statistics have accumulated over time. Use the Change Object Description (CHGOBJD) command to reset the statistics for existing indexes:

CHGOBJD OBJ(schema/index) OBJTYPE(*FILE) USECOUNT(*RESET))

IBM System i





IBM System i


Autonomic Indexes

• Also called Maintained Temporary Index (MTI)

• Classic Query Engine (CQE) has had the ability to create a temporary index but its usage is restricted to a single job and single query only.

• MTIs created by SQE can be shared across queries and jobs like permanent indexes.

IBM System i


Maintained Temporary Index (MTI)

• SQE automatically controls the creation, maintenance and deletion of MTIs.

– Creation occurs when SQE can justify that a query's performance can be enhanced sufficiently by using an MTI.

• An MTI is updated as the base table is updated. – The system cost of maintaining an MTI is similar to the cost of

maintaining a permanent index.

• An MTI is deleted when:– The last access plan, in the system plan cache, that refers to that MTI is

removed (e.g. IPL). Needs QAQQINI parameter CACHE_RESULTS = *SYSTEM

– When a permanent index is created that covers the same columns as the MTI

IBM System i


NoteWhile Index Advised is a highly effective end-user tool for improving query performance, it's also the basis for the creation of another new tool in the Structured Query Engine (SQE) optimizer's toolbox -the MTI. An MTI is a temporary radix index that the SQE optimizer can create to improve a query plan's performance. It has an advantage over other temporary result sets created by the optimizer: it's maintained. Like a permanent radix index, an MTI is updated as the underlying table is updated. Therefore, the cost of maintaining an MTI is similar to the cost of maintaining a permanent radix index. The optimizer creates an MTI for a query plan for two main reasons. One, an index is functionally necessary to implement a query when a temporary result set isn't allowed. A query running under a sensitive cursor is one example where a non-maintained temporary result set can't be used to implement a query. The other reason the optimizer can choose to create an MTI is that the query will perform better with an index. While the Classic Query Engine (CQE) has the ability to create a temporary index, its usage is restricted to a single job and single query only. SQE's MTIs, on the other hand, can be shared across queries and jobs just like a permanent index. SQE automatically controls the creation, sharing and deletion of MTIs. Creation of an MTI occurs when the optimizer determines that a query's performance can be enhanced sufficiently by using the MTI, thus justifying the MTIs create cost. An MTI is deleted when the last plan in the system plan cache referencing it is removed (e.g., IPL) or when a permanent index is created that covers the same columns as the MTI. To avoid incurring MTI create costs after an IPL, consider creating permanent indexes for regularly used MTIs. MTI usage information can be viewed using iSeries* Navigator or the Database Monitor.

IBM System i


MTI Information Example

IBM System i


MTI Control Mechanism

• When SQE sees that a query runs frequently, it is more likely to create the advised MTI for that query.

– Justification for creation : the one-time cost of creating the MTI must be more than offset by the runtime savings in subsequent queries

– SQE interrogates the system plan cache to garner this "historical use information“

• ALLOW_TEMPORARY_INDEXES parameter of QAQQINI control file:– *YES = SQE is free to create MTIs as it sees fit

– *ONLY_REQUIRED = SQE will create MTIs only when they are functionally necessary for sensitive (live data) queries

• To minimize overhead of creating MTIs at query run-time, SQE create them for relatively small and moderately sized files only.

– There is still no substitute for a good indexing strategy.

IBM System i


NoteNow that the SQE optimizer has the ability to create maintained, shareable indexes, it relies on feedback from Index Advised to create those MTIs. When Index Advised recommends an index that may help a query perform better, the optimizer considers the effect on the query performance of creating that index. The cost of creating the index is considered, but it's discounted the more times a query runs. So, the first time a given query runs, creating an MTI based on information from Index Advised may not be justified. Other techniques such as hash join and table scan are typically more appropriate for a query that has only run once. If the optimizer sees that a query runs frequently, it's more likely to create the advised MTI. The justification is based on the fact that the one-time cost of creating the MTI will be more than offset by the runtime savings in subsequent query runs of this query and others. To garner this "historical use information," the optimizer interrogates the system plan cache.

The QAQQINI file gives users the ability to override some of the initialization settings used by the optimizer during optimization. Two QAQQINI parameters affect MTI creation. One is the ALLOW_TEMPORARY_INDEXES parameter, which has two values: *YES and *ONLY_REQUIRED. The default value for this parameter is *YES. When *YES is specified, the optimizer is free to create MTIs as it sees fit. When the *ONLY_REQUIRED value is specified, the optimizer will create MTIs only when they're functionally necessary for sensitive (live data) queries. By specifying *ONLY_REQUIRED, indexes recommended by Index Advised aren't created.

The other QAQQINI parameter that affects MTIs is the CACHE_RESULTS option, which helps determine how long MTIs stay in the system. When the default value of *SYSTEM is in effect, MTIs stay in the system as long as there's a referencing plan in the system plan cache. When the *JOB or *NONE values are in effect, an MTI is deleted when all job's consuming it end. Note that the CACHE_RESULTS QAQQINI parameter only affects reusable MTIs. MTIscreated for sensitive cursor queries are typically deleted when the job ends.

IBM System i


Take Back Control

DB2 for i5/OS V5R4 Enhancements - Summary

System i’s SimplicityEvery component of the System i was designed specifically to support business needs, and it integrates those components for superior functionality. Hardware, OS, database, security, middleware and storage - an all-in-one sum that's truly greater than its parts.

• SQE in V5R4 delivers more versatility in SQL performance tuning area

• These enhancements are easy to use or automated

IBM System i


• APAR SE27116 - OSP-DB ADD INDEX ADVICE CONDENSER

This is an add function PTF to deploy a Database Index Advice Condenser. The Index Advisor, a feature introduced to DB2 for i5/OS V5R4, records statistics on a system's indexing needs. Each time a query is run in which a permanent index is recommended, the Index Advisor records the index which could be used to improve that query's performance. Each time a unique index is advised, a new entry is created in the table. If multiple advice occurs on the same unique index, the information is added to the existing entry. For example, the "Number of Times Advised“ entry is incremented, and the "Last Advised for Query Use" is updated with the current time.

In its raw form, index advice can be unnecessarily verbose. Although all entries in the table are unique, some entries can be redundant. One good example of this occurs when the same columns are advised but in a different order and the order of these keys is unimportant. In this case, the columns can be arranged to create one index that covers both instances of advice.

The following examples assume all advice is on the same table, same partition, with the same index type and NLSS schema and table.

Keys Advised Leading Keys, Order Independent C1, C2, C3 C1, C2, C3 C1, C3, C2 C1, C3, C2

In this case, since all columns are in both sets and all are order independent, the two instances of advice would be condensed into either { C1, C3, C2 } or { C1, C2, C3 }, depending upon the order of the two instances of advice in the Index Advisor table.

Additional Information on Index Advisor Condenser

IBM System i


Objects added by this PTF: QSYS2/Condense_Advice() UDF and QSYS2/CondensedIndexAdvice View.

The CondensedIndexAdvice view can be queried directly to condense advice. iSeries Navigator users will be given a graphical solution in the form of Database Schema action.

Example usage:

-- This query returns uncondensed index advice for a specific table:

select * from qsys2.sysixadv where TABLE_SCHEMA = 'MYlibrary' AND TABLE_NAME = 'Mytable';

-- This query returns condensed index advice for a specific table:

select * from qsys2.CondensedIndexAdvice where

TABLE_SCHEMA = 'MYlibrary' AND TABLE_NAME = 'Mytable';

APAR SE27116 (cont.)

IBM System i


• Indexing and Statistics Strategies whitepaper:

http://www.ibm.com/servers/enable/site/education/abstracts/indxng_abs.html

• iSeries* Navigator:

http://public.boulder.ibm.com/infocenter/iseries/v5r4/index.jsp?topic=/rzajq/queryopt.html

• DB2* UDB for i5/OS SQL and Query Performance Tuning and Monitoring Workshop:

http://www.ibmsystemsmag.com/ME2/Audiences/Segments/Publications/www.ibm.com/eserver/iseries/service/igs/db2performance.html

• QAQQINI Information:

http://publib.boulder.ibm.com/pubs/html/as400/v4r4/ic2924/info/db2/rbafymst02.htm#ToC_689

• Viewing the Query Plan Cache Through iSeries Navigator:

http://publib.boulder.ibm.com/infocenter/iseries/v5r4/index.jsp?topic=/rzajq/rzajqdisplayplancache.html

Additional Information: more reference

IBM System i


• "Preparing for and Tuning the SQL Query Engine on DB2 for i5/OS" (IBM Redbooks): Minimize any SQL or database performance issues when upgrading to OS/400 V5R2 or V5R3 or to IBM i5/OS V5R4:

http://www.redbooks.ibm.com/abstracts/sg246598.html

• "OnDemand Index Advice for DB2 for i5/OS": Read additional information on the DB2 for i5/OS Index Advisor:

http://www.ibm.com/servers/eserver/iseries/db2/index_advisor.html

Additional Information: more reference

Documents

Index Advisor and Autonomic Indexes