Upload
others
View
2
Download
0
Embed Size (px)
Citation preview
Application Development Success
Stories for Performance Issues
on DB2 Z/OS
Speakers: Keith Tidball & Lori Zaremba
Progressive Insurance
Session Code: E06
May 4, 2011 9:45am | Platform: DB2 Z/OS
2 Copyright 2011 Progressive Insurance. May not reproduce without permission.
I. About us
II. How we find performance opportunities
III. Performance success stories
IV. Lessons Learned
V. Q & A
AGENDA
3 Copyright 2011 Progressive Insurance. May not reproduce without permission.
About Us: Speaker Background
Keith Tidball
Lead Application Developer, Progressive Insurance 12 years mainframe development experience 9 years experience working with DB2
Lori Zaremba
Lead Application Developer, Progressive Insurance 15 years experience working with DB2 3 years working with SQL Server
Sorry, no Flo this year at IDUG.
But she promised to upgrade her ‘tricked out name tag’ in our
honor, though…
4 Copyright 2011 Progressive Insurance. May not reproduce without permission.
Claims
DMG
Billing Quoting
Payroll
Two Data Centers
Cleveland Colorado Springs
About Us: DB2 at Progressive
5 Copyright 2011 Progressive Insurance. May not reproduce without permission.
546 Databases
10 Way Data Sharing Env.
31,000+ Tables
Several Tables over 1 BILLION Rows
40 Terabytes of data
About Us: DB2 at Progressive
6 Copyright 2011 Progressive Insurance. May not reproduce without permission.
About Us: Progressive Billing System
ProBill DB2 Facts:
• Total programs = 8,023
• Total Tables = 423
• Total Views = 429
• Total Indexes = 637
• Triggers = 16
• Stored Procs = 21
• Sequence Objects = 8
• Production Data = 5 terabytes
7 Copyright 2011 Progressive Insurance. May not reproduce without permission.
More ProBill Facts: • 24x7 real-time payment processing and billing services for
all Progressive lines of business.
• Interacts with customers more frequently than any other of Progressive's services as many upfront systems access & display Billing data.
• Provides complete and accurate view of our insureds' accounts, coupled with rigorous financial controls.
• Supplies billing information to external customers, agents, internal service reps, and financial staff.
About Us: The Progressive Billing System
8 Copyright 2011 Progressive Insurance. May not reproduce without permission.
How We Find Performance Opportunities
9 Copyright 2011 Progressive Insurance. May not reproduce without permission.
DB2 Support Team
Team consists of: • DB2 Developers • Performance Testers • DBA
Meeting Frequency: Weekly
Meeting Objectives: • Review upcoming production SQL changes • Review/Brainstorm current issues • Update DB2 standards as necessary with new releases • Share Tips/Tricks
How we find performance opportunities
10 Copyright 2011 Progressive Insurance. May not reproduce without permission.
Stress testing A. Changes deemed as performance enhancements are stress
tested B. Measure and compare changes to baseline tests:
• MIPS • CPU • Run Time • Wait Time
C. All measurements are reviewed and signed off prior to
installing to production.
D. Post-production implementation performance review.
How we find performance opportunities
11 Copyright 2011 Progressive Insurance. May not reproduce without permission.
Performance Instrumentation Database
• Collect data to support the analysis and determination of the causes of performance degradation and performance hits.
• Data originates from:
i. SARQ for batch jobs and job step information
ii. DB2 DBA tables for DB2 program information
iii. Billing infrastructure tables for transaction information
• Compare application performance with historical data.
• Anticipate future performance.
• Report performance of online and unattended processes.
• SQL Server Database using SSRS for reports/charts
*** you will see examples of these reports later in the slideshow…
How we find performance opportunities
12 Copyright 2011 Progressive Insurance. May not reproduce without permission.
Post Implementation Issues (yes, they do happen sometimes!)
A. Post implementation performance issues are typically found by:
• DBA/Enterprise Application Performance Engineering groups • Post production implementation performance review • Application development team • Monthly Performance review meetings
B. Once a performance issue is identified – the application team is engaged with the DB2 support team to derive the solution.
How we find performance opportunities
13 Copyright 2011 Progressive Insurance. May not reproduce without permission.
How we find performance opportunities
Work With Our DBA
A. Our DBA is named Matt and he’s not shy about addressing performance related items when it comes to the ProBill database.
B. We continually work with Matt and other DBA’s to identify performance opportunities.
C. There are two reports that are used by the DBA group and ourselves to help with this process.
i. Query to show SQL statement usage times by program
ii. DB2 MIPS usage by program
14 Copyright 2011 Progressive Insurance. May not reproduce without permission.
How we find performance opportunities
SQL Statement Usage by Program
• Query against data compiled from Detector provides a handy report for programs using the most time in DB2.
SELECT DISTINCT(PROGRAM),STMT#,SQL_CALL,
SUM(INDB2_TIME) AS INDB2_TIME,
SUM(SQL_CALLS) AS SUM_SQL_CALLS,
SUM(INDB2_CPU) AS CPU_TIME,
SUM(IO_WTIME) AS IOWAIT,
SUM(LOCK_WTIME) AS LOCKTIME,
SUM(GETPAGE) AS GETPAGE
FROM XXX.PDT_STANDARD_115
WHERE INTERVAL_START BETWEEN '2011-02-10-00.00.00.000000'
AND '2011-02-11-00.00.00.000000‘)
AND PROGRAM LIKE ‘AAA%’
GROUP BY PROGRAM,STMT#,SQL_CALL
HAVING SUM(INDB2_CPU) > 1000
ORDER BY 6 DESC
This query is very
flexible. It’s easily
tailored to be as
general or specific
as you need.
15 Copyright 2011 Progressive Insurance. May not reproduce without permission.
How we find performance opportunities
PROGRAM STMT# SQL CALL INDB2 TIME
SUM SQL
CALLS CPU TIME IOWAIT LOCKTIME GETPAGEAAA 541 PREPARE 14,475 54,719,793 8,917 22 2,245 72,071,626
BBB 583 EXECUTE 39,547 10,613,835 3,651 26,078 1,616 117,097,052
CCC 586 FETCH 44,982 54,719,793 3,526 39,350 1,084 198,728,294
DDD 1860 DECLARE GLOBAL T 4,193 450,635 2,949 3 460 123,023,355
111 1551 OPEN 27,025 4,495,301 2,919 21,362 721 101,918,500
FFF 626 OPEN 7,535 5 2,795 1,779 4 14,104,260
333 2128 SET HOST VAR 7,794 15,096,233 1,793 13 890 31,393,421
HHH 672 OPEN 8,068 5 1,651 3,865 2 13,471,089
DDD 1825 UPDATE 3,106 7,548,142 1,550 9 653 105,818,246
WWW 117 SELECT 2,245 31,161,850 1,537 1 339 62,323,690
444 1882 UPDATE 3,094 7,548,128 1,521 42 784 105,817,883
JJJ 815 OPEN 7,124 8,647 1,521 3,690 22 70,130,429
SQL Statement Usage by Program
• Example of Query results ordered by CPU usage:
16 Copyright 2011 Progressive Insurance. May not reproduce without permission.
How we find performance opportunities
DB2 MIPS Usage by Program
• SMF data from DB2 Traces in DB2PM gets dumped to flat files and then moved to SAS datasets and DB2 tables.
• ITRM is used to submit SAS against the data and this eventually gets dumped to reports.
• Initially created to charge back MIPS to individual apps. Before it was one large ‘pool’ & costs were spread evenly.
• These reports can give us the following stats by programs: i. MIPS used during ‘Peak Times’ (9a – 11a & 2p – 4p)
ii. # SQL executions
iii. UOW breakdowns (wait times, elapsed times, MIPS, etc)
17 Copyright 2011 Progressive Insurance. May not reproduce without permission.
Top 25 MIPS Daily Consumers for Probill DB2 Packages
M-F, Peak Prime Hours (ranked by last day actuals)
0
500
1000
1500
2000
2500
12
/16
/20
10
12
/17
/20
10
12
/20
/20
10
12
/21
/20
10
12
/22
/20
10
12
/23
/20
10
12
/24
/20
10
12
/27
/20
10
12
/28
/20
10
12
/29
/20
10
12
/30
/20
10
12
/31
/20
10
01
/03
/20
11
01
/04
/20
11
01
/05
/20
11
01
/06
/20
11
01
/07
/20
11
01
/10
/20
11
01
/11
/20
11
01
/12
/20
11
01
/13
/20
11
01
/14
/20
11
01
/17
/20
11
01
/18
/20
11
01
/19
/20
11
01
/20
/20
11
01
/21
/20
11
01
/24
/20
11
01
/25
/20
11
01
/26
/20
11
01
/27
/20
11
01
/28
/20
11
Application groups based on AuthID, PackageID, Collection ID. Select "refresh" for current data.
25 - _OTHER
24 - GS2_BIL
23 - PFN_BIL
22 - QLG_BIL
21 - PPS_BIL
20 - PPA_BIL
19 - INS_BIL
18 - PNA_BIL
17 - BPT_BIL
16 – TAB_BIL
15 - BIV_BIL
14 - BED_BIL
13 - PUB_BIL
12 - PNC_BIL
11 - LRT_BIL
10 - DEL _BIL
9 - BBH_BIL
8 - OAX _BIL
7 - OAD _BIL
6 - CPY _BIL
5 - BMS_BIL
4 - PSD_BIL
3 - AAA _BIL
2 - BBP_BIL
1 - SEL_BIL
How we find performance opportunities
Example of MIPS Usage Report
• This report shows the top 25 MIPS consuming programs each day during peak hours for the ProBill Application.
Program Names
18 Copyright 2011 Progressive Insurance. May not reproduce without permission.
Performance Success Stories
OK, some were more successful than others….
19 Copyright 2011 Progressive Insurance. May not reproduce without permission.
EXAMPLE 1:
Success Without Using DB2
20 Copyright 2011 Progressive Insurance. May not reproduce without permission.
PROBLEM:
Contention issues with Critical table that houses rules for all transactions run within the Billing system. Accessed 40+ million times a day by COBOL program (PC04).
Replicated the DB2 table into TableBase and it provided speed & performance needed.
Over the years we ran into a few issues with this model:
• The timing of TableBase refresh was different than that of our programs. • Developers would forget to replicate their changes to it. • ProBill is trying to phase out TableBase usage where possible.
PROPOSED SOLUTION:
Discontinue using TableBase table and use the source table in DB2, leveraging newer features of DB2 that were missing when we first developed the system.
Executed successful stress tests using DB2 version of the table and performance
was on par with TableBase. We shared the results with DBAs and Systems Analysts and no one objected.
All this looked great and it went live in Production….
Example 1
21 Copyright 2011 Progressive Insurance. May not reproduce without permission.
RESULT:
PC04 was performant as expected, but with…
15,000+ seconds of CPU and close to 300 million calls to the DB2 engine daily.
Program PC04 rose to the top of the DBA’s ‘Most Wanted’ list within a week. This resulted in all our stats being reassessed and a call for a non-DB2 solution.
Example 1
Guaranteed Call from
Angry DBA!!
22 Copyright 2011 Progressive Insurance. May not reproduce without permission.
FINAL SOLUTION:
Used a CICS UMT to store the table data and created a more robust replication process from DB2 (via scheduled jobs).
Performance of PC04 is the same, but overall CPU usage down using UMT vs. DB2.
DB2 is still used as failover contingency within PC04 if UMT ever has an issue.
DBA’s now talking with us again (somewhat cordially, too…)
Example 1
Eliminating DB2 Calls =
DBA stopped threatening
us with bodily harm…
23 Copyright 2011 Progressive Insurance. May not reproduce without permission.
EXAMPLE 2:
Success by Archiving DB2 Data
24 Copyright 2011 Progressive Insurance. May not reproduce without permission.
PROBLEM:
Database had 5 Terabytes of data that spanned over 6 years.
Degradation of application program performance against larger tables.
Business users really liked having all the data all the time.
PROPOSED SOLUTION:
Reduce the size of our database.
Identified tables that were really transactional in nature (i.e. data is useful for a set period of time only)
Business Users helped set ‘cutoff’ for how long to keep data.
Example 2
25 Copyright 2011 Progressive Insurance. May not reproduce without permission.
RESULT:
We saw decreases in processing and run time across several programs. Here is one example where DB2 CPU time dropped nicely for program ECD.
Example 2
Ran Reorg w/ Discard
and dropped 65% of the
data rows.
DBA Happy Place
26 Copyright 2011 Progressive Insurance. May not reproduce without permission.
EXAMPLE 3:
Several Attempts Before Reaching Success
27 Copyright 2011 Progressive Insurance. May not reproduce without permission.
PROBLEM:
Batch Job CD77 run time steadily increasing due to called API transitioning from MF to Server.
Quoted
Policy
Data
Populate
needed
reporting info
(CD77 Job)
Quote
Data
API to gather
data from Policy
systems
Existing Policy System was based on the
MF, but its replacement was server
based. Policy volume was slowly growing
in the new system.
Quote
Balancing
Reports
Example 3
28 Copyright 2011 Progressive Insurance. May not reproduce without permission.
Job CD77 run time steadily increasing from 150 minutes to over 300 minutes! (about a 3 month period)
Run time increasing with more calls to server.
Example 3
29 Copyright 2011 Progressive Insurance. May not reproduce without permission.
Solution #1:
Remove calls to API and get data from DB2 instead via queries.
Billing
Tables
Billing
Tables
Quoted
Policy
Data
Populate
needed
reporting info
(CD77 Job)
Quote
Data
API to gather
data from Policy
systems
Quote
Balancing
Reports Billing
Tables
XX
Example 3
30 Copyright 2011 Progressive Insurance. May not reproduce without permission.
Solution #1 Results:
The changes performed poorly in production and had to be demoted.
(Programming Tip: Try to avoid causing HUGE upward spikes in runtime…)
Demoted changes and resumed calling the API.
Example 3
31 Copyright 2011 Progressive Insurance. May not reproduce without permission.
Example 3
Billing
Tables
Billing
Tables
Quoted
Policy
Data
Populate
needed
reporting info
(CD77 Job)
Quote
Data
API to gather
data from Policy
systems
Quote
Balancing
Reports Billing
Tables
X
X
Added one data element
to this table from the
Query that was taking the
majority of time to run
Removed the longest &
costliest call
X
Solution #2:
Cache the data element that was the most inefficient DB2 call and then get remaining data from existing DB2 tables within Billing vs. calling the API.
32 Copyright 2011 Progressive Insurance. May not reproduce without permission.
Solution #2 Results: Better than the first try, but still unacceptable from a performance
standpoint.
Example 3
(Solution #1)
(Solution #2)
33 Copyright 2011 Progressive Insurance. May not reproduce without permission.
Example 3
Solution #3:
Cache ALL data elements and eliminate the SQL & API calls entirely.
Billing
Tables
Billing
Tables
Quoted
Policy
Data
Populate
needed
reporting info
(CD77 Job)
Quote
Data
API to gather
data from Policy
systems
Quote
Balancing
Reports Billing
Tables
X
X
Added ALL data
elements to this table
and eliminated the
queries all together
X
XX
X
34 Copyright 2011 Progressive Insurance. May not reproduce without permission.
Solution #3 Results: Results were impressive. Run times dropped from hours to under 10
minutes! SQL calls and I/O were reduced to almost nil.
Example 3
(Solution #1)
(Solution #2)
(Solution #3)
Third time’s the
charm!!!
35 Copyright 2011 Progressive Insurance. May not reproduce without permission.
EXAMPLE 4:
MIPS for Matt
36 Copyright 2011 Progressive Insurance. May not reproduce without permission.
PROBLEM:
MIPS usage during ‘peak’ hours was driving a need to purchase engine upgrades, but our lease on the mainframes was expiring within 12 mos.
Example 4
37 Copyright 2011 Progressive Insurance. May not reproduce without permission.
PROPOSED SOLUTION:
‘MIPS for Matt’ was born as we felt our DBA was a charity worth supporting fully.
Identified programs for performance improvements by revising or eliminating SQL.
Identified processes that could run during off-peak hours.
Example 4
38 Copyright 2011 Progressive Insurance. May not reproduce without permission.
Example 4
Removed SQL no longer needed
Revised SQL & scheduling
Run during Off-Peak hours
39 Copyright 2011 Progressive Insurance. May not reproduce without permission.
Top 25 MIPS Daily Consumers for Probill DB2 Packages
M-F, Peak Prime Hours (ranked by last day actuals)
0
500
1000
1500
2000
2500
12
/16
/20
10
12
/17
/20
10
12
/20
/20
10
12
/21
/20
10
12
/22
/20
10
12
/23
/20
10
12
/24
/20
10
12
/27
/20
10
12
/28
/20
10
12
/29
/20
10
12
/30
/20
10
12
/31
/20
10
01
/03
/20
11
01
/04
/20
11
01
/05
/20
11
01
/06
/20
11
01
/07
/20
11
01
/10
/20
11
01
/11
/20
11
01
/12
/20
11
01
/13
/20
11
01
/14
/20
11
01
/17
/20
11
01
/18
/20
11
01
/19
/20
11
01
/20
/20
11
01
/21
/20
11
01
/24
/20
11
01
/25
/20
11
01
/26
/20
11
01
/27
/20
11
01
/28
/20
11
Application groups based on AuthID, PackageID, Collection ID. Select "refresh" for current data.
25 - _OTHER
24 - GS2_BIL
23 - PFN_BIL
22 - QLG_BIL
21 - PPS_BIL
20 - PPA_BIL
19 - INS_BIL
18 - PNA_BIL
17 - BPT_BIL
16 - TAB_BIL
15 - BIV_BIL
14 - BED_BIL
13 - PUB_BIL
12 - PNC_BIL
11 - LRT_BIL
10 - DEL _BIL
9 - BBH_BIL
8 - AX _BIL
7 - AD _BIL
6 - PY _BIL
5 - BMS_BIL
4 - PSD_BIL
3 - AA _BIL
2 - BBP_BIL
1 - SEL_BIL
Example 4
RESULTS:
‘MIPS for Matt’ was a success as we ‘raised’ a Peak Hours savings of 800 MIPs in a little over 2 months.
Peak MIPS for the month was barely over 2000 compared with 2900 a couple months prior
40 Copyright 2011 Progressive Insurance. May not reproduce without permission.
EXAMPLE 5:
Limit Trips to DB2
41 Copyright 2011 Progressive Insurance. May not reproduce without permission.
PROBLEM:
Batch Job CD01 had a step that routinely ran for 2+ hours, but it wasn’t doing any serious data processing (Program BL80).
The extended run time often missed the SLA we had for the reports.
Example 5
Asst. Job
Steps
Raw
Report
Data
Program
BL80
Report
Info
Policy
Info
Payment
Info
Financial
ReportAsst. Job
Steps
Asst. Job
Steps
One query was taking
90% of the run time in
the program
42 Copyright 2011 Progressive Insurance. May not reproduce without permission.
SOLUTION:
Program BL80 wasn’t using the most efficient index on the Payment Info table, but BL80 also lacked the data to get a better access path.
Upstream program BL24 also read this table, but used a full keyed read. We cached the data BL80 needed & passed it in BL24 output file.
Example 5
Asst. Job
Steps
Program
BL24
Payment
Info
Raw
Report
Data
Program
BL80
Report
Info
Policy
Info
Payment
Info
Financial
ReportAsst. Job
Steps
Asst. Job
Steps
Output
Data
Select the Postmark
date needed by BL80
downstream & write to
output file. X Since the input file now had
the Postmark date, BL80
didn’t have to query for it.
43 Copyright 2011 Progressive Insurance. May not reproduce without permission.
RESULTS:
Execution time of Job CD01 dropped by 80%. Execution of program BL80 went from 2+ hours to less than 15 minutes.
Example 5
FYI, this is kind of good…
44 Copyright 2011 Progressive Insurance. May not reproduce without permission.
EXAMPLE 6:
Review & Tune the SQL
45 Copyright 2011 Progressive Insurance. May not reproduce without permission.
PROBLEM:
New CICS program CIDE taking far too long to execute the only two SQL statements contained within it.
Example 6
IN DB2 SQL CPU
PROGRAM STMT# TIME SECS CALLS TIME SECS
--------- ----------- ------------------ ----------- ------------------
CIDE 1472 8744 61948 5981
CIDE 1418 3521 61948 2446
The program spent almost 2.5 hours daily in CPU alone!!
This caused the ‘Matt Phone’ to start ringing immediately...
(FYI, the developer responsible for this program was put into protective custody
to guarantee his safety)
46 Copyright 2011 Progressive Insurance. May not reproduce without permission.
SOLUTION:
Review the SQL for tuning opportunities…
Example 6
IN DB2 SQL CPU
PROGRAM STMT# TIME SECS CALLS TIME SECS
--------- ----------- ------------------ ----------- ------------------
CIDE 1472 8744 61948 5981
CIDE 1418 3521 61948 2446
Statement 1472 really only returned one of 3 values, and these could be determined programmatically using the host variables in the WHERE clause.
Statement 1418 could drop a ‘SUBSTR’ predicate and instead use an ‘IN’ predicate against host variables populated to emulate what the SUBSTRing was looking for. This would allow a better index access path to the data.
47 Copyright 2011 Progressive Insurance. May not reproduce without permission.
SOLUTION:
Review SQL in statement 1472 for tuning opportunities.
Example 6
SELECT CARD_BRAND
INTO :PRGGBL-CARD-BRAND
FROM PRG_BIN_GLOBAL PBG
,PRG_BIN_LOAD_CNTL PBLC
WHERE PBLC.PART_NBR = PBG.PART_NBR
AND PBLC.TABLE_IND = 'G'
AND :PRGGBL-PAN-LEN-KEY = PAN_LEN
AND SUBSTR(:BILCCINF-PYMNT-CRD-NBR,
1,BIN_LEN) >= BIN_LOW
AND SUBSTR(:BILCCINF-PYMNT-CRD-NBR,
1,BIN_LEN) <= BIN_HIGH
AND CARD_BRAND IN ('D','M','V')
Statement 1472:
Only 3 Possible Values ever returned by query based upon 1st byte of the
credit card number used.
WHERE clause not filtering well and scanning 10K index rows =
Poor Performance!
48 Copyright 2011 Progressive Insurance. May not reproduce without permission.
RESULT:
The SQL in statement 1472 was replaced with program code to evaluate the 1st byte of the card number to determine the Card Brand.
Example 6
IN DB2 SQL CPU
PROGRAM STMT# TIME SECS CALLS TIME SECS
--------- ----------- ------------------ ----------- ------------------
CIDE 1472 8744 61948 5981
CIDE 1418 3521 61948 2446
Replacing this SQL statement with program code saved 100 minutes of CPU usage per day!
Now it was time to look at the other SQL statement…
49 Copyright 2011 Progressive Insurance. May not reproduce without permission.
SOLUTION:
Review SQL in statement 1418 for tuning opportunities.
Example 6
Statement 1418:
BIN_LEN can be from 1 – 16 bytes based upon a table entry.
WHERE clause not filtering well and scanning ENTIRE index =
Poor Performance!
SELECT FDMS_NETWORK_ID
INTO :PRGPLD-FDMS-NETWORK-ID-KEY
FROM PRG_BIN_PLD PBP
,PRG_BIN_LOAD_CNTL PBLC
WHERE PBLC.PART_NBR = PBP.PART_NBR
AND PBLC.TABLE_IND = 'P'
AND SUBSTR(:BILCCINF-PYMNT-CRD-NBR,
1,BIN_LEN) = BIN_NBR
50 Copyright 2011 Progressive Insurance. May not reproduce without permission.
SOLUTION:
Revise SQL in statement 1418 to replace SUBSTR with an ‘IN’ list.
Example 6
Revised SQL:
Since BIN_NBR could be between 1 – 16 bytes of the Card Number, we used program code to easily load up all
possible variants into host variables and use ‘IN’ List.
WHERE clause now only scans 20 index rows = WAY faster!
SELECT FDMS_NETWORK_ID
INTO :PRGPLD-FDMS-NETWORK-ID-KEY
FROM PRG_BIN_PLD PBP
,PRG_BIN_LOAD_CNTL PBLC
WHERE PBLC.PART_NBR = PBP.PART_NBR
AND PBLC.TABLE_IND = 'P'
AND BIN_NBR IN (:WS-CRD-NBR-1,
:WS-CRD-NBR-2,:WS-CRD-NBR-3,
:WS-CRD-NBR-4,:WS-CRD-NBR-5,
:WS-CRD-NBR-6,:WS-CRD-NBR-7,
:WS-CRD-NBR-8,:WS-CRD-NBR-9,
:WS-CRD-NBR-10,:WS-CRD-NBR-11,
:WS-CRD-NBR-12,:WS-CRD-NBR-13,
:WS-CRD-NBR-14,:WS-CRD-NBR-15,
:WS-CRD-NBR-16)
51 Copyright 2011 Progressive Insurance. May not reproduce without permission.
FINAL RESULTS:
A drastic reduction in SQL execution time and CPU consumption by eliminating one statement and tuning the other!
Program now runs 99% faster* than before.
Example 6
IN DB2 SQL CPU
PROGRAM STMT# TIME SECS CALLS TIME SECS
--------- ----------- ------------------ ----------- ------------------
CIDE 1472 8744 61948 5981
CIDE 1456 188 65189 43
STMT # changed from 1418 to 1456 after
recompiling the program.
Total time spent in DB2 for this query reduced by over 3,300 seconds!
# calls daily roughly the
same.
Total CPU used by this query reduced by 2,400 seconds!
* Results not typical. Please consult with your fellow developers before starting a new DB2 weight loss regimen…
52 Copyright 2011 Progressive Insurance. May not reproduce without permission.
EXAMPLE 7:
Performance Quick Hits
53 Copyright 2011 Progressive Insurance. May not reproduce without permission.
Sort input data in partitioning index order prior to processing it against DB2.
Utilize multi-row fetch on cursors that are ‘narrow & deep’ (few columns selected & many rows returned). *** Multi-row fetch can be
successful in other scenarios, too, it’s just that we’ve seen the biggest bang for the buck on ‘narrow & deep’ cursors.
Try loading several table partitions in parallel for larger DB2 tables.
Adjust commit count levels in long running batch DB2 programs to avoid not committing often enough or doing it way too frequently.
Remove ‘Stage 3’ processing. We’ve found programs fetching lots of data from DB2, then filtering it within the program. Use SQL to filter this data far more efficiently and reduce overall run time.
Example 7
Here are a few Performance ‘Quick Hits’ we’ve used that didn’t take a lot of effort to net nice performance gains:
54 Copyright 2011 Progressive Insurance. May not reproduce without permission.
Lessons Learned
55 Copyright 2011 Progressive Insurance. May not reproduce without permission.
A. Have a good relationship with your DBA • DBA’s really aren’t THAT scary (and they often possess very helpful knowledge). • Discuss performance tuning opportunities as a group and keep your DBA’s in the
loop on upcoming changes - when they’re happy, you’re happy .
B. Research new features of DB2 and your application software • Keep tabs on new release information and see how the newer features can benefit
your application.
C. Keep an open mind to trying solutions that don't fit in the box • i.e. Storing data outside of DB2.
D. Performance can often be an iterative process • You may fix one thing and find more opportunities along the way. • Rewrite & tune poorly performing SQL (i.e. replacing a simple Stage 2 predicate
with a more complex, yet more efficient Stage 1 predicate).
E. Consider how to monitor performance of changes while you’re developing them vs. after they install to production.
F. Network with other DB2 groups both within and outside of your organization • Set up regular meetings/forums to discuss new features, performance
enhancements, products, etc.
G. Most importantly, be careful about suggesting future IDUG topics or you may end up being asked to present that material here!
Lessons Learned
56 Copyright 2011 Progressive Insurance. May not reproduce without permission.
? ? ? Questions ? ? ?
Keith Tidball Progressive Insurance
Session - E06
Title - Application Development Success Stories for Performance Issues on DB2 Z/OS
Lori Zaremba Progressive Insurance