CP - 64...» Mandatory to test virtual environments with production workloads. » Technologies that move virtual machines or their storage can cause problems for the Weblogic cluster

CP - 64Costpoint with MS Sqlserver Database

Steve Sisak, Scott Wilburn Deltek

SQL Server Topics

SI 566451 - CONFIDENTIAL © Deltek, Inc. All Rights Reserved. 2

» Introduction

» Performance Settings

» Memory

» CPU

» IOPS – Input Output Operations per Second

» Virtual Environments

» Degree of Parallelism

» Troubleshooting

» W_App_Status table

» Tracing

» Database reports

» Baselines

Introduction

CONFIDENTIAL © Deltek, Inc. All Rights Reserved. 3

» Technical session on SQL Server database use and configuration.

» Presented from the perspective that these are some of the most common causes of performance issues.

» Will hopefully help you understand all the other factors that can be part of performance issues.

» Best suited to:

» Application administrators interfacing with DBA’s.

» DBA staff responsible for infrastructure related to Costpoint.

» Accountants who would like to understand more about this topic.

» Separate sessions available for Oracle and Advanced System Management (Weblogic).

Performance Settings - Memory


» Memory and Caching are key to both OS and Database performance.

» Caching is the primary way to avoid IO bottlenecks.

» Set minimum and maximum database memory to be the same value.

» Always leave 25% of the OS memory free and not committed to the database.

» Other applications running on the server might require adding memory.

» Add memory if either the database or OS show signs of memory paging to disk.

» Track key memory counters including:

» Page Life Expectancy.

» Buffer Cache Hit Rate.

Performance Settings - CPU


» Always use minimum of 2 Ghz clock speed for CPU.

» Hyperthreading can cause system overload if DOP is set based on hyperthreads.

» For systems 4 CPU cores is the bare minimum today. With modern CPU’s a 2 CPU server can have 20 physical CPU cores.

» OS issues like the Meltdown and Spectre security patches took away 20 – 30% of CPU performance.

» Size beyond the database requirements since the OS and other system applications also need CPU to operate. As with memory have 25% extra capacity.

» IO performance is the first impact if the database uses all the CPU resources.

» Monitor your peak workload time periods.

» Make use of antivirus exclusions or switch from real time to scheduled scans.

» Never use Power Saving settings in the system BIOS, a Virtual Machine guest OS, or a Host OS.

» Deltek minimum CPU recommendations are based on physical servers and virtual CPU’s can be much different than physical ones.

IOPS – Input/Output Operations per Second


» Standard measurement of storage system performance

» Faster / “More” IOPS = Faster Costpoint or more concurrent users..

» Direct 1 to 1 correlation between IOPS increases and performance of key Costpoint closing processes in Deltek’s performance lab.

» Impacts of Memory and CPU tuning on a database are minimal until IO is adequate.

» Deltek lab database test servers have current IOPS ratings of :

» 17,000 – decommissioned in 2018 but was the standard for 7 years.

» 30,000 – 40,000 internal low performance SSD array.

» 70,000 – internal high performance SSD array.

» 106,000 – Nimble AF60 all flash storage array, dual 10Gb active Ethernet storage network.

» Deltek uses Diskspd utility to measure IOPS. (Microsoft Technet). Available through Customer Care.

» Deltek also has a database driven load test that is not Windows specific that can be used to test IO.

IOPS and Hosting


» IOPS are the most important factor for a database system like Costpoint.

» IOPS are critical for performance but often overlooked selecting instance types.

» Hosted IOPS are complex due to options to dedicate or “burst” and different limits that are tied to IOPS like throughput.

» IOPS is the first sizing parameter to consider. Throughput needs to match / sustain IOPS for the duration of a long running Costpoint process.

» Sustainable IOPS are often lower than Peak IOPS be aware of what you are buying and your processing needs.

» Example:

» C5n.2xlarge AWS instance – 20,000 IOPS (30 min limit). 3,500 Mbps throughput (30 min limit).

» Sustainable IOPS = 10,000. Sustainable throughput = 1,750 Mbps.

» Need to understand that a “Fast” Costpoint system can be a 900 lb gorilla of IO.

Virtual Environments


» Best practices with Virtualization.

» Monitor and Manage to meet the database peak workload requirements.

» There are more variables which make sizing and management more

complex.

» Reserve the resources in the VM required for the database server.

» Don’t recommend sharing or pooling CPU or Memory.

» Mandatory to test virtual environments with production workloads.

» Technologies that move virtual machines or their storage can cause

problems for the Weblogic cluster and database. The short system

disconnects or suspension they cause have taken systems offline.

Degree of Parallelism


» A calculation by SQL Server, based on detected hardware or configuration, to speed up SQL using multiple threads.

» SQL Server will apply parallelism based on the database settings and optimizer determination.

» Controlled through two primary database settings:

» Max Degree of Parallelism - Tells SQL Server how many threads to use explicitly or can let SQL Server decide.

» Cost Threshold for Parallelism - Tells SQL Server how “expensive” a query must be before it is parallelized or lets SQL Server parallelize every query with a cost greater than 5.

» Setting these correctly is very dependent on your system and requires testing:

» Hardware and IOPS capability.

» Workload at peak periods.

» Never leave DOP at default. Test values starting at 2 and monitor. Common values are 4 and 8.

» Warning signs of too high a MaxDOP setting:

» High CXPacket wait events.

» High IO latency.

» High CPU utilization and / or high context switching.

» System slowness.

Technical Troubleshooting


» Information about the issue.

» Complete description.

» Is it repeatable and how to repeat.

» Has anything changed.

» A system diagram can summarize the architecture and improve troubleshooting.

» Application Trace.

» W_App_Status report for the impacted process with give the performance history of the job.

» SQL Server or third party performance management tool data may be useful.

» Weblogic diagnostic image.

» Check for errors that happen at the time of the incident

W_App_Status_Info table


» Automatically stores run time information for all reports and processes run in Costpoint.

» The data can be accessed in any screen with an Actions / Report button on the menu.

» A Costpoint report interface allows you to quickly run reports for any Costpoint job like Calculate Revenue. Reports can also be run by time period and in that case you get a list of everything in Costpoint that ran at a particular time.

» Great way to track run time changes in key jobs.

» Answers the question “what else was running” also records other key data like application server used and how the job was submitted.

» Should be used to double check time periods for other analysis reports from OS data or Database data.

» Supply both baselines and trends for Costpoint processes that can help evaluate tuning or hardware impacts.

W_App_Status_Info data


» APP_ID, ACTION_ID and RPT_ID – Identify the work.

» COMPLETE_FL – Completion status.

» EXEC_METHOD_CD – How job was submitted ( UI, API, Async, PrcSrv).

» USER_ID – Submitter.

» SYS_TEXT – Job messages usually completion or error.

» JOB_START_DT, JOB_END_DT or ACTION_START_DT, ACTION_END_DT –Start and stop times and dates to allow calculation and tracking of elapsed time.

» COMPANY_ID – Company run for.

» SERVER_ID – Application server doing the work.

Weblogic Trace Options


» traceSlowStatements

» Maintenance screen only trace for special situations.

» enableSQLTrace

» Weblogic based application text trace. Developed for SQL Server. No

DBA involvement but only elapsed time from each statement. No

database impact.

» For full details see KB’s 96692, 85245

» This traces 1 application per entry in the enterprise.properties file.

» In special situations for maintenance screens use traceSlowStatements

enableSQLTrace Output


» Comparison of the same statement in two time periods.

» 71. Statement:

» MERGE INTO D_585PJPCOMPR_POP POP USING ( SELECT POP2.PROJ_ID AS PROJ_ID, ? AS nMaxLvlNo,

SUM(isnull(POP1.PROJ_V_FEE_AMT,'NUM') + isnull(POP1.PROJ_V_CST_AMT,'NUM')) AS PROJ_V_TOT_AMT, SUM(isnull(POP1.PROJ_F_FEE_AMT,'NUM') /*

C:1 R:All */

» First Executed on: Sat Jun 01 07:06:35 CDT 2019

» Parsed: 8

» Executed : 8

» Has literals: YES

» Total Exec Time: 12485139 milliseconds (200 minutes approximately)

» Invoked by: application.pj.pjpcompr.Pjpcompr.funcGetContractAndFundedValues(Pjpcompr.java:11187)

» 71. Statement:

» MERGE INTO D_585PJPCOMPR_POP POP USING ( SELECT POP2.PROJ_ID AS PROJ_ID, ? AS nMaxLvlNo,

SUM(isnull(POP1.PROJ_V_FEE_AMT,'NUM') + isnull(POP1.PROJ_V_CST_AMT,'NUM')) AS PROJ_V_TOT_AMT, SUM(isnull(POP1.PROJ_F_FEE_AMT,'NUM') /*

C:1 R:All */

» First Executed on: Tue Jun 04 07:06:48 CDT 2019

» Parsed: 8

» Executed : 8

» Has literals: YES

» Total Exec Time: 226934 milliseconds (4 minutes approximately)

» Invoked by: application.pj.pjpcompr.Pjpcompr.funcGetContractAndFundedValues(Pjpcompr.java:11187)

Database Reports


» There is no uniform performance reporting tool for SQL Server.

» Third party tools have filled this gap historically.

» With SQL Server 2016 the new Query Store is the first repository of database performance and management data.

» It still needs to activated.

» Reports are minimal.

» Some data is not yet collected.

» Potential exists for performance issues.

» The following reports were generated with the Query Store and some supplemental scripts by Deltek Operations for internal system management. This tool is not currently a part of any product. We are hopeful that Microsoft will develop the Query Store to include this type of functionality.

» Our goal is to include the most often used data for SQL Server troubleshooting.

Overview data


» Important overall server data.

Top Wait Events


» Wait events are normal and consistent. This section helps spot unusual or abnormal delays in the system.

» New top wait events should always be investigated especially after upgrades or if performance issues are occurring.

Queries by Duration


» Key for identifying problem SQL and making sure there are not other SQL interferences. SQL can also be sorted by Reads, Writes, CPU and Memory Grant.

» Costpoint applications being reviewed should be the longest running SQL.

Diagnostic Messages and Error Log


» Diagnostic view of main wait states and errors during the report period.

IO Performance Counters from SQL Server


» IO performance with “Stall” of less than 10ms is crucial for good performance.

» Can be supplemented by performance monitor OS counters.

SQL Server Status messages


» A variety of status messages from all databases on the server.

SQL Server initialization parameters


» Confirm the configuration has not changed from a known good baseline

and identify any new or modified parameters.

Graphs of Data from SQL and Performance Monitor


» We are testing combining OS statistics with SQL data in our analysis.

This is a feature available in third party tools currently.


» In coordination with accounting and using the W_App_Status report, run the available performance reports covering month end processing for each period of the year.

» Save January and December data as baselines. (Minimum).

» Use them to review performance during the year.

» Evaluate intermittent performance issues.

» Do capacity planning.

» Evaluate performance tests and configuration or hardware changes.

» For long running processes add a trace of the job. Having a known picture of good performance is extremely valuable in troubleshooting.

Performance Baselines


Q & A

Documents

CP - 64...» Mandatory to test virtual environments with production workloads. » Technologies that move virtual machines or their storage can cause problems for the Weblogic cluster