SQL Server Dump Analysis

PRASHANT KUMAR

H T T P : / / S Q L AC T I ON S . COM

PRASHAN T@SQ L AC T I O N S . COM

@PRKUMA

Agenda1. Introduction

2. Windows concepts

3. Tools

4. Windbg Setup and Configuration

5. Symbols

6. Open a dump and load symbols (Demo)

7. SQL Server dump files

8. Analysing SQL Server dump files (Demo)

9. Resources

10. Q n A

Introduction

What is a dump file?A dump is an image of a process’ memory space at a given point of time written to a file for future verification.

Dump Analysis requires understanding of Windows Memory Management and Programming languages.

Managed Vs Non-managed Application.

DumpUser Mode

Kernel Mode

The Art of debugging…The process of ‘Debugging’ is not limited to

just using a debugger.

Debugging

(Identify and dissect the problem)

Knowledge of the code

and expected behaviour

Inspection of logs

Using other tools e.g. Perfmon, Netmon,

eventvieweretc.

Establish a hypothesis.

Test the hypothesis.

Windows Concepts

32-bit Address Space LayoutWindows provides a page-based virtual memory

management scheme that allows applications to realize a 32-bit linear address space for 4 GB of memory.

Each application can address 2 GB of available memory, regardless of how much physical memory actually exists.

Windows employs the PC's hard disk as the memory-backing store, and has a practical limit imposed only by the available disk space.

FFFF FFFF = 1111 1111 1111 1111 1111 1111 1111 1111 = 32bits

DWORD(32 bits/4 bytes) and QWORD(64 bits/8 bytes)

Use dd on a 32-bit dump

Use dq on a 64-bit dump

32-bit vs. 64-bit Virtual Memory

Memory Allocation Settings 32 bit versions 64-bit versions

Total amount of virtual address space 4GB

16 TB (8TB user, 8TB kernel)

Amount of virtual address space per 32-bit process

2GB (3GB if the/3G switch is added to the boot.ini file)

2GB(4GB if using /LARGEADDRESSAWARE)

Amount of virtual address space for the 64-bit processes Not applicable 8TB

Programs, Processes, and ThreadsProgram - A Static Sequence of Instructions

Process – Own resources Reserved for the Thread

Thread - Entities which Execute Instructions

Composed of: Changing set of registers Private storage area One used when running in user mode, and one used in kernel mode

Thread ID

Tools

ToolsDebugging Tools for Windows (Part of Windows SDK) WinDbg

Cdb

Kd

Adplus

DebugDiag

Visual Studio Native Debugger

ProcDump

Other Debuggers

Windbg Setup and Configuration

Choose the right installerSearch internet for “Debugging Tools for Windows” or “windbg”

For Windows 8 and 8.1

http://msdn.microsoft.com/en-US/windows/desktop/bg162891

For Windows 7

http://www.microsoft.com/en-us/download/details.aspx?id=8279

Download the right package – Both x86 and x64 versions available.

Installer screen

Symbols

What are Symbol Files?

Symbols are files.

They contain the data that map the executable code back to the source code.

Symbols hold variety of data which may not be necessary for a program's execution but debugging.

How do symbols help in debugging?

WITHOUT SYMBOLS

Call Site

sqlservr+0xd81879

sqlservr+0x31f04f0

WITH SYMBOLS

Call Site

sqlservr!HoBtFactory::DirtyLockResourceLookup+0x9d

sqlservr!GetHoBtLockInternal+0x185

sqlservr!IsRowsetBTree+0xc5

sqlservr!RowsetNewSS::Init+0x158

sqlservr!OpenRowsetSS::OpenRowset+0x105

sqlservr!OpenSystemTableRowset+0x336

sqlservr!CMEDScanBase::Rowset+0x315

IT CAN CONVERT AND TRANSLATE USEFUL INFORMATION

Public Symbol vs Private Symbol

Public Symbol Files Global variable names

Function names and the address of their entry points

FPO data

Private Symbol Files Local variable names

Source-line numbers

Type information for variables, structures, etc.

Setting the symbol pathDifferent ways to set the symbol path in WinDbg:

Set the _NT_SYMBOL_PATH environment variable to point to the root of the directory tree containing the symbols before starting the debugger.

Use the -y command line option.

Use the .sympath (Set Symbol Path) debugger meta-command.

Use the “Symbol File Path” command in the File menu

Microsoft Symbol ServerMicrosoft Public Symbol Server is http://msdl.microsoft.com/download/symbols

Centralized symbol server. Not browse able. Debugger can download symbol on need basis.

.sympath srv*DownstreamStore*http://msdl.microsoft.com/download/symbols

.symfix+ DownstreamStore

When using the public symbol store, you should always use a downstream store. Otherwise you will end up downloading the same file several times!

Open a Dump and load symbols (Demo)

Demo: Loading a dump fileOpen windbg; set symbol path; load the dump, load the symbols

Verify symbols are loaded correctly. Use !sym –noisy

Use symchk.exe

lmvm

Dialects of debugging Thread call stack

Frame

Registers, variables

Exception Context

Walking thru the stackAlways read from the bottom to top (that’s a stack you know)

Return addresses should always equal the previous stack entry’s symbolic name. In this stack -

Child-SP RetAddr Call Site

00000000`27609f10 00000000`0105ff95 sqlservr!LatchBase::UnpendEligibleWaiters+0x196

00000000`2760a060 00000000`010e01e6 sqlservr!LatchBase::ReleaseInternal+0xca

00000000`2760a0e0 00000000`010607f0 sqlservr!BPool::ReadPageCompletion+0x236

00000000`2760a3d0 00000000`0106041c sqlservr!FCB::IoCompletion+0x90

00000000`0105ff95 should equal to sqlservr!LatchBase::ReleaseInternal+0xca

00000000`2760a060 is the childEBP of sqlservr!LatchBase::ReleaseInternal+0xca

Basic Debugger commands

.sympath [to get the current symbol path]

!sym -noisy [Generate verbose output]

.load [Load a debugger extension]

.unload [Unload a debugger extension]

.reload [Reload symbols]

.reload /f [to reload symbols]

.reload /f sqlservr.exe [to reload symbols for sqlservr]

.cls [to clear debug output window]

.logopen c:\output.txt [to redirect the output into a file]

.logappend <logfile name>

.logclose c:\output.txt [to stop redirection of output to file]

SQL Server dump files

SQL Server dump filesSQL Server generates a dump file when…

Dump files location

Not only minidumps

SQLDumper.exe – What it does

Control the way dumps are generated

Be alert when a dump is generated

SQL Server generates a dump file when… Non-yielding scheduler

Non-yielding resource monitor

Non-yielding IOCP listener

Deadlocked Schedulers

Access Violation (Exception or Assertion)

Database Corruption

Latch Timeout

.NET Framework runtime exception

See http://support.microsoft.com/kb/2028589 for a list of event IDs and messages.

Dump files locationDefault location is \LOG folder

For one occurrence, a set of three files are generated:

SQLDumpNNNN.txt Symtom dump file

SQLDumpNNNN.log Snippet from ERRORLOG

SQLDumpNNNN.mdmp The memory dump file.

To change the default location:

◦ Use SQL Server Error and Usage Reporting under Configurations Tools from Programs Menu.

◦ Alternatively, edit the registry

e.g. for SQL 2012 instance

HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Microsoft SQL Server\MSSQL11.<Instance_Name>\CPE

Value = "ErrorDumpDir“

Not only minidumpsMini dump

Filtered dump

Full user-mode dump

Not only minidumps

Crash dumps : These kinds of dumps are generated in the process crash scenarios. In SQL server, whenever an Exception occurs, SQL Server generates a mini dump. Depending upon the nature of Exception, either SQL Server is terminated, or the particular session is terminated.

Hang dumps : These kinds of dumps are always taken manually (using adplus etc.). In some scenarios, e.g. SQL agent job takes ages to complete a job – we can take a hang dump. Even in high CPU scenarios also , these kinds of dumps are helpful.

Exception and AssertionException – Catch me and throw for an error

Assertion – Raise me if I don’t stand true

SQL Server handles exception and assertion in the same way by generating a minidump

The minidump contains current thread's stack into a minidump

Server * BEGIN STACK DUMP:Server * spid 123Server * ex_handle_except encountered exception C0000005 – Server terminating

0xC0000005 STATUS_ACCESS_VIOLATION Reading or writing to an inaccessible memory location.

* Exception Address = 0021AC24* Exception Code = c0000005 EXCEPTION_ACCESS_VIOLATION* Access Violation occurred writing address 67192000* Input Buffer 48 bytes -* select * from sysindexes

SQLDumper.exe – What it doesSQLDumper.exe is internally called by the SQL Server process to generate a dump file when the process encounters an exception.

SQL Server passes flags to the Sqldumper.exe utility.

You can use trace flags to change the flags that SQL Server passes to the utility in the context of an exception or in the context an assertion.

For details on using Sqldumper.exe, refer to this KB article:How to use the Sqldumper.exe utility to generate a dump file in SQL Server 2005

http://support.microsoft.com/kb/917825

Control the way dumps are generatedUsing SQLDumper.exe to manipulate the parameters. http://support.microsoft.com/kb/917825

Take manual dumps using DBCC STACKDUMP

Schedule to generate dumps on certain errors using DBCC DUMPTRIGGER

Use adplus (especially for hang dumps)

Using task manager in Windows 2008 and above

Be alert when a dump is generated

Scan ERRORLOG for dump generation messagesUsing 'dbghelp.dll' version '4.0.5'

**Dump thread - spid = 0, EC = 0x0000000000000000

***Stack Dump being sent to

X:\Data\MSSQL10.Instance_Name\MSSQL\LOG\SQLDump0008.txt

*

*******************************************************************

************

*

* BEGIN STACK DUMP:

Custom task to monitor Dump directory for recent dump files

Analysing SQL Server dump files

Demo

ResourcesDebugging Applications for Microsoft® .NET and Microsoft Windows®

Windows® Internals

Windows via C/C++

http://msdn.microsoft.com/en-us/library/cc917684.aspx

http://mssqlwiki.com

http://troubleshootingsql.com

http://sqlactions.com

Q n A