SQL Log File: Spelunking

SQL Log File:Spelunking

(for anyone, really, that wants to recover lost data or find out whodunnit)Welcome to my lo-fipresentation (boycott slideshows!)I am Scott Ellis, kCuras Infrastructure Engineering ArchitectThe Scenario!

You (or your client) got HACKED

Symptoms and VerificationImportant tables are missingClients are missingThere is a giant smiley face painted on your application home pageThere are no database errorsDBCC checkDB checks OKYou verify certain tables are, in fact, missing. You restore from backup and now you want to roll forward up to the moment of disaster.

What now?Log file analysis

Wait!

Whats a log file?(SQL calls them "ldf"s)It's a rewind file for the database. a record of everything, circular log, blah blah blah They are often poorly maintained and retain massive amounts of data.Unchecked, they can get HUGE (terabytes in size)Types of database loggingACID compliant databases have logs. MS SQL BULKSIMPLEFULLEmergency

When should you look for Log files?

Any database that is ACID compliant will use them, though the structure won't be the same.AtomicConsistentIsolatedDurable

Where are they?Large, improperly maintained log files are not only common, they are prevalent. They are anywhere that an ACID database is. (on your phones!)On a live database, you can easily see where the logs are kept by viewing its propertiesSo now you want to know:

How do we go about finding out what's in them?

You can also search through log backups (If you have them - the absence of log backups is not suspicious if the transaction log is quite large and the maintenance plan for log backups has been failing.)

For live database log files, uncommitted transactions:Use the undocumented fn_dblog function.

for committed that are still sitting in the log file:DBCC TraceOn (2536) thenfn_dblog

From log backup filesfn_dump_dblog

BEWARE: as of at least 8/13/2013: every time fn_dump_dblog is called, it creates a new hidden SQLOS scheduler and up to three threads, which will never go away and never be reused. It's a bug.

Here's the thing about undocumented features: DON'T USE THEM ON YOUR PRODUCTION SYSTEM!

DISCLAIMER(If you have Relativity, and you break your production system by executing anything I show you tonight, I will deny ever having been here.)

DEMO Time!--First, let's make some backups...ALTER DATABASE readytest SET RECOVERY FULLBACKUP DATABASE [readyTest] TO DISK = N'J:\ReadyTestDemo_Full.bak' WITH INIT;GOBACKUP LOG [readyTest] TO DISK = N'J:\ReadyTestDemoLog_Log1.bak' WITH INIT;GO--Now do the bad thing:DROP TABLE eddsdbo.[4n6ArtifactWords]--Now pretend to rebuild the tables (next three tabs) (I did this ealier because it takes a long time, this is the Bloomberg data)--now, when was the table dropped, and who did it?--First, let's make some backups...ALTER DATABASE readytest SET RECOVERY FULLBACKUP DATABASE [readyTest] TO DISK = N'J:\ReadyTestDemo_Full.bak' WITH INIT;GOBACKUP LOG [readyTest] TO DISK = N'J:\ReadyTestDemoLog_Log1.bak' WITH INIT;GO--Now do the bad thing:DROP TABLE eddsdbo.[4n6ArtifactWords]

demo continued--Now pretend to rebuild the tables (next three tabs) (I did this earlier because it takes a long time, this is the Bloomberg data)--now, when was the table dropped, and who did it?SELECT Top 10suser_sname([Transaction SID]) as 'Login',[Transaction Name],[Begin Time], [Current LSN], [Operation], [Context], [Transaction ID], [Description]FROM fn_dblog (NULL, NULL), (SELECT [Transaction ID] AS [tid] FROM fn_dblog (NULL, NULL) WHERE [Transaction Name] LIKE '%DROPOBJ%') [fd]WHERE [Transaction ID] = [fd].[tid];GO--If you happen to bee googling around and run across this SQL: SELECT suser_sname(convert(varbinary,"Transaction SID")) as 'Login' --it is wrong. Don't use it. --

Performance TipDisk, and CPU will get slammed by this procedure.

Google tip: just the word Fn_dblog will turn up a lot of info. I suggest that you set aside a day and plan stepping through what you find, and create your own documentation about how it works. For examole, Dimitri's info, while useful, doesn't come along with him saying that he actually tested doing this. If you want to recover the log file from a backup of the database that included the tail of the log, then you should try it and make sure this works.

Useful other tools (if you don't want to carve bytes) Light speed, Toad, Apex

Tips: use a machine that has a lot of disk IO and CPU

Some Questions: Can each of these tools work with just a log file, or do they need a functional database behind them? What if I carve out what I believe is a log file, will it work? Where will you find SQL Log files?

More information can be found on SQLskills.com http://www.sqlskills.com/blogs/paul/using-fn_dblog-fn_dump_dblog-and-restoring-with-stopbeforemark-to-an-lsn/

From dimitri furman (msn blog)http://blogs.msdn.com/b/dfurman/archive/2009/11/05/reading-database-transaction-log-with-fn-dump-dblog.aspxUpdate 2012-02-08: I just found out that the function can also work against database backups, not just log backups. A database backup contains a portion of the log that will be rolled forward on restore to make the restored database transactionally consistent, and that portion of the log can be viewed with the fn_dump_dblog() function. This is potentially useful to discover the LSN and the timestamp of the checkpoint that occurs during the backup - look for LOP_END_CKPT in the Operation column.

FINAL Question:

Shutting down a database server: shut down normally or pull the plug? If a database really is ACID, should it matter if I kill the power to it?Also, if you shut it down normally, it's going to finish executing whatever transactions it's doing!

What if it is a cleanup where the log gets truncated and then wiped (because the dude you are after was able to kick of his "Destroy everything" maintenance task before he was tackled.)

Wellllll......

I asked an expert "Is it true that unplugging a database server can corrupt it?" and here is what he said:

"In theory, if it was perfectly configured, it won't, but that's rarely true. People leave write caching on at the RAID controller, for example. When you pull the plug, the data is (hopefully) written to non-volatile RAM, and then when power comes back, it finishes writing the data out to the relevant drives. Perfect example of problems though - we had a datacenter-wide AC outage, and the sysadmins yanked cables to shut down servers faster. When we went to power back up, some of them suffered RAID array failures - when you stop drives that have been running for years, sometimes they don't come back up. So we pulled the drives, put them into another server, and I had corrupt databases because SQL Server's writes hadn't all been done to the transaction log. Some arrays had them, some didn't."~well-known SQL expert Brent Ozar

Evidence Tracking ApplicationClient Information

Evidence Tracking ApplicationProject Information

Evidence Tracking ApplicationCompany Information

Evidence Tracking ApplicationCustodian Information

Evidence Tracking ApplicationEvidence Information

Evidence Tracking ApplicationForensic Image Information

Evidence Tracking ApplicationExaminer Information