42
Automated Real-time and Post Mortem Security Crash Analysis and Categorization Jason Shirk, PM Dave Weinstein, Sr. SDE [email protected] [email protected] Security Science

Jason Shirk, PMDave Weinstein, Sr. SDE [email protected]@microsoft.com [email protected]@microsoft.com Security

Embed Size (px)

Citation preview

Page 1: Jason Shirk, PMDave Weinstein, Sr. SDE jason.shirk@microsoft.comdave.weinstein@microsoft.com jason.shirk@microsoft.comdave.weinstein@microsoft.com Security

Automated Real-time and Post Mortem Security Crash Analysis and Categorization

Jason Shirk, PM Dave Weinstein, Sr. [email protected] [email protected]

Security ScienceMicrosoft Security Engineering Center (MSEC)

Page 2: Jason Shirk, PMDave Weinstein, Sr. SDE jason.shirk@microsoft.comdave.weinstein@microsoft.com jason.shirk@microsoft.comdave.weinstein@microsoft.com Security

Trustworthy Computing

Scaling a Difficult Problem

Problems exist with identifying unique crashes

The same issue can arise multiple timesThe same issue can arise through multiple code pathsThe same issue can be found across multiple machines

Classifying the crashes is another issue entirely

Manual inspection of crash dumps does not scaleIdentifying security issues takes experienced resourcesTakes a lot of time to manually analyze the crash

Testing produces more crashes than there are resources to triage

Automation can help trim down the triagingGrouping crashes by location in code helps

Page 3: Jason Shirk, PMDave Weinstein, Sr. SDE jason.shirk@microsoft.comdave.weinstein@microsoft.com jason.shirk@microsoft.comdave.weinstein@microsoft.com Security

Trustworthy Computing

Fuzzer A9 Issues8 Unique

Fuzzer B23

Issues21

Unique

Constraint Solver24 Issues23 Unique

Fuzzer C7

Issues5

Unique

An Experiment: 1 parser, 2 weeks, 4 fuzzers = 57 distinct crashes

Page 4: Jason Shirk, PMDave Weinstein, Sr. SDE jason.shirk@microsoft.comdave.weinstein@microsoft.com jason.shirk@microsoft.comdave.weinstein@microsoft.com Security

Trustworthy Computing

Fuzzer A7 Issues3 Unique

Fuzzer B10

Issues4

Unique

Constraint Solver

3 Issues1 Unique

The same results, with similar crashes grouped together = 9 unique issues

Page 5: Jason Shirk, PMDave Weinstein, Sr. SDE jason.shirk@microsoft.comdave.weinstein@microsoft.com jason.shirk@microsoft.comdave.weinstein@microsoft.com Security

Trustworthy Computing

Scoping the Problem

Windows Vista® File Fuzzing EffortFuzzed from September ‘05 to November ’06350M iterations total250+ file parsers fuzzed300+ issues fixedLots of people fuzzingOverall, a huge effort

Page 6: Jason Shirk, PMDave Weinstein, Sr. SDE jason.shirk@microsoft.comdave.weinstein@microsoft.com jason.shirk@microsoft.comdave.weinstein@microsoft.com Security

Trustworthy Computing

Realizing the Problem

Vista is just one release of one productMicrosoft is a much larger organization

10,000’s of developers10,000,000’s of lines of code1,000,000,000 + users

What percentage of the security community would we need to just parse the logs?

Everyone in MSEC?Everyone in this room?Everyone qualified in the world?More?

Page 7: Jason Shirk, PMDave Weinstein, Sr. SDE jason.shirk@microsoft.comdave.weinstein@microsoft.com jason.shirk@microsoft.comdave.weinstein@microsoft.com Security

Trustworthy Computing

First Steps

Grouping Similar CrashesNecessary to find duplicate issuesUncomplicated solution implemented by hashing the stack

Simple Categorization by Exception Type

Information is provided by the OSSolutions were custom code in individual tools

Code sharing by “cut and paste”Difficult to determine which tools used which rules

Page 8: Jason Shirk, PMDave Weinstein, Sr. SDE jason.shirk@microsoft.comdave.weinstein@microsoft.com jason.shirk@microsoft.comdave.weinstein@microsoft.com Security

Trustworthy Computing

Moving Forward

Hashing code for crashes existed in multiple tools and in multiple librariesCrash categorization code based on the MSDN article “Analyze Crashes to Find Security Vulnerabilities in Your Apps” (Nov. 2007) existed in multiple tools and in multiple librariesRule of Thumb: “When the same code is in more than one place, it is wrong in at least one of them”It was time to put the classification and analysis logic in one place…

Page 9: Jason Shirk, PMDave Weinstein, Sr. SDE jason.shirk@microsoft.comdave.weinstein@microsoft.com jason.shirk@microsoft.comdave.weinstein@microsoft.com Security

Trustworthy Computing

Requirements for a Solution

Performance in Live DebuggingSecurity tools have to evaluate every first chance exception for security implications

Minimize False PositivesTools must scale, noise blocks that

Avoid False Negatives Categorizing an exploitable crash as not being exploitable is a major fault

Maintainability and ExpandabilityThe state of the art in exploitation changes, tools must keep up

Page 10: Jason Shirk, PMDave Weinstein, Sr. SDE jason.shirk@microsoft.comdave.weinstein@microsoft.com jason.shirk@microsoft.comdave.weinstein@microsoft.com Security

Trustworthy Computing

First Prototype (Early 2008)

Windows Debugger ExtensionDesigned to produce output that would be human readable and also parsed by tools to determine the hash and the severityDetection logic was hand-coded CProof of concept was successful

Page 11: Jason Shirk, PMDave Weinstein, Sr. SDE jason.shirk@microsoft.comdave.weinstein@microsoft.com jason.shirk@microsoft.comdave.weinstein@microsoft.com Security

Trustworthy Computing

Architecture for a Solution

Rules Driven EngineMaintainabilityEase of understanding the rules

Processor independenceThe prototype already had too many special cases for x86 versus x64 cluttering the code flow

Human and Machine readable outputOther tools will be layered on topManual use is still a supported scenario

Implemented as a Windows Debugger extension

Page 12: Jason Shirk, PMDave Weinstein, Sr. SDE jason.shirk@microsoft.comdave.weinstein@microsoft.com jason.shirk@microsoft.comdave.weinstein@microsoft.com Security

Trustworthy Computing

!exploitable Crash Analyzer

Demo

Page 13: Jason Shirk, PMDave Weinstein, Sr. SDE jason.shirk@microsoft.comdave.weinstein@microsoft.com jason.shirk@microsoft.comdave.weinstein@microsoft.com Security

Trustworthy Computing

Rules Engine

Page 14: Jason Shirk, PMDave Weinstein, Sr. SDE jason.shirk@microsoft.comdave.weinstein@microsoft.com jason.shirk@microsoft.comdave.weinstein@microsoft.com Security

Trustworthy Computing

Types of Rules

Gather Rules Query the debugger for informationPerform complex computationsAll results are cached in the state

Report RulesProduce both human and machine readable reports based on the state

Analysis RulesUse the cached state to determine if a case has been met, and determine the resultsCan perform programmatic queries of the cached state

Page 15: Jason Shirk, PMDave Weinstein, Sr. SDE jason.shirk@microsoft.comdave.weinstein@microsoft.com jason.shirk@microsoft.comdave.weinstein@microsoft.com Security

Trustworthy Computing

Rules Engine Flow

Report Results

Gather?

Report?

Get Next Rule

Update State

Report

Matches?

More Rules?

Final?

Analyze

Page 16: Jason Shirk, PMDave Weinstein, Sr. SDE jason.shirk@microsoft.comdave.weinstein@microsoft.com jason.shirk@microsoft.comdave.weinstein@microsoft.com Security

What does a rule look like?

// Rule: Read AVS in a block move are considered probably exploitableBEGIN_ANALYZE_DATA

PROCESSOR_MODE USEREXCEPTION_ADDRESS_RANGE DONT_CAREEXCEPTION_TYPE STATUS_ACCESS_VIOLATIONEXCEPTION_SUBTYPE ACCESS_VIOLATION_TYPE_READEXCEPTION_LEVEL DONT_CAREANALYZE_FUNCTION IsFaultingInstructionBlockDataMoveRESULT_CLASSIFICATION PROBABLY_EXPLOITABLERESULT_DESCRIPTION L"Read Access Violation on Block Data Move"RESULT_SHORTDESCRIPTION L"ReadAVonBlockMove"RESULT_EXPLANATION L"This […] classified as probably exploitable."RESULT_URL NULLRESULT_IS_FINAL true

END_ANALYZE_DATA

Page 17: Jason Shirk, PMDave Weinstein, Sr. SDE jason.shirk@microsoft.comdave.weinstein@microsoft.com jason.shirk@microsoft.comdave.weinstein@microsoft.com Security

Not quite as impressive as it first appears…

#define BEGIN_ANALYZE_DATA ,{ ANALYZE_DATA, NULL, NULL #define PROCESSOR_MODE ,#define EXCEPTION_ADDRESS_RANGE ,#define EXCEPTION_TYPE ,#define EXCEPTION_SUBTYPE ,#define EXCEPTION_LEVEL ,#define ANALYZE_FUNCTION ,#define RESULT_CLASSIFICATION ,#define RESULT_DESCRIPTION ,#define RESULT_SHORTDESCRIPTION ,#define RESULT_EXPLANATION ,#define RESULT_URL ,#define RESULT_IS_FINAL ,#define END_ANALYZE_DATA }

Page 18: Jason Shirk, PMDave Weinstein, Sr. SDE jason.shirk@microsoft.comdave.weinstein@microsoft.com jason.shirk@microsoft.comdave.weinstein@microsoft.com Security

Trustworthy Computing

Rule Conventions

Final Analysis rules which can be determined without further gather steps should happen before more information is gathered

Performance, performance, performance

All heavyweight processing is done in gather rules

To help enforce this, Analysis functions don’t have access to the underlying Debugger objects

Data which is not affected by multiple gather rules should be reported once it is gathered

Data which is gathered by multiple rules must be reported when results are reported

Page 19: Jason Shirk, PMDave Weinstein, Sr. SDE jason.shirk@microsoft.comdave.weinstein@microsoft.com jason.shirk@microsoft.comdave.weinstein@microsoft.com Security

Trustworthy Computing

Expanding the Rules Set

Basic Rules ChangesAdding new analysis rules that use the built-in fields

Complex Rules ChangesAdding new analysis rules that require a custom analysis functionAdding new analysis rules that require additional state and matching gather rules

Advanced Rules ChangesAdding or modifying the built in fields or field constants

Page 20: Jason Shirk, PMDave Weinstein, Sr. SDE jason.shirk@microsoft.comdave.weinstein@microsoft.com jason.shirk@microsoft.comdave.weinstein@microsoft.com Security

Trustworthy Computing

Code Analysis

Page 21: Jason Shirk, PMDave Weinstein, Sr. SDE jason.shirk@microsoft.comdave.weinstein@microsoft.com jason.shirk@microsoft.comdave.weinstein@microsoft.com Security

Trustworthy Computing

Meta-Disassembler

Design Goal:Convert actual assembly into a pseudo-assembly language that includes only the information needed for threat analysis

Implementation Goal:Minimize external dependencies by using the disassembler built into the Windows Debugger

Page 22: Jason Shirk, PMDave Weinstein, Sr. SDE jason.shirk@microsoft.comdave.weinstein@microsoft.com jason.shirk@microsoft.comdave.weinstein@microsoft.com Security

Trustworthy Computing

Meta-Instructions

DATA_MOVEBLOCK_DATA_MOVEBRANCHRETURNINTERRUPT

DATA_EXCHANGECALCULATIONSTACK_PUSHSTACK_POP

Page 23: Jason Shirk, PMDave Weinstein, Sr. SDE jason.shirk@microsoft.comdave.weinstein@microsoft.com jason.shirk@microsoft.comdave.weinstein@microsoft.com Security

Trustworthy Computing

Meta-Instruction Operands

Implicit and Explicit OperandsImplicit come from the instruction definitionExplicit are found in the instruction instance

Types of Operands in an InstructionSource OperandsDestination OperandsDestination Pointer Operands

Source operands that point to where the destination is in memory

Branch or Return OperandsOnly implicitUsed to handle conventions for passing operands to and from functions

Page 24: Jason Shirk, PMDave Weinstein, Sr. SDE jason.shirk@microsoft.comdave.weinstein@microsoft.com jason.shirk@microsoft.comdave.weinstein@microsoft.com Security

Trustworthy Computing

Meta-Instruction Complications

Not all “Destination” operands are actually modified

CMP EAX, EBXOften “Destination” operands are also implicit source operands

Common in calculations and comparisonsIdiomatic use of assembly instructions for side affects can affect source/destination tracking

XOR EAX, EAX

Page 25: Jason Shirk, PMDave Weinstein, Sr. SDE jason.shirk@microsoft.comdave.weinstein@microsoft.com jason.shirk@microsoft.comdave.weinstein@microsoft.com Security

Trustworthy Computing

Registers

Registers are really just strings at the Analysis stage

The meta-disassemblers specific to a processor are responsible for defining the register set

For x86/x64 processors, we add special pseudo-registers

Calculations off of EBP/RBP or ESP/RSP are turned into registersThis lets us treat stack variables as independent registers

Page 26: Jason Shirk, PMDave Weinstein, Sr. SDE jason.shirk@microsoft.comdave.weinstein@microsoft.com jason.shirk@microsoft.comdave.weinstein@microsoft.com Security

Trustworthy Computing

Register Complications

On the x86 and x64 architectures, registers are sub and super-sets of each other

i.e. EAX encapsulates AX, AH, and ALOn most if not all processors, individual flags are subsets of a flags register

Page 27: Jason Shirk, PMDave Weinstein, Sr. SDE jason.shirk@microsoft.comdave.weinstein@microsoft.com jason.shirk@microsoft.comdave.weinstein@microsoft.com Security

What does an Instruction Definition look like?// CallBEGIN_INSTRUCTION

EXACT_MNEMONIC( L"call" )INSTRUCTION_CLASSIFICATION BRANCHOPERAND_ENCODING SOURCE_OPERANDS_ONLYNO_ANALYSIS_FUNCTIONNO_IMPLICIT_SOURCE_REGISTERSNO_IMPLICIT_DESTINATION_REGISTERSNO_IMPLICIT_DESTINATION_POINTER_REGISTERSIMPLICIT_PASSED_OR_RETURNED_REGISTERS OPERAND_ENTRY( STACK_CONTENTS ),

OPERAND_ENTRY( L”ecx” ), OPERAND_ENTRY( L”edx” ), END_OPERAND_LIST

END_INSTRUCTION

Page 28: Jason Shirk, PMDave Weinstein, Sr. SDE jason.shirk@microsoft.comdave.weinstein@microsoft.com jason.shirk@microsoft.comdave.weinstein@microsoft.com Security

Trustworthy Computing

What is being left out?

All pure constantsAll non xBP/xSP based constant relative calculations

[EAX *4] is only tracked as EAX[EAX + ECX] tracks as both EAX and ECX

Not all instructions are modeledAnything that isn’t in our instruction definition set is ignored, resulting in loss of tracking information

Page 29: Jason Shirk, PMDave Weinstein, Sr. SDE jason.shirk@microsoft.comdave.weinstein@microsoft.com jason.shirk@microsoft.comdave.weinstein@microsoft.com Security

Simple Analysis Functions based on Meta-Instructions

/// Determines if the faulting instruction is a block data moveboolIsFaultingInstructionBlockDataMove( const DEBUGGER_STATE &objState ){ INSTRUCTION_CLASS eFaultingClass =

objState.objSourceCode.front()->eClass;

return( eFaultingClass == BLOCK_DATA_MOVE );}

Page 30: Jason Shirk, PMDave Weinstein, Sr. SDE jason.shirk@microsoft.comdave.weinstein@microsoft.com jason.shirk@microsoft.comdave.weinstein@microsoft.com Security

Trustworthy Computing

Extending the Analysis

Analyzing the faulting instruction is not sufficient

There are a set of problems that can only be found by following the flow of tainted data through the basic block

The Meta-Instruction definition shown previously has evolved as the code analysis algorithms and rules were created during development

Page 31: Jason Shirk, PMDave Weinstein, Sr. SDE jason.shirk@microsoft.comdave.weinstein@microsoft.com jason.shirk@microsoft.comdave.weinstein@microsoft.com Security

Trustworthy Computing

Taint Tracking in the Basic Block

Issues we have to handleTransitions on and off of the stackDifferentiating between types of operands

Handled explicitly in the Instruction definition

Tracking taint setting and clearing across the overlapping registers of the x86 and x64 instruction setsTracking the difference between implicit and explicit operands

Handled explicitly in the Instruction definition

Page 32: Jason Shirk, PMDave Weinstein, Sr. SDE jason.shirk@microsoft.comdave.weinstein@microsoft.com jason.shirk@microsoft.comdave.weinstein@microsoft.com Security

Trustworthy Computing

Dealing with the Stack

Lightweight virtual stackTracks tainted data going onto and off of the stackHas some potential processor specific assumptions lurking in the code

Stack operations are either limited to one explicit register OR are driven by the implicit registersImplicit registers are popped from or pushed onto the stack in the order they are listed in the Instruction definition

Page 33: Jason Shirk, PMDave Weinstein, Sr. SDE jason.shirk@microsoft.comdave.weinstein@microsoft.com jason.shirk@microsoft.comdave.weinstein@microsoft.com Security

Trustworthy Computing

Dealing with Overlapping Registers

For each instruction set, register relationships are defined

“Setting” relationships taint everything higher

AH -> AX -> EAX -> RAX

“Clearing” relationships clear everything lower

AX->AH->AL

Page 34: Jason Shirk, PMDave Weinstein, Sr. SDE jason.shirk@microsoft.comdave.weinstein@microsoft.com jason.shirk@microsoft.comdave.weinstein@microsoft.com Security

Trustworthy Computing

General Taint Tracking Algorithm

Maintain two sets of registersThose known to be taintedThose known to be cleared

From this we can determine which registers are tainted at any given point by

Expanding the tainted set using the “set” mappingRemoving from that expanded set any register that is explicitly known to be cleared

At the end of each instruction, taint propagates to these two sets

Page 35: Jason Shirk, PMDave Weinstein, Sr. SDE jason.shirk@microsoft.comdave.weinstein@microsoft.com jason.shirk@microsoft.comdave.weinstein@microsoft.com Security

Rules using complex taint analysis

/// Determines if the branch target is based off of the tainted data////// Important: This makes an assumption that branch selection is based off of/// implicit source registers, and that explicit source registers determine/// the branch target. boolIsTaintedDataUsedToDetermineBranchTarget( const DEBUGGER_STATE &objState ){ INSTRUCTION *pInstruction = objState.objSourceCode.back();

if( pInstruction->eClass == BRANCH ) {

for( OPERAND_SET::const_iterator itOperand = pInstruction->setSourceRegisters.begin(); itOperand != pInstruction->setSourceRegisters.end(); itOperand++ ){ if( (pInstruction->setTaintedInputRegisters.find( *itOperand ) !=

pInstruction->setTaintedInputRegisters.end()) && (pInstruction->setExplicitRegisters.find( *itOperand ) !=

pInstruction->setExplicitRegisters.end() ) ) { return( true ); }}

}

return( false );}

Page 36: Jason Shirk, PMDave Weinstein, Sr. SDE jason.shirk@microsoft.comdave.weinstein@microsoft.com jason.shirk@microsoft.comdave.weinstein@microsoft.com Security

Trustworthy Computing

Wrapping Things Up

Page 37: Jason Shirk, PMDave Weinstein, Sr. SDE jason.shirk@microsoft.comdave.weinstein@microsoft.com jason.shirk@microsoft.comdave.weinstein@microsoft.com Security

Trustworthy Computing

Assumptions and Limitations

Fundamental AssumptionAll input operands in the faulting instruction are under the control of the attacker

LimitationsThe lightweight flow analysis and the above assumption can result in over-assessing riskRegisters are not aliased by value

Even if EAX and ECX were set to the same value before the faulting instruction, we will not consider them equivalently taintedChanges in registers don’t affect special cases

EBP-10 is a pseudo-register, unaffected by changes to EBP itself

Page 38: Jason Shirk, PMDave Weinstein, Sr. SDE jason.shirk@microsoft.comdave.weinstein@microsoft.com jason.shirk@microsoft.comdave.weinstein@microsoft.com Security

Trustworthy Computing

Use Inside of Microsoft

Used by Security Tools for crash categorization and bucketizationUsed by Teams and Developers to evaluate the implications of crashes on products in development

Page 39: Jason Shirk, PMDave Weinstein, Sr. SDE jason.shirk@microsoft.comdave.weinstein@microsoft.com jason.shirk@microsoft.comdave.weinstein@microsoft.com Security

Trustworthy Computing

Questions

Technical Questions for DaveNon-Technical Questions for Jason

These will be answered after the technical questions

Page 40: Jason Shirk, PMDave Weinstein, Sr. SDE jason.shirk@microsoft.comdave.weinstein@microsoft.com jason.shirk@microsoft.comdave.weinstein@microsoft.com Security

Trustworthy Computing

Thanks!

Michael Eddington (Leviathan)

Microsoft - Adel Abouchaev, Dustin Duran, Tom Gallagher, Damian Hasse, Rob Hensing, Shawn Hernan, Vassilii Khachaturov, Scott Lambert, Michael Levin, Dan Margolis, Nachi Nagappan, Peter Oehlert, Andy Renk, Hassan Sultan, Gavin Thomas, Chris Walker

Page 41: Jason Shirk, PMDave Weinstein, Sr. SDE jason.shirk@microsoft.comdave.weinstein@microsoft.com jason.shirk@microsoft.comdave.weinstein@microsoft.com Security

Trustworthy Computing

Links

For more information on Microsoft’s Security Science and !exploitable Crash Analyzer, please visit:

http://www.microsoft.com/security/msec/default.mspx 

And the Security Research & Defense (SRD) blog:

http://blogs.technet.com/swi/default.aspx

Page 42: Jason Shirk, PMDave Weinstein, Sr. SDE jason.shirk@microsoft.comdave.weinstein@microsoft.com jason.shirk@microsoft.comdave.weinstein@microsoft.com Security

Trustworthy Computing

© 20097 Microsoft Corporation. All rights reserved. Microsoft, Windows, Windows Vista and other product names are or may be registered trademarks and/or trademarks in the U.S. and/or other countries.

The information herein is for informational purposes only and represents the current view of Microsoft Corporation as of the date of this presentation. Because Microsoft must respond to changing market conditions, it should not be interpreted to be a commitment on the part of Microsoft, and Microsoft cannot guarantee the accuracy of any information provided after

the date of this presentation. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.