Incident Response and Digital Forensics –Denver Chapter of ISACA

Outline Introduction The Incident Response Process

Preparation Identification Containment Eradication Recovery Lessons Learned

The Attacker Process Reconnaissance Scanning Exploitation Keeping Access Covering Tracks

Conclusion

Introduction

FishNet Security

Started in 1996, largest InfoSec VAR Support, Consulting, Training, Products Locations – Boston, Omaha, Kansas City,

St. Louis, Dallas, Minnesota Assessment Team: Policy, Network,

WebApp and DB, Wireless, IR and Digital Forensics

Introduction

Trey Keifer - (trey.keifer@fishnetsecurity.com)

Level II Security Engineer

Sr. Incident Response Analyst Coordinator of Incident Response

Program SANS Certified Incident Handler (GCIH)

Incident Response and Digital Forensics

One of the least practiced, most stressful, highly scrutinized areas of Information Security.

Every incident is unique and can incorporate many different areas of the affected organization.

Incident analysts must be able to think quickly, remain calm and consider all possibilities.

Common Incident Types

Economic Espionage Intellectual Property Theft

Unauthorized Access Stolen Passwords and Data

Unauthorized Use Inappropriate E-Mail and Web Habits

Malicious Code Worms with Backdoors (Sasser)

Insider Threats

6 Steps of the Incident Handler Methodology

Preparation Identification

Containment Eradication Recovery Lessons Learned

Preparation:

The key to a successful response is preparation.

Form a strategy. Design a procedure. Gather Resources. Practice, practice, practice.

Preparation:

Identify the “Core Team”

Technical (IT, InfoSec and System Owners) Management Legal Department Forensics Public Relations Human Resources Physical Security and Maintenance Telecommunications

Preparation:

Organizing Individuals All members of the CSIRT team should know their

role and how they will interact with the other members.

Outsourced or “third party” members should have contracts in place.

Contacts for Law Enforcement should be known and situations for their involvement discussed.

Preparation:

Develop a Procedure Incident response can be a high-stress time. A

well documented procedure, that is easy to follow, can greatly reduce the anxiety.

Develop a call tree and notification procedures Brainstorm likely scenarios. Identify general information needed in most

scenarios ahead of time. Make checklists and forms for as much as

possible.

Preparation:

Communication Communication is incredibly important during an

incident. Not only the people involved, but the method which it is done.

Updates should be frequent. Out-of-Band Communications are very important.

Faxes Cell Phones Be careful with the Blackberry’s

Preparation:

Access Rights The incident response team must have access to

systems without the administrators authorization.

Controversial Issue User Accounts, Passwords and Encryption keys Third-party storage methods are available

Preparation:

Policies Protect the organization from legal liability and

allow investigators to do their job.

Warning Banners are readily displayed. Search policy is detailed in employee manual. Human Resources and Legal have signed off. Employees have acknowledged knowing their

expectations on privacy. Beware of international laws (European Privacy

Directive)

Preparation:

Gathering Resources Incident analysts should have all information

ready and be able to respond to the incident.

Procedures, Checklists and Forms are ready. Access credentials are available or individuals

with them are known. System information, network diagrams, software

and intellectual property are documented thoroughly.

Preparation:

Training SANS Institute and GIAC Certifications

Track 4: Incident Response and Hacker Techniques Track ??: Digital Forensics

Vendor Training Guidance Software Access Data

Partners Incident Response Scenarios

Identification:

“Incidents can’t always be prevented, but must always be detected.”

Incident: Intentional or Unintentional

Multiple failed logins to the domain administrator account.

Administrator credentials were cached on a users workstation and they are attempting to login.

Someone is actively attempting to brute-force the account.

Identification:

Goals

Determine Scope Identify what systems, people and informational

assets are involved in the event.

Preserve Evidence Protect the facts of the incident while determining

the scenario.

Identification: Suspicious Events

Unexplained Occurrences

New Accounts or Files File Modifications IDS Triggers Firewall Entries Accounting Discrepancies Poor Performance/Unresponsive services System Instability

Identification: Passive Identification

Sniffers and Traffic Analysis

Cyclical Buffers allow full recording of events at the packet level to a point, depending on size and utilization.

Target machine evidence is still preserved. Assist in determining new attacks for which

signatures have not yet been written.

Identification: Passive Identification

Intrusion Detection Systems

Least invasive method Target machine evidence is preserved Logs must still be protected

Write-Once, Read-Many Media

Identification: Passive Identification

Tripwire-style File Modification A hash of the file is taken and stored in a secure

database. Any modification to that file results in a change of the hash.

Very indicative of a successful compromise. Can be noisy during patching and must be tuned

after every software upgrade.

Identification: HoneyPots and HoneyTokens

Specific systems or accounts with additional logging and notification to alert on suspicious activity.

Operators must be careful of entrapment. Systems have to be secured and heavily

monitored. Systems cannot invite intruders –

No “hackme” accounts No “Salary Database” systems

Identification: Chain of Custody

Evidence must be accounted for from the time it is collected until the time it is submitted to the court.

Each piece of evidence must be under the control of one, identifiable person at all times.

A change in control of the evidence must be recorded.

Evidence in storage must be protected from contamination. (ie… sealed and secured)

Containment -

Now that the events have been identified as an incident and a chain-of-custody for evidence has been established, we will take the first step into system modification by beginning our containment.

Containment:

Vendor Coordination Work closely with your vendors and know how to

open security-related tickets with high priority.

ISPs can prevent some Denial of Service situations. They are more familiar with attacks because they have

seen them with other clients and are up-to-date on advisories.

Additional people working towards identification, containment and recovery.

We are used to the pressure!

Containment:

Identifying the Trust Model The trust model identifies not only the

technology, but also the people that are involved in the incident.

What connectivity does the network or system have to other areas in the organization?

What information is contained within it? Who needs to be involved and to what extent?

Containment:

Documentation Strategies Documentation should be collected from

most volatile to least volatile and least invasive to most invasive.

Volatile evidence includes RAM, running processes and active connections.

Be careful of running system commands from anything but recovery media.

Containment:

Should we Quarantine? Changes to a system may be easily

observed by an active attacker.

Rootkits may identify a pulled network connection or extensive system modification and protect the attacker.

Some exploits are entirely memory resident and will disappear when the power is pulled.

Containment:

Initial Analysis

Keep a low profile Never analyze the original Make frequent updates to CSIRT Acquire log files Stick to the facts and avoid blame Consider all possibilities but keep it simple

Containment:

Backups Numerous backups allow both

investigation and preservation of evidence.

Different strategies exist and depend on the situation.

Original is kept as evidence Backup 1 – Placed back in production Backup 2 – Forensic Analysis Backup 3, 4, etc… separate copies for analysis

Containment:

Digital Forensics Numerous separate analysis all yield the

same results.

Requires specialty hardware, software and training.

Bit by Bit copying and analysis of data. Recovery of deleted data. Identification of altered system files (trojans) and

binaries in a safe environment.

Containment:

Digital Forensics: Hardware Write Blockers No modification to the data itself, we want

to observe and duplicate only.

Hardware device or driver between acquisition machine and target system.

May use NIC, USB, FireWire or IDE/SCSI channels. Intercepts write commands and gives logical

return results. Allows browsing of the filesystem during

acquisition.

Containment:

Digital Forensics: Forensic Software Allows quick and efficient analysis of the

information contained on the device.

Guidance Software’s EnCase used by law enforcement.

Linux Forensics CD’s are coming along in maturity. (still must use write blockers!!!)

Scripts allow quick searching of keywords in files and deleted data.

Hash comparisons verify original files, known dangerous applications and aid the examiner in avoiding the bad stuff.

Containment:

Digital Forensics: What are we looking for? Many areas of interesting data are

forgotten about.

Cached web content Email Files (PST’s) Recoverable Deleted Files Specific Incidents: CAD drawings, Engineering

diagrams, Pornography Known file signatures of hacking tools, backdoors,

etc…

Containment:

Digital Forensics: Other devices? May not be able to submit as evidence in

court, but can assist the Incident Handler in their investigation.

Personal Organizers (PIMs): Blackberry, Palm Pilots, IPAQ’s.

SIM Cards/Cell phones USB Tokens/Flash Drives

Containment:

Digital Forensics: Not Perfect! Some tools have been written specifically

to defeat forensics software.

DoD: 7-Pass, random-write method for secure deletion of magnetic media. (Rainbow Method)

Windows: Eraser Unix: Wipe

Containment:

Slowing the Attack Change passwords and access rights. Change hostnames and IPs. Null Route suspicious traffic. Block IPs or Networks. Apply Patches to similar systems. Shutdown services.

Eradication -

Once an incident has been contained we attempt the total removal of malicious applications from a system or network.

Eradication:

Remove or Restore The decision of whether to remove malicious files

or restore from backups is a difficult task.

Rootkits almost always demand a rebuild. Verification of backups is a must. Patches may not be available and a total change

of architectures may be necessary.

Eradication:

Improve Defenses Implement additional detection and protection

methods and strengthen existing technologies and processes.

Apply firewall and router filters. Perform “mini-assessments” using the same tools

and techniques as your attackers. Look for the same exploits and backdoors on

multiple machines.

Recovery -

Once the threat has been removed the organization must begin the process of returning the business to normal operation.

Recovery:

Returning to Operation System owners make the final call on returning to

production.

Owners depend on the systems and know their true value.

If a disagreement occurs on whether to return to production or not it should be documented by the analysts and the owner should acknowledge responsibility.

Recovery:

Monitoring At this point in the process you should have

enough information to identify the attack if it occurs again.

Create custom IDS signatures if possible. Verify proper operation to baseline configurations. Implement additional logging on network, hosts

and applications.

Lessons Learned -

The lessons learned meeting provides a method for the organization to coordinate knowledge of an incident, suggest changes in procedures and policies for the future and justify the implementation of new safeguards.

Lessons Learned:

Recap Meeting Should occur promptly after eradication of

an incident while details are fresh in the team members heads.

Create a timeline of events. Provide a consensus of notes and

documentation. Finalize facts for a final report.

7 Deadly Sins

Failure to report/ask for help Incomplete/Non-Existent Notes Mishandling/Damaging Evidence Failure to create backups Failure to eradicate or contain Failure to prevent re-infection Failure to apply lessons learned

Attacker Methodology

Reconnaissance Profiling the Target

Scanning Identifying Weaknesses

Exploitation Breaking the Law

Keeping Access Backdoors

Covering Tracks Staying out of Jail

Reconnaissance:

The target is profiled –

Employee Information (name, numbers, titles) Systems Information (usenet postings, job listings) Process Information (vendors and transactions) Location Information (external networks, physical

locations)

Scanning:

Port and Vulnerability scanners are run to identify vulnerable systems.

Open Ports and Services Vulnerable Applications Default Usernames and Passwords Weak Encryption Implementations

Exploitation:

Execution of attack – usually the first point at which the law is broken.

Goals Gaining Access Elevating Access Extracting Information Denying Service (DoS)

Keeping Access:

Addition of Admin-level User Accounts Enabling of default, insecure services Installation of “Backdoor” or “root kit”

applications allowing the attacker to retain access despite system modifications.

Application Level Traditional Rootkit Kernel Level Rootkit

Covering Tracks:

Modification of system logs, applications and processes to prevent identification by administrators. Hiding files and Directories (… and alt-255

dirs) Changes in /var/log Changes in shell history Removal of events (windows)

Our Example Scenario

An attacker uses a “0-day” exploit to infiltrate the target organization, install a backdoor and retrieve critical intellectual property for a competitor.

Normal security procedures alert the administrators to suspicious activity and the incident response plan is activated.

Attacker Perspective: Reconnaissance

Google and the corporate web site are used to identify the organizational structure of key personnel including HR managers and executive management.

Low-Profile, no data sent directly to organization.

Impossible to detect.

Attacker Perspective: Harvesting

Freely-available scanningtools are used to identifyemail addresses from thecorporate website.

Same method as SPAMgroups.

Many sites do not usegeneric web addresses.

Attacker Perspective: Exploitation

Attacker sends malicious application to email addresses obtained during scanning.

Users open emails (possibly through social engineering) and are immediately infected.

Attacker can be listening for connections from infected machines and have immediate control over systems.

Attacker Perspective: Keeping Access

Incident Timeline

Incident Timeline: Preparation

IR Team established and roles defined. Daily procedures established for log

analysis and identification. Containment procedures are outlined in

policy. (Restoration takes priority) Roles and Responsibilities are defined

Incident Timeline: Identification

Bandwidth graphing shows abnormal usage

Passive sniffing identifies responsible host

Incident Timeline: Containment

No “watch and learn” policy, power is pulled from the host.

System is imaged using forensic tools and Hardware Write-Blockers which prevent alteration of data during backup.

Employee is interviewed to determine method of infection.

Incident Timeline: Eradication and Recovery

System is restored from the organizations hardened base image and patches are applied. (Analysis can continue through restore)

Incident Timeline: Lessons Learned

Social Engineering Awareness File attachment blocking Firewall Rule Revisions IDS Signature changes Patch Management Advisory Alert Services

Questions?