Inside PCI Forensic Investigations: What Every Company & Investigator Needs to Know

Presented by Kat Valentine and Walter Conway

Computer Forensics Show

October 2011

Goals and Takeaways

Discussing cardholder data breaches – what really happens?

Merchants: We want you to never need our services How to best prepare for this scenario

Law Enforcement Officers (LEOs): Helpful to know what the merchant is about to go through and what their requirements are

Forensic Analysts: There's a whole other industry made up of investigators who are not LEOs

The card brands care about your response to the incident and what/who caused the breach

Agenda

Who are we?

What is the PCI Data Security Standard (DSS)?

What is a PFI?

Common attack vectors

You’ve been breached… now what?

Incident Response and Forensic Investigation detailed

Protecting yourself by preventing this scenario

Really uncomfortable merchant situations

Up for debate

Questions?

Who is 403 Labs, LLC?

Full-service information security consulting firm

Specializes in the Payment Card Industry (PCI) space

PCI Forensic Investigator in addition to being a QSA, PA-QSA and an ASV

Helps clients protect critical business and customer data

Provides a full suite of security services including assessments and penetration testing

Works forensic cases of all types, criminal and civil

Interacts with all levels of law enforcement – local, state, and federal

Assists in prosecutions

What is the PCI DSS?

Payment Card Industry Data Security Standard Set of security practices to protect card data Unified security standards from individual card brands Affects anyone taking cardholder data from small merchants to

globally recognized brands Required for all entities that process, store or transmit cardholder

data, regardless of transaction volume 12 common-sense security requirements

Meant to provide guidance in the creation of a secure network Over 280 specific sub-requirements Some requirements in place with forensic investigations in mind

What is the PCI DSS?

PCI compliance is not enough to avoid being breached, but makes the merchant a harder target Difference between compliance and security Newer attack vectors and “zero-days” may not be covered by

existing security controls PCI DSS only a minimum set of requirements -- one size does

not fit all Many PCI assessments lack proper scoping and rigor Assessments are merely a snapshot in time Compliance and security need to be ongoing efforts

What is a PFI?

Payment Card Industry Forensic Investigator Forensic agency that specializes in credit card breaches Approved and governed by PCI SSC Required to be both a QSA and PA-QSA firm Evolved from Visa’s Qualified Incident Response Assessor

(QIRA) program

What is a PFI?

PFI program replaced QIRA program October 2010 Investigates incidents on-site Assures acquired data is forensically sound and could be used

in court of law Identifies cardholder data (CHD) environment and compromised

hosts/networks/devices Oversees remediation Provides final assurance to card brands that breached entities

have been secured and returned to a compliant state As of today, 14 companies approved to be PFI firms Only nine PFI firms approved for the United States All 10 firms from QIRA program grandfathered into PFI program

What is a PFI?

Actual guidelines PFIs need to follow to get approved by PCI Council Must have designated core forensic investigators Cannot take on cases where the firm was a QSA for breached

entity; can take cases as the PA-QSA for breached entity's point of sale (POS) device -- must maintain independence

PCI Council checks your forensic procedures and references For every geographic zone you wish to do work for (service

markets), need to have Certified Forensic Investigator (CFI) for that market

Players with a stake in the investigation:

Card brands VISA, MasterCard, AMEX, Discover, JCB International

Processor / Merchant bank

Gateways

Your QSA

Your POS’s PA-QSA

Vendors Hardware/Software and Implementation companies

Who answers to whom?

Processing bank answers to card brands

Compromised merchant answers to bank

Hardware / software vendors = complicated

Implementation vendors = also complicated

How are the majority of CHD breaches discovered?

*From the Verizon 2011 Data Breach Investigations Report

Fun fact: <1% of merchants detect their own breaches

Common Attack Vectors - Physical

Installing rogue WAP Attaching wireless devices to networks

USB w/ malware, keyloggers, etc. Attaching external devices to capture keystrokes or drop

malware to the POS

Attaching recording devices to phones for mail order or telephone order (MOTO) transactions

Theft of endpoints (laptops) or back-office server POS

Really Interesting Physical Attack!

“Burn” phone + Arduino + Lithium batteries + unmonitored public terminals = MONTHS of CHD!

Drop and walk

Burn phone texts or e-mails CHD at the swipe

*Photo by Mikko Hypponen

Common Attack Vectors - Logical

Logical vectors (illegal access to systems) Wireless Malware Remote access Really weak passwords Web applications Storage of CHD POS flaws

Common Attack Vectors

*From the Verizon 2011 Data Breach Investigations Report

You’ve been breached… Now what? (50 ft.)

1. Merchant identified as Common Point of Purchase (CPP) All stolen cards were used at this merchant location before fraud

activity Identified by Merchant ID (MID) -- usually tied to one physical

location, even if there are multiple locations

2. Merchant directed by card brands to get a PFI involved Has to go to a PCI Council-blessed PFI firm, not just any forensic

agency

You’ve been breached… Now what? (50 ft.)

3. Merchant contacts PFI agency; initial scope is defined Processor / card brands play a part in determining scope, but

scope might get bigger in time Documentation? (Network diagram, data flow and storage

diagram, etc.) Any public-facing POS terminals or pay-at-the-pump / Redbox

devices? Provide analyst with make/model of unattended devices so we can

come prepared with stock photographs and identify any differences Any cameras on sensitive areas?

You’ve been breached… Now what? (50 ft.)

3. Merchant contacts PFI agency; initial scope is defined (continued) Multiple locations? Multiple POS solutions? Inventory system?

Stand-alone POS? Integrated inventory system? Mode of connectivity for the POS? Are multiple locations connected to one another?

Sometimes cheapest option might be to send someone from PFI onsite and forego interview process

Sometimes IT staff is POS vendor Sometimes IT staff doesn't have an inventory or a clue

You’ve been breached… Now what? (50 ft.)

4. On-site data collection / acquisition Interviews Confirm initial scope Sweep to look for physical intrusions Documentation of the environment (pictures, video) Live memory acquisition Network captures Drive acquisition Digital Media Evidence (DME) collection (think DVR system)

You’ve been breached… Now what? (50 ft.)

5. Analysis Chain of custody maintained Working copy created Analysis in PCI SSC-approved lab

Live memory Running processes Active network connections

Network captures Drives

Unallocated space Malware analysis

Timeline of events – Piecing it together

You’ve been breached… Now what? (50 ft.)

6. Write and submit to bank / card brands Preliminary report

Type of account data exposed (PAN, track, CVV2, etc.) Steps taken in investigation thus far Initial thoughts on nature of the breach

Forensic report How the breach occurred Number of compromised cards confirmed Merchants PCI DSS compliance status at the time of the breach Verifying eradication and recovery efforts were effective Verification merchant is now compliant with PCI DSS

You’ve been breached… Now what? (50 ft.)

7. Follow-up investigations if scope widens Potential for further investigations Potential for penetration test, pre- and post-eradication Potential for additional PCI assessment by QSA to identify any

gaps in compliance and prescribe a detailed remediation plan

Incident Response and PFI in Detail

Incident Response (IR) 1. Preparation

2. Identification

3. Containment

4. Eradication

5. Recovery

6. Lessons Learned

Incident Response and PFI in Detail

Incident Response asides: PCI DSS doesn’t really provide detailed guidelines regarding

incident handling Card brands REALLY care about IR Some requirements apply to preparation and identification to aid

in forensic investigations, so your PFI has valuable data to analyze

Card brands have specific requirements for containment, eradication and recovery phases

IR: Preparation

Goal: Get the company ready to handle different security incidents per PCI DSS before any incidences occur

Card brands have different security compliance programs and different approaches to deal with a security breach Get familiar with them and keep in mind - they change Lack of prep results in additional fines by the brands -they take

IR VERY seriously

IR: Preparation

Investigate who you want to work with before an incident occurs PFI firms

Your processor is a good source of unbiased information (...maybe) -- they know the players and have had exposure to several breaches and thus, several PFI agencies

Call & interview them! Lawyers

Ask if they have an on-staff forensic investigator and experience with data breach scenarios

PR firms

IR: Preparation

Identify scenarios where breach should be reported to LEOs and have an idea of specific law enforcement agencies for specific situations Local for physical intrusions FBI and Secret Service for major data intrusions Know what getting an LEO involved means for business

Identify internal staff who know everything about everything Can your own company conduct its own internal investigation

without corrupting valuable data?

IR: Preparation

Business continuity versus forensic integrity Have a backup plan, whether it’s parallel networks or simple

dial-up terminals Made more difficult with POS/inventory integration

Payment systems / environment Shut down? Disconnect from network? Business as usual? Depends on specific scenarios – start imagining now Lean on the PFI for guidance – we know you have a business to

run!

IR: Preparation

But whatever you do… Don’t cover up a breach – we'll find evidence of that, and it won't

be pretty (regarding card brands) Regularly test IR plans (12.9) Know how your POS works before a breach

Make sure you get an implementation guide... AND READ IT Disable debug logs -- POSs put stupid data in debug logs, like track

data from memory dumps Encryption key rotation – Do you handle that? Do they handle that?

Nobody handles it?

IR: Preparation

PCI DSS requirements establish a foundation for effective incident handling and forensic investigation process Documentation

Network diagram Standard system builds / configs Change control documentation Digital Media Evidence Audit trails

Processes Security awareness program / training Log reviews

IR: Preparation -/- Documentation

Network Diagrams – Requirement 1.1.2a and b Quick and dirty picture of CHD environment Always need to be up-to-date and accurate

Pretty important, as it could slow identification of scope, re-work and eradication of the breach cause

Include dataflow information in the network diagram In motion – Internal and external connections to CHD environment Resting data repositories – Depict databases and files containing

CHD

IR: Preparation -/- Documentation

Documentation and business justification for services, protocols, and ports allowed – Requirement 1.1.5 During PFI, all firewall and router configs to be reviewed Investigators want business justification for services and

protocols allowed if involved in breach (most cases it is) Also to be identified and included in the report = non-approved

rules and access control lists (ACLs) Was poor firewall and router config responsible? Were their firewall or router config changed or compromised?

IR: Preparation -/- Documentation

Documentation of standard system builds/configs – Requirements 2.2.a, b and c Identification of normal applications and processes that should

be running on system -- helps in identifying potential malware used in attack

Current information contributes to the IR process by providing assurance that OS and apps were not altered

Requirement 2.2 – Industry-accepted system hardening standards

Change control documentation – Req. 6.4 Change control processes the org can trust

IR: Preparation -/- Documentation

Audit trails of all system components in the CHD environment – Requirement 10 Good idea -- logs exported from live systems to secured server

to avoid alterations from hackers attempting to cover their tracks 90 days available immediately... one year available from

backup/storage While we’re at it – Consistent time across all systems via

Network Time Protocol (NTP)

Antivirus audit logs Did AV detect malware used by attackers? Make sure AV is configured to quarantine, not just eradicate

IR: Preparation -/- Documentation

Video camera data (or Digital Media Evidence / DME) – Requirement 9 Can be critical in investigations where physical compromise was

a factor Potential to identify rogue access points, modem deployments,

custom hardware deployments, skimming by employees, etc. Three months of footage to be immediately accessible; one year

stored

IR: Preparation -/- Processes

Daily log reviews to immediately detect potential breaches – Requirements 12.2, 10.6, 12.5.2 and 12.5.5 Daily operational security procedures to cut time between when

breach occurred and when breach was discovered Fun fact (again): <1% of merchants detect their own breaches Its obvious whether or not you're doing daily security reviews via

who identifies you as a CPP

IR: Preparation -/- Processes

Key management processes related to CHD encryption – Requirement 3 CHD must be unreadable anywhere it is stored Encryption often the last layer of defense PFIs and LEOs need to know how encryption keys are handled

to verify if they have been compromised Generation Distribution Storage Destruction Revocation Replacement

Re-encryption?

IR: Preparation -/- Processes

Formal security awareness program (12.6)

Annual risk assessments (12.1.2)

Annual penetration tests (11.2)

External and internal vulnerability scanning (11.3)

Mailing lists and security newsletters from your vendors (6.2.b)

Ongoing IR training Electronic evidence preservation best practices for internal

employees Legal ramifications and legal considerations

IR: Identification

Goal: Identify scope and containment / eradication next-steps The clock starts when card brands confirm you to be a CPP Merchant may have as few as three days to sign with a PFI

In some cases, if the merchant refuses, Visa will hire one and charge the merchant

Preliminary report - Five days after first day on-site Contains findings thus far and suspected / potentially compromised

account details Final report – Ten days after analysis in the lab completed

IR: Identification Within 10 business days, Visa wants a list of all known

compromised cards Visa then shares potential compromised cards with issuing banks Issuing banks monitor / confirm activity While monitoring is going on, investigation starts happening

PFI firm to acquire live memory, network captures and disk images

PFI agency to analyze acquired data -- determine cause from forensically sound data

Document all events into timeline and correlate

IR: Identification Merchants, I know it’s tempting, but don’t start eradication just yet!

Don't access or alter confirmed compromised systems without guidance from your PFI

Don't change passwords – tips off attackers, compromise new password

Isolate compromised systems (unplug network cable if you have to) Know the type of CHD at risk (account numbers, expiration dates,

forbidden fruit, a.k.a. track data?) Log all actions taken internally (court) In case of WAP, change the service set identifier (SSID) ASAP and

document the change Potential: set up honeypot Be on high alert

IR: Containment, Recovery and Eradication

Contain Segmentation

Recovery Follow business continuity plan Keep in mind: Is it possible to rebuild on the existing network?

Potential game-changers like switching POS vendors

Eradication Happens AFTER investigation -- you need to know what’s

affected first, so don’t eradicate yet -- Could ruin chances of figuring out the full scope

Examples: Discover / destroy malware, harden systems, etc.

Repercussions of CHD Breach

Associated costs Card brands assess fines

Trickle-down – card brands --> processors --> gateway --> merchant

Fines are a trade secret – can’t know for sure Fines per location / Merchant ID (MID) Under 10k cards exposed – things get loose and brands may

assess fairly minor fines that are defined based on situation Over 10k cards exposed… good luck! Fines = reissue of cards, data protection services for

customers (credit watch), merchant punishment, overhead costs, etc.

Repercussions of CHD Breach

Associated costs Fraud transactions charged to compromised cards -- those

transactions, at the discretion of the card brands, get handed off to the CPP

Hiring PFI firm and the investigation itself PCI assessment costs (post-incident) Legal fees PR costs Loss of employee productivity

Failing to report the breach: Additional fines (of course) Reporting it actually makes your company look good – shows

you had requirements in place to identify breaches

Protecting Yourself by Preventing This:

PCI DSS is the minimum -- Do at least that

Avoid storing CHD

Know your scope Merchants shocked to learn their VoIP call center may be in scope Reduce entry points while reducing scope

Monitor physical controls Look for changes via line of sight and weight

Watch for social engineering (SE) tricks A guy in a jumpsuit saying they need to change hardware out

Web presence -- Consider your databases (DB) Is that public-facing, low-priority DB connected to the CHD DB?

Really Uncomfortable Merchant Situation:

A lot of the time, it’s not the card brands who first discover you’re a CPP Really common to get notified by issuing / processing banks Creates several weeks of limbo between the merchant being

notified by a reliable source that they've been breached and the card brands actually mandating the PFI

During this limbo time, merchants go to their POS vendors... which may or may not have had a hand in the breach

While merchant and processing bank try to guess if the breach will require a PFI, time to acquire meaningful data elapses

Up for Debate:

Business continuity versus forensic integrity Very difficult to juggle Backup plan to continue business? From PFI perspective, doesn't matter if you catch the crook, but...

you might want to recover damages and have card brands go a little easier on you

What happens when cloud computing needs a forensic investigation? .vmdk file might make things easier? …Maybe?

Questions?

Thank you!

Kat Valentine, ASV, CCNA – kvalentine[at]403labs[dot]com

Researcher & Forensic Analyst

Walter Conway, QSA – wconway[at]403labs[dot]com

Manager, Author

403 Labs, LLC

www.403labs.com

877.403.LABS