Security Metrics

Spring 2005 CS996

Information Security ManagementPolytechnic University

Michael Aiello

Overview• Why do we care?• What is a metric?• How do we decide which metrics to collect?• How are they collected?• Effective risk analysis through security metrics• How do security metrics make a corporation money

(operational risk)?• “Compliance competition” and security ROI• POSA Example• Problems Experienced• Homework

Why do we care?

• How do we know how “secure” an organization is?– Metrics help define “secure”– Metrics let us benchmark our security

investments against other organizations – Compliance– The metrics “gathering” process often leads to

identification of security inconsistencies or holes

Why do we care: Example

• Manager asks, “Are we secure?”• Without metrics:

“Well that depends on how you look at it.”• With metrics:

“No doubt about it. Look at our risk score before we implemented that firewall project. It’s down 10 points. We are definitely more secure today than we were before.”

Why do we care: Example

• Manager Asks: “Have the changes that we implemented improved our security posture?”

• Without metrics:“Sure. They must have, right?”

• With metrics:“Absolutely. Look at our risk score before we made the recommended changes, and now it’s down 25 points. No question, the changes reduced our security risk.”

Motorola CISO on Metrics

• “Security experts can't measure their success without security metrics, and what can't be measured can't be effectively managed.” (William Boni, PresidentCISO, Motorola Inc. www.secmet.org)

What is a metric?• The National Institute of Standards and Technology

(NIST) define metrics as tools designed to facilitate decision-making and improve performance and accountability through collection, analysis, and reporting of relevant performance-related data. Metrics are simply a standard or system of measurement. In this case, it is a standard for measuring security, specifically measuring an organization’s security posture. Although there are some published standards for measuring security, ideally security metrics should be adjusted and tuned to fit a specific organization or situation.

Examples of metrics• Total number of remote connections over a one

month period (VPN, ISDN, dial-up, remote desktop)

• Maximum number of concurrent remote by user• The percentage of total applications that have a

contingency plan by application criticality.• Time to analyze and recommend action on a

security event• Number of Linux servers at least 90% compliant

with the Linux platform security standard

Security Metric Categories• Platform

– Number of Linux servers that are compliant with EFS policy• Network

– DMZ port scans• Incident

– Number of hosts infected with worm XYZ• Vendor

– Average security rating for vendors that touch active customer files• People

– Number of terminated employees with administrator access • Industry

– Number of public security incidents in sector ABC with severity score Z• Political

– Hacktivism scores, amount of sites listing sector/company ABC as potential target

Security Metric Types• Real Time

– Number of concurrent connections to VPN – Usually from incident response systems

• Polled– Number of password reset requests (monthly), – Usually from SA’s or SME’s

• Incident based– Number of machines infected with worm XYZ– Number of vendors suffering from infections of worm

XYZ– Usually from industry intelligence/incident

response/SA’s and SME’s

How do we decide which one’s to collect?

• Policy Mining / Easy to Spot Anomalies• Risk Scoring• ROI / Vendor Evaluations• Regulatory / Cover the industry standards• “Tips” / Visionaries

Policy Mining Example• Policy Statement: All users who connect remotely must be uniquely

authenticated.• Enforcement Mechanism: Users are required to authenticate with a

username, password and securID token to gain access to the internal network.

• Network Policy (VPN): A user Kerberos account must authenticate with both the Radius and securID privilege granting servers before VPN connectivity is established

• Question: How do we tell if a user is uniquely authenticated?• Metric: Maximum number of remote connections by user in a month.• Metric: Maximum number of concurrent connections for a single user• Metric: Total time connected in a single month• Metric: Number of users granted remote VPN privileges• Metric: Number of securID reset requests in a given month• Metric/Alert: user connecting to VPN from different countries simultaneously

Risk Scoring

• Metric: Maximum number of remote connections by user in a month.– Impact: 6/10 (we care about this 6/10 relative

to other metrics in this policy’s risk, this score may come from SME’s/upper management/industry direction)

– Risk: 20% + (10% * last month’s count)– this is where the “soft” analysis takes place

ROI / Vendor Evaluations • “We spent $XXXXX on 4 new application

penetration testers, are our applications more secure now? Should we hire another one?”– Metrics specific to applications not pen-tested, and

those that are• “We are spending $YYYY on product XYZ, is it

worth it to renew the contract or should we start looking for a new solution?”– Metrics specifically surrounding product XYZ and the

problem it is solving• Number of successful social engineering attacks and their

impact before and after the online training seminar

Regulatory (doesn’t yet exist for most industries)

• Baseline metrics (from Spire Security)– Number of patched machines / total– Number services running on external facing

machines– Port Scanning– Incidents

Standards (from Spire Security)Finances

Market CapOverall Revenue/Funding levelOverall Expenses

WorkforceNumber of EmployeesNumber of contractors/tempsNumber of locations with dedicated IT

IT SpendingBudgets for Operations, Maintenance, Capital Employees

EquipmentCount of Servers, appliances, databases, client PCs, Laptops,

PDAsNetwork Traffic

Count of flows, possible flows, actual flows, blocked flows, sessions, commands, transactions

Security SpendingOperations, Maintenance, Capital Expenditures, Number of Security FTEs

Standards contd.Identity Management

Management budgetManagement FTEsTotal User RepositoriesTotal User AccountsCount of user accounts created, accounts modified, password

resets, accounts disabled/deleted, accounts evaluatedAuthentication Events

Number of failed authentications Vulnerability Management Spending

Number of servers/applications/PCs scannedNumber of Vulnerability Management FTEsCount of open ports, known vulnerabilities, patches, configuration changes

Trust Management spendingCount of Trust Management FTEs, policies written, certificates

issued, signed documents, encrypted documentsThreat Management Spending

Count of Threat Management FTEs, alerts, compromised systems

“Tips” / Visionaries

• Investigations/Government/Regulator may ask information security to “monitor” specific activities

• A “visionary” (author/upper management/consultant) will come up with a new/derived metric to collect in order to report on a new phenomenon

How are metrics collected?

• Categorize and define the metric and its owner• Determine and document metric source

– Automated• database connection• Script file output

– Manual• Email polling• Form entry• Manual file updating• Report analysis/research

How are metrics collected?• Define/document collection process for each metric

– A pull replication query mirrors the critical IDS alerts from server ABC database BCA to the metrics collection server DEF database BCA. DEF then sums and categorizes the alerts. The final counts are archived in table QRS in database BCA on server DEF.

– Joe runs a stored procedure on server XYS database YZD which he manually correlates with Radius logs aging over the past 3 months. The report is then stored on share ABCD and Joe sends an email to Sally indicating the metric is updated. Sally then enters the metric information into the metric collection database using the form at URLQYZX

Effective risk analysis through security metrics

• How do we make decisions based on the metrics now that we have them?

• Metrics which are collected should match high impact risk items. (only spend money collecting those with high risk scores)

Risk Breakdown Example• Risk Measurement: Federation information security risk

score (akin to homeland security colors extremely vague, policy generally shouldn’t be created based on such high indicators)

• Risk Components: Network, Incident, Vendor, People, Industry, Political

• Subcomponent: Federation-Global-Network-Trading-FTSE

• Metric inventory for subcomponent: ID4786(A),ID2235(B),ID8674(C)

• Subcomponent risk score calculation: • 50%(A*(∆last 4 months(B))) + (50% * C’s rolling average)• Security risk analysts and SMEs create score weightings

This is complicated and expensive, why do we do it this way?

• As people, we are generally bad at concentrating on more than 7 factors/metrics/indicators at a time

• Risk scoring lets us define and objectively monitor the “big picture” information security view

• Correlations• Alerting / “Smarter” automated responses

How do security metrics make a corporation money (operational risk)?

• Legislation (Basel II) says “you have to withhold 15% of last year’s revenue unless you can prove that you have mitigated your risk”

• Metrics are your proof, risk scores are your slice description of the “state of the union”

• In general, the less money you have to withhold/spend on insurance, the more money you make

“Compliance competition” and security ROI

• The faster/better you can prove compliance the higher the bar is for your competition.– Reports and systems that are easy for auditors to

use.– Meaningful and provable risk scores with metrics– Higher bar for competition means they spend more

time/money/effort complying• Metrics give us a way of measuring security ROI

– Pretend we are auditors evaluating our own organization: now we can measure our security posture before and after implementing a solution

POSA ExamplePolytechnic University

CS996 Information Security Management(example POSA system here)

• Quick reminder of system

POSA

CFAC

USER

1 Sale information7 Complete Trans.

Register

5 Y/N

4 Sale & user information8 Complete transaction

3 User CCinformation

6 Y/N 2 DisplaySale Info

PSOA Metrics

• After doing thorough risk analysis and identifying Assets/Threats/Vulnerabilities assume the following combinations are deemed the most important:

Asset Threat Vuln

Credit Card Database on CFAC

Insider Employee Mal-intent System Administrators

Credit Card Database on CFAC

External “hackers” Network hopping/ Sniffing

Payment Processing Availability

Competition/HactivistsNatural Disasters

Denial of servicePhysical equipment destruction

Ask the question, do we have policy and guidelines to mitigate/monitor these

combinations?

Asset Threat VulnCredit Card Database on CFAC

Insider Employee Mal-intent System Administrators

Credit Card Database on CFAC

External “hackers” Network hopping/ Sniffing

Payment Processing Availability

Competition/HactivistsNatural Disasters

Denial of servicePhysical equipment destruction

• If not, create them• In our case, assume we do• Mine these policies for potential metrics

Risk Measurement: Customer Credit Card Privacy Score

• Risk Components: Network, Incident, Vendor, People, Industry• Subcomponent: Firmwide-Platform-Database-Internal-Access

Control• Metric inventory for subcomponent:

(A)(Real-Time)Number of databases with more than 100 credit cards which do not store credit card numbers in an encrypted manner:(B)(Real-Time)Number of system administrators with view level access to non-encrypted databases with more than 100 credit cards:(C)(Real-Time)Number of system administrators who have criminal history with view level access to non-encrypted databases with more than 100 credit cards(D)(Real-Time)Number of system administrators who have criminal history with view level access to encrypted databases with more than 100 credit cards(E)(Monthly) Number of employees leaving the firm which have had access to non-encrypted databases with more than 100 credit cards:(F)(Real-Time)Number of databases with encrypted credit cards(G)(Real-Time)Number of administrators with ability to decrypt encrypted credit cards(H)(Real-Time)Age of keys used to encrypt credit cards (days)(I)(Manual)Industry time to break one credit card encryption (days)(J)(Manual)% of administrators who have attended social engineering seminarsSubcomponent risk score calculation: (0.75 * A * ((B * .1) + (C * .3))) * (0.25 * F * ( ....) )

How do we do the risk score calculation?

• Risk = Threat * Vulnerability * Expected Loss• ALE (Average Loss Expectancy) = probability of loss *

total loss potential• Asset Valuing

– Productivity Value– Revenue Value– Liquid Financial Assets Value– Intellectual Property Value

• Potential Loss– Confidentiality– Integrity– Availability– Productivity– Liability

How do we do the risk score calculation?

• Measuring Risk– Manifest risk = ratio of malicious events to total

events• (sessions, commands transactions)

– Inherent risk = likelihood that a configuration will contribute to a compromise

• Open ports and services running compared to historical vulnerabilities on those ports

– Contributory risk = measure of process errors during normal course of operations that contribute to a compromise

• User Account Management procedures

How do we do the risk score calculation?

• Correlation– Expert systems, obvious correlations

• Historical testing– Look at data leading up to an incident, see what changed

• SME predictions/insight– Sometimes they just know

• Industry trends– We don’t really care about 1024 bit key breaking in 2005, will we

in 2012? • Firm specific

– Explosion in certain incident types may necessitate a change in the equation

Optimization• After doing risk calculation, don’t spend

time/money to collect metrics that don’t change the final score that much

• Analyze risk score equation against reality, are we really reporting the proper state of the union?– Add “reliability” fields to metrics and weigh according

to them as well.• Correlation/expert system analysis• Prioritize future metrics to be collected (find the

“value” of a metric (risk of not collecting the metrics))

Problems Experienced• Systems unable to report highly critical metrics• System integration• SAs not willing/able to invest time setting up processes to collect

metrics• Ad-Hoc requests/reports and their implication on the overall view

– We didn’t do a penetration test on most of our servers, how can call our network secure?

• Vague risk descriptions, Vague Metric Requests• Impossible metrics (usually external)

– Number of credit card accounts compromised globally across any firm

• Missing/incomplete historical data• Mistrust/inaction/devaluing because of qualitative components• Complete trust

References

• http://www.secmet.org• http://csrc.nist.gov/publications/nistpubs/8

00-55/sp800-55.pdf• http://www1.netsec.net/content/securitybri

ef/archive/2004-09_Metrics.pdf• http://www.cert.org/octave/

Homework• Pretend you are a security analyst at Polytechnic. You have just had the

following (highly simplified and fictitious) conversation with a senior manager.

• Manager: “Another engineering school was just sued because student’s transcripts could be accessed by anyone online. How secure is our new grade transaction server?”

• You: “We believe the new design is secure, however, haven’t allocated time, money and effort to post-implementation security evaluation.

• Manager: “I need to show the board that we are not prone to this type of humiliation, what can you pull together for me in the next 3 months?”

• Your assignment is to describe how you would structure your new metrics proposal which includes the following sections.

– Description of which metrics you will be collecting (Based on risk analysis. Remember, this should be a minimal set, you only have 3 months to set this up)

– A metric collection process example for one of the metrics– Suggested, simple weighting of metrics to calculate the overall risk score of the

system.