Fast, Cheap and In Control: A Step Towards Pain Free Security! · 2018-09-13 · Fast, Cheap and In Control: A Step Towards Pain Free Security! Sandeep Bhatt, Cat Okita, and Prasad

Keyword(s): Abstract:

©

Fast, Cheap and In Control: A Step Towards Pain Free Security!

Sandeep Bhatt, Cat Okita, Prasad Rao

HP LaboratoriesHPL-2008-111

firewall, network, security metrics

We hypothesize that it is possible to obtain significant gains in operational efficiency through theapplication of simple analysis techniques to firewall rule sets. This paper describes our experiences with afirewall analysis tool and metrics that we have designed and used to help manage large production rule sets.Firewall rule sets typically become increasingly unwieldy over time. It is common for firewalls to havehundreds, or even thousands, of rules. Not surprisingly, administrators have a hard time keeping track ofhow the rules interact with each other, resulting in many partially effective or completely ineffective rules,and unpredictable behavior. Our tool can be used to identify these problematic rules. Further, given tworule sets, our tool produces a comprehensive list of the traffic that is only permitted or denied by one ruleset, rather than both. As such, we can compare the existing rule set with a second rule set containing theproposed changes. The administrator can then visually check if the difference in traffic patternscorresponds to what he or she intended in proposing the changes. Additionally our tool collects variousmetrics that help the administrator to gauge the 'health' of the firewall. The tool is designed to be extensibleto multiple vendor products.

External Posting Date: September 21, 2008 [Fulltext] Approved for External PublicationInternal Posting Date: September 21, 2008 [Fulltext]

To be published and presented at 22nd Large Installation System Administration Conference (LISA '08), San Diego, CA November9-14, 2008

Copyright 22nd Large Installation System Administration Conference (LISA '08)

Fast, Cheap and In Control: A StepTowards Pain Free Security!

Sandeep Bhatt, Cat Okita, and Prasad Rao – Hewlett-Packard

ABSTRACT

We hypothesize that it is possible to obtain significant gains in operational efficiency throughthe application of simple analysis techniques to firewall rule sets. This paper describes ourexperiences with a firewall analysis tool and metrics that we have designed and used to helpmanage large production rule sets. Firewall rule sets typically become increasingly unwieldy overtime. It is common for firewalls to have hundreds, or even thousands, of rules. Not surprisingly,administrators have a hard time keeping track of how the rules interact with each other, resultingin many partially effective or completely ineffective rules, and unpredictable behavior. Our toolcan be used to identify these problematic rules. Further, given two rule sets, our tool produces acomprehensive list of the traffic that is only permitted or denied by one rule set, rather than both.As such, we can compare the existing rule set with a second rule set containing the proposedchanges. The administrator can then visually check if the difference in traffic patterns correspondsto what he or she intended in proposing the changes. Additionally our tool collects various metricsthat help the administrator to gauge the ‘health’ of the firewall. The tool is designed to beextensible to multiple vendor products.

Introduction

Securing the increasingly complex and highlynetworked environments of today is a challenging andfrustrating task. Between compliance demands, suchas Sarbanes-Oxley and PCI, new applications and ser-vices, and increased end-user awareness of securityissues, it is a challenge to maintain security in anyenvironment. As the complexity of an environmentincreases, the complexity of the configurationsrequired to secure access to the environment increases,as does the amount of time required to maintain,update and validate the configurations. Changes to thesecurity configurations start to produce unexpectedside effects, and often result in a reluctance to makeany changes at all!

Similarly, the amount of time required to debugobvious access issues increases dramatically with thesize and complexity of the configurations involved –and skyrockets when multiple groups are involved indebugging any single issue. Configuration issueswhich do not result in immediately obvious issuessuch as permitting additional access that either fails tobe noticed (or is so useful that nobody wants to men-tion it!) are often missed, and are hard to find withoutextensive, repeated effort on the part of an experi-enced system administrator (nowadays affectionatelyreferred to as a ‘resource’).

Migration between vendors, or versions of secu-rity devices is also challenging – there is no standardconfiguration language for security devices like fire-walls, and most vendors do not provide migrationtools between their own OS releases, let alone fromthe OSes of other vendors.

Compounding the above issues, since the secu-rity of an environment is often a case of proving anegative, there is a definite need for straightforward,easily understood metrics that can be used to describethe state of a given configuration. Alas, while there aremany tools that can be used to secure an environment– firewalls, intrusion detection/prevention devices,virus scanners, et al. – there is a dearth of simple,multi-platform tools that can be used to analyze andmeasure existing installations.

Problem

In this work, we have elected to focus on ways toimprove firewall rule set management, which manysecurity managers have identified as an ongoing chal-lenge. Typical operational issues include how to deter-mine the effect of adding or removing a firewall rule,clean up messy firewall rule sets and debug firewallrule related issues; other challenges include reportingthe ongoing status of firewall operations in an easy toevaluate format.

The increasing commoditization of firewall man-agement through outsourced and managed services, aswell as decreasing amount of time or skill (or both)available to manage firewalls in house has also led toan increase in ‘‘cargo cult’’ style firewall operations,and a ‘‘once in, never out’’ rule set management.

Methodology

Our goal was to produce a high value, fast, light-weight tool that can be used to improve firewall ruleset management and acts as an easy to implement sup-plement to existing systems and processes, rather thanadding overhead and cost.

22nd Large Installation System Administration Conference (LISA ’08) 1

Fast, Cheap and In Control: A Step Towards Pain Free Security! Bhatt, Okita, and Rao

The methods used must be vendor and productindependent, and be easily extensible to additionalproducts.

Approach

To meet our goal of producing a minimally intru-sive and maximally effective tool, we elected to restrictourselves to analysis of security configurations (e.g.,firewall configurations and router ACLs), and morespecifically to the structural properties of the rule sets.

We do not evaluate whether a given configura-tion enforces the ‘correct’ policy, or adheres to somespecific set of best practices which may or may not beapplicable.

These decisions have multiple advantages – anal-ysis takes place offline, with no requirement for agentsor special access to the security infrastructure. Analy-sis is repeatable, and can be used to show improve-ment (or lack of improvement) in the configuration(s)over time. Further, as the analysis is based on thestructural properties of the rule sets, it is vendoragnostic, and allows for rule set comparison betweendifferent products.

We first identify the minimal set of informationrequired to describe the action of the rule sets fromvarious configurations, which allows us to translatethe configurations to a ‘‘standard’’ grammar. As aresult, it becomes extremely straightforward to handleand compare multiple types of configurations.

We next identify a set of universal issues thattypically cause hard to predict, difficult to diagnose(or inexplicable) effects on firewall management, andthen develop a set of common ideal conditions forfirewall rule sets to avoid those issues, and define met-rics which can be used to measure the degree to whicha given firewall rule set differs from the ideal.

Finally we describe the implementation and useof the prototype tool to improve the management ofproduction firewall rule sets, and lessons learned.

Sidebar: Terminology

Location: Source or destination in a ruleService: Port or protocolAction: What to do when a rule matchesRule: A source, destination, service & action combi-

nationObject: Location or ServiceBlock: A subset of a Rule or Object

For the purposes of this paper, a firewall rule setis a collection of declarations consisting of a sourcelocation, destination location, service and action decla-ration. We do not address transformations such as NATor PAT in this paper, although the work is extensible.Since each location and service can be a group of loca-tions and services, and there are no constraints pre-venting overlapping locations or services, it is unfortu-nately simple to define (and hard to discover) ruleswhich are partially or completely similar to other rules.

Realization

There are policy and vendor independent charac-teristics that can be generalized as being universallycommon to a well managed firewall rule set, andwhich can be used as a basis for metrics to evaluateand improve firewall rule sets.

We posit that the ideal structural properties of afirewall rule set are:

• Non-InterferenceRules should not interfere with other rules.Rules should not partially or completelyoverlap other rules, or be partially or com-pletely overlapped by other rules except inthe case where the more specific rule isacted on first, and is completely over-lapped by the less specific rule, and theaction of the less specific rule differs.Overlapping rules are described as ‘eclips-ing’ or ‘eclipsed’ rules, and are brokendown into ‘interfering blocks.’Objects are unique. Objects do not definethe same location or service.

• SimplicityThere are no unused non-default objectsdefinedOnly rules which can be triggered aredefined in the rule setRules permit only what is required by pol-icy

• ConsistencyRules actions are consistent. If rules inter-fere, except in the case noted above, theyshould have the same action.Object naming style is consistent if namedobjects are used.

Unfortunately, since firewalls are highly idiosyn-cratic, it is not possible to discuss firewall rule setswithout noting the existence of configuration anddevice specific corner cases.

• Object Template Re-Use: Objects in multiplefirewall policies used by a common manage-ment interface must remain identical across allfirewalls using them.

• Rule Set Commonality: Rules which can not betriggered may only be permitted if a single ruleset common to multiple firewalls is in use.

Interpretation

Given the ideal characteristics shown above, wedescribe a set of metrics that we can use to measurehow well a firewall rule set is managed. In order to bemeaningful across multiple firewalls, and multipletypes of firewalls, the metrics selected must also bepolicy independent and vendor independent.

We use Non-Interference and Consistency to pro-vide an understanding of the complexity of a givenfirewall rule set, while Simplicity and Effectivenessmeasure the functionality of the rule set.

2 22nd Large Installation System Administration Conference (LISA ’08)

Bhatt, Okita, and Rao Fast, Cheap and In Control: A Step Towards Pain Free Security!

• Non-Interference (Rules): The fraction of rulesin a firewall rule set which do not interfere withother rules.

• Non-Interference (Objects): The fraction ofobjects in a firewall rule set which do notdefine the same location or service.

• Simplicity (Rules): The fraction of rules in afirewall rule set which can be triggered.

• Simplicity (Objects): The fraction of objects ina firewall rule set which are defined and used.

Since we have explicitly stated that policy analy-sis is out of scope for the purposes of our analysis, wewill avoid the question of how to measure ‘permitonly what is required by policy’ at this time.

• Consistency (Rules): The fraction of interferingrules in a firewall rule set which have the sameaction.

Measuring inconsistent object naming style andobject template re-use is challenging, and is left tofuture work.

Based on the above properties, we propose a newmetric – effectiveness – that can be used to evaluatethe complexity of a rule set. This metric essentiallycaptures the degree to which different rules are inde-pendent of one another; the intuition is that the greaterthe overlap, the more complex the rule set and hencemore costly to manage. This will also allow us to trackthe effectiveness of firewalls over time as their rulesets evolve.

• Effectiveness: A measure of the fraction of agiven Rule or Object that is not interfered with.

We also investigate other metrics such as fre-quency of log hits and interference counts, simplicitymeasures, and consistency measures over rules andobjects.

Implementation

Our prototype tool currently handles Checkpointconfigurations; the Checkpoint configuration file for-mats are notably different from the single file, singleline style configurations of most other firewalls. Wehave done proof of concept checks against CiscoPIX/ASA, pf and ipfilter to confirm our ideal statehypothesis, but have not yet implemented parsers forthose configurations.

Given two rule sets, the tool produces a compre-hensive list of the traffic that one rule set will letthrough but not the other one. As such, we can use itto compare the existing rule set with a second rule setcontaining the proposed changes. The administratorcan visually check if the difference in patterns ofallowed packets corresponds to what he or sheintended in proposing the changes.

The tool is implemented in Java SDK 1.6. It canbe invoked either from a command line or from a Webinterface. This Web interface is implemented usingJetty (Version 6.1). The tool requires the configuration

files (object files and rule files) to be transferred to adirectory accessible to the firewall analyzer (read onlypermission is enough). The user can use the web frontend to explore the results of the analysis, compare theresults of two analyses and run queries via a forminterface.

The raw configuration files are parsed using arecursive descent parser that converts the raw configu-ration files to an intermediate format. The tool inter-prets rules and objects (represented in this intermediateformat) geometrically and computes overlaps betweenthem using computational geometry algorithms. Theseresults are stored in a data structure with multipleindices (rule ids, object ids etc.) so that they are effi-ciently retrievable by the servlet and query algorithms.(Details in http://www.hpl.hp.com/techreports/2007/HPL-2007-154R1.html .)

Case Studies

The challenges described by firewall managerscan be split into three groups – comparison, remedia-tion and reporting.

The anonymized examples below are taken fromlive production environments, and showcase the use ofthe tool and metrics which we have described for fire-wall rule set comparison, remediation and reporting.

Case Study 1: Rule Set Comparison

In this case study, we describe the use of our toolto compare, identify and resolve the differencesbetween two ostensibly identical rule sets.

An end-of-life Checkpoint Firewall-1 NG FP3firewall was scheduled for replacement with a pair ofredundant firewalls running Checkpoint NGX R60,centrally managed by Checkpoint Provider-1.

As the Checkpoint configurations were not com-patible between versions, and Checkpoint did not pro-vide a migration tool, a manual rule and object trans-fer between the old and new environments wasrequired. Since this migration was a bug-for-bug fire-wall rule set migration, the task of manually re-enter-ing the firewall rules from the old environment to thenew environment was delegated to front line supportstaff, with validation of the copied rules being per-formed by a senior engineer.

An initial comparison of the old and new rule setrule effectiveness made it immediately obvious thatthe two configurations were significantly different.

The gross difference in rule sets was swiftlyexplained by noting that although the total number ofactive rules was correct (the new firewall has an addi-tional rule for state synchronization), the old configu-ration had six deny rules, while the new configurationhad 12.

A quick visual comparison of the two rule setsalso revealed a number of rules that were missing, outof order, with incorrect actions, or missing objects.



New New New OldDestination Firewall Firewall Firewall Firewall

Source Firewall Action Rule ID Action Rule ID

10.0.6.230 192.168.10.21 322 accept 352 drop10.0.6.231 192.168.10.21 322 accept 352 drop10.0.6.232 192.168.10.21 322 accept 352 drop

172.16.0.0-172.16.32.20 192.168.11.150 311 accept 352 drop172.16.0.0-172.16.32.20 192.168.10.150 353 drop 310 accept

172.16.32.22-172.16.35.255 192.168.11.150 311 accept 352 drop172.16.32.22-172.16.35.255 192.168.10.150 353 drop 310 accept172.16.64.0-172.16.72.255 192.168.11.150 311 accept 352 drop172.16.64.0-172.16.72.255 192.168.10.150 353 drop 310 accept

172.16.128.0-172.16.144.255 192.168.11.150 311 accept 352 drop172.16.128.0-172.16.144.255 192.168.10.150 353 drop 310 accept

Table 1: Case Study 1 – Rule set comparison – Object TCP-3389 interference.

0

50

100

150

200

250

300

350

Effectiveness (%)

Nu

mb

er

of

Ru

les

Old Firewall (baseline) 2 2 13 27 72 237

New Firewall (baseline) 327 0 1 0 13 13

0 0 to 25.0 25 to 50.0 50.0 to 75.0 75 to 100.0 100

Figure 1: Case Study 1 – Rule set comparison baseline.

0

50

100

150

200

250

Effectiveness (%)

Nu

mb

er

of

Ru

les

Old Firewall (Baseline) 2 2 13 27 72 237

New Firewall (First Pass) 2 2 17 30 69 234

0 0 to 25.0 25 to 50.0 50.0 to 75.0 75 to 100.0 100

Figure 2: Case Study 1 – Rule set comparison first pass.



These issues were straightforward to identify,and correct, and would certainly have been identifiedthrough manual examination. Once the first pass forgross errors was complete, the effectiveness of boththe old and new rule sets was nearly identical.

Old Firewall New Firewall(Baseline) (Baseline)

Drop Rules 6 12Accept Rules 347 342Total Rules 353 354

0

20

40

60

80

100

120

140

160

0 10 100 1,000 10,000 100,000 1,000,000 10,000,000

Frequency of Log Hits (6 months)

Nu

mb

er

of

Ru

les

Figure 3: Case Study 2 – Rule set remediation – Rule hit frequency figure.

Although the configurations appear to be nearlyidentical from a rule effectiveness standpoint, compar-ing used objects showed a high number of objects inuse in only one of the two configurations – 65 objectsunique to the old configuration, and 77 objects uniqueto the new configuration. 100 objects in the two con-figurations interfered.

Further, as demonstrated by the object interfer-ence example in Table 1, each object could interferewith multiple rules, and either partially or completely.

More specific examination of the interferingobjects revealed that the object definitions also variedbetween the old and new firewalls, and object nameshad not been consistently defined between the old andnew firewalls. Object names had been entered withvarying case (TCP vs tcp vs Tcp), different separatingcharacters (TCP-22 vs TCP_22 vs tcp22) and withcompletely different names (TCP-22 vs SSH).

Resolving the interfering objects then became aniterative process of resolving one set of interferenceand using the tool to compare the configurationsagain, as resolving interference in one rule frequentlyaffected interference in other rules.

Case Study 2: Rule Set Remediation

In this case study, we describe the use of our toolin combination with log file analysis to identify par-tially and completely ineffective rules, and then exam-ine the effect of removing the identified rules from therule set.

We were asked to identify what rules in a givenset of approximately 350 rules were not being used,based on six months of firewall logs, and identify theimpact(s) of removing those rules.

The method we selected was a combination ofconfiguration analysis and log analysis. Log analysiswas used to identify those rules which had few or nohits during the monitored period, and could beremoved, while configuration analysis was used toidentify eclipsed rules, and the impact of removingrules.

As shown in Figure 3, log analysis revealed thata sizeable number of rules had received no hits at allduring the six month analysis period.

Based on input from the customer and review ofthe rules, the decision was made to remove all ruleswhich had fewer than 1,000 hits over six months, withthe exception of two specified IP ranges.

Based on that specification, a new version of therule set was created, and the old and new rule setscompared to determine the impact of the proposedchanges.

As shown in the figures below, removing the sel-dom and never used rules resulted in a dramatic reduc-tion in both the number of rules and the number ofobjects in use.

While the number of rules and objects in the newconfigurations had decreased significantly, there werestill a number of interfering objects and eclipsing rules



0

50

100

150

200

250

300

350

400

Old Rule Set New Rule Set

Rule Set Name

Nu

mb

er

of

Ru

les

Figure 4a: Case Study 2 – Rule remediation – Number of rules.

0

500

1000

1500

2000

2500

Old Rule Set New Rule Set

Rule Set Name

Nu

mb

eof

Ob

jects

Figure 4b: Case Study 2 – Rule remediation – Number of objects.

Object Name (Numeric)

Nu

mb

er

of

In

terfe

rin

gB

locks

Figure 5: Case Study 2 – Rule set remediation – Number of interfering blocks in rule set.



in the new configuration, pointing towards a numberof additional rule set improvements which we did notinvestigate at that time.

y = -0.0003x2 + 0.7758x - 0.3804

R2 = 0.9917

0

50

100

150

200

250

300

350

400

450

500

0 50 100 150 200 250 300 350 400 450 500

Total Rules

Nu

mb

er

Of

Eff

ecti

ve

Ru

les

Completely Effective Rules All Rules Linear (Completely Effective Rules)

Poly. (Completely Effective Rules) Linear (All Rules)

Figure 6a: Case Study 3 – Rule set reporting – Non-interference (rules).

y = -0.0003x2 + 0.7758x - 0.3804

R2 = 0.9917

y = 0.3465x - 5.8035

R2 = 0.9657

y = 0.0003x2 + 0.2205x + 0.1417

R2

= 0.9726

-100

0

100

200

300

400

500

0 50 100 150 200 250 300 350 400 450 500

Total Rules

Nu

mb

er

Of

Eff

ecti

ve

Ru

les

Completely Effective Rules Partially Effective Rules All Rules

Linear (Completely Effective Rules) Poly. (Completely Effective Rules) Linear (Partially Effective Rules)

Poly. (Partially Effective Rules) Linear (All Rules)

Figure 6b: Case Study 3 – Rule set reporting – Non-interference (rules).

Rule Set Reporting

We previously described six metrics which webelieve are a good measure of how well a firewall ruleset is managed. In the first four sections we applythese metrics to a collection of 50+ Checkpoint Fire-wall-1 configurations spanning 2+ years, 34 differentfirewalls and multiple business types, and discuss theresults; the last section looks at a single firewall overtime, and through a migration to new hardware.

• Non-Interference (Rules): The fraction of rulesin a firewall rule set which do not interfere withother rules.

• Non-Interference (Objects): The fraction ofobjects in a firewall rule set which do notdefine the same location or service.

• Simplicity (Rules): The fraction of rules in afirewall rule set which can be triggered.

• Simplicity (Objects): The fraction of objects ina firewall rule set which are defined and used.

• Consistency (Rules): The fraction of interferingrules in a firewall rule set which have the sameaction.

• Effectiveness: A measure of the fraction of agiven Rule or Object that is not interfered with.

Non-Interference

As one might hope, in general, as the number ofrules in a firewall rule set increases, the number of



Number of Locations Used

Nu

mb

er

of

Lo

cati

on

Co

nfl

icts

Number of Interfering Locations

Figure 7: Case Study 3 – Rule set reporting – Non-interference (objects).

y = 0.9953x -

0.1933

R2 = 0.9999

0

50

100

150

200

250

300

350

400

450

500

0 50 100 150 200 250 300 350 400 450 500

Number of Rules

Nu

mb

er

of

Tri

gg

erab

leR

ule

s

Potentially Active Rules Linear (Potentially Active Rules)

Figure 8a: Case study 3 – Rule set reporting – Simplicity (rules).

y = -0.0003x2 + 0.7758x - 0.3804

R2 = 0.9917

y = 0.9953x -

0.1933

R2 = 0.9999

y = 0.6488x + 5.6102

R2

= 0.9896

0

50

100

150

200

250

300

350

400

450

500

0 50 100 150 200 250 300 350 400 450 500

Number of Rules

Nu

mb

er

of

Tri

gg

erab

leR

ule

s

Potentially Active Rules Completely Effective Rules Poly. (Completely Effective Rules)

Linear (Potentially Active Rules) Linear (Completely Effective Rules)

Figure 8b: Case study 3 – Rule set reporting – Simplicity (rules).



non-interfering rules also increases. However, asshown in Figure 6a, as the number of rules increases,the number of non-interfering rules drops away fromthe total number of rules.

Number of Locations

Nu

mb

er

of

Lo

cati

on

sU

sed

Number of Locations Used

Figure 9: Case Study 3 – Rule set reporting – Simplicity (Objects).

y = 0.4384x - 12.621

R2

= 0.9857

y = 0.0002x2

+ 0.2811x - 1.6324

R2

= 0.9924

-50

0

50

100

150

200

250

300

350

400

450

500

0 200 400 600 800 1000

Number of Higher Priority Interfering Blocks

Nu

mb

er

of

In

terfe

rin

gB

locks

wit

hD

iffe

rin

gA

cti

on

s

High Priority Blocks with DifferingActions

Linear (High Priority Blocks withDiffering Actions)

Poly. (High Priority Blocks withDiffering Actions)

Figure 10: Case Study 3 – Rule set reporting – Consistency.

More clearly, Figure 6b suggests that the numberof interfering rules is proportionate to, and increaseswith, the total number of rules. There is also an inter-esting hint that the number of non-interfering andinterfering rules may intersect, and that the number ofinterfering rules will exceed, and eventually over-whelm the number of non- interfering rules, for a suf-ficiently large number of rules.

As Checkpoint uses shared object definitions forall firewalls being managed by the same managementstation, the number of non-interfering objects is heav-ily dependent on the number of objects defined in that

management instance, as the loose plot below demon-strates. It is clear, however, that interference betweenobjects is to be expected in all rule sets, and increaseswith the number of objects defined on a given com-mon management instance.

Simplicity

If we consider Simplicity of rules strictly fromthe standpoint of rules which could be triggered – thatis to say, any rule which is not completely eclipsed,Figure 8a shows that the relationship between numberof rules, and number of potentially active rules is lin-ear, and almost exact.

This is misleading, as Figure 8b shows; as thenumber of rules increases, the number of completelyeffective rules immediately drops away from the num-ber of potentially active rules, showing that rules can



be expected to interact in unexpected ways. The dataobtained for object simplicity is inconclusive, and sug-gests that our approach for examining objects inCheckpoint configurations managed by a shared man-agement instance needs to be revisited.

y = 0.3465x - 5.8035

R2

= 0.9657

y = 0.0003x2 + 0.2205x + 0.1417

R2

= 0.9726

-50

0

50

100

150

200

0 50 100 150 200 250 300 350 400 450 500

Number of Rules

Nu

mb

er

of

Pa

rtia

lly

Eff

ecti

ve

Ru

les

Number Of Partially Effective Rules Linear (Number Of Partially Effective Rules)

Poly. (Number Of Partially Effective Rules)

Figure 11a: Case Study 3: Rule set reporting – Effectiveness.

Number of Rules

Eff

ecti

ven

ess

of

Ru

les

(%

)

Minimum Effectiveness of Rules

Figure 11b: Case Study 3: Rule set reporting – Effectiveness.

Consistency

Here we look specifically at the case of interfer-ing rules, where the action of the rule taking prioritydiffers from the action of the rule(s) being eclipsed.

Unsurprisingly, we see that the number of inter-fering rules with inconsistent actions increases as thesize of a rule set increases. We have not calculated thecase where a more specific rule is acted on first, and iscompletely overlapped by a less specific rule sepa-rately, but other configuration analysis work suggests

Rule Effectiveness (%) R2 Value (poly)

Ineffective Rules 0.66840-25 0.636725-50 0.919550-75 0.942975-100 0.9449Partially EffectiveRules

0.9726

Completely EffectiveRules

0.9917

Table 2: Case Study 3 – Rule set reporting – Effec-tiveness.

that this case does not account for the majority of ruleinterference.



Effectiveness

As shown above, there is a direct correlationbetween the number of rules, and the number of par-tially effective rules. Less obviously, the variance inrule effectiveness also increases as the rule set size

increases. While the overall R2 value for partiallyeffective rules is excellent, breaking the rule effective-ness into ranges is suggestive. The lowest correlationvalues are associated with the most ineffective rules,suggesting that these rules may be easier to detect andresolve.

y = -2E-13x2

+ 0.0004x - 246327

R2 = 0.9474

y = 5E-13x2 - 0.0011x + 676716

R2

= 0.9532

0

500

1000

1500

2000

2500

117...

117...

118...

118...

119...

119...

120...

120...

121...

121...

122...

122...

Time Since Epoch

Nu

mb

er

of

Loca

tion

sU

sed

Locations Used (Old Firewall) Locations Used (New Firewall)

Poly. (Locations Used (Old Firewall)) Poly. (Locations Used (New Firewall))

Figure 12a: Number of locations used over time.

y = 1E-13x2 - 0.0002x + 149075

R2

= 0.9661

y = 1E-13x2 - 0.0002x + 134606

R2 = 0.9481

0

100

200

300

400

500

600

117...

117...

118...

118...

119...

119...

120...

120...

121...

121...

122...

122...

Time Since Epoch

Nu

mb

er

of

Ru

les

Total Rules (Old) Total Rules (New) Poly. (Total Rules (Old)) Poly. (Total Rules (New))

Figure 12b: Number of rules over time.

Trends Over Time

The following data is drawn from an ongoingfirewall migration project which has been in progressfor nearly two years, and continues onward with an

extended period of overlapping operation for the oldand new firewalls.

When the configuration was initially migratedfrom the old firewall to the new firewall, a manualcleanup of the rule set took place; currently most ruleset changes are expected to be implemented on boththe old and new firewalls.

Unsurprisingly, we see that both the old and newrule sets show a steady increase in the number of rulesand locations defined over time

Similarly, we also see the number of partiallyeffective rules increasing over time, as the number ofrules in the rule set increases.

The number of partially effective rules in thenew firewall rule set is initially lower than the number



of partially effective rules in the old firewall rule set,thanks to a manual clean up of the rule set, prior tomigration. It is abundantly clear, however, that theeffect of the rule set clean up was only temporary.

y = 4E-14x2 - 1E-04x + 57054

R2 = 0.9158

y = -2E-15x2 + 6E-06x - 4260.5

R2 = 0.9464

185

190

195

200

205

210

215

220

1170000

000

1175000

000

1180000

000

1185000

000

1190000

000

1195000

000

1200000

000

1205000

000

1210000

000

1215000

000

1220000

000

1225000

000

Number of Rules

Nu

mb

er

of

Pa

rtia

lly

Eff

ecti

ve

Ru

les

Number of Partially Effective Rules (Old) Number of Partially Effective Rules (New)

Poly. (Number of Partially Effective Rules (New)) Poly. (Number of Partially Effective Rules (Old))

Figure 13a: Trends over time – Rule interference.

Time Since Epoch

Nu

mb

er

of

Ru

les

0-75 (Old) 75-100 (Old) 100 (Old) 0-75 (New)

75-100 (New) 100 (New) Poly. (100 (New)) Poly. (100 (Old))

Poly. (75-100 (New)) Poly. (75-100 (Old)) Linear (0-75 (New)) Linear (0-75 (Old))

Figure 13b: Trends over time – Distribution of paritally effective rules.

Further, if we examine [FIGURE: Trends OverTime – Distribution of Partially Effective Rules], it isclear that the majority of rule changes which result inpartially effective rules are for rules in the 75-100%effectiveness range.

Discoveries

Our findings in this work range from confirmingrelatively obvious intuitions (such as simpler configu-rations are better) to some more surprising results. Ourdiscoveries here summarize the case studies, as well asadditional experiences not described in this paper.

1. The number of rules in a rule set is highly cor-related to the number of partially and com-pletely effective rules. This suggests that ourintuition that larger rule sets are more complexis likely to be correct.

2. There is a high correlation between the numberof interfering rules, and the number of interfer-ing rules with conflicting actions. While it islikely that some of these interfering rules havebeen knowingly configured (e.g., a permit rulefor a host in a larger subnet which is denied), inpractice it appears that most interfering rulesare unintended.

3. The majority of partially effective rules are25-99% effective. Further, when a rule setundergoes a manual clean up process, the



number of completely ineffective rules is typi-cally reduced dramatically, while the number ofpartially effective rules remains relativelyunchanged. This suggests that complex interfer-ence patterns are more difficult for people todetect and resolve.

4. The effectiveness of a rule set decays visiblyover time. If a rule set is cleaned up, the effec-tiveness immediately begins to decay again.

5. Most firewall rule sets have some amount ofinterference, but the amount of interferencevaries dramatically, and increases as number ofrules increases.

6. While effectiveness varies dramatically, in gen-eral, more effective rule sets have less variancein effectiveness, suggesting that less confusingrule sets are easier to manage.

7. In general, in the absence of a major clean upeffort, the number of rules and objects used in agiven rule set always increases. Since the num-ber of partially and completely effective rulesalso increases as the number of rules increase,this points to the management of firewall rulesets as write-once, remove-never devices.

8. Even when a rule set is cleaned up, the numberof rules decreases, but the number of objectsremains constant or increases, suggesting thatobject management is an ongoing challenge.

9. There is no clear relationship between the num-ber of rules and number of locations.

Unfortunately, it appears that objects are a weaksource for information about the state of rule sets, atleast as we are currently measuring and examiningobjects. It is likely that the use of shared objects by allfirewalls managed by a given Checkpoint manage-ment instance is the cause of this issue, and we hopeto better address the role of objects as a means of mea-suring firewall rule sets in the future.

Ultimately, our holy grail is to be able to corre-late the above metrics with quantities dear to higherlevel management such as cost, ‘ease of management’etc. We need to measure these ‘manager-friendly’ met-rics to see what configuration and network metricscorrelate with them – this is a part of our ongoingefforts.

Usage

The tool and metrics we describe are extremelyuseful for rule set comparison and clean up, as theyautomate the process of finding conflicts, vastlyreducing the amount of time and effort required.

As well as being used for massive rule set reme-diation projects, we suggest that an ideal usage wouldbe to include rule set analysis as a part of routinechange management, to identify unanticipated effectsfrom rule and objects additions and removals.

We also hope that our metrics work will serve asa measure of health for firewalls – and can drive

decisions like ‘‘when should I clean up my rule set?’’or ‘‘Is my outsourced firewall administrator doingtheir job properly?’’

Future Work

Going forward, we intend to improve our under-standing of firewall management. Specific areas ofinterest include:

• Analyzing configurations from a wider varietyof firewalls to discover commonalities and con-sider additional metrics.

• Tracking the effect of using the metrics wehave described to improve firewall manage-ment over time.

• Correlating the metrics we have described tothe rate of change, number of incidents, andtime to resolve incidents in the environment.

• Correlating the metrics we have described tocost.

Related Work

There has been a flurry of work on firewall anal-ysis in recent years. The most relevant is the FirewallAnalyzer (FA) product from Algorithmic Security Inc[ALG] that builds on previous research reported in[MWZ] and [WOO]. FA analyzes a number of differ-ent types of firewalls, reports common vulnerabilitiesfound in the rule set, and detects rules that are redun-dant (i.e., are eclipsed by higher-priority rules). FAdoes not check equivalence of rule sets.

The Solsoft Firewall Manager [SOL] convertshigh-level specifications of allowed and prohibitedtraffic into firewall rule sets. However, it does not ana-lyze existing rule sets on a firewall to see if they com-ply with the high-level specifications.

There are also a number of academic papers onfirewall analysis. The paper [AH1] recognizes differ-ent types of conflicts that can occur between a pair ofrules, but does not address the more general problemof one rule being eclipsed by a set of rules. Theauthors extend their study to rules on multiple fire-walls in a tree network [AHB], but the results areagain limited to interactions between pairs of rules.The paper [GLI] proposes a simplistic way to designrule sets such that every packet is associated withexactly one rule; the problem with this approach isthat one can design much more compact rule sets ifmultiple rules (resolved by a priority mechanism) canapply to any packet. The paper [EMU] investigatesefficient data structures to process basic firewallqueries.

The paper [VPR] addresses problems of generat-ing and analyzing rule sets for networks with multiplefirewalls and addresses the problem of checkingequivalence of rule sets. However, the paper relies onexhaustive analysis of all possible packets, and it isunclear whether the methods scale to large networks.



The paper [BRR] describes a tool to configure andanalyze rule sets for networks with multiple firewalls.It employs efficient algorithms that scale for large net-works, and while the techniques are relevant forchecking rule set equivalence, the paper does notexplicitly address the equivalence problem.

The paper [MKE] proposes the use of binarydecision diagrams to analyze rule sets. These diagramsallow you to query the firewall rule set but do not eas-ily support the comparison of two rule sets.

Conclusions

Offline configuration analysis is a fast, light-weight, and reliable way to identify the effectivenessof existing firewall configurations, improve maintain-ability, hasten debugging, and assist with policy, auditand compliance.

Author Biographies

Sandeep Bhatt is a researcher in the SystemsSecurity Laboratory at HP Labs. Since joining HPLabs in 2004, he has focused on the management ofdistributed access control in enterprise applications, onalgorithms for firewall analysis, and on end-to-endaccess control in enterprise networks. Before joiningHP Labs, Sandeep led the Systems Performance teamat Akamai Technologies, and was Director of NetworkAlgorithms at Telcordia Technologies where he led thedevelopment of fault diagnosis systems for large-scaletelecommunication networks. Sandeep was also on thefaculty of Computer Science at Yale University, withappointments at Caltech and Rutgers University. Hisresearch interests have included algorithms for parallelcomputation, network communication, and VLSI lay-out. He received Ph.D, S.M and S.B degrees in Com-puter Science from MIT.

Cat Okita has more than 10 years of experienceas a senior systems, security and network professionalin the Financial, Internet, Manufacturing and Telecomsectors. She has designed, managed and contributed toa variety of geographically diverse projects and instal-lations. Her assignments have included proposing andseeing to completion a wide variety of security, Inter-net, enterprise and monitoring projects, includinghighly available, redundant fault tolerant systems forsecurity, web services, logging, monitoring, statisticsgathering and analysis. Her research interests includeprivacy, reputation and practical security.

Cat has spoken at LISA and Defcon about iden-tity and reputation, co-chairs the ‘Managing Sysad-mins’ workshop at LISA, and programs for fun, in herspare time.

Prasad Rao is a member of the Systems SecurityLaboratories within HP labs, where he works in secu-rity analytics and metrics. At HP he has been active inthe Vantage project that analyzes end-to-end accesscontrol across various layers in the enterprise. As a

part of this project, he has built technologies to ana-lyze firewalls with very large rule sets (more than7500 rules) and objects (more than 12000 objects) forpresence of operator errors and vulnerabilities – thistechnology is currently being tested within HP. Inaddition he has contributed to ongoing work in otherareas such as role discovery, securing printing soft-ware and privacy compliance.

At Telcordia Technologies he was the technicallead and architect of the Smart Firewalls project (apart of the Dynamic Coalitions Program). Addition-ally, at Telcordia technologies he built a deductiverule-based system in large scale worker schedulingprograms, which had to optimize worker utilitizationand cost under arbitrarily stated union rules and practi-cal constraints.

For his Ph.D. thesis in logic programming anddeductive databases at the State University of NewYork at Stony Brook, Dr Rao built designed andimplemented the tabling engine of the XSB logic pro-gramming language – which set speed records forstandard benchmarks in the field of deductive data-bases compared to the state of the art in the early ’90s.

Bibliography

[ALG] Algorithmic Security Inc., Firewall Analyzer:Make Your Firewall Really Safe, white paper,2006, http://www.algosec.com .

[AH1] Al-Shaer, E. and H. Hamed, ‘‘Firewall PolicyAdvisor for Anomaly Discovery and Rule Edit-ing,’’ Proceedings IEEE/IFIP Integrated Man-agement Conference, 2003.

[AHB] Al-Shaer, E., H. Hamed, R. Boutaba, and M.Hasan, ‘‘Conflict Classification and Analysis ofDistributed Firewall Policies,’’ JSAC, 2005.

[BRR] Bhatt, S., S. Rajagopalan, P. Rao, ‘‘AutomaticManagement of Network Security Policy,’’ Pro-ceedings MILCOM, 2003.

[EMU] Eppstein, D. and S. Muthukrishnan, ‘‘InternetPacket Filter Management and Rectangle Geome-try,’’ Proceedings ACM SODA, 2001.

[GLI] Gouda, M. and X-Y. Liu, ‘‘Firewall Design:Consistency, Completeness and Compactness,’’Proceedings IEEE International Conference onDistributed Computing Systems, 2004.

[MKE] Marmorstein, R. and P. Kearns, ‘‘An OpenSource Solution for Testing NAT’d and Nestediptables Firewalls,’’ Proceedings LISA ’05, 2005.

[MWZ] Mayer, A., A. Wool and E. Ziskind, ‘‘Fang: AFirewall Analysis Engine,’’ Proceedings IEEESymposium on Security and Privacy, 2000.

[SOL] Solsoft Inc., http://www.solsoft.com .

[VPR] Verma, P. and A. Prakash, ‘‘FACE: A FirewallAnalysis and Configuration Engine,’’ IEEE Sym-posium on Applications and the Internet, 2005.

[WOO] Wool, A., ‘‘Architecting the Lumeta FirewallAnalyzer,’’ Proceedings 10th USENIX SecuritySymposium, 2001.



Appendix 1

To briefly demonstrate the tool, consider the following first match firewall rule set, which contains severalerrors:

Action Service Source DestinationPermit SSH 10.0.0.0/8 10.0.0.0/8Deny SSH 10.0.0.0/8 10.3.1.0/24Permit SSH,https 10.0.0.0/8 10.3.1.61/32Deny ANY ANY ANY

The tool produces the following overall summary:

From the Summary, we can then obtain more specific information about the rule set...



... and we can drill down for detail about the overlapping rules:

... and finally procure further details about the specific overlaps:


Documents

Fast, Cheap and In Control: A Step Towards Pain Free Security! · 2018-09-13 · Fast, Cheap and In Control: A Step Towards Pain Free Security! Sandeep Bhatt, Cat Okita, and Prasad