UMBC Resnet IDS and Tipping Point IPS

Mark Cather

Office of Information Technology / UMBC

UMBC Campus Network

The UMBC Resnet IDS Implementation

Layout of UMBC Residential Network

Possible Methods for Detecting Infected Resnet Machines

• Scan Machines with a Nessus-type scanner as users login.– Windows SP2 blocks your ability to scan.– Linksys and other NAT routers can block or confuse your scan.– Scanning won’t detect an active virus on the inside of a machine.– Users may get a firewall alert message on their screen as you scan their

system. – Many of the open-source scanning solutions have trouble scaling to a

high rate of logins.

• Install a Host-Side Agent (like Cisco Perfigo SmartEnforcer)– Agent can cause some machines to become unstable.– Agents are not available for all operating systems.– NAT routers will block your ability to connect to the agent.– How do you handle devices like Xbox’s and Playstation2’s?

Possible Methods for Detecting Infected Resnet Machines (cont)

• Use an Intrusion Detection System to Watch Resnet Traffic for Signs of Infected Machines.– Pros

• It can look for infections from any type of device.• It doesn’t trigger the user’s firewall notifications.• It doesn’t require the user to install anything on their computer.• It doesn’t matter if the user has a NAT router.• It doesn’t matter how many devices the user may have behind their

NAT router.– Cons

• It can be difficult to scale.• Developing and maintaining an IDS signature database is difficult

and time-consuming.

In the End, UMBC Chose to Implement an IDS Solution.

• Our policies don’t allow us to restrict users from installing NAT routers in the Dorm rooms.

• Our users are allowed to use whatever operating systems they like.

• While we felt that scaling an IDS solution to manage our traffic would be difficult, we expected that we would sooner or later catch most of our infected users.

• We feel that this solution offers us greater flexibility in identifying security issues.– We can lockout a user for any defined IDS rule.

UMBC Residential Network Authentication

• The authentication system is a major component of any lockout system.– Without a robust authentication system, you can not lockout a “User”.

You can only block their machine (by IP or hardware address).

– Many of the options for blocking a machine would cut off all network access to the infected machine.

• User’s then have to call a Helpdesk to report that their network jack is dead. • If a widespread virus gets loose, you could have hundreds of users calling

your helpdesk.

– Some methods of blocking an infected machine involve a browser redirection mechanism.

• If users are regularly made to open web browsers on all their network jacks, this method could work.

UMBC Resnet Authentication System

• All users must go through a web-based login system before their network jack is enabled.

• During the login process, the users are required to agree the terms of our Acceptable Use Policy.

• They also must agree to take full responsibility for any traffic that comes from their network jack. – This has saved us a lot of trouble with copyright complaints and

NAT abuse cases.

The Nuts and Bolts of the System

• All of our network jacks are assigned a separate /30 subnet. • This method offers many benefits:

– Rogue DHCP servers are not a problem.

– Users can’t hardcode another user’s IP address and cause an address conflict.

– The users only have one useable IP address per jack. This limits the ability for another person to use another user’s jack.

– We have a router boundary on every jack. This means we can apply ACLs to every network jack in our Resnet.

• The problem is that we end up using 4 addresses for every jack. Since we have about 4200 residents on campus, we are now tying up more than 16000 IP addresses for our Resnet.– This year we had to begin using NAT at our Resnet Border.

Resnet Authentication Process

Where Does the IDS Connect?

IDS Lockout Implementation

• We use Snort for our Resnet IDS.• The Resnet IDS has a set of rules that are as free of

false positives as we can determine.• It watches as much traffic as it can keep up with. If it

misses something, we figure we will catch them the next time.– Currently we are keeping up with ~60% of our total traffic.

• Users can be locked out for multiple reasons at the same time.

What does the User See?

• When they are locked out, the users abruptly lose their network connections.

• Our users are accustomed to going to our login page whenever they lose access.

• When a locked out user tries to log back in again, they are given a page that indicates that they are locked out and the reasons why they are locked out.

• Each of the reasons listed is a link that further explains the lockout reason and how to fix the problem.

• Once a user thinks they have fixed their problem, they can release their own lockout. (for their first two incidences).

• If a user is locked out for a third time (per year), the user is not allowed to release their own lockout. The user must schedule with OIT to come out to their room and verify that they have successfully cleaned their machine.

What Benefits have We Seen from the Resnet IDS Implementation?

• It allows us to deal with each individual virus or worm; not every individual infected user.– If we can isolate a signature for a security problem, we can deal

with everyone who has that problem at one time.– The number of users who are infected with a particular problem

does not matter as much.

• It allows the users to fix their own problems (without having to call OIT for assistance).– This speeds up the response time and makes our users happier.

• The same lockout mechanism can be used for administrative security issues.– Copyright Violations and AUP Violations.– Three-Strike rule doesn’t apply to administrative sanctions.

Keeping Snort Up-To-Date

• Keeping Snort up to date is the biggest problem.• There are a couple of mechanisms that we use to obtain

Snort rules:– Self-written

• As we find infected machines, we capture the virus executables. • We usually find the infected machines through outside complaints or

our other IDS / IPS solutions.• The captured virus executables are analyzed to characterize their

activity.• If we need to, we will do some reverse engineering on the virus

executable to see what makes it tick.

– Outside Signature Sources• Bleeding Snort (http://www.bleedingsnort.com/)• Snort.org Signature Tarballs and Mailing Lists.

UMBC and TippingPoint

Which IDS / IPS?

• During the Summer of 2004, we began the process of finding and procuring a commercial IDS / IPS solution for the campus.

• Up until this point, we had been using Snort as our only IDS solution.– Snort did not scale well. We only analyzed a small percentage of our

outbound WAN traffic or academic / administrative building traffic.

– We could never keep the signatures up-to-date.• There was typically a 1 - 4 week delay between the release of a piece of

malware and the release of a good signature.

– We had no visibility to the interior networks on campus.

– Our Snort system was not tied into anything to proactively block attacks.

– Our incident response group spent their time running from hacked machine to hacked machine. They were not able to keep up with the newly released malware and the previously hacked machines.

Requirements for a Commercial Solution

• Based on the issues we had with our previous IDS, we came up with the following requirements for our IDS evaluation:– Staff Augmentation - The new IDS / IPS had to be automated enough that

we wouldn’t have to spend very much time updating it’s signatures or software.

– Signatures - Someone else should come up with the needed signatures in a timely manner (1 - 2 weeks max). The Signatures should have a very low false positive rate.

– Proactive Action - The new IDS / IPS had to have a component that would automatically block common and obvious attacks.

– Capacity - The new IDS / IPS has to have a minimum capacity of 1Gbps.– Expandability - The new hardware should be able to integrate multiple

sensing and blocking components under one management system.– Reporting - The new hardware should have a reporting feature that easily

identifies infected hosts. – Procurement - Anything we purchased had to be through a state contract.

We did not want to go the the RFP process.

The TippingPoint UnityOne System has Met All our Needs

Staff Augmentation

• Before the TippingPoint, we would spend all our time on Incident Response.– There were new variants of worms and viruses released every week.– By the time we obtained a signature for a new piece of malware three

more pieces of malware had been released. – The malware had a good 2 - 4 week lead on us obtaining signatures. – Updating and maintaining the Snort server and Snort itself also took

large amounts of time.

• Now, We don’t need to spend any time at all managing the TippingPoint itself.– The TippingPoint system automatically checks and updates it’s

signature files daily. As new firmware for the hardware becomes available, we receive email telling us that an update is available. We can then update the hardware whenever our downtime schedule will allow.

Signatures

• As a part of our service contract, TippingPoint maintains the signature lists. Our system automatically goes out and checks for a new signature file daily. Unless a new security threat is announced, TippingPoint typically come out with new signature sets once a week.

• We have only had one problem with false positives.• TippingPoint writes it’s signatures to detect the abuse of a

vulnerability, not a particular piece of malware.– As an example, a buffer overflow attack on the MS-RPC

interface of a machine could be used by many different pieces of malware (both new and old). TippingPoint has one rule that triggers on buffer overflow attempts, no matter what the source.

Signatures (cont)

• Signatures are grouped into three classifications:– Application Protection

• Worms and Viruses• Buffer Overflow Detection• Spyware• Scan, Sweeps, and Probes

– Infrastructure Protection• DDOS Activity• Network Equipment Attacks• Traffic Abnormalities

– Malformed Packet Headers– TCP flags that make no sense.

– Performance Protection• Peer to Peer File Sharing

Proactive Action

• In the daily signature updates, TippingPoint indicates whether or not a signature is recommended.

• From our experience, signatures that are recommended have an extremely low false positive rate (we’ve never had a false positive on a recommended signature).

• Recommended signatures also are typically related to security threats, not new features or policy based rules.

• We are configured to Block and Notify any traffic that matches a “Recommended” signature.

Capacity• TippingPoint’s hardware comes in a couple of different sizes:

– 50Mbps, 100Mbps, 200Mbps, 400Mbps, 1.2Gbps, and 2Gbps.– A 5Gbps product will be available in mid-March.

• The TippingPoint UnityOne systems monitor and protect in both directions at the same time.

– They detect and protect your users from the outside and the outside from your users.

• The hardware with a capacity of 200 Mbps or more is broken up into multiple segments.– The 200Mbps product has 2 segments.– The 400Mbps - 2Gbps products have 4 segments.– Segments can be a mix of UTP and Fiber, but they must be ordered

from the factory in the desired configuration. The configuration can not be changed later.

– The segments are isolated from each other. No traffic can pass from one segment to another.

• The amount of traffic that a unit can handle is the total across all it’s segments, in both directions.

TippingPoint UnityOne Appliance

Expandability• The product line consists of two parts:

– The UnityOne IPS Appliance• This is the unit that comes in sizes from 50Mbps - 2Gbps.• Depending on capacity, a single appliance can have as many as four

segments. • In the case of UMBC, we have purchased a 1.2Gbps Appliance. The

appliance is in between the following network areas and the campus core:– Residential Networking– Wireless Networking– Remote Access Networking– Academic and Administrative Building Networking

– The UnityOne Security Management Server• This appliance manages one or more IPS appliances (up to ~25 IPS

appliances).• The SMS is what manages the signature files and firmware updates.• All the log messages from all the IPS appliances are collected together on

the SMS. The SMS produces incident reports and allows you to filter / search through all the log messages.

Reporting

• All of the log messages for all the UnityOne IPS appliances are viewable through the SMS.

• You can search or sort any way you would like.• Columns in the display can be aggregated to show the number of

incidences per rule, the number of rules broken per source or destination or any other value you would like.

• The client for the SMS system is Java based and works on Windows, Macintosh, Sun and Linux systems.

• The only client version that is fully supported is the Windows version.

• Canned reports are also available. • Custom reports can be written as needed.

Procurement

• Dell resells the whole TippingPoint product line through their contract.

• When we purchased our hardware, last July, the list pricing was as follows:– 50Mbps - $9,995– 200Mbps - $24,995– 400Mbps - $42,995– 1.2Gbps - $64,995– 2Gbps - $89,995– Security Management Server - $9,995

Threat Suppression Engine

• The Threat Suppression Engine is what makes this product perform.

• The pattern matching rules for all blocking and traffic shaping signatures are downloaded into hardware ASICs.

• This is how they can pump through 2Gbps, at full line rate, and the number of rules in service won’t impact performance.

High Availability

• The UnityOne products are not active layer-2 or layer-3 devices. They say that they are a “bump in the wire”.

• Layer-2 Fallback– In the case of the major software issue or unit overload, the unit

directly forwards traffic across the segments at layer-2.

• Zero Power High Availability– In the event of a complete power loss, the copper interfaces can

be setup to directly bridge within each segment.

• Two units can be installed in parallel.– The two devices can be setup to pass state information.– Since the devices don’t interact with spanning tree of routing

protocols, any standard network redundancy solutions will continue to work as they do now.

• All units have hot-swappable dual power supplies.

We Have Had Problems…• Blue Status Indicator

– The TippingPoint products do some of their own checks to see if something is a false positive.

– There is a bug in the version of firmware, that we are running, that incorrectly logs a caught false-positive as a memory error.

• Load Issue– When we first received our hardware, we set it to permit/notify us

on all rules.– Since permit/notify traffic needs to go through the main process

(rather than the high-speed ASICs), the processor overloaded.

Problems (cont)

• Novell Outage - – Rule 7120 blocks TCP traffic that is not RFC compliant.– Novell traffic is not compliant with RFC standards.– They reuse sequence numbers.

• Mail messages with Viruses will be detected.– When the mail message with the virus is detected, the TCP

session for that message download will be cutoff.– In some cases, mail clients download all their messages over

one connection.– When that connection gets cut off, the user’s mail client will

hang.

Future Directions

• In the next 12 - 18 months, I would like to tie the output of the TippingPoint into our Resnet Lockout system.

• We need to find a good way to setup a resnet-style lockout and remediation system in our academic/administrative building network segment.– This could also be triggered by the log output of the

TippingPoint.

Thank You

Mark Cather

Assistant Director of Networks and Security

Office of Information Technology / UMBC

mark.cather@umbc.edu