NIPS and HIPS

To the general public, an article called "NIPS and HIPS" might sound like a discussion about intrusive plastic surgery. For security administrators, though, "NIPS and HIPS" should sound like a dream come true: preventive remedies for fending off a long laundry list of network attacks.

NIPS and HIPS are two types of Intrusion Prevention Systems (IPSs). Some security administrators believe IPS is just a marketing term that lets vendors promote Intrusion Detection Systems (IDSs) in a new way. Other people are less skeptical and see IPS as the next evolutionary step in network protection devices. These opinions are commonly based on the various definitions of IPS. Even the IPS vendors can't agree on a standardized definition or technology model. However, this technology is new. Only time will tell if the market will embrace it.

The most commonly agreed-on definition is that an IPS is an inline device that is a combination of an IDS and application-layer firewall. Most organizations don't use firewalls that work at the application layer of the network stack because of the performance hit that occurs with having to use so much processing power to dig through all of the components of each and every packet to try to identify something malicious. Today's firewalls mainly make their access decisions based on the network and transport layers of a packet, which misses many of the crucial portions that can be carrying malicious payloads.

As Figure 1 shows, firewalls use access criteria based mainly on IP addresses, port numbers, and a limited amount of information pertaining to the protocols the packets are using. IDSs evaluate the traffic but can't stop the traffic from entering the network. IPS evaluates traffic at a deeper level than most firewalls before it allows the traffic in through a port—the best of both worlds. However, current IPS products are constrained by a fundamental limitation: They can block only the traffic they see. Nowadays, more and more environments are switched—and if IPS is to monitor all the individual communication channels through the switching fabric, many IPS devices are needed, which is cost prohibitive. This means that an inline IPS can't cover the entire network until network infrastructure vendors are able to cost-effectively replace conventional switches with combination switch-IPS products. Some infrastructure companies are starting to build security intelligence into their network devices and protocols, which will provide a more holistic and integrated approach to security, but it'll take them a few years to get to that point. So, let's take a look at what you can do today with the various IPS products available.

Just as there are network IDS (NIDS) and host IDS (HIDS) solutions, there are network IPS (NIPS) and host IPS (HIPS) solutions. NIPS solutions evaluate traffic before it's allowed into a network or subnet. HIPS solutions evaluate packets before they're allowed to enter a computer.

Besides the NIPS and HIPS differentiation, IPSs can be differentiated by the type of product. IPS functionality can be

• packaged as a dedicated appliance. Dedicated IPS appliances are standalone products. They're usually inline NIPS devices, which means all traffic must pass through them to gain access to the network.
• integrated into other products. Some vendors have started integrating IPS functionality into their existing security products. For example, firewall vendors Check Point Software Technologies and Juniper Networks have integrated IPS functionality into their Fire-Wall-1 and NetScreen-5GT firewalls, respectively.

Because most organizations already have firewalls and are looking to supplement rather than replace them, let's concentrate on dedicated NIPS appliances and HIPS solutions.

Dedicated NIPS Appliances
Dedicated NIPS appliances have no MAC or IP address, so hackers can't attack them directly. The appliances use either rate-based functionality or content-based functionality.

Rate based. Rate-based IPS appliances use thresholds that detect when there are too many connections, errors, or packets coming into the network. The way in which NIPS appliance vendors address rate-based protection differs between products. However, all NIPS appliances let administrators define the computers, ports, and applications that need to be protected. Source and destination IP addresses and port numbers are used so that a certain baseline of traffic can be set for each computer and each service that the computer is providing. Administrators typically use wildcard values for the source IP addresses and port numbers because it's impossible to know about all the systems that are going to initiate contact.

Some NIPS appliances let administrators set the rate baselines by using quantitative bandwidth values. Other appliances use qualitative values, such as high, medium, or low. The Captus Networks' Captus IPS 4000 series can identify when a specific service is being overwhelmed and can start throttling the traffic. If the same amount of traffic continues, the product can disconnect access to the service from that client.

Besides letting you set a threshold for the appropriate level of traffic allowed into a particular system, Top Layer Networks' Attack Mitigator IPS 5500 provides an interesting functionality called connection proxying that other NIPS products don't appear to have. Connection proxying lets you limit the number of incomplete TCP connections. An incomplete TCP connection usually indicates malicious activity, such as a SYN flood attack. Once the threshold for the number of allowed incomplete TCP connections is reached, Attack Mitigator works as a proxy between the sender and receiver. The connection packets will still come to Attack Mitigator, but it allows the traffic to continue to the destined computer only if the TCP connect completes (SYN, SYN/ACK, ACK packet sequence).

Depending on how you configure dedicated NIPS appliances, once the threshold for the appropriate level of traffic has been exceeded, they can throttle the traffic, drop packets entirely, reroute traffic, or send an alert to an administrator. Thus, you have an effective countermeasure for Denial of Service (DoS) attacks. The trick is to know how to properly calibrate a rate-based IPS so that the right level of protection is being provided and still not drop or reroute legitimate traffic. For example, what would be the acceptable number of packets that should be going to your DNS server, Web servers, and mail servers? An administrator would have to learn the correct thresholds through on-the-job training and continually tweak the IPS appliance until just the right amount of traffic is allowed to the different destinations.

Customers who implement rate-based NIPS appliances often complain that it's hard to determine the correct thresholds for host-to-host traffic and even port-to-port traffic. A few vendors provide the methodologies and tools necessary to monitor traffic so that you can set the correct thresholds, but this is rare. More often, vendors provide overly complex steps or send an engineer to create the baselines, which later might have to be recalibrated as the environment changes or traffic loads change.

Content based. Content-based NIPS appliances look for anomalous behavior and protocol anomalies to detect malicious payloads. Traditional IDSs work the same way, so content-based NIPS appliances don't represent a tremendous leap in ingenuity.

Content-based NIPS appliances look for anomalous behavior, such as FTP traffic going toward port 53, binary code within a user password, or an excessive number of bytes coming from a Web browser. In addition, they use signatures to look for protocol anomalies to identify packets that aren't compliant to specific protocol Request for Comments (RFCs). This has caused many false positives because many vendors don't choose to follow the protocol RFCs completely. Content-based IPS appliances also look for specific malicious protocol anomalies. For example, here are some potentially malicious modifications to the network and transport headers of a packet:

• incorrect length of header or field
• corrupt checksums
• incorrect TCP segmentation overlaps
• inconsistent use of flags within header fields

Some content-based IPS appliances dig deeper into the packet to look for potentially malicious activity within the application-layer headers. Application-layer protocol anomalies that can be detected include

• illegal protocol command usage
• unusually long or short field lengths, which might indicate a buffer overflow
• using a protocol for unusual purposes
• mapping a protocol to an unusual port number
• incorrect field values and combinations

Many content-based NIPS appliances are shipped with the same type of signature database you would find in IDSs. Because one of the biggest problems with IDSs is false positives, some IPS vendors have only about a fourth of their signature rules enabled out of the box. You can enable the other signatures as you see fit. As with rate-based NIPS appliances, content-based NIPS appliances can be configured to drop packets, reset connections, and even create a short-lived blacklist of IP addresses that are sending malicious traffic.

Some IPS products attempt to combine content-and rate-based features. However, it's difficult for one product to do both well at the same time because of the amount of resources required for these different types of packet inspections.

HIPS
Unlike HIDS solutions, which tell you only that a suspicious event took place, HIPS solutions attempt to stop the suspicious activity from happening in the first place. Like NIPS appliances, HIPS solutions can use signature-or behavioral-based approaches. For example, suppose an attacker wants to carry out a buffer overflow so that his malicious code can run in the memory space of the kernel. To prevent this type of activity, the HIPS solution will review the system call and compare it to either a list of signatures or a list of known good behaviors. If the HIPS solution identifies the call as malicious, it doesn't allow access. Vendors can use one or both approaches in their products. For example, McAfee's Entercept uses signature-and behavioral-based methods, whereas Cisco System's Cisco Security Agent (formerly known as Okena StormWatch technology) uses a purely behavioral-based approach.

Although the various HIPS solutions might use different approaches, most of them employ agents, which are centrally managed, on the systems needing protection. The agents examine system and API calls to identify when an attack is being attempted. The agent must understand the security context in which the process is running, the command requests being sent to the interface, and the resource that the process is attempting to access. When a call comes in, signature-based HIPS solutions check what is usually a long list of illegal call patterns that have been identified with certain types of attacks. If the incoming call contains one of the identified patterns, they don't allow access. Behavioral-based HIPS solutions usually have specific modules for individual system-service APIs. For example, there might be a module that reviews requests between processes and the file system, a module that reviews network stack requests, a module that monitors registry requests, and so on. There are also modules for commonly used services and applications, such as DNS, DHCP, and Microsoft SQL Server.

HIPS solutions can protect against many types of attacks. For example, they can

• prevent access to email clients' contact lists so that viruses and worms can't be spread through this means
• prevent privilege escalation exploits in which a user account tries to obtain administrative or root access
• prevent the loading of root kits, backdoors, and Trojan horses
• prevent the alteration of system files, registry settings, and user accounts
• prevent buffer overflow exploits

What You Need to Know
Now that you know about the different types of IPSs, you can determine which type is best for your network. First, you need to figure out what type of protection your organization requires. Does it just need perimeter protection to identify any malicious traffic that has passed through the firewall? If so, you need to obtain a dedicated NIPS appliance and place it behind your firewall, most likely within your demilitarized zone (DMZ). Does your organization need to look for malicious activity within the network? If so, you need to deploy several NIPS appliances or purchase network products that have this functionality integrated into them. Do you need individual system protection? If so, you need to investigate the HIPS solutions currently on the market. Do you mainly need protection from DoS attacks (in which case you need rate-based IPS), other types of attacks (in which case you need content-based IPS), or both?

When you start talking with vendors about an IPS product, get answers to the following applicable questions:

• If it's an inline product, does it fail open or closed? If the product fails closed, you run the risk of blocking all network traffic at that point, so there needs to be a redundant component to ensure that this doesn't take place. If the product fails open, all traffic will be entering the network without being properly analyzed.
• If it's an inline product, what type of redundancy is built into it?
• Where in the network is the product designed to reside?
• What are the product's performance metrics (e.g., throughput, latency)?
• If it's a rate-based product, how difficult is it to set and maintain rate baselines?
• If it's a content-based product, what types of attacks does it look for?
• If it's a content-based product, how large is the signature base and how many signatures are enabled? (More enabled signatures can equate to more false negatives.)
• To what degree can the product be customized for your specific environment?
• What type of traffic does the product block (e.g., protocol anomalies, fragmentation attacks, buffer overflows, spoofing attacks)?
• Does the product monitor traffic coming into and going out of the network?
• How will the product work with your existing firewall and what are the proper configurations for both the product and the firewall?
• Can the product be centrally managed and configured?
• What are the product's logging and alerting capabilities?

These are good questions to start with, but you'll probably want to drill down further to ensure that you purchase the IPS product that meets your company's needs. Before purchasing the IPS product, check to see whether any reviews or comparative analyses have been done. In addition, get feedback from companies that have implemented the product.

An Immature But Promising Technology
There's no doubt that IPS is an immature technology. However, the idea behind the technology—that is, the idea of stopping all types of malicious attacks before they enter the network—seems to be a sound one. Equally important, the security industry seems to be heading in the right direction by starting to develop IPS products that provide a more holistic and integrated approach to security.