What to Do When Your Network Is Attacked

It's finally Friday. You're looking forward to a long 3-day weekend at that exclusive resort you've been wanting to try. Just before you leave the office, a user complains that the network is running slowly, particularly when he tries to access the Internet. You check it out and discover that the server and network are indeed incredibly slow. You check your firewall's traffic statistics and notice unusually heavy Internet traffic. You run the Netstat command on the server and find several unauthorized connections to your server that look like they're coming from the Internet. You check the server's registry and notice several unfamiliar programs set to automatically load. Cancel your plans; you have a long weekend of work ahead. You've just been hacked.

Depending on the nature of the attack, it's not always easy to determine that you've been hacked. Knowing where to look and what to look for can help you discover hacks and take remedial action before they cause additional damage. I'll show you where to begin looking for malicious programs that can wreak havoc on your systems, and I'll help you develop a hacking recovery plan. I'll wrap up with three case studies that show how I used these tactics to help organizations detect and recover from a network hack and prevent future attacks.

Where to Look
Obviously, you need to find a hack before you can take measures to stop the attack and recover from it. Where do you begin? Every hack is unique, but you should always check certain places first. Here are the key locations in which to start your search.

Registry subkeys. If you suspect that a particular machine has been hacked, check the Run subkeys in that machine's registry first. Look for any unfamiliar programs that load from these subkeys. Not only do attackers favor the Run subkeys as a launching point for rogue programs, but intruders can launch viruses from those subkeys as well. The subkeys apply to Windows Server 2003, Windows XP, Windows 2000, Windows NT, Windows Me, and Windows 9x. The specific subkeys to check are:

If you're running Windows 2003, XP, Win2K, or NT systems, you also need to check the HKEY_LOCAL_MACHINESOFTWAREMicrosoft
WindowsCurrentVersionPoliciesExplorerRun subkey.

Any program that you don't recognize is a potential hacking program. Use Google or a similar search engine to search the Internet for the program name and determine whether the program is legitimate. You should be especially suspicious of programs that load from C:, C:windows, and C:windowssystem32. I strongly suggest that you make a habit of regularly reviewing these registry keys so you become familiar with all the programs that are set to automatically load on your computers.

The following subkeys are less commonly used to launch hacking programs, but you need to check them also. These subkeys apply to all Windows OSs. If the default registry key contains a value other than "%1" %*, the program is most likely a hacker program.

Services. Review the HKEY_LOCAL_MACHINESYSTEMCurrentControlSetServices registry subkey on all Windows OSs. The entries under this subkey specify the services that are defined on your computer. I suggest that you look directly in the registry instead of using Windows' Services GUI because some services (e.g., Type 1 services) don't show up in the Services GUI. Again, check for programs you don't recognize. If possible, compare the Services subkey entries and values to a machine that you know is hack-free and investigate any differences you find.

Startup Folder. Check the C:Documents and SettingsAll UsersStart MenuProgramsStartup and C:Documents and Settingsuser_name>Start MenuProgramsStartup folders for unfamiliar programs and hidden files. To display a list of hidden files in the current folder and any subfolders, at a command prompt, enter

dir /a h /s

Task Scheduler. Check the C:windowstasks folder for unauthorized tasks. Investigate any scheduled task that you don't recognize.

Win.ini. Malicious users can load hacking programs automatically from C:windowswin.ini. Look in the following section of the win.ini file:

[windows]Run=Load=

Any program listed after Run= or Load= will load automatically when Windows starts.

System.ini. Intruders can use shell commands to load programs in C:windowssystem.ini. Search system.ini for:

[boot]shell=explorer.exe 

Any program listed after explorer.exe will load automatically when Windows starts.

Other locations exist from which a hacker can automatically load programs to launch when Windows starts. Sysinternals' Autoruns freeware utility shows you which programs are configured to load during startup on NT and later systems. You can download the tool from http://www.sysinternals.com/ntw2k/freeware/autoruns.shtml.

Open Ports and Unauthorized Users
After you've run your initial key-locations check for hacking activity, look for unexpected or suspicious open ports.

Root kits are stealth programs that run at the OS level and open ports on a compromised machine that an intruder can use for remote access. Root kits are common in the UNIX world, but more and more malicious hackers are writing them to exploit Windows. To determine connections and listening ports on a Windows-based computer, open a command line and run the command

Netstat -a

Table 1 lists ports that you'll typically find open on an XP computer. Don't be alarmed if you see more open ports on your particular workstation or server. Ports can be dynamically assigned depending on the type of service. For example, remote procedure calls (RPCs) use dynamic ports when you remotely administer DHCP and WINs. For more information, refer to the Microsoft article "How To Configure RPC Dynamic Port Allocation to Work with Firewall" at http://support.microsoft.com/?kbid=15459. When you run Netstat, look for the following items:

Some hacking tools can block Netstat from displaying open ports on a computer. If Netstat shows no suspicious open ports but you still suspect that some exist, run a port scanner tool such as the Network Mapper (nmap) open-source utility--which you can download at http://www.insecure.org/nmap--from another computer to see which ports are open on the target computer.

Rogue users in AD. When an intruder compromises a system, he or she will sometimes create one or more rogue users in Active Directory (AD). Often, intruders create these user accounts with a blank description. To combat this tactic, I suggest you add a description (following a specific naming convention) for every authorized user in AD. Then, you can sort your users by description, and all users without a description will appear at the top of the list.

Unauthorized users in privileged groups. One primary hacking goal is privilege escalation. Check the privileged groups in AD (e.g., Administrators, Domain Admins, Enterprise Admins, Server Operators) for unauthorized group membership. Make sure that you limit membership in these groups to make identifying unauthorized users easier.

Stop the Bleeding: A Hack Recovery Plan
If you discover that one of your systems has been hacked, don't panic. You need to keep your cool and proceed in a logical fashion. The following plan of action can help you limit the damage.

1.Isolate the network. Shut down all external interfaces on your network, including Internet, WAN, VPN, and dial-up connections, and disconnect all lines from routers, wireless Access Points (APs), and any other devices that connect your network to the outside world. This action can stop an active attack and prevent the intruder from compromising other systems.

2.Perform a wireless sweep. Use a wireless sniffer such as Airscanner Mobile Sniffer or NetStumbler.com's NetStumbler to locate any rogue APs in the area. Be sure to install the sniffer on a card that supports all current wireless standards (i.e., 802.11a, 802.11b, and 802.11g).

3.Check for other compromised machines. Use the techniques in this article to discover whether you have additional hacked machines.

4.Review firewall configuration. Look for any unauthorized rules, unauthorized open ports to the outside world, and unauthorized Network Address Translation (NAT) rules. Examine the firewall logs for any suspicious activity. I recommend that you always restrict outbound traffic to only necessary outbound ports and make sure that only authorized computers can send outgoing mail through the firewall.

5.Inspect AD. Look for any unauthorized user accounts and disable any you find.

6.Change passwords for every account on the network. For accounts with escalated privileges, I suggest you create a password (or pass phrase) of at least 15 characters. Passwords of this length are harder to crack because LAN Manager (LM) password hashes aren't stored on the server for passwords longer than 14 characters.

7.Replace hard disks on hacked computers. Replacing disks isolates and preserves the hacking activity. You can review the data on the old disks to gain valuable information about the attack.

8.Identify and address the vulnerability. Try to determine how the hacker accessed the network. This is often easier said than done (and outside the scope of this article). If you can't identify the vulnerability, consider hiring a security consultant to help.

9.Rebuild the compromised machine. It's almost impossible to completely clean a hacked computer. If one or more hacking tools remain on the machine, the intruder can regain access to the machine. The only way to ensure a clean computer is to format the hard disk and rebuild the machine from scratch, making sure you don't restore any previously installed hacking tools. You should reinstall all programs from CD-ROM, manually install any patches, and restore only data files. Never restore the registry, OS, or any programs from tape.

10.Run full virus scans on all machines. Be aware that antivirus software can sometimes identify hacking tools as legitimate programs. If a machine scans clean but you still suspect that it's been hacked, I recommend you rebuild the machine from scratch.

11.Reconnect the WAN lines. Reconnect and carefully monitor WAN lines to make sure you've closed holes on your network. Watch for heavy bandwidth usage on the network, closely monitor the firewall logs, and enable security auditing on all servers.

12.Perform forensic analysis on hacked hard disks. Install the hacked hard disks on a standalone computer and examine them to gain more information about the hack. Although intruders often spoof their IP addresses, the IP address is a good place to start tracking the attack's source. You can obtain a list of IP address allocations from the Internet Assigned Numbers Authority (IANA) Web site at http://www.iana.org.

13.Notify authorities. The FBI runs the Internet Fraud Complaint Center (IFCC--http://www.ifccfbi.gov/index.asp) for reporting suspicious Internet activity, and most FBI field offices have Cyber Action Teams (CATs). No one likes to admit to having been been hacked, but notifying the proper authorities can prevent a hacker from doing more damage. To contact your local FBI office, go to http://www.fbi.gov/contact/fo/fo.htm.

You can use these steps to design a customized hacking recovery plan. Tailor the steps to your organization, and integrate them into your company's disaster recovery plan.

Learning By Example
In my consulting practice, I come across many situations in which organizations have experienced attacks against their networks. Learning from others' experiences can help you detect vulnerabilities in your networks and help you recover from similar attacks. So, let's look at some real-life hacking scenarios.

IIS Attack in the DMZ
One of my clients called me, saying that users couldn't access certain folders on a Win2K Server system. When I discovered that all rights had been removed from the folders, I suspected that someone had compromised the system.

Identifying the hack. I opened the Microsoft Management Console (MMC) Active Directory Users and Computers snap-in and looked at the user accounts. Someone had created a rogue user account that was a member of the Administrators group. I knew an intruder had hacked the network, so I shut down all external connections and started looking for a compromised computer. It didn't take long. My client had two Web servers in the demilitarized zone (DMZ). One server ran the company's Web page; the other ran timekeeping software. I checked the registry Run subkeys and investigated suspicious batch files on the C drives of both servers. The server running the timekeeping software was severely compromised. It held rogue FTP and SMTP programs with multiple root kits. This server connected to a Microsoft SQL Server system on the LAN side and allowed only SQL Server traffic to pass from the DMZ to the LAN. The server was running Win2K with Service Pack 2 (SP2) and was missing a lot of critical Win2K updates.

Repairing the damage. I rebuilt the server from scratch, moved the server to the LAN side of the firewall, and eliminated public access to it. I then reconnected the external lines and monitored them closely for suspicious activity.

Lessons learned. Before you place a publicly accessible server in the DMZ, verify with the software vendors that any programs you run on the server are secure enough for public access. Keep all servers, not just those in the DMZ, updated with the latest service packs and critical updates. Make sure that applications use SQL Server­integrated security and don't use a connection string in their code to connect to an SQL server. Embedding a connection string in the Web server code instantly gives an intruder a valid username and password. For more information about establishing a SQL Server connection from a Web server, refer to the Microsoft article "Recommendations for Connecting to Databases Through Internet Information Services" (http://support.microsoft.com/?kbid=258939). Make sure Microsoft IIS uses stored procedures to access SQL Server data, and don't let the IIS server run SQL statements. Because these steps let you grant an authenticated user Execute permissions for only specific stored procedures, you can restrict the user from running any SELECT statements on SQL Server.

VPN Client Attack
Another client's Exchange 2000 Server machine was experiencing backup problems and poor server performance when sending and receiving email. I arrived to find the problem was more serious than a failed tape drive and slow server. The server had a lot of disk activity and high CPU utilization. I opened Windows Task Manager and sorted the processes by CPU utilization. Store.exe was taking up most of the CPU cycles. The company wasn't a heavy email user and had only 15 users connected to the server. With so few users, Exchange shouldn't consume as many resources as this server was. I suspected a corrupt mail store.

Identifying the hack. I started Exchange System Manager (ESM) and selected Administrative Groups, Admin_Group_Name, Servers, Server_Name, Protocols, Smtp, Default SMTP Virtual Server, Current Sessions. I noticed that six sessions had been connected to the SMTP virtual server for longer than 5 minutes--a clue that something was very wrong on the server. Typically, an Exchange Server session lasts only a few seconds unless the connection is sending or receiving a message with a large attachment. I looked at the queues on the default SMTP virtual server and discovered more than 50 queues in various states of sending mail or waiting for a retry. Someone was using the mail server as a relay, but how? The server had the latest service packs (Win2K SP4 and Exchange 2000 SP3) and the latest critical updates, so I used the Open Relay Database's (ORDB's) test at http://www.ordb.org, which checks submitted host systems for open relays, to ensure that the relay was closed.

Whenever I tried to clear a connection to the default SMTP virtual server, the connection would reappear, usually with a different domain name but from the same IP scheme. I used the IANA to trace the IP addresses to a block allocated by an ISP in China. After verifying that the server wasn't an open relay, I concluded that someone was probably authenticating to the server and sending mail from it. Backup was failing because it was trying to back up all the mail the spammer was trying to send. I opened the Active Directory Users and Computers snap-in and removed all invalid users. I also noticed some unauthorized users in the Administrators Group and removed them. I then checked the registry and found no hacking programs loaded in the Run subkeys. I also ran a virus scan on the server, and the server was clean.

To prevent the spammer from sending more messages, I disconnected the firewall from the Internet and deleted all the active sessions from the Exchange server. I tried to use ESM to delete the messages from the mail queues, but that took too long. So I stopped all Exchange services, opened a command prompt, and used the Del command to delete the messages from the directory D:exchsrvrmailrootvsi 1queue. As soon as I stopped the Exchange services, the server's performance greatly improved. Even so, deleting the 10,000 queued messages took me more than an hour. I then looked at the bad-mail directory in D:exchsrvrmailrootvsi 1badmail. It took me approximately 8 more hours to delete all those messages. Finally, I changed the passwords for every user on the network and created a rule on the firewall to deny traffic from the IP ranges where the spam originated. After making these changes, I reconnected the firewall to the Internet and monitored the server. The spam connection didn't reappear.

This particular network had several remote sites that ran VPN tunnels. At one of the remote sites, I discovered that the remote machine contained these hacking programs: Bat.Mumu.A.Worm, Hacktool, W32.Valla.2048, W32.HLLW.Lovgate.J@mm, Bat.Boohoo.Worm, and MSBlast.

My client said that this computer was left running all the time, with the VPN tunnel active. With such an arrangement, it's only a matter of time before someone hacks the machine. I always recommend that remote clients sit behind a firewall, especially if they use a broadband connection. If you run XP over a dial-up or wireless connection, make sure you use XP's Windows Firewall (formerly Internet Connection Firewall--ICF) to protect your computer while it's connected to the Internet.

Repairing the damage. I rebuilt the workstation, placed it behind SonicWALL's TELE3 firewall, and let the firewall create the tunnel back to the corporate office. Fortunately for this client, the intruder used the server only to send spam--he or she could have caused a lot more damage.

Lessons learned. Because of this hack, the company no longer lets client machines use a mobile VPN client on a broadband connection without a firewall. If you have remote sites with VPN tunnels and broadband connections, install a firewall--or at least train users to turn off their computers when they're not in use. Also make sure each user knows how to deactivate the VPN tunnel when it's not in use.

Exchange Server SMTP AUTH Attack
A third client's Internet connection was running slowly because of heavy Internet traffic. After I asked all users to disconnect from the Internet, the traffic was still heavy. I looked at the outgoing queues on the Exchange 2000 server and discovered more than 100, with a significant number of messages in each queue. Using ESM, I inspected messages in several random queues. I discovered messages whose sender or recipient wasn't from the local domain, which means the mail server was likely being used as a mail relay. By default, Exchange 2000 and later systems allow relaying if a message sender can successfully authenticate to the mail server.

Identifying the attack. Hackers can use a couple of different methods to get a valid username and password. They can repeatedly guess a guest or user's password until they stumble upon a valid one, or they can launch a hack to obtain a valid username and password. A spammer needs only one valid username and password to relay mail, even if your mail server isn't an open relay. To determine which account the spammer was using, I started ESM and clicked Organization, Administrative Groups, Organizational Unit, Servers, then right-clicked Server Name, Properties. I selected the Diagnostics Logging tab. In the Services window, I clicked MSExchangeTransport, and in the Categories window I increased the logging level to maximum for the categories Routing Engine, Categorizer, Connection Manager, Queuing Engine, Exchange Store Driver, SMTP protocol, and NTFS store driver. I then checked the event log, looking for an authentication from an external or unknown mail server. Unsuccessful logon attempts will show up in the Security log with event ID 680. I discovered that an intruder was using a user account that wasn't a local Exchange Server account to authenticate to the mail server.

Repairing the damage. After I identified the authentication account, I took the following steps to secure the Exchange server.

1. I changed the password for the account the spammer was using. If, in a similar situation, you think a spammer might have more than one valid user ID and password, change the passwords for all users on your network. I also disabled the Guest account and set up dedicated accounts to start services on the server. Don't use the Administrator account to start services. If a machine is hacked, the account used to start the service can be compromised.

2. I disabled authentication on the outward-facing Exchange server. To do so, I started ESM and selected Organization, Administrative Groups, Organizational Unit, Servers,ServerName, Protocols, SMTP, then right-clicked the default SMTP virtual server. I selected Properties, clicked the Access tab, then clicked Authentication. I left Anonymous access enabled but cleared the Basic authentication and Integrated Windows Authentication check boxes. Clearing these check boxes essentially disables the Auth command on the SMTP server.

3. I enabled relaying for other internal Exchange servers to ensure that they can send mail to the outward-facing Exchange server. I opened ESM, right-clicked the virtual SMTP server, and selected Properties. Under the Access tab, I clicked Relay, selected Only the List Below, and listed the internal mail servers that are allowed to relay mail to the outward-facing server.

4. After making these changes, I thoroughly tested the configuration. I tested mail flow to and from the Internet and to and from all mail servers in the organization. The changes have the potential to disrupt mail flow between servers, so you might want to wait until the weekend to make them. Better yet, test these changes in a lab environment before implementing them in production.

5. In this particular incident, I discovered a machine that was severely compromised, which I completely rebuilt. In situations you might encounter, you'll need to identify all compromised machines and repair or rebuild them.

6. I checked the ORDB to determine whether the client's mail server had been blacklisted for being an open relay. Fortunately, I discovered and repaired the hack before the client's mail server was blacklisted. A mail server can be blacklisted if it's an open relay or if the mail server is identified as a server that sends large amounts of spam. Many open-relay databases exist. You can see a list of some of these databases at http://dmoz.org/computers/internet/abuse/spam/blacklists.

If your mail server is blacklisted, you can either submit a request to remove the server from the blacklist or change the outside IP address of your mail server. If you change the mail server address, you must also update the mail exchanger (MX) record for your mail server, or incoming mail will be blocked.

Lessons learned. To repair Exchange Server SMTP AUTH attacks and prevent future ones, I strongly suggest that you take the steps I did. If an intruder procures a valid user ID and password and is able to relay mail, your mail server will be placed on various email blacklists. You'll spend significantly less time preventing these attacks than troubleshooting mail delivery problems, removing your server from blacklists, and fixing the vulnerability.

Don't Panic; Be Prepared
An attack recovery plan is part of any sound IT structure. It will help you respond efficiently to a network hack instead of going into a panic. Be familiar with the tools and methods that malicious intruders use and take a proactive approach to preventing them from hacking your network. I'll discuss this subject in more detail at the next Windows Connections Conference, from October 24 to October 27 in Orlando, Florida. Hope to see you there.