Stopping The Flood

If you notice that your servers are up, but that certain services seem slow or unavailable, then it's possible that you're being SYN-flooded. Any service based on TCP (as opposed to UDP) is vulnerable to SYN-flooding: SMTP (mail), NNTP (news), HTTP (web), and Gopher services are all vulnerable to attack.

TCP/IP services operate over "connections," and SYN-flooding programs send out many connection establishment requests (called "SYNs") per second, attempting to overflow the remote computer's "queue" of pending connection requests. Since the computer offering the TCP-based service has no way to tell the difference between forged and real requests, legitimate users are locked out of your TCP-based services.

SYN FLOOD ATTACKS

In the Summer and Fall of 1996, code for a SYN flooding was published, first in an electronic hacker magazine, and then in print form by 2600 Magazine. In order to generate devastating attacks, all you need is a UNIX box connected to a dial-up phone line. SYN flooding is called a "Denial of Service" (DoS) attack.

Other forms of DoS attacks can cripple computers and destroy data, but SYN flooding "just" renders Internet servers temporarily unreachable. Even so, if your business is web serving _ or even simple delivery of mail to your users - a sustained SYN flood attack will put you out of business.

This past fall, Panix, the original dial-up ISP in New York City, was attacked repeatedly by SYN-floods. Luckily, Panix has a number of people who are very familiar with TCP/IP and UNIX, so figuring out what was happening didn't take very long. Working with Net Access in Philadelphia, some interim solutions were developed that held off the attacks and allowed normal functioning of the Panix servers, but it took many late nights and much wasted time. The Panix attack was the first widely-reported incident. Since then, hundreds of ISPs have reported slow or unresponsive servers due to SYN flooding.

Luckily, there are now fixes to almost every major vendor's TCP/IP code. You don't need to fix the web or mail server software - the fix is at the OS level.

WHAT IS A SYN?

SYN is the abbreviation for SYNchronizing segment. It is the first segment sent by the TCP protocol and is used to synchronize the two ends of a connection in preparation for opening a connection.

Every TCP server application listens on a certain port (See Fig 1). That port, while the server is listening, has a queue of pending connection requests. Each client sends a SYN to the server, asking "will you talk to me?" If there is no server listening, then the OS on the server will send a packet back saying "No, go away" (a RST packet _ the reset flag). If the server is listening and ready, it sends back a SYNACK (an acknowledgment of the SYN) to the client. When the client receives the SYN-ACK, it sends back yet another packet to acknowledge it (the ACK of the SYN-ACK). When the server receives this packet, it removes the SYN from the pending queue and sets up a TCP connection.

However, if the queue of pending connection requests (SYNs) is full, then new SYNs can't be entered and the client will just hang trying to connect to the server. It won't be refused or accepted, instead it will just wait until its SYN can get into the queue (See Fig 2).

RANDOM SOURCE ADDRESSES

Every IP packet has a source address and a destination address. One key problem with Internet security is that source addresses are easy (if you have the right software) to forge. Much of the planning of the Internet involved the assumption that the actual machines on the net were large, monitored, and somewhat secure. However, everyone is "root" on their own Linux or Win95 machine, and anyone can easily pop a program in to generate nasty attack packets.

SYN-flooding from a host without forging the source address doesn't work well because:

• The server will send SYN-ACKs back to the SYN-flooding host, but the SYN flooding host will say, "Hey, nope, I didn't ask to connect to you." It will then issue a RST packet, which will knock the bogus packet out of the SYN queue on the server and lessen the effect of the attack.

• People can see the actual source address of the attack machine, filter those packets out, and try to find and prosecute the attacker.

But if the source addresses are forged, there's no simple way for the ISP being attacked to defend against the attacks _ and tracking down the attackers is much more difficult. Unfortunately, most of the code out there does randomize the source address.

HOW DO YOU KNOW YOU'RE UNDER ATTACK?

Presumably you're asking this question at a time when response from your servers is slow or nonexistent, yet the servers themselves seem fine when you log into them.

If you do a 'netstat -an | grep SYN' on a UNIX machine, it should show you the pending connections waiting in the SYNRECVD state. Note that not all of these are necessarily SYN attacks! A slow dial-up user, who is a few seconds away from you IP-wise, will show up as a pending connection request for a few seconds. If you see tens of connect attempts to a particular service (port 23, 25, 110, 119, etc...) - in particular, from very random looking source addresses - then you have cause to suspect that you're being attacked.

SO WHAT CAN YOU DO?

Well, it would be nice to get involved and try to find SYN-flooding attackers so that some can be caught and punished. But probably it's best to just fix your hosts so they're not vulnerable to SYN attacks, and go on with your life.

You can, if you want, contact your provider upstream and ask to trace the attacks. They may be interested in helping, or may tell you to solve the problem on your end. This might be intensely frustrating for you, as it was for Panix when they were attacked. When Panix was attacked, good fixes weren't available, and Panix couldn't get timely cooperation from either of their upstream providers to try to trace the source of the attack.

David Dennis, maintainer of the inet-access FAQ, has a SYN-flood FAQ at http://cgi.amazing.com/internet. While you're there, I strongly suggest that you look at the inet-access FAQ and join the inet-access mailing list, at least in a read-only mode. There's a lot of information flowing by, but technical and business issues of interest to every ISP are discussed (and re-discussed) on inet-access.

DETAILS OF THE FIX

There are some simple approaches to fixing the SYN problem:

• Decrease the length of time that pending connections are allowed to sit around in the OS queues

• Increase the length of the queues (linked-lists, actually)

• Use better kernel data structures to store the list of pending connections

The standard UNIX kernels use 75 seconds as the hold time, and often limit the per-port list of pending connects to 10! Decreasing the hold time and increasing the list length (to a few hundred) will help in cases of attack, but it's not a perfect fix. Examples of how to do this under SunOS are available at http://www.net axs.com/~freedman/syn.

If you're running BSDi, use David Borman's excellent fix, which uses a mix of approaches and is almost a total fix. Basically, it does nothing special when the machine is not under attack, but uses compressed and efficient data structures to store the pending connections when the machine is being attacked. You can find this patch at ftp://ftp.bsdi.com/patches.

Jeff Weisberg, of Op.net, coded the SunOS patch that many are using. It is a total fix for the SYN attack problem, since the kernel actually doesn't have any queues. A description of this fix can be found, with the fix itself, at http://www.op.net/~jaw/syn.html. At Net Access, we run this patch on all of our SunOS machines. There is also some source code for applying the same fix to any BSD-based kernel. It is available at ftp://ftp.op.net/pub/src/synprophylactia.

Also, some firewall vendors make fairly expensive "proxy" additions which actually protect your whole network by only allowing valid SYN requests through. These additions maintain queues, negotiate with the remote clients, and then do some mild spoofing to make it look to the client and server that the negotiation happened normally. Checkpoint's FireWall was one of the first to implement SYN defenses. A great description of Checkpoint's SYN defense approach is at http://WWW.CheckPoint.COM/products/additions/syndefender.

YOU CAN HELP PREVENT ATTACKS

After Alexis Rosen of Panix made his hosts SYN-resistant, he started a campaign to educate other ISPs about preventing SYN attacks. A site being attacked can't block packets from random source addresses, but ISPs can block packets that have bogus source addresses. A router with "IP filtering" capability can do this with a very simple set of "filter rules." Basically, allow any packet with a reasonable source address (within your block of network addresses) to leave your network and deny others.

If all or almost all Internet providers put these filters into place, then it would be much more difficult for people to launch SYN-attacks, which would save everyone much time and effort.

Additionally, there are other known, but unimplemented, attacks which require being able to forge the source address - and some of these attacks have no known good defenses yet. Implementing outgoing "sanity" filters prevents your network from being used as a base for every type of forged-source-address DoS attack. Pointers to instructions for implementing these garbage filters on your router are available at http://cgi.amazing.com/internet.