How to make BSD (SunOS) kernels SYN-attack resistant

The quick version: Grab the appropriate object files and rebuild your kernel

The two key fixes in BSD-derived TCP OS code.
For SunOS, I've supplied the three modules you'll need to replace
for Sun4c and Sun4m architectures.

The quick fix is to put these three modules in your 
/usr/kvm/sys/sun4c/OBJ or /usr/kvm/sys/sun4m/OBJ directory and
build a new kernel.

For sun4c: uipc_socket2.o tcp_input.o tcp_usrreq.o

For sun4m: uipc_socket2.o tcp_input.o tcp_usrreq.o

Our experience with the patches is that even on a lowly Sparc 1+, it can
easily repel 50-200 SYNs/second.  And a Sparc 10 model 30 can repel 1000-2000
packets/second without a sweat.

The technical version (requires kernel source): Part 1

Set the timeout from 75 seconds to 15 seconds on pending TCP connections.
This will expire any sockets and PCBs from connections for which we've
only recevied SYNs.

 a) In src/sys/netinet/tcp_timer.h, change:

    #define	TCPTV_KEEP_INIT	(75*PR_SLOWHZ)		/* initial connect keep alive */

    to:

    #define	TCPTV_KEEP_INIT	(15*PR_SLOWHZ)		/* initial connect keep alive */

    Or whatever timeout value, in seconds, you want.

    Then, make sure that tcp_input.c and tcp_usrreq.c get recompiled
    (they're the two modules in netinet which use TCPTV_KEEP_INIT).

The technical version (requires kernel source): Part 2

Change the algorithm to allow either an infinite or a very large
(many thousand) number of embryonic TCP sockets to be queued up
"on" your server socket (in so_q0).

The default algorithm is:

In src/sys/os/uipc_socket2.c (under SunOS; FreeBSD and NetBSD are in sys/kern):
  In sonewconn(), the first few lines of code should be something like:

	if (head->so_qlen + head->so_q0len > 3 * head->so_qlimit / 2)
		goto bad;

This says "If (the number of established TCP connections which we
haven't accept()ed yet + the number of unestablished TCP connections
we're waiting on to become established) is mre than 3/2 (a fudge
factor) * the so_qlimit) then dump this embryonic (unestablished) TCP
connection and free the data structures ("goto bad")."

so_qlimit is set in uipc_socket.c to the minimum of either the number
the application requests in the listen() syscall or SOMAXCONN.  You can
just crank SOMAXCONN, but people have reported difficulty doing it above
256 or 1024 or something under SunOS.  Also, if you modify the algorithm
here, you don't have to recompile your applications.

The sun4c object above has these two lines commented out.  For some reason,
the sun4m kernels don't like that, so the sun4m kernel has the 3/2 changed
to 3000...