To understand how a TCP SYN flooding Denial of Service (DoS) attack works, you need to know how a TCP connection between two hosts (aka a TCP 3-way handshake) works. A TCP connection starts with the client sending a SYN message to the server. A SYN message is a message in which the TCP header has the SYN bit set on, which lets the receiver know that the sender wants to establish a TCP-based connection. The server replies to the SYN message with a SYN/ACK message to acknowledge that it’s received the initial SYN message. After this exchange, the TCP connection is half open. To open the TCP connection completely, the client must reply to the server with another ACK message. Then, data can move between the client and the server in both directions.
In a TCP SYN flooding DoS attack, an attacker sends out many SYN messages with forged IP addresses (this type of attack is called spoofing). The server replies with SYN/ACK messages, but the attacker never acknowledges these messages, thereby leaving many half-open connections on the server. The intruder can continue sending SYN messages until the server reaches its half-open-connection limit and can’t respond to any new incoming requests.