Causes of TCP TIME_WAIT Increase

The TIME_WAIT state occurs during the final stage of the TCP connection termination process, the 4-way handshake.

It refers to the state where the active close side, which initiated the connection termination, does not immediately clean up the socket after sending the final ACK, but rather leaves it open for a certain period (typically 2MSL, about 2-4 minutes).

This allows for delayed packet handling, preventing old duplicates that arrive late due to network delays from being mistaken for data from a new connection.

It also ensures connection termination, allowing the system to re-process a FIN if the peer retransmits it in case the final ACK was lost.

Problem Situation #

The main cause is who initiates the connection termination.

In server environments, TIME_WAIT often surges because the server becomes the initiator of the active close.

• HTTP/1.0 and no keep-alive: If a connection is established and terminated for every request, TIME_WAIT occurs every time.
• Short-lived connections: Occurs when frequently making short communications with external APIs like Redis or databases.
• Reverse Proxy Configuration: If a reverse proxy server like Nginx does not use keep-alive when communicating with an upstream (app), the proxy server's local ports will be exhausted.

For example, let's assume a server processing 5,000 requests per second, with a system local port range of 32,768 to 60,999, which is about 28,000 ports.

If TIME_WAIT is 60 seconds, even consuming only 500 ports per second will exhaust the ports in one minute. This results in a Can't assign requested address error.

netstat -ant | awk '/^tcp/ {print $6}' | sort | uniq -c | sort -n

2 CLOSE_WAIT
15 ESTABLISHED
8 LISTEN
12 SYN_SENT
15420 TIME_WAIT


# 또는 ss 명령어 (더 빠름)
ss -s

Total: 15600 (kernel 16000)
TCP:   15500 (estab 15, closed 15420, orphaned 0, timewait 15420)

Transport Total     IP        IPv6
RAW	      0         0         0        
UDP	      5         3         2        
TCP	      80        15500     5

Use the command above to check how many TIME_WAIT sockets are in the overall state.

netstat -atn | grep TIME_WAIT | awk '{print $5}' | cut -d: -f1 | sort | uniq -c | sort -nr

8500 10.0.1.50
6200 172.16.0.12
700 127.0.0.1
20 211.234.xx.xx

You can also check which IP port is experiencing a high number of occurrences.

Solution #

The fundamental solution is connection reuse.

1. HTTP Keep-Alive Activation: Maintain connections between clients and servers to reduce the number of handshakes.
2. Use Connection Pool: When connecting to databases, Redis, or external APIs, use pre-established pools instead of creating and tearing down connections every time.

This can also be done by tuning kernel parameters.

1. net.ipv4.tcp_tw_reuse=1: Allows sockets in the TIME_WAIT state to be reused for new connections.
2. net.ipv4.ip_local_port_range: Expands the range of available local ports.

sysctl -w net.ipv4.ip_local_port_range="1024 65535"

3. net.core.somaxconn: Increases the LISTEN backlog queue size to enhance connection request capacity.

Never use net.ipv4.tcp_tw_recycle. It was removed after Linux kernel 4.12 and causes packet drop issues in NAT environments.

Finally, at the architectural level, you can place a Load Balancer (LB) to manage connections, allowing the actual services to maintain only lightweight communication.

Delayed Packet Handling, Guaranteed Connection Termination #

This might raise a question: if we use keep-alive or connection pools to continuously maintain or reuse connections without terminating them,

wouldn't this create issues regarding the original reasons for TIME_WAIT's existence, namely delayed packet handling and guaranteed connection termination?

Guaranteed Connection Termination #

TIME_WAIT is a state that occurs when a connection is terminated. However, if keep-alive and connection pools are used, connections are not terminated in the first place.

• Traditional method: Request - Response - Connection Termination FIN - TIME_WAIT occurs
• Improved method: Request - Response - Connection Maintained - Next Request - Response ...

Delayed Packet Handling (Old Duplicate) #

While a connection is maintained, TCP sequence numbers continue sequentially.

Even if a very old, delayed packet arrives late, it will naturally be discarded at the kernel level because it does not match the currently active sequence number range.

In other words, there's no need to wait in the TIME_WAIT state; TCP's standard sequence validation logic within a live connection filters out duplicate packets.

Furthermore, delayed packet and termination issues are eventually handled when the connection is finally closed.

Of course, connections in a connection pool are not maintained indefinitely. They are reset if there's an idle timeout or server restart.

The mechanism when a connection is terminated is as follows:

1. Minimized Occurrence: If sending 1,000 requests previously resulted in 1,000 TIME_WAITs, using pooling means TIME_WAIT occurs only for the occasional single connection termination. The system can easily handle this.
2. Safe Termination: When a decision is made to terminate a connection, a normal 4-way handshake occurs, entering the TIME_WAIT state to handle delayed packets and ensure termination.

Essentially, keep-alive and connection pool strategies drastically reduce the number of risky active closes,

and rather than abandoning delayed packet handling, they control the situations that require it to a manageable level.