Time_waits and what they mean

Below are a couple good articles about Time_wait and what they mean as well as ideas on how to improve performance on busy IIS webservers.


http://blog.port80software.com/2004/12/07/hurry-up-and-time_wait/
So, ever wonder what all those TIME_WAITs are doing in your netstat listing?




Okay, for those of you who don’t spend all your waking hours fooling around with Web servers, let me back up a little and explain what that sentence meant.



Netstat is a little utility that many administrators use to monitor the network connections on their servers. It is quite useful for tracking down that small subset of performance bottlenecks that aren’t attributable to yet another piece of convoluted application code that some careless programmer wrote and now you have to take care of. But I digress.



When you run netstat on your busy IIS box, you might get something that looks like this:



C:\>netstat -np tcp



Active Connections



Proto Local Address Foreign Address State



TCP 192.168.0.1:80 192.168.0.12:1217 ESTABLISHED

TCP 192.168.0.1:80 192.168.0.5:1218 TIME_WAIT

TCP 192.168.0.1:80 192.168.0.234:1252 TIME_WAIT

TCP 192.168.0.1:80 192.168.0.37:1267 ESTABLISHED

TCP 192.168.0.1:80 192.168.0.23:1298 TIME_WAIT

TCP 192.168.0.1:80 192.168.0.32:1345 TIME_WAIT



And so on and on, for many, many lines. Each line here represents a connection between a TCP socket your server and a matching one on some other machine–usually an HTTP client such as a browser or proxy server, but depending on your architecture you might also see connections to other kinds of servers (database, application, directory, etc.). Each connection has a unique combination of IP addresses and port numbers that identify the endpoints to which the sockets are bound. More to the point, each one also has a state indicator. As connections are set up used and torn down, they pass through a variety of these states, most of which aren’t shown here, because they come and go quite quickly).



The connections in the ESTABLISHED state are, well, established–they are neither being set up nor torn down but just used. This is what you will often see the most of. But what about the others? On a busy HTTP server, the number of sockets in this TIME_WAIT state can far exceed those in the ESTABLISHED state. For instance, I checked an IIS 6.0 box that serves a fairly busy corporate site earlier today and got 124 ESTABLISHED connections versus 431 in TIME_WAIT.



What does this all mean? More importantly, is it something you should be worried about?



The answers are:



1. It’s complicated.



2. Maybe.



To understand what all those TIME_WAITs are doing there, it’s useful to review (or learn) a little TCP. I’ll wait here while you brush up on RFC793.



That was fast. Just kidding. The bit you need to know is so simple, even I can explain it.



As you know, TCP provides a reliable connection between two endpoints, across which data can be sent in segmented form. As part of this, TCP also provides a mechanism for gracefully shutting down such connections. This is accomplished with a full duplex handshake, which can be diagrammed like so:



Server Client



————– FIN ————–>



<————- ACK —————



<————- FIN —————



————– ACK ————->



As you can see by this very sophisticated diagram, a graceful shutdown requires the two endpoints to exchange some TCP/IP packets with the FIN and ACK bits set, in a certain sequence. This exchange of packets in turn corresponds to certain state changes on each side of the connection. In the diagram, I’ve labeled the two sides “Server” and “Client” such that the sequence of events mirrors what usually happens when connections are closed by HTTP.



Here is what happens, step-by-step:



1. First the application at one endpoint–in this example, that would be the Web server–initiates what is called an “active close.” The Web server itself is now done with the connection, but the TCP implementation that supplied the socket it was using still has some work to do. It sends a FIN to the other endpoint and goes into a state called FIN_WAIT_1.



2. Next the TCP endpoint on the browser’s side of the connection acknowledges the server’s FIN by sending back an ACK, and goes into a state called CLOSE_WAIT. When the server side receives this ACK, it switches to a state called FIN_WAIT_2. The connection is now half-closed.



3. At this point, the socket on the client side is in a “passive close,” meaning it waits for the application that was using it (the browser) to close. When this happens, the client sends its own FIN to the server, and deallocates the socket on the client side. It’s done.



4. When the server gets that last FIN, it of course sends back on ACK to acknowledge it, and then goes into the infamous TIME_WAIT state. For how long? Ah, there’s the rub.



The socket that initiated the close is supposed to stay in this state for twice the Maximum Segment Lifetime–2MLS in geek speak. The MLS is supposed to be the length of time a TCP segment can stay alive in the network. So, 2MLS makes sure that any segments still out there when the close starts have time to arrive and be discarded. Why bother with this, you ask?



Because of delayed duplicates, that’s why. Given the nature of TCP/IP, it’s possible that, after an active close has commenced, there are still duplicate packets running around, trying desperately to make their way to their destination sockets. If a new socket binds to the same IP/port combination before these old packets have had time to get flushed out of the network, old and new data could become intermixed. Imagine the havoc this could cause around the office: “You got JavaScript in my JPEG!”



So, TIME_WAIT was invented to keep new connections from being haunted by the ghosts of connections past. That seems like a good thing. So what’s the problem?



The problem is that 2MLS happens to be a rather long time–240 seconds, by default. There are several costs associated with this. The state for each socket is maintained in a data structure called a TCP Control Block (TCB). When IP packets come in they have to be associated with the right TCB and the more TCBs there are, the longer that search takes. Modern implementations of TCP combat this by using a hash table instead of a linear search. Also, since each TIME_WAIT ties up an IP/port combination, too many of them can lead to exhaustion of the default number of ephemeral ports available for handling new requests. And even if the TCB search is relatively fast, and even if there are plenty of ports to bind to, the extra TCBs still take up memory on the server side. In short, the need to limit the costs of TIME_WAIT turns out to be a long-standing problem. In fact, this was part of the original case for persistent connections in HTTP 1.1.



The good news is that you can address this problem by shortening the TIME_WAIT interval. This article by Brett Hill explains how to do so for IIS. As Brett explains, four minutes is probably longer than needed for duplicate packets to flush out of the network, given that modern network latencies tend to be much shorter than that. The bad news is that, while shortening the interval is quite common, it still entails risks. As Faber, Touch and Yue (who are the real experts on this) explain: “The size of the MSL to maintain a given memory usage level is inversely proportional to the connection rate.” In other words, the more you find yourself needing to reduce the length of TIME_WAIT, the more likely doing so will cause problems.



How’s that for a Catch-22?


http://www.windowsitpro.com/article/registry2/the-time_wait-state-s-effect-on-iis-performance.aspx

My IIS 5.0 server is usually under a large load, and performance slows periodically. While investigating the problem, I've discovered more than 1000 TCP ports in the TIME_WAIT state. Is this state typical on an active server, and could it be affecting performance?




The TIME_WAIT state is typical, but it can also affect performance. You might want to adjust the amount of time your IIS machine keeps a TCP/IP session open after the client has disconnected. After a connection is closed, the server port goes into a TIME_WAIT state when the client sends the server a FIN packet. By default, the server then keeps the connection alive for 4 minutes. (Internet Engineering Task Force—IETF—Request for Comments—RFC—793 provides the rationale for keeping the connection alive this long.) As long as the server port is in TIME_ WAIT state, no other connections can be made to the port. This default time is protection against delayed network communication from an old session intermingling with communication from a new session and resulting in unpredictable consequences.



Network traffic on today's Internet and intranets is unlikely to have a 4-minute latency. Consequently, you can usually decrease the TIME_WAIT interval to 1 minute or less. This change will cause your server to recycle connections almost four times faster than the default configuration, resulting in more efficient use of your server's resources. You can use the Netstat (netstat.exe) utility to determine how many ports are in the TIME_WAIT state. (For more information about TCP connections and the Netstat utility, see the Microsoft article "TCP Connection States and Netstat Output" at http://support.microsoft.com/support/kb/articles/q137/9/84.asp.)



The setting you need to adjust to change the TIME_WAIT interval doesn't exist by default. You must add the Tcp TimedWaitDelay REG_DWORD value to the HKEY_LOCAL_MACHINE\ SYSTEM\CurrentControlSet\Services\ Tcpip\Parameters registry subkey. Then, you set the delay to the number of seconds (in decimal form):



Value Type: REG_DWORD--

Time in seconds

Valid Range: 30­300 (decimal)

Default: 0xF0 (240 decimal)Of course, before you make any changes to the registry, be sure you have a good recovery system in place. For more information about this registry change, see the Microsoft article "TCP/IP and NBT Configuration Parameters for Windows" (http://support.microsoft.com/support/kb/articles/q120/6/42.asp) and the Windows Registry Guide Web site (http://www.winguides.com/registry/display.php/878).



Another factor that can affect performance is that you might be running low on ephemeral ports—ports used to create connections to the client computers from the server and between COM server objects. By default, these ports range from 1024 to 5000, and if you run out of them, connection problems can occur. As the Microsoft article "Unable to Connect from TCP Ports Above 5000" (http://support.microsoft.com/support/kb/articles/q196/2/71.asp) states, you can increase the number of ports available to 65,535. Each port uses 2KB to 4KB of memory, so be certain you have ample RAM to support the number of connections you intend to use. (Each connection requires at least one port.)