Article ID: 214397
When you need to send small data packets over TCP, the design of your Winsock application is especially critical. A design that does not take into account the interaction of delayed acknowledgment, the Nagle algorithm, and Winsock buffering can drastically effect performance. This article discusses these issues, using a couple of cases studies, and derives a series of recommendations for sending small data packets efficiently from a Winsock application.
BackgroundWhen a Microsoft TCP stack receives a data packet, a 200-ms delay timer goes off. When an ACK is eventually sent, the delay timer is reset and will initiate another 200-ms delay when the next data packet is received. To increase the efficiency in both Internet and the intranet applications, Microsoft TCP stack uses the following criteria to decide when to send one ACK on received data packets:
Winsock uses the following rules to indicate a send completion to the application (depending on how the send is invoked, the completion notification could be the function returning from a blocking call, signaling an event or calling a notification function, and so forth):
Case Study 1
Overview:A Winsock TCP client needs to send 10000 records to a Winsock TCP server to store in a database. The size of the records varies from 20 bytes to 100 bytes long. To simplify the application logic, the design is as follows:
Performance:During testing, the developer finds the client could only send five records per second to the server. The total 10000 records, maximum at 976K bytes of data (10000 * 100 / 1024), takes more than half an hour to send to the server.
Analysis:Because the client does not set the TCP_NODELAY option, the Nagle algorithm forces the TCP stack to wait for an ACK before it can send another packet on the wire. However, the client has disabled the Winsock buffering by setting the SO_SNDBUF option to 0. Therefore, the 10000 send calls have to be sent and ACK'ed individually. Each ACK is delayed 200-ms because the following occurs on the server's TCP stack:
How to Improve:There are two problems with this design. First, there is the delay timer issue. The client needs to be able to send two packets to the server within 200-ms. Because the client uses the Nagle algorithm by default, it should just use the default Winsock buffering and not set SO_SNDBUF to 0. Once the TCP stack has coalesced a buffer larger than the Maximum Transmission Unit (MTU), a full-sized packet is sent immediately without waiting for the ACK from the remote host.
Secondly, this design calls one send for each record of such small size. Sending this small of a size is not very efficient. In this case, the developer might want to pad each record to 100 bytes and send 80 records at a time from one client send call. To let the server know how many records will be sent in total, the client might want to start off the communication with a fix-sized header containing the number of records to follow.
Case Study 2
Overview:A Winsock TCP client application opens two connections with a Winsock TCP server application providing stock quotes service. The first connection is used as a command channel to send the stock symbol to the server. The second connection is used as a data channel to receive the stock quote. After the two connections have been established, the client sends a stock symbol to the server through the command channel and waits for the stock quote to come back through the data channel. It sends the next stock symbol request to the server only after the first stock quote has been received. The client and the server do not set the SO_SNDBUF and TCP_NODELAY option.
Performance:During testing, the developer finds the client could only get five quotes per second.
Analysis:This design only allows one outstanding stock quote request at a time. The first stock symbol is sent to the server through the command channel (connection) and a response is immediately sent back from the server to the client over the data channel (connection). Then, the client immediately sends the second stock symbol request and the send returns immediately as the request buffer in the send call is copied to the Winsock kernel buffer. However, the client TCP stack cannot send the request from its kernel buffer immediately because the first send over the command channel is not acknowledged yet. After the 200-ms delay timer at the server command channel expires, the ACK for the first symbol request comes back to the client. Then, the second quote request is successfully sent to the server after being delayed for 200-ms. The quote for the second stock symbol comes back immediately through the data channel because, at this time, the delay timer at the client data channel has expired. An ACK for the previous quote response is received by the server. (Remember that the client could not send a second stock quote request for 200-ms, thus giving time for the delay timer on the client to expire and send an ACK to the server.) As a result, the client gets the second quote response and can issue another quote request, which is subject to the same cycle.
How to Improve:The two connection (channel) design is unnecessary here. If you use only one connection for the stock quote request and response, the ACK for the quote request can be piggybacked on the quote response and come back immediately. To further improve the performance, the client could "multiplex" multiple stock quote requests into one send call to the server and the server could also "multiplex" multiple quote responses into one send call to the client. If the two unidirectional channel design is really necessary for some reason, both sides should set the TCP_NODELAY option so that the small packets can be sent immediately without having to wait for an ACK for the previous packet.
Recommendations:While these two case studies are fabricated, they help to illustrate some worst case scenarios. When you design an application that involves extensive small data segment sends and recvs, you should consider the following guidelines:
For more information about Delayed Acknowledgment and the Nagle algorithm, please see the following:
Braden, R., RFC 1122, Requirements for Internet Hosts--Communication Layers, Internet Engineering Task Force.
Article ID: 214397 - Last Review: June 22, 2014 - Revision: 5.0