As a general rule, reset (RST) must be sent whenever a segment arrives which apparently is not intended for the current connection. A reset must not be sent if it is not clear that this is the case ...
If the connection does not exist (CLOSED) then a reset is sent in response to any incoming segment except another reset. In particular SYNs addressed to a non-existent connection are rejected by this means.
Reset Processing [page 36]
... In the SYN-SENT state (a RST received in response to an initial SYN), the RST is acceptable if the ACK field acknowledges the SYN.
This behavior may result in poor performance if for some reason a process repeatedly issues connect() calls to ports with nothing listening there, resulting in the error WSAECONNREFUSED. Note that with other implementations of TCP, such as those commonly found in many UNIX systems, the connect() fails immediately upon the receipt of the first ACK/RST packet, resulting in the awareness of an error very quickly. However, this behavior is not specified in the RFCs and is left to each implementation to decide. The approach of Microsoft platforms is that the system administrator has the freedom to adjust TCP performance-related settings to their own tastes, namely the maximum retry that defaults to 3. The advantage of this is that the service you're trying to reach may have temporarily shut down and might resurface in between SYN attempts. In this case, it's convenient that the connect() waited long enough to obtain a connection since the service really was there.
It contains the complete listing of NT TCP registry values. Of particular interest is the key:
Key : TcpMaxConnectRetransmissions
Value Type : REG_DWORD - Number
Valid Range : 0 - 0xFFFFFFFF
Default : 3 (in Windows NT)
Default : 2 (in Windows 2000)
Description : This parameter determines the number of times TCP will retransmit a connect request (SYN) before aborting the attempt. The retransmission time-out is doubled with each successive retransmission in a given connect attempt (except in the situation discussed above). The initial time-out value is three seconds (since an ACK/RST was received in the case above, this is irrelevant).
Key : MaxConnectRetries
Value Type : DWORD - 32 bit number
Default : 3
Description : Specifies the number of times a connection attempt (SYN) will be retransmitted before giving up. The initial retransmission time-out is 3 seconds (irrelevant in the above case), and it is doubled each time (the case above is an exception) up to a maximum of 2 minutes.
WARNING: Modifying these values affects any and all TCP outgoing connection requests from the affected computer. If all of the TCP activity on the computer has little latency and little distance to travel round-trip, reducing this as low as zero (although Microsoft does not recommend this) will make no difference. However, if applications or connection attempts fail regularly with the standard Winsock errors and the target remote servers are known to be available, it is likely that these registry entries are the cause. If so, you may have to raise the maximum retries so that TCP will double the SYN packet's time-to-live value in its IP header at successive retries until it has a long enough life span to reach the target host and receive an ACK/SYN.