Network Speed and Throughput Issues – Some Things to Look For

I wrote a post about using iperf, a tool to test network speed and throughput. I just want to quickly go over some things that could affect those speeds.

Below are some things to look for that could affect maximum throughput on a single connection:

Bandwidth/Throughput – amount of data being sent/received
Packet loss – number of packets lost in transit
Latency – time required to transmit packets across the network
     Measured by: one-way, round trip,etc…
     Impacted by: PC/Server, LAN/WAN Links, Routers, Firewalls, Fiber light speeds, Bandwidth/Throughput,etc…
Network congestion – usually a when a link hits sustained 70% utilization, it’s considered saturated or congested
*Send window (buffer size) – Windows size on the Server/Device
*Receive window (buffer size) – Windows size on the Server/Device

When looking into any of the above, don’t forget to look for hardware devices like Routers, Switches, or Firewall that have settings configured which could filter/alter traffic.

I put asterisks next to the windows size items above because you use to be able to change these values and it would have an effect. Today those changes don’t have any effect (see below).  Sometimes those values aren’t even available. Windows has Windows Scaling turned on by default so there’s no changes you need to do on the Windows side. It’s important to point out that I’m talking about Windows here and not Linux. I’ll get more into Windows Scaling below.

TCP is a connection-oriented protocol which makes it a reliable option because it keeps track of lost packets and resends them if they do get lost.  It uses sequence numbers to make sure the packet arrives in order. UDP is the opposite where it doesn’t care if the packets are lost or arrive in order. TCP requires more resources than UDP as you could guess because it must do more work to get the packets to the other end in a reliable fashion. So, there’s more overhead with TCP than UDP.  This is the reason why when people do UDP speed tests with tools likeiperf, UDP is usually always has better results. Another thing to keep in mind is that UDP Throughput is not impacted by latency like TCP is. I’ve seen people say, “But why is UDP so much better than my TCP test?”. Well, above is the reason.  The cost you’re paying for reliable delivery with TCP is a loss in goodput in contrast to UDP which doesn’t guarantee delivery at the expense of throughput.

The “goodput” is the amount of useful information that is delivered per second to the application layer protocol. Dropped packets or packet retransmissions as well as protocol overhead are excluded. Because of that, the “goodput” is lower than the throughput.

Based on Microsoft’s site:
Note Unlike in versions of Windows that are earlier than Windows 10 or Windows Server 2019, the TCP Receive Window size can no longer be configured through the registry.

TCP Receive Window size is the basis of AutoTuningLevel negotiation. TCP Receive Window size is determined by whether the application is defined for it.

Note Some applications define the amount of data that the window should use.

If the application does not define the size, a default size is determined by link speed, as follows:

Less than 1 megabits per second (Mbps): 8 kilobytes (KB)
1 Mbps to 100 Mbps: 17 KB
100 Mbps to 10 gigabits per second (Gbps): 64 KB
10 Gbps or faster: 128 KB
For example, on a computer that has a 1 Gbps network adapter installed, the window size should be 64 KB.

Summary of level values

The different values for AutoTuningLevel are as follows.

Level                        Value
Normal (default)     0x8 (scale factor of 8)
Restricted                 0x4 (scale factor of 4)
Highly Restricted    0x2 (scale factor of 2)
Disabled                    No scale factor available
Experimental           0xE (scale factor of 14)

Taking the values above from Microsoft, we can get the Windows size values:

Scale/Shift Factor of 2 = 65,536 bytes * 2`2 = 262,144 bytes = .26MB
Scale/Shift Factor of 4 = 65,536 bytes * 2`4 = 1,048,576 bytes = 1MB
Scale/Shift Factor of 8 = 65,536 bytes * 2`8 = 16,777,216 bytes = 16MB
Scale/Shift Factor of 14 = 65,536 bytes * 2`14 = 1,073,741,824 bytes = 1073MB or 8.5 Gbps

To add to this, now let’s see how much of Windows we need by looking a the RTT and Circuit speed.  For example, below we are looking at a 1G connection that has a RTT of 40ms.

Circuit CIR = 1G
RTT = 40ms

1,000,000,000 bits/sec / 8 = 125,000,000 bytes/sec
40ms / 1000 = .04
125,000,000 bytes/sec * .04 seconds = 5,000,000 bytes = 5MB

To fill up the 1G pipe we must use a TCP window of 5MB given the RTT of 40ms. Based on that you can see which level you would want to use. But keep in mind that you don’t want to kill/congest/over saturate the link so it’s best to do some testing. Play around with the settings in a test nonProd environment if possible.