Troubleshooting AWS Direct Connect Performance: What iperf3 and MTR Can and Cannot Tell You

Direct Connect performance complaints usually arrive in the same form: the circuit is 1 Gbps, the transfer is running at 300 Mbps, and someone wants to know why AWS is throttling the connection. In most cases nothing is being throttled. The test is measuring a limit that has nothing to do with the circuit.

This post assumes you already know how to drive iperf3 and read an mtr report. What follows is what changes when the path is a Direct Connect virtual interface, where the real ceilings sit, and which of these tools stops being useful.

Build a test environment worth trusting

Every number you produce is bounded by the endpoints generating it. Before testing anything, set up a dedicated on-premises test machine and an EC2 instance in the target VPC.

The EC2 instance matters more than people expect. Instance size determines network bandwidth allocation, and smaller instance types use a burst credit model: they deliver high throughput briefly, exhaust their credits, then settle to a much lower baseline. A test that looks excellent for thirty seconds and then degrades is usually describing credit exhaustion rather than the network.

• Use a current-generation instance sized well above the throughput you intend to measure. C5n, M5n, and similar network-optimized types are appropriate.
• Avoid burstable types entirely. Results will not be reproducible.
• Confirm the instance supports enhanced networking, and that the ENA driver is present and current.
• Place the test instance in the subnet and Availability Zone your production traffic actually uses.
• Keep the same pair of endpoints for every test so results remain comparable.

The central rule: on Direct Connect, the circuit is rarely the first thing to saturate. Establish that the instance, the flow, and the MTU are not the constraint before raising a ticket about the connection.

The single-flow limit is the usual culprit

This is the most common cause of Direct Connect performance tickets, and it is not a defect.

Traffic between an EC2 instance and Direct Connect is subject to a per-flow bandwidth limit. A flow is identified by the five-tuple: source IP, destination IP, protocol, source port, and destination port. Baseline bandwidth for single-flow traffic is limited to approximately 5 Gbps when instances are not in the same cluster placement group.

The practical consequences:

• A single TCP connection cannot exceed roughly 5 Gbps regardless of circuit size.
• On a 10 Gbps, 100 Gbps, or 400 Gbps Direct Connect, one flow will never fill the port.
• An application that uses one connection for bulk transfer is capped by this limit, not by the circuit.
• Multiple flows can aggregate above it, which is why parallel-stream tests look so different from single-stream tests.

This is exactly the single-stream versus parallel-stream comparison from the iperf3 side of things, and on Direct Connect it has a specific, documented number attached to it. Run both:

# On the EC2 instance (server side)
iperf3 -s

# Single stream, forward — establishes the per-flow ceiling
iperf3 -c 10.0.1.50 -t 30 -O 3 -i 1 -T "DX-1Stream-Forward"

# Single stream, reverse
iperf3 -c 10.0.1.50 -t 30 -O 3 -i 1 -T "DX-1Stream-Reverse" -R

# Eight streams — establishes aggregate capability
iperf3 -c 10.0.1.50 -P 8 -t 30 -O 3 -i 1 -T "DX-8Streams"

# Eight streams, reverse
iperf3 -c 10.0.1.50 -P 8 -t 30 -O 3 -i 1 -T "DX-8Streams-Reverse" -R

If one stream is limited but eight streams aggregate to near circuit capacity, the connection is healthy and the finding belongs to the application team rather than the network team. That distinction saves a great deal of time.

Important: if the application uses a single TCP connection, aggregate throughput is irrelevant to it. Fixing this means changing the application to parallelize transfers, not changing the circuit.

Check MTU before blaming anything else

MTU mismatch is the quietest performance problem on Direct Connect, because it rarely breaks connectivity outright. Small packets succeed, large transfers stall or run far below expectation.

Direct Connect supports jumbo frames, but the supported values differ by virtual interface type:

The 8500 value for transit VIFs catches people who configured 9001 everywhere on the assumption it applied uniformly. If traffic crosses a Transit Gateway, 9001 is not the right number.

Verify the real path MTU rather than the configured one, using packets with the do-not-fragment bit set:

# Linux: 8972 payload + 28 bytes overhead = 9000
ping -M do -s 8972 -c 4 10.0.1.50

# Step down until it succeeds to find the real path MTU
ping -M do -s 1472 -c 4 10.0.1.50

# Windows
ping -f -l 8972 10.0.1.50

Also confirm the instance interface itself is configured for jumbo frames. An instance defaulting to 9001 while the VIF is set to 1500, or the reverse, produces exactly the symptom you are chasing.

Note that enabling jumbo frames on a VIF can trigger an update to the underlying physical connection, which disrupts connectivity for all virtual interfaces on that connection for up to thirty seconds. Schedule it.

Why MTR tells you less here than you expect

Run MTR across a Direct Connect private VIF and the output is short and unhelpful. You see your own equipment, the Direct Connect router, and then very little before the destination.

This is expected. The AWS side of the path does not expose intermediate hops the way a transit provider does. Applying the standard reading rule — start at the destination and work backward — the useful question is narrow: does the final hop show loss, and does the latency match the expected distance to the Direct Connect location?

Within those limits, MTR still earns its place:

• It confirms whether loss exists end to end, which throughput tests alone cannot separate from congestion.
• It measures baseline latency to the DX location, which should closely match the physical distance.
• Run from both ends, it exposes asymmetry between the on-premises and AWS return paths.
• TCP mode on the application’s real port reveals policy differences that ICMP hides.

# On-premises toward AWS, ICMP baseline
mtr --report --report-cycles 200 --show-ips --no-dns 10.0.1.50

# On-premises toward AWS, TCP on the real application port
mtr --report --report-cycles 200 --tcp --port 443 --show-ips --no-dns 10.0.1.50

# From the EC2 instance back toward on-premises
mtr --report --report-cycles 200 --tcp --port 443 --show-ips --no-dns 192.0.2.25

Use the port your application actually uses. Probing an arbitrary port tests a path that no production traffic takes, which defeats the purpose of TCP mode.

Security groups and network ACLs will affect these results. A TCP probe to a port the security group does not permit fails at the instance, not in the network, and that failure looks like loss at the final hop.

On Direct Connect, BGP is the diagnostic layer

Because the path is largely invisible to traceroute, routing state carries the information a per-hop view would otherwise provide. When traffic takes an unexpected route or fails over unexpectedly, BGP is where that shows up.

Things worth checking:

• Session state on every VIF, and whether any session has recently flapped
• Prefixes advertised from on-premises, and prefixes received from AWS
• Local preference communities, which control which path AWS prefers for return traffic
• AS path length where multiple connections or providers exist
• Whether traffic is using the connection you believe it is using

Two limits are worth knowing before troubleshooting a session that will not stay up. Direct Connect accepts a maximum of 100 routes per BGP session on a private or transit VIF — advertise more and the session goes to idle, which takes down all traffic on that VIF until the count is reduced. Public VIFs allow 1,000.

Asymmetry deserves particular attention. Return-path selection from AWS is driven by BGP attributes you control from the on-premises side. A performance problem that appears in only one direction is frequently a routing preference issue rather than a capacity issue, and the reverse MTR test is what surfaces it.

Read the CloudWatch metrics that matter

Direct Connect publishes metrics under the DX namespace at 30-second intervals, aggregated by CloudWatch to one or five-minute intervals. These are the ones worth graphing during a performance investigation:

The correlation to look for is simple. If throughput is disappointing while ConnectionBpsEgress sits well below port speed and ConnectionErrorCount is zero, the circuit is not the constraint. Look at flows, instance limits, and MTU instead.

On a hosted virtual interface, the partner may have provisioned capacity below the physical port speed, and you will not see their side of it. Confirm the provisioned rate with the connection owner before assuming the full port is available.

Where the ceiling actually sits

Several distinct limits can produce identical symptoms. Work down this list in order:

Only the last two are Direct Connect problems in the sense that AWS or the provider can act on them. The rest are yours to fix, and identifying which one you have is most of the work.

Capture packets when the numbers disagree

When throughput tests and path tests point in different directions, a capture from both ends resolves it:

# On-premises host
sudo tcpdump -i eth0 host 10.0.1.50 and port 443 -w onprem-capture.pcap

# EC2 instance, simultaneously
sudo tcpdump -i ens5 host 192.0.2.25 and port 443 -w aws-capture.pcap

Capture at both ends at the same time. A one-sided capture cannot distinguish a packet that was never sent from one that was sent and lost, which is usually the question.

What to look for:

• Retransmissions and duplicate ACKs indicating real loss
• Fragmentation or ICMP fragmentation-needed messages indicating MTU trouble
• Zero-window advertisements indicating a receiver that cannot keep up
• Window scaling being negotiated, and whether the window actually grows
• Packets present in one capture and absent from the other, which localizes loss to the segment between them

A repeatable sequence

# 1. Confirm instance type, ENA driver, and interface MTU
ethtool -i ens5 && ip link show ens5

# 2. Verify real path MTU with DF set
ping -M do -s 8972 -c 4 10.0.1.50

# 3. Single-stream baseline, both directions
iperf3 -c 10.0.1.50 -t 30 -O 3 -i 1 -T "DX-1Stream-Forward"
iperf3 -c 10.0.1.50 -t 30 -O 3 -i 1 -T "DX-1Stream-Reverse" -R

# 4. Parallel streams, both directions
iperf3 -c 10.0.1.50 -P 8 -t 30 -O 3 -i 1 -T "DX-8Streams"
iperf3 -c 10.0.1.50 -P 8 -t 30 -O 3 -i 1 -T "DX-8Streams-Reverse" -R

# 5. Path check on the real application port, both directions
mtr --report --report-cycles 200 --tcp --port 443 --show-ips --no-dns 10.0.1.50

# 6. Correlate against CloudWatch DX metrics for the same window

# 7. Capture both ends only if the results still disagree

What to document

• Direct Connect connection ID, VIF ID, and VIF type
• Port speed, and provisioned rate if the VIF is hosted
• Whether traffic crosses a virtual private gateway, Direct Connect gateway, or Transit Gateway
• EC2 instance type, Availability Zone, and configured MTU
• Configured VIF MTU and measured path MTU
• Single-stream and parallel-stream results in both directions
• CloudWatch throughput, error count, and optical levels for the same window
• BGP session state and prefix counts
• Whether the application uses one connection or many
• A healthy baseline from before the problem, if one exists

That last item carries more weight than anything else when escalating. A report showing that a path changed gets a different response than one asserting a path is slow.

Before escalating: confirm the result is not a single-flow limit, not an instance ceiling, and not an MTU mismatch. These three account for the large majority of Direct Connect performance tickets, and all three are resolved on the customer side.

Final takeaway

Direct Connect performance work is mostly a process of elimination, and the tools tell you different things than they do on a general network path.

Test one flow and many flows. The gap between them is the single-flow limit, and it explains most complaints.

Size the test instance deliberately. An undersized instance measures itself and nothing else.

Check MTU early. 9001 on private VIFs, 8500 on transit VIFs, and verify the real path rather than the configuration.

Expect MTR to be quiet. The AWS side does not expose hops. Use it to confirm loss and latency, not to locate a bad router.

Treat BGP as the path view. Routing state carries the information traceroute cannot.

Correlate with CloudWatch. If throughput is low and the circuit is not saturated and errors are zero, the circuit is not the problem.

// the circuit is the last thing to suspect. the flow, the instance, and the MTU are the first three.