A throughput test tells you that a path is slow. It does not tell you where the problem is. mtr is usually the next tool to reach for, because it combines traceroute and ping into a single continuously updating view of every hop between two endpoints.
It is also one of the most frequently misread tools in networking. A single ten-second run showing forty percent loss at hop six looks alarming and very often means nothing at all. The value of MTR comes from running it correctly, running it in both directions, and knowing which loss is real.
What MTR actually measures
mtr sends packets with incrementing TTL values, exactly as traceroute does. Each router that decrements the TTL to zero returns an ICMP Time Exceeded message, which reveals its address. Unlike traceroute, MTR repeats this continuously and accumulates statistics for every hop.
The result is a table with one row per hop showing loss percentage, packets sent, and latency distribution. That table describes:
- The forward path from the source toward the destination
- The latency observed when each hop responded
- The proportion of probes each hop did not answer
- How much latency varied across the sampling period
What it does not describe is equally important. MTR shows the forward path only. The response from each intermediate hop travels back along whatever return path that router chooses, and you have no visibility into that return path from the source side.
The central rule: loss reported at an intermediate hop that does not persist through to the destination is almost never real packet loss. It usually means the router deprioritized or rate-limited its own ICMP responses while forwarding transit traffic normally.
Reading the output columns
A report-mode run produces output in this form:
HOST: workstation01 Loss% Snt Last Avg Best Wrst StDev 1.|-- 192.168.1.1 0.0% 200 0.6 0.7 0.5 2.1 0.2 2.|-- 10.240.0.1 0.0% 200 3.4 3.6 3.1 12.4 0.7 3.|-- 172.20.14.9 28.5% 200 8.2 8.4 7.9 19.6 1.1 4.|-- 203.0.113.17 0.0% 200 14.1 14.3 13.8 28.2 1.4 5.|-- 198.51.100.44 0.0% 200 22.7 22.9 22.1 41.5 2.0
Each column answers a different question:
| Column | What it means |
|---|---|
Loss% |
Percentage of probes that hop did not answer. Meaningful only on the final hop unless it persists downward. |
Snt |
Probes sent to that hop. Small values make the loss percentage statistically meaningless. |
Last |
Round-trip time of the most recent probe. |
Avg |
Mean round-trip time across all probes to that hop. |
Best |
Lowest observed round-trip time. Approximates the unloaded path latency. |
Wrst |
Highest observed round-trip time. A single outlier can dominate this value. |
StDev |
Standard deviation of latency. High values indicate jitter, queueing, or inconsistent handling. |
Read Best and Avg together. If they are close, the path is stable. A large gap between them, or a high StDev, indicates variable queueing somewhere at or before that hop.
Why intermediate loss usually is not loss
In the sample output above, hop 3 shows 28.5 percent loss while hops 4 and 5 show none. This is the single most misdiagnosed pattern in network troubleshooting.
Generating an ICMP Time Exceeded message is control-plane work. The router must punt the packet to its CPU rather than forwarding it in hardware. Nearly every production router rate-limits this work to protect itself. When it hits that limit, it silently declines to answer some probes while continuing to forward transit traffic at line rate.
The consequence is straightforward. If a hop shows loss but every hop after it shows none, traffic passed through that router successfully. Only loss that begins at a hop and continues through to the destination reflects packets actually being dropped.
Intermediate loss can appear because of:
- ICMP rate limiting on the control plane
- Control-plane policing configured by the operator
- Routers configured not to generate Time Exceeded messages at all
- Firewalls or providers filtering ICMP selectively
- Load balancers or anycast responders behaving inconsistently
- CPU pressure on the responding device unrelated to forwarding
Important: read the loss column from the bottom up. Start at the destination. If the final hop shows no loss, the path is not dropping traffic regardless of what intermediate hops report.
Use report mode for anything you intend to share
Interactive mode is useful for watching a problem develop. Report mode is what belongs in a ticket, because it produces a fixed, complete sample that another engineer can read without context.
mtr --report --report-cycles 200 --show-ips --no-dns 198.51.100.44
The options used are:
| Option | Purpose |
|---|---|
--report |
Runs a fixed number of cycles and prints a summary table |
--report-cycles 200 |
Sends 200 probes per hop instead of the default 10 |
--show-ips |
Displays IP addresses alongside resolved hostnames |
--no-dns |
Skips reverse DNS, which avoids delays and misleading stale names |
The default of ten cycles is not a sample. It is a glance. Ten probes cannot distinguish one percent loss from zero, and intermittent problems will not appear at all. Two hundred cycles takes a few minutes and produces a result worth acting on.
Choose the protocol deliberately
By default MTR probes with ICMP. Many networks treat ICMP differently from the traffic you actually care about, which means an ICMP-based result can misrepresent the path your application takes.
MTR can also probe with UDP or TCP:
# Default ICMP probes mtr --report --report-cycles 200 198.51.100.44 # UDP probes mtr --report --report-cycles 200 --udp 198.51.100.44 # TCP probes to a specific service port mtr --report --report-cycles 200 --tcp --port 443 198.51.100.44
TCP mode is often the most representative test. It probes the same port your application uses, so it traverses the same firewall rules, the same QoS classification, and frequently the same ECMP hash result as production traffic.
Comparing protocols is itself diagnostic. If ICMP shows loss but TCP to port 443 is clean, the loss is in ICMP handling rather than in the data path. If TCP to a specific port fails while ICMP succeeds, the problem is likely policy rather than transport.
| Protocol | When to use it |
|---|---|
| ICMP | General path discovery and a first look at latency |
| UDP | Approximating real-time or DNS traffic, and probing paths that filter ICMP |
| TCP | Matching application traffic through firewalls, QoS policies, and ECMP hashing |
Always run MTR from both ends
This is MTR’s most important limitation and the one most often ignored. A trace from A to B shows the forward path from A. It does not show the path traffic takes returning from B to A, and internet routing is frequently asymmetric.
Run the same test from the far end:
# From the local endpoint toward the remote endpoint mtr --report --report-cycles 200 --tcp --port 443 198.51.100.44 # From the remote endpoint back toward the local endpoint mtr --report --report-cycles 200 --tcp --port 443 203.0.113.88
Two traces taken from opposite ends of the same path routinely disagree, because they are describing different sets of routers. Without both, you can spend a long time asking a provider about a hop that carries traffic in only one direction.
When you cannot get shell access on the far end, looking glass servers operated by carriers and exchanges provide a reasonable substitute for the reverse view.
How to interpret common patterns
| Observed pattern | What it may indicate |
|---|---|
| Loss at one hop, none afterward | ICMP rate limiting or control-plane policing. Not forwarding loss. |
| Loss begins at a hop and continues to the destination | Probable real loss at or beyond that hop. Worth escalating. |
| Latency jumps at one hop and stays elevated | Expected across a long-distance or satellite link. Compare against known distance. |
| Latency jumps at one hop and returns to normal after | Slow control-plane response at that router, not path latency. |
| High StDev across later hops | Queueing, congestion, or buffer bloat somewhere at or before the first affected hop. |
Hop shows as ??? but later hops respond |
Router configured not to answer, or ICMP filtered. Not a break in the path. |
| Addresses at one hop change between runs | ECMP hashing across parallel paths. Probes are not all taking the same route. |
| Everything clean but the application is slow | Look at TCP behavior, endpoints, or the application. MTR has ruled out path loss. |
ECMP and the paths MTR cannot see
When multiple equal-cost paths exist, routers hash each flow across them. MTR’s probes may not all hash identically, which means the displayed path can be a blend of several real paths rather than any single one.
This shows up as a hop whose address changes between runs, or alternating addresses within one run. It also means an intermittent problem on one of several parallel links will appear as partial loss rather than total failure.
Two practical responses:
- Use TCP mode with a fixed destination port. A consistent flow tuple is more likely to hash consistently and to match your application’s actual path.
- Run the test several times. If the path is stable across runs, ECMP variation is not confusing the result.
MPLS is a related blind spot. Traffic crossing a provider’s MPLS core may traverse several routers that never appear as hops, because label switching does not decrement the TTL in the way the tool depends on. Where the provider exposes them, MTR can display label information:
mtr --report --report-cycles 200 --mpls 198.51.100.44
A trace that appears to cross a continent in one hop is usually an MPLS core, not a single physical link.
A repeatable troubleshooting sequence
# 1. Confirm the installed version mtr --version # 2. Baseline ICMP trace, long enough to mean something mtr --report --report-cycles 200 --show-ips --no-dns 198.51.100.44 # 3. Repeat with TCP on the application port mtr --report --report-cycles 200 --tcp --port 443 --show-ips --no-dns 198.51.100.44 # 4. Run the same TCP test from the remote endpoint mtr --report --report-cycles 200 --tcp --port 443 --show-ips --no-dns 203.0.113.88 # 5. Repeat the forward test to check for ECMP path variation mtr --report --report-cycles 200 --tcp --port 443 --show-ips --no-dns 198.51.100.44 # 6. Capture machine-readable output for the ticket mtr --report --report-cycles 200 --json 198.51.100.44 > mtr-forward.json
Steps 3 and 4 together are the ones that resolve most disputes with a provider. A forward trace alone invites the response that the problem is on your side.
What to document with every test
An MTR result without context is difficult to compare or escalate. Record the following:
- Source and destination hostnames and IP addresses
- MTR version and operating system on both endpoints
- Date, time, and time zone of each run
- Direction of the test
- Protocol and destination port used
- Number of report cycles
- Whether the problem was occurring during the capture
- A matching capture taken while the path was healthy, for comparison
- Known ECMP, MPLS, or asymmetric routing in the path
- Any related throughput or application measurements
The healthy baseline matters more than it appears to. Providers respond very differently to a report showing that a path changed than to one asserting that a path is bad.
For automated collection or long-term comparison, use structured output:
mtr --report --report-cycles 200 --json 198.51.100.44 > mtr-result.json mtr --report --report-cycles 200 --xml 198.51.100.44 > mtr-result.xml
Operational considerations
MTR is lightweight, but it is not free of consequences:
- Long runs against a busy router generate sustained control-plane load. Use a reasonable interval rather than the fastest possible.
- Raw socket access is required, so MTR typically needs elevated privileges or a capability grant.
- Probing infrastructure you do not own can trigger security alerting. Coordinate before running extended tests against a third party.
- Reverse DNS results are frequently stale and can attribute a hop to the wrong network. Verify against routing data rather than trusting hostnames.
Escalation warning: before reporting a hop to a provider, confirm the loss persists to the destination and confirm it appears in both directions. Reports based on intermediate ICMP loss are the most common reason a ticket gets closed without action.
Final takeaway
mtr is not a tool for proving a network is broken. It is a tool for narrowing where a problem could be, and for ruling out the path entirely when the path is fine.
Read loss from the destination backward. Intermediate loss that does not persist is control-plane behavior, not forwarding loss.
Run enough cycles. Ten probes cannot characterize a path. Two hundred can.
Match the protocol to the traffic. TCP on the application port reflects reality better than ICMP does.
Test both directions. A single-sided trace describes half the path and invites the wrong conclusion.
Keep a healthy baseline. Change is far easier to escalate than a bare assertion that something is slow.
Know what it cannot see. ECMP variation, MPLS cores, and return paths are all invisible or misleading from one endpoint.
// the loss column is not a list of suspects. it is a list of routers that were busy answering you. read it from the bottom up.