ICMP Packet Loss: Why Ping Doesn’t Always Tell the Truth

Kerry Cordero
Routing & Switching
12 min read

Introduction

One of the most common mistakes I see in enterprise network troubleshooting is engineers running a ping across the Internet, observing packet loss, and immediately concluding that there is a network problem.

This happens at all experience levels, including among engineers who have spent years working with enterprise networks.

Ping is one of the oldest and most useful troubleshooting utilities available. However, interpreting its results requires understanding how modern routers process traffic and how Internet service providers protect their infrastructure.

ICMP packet loss does not automatically mean production traffic is experiencing packet loss.

Before declaring an issue, engineers must understand what they are testing, how the device processes that traffic, and whether the observed behavior affects actual application flows.

Let’s examine the technical reasons.

1. Understanding the Control Plane and Data Plane

Modern enterprise and service-provider routers separate traffic processing into different functions.

Control plane

The control plane is responsible for functions such as:

  • Routing protocol processing, including BGP and OSPF.
  • Routing information and route selection.
  • Management and monitoring services.
  • Processing certain ICMP messages addressed to the router.
  • Handling packets requiring special processing or exceptions.

Control-plane traffic often requires processing resources separate from the router’s normal high-speed forwarding path.

Data plane

The data plane is responsible for forwarding packets through the network.

Modern routers generally use dedicated forwarding hardware, such as ASICs or network processing units, to forward transit traffic at high rates.

Consider a service-provider router with multiple 100 Gbps interfaces. It may be forwarding hundreds of gigabits of production traffic while applying strict limits to packets addressed to the router itself.

That distinction is fundamental.

A router can be forwarding production traffic without packet loss while simultaneously dropping ICMP requests directed at its own interfaces.

One important technical clarification: ICMP itself is not exclusively control-plane traffic. ICMP packets traversing a router can be handled by its data plane. The distinction depends on whether the traffic is being forwarded through the device or processed by the device.

2. Why ISPs Rate-Limit ICMP

Service providers must protect their routing infrastructure against excessive or malicious traffic.

One common mechanism is Control-Plane Policing (CoPP).

CoPP uses traffic classification and policing policies to control how much traffic can reach protected control-plane resources.

Depending on the network platform and configuration, these policies may restrict:

  • ICMP Echo Requests addressed to router interfaces.
  • ICMP Time Exceeded messages generated during traceroute.
  • Management traffic.
  • Routing protocol traffic, with separate classifications and limits.
  • Other packets requiring exception processing.

For example, a service provider may configure relatively restrictive ICMP policing while reserving sufficient processing resources for critical routing protocols.

If ICMP traffic exceeds the configured threshold, some requests may be dropped or responses suppressed.

This does not necessarily affect ordinary transit traffic.

In fact, this is intentional behavior. Protecting the router’s control plane helps maintain the stability of the network.

The IETF acknowledges ICMP rate limiting in RFC 1812, and network equipment manufacturers document the same behavior in their control-plane protection implementations.

3. A Practical Traceroute Example

Consider the following simplified traceroute results.

Hop    Latency      Packet Loss
---------------------------------
1       1 ms          0%
2       8 ms          0%
3      22 ms         35%
4      26 ms          0%
5      31 ms          0%
6      35 ms          0%

An engineer might see 35% packet loss at Hop 3 and immediately escalate the issue to the ISP.

But look carefully.

Hop 3 shows substantial loss when responding to probes directed at that hop.

However, Hops 4, 5, and 6 respond without packet loss.

What does that tell us?

The probes reaching the later hops must traverse the earlier portion of the path, assuming the probes follow the same forwarding path.

If Hop 3 were consistently discarding 35% of the transit traffic along that path, we would generally expect corresponding loss at subsequent hops.

Instead, the evidence suggests that Hop 3 may be deprioritizing or rate-limiting its own ICMP responses.

The router can still be forwarding traffic normally.

The key troubleshooting principle

Packet loss at an intermediate hop that does not continue to subsequent hops is not sufficient evidence of transit packet loss.

This is particularly important when using tools such as traceroute, MTR, and monitoring platforms that visualize hop-by-hop latency and loss.

There are exceptions involving ECMP, path changes, probe classification, and return-path behavior. Therefore, downstream results provide useful evidence, not an absolute mathematical guarantee.

4. Understanding Traceroute and TTL Expiration

Traceroute works by transmitting packets with progressively increasing IP Time to Live (TTL) values.

For example:

  1. A probe with TTL 1 expires at the first router.
  2. A probe with TTL 2 expires at the second router.
  3. A probe with TTL 3 expires at the third router.
  4. The process continues until the destination is reached or the configured limit is exceeded.

When a transit router decrements the TTL to zero, it normally discards that packet and generates an ICMP Time Exceeded response.

That response allows traceroute to identify the intermediate device.

However, the response is generated by the router, rather than being an ordinary forwarded transit packet.

The router may rate-limit these responses.

As a result, traceroute can show missing responses or elevated latency at intermediate hops even when actual application traffic is forwarding normally.

Furthermore, not all traceroute implementations use the same probe protocol. Depending on the operating system and configuration, probes may use UDP, ICMP, or TCP.

This can affect the path selected by load-balancing devices and how network security policies treat the traffic.

5. Why High Latency at One Hop Can Also Be Misleading

Packet loss isn’t the only metric engineers sometimes misinterpret.

Consider another example.

Hop    Latency
----------------
1       1 ms
2       9 ms
3     180 ms
4      28 ms
5      32 ms
6      35 ms

Hop 3 reports 180 milliseconds of latency.

Should we assume the router is introducing an additional 150 milliseconds of forwarding delay?

No.

The reported latency measures the round-trip time associated with that hop’s response. It does not isolate the forwarding delay introduced by that router.

The device may delay generating or scheduling its response while continuing to forward transit packets efficiently.

Subsequent hops showing normal latency strongly suggest that the elevated response time at Hop 3 is not being imposed on all forwarded traffic.

This is another example of why engineers must distinguish between the performance of a router’s diagnostic responses and its forwarding performance.

6. When ICMP Packet Loss Should Be Investigated

None of this means engineers should ignore ICMP packet loss.

Ping can be extremely valuable when used correctly.

Consider these scenarios.

Scenario A: Intermediate-hop loss only

Hop 3:          40% loss
Hop 4:           0% loss
Hop 5:           0% loss
Destination:     0% loss

Interpretation: Likely ICMP response policing, deprioritization, or another hop-specific response behavior.

There is no demonstrated end-to-end packet loss.

Scenario B: Loss begins and continues downstream

Hop 3:           0% loss
Hop 4:          15% loss
Hop 5:          15% loss
Destination:    15% loss

Interpretation: Potential forwarding loss near Hop 4, but additional investigation is necessary.

The evidence becomes stronger if the loss is reproducible, the probes follow a consistent path, and application traffic is also affected.

Scenario C: Destination-only ICMP loss

Hop 3:           0% loss
Hop 4:           0% loss
Hop 5:           0% loss
Destination:    20% loss

Interpretation: The destination may be rate-limiting ICMP, or genuine loss may exist near the endpoint or on the return path.

This result should not automatically be classified as an ISP forwarding issue.

Scenario D: Packet loss with application degradation

Suppose engineers observe:

  • Intermittent application timeouts.
  • Elevated TCP retransmissions.
  • Reduced throughput.
  • Increased connection establishment time.
  • Sustained packet loss to a responsive endpoint.

Now there is meaningful evidence supporting a potential network performance problem.

The next step is identifying the responsible segment, device, direction, or application component.

7. ECMP and Asymmetric Routing Complicate the Picture

In modern enterprise networks, Equal-Cost Multipath (ECMP) routing is common.

Networks may use multiple active paths across:

  • Internet transit providers.
  • Data center interconnects.
  • Cloud connectivity.
  • WAN architectures.
  • Service-provider backbones.

Depending on the device, ECMP forwarding decisions may consider source and destination IP addresses, transport protocols, and source and destination ports.

Traceroute probes can produce different flow hashes, potentially traversing different ECMP paths.

Consequently, hop-by-hop results can become misleading.

A traditional traceroute may combine observations from different forwarding paths into what appears to be one continuous route.

Asymmetric routing introduces another complication.

The forward path taken by a probe and the return path taken by an ICMP response may be different.

An elevated round-trip time or missing ICMP response does not independently identify which direction experienced a problem.

For difficult cases, engineers should consider flow-consistent probing, such as Paris traceroute techniques, and compare forward- and reverse-path telemetry where available.

8. How Engineers Should Actually Troubleshoot Internet Performance

Rather than immediately escalating an ICMP result, engineers should follow a repeatable troubleshooting methodology.

Step 1: Identify the actual symptom

Start by determining what is failing.

Is the user experiencing slow application response times, failed connections, intermittent timeouts, or reduced throughput?

Or did a monitoring system simply report ICMP loss?

These are different situations and should not be treated as equivalent.

Step 2: Identify what the test is measuring

Determine whether the engineer is testing:

  • A router interface.
  • An intermediate ISP hop.
  • A remote server.
  • A load balancer.
  • An application endpoint.

Understand whether the tested device responds to ICMP consistently and whether its response path is representative of application traffic.

Step 3: Validate using TCP

When troubleshooting a TCP-based application, test the relevant protocol and destination port.

For example, on Linux:

nc -vz example.com 443

This verifies whether a TCP connection can be established, but it does not measure ongoing packet loss.

For repeated TCP path measurements, MTR can be used in TCP mode:

sudo mtr -T -P 443 example.com

Remember that intermediate devices can also limit traceroute-related responses, even when the probes use TCP.

Step 4: Perform application-layer testing

For an HTTPS application:

curl -o /dev/null -s \
  -w 'DNS: %{time_namelookup}\n\
TCP: %{time_connect}\n\
TLS: %{time_appconnect}\n\
TTFB: %{time_starttransfer}\n\
Total: %{time_total}\n' \
  https://example.com

This provides application transaction timing and can help distinguish DNS, TCP connection, TLS negotiation, and server response delays.

The measurements are cumulative timestamps, not independent phase durations.

Step 5: Correlate with network telemetry

Depending on the environment, examine:

  • Interface errors and discards.
  • Queue congestion and QoS drops.
  • TCP retransmissions and round-trip-time trends.
  • Link utilization.
  • Routing changes and ECMP behavior.
  • Firewall or load-balancer session statistics.
  • Flow telemetry and packet captures.

TCP retransmissions are useful supporting evidence, but retransmissions alone do not prove an ISP fault. They must be correlated with the actual traffic path and possible causes.

Step 6: Establish evidence before escalation

Before opening an ISP incident, engineers should be able to document:

  1. The affected application or destination.
  2. The source network and actual traffic path.
  3. The observed symptom and its timing.
  4. Evidence of sustained or reproducible performance degradation.
  5. Supporting data distinguishing diagnostic response behavior from actual forwarding loss.

An escalation supported by multiple correlated observations is far more effective than one based solely on an intermediate-hop ping.

9. What Senior Network Engineers Should Understand

Troubleshooting is not simply about running commands.

It requires understanding what those commands measure.

A network engineer should know the differences between:

  • Control-plane processing and data-plane forwarding.
  • ICMP Echo Replies and ICMP Time Exceeded messages.
  • Traffic policing and actual interface congestion.
  • Intermediate-hop response loss and end-to-end packet loss.
  • Round-trip-time measurements and per-hop forwarding delay.
  • TCP retransmissions and ICMP diagnostic results.
  • Application-level symptoms and network-level observations.

The objective is not to memorize every command or every platform-specific implementation.

It is to understand the underlying networking principles well enough to interpret the evidence correctly.

Senior engineers also have a responsibility to help develop these skills in less-experienced engineers.

Simply reproducing the same ping test does not validate the original conclusion.

Conclusion: Understand What You’re Measuring

Ping and traceroute remain essential network diagnostic tools. They are simple, widely available, and extremely valuable when interpreted correctly.

But they have limitations, particularly across the public Internet, where engineers do not control intermediate routing infrastructure or ICMP response policies.

A router dropping diagnostic packets directed at itself is not necessarily dropping production traffic.

A traceroute showing high latency at one intermediate hop does not necessarily indicate forwarding delay.

And a monitoring platform reporting ICMP loss is not automatically evidence of degraded application performance.

The correct approach is to understand the network architecture, distinguish control-plane behavior from data-plane forwarding, and correlate multiple measurements before reaching a conclusion.

The most important networking skill isn’t knowing which command to run. It’s understanding what the output actually proves.

Technical References