Traceroute: the middle hops lie and the last one does not

A traceroute with asterisks in the middle gets forwarded to a hosting provider every day, with a note saying hop 7 is dropping packets. Hop 7 is almost never dropping packets. It is declining to answer you, which is a completely different thing, and it is doing it on purpose.

Understanding why turns traceroute from a source of false leads into the tool that settles arguments about whose network is at fault.

What the tool is actually doing

Traceroute sends packets with a deliberately small time-to-live. The first has a TTL of 1, so the first router decrements it to zero, discards it, and sends back an ICMP "time exceeded" message that includes its own address. Then TTL 2, and the second router does the same. Repeat until something answers from the destination.

Read that again with the emphasis in the right place: each line is a router that chose to generate an error message for you. Nothing in the process measures whether that router forwards real traffic correctly. Those are two separate functions on two separate parts of the hardware.

On serious routers, forwarding happens in silicon at line rate, and generating ICMP happens on a general-purpose processor that also runs the routing protocols holding the internet together. That processor is protected. Rate limiting ICMP generation, or deprioritising it, is standard, correct configuration. A router doing this will show 100% loss in your traceroute while passing every packet of your actual traffic.

The rule that makes the output readable

Read from the bottom.

Loss at a middle hop that does not appear at the hops below it is not loss. If hop 7 shows 60% and hops 8, 9 and the destination show 0%, then everything traversed hop 7 successfully. Hop 7 was too busy to chat.

Loss that starts at a hop and persists all the way down is real. That is the only pattern worth reporting, and the first hop where it begins is where to point.

Latency behaves the same way. A hop showing 300 ms when the destination shows 40 ms has not added 260 ms to anything. It generated its reply slowly. Only a latency increase that carries through to the bottom is a latency increase.

And one thing you cannot see at all: the return path. Each measurement is a round trip, and the way back from hop 7 is not the way back from hop 8. Routing on the internet is asymmetric by default, so a bad return path from one intermediate router colours a line in a way that has nothing to do with your traffic. This is also why a traceroute from you to the server tells you nothing about the path from the server back to you, and why the second half of any real investigation is asking the other end to run one in the opposite direction.

Trace the protocol you actually use

The defaults differ by platform, and they matter. Linux traceroute sends UDP to high ports. Windows tracert sends ICMP echo. Neither is what your website traffic looks like, and firewalls treat all three differently. A path that appears broken over UDP is frequently perfect over TCP to port 443, because the middle of the internet is full of devices that filter unfamiliar UDP and pass web traffic without a thought.

sudo traceroute -T -p 443 example.com

That sends TCP SYN packets to port 443, which is the traffic you care about. It needs root, and it is worth it. If a UDP trace dies at hop 9 and a TCP trace on 443 completes cleanly, there was never a problem.

mtr, which is what you should be sending your host

A traceroute is one sample. Network problems are statistical. Sending a single traceroute to a provider is like reporting a slow page from one page load.

mtr -rwzbc 200 -T -P 443 example.com

That runs two hundred cycles and prints a report: for each hop, the loss percentage, the number of packets sent, and the best, average, worst and standard deviation of latency. The flags are worth knowing individually. -r gives a report rather than a live display, so it pastes into a ticket. -w keeps long hostnames intact. -z adds the AS number of each hop, which tells you which network you are in and therefore whose problem it is. -b shows both name and address. -T -P 443 uses TCP, for the reason above.

Two hundred cycles takes a few minutes. Run it while the problem is happening, and run it again when it is not, so you have a baseline. A support ticket containing both, from both directions, gets a real engineer's attention. A single screenshot with red numbers in the middle gets a templated reply about how ICMP is deprioritised, which will be correct.

When to reach for it, and when not to

Reach for it when the failure is a timeout rather than a refusal, when it depends on where you are testing from, or when a client on one network sees something nobody else sees. Silence on the wire is the traceroute case: something is dropping packets and you need to know whose equipment. A connection that is actively refused, on the other hand, has already told you the host is alive and reachable, and no amount of tracing will add to that; the distinction between refused, timed out and unresolved is the first fork in connection refused.

Do not reach for it for slow pages. If the connection establishes in 30 ms and the first byte takes eight seconds, the network delivered your request promptly and the application spent the time. A traceroute will show a healthy path and you will have learned nothing you did not already know.

curl -sS -o /dev/null -w 'dns %{time_namelookup}  connect %{time_connect}  ttfb %{time_starttransfer}\n' \
  https://example.com/

Run that first, every time. It splits the total into name resolution, connection and server thinking time, and it tells you in one line which of the three deserves the next hour. A large time_namelookup sends you to the resolver rather than the route, and resolution has its own set of failures that look like network problems and are not: records are answered from caches that expire on a schedule.

The part that concerns anyone watching sites for other people

Route problems are geographic and they are transient. A peering dispute or a fibre cut affects visitors on one network, in one region, for a few hours, and it is completely invisible from your office and from a monitoring probe in a single data centre. The client's customers in one city cannot reach the site. Your dashboard is green and it is not wrong.

Checking from more than one location is the only thing that catches this, and it is worth being honest about what you gain: not faster detection of real outages, which any single probe finds, but the ability to tell a client that the problem is on their visitors' route rather than on their site. That answer is worth having, because the alternative is spending a day looking for a fault in a server that never had one.

One ordering note, because it saves the most time: resolve the name before you trace the route. A trace to the wrong address is a perfectly valid trace, and it will look like a network fault for as long as you believe the address. On a machine that is not yours, the tool that tells you what that machine resolves is nslookup, in one-shot form.