Functionality: Traceroute shows all Layer 3 (routing layer) hops between the hosts. This is achieved by sending packets to the remote destination with increasing TTL (Time To Live) value (starting at 1). The TTL field is a field in the IP packet which gets decreased by one at every router. Once the TTL hits zero, the packet gets discarded and a “TTL exceeded” ICMP message is returned to the sender. This approach is used to avoid routing loops; packets cannot loop continuously because the TTL field will eventually decrement to 0. By default the OS sets the TTL value to a high value (64, 128, 255 or similar), so this should only ever be reached in abnormal situations.
So traceroute sends packets first with TTL value of 1, then TTL value of 2, etc., causing these packets to expire at the first/second/etc. router in the path. It then takes the source IP/host of the ICMP TTL exceeded message returned to show the name/IP of the intermediate hop. Once the TTL is high enough, the packet reaches the destination, and the destination responds.
The type of packet sent varies by implementation. Under Linux, UDP packets are sent to a high, unused port. So the final destination responds with an ICMP Port Unreachable. Windows and the mtr tool by default use ICMP echo requests (like ping), so the final destinations answers with an ICMP echo reply.
- Last hop on traceroute is not destination: This is true for nearly all external examples. The reason for this is that traceroute performs a reverse DNS lookup for every host in the path. The reverse lookup for the last host might be not implemented (e.g. www.stackoverflow.com) or might be different than the name given for the forward DNS (e.g. www.gnu.org)
- Traceroute shows only stars at the end: This means there is probably a firewall in-between blocking either the incoming UDP/ICMP packets or the outgoing ICMP packets (or both). With some hosts (e.g. www.wikipedia.org) you observe different behaviour with traceroute or mtr, which shows that UDP packets only seem to be discarded.
- Other VMs (even on different continents), www.google.com, www.adcash.com seem only one hop away: This is due to the network virtualization layer. In certain settings, the TTL of the inner packet is never decreased, although there are many physical hosts in-between. www.google.com and www.adcash.com (their website is hosted on Google Cloud Platform) both are cases where a routing happens mostly encapsulated due to packets staying inside the Google (Software Defined) Network.
- Multiple paths showing: Traceroute always sends three packets with the same TTL, and those might be routed over different paths (for example, different MPLS TE paths or ECMP routing). So this is nothing to worry about.
- Traceroute shows stars in the middle: This is because a host in the middle might not respond correctly with TTL exceeded messages or those might be filtered somewhere on the way.
- Traceroute to bad.horse looks funny: This is an intended easter egg and can be built with a bunch of public IPs and virtual routers. See this post on how to create such a traceroute if you’re interested.
