Probably the least glamorous software program on the Internet today is a program titled TRACEROUTE. It's almost universally available, it comes in almost every operating system including the ever more ubiquitous Windows95. But, as part of compiling our Boardwatch Directory of Internet Service Providers, we recently began to ask ISP's for our directory where THEY got THEIR connection to the Internet. We were surprised to have several of them actually tell us that their connection to the Internet was "proprietary" and that they didn't want anyone to know from who or topographically how they got their connection. We found this a bit bizarre in that anyone can run a traceroute and tell in some detail a number of things about the connection within seconds. In the hoopla over the World Wide Web and the many client/server tools available, in some cases the basic knowledge of such common tools as traceroute and ping has somehow been missed. Then too, there has been some recent developments with traceroute that make this a NEW tool in some interesting ways.

So despite it's rather nakedly plain heritage and operation, perhaps it would actually serve some purpose to do an article on TRACEROUTE. Traceroute, in its most basic form, allows you to print out a list of all the intermediate routers between two destinations on the Internet. It allows you to diagram a path through the network.

Why would you want to do that? We can think of a number of reasons. First, almost everyone connected to the Internet has some curiosity about what they are connected to. A traceroute from your desktop machine to almost anywhere else will provide a listing of each machine between you and the destination - essentially diagramming in the first three to five lines YOUR connection to the Internet.

Secondly, many individuals and companies are setting up their own servers of one form or another - typically a web server. Once set up, if the 50 million people alleged to be on the Internet, or even the fifteen million who actually are don't show up, you might want to check to see if any of them CAN make the connection. Despite the ubiquity of the Internet, we are increasingly finding sidespurs to the net where "you can't get there from here - you have to go somewhere else first." You might be surprised to learn that your web site is ONLY available to people on Sprint's backbone, but not on the AGIS backbone., or on the internetMCI backbone, but NOT to those on the Sprint backbone. I'm assured this doesn't actually happen of course. But on the off chance that my ravings are not purely hallucinatory, you might try a few traceroutes to see for yourself.

The ability to make the connection to your web site is a function of proper connectivity across the network. But it is also a function of the efficiency of the domain name server system - a system that is growing increasingly creaky and quirky in operation as it grows without limit, and more importantly without any mechanism for cleaning up all the little DNS messes of the past. Companies come and go, web sites come and go, but DNS is apparently forever as essentially no one ever goes in and cleans any of it up.

In any event, getting a domain name assigned and putting up a server may be the right moves, but you might want to actually check to see if you are available from different backbones and locations. Traceroute will show you with some confidence the path through the network, and it looks up each machine along the way, including yours, in the DNS structure. If you can traceroute to a site, you can almost always connect to a web server there.

Finally, in selecting an Internet Service Provider, backbone connection, or a service to host your web site, many savvy Internauts like to check out the connectivity of the site and precisely how it is connected to whom where. A bit of thought and the traceroute program will actually allow you to get a pretty good idea.

Traceroute was originally written by Van Jacobson of Lawrence Berkeley National Laboratory (van@helios.ee.lbl.gov) and the traceroute code is still freely available from version 1.0 through version 1.3.2 at ftp://ee.lbl.gov/old. The program derived from a conversation with Steve Deering of Stanford University in 1988.

In it's simplest usage, traceroute is simply a command line program where you enter a domain name:

traceroute boardwatch.com

In Windows95, traceroute is renamed TRACERT for reasons only apparent to Microsoftians. It is also only available from a DOS window. Drop to a DOS window and enter TRACERT BOARDWATCH.COM to get essentially the same results.

In the example above, the first machine encountered was 205.139.105.254 in Grants Pass, Oregon. The machine failed a reverse domain name lookup and so only the IP number is listed. This is actually a traceroute from a web server in Grants Pass TO www.boardwatch.com.

The second machine is a border router of MCI's network in Seattle where we join MCI and machine three is one of the MCI core backbone routers in Seattle. In machine four, we connect to Sprint in Stockton California. It's worth noting that we did not go through one of the NAP's apparently such as the PacBell SF NAP or MAE-WEST and so we've most likely discovered a private peering exchange between Sprint and MCI in Stockton California. The concept that all connections between private backbones exist at one of the four NSF funded Network Access Points or NAPS is simply erroneous at this point with hundreds of such private NAPS between two carriers cropping up all over the country at any target of geographic proximity/opportunity.

At lines 6 we pass through what looks like a Stockton router where the interface is designated Cheyenne. At line 7 we enter Sprint's hub facility in Cheyenne and line 8 is the exit point on the way to Colorado Internet Cooperative Association, also known as the Boulder COOP in line 9. Line 10 appears to be a Boulder Coop router. In line 11 we go through an unidentified machines, actually at eSoft, Inc. in Aurora Colorado. Line 12 is a T-1 connection to our gateway router here at Boardwatch and line 13 is the 204.144.169.6 machine we use as a web server on our LAN.

Note the time indications for each line - three time values in milliseconds. These are roundtrip times for each leg, not sums, differences, or otherwise related to each other. From Grants Pass to Littleton, across two major backbones, and BACK looks like about 82 ms, 84 ms, and 74 ms for the three packets involved. Not bad actually. The point is, times are complete roundtrip times for packets to THAT router on THAT line and are unrelated to anything on previous or subsequent lines. Traceroute is a relatively low priority service on most machines, and as such isn't really a very good indicator of roundtrip times. The PING program actually works slightly better for that purpose.

That said, you CAN derive some indication of how busy the Internet is from examining these traceroute roundtrip times. If the three times on any one particular line are fairly close together, this indicates some consistency between the packets. If they are more variable, it usually indicates some dynamic change in traffic levels between the three packets. Similarly, if you get a relatively good time roundtrip to one router, while the very next machine gives a very poor series of times, and then the times are pretty good again on the NEXT router, you are seeing some traffic surges on the network. Smaller pipes tend to be somewhat more sensitive to such traffic surges, while larger ones tend to be a bit more forgiving.

HOW TRACEROUTE WORKS

Traceroute wasn't really designed as any of the normal control message elements of any protocol. It is actually a small program that relies on some predictable response activity of routers and servers on the Internet.

Routers receive Internet Protocol packets, and in fact packets of other protocols of various sizes and flavors, and in most cases simply pass them on. A router will have several interfaces usually. It receives a packet in one interface, opens the packet to find the DESTINATION ADDRESS, and then does a lookup in a routing table to determine which of its other interfaces (to other routers) is the best place to "route" the packet. In basic operation, it's actually not a terribly smart device. To a router with three interfaces, for example, the entire global Internet falls into three categories - Interface 1, Interface 2, and Interface 3. The Interfaces can be serial ports, ethernet cards, V.32 ports, or a specialized smoke signal rapidly-flappable-wet-blanket (RFWB) interface device. It doesn't matter. Open the packet, decide which port, and mail the puppy. In almost all cases, a router is the ultimate finger pointer - "This packet is somebody else's problem, and I'll pass it out port X for them to deal with."

In the early days of TCP/IP networks, it was discovered pretty quickly that if you sent out a packet and there was some configuration anomaly in a single router in the network, you could create a routing loop where packets could enter, but they would never get back out. The packet would simply move from one router to another in a circle. These packets would accumulate in the network quite quickly and seriously slow things down for those packets that were actually going somewhere.

The solution was the addition of an 8-bit Time-To-Live (TTL) field in the Internet Protocol packet header. The sending machine could set this to any value between 0 and 255. Each router that handled the packet decremented this value in the packet header by one when it passed on the packet. If it received a packet that had a TTL of 0 or 1, instead of passing on the packet, it killed it. In this way, after a set number of "hops" through routers, in no case larger than 255, a packet died and was removed from the network. In practice, TTL is the maximum number of hops a packet can transit before death. The 255 hop total is also the maximum "span" of the Internet today. If you are more than 255 hops from anywhere, you can't get there from here using the existing Internet Protocol.

When a router kills a packet, it also sends out an Internet Control Message Protocol (ICMP) error message to the packet originator address indicating TIME EXCEEDED IN TRANSIT. This message contains the IP address of the router sending the error message, as well as the address of the machine that sent the original IP packet and any of the original packet contents.

Traceroute takes advantage of these two predictable reactions. When you enter TRACERT WWW.BOARDWATCH.COM, traceroute first does a DNS lookup of WWW.BOARDWATCH.COM to get the 204.144.169.6 IP address. It then uses the somewhat more efficient User Datagram Protocol (UDP) to create three small, typically 40 byte, packets containing the originating address, the destination address, and a time stamp of when the packets were created. It sends out three of these packets with the TTL value set to 1. These packets arrive at the first router in the path to Boardwatch and that router immediately decrements the TTL, notes that it is now zero, and issues an ICMP TIME EXCEEDED IN TRANSIT error message back to the sending machine - including the original time stamp and the IP number of the router sending the error message. The original machine that sent out the three packets receives this ICM error message and notes the time of receipt, as well as the IP number of the router that sent it. It then examines the time stamp information returned inside the ICMP packet. It can then calculate the difference between the time stamp information sent, and the time the ICMP packet was received. This is how it calculates the round trip transit time in milliseconds.

Traceroute then extracts the IP number of the machine issuing the error message and does a reverse DNS lookup to retrieve the name of the machine. It prints a sequence number, in this case 1, followed by the name of the machine, the IP number of the machine, and the round trip time for each of the three test packets. The "first" machine in route is now diagrammed.

Traceroute then increments the TTL to 2 and sends out three more packets with new time stamps in them - again addressed to 204.144.169.6 - still the ultimate destination. The first router in the path receives the packet, decrements TTL from 2 to 1 this time, and since TTL has NOT expired, passes the packets on to the NEXT router in the path. THAT router now decrements TTL from 1 to 0, and issues the ICMP error messages back to the originating machine in the same way.

Tracert continues to increment TTL and send out sorties of three packets each time. Each time TTL is incremented, the packets make it to another router down the path before expiring and causing the tattletale ICMP error message that allows traceroute to identify the router. The traceroute machine will wait a fixed amount of time for a reply. If this expires, it will print a series of ASTERISKS (*) indicating that there is a machine there in the path, but it can't get it to respond with an ICMP error message within the default time. It then increments TTL again and continues.

When TTL reaches a value where the UDP datagram actually reaches the ultimate intended host, in this case www.boardwatch.com, that would normally be the end of the packet. But the host will be surprised to learn that the destination port number in the packet header is a ridiculously implausible port number - usually 33,434 but in any case something not ever recognized as a port. Web sites usually monitor port number 80 for example while an e-mail server will monitor port 25. Port 33,434 is not only not one of the normal ports, but is not likely to ever be. So the destination machine issues an ICMP error message as well - in this case the PORT UNREACHABLE message. Traceroute reads this as a kind of "mission accomplished" termination.

USING SOURCE ROUTING TO MANGLE THE PATH

Traceroute can be very useful to list the default path from your site to another site and that is its most common use. However, we often use traceroute to examine backbone topologies and this at times requires some bending of the rules. We often want to run packets around in circles so to speak, and watch how they get there.

Conceptually, there are two options to UDP datagrams that traceroute can take advantage of. These are the Loose Soure Route Record (LSRR) and Strict Source Route Record (SSRR) usually referred to as Loose Source Routing and Strict Source Routing. Rumor has it that Van Jacobson had both in the original 1988 TRACEROUTE and disabled them at the request of some administrators who didn't want it known how badly broken their gateways really were. We can't verify that rumor, but find it believable.

We haven't seen much in the way of traceroute variants that do the Strict Source Routing, which in theory would allow you to specify the precise path from one point to the next. But there are several that support Loose Source Routing, which allows you to indicate intermediate destinations.

Basically, this requires a bit more detailed description of UDP packets. In the case of LSRR, you can enter up to nine intermediate routers to define a path through the network. In most UNIX traceroutes, this is the -g option switch where in the Windows95 TRACERT it is the -J option.

To see why this might be valuable, let's take a look at a regular traceroute to Mustang Software Company (diagrammed above) in Bakersfield California.

Tracing route to mustang.com [204.250.9.1] over a maximum of 30 hops:
1	2 ms	2 ms	2 ms	t1.boardwatch.com [204.144.169.2]
2	9 ms	7 ms	7 ms	199.45.143.15
3	11 ms	10 ms	12 ms	t1.esoft.com [199.45.143.14]
4	18 ms	16 ms	16 ms	199.45.130.13
5	56 ms	18 ms	18 ms	gw21-backbone-1.boulder.co.coop.net [199.45.132.33]
6	23 ms	20 ms	20 ms	sl-che-3-H12/0-T3.sprintlink.net [144.224.13.5]
7	24 ms	32 ms	22 ms	sl-che-1-F0/0.sprintlink.net [144.224.10.1]
8	44 ms	49 ms	45 ms	sl-stk-1-H3/0-T3.sprintlink.net [144.228.10.89]
9	76 ms	48 ms	43 ms	sl-stk-2-F/T.sprintlink.net [198.67.6.2]
10	223 ms	224 ms	307 ms	sl-ana-2-H4/0-T3.sprintlink.net [144.228.10.26]
11	53 ms	54 ms	82 ms	sl-ana-4-F0/0.sprintlink.net [144.228.70.4]
12	62 ms	62 ms	66 ms	sl-mustang-1-s0-T1.sprintlink.net [144.228.74.46]
13	85 ms	73 ms	69 ms	mustang.com [204.250.9.1]
Trace complete

This is a typical traceroute. Mustang is in California and like Boardwatch, is basically connected to the Sprint backbone. As you can see, we start at Boardwatch in line one, and then sequence through an in and out interface at eSoft, Inc. in Aurora. We then go to the Boulder Coop, up through two Sprint backbone routers, in Cheyenne Wyoming, and from their to Stockton California on the Sprint backbone. From Stockton, we drop down to Anaheim California, and from their to Mustang.

Note an interesting anomaly: We went through DIFFERENT routers going OUT through Cheyenne and Stockton than we did coming IN from Grants Pass. Backbone systems can be considerably more complex than highway systems actually with multiple "lanes" for redundancy, capacity, etc. But like a highway system, you often take a different exit going north than you do going south . This also explains why roundtrip times are not terribly valuable, the route the ICMP error message takes BACK to you is unlikely to be the route the original sortie of three packets took OUT to the machine in the path. The point is, traceroute diagrams paths in only one direction. Reversing the direction often diagrams an entirely different path.

Well and good enough. But we're pretty familiar with the Cheyenne/Stockton leg. What we would like to do is find out what is going on going EAST to Kansas City from Cheyenne. We're pretty sure that Kansas City is connected to Fort Worth, but Sprint might have built a new connection directly from Kansas City to Anaheim. From previous traceroute explorations, we know there is a Sprint router in Kansas City - 144.228.10.81

Tracing route to ns.mustang.com [204.250.9.1] over a maximum of 30 hops:
1	2 ms	2 ms	2 ms	t1.boardwatch.com [204.144.169.2]
2	9 ms	9 ms	8 ms	199.45.143.15
3	12 ms	12 ms	11 ms	t1.esoft.com [199.45.143.14]
4	18 ms	18 ms	19 ms	199.45.130.13
5	22 ms	40 ms	20 ms	gw21-backbone-1.boulder.co.coop.net [199.45.132.33]
6	60 ms	214 ms	233 ms	sl-che-3-H12/0-T3.sprintlink.net [144.224.13.5]
7	253 ms	219 ms	208 ms	sl-che-2-F0/0.sprintlink.net [144.224.10.2]
8	38 ms	38 ms	38 ms	sl-kc-1-H2/0-T3.sprintlink.net [144.228.10.81]
9	42 ms	40 ms	39 ms	sl-kc-2-F0/0.sprintlink.net [144.224.20.2]
10	116 ms	252 ms	232 ms	sl-fw-5-H3/0-T3.sprintlink.net [144.228.10.78]
11	190 ms	211 ms	204 ms	sl-fw-6-F/T.sprintlink.net [208.12.128.6]
12	249 ms	257 ms	117 ms	sl-ana-1-H2/0-T3.sprintlink.net [144.228.10.30]
13	261 ms	227 ms	249 ms	sl-mustang-1-s0-T1.sprintlink.net [144.228.74.46]
14	507 ms	466 ms	480 ms	ns.mustang.com [204.250.9.1]
Trace complete

TRACERT -J 144.228.10.81 204.250.9.1

Examine this command line. We are listing two IP numbers with the -J option. The rightmost number is actually our destination, 204.250.9.1, the address of the MUSTANG.COM machine. In this case, we have no interest whatsoever in this machine. We're actually trying to verify the Sprint backbone through Cheyenne to Kansas City and Fort Worth to Anaheim. A normal traceroute to MUSTANG.COM would actually go through Cheyenne, to San Francisco, and down to Anaheim. But in this case, we want to force a path east and out the south end of the backbone. Our interest is in the backbone, and the Mustang site is a handy endpoint of known connection that we can use.

The first IP number, 144.228.10.81 is a Sprint router in Kansas City. The second is the MUSTANG.COM destination - picked only because we know it is attached in Anaheim.

In the UDP datagram, the first address is listed as a DESTINATION address, but a pointer in the datagram points to the MUSTANG.COM address as the DESTINATION address. When the router in Kansas City is finally reached, it notes the discrepancy between the pointer and the destination address. It discards its own address as destination, shifts all addresses to the left, and sends the packets on with the MUSTANG.COM address now appearing to be the destination.

We could put up to nine intermediate IP addresses in the command line. That is all the room available in the header. As each address is reached, it is discarded and the remaining addresses shifted left one position. In this way, we can "steer" a bit through the Internet. But in practice, you have to have some knowledge of the possible Intermediate addresses.

The advantage of loose source routing is that you don't have to know ALL the routers in line as in Strict Source Routing. But the routes you enter DO have to make some sort of sense, and the ultimate destination has to make some sort of sense. But it does allow you to "steer" a bit in exploring paths through the Internet.

As we can see from the results, we can go from Boulder, to Cheyenne, and out to Kansas City. Kansas City is connected to Sprint routers in Fort Worth, and Fort Worth is connected directly to Anaheim. There is no link directly between Kansas City and Anaheim. We already knew that Mustang was connected to Sprint at Anaheim. But we've just successfully mapped the route from Cheyenne to Kansas City to Fort Worth to Anaheim. We can now use the router numbers we get from those cities, to go exploring further if we like.

TRIANGULATING THE INTERNET

Give me a lever and a place to stand and I can move the world. Actually, whoever said this was a bit off. Over and over, and in multiple fields of endeavor, we've found that you generally need THREE places to stand in this world to do anything.

On the Internet, there are many backbones, and your best source of reference is the one you are connected to. But it can be of a powerful advantage to be able to see your site, and others, from other vantage points on the Internet. As we noted, the path in one direction is rarely the same path in the reverse direction. It can be of powerful advantage to run traceroutes from OTHER locations back to US. In addition, this can verify our "visibility" from other backbones and even other continents.

Ray Davis of Carpe Net in Hofheim Germany (ray@carpe.net) has actually presented me personally with a bit of a rose in this respect and the entire Internet it would appear as well. Ray has done a small, fairly simple thing really quite well. He connected traceroute and the World Wide Web with a very simple shell script.. Anyone can run this shell script on their web and provide the world with an online traceroute function accessible from a browser.

Davis is actually from Colorado, but moved to Germany to serve as Technical Director at carpeNet Information Technologies GmbH (carpe diem - seize the day, carpe net - seize the net).. CarpeNet is connected to the central hub of one of the fastest and most extensive European networks (Nacamar/www.nacamar.net). And soon they'll be one of the highest bandwidth ISPs in Europe - with a 10 Mbps ethernet link into the new MAE-Frankfurt building - destined to be a major European NAP. They hope to develop some business as a European mirror site for popular U.S. web sites and they seem to be making some interesting moves in getting there. http://www.carpe.net/index.html

In any event, Ray wrote a little bit of an HTML script and a Common Gateway Interface (CGI) program script to allow anyone to visit his web site and run traceroute. In fact, you can enter ANY destination you want on the HTML form, and carpeNet's server will dutifully traceroute from their location to the entered destination and print out the results on your screen.

According to Davis: "Before we founded carpeNet, I spent over a year researching networks here in Germany and as many possibilities of connecting to the Internet as I could find. I often found myself wanting to trace the route not only from where I am but from other places back to me or from other places to other places. This way I could figure out how various nets were utilized, who was peering with who, how logical their routing was and something about their performance.

"Some months later, after carpeNet was founded, I wrote a simple sh cgi script and put an ISINDEX web page in front of it. Then I send a message to the inet-access mailing list asking if anybody else thought it was useful to be able to traceroute from elsewhere via a web page. I also included my traceroute URL and told everyone to pick up the cgi source if they wanted to set up a traceroute page too.

"Quite a few people thought it was a great idea, and some of them also put up a traceroute page and announced it. A couple days later David Dennis (http://www.amazing.com) put up some of these URLs on a page he called Club Traceroute."

Davis work has had several interesting effects. First, you can check to see if your web site is really reachable from Hofheim Germany. This will check to see that your Domain Name Service entry has migrated through the system to Germany, and also that you are actually reachable over the network. Finally, you get to see the connections between European backbones to the U.S. and across the U.S. backbones. By selecting different end points, you can again draw some very interesting routes.

So interesting that others wanted to setup webserver traceroutes on THEIR web sites. And the more of these that go up, the more interesting and useful it becomes. The accompanying table lists 88 of our favorites located in an afternoon. If you don't feel like hand entering them, we've also put them up on a page on our web site at http://www.boardwatch.com/isp/trace.htm. But there appear to be hundreds available in Japan, North America, Germany, France, Belgium, Russia, Israel, Mexico, Czech Republic, Sweden, Australia, and more. Just in North America, we can now trace different sites both from and two on MCI, AGIS, UUNET, IBM/Advantis, Sprint and Sprint's overseas cooperative, Global One, which is apparently becoming ever so popular.

This growing fleet of web traceroute servers allows us to pick spots on different backbones and even different continents to traceroute FROM.

Davis has actually moved beyond CGI scripts and actually rewritten a C language traceroute to output HTML and BE the CGI - somewhat more efficient on the server and perhaps more secure as well.. The C language source code is available at http://www.carpe.net/src/index.html.

ISP's particularly are putting up these simple little pages all over the world. Soon, it would appear there will be enough of them that we will have the illusion of being able to specify BOTH ends of a traceroute, the source and destination. At that point, it will be fairly easy to map the Internet to any degree of specificity desired, using the common traceroute program.

INSTALLING A CGI SCRIPT WEB SERVER

The following script was written by Ray Davis of carpeNet in Hofhiem Germany. To install it on a UNIX web server:

Edit the script with an editor to indicate your site information instead of BOARDWATCH for example.

Login to your system as a Super User (SU)

Place the script in the web server /cgi-bin directory under the filename traceroute.

Make the file traceroute an executable (chmod 555 traceroute)

Test with a browser to http://yourwebsite.com/cgi-bin/traceroute

#TRACEROUTE SCRIPT #!/bin/sh # # trace - cgi to run traceroute using ISINDEX # # # edit/usr/sbin/traceroute to point to traceroute TRACEROUTE=/usr/sbin/traceroute TOPSTUFF='Content-type: text/html <HTML> <HEAD> <TITLE>Boardwatch Traceroute </HEAD> <BODY BGCOLOR="#e8e8ff"> <P ALIGN=Left> ' if [ -x $TRACEROUTE ]; then if [ $# = 0 ]; then echo "$TOPSTUFF<H1>Traceroute</H1>" # edit this line to match your system echo '<B>Perform a traceroute from www.boardwatch.com</B><ISINDEX prompt="Enter domain name to traceroute to:">' else # edit this line to match your system echo "$TOPSTUFF<H1>Traceroute Output</H1><B>FROM www.boardwatch.com TO "$*".</B><P<>PRE>" $TRACEROUTE "$*" 2>&1 fi else echo ERROR: cannot find traceroute on this system. fi