San Diego Road Runner Architecture

Phil Karn
June 8, 1997
Revised 13 July 1998

Here is a description of the San Diego Road Runner network architecture as I've been able to piece it together by the use of standard Internet network diagnostic tools, from comments made by Road Runner staff, and from discussions with others familiar with the technology. It is entirely possible that my information contains errors and/or is incomplete; additions and corrections are welcome.

Overview

The San Diego Road Runner network is organized as a tree. The backbone of the whole network is an ATM switch located at the Ware Ct. headquarters. The proxy, netnews and mail servers, all physically located at Ware Ct, have direct connections to this ATM backbone. So does a Cisco 7507 router that serves as the gateway to the outside Internet through MCI and CERFnet.

Also connected to the ATM backbone via point-to-point fiber links are fifteen Toshiba Authentication Systems (TASes) at the various regional hubs around the city. TAS-7 is located at Ware Ct where it is used for local testing. TAS 14 does not exist. The TASes provide DHCP and DNS service to the users, and they also implement an ad-hoc login protocol that must be executed on the user's computer before upstream packets are permitted to pass to the ATM backbone.

Colocated with each TAS and connected to it with a small (2 host) FDDI ring is a Motorola Cable Router, the device that talks directly to the Motorola CyberSURFR cable modem in each subscriber's home.

Here is a diagram of the network courtesy of Mike Maculsay.

Outbound packet path

A packet sent from a subscriber to the external Internet takes the following path:

  • From the customer's computer by 10 Mb/s Ethernet to the CyberSURFR cable modem.
  • RF from the CyberSURFR modem via coaxial cable to the neighborhood coax/fiber concentrator.
  • RF via fiber to the regional hub.
  • The Cable Router at the regional hub demodulates the packet and passes it over FDDI to the co-located TAS.
  • The TAS sends it over the fiber ATM link to the ATM switch at Ware Ct.
  • The ATM switch routes it to the Cisco 7507.
  • The Cisco 7507 routes it via leased line to MCI (for general internet traffic) or to CERFnet (for local San Diego sites on CERFnet).

    The MCI connection is seven downward multiplexed T1s to MCI's Los Angeles POP. The connection to CERFnet (installed 31 July 1997) is now a full DS3 (43 Mb/s).

    The Cisco 7507 has a third external connection: a frame relay link to the Time Warner Pathfinder site.

    Ware Ct. has several miscellaneous LANs. Two, a 100 megabit and a 10 megabit ethernet, apparently use the mail server (mail.san.rr.com, 204.210.0.1) as a router to the ATM backbone. Another two LANs, probably Ethernets, use a Cisco 4700 as a router to the ATM backbone.

    Addressing

    The San Diego Road Runner system uses the IP address block 204.210.0.0 through 204.210.63.255. In CIDR terms, this is 204.210.0.0/18. The rest of the 204.210 prefix is used by Road Runner and Excalibur networks in other cities.

    The San Diego address space is divided into 64 256-host ("Class C") nets as follows:

    subnet,
    204.210.nnn    		 usage
    0ATM backbone
    1Ware Ct 100Base-T network
    2Ware Ct 10Base-T network
    3unused
    4Ware Ct network (32-48 only - via Cisco 4700)
    5Ware Ct 10Base-T network (via Cisco 4700)
    6unused
    7regional hub FDDI subnets (see below)
    8Hub 7 user subnets (Ware Ct test)
    9-11Hub 2 user subnets (Mira Mesa)
    12-18Hub 1 user subnets (Pacific Beach)
    19-21Hub 3 user subnets (Scripps Ranch)
    22-26Hub 8 user subnets (La Jolla)
    27-30Hub 4 user subnets (Rancho Penasquitos)
    31-34Hub 5 user subnets (North Clairemont)
    35-38Hub 6 user subnets (University City/UTC)
    39-41Hub 15 user subnets (Carmel Valley)
    42-45Hub 10 user subnets (South Clairemont)
    46-49Hub 9 user subnets (Kearney Mesa)
    50-51Hub 11 user subnets (Rancho Bernardo West)
    52-55Hub 12 user subnets (Rancho Bernardo East)
    56-57Hub 13 user subnets (Tierrasanta)
    58-59Hub 16 user subnets (Coronado)
    60Hub 15 (Carmel Valley) overflow
    61Hub 3 (Scripps Ranch) overflow
    62Hub 6 (University City/UTC) overflow
    63Ware Ct network (via Cisco 4700)

    In each user subnet, the .1 address (e.g., 204.210.35.1) is assigned to the cable router. It has a domain name of the form tas[1-16]-hfc[1-9].san.rr.com. The remaining addresses are available for DHCP assignment to users. They have domain names of the form dt[1-16]h[1-9]n[2-fe].san.rr.com. The first numeric field gives the hub number, the second gives the subnet number within the hub starting at 1, and the third field gives the host part of the IP address in hexadecimal.

    The Motorola Cable Routers do not support classless IP subnetting, so the address blocks they manage must be aligned on full Class C boundaries.

    Note that the hubs are not assigned equal numbers of subnets, probably because they serve different numbers of customers.

    The 204.210.7 subnet is further subdivided into small 4-host subnets for the FDDI ring in each regional hub. Each hub uses the the four available addresses as follows:

    FDDI host
    number    		usagedomain name, #=hub number
    0unused
    1TAStas#-fddi.san.rr.com
    2Motorola Cable Routermcr#-fddi.san.rr.com
    3unused

    Regional hubs and addresses

    For each Road Runner hub, this table shows the hub number, the area it serves, and the user and internal IP addresses assigned to each one.

    hub# region user
    subnets,
    204.210.nnn    		 Cable router
    FDDI address
    mcr#-fddi.san.rr.com    		 TAS
    FDDI address
    tas#-fddi.san.rr.com    		 TAS
    ATM address
    tas#-atm.san.rr.com
    1 Pacific Beach
    Mission Beach    		 12-18 204.210.7.6 204.210.7.5 204.210.0.240
    2 Mira Mesa 9-11 204.210.7.10 204.210.7.9 204.210.0.241
    3 Scripps Ranch 19-21,61 204.210.7.14 204.210.7.13 204.210.0.242
    4 Rancho
    Penasquitos    		 27-30 204.210.7.18 204.210.7.17 204.210.0.243
    5 North
    Clairemont    		 31-34 204.210.7.22 204.210.7.21 204.210.0.244
    6 University City
    UTC    		 35-38,62 204.210.7.26 204.210.7.25 204.210.0.245
    7 Ware ct test 8 204.210.7.30 204.210.7.29 204.210.0.246
    8 La Jolla 22-26 204.210.7.34 204.210.7.33 204.210.0.247
    9 Kearney Mesa 46-49 204.210.7.38 204.210.7.37 204.210.0.100
    10 South
    Clairemont    		 42-45 204.210.7.42 204.210.7.41 204.210.0.101
    11 Rancho Bernardo
    West    		 50-51 204.210.7.46 204.210.7.45 204.210.0.102
    12 Rancho Bernardo
    East    		 52-55 204.210.7.50 204.210.7.49 204.210.0.103
    13 Tierrasanta 56-57 204.210.7.54 204.210.7.53 204.210.0.104
    14 unused?
    15 Carmel Valley 39-41,60 204.210.7.58 204.210.7.57 204.210.0.106
    16 Coronado 58-59 204.210.7.62 204.210.7.61 204.210.0.105

    Some traceroutes

    Traceroute is a utility that determines the path an IP packet takes through the Internet by cleverly using the Time to Live (TTL) field in the IP packet header and looking for ICMP Destination Unreachable (TTL exceeded) messages generated by the routers along the path.

    When looking at a traceroute, bear in mind that routers generally have a different IP address for each interface, so the IP address shown in a traceroute is always that of the interface on which the router received the packet.

    Because of this rule, a traceroute done in the reverse direction may show a completely different set of IP addresses and domain names even when the exact same routers are traversed in reverse order.

    An upstream traceroute

    Here is the annotated output of a "traceroute" command ("tracert" in Windows 95) performed from a host in the University City area through the Road Runner system to a host on the external Internet. The nodes beyond the entry point to MCI are not shown. 

     1  * * *
 2  tas6-fddi.san.rr.com (204.210.7.25)  125.26 ms  258.222 ms *
 3  7507-atm.san.rr.com (204.210.0.254)  244.646 ms  39.975 ms  48.9 ms
 4  border7-serial4-2.LosAngeles.mci.net (204.70.252.105)  52.217 ms  263.338 ms  19.007 ms

    Node 1: The Cable Router should have responded here with its IP address (204.210.35.1 in this case), but a bug in the current firmware prevents this. I have heard the new release (1.3) fixes this, but I'm not sure.

    Node 2: This is the FDDI interface on TAS 6, the side "facing" the cable router (and me).

    Node 3: This is the ATM interface on RR's main Cisco router. Again, this is the side that faces the TAS, the cable router and me.

    Node 4: This is the input interface on MCI's router in Los Angeles.

    A downstream traceroute

    Now here is the last part of a traceroute performed in the reverse direction from a machine outside the RR network. Although the very same nodes are traversed in reverse order, the names and addresses are different because traceroute reports the interfaces away from the destination. 

    11  * * time-inc-new-media.LosAngeles.mci.net (204.70.252.106)  43 ms
12  tas6-atm.san.rr.com (204.210.0.245)  44 ms *  44 ms
13  * * *
14  dt6h1n34.san.rr.com (204.210.35.52)  70 ms  47 ms  149 ms

    Node 11: This is the MCI leased line interface on RR's Cisco 7507 router. Don't be fooled by the name, it is standard practice to assign carrier-owned IP addresses and domain names to the external interface on a customer's border router.

    Node 12: This is the ATM interface on TAS 6, the side of TAS 6 facing away from the end user (me).

    Node 13: Again, the Motorola cable router refuses to respond. It should have responded with 204.210.7.26, its FDDI interface address.

    Node 14: We reach our destination.