Multihoming for the Small ISP

Author's note:
Please send comments to avi@freedman.net with a subject containing 'Multihoming FAQ Comments'. Please send questions to inet-access@earth.com with a subject containing 'Multihoming Question'. If you are not subscribed to inet-access, send e-mail to inet-access-request@earth.com with a Subject of 'subscribe'. Otherwise you probably won't get a response from me.

What is Multihoming?

You are multi-homed when you have two "upstream providers". These could be other ISPs that you exchange backup service with (more later); large backbones such as MCI, Sprint, UUNet, or Agis/Net99 (PSI and ANS don't sell to resellers); regional backbone providers; or other local ISPs.

Why Multihome?

Because any one provider may have huge problems at any time. I won't name names here, and the Best Provider of Today could be the Shit Provider of Tomorrow.

So the best way to achieve redundancy is to multi-home (connect up to two different providers). Right now, this FAQ only covers "full multi-homing" - not backup arrangements that can be used in case of emergency, but cases where you are fully operational immediately if either (any) of your lines or providers gets hosed.

What is the Job of Your Provider?

This may seem obvious, but there are two things your provider is supposed to do for you: (1) Bring data from the rest of the Internet to your network, and (2) Take data from your network to the rest of the Internet.

About #1: Your Provider Bringing Data to You

Providers exchange routes with each other. A route is both a description of a section of IP address space and a "promise to accept data" for that section of IP address space. Unfortunately, you must at least know what: routes and IP address space are to get anything out of this FAQ.

So how does someone someplace on the Net send data to you? They send a packet to their provider. If you are using a different provider, then their provider sends that packet to your provder based on a "route announcement" having occurred between those two providers.

So outgoing route announcements bring data in to your network.

Either your provider "nails" your routes into their internal routing table and external route-announcement table somehow, or you announce your routes to your provider and then those routes go into their internal & external-announcement routing tables. Of course, if you got your IP address space from that provider, they will not be announcing your smaller sub-route to the rest of the Internet at the exchange points.

[Insert link to the CIDR FAQ.]

About #2: You Sending Data to the Internet

One of your machines generates a packet of data, and it's not destined to somewhere within your network. Where do you send it? To one of your provider. How do you make that decision? There's a route in your "border router"(s).

Default route: If you have a default route (also written as the 0.0.0.0 route), then you are just sending all data that you don't know what to do with the one of your providers. Even with functional multi-homing, you may still be doing this.

Taking routes from your provider: If you have one provider, you *could* take all of the routes on the Internet (about 32,000 of them on 12/15/95). This WILL fit in a 16mb Cisco 2500 series router. But why would you want to do that if you only have one provider? It gives you the same net effect as if you just defaulted into that provider.

Sample Description 1: What if you are single-homed?

Usual Configuration:
Your provider statically inserts all routes for your network. This means that if you have a customer with a "legacy Class C" that you can't convince to renumber immediately, you'll have to ask your provider to insert a route for you.

You default route into your provider.

int e0
  ip add [local_ip_adx] [local_ip_mask]
int s0
  ip add [local_t1_adx] 255.255.255.252
ip route 0.0.0.0 s0

note: Usually the mask on a serial is 255.255.255.252.
      This is to conserve ip address space.

Slighly More Desirable Configuration:
You speak BGP with your provider to announce your routes. This way, you can add routes without having to call their NOC or "routing department" to do it for you. If you repeatedly (or even ever, sometimes) screw it up, though, expect to have them either put in a filter list for your route announcements - or switch you to static routing. (Note: We build specific lists of routes that our BGP customers can announce into us to prevent them from blowin nasty routes into us...)

You default route into your provider.

int e0
  ip add [local_ip_adx] [local_ip_mask]
int s0
  ip add [local_t1_adx] 255.255.255.252
router bgp [your-asn]
  network [net1]
  network [net2]  mask 255.255.254.0 
  network [net3]  mask 255.255.252.0
  neighbor [remote_t1_adx] remote-as [provider-as]
ip route 0.0.0.0 s0
ip route [net1] dest1
ip route [net2] 255.255.254.0 dest2
ip route [net3] 255.255.252.0 dest3

note: This way ensures that wild routes won't creep in.
      But this will ONLY be enough if you have a single provider.

BGP NOTE:
If you only have one provider, you don't have to worry as much. If you have multiple providers, you MUST make sure you don't announce garbage routes into them (or routes for all of the net). Providers are usually set up to trust customer routes above any routes heard externally, so if you announce MCI routes into UUNET, you may find UUNET using your MCI T1 to send all of its local data destined for MCI to MCI!!! So you don't have to worry about this until you're dual-homed.

Sample Description 2: What if you are dual-homed?

Incoming data (outgoing route announcements)

Usual Configuration:
You speak BGP to both of your providers to announce your routes.
What will work:
There is NO reason that you "must" speak BGP to your providers. They can both statically insert your routes, but the assumption is that you want to speak BGP because: (1) You want to be able to take full routes from them to make more intelligent routing decisions, and/or (2) You want to be able to add routes without interfacing person-person with them.

Outgoing data (incoming route announcements and/or default route(s))

Method 1: Only defaults: Even weights

Either used unnumbered interfaces to your providers, or use the same local IPs on your "serial ends" - your local IP address(es) for your T1s. This assumes that both providers are on the same router at your end. Just put in defaults to both routers with equal weights. Set 'ip route-cache' on both serial lines.

You *can* do this while speaking BGP to one or both providers, but if you are speaking BGP you probably want to take "customer routes" (see Method 3).

Of course, if you want to speak BGP for outgoing route announcements, you can just filter all incoming routes and know that you can multi-home using this method on even a 1mb Cisco 2501.

Method 2: Only defaults: One is a backup

Have two default routes - one to each provider, but have one set as a lower weight so that it only takes effect if your link to your primary provider goes down [note: on Ciscos, routes (including default routes) "go away" when the interface that they're associated with is down].

This might work when one provider is not as good - and when you don't need the bandwidth of the 2nd connection.

Method 3: Take "customer routes" from each provider

Speak BGP to both providers. Take all of provider X's customer routes from provider X and the same for provider Y. Then, either default into both with equal weights or default into one with a backup default into the other. This will definitely work in a 16mb 2501.

Method 4: Take "full routes" from each provider

Speak BGP to both providers. Take all of each provider's routes for everything. This *might* fit in a 16mb 2501, but probably would not. A 32mb 4000 or 4500 would be fine, as would a Morningstar/gated or PC/gated solution. With this configuration, you *can* run "defaultless" - without a default route, since you should have a fairly specific route for every active site on the net. But there's no need to do this, since if one or both providers go screwy and are not feeding you full routes, you lose connectivity by being defaultless. And having default routes in while taking full routes doesn't hurt anything.

Method 5: The creative route-balancing method

This is halfway between Method 3 and Method 4. Speak BGP to both providers. Take "customer routes" from each provider. Then, split up the rest of the Internet by their key transit provider's AS:

MCI (AS3561)

Sprint (AS1239)

ANS (AS690)

UUNET (AS701)

PSI (AS174)

Net99 (AS3830)

AGIS (AS4200)

These numbers might be wrong for you! You'll have to look at your routing table (at the AS-paths for ftp.uu.net, ftp.psi.com, ftp.sprintlink.net, etc...) to figure it out.

Then, just decide to hear MCI routes through provider A and Sprint routes through provider B. Or whatever mix you want. Balance whose routes you hear through which provider until your average utilization looks balanced. Add default routes to both (equal weighted) or to one as primary and one as secondary to suit.

To be added:

as-paths, routing decisions made based on

making sure that if your provider statically-routes you that they go away if your line to them goes down, even inside of their network

where to get address space from

subnet faq

t1s vs. 56k for multiple links

need bgp? - livingston

---------------

terry@spcvxb.spc.edu  Terry Kennedy, Operations Mgr. at St. Peter's College, US

In article <1995Dec18.164944.1@hujicc>, yehavi@vms.huji.ac.il (Yehavi Bourvine (58-4279)) writes:
>   An ISP is going soon to be dual-homed to two ISPs which are directly
> connected to the NAPs. THe question is which router to use: 4,500 with 32MB or
> 7010 with 64MB. My main concern is the routing tables (full BGP tables) and the
> amount of memory they use. WIll I have enough free memory on the 4,500 for the
> forseen future size of the tables, or shall we take a 7010 with 64MB to be on
> the safe side? 7,000 and 7,500 are out of question due to the price.

  There are conflicting answers to this question. I posted some numbers a
while back from the CIX router, and people told me "the CIX router filters
many routes and you'll have problems with full routing".

  Well, I went ahead and did it anyway. I'm using a 4500M with 32MB main
memory and 16MB I/O memory. It is dual-connected to Sprint and Alternet
and maintains full routing. Here's the output from sho ip bgp sum and sho
mem:

router>sh ip bgp sum
BGP table version is 2436598, main routing table version 2436598
32175 network entries (61909/64360 paths) using 5679608 bytes of memory
3550 BGP path attribute entries using 413828 bytes of memory
0 BGP route-map cache entries using 0 bytes of memory
5246 BGP filter-list cache entries using 83936 bytes of memory

Neighbor        V    AS MsgRcvd MsgSent   TblVer  InQ OutQ Up/Down  State
xxx.xxx.xxx.xxx 4   701  432288   20337  2436532    0    0 2d13
xxx.xxx.xxx.xxx 4  1239  411988   20314  2436532    0    0 1w6d
xxx.xxx.xxx.xxx 4  xxxx   19984   19978  2436594    0    0 3d04
xxx.xxx.xxx.xxx 4  xxxx   20222   19979  2436532    0    0 4d23
xxx.xxx.xxx.xxx 4  xxxx   19512   19881  2436532    0    0 2d17
router>sh mem
               Head  FreeList    Total(b)     Used(b)     Free(b)  Largest(b)
Processor  60508B40  604ACA78    28275904    15935728    12340176    11647024
      I/O  40000000  604AD78C    16777216     1842080    14935136    14877616

        Terry Kennedy             Operations Manager, Academic Computing
        terry@spcvxa.spc.edu      St. Peter's College, Jersey City, NJ USA
        +1 201 915 9381 (voice)   +1 201 435-3662 (FAX)