ONLamp.com    
 Published on ONLamp.com (http://www.onlamp.com/)
 See this if you're having trouble printing code examples


O'Reilly Book Excerpts: BGP

Traffic Engineering: Incoming Traffic

Related Reading

BGP
Building Reliable Networks with the Border Gateway Protocol
By Iljitsch van Beijnum

by Iljitsch van Beijnum

Editor's Note: In this third installment on Traffic Engineering, excerpted from O'Reilly's BGP, learn how to balance inbound traffic.

Traffic Engineering for Incoming Traffic

Because the local router determines the route taken by outgoing packets, it isn't difficult to balance outbound traffic over multiple connections. The situation for inbound traffic is different. There are only a few routes you can influence to shift incoming traffic patterns: one for each address block for each ISP you connect to, instead of tens of thousands for outgoing traffic. In the typical multihoming case, with one address block and two ISPs, this leaves you with just two routes that can be manipulated to change inbound traffic distribution. This manipulation can take the form of:

You can also decide to "cheat" and break up a single address block that would normally be announced as a single route into several smaller blocks, so you can announce each separately, with different properties, for more fine-grained control.

Setting the MED

The MED metric is intended to be used only between two neighboring ASes. It isn't communicated to ASes beyond the neighboring AS. For this reason, the use of the MED in balancing incoming traffic is limited to the situation where there is more than one connection between two ASes: setting a higher MED for one route will make the traffic flow over the other. This is useful when one of the connections is of a much higher bandwidth, and the second one is a lower-bandwidth backup. Because you don't know whether the bgp bestpath med missing-as-worst command is in effect on the router terminating your connections at the other end, always set MEDs for the routes over both connections, as is shown in Example 6-11.

Example 6-11: Setting outbound MED values

!
router bgp 60055
 neighbor 192.0.254.17 remote-as 40077
 neighbor 192.0.254.17 route-map ispa-out out
 neighbor 219.2.19.1 remote-as 50066
 neighbor 219.2.19.1 route-map ispb-out out
!
route-map ispa-out permit 10
 set metric 10
!
route-map ispb-out permit 10
 set metric 20
!

We are now trying to influence incoming traffic, so we have to manipulate outgoing routing updates and apply the route maps to the neighbors using the neighbor ... route-map ... out command.

In This Series

Traffic Engineering: Queuing, Traffic Shaping, and Policing
In the fifth and final installment in this series of excerpts on Traffic Engineering from O'Reilly's BGP, learn how to increase performance for certain protocols or sessions using special queuing strategies, traffic shaping, and rate limiting.

Traffic Engineering: Specific Routes
In this fourth installment on Traffic Engineering, excerpted from O'Reilly's BGP, learn how to balance incoming traffic by announcing more specific routes.

Traffic Engineering: Local Routing Policy
In this second installment on Traffic Engineering, excerpted from O'Reilly's BGP, learn how to influence the BGP path-selection process.

Traffic Engineering: Finding the Right Route
In this first installment on Traffic Engineering, excerpted from O'Reilly's BGP, learn how to find the best route in a multihomed setup--the one that will take advantage of all available bandwidth.

TIP: The MED metric you see in the BGP table is never announced to eBGP neighbors. If you want a neighbor to receive a MED, you have to configure an outbound route map to set the MED for this neighbor.

Prepending Outbound AS Paths

When you bring up your second BGP session, you soon get to see how much traffic your routes attract over both ISPs. In many cases, the traffic will be distributed fairly equally over both connections, or one connection receives more traffic but there is enough spare capacity (for inbound traffic) so this isn't a problem. But maybe one connection attracts more traffic than it can handle, or you have one big pipe and a smaller one, and the traffic volumes are equal (or at least they try to be). Under these circumstances, you'll want to shift part of the incoming traffic load from one connection to the other. The most powerful option to change incoming traffic patters is making the AS path longer. This is effective, because the path is preserved between ASes, and BGP implementations use the path length early in the route selection algorithm. The biggest problem with making the AS path longer by prepending your own AS number to the path one or more extra times is that it may be too effective. Example 6-12 shows a configuration that prepends the path for the routes announced to an upstream ISP.

Example 6-12: Prepending the path for outbound routes

!
router bgp 60055
 neighbor 219.2.19.1 remote-as 50066
 neighbor 219.2.19.1 route-map ispb-out out
!
route-map ispb-out permit 10
 set as-path prepend 60055
!

The way the route map works should be familiar by now. Rather than applying the route map for incoming routes, the ispb-out permit 10 route map is used for outbound route updates. The number 10 is superfluous here, because there is only a single route map, but the router adds it to the configuration if you don't type it in yourself. The set clause adds 60055 to all routes.

Make sure all routes with prepended paths are accepted by your ISP and upstream networks. It isn't uncommon for the AS path filters that ISPs use to filter routes from customers not to allow path prepending. There usually isn't a good reason for this; it's just that allowing path prepending makes for more complex filters. If, after configuring path prepending, you use one or more looking glasses to see if they now receive your route in its prepended state, you may see only the unprepended route over your other ISP. It isn't always clear whether this means the route wasn't accepted, or routers further upstream just prefer the unprepended path over your other ISP because of the shorter AS path. The only way to make sure is to temporarily disable the BGP session to the nonprepended ISP:

!
router bgp 60055
 neighbor 192.0.254.17 shutdown
!

If the prepended route doesn't show up on remote looking glasses, or remote destinations on the Net become unreachable after shutting down your unprepended ISP, there must be a filter somewhere. Don't forget to let route propagation settle for a minute or two before drawing conclusions. You can determine where filtering takes place by tracerouting to an unreachable destination. The ASes that show up in the traceroute don't filter the prepended AS path. The filter must be between the last AS that shows up in the traceroute and the first one that doesn't. If your ISP is the one filtering out prepended routes, you can ask them to change their filters, but if the problem is further upstream, there is probably not a lot you can do. Don't forget to reenable the BGP session to your other ISP:

!
router bgp 60055
 no neighbor 192.0.254.17 shutdown
!

TIP: The filter that prohibits routes with prepended paths from finding their way may be located inside your own router. It's best always to allow prepending your own AS, even if you don't plan on prepending in the near future:

ip as-path access-list 2 permit ^(60055_)*$

This regular expression matches all AS paths consisting of the beginning of the line (^), zero or more (()*) times the AS number, a space (_), and then the end of the line ($). In other words: an empty AS path or an AS path with just the AS number 60055.

The Effect of AS Path Prepending

Suppose you multihome to two ISPs that are similar: they interconnect at mostly the same NAPs and Internet Exchanges, and they peer with mostly the same networks. Under these circumstances, other networks see two similar paths for the routes you announce. Figure 6-4 shows an example of this.

Figure 6-4. Multihoming to similar ISPs

If AS E (the example multihomed network, AS 60055) wants to receive more traffic over ISP A and thus makes the AS path longer over ISP B, the majority of traffic will flow over the ISP A, which now has the shorter path. In a situation with two similar ISPs, AS path prepending gives them only three choices:

Table 6-1 shows which route is preferred in the situation shown in Figure 6-4 without path prepending, with prepending the path to ISP A, and with prepending the path to ISP B.

Table 6-1: Prepended paths over similar ISPs

  AS X AS Y AS Z Traffic distribution
Prepend to A AEE
BE
AEE
BE
AEE
BE
ISP A: 15%
ISP B: 85%
No prepending AE
BE
AE
BE
AE
BE
ISP A: 40%
ISP B: 60%
Prepend to B AE
BEE
AE
BEE
AE
BEE
ISP A: 90%
ISP B: 10%

For the purposes of calculating the traffic distribution, it's assumed that A always handles 15% of the traffic, B always 10%, and ASes X, Y, and Z are all the source of 25% of incoming traffic. ASes with "even" letters (X, Z) prefer to send traffic over ISP B when the paths are of equal length; "odd" ASes (Y) prefer ISP A in this example. The preferred path is listed in bold type in the table.

When the two ISPs are not as similar, increasing the length of the AS path has a more gradual effect, because the paths over ISPs A and B aren't the same for all networks. Figure 6-5 shows multihoming to dissimilar ISPs.

Figure 6-5. Multihoming to dissimilar ISPs

In this example, ISP B is a much smaller ISP that doesn't peer with networks X and Y, but rather buys transit service from AS C to reach those networks. Networks V and W don't peer directly with ISP A, so even if the path over ISP B becomes a lot longer, they'll still prefer to send traffic to AS E over ISP B. Because this is a peering link, sending the traffic over this route is cheaper. Network Z, on the other hand, will immediately route traffic over ISP A when the path over ISP B is prepended, because the connections to both A and B are peering links. Table 6-2 shows the possible traffic distribution using prepending.

Table 6-2: Prepended paths over dissimilar ISPs

  AS C AS V AS W AS X AS Y AS Z Traffic distribution
2 to ISP A BE BE BE
CBE
AEEE
CBE
AEEE
CBE
AEEE
BE
A: 15%
B: 85%
1 to ISP A BE BE BE
CBE
AEE
CBE
AEE
CBE
AEE
BE
A: 35%
B: 65%
No prepending BE BE BE
CBE
AE
CBE
AE
CBE
AE
BE
A: 55%
B: 45%
1 to ISP B BEE BEE BEE
CBEE
AE
CBEE
AE
CBEE
AE
BEE
A: 75%
B: 25%

The traffic distribution in this example is 15% from ISP A, 5% from ISP B and ASes V and W, 10% from AS C, and 20% from ASes X, Y, and Z.

TIP: All else being equal, it's a good idea to select dissimilar ISPs, for instance, one tier-1 ISP that peers with all the other large networks, and one tier-2 ISP that peers with many small networks. This way, you have a wide range of traffic engineering options.

Setting Outbound Communities

In many cases you'll want to prepend the path for certain upstream networks or peers of a transit ISP and not for others. For instance, if two of your ISPs have a transit network in common, you might want to have one ISP announce a prepended path to this transit network without changing the path that other transit networks and peers see over that ISP. To avoid spending a lot of time implementing this type of policy upon customer request, many ISPs provide their customers (and sometimes their peers) with a list of communities that trigger actions such as path prepending and setting the Local Preference. This can then be done for each route individually.

Well-Known Communities

Communities were introduced in BGP by RFC 1997. This RFC also defines the three well-known communities listed in Table 6-3.

Table 6-3: Well-known communities

Well-known community Action
no-export (0xFFFFFF01) Don't advertise this route to eBGP peers.
no-advertise (0xFFFFFF02) Don't advertise this route to any peers, iBGP, or eBGP.
no-export-subconfed (0xFFFFFF03) Advertise this route to iBGP peers with the same AS number, but not to other confederation members.

These communities can be useful under certain circumstances, for example, if an ISP wants to set the no-export community on routes it sends to a customer to make sure the customer doesn't accidentally announce the ISP's routes to another ISP. If the customer provides transit services to customers of his own, however, they won't receive the route unless the original customer of the ISP removes the no-advertise community.

WARNING: Setting the no-export, no-advertise, or no-export-subconfed communities can have the (possibly unwanted) side effect that no routes are announced, even if there are other routes that would otherwise be eligible for announcement.

For instance, if you set the no-advertise community on routes announced to ISP B, other customers of ISP B won't see these routes because they aren't advertised. This is as intended. But routes with the same NLRI that ISP B has learned from ISP A will not be advertised either, because ISP B considers the directly received routes with the no-advertise community best, and only the best route is eligible for further announcement over BGP.

Common Community Actions

ISPs accepting communities provide their customers with a list of communities they use and what action is taken for each community. It's possible to set several communities for a single route, but the results may be unpredictable if your ISP doesn't expect this. Many networks list the communities they accept in their AUT-NUM object in the Routing Registry they use. Table 6-4 shows a fairly typical list of communities an ISP might accept.

Table 6-4: An example of communities an ISP accepts

Community Action
50066:70 Set Local Preference to 70.
50066:90 Set Local Preference to 90.
50066:110 Set Local Preference to 110.
50066:5010 Announce this route for transit to ISP C.
50066:5020 Don't announce this route to transit ISP or peer C.
50066:5040 Prepend AS path to C once.
50066:5041 Prepend AS path to C twice.
50066:5042 Prepend AS path to C three times.
50066:10040 Prepend AS path once at interconnect point I.
50066:10041 Prepend AS path twice at interconnect point I.
50066:10042 Prepend AS path three times at interconnect point I.

Some ISPs require you to set a community indicating a route should be announced for transit, 50066:5010 in this example. This isn't common: most networks do the opposite and allow you to set a community indicating the route shouldn't be announced to transit networks. Be sure to notice the subtle difference between not announcing for transit and not announcing to transit networks: in the first case, the potential transit network still receives the announcement, but the route is treated as a peering route and not announced to peers and upstream networks of the transit network. In the latter case, the transit network doesn't get to hear the route at all.

Influencing the Local Preference in Upstream ASes

The MED is specifically intended to inform an upstream ISP that one connection should be preferred over another, but today this is often done with communities. Using communities instead of the MED may have benefits internal to the ISP network. For example, the ISP is then free to use the MED for another purpose, as we did for outbound traffic engineering in the beginning of this chapter. And, unlike the MED, using a community to set the Local Preference inside an ISP network also makes it possible to use a link to an ISP as a backup for a link to another ISP. When the Local Preference is set sufficiently low for the intended backup connection, the ISP it connects to will completely ignore the route and always send traffic over the other ISP as long as there is a route present over this ISP. This can't be accomplished with the MED; the AS path length overrides it, the MED isn't communicated from AS to AS, and by default, the MED is looked at only when two connections terminate at the same AS.

The impact of changing the Local Preference depends on the Local Preference values an ISP uses for routes learned from transit, peers, and customers. In this example, that would be 80 for transit, 100 for peers, and 120 for customers. If you have a fast main connection to this ISP along with a slower backup connection, you'll probably want to set community 50066:110 for routes announced over the backup connection. This makes sure all traffic flows over the main connection and the backup connection is used only when the main connection is unavailable. It's also possible to do this when you connect to two different ISPs: to ISP A with a main connection, and to ISP B with a backup connection. Then you'll want to set community 50066:90 or even 50066:70 so ISP B sends all traffic for you over a peering or transit connection to ISP A.

Example 6-13 shows a configuration for the router terminating the backup connection to ISP B. The main, high-bandwidth connection terminates at another router.

Example 6-13: Setting a community to indicate a backup route

!
router bgp 60055
 neighbor 219.2.19.1 remote-as 50066
 neighbor 219.2.19.1 route-map ispb-backup-out out
 neighbor 219.2.19.1 send-community
!
route-map ispb-backup-out permit 10
 set community 50066:5010 50066:70
!

The community 50066:5010 makes this route eligible for announcement as a transit route over AS C, but the community 50066:70 makes sure this route has the lowest possible Local Preference in ISP B's network. Thus, ISP B won't use it as long as there is any other route with the same NLRI (prefix), even if this means routing the traffic over ISP A.

TIP: By default, Cisco routers accept incoming communities but don't transmit them over iBGP or eBGP. The send-community command enables sending communities to a neighbor.

Prepending the AS Path

Some smaller ISPs have path-prepending communities for each peer, but even medium-sized ISPs peer with many networks, so this soon gets out of hand. More often, an ISP has communities to prepend the path to each of its transit ISPs individually, as well as communities to prepend the path for an entire interconnect point. Our example ISP B (AS 50066) accepts path prepending communities for ISP X and interconnect point I.

AS W in Figure 6-5 (shown earlier this chapter) connects both directly to ISP B and also over transit AS C and then ISP B. Supposing the peering link between AS W and ISP B is congested, we'll want incoming traffic from AS W to flow over ISP C. This is accomplished by prepending the path ISP B announces to AS W twice, as is done in the configuration in Example 6-14.

Example 6-14: Setting a community to prepend the path

!
router bgp 60055
 neighbor 219.2.19.1 remote-as 50066
 neighbor 219.2.19.1 route-map ispb-out out
 neighbor 219.2.19.1 send-community
!
route-map ispb-out permit 10
 set community 50066:5010 50066:10041
!

Unfortunately, it's not possible to do something similar for outbound traffic: this will still flow over the congested connection between ISP B and AS C. This isn't the case if it's routed over ISP A and not over ISP B, of course, as is done in Example 6-4 earlier this chapter (in a previous article). Also, setting community 50066:10041 means the path is prepended twice towards all peers at this interconnect point. This may be undesirable, for instance if AS Z connects to ISP B over the same interconnect point as AS W. AS Z now sees the path C B B E over ISP B and the much shorter path AS E over ISP A, so traffic from AS Z will now come in over the connection to ISP A.

In the next installment learn how to announce more specific routes. When prepending the path and setting communities for outbound routes aren't enough to balance incoming traffic, this is the only step left to take. However, because a more specific route always takes precedence over a less specific route, this technique always works.

Iljitsch van Beijnum has been working with BGP in ISP and end-user networks since 1996.


Return to ONLamp.com.

Copyright © 2009 O'Reilly Media, Inc.