This will be the BGP Topology you will very likely be working Trouble tickets from on the TSHOOT, it could be iBGP or eBGP, and there is a whole lot of BGP to know even for that!
Review of BGP
BGP is called a “Path Vector” Protocol that basically is the internet at Layer 3, being an Exterior Gateway Protocol (EGP) that exchanges route information between domains under different organizations control, where an Interior Gateway Protocol (IGP) is meant to be under a single entities administrative control and is not an open transport protocol.
One thing to touch on before diving into BGP is the show commands, as there is two flavors, one is the basic ones which I’ve always used a newer style:
- “sh ip bgp”
- “sh ip bgp nei”
- “sh ip bgp summ”
- “sh bgp ipv4 unicast (nei/summ)” – This is the newer verification to me
^^ I never used that last one while labbing, but am trying to use it throughout this post to burn it into my brain for exam day (and real world), to prepare myself for issuing ipv4 and ipv6 commands fluently.
BGP neighbors are very intentional and must be configured manually on both sides to become neighbors with the “neighbor x.x.x.x remote-as #” on each side, if the remote-as # value matches they are considered iBGP and if they are different the neighbors are considered eBGP which behave much differently from each other.
Some commands to know for exam day to troubleshoot BGP:
- “sh bgp ipv4 unicast summary” – Shows the local BGP RID, local AS #, Neighbor IP Address (from neighbor statement), Neighbor AS #, Up/Down time, etc
- “sh bgp ipv4 unicast neighbors” – Shows this same information with the addition of a lot more granular info, not needed for initial neighbor verification
If configuring a neighbor, a syslog message will be generated such as that seen from the Redistribution Part 1 post where iBGP neighbors were configured:
*Mar 1 03:26:11.720: %BGP-5-ADJCHANGE: neighbor 2.2.2.2 Up
The above adjacency was configured using loopback IP Address with static routes configured on each router, pointing toward the other router as the next hop to reach the static route, though using the connected interface as for the neighbor statement is completely acceptable to do as well (I just used the loopback to demonstrate how it is done in that short configuration lab).
A list of reasons why BGP Neighbors may not form an adjacency:
- Interface is Down – Always confirm interfaces are up in any scenario
- Layer 3 Connectivity Issue – Confirm the neighbor IP is reachable, if not you will need to troubleshoot how why its not / how to get it reachable
- Path to neighbor is default route – Neighbors must be reachable from a non-default route type
- Neighbor does not have route back to local router – No route back to local router
- Incorrect neighbor statement – Wrong IP or AS # in “neighbor x.x.x.x remote-as #” statement configured in “router bgp #” (#=local AS) configuration
- ACL – BGP Communicated over TCP Port 179
- BGP Packets have wrong source address – Inbound BGP Packets match the local neighbor statements IP Address (whether connected interface or loopback)
- TTL of BGP packet expires – Peer is further away than permitted by TTL
- Authentication Mismatch – BGP Routers do not have matching Authentication parameters on both routers
- Misconfigured Peer Group – Peer groups can be configured to simplify repetitive BGP Configurations, however Peer Group misconfigurations can prevent BGP neighbors from forming relationships and exchanging routes
- Timers – Timers do NOT need to match! However if neighbors “minimum holddown from neighbor” timer is not set it can prevent neighbor adjacencies
Interfaces being down for BGP can be either Physical or Logical, as seen in my configuration in the Part 1 Redistribution lab I used Loopback interfaces instead of Physical interfaces to create the Adjacency, and if either side loopback was not in an Up/Up state the neighbor Adjacency would not form.
Layer 3 connectivity to the neighbor can be tested with the “ping” command between neighbor routers, and neighbor will also show a “idle” in the “sh bgp ipv4 unicast summary” output, or if no neighbor adjacency is formed look up the running configuration router bgp section, this is likely solved by pointing a static route to the neighbor IP as depending on Dynamic Routing protocols to share a router to the neighbor adds another layer of troubleshooting to the BGP Neighbor adjacency.
Default routes can allow the “ping” test to work as Layer 3 connectivity is there, but because BGP will not form an adjacency from a default route the command “sh ip route (neighbor IP)” should be issued to see if a route exists specifically for the neighbor IP, if not checking the “sh ip route” should confirm a default route is allowing the ping to succeed, and likely a static route is needed.
Neighbors not having a return route is also an issue that will cause an “idle” status on the “sh bgp ipv4 unicast summ” command output, if the local router is not using a default route and the remote neighbor IP is responding to pings, it is very likely the remote BGP neighbor is using a default route!
^^^ Very important concept to know, as it is obviously not completely missing a route if it returns pings, however if it is not establishing a TCP Session / BGP Adjacency over a default route so this is likely the issue if you cannot look at the BGP Peer in the cloud for any reason!
Incorrect neighbor statements can result in two different issues depending on the portion of “neighbor 10.10.10.1 remote-as 65000″ statement, if the IP Address is incorrect the neighbor will remain in an “idle” state in the “sh bgp ipv4 summ” command, if the IP is correct but the remote-as is wrong the TCP Handshake will finish and the adjacency will flap between Active and Idle.
To troubleshoot this the “sh run | sec router bgp” to review the neighbor statements is useful along with “sh tcp brief all” to verify TCP Sessions open on the local router.
BGP Packets sourced from the wrong IP Address can be an issue if the neighbor IP used is tied to a physical interface that can become unavailable, as the BGP Router may be able to reach the router through a different route, however that BGP Hello / Session will now be coming from a source that is not the neighbors defined IP Address.
This is a problem that is easily solved by using Loopback interfaces for Peering!
I have always practiced setting up BGP neighbors by using a loopback interface as it cannot go down unless Admin shutdown, then adding “neighbor x.x.x.x update-source lo#” after you enter the initial “neighbor x.x.x.x remote-as #” statement.
This can be verified reviewing the “sh run | sec router bgp” output and the “sh ip int bri” command to verify if the link assigned to the neighbor IP is admin down or there is some other issue or the interface is Up/Down figuring out what the logical issue is.
ACL’s are extremely tricky with BGP and require a thorough explanation
Seeing an ACL that is blocking BGP in one direction will not stop a neighbor relationship from forming under the right circumstances, with these two types of ACL’s for example:
deny any any eq bgp
or
deny any eq bgp any
This is because both BGP routers both attempt to initiate the TCP Session over Port 179, however one of them will end up maintaining the session on Port 179 (This is the BGP Server), and the other will use a random TCP Port to maintain the session (This is the BGP Client).
I think of it very similar to IPSec VPN in the way that one end of the Tunnel is the “Initiator” of the Site-to-Site VPN Session, and the other is not, though VPN does have set ports it utilizes on both sides so that is a difference from BGP.
To go one level deeper, BGP will use a random “source” Port when initiating the TCP Session and use TCP 179 for the Destination for the 3-way handshake, however the Server/Client relationship can be influenced so it does not matter if BGP is denied from sending 179 from one way or the either but it cannot be blocked in both directions!
The command within the router bgp process that allows the ACL’s to not matter (if configured correctly) is “neighbor x.x.x.x transport connection-mode (active/passive)” which is a little counter-intuitive because the router configured with “active” become the BGP Client and the Passive router will become the BGP Server.
This is because of how both sides send their initial TCP Hello with a Destination of 179 and a random TCP Source Port, the side Actively trying to establish the Adjacency will tell the remote router “I want to talk at you on Port 179” and the router then responds “OK I will talk to you on your random Port you sent this to me on”
The random port technically being called an “ephemeral port” which is not technically “random” by definition but for the purpose of simplicity, I am calling them random 🙂
To visually demonstrate exactly how this can and cannot work I’ve drawn this up:
The direction (in/out) and ACL # with a red X through it will not work in the way the TCP Hello packet is going based on the BGP routers configured connection-mode, this is a very tricky niche I might expect to see on an exam such as TSHOOT.
Take some time to look at the ACL, at the BGP Router Type, and the direction!
It took me about 10 minutes to REALLY think about the effects in both direction for both lists on both routers, and that is something you will want to understand BEFORE you have a clock ticking down on the exam PC screen 🙂
You can view which port is being used locally to tell whether it is using 179 indicating the local router is the BGP Server, or if its using an ephemeral port indicating it is the BGP Client in the relationship, in the “sh bgp ipv4 unicast nei” command somewhere in the output.
To see specifically the port #’s, issue “sh bgp ipv4 unicast nei | i ^BGP neighbor|Local port|Foreign port” exactly, however it may just be easier to parse through the output rather than memorize that syntax – However it is there to use!
I’ve also included the Loopback neighbor adjacency in there as well to demonstrate how that works, all that is required is a static route pointing the routers to those loopbacks across the link and those neighbors are formed over loopback addresses.
/acl rant
TTL BGP Packet expires are a result of a neighbor being too many hops away, as an eBGP Peer is expected to be directly connected or one hop away, whereas iBGP Peers can be several hops away (a maximum of 255 to be exact).
The hop count can also be seen in the “sh bgp ipv4 unicast neigh” output, with a narrower search of “sh bgp ipv4 unicast nei | i BGP nei|TTL” to show the exact hop counts to neighbors, which the trend will be seen that any neighbor in a remote-as that does not match the local router bgp process # / AS # will be 1 hop.
***Quick side note for BGP verification, remember “unicast” in there!***
For some reason it is incredibly forgettable for me, so wanted to make that a point 🙂
Now, back to hopping, and this is THE important point of it for exam day!
Being that eBGP Peers are expected to be directly connected, you CANNOT use loopback interfaces to form neighbor adjacencies for eBGP Peers by default, because the neighbor can only be 1 hop away and a Loopback interfaces first hop is the local router!
iBGP can have up to 255 hops before the TTL packets expires, however by default, eBGP neighbors CANNOT be more than 1 hop away or directly connected.
This is from the Redistribution Part 1 (lab config):
R2(config)#ip route 3.3.3.3 255.255.255.255 10.1.23.2
R2(config)#router bgp 200
R2(config-router)#neighbor 3.3.3.3 remote-as 100
R2(config-router)#neighbor 3.3.3.3 update-source lo22
R2(config-router)#neighbor 3.3.3.3 ebgp-multihop 2
R3(config)#ip route 2.2.2.2 255.255.255.255 10.1.23.1
R3(config)#router bgp 100
R3(config-router)#neighbor 2.2.2.2 remote-as 200
R3(config-router)#neighbor 2.2.2.2 update-source lo33
R3(config-router)#neighbor 2.2.2.2 ebgp-multihop 2
Even though these two BGP routers are directly connected, the loopback interface configuration to form the adjacency adds a hop, so the “ebgp-multihop #” neighbor statement is needed as well for each hop away the neighbor is.
This may require a higher value if there is multiple hops to the neighbor, so use as needed, but be aware that eBGP Peers will not form unless they are configured directly to each others directly connected interface or the multihop configuration is in use!
Authentication with BGP is amazingly easy, even though it does use MD5, to enable it you simply need to type “neighbor x.x.x.x password (something)” that matches on each side and it is just that easy. No Key Chains, no Key IDs, just the password command.
It will be in plain text if “service password-encryption” is not issued on the router, so it is advised to issue this on both routers, especially if comparing passwords and only one side has the password encrypted via service password-encryption.
Peer Group Misconfiguration can fall into a few different categories, but first I want to compare from my output above, and do it for per neighbor and then with a Peer Group:
R2(config)#router bgp 200
R2(config-router)#neighbor 3.3.3.3 remote-as 100
R2(config-router)#neighbor 3.3.3.3 update-source lo22
R2(config-router)#neighbor 3.3.3.3 ebgp-multihop 2
R2(config-router)#neighbor 4.4.4.4 remote-as 100
R2(config-router)#neighbor 4.4.4.4 update-source lo22
R2(config-router)#neighbor 4.4.4.4 ebgp-multihop 2
R2(config-router)#neighbor 5.5.5.5 remote-as 100
R2(config-router)#neighbor 5.5.5.5 update-source lo22
R2(config-router)#neighbor 5.5.5.5 ebgp-multihop 2
R2(config-router)#neighbor 6.6.6.6 remote-as 100
R2(config-router)#neighbor 6.6.6.6 update-source lo22
R2(config-router)#neighbor 6.6.6.6 ebgp-multihop 2
R2(config-router)#neighbor 7.7.7.7 remote-as 100
R2(config-router)#neighbor 7.7.7.7 update-source lo22
R2(config-router)#neighbor 7.7.7.7 ebgp-multihop 2
That was hard to do just using absolutely generic IP’s with 2 configurations, yuck.
Using a Peer Group, that same configuration is entered with the following syntax:
R2(config)#router bgp 200
R2(config-router)#neighbor TSHOOT peer-group
R2(config-router)#neighbor TSHOOT remote-as 100
R2(config-router)#neighbor TSHOOT update-source lo22
R2(config-router)#neighbor TSHOOT ebgp-multihop 2
R2(config-router)#neighbor 3.3.3.3 peer-group TSHOOT
R2(config-router)#neighbor 4.4.4.4 peer-group TSHOOT
R2(config-router)#neighbor 5.5.5.5 peer-group TSHOOT
R2(config-router)#neighbor 6.6.6.6 peer-group TSHOOT
R2(config-router)#neighbor 7.7.7.7 peer-group TSHOOT
R2(config-router)#neighbor 8.8.8.8 peer-group TSHOOT
Peer-Groups can contain any kind of modifiers, including route filters / route-maps / transport connection-mode (active/passive) / next-hop-self / Etc.
So one Peer Group is not one size fits all, however for neighbors that it does fit, that is a POWERFUL command to have at your disposal. Being that it is so powerful and important to understand, I fully expect to see it on the TSHOOT exam!
So we better know what can wrong with them specifically:
- Missing or misconfigured neighbor IP – The Peer Group won’t function properly if the neighbor statement associated with the peer-group is incorrect!
- One size doesn’t fit all – Neighbors should not be associated with the Peer Group that it doesn’t work for, for example a neighbor that is more than 2 hops away from my example Peer Group configured above
- Route-Filter or Route-Map for all neighbors – Essentially the same as above, if a Route Filter or Route-Map is configured to the Peer Group, check it carefully for configurations that might work for one Peer but not another
- Neighbor Statements override Peer Group statements – Going back to my example above, if a statement was entered “neighbor 4.4.4.4 update-source lo30” then it will override the update-source command in the Peer Group, though the neighbor will still have the modifiers from the Peer Group that single neighbor statement will override the statement defined by the Peer Group
That bottom one is a double-edge sword as it can cause issues to occur, however it also adds some flexibility to add special configs for a neighbor if needed, though it really defeats the point of the neighbor being part of the Peer Group in general in cutting down on the CPU / Bandwidth cost as that neighbor will need it own separate update.
So a rogue neighbor statement won’t cause the neighbor necessarily to fail, though if seen on exam day (or in the real world), it is something to verify whether its an issue.
Timers with BGP do not have to match for an Adjacency to form, and the default timers are 60sec Keepalive / 180 Holdtime, and the neighbors will adjust to use the lower holdtime if connected with one caveat.
If a “minimum holdtime” is configured on a neighbor, that is saying that it will not accept any holdtime lower than that value, it will go up in value from that hold time but not below this. This is to ensure a neighbor is not too aggressive with its timers.
The point being with Timers is they do not have to match, but if a “minimum holdtime” is set than the neighbors holdtime cannot be lower than that, or the adjacency won’t form.
Verify with “sh bgp ipv4 unicast | i hold time|holdtime” or “sh run | sec router bgp” and to adjust timers in router bgp process with “bgp timers # #” for the local router and “neighbor x.x.x.x timer # #” to set a neighbors timer values to match.
Timers should also be viewed via “sh ip bgp neighbors” as well.
Troubleshooting BGP routes
Once a neighbor adjacency is formed between BGP neighbors they will exchange routes, and if they do are adjacent but not exchanging routes, knowing the possible reasons why is critical to getting through the TSHOOT exam and real world!
The following is a list of possible issues prevent route exchange with BGP:
- Missing or incorrect network mask – Must match the exact prefix in the IP Route Table to be advertised into BGP
- Next-Hop router unreachable – Whether interface is down, device is down, or ebgp multi-hop is not configured
- BGP Split Horizon – iBGP Peers do not shares routes with other iBGP Peers if the route was learned via BGP
- Better source for route – Administrative Distance
- Route Filtering – Some mechanism preventing the route from either being shared from the remote neighbor or learned on the local router
First things first, to verify routes in the local BGP Routing Process, use “sh ip bgp” or “sh bgp ipv4 unicast” to view known routes, routes will appear here if they:
- Is learned for another BGP Neighbor advertisement
- “network mask” command matches a local route in the IP Route Table
- Redistribution is happening from another Routing Domain
- A summary route is created within BGP
It can be difficult to determine where routes come from the output, but there are a few tell tale signs in the output that can ID routes:
R3#sh ip bgp
BGP table version is 3, local router ID is 33.33.33.3
Status codes: s suppressed, d damped, h history, * valid, > best, i – internal,
r RIB-failure, S Stale, m multipath, b backup-path, x best-external, f RT-Filter
Origin codes: i – IGP, e – EGP, ? – incomplete
Network Next Hop Metric LocPrf Weight Path
* i10.1.23.0/30 2.2.2.2 0 100 0 i
*> 0.0.0.0 0 32768 i
*> 33.33.33.0/24 0.0.0.0 0 32768 i
R3#
From the above output a few things are indicative of where routes are learned from:
- “i” in Path means route was injected via “network” or “summary-address”
- “?” in Path means route was Redistributed from another Routing Domain
- 0.0.0.0 Next Hop indicates local router originated the route
- 2.2.2.2 (or another IP) Next Hop indicates the route originated from the neighbor with that IP Address for the Next Hop
If course “sh ip route” will show the BGP routes that are injected into the IP Roue table.
For the network / mask be sure the exact prefix is in the IP Route table, or it will not be injected into BGP.
The Next Hop router not being available can be fixed in several ways via advertising route via IGP Protocol, static route, or if iBGP using the next-hop-self command which is fully “neighbor 2.2.2.2 next-hop-self” for iBGP peers to advertise the local router as the Next Hop to a downstream iBGP Peer which avoids needing routing through multiple hops to the eBGP peer upstream.
This would change the “Next Hop” column in the router unable to reach the Next Hop peer address to the actual next hop router the command is issued on, which will bridge the gap between one hop to the other.
^^^ I would expect to see the next-hop-self” in some form on exam day.
These can be spotted in the “sh ip bgp” or “sh bgp ipv4 unicast” output at the very left of the Next Hop address codes, if a Route is valid but is not flagged as a best route candidate, then it is indicative that the Next Hop may be unreachable.
BGP Split Horizon is different from Distance Vector protocol split horizon in the way that the rule states that a route learned via an iBGP Peering shall not advertise the route to another iBGP Peer, and that is a very important rule and distinction to make.
Pulled from a lab I did on this subject found here this is a way to verify BGP Split-Horizon by viewing a specific route with “sh ip bgp x.x.x.x” :
R1#sh ip bgp 5.5.5.5
BGP routing table entry for 5.5.5.5/32, version 2
Paths: (1 available, best #1, table Default-IP-Routing-Table)
Not advertised to any peer
Local
172.12.15.5 from 172.12.15.5 (5.5.5.5)
Origin IGP, metric 0, localpref 100, valid, internal, best
R1#
Again the BGP Split-Horizon rule is “A BGP Router that receives a BGP Route via an iBGP Peering shall not advertise that route to another router that is an iBGP Peer”
So if we are talking iBGP, we have next-hop-self to bridge gaps to eBGP Neighbors that are otherwise unreachable, and also iBGP Split Horizon to watch out for!
Better source for information depends wildly on whether it is learned via eBGP (AD 20) or iBGP (AD 200) whether the route will be preferred, this huge difference in AD is because BGP is meant to be a Transport protocol that exchanges routes between Autonomous Systems, so to get to that remote eBGP AS the AD is lower than IGP AD’s so their route is preferred while the iBGP routes less preferred that IGP’s AD values.
In the “sh ip bgp” or “sh bgp ipv4 unicast” information it may show a route with “r>” instead of “*>” which indicates a RIB failure, which means not that the route is necessarily invalid but that a better source for it was found.
To determine that source use “sh ip route x.x.x.x” for verification, or “sh bgp ipv4 unicast rib-failure” command to see the RIB failure reasons, which will show “Higher admin distances” if a better source for information was found.
Route filtering is a gigantic topic that will only be covered from a route verification and possibly point of contention standpoint here, so for this topic I will go over some commands and possible route filter mechanisms to watch for on exam day:
- “sh bgp ipv4 unicast nei x.x.x.x routes” – This will show all BGP learned routes from the specified neighbor ID installed in the BGP Route table
- “sh bgp ipv4 unicast nei x.x.x.x advertised-routes” – This will show the routes being advertised from the specified neighbor IP even though they may not appear in the BGP Route table indicating they are being filtered
- “sh ip proto” – Shows inbound and outbound filtering via Distribute-List / ACL / Route-Map / Prefix-List to then verify via the appropriate show command if the route expected is being filtered
BGP Path Selection troubleshooting
BGP uses the following 11 attributes in order to make its path selection:
- Highest weight – Default is 0
- Highest local preference – Default is 100
- Route originated from local router – Next Hop 0.0.0.0
- Shortest AS Path – The remote-as # to route
- Lowest origin code – IGP (configured via network statement) is preferred over EGP
- Lowest MED (metric) – Default Metric is 0 unless configured via Route-Map
- External preferred over Internal – eBGP preferred over iBGP
- Path to closest IGP neighbor – Closest IGP Metric to destination (lowest)
- Oldest route for eBGP path – Age of routes in “sh ip bgp”
- Path with lowest neighbor BGP RID – Lowest configured RID wins, if none tied
- Path with lowest neighbor IP Address – Lowest neighbor IP final tie breaker
To check these the following commands can be used:
- “sh ip bgp”
- “sh ip bgp x.x.x.x”
- “sh ip bgp summ”
- “sh bgp ipv4 unicast …” – The other way of inputting above commands
R3#sh ip bgp
BGP table version is 3, local router ID is 33.33.33.3
Status codes: s suppressed, d damped, h history, * valid, > best, i – internal,
r RIB-failure, S Stale, m multipath, b backup-path, x best-external, f RT-Filter
Origin codes: i – IGP, e – EGP, ? – incomplete
Network Next Hop Metric LocPrf Weight Path
* i10.1.23.0/30 2.2.2.2 0 100 0 i
*> 0.0.0.0 0 32768 i
*> 33.33.33.0/24 0.0.0.0 0 32768 i
R3#
R1#sh ip bgp 5.5.5.5
BGP routing table entry for 5.5.5.5/32, version 2
Paths: (1 available, best #1, table Default-IP-Routing-Table)
Not advertised to any peer
Local
172.12.15.5 from 172.12.15.5 (5.5.5.5)
Origin IGP, metric 0, localpref 100, valid, internal, best
R1#
R1#sh ip bgp summary
BGP router identifier 22.22.22.22, local AS number 100
BGP table version is 1, main routing table version 1
Neighbor V AS MsgRcvd MsgSent TblVer InQ OutQ Up/Down State/PfxRcd
172.12.15.5 4 500 35 33 1 0 0 00:12:52 0
172.12.123.3 4 100 12 12 1 0 0 00:08:18 0
R1#
These three verification commands should pull you through best path selection, also another great one of course is to just review the running configuration with a “sh run | i router bgp” output, although for Path Selection you will need dynamically learned information from neighbors that can’t be found in local configuration!
BGP for IPv6 / Multi-Protocol BGP / MP-BGP Troubleshooting
As with all protocols, it operates exactly the same on IPv6 as it does on IPv4, and TCP sessions to form neighbor adjacencies can be configured with an “Address Family” using the IPv4 AF configuration to enter the Neighbor statements, then the IPv6 AF configuration to enter Neighbor statements.
The only real difference between the two is the usual differences in any Dynamic Routing Protocol using IPv6, however the troubleshooting verification commands will use “sh bgp ipv6 unicast …” instead of “sh bgp ipv4 unicast …” and that is really the biggest difference there is to mentioned.
Beyond that there is honestly no more that needs to be mentioned for MP-BGP as it operates exactly the same with Path Selection and everything as IPv4 BGP, its just being able to understand the IPv6 addressing that may come up in question.
However given the Cisco published TSHOOT exam topologies, I would not expect to see MP-BGP at all on exam day, unless possibly a multiple choice question, which if you understand how BGP works for IPv4 and IPv6 Addressing you will be gold for exam day!
This will conclude this monster of a BGP Troubleshooting article
I believe this review will be more than enough for BGP on the TSHOOT, as again all published Topologies have a single entry / exit point for BGP, so I would be amazed if even half of the information on this page will be necessary to know.
However I would make sure to understand these concepts before sitting the exam, as in my experience with Cisco exams, the topics you are tested on are the ones you least expected to be tested on 🙂
The DEVNET GRIND! 