MPLS – Now to apply the Network Ninja troubleshooting to get these CE Routers talking across the MPLS Network!

MPLS_EVE_WAN

Above is the full scope of the Topology that will be configured by the end of this, utilizing all different protocols for CE to PE, mainly to demonstrate it flexibility with accepting IGPs for all walk of life which is what this blog is all about 🙂

NOW, I had went ahead and used Keith Barkers uploaded config files to copy pasta a lot of node configurations (which I later removed to demo) to all CE / PE / P devices, and found I am unable to even ping between PE and CE routers in most cases.

One thing I did find interesting and wanted to really point this out

I thought when looking at “Back2” routers BGP config it must be wrong or not even possible as far as I learned it during my CCNP studies, as routers can only have a single “router bgp #” running at any single time – However this configuration is a way around this pesky limitation configured on R5-PE in Address-Family mode:

router bgp 65536
bgp log-neighbor-changes
neighbor 1.1.1.1 remote-as 65536
neighbor 1.1.1.1 update-source Loopback0
!
address-family vpnv4
neighbor 1.1.1.1 activate
neighbor 1.1.1.1 send-community extended
neighbor 1.1.1.1 next-hop-self
exit-address-family
!
address-family ipv4 vrf 101:Looped
redistribute rip
exit-address-family
!
address-family ipv4 vrf 102:Back
neighbor 192.168.202.2 remote-as 65537
neighbor 192.168.202.2 activate
exit-address-family

First we see the Autonomous System 65536 being used for our mBGP / VPNv4 configuration across the WAN, but then I noticed the bottom portion showing it connecting to what appears to be Backed2 sites BGP AS 65537:

BGP3

Both routes are Best and Valid on this table, indicating to me they are ready to rock

However I also highlighted in blue the VRF 101:Looped2 in the BGP Address-Family configuration, as I noticed its main config that pops up to me is the “Redistribution” obviously indicating that it is Redistributing this IGP into mBGP across the L3 VPN.

Now to begin the Configuration / Fix starting with Looped1 CE and R1-PE

I really didn’t think it would be this difficult, but it is fighting me here:

Looped1(config)#router ospf 1
Looped1(config-router)#network 172.16.0.0 0.0.15.255 area 101
Looped1(config-router)#log-adjacency-changes
Looped1(config-router)#^Z
Looped1#do
*Nov 25 04:00:17.253: %SYS-5-CONFIG_I: Configured from console by console
Looped1#wr

The OSPF Process is locally significant only, and is only taken into consideration when running multiple OSPF Processes on a single router, so this should work fine with OSPF Router process 101 configured on the other side (as it uses Process #1 for the WAN):

R1-PE(config)#router ospf 101 vrf 101:Looped
R1-PE(config-router)#network 172.16.0.0 0.0.15.255 area 101
R1-PE(config-router)#log-adjacency-changes
R1-PE(config-router)#^Z
R1-PE#wr

Adding the OSPF Process tied to the VRF, network 172.16.0.0/12 in Area 101

R1-PE(config)#router bgp 65536
R1-PE(config-router)#address-family ipv4 vrf 101:Looped
R1-PE(config-router-af)#redistribute ospf 101
R1-PE(config-router-af)#exit
R1-PE(config-router)#exit
R1-PE(config)#do wr
Building configuration…
[OK]
R1-PE(config)#

Redistributing the VRF Instance for Customer “Looped1” into the mBGP VPN, however I am still getting this and don’t fully understand why this is:

R1-PE(config)#do sh ip ospf nei

Neighbor ID Pri State Dead Time Address Interface
2.2.2.2 1 FULL/DR 00:00:32 10.12.0.2 FastEthernet0/0
R1-PE(config)#

Ah Ha! I took a page out of my TSHOOT studies and figured it out, as I was trying to enter matching “network” statements, I forgot that just enabled the directly connected interfaces in the common Area they would share (although it takes EVE FOREVER to bring up an Adjacency) it worked!!! :

Looped1(config)#int fa1/0
Looped1(config-if)#ip ospf 1 area 101
Looped1(config-if)#
*Nov 25 04:11:38.861: %OSPF-5-ADJCHG: Process 1, Nbr 172.16.101.1 on FastEthernet1/0 from LOADING to FULL, Loading Done
Looped1(config-if)#

I was just starting to say what on Earth now and look into config examples, and Up comes the Adjacency, hooray! So I will go around making changes, and documenting it here for my own review as well as yours should you happen to care at all 🙂

For R1-PE’s EIGRP connection to Customer Back1 site, we have this:

!
router eigrp 1
network 172.16.0.0
redistribute bgp 65536 metric 1 1 1 1 1
!

That is definitely not going to cut it, as we have to define the VRF instance as well:

R1-PE(config)#router eigrp ?
<1-65535> Autonomous System
WORD EIGRP Virtual-Instance Name

R1-PE(config)#router eigrp 1
R1-PE(config-router)#address-family ipv4 ?
unicast Address Family Unicast
vrf Specify a specific virtual routing/forwarding instance

R1-PE(config-router)#address-family ipv4 vrf ?
WORD Virtual Routing/Forwarding instance name

R1-PE(config-router)#address-family ipv4 vrf 102:Back
R1-PE(config-router-af)#network 172.16.0.0
R1-PE(config-router-af)#redistribute bgp 65536 metric 1.1.1.1

Got kind of an odd message exiting Address-Family EIGRP mode on R1-PE:

R1-PE(config-router-af)#redistribute bgp 65536 metric 1 1 1 1 1
R1-PE(config-router-af)#^Z
%Warning: EIGRP Autonomous-System number must be provided
R1-PE#s
*Nov 25 05:47:05.376: %SYS-5-CONFIG_I: Configured from console by console
R1-PE#

Not sure what that is going to do, however lets hop onto Customer Back1 site:

!
router eigrp 102
network 0.0.0.0
!

So that is kind of an “Advertise Anything” command for Back1 CE Router, so I am not entirely understanding the issue here, maybe there is none?

R5-PE#sh ip eigrp nei
R5-PE#

Nope, that is an issue, if I recall I think EIGRP is actually the IGP that requires matching network numbers EXACTLY and in fact timers as well, so let me adjust this up here:

Found it, doh:

R1-PE(config)#router eigrp 1
R1-PE(config-router)#address-family ipv4 ?
unicast Address Family Unicast
vrf Specify a specific virtual routing/forwarding instance

This is where things must have gone wrong, so now to do some fixing up here:

R1-PE(config)#no router eigrp 1
R1-PE(config)#router eigrp 102
R1-PE(config-router)#address-family ipv4 vrf 102:Back
R1-PE(config-router-af)#network 172.16.0.0
R1-PE(config-router-af)#redistribute bgp 65536 metric 1 1 1 1 1
R1-PE(config-router-af)#

Unlike OPSF where the “router … #” is a locally significant process, the “router … #” for EIGRP needs to match for routers to be considered in the same AS, which is why it sounds right that all other values to form neighbor relationships without looking seem picky to me about EIGRP.

And there is no “ip eigrp 102” for Interface level commands, and in fact nothing for EIGRP as far as the interface goes to make things easy, so debugging it is after just a bit more stare and compare of configs here – I am determined not to look for help in training materials or cisco documentation!

Back1#debug ip eigrp
EIGRP-IPv4 Route Event debugging is on
Back1#
Back1#
Back1#! ummmm
Back1#u all
All possible debugging has been turned off
Back1#

How often do you debug a protocol for ALL traffic, and get absolutely no output?

R1-PE(config)#no router eigrp 102
R1-PE(config)#router eigrp 1
R1-PE(config-router)#address-family ipv4 vrf 102:Back autonomous-system 102
R1-PE(config-router-af)#network 172.16.0.0
R1-PE(config-router-af)#
*Nov 25 06:02:05.220: %DUAL-5-NBRCHANGE: EIGRP-IPv4 102: Neighbor 172.16.101.2 (FastEthernet2/0) is up: new adjacency
R1-PE(config-router-af)#

That is why I was getting griped at about not having an AS, interestingly I can make the router AS # whatever within the initial configuration, but it is the AS # configured within the Address-Family vrf AS # that will define what will be matched for AS to the CE router.

So with EIGRP / MPLS / VRF / mBGP VPNv4 stuff, use an AS # on the end of the Address-Family configuration to define which AS is configured on the customer site router 🙂

Now onto the R5-PE to see if we have any customer activity over that way

R5-PE#sh ip rip nei
R5-PE#

RIPv2 is not happening at the moment.

BGP4

There isn’t any BGP connections showing between R5-PE to Back2 routers, however our mBGP / VPNv4 tunnel is up and passing traffic between the two neighbors.

!
router rip
version 2
!

I am not sure what I was actually thinking with that small barren configuration on R5-PE for RIPv2, so lets give this another try here:

R5-PE#conf t
Enter configuration commands, one per line. End with CNTL/Z.
R5-PE(config)#no router rip
R5-PE(config)#router rip
R5-PE(config-router)#ver 2
R5-PE(config-router)#address-family ipv4 vrf 101:Looped
R5-PE(config-router-af)#network 192.168.202.0
R5-PE(config-router-af)#no auto
R5-PE(config-router-af)#

That seems like the more reasonable configuration on the PE Router given the complexity of its configuration given all this complexity, but I decided to do a “clear ip bgp *” on R5 just to see if we there is any life from “Back2” site:

R5-PE#clear ip bgp *
R5-PE#
R5-PE#
*Nov 25 05:32:41.968: %BGP-5-ADJCHANGE: neighbor 1.1.1.1 Down User reset
*Nov 25 05:32:41.968: %BGP_SESSION-5-ADJCHANGE: neighbor 1.1.1.1 VPNv4 Unicast topology base removed from session User reset
*Nov 25 05:32:41.972: %BGP_SESSION-5-ADJCHANGE: neighbor 1.1.1.1 IPv4 Unicast topology base removed from session User reset
*Nov 25 05:32:41.976: %BGP-5-ADJCHANGE: neighbor 192.168.202.2 vpn vrf 102:Back Down User reset
*Nov 25 05:32:41.976: %BGP_SESSION-5-ADJCHANGE: neighbor 192.168.202.2 IPv4 Unicast vpn vrf 102:Back topology base removed from session User reset
*Nov 25 05:32:42.916: %BGP-5-ADJCHANGE: neighbor 192.168.202.2 vpn vrf 102:Back Up
R5-PE#
*Nov 25 05:32:43.252: %BGP-5-ADJCHANGE: neighbor 1.1.1.1 Up
R5-PE#

Sure enough there is, so its not that BGP was not forming to “Back2” CE Router, there is just a difference way of looking at VRF specific information.

Back on “Looped1” CE to see if RIP is cooking:

RIP

Looks like since updating the configuration and letting it sit for awhile, it connected up and is chugging along! Now comes the big test, site to site connectivity:

MPLS_EVE_WAN

 

Upon failing from “Looped1” CE to ping “Looped2” CE’s loopback, I realized in a classful routing protocol such as RIPv2, it will summarize routes to a Class C /24 address, meaning I would need to add the loopback 192.168.111.0 to the network statements to get a response from its loopback at from any devices that can reach the router:

Looped2(config)#router rip
Looped2(config-router)#ver 2
Looped2(config-router)#network 192.168.111.0
Looped2(config-router)#network 8.0.0.0
Looped2(config-router)#

Aaaaaaaaaaand:

Looped1#ping 192.168.111.2
Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 192.168.111.2, timeout is 2 seconds:
…..
Success rate is 0 percent (0/5)
Looped1#

This seems like a good place to halt my horses, I am completely brain dead tired, and I seem to be doing more harm then good towards a solution right at the moment, so I will pick up where I left off here in the next post to make sure we have a fully functioning MPLS Network with two customers passing traffic to take a look at a WireShark packet capture with all the different traffic labeling attached to packets as they ride the MPLS!

So that will conclude tonight labbing!

Would have liked to finish that up, but I know when to throw in the towel when staring at the CLI longer than I should be, so I will tackle this next round – See you then!

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s