Above is a graphic of my own Meraki stack at my home, however this will only be reviewed in discussion of network “Planes” before we close out MPLS discussion 🙂
In this final MPLS post I wanted to first get a very clear understanding of Management Plane vs Control Plane vs Data Plane, then take a look at how VPNv4 traffic is traversing the MPLS network, and finally a quick look at VRF’s and the Import / Export function to make sense of what is happening when routes are shared via VRF and mBGP!
Management Plane vs Control Plane vs Data Plane
Wanted to hit this quick first to make sure we are all on the same page up front.
The Management Plane as I had discussed in TSHOOT is how we as Network Engineers access the network devices, whether it be a CLI tool such as Telnet or SSH, or something like ASDM / CCP / Web GUI such as a Meraki Dashboard.
The Meraki Dashboard is a perfect example of mixing the Management Plane with the Control Plane, as you both “Manage” and “Control” your network traffic from the same window pane in a web browser as shown here on my home Meraki Dashboard:
This is how my Meraki Gear is managed (no CLI access at all), this is the actual “Management Plane” for my Meraki stack as this is how I manage it, however it is also the MPP (Management Plane Protection) and the “Control Plane” for my gear as I control rules and policies that dictates what traffic goes where.
There is of course other pages for Traffic Shaping / QoS / IP Routing / VPN / Etc, however I felt this was a perfect example of both Management and Control Planes visually from a Web GUI (which will be increasingly the utility going forward I believe), whereas with the CLI its a bit less visual.
Of course we are used to seeing both the Management and Control Planes as this:
A Telnet or SSH Client for “Management” of a device, the “Control Plane” building from protocols coming up that will form our network (OSPF, EIGRP, LDP, BGP), which is where we have been working at for the last 20+ years.
The “Data Plane” is simply the hardware doing the “Control Planes” bidding, and forwarding the network traffic as absolutely fast as possible, which the more we remove the “Control Plane” from the hardware device the more power it can dedicate to the “Data Plane” or forwarding that traffic double time!
For this reason Cisco appears to be moving the Control Plane away from the devices, and using centralized Management / Control Plane “single pane” solutions such as Meraki Dashboard / Velocloud Orchestrator / Viptela vManagement, and this is a huge step away from traditional WAN Routing and Switching that we must know!
Not to scare anyone away from the future of how we interact with the networks, as I for one have been dubbed a CLI junky by more than one colleague, however looking forward to my studies of SD-WAN and Software Driven Networking is that the brains are being taken out of the individual devices and centralized.
Routers and Switches will not become door stops, as there will always need to be hardware on the ground to perform traffic forwarding, however I wanted to take a moment to really dissect these different “Planes” of existence and where they are headed with regards to the future of WAN Networking and SD-WAN which I assume is going to dominate the future of how companies connect to the Internet. Big Time.
Now that I have completely dragged MPLS and the CLI through the mud, lets take a look at what makes it work, and play with our now working MPLS network 🙂
Understanding all the MPLS Mechanisms that makes this network work
Though SD-WAN absolutely looks like the future of Networking, traditional MPLS / VPLS / OTT w/ MFA (Internet + Multi-Factor Auth) Cloud Applications will be around for a long time yet, so its a good idea to know and love them as we someday will SD-WAN!
For this reason we’ll review the “Control Plane” of our current network, starting with R1-PE booting up, and all the Control Plane protocols it has firing off immediately:
*Nov 29 09:45:06.351: %DUAL-5-NBRCHANGE: EIGRP-IPv4 102: Neighbor 172.16.101.2 (FastEthernet2/0) is up: new adjacency
*Nov 29 09:45:47.571: %OSPF-5-ADJCHG: Process 101, Nbr 172.16.111.2 on FastEthernet1/0 from LOADING to FULL, Loading Done
*Nov 29 09:45:48.179: %OSPF-5-ADJCHG: Process 1, Nbr 2.2.2.2 on FastEthernet0/0 from LOADING to FULL, Loading Done
*Nov 29 09:45:55.039: %LDP-5-NBRCHG: LDP Neighbor 2.2.2.2:0 (1) is UP
*Nov 29 09:46:04.299: %BGP-5-ADJCHANGE: neighbor 5.5.5.5 Up
EIGRP to “Back1” / OSPF to “Looped1” / OSPF to “R2-P” / LDP to “R2-P” / BGP to “R5-PE”
LDP is just represented in Green text because in this diagram I have no color assigned.
All of this configured on this network allows it to not only forward but tag traffic for downstream devices to know how to route them, all working together in sweet harmony, as shown again by this traceroute from Looped1 CE Router to the Looped2 CE Routers 8.8.8.8 Loopback Address:
Looped1#traceroute 8.8.8.8
Type escape sequence to abort.
Tracing the route to 8.8.8.8
VRF info: (vrf in name/id, vrf out name/id)
1 172.16.101.1 48 msec 48 msec 48 msec
2 10.12.0.2 [MPLS: Labels 201/507 Exp 0] 96 msec 96 msec 80 msec
3 10.23.0.3 [MPLS: Labels 305/507 Exp 0] 120 msec 72 msec 108 msec
4 10.34.0.4 [MPLS: Labels 405/507 Exp 0] 92 msec 72 msec 92 msec
5 192.168.202.1 [MPLS: Label 507 Exp 0] 72 msec 72 msec 76 msec
6 192.168.202.2 92 msec * 84 msec
Looped1#
To break down how this packet gets routed hop by hop:
- Packet is sent to R1-PE’s VRF 101:Looped Table, CEF shows both a VPNv4 and an MPLS Next-Hop Label to be imposed on the packet, and away it goes to R2-P
- R2-P only looks at the Top of Stack label 201, knows that is the route to 8.8.8.8, and swaps the label for 305 before sending to R3-P
- R3-P repeats this process
- R4-P Receives this packet, performs the PHP Label Popping operation on the TOP Label, leaves the VPNv4 BoS Label attached, and forwards the IP Packet
- R5-PE receives the IP Packet with a BoS Label bit set to 1, indicating this is a VPNv4 packet, and this directs it as to how to forward the packet to its final hop
- Looped2 CE Router receives the packet as though no special routing has occurred and the traffic is successfully delivered
Now first to point out for those who have never worked with a production MPLS network, you will NEVER(!!!) get label information back from a traceroute through it, there is a TTL configuration to disable to prevent that info from responding.
Next to take a closer look at a ping to 8.8.8.8 from “Looped1” showing both the Next-Hop Label along with the VPNv4 Label from a screen snip of a quick Wireshark capture I took of the ping between R2-P and R3-P Routers:
A few things here to note:
- The 3 “Traffic Class” bits are shown as “Experimental” which is why I earlier referred to them as both, this is the “QoS” field built into MPLS
- The TTL on the Top of Stack label that is actually being swapped decrements as it moves along, the VPNv4 Label does not because it is in “Transit” through the network meaning some other protocol / mechanism is moving the data
- The VPNv4 Label never changes, never swaps, it literally is just in transit like cargo on a delivery truck – Only when the truck runs out of gas and has to stop will its cargo stop as well while in transit!
Also going back to Hop #1 in our traceroute and referring to CEF making this decision, that is because CEF will always be referred to when performing Packet / Hardware Switching (Traffic Forwarding) decisions:
R1-PE#sh ip cef 8.8.8.8/32
%Prefix not found
R1-PE#
The Prefix is not found because this command is asking CEF to check its entry for the Global Routing table, which this will not be in, but rather will have been “Imported” via mBGP / VPNv4 to the VRF for Looped sites:
R1-PE#sh ip cef vrf 101:Looped 8.8.8.8/32
8.8.8.8/32
nexthop 10.12.0.2 FastEthernet0/0 label 201 507
R1-PE#
In fact lets take a look at ALL routes for both VRF’s configured on the CLI:
Note that although both networks on the left hand side have literally the exact same “WAN” connections IP Addressing, the VRF instances configured on the Provider Edge router interfaces point their routing information to their own respective VRF’s, thus allowing them to route independent of each other!
All of this information is figured out by the “Control Plane” the moment you see the Protocols show as “Up” when the routers boots up, and it is off to the races!
Adding an interface to a remote site, and watching it propagate the VPNv4 Network
One thing I thought was cool and wanted to demo here quick was the ability to Telnet from Looped1 to Looped2, create a new loopback interface on the router, and have it propagate back to Looped1 CE Router all in the same CLI session:
Looped1#sh ip route
(Codes redacted)
Gateway of last resort is not set
8.0.0.0/32 is subnetted, 1 subnets
O E2 8.8.8.8 [110/1] via 172.16.101.1, 03:45:52, FastEthernet1/0
172.16.0.0/16 is variably subnetted, 3 subnets, 2 masks
C 172.16.101.0/30 is directly connected, FastEthernet1/0
L 172.16.101.2/32 is directly connected, FastEthernet1/0
C 172.16.111.2/32 is directly connected, Loopback0
192.168.111.0/32 is subnetted, 1 subnets
O E2 192.168.111.2 [110/1] via 172.16.101.1, 03:45:52, FastEthernet1/0
192.168.202.0/30 is subnetted, 1 subnets
O E2 192.168.202.0 [110/1] via 172.16.101.1, 03:45:52, FastEthernet1/0
Looped1#
Looped1#
Looped1#telnet 192.168.202.2
Trying 192.168.202.2 … Open
Looped2#conf t
Enter configuration commands, one per line. End with CNTL/Z.
Looped2(config-if)#ip add 9.9.9.9 255.255.255.255
Looped2(config-if)#router rip
Looped2(config-router)#network 9.0.0.0
Looped2(config-router)#^Z
Looped2#wr
Building configuration…
[OK]
Looped2#exit
[Connection to 192.168.202.2 closed by foreign host]
Looped1#
Looped1#sh ip route
(Codes redacted)
Gateway of last resort is not set
8.0.0.0/32 is subnetted, 1 subnets
O E2 8.8.8.8 [110/1] via 172.16.101.1, 03:48:28, FastEthernet1/0
9.0.0.0/32 is subnetted, 1 subnets
O E2 9.9.9.9 [110/1] via 172.16.101.1, 00:00:17, FastEthernet1/0
172.16.0.0/16 is variably subnetted, 3 subnets, 2 masks
C 172.16.101.0/30 is directly connected, FastEthernet1/0
L 172.16.101.2/32 is directly connected, FastEthernet1/0
C 172.16.111.2/32 is directly connected, Loopback0
192.168.111.0/32 is subnetted, 1 subnets
O E2 192.168.111.2 [110/1] via 172.16.101.1, 03:48:28, FastEthernet1/0
192.168.202.0/30 is subnetted, 1 subnets
O E2 192.168.202.0 [110/1] via 172.16.101.1, 03:48:28, FastEthernet1/0
Looped1#
We would of course do not see Loopback9 interface go “Up” via line session because we do not have “term mon” command on to see console messages, but sure enough as quickly as we get back to Looped1 we now have a route propagated across the MPLS network up to Looped1 CE Router from Looped2 CE.
From the perspective of R1-PE, this route is being learnt from mBGP over the VPN:
R1-PE#sh ip route vrf 101:Looped
Routing Table: 101:Looped
Codes: L – local, C – connected, S – static, R – RIP, M – mobile, B – BGP
D – EIGRP, EX – EIGRP external, O – OSPF, IA – OSPF inter area
N1 – OSPF NSSA external type 1, N2 – OSPF NSSA external type 2
E1 – OSPF external type 1, E2 – OSPF external type 2
i – IS-IS, su – IS-IS summary, L1 – IS-IS level-1, L2 – IS-IS level-2
ia – IS-IS inter area, * – candidate default, U – per-user static route
o – ODR, P – periodic downloaded static route, H – NHRP, l – LISP
+ – replicated route, % – next hop override
Gateway of last resort is not set
8.0.0.0/32 is subnetted, 1 subnets
B 8.8.8.8 [200/1] via 5.5.5.5, 03:14:52
9.0.0.0/32 is subnetted, 1 subnets
B 9.9.9.9 [200/1] via 5.5.5.5, 00:06:52
10.0.0.0/24 is subnetted, 1 subnets
S 10.10.10.0 [1/0] via 172.16.101.2
172.16.0.0/16 is variably subnetted, 2 subnets, 2 masks
C 172.16.101.0/30 is directly connected, FastEthernet1/0
L 172.16.101.1/32 is directly connected, FastEthernet1/0
192.168.111.0/32 is subnetted, 1 subnets
B 192.168.111.2 [200/1] via 5.5.5.5, 03:14:52
192.168.202.0/30 is subnetted, 1 subnets
B 192.168.202.0 [200/0] via 5.5.5.5, 03:14:52
R1-PE#
To walk through this process of Import / Export step by step:
- Looped2 advertises the new route down to R5-PE via RIP
- R5-PE receives the route on the 102:Looped VRF instance which is configured to “Export” to 1.1.1.1 via mBGP (Exporting from VRF to BGP)
- The route is propagate across the VPNv4 / MPLS network to R1-PE
- R1-PE receives this route with the Route Target of VRF 101:Looped, which then “Imports” the route as defined with “import 5.5.5.5:101” in its configuration
- With the route now in the VRF’s IGP of OSPF, it shares the route up to Looped1
This is a very VERY straight forward way of how VPNv4 works, as there are ways to configure the Route Targets / Import and Export configurations to allow or “Route Leaking” between MPLS Rings either from the Global Route Table into a VRF instance or from VRF to VRF.
More on Route Leaking can be found here in Cisco official docs on it!
Here is a quick review of MPLS / VPNv4 / VRF troubleshooting commands
I was going to skip this as I am just fried from studying this A LOT to fully grasp the concepts, I wanted to throw in a few VRF / VPNv4 commands to assist in troubleshooting:
For showing “VRF” tables for basically anything, you just need to define the correct VRF in front of it, such as:
“sh ip route vrf 101:Looped”
“sh mpls forwarding-table vrf 101:Looped” – Will dig into this below
“sh ip cef vrf 101:Looped”
Etc.
The seemingly tricky “MPLS Forwarding-Table” output for VRF’s output is shown here, and we will decipher it to make sense:
Being this VRF / VPNv4 traffic is considered “Transit” traffic (Cargo on a truck), though this does not confirm from R1-PE that this label has been advertised to R5-PE, or that R5’s Labels have been advertised to this router.
For this we have a very special VPN specific verification command:
This command does require the newer format of BGP commands to identify specifically vpnv4 routing (like you would with “sh bgp ipv4 unicast …”), which shows a table of all the VPN Label Mappings the local router knows about.
Some other VRF related utilities you might use is to add a static IP route to a VRF:
R1-PE#conf t
Enter configuration commands, one per line. End with CNTL/Z.
R1-PE(config)#ip route vrf 101:Looped 100.100.100.0 255.255.255.0 10.12.0.2
R1-PE(config)#
That of course goes nowhere, but to demonstrate how that is done.
Also to ping or traceroute via a specific VRF:
R1-PE#ping 9.9.9.9
Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 9.9.9.9, timeout is 2 seconds:
…..
Success rate is 0 percent (0/5)
R1-PE#ping vrf 101:Looped 9.9.9.9
Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 9.9.9.9, timeout is 2 seconds:
!!!!!
Success rate is 100 percent (5/5), round-trip min/avg/max = 92/104/116 ms
R1-PE#
I will spare the traceroute for this specifically as it works the same with adding the VRF instance, however note that the first is trying to route from the Global Routing Table (where 9.9.9.9 does not exist), while the second uses the VRF where it DOES exist!
And finally for my MPLS / VRF troubleshooting commands, I have one more for you here:
R1-PE#traceroute mpls ipv4 5.5.5.5 255.255.255.255
Tracing MPLS Label Switched Path to 5.5.5.5/32, timeout is 2 seconds
Codes: ‘!’ – success, ‘Q’ – request not sent, ‘.’ – timeout,
‘L’ – labeled output interface, ‘B’ – unlabeled output interface,
‘D’ – DS Map mismatch, ‘F’ – no FEC mapping, ‘f’ – FEC mismatch,
‘M’ – malformed request, ‘m’ – unsupported tlvs, ‘N’ – no label entry,
‘P’ – no rx intf label prot, ‘p’ – premature termination of LSP,
‘R’ – transit router, ‘I’ – unknown upstream index,
‘l’ – Label switched with FEC change, ‘d’ – see DDMAP for return code,
‘X’ – unknown return code, ‘x’ – return code 0
Type escape sequence to abort.
0 10.12.0.1 MRU 1500 [Labels: 201 Exp: 0]
L 1 10.12.0.2 MRU 1500 [Labels: 305 Exp: 0] 88 ms
L 2 10.23.0.3 MRU 1500 [Labels: 405 Exp: 0] 44 ms
L 3 10.34.0.4 MRU 1504 [Labels: implicit-null Exp: 0] 448 ms
! 4 10.45.0.5 80 ms
R1-PE#
Whole new code table there to check out, pretty cool eh?
What is also cool is that this command will specifically check if the traffic is being Label Switched to its destination (and with which labels) so that you can identify the “break” in the path, this command is specific to MPLS is is probably geared more towards the deeper dive into subject matters that I will not be doing at this time.
With that I will conclude my MPLS Fundamentals here
I think that is more than enough understanding of MPLS for your modern day Network Engineer to understand working with and troubleshooting it, I hope this helps my fellow Cisco geeks out there across the globe, and I will see you soon with the next topic! 🙂
The DEVNET GRIND! 