I have to honestly say, I am so relieved (for now) to have this smaller Topology to demonstrate how and why Front Door VRF’s work, why they work and why we care about them at all when no Cisco training material mentions them.
Warning – This is another post that doesn’t really teach you how to do anything, but shows my struggle to get a technology working, that eventually fails and I tap out.
I am very much actively looking into getting this working, and amending this post when I can conquer this problem, but for now I am retreating for further studies to emerge victorious at a later time 🙂
So don’t expect to learn much here except maybe my troubleshooting train of thought.
What is a Front Door VRF and why do I can if its configured on my network?
Front Door VRF’s confused the daylights out of me (and still kind of do) when I first saw them, and upon researching, it appears it kind of tucks away your “Underlay” routing information into its own VRF Instance while leaving the Global IP Route Table alone at Branch locations – So kind of a way of just cleaning up Routing at all Branch locations.
This also makes sure a Branch Manager can’t order a secondary Ethernet pipe on their own to increase bandwidth without consulting their IT Department (like they would ever do that?), and messing up site routing / possibly taking the site down until IT can swoop in and save the day amidst pitch fork and torch mobs chanting about Facebook access!
That said lets configure us some Front Door VRF on our Hub and Spoke routers!
I’ll go through the step by step, explaining the steps as we go, as always for my readers:
Step 1 – Create a “common” VRF to bed used by all sites
PHX1(config)#vrf def FrontDoor
PHX1(config-vrf)#address-family ipv4
PHX1(config-vrf-af)#exit
Seriously do not forgot the “address-family ipv4” or any interface you try to apply it to will kick an error saying the af IP # has not been defined, so don’t forget this step!
Step 2 – Applying it to the WAN Interface
PHX1(config-if)#vrf forwarding FrontDoor
% Interface FastEthernet0/0 IPv4 disabled and address(es) removed due to enabling VRF FrontDoor
PHX1(config-if)#
*Dec 11 10:39:15.207: %LINEPROTO-5-UPDOWN: Line protocol on Interface Tunnel0, changed state to down
*Dec 11 10:39:15.271: %OSPF-5-ADJCHG: Process 1, Nbr 172.16.123.14 on FastEthern et0/0 from FULL to DOWN, Neighbor Down: Interface down or detached
*Dec 11 10:39:15.499: %DUAL-5-NBRCHANGE: EIGRP-IPv4 100: Neighbor 192.168.1.2 (T unnel0) is down: interface down
*Dec 11 10:39:15.531: %DUAL-5-NBRCHANGE: EIGRP-IPv4 100: Neighbor 192.168.1.1 (T unnel0) is down: interface down
*Dec 11 10:39:15.567: %DUAL-5-NBRCHANGE: EIGRP-IPv4 100: Neighbor 192.168.1.3 (T unnel0) is down: interface down
PHX1(config-if)#ip add 172.16.123.1 255.255.255.0
%FastEthernet0/0 is linked to a VRF. Enable IPv4 on that VRF first.
I wanted to leave that tail end error in their to remind, define the address-family of ipv4 in the VRF instance defined, no further VRF config is necessary but it must be defined!
Once turning on Address-Family IPv4 in VRF Instance FrontDoor:
PHX1(config-if)#ip add 172.16.123.1 255.255.255.0
PHX1(config-if)#
*Dec 11 10:40:54.407: %LINEPROTO-5-UPDOWN: Line protocol on Interface Tunnel0, changed state to up
*Dec 11 10:40:54.611: %CRYPTO-6-ISAKMP_ON_OFF: ISAKMP is OFF
*Dec 11 10:40:54.647: %CRYPTO-6-ISAKMP_ON_OFF: ISAKMP is ON
*Dec 11 10:40:54.691: %CRYPTO-6-ISAKMP_ON_OFF: ISAKMP is OFF
*Dec 11 10:40:54.711: %CRYPTO-6-ISAKMP_ON_OFF: ISAKMP is ON
PHX1(config-if)#
Beautiful! However we are far from done!
Step 3 – Turning OSPF into a OSPF VRF Instance
PHX1(config)#no router ospf 1
PHX1(config)#router ospf 1 ?
vrf VPN Routing/Forwarding Instance
<cr>
PHX1(config)#router ospf 1 vrf FrontDoor ?
<cr>
PHX1(config)#router ospf 1 vrf FrontDoor
PHX1(config-router)#network 172.12.123.0 0.0.0.255 area 0
Step 4 – Setting the Tunnel Options on the Tunnel Interface
This would include “ip ospf pri 0” and “ip ospf network broadcast” normally, but because this is not my first rodeo those were already configured, so I just configured tunnel vrf :
PHX1(config)#int tu0
PHX1(config-if)#tunnel vrf FrontDoor
PHX1(config-if)#
*Dec 11 11:02:08.499: %CRYPTO-6-ISAKMP_ON_OFF: ISAKMP is OFF
*Dec 11 11:02:08.519: %CRYPTO-6-ISAKMP_ON_OFF: ISAKMP is ON
PHX1(config-if)#
Now this has been configured around on all Hub and Spoke routers exactly the same, though somehow I am getting the same DMVPN state on the routers:
MIL1(config-if)#do sh dmvpn
Legend: Attrb –> S – Static, D – Dynamic, I – Incomplete
N – NATed, L – Local, X – No Socket
# Ent –> Number of NHRP entries with same NBMA peer
NHS Status: E –> Expecting Replies, R –> Responding, W –> Waiting
UpDn Time –> Up or Down Time for a Tunnel
==========================================================================
Interface: Tunnel0, IPv4 NHRP Details
Type:Spoke, NHRP Peers:1,
# Ent Peer NBMA Addr Peer Tunnel Add State UpDn Tm Attrb
—– ————— ————— —– ——– —–
1 172.16.123.1 192.168.1.10 IKE 00:00:44 S
MIL1(config-if)#
I am honestly way too tired to begin troubleshooting this late, and would probably do more harm than good, so I will book mark it here to resume and finish tomorrow.
The issue I found was a very slight oversight of assigning the outside interfaces a mask of /24 rather than /30 to OSPF and the IP Address, and correct as such:
ATL1(config)#no router ospf 1
ATL1(config)#router ospf 1 vrf FrontDoor
ATL1(config-router)#network 172.16.123.8 0.0.0.3 area 0
ATL1(config-router)#int fa0/0
ATL1(config-if)#ip add 172.16.123.9 255.255.255.252
ATL1(config-if)#
*Dec 11 11:36:24.319: %CRYPTO-6-ISAKMP_ON_OFF: ISAKMP is OFF
*Dec 11 11:36:24.335: %CRYPTO-6-ISAKMP_ON_OFF: ISAKMP is ON
ATL1(config-if)#
*Dec 11 11:36:30.147: %OSPF-5-ADJCHG: Process 1, Nbr 172.16.123.14 on FastEthernet0/0 from LOADING to FULL, Loading Done
ATL1(config-if)#
Once I did this to all DMVPN routers, we’ll take a look at MIL1 “sh dmvpn” now:
MIL1(config)#do sh dmvpn
Legend: Attrb –> S – Static, D – Dynamic, I – Incomplete
N – NATed, L – Local, X – No Socket
# Ent –> Number of NHRP entries with same NBMA peer
NHS Status: E –> Expecting Replies, R –> Responding, W –> Waiting
UpDn Time –> Up or Down Time for a Tunnel
==========================================================================
Interface: Tunnel0, IPv4 NHRP Details
Type:Spoke, NHRP Peers:1,
# Ent Peer NBMA Addr Peer Tunnel Add State UpDn Tm Attrb
—– ————— ————— —– ——– —–
1 172.16.123.1 192.168.1.10 IKE 00:59:26 S
MIL1(config)#
That is really a bummer to see, I had thought I nailed that one on the head, well crap…
There is one more solution that might work to get this rolling, as I’m not trying to get too far into Cisco CLI Crypto settings, however it is an “ISAKMP Profile” to insert into the IPSec Profile applied to our DMVPN Tunnel Interface:
Step 1 – Define Crypto Keyring
PHX1(config)#cry keyring SOMETHING vrf FrontDoor
PHX1(conf-keyring)#pre-shared address 0.0.0.0 key PassWord!!
PHX1(conf-keyring)#exit
Step 2 – Define Crypto ISAKMP Profile
PHX1(config)#cry isa prof SecureVRF
% A profile is deemed incomplete until it has match identity statements
PHX1(conf-isa-prof)#keyring SOMETHING
PHX1(conf-isa-prof)#match identity address 0.0.0.0
PHX1(conf-isa-prof)#exit
Step 3 – Add ISAKMP Profile to IPSec Profile
PHX1(config)#cryp ipsec profile ProtectTheTunnel
PHX1(ipsec-profile)#set isakmp-profile SecureVRF
PHX1(ipsec-profile)#
PHX1(ipsec-profile)#exit
PHX1(config)#
Now to check all routers “sh dmvpn” output to see what ‘State’ its in
I will remove the table codes from the output to keep the output concise.
PHX1
PHX1(config)#do sh dmvpn
Interface: Tunnel0, IPv4 NHRP Details
Type:Hub, NHRP Peers:1,
# Ent Peer NBMA Addr Peer Tunnel Add State UpDn Tm Attrb
—– ————— ————— —– ——– —–
1 172.16.123.9 192.168.1.2 UP 01:11:49 D
MPLS1
MPLS1(config-if)#do sh dmvpn
MPLS1(config-if)#
ATL1
ATL1(config-if)#do sh dmvpn
Interface: Tunnel0, IPv4 NHRP Details
Type:Spoke, NHRP Peers:1,
# Ent Peer NBMA Addr Peer Tunnel Add State UpDn Tm Attrb
—– ————— ————— —– ——– —–
1 172.16.123.1 192.168.1.10 UP 01:10:27 S
MIL1
MIL1#sh dmvpn
Interface: Tunnel0, IPv4 NHRP Details
Type:Spoke, NHRP Peers:1,
# Ent Peer NBMA Addr Peer Tunnel Add State UpDn Tm Attrb
—– ————— ————— —– ——– —–
1 172.16.123.1 192.168.1.10 IKE 02:36:30 S
MIL1#
So right now the only Tunnels showing “Up” on the DMVPN is between the Hub PHX1 and Spoke ATL1, however a real oddity of even this small victory shows here:
ATL2#ping 10.10.10.254
Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 10.10.10.254, timeout is 2 seconds:
…..
Success rate is 0 percent (0/5)
ATL2#
Though the tunnel is “Up” it seems, it is definitely not passing traffic or propagating route to PHX1 Hub in the Global or VRF IP Route Tables:
PHX1(config)#do sh ip route
(Codes redacted)
Gateway of last resort is not set
10.0.0.0/8 is variably subnetted, 3 subnets, 2 masks
D 10.0.10.0/24 [90/30720] via 10.10.10.253, 02:11:18, FastEthernet1/0
C 10.10.10.0/24 is directly connected, FastEthernet1/0
L 10.10.10.254/32 is directly connected, FastEthernet1/0
The only Dynamic route worth noting is the LAN segment behind PHX1 learned from PHX2 via EIGRP, lets look at the VRF Route Table:
PHX1(config)#do sh ip route vrf FrontDoor
Routing Table: FrontDoor
(Codes redacted)
Gateway of last resort is not set
172.16.0.0/16 is variably subnetted, 5 subnets, 2 masks
C 172.16.123.0/30 is directly connected, FastEthernet0/0
L 172.16.123.1/32 is directly connected, FastEthernet0/0
O 172.16.123.4/30 [110/2] via 172.16.123.2, 02:10:12, FastEthernet0/0
O 172.16.123.8/30 [110/2] via 172.16.123.2, 02:10:12, FastEthernet0/0
O 172.16.123.12/30 [110/2] via 172.16.123.2, 02:10:12, FastEthernet0/0
192.168.1.0/24 is variably subnetted, 2 subnets, 2 masks
C 192.168.1.0/24 is directly connected, Tunnel0
L 192.168.1.10/32 is directly connected, Tunnel0
PHX1(config)#
I assume that is exactly what we want, the “Transport” information is tucked away in its own VRF IP Table as the Global IP Route table only needs to know about non-Transport routes to begin with!
This is kind of baffling, so I am going to remove VRF from the Physical and Tunnel Interface (I don’t believe it should matter leaving it on OSPF), and see if we can restore some sanity to this Topology.
Behind the scenes, this is what I am doing (or undoing) on every router, I am nuking everything VRF related to get this DMVPN up and running again at all:
MPLS1(config-if)#router ospf 1 vrf FrontDoor
MPLS1(config-router)#no network 172.16.123.0 0.0.0.3 area 0
MPLS1(config-router)#int tu0
MPLS1(config-if)#no tunnel vrf FrontDoor
MPLS1(config-if)#int fa0/0
MPLS1(config-if)#no vrf forwarding FrontDoor
% Interface FastEthernet0/0 IPv4 disabled and address(es) removed due to enabling VRF
MPLS1(config)#no router ospf 1 vrf FrontDoor <—- Have to completely nuke it
MPLS1(config-if)#
*Dec 11 13:52:37.943: %OSPF-5-ADJCHG: Process 1, Nbr 172.16.123.14 on FastEthernet0/0 from FULL to DOWN, Neighbor Down: Interface down or detached
*Dec 11 13:52:38.107: %DEC21140-5-REMOVE_HWADDR_FAIL: Interface FastEthernet0/0 failed to remove Addr:=0100.5e00.0005 from HWAF
MPLS1(config-if)#ip add 172.16.123.5 255.255.255.252
MPLS1(config-if)#exit
MPLS1(config)#int fa0/0
MPLS1(config-if)#ip ospf 1 area 0
MPLS1(config-if)#
*Dec 11 13:56:49.195: %OSPF-5-ADJCHG: Process 1, Nbr 172.16.123.14 on FastEthernet0/0 from LOADING to FULL, Loading Done
MPLS1(config-if)#
- Remove the “Underlay” from the OSPF VRF Instance
- “no tunnel (VRF)” from Tunnel Interface
- “no forwarding (VRF)” from Physial Interface – Re-added IP (of course)
- Tried to add “ip ospf 1 area 0” on the physical interface, some VRF message saying nothing happening, removed “ospf 1 vrf FrontDoor” entirely, re-added “ip ospf 1 area 0” command to WAN facing Interface Fa0/0
This has been done to ALL Branches WAN routers, with only “ip ospf 1 area 0” being applied to the physical Fa0/0 interface, HOWEVER I HAVE FORGOTTEN 2 RULES OF PLAYING WITH EIGRP VIA DMVPN I AM KICKING MYSELF FOR!!!!
- “no ip split horizon eigrp 100”
- “no ip next-hop-self eigrp 100”
Ooooh I bet I wasted so much time yanking all the VRF configs off these routers for this to work due to those two commands missing from this fresh lab where those configs were NOT saved from the previous DMVPN lab!
PHX1(config-if)#
PHX1(config-if)#int tu0
PHX1(config-if)#no ip split-horizon eigrp 100
PHX1(config-if)#no ip next-hop-self eigrp 100
PHX1(config-if)#
No immediate mind-blowing impact on the Topology, lets check out if anything is more pingable or whats going on at this point:
– I tried to ping and got …’s, and upon looking at the Tunnel Interface I must have REALLY jammed it up to produce this status:
MPLS1#sh int tu0
Tunnel0 is reset, line protocol is down
Yeeeesh! I am digging myself into a hole here quick!
So I go around to all router and do a “shut” then “no shut” as this is how you bring a Tunnel interface out of a reset/down state, you cannot just “no shut” from this:
MPLS1(config-if)#shut
MPLS1(config-if)#
*Dec 11 14:21:57.991: %LINK-5-CHANGED: Interface Tunnel0, changed state to administratively down
MPLS1(config-if)#no shut
MPLS1(config-if)#
*Dec 11 14:22:01.575: %CRYPTO-6-ISAKMP_ON_OFF: ISAKMP is ON
MPLS1(config-if)#
*Dec 11 14:22:03.499: %LINEPROTO-5-UPDOWN: Line protocol on Interface Tunnel0, changed state to up
MPLS1(config-if)#
*Dec 11 14:22:03.527: %LINK-3-UPDOWN: Interface Tunnel0, changed state to up
MPLS1(config-if)#
So this will be done to all 3 other site locations, then we’ll see if we fiiinally have DMVPN!
I think I’ve caught the sneaky culprit, hiding above the Tunnel0 IP Address, barely in view for me to remove from the configuration (this is before I knew the “forwarding” should be configured on the physical interface):
!
interface Tunnel0
vrf forwarding FrontDoor <——————— GOTCHA!!!!
ip address 192.168.1.10 255.255.255.0
So this was removed, however I want to see if we have basic “Underlay” connectivity, cause after bouncing OSPF if we don’t have that we don’t have DMVPN:
PHX1(config-if)#do ping 172.16.123.5
Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 172.16.123.5, timeout is 2 seconds:
!!!!!
Success rate is 100 percent (5/5), round-trip min/avg/max = 68/72/80 ms
PHX1(config-if)#do ping 172.16.123.9
Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 172.16.123.9, timeout is 2 seconds:
.!!!!
Success rate is 80 percent (4/5), round-trip min/avg/max = 48/49/52 ms
PHX1(config-if)#do ping 172.16.123.13
Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 172.16.123.13, timeout is 2 seconds:
.!!!!
Success rate is 80 percent (4/5), round-trip min/avg/max = 44/51/60 ms
PHX1(config-if)#
Understandably after all that “Malarkey” as some may call it, we now have Underlay connectivity, but what about “Overlay” connectivity:
PHX1(config-if)#do ping 192.168.1.1
Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 192.168.1.1, timeout is 2 seconds:
…..
Success rate is 0 percent (0/5)
PHX1(config-if)#do ping 192.168.1.2
Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 192.168.1.2, timeout is 2 seconds:
…..
Success rate is 0 percent (0/5)
PHX1(config-if)#do ping 192.168.1.3
Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 192.168.1.3, timeout is 2 seconds:
…..
Success rate is 0 percent (0/5)
PHX1(config-if)#
What is missing here is an IGP to tell all these different routers about their Overlay IP Addressing, so I am going to try to sneak a VRF OSPF instance in to do this, as I highly suspect that “vrf forwarding” statement on the Phoenix tunnel caused all kinds of issues:
- Remove “ip ospf 1 area 0” from Fa0/0 to allow VRF OSPF Instance
- Add network “172.16.123.X 0.0.0.3 area 0” to all VRF OSPF Instances
- Cross fingers
So this is the last router:
MIL1(config-if)#int fa0/0
MIL1(config-if)#no ip ospf 1 area 0
MIL1(config-if)#
*Dec 11 14:40:36.834: %OSPF-5-ADJCHG: Process 1, Nbr 172.16.123.14 on FastEthernet0/0 from FULL to DOWN, Neighbor Down: Interface down or detached
MIL1(config-if)#no router ospf 1
MIL1(config)#router ospf 1 vrf FrontDoor
MIL1(config-router)#
*Dec 11 14:45:19.046: %OSPF-4-NORTRID: OSPF process 1 failed to allocate unique router-id and cannot start
MIL1(config-router)#router-id 3.3.3.3
MIL1(config-router)#network 172.16.123.12 0.0.0.3 area 0
MIL1(config-router)#
And even with this, PHX1 is showing ZERO(!!!) OSPF Neighbors!
I am going to take a chance, and enable “vrf forwarding FrontDoor” again on all physical interfaces to begin with, to see if that starts routing for that VRF OSPF process:
MIL1(config-router)#int fa0/0
MIL1(config-if)#vrf forwarding FrontDoor
% Interface FastEthernet0/0 IPv4 disabled and address(es) removed due to enabling VRF FrontDoor
MIL1(config-if)#
*Dec 11 14:48:54.626: %LINEPROTO-5-UPDOWN: Line protocol on Interface Tunnel0, changed state to down
*Dec 11 14:48:54.890: %CRYPTO-6-ISAKMP_ON_OFF: ISAKMP is OFF
MIL1(config-if)#ip add 172.16.123.13 255.255.255.0
MIL1(config-if)#
*Dec 11 14:49:00.798: %LINEPROTO-5-UPDOWN: Line protocol on Interface Tunnel0, changed state to up
MIL1(config-if)#
*Dec 11 14:49:01.030: %CRYPTO-6-ISAKMP_ON_OFF: ISAKMP is ON
*Dec 11 14:49:01.134: %CRYPTO-6-ISAKMP_ON_OFF: ISAKMP is OFF
*Dec 11 14:49:01.158: %CRYPTO-6-ISAKMP_ON_OFF: ISAKMP is ON
MIL1(config-if)#
Nothing is happening on PHX1 as far as neighbor relationships, I think this about wraps this up for now unfortunately, as a last ditch effort I even tried tweaking the ISP:
ISP(config)#no router ospf 1
ISP(config)#router ospf 1 vrf FrontDoor
Unknown VRF specified
ISP(config)#vrf def FrontDoor
ISP(config-vrf)#address-family ipv4
ISP(config-vrf-af)#exit
ISP(config-vrf)#exit
ISP(config)#router ospf 1 vrf FrontDoor
*Dec 11 14:44:49.118: %OSPF-4-NORTRID: OSPF process 1 failed to allocate unique router-id and cannot start
ISP(config-router)#router-id 100.100.100.100
ISP(config-router)#network 172.16.123.0 0.0.0.3 area 0
ISP(config-router)#network 172.16.123.4 0.0.0.3 area 0
ISP(config-router)#network 172.16.123.8 0.0.0.3 area 0
ISP(config-router)#network 172.16.123.12 0.0.0.3 area 0
ISP(config-router)#
Not even adding the ISP Router to the same VRF as the
So I am going to end these studies here for now until I better undstand them
I’m going to keep this Topology and circle back around to this, but I feel I’ve been spinning my wheels for almost a week now, and I need to make some forward progress with finishing MPLS studies and something that offers rock solid study material 🙂
The DEVNET GRIND! 