A very frustrating end to my labbing last night ended with myself not being able to ping from spoke R2 to R3 and likewise, and I called it for the night earlier on, but went back to it as I wanted to see what broke it.
So first I turned off the interface running OSPF, then “no router ospf 1” on R3, then ever removed RIP from all 3 Routers (and took EIGRP off R1) and IT STILL DOES NOT PING. This is basically now just the Frame-Relay connection that will ping without protocols or route statements, because the NBMA switch is programmed with DLCI’s to allow communication.
So I did a ‘write erase’ on R1, R2, and R3, and place my usual Frame-Relay and IP address only base configs I’ve used several dozen times and still the same problem. I always verify after pasting the base configs out of text files (to make sure they are always correct), I would ping each NBMA remote IP before beginning labbing. This time I applied those same configs, and I can only ping Hub to Spoke, Spoke to Hub, but not Spoke to Spoke.
Starting at R2 with 10,000 pings aimed at R3, I begin a 10,000 ping sequence to check my PVC’s to see if they are on issue:
R2#ping
Protocol [ip]:
Target IP address: 172.12.123.3
Repeat count [5]: 10000
Datagram size [100]:
Timeout in seconds [2]:
Extended commands [n]:
Sweep range of sizes [n]:
Type escape sequence to abort.
Sending 10000, 100-byte ICMP Echos to 172.12.123.3, timeout is 2 seconds:
…
ASR#1
[Resuming connection 1 to r1 … ]
R1#
R1#show frame pvc
PVC Statistics for interface Serial0/0 (Frame Relay DTE)
Active Inactive Deleted Static
Local 2 0 0 0
Switched 0 0 0 0
Unused 0 0 0 0
DLCI = 122, DLCI USAGE = LOCAL, PVC STATUS = ACTIVE, INTERFACE = Serial0/0
input pkts 0 output pkts 0 in bytes 0
out bytes 0 dropped pkts 0 in pkts dropped 0
out pkts dropped 0 out bytes dropped 0
in FECN pkts 0 in BECN pkts 0 out FECN pkts 0
out BECN pkts 0 in DE pkts 0 out DE pkts 0
out bcast pkts 0 out bcast bytes 0
5 minute input rate 0 bits/sec, 0 packets/sec
5 minute output rate 0 bits/sec, 0 packets/sec
pvc create time 00:05:57, last time pvc status changed 00:05:27
DLCI = 123, DLCI USAGE = LOCAL, PVC STATUS = ACTIVE, INTERFACE = Serial0/0
input pkts 0 output pkts 0 in bytes 0
out bytes 0 dropped pkts 0 in pkts dropped 0
out pkts dropped 0 out bytes dropped 0
in FECN pkts 0 in BECN pkts 0 out FECN pkts 0
out BECN pkts 0 in DE pkts 0 out DE pkts 0
out bcast pkts 0 out bcast bytes 0
5 minute input rate 0 bits/sec, 0 packets/sec
5 minute output rate 0 bits/sec, 0 packets/sec
pvc create time 00:06:02, last time pvc status changed 00:06:02
R1#
My DLCI 122 shows no packets in, out, or dropped while it still has about 9,946 more pings trying to cross it.
Before moving onto what “debug ip packet” reveals about the spokes, here is the proof in the pudding the configuration is correct on R1:
R1#show frame map
Serial0/0 (up): ip 172.12.123.2 dlci 122(0x7A,0x1CA0), static,
broadcast,
CISCO, status defined, active
Serial0/0 (up): ip 172.12.123.3 dlci 123(0x7B,0x1CB0), static,
broadcast,
CISCO, status defined, active
So this is what it looks like from the R2, I will first post the output from “debug ip pack” for a failed ping, then the same for a ping to the Hub that get responses:
(Hit ctrl +shft +6 twice to break the 10,000 ping sequence slowly dotting by)
R2#debug ip pack
IP packet debugging is on
R2#ping 172.12.123.3
Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 172.12.123.3, timeout is 2 seconds:
*Mar 1 12:18:40.171: IP: tableid=0, s=172.12.123.2 (local), d=172.12.123.3 (Serial0/0), routed via RIB
*Mar 1 12:18:40.171: IP: s=172.12.123.2 (local), d=172.12.123.3 (Serial0/0), len 100, sending
*Mar 1 12:18:40.175: IP: s=172.12.123.2 (local), d=172.12.123.3 (Serial0/0), len 100, encapsulation failed.
*Mar 1 12:18:42.170: IP: tableid=0, s=172.12.123.2 (local), d=172.12.123.3 (Serial0/0), routed via RIB
*Mar 1 12:18:42.170: IP: s=172.12.123.2 (local), d=172.12.123.3 (Serial0/0), len 100, sending
*Mar 1 12:18:42.170: IP: s=172.12.123.2 (local), d=172.12.123.3 (Serial0/0), len 100, encapsulation failed.
*Mar 1 12:18:44.174: IP: tableid=0, s=172.12.123.2 (local), d=172.12.123.3 (Serial0/0), routed via RIB
*Mar 1 12:18:44.174: IP: s=172.12.123.2 (local), d=172.12.123.3 (Serial0/0), len 100, sending
*Mar 1 12:18:44.174: IP: s=172.12.123.2 (local), d=172.12.123.3 (Serial0/0), len 100, encapsulation failed.
*Mar 1 12:18:46.177: IP: tableid=0, s=172.12.123.2 (local), d=172.12.123.3 (Serial0/0), routed via RIB
*Mar 1 12:18:46.177: IP: s=172.12.123.2 (local), d=172.12.123.3 (Serial0/0), len 100, sending
*Mar 1 12:18:46.177: IP: s=172.12.123.2 (local), d=172.12.123.3 (Serial0/0), len 100, encapsulation failed.
*Mar 1 12:18:48.180: IP: tableid=0, s=172.12.123.2 (local), d=172.12.123.3 (Serial0/0), routed via RIB
*Mar 1 12:18:48.180: IP: s=172.12.123.2 (local), d=172.12.123.3 (Serial0/0), len 100, sending
*Mar 1 12:18:48.180: IP: s=172.12.123.2 (local), d=172.12.123.3 (Serial0/0), len 100, encapsulation failed.
Success rate is 0 percent (0/5)
R2#
It shows that it is routed to the RIB which I have not learned specifically but I believe it’s the ‘Routing Information Base’ without looking at google, it begins sending, and encapsulation fails? Why? What? So this is what it SHOULD look like:
Sending 5, 100-byte ICMP Echos to 172.12.123.1, timeout is 2 seconds:
!!!!!
Success rate is 100 percent (5/5), round-trip min/avg/max = 68/69/72 ms
R2#
*Mar 1 12:21:57.090: IP: tableid=0, s=172.12.123.2 (local), d=172.12.123.1 (Serial0/0), routed via FIB
*Mar 1 12:21:57.090: IP: s=172.12.123.2 (local), d=172.12.123.1 (Serial0/0), len 100, sending
*Mar 1 12:21:57.158: IP: tableid=0, s=172.12.123.1 (Serial0/0), d=172.12.123.2 (Serial0/0), routed via RIB
*Mar 1 12:21:57.158: IP: s=172.12.123.1 (Serial0/0), d=172.12.123.2 (Serial0/0), len 100, rcvd 3
*Mar 1 12:21:57.158: IP: tableid=0, s=172.12.123.2 (local), d=172.12.123.1 (Serial0/0), routed via FIB
*Mar 1 12:21:57.162: IP: s=172.12.123.2 (local), d=172.12.123.1 (Serial0/0), len 100, sending
*Mar 1 12:21:57.226: IP: tableid=0, s=172.12.123.1 (Serial0/0), d=172.12.123.2 (Serial0/0), routed via RIB
*Mar 1 12:21:57.230: IP: s=172.12.123.1 (Serial0/0), d=172.12.123.2 (Serial0/0), len 100, rcvd 3
*Mar 1 12:21:57.230: IP: tableid=0, s=172.12.123.2 (local), d=172.12.123.1 (Serial0/0), routed via FIB
*Mar 1 12:21:57.230: IP: s=172.12.123.2 (local), d=172.12.123.1 (Serial0/
R2#0), len 100, sending
*Mar 1 12:21:57.298: IP: tableid=0, s=172.12.123.1 (Serial0/0), d=172.12.123.2 (Serial0/0), routed via RIB
*Mar 1 12:21:57.298: IP: s=172.12.123.1 (Serial0/0), d=172.12.123.2 (Serial0/0), len 100, rcvd 3
*Mar 1 12:21:57.302: IP: tableid=0, s=172.12.123.2 (local), d=172.12.123.1 (Serial0/0), routed via FIB
*Mar 1 12:21:57.302: IP: s=172.12.123.2 (local), d=172.12.123.1 (Serial0/0), len 100, sending
*Mar 1 12:21:57.370: IP: tableid=0, s=172.12.123.1 (Serial0/0), d=172.12.123.2 (Serial0/0), routed via RIB
*Mar 1 12:21:57.370: IP: s=172.12.123.1 (Serial0/0), d=172.12.123.2 (Serial0/0), len 100, rcvd 3
*Mar 1 12:21:57.370: IP: tableid=0, s=172.12.123.2 (local), d=172.12.123.1 (Serial0/0), routed via FIB
*Mar 1 12:21:57.370: IP: s=172.12.123.2 (local), d=172.12.123.1 (Serial0/0), len 100, sending
*Mar 1 12:21:57.439: IP: tableid=0, s=172.12.123.1 (Serial0/0), d=172.12.123.2 (Serial0/0), routed via RIB
*Mar 1 12:21:57.439: IP: s=172.12.123.1 (Serial0/0), d=172.12.123.2 (Serial0/0), len 100, rcvd 3
R2#
Now this acronym I did google last night while I was middle of the night troubleshooting, it is the ‘Forwarding Information Base’ that is replying to the pings sent out via the RIB.
*** Worthy to note RIB is what the local router uses to send, FIB is what remote routers use to send traffic back to your local router ***
The only answers I have found so far is misconfigs of frame map statements like getting DLCI’s or remote IP’s mixed up, my output from the interfaces of all routers are as shown below:
R1:
interface Serial0/0
ip address 172.12.123.1 255.255.255.0
encapsulation frame-relay
frame-relay map ip 172.12.123.2 122 broadcast
frame-relay map ip 172.12.123.3 123 broadcast
no frame-relay inverse-arp
R2:
interface Serial0/0
ip address 172.12.123.2 255.255.255.0
encapsulation frame-relay
frame-relay map ip 172.12.123.1 221 broadcast
frame-relay map ip 172.12.123.3 321 <- OMG I CANNOT BELIEVE I MISSED THAT!
no frame-relay inverse-arp
R3:
interface Serial0/2
ip address 172.12.123.3 255.255.255.0
encapsulation frame-relay
frame-relay map ip 172.12.123.1 321 broadcast
frame-relay map ip 172.12.123.2 321
no frame-relay inverse-arp
Oh man…. I got so caught up with constantly making R4 and R5 weird Areas that I would ping from THOSE to a spoke router, but never pinged between the spokes because I just assumed connectivity because routes were propagating.
So of course once I changed that route map on R2 to point traffic destined for R4 towards the Hub, being that it’s a Hub and Spoke network, of course pings were successful. The most valuable learning lesson here, is don’t overlook the painfully easy mistakes, or you can spend hours looking at the complex ones and possibly breaking more along the way!
The DEVNET GRIND! 