OSPF_NBMA_15

Back with the Topology, because we are labbing this time! I have three more videos I want to compress into this one lesson as they are all pretty short and to the point, and then it is time to start the extremely large BGP section which I am really looking forward to labbing (absolutely new territory).

Anyways, I’ve setup OSPF for demonstration purposes, because of its occasional Hello’s multicast to 224.0.0.5 to examplify slimming down the output of “debug ip packet” though I would strongly recommend not doing this in a production network!

 

Part 1 – Filtering “debug ip packet” to see only the output you need

 

So just to verify we are all neighbors, lets take a look at R1:

R1#sh ip ospf nei

Neighbor ID     Pri   State           Dead Time   Address         Interface
2.2.2.2           0   FULL/DROTHER    00:01:48    172.12.123.2    Serial0/0
3.3.3.3           0   FULL/DROTHER    00:01:59    172.12.123.3    Serial0/0
5.5.5.5           1   FULL/DR         00:00:35    172.12.15.5     FastEthernet0/1
R1#

All neighborinos, good deal, now I’ll run an initial “debug ip pack” to see how much data we get spilling onto the CLI:

R1#debug ip pack
IP packet debugging is on
R1#
*Mar 30 22:36:42.978: IP: s=172.12.15.5 (FastEthernet0/1), d=224.0.0.5, len 80, rcvd 0
R1#
*Mar 30 22:36:46.636: IP: s=172.12.15.1 (local), d=224.0.0.5 (FastEthernet0/1), len 80, sending broad/multicast
R1#
*Mar 30 22:36:49.893: IP: s=172.12.123.1 (local), d=172.12.123.2 (Serial0/0), len 84, sending
*Mar 30 22:36:49.893: IP: s=172.12.123.1 (local), d=172.12.123.3 (Serial0/0), len 84, sending
R1#
*Mar 30 22:36:52.578: IP: s=172.12.15.5 (FastEthernet0/1), d=224.0.0.5, len 80, rcvd 0
R1#
*Mar 30 22:36:56.636: IP: s=172.12.15.1 (local), d=224.0.0.5 (FastEthernet0/1), len 80, sending broad/multicast
R1#
*Mar 30 22:36:58.972: IP: s=172.12.123.2 (Serial0/0), d=172.12.123.1, len 80, rcvd 0
R1#u a
*Mar 30 22:37:02.514: IP: s=172.12.15.5 (FastEthernet0/1), d=224.0.0.5, len 80, rcvd 0
R1#u all
All possible debugging has been turned off
R1#

So relatively nothing, just your standard Hello’s, so lets add some traffic to see a REAL debug ip pack output with even just ping traffic constantly hitting R1:

R2#ping 172.12.123.1 repeat 10000

Type escape sequence to abort.
Sending 10000, 100-byte ICMP Echos to 172.12.123.1, timeout is 2 seconds:
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
ASR#1
[Resuming connection 1 to r1 … ]

R1#debug ip pack
IP packet debugging is on
R1#
*Mar 30 22:38:39.805: IP: tableid=0, s=172.12.123.2 (Serial0/0), d=172.12.123.1 (Serial0/0), routed via RIB
*Mar 30 22:38:39.809: IP: s=172.12.123.2 (Serial0/0), d=172.12.123.1 (Serial0/0), len 100, rcvd 3
*Mar 30 22:38:39.809: IP: tableid=0, s=172.12.123.1 (local), d=172.12.123.2 (Serial0/0), routed via FIB
*Mar 30 22:38:39.809: IP: s=172.12.123.1 (local), d=172.12.123.2 (Serial0/0), len 100, sending
*Mar 30 22:38:39.877: IP: tableid=0, s=172.12.123.2 (Serial0/0), d=172.12.123.1 (Serial0/0), routed via RIB
*Mar 30 22:38:39.877: IP: s=172.12.123.2 (Serial0/0), d=172.12.123.1 (Serial0/0), len 100, rcvd 3
*Mar 30 22:38:39.877: IP: tableid=0, s=172.12.123.1 (local), d=172.12.123.2 (Serial0/0), routed via FIB
*Mar 30 22:38:39.881: IP: s=172.12.123.1 (local), d=172.12.123.2 (Serial0/0), len 100, sending
*Mar 30 22:38:39.946: IP: tableid=0, s=172.12.123.2 (Serial0/0), d=172.12.123.1 (Serial0/0), routed via RIB
*Mar 30 22:38:39.950: IP: s=172.12.123.2 (Serial0/0), d=172.12.123.1 (
R1#Serial0/0), len 100, rcvd 3
*Mar 30 22:38:39.950: IP: tableid=0, s=172.12.123.1 (local), d=172.12.123.2 (Serial0/0), routed via FIB
*Mar 30 22:38:39.950: IP: s=172.12.123.1 (local), d=172.12.123.2 (Serial0/0), len 100, sending
*Mar 30 22:38:39.958: IP: s=172.12.123.3 (Serial0/0), d=172.12.123.1, len 80, rcvd 0
*Mar 30 22:38:40.018: IP: tableid=0, s=172.12.123.2 (Serial0/0), d=172.12.123.1 (Serial0/0), routed via RIB
*Mar 30 22:38:40.018: IP: s=172.12.123.2 (Serial0/0), d=172.12.123.1 (Serial0/0), len 100, rcvd 3
*Mar 30 22:38:40.022: IP: tableid=0, s=172.12.123.1 (local), d=172.12.123.2 (Serial0/0), routed via FIB
*Mar 30 22:38:40.022: IP: s=172.12.123.1 (local), d=172.12.123.2 (Serial0/0), len 100, sending

Now that is just one ping, so full on network traffic can take down a router within seconds of issuing the command if not in microseconds. However, if we need to hone in on one host to see the traffic coming from it and going to it, there is a way to get all other traffic out of the debug.

This is done with our all time hero, Access Control Lists! For this example I want to focus on traffic only coming from, or going to R5’s 172.12.15.5 IP address, so we will need two ACL’s to accomplish this as you cannot set the source and destination to 172.12.15.5 to accomplish this task (as much as we’d like it to be as easy):

R1(config)#access-list 123 permit ip host 172.12.15.5 any
R1(config)#access-list 123 permit ip any host 172.12.15.5
R1(config)#

Tada! Now any traffic with a source or destination of any with a corresponding source or destination of 172.12.15.5 will be seen, and here is how to configure the debug:

R1#debug ip pack ?
  <1-199>      Access list
  <1300-2699>  Access list (expanded range)
  detail       Print more debugging detail
  <cr>

R1#debug ip pack 123 ?
  detail  Print more debugging detail
  <cr>

R1#debug ip pack 123
IP packet debugging is on for access list 123

We are now debugging with the parameters of access-list 123, and I’ve even highlighted in red the output from the console will tell you that you’re debugging using the ACL.

So let’s check out the output from this:
R1#
*Mar 30 22:47:54.242: IP: s=172.12.15.5 (FastEthernet0/1), d=224.0.0.5, len 80, rcvd 0
R1#
ASR#2
[Resuming connection 2 to r2 … ]

R2#ping 172.12.123.1 repeat 1000

Type escape sequence to abort.
Sending 1000, 100-byte ICMP Echos to 172.12.123.1, timeout is 2 seconds:
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
ASR#1
[Resuming connection 1 to r1 … ]

*Mar 30 22:48:03.638: IP: s=172.12.15.5 (FastEthernet0/1), d=224.0.0.5, len 80, rcvd 0
R1#
*Mar 30 22:48:13.073: IP: s=172.12.15.5 (FastEthernet0/1), d=224.0.0.5, len 80, rcvd 0
R1#
*Mar 30 22:48:22.537: IP: s=172.12.15.5 (FastEthernet0/1), d=224.0.0.5, len 80, rcvd 0
R1#
*Mar 30 22:48:31.860: IP: s=172.12.15.5 (FastEthernet0/1), d=224.0.0.5, len 80, rcvd 0

So it is working because of that implicit deny on the end of the ACL, we aren’t seeing those pings, but another interesting thing that I thought was great to point out is that you see Hello’s sourced from R5 but none from R1 with a destination of R5’s 172.12.15.5 interface.

This is because the source would be 172.12.15.1 and the destination would be 224.0.0.5, so it also does not match the ACL criteria, so it is working perfectly.

Now one interesting behavior to note, is when you remove the ACL while the debug is running, lets check it out:

R1#debug ip pack 123
IP packet debugging is on for access list 123
R1#
*Mar 30 22:53:35.021: IP: s=172.12.15.5 (FastEthernet0/1), d=224.0.0.5, len 80, rcvd 0
R1#conf t
Enter configuration commands, one per line.  End with CNTL/Z.
R1(config)#no
*Mar 30 22:53:44.444: IP: s=172.12.15.5 (FastEthernet0/1), d=224.0.0.5, len 80, rcvd 0
R1(config)#no access-list 123
IP packet debugging is off

Turning off all possible debugging on ACL 123

R1(config)#do sh access-list

R1(config)#do sh debug

R1(config)#

It immediately stops the debugging of both IP Packet, and all possible debugging for that particular ACL, so at the same time this removes the ACL and stops the debug on the router be sure to verify that both ACL and debug are gone before running for the door.

One more thing you can see on the output for debug ip pack 123 is “detail” as a modifier, which lets see what more possible detail it wants to give us:

R1#debug ip pack 123 detail
IP packet debugging is on (detailed) for access list 123
R1#
*Mar 30 22:57:31.348: IP: s=172.12.15.5 (FastEthernet0/1), d=224.0.0.5, len 80, rcvd 0, proto=89
R1#
*Mar 30 22:57:41.329: IP: s=172.12.15.5 (FastEthernet0/1), d=224.0.0.5, len 80, rcvd 0, proto=89
R1#
*Mar 30 22:57:50.472: IP: s=172.12.15.5 (FastEthernet0/1), d=224.0.0.5, len 80, rcvd 0, proto=89
R1#
*Mar 30 22:58:00.051: IP: s=172.12.15.5 (FastEthernet0/1), d=224.0.0.5, len 80, rcvd 0, proto=89
R1#

Not really much additional detail, except a proto=89 tagged on the end there, so there isn’t a whole lot to that to know at this time in ROUTE studies that it exists and doesn’t give a whole lot of detail contrary to its name!

 

Spotting memory issues in Router Memory

 

A couple of ways to spot memory issues that the router will have in its output or behaviors that will happen if memory is getting low, I will list bullet point style:

  • If the router rejects a telnet session it may be low on memory
  • If the router shows the output “show processor memory” regardless of the command you enter that’s a definite memory issue
  • If your router literally has the output “low on memory” or “Unable to create EXEC – no memory or too many processes” you also definitely have a memory problem
  • Show commands not giving you any output / freezing up / console sessions hanging

Cisco recommends if you have an issue connecting to the console port, to unplug the LAN and WAN cables so it is not processing packets, which should free up some memory to connect. If you still cannot connect, you are up chocolate creek without a popsicle stick!

Cisco also recommends for “show” commands in terms of showing memory issues here, but they are very cumbersome commands (out of scope for the CCNP) but wanted to show them here. The two commands are “show memory allocating-process totals” and “show memory summary” which will give you huge outputs that again are way beyond the scope of CCNP and probably is more meant for Cisco TAC (unless you are with TAC):

I’ll be demonstrating on R5 running the 15.x code, and go through each so you get a taste of both commands output:

R5#sh mem ?
  allocating-process  Show allocating process name
  dead                Memory owned by dead processes
  debug               Memory debugging commands
  failures            Memory failures
  fast                Fast memory stats
  fragment            Summary of memory fragment information
  free                Free memory stats
  io                  IO memory stats
  multibus            Multibus memory stats
  overflow            memory overflow corrections
  pci                 PCI memory stats
  processor           Processor memory stats
  statistics          Mempool Statistics
  summary             Summary of memory usage per alloc PC
  transient           Transient memory stats
  |                   Output modifiers
  <cr>

So here we can actually see just “sh mem” on its own is a valid command, so lets look at the output of that once:

R5#sh mem
                Head    Total(b)     Used(b)     Free(b)   Lowest(b)  Largest(b)
Processor   64A4B380   165366912    21050552   144316360   141933980   140903024
      I/O   EE800000    25165824     4500520    20665304    20658976    20658940

          Processor memory

 Address      Bytes     Prev     Next Ref     PrevF    NextF Alloc PC  what
64A4B380 0000044004 00000000 64A55F94 001  ——– ——– 61EC6DB0  Exec
64A55F94 0000000028 64A4B380 64A55FE0 000  657898EC 653F9A7C 61EC6DB0  (fragment)
64A55FE0 0000020004 64A55F94 64A5AE34 001  ——– ——– 62E38678  OSPF redist ro
64A5AE34 0000032772 64A55FE0 64A62E68 001  ——– ——– 61E47414  OSPF work
64A62E68 0000020004 64A5AE34 64A67CBC 001  ——– ——– 61EA72F8  OSPF path
64A67CBC 0000032772 64A62E68 64A6FCF0 001  ——– ——– 61E70AF8  OSPF rt
64A6FCF0 0000001996 64A67CBC 64A704EC 001  ——– ——– 61E8474C  OSPF-1 Router
64A704EC 0000000492 64A6FCF0 64A70708 001  ——– ——– 61E81B34  OSPF-1 Router
64A70708 0000020004 64A704EC 64A7555C 001  ——– ——– 616148A0  IP ARP Adjacen
64A7555C 0000020004 64A70708 64A7A3B0 001  ——– ——– 61E73298  OSPF rtr path
64A7A3B0 0002829868 64A7555C 64D2D20C 000  0        6619FD5C 60F90F58  (coalesced)
 –More–

Notice there is more! However, I think that is enough to clarify these commands are strictly Cisco recommendations to keep in mind for exam day. Now for “sh mem allocating-process totals” :
R5#sh mem allocating-process totals
                Head    Total(b)     Used(b)     Free(b)   Lowest(b)  Largest(b)
Processor   64A4B380   165366912    21061272   144305640   141933980   140903024
      I/O   EE800000    25165824     4500520    20665304    20658976    20658940

Allocator PC Summary for: Processor

    PC          Total   Count  Name
0x620CBDE0    2766792      84  CCE dp subbloc
0x607227D8    1933428     239  Process Stack
0x604700F0    1247876    1087  *Packet Header*
0x627501A8     879940      85  TW Buckets
0x60F5F63C     838292      17  Init
0x627718CC     677960      16  pak subblock chunk
0x6027EEB8     364544     256  USB Startup
0x62A23A50     268528      33  Init
0x601BBF3C     262352       4  IPC Message He
0x61270168     262248       2  CEF: hash table
0x604A5350     253552      92  Normal
0x62A25518     221732    1050  *Init*
0x626F9B48     207296      10  CCE dp subbloc
 –More–

Also more, but I will spare the output, as we have one more and that is “sh mem summary” :

R5#sh mem summ
                Head    Total(b)     Used(b)     Free(b)   Lowest(b)  Largest(b)
Processor   64A4B380   165366912    21050552   144316360   141933980   140903024
      I/O   EE800000    25165824     4500520    20665304    20658976    20658940

          Processor memory

Alloc PC        Size     Blocks      Bytes    What

0x600142C4 0000000380 0000000001 0000000380    Init
0x600181C0 0000001024 0000000002 0000002048    AF filter
0x60018380 0000000028 0000000001 0000000028    AF entry
0x60018380 0000000072 0000000004 0000000288    AF entry
0x60018380 0000000084 0000000002 0000000168    AF entry
0x60018380 0000000100 0000000005 0000000500    AF entry
0x600193AC 0000001096 0000000008 0000008768    *In-use Packet Header*
0x6001A4B0 0000000024 0000000002 0000000048    Init
0x6001A4CC 0000000024 0000000002 0000000048    Init
0x6001A4E8 0000000024 0000000002 0000000048    Init
0x6001A5A0 0000000256 0000000008 0000002048    Init
0x6001A60C 0000001536 0000000004 0000006144    Init
0x6001C5B8 0000000768 0000000002 0000001536    Init
0x6001D808 0000001096 0000000002 0000002192    *In-use Packet Header*
0x6001E360 0000000136 0000000001 0000000136    Init
0x6001E53C 0000000128 0000000004 0000000512    Init
0x6001E56C 0000000128 0000000004 0000000512    Init
 –More–

Another big block of blah, so enough of that.

One real world tip to drop here, and I’m surprised it’s not included in the part of the course, is “sh proc cpu history” which will show you how close to pegged your CPU is along with charts to show spikes of usage over time, which I’ll throw up some quick examples:

R5#sh proc cpu history

R5   12:14:14 AM Thursday Apr 6 2017 UTC

         11111     22222                              11111
  100
   90
   80
   70
   60
   50
   40
   30
   20
   10
     0….5….1….1….2….2….3….3….4….4….5….5….6
               0    5    0    5    0    5    0    5    0    5    0
               CPU% per second (last 60 seconds)

                2
      121111111101111111111111111111111111111111111111111111111111
  100
   90
   80
   70
   60
   50
   40
   30
   20           *
   10           *
     0….5….1….1….2….2….3….3….4….4….5….5….6
               0    5    0    5    0    5    0    5    0    5    0
               CPU% per minute (last 60 minutes)
              * = maximum CPU%   # = average CPU%

      4
  100
   90
   80
   70
   60
   50
   40
   30
   20
   10
     0….5….1….1….2….2….3….3….4….4….5….5….6….6….7..
               0    5    0    5    0    5    0    5    0    5    0    5    0
                   CPU% per hour (last 72 hours)
                  * = maximum CPU%   # = average CPU%

R5#

Since there is nothing going on with R5 other than booting up and some quick ospf configs, but this is really invaluable for troubleshooting real world issues with CPU issues, as you can see spikes by the minute, hour, and 3 day range in a graph format.

This can be used on switches as well, and can tell you if a LAN slowness issue is being caused by switch with a pegged CPU, so really good Real World command to know!!

 

Making a Core Dump!

 

A core dump is a copy of the routers memory contents, which can be created on a router fairly easily, but Cisco has an official statement on making core dumps:

“CAUTION: Core dumps are not necessary to solve most crash cases. Creation of a core dump when the router is functioning in a network can disrupt network operation. Use… only under the direction of a technical support representative”

There are several methods to retrieve the core dump including FTP, TFTP, Flash Disk, etc.

Now again it’s not recommended to do this on a production router or really ever unless directed by Cisco TAC, however here are the command to do so:

  • “exception core-file (name)” this is used to change the default name given to the core dump, by default will be ‘Hostname-core’ so on R5 it would be “R5-core”
  • “exception region-size (#)” reserves a chunk of memory defined by the # to be used by the Core Dump in case the larger memory pool gets corrupted, the default size of this is 16384

Now the actual command to create the dump file is this:

R5#write core 172.12.15.100 ?
  LINE
  <cr>

R5#write core 172.12.15.100
Base name of core files to write [R5-core]?

Now I don’t have any kind of FTP server to back it up, and this probably won’t even show up on the ROUTE exam, but this is good to know material just in case it rears its ugly head.

With that, Security section is finished, and we start BGP next post! 🙂