Back with the Topology, because we are labbing this time! I have three more videos I want to compress into this one lesson as they are all pretty short and to the point, and then it is time to start the extremely large BGP section which I am really looking forward to labbing (absolutely new territory).
Anyways, I’ve setup OSPF for demonstration purposes, because of its occasional Hello’s multicast to 224.0.0.5 to examplify slimming down the output of “debug ip packet” though I would strongly recommend not doing this in a production network!
Part 1 – Filtering “debug ip packet” to see only the output you need
So just to verify we are all neighbors, lets take a look at R1:
R1#sh ip ospf nei
Neighbor ID Pri State Dead Time Address Interface
2.2.2.2 0 FULL/DROTHER 00:01:48 172.12.123.2 Serial0/0
3.3.3.3 0 FULL/DROTHER 00:01:59 172.12.123.3 Serial0/0
5.5.5.5 1 FULL/DR 00:00:35 172.12.15.5 FastEthernet0/1
R1#
All neighborinos, good deal, now I’ll run an initial “debug ip pack” to see how much data we get spilling onto the CLI:
R1#debug ip pack
IP packet debugging is on
R1#
*Mar 30 22:36:42.978: IP: s=172.12.15.5 (FastEthernet0/1), d=224.0.0.5, len 80, rcvd 0
R1#
*Mar 30 22:36:46.636: IP: s=172.12.15.1 (local), d=224.0.0.5 (FastEthernet0/1), len 80, sending broad/multicast
R1#
*Mar 30 22:36:49.893: IP: s=172.12.123.1 (local), d=172.12.123.2 (Serial0/0), len 84, sending
*Mar 30 22:36:49.893: IP: s=172.12.123.1 (local), d=172.12.123.3 (Serial0/0), len 84, sending
R1#
*Mar 30 22:36:52.578: IP: s=172.12.15.5 (FastEthernet0/1), d=224.0.0.5, len 80, rcvd 0
R1#
*Mar 30 22:36:56.636: IP: s=172.12.15.1 (local), d=224.0.0.5 (FastEthernet0/1), len 80, sending broad/multicast
R1#
*Mar 30 22:36:58.972: IP: s=172.12.123.2 (Serial0/0), d=172.12.123.1, len 80, rcvd 0
R1#u a
*Mar 30 22:37:02.514: IP: s=172.12.15.5 (FastEthernet0/1), d=224.0.0.5, len 80, rcvd 0
R1#u all
All possible debugging has been turned off
R1#
So relatively nothing, just your standard Hello’s, so lets add some traffic to see a REAL debug ip pack output with even just ping traffic constantly hitting R1:
R2#ping 172.12.123.1 repeat 10000
Type escape sequence to abort.
Sending 10000, 100-byte ICMP Echos to 172.12.123.1, timeout is 2 seconds:
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
ASR#1
[Resuming connection 1 to r1 … ]
R1#debug ip pack
IP packet debugging is on
R1#
*Mar 30 22:38:39.805: IP: tableid=0, s=172.12.123.2 (Serial0/0), d=172.12.123.1 (Serial0/0), routed via RIB
*Mar 30 22:38:39.809: IP: s=172.12.123.2 (Serial0/0), d=172.12.123.1 (Serial0/0), len 100, rcvd 3
*Mar 30 22:38:39.809: IP: tableid=0, s=172.12.123.1 (local), d=172.12.123.2 (Serial0/0), routed via FIB
*Mar 30 22:38:39.809: IP: s=172.12.123.1 (local), d=172.12.123.2 (Serial0/0), len 100, sending
*Mar 30 22:38:39.877: IP: tableid=0, s=172.12.123.2 (Serial0/0), d=172.12.123.1 (Serial0/0), routed via RIB
*Mar 30 22:38:39.877: IP: s=172.12.123.2 (Serial0/0), d=172.12.123.1 (Serial0/0), len 100, rcvd 3
*Mar 30 22:38:39.877: IP: tableid=0, s=172.12.123.1 (local), d=172.12.123.2 (Serial0/0), routed via FIB
*Mar 30 22:38:39.881: IP: s=172.12.123.1 (local), d=172.12.123.2 (Serial0/0), len 100, sending
*Mar 30 22:38:39.946: IP: tableid=0, s=172.12.123.2 (Serial0/0), d=172.12.123.1 (Serial0/0), routed via RIB
*Mar 30 22:38:39.950: IP: s=172.12.123.2 (Serial0/0), d=172.12.123.1 (
R1#Serial0/0), len 100, rcvd 3
*Mar 30 22:38:39.950: IP: tableid=0, s=172.12.123.1 (local), d=172.12.123.2 (Serial0/0), routed via FIB
*Mar 30 22:38:39.950: IP: s=172.12.123.1 (local), d=172.12.123.2 (Serial0/0), len 100, sending
*Mar 30 22:38:39.958: IP: s=172.12.123.3 (Serial0/0), d=172.12.123.1, len 80, rcvd 0
*Mar 30 22:38:40.018: IP: tableid=0, s=172.12.123.2 (Serial0/0), d=172.12.123.1 (Serial0/0), routed via RIB
*Mar 30 22:38:40.018: IP: s=172.12.123.2 (Serial0/0), d=172.12.123.1 (Serial0/0), len 100, rcvd 3
*Mar 30 22:38:40.022: IP: tableid=0, s=172.12.123.1 (local), d=172.12.123.2 (Serial0/0), routed via FIB
*Mar 30 22:38:40.022: IP: s=172.12.123.1 (local), d=172.12.123.2 (Serial0/0), len 100, sending
Now that is just one ping, so full on network traffic can take down a router within seconds of issuing the command if not in microseconds. However, if we need to hone in on one host to see the traffic coming from it and going to it, there is a way to get all other traffic out of the debug.
This is done with our all time hero, Access Control Lists! For this example I want to focus on traffic only coming from, or going to R5’s 172.12.15.5 IP address, so we will need two ACL’s to accomplish this as you cannot set the source and destination to 172.12.15.5 to accomplish this task (as much as we’d like it to be as easy):
R1(config)#access-list 123 permit ip host 172.12.15.5 any
R1(config)#access-list 123 permit ip any host 172.12.15.5
R1(config)#
Tada! Now any traffic with a source or destination of any with a corresponding source or destination of 172.12.15.5 will be seen, and here is how to configure the debug:
R1#debug ip pack ?
<1-199> Access list
<1300-2699> Access list (expanded range)
detail Print more debugging detail
<cr>
R1#debug ip pack 123 ?
detail Print more debugging detail
<cr>
R1#debug ip pack 123
IP packet debugging is on for access list 123
We are now debugging with the parameters of access-list 123, and I’ve even highlighted in red the output from the console will tell you that you’re debugging using the ACL.
So let’s check out the output from this:
R1#
*Mar 30 22:47:54.242: IP: s=172.12.15.5 (FastEthernet0/1), d=224.0.0.5, len 80, rcvd 0
R1#
ASR#2
[Resuming connection 2 to r2 … ]
R2#ping 172.12.123.1 repeat 1000
Type escape sequence to abort.
Sending 1000, 100-byte ICMP Echos to 172.12.123.1, timeout is 2 seconds:
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
ASR#1
[Resuming connection 1 to r1 … ]
*Mar 30 22:48:03.638: IP: s=172.12.15.5 (FastEthernet0/1), d=224.0.0.5, len 80, rcvd 0
R1#
*Mar 30 22:48:13.073: IP: s=172.12.15.5 (FastEthernet0/1), d=224.0.0.5, len 80, rcvd 0
R1#
*Mar 30 22:48:22.537: IP: s=172.12.15.5 (FastEthernet0/1), d=224.0.0.5, len 80, rcvd 0
R1#
*Mar 30 22:48:31.860: IP: s=172.12.15.5 (FastEthernet0/1), d=224.0.0.5, len 80, rcvd 0
So it is working because of that implicit deny on the end of the ACL, we aren’t seeing those pings, but another interesting thing that I thought was great to point out is that you see Hello’s sourced from R5 but none from R1 with a destination of R5’s 172.12.15.5 interface.
This is because the source would be 172.12.15.1 and the destination would be 224.0.0.5, so it also does not match the ACL criteria, so it is working perfectly.
Now one interesting behavior to note, is when you remove the ACL while the debug is running, lets check it out:
R1#debug ip pack 123
IP packet debugging is on for access list 123
R1#
*Mar 30 22:53:35.021: IP: s=172.12.15.5 (FastEthernet0/1), d=224.0.0.5, len 80, rcvd 0
R1#conf t
Enter configuration commands, one per line. End with CNTL/Z.
R1(config)#no
*Mar 30 22:53:44.444: IP: s=172.12.15.5 (FastEthernet0/1), d=224.0.0.5, len 80, rcvd 0
R1(config)#no access-list 123
IP packet debugging is off
Turning off all possible debugging on ACL 123
R1(config)#do sh access-list
R1(config)#do sh debug
R1(config)#
It immediately stops the debugging of both IP Packet, and all possible debugging for that particular ACL, so at the same time this removes the ACL and stops the debug on the router be sure to verify that both ACL and debug are gone before running for the door.
One more thing you can see on the output for debug ip pack 123 is “detail” as a modifier, which lets see what more possible detail it wants to give us:
R1#debug ip pack 123 detail
IP packet debugging is on (detailed) for access list 123
R1#
*Mar 30 22:57:31.348: IP: s=172.12.15.5 (FastEthernet0/1), d=224.0.0.5, len 80, rcvd 0, proto=89
R1#
*Mar 30 22:57:41.329: IP: s=172.12.15.5 (FastEthernet0/1), d=224.0.0.5, len 80, rcvd 0, proto=89
R1#
*Mar 30 22:57:50.472: IP: s=172.12.15.5 (FastEthernet0/1), d=224.0.0.5, len 80, rcvd 0, proto=89
R1#
*Mar 30 22:58:00.051: IP: s=172.12.15.5 (FastEthernet0/1), d=224.0.0.5, len 80, rcvd 0, proto=89
R1#
Not really much additional detail, except a proto=89 tagged on the end there, so there isn’t a whole lot to that to know at this time in ROUTE studies that it exists and doesn’t give a whole lot of detail contrary to its name!
Spotting memory issues in Router Memory
A couple of ways to spot memory issues that the router will have in its output or behaviors that will happen if memory is getting low, I will list bullet point style:
- If the router rejects a telnet session it may be low on memory
- If the router shows the output “show processor memory” regardless of the command you enter that’s a definite memory issue
- If your router literally has the output “low on memory” or “Unable to create EXEC – no memory or too many processes” you also definitely have a memory problem
- Show commands not giving you any output / freezing up / console sessions hanging
Cisco recommends if you have an issue connecting to the console port, to unplug the LAN and WAN cables so it is not processing packets, which should free up some memory to connect. If you still cannot connect, you are up chocolate creek without a popsicle stick!
Cisco also recommends for “show” commands in terms of showing memory issues here, but they are very cumbersome commands (out of scope for the CCNP) but wanted to show them here. The two commands are “show memory allocating-process totals” and “show memory summary” which will give you huge outputs that again are way beyond the scope of CCNP and probably is more meant for Cisco TAC (unless you are with TAC):
I’ll be demonstrating on R5 running the 15.x code, and go through each so you get a taste of both commands output:
R5#sh mem ?
allocating-process Show allocating process name
dead Memory owned by dead processes
debug Memory debugging commands
failures Memory failures
fast Fast memory stats
fragment Summary of memory fragment information
free Free memory stats
io IO memory stats
multibus Multibus memory stats
overflow memory overflow corrections
pci PCI memory stats
processor Processor memory stats
statistics Mempool Statistics
summary Summary of memory usage per alloc PC
transient Transient memory stats
| Output modifiers
<cr>
So here we can actually see just “sh mem” on its own is a valid command, so lets look at the output of that once:
R5#sh mem
Head Total(b) Used(b) Free(b) Lowest(b) Largest(b)
Processor 64A4B380 165366912 21050552 144316360 141933980 140903024
I/O EE800000 25165824 4500520 20665304 20658976 20658940
Processor memory
Address Bytes Prev Next Ref PrevF NextF Alloc PC what
64A4B380 0000044004 00000000 64A55F94 001 ——– ——– 61EC6DB0 Exec
64A55F94 0000000028 64A4B380 64A55FE0 000 657898EC 653F9A7C 61EC6DB0 (fragment)
64A55FE0 0000020004 64A55F94 64A5AE34 001 ——– ——– 62E38678 OSPF redist ro
64A5AE34 0000032772 64A55FE0 64A62E68 001 ——– ——– 61E47414 OSPF work
64A62E68 0000020004 64A5AE34 64A67CBC 001 ——– ——– 61EA72F8 OSPF path
64A67CBC 0000032772 64A62E68 64A6FCF0 001 ——– ——– 61E70AF8 OSPF rt
64A6FCF0 0000001996 64A67CBC 64A704EC 001 ——– ——– 61E8474C OSPF-1 Router
64A704EC 0000000492 64A6FCF0 64A70708 001 ——– ——– 61E81B34 OSPF-1 Router
64A70708 0000020004 64A704EC 64A7555C 001 ——– ——– 616148A0 IP ARP Adjacen
64A7555C 0000020004 64A70708 64A7A3B0 001 ——– ——– 61E73298 OSPF rtr path
64A7A3B0 0002829868 64A7555C 64D2D20C 000 0 6619FD5C 60F90F58 (coalesced)
–More–
Notice there is more! However, I think that is enough to clarify these commands are strictly Cisco recommendations to keep in mind for exam day. Now for “sh mem allocating-process totals” :
R5#sh mem allocating-process totals
Head Total(b) Used(b) Free(b) Lowest(b) Largest(b)
Processor 64A4B380 165366912 21061272 144305640 141933980 140903024
I/O EE800000 25165824 4500520 20665304 20658976 20658940
Allocator PC Summary for: Processor
PC Total Count Name
0x620CBDE0 2766792 84 CCE dp subbloc
0x607227D8 1933428 239 Process Stack
0x604700F0 1247876 1087 *Packet Header*
0x627501A8 879940 85 TW Buckets
0x60F5F63C 838292 17 Init
0x627718CC 677960 16 pak subblock chunk
0x6027EEB8 364544 256 USB Startup
0x62A23A50 268528 33 Init
0x601BBF3C 262352 4 IPC Message He
0x61270168 262248 2 CEF: hash table
0x604A5350 253552 92 Normal
0x62A25518 221732 1050 *Init*
0x626F9B48 207296 10 CCE dp subbloc
–More–
Also more, but I will spare the output, as we have one more and that is “sh mem summary” :
R5#sh mem summ
Head Total(b) Used(b) Free(b) Lowest(b) Largest(b)
Processor 64A4B380 165366912 21050552 144316360 141933980 140903024
I/O EE800000 25165824 4500520 20665304 20658976 20658940
Processor memory
Alloc PC Size Blocks Bytes What
0x600142C4 0000000380 0000000001 0000000380 Init
0x600181C0 0000001024 0000000002 0000002048 AF filter
0x60018380 0000000028 0000000001 0000000028 AF entry
0x60018380 0000000072 0000000004 0000000288 AF entry
0x60018380 0000000084 0000000002 0000000168 AF entry
0x60018380 0000000100 0000000005 0000000500 AF entry
0x600193AC 0000001096 0000000008 0000008768 *In-use Packet Header*
0x6001A4B0 0000000024 0000000002 0000000048 Init
0x6001A4CC 0000000024 0000000002 0000000048 Init
0x6001A4E8 0000000024 0000000002 0000000048 Init
0x6001A5A0 0000000256 0000000008 0000002048 Init
0x6001A60C 0000001536 0000000004 0000006144 Init
0x6001C5B8 0000000768 0000000002 0000001536 Init
0x6001D808 0000001096 0000000002 0000002192 *In-use Packet Header*
0x6001E360 0000000136 0000000001 0000000136 Init
0x6001E53C 0000000128 0000000004 0000000512 Init
0x6001E56C 0000000128 0000000004 0000000512 Init
–More–
Another big block of blah, so enough of that.
One real world tip to drop here, and I’m surprised it’s not included in the part of the course, is “sh proc cpu history” which will show you how close to pegged your CPU is along with charts to show spikes of usage over time, which I’ll throw up some quick examples:
R5#sh proc cpu history
R5 12:14:14 AM Thursday Apr 6 2017 UTC
11111 22222 11111
100
90
80
70
60
50
40
30
20
10
0….5….1….1….2….2….3….3….4….4….5….5….6
0 5 0 5 0 5 0 5 0 5 0
CPU% per second (last 60 seconds)
2
121111111101111111111111111111111111111111111111111111111111
100
90
80
70
60
50
40
30
20 *
10 *
0….5….1….1….2….2….3….3….4….4….5….5….6
0 5 0 5 0 5 0 5 0 5 0
CPU% per minute (last 60 minutes)
* = maximum CPU% # = average CPU%
4
100
90
80
70
60
50
40
30
20
10
0….5….1….1….2….2….3….3….4….4….5….5….6….6….7..
0 5 0 5 0 5 0 5 0 5 0 5 0
CPU% per hour (last 72 hours)
* = maximum CPU% # = average CPU%
R5#
Since there is nothing going on with R5 other than booting up and some quick ospf configs, but this is really invaluable for troubleshooting real world issues with CPU issues, as you can see spikes by the minute, hour, and 3 day range in a graph format.
This can be used on switches as well, and can tell you if a LAN slowness issue is being caused by switch with a pegged CPU, so really good Real World command to know!!
Making a Core Dump!
A core dump is a copy of the routers memory contents, which can be created on a router fairly easily, but Cisco has an official statement on making core dumps:
“CAUTION: Core dumps are not necessary to solve most crash cases. Creation of a core dump when the router is functioning in a network can disrupt network operation. Use… only under the direction of a technical support representative”
There are several methods to retrieve the core dump including FTP, TFTP, Flash Disk, etc.
Now again it’s not recommended to do this on a production router or really ever unless directed by Cisco TAC, however here are the command to do so:
- “exception core-file (name)” this is used to change the default name given to the core dump, by default will be ‘Hostname-core’ so on R5 it would be “R5-core”
- “exception region-size (#)” reserves a chunk of memory defined by the # to be used by the Core Dump in case the larger memory pool gets corrupted, the default size of this is 16384
Now the actual command to create the dump file is this:
R5#write core 172.12.15.100 ?
LINE
<cr>
R5#write core 172.12.15.100
Base name of core files to write [R5-core]?
Now I don’t have any kind of FTP server to back it up, and this probably won’t even show up on the ROUTE exam, but this is good to know material just in case it rears its ugly head.
With that, Security section is finished, and we start BGP next post! 🙂
The DEVNET GRIND! 