Category Archives: CCNP – Security

Security Triple-Header to finish the Security Section: Filtering “debug ip packet”, spotting memory issues, creating and understanding core dumps!

OSPF_NBMA_15

Back with the Topology, because we are labbing this time! I have three more videos I want to compress into this one lesson as they are all pretty short and to the point, and then it is time to start the extremely large BGP section which I am really looking forward to labbing (absolutely new territory).

Anyways, I’ve setup OSPF for demonstration purposes, because of its occasional Hello’s multicast to 224.0.0.5 to examplify slimming down the output of “debug ip packet” though I would strongly recommend not doing this in a production network!

 

Part 1 – Filtering “debug ip packet” to see only the output you need

 

So just to verify we are all neighbors, lets take a look at R1:

R1#sh ip ospf nei

Neighbor ID     Pri   State           Dead Time   Address         Interface
2.2.2.2           0   FULL/DROTHER    00:01:48    172.12.123.2    Serial0/0
3.3.3.3           0   FULL/DROTHER    00:01:59    172.12.123.3    Serial0/0
5.5.5.5           1   FULL/DR         00:00:35    172.12.15.5     FastEthernet0/1
R1#

All neighborinos, good deal, now I’ll run an initial “debug ip pack” to see how much data we get spilling onto the CLI:

R1#debug ip pack
IP packet debugging is on
R1#
*Mar 30 22:36:42.978: IP: s=172.12.15.5 (FastEthernet0/1), d=224.0.0.5, len 80, rcvd 0
R1#
*Mar 30 22:36:46.636: IP: s=172.12.15.1 (local), d=224.0.0.5 (FastEthernet0/1), len 80, sending broad/multicast
R1#
*Mar 30 22:36:49.893: IP: s=172.12.123.1 (local), d=172.12.123.2 (Serial0/0), len 84, sending
*Mar 30 22:36:49.893: IP: s=172.12.123.1 (local), d=172.12.123.3 (Serial0/0), len 84, sending
R1#
*Mar 30 22:36:52.578: IP: s=172.12.15.5 (FastEthernet0/1), d=224.0.0.5, len 80, rcvd 0
R1#
*Mar 30 22:36:56.636: IP: s=172.12.15.1 (local), d=224.0.0.5 (FastEthernet0/1), len 80, sending broad/multicast
R1#
*Mar 30 22:36:58.972: IP: s=172.12.123.2 (Serial0/0), d=172.12.123.1, len 80, rcvd 0
R1#u a
*Mar 30 22:37:02.514: IP: s=172.12.15.5 (FastEthernet0/1), d=224.0.0.5, len 80, rcvd 0
R1#u all
All possible debugging has been turned off
R1#

So relatively nothing, just your standard Hello’s, so lets add some traffic to see a REAL debug ip pack output with even just ping traffic constantly hitting R1:

R2#ping 172.12.123.1 repeat 10000

Type escape sequence to abort.
Sending 10000, 100-byte ICMP Echos to 172.12.123.1, timeout is 2 seconds:
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
ASR#1
[Resuming connection 1 to r1 … ]

R1#debug ip pack
IP packet debugging is on
R1#
*Mar 30 22:38:39.805: IP: tableid=0, s=172.12.123.2 (Serial0/0), d=172.12.123.1 (Serial0/0), routed via RIB
*Mar 30 22:38:39.809: IP: s=172.12.123.2 (Serial0/0), d=172.12.123.1 (Serial0/0), len 100, rcvd 3
*Mar 30 22:38:39.809: IP: tableid=0, s=172.12.123.1 (local), d=172.12.123.2 (Serial0/0), routed via FIB
*Mar 30 22:38:39.809: IP: s=172.12.123.1 (local), d=172.12.123.2 (Serial0/0), len 100, sending
*Mar 30 22:38:39.877: IP: tableid=0, s=172.12.123.2 (Serial0/0), d=172.12.123.1 (Serial0/0), routed via RIB
*Mar 30 22:38:39.877: IP: s=172.12.123.2 (Serial0/0), d=172.12.123.1 (Serial0/0), len 100, rcvd 3
*Mar 30 22:38:39.877: IP: tableid=0, s=172.12.123.1 (local), d=172.12.123.2 (Serial0/0), routed via FIB
*Mar 30 22:38:39.881: IP: s=172.12.123.1 (local), d=172.12.123.2 (Serial0/0), len 100, sending
*Mar 30 22:38:39.946: IP: tableid=0, s=172.12.123.2 (Serial0/0), d=172.12.123.1 (Serial0/0), routed via RIB
*Mar 30 22:38:39.950: IP: s=172.12.123.2 (Serial0/0), d=172.12.123.1 (
R1#Serial0/0), len 100, rcvd 3
*Mar 30 22:38:39.950: IP: tableid=0, s=172.12.123.1 (local), d=172.12.123.2 (Serial0/0), routed via FIB
*Mar 30 22:38:39.950: IP: s=172.12.123.1 (local), d=172.12.123.2 (Serial0/0), len 100, sending
*Mar 30 22:38:39.958: IP: s=172.12.123.3 (Serial0/0), d=172.12.123.1, len 80, rcvd 0
*Mar 30 22:38:40.018: IP: tableid=0, s=172.12.123.2 (Serial0/0), d=172.12.123.1 (Serial0/0), routed via RIB
*Mar 30 22:38:40.018: IP: s=172.12.123.2 (Serial0/0), d=172.12.123.1 (Serial0/0), len 100, rcvd 3
*Mar 30 22:38:40.022: IP: tableid=0, s=172.12.123.1 (local), d=172.12.123.2 (Serial0/0), routed via FIB
*Mar 30 22:38:40.022: IP: s=172.12.123.1 (local), d=172.12.123.2 (Serial0/0), len 100, sending

Now that is just one ping, so full on network traffic can take down a router within seconds of issuing the command if not in microseconds. However, if we need to hone in on one host to see the traffic coming from it and going to it, there is a way to get all other traffic out of the debug.

This is done with our all time hero, Access Control Lists! For this example I want to focus on traffic only coming from, or going to R5’s 172.12.15.5 IP address, so we will need two ACL’s to accomplish this as you cannot set the source and destination to 172.12.15.5 to accomplish this task (as much as we’d like it to be as easy):

R1(config)#access-list 123 permit ip host 172.12.15.5 any
R1(config)#access-list 123 permit ip any host 172.12.15.5
R1(config)#

Tada! Now any traffic with a source or destination of any with a corresponding source or destination of 172.12.15.5 will be seen, and here is how to configure the debug:

R1#debug ip pack ?
  <1-199>      Access list
  <1300-2699>  Access list (expanded range)
  detail       Print more debugging detail
  <cr>

R1#debug ip pack 123 ?
  detail  Print more debugging detail
  <cr>

R1#debug ip pack 123
IP packet debugging is on for access list 123

We are now debugging with the parameters of access-list 123, and I’ve even highlighted in red the output from the console will tell you that you’re debugging using the ACL.

So let’s check out the output from this:
R1#
*Mar 30 22:47:54.242: IP: s=172.12.15.5 (FastEthernet0/1), d=224.0.0.5, len 80, rcvd 0
R1#
ASR#2
[Resuming connection 2 to r2 … ]

R2#ping 172.12.123.1 repeat 1000

Type escape sequence to abort.
Sending 1000, 100-byte ICMP Echos to 172.12.123.1, timeout is 2 seconds:
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
ASR#1
[Resuming connection 1 to r1 … ]

*Mar 30 22:48:03.638: IP: s=172.12.15.5 (FastEthernet0/1), d=224.0.0.5, len 80, rcvd 0
R1#
*Mar 30 22:48:13.073: IP: s=172.12.15.5 (FastEthernet0/1), d=224.0.0.5, len 80, rcvd 0
R1#
*Mar 30 22:48:22.537: IP: s=172.12.15.5 (FastEthernet0/1), d=224.0.0.5, len 80, rcvd 0
R1#
*Mar 30 22:48:31.860: IP: s=172.12.15.5 (FastEthernet0/1), d=224.0.0.5, len 80, rcvd 0

So it is working because of that implicit deny on the end of the ACL, we aren’t seeing those pings, but another interesting thing that I thought was great to point out is that you see Hello’s sourced from R5 but none from R1 with a destination of R5’s 172.12.15.5 interface.

This is because the source would be 172.12.15.1 and the destination would be 224.0.0.5, so it also does not match the ACL criteria, so it is working perfectly.

Now one interesting behavior to note, is when you remove the ACL while the debug is running, lets check it out:

R1#debug ip pack 123
IP packet debugging is on for access list 123
R1#
*Mar 30 22:53:35.021: IP: s=172.12.15.5 (FastEthernet0/1), d=224.0.0.5, len 80, rcvd 0
R1#conf t
Enter configuration commands, one per line.  End with CNTL/Z.
R1(config)#no
*Mar 30 22:53:44.444: IP: s=172.12.15.5 (FastEthernet0/1), d=224.0.0.5, len 80, rcvd 0
R1(config)#no access-list 123
IP packet debugging is off

Turning off all possible debugging on ACL 123

R1(config)#do sh access-list

R1(config)#do sh debug

R1(config)#

It immediately stops the debugging of both IP Packet, and all possible debugging for that particular ACL, so at the same time this removes the ACL and stops the debug on the router be sure to verify that both ACL and debug are gone before running for the door.

One more thing you can see on the output for debug ip pack 123 is “detail” as a modifier, which lets see what more possible detail it wants to give us:

R1#debug ip pack 123 detail
IP packet debugging is on (detailed) for access list 123
R1#
*Mar 30 22:57:31.348: IP: s=172.12.15.5 (FastEthernet0/1), d=224.0.0.5, len 80, rcvd 0, proto=89
R1#
*Mar 30 22:57:41.329: IP: s=172.12.15.5 (FastEthernet0/1), d=224.0.0.5, len 80, rcvd 0, proto=89
R1#
*Mar 30 22:57:50.472: IP: s=172.12.15.5 (FastEthernet0/1), d=224.0.0.5, len 80, rcvd 0, proto=89
R1#
*Mar 30 22:58:00.051: IP: s=172.12.15.5 (FastEthernet0/1), d=224.0.0.5, len 80, rcvd 0, proto=89
R1#

Not really much additional detail, except a proto=89 tagged on the end there, so there isn’t a whole lot to that to know at this time in ROUTE studies that it exists and doesn’t give a whole lot of detail contrary to its name!

 

Spotting memory issues in Router Memory

 

A couple of ways to spot memory issues that the router will have in its output or behaviors that will happen if memory is getting low, I will list bullet point style:

  • If the router rejects a telnet session it may be low on memory
  • If the router shows the output “show processor memory” regardless of the command you enter that’s a definite memory issue
  • If your router literally has the output “low on memory” or “Unable to create EXEC – no memory or too many processes” you also definitely have a memory problem
  • Show commands not giving you any output / freezing up / console sessions hanging

Cisco recommends if you have an issue connecting to the console port, to unplug the LAN and WAN cables so it is not processing packets, which should free up some memory to connect. If you still cannot connect, you are up chocolate creek without a popsicle stick!

Cisco also recommends for “show” commands in terms of showing memory issues here, but they are very cumbersome commands (out of scope for the CCNP) but wanted to show them here. The two commands are “show memory allocating-process totals” and “show memory summary” which will give you huge outputs that again are way beyond the scope of CCNP and probably is more meant for Cisco TAC (unless you are with TAC):

I’ll be demonstrating on R5 running the 15.x code, and go through each so you get a taste of both commands output:

R5#sh mem ?
  allocating-process  Show allocating process name
  dead                Memory owned by dead processes
  debug               Memory debugging commands
  failures            Memory failures
  fast                Fast memory stats
  fragment            Summary of memory fragment information
  free                Free memory stats
  io                  IO memory stats
  multibus            Multibus memory stats
  overflow            memory overflow corrections
  pci                 PCI memory stats
  processor           Processor memory stats
  statistics          Mempool Statistics
  summary             Summary of memory usage per alloc PC
  transient           Transient memory stats
  |                   Output modifiers
  <cr>

So here we can actually see just “sh mem” on its own is a valid command, so lets look at the output of that once:

R5#sh mem
                Head    Total(b)     Used(b)     Free(b)   Lowest(b)  Largest(b)
Processor   64A4B380   165366912    21050552   144316360   141933980   140903024
      I/O   EE800000    25165824     4500520    20665304    20658976    20658940

          Processor memory

 Address      Bytes     Prev     Next Ref     PrevF    NextF Alloc PC  what
64A4B380 0000044004 00000000 64A55F94 001  ——– ——– 61EC6DB0  Exec
64A55F94 0000000028 64A4B380 64A55FE0 000  657898EC 653F9A7C 61EC6DB0  (fragment)
64A55FE0 0000020004 64A55F94 64A5AE34 001  ——– ——– 62E38678  OSPF redist ro
64A5AE34 0000032772 64A55FE0 64A62E68 001  ——– ——– 61E47414  OSPF work
64A62E68 0000020004 64A5AE34 64A67CBC 001  ——– ——– 61EA72F8  OSPF path
64A67CBC 0000032772 64A62E68 64A6FCF0 001  ——– ——– 61E70AF8  OSPF rt
64A6FCF0 0000001996 64A67CBC 64A704EC 001  ——– ——– 61E8474C  OSPF-1 Router
64A704EC 0000000492 64A6FCF0 64A70708 001  ——– ——– 61E81B34  OSPF-1 Router
64A70708 0000020004 64A704EC 64A7555C 001  ——– ——– 616148A0  IP ARP Adjacen
64A7555C 0000020004 64A70708 64A7A3B0 001  ——– ——– 61E73298  OSPF rtr path
64A7A3B0 0002829868 64A7555C 64D2D20C 000  0        6619FD5C 60F90F58  (coalesced)
 –More–

Notice there is more! However, I think that is enough to clarify these commands are strictly Cisco recommendations to keep in mind for exam day. Now for “sh mem allocating-process totals” :
R5#sh mem allocating-process totals
                Head    Total(b)     Used(b)     Free(b)   Lowest(b)  Largest(b)
Processor   64A4B380   165366912    21061272   144305640   141933980   140903024
      I/O   EE800000    25165824     4500520    20665304    20658976    20658940

Allocator PC Summary for: Processor

    PC          Total   Count  Name
0x620CBDE0    2766792      84  CCE dp subbloc
0x607227D8    1933428     239  Process Stack
0x604700F0    1247876    1087  *Packet Header*
0x627501A8     879940      85  TW Buckets
0x60F5F63C     838292      17  Init
0x627718CC     677960      16  pak subblock chunk
0x6027EEB8     364544     256  USB Startup
0x62A23A50     268528      33  Init
0x601BBF3C     262352       4  IPC Message He
0x61270168     262248       2  CEF: hash table
0x604A5350     253552      92  Normal
0x62A25518     221732    1050  *Init*
0x626F9B48     207296      10  CCE dp subbloc
 –More–

Also more, but I will spare the output, as we have one more and that is “sh mem summary” :

R5#sh mem summ
                Head    Total(b)     Used(b)     Free(b)   Lowest(b)  Largest(b)
Processor   64A4B380   165366912    21050552   144316360   141933980   140903024
      I/O   EE800000    25165824     4500520    20665304    20658976    20658940

          Processor memory

Alloc PC        Size     Blocks      Bytes    What

0x600142C4 0000000380 0000000001 0000000380    Init
0x600181C0 0000001024 0000000002 0000002048    AF filter
0x60018380 0000000028 0000000001 0000000028    AF entry
0x60018380 0000000072 0000000004 0000000288    AF entry
0x60018380 0000000084 0000000002 0000000168    AF entry
0x60018380 0000000100 0000000005 0000000500    AF entry
0x600193AC 0000001096 0000000008 0000008768    *In-use Packet Header*
0x6001A4B0 0000000024 0000000002 0000000048    Init
0x6001A4CC 0000000024 0000000002 0000000048    Init
0x6001A4E8 0000000024 0000000002 0000000048    Init
0x6001A5A0 0000000256 0000000008 0000002048    Init
0x6001A60C 0000001536 0000000004 0000006144    Init
0x6001C5B8 0000000768 0000000002 0000001536    Init
0x6001D808 0000001096 0000000002 0000002192    *In-use Packet Header*
0x6001E360 0000000136 0000000001 0000000136    Init
0x6001E53C 0000000128 0000000004 0000000512    Init
0x6001E56C 0000000128 0000000004 0000000512    Init
 –More–

Another big block of blah, so enough of that.

One real world tip to drop here, and I’m surprised it’s not included in the part of the course, is “sh proc cpu history” which will show you how close to pegged your CPU is along with charts to show spikes of usage over time, which I’ll throw up some quick examples:

R5#sh proc cpu history

R5   12:14:14 AM Thursday Apr 6 2017 UTC

         11111     22222                              11111
  100
   90
   80
   70
   60
   50
   40
   30
   20
   10
     0….5….1….1….2….2….3….3….4….4….5….5….6
               0    5    0    5    0    5    0    5    0    5    0
               CPU% per second (last 60 seconds)

                2
      121111111101111111111111111111111111111111111111111111111111
  100
   90
   80
   70
   60
   50
   40
   30
   20           *
   10           *
     0….5….1….1….2….2….3….3….4….4….5….5….6
               0    5    0    5    0    5    0    5    0    5    0
               CPU% per minute (last 60 minutes)
              * = maximum CPU%   # = average CPU%

      4
  100
   90
   80
   70
   60
   50
   40
   30
   20
   10
     0….5….1….1….2….2….3….3….4….4….5….5….6….6….7..
               0    5    0    5    0    5    0    5    0    5    0    5    0
                   CPU% per hour (last 72 hours)
                  * = maximum CPU%   # = average CPU%

R5#

Since there is nothing going on with R5 other than booting up and some quick ospf configs, but this is really invaluable for troubleshooting real world issues with CPU issues, as you can see spikes by the minute, hour, and 3 day range in a graph format.

This can be used on switches as well, and can tell you if a LAN slowness issue is being caused by switch with a pegged CPU, so really good Real World command to know!!

 

Making a Core Dump!

 

A core dump is a copy of the routers memory contents, which can be created on a router fairly easily, but Cisco has an official statement on making core dumps:

“CAUTION: Core dumps are not necessary to solve most crash cases. Creation of a core dump when the router is functioning in a network can disrupt network operation. Use… only under the direction of a technical support representative”

There are several methods to retrieve the core dump including FTP, TFTP, Flash Disk, etc.

Now again it’s not recommended to do this on a production router or really ever unless directed by Cisco TAC, however here are the command to do so:

  • “exception core-file (name)” this is used to change the default name given to the core dump, by default will be ‘Hostname-core’ so on R5 it would be “R5-core”
  • “exception region-size (#)” reserves a chunk of memory defined by the # to be used by the Core Dump in case the larger memory pool gets corrupted, the default size of this is 16384

Now the actual command to create the dump file is this:

R5#write core 172.12.15.100 ?
  LINE
  <cr>

R5#write core 172.12.15.100
Base name of core files to write [R5-core]?

Now I don’t have any kind of FTP server to back it up, and this probably won’t even show up on the ROUTE exam, but this is good to know material just in case it rears its ugly head.

With that, Security section is finished, and we start BGP next post! ūüôā

Ip helper-address command explanation and configuration, and some important details and one additional helper cmd!

This will be a quicky I hope, pretty straight forward topic, but I always say that so lets get started.

The IP helper-address is a command issued at the interface level, on the interface of incoming broadcasts, and its purpose is to forward UDP broadcasts on (as routers DO NOT forward broadcasts) to other routers in Unicast format.

The command itself is fairly simple:

R1(config-if)#ip helper-address ?
  A.B.C.D  IP destination address
  global   Helper-address is global
  vrf      VRF name for helper-address (if different from interface VRF)

R1(config-if)#ip helper-address 172.12.123.2 ?
  redundancy  defines VRG group name
  <cr>

R1(config-if)#ip helper-address 172.12.123.2
R1(config-if)#

In red is what we are focusing on, configuring an IP address, though it is worth noting that you can assign VRF instances for it as well as a global command.

Now the global command was not covered in my materials thus far, but from what I researched (fairly quickly), it works in conjunction with VRF to allow ip helper-address to work with VRF – For the CCNP ROUTE these very well could be unnecessary to know but I wanted to point them out.

Now, back to business.

This command will primarily be used to forward BOOTP (DHCP) requests from host machines onto Domain Controllers running a DHCP server, but it is also VERY important for Cisco Voice as VOIP phones sometimes need a helper-address to reach their TFTP server to pull down their settings. Wanted to drop that little bit of real world knowledge.

By default, the “ip helper-address x.x.x.x” command only forward 8 UDP port #’s:

  • TIME (TimeServer) = 37¬† *** Note this is not NTP and has nothing to do with NTP ***
  • TACACS = 49
  • DNS = 53
  • BOOTP (DHCP SERVER) = 67
  • BOOTP (DHCP CLIENT) = 68
  • TFTP (Trivial File Transfer Protocol) = 69
  • NETBIOS Name Service = 137
  • NETBIOS Datagram Service = 138

So we get kind of a win with memorizing these with 67-69 being in sequential order, but the others if you don’t work with them will be good to commit to memory. This command is obviously very oriented on reaching server types, and is mainly used for DHCP / TFTP / DNS so I would really burn those port numbers into your mind.

Remember, it is configured on the interface of the incoming broadcasts, and can be verified with “sh IP int s0/0” and it is becoming so clearly important to remember that “sh ip int” and “sh int” give completely different output so I would recommend really committing the output to memory (as with everything else… of course):

R1#sh ip int s0/0
Serial0/0 is up, line protocol is up
  Internet address is 172.12.123.1/24
  Broadcast address is 255.255.255.255
  Address determined by non-volatile memory
  MTU is 1500 bytes
  Helper address is 172.12.123.2
  Directed broadcast forwarding is disabled
  Outgoing access list is not set
  Inbound  access list is not set
  Proxy ARP is enabled
  Local Proxy ARP is disabled
  Security level is default
  Split horizon is disabled
  ICMP redirects are always sent
  ICMP unreachables are always sent
  ICMP mask replies are never sent
  IP fast switching is enabled
  IP fast switching on the same interface is enabled
  IP Flow switching is disabled
  IP CEF switching is enabled
  IP CEF Fast switching turbo vector
  IP multicast fast switching is enabled
  IP multicast distributed fast switching is disabled
  IP route-cache flags are Fast, CEF
  Router Discovery is disabled
  IP output packet accounting is disabled
  IP access violation accounting is disabled
  TCP/IP header compression is disabled
  RTP/IP header compression is disabled
  Policy routing is disabled
  Network address translation is disabled
  BGP Policy Mapping is disabled
  WCCP Redirect outbound is disabled
  WCCP Redirect inbound is disabled
  WCCP Redirect exclude is disabled
R1#

I was going to stop at the helper address, but I wanted to exemplify all the different information you can get from “sh ip int x/x” vs “sh int x/x” so really know the differences for exam day!

So we are unable to add protocols to be forwarded via “ip helper-address” (also note the hyphen is between helper-address), so we will need another command:

R1(config)#ip forward-protocol

“ip forward-protocol” comes to save the day, and I want to walk through the output modifiers for clarity on maybe a couple things:

R1(config)#ip forward-protocol ?
¬† nd¬†¬†¬†¬†¬†¬†¬†¬†¬†¬†¬†¬† Sun’s Network Disk protocol
  sdns           Network Security Protocol
  spanning-tree  Use transparent bridging to flood UDP broadcasts
  turbo-flood    Fast flooding of UDP broadcasts
  udp            Packets to a specific UDP port

R1(config)#ip forward-protocol udp ?
  <0-65535>      Port number
  biff           Biff (mail notification, comsat, 512)
  bootpc         Bootstrap Protocol (BOOTP) client (68)
  bootps         Bootstrap Protocol (BOOTP) server (67)
  discard        Discard (9)
  dnsix          DNSIX security protocol auditing (195)
  domain         Domain Name Service (DNS, 53)
  echo           Echo (7)
  isakmp         Internet Security Association and Key Management Protocol
                 (500)
  mobile-ip      Mobile IP registration (434)
  nameserver     IEN116 name service (obsolete, 42)
  netbios-dgm    NetBios datagram service (138)
  netbios-ns     NetBios name service (137)
  netbios-ss     NetBios session service (139)
  non500-isakmp  Internet Security Association and Key Management Protocol
                 (4500)
  ntp            Network Time Protocol (123)
  pim-auto-rp    PIM Auto-RP (496)
  rip            Routing Information Protocol (router, in.routed, 520)
  snmp           Simple Network Management Protocol (161)
  snmptrap       SNMP Traps (162)

R1(config)#ip forward-protocol udp ntp ?
  <cr>

R1(config)#ip forward-protocol udp ntp
R1(config)#

A few things to note with “ip forward-protocol …” command:

  • It is configured on the global configuration level, not the interface
  • It also defines in the first ? that there is no TCP options available
  • It does not require an IP address to send it to, however it will forward it

Really the main difference that I have found between the two protocols is in terms of vlans, that “ip helper-address” will forward broadcasts onto its same vlan that the interface is part of that it’s configured on, while “ip forward-protocol” will forward to all attached vlans.

So the helper-address is a more focused command, while the forward-protocol is a bit more general in allowing traffic to flow around.

That is actually it for this, I have one more lab on quieting down “sh ip packet” so you can run it on a router without pegging the CPU, but I think it would take voodoo for that to work and then it is on to the massive topic of BGP!

So, I will wrap up the security stuff with that, and then it is going to be some epic labbing from what I understand of BGP – It’s going to be like catching King Kong in a net ūüôā Can’t wait!

Unicast Reverse Path Forwarding (RPF) discussion, configuration, and a LOT of great commands and output examples!

Not to be confused with RPF (Reverse Path Forwarding) which is a multicast feature, Unicast RPF is a Unicast feature, which allows a router to verify an incoming packets source IP address by ensuring it is reachable.

To do this, it uses the FIB (Forwarding Information Base) to perform this check. To accomplish this “ip cef” (Cisco Express Forwarding) must be running, which can be done by issuing the following:

R1(config)#ip cef

Bam, that easy, and verification that it is running is just as easy:
R1(config)#do sh ip cef
Prefix              Next Hop             Interface
0.0.0.0/0           drop                 Null0 (default route handler entry)
0.0.0.0/32          receive
1.1.1.1/32          receive
172.12.15.0/24      attached             FastEthernet0/1
172.12.15.0/32      receive
172.12.15.1/32      receive
172.12.15.255/32    receive
172.12.123.0/24     attached             Serial0/0
172.12.123.0/32     receive
172.12.123.1/32     receive
172.12.123.2/32     172.12.123.2         Serial0/0
172.12.123.3/32     172.12.123.3         Serial0/0
172.12.123.255/32   receive
224.0.0.0/4         drop
224.0.0.0/24        receive
255.255.255.255/32  receive
R1(config)#

Here we see our directly connected routers that R1 knows about, which are the two spokes R2 / R3, as well as R5 directly connected over FastEthernet. An important note is you can tell there is no default route set on the router because 0.0.0.0’s network interface is Null0 (trash can). There is no routing at all configured at this time as my NBMA network acts almost like an MPLS with L3 reachability at Layer 2 as can be seen with a test ping to make sure its rockin:

R1(config)#do ping 172.12.123.2

Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 172.12.123.2, timeout is 2 seconds:
!!!!!
Success rate is 100 percent (5/5), round-trip min/avg/max = 64/65/69 ms
R1(config)#

So once CEF is enabled, there is no more configuration of CEF itself for that for Unicast RFP to work, you just need to configure Unicast RFP to be in one of two modes:

  • Loose mode – Verifies the source IP address of the incoming packet is reachable by any means, it doesn’t matter how
  • Strict mode – Verifies that the source IP address is reachable on the same interface that in came in on – THIS IS THE BIG DIFFERENCE BETWEEN THE TWO!

The command is almost the same, long and complex for both with slight variations for the different modes at the end, and configured on the interface level:

R1(config)#int s0/0
R1(config-if)#ip verify unicast source reachable-via ?
  any  Source is reachable via any interface
  rx   Source is reachable via interface on which packet was received

That is correct, “ip verify unicast source reachable-via …” Easy enough to remember right? As seen above, “any” is for Loose mode, and “rx” is to configure Strict mode.

** One thing to note, a variation of the command on older routers is “ip verify unicast reverse-path to be backwards compatible as seen here, it even shows in IOS help that it is in old command format so it must be ancient:

R1(config-if)#ip verify unicast ?
  reverse-path  Reverse path validation of source address (old command format)
  source        Validation of source address

So back to business, I wanted to see if there were further modifiers, so I ? after both any and rx and this is what I got for either:

R1(config-if)#ip verify unicast source reachable-via rx ?
  <1-199>          A standard IP access list number
  <1300-2699>      A standard IP expanded access list number
  allow-default    Allow default route to match when checking source address
  allow-self-ping  Allow router to ping itself (opens vulnerability in
                   verification)
  <cr>

R1(config-if)#ip verify unicast source reachable-via any ?
  <1-199>          A standard IP access list number
  <1300-2699>      A standard IP expanded access list number
  allow-default    Allow default route to match when checking source address
  allow-self-ping  Allow router to ping itself (opens vulnerability in
                   verification)
  <cr>

So some good things to note from the above output:

  • Only standard ACL’s are being allowed as modifiers
  • There is a <cr> indicating none of the above modifiers must be used
  • There is also something about allowing default routes
  • “allow-self-ping” is marked by the IOS as opening vulnerabilities with verification

Now one gotcha about this command, you have to make sure the return router it will be sending Unicast RPF’s to has a return route, so on that note I will also demonstrate the output of “sh ip cef” after configuring it and a default route on R2:

R2(config)#ip route 0.0.0.0 0.0.0.0 172.12.123.1
R2(config)#ip cef
R2(config)#do show ip cef
Prefix              Next Hop             Interface
0.0.0.0/0           172.12.123.1         Serial0/0
0.0.0.0/32          receive
2.2.2.2/32          receive
172.12.23.0/24      attached             FastEthernet0/0
172.12.23.0/32      receive
172.12.23.2/32      receive
172.12.23.255/32    receive
172.12.123.0/24     attached             Serial0/0
172.12.123.0/32     receive
172.12.123.1/32     172.12.123.1         Serial0/0
172.12.123.2/32     receive
172.12.123.3/32     172.12.123.3         Serial0/0
172.12.123.255/32   receive
224.0.0.0/4         drop
224.0.0.0/24        receive
255.255.255.255/32  receive
R2(config)#

So it does still have a ‘receive’ line in there, however the default route now is defined.

So back to R1 I will actually put the config for verification on S0/0 and demonstrate another verification command that verification is running (ironic?):

R1(config)#int s0/0
R1(config-if)#ip verify unicast source reachable-via any

Now it is configured, and to verify it’s configured, you can use “sh cef int (int)” :
R1(config-if)#do sh cef int s0/0
Serial0/0 is up (if_number 6)
  Corresponding hwidb fast_if_number 6
  Corresponding hwidb firstsw->if_number 6
  Internet address is 172.12.123.1/24
  ICMP redirects are always sent
  Per packet load-sharing is disabled
  IP unicast RPF check is enabled
  Inbound access list is not set
  Outbound access list is not set
  Hardware idb is Serial0/0
  Fast switching type 5, interface type 60
  IP CEF switching enabled
  IP CEF Feature Fast switching turbo vector
  Input fast flags 0x4000, Input fast flags2 0x0, Output fast flags 0x0, Output fast flags2 0x0
  ifindex 3(3)
  Slot 0 Slot unit 0 Unit 0 VC -1
  Transmit limit accumulator 0x0 (0x0)
  IP MTU 1500
R1(config-if)#

Lots of other output, but to verify specifically for Unicast RPF, that is the command to commit to memory… like everything.

One thing that needs it’s own bold red look at me is the “sh int s0/0” vs “sh ip int s0/0” as the initial command will give you more interface configurations and errors, while the latter command shows you the IP services running on the interface like IP CEF, BGP Mapping, and IP verify verification drops! I want to show both here to examplify:

“sh int s0/0”:

R1(config-if)#do sh int s0/0
Serial0/0 is up, line protocol is up
  Hardware is PowerQUICC Serial
  Internet address is 172.12.123.1/24
  MTU 1500 bytes, BW 1544 Kbit, DLY 20000 usec,
     reliability 255/255, txload 1/255, rxload 1/255
  Encapsulation FRAME-RELAY, loopback not set
  Keepalive set (10 sec)
  CRC checking enabled
  LMI enq sent  557, LMI stat recvd 557, LMI upd recvd 0, DTE LMI up
  LMI enq recvd 0, LMI stat sent  0, LMI upd sent  0
  LMI DLCI 1023  LMI type is CISCO  frame relay DTE
  FR SVC disabled, LAPF state down
  Broadcast queue 0/64, broadcasts sent/dropped 156/0, interface broadcasts 79
  Last input 00:00:00, output 00:00:00, output hang never
¬† Last clearing of “show interface” counters 01:34:13
  Input queue: 0/75/0/0 (size/max/drops/flushes); Total output drops: 0
  Queueing strategy: weighted fair
  Output queue: 0/1000/64/0 (size/max total/threshold/drops)
     Conversations  0/1/256 (active/max active/max total)
     Reserved Conversations 0/0 (allocated/max allocated)
     Available Bandwidth 1158 kilobits/sec
  5 minute input rate 0 bits/sec, 0 packets/sec
  5 minute output rate 0 bits/sec, 0 packets/sec
     721 packets input, 59837 bytes, 0 no buffer
     Received 0 broadcasts, 0 runts, 0 giants, 0 throttles
     1 input errors, 0 CRC, 1 frame, 0 overrun, 0 ignored, 0 abort
     720 packets output, 59580 bytes, 0 underruns
     0 output errors, 0 collisions, 2 interface resets
     0 output buffer failures, 0 output buffers swapped out
     4 carrier transitions
     DCD=up  DSR=up  DTR=up  RTS=up  CTS=up
“sh ip int s0/0”:
R1(config-if)#do sh ip int s0/0
Serial0/0 is up, line protocol is up
  Internet address is 172.12.123.1/24
  Broadcast address is 255.255.255.255
  Address determined by setup command
  MTU is 1500 bytes
  Helper address is not set
  Directed broadcast forwarding is disabled
  Outgoing access list is not set
  Inbound  access list is not set
  Proxy ARP is enabled
  Local Proxy ARP is disabled
  Security level is default
  Split horizon is disabled
  ICMP redirects are always sent
  ICMP unreachables are always sent
  ICMP mask replies are never sent
  IP fast switching is enabled
  IP fast switching on the same interface is enabled
  IP Flow switching is disabled
  IP CEF switching is enabled
  IP CEF Feature Fast switching turbo vector
  IP multicast fast switching is enabled
  IP multicast distributed fast switching is disabled
  IP route-cache flags are Fast, CEF
  Router Discovery is disabled
  IP output packet accounting is disabled
  IP access violation accounting is disabled
  TCP/IP header compression is disabled
  RTP/IP header compression is disabled
  Policy routing is disabled
  Network address translation is disabled
  BGP Policy Mapping is disabled
  WCCP Redirect outbound is disabled
  WCCP Redirect inbound is disabled
  WCCP Redirect exclude is disabled
  IP verify source reachable-via ANY
  5 verification drops
  0 suppressed verification drops
R1(config-if)#

As can be seen, the second command lists more IP services that statistics for the interface.

Also on the note of that terribly long output, if you are looking for one piece of info in a monster pile of info, you can put ” | i (word)” at the end of the command to only see lines that include that word as shown here:

R1(config-if)#do sh ip int s0/0 | i verify
  IP verify source reachable-via ANY
R1(config-if)#do sh ip int s0/0 | i verification
  5 verification drops
  0 suppressed verification drops

R1(config-if)#

Notice I got burned there on the first one because that just shows me my verify criteria, but what I was actually looking for was the drop count from R2 because it’s using that default route.

Also to note, that WILL ONLY INCREMENT FROM INCOMING PACKETS ! It is verifying the source of incoming packets, so any outgoing traffic WILL NOT increment the verification drops! That is very important to note as it’s so easy to forget!

One more show command to verify a lot of different things, is “sh ip traffic”, and so you have been warned this output is another monster but a TON of good info:

R1#sh ip traffic
IP statistics:
  Rcvd:  396 total, 391 local destination
         0 format errors, 0 checksum errors, 0 bad hop count
         0 unknown protocol, 0 not a gateway
         0 security failures, 0 bad options, 0 with options
  Opts:  0 end, 0 nop, 0 basic security, 0 loose source route
         0 timestamp, 0 extended security, 0 record route
         0 stream ID, 0 strict source route, 0 alert, 0 cipso, 0 ump
         0 other
¬† Frags: 0 reassembled, 0 timeouts, 0 couldn’t reassemble
¬†¬†¬†¬†¬†¬†¬†¬† 0 fragmented, 0 fragments, 0 couldn’t fragment
  Bcast: 381 received, 317 sent
  Mcast: 0 received, 0 sent
  Sent:  317 generated, 0 forwarded
  Drop:  5 encapsulation failed, 0 unresolved, 0 no adjacency
         0 no route, 5 unicast RPF, 0 forced drop
         0 options denied
  Drop:  0 packets with source IP address zero
  Drop:  0 packets with internal loop back IP address

ICMP statistics:
  Rcvd: 0 format errors, 0 checksum errors, 0 redirects, 0 unreachable
        5 echo, 5 echo reply, 0 mask requests, 0 mask replies, 0 quench
        0 parameter, 0 timestamp, 0 timestamp replies, 0 info request, 0 other
        0 irdp solicitations, 0 irdp advertisements
  Sent: 0 redirects, 0 unreachable, 21 echo, 5 echo reply
        0 mask requests, 0 mask replies, 0 quench, 0 timestamp, 0 timestamp replies
        0 info reply, 0 time exceeded, 0 parameter problem
        0 irdp solicitations, 0 irdp advertisements

TCP statistics:
  Rcvd: 0 total, 0 checksum errors, 0 no port
  Sent: 0 total

BGP statistics:
  Rcvd: 0 total, 0 opens, 0 notifications, 0 updates
        0 keepalives, 0 route-refresh, 0 unrecognized
  Sent: 0 total, 0 opens, 0 notifications, 0 updates
        0 keepalives, 0 route-refresh

IP-EIGRP statistics:
  Rcvd: 0 total
  Sent: 0 total

PIMv2 statistics: Sent/Received
  Total: 0/0, 0 checksum errors, 0 format errors
  Registers: 0/0 (0 non-rp, 0 non-sm-group), Register Stops: 0/0,  Hellos: 0/0
  Join/Prunes: 0/0, Asserts: 0/0, grafts: 0/0
  Bootstraps: 0/0, Candidate_RP_Advertisements: 0/0
  Queue drops: 0
  State-Refresh: 0/0

IGMP statistics: Sent/Received
  Total: 0/0, Format errors: 0/0, Checksum errors: 0/0
  Host Queries: 0/0, Host Reports: 0/0, Host Leaves: 0/0
  DVMRP: 0/0, PIM: 0/0
  Queue drops: 0

UDP statistics:
  Rcvd: 381 total, 0 checksum errors, 381 no port
  Sent: 312 total, 0 forwarded broadcasts

OSPF statistics:
  Rcvd: 0 total, 0 checksum errors
        0 hello, 0 database desc, 0 link state req
        0 link state updates, 0 link state acks

  Sent: 0 total
        0 hello, 0 database desc, 0 link state req
        0 link state updates, 0 link state acks

ARP statistics:
  Rcvd: 0 requests, 0 replies, 92 reverse, 0 other
  Sent: 0 requests, 4 replies (0 proxy), 162 reverse
R1#

It includes a TON of great stats, but lets go a step further with the | (pipe) in the command, because this is a monster output that we can cut down by a lot:

R1# sh ip traffic | i rpf
R1# sh ip traffic | i RPF
         0 no route, 5 unicast RPF, 0 forced drop
R1#

So it is case sensitive because it is looking to match an exact word in the output, so you could also put unicast and probably get some more info along with RPF, but this can also be used for whole sections:

R1#sh ip traffic | ?
  append    Append redirected output to URL (URLs supporting append operation
            only)
  begin     Begin with the line that matches
  exclude   Exclude lines that match
  include   Include lines that match
  redirect  Redirect output to URL
  section   Filter a section of output
  tee       Copy output to URL

R1#sh ip traffic | s OSPF
OSPF statistics:
  Rcvd: 0 total, 0 checksum errors
        0 hello, 0 database desc, 0 link state req
        0 link state updates, 0 link state acks
R1#

So you can use include to match a single word (but it will only show you that single line that includes the word), you can use begin to begin viewing output at your defined criteria, or like I used was section to just see a section that may be relevant to me.

I feel it’s very important to know you can use the | (above enter key called pipe) to dig straight to your info you need both for the exam but also real world network troubleshooting.

One last thing to wrap up, as seen there were 5 drops on there, and that was from testing from R2 because I had to test that verification on incoming packets that then get verify response sent back to the source:

R2# ping 172.12.123.1 source 2.2.2.2

Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 172.12.123.1, timeout is 2 seconds:
Packet sent with a source address of 2.2.2.2
…..

Success rate is 0 percent (0/5)

This fails because by default, a default route is not a valid verification source for Unicast RFP, and was actually in the configuration options just below the ACL modifiers:

R1(config-if)#ip verify unicast source reachable-via any ?
  <1-199>          A standard IP access list number
  <1300-2699>      A standard IP expanded access list number
  allow-default    Allow default route to match when checking source address
  allow-self-ping  Allow router to ping itself (opens vulnerability in
                   verification)
  <cr>

So as best practice, I will remove the entire command and re-apply it with “allow-default” in the command, and give it another go here:

R1(config-if)#no ip verify unicast source reachable-via any
R1(config-if)#ip verify unicast source reachable-via any allow-default
R1(config-if)#
ASR#2
[Resuming connection 2 to r2 … ]

R2(config)#do ping 172.12.123.1 source 2.2.2.2

Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 172.12.123.1, timeout is 2 seconds:
Packet sent with a source address of 2.2.2.2
!!!!!
Success rate is 100 percent (5/5), round-trip min/avg/max = 64/66/69 ms
R2(config)#

And that is about all I have to say about that. I start these posts thinking easy topic, should be a short post, and end up with half a book written. Next up will be some useful security commands, so it will be a shorter post hopefully (famous last words).

NTP Authentication and ACL configuration, odd behaviors explained, and issues to troubleshoot as always!

OSPF_Base_Topology_NTP

So to begin this, I apparently completely spaced writing R1 so nothing got saved, and we are back in time again:

R1#sh clock
*22:38:45.666 UTC Fri Mar 1 2002
R1#

So I’ve decided to go right into Authentication first, as it’s fairly straight forward with an odd behavior to note, then continue my battle with R4 over the virtual-link to see if I can maybe use the “peer” command between R3 and R4 to resolve the issue.

So I’ve removed all NTP settings from all routers involved in the last lab, as I will need to reconfigure them for authentication. Now to configure authentication, it actually only takes 3 commands on the Master / Server, but 4 commands on the clients as seen here:

R1(config)#ntp authenticate
R1(config)#

^This command sets NTP authentication to run

R1(config)#ntp authentication-key ?
  <1-4294967295>  Key number

R1(config)#ntp authentication-key 1 ?
  md5  MD5 authentication

R1(config)#ntp authentication-key 1 md5 ?
  WORD  Authentication key

R1(config)#ntp authentication-key 1 md5 CCNP ?
  <0-4294967295>  Authentication key encryption type
  <cr>

R1(config)#ntp authentication-key 1 md5 CCNP
R1(config)#

^This command is literally one way to type it, straight forward, CCNP is my keys “password” to authenticate to NTP clients. I have no idea what the last value is, so I will just leave it as configured.
R1(config)#ntp trusted-key ?
  <1-4294967295>  Key number

R1(config)#ntp trusted-key 1 ?
  <cr>

R1(config)#ntp trusted-key 1
R1(config)#

^Again just a very straight forward command, just identifying which one of its keys is a trusted key.

And that is it for the server, it is now “offering” authentication for NTP to potential clients, which sounds odd for authentication as it definitely should.

So on R3 I repeat the same thing:

R3(config)#ntp authenticate
R3(config)#ntp authentication-key 1 md5 CCNP
R3(config)#ntp trusted-key 1
R3(config)#ntp server 172.12.123.1 ? <– The 4th command that is required for clients!
  key      Configure peer authentication key
  prefer   Prefer this peer when possible
  source   Interface for source address
  version  Configure NTP version
  <cr>

R3(config)#ntp server 172.12.123.1 key ?
  <0-4294967295>  Peer key number

R3(config)#ntp server 172.12.123.1 key 1 ?
  prefer   Prefer this peer when possible
  source   Interface for source address
  version  Configure NTP version
  <cr>

R3(config)#ntp server 172.12.123.1 key 1
R3(config)#

So really the NTP clients ONLY additional command to get its time from the server in any case is that mysterious 4th command while R1 has 3 (not including the clock set again).

After waiting a few minutes to see R3 populate, I realized two things: 1. I forgot that yesterday R1 had no neighbor statements for OSPF on my Hub for my spoke routers, and 2. I forgot “ntp master 1” on R1.

So really to set up its 4 commands on each if you include the ntp master on the time server, however it is 3 and 4 if you assume that is part of the normal configuration and adding “key 1” to the “ntp server x.x.x.x …” command.

Now that we’ve beat that horse to death, lets see whats happening on R3:

R3(config)#do sh ntp assoc

      address         ref clock     st  when  poll reach  delay  offset    disp
*~172.12.123.1     .LOCL.            1     1    64   17    65.4   -1.01  1877.2
 * master (synced), # master (unsynced), + selected, Рcandidate, ~ configured
R3(config)#do sh clock
18:32:54.346 UTC Tue Mar 21 2017
R3(config)#

Alright so that is now working as expected, and I just reconfigured R4 pointed at 172.12.123.1 because it makes no sense from yesterday that it cannot sync up with R1 (which I will visit shortly), but I also configured R2 with absolutely no authentication commands and shortly after I pointed it at R1 as the Time Server we get this:

R2(config)#do sh ntp assoc

      address         ref clock     st  when  poll reach  delay  offset    disp
*~172.12.123.1     .LOCL.            1     9    64  377    53.7   -0.43     0.3
 * master (synced), # master (unsynced), + selected, Рcandidate, ~ configured
R2(config)#do sh clock
18:39:25.526 UTC Tue Mar 21 2017
R2(config)#

So this is the weird thing with NTP Authentication, is when I say it is “offered” when set by the Master, it is usable without authenticating which sort of defeats the concept of authentication completely – REMEMBER THIS FOR EXAM DAY!

In not so red of text, a client can be configured with no authentication to the time server, and it will still get time from that server (defeating the purpose of authentication).

So I configured R4 with authentication commands and pointed it to R1, and to my surprise:

ASR#4
[Resuming connection 4 to r4 … ]

R4(config)#do sh ntp assoc

address         ref clock       st   when   poll reach  delay  offset   disp
*~172.12.123.1    .LOCL.           1     43     64    77 64.412 -38.023 188.77
* sys.peer, # selected, + candidate, – outlyer, x falseticker, ~ configured
R4(config)#
R4(config)#do sh ntp status
Clock is synchronized, stratum 2, reference is 172.12.123.1
nominal freq is 250.0000 Hz, actual freq is 250.0000 Hz, precision is 2**24
reference time is DC7BF1C5.0072A487 (18:39:01.001 UTC Tue Mar 21 2017)
clock offset is -38.0238 msec, root delay is 64.41 msec
root dispersion is 506.73 msec, peer dispersion is 5.72 msec
loopfilter state is ‘CTRL’ (Normal Controlled Loop), drift is -0.000000202 s/s
system poll interval is 64, last update was 471 sec ago.
R4(config)#

Hooray! No nitty gritty troubleshooting, the lab must know I feel like I am getting sick too!

In the above output of the two “show” verification commands we know of thus far, you see nothing about NTP authentication, but it is all in the “detail” so to say:

R4(config)#do sh ntp assoc detail
172.12.123.1 configured, authenticated, our_master, sane, valid, stratum 1
ref ID .LOCL., time DC7BF403.9FB9E5B6 (18:48:35.623 UTC Tue Mar 21 2017)
our mode client, peer mode server, our poll intvl 64, peer poll intvl 64
root delay 0.00 msec, root disp 0.03, reach 377, sync dist 67.60
delay 64.58 msec, offset -101.6407 msec, dispersion 2.66
precision 2**18, version 4
org time DC7BF405.D9A56AD6 (18:48:37.850 UTC Tue Mar 21 2017)
rec time DC7BF405.FBEF0202 (18:48:37.984 UTC Tue Mar 21 2017)
xmt time DC7BF405.EB34C2F5 (18:48:37.918 UTC Tue Mar 21 2017)
filtdelay =    64.58   64.68   65.65   64.73   64.64   64.76   64.60   64.63
filtoffset = -101.64  -95.55  -87.40  -80.41  -72.88  -65.64  -58.45  -51.39
filterror =     0.00    0.94    1.90    2.88    3.85    4.83    5.80    6.76
minpoll = 6, maxpoll = 10

R4(config)#

I tripped over this message a couple times, because the huge output made me think “sh ntp status” however it is “sh ntp assoc detail” and NOT detail”s”!

So back to authentication not really making hosts authenticate to use the server as a time source, to limit hosts receiving time from our NTP to who we want, Access-Lists’s come to¬† the rescue!

So the creation of the access-list:

R1(config)#access-list 10 permit 172.12.123.0 0.0.0.255
R1(config)#access-list 10 permit 172.12.34.0 0.0.0.255

Pretty simple, allows R2 / R3 / R4 to get time from R1 with of course the implicit deny at the end to deny all other networks, and now for the NTP portion of applying the ACL:

R1(config)#ntp ?
  access-group        Control NTP access
  authenticate        Authenticate time sources
  authentication-key  Authentication key for trusted time sources
  broadcastdelay      Estimated round-trip delay
  clock-period        Length of hardware clock tick
  logging             Enable NTP message logging
  master              Act as NTP master clock
  max-associations    Set maximum number of associations
  peer                Configure NTP peer
  server              Configure NTP server
  source              Configure interface for source address
  trusted-key         Key numbers for trusted time sources

R1(config)#ntp access-group ?
  peer        Provide full access
  query-only  Allow only control queries
  serve       Provide server and query access
  serve-only  Provide only server access

R1(config)#ntp access-group serve ?
  <1-99>       Standard IP access list
  <1300-1999>  Standard IP access list (expanded range)

R1(config)#ntp access-group serve 10

And that officially applies it, so I just reloaded R2 / R3 / R4 to see how they would come back up and get their time, meanwhile I went on the 172.12.23.0 Ethernet segment and set SW1 to also point to 172.12.123.1 as its NTP server and give it some time to sync until my routers reloaded.

Now all routers have reloaded, lets go around the room, and see who has the correct time:

R2#sh clock
18:34:48.760 UTC Wed Mar 22 2017
R2#
ASR#3
[Resuming connection 3 to r3 … ]

R3#sh clock
18:35:04.335 UTC Wed Mar 22 2017
R3#
ASR#4
[Resuming connection 4 to r4 … ]

R4#sh clock
18:35:14.577 UTC Wed Mar 22 2017
R4#
ASR#5
[Resuming connection 5 to sw1 … ]

SW1#sh clock
*01:30:37.361 UTC Mon Mar 1 1993
SW1#sh ntp assoc

      address         ref clock     st  when  poll reach  delay  offset    disp
¬†~172.12.123.1¬†¬†¬†¬† 0.0.0.0¬†¬†¬†¬†¬†¬†¬†¬†¬† 16¬†¬†¬†¬† –¬†¬†¬† 64¬†¬†¬† 0¬†¬†¬†¬† 0.0¬†¬†¬† 0.00¬† 16000.
 * master (synced), # master (unsynced), + selected, Рcandidate, ~ configured

Our switch is existing in probably my favorite era of my lifetime, the 90’s, and will remain there unless we let it back into the present (but that might be cruel) ūüôā

So I wanted to post up the output of SW1’s “sh ntp status” and “sh ntp assoc det” because there is what I found in my voice days hilarious, but serious NTP status:
SW1#sh ntp status
Clock is unsynchronized, stratum 16, no reference clock
nominal freq is 119.2092 Hz, actual freq is 119.2092 Hz, precision is 2**18
reference time is 00000000.00000000 (00:00:00.000 UTC Mon Jan 1 1900)
clock offset is 0.0000 msec, root delay is 0.00 msec
root dispersion is 0.00 msec, peer dispersion is 0.00 msec

Not a whole lot there, except showing the clock isn’t synchronized, but with “sh ntp assoc det” we see a very… odd and awesome way of putting it:
SW1#sh ntp assoc det
172.12.123.1 configured, insane, invalid, unsynced, stratum 16
ref ID 0.0.0.0, time 00000000.00000000 (00:00:00.000 UTC Mon Jan 1 1900)
our mode client, peer mode unspec, our poll intvl 64, peer poll intvl 64
root delay 0.00 msec, root disp 0.00, reach 0, sync dist 0.000
delay 0.00 msec, offset 0.0000 msec, dispersion 16000.00
precision 2**5, version 3
org time 00000000.00000000 (00:00:00.000 UTC Mon Jan 1 1900)
rcv time 00000000.00000000 (00:00:00.000 UTC Mon Jan 1 1900)
xmt time AF3BE5E5.8494D4E8 (01:31:17.517 UTC Mon Mar 1 1993)
filtdelay =     0.00    0.00    0.00    0.00    0.00    0.00    0.00    0.00
filtoffset =    0.00    0.00    0.00    0.00    0.00    0.00    0.00    0.00
filterror =  16000.0 16000.0 16000.0 16000.0 16000.0 16000.0 16000.0 16000.0

SW1#

This switch is INSANE. When I first saw that, it was so awesome, the terminology just tickled me right in my Cisco soft spot. So what is “insane” you might ask (in terms of Cisco switches)? It is when the a network device is configured with an NTP server to get time from, but cannot reach that time source.

So lets see if we can get this switch SANE, and back to the present date. My brain is already fried like chicken so I accidentally exited until I found myself back at “user priv” (square one) mode, so I reconfigured SW1 to point at 172.12.123.1 for NTP and lets see:

SW1#conf t
Enter configuration commands, one per line.  End with CNTL/Z.
SW1(config)#ntp server 172.12.123.1
SW1(config)#do sh clock
*01:46:27.731 UTC Mon Mar 1 1993

Hmm.. that’s been cooking for about 2-3 minutes prior to that output, so it’s time to investigate this, and as I’ve learned with R4 I need to make sure I can ping R1 first:

SW1#ping 172.12.123.1
% Unrecognized host or address, or protocol not running.

Crap. So I forgot I only gave this a name when I brought it online, so to make it able to ping over to R1 I input the following commands:

SW1#conf t
Enter configuration commands, one per line.  End with CNTL/Z.
SW1(config)#ip routing
SW1(config)#int vlan1
SW1(config-if)#ip address 172.12.23.1 255.255.255.0
SW1(config-if)#no shut
SW1(config-if)#exit
01:50:31: %LINK-3-UPDOWN: Interface Vlan1, changed state to up
01:50:32: %LINEPROTO-5-UPDOWN: Line protocol on Interface Vlan1, changed state to up
SW1(config)#exit
SW1#ping 172.12.123.1

Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 172.12.123.1, timeout is 2 seconds:
…..
Success rate is 0 percent (0/5)
SW1#

So we are routing now, but still no dice, I sense an OSPF network not included issue:

R1#sh ip route ospf
     2.0.0.0/32 is subnetted, 1 subnets
O IA    2.2.2.2 [110/65] via 172.12.123.2, 00:30:22, Serial0/0
     3.0.0.0/32 is subnetted, 1 subnets
O IA    3.3.3.3 [110/65] via 172.12.123.3, 00:30:22, Serial0/0
     4.0.0.0/32 is subnetted, 1 subnets
O IA    4.4.4.4 [110/66] via 172.12.123.3, 00:30:22, Serial0/0
     5.0.0.0/32 is subnetted, 1 subnets
O       5.5.5.5 [110/2] via 172.12.15.5, 01:48:21, FastEthernet0/1
     172.12.0.0/24 is subnetted, 4 subnets
O IA    172.12.34.0 [110/65] via 172.12.123.3, 00:30:22, Serial0/0
O IA    172.12.23.0 [110/65] via 172.12.123.3, 00:30:11, Serial0/0
                    [110/65] via 172.12.123.2, 00:30:11, Serial0/0
     44.0.0.0/32 is subnetted, 1 subnets
O IA    44.44.44.1 [110/66] via 172.12.123.3, 00:30:22, Serial0/0
R1#ping 172.12.23.1

Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 172.12.23.1, timeout is 2 seconds:
…..
Success rate is 0 percent (0/5)
R1#sh access-list
Standard IP access list 10
    10 permit 172.12.123.0, wildcard bits 0.0.0.255 (76 matches)
    20 permit 172.12.34.0, wildcard bits 0.0.0.255 (94 matches)
    30 permit 172.12.23.0, wildcard bits 0.0.0.255
R1#

So in the route table it knows of the 172.12.23.0 network via R2, so I hopped on there:
R2#ping 172.12.23.1

Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 172.12.23.1, timeout is 2 seconds:
!!!!!
Success rate is 100 percent (5/5), round-trip min/avg/max = 1/3/4 ms
R2#ping 172.12.123.1

Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 172.12.123.1, timeout is 2 seconds:
!!!!!
Success rate is 100 percent (5/5), round-trip min/avg/max = 64/66/68 ms
R2#
ASR#5
[Resuming connection 5 to sw1 … ]

SW1#ping 172.12.23.2

Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 172.12.23.2, timeout is 2 seconds:
!!!!!
Success rate is 100 percent (5/5), round-trip min/avg/max = 1/2/8 ms
SW1#

So R2 can ping R1 and SW1, and is the go between for them, and SW1 can ping R2 which is its middle man to R1. So I go back to SW1 to give a traceroute a try to see if it’s even getting a response from R2 when sending traffic:

SW1#traceroute 172.12.123.1

Type escape sequence to abort.
Tracing the route to 172.12.123.1

  1  *  *  *
  2  *  *  *
  3  *  *
SW1#sh ip route

Gateway of last resort is not set

     172.12.0.0/24 is subnetted, 1 subnets
C       172.12.23.0 is directly connected, Vlan1

There’s the issue, it has no route to 172.12.123.1, so at this point of going brain dead from work / study I will just create the static route to bring sanity back to this switch:

SW1(config)#ip route 172.12.123.0 255.255.255.0 172.12.23.2
SW1(config)#exit
SW1#ping 172.12.123.1

Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 172.12.123.1, timeout is 2 seconds:
!!!!!

I love when logic works, makes my head hurt less. So I issued a write and reload, and after it booted back up, this is how the clock is now looking:

SW1#sh clock
*00:01:08.501 UTC Mon Mar 1 1993
SW1#sh ntp assoc

      address         ref clock     st  when  poll reach  delay  offset    disp
¬†~172.12.123.1¬†¬†¬†¬† 0.0.0.0¬†¬†¬†¬†¬†¬†¬†¬†¬† 16¬†¬†¬†¬† –¬†¬†¬† 64¬†¬†¬† 0¬†¬†¬†¬† 0.0¬†¬†¬† 0.00¬† 16000.
 * master (synced), # master (unsynced), + selected, Рcandidate, ~ configured
SW1#

So starting to lose my mind wondering why on Earth this thing will not sync, I got on R1 and ran “debug ntp packet” and I’ll spare you all the output EXCEPT WHEN SW1 FINALLY HIT THIS SUCKER AND GOT BROUGHT BACK FROM THE FUTURE… OR PAST… WHICHEVER:

.Mar 22 19:16:31.578: NTP: rcv packet from 172.12.23.1 to 172.12.123.1 on Serial0/0:
.Mar 22 19:16:31.578:  leap 3, mode 3, version 3, stratum 0, ppoll 64
.Mar 22 19:16:31.578:  rtdel 0000 (0.000), rtdsp 10001 (1000.015), refid 00000000 (0.0.0.0)
.Mar 22 19:16:31.578:  ref 00000000.00000000 (00:00:00.000 UTC Mon Jan 1 1900)
.Mar 22 19:16:31.578:  org 00000000.00000000 (00:00:00.000 UTC Mon Jan 1 1900)
.Mar 22 19:16:31.582:  rec 000
R1#00000.00000000 (00:00:00.000 UTC Mon Jan 1 1900)
.Mar 22 19:16:31.582:  xmt AF3BD148.1181126D (00:03:20.068 UTC Mon Mar 1 1993)
.Mar 22 19:16:31.582:  inp DC7D4C0F.945A1DF3 (19:16:31.579 UTC Wed Mar 22 2017)
.Mar 22 19:16:31.582: NTP: stateless xmit packet to 172.12.23.1:
.Mar 22 19:16:31.582:  leap 3, mode 4, version 3, stratum 0, ppoll 64
.Mar 22 19:16:31.582:  rtdel 0000 (0.000), rtdsp 6002 (375.031), refid 4C4F434C (76.79.67.76)
.Mar 22 19:16:31.586:  ref DC7D400B.EEB63084 (18:25:15.932 UTC Wed Mar 22 2017)
.Mar 22 19:16:31.586:  org AF3BD148.1181126D (00:03:20.068 UTC Mon Mar 1 1993)
.Mar 22 19:16:31.586:  rec DC7D4C0F.945A1DF3 (19:16:31.579 UTC Wed Mar 22 2017)
.Mar 22 19:16:31.586:  xmt DC7D4C0F.95A82567 (19:16:31.584 UTC Wed Mar 22 2017)
R1#u all
R1#

I left the big crap ton of output to see the exchange and references, I am not sure what most of it means, but it does show the switches old time and the current time being exchanged (as well as referring to 1900 for some odd reason).

So when we go back to SW1, we should finally have some sanity, before I lose mine:

SW1#sh clock
*00:10:24.859 UTC Mon Mar 1 1993
SW1#sh ntp assoc det
172.12.123.1 configured, insane, invalid, unsynced, stratum 16
ref ID 0.0.0.0, time 00000000.00000000 (00:00:00.000 UTC Mon Jan 1 1900)
our mode client, peer mode unspec, our poll intvl 64, peer poll intvl 64
root delay 0.00 msec, root disp 0.00, reach 0, sync dist 0.000
delay 0.00 msec, offset 0.0000 msec, dispersion 16000.00
precision 2**5, version 3
org time DC7D4D8F.933D330C (19:22:55.575 UTC Wed Mar 22 2017)
rcv time AF3BD2C8.1E2EB9CE (00:09:44.117 UTC Mon Mar 1 1993)
xmt time AF3BD2C8.10B78FA5 (00:09:44.065 UTC Mon Mar 1 1993)
filtdelay =     0.00    0.00    0.00    0.00    0.00    0.00    0.00    0.00
filtoffset =    0.00    0.00    0.00    0.00    0.00    0.00    0.00    0.00
filterror =  16000.0 16000.0 16000.0 16000.0 16000.0 16000.0 16000.0 16000.0

SW1#

What??? If you examine the details, other than insanity still being awesome terminology, you can see what I’ve highlighted in red that it does have the correct network time in the output which its getting from R1, but show clock does not show the correct time.

So at this point, I did try to put “router ospf 1” on the switch which it does show as a valid command, but it does not drop me into ospf configuration mode to see if that gives it the kick it needs. You can even see it getting hits on R1’s ACL for NTP requests:

R1#sh access-list
Standard IP access list 10
    10 permit 172.12.123.0, wildcard bits 0.0.0.255 (130 matches)
    20 permit 172.12.34.0, wildcard bits 0.0.0.255 (105 matches)
    30 permit 172.12.23.0, wildcard bits 0.0.0.255 (15 matches)
R1#

So at this point, the next segment of my course is running NTP in broadcast mode, and I want to see what that has to say there to see if we can maybe salvage this SW1 situation.

NTP – Network Time Protocol discussion, configuration, and some NTP / routing troubleshooting as usual!

OSPF_Base_Topology_NTP

For this lab I will be using this OSPF Topology, which has the virtual-link to R4 to bring it into the entirety of the network, and to demonstrate how to configure each of these routers so they keep in sync with the entire network via NTP even if their primary time source goes down.

So this is a subject near and dear to my heart, working previously a lot with Cisco Voice systems / servers, if the network NTP is not near perfectly synchronized your phones / voicemails / servers are going to go absolutely haywire.

In the CCNP ROUTE context, I believe it is mostly beneficial for troubleshooting so your logging is on the same time, so you don’t have to compare different time frames to guess where the event happened on different devices because they had different times – I know this because network time on Small / Medium sized businesses can be off an amazing amount of time.

To begin any NTP discussion you have to first begin with “Stratum” and what it is. Stratum is the metric (like hop count) as to how close to a Stratum 0 device you are to gauge how accurate your time is, this again is especially important for VOIP as I believe only Stratum 3 or lower is acceptable for time differential.

So with Stratum, the lower the better, the best obviously being Stratum 0 which are actually referred to as atomic clocks that are the size of datacenters on naval bases. You will not be able to connect directly to a stratum 0 device (or make a router stratum 0 as you will see), but there are “Time Servers” on the internet that are Stratum 1, that you can point your edge device to (or multiple of them in case one disappears).

Each hop you get away from that Time Server, the more your “Stratum #” will increase when you are running show commands for NTP on your router, and goes up to a maximum of 15 which means as unreliable as it gets before Stratum 16 which means unreachable or unreliable. Now for a couple important notes before we dive into some configuration:

  • When you “write erase” / “reload” a device to wipe it, you are wiping the time, so in the real world or in your lab don’t forget it needs to be reset or chaos will ensue!
  • NTP uses UDP port 123, so do block it on the devices on your network

Also worth a bullet point style explanation, are the 3 different types of NTP router:

  • NTP Server – Set time on this device, it will send out Time Sync messages to NTP clients on the network
  • NTP Client – Receives Time Sync messages from Server, DOES NOT send time sync messages back to server
  • NTP Peer – Can be both Client and Peer, Peers can share time with eachother

NTP can be run in broadcast mode, or multicast mode, depending on your network needs. Its odd that this part of the topic is kind of just left at you needing to figure out what works with your network best, so I imagine trying to lab it over an NBMA running OSPF should be fun!

Also a note on configuring an NTP Server or “Master” as your edge device, it is highly recommended that you not only use a public time server(s) as your time source as opposed to setting it, but also it is necessary to use authentication and / or ACL’s to stop other routers from using ours as an NTP time source even if it is just for time synching reasons we don’t want the extra workload on the edge device.

Ahhhh, the perfect segway into the security side of NTP ūüôā However for all this discussion, there has been no labbing, and not labbing bores me to tears so lets get into some configuration and see what we can break:

R1#sh ntp assoc
R1#sh clock
*23:21:30.839 UTC Fri Mar 1 2002
R1#

As show with “sh ntp associations” we have nothing configured with any other routers (including this one), and we have traveled back 15 years to 2002. So given that my lab is connected to the internet, I am going to set R1 as the NTP Server / Master, and all other routers as clients / peers.

So first, we need to get R1 rocking as the NTP Master of the network, so lets get that done:

(I actually forgot how to set the time and referred back to my time-based ACL’s post, and that’s why I say it over and over it’s so important to start your own blog or something equal in being able to refer back to examples of these things quickly!!)

R1#clock set 19:43:00 20 mar 2017
R1#
*Mar 20 19:43:00.000: %SYS-6-CLOCKUPDATE: System clock has been updated from 23:30:06 UTC Fri Mar 1 2002 to 19:43:00 UTC Mon Mar 20 2017, configured from console by console.
R1#conf t
Enter configuration commands, one per line.  End with CNTL/Z.
R1(config)#ntp ?
  access-group        Control NTP access
  authenticate        Authenticate time sources
  authentication-key  Authentication key for trusted time sources
  broadcastdelay      Estimated round-trip delay
  clock-period        Length of hardware clock tick
  logging             Enable NTP message logging
  master              Act as NTP master clock
  max-associations    Set maximum number of associations
  peer                Configure NTP peer
  server              Configure NTP server
  source              Configure interface for source address
  trusted-key         Key numbers for trusted time sources

R1(config)#ntp master ?
  <1-15>  Stratum number
  <cr>

R1(config)#ntp master 1 ?
  <cr>

R1(config)#ntp master
R1(config)#

A couple things here, I left the ? output after NTP so you can see the command modifiers of which I used master, but as I went on also that my options were 1-15 from most trusted to least. I was going to make it Stratum 1 but I left off the # as I curious what Stratum # it gets by default when configured as an NTP master:

R1(config)#do sh ntp assoc

      address         ref clock     st  when  poll reach  delay  offset    disp
*~127.127.7.1      127.127.7.1       7    51    64  377     0.0    0.00     0.0
 * master (synced), # master (unsynced), + selected, Рcandidate, ~ configured
R1(config)#

Ha, so this router isn’t giving me too much credibility, under “st” which refers to Stratum # is 7, so I am neither very trusted or very unreliable. I’ll switch it back so that it is a Stratum 1, because I like my NTP master of my network to have some credibility.

Also what I have highlighted in red is very important, especially the * when it comes to clients, because that means they are fully synced with that address. Being this is the master it uses its loopback 127.127.7.1, so if you see * in “sh ntp assoc” with a loopback address you know that router is set as the NTP Master.

Now lets configure R2 and R3 over our NBMA and our Area 0 in OSPF as clients of R1:

R2#conf t
Enter configuration commands, one per line.  End with CNTL/Z.
R2(config)#ntp ?
  access-group        Control NTP access
  authenticate        Authenticate time sources
  authentication-key  Authentication key for trusted time sources
  broadcastdelay      Estimated round-trip delay
  clock-period        Length of hardware clock tick
  logging             Enable NTP message logging
  master              Act as NTP master clock
  max-associations    Set maximum number of associations
  peer                Configure NTP peer
  server              Configure NTP server
  source              Configure interface for source address
  trusted-key         Key numbers for trusted time sources

R2(config)#ntp server ?
  Hostname or A.B.C.D  IP address of peer
  vrf                  VPN Routing/Forwarding Information

R2(config)#ntp server 172.12.123.1 ?
  key      Configure peer authentication key
  prefer   Prefer this peer when possible
  source   Interface for source address
  version  Configure NTP version
  <cr>

R2(config)#ntp server 172.12.123.1
R2(config)#
ASR#3
[Resuming connection 3 to r3 … ]

R3#conf t
Enter configuration commands, one per line.  End with CNTL/Z.
R3(config)#ntp server 172.12.123.1
R3(config)#do sh ntp assoc

      address         ref clock     st  when  poll reach  delay  offset    disp
¬†~172.12.123.1¬†¬†¬†¬† 0.0.0.0¬†¬†¬†¬†¬†¬†¬†¬†¬† 16¬†¬†¬†¬† –¬†¬†¬† 64¬†¬†¬† 0¬†¬†¬†¬† 0.0¬†¬†¬† 0.00¬† 16000.
 * master (synced), # master (unsynced), + selected, Рcandidate, ~ configured
R3(config)#

I’ve highlighted the “prefer” option after pointing at 172.12.123.1, to show that it is there, and is used when assigning multiple NTP servers as backup but you prefer to use a specific server as the time source.

Also highlighted in red, before it could sync I did a quick “sh ntp assoc” to demonstrate what it looks like when a router is not synced, and why I stressed seeing the * next to the IP address means that it is fully synced.

Also, that Stratum 16 is the equivalent to RIP’s metric of 16, it’s not even barely reliable but is actually an invalid time source at Stratum 16 – This is important to note!

So let’s see if we have some synchronization going on with R2 and R3:

R2(config)#do sh ntp assoc

      address         ref clock     st  when  poll reach  delay  offset    disp
*~172.12.123.1     .LOCL.            1     1    64  377    53.5   -1.53     0.4
 * master (synced), # master (unsynced), + selected, Рcandidate, ~ configured
R2(config)#do sh clock
20:03:17.432 UTC Mon Mar 20 2017
R2(config)#
ASR#3
[Resuming connection 3 to r3 … ]

R3(config)#do sh ntp assoc

      address         ref clock     st  when  poll reach  delay  offset    disp
*~172.12.123.1     .LOCL.            1    18    64  377    53.5   -3.38     0.5
 * master (synced), # master (unsynced), + selected, Рcandidate, ~ configured
R3(config)#do sh clock
20:03:39.637 UTC Mon Mar 20 2017
R3(config)#

Both are looking good, except I do see it references this again as UTC time, and for lab purposes I do not intend to spend time digging into getting the time into my timezone ūüôā

Another important command to know for checking NTP settings on the local router is “sh ntp status” as demonstrated here:

R3#sh ntp status
Clock is synchronized, stratum 2, reference is 172.12.123.1
nominal freq is 249.5901 Hz, actual freq is 249.5903 Hz, precision is 2**18
reference time is DC7AB544.FFF63BFD (20:08:36.999 UTC Mon Mar 20 2017)
clock offset is -3.8317 msec, root delay is 53.48 msec
root dispersion is 4.06 msec, peer dispersion is 0.18 msec
R3#

So we see it’s synchronized now back to the server from R2 and R3 which right now is just using the Client / Server model, however I had to have my first Derp of the night – Configuring R4 correctly for NTP however not testing connectivity to the server address:

R4(config)#do sh ntp assoc

  address         ref clock       st   when   poll reach  delay  offset   disp
¬†~172.12.123.1¬†¬†¬† .INIT.¬†¬†¬†¬†¬†¬†¬†¬†¬† 16¬†¬†¬†¬†¬† –¬†¬†¬†¬† 64¬†¬†¬†¬† 0¬† 0.000¬†¬† 0.000 15937.
 * sys.peer, # selected, + candidate, Рoutlyer, x falseticker, ~ configured
R4(config)#do ping 172.12.123.1
Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 172.12.123.1, timeout is 2 seconds:
…..
Success rate is 0 percent (0/5)
R4(config)#

The sad part is I’ve been waiting for about 5 minutes or so for that to sync, because I know it can take 5+ minutes, but a ping even before ntp configuration would have been a good way to start the configuration! So lets take a look at R3:

R3#sh ip int bri
Interface                  IP-Address      OK? Method Status                Protocol
FastEthernet0/0            172.12.23.3     YES NVRAM  up                    up
FastEthernet0/1            172.12.34.3     YES NVRAM  up                    up
Serial0/2                  172.12.123.3    YES NVRAM  up                    up
Serial0/3                  unassigned      YES NVRAM  administratively down down
Loopback3                  3.3.3.3         YES NVRAM  up                    up
R3#ping 4.4.4.4

Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 4.4.4.4, timeout is 2 seconds:
!!!!!
Success rate is 100 percent (5/5), round-trip min/avg/max = 1/2/4 ms
R3#sh ip ospf nei

Neighbor ID     Pri   State           Dead Time   Address         Interface
44.44.44.1¬†¬†¬†¬†¬†¬†¬† 0¬†¬† FULL/¬† –¬†¬†¬†¬†¬†¬†¬†¬†¬†¬† –¬†¬†¬†¬†¬†¬†¬† 172.12.34.4¬†¬†¬†¬† OSPF_VL0
2.2.2.2           1   FULL/BDR        00:00:38    172.12.23.2     FastEthernet0/0
44.44.44.1        1   FULL/DR         00:00:38    172.12.34.4     FastEthernet0/1
R3#sh ip proto
Routing Protocol is “ospf 1”
  Outgoing update filter list for all interfaces is not set
  Incoming update filter list for all interfaces is not set
  Router ID 3.3.3.3
  It is an area border router
  Number of areas in this router is 4. 4 normal 0 stub 0 nssa
  Maximum path: 4
  Routing for Networks:
    3.3.3.3 0.0.0.0 area 3
    172.12.23.0 0.0.0.255 area 23
    172.12.34.0 0.0.0.255 area 34
    172.12.123.0 0.0.0.255 area 0
 Reference bandwidth unit is 100 mbps
  Routing Information Sources:
    Gateway         Distance      Last Update
    44.44.44.1           110      01:58:30
  Distance: (default is 110)

R3#sh ip ospf virtual-link
Virtual Link OSPF_VL0 to router 44.44.44.1 is up
  Run as demand circuit
  DoNotAge LSA allowed.
  Transit area 34, via interface FastEthernet0/1, Cost of using 1
  Transmit Delay is 1 sec, State POINT_TO_POINT,
  Timer intervals configured, Hello 10, Dead 40, Wait 40, Retransmit 5
    Hello due in 00:00:03
    Adjacency State FULL (Hello suppressed)
    Index 1/2, retransmission queue length 0, number of retransmission 0
    First 0x0(0)/0x0(0) Next 0x0(0)/0x0(0)
    Last retransmission scan length is 0, maximum is 0
    Last retransmission scan time is 0 msec, maximum is 0 msec
R3#ping 172.12.123.1

Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 172.12.123.1, timeout is 2 seconds:
!!!!!
Success rate is 100 percent (5/5), round-trip min/avg/max = 64/66/68 ms
R3#

Uhhhh…. So R4 has some issue? It should have been ruled out by R3 pinging it’s loopback, but I’ll play ball, lets take a look:

R4#traceroute 172.12.123.1
Type escape sequence to abort.
Tracing the route to 172.12.123.1
VRF info: (vrf in name/id, vrf out name/id)
  1 172.12.34.3 4 msec 0 msec 4 msec
  2  *  *  *
  3  *  *  *
  4  *  *
R4#
R4#sh ip route ospf

Gateway of last resort is not set

      3.0.0.0/32 is subnetted, 1 subnets
O IA     3.3.3.3 [110/2] via 172.12.34.3, 02:04:27, FastEthernet0/1
      172.12.0.0/16 is variably subnetted, 4 subnets, 2 masks
O IA     172.12.23.0/24 [110/2] via 172.12.34.3, 02:03:49, FastEthernet0/1
O        172.12.123.0/24 [110/65] via 172.12.34.3, 02:04:27, FastEthernet0/1
R4#
R4#ping 172.12.123.3
Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 172.12.123.3, timeout is 2 seconds:
!!!!!
Success rate is 100 percent (5/5), round-trip min/avg/max = 1/2/4 ms
R4#ping 172.12.123.1
Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 172.12.123.1, timeout is 2 seconds:
…..
Success rate is 0 percent (0/5)
R4#

So it is being processed by R3, as it can ping the serial interface on the NBMA network, so the issue is either somewhere on the frame switch or on R1. It’s too late and I’m too fried to really want to dig into this weird behavior (I will just wipe / reconfigure them if push comes to shove), but for the heck of it lets look at R1:

!
interface Serial0/1
 ip address 172.12.13.1 255.255.255.252
 clock rate 2000000
!
router ospf 1
 log-adjacency-changes
¬†area 34 virtual-link 44.44.44.1 <— What???
 network 1.1.1.1 0.0.0.0 area 1
 network 172.12.15.0 0.0.0.255 area 15
 network 172.12.123.0 0.0.0.255 area 0
!
!

I am wondering if I was so tired configuring this, that I entered that command on R1, and that is what is jamming up the traffic, lets put a stop to this silliness:

R1#conf t
Enter configuration commands, one per line.  End with CNTL/Z.
R1(config)#router ospf 1
R1(config-router)#no area 34 virtual-link 44.44.44.1
R1(config-router)#

I can confirm it is now gone, lets see about some pings:

R1(config-router)#do ping 4.4.4.4

Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 4.4.4.4, timeout is 2 seconds:
…..
Success rate is 0 percent (0/5)
R1(config-router)#do ping 172.12.34.4

Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 172.12.34.4, timeout is 2 seconds:
…..
Success rate is 0 percent (0/5)
R1(config-router)#do ping 172.12.34.3

Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 172.12.34.3, timeout is 2 seconds:
…..
Success rate is 0 percent (0/5)
R1(config-router)#do ping 172.12.123.3

Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 172.12.123.3, timeout is 2 seconds:
!!!!!
Success rate is 100 percent (5/5), round-trip min/avg/max = 64/66/68 ms
R1(config-router)#

So R3 will process packets from R4 to the IP on its NBMA serial Interface, but R1 cannot ping to the Fa0/1 interface of Area 34. Let me stare at the show run for a moment, I am losing my sense of humor now ūüôā

Wow, I must have been half asleep, I found some leftover access-lists from when I was doing that section on R3 that was messing with traffic, let me remove these AND LET THE TRAFFIC FLOWETH THROUGHOUT THE NETWORK!!! :

R3#conf t
Enter configuration commands, one per line.  End with CNTL/Z.
R3(config)#no access-list 15
R3(config)#no access-list 111
R3(config)#int fa0/1
R3(config-if)#no ip access-group 111 in
R3(config-if)#no ip access-group 15 out
R3(config-if)#
ASR#4
[Resuming connection 4 to r4 … ]

R4#ping 172.12.123.1
Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 172.12.123.1, timeout is 2 seconds:
…..
Success rate is 0 percent (0/5)

GAH!!!

So at this point, I’m giving R3 a write / reload, and if R4 cannot ping the server, I’ll have to wipe and confirm cabling / reconfigure the network as the previously labs are having impact somewhere in the network.

So as R3 loaded back up, and I watched the Adjacency form, I never saw the relationship back to Area 0 on the NBMA network form (I even waited the 10 years it takes over serial links):

R3>en
Password:
R3#
Mar 20 21:16:43.574: %OSPF-5-ADJCHG: Process 1, Nbr 44.44.44.1 on FastEthernet0/1 from LOADING to FULL, Loading Done
R3#
Mar 20 21:16:46.915: %OSPF-5-ADJCHG: Process 1, Nbr 2.2.2.2 on FastEthernet0/0 from LOADING to FULL, Loading Done
R3#
Mar 20 21:16:58.655: %OSPF-5-ADJCHG: Process 1, Nbr 44.44.44.1 on OSPF_VL0 from LOADING to FULL, Loading Done
R3#sh ip ospf nei

Neighbor ID     Pri   State           Dead Time   Address         Interface
44.44.44.1¬†¬†¬†¬†¬†¬†¬† 0¬†¬† FULL/¬† –¬†¬†¬†¬†¬†¬†¬†¬†¬†¬† –¬†¬†¬†¬†¬†¬†¬† 172.12.34.4¬†¬†¬†¬† OSPF_VL0
2.2.2.2           1   FULL/DR         00:00:37    172.12.23.2     FastEthernet0/0
44.44.44.1        1   FULL/DR         00:00:37    172.12.34.4     FastEthernet0/1
R3#

So now it is an OSPF issue that R4 is not getting it’s time, which raises another good point, that time servers should not be reliant (if possible) on a dynamic protocol to reach it’s time source.

So for the hell of it since its this late and I’m already this friend, I might as well try to see this though to the gruesome end, so I took at what routes it DOES see and this is what I got:

R1#sh ip route

Gateway of last resort is not set

     1.0.0.0/32 is subnetted, 1 subnets
C       1.1.1.1 is directly connected, Loopback1
     5.0.0.0/32 is subnetted, 1 subnets
O       5.5.5.5 [110/2] via 172.12.15.5, 00:28:51, FastEthernet0/1
     172.12.0.0/24 is subnetted, 2 subnets
C       172.12.15.0 is directly connected, FastEthernet0/1
C       172.12.123.0 is directly connected, Serial0/0
R1#

Nothing over my NBMA. OMG I forgot the neighbor statements on R1 in my tired stupor, *slams head against desk* :

R1(config)#router ospf 1
R1(config-router)#neighbor 172.12.123.2
R1(config-router)#neighbor 172.12.123.3
R1(config-router)#
.Mar 20 21:31:26.994: %OSPF-5-ADJCHG: Process 1, Nbr 2.2.2.2 on Serial0/0 from LOADING to FULL, Loading Done
.Mar 20 21:31:27.102: %OSPF-5-ADJCHG: Process 1, Nbr 3.3.3.3 on Serial0/0 from LOADING to FULL, Loading Done
R1(config-router)#
ASR#4
[Resuming connection 4 to r4 … ]

*Mar 21 01:56:05.859: %OSPF-5-ADJCHG: Process 1, Nbr 3.3.3.3 on OSPF_VL0 from FULL to DOWN, Neighbor Down: Interface down or detached
R4#
*Mar 21 01:56:12.583: %LINEPROTO-5-UPDOWN: Line protocol on Interface FastEthernet0/1, changed state to down
*Mar 21 01:56:12.583: %OSPF-5-ADJCHG: Process 1, Nbr 3.3.3.3 on FastEthernet0/1 from FULL to DOWN, Neighbor Down: Interface down or detached
R4#ping 172.12.123.1
Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 172.12.123.1, timeout is 2 seconds:
!!!!!
Success rate is 100 percent (5/5), round-trip min/avg/max = 64/65/68 ms
R4#sh ntp assoc

  address         ref clock       st   when   poll reach  delay  offset   disp
¬†~172.12.123.1¬†¬†¬† .INIT.¬†¬†¬†¬†¬†¬†¬†¬†¬† 16¬†¬†¬†¬†¬† –¬†¬† 1024¬†¬†¬†¬† 0¬† 0.000¬†¬† 0.000 15937.
 * sys.peer, # selected, + candidate, Рoutlyer, x falseticker, ~ configured
R4#sh ntp assoc

  address         ref clock       st   when   poll reach  delay  offset   disp
¬†~172.12.123.1¬†¬†¬† .INIT.¬†¬†¬†¬†¬†¬†¬†¬†¬† 16¬†¬†¬†¬†¬† –¬†¬† 1024¬†¬†¬†¬† 0¬† 0.000¬†¬† 0.000 15937.
 * sys.peer, # selected, + candidate, Рoutlyer, x falseticker, ~ configured

So it is just not synching for whatever reason I don’t care to dig in now that its 10pm, so I will continue this battle in my next post, which will include NTP Authentication!

 

SNMP (Simple Network Management Protocol) Fundamentals, very important configuration notes for exam day!

I will keep this brief and right to the points of SNMP, as it is St. Patricks Day night, and I really don’t have any other better plans… I’m just tired from work and its about nap time.

That being said, SNMP is a “polling” protocol used to carry network information traffic on UDP port 161, between an SNMP Manager and SNMP Agents., speaking of which the 3 main components of SNMP are:

  • The SNMP Manager
  • The SNMP Agents
  • The SNMP MIB (Management Information Base)

The MIB is a database that resides on agents, which contains “variables” about the agent, which we will get to variables in a moment, but first lets see the two types of traffic sent from the Manager to its SNMP Agents:

  • GET = A request for some form of information
  • SET = A request to request a certain variable be set to the value indicated in the SET

So from my understanding of this, the Manager defines the information it wants each individual Agent to retain based on what is SET for that Agent, so that it can send a GET at any time to request the defined SET information.

As mentioned above the “polling” for events at set times, which would result in extremely slow notification time if an event were to happen if the polling is not happening every 5 seconds, and if it is polling every 5 seconds the network is going to take a big hit constantly on bandwidth and hardware resources (particularly the Managers).

To get a quick notification to the Manager without overloading it, is to set SNMP Traps on managed devices, which allows the Agent to generate SNMP traffic to send to the Manager if a critical variable changes between GET’s.

On a change of topic, there are 3 different flavors of SNMP currently in use: 1, 2c, and 3.

Versions 1 and 2c DO NOT have authentication and encryption, whereas 3 does, so there is some major security flaws with running them rather than v3 on your network.

If v1 or v2c are used, they should be using something called “SNMP Community strings” which are is a combination of password and authority level, and allows to choose whether its read-only or read-write access.

Now brace yourself for some huge output as I want the modifiers to be seen here, but I will highlight in red what I am using for my input on this router:

R5(config)#snmp-server ?
  chassis-id        String to uniquely identify this chassis
  community         Enable SNMP; set community string and access privs
  contact           Text for mib object sysContact
  context           Create/Delete a context apart from default
  drop              Silently drop SNMP packets
  enable            Enable SNMP Traps
  engineID          Configure a local or remote SNMPv3 engineID
  file-transfer     File transfer related commands
  group             Define a User Security Model group
  host              Specify hosts to receive SNMP notifications
  ifindex           Enable ifindex persistence
  inform            Configure SNMP Informs options
  ip                IP ToS configuration for SNMP traffic
  location          Text for mib object sysLocation
  manager           Modify SNMP manager parameters
  packetsize        Largest SNMP packet size
  queue-length      Message queue length for each TRAP host
  queue-limit       Message queue size for different queues
  source-interface  Assign an source interface
  system-shutdown   Enable use of the SNMP reload command
  tftp-server-list  Limit TFTP servers used via SNMP
  trap              SNMP trap options
  trap-source       Assign an interface for the source address of all traps
  trap-timeout      Set timeout for TRAP message retransmissions
  user              Define a user who can access the SNMP engine
  view              Define an SNMP MIB view

R5(config)#snmp-server community ?
  WORD  SNMP community string

R5(config)#snmp-server community CCNP ?
  <1-99>       Std IP accesslist allowing access with this community string
  <1300-1999>  Expanded IP accesslist allowing access with this community
               string
  WORD         Access-list name
  ipv6         Specify IPv6 Named Access-List
  ro           Read-only access with this community string
  rw           Read-write access with this community string
  view         Restrict this community to a named MIB view
  <cr>

R5(config)#snmp-server community CCNP ro ?
  <1-99>       Std IP accesslist allowing access with this community string
  <1300-1999>  Expanded IP accesslist allowing access with this community
               string
  WORD         Access-list name
  ipv6         Specify IPv6 Named Access-List
  <cr>

R5(config)#snmp-server community CCNP ro 15 ?
  <cr>

R5(config)#snmp-server community CCNP ro 15
R5(config)#

So the above command will allow hosts defined in ACL 15 as permitted to have read-only access to SNMP objects specified by the community string.

With SNMP3, it is more secure, but it is more complex to configure and the commands are a bit more long winded to configure it:

R5(config)#snmp-server group CCNP v3 ?
  auth    group using the authNoPriv Security Level
  noauth  group using the noAuthNoPriv Security Level
  priv    group using SNMPv3 authPriv security level

R5(config)#snmp-server group CCNP v3

With the above output, it’s really about breaking down each line in it entirety, individually:

  1. ¬†auth –¬† group using the authNoPriv Security Level – As Priv is referring to Privacy or Encryption, it is referring to this option offering Authentication, but no encryption
  2. noauth –¬†group using the noAuthNoPriv Security Level – Really flushing security down the toilet with this option, no Authentication and no Encryption
  3. priv –¬†group using SNMPv3 authPriv security level – As discussed in the beginning of this post, and indicated by authPriv, we do have Authentication and Encryption with this option using SNMPv3

Notice there wasn’t a <cr> there, so lets look at the continuation of the output of ? after that command to see what the options are:

R5(config)#snmp-server group CCNP v3 priv ?
  access   specify an access-list associated with this group
  context  specify a context to associate these views for the group
  match    context name match criteria
  notify   specify a notify view for the group
  read     specify a read view for the group
  write    specify a write view for the group
  <cr>

Configuration is beyond the scope of the exam, but some key notes to try to keep in the back of your mind regarding SNMPv3 options above:

  • If no read view is defined, all objects can be read
  • If no write view is defined, no objects can be written
  • If no notify view is defined, group members are not sent notifications

Speaking of users, I’ll create a user here use SHA for Auth and AES 128-bit encryption, and as a warning this is going to be a LOT of output as I show the ? modifiers:

R5(config)#snmp-server user Dave ?
  WORD  Group to which the user belongs

R5(config)#snmp-server user Dave CCNP ?
  remote  Specify a remote SNMP entity to which the user belongs
  v1      user using the v1 security model
  v2c     user using the v2c security model
  v3      user using the v3 security model

R5(config)#snmp-server user Dave CCNP v3 ?
  access     specify an access-list associated with this group
  auth       authentication parameters for the user
  encrypted  specifying passwords as MD5 or SHA digests
  <cr>

R5(config)#snmp-server user Dave CCNP v3 auth ?
  md5  Use HMAC MD5 algorithm for authentication
  sha  Use HMAC SHA algorithm for authentication

R5(config)#snmp-server user Dave CCNP v3 auth sha ?
  WORD  authentication pasword for user

R5(config)#snmp-server user Dave CCNP v3 auth sha CCIE ?
  access  specify an access-list associated with this group
  priv    encryption parameters for the user
  <cr>

R5(config)#snmp-server user Dave CCNP v3 auth sha CCIE priv ?
  3des  Use 168 bit 3DES algorithm for encryption
  aes   Use AES algorithm for encryption
  des   Use 56 bit DES algorithm for encryption

R5(config)#snmp-server user Dave CCNP v3 auth sha CCIE priv aes ?
  128  Use 128 bit AES algorithm for encryption
  192  Use 192 bit AES algorithm for encryption
  256  Use 256 bit AES algorithm for encryption

R5(config)#snmp-server user Dave CCNP v3 auth sha CCIE priv aes 128 ?
  WORD  privacy pasword for user

R5(config)#snmp-server user Dave CCNP v3 auth sha CCIE priv aes 128 CCNA ?
  access  specify an access-list associated with this group
  <cr>

R5(config)#snmp-server user Dave CCNP v3 auth sha CCIE priv aes 128 CCNA
R5(config)#
*Mar 19 04:24:28.623: Configuring snmpv3 USM user, persisting snmpEngineBoots. Please Wait…

R5(config)#

So I actually hit enter there without creating the above “group” config, just to see what happens, and I got this message. I am waiting for my routing to start sparking or smoking, but apparently when you hit enter you apparently start snmpEngineBoots.

So I hope the above ? output all looks fairly self explanatory, it’s just a mouthful to configure, which has me thankful its beyond the scope of this course (I hope) ūüôā

Now to give you some output to chew on as to configuring the traps, I won’t actually be configuring one, but here is the output of the beginning of the configuration:

R5(config)#snmp-server host ?
  WORD                                                  Hostname or IP address
                                                        of SNMP notification
                                                        host
  http://<Hostname or A.B.C.D>[:<port number>][/<uri>]  HTTP address of XML
                                                        notification host

R5(config)#snmp-server host 172.12.15.1 ?
  WORD     SNMPv1/v2c community string or SNMPv3 user name
  informs  Send Inform messages to this host
  traps    Send Trap messages to this host
  version  SNMP version to use for notification messages
  vrf      VPN Routing instance for this host

R5(config)#snmp-server host 172.12.15.1 traps ?
  WORD     SNMPv1/v2c community string or SNMPv3 user name
  version  SNMP version to use for notification messages

R5(config)#snmp-server host 172.12.15.1 traps version ?
  1   Use SNMPv1
  2c  Use SNMPv2c
  3   Use SNMPv3

R5(config)#snmp-server host 172.12.15.1 traps version 3 ?
  auth    Use the SNMPv3 authNoPriv Security Level
  noauth  Use the SNMPv3 noAuthNoPriv Security Level
  priv    Use the SNMPv3 authPriv Security Level

R5(config)#snmp-server host 172.12.15.1 traps version 3 priv ?
  WORD  SNMPv1/v2c community string or SNMPv3 user name

R5(config)#snmp-server host 172.12.15.1 traps version 3 priv Dave ?
  aaa_server               Allow SNMP AAA traps
  adslline                 Allow ADSL LINE-MIB traps
  atm                      Allow SNMP atm traps
  authenticate-fail        Allow SNMP 802.11 Authentication Fail Trap
  bgp                      Allow BGP state change traps
  bulkstat                 Allow Data-Collection-MIB traps
  c3g                      Allow Cellular 3G modem reset traps
  call-home                Allow SNMP CISCO-CALLHOME-MIB traps
  cnpd                     Allow NBAR Protocol Discovery traps
  config                   Allow SNMP config traps
  config-copy              Allow SNMP config-copy traps
  config-ctid              Allow SNMP config-ctid traps
  cpu                      Allow cpu related traps
  deauthenticate           Allow SNMP 802.11 Deauthentication Trap
  disassociate             Allow SNMP 802.11 Disassociation Trap
  dot11-mibs               Allow dot11 traps
  dot11-qos                Allow SNMP 802.11 QoS Change Trap
  ds0-busyout              Allow ds0-busyout traps
  ds1                      Allow SNMP ds1 traps
  ds1-loopback             Allow ds1-loopback traps
  dsp                      Allow SNMP DSP traps
  eigrp                    Allow SNMP EIGRP traps
¬†–More–

And on and on it goes, which I won’t drown you or myself looking back on the modifier output, but the main take away if nothing else is REMEMBER THOSE PRIVILEGE LEVELS AND WHAT THEY DO (auth, noauth, priv)! Also, that SNMPv3 is the only one that offers both authentication and encryption.

That should cover what is needed for the ROUTE exam, next up is NTP and securing it, as it is possibly one of the most important protocols on your network to keep all network devices working. This is especially true for Cisco Voice stuff, but that’s a discussion for another day.

Router password fundamentals, configuration, a very good brush up on passwords unless you just took the CCNA

OSPF_Base_Topology

I have the above network configured,  but again will just use R1 and R5 for the discussion and examples of router output and configuration, as I am a walking zombie today due to lack of sleep and life stuff.

Speaking of life stuff, I said I was thinking about pushing my date out for CCNP ROUTE, and I got an email notification shortly after notifying me my test center I’ve always gone to is shutting down the week before my exam date so I had to reschedule it – So be careful what you wish for ūüôā 4/28 is the new date that I pass this exam and move onto SWITCH.

This will be brief, as it’s really CCNA refresher material, but if you haven’t taken the CCNA in years like myself then it’s good to know the command syntax and options.

That being said, lets start with the most basic password concepts and end on the least:

  • Enable password vs Secret – Enable secret will still be preferred over the enable password when prompted for a password for user exec in IOS 15.x
  • Password must be enabled on the VTY lines or connection will be refused
  • “service password-encryption” encrypts all current and future passwords in clear test in the router running configuration
  • Must configure “login” if only setting a password, and use “login local” to enable the use of username and password local database to log in

Speaking of local username / password database, lets configure a few usernames here to demonstrate “login local” as mentioned above briefly. Now generally in any show run you just see “login” on my vty lines, because in a lab environment that is ok, but real world you may want to have a username and password combination for router access.

Still on 15.x IOS, the username does still appear when you type it, but the password does not. So lets get to the configuration of our two users, the bobs:

R1(config)#username the password bobs
R1(config)#user bobs ?
  aaa                  AAA directive
  access-class         Restrict access by access-class
  autocommand          Automatically issue a command after the user logs in
  callback-dialstring  Callback dialstring
  callback-line        Associate a specific line with this callback
  callback-rotary      Associate a rotary group with this callback
  dnis                 Do not require password when obtained via DNIS
  nocallback-verify    Do not require authentication after callback
  noescape             Prevent the user from using an escape character
  nohangup             Do not disconnect after an automatic command
  nopassword           No password is required for the user to log in
  password             Specify the password for the user
  privilege            Set user privilege level
  secret               Specify the secret for the user
¬† user-maxlinks¬†¬†¬†¬†¬†¬†¬† Limit the user’s number of inbound links
  view                 Set view name
  <cr>

R1(config)#user bobs privilege ?
  <0-15>  User privilege level

R1(config)#user bobs privilege 15 ?
  aaa                  AAA directive
  access-class         Restrict access by access-class
  autocommand          Automatically issue a command after the user logs in
  callback-dialstring  Callback dialstring
  callback-line        Associate a specific line with this callback
  callback-rotary      Associate a rotary group with this callback
  dnis                 Do not require password when obtained via DNIS
  nocallback-verify    Do not require authentication after callback
  noescape             Prevent the user from using an escape character
  nohangup             Do not disconnect after an automatic command
  nopassword           No password is required for the user to log in
  password             Specify the password for the user
  privilege            Set user privilege level
  secret               Specify the secret for the user
¬† user-maxlinks¬†¬†¬†¬†¬†¬†¬† Limit the user’s number of inbound links
  view                 Set view name
  <cr>

R1(config)#user bobs privilege 15 password the

So the first username as you can see at the top, I just typed “username the password bobs” and it’s really just as easy as that to configure a user, however for ‘bobs the’ I let the ? output flow because there are some very weird command modifiers I want to be known.

For example after I entered the priv 15, we can enter “nopassword” or “secret” for the user, as well as even restricting the username by “access-class” which we all know from the last post is an ACL to lock down incoming connections! However I took the slacker road and just entered a username and password.

So lets look at the show run currently for this bad boy, and I’ll highlight in red all the security derps we have going on in it:

R1# sh run
Building configuration…

Current configuration : 1462 bytes
!
version 12.4
service timestamps debug datetime msec
service timestamps log datetime msec
no service password-encryption
!
hostname R1
!
boot-start-marker
boot-end-marker
!
enable secret 5 $1$aHDM$YMgDe3WXGwGCHctjWlGr71
enable password CCNA
!
no aaa new-model
!
resource policy
!
no network-clock-participate slot 1
no network-clock-participate wic 0
ip cef
!
!
!
!
no ip domain lookup
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
username the password 0 bobs
username bobs privilege 15 password 0 the
!
!
!
!
!
!
!
interface Loopback1
 ip address 1.1.1.1 255.255.255.255
!
interface FastEthernet0/0
 no ip address
 shutdown
 duplex auto
 speed auto
!
interface Serial0/0
 ip address 172.12.123.1 255.255.255.0
 encapsulation frame-relay
 frame-relay map ip 172.12.123.2 122 broadcast
 frame-relay map ip 172.12.123.3 123 broadcast
 no frame-relay inverse-arp
 frame-relay lmi-type cisco
!
interface FastEthernet0/1
 ip address 172.12.15.1 255.255.255.0
 duplex auto
 speed auto
!
interface Serial0/1
 ip address 172.12.13.1 255.255.255.252
 clock rate 2000000
!
router ospf 1
 log-adjacency-changes
 area 34 virtual-link 44.44.44.1
 network 1.1.1.1 0.0.0.0 area 1
 network 172.12.15.0 0.0.0.255 area 15
 network 172.12.123.0 0.0.0.255 area 0
!
!
!
ip http server
no ip http secure-server
!
!
!
!
!
!
control-plane
!
!
!
!
!
!
!
!
!
!
!
line con 0
 exec-timeout 0 0
 logging synchronous
line aux 0
line vty 0 4
 password CCNP
 logging synchronous
 login
!
!
end

R1#

First we do not have the service “password-encryption” running, so you can see all non-secret (almost every) password because they are in plain text.

A never ending exec mode timeout can lead to leaving sessions open, and allowing others to stumble upon an open router at it’s prompt, or use it to get around time based ACL’s (as discussed in the last post).

Another issue with the vty lines, is they are using telnet so all data is transferred in plain text INCLUDING THE PASSWORD TO LOGIN, but it is also using a single password instead of the local username / password database.

So lets tighten some things up:

R1#conf t
Enter configuration commands, one per line.  End with CNTL/Z.
R1(config)#service password-encryption
R1(config)#do sh run | i username
username the password 7 094E410B0A
username bobs privilege 15 password 7 105A011C
R1(config)#

After enabling password-encryption in global config, a sh run | i username shows they are now encrypted in the running configuration, and in case you are wondering the pipe include ( | i ) will only give you output with the keyword or number you specify after it.

So lets make sure even though service password-encryption is running, that the secret takes precedence as the password used for enable:

R5#telnet 172.12.15.1
Trying 172.12.15.1 … Open

User Access Verification

Password:¬† <—- Set on VTY lines as CCNP (about to change that)
R1>en
Password:¬†¬† <—- Tried CCNA
Password:¬†¬† <—- Tried CCNP
R1#

So lets first change this so that we are using usernames and password combinations at the least:

R1(config)#
R1(config)#line vty 0 4
R1(config-line)#login local
R1(config-line)#^Z
R1#
*Mar  1 23:09:21.103: %SYS-5-CONFIG_I: Configured from console by console
R1#
ASR#2
[Resuming connection 2 to r5 … ]

R5#telnet 172.12.15.1
Trying 172.12.15.1 … Open

User Access Verification

Username: the
Password:
R1>en
Password:¬†¬† <—- CCNA
Password:¬†¬† <—- CCNP
R1#

Now I’m having too much fun with the enable secret precedence. As seen, “the” is visible while “bobs” is not, so that is how that goes. However, that priv 15 should kick us straight into user exec mode when telnet’ing in, lets check it out:

R1#exit

[Connection to 172.12.15.1 closed by foreign host]
R5#telnet 172.12.15.1
Trying 172.12.15.1 … Open

User Access Verification

Username: bobs
Password:
R1#

And that it did, right into user exec mode. We are going in the right direction, but we need to take care of using telnet to log into the router, as we are security minded CCNP candidates.

This takes actually a couple steps and we’ve already taken care of one of them by forcing the login local for remote connections, as ssh requires a username / password whether its an AAA server doing authentication or whether its authentication LOCAL.

The next step is defining on the vty lines, what kind of remote management protocols you want to allow to access those lines:

R1#conf t
Enter configuration commands, one per line.  End with CNTL/Z.
R1(config)#line vty 0 4
R1(config-line)#transport ?
  input      Define which protocols to use when connecting to the terminal
             server
  output     Define which protocols to use for outgoing connections
  preferred  Specify the preferred protocol to use

R1(config-line)#transport input ?
  all     All protocols
  lat     DEC LAT protocol
  mop     DEC MOP Remote Console Protocol
  none    No protocols
  pad     X.3 PAD
  rlogin  Unix rlogin protocol
  ssh     TCP/IP SSH protocol
  telnet  TCP/IP Telnet protocol
  udptn   UDPTN async via UDP protocol
  v120    Async over ISDN

R1(config-line)#transport input ssh
R1(config-line)#exit
R1(config)#exit
R1#exit

[Connection to 172.12.15.1 closed by foreign host]
R5#

Oddly I kind of expected it to kill my telnet connection to it from R5, but perhaps because the TCP connection was already made, you must wait for it to break and try reconnecting for the limitation or rule to kick in which seems to be a universal rule with TCP connections:

R5#telnet 172.12.15.1
Trying 172.12.15.1 …
% Connection refused by remote host

R5#

And so it is, TCP connections will maintain their connection until torn down, newly added configs during the duration of the connection will not effect the TCP connection(s).

However we are not done with the SSH setup yet, it requires for a domain name to be added to the router, along with a crypto key to be generated:

R1#conf t
Enter configuration commands, one per line.  End with CNTL/Z.
R1(config)#ip domain ?
  list         Domain name to complete unqualified host names
  lookup       Enable IP Domain Name System hostname translation
  multicast    Define the domain name for multicast address lookups
  name         Define the default domain name
  retry        Specify times to retry sending a DNS query
  round-robin  Round-robin multiple IP addresses in cache
  timeout      Specify timeout waiting for response to a DNS query

R1(config)#ip domain-name ?
  WORD  Default domain name
  vrf   Specify VRF

R1(config)#ip domain-name loopedback.com
R1(config)#crypto key generate ?
  rsa  Generate RSA keys
  <cr>

R1(config)#crypto key generate rsa
The name for the keys will be: R1.loopedback.com
Choose the size of the key modulus in the range of 360 to 2048 for your
  General Purpose Keys. Choosing a key modulus greater than 512 may take
  a few minutes.

How many bits in the modulus [512]: 1024
% Generating 1024 bit RSA keys, keys will be non-exportable…[OK]

R1(config)#
*Mar  1 23:22:59.697: %SSH-5-ENABLED: SSH 1.99 has been enabled
R1(config)#

As you can see by that last console message, we now have ssh success! Now a couple things, you would of course use the local domain name of the network the router is an edge device on, also the crypto key size can be 512 (and is by default) but should be made to at least 1024 bit encryption in the real world (at very least) so I did here.

So lets see if you can ssh from router to router, I’m not sure if I have actually ever tried:

R5#ssh 172.12.15.1
% No user specified nor available for SSH client
R5#ssh ?
  -c    Select encryption algorithm
  -l    Log in using this user name
  -m    Select HMAC algorithm
  -o    Specify options
  -p    Connect to this port
  -v    Specify SSH Protocol Version
  -vrf  Specify vrf name
  WORD  IP address or hostname of a remote system

R5#telnet ?
  WORD  IP address or hostname of a remote system
  <cr>

R5#ssh -l ?
  WORD  Login name

R5#ssh -l the ?
  -c    Select encryption algorithm
  -m    Select HMAC algorithm
  -o    Specify options
  -p    Connect to this port
  -v    Specify SSH Protocol Version
  -vrf  Specify vrf name
  WORD  IP address or hostname of a remote system

R5#ssh -l the 172.12.15.1

Password:

R1>

Huh, I didn’t think I’d figure it out that easy. So “ssh -l (username) (remote IP)” and you will be prompted for your password and get logged in. I have never configured that before, that is very good to know.

That’s going to do it for passwords, I am officially fried, thee ya!