NTP Broadcast better demonstrated over an Ethernet segment, pretty brief and to the point to finish off the NTP section!

ACL_Refresher_R1toR5

I have done a “wr er” / “reload” on R1 and R5, configuring only loopbacks, and opened Fa0/1 interfaces to be Area 0 between the two. No NBMA, no oddities (hopefully), and see how it works with a lot of troubleshooting.

So to begin, I’ll set R1 first as the NTP master with a clock time:

R1#clock set 18:00:00 30 mar 2017
R1#
*Mar 30 18:00:00.000: %SYS-6-CLOCKUPDATE: System clock has been updated from 18:43:02 UTC Thu Mar 30 2017 to 18:00:00 UTC Thu Mar 30 2017, configured from console by console.
R1#conf t
Enter configuration commands, one per line.  End with CNTL/Z.
R1(config)#ntp master ?
  <1-15>  Stratum number
  <cr>

R1(config)#ntp master 3
R1(config)#int fa0/1
R1(config-if)#ntp broadcast ?
  client       Listen to NTP broadcasts
  destination  Configure broadcast destination address
  key          Configure broadcast authentication key
  version      Configure NTP version
  <cr>

R1(config-if)#ntp broadcast
R1(config-if)#

So that is all configured, unfortunately R1 is running on 12.x IOS code, so it is using NTPv3 (and is only capable of running version 3). So lets go to R5 and see if we can get this working:

R5#conf t
Enter configuration commands, one per line.  End with CNTL/Z.
R5(config)#int fa0/1
R5(config-if)#ntp ?
  broadcast  Configure NTP broadcast service
  disable    Disable NTP traffic (both IP and IPv6)
  multicast  Configure NTP multicast service

R5(config-if)#ntp broadcast ?
  client       Listen to NTP broadcasts
  destination  Configure broadcast destination address
  key          Configure broadcast authentication key
  version      Configure NTP version
  <cr>

R5(config-if)#ntp broadcast client ?
  <cr>

R5(config-if)#ntp broadcast client
R5(config-if)#^Z
R5#debug
*Mar 30 17:05:01.491: %SYS-5-CONFIG_I: Configured from console by console
R5#debug ntp pack
NTP packets debugging is on
R5#
*Mar 30 17:05:10.659: NTP message received from 172.12.15.1 on interface ‘FastEthernet0/1’ (255.255.255.255).
R5#sh clock
*17:05:27.923 UTC Thu Mar 30 2017
R5#

Just as easy as that, that is how it is supposed to work 🙂

One thing I noticed on R5 running 15.1 IOS code is the NTP messages are so much smaller and concise, a lot of the basic infrastructure works the same between IOS versions but I do like some of the subtle differences.

So now I’ll reverse roles with debug still running, and see what happens starting with R5:

R5#sh clock
.18:45:28.132 UTC Thu Mar 30 2017
R5#conf t
Enter configuration commands, one per line.  End with CNTL/Z.
R5(config)#ntp master ?
  <1-15>  Stratum number
  <cr>

R5(config)#ntp master 3 ?
  <cr>

R5(config)#ntp master 3
R5(config)#int fa0/1
R5(config-if)#ntp broadcast ?
  client       Listen to NTP broadcasts
  destination  Configure broadcast destination address
  key          Configure broadcast authentication key
  version      Configure NTP version
  <cr>

R5(config-if)#ntp broadcast version ?
  <2-4>  NTP version number

R5(config-if)#ntp broadcast version 4
R5(config-if)#^Z
R5#deb
.Mar 30 18:49:32.833: %SYS-5-CONFIG_I: Configured from console by console
R5#debug ntp pack
.Mar 30 18:49:38.737: NTP message sent to 255.255.255.255, from interface ‘FastEthernet0/1’ (172.12.15.5).
R5#debug ntp pack
NTP packets debugging is on
R5#

I changed it to Version 4 NTP to see if R1 who only understands up to version 3 can still pick up the time from R5, so debugs running lets see what happens with R1:

R1#conf t
Enter configuration commands, one per line.  End with CNTL/Z.
R1(config)#int fa0/1
R1(config-if)#ntp broadcast client
R1(config-if)#do sh clock
18:54:12.613 UTC Thu Mar 30 2017
R1(config-if)#
ASR#2
[Resuming connection 2 to r5 … ]

Mar 30 18:52:48.736: NTP message sent to 255.255.255.255, from interface ‘FastEthernet0/1’ (172.12.15.5).
R5#
Mar 30 18:53:54.736: NTP message sent to 255.255.255.255, from interface ‘FastEthernet0/1’ (172.12.15.5).
R5#
Mar 30 18:54:57.735: NTP message sent to 255.255.255.255, from interface ‘FastEthernet0/1’ (172.12.15.5).
R5#sh clock
18:55:04.327 UTC Thu Mar 30 2017
R5#clock set 21:00:00 30 mar 2017
R5#
.Mar 30 21:00:00.000: %SYS-6-CLOCKUPDATE: System clock has been updated from 18:55:35 UTC Thu Mar 30 2017 to 21:00:00 UTC Thu Mar 30 2017, configured from console by console.
R5#
.Mar 30 21:00:26.763: NTP message sent to 255.255.255.255, from interface ‘FastEthernet0/1’ (172.12.15.5).
R5#
ASR#1
[Resuming connection 1 to r1 … ]

R1(config-if)#do sh clock
18:56:17.338 UTC Thu Mar 30 2017
R1(config-if)#

So it is not updating, as I believe it just doesn’t understand NTP version 4, so I’ll change it to version 3 and see if we can get this working:

R5(config)#int fa0/1
R5(config-if)#no ntp broadcast version 4
R5(config-if)#ntp broadcast version 3
R5(config-if)#do sh clock
21:03:18.143 UTC Thu Mar 30 2017
R5(config-if)#
Mar 30 21:03:55.763: NTP message sent to 255.255.255.255, from interface ‘FastEthernet0/1’ (172.12.15.5).
R5(config-if)#
ASR#1
[Resuming connection 1 to r1 … ]

R1(config-if)#do sh clock
18:59:56.341 UTC Thu Mar 30 2017

So it’s not liking that, so I tried removing and re-adding the the “ntp broadcast client” on R1 to see if that’d kick it into gear but it did not so I did have to reboot R1, however:

R1>en
Password:
R1#sh clock
21:24:00.844 UTC Thu Mar 30 2017
R1#

Almost gets me teary how proud I am of my routers to work as expected 🙂

So that is it, done with NTP for now, we have officially beaten that dead horse into the ground!

One thing to note, as this NTP battle has underscored running IOS code 12.x sucks for the current CCNP ROUTE exam, all of the concepts we have troubleshot have been overcome by logic, and not incompatibility between the versions. I will definitely demonstrate on my 2 routers I have running IOS 15.x any critical to know material that are not on my NBMA routers (R1 / R2 / R3 are on 12.x IOS code), however it really does not alter the behaviors of most protocols beyond newer versions of them – So don’t take this as irrelevant due to the 12.x code over the NBMA I would just rather invest that money into the exams

NTP Finale – Configuring broadcast / multicast / version #, and disabling NTP all together – Time to see what breaks!

OSPF_Base_Topology_NTP

After quite a few days out of the study game due to illness and actually giving myself time to recover from it this time, I am back, and I am already sick of going over NTP – So this is the last but important concepts for NTP.

One oddity I have noticed working with NTP on my routers over the NBMA, is that R1 for some reason does not save the time after I “wr mem” so I have to use the “clock set …” command every time for it, however R2 and R3 never catch up after I reset the clock and get their time:

R3#sh clock
*00:33:12.545 UTC Sat Mar 2 2002
R3#sh ntp assoc

      address         ref clock     st  when  poll reach  delay  offset    disp
 ~172.12.123.1     0.0.0.0          16    45    64    0     0.0    0.00  16000.
 * master (synced), # master (unsynced), + selected, – candidate, ~ configured
R3#
ASR#2
[Resuming connection 2 to r2 … ]

R2#sh clock
*00:32:47.895 UTC Sat Mar 2 2002
R2#

That is, unless I reload them, I let them sit for 10 minutes after resetting R1 but still nothing. So, given the frame switch is a NON-BROADCAST MultiAccess network between all of them, I am going to change this up a bit to see if I can wake R2 and R3 up:

R1#conf t
Enter configuration commands, one per line.  End with CNTL/Z.
R1(config)#int s0/0
R1(config-if)#ntp ?
  broadcast  Configure NTP broadcast service
  disable    Disable NTP
  multicast  Configure NTP multicast service

R1(config-if)#ntp multicast ?
  A.B.C.D  Multicast group IP address
  client   Listen to NTP multicasts
  key      Configure multicast authentication key
  ttl      TTL of the multicast packet
  version  Configure NTP version
  <cr>

R1(config-if)#ntp multicast
R1(config-if)#
ASR#2
[Resuming connection 2 to r2 … ]

R2#conf t
Enter configuration commands, one per line.  End with CNTL/Z.
R2(config)#int s0/0
R2(config-if)#ntp multicast
R2(config-if)#
ASR#3
[Resuming connection 3 to r3 … ]

R3#conf t
Enter configuration commands, one per line.  End with CNTL/Z.
R3(config)#int s0/2
R3(config-if)#ntp multicast
R3(config-if)#
ASR#1
[Resuming connection 1 to r1 … ]

R1(config-if)#^Z

I am leaving R4 out of this lab, because it is late, and I want to play with a few of the extended command highlighted in red – specifically version # to see if NTP will jive between different version numbers.

So I threw on some debugging to see what we have going on with R2 and R3:
R1#debug ntp pack
NTP packets debugging is on
R1#
.Mar 29 20:43:46.821: NTP: rcv packet from 172.12.123.2 to 172.12.123.1 on Serial0/0:
.Mar 29 20:43:46.821:  leap 3, mode 3, version 3, stratum 0, ppoll 64
.Mar 29 20:43:46.821:  rtdel 0000 (0.000), rtdsp 10001 (1000.015), refid 00000000 (0.0.0.0)
.Mar 29 20:43:46.825:  ref 00000000.00000000 (00:00:00.000 UTC Mon Jan 1 1900)
.Mar 29 20:43:46.825:  org DC869AC2.D297792B (20:42:42.822 UTC Wed Mar 29 2017)
.Mar 29 20:43:46.825:  rec C02A9DA9.4FAC7075 (00:39:05.311 UTC Sat Mar 2 2002)
.Mar 29 20:43:46.825:  xmt C02A9DE9.42276D52 (00:40:09.258 UTC Sat Mar 2 2002)
.Mar 29 20:43:46.825:  inp DC869B02.D2BFA7C5 (20:43:46.823 UTC Wed Mar 29 2017)
R1#
.Mar 29 20:43:46.829: NTP: stateless xmit packet to 172.12.123.2:
.Mar 29 20:43:46.829:  leap 3, mode 4, version 3, stratum 0, ppoll 64
.Mar 29 20:43:46.829:  rtdel 0000 (0.000), rtdsp 10001 (1000.015), refid 00000000 (0.0.0.0)
.Mar 29 20:43:46.829:  ref 00000000.00000000 (00:00:00.000 UTC Mon Jan 1 1900)
.Mar 29 20:43:46.829:  org C02A9DE9.42276D52 (00:40:09.258 UTC Sat Mar 2 2002)
.Mar 29 20:43:46.829:  rec DC869B02.D2BFA7C5 (20:43:46.823 UTC Wed Mar 29 2017)
.Mar 29 20:43:46.829:  xmt DC869B02.D41D2E3F (20:43:46.828 UTC Wed Mar 29 2017)
R1#
.Mar 29 20:43:54.894: NTP: rcv packet from 172.12.123.3 to 172.12.123.1 on Serial0/0:
.Mar 29 20:43:54.894:  leap 3, mode 3, version 3, stratum 0, ppoll 64
.Mar 29 20:43:54.894:  rtdel 0000 (0.000), rtdsp 10001 (1000.015), refid 00000000 (0.0.0.0)
.Mar 29 20:43:54.894:  ref 00000000.00000000 (00:00:00.000 UTC Mon Jan 1 1900)
.Mar 29 20:43:54.898:  org DC869ACA.E5739DBB (20:42:50.896 UTC Wed Mar 29 2017)
.Mar 29 20:43:54.898:  rec C02A9DE2.574AB8D3 (00:40:02.340 UTC Sat Mar 2 2002)
.Mar 29 20:43:54.898:  xmt C02A9E22.46D2D084 (00:41:06.276 UTC Sat Mar 2 2002)
.Mar 29 20:43:54.898:  inp DC869B0A.E5776373 (20:43:54.896 UTC Wed Mar 29 2017)
R1#
.Mar 29 20:43:54.898: NTP: stateless xmit packet to 172.12.123.3:
.Mar 29 20:43:54.902:  leap 3, mode 4, version 3, stratum 0, ppoll 64
.Mar 29 20:43:54.902:  rtdel 0000 (0.000), rtdsp 10001 (1000.015), refid 00000000 (0.0.0.0)
.Mar 29 20:43:54.902:  ref 00000000.00000000 (00:00:00.000 UTC Mon Jan 1 1900)
.Mar 29 20:43:54.902:  org C02A9E22.46D2D084 (00:41:06.276 UTC Sat Mar 2 2002)
.Mar 29 20:43:54.902:  rec DC869B0A.E5776373 (20:43:54.896 UTC Wed Mar 29 2017)
.Mar 29 20:43:54.906:  xmt DC869B0A.E6CF23AF (20:43:54.901 UTC Wed Mar 29 2017)
R1#

So as can be seen, it shows their old time, and the new time, but we have seen this with the switch that stays insane on the Ethernet segment so let us check R2 and R3:

R2(config-if)#do sh clock
*00:47:10.551 UTC Sat Mar 2 2002
R2(config-if)#
ASR#3
[Resuming connection 3 to r3 … ]

R3(config-if)#do sh clock
*00:48:09.668 UTC Sat Mar 2 2002
R3(config-if)#do sh ntp assoc

      address         ref clock     st  when  poll reach  delay  offset    disp
 ~172.12.123.1     0.0.0.0          16    44    64    0     0.0    0.00  16000.
 * master (synced), # master (unsynced), + selected, – candidate, ~ configured
R3(config-if)#do sh ntp assoc det
172.12.123.1 configured, authenticated, insane, invalid, unsynced, stratum 16
ref ID 0.0.0.0, time 00000000.00000000 (00:00:00.000 UTC Mon Jan 1 1900)
our mode client, peer mode unspec, our poll intvl 64, peer poll intvl 64
root delay 0.00 msec, root disp 0.00, reach 0, sync dist 0.000
delay 0.00 msec, offset 0.0000 msec, dispersion 16000.00
precision 2**5, version 3
org time DC869CCA.E3EA868D (20:51:22.890 UTC Wed Mar 29 2017)
rcv time C02A9FE2.56ACF13C (00:48:34.338 UTC Sat Mar 2 2002)
xmt time C02A9FE2.45D78CAD (00:48:34.272 UTC Sat Mar 2 2002)
filtdelay =     0.00    0.00    0.00    0.00    0.00    0.00    0.00    0.00
filtoffset =    0.00    0.00    0.00    0.00    0.00    0.00    0.00    0.00
filterror =  16000.0 16000.0 16000.0 16000.0 16000.0 16000.0 16000.0 16000.0

R3(config-if)#

So it has the same issue as the switch, I’m having a hard time grasping what this issue is, however thinking upon it if you configure an interface for “ntp broadcast” or “ntp multicast” you don’t actually don’t have to have a server command pointing to R1 as it will listen on that interface for NTP broadcasts regardless.

So I am going to make a change to R1 and throw it into broadcast mode, as it should actually before forwarding broadcasts with given the ‘broadcast’ options on my frame relay statements, reload both of them to see what we get:

R3(config-if)#exit
R3(config)#no ntp server 172.12.123.1

R3(config)#int s0/2
R3(config-if)#ntp broadcast
R3(config-if)#^Z
R3#wr
*Mar  2 00:58:24.845: %SYS-5-CONFIG_I: Configured from console by console
R3#wr
Building configuration…
[OK]
R3#reload
Proceed with reload? [confirm]

ASR#2
[Resuming connection 2 to r2 … ]

R2(config-if)#ntp broadcast
R2(config-if)#exit
R2(config)#no ntp server 172.12.123.1
R2(config)#exit
R2#wr
Building configuration…

*Mar  2 00:58:18.581: %SYS-5-CONFIG_I: Configured from console by console[OK]
R2#reload
Proceed with reload? [confirm]

*Mar  2 00:58:35.857: %SYS-5-RELOAD: Reload requested by console. Reload Reason: Reload Command.
ASR#1
[Resuming connection 1 to r1 … ]

R1#conf t
Enter configuration commands, one per line.  End with CNTL/Z.
R1(config)#int s0/0
R1(config-if)#ntp broadcast
R1(config-if)#exit
R1(config)#exit
R1#debug
.Mar 29 21:02:31.933: %SYS-5-CONFIG_I: Configured from console by console
R1#debug ntp pack
NTP packets debugging is on
R1#

So now Serial0/0 on R1 is rocking broadcast mode, along with the serial interfaces of R2 and R3, and as they reload I have a debug running to see what output we get upon them coming up, lets see NTP come through for us just one more time so I can move on to a new topic:

(An off topic note, the slowness of a 2600 series router booting, mixed with a serial link speed to bring up an OSPF adjacency makes me want to jump off a roof. Fortunately my roof is only high enough to break my leg or something, so I won’t jump off it quite yet)

R1#debug
.Mar 29 21:02:31.933: %SYS-5-CONFIG_I: Configured from console by console
R1#debug ntp pack
NTP packets debugging is on
R1#
.Mar 29 21:03:38.574: %OSPF-5-ADJCHG: Process 1, Nbr 3.3.3.3 on Serial0/0 from FULL to DOWN, Neighbor Down: Dead timer expired
R1#
.Mar 29 21:04:14.165: %OSPF-5-ADJCHG: Process 1, Nbr 2.2.2.2 on Serial0/0 from FULL to DOWN, Neighbor Down: Dead timer expired
R1#sh access-list
Standard IP access list 10
    10 permit 172.12.123.0, wildcard bits 0.0.0.255 (149 matches)
    20 permit 172.12.34.0, wildcard bits 0.0.0.255 (26 matches)
    30 permit 172.12.23.0, wildcard bits 0.0.0.255 (73 matches)
R1#
.Mar 29 21:09:38.997: %OSPF-5-ADJCHG: Process 1, Nbr 3.3.3.3 on Serial0/0 from LOADING to FULL, Loading Done
R1#
.Mar 29 21:10:14.483: %OSPF-5-ADJCHG: Process 1, Nbr 2.2.2.2 on Serial0/0 from LOADING to FULL, Loading Done
R1#

So after 6 or so minutes of staring at the CLI, my adjacencies finally creaked their way out of their coffin to say Hello to R1, however no NTP debug output at all?

And as soon as I typed that, of course I DID get some output from the debug:

R1#
.Mar 29 21:12:07.224: NTP: rcv packet from 172.12.23.1 to 172.12.123.1 on Serial0/0:
.Mar 29 21:12:07.224:  leap 3, mode 3, version 3, stratum 0, ppoll 64
.Mar 29 21:12:07.224:  rtdel 0000 (0.000), rtdsp 10001 (1000.015), refid 00000000 (0.0.0.0)
.Mar 29 21:12:07.228:  ref 00000000.00000000 (00:00:00.000 UTC Mon Jan 1 1900)
.Mar 29 21:12:07.228:  org DC869F27.39F0071C (21:01:27.226 UTC Wed Mar 29 2017)
.Mar 29 21:12:07.228:  rec AF3BE488.1E7958AA (01:25:28.119 UTC Mon Mar 1 1993)
.Mar 29 21:12:07.228:  xmt AF3BE708.11A18045 (01:36:08.068 UTC Mon Mar 1 1993)
.Mar 29 21:12:07.228:  inp DC86A1A7.3A1C931F (21:12:07.226 UTC Wed Mar 29 2017)
R1#
.Mar 29 21:12:07.232: NTP: stateless xmit packet to 172.12.23.1:
.Mar 29 21:12:07.232:  leap 3, mode 4, version 3, stratum 0, ppoll 64
.Mar 29 21:12:07.232:  rtdel 0000 (0.000), rtdsp 10001 (1000.015), refid 00000000 (0.0.0.0)
.Mar 29 21:12:07.232:  ref 00000000.00000000 (00:00:00.000 UTC Mon Jan 1 1900)
.Mar 29 21:12:07.232:  org AF3BE708.11A18045 (01:36:08.068 UTC Mon Mar 1 1993)
.Mar 29 21:12:07.232:  rec DC86A1A7.3A1C931F (21:12:07.226 UTC Wed Mar 29 2017)
.Mar 29 21:12:07.232:  xmt DC86A1A7.3B65703B (21:12:07.232 UTC Wed Mar 29 2017)
R1#

So now the only debug information I am seeing is coming from the Ethernet segment which we did not alter, so I’m afraid to look at R2 and R3 but given they are configured only on the interface level to listen for NTP broadcasts maybe they just silently take the time and go with it like NTP ninjas?

Lets see:

R2>en
Password:
R2#sh clock
*01:05:28.941 UTC Sat Mar 2 2002
R2#

I did a “sh run” and forgot one of my cardinal rules, I’m already getting rusty – remove old configurations before entering new configurations for a command:

!
interface Serial0/0
 ip address 172.12.123.2 255.255.255.0
 encapsulation frame-relay
 ip ospf priority 0
 ntp broadcast
 ntp multicast
 frame-relay map ip 172.12.123.3 221
 frame-relay map ip 172.12.123.1 221 broadcast
 no frame-relay inverse-arp
 frame-relay lmi-type cisco

So I’ll save the output, but I removed NTP Multicast from all routers, and it didn’t make a difference.

And I just figured out what I goofed here:

R2(config-if)#ntp broadcast ?
  client       Listen to NTP broadcasts
  destination  Configure broadcast destination address
  key          Configure broadcast authentication key
  version      Configure NTP version
  <cr>

R2(config-if)#ntp broadcast client ?
  <cr>

R2(config-if)#ntp broadcast client

And just a few moments later:

R2(config-if)#do sh clock
.23:32:32.694 UTC Wed Mar 29 2017

So one big note worth mentioning on here that I completely derp’d, on the server you only need to set “broadcast”, however with clients listening on their interfaces you must add that “client” to the end as you are not pointing it to a master making it a client by default so you must configure it as a client. Clear as mud? Good, lets move on.

What a huge oversight, that is what facepalms are made of. However, lets see if the same works for multicasts as well then eh?

Again, after making the change on both clients to “ntp multicast client” on the serial interface I am surprisingly seeing no output from R1, and I’ve set it’s Serial0/0 interface as “ntp multicast” before configuring the others. Lets look at times around the network and see what we have:

R2(config-if)#do sh clock
.23:43:12.229 UTC Wed Mar 29 2017
R2(config-if)#do sh ntp assoc det

R2(config-if)#exit
R2(config)#exit
R2#sh n
.Mar 29 23:43:47.383: %SYS-5-CONFIG_I: Configured from console by console
R2#sh ntp ?
  associations  NTP associations
  status        NTP status

R2#sh ntp status
Clock is unsynchronized, stratum 16, no reference clock
nominal freq is 249.5901 Hz, actual freq is 249.5907 Hz, precision is 2**16
reference time is DC86C1C0.16D0FDAD (23:29:04.089 UTC Wed Mar 29 2017)
clock offset is 0.0000 msec, root delay is 0.00 msec
root dispersion is 7875.02 msec, peer dispersion is 7875.02 msec

So it has the time that it’s probably held on to from the broadcast configuration, but as can be seen there is no ntp assoc, no reference clock, notta.

So I’m going to set everything back into broadcast mode, and see if we can get everything at least in order again, and again I’ll spare the output of the commands to keep this post that should be brief, as brief as possible.

Also, I’m just leaving “debug ntp pack” on R1 the entire time, I want to see every 1 and 0 that passes through that dang NTP Master to see if I can spot a clue as to what is going on here:

R2(config)#
R2(config)#int s0/0
R2(config-if)#no ntp multicast client
R2(config-if)#ntp broadcast client
R2(config-if)#exit
R2(config)#exit
R2#wr
Building configuration…

.Mar 29 23:49:25.404: %SYS-5-CONFIG_I: Configured from console by console[OK]
R2#
ASR#3
[Resuming connection 3 to r3 … ]

R3(config-if)#no ntp multicast client
R3(config-if)#ntp broadcast client
R3(config-if)#^Z
R3#wr
Building configuration…

*Mar  2 01:42:09.543: %SYS-5-CONFIG_I: Configured from console by console[OK]
R3#
R3#reload
Proceed with reload? [confirm]

*Mar  2 01:42:33.590: %SYS-5-RELOAD: Reload requested by console. Reload Reason: Reload Command.
ASR#2
[Resuming connection 2 to r2 … ]

R2#reload
Proceed with reload? [confirm]

.Mar 29 23:50:47.210: %SYS-5-RELOAD: Reload requested by console. Reload Reason: Reload Command.
ASR#1
[Resuming connection 1 to r1 … ]

R1#conf t
Enter configuration commands, one per line.  End with CNTL/Z.
R1(config)#int s0/0
R1(config-if)#no ntp multicast
R1(config-if)#ntp broadcast
R1(config-if)#^Z
R1#
.Mar 29 21:52:28.573: %SYS-5-CONFIG_I: Configured from console by console
R1#debug ntp pack
NTP packets debugging is on
R1#

I did that in kind of a weird order, as I wanted R2 and R3 to reload at relatively the same time, then reset R1 for broadcast mode as they re-cook, and now that I have 6 minutes to wait I put on my debug on R1 and going to start my PS4 and get food ready, run a marathon, and go sky diving so when I get back it should finally be back up and cooking.

So I literally cleaned my kitchen, did dishes, and came back to this:

R1#debug ntp pack
NTP packets debugging is on
R1#
.Mar 29 21:53:54.190: %OSPF-5-ADJCHG: Process 1, Nbr 3.3.3.3 on Serial0/0 from FULL to DOWN, Neighbor Down: Dead timer expired
R1#
.Mar 29 21:54:03.565: %OSPF-5-ADJCHG: Process 1, Nbr 2.2.2.2 on Serial0/0 from FULL to DOWN, Neighbor Down: Dead timer expired
R1#
.Mar 29 21:59:32.168: NTP: rcv packet from 172.12.34.4 to 172.12.123.1 on Serial0/0:
.Mar 29 21:59:32.168:  leap 3, mode 3, version 4, stratum 0, ppoll 256
.Mar 29 21:59:32.172:  rtdel 0000 (0.000), rtdsp 2095 (127.274), refid 41555448 (65.85.84.72)
.Mar 29 21:59:32.172:  ref 00000000.00000000 (00:00:00.000 UTC Mon Jan 1 1900)
.Mar 29 21:59:32.172:  org 00000000.00000000 (00:00:00.000 UTC Mon Jan 1 1900)
.Mar 29 21:59:32.172:  rec 00000000.00000000 (00:00:00.000 UTC Mon Jan 1 1900)
.Mar 29 21:59:32.172:  xmt DC870C00.8A152270 (04:45:52.539 UTC Thu Mar 30 2017)
.Mar 29 21:59:32.172:  inp DC86ACC4.2BCC2DF4 (21:59:32.171 UTC Wed Mar 29 2017)
R1#
.Mar 29 21:59:32.176: NTP: stateless xmit packet to 172.12.34.4:
.Mar 29 21:59:32.176:  leap 3, mode 4, version 4, stratum 0, ppoll 256
.Mar 29 21:59:32.176:  rtdel 0000 (0.000), rtdsp 10001 (1000.015), refid 00000000 (0.0.0.0)
.Mar 29 21:59:32.176:  ref 00000000.00000000 (00:00:00.000 UTC Mon Jan 1 1900)
.Mar 29 21:59:32.176:  org DC870C00.8A152270 (04:45:52.539 UTC Thu Mar 30 2017)
.Mar 29 21:59:32.176:  rec DC86ACC4.2BCC2DF4 (21:59:32.171 UTC Wed Mar 29 2017)
.Mar 29 21:59:32.176:  xmt DC86ACC4.2D26835D (21:59:32.176 UTC Wed Mar 29 2017)
R1#
.Mar 29 21:59:54.624: %OSPF-5-ADJCHG: Process 1, Nbr 3.3.3.3 on Serial0/0 from LOADING to FULL, Loading Done
R1#
.Mar 29 22:00:03.884: %OSPF-5-ADJCHG: Process 1, Nbr 2.2.2.2 on Serial0/0 from LOADING to FULL, Loading Done
R1#

The seemingly never ending OSPF Adj reform, but also again no NTP messages between R1 and either spoke, so lets take a look at the spokes here:
R2>en
Password:
R2#sh clock
*23:57:14.213 UTC Wed Mar 29 2017
R2#
ASR#3
[Resuming connection 3 to r3 … ]

System Bootstrap, Version 12.2(8r) [cmong 8r], RELEASE SOFTWARE (fc1)
Copyright (c) 2003 by cisco Systems, Inc.
C2600 platform with 131072 Kbytes of main memory

R3>en
Password:
R3#sh clock
*01:50:15.588 UTC Sat Mar 2 2002
R3#

I just don’t get some of these behaviors, and yes that clock does show its actually now past midnight, and this is one of those things that will drive me up a freegin wall if I do not figure out this behavior (Although as seen in other lab posts, there are bugs (and screenshots of Cisco’s webpage describing the bug) for some of these IOS’s, so it very well may be a bug.

So lets get to troubleshooting here and figure out what in the world is going on. So first, I want to stare and compare any conflicts on the “sh run” of all 3 routers which I’ll display here:

R1:

R1#sh run int serial0/0
Building configuration…

Current configuration : 258 bytes
!
interface Serial0/0
 ip address 172.12.123.1 255.255.255.0
 encapsulation frame-relay
 ntp broadcast
 frame-relay map ip 172.12.123.3 123 broadcast
 frame-relay map ip 172.12.123.2 122 broadcast
 no frame-relay inverse-arp
 frame-relay lmi-type cisco
end

R1#

R2:
R2#sh run int s0/0
Building configuration…

Current configuration : 275 bytes
!
interface Serial0/0
 ip address 172.12.123.2 255.255.255.0
 encapsulation frame-relay
 ip ospf priority 0
 ntp broadcast client
 frame-relay map ip 172.12.123.1 221 broadcast
 frame-relay map ip 172.12.123.3 221
 no frame-relay inverse-arp
 frame-relay lmi-type cisco
end

R2#

R3:

R3#sh run int s0/2
Building configuration…

Current configuration : 247 bytes
!
interface Serial0/2
 ip address 172.12.123.3 255.255.255.0
 encapsulation frame-relay
 ip ospf priority 0
 ntp broadcast client
 frame-relay map ip 172.12.123.2 321
 frame-relay map ip 172.12.123.1 321 broadcast
 no frame-relay inverse-arp
end

R3#

These all look good, how about pings, I guess what has burnt me in the past is Layer 3 connectivity:

R1#ping 172.12.123.2

Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 172.12.123.2, timeout is 2 seconds:
!!!!!
Success rate is 100 percent (5/5), round-trip min/avg/max = 64/66/69 ms
R1#ping 172.12.123.3

Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 172.12.123.3, timeout is 2 seconds:
!!!!!
Success rate is 100 percent (5/5), round-trip min/avg/max = 65/67/68 ms
R1#

Nope. So to try to force some action, I have set “debug ntp pack” on R2 and R3, and will do so now and remove the ntp broadcast command and switch it to a version number not used by the other routers that are using NTPv3 from what I see in past output, and see what happens:

R1#conf t
Enter configuration commands, one per line.  End with CNTL/Z.
R1(config)#int s0/0
R1(config-if)#no ntp broadcast version 2
R1(config-if)#^Z
R1#deb
.Mar 29 22:20:34.140: %SYS-5-CONFIG_I: Configured from console by console
R1#debug ntp pack
NTP packets debugging is on
R1#conf t
Enter configuration commands, one per line.  End with CNTL/Z.
R1(config)#int s0/0
R1(config-if)#ntp broadcast version 2
R1(config-if)#
ASR#2
[Resuming connection 2 to r2 … ]

R2#sh clock
*00:12:33.711 UTC Thu Mar 30 2017
R2#
ASR#3
[Resuming connection 3 to r3 … ]

R3#sh clock
*02:05:31.201 UTC Sat Mar 2 2002

Across the NBMA this gets all sorts of buggy, I must have gotten cursed code for my routers when it comes to NTP, however lets point back to a master so I can sleep tonight knowing my clocks are in sync:

R2#conf t
Enter configuration commands, one per line.  End with CNTL/Z.
R2(config)#int s0/0
R2(config-if)#no ntp broadcast client
R2(config-if)#exit
R2(config)#ntp server 172.12.123.1
R2(config)#
ASR#3
[Resuming connection 3 to r3 … ]

R3#conf t
Enter configuration commands, one per line.  End with CNTL/Z.
R3(config)#int s0/2
R3(config-if)#no ntp broadcast client
R3(config-if)#exit
R3(config)#ntp server 172.12.123.1
R3(config)#
ASR#1
[Resuming connection 1 to r1 … ]

.Mar 29 22:25:05.950: NTP: rcv packet from 172.12.34.4 to 172.12.123.1 on Serial0/0:
.Mar 29 22:25:05.950:  leap 3, mode 3, version 4, stratum 0, ppoll 1024
.Mar 29 22:25:05.950:  rtdel 0000 (0.000), rtdsp 2679 (150.284), refid 41555448 (65.85.84.72)
.Mar 29 22:25:05.950:  ref 00000000.00000000 (00:00:00.000 UTC Mon Jan 1 1900)
.Mar 29 22:25:05.954:  org DC86AEC3.1A2B1433 (22:08:03.102 UTC Wed Mar 29 2017)
.Mar 29 22:25:05.954:  rec DC870DFF.9B9DF69E (04:54:23.607 UTC Thu Mar 30 2017)
.Mar 29 22:25:05.954:  xmt DC8711FE.89175915 (05:11:26.535 UTC Thu Mar 30 2017)
.Mar 29 22:25:05.958:  inp DC86B2C1.F3D1CFC4 (22:25:05.952 UTC Wed Mar 29 2017)
R1(config)#0F7FD7A (22:25:29.066 UTC Wed Mar 29 2017)
.Mar 29 22:25:30.057: NTP: rcv packet from 172.12.123.2 to 172.12.123.1 on Serial0/0:
.Mar 29 22:25:30.057:  leap 3, mode 3, version 3, stratum 0, ppoll 64
.Mar 29 22:25:30.057:  rtdel 0000 (0.000), rtdsp 10001 (1000.015), refid 00000000 (0.0.0.0)
.Mar 29 22:25:30.057:  ref 00000000.00000000 (00:00:00.000 UTC Mon Jan 1 1900)
.Mar 29 22:25:30.057:  org DC86B2D9.10F7FD7A (22:25:29.066 UTC Wed Mar 29 2017)
.Mar 29 22:25:30.062:  rec DC86CCEF.534DC1DA (00:16:47.325 UTC Thu Mar 30 2017)
.Mar 29 22:25:30.062:  xmt DC86CCF0.43AD06C4 (00:16:48.264 UTC Thu Mar 30 2017)
.Mar 29 22:25:30.062:  inp DC86B2DA.0F1B0C1B (22:25:30.059 UTC Wed Mar 29 2017)
.Mar 29 22:25:30.062: NTP: stateless xmit packet to 172.12.123.2:
.Mar 29 22:25:30.062:  leap 3, mode 4, version 3, stratum 0, ppoll 64
.Mar 29 22:25:30.062:  rtdel 0000 (0.000), rtdsp 10001 (1000.015), refid 00000000 (0.0.0.0)
.Mar 29 22:25:30.066:  ref 00000000.00000000 (00:00:00.000 UTC Mon Jan 1 1900)
.Mar 29 22:25:30.066:  org DC86CCF0.43AD06C4 (00:16:48.264 UTC Thu Mar 30 2017)
.Mar 29 22:25:30.066:  rec DC86B2DA.0F1B0C1B (22:25:30.059 UTC Wed Mar 29 2017)
.Mar 29 22:25:30.066:  xmt DC86B2DA.10675AC7 (22:25:30.064 UTC Wed Mar 29 2017)
.Mar 29 22:25:31.059: NTP: rcv packet from 172.12.123.2 to 172.12.123.1 on Serial0/0:
.Mar 29 22:25:31.059:  leap 3, mode 3, version 3, stratum 0, ppoll 64
.Mar 29 22:25:31.059:  rtdel 0000 (0.000), rtdsp 10001 (1000.015), refid 00000000 (0.0.0.0)
.Mar 29 22:25:31.059:  ref 00000000.00000000 (00:00:00.000 UTC Mon Jan 1 1900)
.Mar 29 22:25:31.059:  org DC86B2DA.10675AC7 (22:25:30.064 UTC Wed Mar 29 2017)
.Mar 29 22:25:31.059:  rec DC86CCF0.52B2A86D (00:16:48.323 UTC Thu Mar 30 2017)
.Mar 29 22:25:31.063:  xmt DC86CCF1.44195CA6 (00:16:49.266 UTC Thu Mar 30 2017)
.Mar 29 22:25:31.063:  inp DC86B2DB.0F712205 (22:25:31.060 UTC Wed Mar 29 2017)
.Mar 29 22:25:31.063: NTP: stateless xmit packet to 172.12.123.2:
.Mar 29 22:25:31.063:  leap 3, mode 4, version 3, stratum 0, ppoll 64
.Mar 29 22:25:31.063:  rtdel 0000 (0.000), rtdsp 10001 (1000.015), refid 00000000 (0.0.0.0)
.Mar 29 22:25:31.063:  ref 00000000.00000000 (00:00:00.000 UTC Mon Jan 1 1900)
.Mar 29 22:25:31.067:  org DC86CCF1.44195CA6 (00:16:49.266 UTC Thu Mar 30 2017)
.Mar 29 22:25:31.067:  rec DC86B2DB.0F712205 (22:25:31.060 UTC Wed Mar 29 2017)
.Mar 29 22:25:31.067:  xmt DC86B2DB.10BC944D (22:25:31.065 UTC Wed Mar 29 2017)
.Mar 29 22:25:32.061: NTP: rcv packet from 172.12.123.2 to 172.12.123.1 on Serial0/0:
.Mar 29 22:25:32.061:  leap 3, mode 3, version 3, stratum 0, ppoll 64
.Mar 29 22:25:32.061:  rtdel 0000 (0.000), rtdsp 10001 (1000.015), refid 00000000 (0.0.0.0)
.Mar 29 22:25:32.061:  ref 00000000.00000000 (00:00:00.000 UTC Mon Jan 1 1900)
.Mar 29 22:25:32.061:  org DC86B2DB.10BC944D (22:25:31.065 UTC Wed Mar 29 2017)
.Mar 29 22:25:32.061:  rec DC86CCF1.530371D5 (00:16:49.324 UTC Thu Mar 30 2017)
.Mar 29 22:25:32.065:  xmt DC86CCF2.4483F9C0 (00:16:50.267 UTC Thu Mar 30 2017)
.Mar 29 22:25:32.065:  inp DC86B2DC.0FE71261 (22:25:32.062 UTC Wed Mar 29 2017)
.Mar 29 22:25:32.065: NTP: stateless xmit packet to 172.12.123.2:
.Mar 29 22:25:32.065:  leap 3, mode 4, version 3, stratum 0, ppoll 64
.Mar 29 22:25:32.065:  rtdel 0000 (0.000), rtdsp 10001 (1000.015), refid 00000000 (0.0.0.0)
.Mar 29 22:25:32.065:  ref 00000000.00000000 (00:00:00.000 UTC Mon Jan 1 1900)
.Mar 29 22:25:32.069:  org DC86CCF2.4483F9C0 (00:16:50.267 UTC Thu Mar 30 2017)
.Mar 29 22:25:32.069:  rec DC86B2DC.0FE71261 (22:25:32.062 UTC Wed Mar 29 2017)
.Mar 29 22:25:32.069:  xmt DC86B2DC.1136D29C (22:25:32.067 UTC Wed Mar 29 2017)
.Mar 29 22:25:33.058: NTP: rcv packet from 172.12.123.2 to 172.12.123.1 on Serial0/0:
.Mar 29 22:25:33.058:  leap 3, mode 3, version 3, stratum 0, ppoll 64
.Mar 29 22:25:33.058:  rtdel 0000 (0.000), rtdsp 10001 (1000.015), refid 00000000 (0.0.0.0)
.Mar 29 22:25:33.058:  ref 00000000.00000000 (00:00:00.000 UTC Mon Jan 1 1900)
.Mar 29 22:25:33.058:  org DC86B2DC.1136D29C (22:25:32.067 UTC Wed Mar 29 2017)
.Mar 29 22:25:33.058:  rec DC86CCF2.5387E2A1 (00:16:50.326 UTC Thu Mar 30 2017)
.Mar 29 22:25:33.062:  xmt DC86CCF3.43E9BCB7 (00:16:51.265 UTC Thu Mar 30 2017)
.Mar 29 22:25:33.062:  inp DC86B2DD.0F48A96F (22:25:33.059 UTC Wed Mar 29 2017)
.Mar 29 22:25:33.062: NTP: stateless xmit packet to 172.12.123.2:
.Mar 29 22:25:33.062:  leap 3, mode 4, version 3, stratum 0, ppoll 64
.Mar 29 22:25:33.062:  rtdel 0000 (0.000), rtdsp 10001 (1000.015), refid 00000000 (0.0.0.0)
.Mar 29 22:25:33.062:  ref 00000000.00000000 (00:00:00.000 UTC Mon Jan 1 1900)
.Mar 29 22:25:33.066:  org DC86CCF3.43E9BCB7 (00:16:51.265 UTC Thu Mar 30 2017)
.Mar 29 22:25:33.066:  rec DC86B2DD.0F48A96F (22:25:33.059 UTC Wed Mar 29 2017)
.Mar 29 22:25:33.066:  xmt DC86B2DD.1094F81A (22:25:33.064 UTC Wed Mar 29 2017)
.Mar 29 22:25:37.818: NTP: rcv packet from 172.12.123.3 to 172.12.123.1 on Serial0/0:
.Mar 29 22:25:37.818:  leap 3, mode 3, version 3, stratum 0, ppoll 64
.Mar 29 22:25:37.818:  rtdel 0000 (0.000), rtdsp 10001 (1000.015), refid 00000000 (0.0.0.0)
.Mar 29 22:25:37.818:  ref 00000000.00000000 (00:00:00.000 UTC Mon Jan 1 1900)
.Mar 29 22:25:37.818:  org 00000000.00000000 (00:00:00.000 UTC Mon Jan 1 1900)
.Mar 29 22:25:37.818:  rec 00000000.00000000 (00:00:00.000 UTC Mon Jan 1 1900)
.Mar 29 22:25:37.818:  xmt C02AB2EA.4A4B18B3 (02:09:46.290 UTC Sat Mar 2 2002)
.Mar 29 22:25:37.822:  inp DC86B2E1.D1AC185A (22:25:37.819 UTC Wed Mar 29 2017)
.Mar 29 22:25:37.822: NTP: stateless xmit packet to 172.12.123.3:
.Mar 29 22:25:37.822:  leap 3, mode 4, version 3, stratum 0, ppoll 64
.Mar 29 22:25:37.822:  rtdel 0000 (0.000), rtdsp 10001 (1000.015), refid 00000000 (0.0.0.0)
.Mar 29 22:25:37.822:  ref 00000000.00000000 (00:00:00.000 UTC Mon Jan 1 1900)
.Mar 29 22:25:37.822:  org C02AB2EA.4A4B18B3 (02:09:46.290 UTC Sat Mar 2 2002)
.Mar 29 22:25:37.826:  rec DC86B2E1.D1AC185A (22:25:37.819 UTC Wed Mar 29 2017)
.Mar 29 22:25:37.826:  xmt DC86B2E1.D2F33CAE (22:25:37.824 UTC Wed Mar 29 2017)
.Mar 29 22:25:38.820: NTP: rcv packet from 172.12.123.3 to 172.12.123.1 on Serial0/0:
.Mar 29 22:25:38.820:  leap 3, mode 3, version 3, stratum 0, ppoll 64
.Mar 29 22:25:38.820:  rtdel 0000 (0.000), rtdsp 10001 (1000.015), refid 00000000 (0.0.0.0)
.Mar 29 22:25:38.820:  ref 00000000.00000000 (00:00:00.000 UTC Mon Jan 1 1900)
.Mar 29 22:25:38.820:  org DC86B2E1.D2F33CAE (22:25:37.824 UTC Wed Mar 29 2017)
.Mar 29 22:25:38.820:  rec C02AB2EA.593EA3B9 (02:09:46.348 UTC Sat Mar 2 2002)
.Mar 29 22:25:38.824:  xmt C02AB2EB.4AB74497 (02:09:47.291 UTC Sat Mar 2 2002)
.Mar 29 22:25:38.824:  inp DC86B2E2.D215FB3E (22:25:38.820 UTC Wed Mar 29 2017)
.Mar 29 22:25:38.824: NTP: stateless xmit packet to 172.12.123.3:
.Mar 29 22:25:38.824:  leap 3, mode 4, version 3, stratum 0, ppoll 64
.Mar 29 22:25:38.824:  rtdel 0000 (0.000), rtdsp 10001 (1000.015), refid 00000000 (0.0.0.0)
.Mar 29 22:25:38.824:  ref 00000000.00000000 (00:00:00.000 UTC Mon Jan 1 1900)
.Mar 29 22:25:38.824:  org C02AB2EB.4AB74497 (02:09:47.291 UTC Sat Mar 2 2002)
.Mar 29 22:25:38.828:  rec DC86B2E2.D215FB3E (22:25:38.820 UTC Wed Mar 29 2017)
.Mar 29 22:25:38.828:  xmt DC86B2E2.D36249E9 (22:25:38.825 UTC Wed Mar 29 2017)
.Mar 29 22:25:39.817: NTP: rcv packet from 172.12.123.3 to 172.12.123.1 on Serial0/0:
.Mar 29 22:25:39.817:  leap 3, mode 3, version 3, stratum 0, ppoll 64
.Mar 29 22:25:39.817:  rtdel 0000 (0.000), rtdsp 10001 (1000.015), refid 00000000 (0.0.0.0)
.Mar 29 22:25:39.817:  ref 00000000.00000000 (00:00:00.000 UTC Mon Jan 1 1900)
.Mar 29 22:25:39.817:  org DC86B2E2.D36249E9 (22:25:38.825 UTC Wed Mar 29 2017)
.Mar 29 22:25:39.817:  rec C02AB2EB.5BF27F97 (02:09:47.359 UTC Sat Mar 2 2002)
.Mar 29 22:25:39.822:  xmt C02AB2EC.4A1B24F3 (02:09:48.289 UTC Sat Mar 2 2002)
.Mar 29 22:25:39.822:  inp DC86B2E3.D189A67E (22:25:39.818 UTC Wed Mar 29 2017)

Woah! That is what I needed to see, some life from my spokes!! If this does not show the correct time on my NTP routers now, I might just throw my rack off the roof instead of myself. I did check R2 which has the correct time, but no NTP association.

The part that drives me crazy, and as seen below, I will do a “wr” with the new ntp master on their and the interface configs removed, and it will come back up as it should. I may revisit this on an Ethernet only segment to cut out the oddities NBMA might be throwing in there.

R3#wr
Building configuration…

*Mar  2 02:15:21.301: NTP: xmit packet to 172.12.123.1:
*Mar  2 02:15:21.301:  leap 3, mode 3, version 3, stratum 0, ppoll 64
*Mar  2 02:15:21.301:  rtdel 0000 (0.000), rtdsp 10001 (1000.015), refid 00000000 (0.0.0.0)
*Mar  2 02:15:21.301:  ref 00000000.00000000 (00:00:00.000 UTC Mon Jan 1 1900)
*Mar  2 02:15:21.305:  org DC86B3F0.D0EA111D (22:30:08.816 UTC Wed Mar 29 2017)
*Mar  2 02:15:21.305:  rec C02AB3F9.57D528A6 (02:14:17.343 UTC Sat Mar 2 2002)
*Mar  2 02:15:21.305:  xmt C02AB439.4DA1B594 (02:15:21.303 UTC Sat Mar 2 2002)
*Mar  2 02:15:21.506: NTP: rcv packet from 172.12.123.1 to 172.12.123.3 on Serial0/2:
*Mar  2 02:15:21.510:  leap 3, mode 4, version 3, stratum 0, ppoll 64
*Mar  2 02:15:21.510:  rtdel 0000 (0.000), rtdsp 10001 (1000.015), refid 00000000 (0.0.0.0)
*Mar  2 02:15:21.510:  ref 00000000.00000000 (00:00:00.000 UTC Mon Jan 1 1900)
*Mar  2 02:15:21.510:  org C02AB439.4DA1B594 (02:15:21.303 UTC Sat Mar 2 2002)
*Mar  2 02:15:21.510:  rec DC86B430.D46F260E (22:31:12.829 UTC Wed Mar 29 2017)
*Mar  2 02:15:21.510:  xmt DC86B430.D472979E (22:31:12.829 UTC Wed Mar 29 2017)
*Mar  2 02:15:21.514:  inp C02AB439.8260000E (02:15:21.509 UTC Sat Mar 2 2002)[OK]
R3#
R3#

This is what is driving me crazy, is it is receiving the correct time, but then reverts itself back to its old time.

So I will maybe revisit this quickly between R1 and R5 directly connected over a FastEthernet connection to see if it works any better on there, otherwise a few take aways from NTP:

  • Authentication doesn’t really force authentication, so use ACL’s to limit access
  • ACL command to apply to ntp is “ntp access-group #” from global config
  • On the interface level, at this point theoretically, you shouldn’t need to configure an NTP Server but just an interface level command “ntp (broadcast/multicast) client” to listen for NTP broadcasts from a master on that interface
  • NTP can also be disabled (which I did not get to) on the interface level in place of where you configure it to be broadcast or multicast
  • For all intensive purposes, over an NBMA, the server/client model is the way to go

I am going to go blink a few times before bed now, I am not sure exactly the subject matter of the next section, however I really would like to go over this subject again on just an Ethernet segment between R1 and R5 to show how it actually works.

I did want to keep this post up, just to show behaviors of ntp (configurations and command variables of course), but also its odd behaviors for me on my home lab, I assume in the Cisco exam environment they won’t have a shoddy 2621XM Frame Switch serving as their NBMA (I hope) 🙂 That is all for now!

NTP Authentication and ACL configuration, odd behaviors explained, and issues to troubleshoot as always!

OSPF_Base_Topology_NTP

So to begin this, I apparently completely spaced writing R1 so nothing got saved, and we are back in time again:

R1#sh clock
*22:38:45.666 UTC Fri Mar 1 2002
R1#

So I’ve decided to go right into Authentication first, as it’s fairly straight forward with an odd behavior to note, then continue my battle with R4 over the virtual-link to see if I can maybe use the “peer” command between R3 and R4 to resolve the issue.

So I’ve removed all NTP settings from all routers involved in the last lab, as I will need to reconfigure them for authentication. Now to configure authentication, it actually only takes 3 commands on the Master / Server, but 4 commands on the clients as seen here:

R1(config)#ntp authenticate
R1(config)#

^This command sets NTP authentication to run

R1(config)#ntp authentication-key ?
  <1-4294967295>  Key number

R1(config)#ntp authentication-key 1 ?
  md5  MD5 authentication

R1(config)#ntp authentication-key 1 md5 ?
  WORD  Authentication key

R1(config)#ntp authentication-key 1 md5 CCNP ?
  <0-4294967295>  Authentication key encryption type
  <cr>

R1(config)#ntp authentication-key 1 md5 CCNP
R1(config)#

^This command is literally one way to type it, straight forward, CCNP is my keys “password” to authenticate to NTP clients. I have no idea what the last value is, so I will just leave it as configured.
R1(config)#ntp trusted-key ?
  <1-4294967295>  Key number

R1(config)#ntp trusted-key 1 ?
  <cr>

R1(config)#ntp trusted-key 1
R1(config)#

^Again just a very straight forward command, just identifying which one of its keys is a trusted key.

And that is it for the server, it is now “offering” authentication for NTP to potential clients, which sounds odd for authentication as it definitely should.

So on R3 I repeat the same thing:

R3(config)#ntp authenticate
R3(config)#ntp authentication-key 1 md5 CCNP
R3(config)#ntp trusted-key 1
R3(config)#ntp server 172.12.123.1 ? <– The 4th command that is required for clients!
  key      Configure peer authentication key
  prefer   Prefer this peer when possible
  source   Interface for source address
  version  Configure NTP version
  <cr>

R3(config)#ntp server 172.12.123.1 key ?
  <0-4294967295>  Peer key number

R3(config)#ntp server 172.12.123.1 key 1 ?
  prefer   Prefer this peer when possible
  source   Interface for source address
  version  Configure NTP version
  <cr>

R3(config)#ntp server 172.12.123.1 key 1
R3(config)#

So really the NTP clients ONLY additional command to get its time from the server in any case is that mysterious 4th command while R1 has 3 (not including the clock set again).

After waiting a few minutes to see R3 populate, I realized two things: 1. I forgot that yesterday R1 had no neighbor statements for OSPF on my Hub for my spoke routers, and 2. I forgot “ntp master 1” on R1.

So really to set up its 4 commands on each if you include the ntp master on the time server, however it is 3 and 4 if you assume that is part of the normal configuration and adding “key 1” to the “ntp server x.x.x.x …” command.

Now that we’ve beat that horse to death, lets see whats happening on R3:

R3(config)#do sh ntp assoc

      address         ref clock     st  when  poll reach  delay  offset    disp
*~172.12.123.1     .LOCL.            1     1    64   17    65.4   -1.01  1877.2
 * master (synced), # master (unsynced), + selected, – candidate, ~ configured
R3(config)#do sh clock
18:32:54.346 UTC Tue Mar 21 2017
R3(config)#

Alright so that is now working as expected, and I just reconfigured R4 pointed at 172.12.123.1 because it makes no sense from yesterday that it cannot sync up with R1 (which I will visit shortly), but I also configured R2 with absolutely no authentication commands and shortly after I pointed it at R1 as the Time Server we get this:

R2(config)#do sh ntp assoc

      address         ref clock     st  when  poll reach  delay  offset    disp
*~172.12.123.1     .LOCL.            1     9    64  377    53.7   -0.43     0.3
 * master (synced), # master (unsynced), + selected, – candidate, ~ configured
R2(config)#do sh clock
18:39:25.526 UTC Tue Mar 21 2017
R2(config)#

So this is the weird thing with NTP Authentication, is when I say it is “offered” when set by the Master, it is usable without authenticating which sort of defeats the concept of authentication completely – REMEMBER THIS FOR EXAM DAY!

In not so red of text, a client can be configured with no authentication to the time server, and it will still get time from that server (defeating the purpose of authentication).

So I configured R4 with authentication commands and pointed it to R1, and to my surprise:

ASR#4
[Resuming connection 4 to r4 … ]

R4(config)#do sh ntp assoc

address         ref clock       st   when   poll reach  delay  offset   disp
*~172.12.123.1    .LOCL.           1     43     64    77 64.412 -38.023 188.77
* sys.peer, # selected, + candidate, – outlyer, x falseticker, ~ configured
R4(config)#
R4(config)#do sh ntp status
Clock is synchronized, stratum 2, reference is 172.12.123.1
nominal freq is 250.0000 Hz, actual freq is 250.0000 Hz, precision is 2**24
reference time is DC7BF1C5.0072A487 (18:39:01.001 UTC Tue Mar 21 2017)
clock offset is -38.0238 msec, root delay is 64.41 msec
root dispersion is 506.73 msec, peer dispersion is 5.72 msec
loopfilter state is ‘CTRL’ (Normal Controlled Loop), drift is -0.000000202 s/s
system poll interval is 64, last update was 471 sec ago.
R4(config)#

Hooray! No nitty gritty troubleshooting, the lab must know I feel like I am getting sick too!

In the above output of the two “show” verification commands we know of thus far, you see nothing about NTP authentication, but it is all in the “detail” so to say:

R4(config)#do sh ntp assoc detail
172.12.123.1 configured, authenticated, our_master, sane, valid, stratum 1
ref ID .LOCL., time DC7BF403.9FB9E5B6 (18:48:35.623 UTC Tue Mar 21 2017)
our mode client, peer mode server, our poll intvl 64, peer poll intvl 64
root delay 0.00 msec, root disp 0.03, reach 377, sync dist 67.60
delay 64.58 msec, offset -101.6407 msec, dispersion 2.66
precision 2**18, version 4
org time DC7BF405.D9A56AD6 (18:48:37.850 UTC Tue Mar 21 2017)
rec time DC7BF405.FBEF0202 (18:48:37.984 UTC Tue Mar 21 2017)
xmt time DC7BF405.EB34C2F5 (18:48:37.918 UTC Tue Mar 21 2017)
filtdelay =    64.58   64.68   65.65   64.73   64.64   64.76   64.60   64.63
filtoffset = -101.64  -95.55  -87.40  -80.41  -72.88  -65.64  -58.45  -51.39
filterror =     0.00    0.94    1.90    2.88    3.85    4.83    5.80    6.76
minpoll = 6, maxpoll = 10

R4(config)#

I tripped over this message a couple times, because the huge output made me think “sh ntp status” however it is “sh ntp assoc detail” and NOT detail”s”!

So back to authentication not really making hosts authenticate to use the server as a time source, to limit hosts receiving time from our NTP to who we want, Access-Lists’s come to  the rescue!

So the creation of the access-list:

R1(config)#access-list 10 permit 172.12.123.0 0.0.0.255
R1(config)#access-list 10 permit 172.12.34.0 0.0.0.255

Pretty simple, allows R2 / R3 / R4 to get time from R1 with of course the implicit deny at the end to deny all other networks, and now for the NTP portion of applying the ACL:

R1(config)#ntp ?
  access-group        Control NTP access
  authenticate        Authenticate time sources
  authentication-key  Authentication key for trusted time sources
  broadcastdelay      Estimated round-trip delay
  clock-period        Length of hardware clock tick
  logging             Enable NTP message logging
  master              Act as NTP master clock
  max-associations    Set maximum number of associations
  peer                Configure NTP peer
  server              Configure NTP server
  source              Configure interface for source address
  trusted-key         Key numbers for trusted time sources

R1(config)#ntp access-group ?
  peer        Provide full access
  query-only  Allow only control queries
  serve       Provide server and query access
  serve-only  Provide only server access

R1(config)#ntp access-group serve ?
  <1-99>       Standard IP access list
  <1300-1999>  Standard IP access list (expanded range)

R1(config)#ntp access-group serve 10

And that officially applies it, so I just reloaded R2 / R3 / R4 to see how they would come back up and get their time, meanwhile I went on the 172.12.23.0 Ethernet segment and set SW1 to also point to 172.12.123.1 as its NTP server and give it some time to sync until my routers reloaded.

Now all routers have reloaded, lets go around the room, and see who has the correct time:

R2#sh clock
18:34:48.760 UTC Wed Mar 22 2017
R2#
ASR#3
[Resuming connection 3 to r3 … ]

R3#sh clock
18:35:04.335 UTC Wed Mar 22 2017
R3#
ASR#4
[Resuming connection 4 to r4 … ]

R4#sh clock
18:35:14.577 UTC Wed Mar 22 2017
R4#
ASR#5
[Resuming connection 5 to sw1 … ]

SW1#sh clock
*01:30:37.361 UTC Mon Mar 1 1993
SW1#sh ntp assoc

      address         ref clock     st  when  poll reach  delay  offset    disp
 ~172.12.123.1     0.0.0.0          16     –    64    0     0.0    0.00  16000.
 * master (synced), # master (unsynced), + selected, – candidate, ~ configured

Our switch is existing in probably my favorite era of my lifetime, the 90’s, and will remain there unless we let it back into the present (but that might be cruel) 🙂

So I wanted to post up the output of SW1’s “sh ntp status” and “sh ntp assoc det” because there is what I found in my voice days hilarious, but serious NTP status:
SW1#sh ntp status
Clock is unsynchronized, stratum 16, no reference clock
nominal freq is 119.2092 Hz, actual freq is 119.2092 Hz, precision is 2**18
reference time is 00000000.00000000 (00:00:00.000 UTC Mon Jan 1 1900)
clock offset is 0.0000 msec, root delay is 0.00 msec
root dispersion is 0.00 msec, peer dispersion is 0.00 msec

Not a whole lot there, except showing the clock isn’t synchronized, but with “sh ntp assoc det” we see a very… odd and awesome way of putting it:
SW1#sh ntp assoc det
172.12.123.1 configured, insane, invalid, unsynced, stratum 16
ref ID 0.0.0.0, time 00000000.00000000 (00:00:00.000 UTC Mon Jan 1 1900)
our mode client, peer mode unspec, our poll intvl 64, peer poll intvl 64
root delay 0.00 msec, root disp 0.00, reach 0, sync dist 0.000
delay 0.00 msec, offset 0.0000 msec, dispersion 16000.00
precision 2**5, version 3
org time 00000000.00000000 (00:00:00.000 UTC Mon Jan 1 1900)
rcv time 00000000.00000000 (00:00:00.000 UTC Mon Jan 1 1900)
xmt time AF3BE5E5.8494D4E8 (01:31:17.517 UTC Mon Mar 1 1993)
filtdelay =     0.00    0.00    0.00    0.00    0.00    0.00    0.00    0.00
filtoffset =    0.00    0.00    0.00    0.00    0.00    0.00    0.00    0.00
filterror =  16000.0 16000.0 16000.0 16000.0 16000.0 16000.0 16000.0 16000.0

SW1#

This switch is INSANE. When I first saw that, it was so awesome, the terminology just tickled me right in my Cisco soft spot. So what is “insane” you might ask (in terms of Cisco switches)? It is when the a network device is configured with an NTP server to get time from, but cannot reach that time source.

So lets see if we can get this switch SANE, and back to the present date. My brain is already fried like chicken so I accidentally exited until I found myself back at “user priv” (square one) mode, so I reconfigured SW1 to point at 172.12.123.1 for NTP and lets see:

SW1#conf t
Enter configuration commands, one per line.  End with CNTL/Z.
SW1(config)#ntp server 172.12.123.1
SW1(config)#do sh clock
*01:46:27.731 UTC Mon Mar 1 1993

Hmm.. that’s been cooking for about 2-3 minutes prior to that output, so it’s time to investigate this, and as I’ve learned with R4 I need to make sure I can ping R1 first:

SW1#ping 172.12.123.1
% Unrecognized host or address, or protocol not running.

Crap. So I forgot I only gave this a name when I brought it online, so to make it able to ping over to R1 I input the following commands:

SW1#conf t
Enter configuration commands, one per line.  End with CNTL/Z.
SW1(config)#ip routing
SW1(config)#int vlan1
SW1(config-if)#ip address 172.12.23.1 255.255.255.0
SW1(config-if)#no shut
SW1(config-if)#exit
01:50:31: %LINK-3-UPDOWN: Interface Vlan1, changed state to up
01:50:32: %LINEPROTO-5-UPDOWN: Line protocol on Interface Vlan1, changed state to up
SW1(config)#exit
SW1#ping 172.12.123.1

Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 172.12.123.1, timeout is 2 seconds:
…..
Success rate is 0 percent (0/5)
SW1#

So we are routing now, but still no dice, I sense an OSPF network not included issue:

R1#sh ip route ospf
     2.0.0.0/32 is subnetted, 1 subnets
O IA    2.2.2.2 [110/65] via 172.12.123.2, 00:30:22, Serial0/0
     3.0.0.0/32 is subnetted, 1 subnets
O IA    3.3.3.3 [110/65] via 172.12.123.3, 00:30:22, Serial0/0
     4.0.0.0/32 is subnetted, 1 subnets
O IA    4.4.4.4 [110/66] via 172.12.123.3, 00:30:22, Serial0/0
     5.0.0.0/32 is subnetted, 1 subnets
O       5.5.5.5 [110/2] via 172.12.15.5, 01:48:21, FastEthernet0/1
     172.12.0.0/24 is subnetted, 4 subnets
O IA    172.12.34.0 [110/65] via 172.12.123.3, 00:30:22, Serial0/0
O IA    172.12.23.0 [110/65] via 172.12.123.3, 00:30:11, Serial0/0
                    [110/65] via 172.12.123.2, 00:30:11, Serial0/0
     44.0.0.0/32 is subnetted, 1 subnets
O IA    44.44.44.1 [110/66] via 172.12.123.3, 00:30:22, Serial0/0
R1#ping 172.12.23.1

Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 172.12.23.1, timeout is 2 seconds:
…..
Success rate is 0 percent (0/5)
R1#sh access-list
Standard IP access list 10
    10 permit 172.12.123.0, wildcard bits 0.0.0.255 (76 matches)
    20 permit 172.12.34.0, wildcard bits 0.0.0.255 (94 matches)
    30 permit 172.12.23.0, wildcard bits 0.0.0.255
R1#

So in the route table it knows of the 172.12.23.0 network via R2, so I hopped on there:
R2#ping 172.12.23.1

Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 172.12.23.1, timeout is 2 seconds:
!!!!!
Success rate is 100 percent (5/5), round-trip min/avg/max = 1/3/4 ms
R2#ping 172.12.123.1

Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 172.12.123.1, timeout is 2 seconds:
!!!!!
Success rate is 100 percent (5/5), round-trip min/avg/max = 64/66/68 ms
R2#
ASR#5
[Resuming connection 5 to sw1 … ]

SW1#ping 172.12.23.2

Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 172.12.23.2, timeout is 2 seconds:
!!!!!
Success rate is 100 percent (5/5), round-trip min/avg/max = 1/2/8 ms
SW1#

So R2 can ping R1 and SW1, and is the go between for them, and SW1 can ping R2 which is its middle man to R1. So I go back to SW1 to give a traceroute a try to see if it’s even getting a response from R2 when sending traffic:

SW1#traceroute 172.12.123.1

Type escape sequence to abort.
Tracing the route to 172.12.123.1

  1  *  *  *
  2  *  *  *
  3  *  *
SW1#sh ip route

Gateway of last resort is not set

     172.12.0.0/24 is subnetted, 1 subnets
C       172.12.23.0 is directly connected, Vlan1

There’s the issue, it has no route to 172.12.123.1, so at this point of going brain dead from work / study I will just create the static route to bring sanity back to this switch:

SW1(config)#ip route 172.12.123.0 255.255.255.0 172.12.23.2
SW1(config)#exit
SW1#ping 172.12.123.1

Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 172.12.123.1, timeout is 2 seconds:
!!!!!

I love when logic works, makes my head hurt less. So I issued a write and reload, and after it booted back up, this is how the clock is now looking:

SW1#sh clock
*00:01:08.501 UTC Mon Mar 1 1993
SW1#sh ntp assoc

      address         ref clock     st  when  poll reach  delay  offset    disp
 ~172.12.123.1     0.0.0.0          16     –    64    0     0.0    0.00  16000.
 * master (synced), # master (unsynced), + selected, – candidate, ~ configured
SW1#

So starting to lose my mind wondering why on Earth this thing will not sync, I got on R1 and ran “debug ntp packet” and I’ll spare you all the output EXCEPT WHEN SW1 FINALLY HIT THIS SUCKER AND GOT BROUGHT BACK FROM THE FUTURE… OR PAST… WHICHEVER:

.Mar 22 19:16:31.578: NTP: rcv packet from 172.12.23.1 to 172.12.123.1 on Serial0/0:
.Mar 22 19:16:31.578:  leap 3, mode 3, version 3, stratum 0, ppoll 64
.Mar 22 19:16:31.578:  rtdel 0000 (0.000), rtdsp 10001 (1000.015), refid 00000000 (0.0.0.0)
.Mar 22 19:16:31.578:  ref 00000000.00000000 (00:00:00.000 UTC Mon Jan 1 1900)
.Mar 22 19:16:31.578:  org 00000000.00000000 (00:00:00.000 UTC Mon Jan 1 1900)
.Mar 22 19:16:31.582:  rec 000
R1#00000.00000000 (00:00:00.000 UTC Mon Jan 1 1900)
.Mar 22 19:16:31.582:  xmt AF3BD148.1181126D (00:03:20.068 UTC Mon Mar 1 1993)
.Mar 22 19:16:31.582:  inp DC7D4C0F.945A1DF3 (19:16:31.579 UTC Wed Mar 22 2017)
.Mar 22 19:16:31.582: NTP: stateless xmit packet to 172.12.23.1:
.Mar 22 19:16:31.582:  leap 3, mode 4, version 3, stratum 0, ppoll 64
.Mar 22 19:16:31.582:  rtdel 0000 (0.000), rtdsp 6002 (375.031), refid 4C4F434C (76.79.67.76)
.Mar 22 19:16:31.586:  ref DC7D400B.EEB63084 (18:25:15.932 UTC Wed Mar 22 2017)
.Mar 22 19:16:31.586:  org AF3BD148.1181126D (00:03:20.068 UTC Mon Mar 1 1993)
.Mar 22 19:16:31.586:  rec DC7D4C0F.945A1DF3 (19:16:31.579 UTC Wed Mar 22 2017)
.Mar 22 19:16:31.586:  xmt DC7D4C0F.95A82567 (19:16:31.584 UTC Wed Mar 22 2017)
R1#u all
R1#

I left the big crap ton of output to see the exchange and references, I am not sure what most of it means, but it does show the switches old time and the current time being exchanged (as well as referring to 1900 for some odd reason).

So when we go back to SW1, we should finally have some sanity, before I lose mine:

SW1#sh clock
*00:10:24.859 UTC Mon Mar 1 1993
SW1#sh ntp assoc det
172.12.123.1 configured, insane, invalid, unsynced, stratum 16
ref ID 0.0.0.0, time 00000000.00000000 (00:00:00.000 UTC Mon Jan 1 1900)
our mode client, peer mode unspec, our poll intvl 64, peer poll intvl 64
root delay 0.00 msec, root disp 0.00, reach 0, sync dist 0.000
delay 0.00 msec, offset 0.0000 msec, dispersion 16000.00
precision 2**5, version 3
org time DC7D4D8F.933D330C (19:22:55.575 UTC Wed Mar 22 2017)
rcv time AF3BD2C8.1E2EB9CE (00:09:44.117 UTC Mon Mar 1 1993)
xmt time AF3BD2C8.10B78FA5 (00:09:44.065 UTC Mon Mar 1 1993)
filtdelay =     0.00    0.00    0.00    0.00    0.00    0.00    0.00    0.00
filtoffset =    0.00    0.00    0.00    0.00    0.00    0.00    0.00    0.00
filterror =  16000.0 16000.0 16000.0 16000.0 16000.0 16000.0 16000.0 16000.0

SW1#

What??? If you examine the details, other than insanity still being awesome terminology, you can see what I’ve highlighted in red that it does have the correct network time in the output which its getting from R1, but show clock does not show the correct time.

So at this point, I did try to put “router ospf 1” on the switch which it does show as a valid command, but it does not drop me into ospf configuration mode to see if that gives it the kick it needs. You can even see it getting hits on R1’s ACL for NTP requests:

R1#sh access-list
Standard IP access list 10
    10 permit 172.12.123.0, wildcard bits 0.0.0.255 (130 matches)
    20 permit 172.12.34.0, wildcard bits 0.0.0.255 (105 matches)
    30 permit 172.12.23.0, wildcard bits 0.0.0.255 (15 matches)
R1#

So at this point, the next segment of my course is running NTP in broadcast mode, and I want to see what that has to say there to see if we can maybe salvage this SW1 situation.

NTP – Network Time Protocol discussion, configuration, and some NTP / routing troubleshooting as usual!

OSPF_Base_Topology_NTP

For this lab I will be using this OSPF Topology, which has the virtual-link to R4 to bring it into the entirety of the network, and to demonstrate how to configure each of these routers so they keep in sync with the entire network via NTP even if their primary time source goes down.

So this is a subject near and dear to my heart, working previously a lot with Cisco Voice systems / servers, if the network NTP is not near perfectly synchronized your phones / voicemails / servers are going to go absolutely haywire.

In the CCNP ROUTE context, I believe it is mostly beneficial for troubleshooting so your logging is on the same time, so you don’t have to compare different time frames to guess where the event happened on different devices because they had different times – I know this because network time on Small / Medium sized businesses can be off an amazing amount of time.

To begin any NTP discussion you have to first begin with “Stratum” and what it is. Stratum is the metric (like hop count) as to how close to a Stratum 0 device you are to gauge how accurate your time is, this again is especially important for VOIP as I believe only Stratum 3 or lower is acceptable for time differential.

So with Stratum, the lower the better, the best obviously being Stratum 0 which are actually referred to as atomic clocks that are the size of datacenters on naval bases. You will not be able to connect directly to a stratum 0 device (or make a router stratum 0 as you will see), but there are “Time Servers” on the internet that are Stratum 1, that you can point your edge device to (or multiple of them in case one disappears).

Each hop you get away from that Time Server, the more your “Stratum #” will increase when you are running show commands for NTP on your router, and goes up to a maximum of 15 which means as unreliable as it gets before Stratum 16 which means unreachable or unreliable. Now for a couple important notes before we dive into some configuration:

  • When you “write erase” / “reload” a device to wipe it, you are wiping the time, so in the real world or in your lab don’t forget it needs to be reset or chaos will ensue!
  • NTP uses UDP port 123, so do block it on the devices on your network

Also worth a bullet point style explanation, are the 3 different types of NTP router:

  • NTP Server – Set time on this device, it will send out Time Sync messages to NTP clients on the network
  • NTP Client – Receives Time Sync messages from Server, DOES NOT send time sync messages back to server
  • NTP Peer – Can be both Client and Peer, Peers can share time with eachother

NTP can be run in broadcast mode, or multicast mode, depending on your network needs. Its odd that this part of the topic is kind of just left at you needing to figure out what works with your network best, so I imagine trying to lab it over an NBMA running OSPF should be fun!

Also a note on configuring an NTP Server or “Master” as your edge device, it is highly recommended that you not only use a public time server(s) as your time source as opposed to setting it, but also it is necessary to use authentication and / or ACL’s to stop other routers from using ours as an NTP time source even if it is just for time synching reasons we don’t want the extra workload on the edge device.

Ahhhh, the perfect segway into the security side of NTP 🙂 However for all this discussion, there has been no labbing, and not labbing bores me to tears so lets get into some configuration and see what we can break:

R1#sh ntp assoc
R1#sh clock
*23:21:30.839 UTC Fri Mar 1 2002
R1#

As show with “sh ntp associations” we have nothing configured with any other routers (including this one), and we have traveled back 15 years to 2002. So given that my lab is connected to the internet, I am going to set R1 as the NTP Server / Master, and all other routers as clients / peers.

So first, we need to get R1 rocking as the NTP Master of the network, so lets get that done:

(I actually forgot how to set the time and referred back to my time-based ACL’s post, and that’s why I say it over and over it’s so important to start your own blog or something equal in being able to refer back to examples of these things quickly!!)

R1#clock set 19:43:00 20 mar 2017
R1#
*Mar 20 19:43:00.000: %SYS-6-CLOCKUPDATE: System clock has been updated from 23:30:06 UTC Fri Mar 1 2002 to 19:43:00 UTC Mon Mar 20 2017, configured from console by console.
R1#conf t
Enter configuration commands, one per line.  End with CNTL/Z.
R1(config)#ntp ?
  access-group        Control NTP access
  authenticate        Authenticate time sources
  authentication-key  Authentication key for trusted time sources
  broadcastdelay      Estimated round-trip delay
  clock-period        Length of hardware clock tick
  logging             Enable NTP message logging
  master              Act as NTP master clock
  max-associations    Set maximum number of associations
  peer                Configure NTP peer
  server              Configure NTP server
  source              Configure interface for source address
  trusted-key         Key numbers for trusted time sources

R1(config)#ntp master ?
  <1-15>  Stratum number
  <cr>

R1(config)#ntp master 1 ?
  <cr>

R1(config)#ntp master
R1(config)#

A couple things here, I left the ? output after NTP so you can see the command modifiers of which I used master, but as I went on also that my options were 1-15 from most trusted to least. I was going to make it Stratum 1 but I left off the # as I curious what Stratum # it gets by default when configured as an NTP master:

R1(config)#do sh ntp assoc

      address         ref clock     st  when  poll reach  delay  offset    disp
*~127.127.7.1      127.127.7.1       7    51    64  377     0.0    0.00     0.0
 * master (synced), # master (unsynced), + selected, – candidate, ~ configured
R1(config)#

Ha, so this router isn’t giving me too much credibility, under “st” which refers to Stratum # is 7, so I am neither very trusted or very unreliable. I’ll switch it back so that it is a Stratum 1, because I like my NTP master of my network to have some credibility.

Also what I have highlighted in red is very important, especially the * when it comes to clients, because that means they are fully synced with that address. Being this is the master it uses its loopback 127.127.7.1, so if you see * in “sh ntp assoc” with a loopback address you know that router is set as the NTP Master.

Now lets configure R2 and R3 over our NBMA and our Area 0 in OSPF as clients of R1:

R2#conf t
Enter configuration commands, one per line.  End with CNTL/Z.
R2(config)#ntp ?
  access-group        Control NTP access
  authenticate        Authenticate time sources
  authentication-key  Authentication key for trusted time sources
  broadcastdelay      Estimated round-trip delay
  clock-period        Length of hardware clock tick
  logging             Enable NTP message logging
  master              Act as NTP master clock
  max-associations    Set maximum number of associations
  peer                Configure NTP peer
  server              Configure NTP server
  source              Configure interface for source address
  trusted-key         Key numbers for trusted time sources

R2(config)#ntp server ?
  Hostname or A.B.C.D  IP address of peer
  vrf                  VPN Routing/Forwarding Information

R2(config)#ntp server 172.12.123.1 ?
  key      Configure peer authentication key
  prefer   Prefer this peer when possible
  source   Interface for source address
  version  Configure NTP version
  <cr>

R2(config)#ntp server 172.12.123.1
R2(config)#
ASR#3
[Resuming connection 3 to r3 … ]

R3#conf t
Enter configuration commands, one per line.  End with CNTL/Z.
R3(config)#ntp server 172.12.123.1
R3(config)#do sh ntp assoc

      address         ref clock     st  when  poll reach  delay  offset    disp
 ~172.12.123.1     0.0.0.0          16     –    64    0     0.0    0.00  16000.
 * master (synced), # master (unsynced), + selected, – candidate, ~ configured
R3(config)#

I’ve highlighted the “prefer” option after pointing at 172.12.123.1, to show that it is there, and is used when assigning multiple NTP servers as backup but you prefer to use a specific server as the time source.

Also highlighted in red, before it could sync I did a quick “sh ntp assoc” to demonstrate what it looks like when a router is not synced, and why I stressed seeing the * next to the IP address means that it is fully synced.

Also, that Stratum 16 is the equivalent to RIP’s metric of 16, it’s not even barely reliable but is actually an invalid time source at Stratum 16 – This is important to note!

So let’s see if we have some synchronization going on with R2 and R3:

R2(config)#do sh ntp assoc

      address         ref clock     st  when  poll reach  delay  offset    disp
*~172.12.123.1     .LOCL.            1     1    64  377    53.5   -1.53     0.4
 * master (synced), # master (unsynced), + selected, – candidate, ~ configured
R2(config)#do sh clock
20:03:17.432 UTC Mon Mar 20 2017
R2(config)#
ASR#3
[Resuming connection 3 to r3 … ]

R3(config)#do sh ntp assoc

      address         ref clock     st  when  poll reach  delay  offset    disp
*~172.12.123.1     .LOCL.            1    18    64  377    53.5   -3.38     0.5
 * master (synced), # master (unsynced), + selected, – candidate, ~ configured
R3(config)#do sh clock
20:03:39.637 UTC Mon Mar 20 2017
R3(config)#

Both are looking good, except I do see it references this again as UTC time, and for lab purposes I do not intend to spend time digging into getting the time into my timezone 🙂

Another important command to know for checking NTP settings on the local router is “sh ntp status” as demonstrated here:

R3#sh ntp status
Clock is synchronized, stratum 2, reference is 172.12.123.1
nominal freq is 249.5901 Hz, actual freq is 249.5903 Hz, precision is 2**18
reference time is DC7AB544.FFF63BFD (20:08:36.999 UTC Mon Mar 20 2017)
clock offset is -3.8317 msec, root delay is 53.48 msec
root dispersion is 4.06 msec, peer dispersion is 0.18 msec
R3#

So we see it’s synchronized now back to the server from R2 and R3 which right now is just using the Client / Server model, however I had to have my first Derp of the night – Configuring R4 correctly for NTP however not testing connectivity to the server address:

R4(config)#do sh ntp assoc

  address         ref clock       st   when   poll reach  delay  offset   disp
 ~172.12.123.1    .INIT.          16      –     64     0  0.000   0.000 15937.
 * sys.peer, # selected, + candidate, – outlyer, x falseticker, ~ configured
R4(config)#do ping 172.12.123.1
Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 172.12.123.1, timeout is 2 seconds:
…..
Success rate is 0 percent (0/5)
R4(config)#

The sad part is I’ve been waiting for about 5 minutes or so for that to sync, because I know it can take 5+ minutes, but a ping even before ntp configuration would have been a good way to start the configuration! So lets take a look at R3:

R3#sh ip int bri
Interface                  IP-Address      OK? Method Status                Protocol
FastEthernet0/0            172.12.23.3     YES NVRAM  up                    up
FastEthernet0/1            172.12.34.3     YES NVRAM  up                    up
Serial0/2                  172.12.123.3    YES NVRAM  up                    up
Serial0/3                  unassigned      YES NVRAM  administratively down down
Loopback3                  3.3.3.3         YES NVRAM  up                    up
R3#ping 4.4.4.4

Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 4.4.4.4, timeout is 2 seconds:
!!!!!
Success rate is 100 percent (5/5), round-trip min/avg/max = 1/2/4 ms
R3#sh ip ospf nei

Neighbor ID     Pri   State           Dead Time   Address         Interface
44.44.44.1        0   FULL/  –           –        172.12.34.4     OSPF_VL0
2.2.2.2           1   FULL/BDR        00:00:38    172.12.23.2     FastEthernet0/0
44.44.44.1        1   FULL/DR         00:00:38    172.12.34.4     FastEthernet0/1
R3#sh ip proto
Routing Protocol is “ospf 1”
  Outgoing update filter list for all interfaces is not set
  Incoming update filter list for all interfaces is not set
  Router ID 3.3.3.3
  It is an area border router
  Number of areas in this router is 4. 4 normal 0 stub 0 nssa
  Maximum path: 4
  Routing for Networks:
    3.3.3.3 0.0.0.0 area 3
    172.12.23.0 0.0.0.255 area 23
    172.12.34.0 0.0.0.255 area 34
    172.12.123.0 0.0.0.255 area 0
 Reference bandwidth unit is 100 mbps
  Routing Information Sources:
    Gateway         Distance      Last Update
    44.44.44.1           110      01:58:30
  Distance: (default is 110)

R3#sh ip ospf virtual-link
Virtual Link OSPF_VL0 to router 44.44.44.1 is up
  Run as demand circuit
  DoNotAge LSA allowed.
  Transit area 34, via interface FastEthernet0/1, Cost of using 1
  Transmit Delay is 1 sec, State POINT_TO_POINT,
  Timer intervals configured, Hello 10, Dead 40, Wait 40, Retransmit 5
    Hello due in 00:00:03
    Adjacency State FULL (Hello suppressed)
    Index 1/2, retransmission queue length 0, number of retransmission 0
    First 0x0(0)/0x0(0) Next 0x0(0)/0x0(0)
    Last retransmission scan length is 0, maximum is 0
    Last retransmission scan time is 0 msec, maximum is 0 msec
R3#ping 172.12.123.1

Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 172.12.123.1, timeout is 2 seconds:
!!!!!
Success rate is 100 percent (5/5), round-trip min/avg/max = 64/66/68 ms
R3#

Uhhhh…. So R4 has some issue? It should have been ruled out by R3 pinging it’s loopback, but I’ll play ball, lets take a look:

R4#traceroute 172.12.123.1
Type escape sequence to abort.
Tracing the route to 172.12.123.1
VRF info: (vrf in name/id, vrf out name/id)
  1 172.12.34.3 4 msec 0 msec 4 msec
  2  *  *  *
  3  *  *  *
  4  *  *
R4#
R4#sh ip route ospf

Gateway of last resort is not set

      3.0.0.0/32 is subnetted, 1 subnets
O IA     3.3.3.3 [110/2] via 172.12.34.3, 02:04:27, FastEthernet0/1
      172.12.0.0/16 is variably subnetted, 4 subnets, 2 masks
O IA     172.12.23.0/24 [110/2] via 172.12.34.3, 02:03:49, FastEthernet0/1
O        172.12.123.0/24 [110/65] via 172.12.34.3, 02:04:27, FastEthernet0/1
R4#
R4#ping 172.12.123.3
Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 172.12.123.3, timeout is 2 seconds:
!!!!!
Success rate is 100 percent (5/5), round-trip min/avg/max = 1/2/4 ms
R4#ping 172.12.123.1
Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 172.12.123.1, timeout is 2 seconds:
…..
Success rate is 0 percent (0/5)
R4#

So it is being processed by R3, as it can ping the serial interface on the NBMA network, so the issue is either somewhere on the frame switch or on R1. It’s too late and I’m too fried to really want to dig into this weird behavior (I will just wipe / reconfigure them if push comes to shove), but for the heck of it lets look at R1:

!
interface Serial0/1
 ip address 172.12.13.1 255.255.255.252
 clock rate 2000000
!
router ospf 1
 log-adjacency-changes
 area 34 virtual-link 44.44.44.1 <— What???
 network 1.1.1.1 0.0.0.0 area 1
 network 172.12.15.0 0.0.0.255 area 15
 network 172.12.123.0 0.0.0.255 area 0
!
!

I am wondering if I was so tired configuring this, that I entered that command on R1, and that is what is jamming up the traffic, lets put a stop to this silliness:

R1#conf t
Enter configuration commands, one per line.  End with CNTL/Z.
R1(config)#router ospf 1
R1(config-router)#no area 34 virtual-link 44.44.44.1
R1(config-router)#

I can confirm it is now gone, lets see about some pings:

R1(config-router)#do ping 4.4.4.4

Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 4.4.4.4, timeout is 2 seconds:
…..
Success rate is 0 percent (0/5)
R1(config-router)#do ping 172.12.34.4

Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 172.12.34.4, timeout is 2 seconds:
…..
Success rate is 0 percent (0/5)
R1(config-router)#do ping 172.12.34.3

Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 172.12.34.3, timeout is 2 seconds:
…..
Success rate is 0 percent (0/5)
R1(config-router)#do ping 172.12.123.3

Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 172.12.123.3, timeout is 2 seconds:
!!!!!
Success rate is 100 percent (5/5), round-trip min/avg/max = 64/66/68 ms
R1(config-router)#

So R3 will process packets from R4 to the IP on its NBMA serial Interface, but R1 cannot ping to the Fa0/1 interface of Area 34. Let me stare at the show run for a moment, I am losing my sense of humor now 🙂

Wow, I must have been half asleep, I found some leftover access-lists from when I was doing that section on R3 that was messing with traffic, let me remove these AND LET THE TRAFFIC FLOWETH THROUGHOUT THE NETWORK!!! :

R3#conf t
Enter configuration commands, one per line.  End with CNTL/Z.
R3(config)#no access-list 15
R3(config)#no access-list 111
R3(config)#int fa0/1
R3(config-if)#no ip access-group 111 in
R3(config-if)#no ip access-group 15 out
R3(config-if)#
ASR#4
[Resuming connection 4 to r4 … ]

R4#ping 172.12.123.1
Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 172.12.123.1, timeout is 2 seconds:
…..
Success rate is 0 percent (0/5)

GAH!!!

So at this point, I’m giving R3 a write / reload, and if R4 cannot ping the server, I’ll have to wipe and confirm cabling / reconfigure the network as the previously labs are having impact somewhere in the network.

So as R3 loaded back up, and I watched the Adjacency form, I never saw the relationship back to Area 0 on the NBMA network form (I even waited the 10 years it takes over serial links):

R3>en
Password:
R3#
Mar 20 21:16:43.574: %OSPF-5-ADJCHG: Process 1, Nbr 44.44.44.1 on FastEthernet0/1 from LOADING to FULL, Loading Done
R3#
Mar 20 21:16:46.915: %OSPF-5-ADJCHG: Process 1, Nbr 2.2.2.2 on FastEthernet0/0 from LOADING to FULL, Loading Done
R3#
Mar 20 21:16:58.655: %OSPF-5-ADJCHG: Process 1, Nbr 44.44.44.1 on OSPF_VL0 from LOADING to FULL, Loading Done
R3#sh ip ospf nei

Neighbor ID     Pri   State           Dead Time   Address         Interface
44.44.44.1        0   FULL/  –           –        172.12.34.4     OSPF_VL0
2.2.2.2           1   FULL/DR         00:00:37    172.12.23.2     FastEthernet0/0
44.44.44.1        1   FULL/DR         00:00:37    172.12.34.4     FastEthernet0/1
R3#

So now it is an OSPF issue that R4 is not getting it’s time, which raises another good point, that time servers should not be reliant (if possible) on a dynamic protocol to reach it’s time source.

So for the hell of it since its this late and I’m already this friend, I might as well try to see this though to the gruesome end, so I took at what routes it DOES see and this is what I got:

R1#sh ip route

Gateway of last resort is not set

     1.0.0.0/32 is subnetted, 1 subnets
C       1.1.1.1 is directly connected, Loopback1
     5.0.0.0/32 is subnetted, 1 subnets
O       5.5.5.5 [110/2] via 172.12.15.5, 00:28:51, FastEthernet0/1
     172.12.0.0/24 is subnetted, 2 subnets
C       172.12.15.0 is directly connected, FastEthernet0/1
C       172.12.123.0 is directly connected, Serial0/0
R1#

Nothing over my NBMA. OMG I forgot the neighbor statements on R1 in my tired stupor, *slams head against desk* :

R1(config)#router ospf 1
R1(config-router)#neighbor 172.12.123.2
R1(config-router)#neighbor 172.12.123.3
R1(config-router)#
.Mar 20 21:31:26.994: %OSPF-5-ADJCHG: Process 1, Nbr 2.2.2.2 on Serial0/0 from LOADING to FULL, Loading Done
.Mar 20 21:31:27.102: %OSPF-5-ADJCHG: Process 1, Nbr 3.3.3.3 on Serial0/0 from LOADING to FULL, Loading Done
R1(config-router)#
ASR#4
[Resuming connection 4 to r4 … ]

*Mar 21 01:56:05.859: %OSPF-5-ADJCHG: Process 1, Nbr 3.3.3.3 on OSPF_VL0 from FULL to DOWN, Neighbor Down: Interface down or detached
R4#
*Mar 21 01:56:12.583: %LINEPROTO-5-UPDOWN: Line protocol on Interface FastEthernet0/1, changed state to down
*Mar 21 01:56:12.583: %OSPF-5-ADJCHG: Process 1, Nbr 3.3.3.3 on FastEthernet0/1 from FULL to DOWN, Neighbor Down: Interface down or detached
R4#ping 172.12.123.1
Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 172.12.123.1, timeout is 2 seconds:
!!!!!
Success rate is 100 percent (5/5), round-trip min/avg/max = 64/65/68 ms
R4#sh ntp assoc

  address         ref clock       st   when   poll reach  delay  offset   disp
 ~172.12.123.1    .INIT.          16      –   1024     0  0.000   0.000 15937.
 * sys.peer, # selected, + candidate, – outlyer, x falseticker, ~ configured
R4#sh ntp assoc

  address         ref clock       st   when   poll reach  delay  offset   disp
 ~172.12.123.1    .INIT.          16      –   1024     0  0.000   0.000 15937.
 * sys.peer, # selected, + candidate, – outlyer, x falseticker, ~ configured

So it is just not synching for whatever reason I don’t care to dig in now that its 10pm, so I will continue this battle in my next post, which will include NTP Authentication!

 

SNMP (Simple Network Management Protocol) Fundamentals, very important configuration notes for exam day!

I will keep this brief and right to the points of SNMP, as it is St. Patricks Day night, and I really don’t have any other better plans… I’m just tired from work and its about nap time.

That being said, SNMP is a “polling” protocol used to carry network information traffic on UDP port 161, between an SNMP Manager and SNMP Agents., speaking of which the 3 main components of SNMP are:

  • The SNMP Manager
  • The SNMP Agents
  • The SNMP MIB (Management Information Base)

The MIB is a database that resides on agents, which contains “variables” about the agent, which we will get to variables in a moment, but first lets see the two types of traffic sent from the Manager to its SNMP Agents:

  • GET = A request for some form of information
  • SET = A request to request a certain variable be set to the value indicated in the SET

So from my understanding of this, the Manager defines the information it wants each individual Agent to retain based on what is SET for that Agent, so that it can send a GET at any time to request the defined SET information.

As mentioned above the “polling” for events at set times, which would result in extremely slow notification time if an event were to happen if the polling is not happening every 5 seconds, and if it is polling every 5 seconds the network is going to take a big hit constantly on bandwidth and hardware resources (particularly the Managers).

To get a quick notification to the Manager without overloading it, is to set SNMP Traps on managed devices, which allows the Agent to generate SNMP traffic to send to the Manager if a critical variable changes between GET’s.

On a change of topic, there are 3 different flavors of SNMP currently in use: 1, 2c, and 3.

Versions 1 and 2c DO NOT have authentication and encryption, whereas 3 does, so there is some major security flaws with running them rather than v3 on your network.

If v1 or v2c are used, they should be using something called “SNMP Community strings” which are is a combination of password and authority level, and allows to choose whether its read-only or read-write access.

Now brace yourself for some huge output as I want the modifiers to be seen here, but I will highlight in red what I am using for my input on this router:

R5(config)#snmp-server ?
  chassis-id        String to uniquely identify this chassis
  community         Enable SNMP; set community string and access privs
  contact           Text for mib object sysContact
  context           Create/Delete a context apart from default
  drop              Silently drop SNMP packets
  enable            Enable SNMP Traps
  engineID          Configure a local or remote SNMPv3 engineID
  file-transfer     File transfer related commands
  group             Define a User Security Model group
  host              Specify hosts to receive SNMP notifications
  ifindex           Enable ifindex persistence
  inform            Configure SNMP Informs options
  ip                IP ToS configuration for SNMP traffic
  location          Text for mib object sysLocation
  manager           Modify SNMP manager parameters
  packetsize        Largest SNMP packet size
  queue-length      Message queue length for each TRAP host
  queue-limit       Message queue size for different queues
  source-interface  Assign an source interface
  system-shutdown   Enable use of the SNMP reload command
  tftp-server-list  Limit TFTP servers used via SNMP
  trap              SNMP trap options
  trap-source       Assign an interface for the source address of all traps
  trap-timeout      Set timeout for TRAP message retransmissions
  user              Define a user who can access the SNMP engine
  view              Define an SNMP MIB view

R5(config)#snmp-server community ?
  WORD  SNMP community string

R5(config)#snmp-server community CCNP ?
  <1-99>       Std IP accesslist allowing access with this community string
  <1300-1999>  Expanded IP accesslist allowing access with this community
               string
  WORD         Access-list name
  ipv6         Specify IPv6 Named Access-List
  ro           Read-only access with this community string
  rw           Read-write access with this community string
  view         Restrict this community to a named MIB view
  <cr>

R5(config)#snmp-server community CCNP ro ?
  <1-99>       Std IP accesslist allowing access with this community string
  <1300-1999>  Expanded IP accesslist allowing access with this community
               string
  WORD         Access-list name
  ipv6         Specify IPv6 Named Access-List
  <cr>

R5(config)#snmp-server community CCNP ro 15 ?
  <cr>

R5(config)#snmp-server community CCNP ro 15
R5(config)#

So the above command will allow hosts defined in ACL 15 as permitted to have read-only access to SNMP objects specified by the community string.

With SNMP3, it is more secure, but it is more complex to configure and the commands are a bit more long winded to configure it:

R5(config)#snmp-server group CCNP v3 ?
  auth    group using the authNoPriv Security Level
  noauth  group using the noAuthNoPriv Security Level
  priv    group using SNMPv3 authPriv security level

R5(config)#snmp-server group CCNP v3

With the above output, it’s really about breaking down each line in it entirety, individually:

  1.  auth –  group using the authNoPriv Security Level – As Priv is referring to Privacy or Encryption, it is referring to this option offering Authentication, but no encryption
  2. noauth – group using the noAuthNoPriv Security Level – Really flushing security down the toilet with this option, no Authentication and no Encryption
  3. priv – group using SNMPv3 authPriv security level – As discussed in the beginning of this post, and indicated by authPriv, we do have Authentication and Encryption with this option using SNMPv3

Notice there wasn’t a <cr> there, so lets look at the continuation of the output of ? after that command to see what the options are:

R5(config)#snmp-server group CCNP v3 priv ?
  access   specify an access-list associated with this group
  context  specify a context to associate these views for the group
  match    context name match criteria
  notify   specify a notify view for the group
  read     specify a read view for the group
  write    specify a write view for the group
  <cr>

Configuration is beyond the scope of the exam, but some key notes to try to keep in the back of your mind regarding SNMPv3 options above:

  • If no read view is defined, all objects can be read
  • If no write view is defined, no objects can be written
  • If no notify view is defined, group members are not sent notifications

Speaking of users, I’ll create a user here use SHA for Auth and AES 128-bit encryption, and as a warning this is going to be a LOT of output as I show the ? modifiers:

R5(config)#snmp-server user Dave ?
  WORD  Group to which the user belongs

R5(config)#snmp-server user Dave CCNP ?
  remote  Specify a remote SNMP entity to which the user belongs
  v1      user using the v1 security model
  v2c     user using the v2c security model
  v3      user using the v3 security model

R5(config)#snmp-server user Dave CCNP v3 ?
  access     specify an access-list associated with this group
  auth       authentication parameters for the user
  encrypted  specifying passwords as MD5 or SHA digests
  <cr>

R5(config)#snmp-server user Dave CCNP v3 auth ?
  md5  Use HMAC MD5 algorithm for authentication
  sha  Use HMAC SHA algorithm for authentication

R5(config)#snmp-server user Dave CCNP v3 auth sha ?
  WORD  authentication pasword for user

R5(config)#snmp-server user Dave CCNP v3 auth sha CCIE ?
  access  specify an access-list associated with this group
  priv    encryption parameters for the user
  <cr>

R5(config)#snmp-server user Dave CCNP v3 auth sha CCIE priv ?
  3des  Use 168 bit 3DES algorithm for encryption
  aes   Use AES algorithm for encryption
  des   Use 56 bit DES algorithm for encryption

R5(config)#snmp-server user Dave CCNP v3 auth sha CCIE priv aes ?
  128  Use 128 bit AES algorithm for encryption
  192  Use 192 bit AES algorithm for encryption
  256  Use 256 bit AES algorithm for encryption

R5(config)#snmp-server user Dave CCNP v3 auth sha CCIE priv aes 128 ?
  WORD  privacy pasword for user

R5(config)#snmp-server user Dave CCNP v3 auth sha CCIE priv aes 128 CCNA ?
  access  specify an access-list associated with this group
  <cr>

R5(config)#snmp-server user Dave CCNP v3 auth sha CCIE priv aes 128 CCNA
R5(config)#
*Mar 19 04:24:28.623: Configuring snmpv3 USM user, persisting snmpEngineBoots. Please Wait…

R5(config)#

So I actually hit enter there without creating the above “group” config, just to see what happens, and I got this message. I am waiting for my routing to start sparking or smoking, but apparently when you hit enter you apparently start snmpEngineBoots.

So I hope the above ? output all looks fairly self explanatory, it’s just a mouthful to configure, which has me thankful its beyond the scope of this course (I hope) 🙂

Now to give you some output to chew on as to configuring the traps, I won’t actually be configuring one, but here is the output of the beginning of the configuration:

R5(config)#snmp-server host ?
  WORD                                                  Hostname or IP address
                                                        of SNMP notification
                                                        host
  http://<Hostname or A.B.C.D>[:<port number>][/<uri>]  HTTP address of XML
                                                        notification host

R5(config)#snmp-server host 172.12.15.1 ?
  WORD     SNMPv1/v2c community string or SNMPv3 user name
  informs  Send Inform messages to this host
  traps    Send Trap messages to this host
  version  SNMP version to use for notification messages
  vrf      VPN Routing instance for this host

R5(config)#snmp-server host 172.12.15.1 traps ?
  WORD     SNMPv1/v2c community string or SNMPv3 user name
  version  SNMP version to use for notification messages

R5(config)#snmp-server host 172.12.15.1 traps version ?
  1   Use SNMPv1
  2c  Use SNMPv2c
  3   Use SNMPv3

R5(config)#snmp-server host 172.12.15.1 traps version 3 ?
  auth    Use the SNMPv3 authNoPriv Security Level
  noauth  Use the SNMPv3 noAuthNoPriv Security Level
  priv    Use the SNMPv3 authPriv Security Level

R5(config)#snmp-server host 172.12.15.1 traps version 3 priv ?
  WORD  SNMPv1/v2c community string or SNMPv3 user name

R5(config)#snmp-server host 172.12.15.1 traps version 3 priv Dave ?
  aaa_server               Allow SNMP AAA traps
  adslline                 Allow ADSL LINE-MIB traps
  atm                      Allow SNMP atm traps
  authenticate-fail        Allow SNMP 802.11 Authentication Fail Trap
  bgp                      Allow BGP state change traps
  bulkstat                 Allow Data-Collection-MIB traps
  c3g                      Allow Cellular 3G modem reset traps
  call-home                Allow SNMP CISCO-CALLHOME-MIB traps
  cnpd                     Allow NBAR Protocol Discovery traps
  config                   Allow SNMP config traps
  config-copy              Allow SNMP config-copy traps
  config-ctid              Allow SNMP config-ctid traps
  cpu                      Allow cpu related traps
  deauthenticate           Allow SNMP 802.11 Deauthentication Trap
  disassociate             Allow SNMP 802.11 Disassociation Trap
  dot11-mibs               Allow dot11 traps
  dot11-qos                Allow SNMP 802.11 QoS Change Trap
  ds0-busyout              Allow ds0-busyout traps
  ds1                      Allow SNMP ds1 traps
  ds1-loopback             Allow ds1-loopback traps
  dsp                      Allow SNMP DSP traps
  eigrp                    Allow SNMP EIGRP traps
 –More–

And on and on it goes, which I won’t drown you or myself looking back on the modifier output, but the main take away if nothing else is REMEMBER THOSE PRIVILEGE LEVELS AND WHAT THEY DO (auth, noauth, priv)! Also, that SNMPv3 is the only one that offers both authentication and encryption.

That should cover what is needed for the ROUTE exam, next up is NTP and securing it, as it is possibly one of the most important protocols on your network to keep all network devices working. This is especially true for Cisco Voice stuff, but that’s a discussion for another day.

Router password fundamentals, configuration, a very good brush up on passwords unless you just took the CCNA

OSPF_Base_Topology

I have the above network configured,  but again will just use R1 and R5 for the discussion and examples of router output and configuration, as I am a walking zombie today due to lack of sleep and life stuff.

Speaking of life stuff, I said I was thinking about pushing my date out for CCNP ROUTE, and I got an email notification shortly after notifying me my test center I’ve always gone to is shutting down the week before my exam date so I had to reschedule it – So be careful what you wish for 🙂 4/28 is the new date that I pass this exam and move onto SWITCH.

This will be brief, as it’s really CCNA refresher material, but if you haven’t taken the CCNA in years like myself then it’s good to know the command syntax and options.

That being said, lets start with the most basic password concepts and end on the least:

  • Enable password vs Secret – Enable secret will still be preferred over the enable password when prompted for a password for user exec in IOS 15.x
  • Password must be enabled on the VTY lines or connection will be refused
  • “service password-encryption” encrypts all current and future passwords in clear test in the router running configuration
  • Must configure “login” if only setting a password, and use “login local” to enable the use of username and password local database to log in

Speaking of local username / password database, lets configure a few usernames here to demonstrate “login local” as mentioned above briefly. Now generally in any show run you just see “login” on my vty lines, because in a lab environment that is ok, but real world you may want to have a username and password combination for router access.

Still on 15.x IOS, the username does still appear when you type it, but the password does not. So lets get to the configuration of our two users, the bobs:

R1(config)#username the password bobs
R1(config)#user bobs ?
  aaa                  AAA directive
  access-class         Restrict access by access-class
  autocommand          Automatically issue a command after the user logs in
  callback-dialstring  Callback dialstring
  callback-line        Associate a specific line with this callback
  callback-rotary      Associate a rotary group with this callback
  dnis                 Do not require password when obtained via DNIS
  nocallback-verify    Do not require authentication after callback
  noescape             Prevent the user from using an escape character
  nohangup             Do not disconnect after an automatic command
  nopassword           No password is required for the user to log in
  password             Specify the password for the user
  privilege            Set user privilege level
  secret               Specify the secret for the user
  user-maxlinks        Limit the user’s number of inbound links
  view                 Set view name
  <cr>

R1(config)#user bobs privilege ?
  <0-15>  User privilege level

R1(config)#user bobs privilege 15 ?
  aaa                  AAA directive
  access-class         Restrict access by access-class
  autocommand          Automatically issue a command after the user logs in
  callback-dialstring  Callback dialstring
  callback-line        Associate a specific line with this callback
  callback-rotary      Associate a rotary group with this callback
  dnis                 Do not require password when obtained via DNIS
  nocallback-verify    Do not require authentication after callback
  noescape             Prevent the user from using an escape character
  nohangup             Do not disconnect after an automatic command
  nopassword           No password is required for the user to log in
  password             Specify the password for the user
  privilege            Set user privilege level
  secret               Specify the secret for the user
  user-maxlinks        Limit the user’s number of inbound links
  view                 Set view name
  <cr>

R1(config)#user bobs privilege 15 password the

So the first username as you can see at the top, I just typed “username the password bobs” and it’s really just as easy as that to configure a user, however for ‘bobs the’ I let the ? output flow because there are some very weird command modifiers I want to be known.

For example after I entered the priv 15, we can enter “nopassword” or “secret” for the user, as well as even restricting the username by “access-class” which we all know from the last post is an ACL to lock down incoming connections! However I took the slacker road and just entered a username and password.

So lets look at the show run currently for this bad boy, and I’ll highlight in red all the security derps we have going on in it:

R1# sh run
Building configuration…

Current configuration : 1462 bytes
!
version 12.4
service timestamps debug datetime msec
service timestamps log datetime msec
no service password-encryption
!
hostname R1
!
boot-start-marker
boot-end-marker
!
enable secret 5 $1$aHDM$YMgDe3WXGwGCHctjWlGr71
enable password CCNA
!
no aaa new-model
!
resource policy
!
no network-clock-participate slot 1
no network-clock-participate wic 0
ip cef
!
!
!
!
no ip domain lookup
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
username the password 0 bobs
username bobs privilege 15 password 0 the
!
!
!
!
!
!
!
interface Loopback1
 ip address 1.1.1.1 255.255.255.255
!
interface FastEthernet0/0
 no ip address
 shutdown
 duplex auto
 speed auto
!
interface Serial0/0
 ip address 172.12.123.1 255.255.255.0
 encapsulation frame-relay
 frame-relay map ip 172.12.123.2 122 broadcast
 frame-relay map ip 172.12.123.3 123 broadcast
 no frame-relay inverse-arp
 frame-relay lmi-type cisco
!
interface FastEthernet0/1
 ip address 172.12.15.1 255.255.255.0
 duplex auto
 speed auto
!
interface Serial0/1
 ip address 172.12.13.1 255.255.255.252
 clock rate 2000000
!
router ospf 1
 log-adjacency-changes
 area 34 virtual-link 44.44.44.1
 network 1.1.1.1 0.0.0.0 area 1
 network 172.12.15.0 0.0.0.255 area 15
 network 172.12.123.0 0.0.0.255 area 0
!
!
!
ip http server
no ip http secure-server
!
!
!
!
!
!
control-plane
!
!
!
!
!
!
!
!
!
!
!
line con 0
 exec-timeout 0 0
 logging synchronous
line aux 0
line vty 0 4
 password CCNP
 logging synchronous
 login
!
!
end

R1#

First we do not have the service “password-encryption” running, so you can see all non-secret (almost every) password because they are in plain text.

A never ending exec mode timeout can lead to leaving sessions open, and allowing others to stumble upon an open router at it’s prompt, or use it to get around time based ACL’s (as discussed in the last post).

Another issue with the vty lines, is they are using telnet so all data is transferred in plain text INCLUDING THE PASSWORD TO LOGIN, but it is also using a single password instead of the local username / password database.

So lets tighten some things up:

R1#conf t
Enter configuration commands, one per line.  End with CNTL/Z.
R1(config)#service password-encryption
R1(config)#do sh run | i username
username the password 7 094E410B0A
username bobs privilege 15 password 7 105A011C
R1(config)#

After enabling password-encryption in global config, a sh run | i username shows they are now encrypted in the running configuration, and in case you are wondering the pipe include ( | i ) will only give you output with the keyword or number you specify after it.

So lets make sure even though service password-encryption is running, that the secret takes precedence as the password used for enable:

R5#telnet 172.12.15.1
Trying 172.12.15.1 … Open

User Access Verification

Password:  <—- Set on VTY lines as CCNP (about to change that)
R1>en
Password:   <—- Tried CCNA
Password:   <—- Tried CCNP
R1#

So lets first change this so that we are using usernames and password combinations at the least:

R1(config)#
R1(config)#line vty 0 4
R1(config-line)#login local
R1(config-line)#^Z
R1#
*Mar  1 23:09:21.103: %SYS-5-CONFIG_I: Configured from console by console
R1#
ASR#2
[Resuming connection 2 to r5 … ]

R5#telnet 172.12.15.1
Trying 172.12.15.1 … Open

User Access Verification

Username: the
Password:
R1>en
Password:   <—- CCNA
Password:   <—- CCNP
R1#

Now I’m having too much fun with the enable secret precedence. As seen, “the” is visible while “bobs” is not, so that is how that goes. However, that priv 15 should kick us straight into user exec mode when telnet’ing in, lets check it out:

R1#exit

[Connection to 172.12.15.1 closed by foreign host]
R5#telnet 172.12.15.1
Trying 172.12.15.1 … Open

User Access Verification

Username: bobs
Password:
R1#

And that it did, right into user exec mode. We are going in the right direction, but we need to take care of using telnet to log into the router, as we are security minded CCNP candidates.

This takes actually a couple steps and we’ve already taken care of one of them by forcing the login local for remote connections, as ssh requires a username / password whether its an AAA server doing authentication or whether its authentication LOCAL.

The next step is defining on the vty lines, what kind of remote management protocols you want to allow to access those lines:

R1#conf t
Enter configuration commands, one per line.  End with CNTL/Z.
R1(config)#line vty 0 4
R1(config-line)#transport ?
  input      Define which protocols to use when connecting to the terminal
             server
  output     Define which protocols to use for outgoing connections
  preferred  Specify the preferred protocol to use

R1(config-line)#transport input ?
  all     All protocols
  lat     DEC LAT protocol
  mop     DEC MOP Remote Console Protocol
  none    No protocols
  pad     X.3 PAD
  rlogin  Unix rlogin protocol
  ssh     TCP/IP SSH protocol
  telnet  TCP/IP Telnet protocol
  udptn   UDPTN async via UDP protocol
  v120    Async over ISDN

R1(config-line)#transport input ssh
R1(config-line)#exit
R1(config)#exit
R1#exit

[Connection to 172.12.15.1 closed by foreign host]
R5#

Oddly I kind of expected it to kill my telnet connection to it from R5, but perhaps because the TCP connection was already made, you must wait for it to break and try reconnecting for the limitation or rule to kick in which seems to be a universal rule with TCP connections:

R5#telnet 172.12.15.1
Trying 172.12.15.1 …
% Connection refused by remote host

R5#

And so it is, TCP connections will maintain their connection until torn down, newly added configs during the duration of the connection will not effect the TCP connection(s).

However we are not done with the SSH setup yet, it requires for a domain name to be added to the router, along with a crypto key to be generated:

R1#conf t
Enter configuration commands, one per line.  End with CNTL/Z.
R1(config)#ip domain ?
  list         Domain name to complete unqualified host names
  lookup       Enable IP Domain Name System hostname translation
  multicast    Define the domain name for multicast address lookups
  name         Define the default domain name
  retry        Specify times to retry sending a DNS query
  round-robin  Round-robin multiple IP addresses in cache
  timeout      Specify timeout waiting for response to a DNS query

R1(config)#ip domain-name ?
  WORD  Default domain name
  vrf   Specify VRF

R1(config)#ip domain-name loopedback.com
R1(config)#crypto key generate ?
  rsa  Generate RSA keys
  <cr>

R1(config)#crypto key generate rsa
The name for the keys will be: R1.loopedback.com
Choose the size of the key modulus in the range of 360 to 2048 for your
  General Purpose Keys. Choosing a key modulus greater than 512 may take
  a few minutes.

How many bits in the modulus [512]: 1024
% Generating 1024 bit RSA keys, keys will be non-exportable…[OK]

R1(config)#
*Mar  1 23:22:59.697: %SSH-5-ENABLED: SSH 1.99 has been enabled
R1(config)#

As you can see by that last console message, we now have ssh success! Now a couple things, you would of course use the local domain name of the network the router is an edge device on, also the crypto key size can be 512 (and is by default) but should be made to at least 1024 bit encryption in the real world (at very least) so I did here.

So lets see if you can ssh from router to router, I’m not sure if I have actually ever tried:

R5#ssh 172.12.15.1
% No user specified nor available for SSH client
R5#ssh ?
  -c    Select encryption algorithm
  -l    Log in using this user name
  -m    Select HMAC algorithm
  -o    Specify options
  -p    Connect to this port
  -v    Specify SSH Protocol Version
  -vrf  Specify vrf name
  WORD  IP address or hostname of a remote system

R5#telnet ?
  WORD  IP address or hostname of a remote system
  <cr>

R5#ssh -l ?
  WORD  Login name

R5#ssh -l the ?
  -c    Select encryption algorithm
  -m    Select HMAC algorithm
  -o    Specify options
  -p    Connect to this port
  -v    Specify SSH Protocol Version
  -vrf  Specify vrf name
  WORD  IP address or hostname of a remote system

R5#ssh -l the 172.12.15.1

Password:

R1>

Huh, I didn’t think I’d figure it out that easy. So “ssh -l (username) (remote IP)” and you will be prompted for your password and get logged in. I have never configured that before, that is very good to know.

That’s going to do it for passwords, I am officially fried, thee ya!

Time based ACL’s, configuring time-range and differences in types of ranges, using time-based ACL’s to limit telnet access

OSPF_Base_Topology

I’m was going to wait for the NTP part of the course to go through this, but since it looks like ACL material finishes with this I will use the time-range command for now rather than synchronizing the network to an NTP server.

A bit of a refresher from CCNA material, but it can’t help to get a refresh on subjects when it comes to Cisco. Time based ACL’s are exactly what they sound like, ACL’s that are only active during the period of time they are set for. This, however, implies that you have the correct time set on your network device which is where the “time-range” command comes in.

So you can set multiple time ranges on a router, as each time you enter “time-range (word)” it will drop you into time-range configure mode. I will work just between R5 and R1 to demonstrate how this works, and have removed the ACL’s from the previous lab so we get a fresh start! So we will start on R1 with our time-range setting and explanations:

R1#conf t
Enter configuration commands, one per line.  End with CNTL/Z.
R1(config)#time-range ?
WORD  Time range name

R1(config)#time-range CCNP ?
<cr>

R1(config)#time-range CCNP
R1(config-time-range)#?
Time range configuration commands:
absolute  absolute time and date
default   Set a command to its defaults
exit      Exit from time-range configuration mode
no        Negate a command or set its defaults
periodic  periodic time and date

R1(config-time-range)#

As can be seen, we are now in time-range configuration mode, so I did ? to see what are options are and there are only two we need to concern ourselves with – absolute and periodic.

Absolute time-range’s are static starting at this time and ending at this time with no recurrence options which makes it not ideal for most situations, but I’ll demonstrate what it looks like with the ? output to show you the modifiers to go with it:

R1(config-time-range)#absolute ?
  end    ending time and date
  start  starting time and date

R1(config-time-range)#absolute start ?
  hh:mm  Starting time

R1(config-time-range)#absolute start 18:00 ?
  <1-31>  Day of the month

R1(config-time-range)#absolute start 18:00 15 ?
  MONTH  Month of the year [eg: Jan for January, Jun for June]

R1(config-time-range)#absolute start 18:00 15 Mar ?
  <1993-2035>  Year

R1(config-time-range)#absolute start 18:00 15 Mar 2017 ?
  end  ending time and date
  <cr>

R1(config-time-range)#absolute start 18:00 15 Mar 2017 end ?
  hh:mm  Ending time – stays valid until beginning of next minute

R1(config-time-range)#absolute start 18:00 15 Mar 2017 end

A couple of things to note here, after the year 2035, no more timed ACL’s according to IOS so get em while the getting is good. Seriously though, it’s very straight forward, and as can be seen at the end there that you can place the end of the time on the same command or enter it with “absolute end …” and do note it does stay active until the next minute!

I am not exactly sure what purpose this would serve unless you perhaps had a consultant coming in from x time and leaving y time, and don’t want them to access certain things in that range only and be done with it. Either way, that is not our scenario, so lets move on to period time-range’s:

R1(config-time-range)#periodic ?
  Friday     Friday
  Monday     Monday
  Saturday   Saturday
  Sunday     Sunday
  Thursday   Thursday
  Tuesday    Tuesday
  Wednesday  Wednesday
  daily      Every day of the week
  weekdays   Monday thru Friday
  weekend    Saturday and Sunday

R1(config-time-range)#periodic weekdays ?
  hh:mm  Starting time

R1(config-time-range)#periodic weekdays 08:00 ?
  to  ending day and time

R1(config-time-range)#periodic weekdays 08:00 to 17:00 ?
  <cr>

R1(config-time-range)#periodic weekdays 08:00 to 17:00
R1(config-time-range)#

I love the options, the simplicity of setting the values, and that Cisco was human enough to put “weekdays” and “weekends” as values so you don’t have to add ranges for each separate weekday or weekend day.

So I set mine with that period command, so whatever ACL I apply that to is open for business the same time I am, M-F 8am-5pm (though we may need to tweak some times on the routers to demonstrate some reactions and output).

So we now have a time range, and here is how to view it:

R1#sh time-range
time-range entry: CCNP (inactive)
   periodic weekdays 8:00 to 17:00
R1#

If you have more than one it will show all of them, but it will also show which ones are active and inactive, which can be a way to tell what time it is or isn’t loosely on a router on exam day if asked what time of day it is on the router. Speaking of time on routers, and since we are in User Exec mode which really surprised me this is where it gets configured, lets set the time for R1 and R5 simultaneously since we are not doing an NTP lab just yet:

R1#clock set ?
  hh:mm:ss  Current Time

R1#clock set 16:04:00 ?
  <1-31>  Day of the month
  MONTH   Month of the year

R1#clock set 16:04:00 15 ?
  MONTH  Month of the year

R1#clock set 16:04:00 15 Mar ?
  <1993-2035>  Year

R1#clock set 16:04:00 15 Mar 2017 ?
  <cr>

R1#clock set 16:04:00 15 Mar 2017
R1#
*Mar 15 16:04:00.000: %SYS-6-CLOCKUPDATE: System clock has been updated from 23:05:02 UTC Fri Mar 1 2002 to 16:04:00 UTC Wed Mar 15 2017, configured from console by console.
R1#
ASR#5
[Resuming connection 5 to r5 … ]

R5#clock set 16:04:00 15 Mar 2017
R5#
*Mar 15 16:04:00.000: %SYS-6-CLOCKUPDATE: System clock has been updated from 23:58:24 UTC Wed Mar 15 2017 to 16:04:00 UTC Wed Mar 15 2017, configured from console by console.
R5#

I am not in the UTC time zone, but I will address changing that in the NTP lab, as I have set up IOS devices to be NTP Servers for networks and I don’t want to pile extra stuff onto this lab tonight, however our routers are about as close as I could get them to the correct time.

ONE MAJOR THING TO NOTE, THE CLOCK IS SET IN USER EXEC MODE, NOT GLOBAL CONFIG MODE WHICH I THOUGHT WAS VERY WEIRD, SO WATCH THAT ON EXAM DAY!

Ok, so we have a time range, clocks are set, and we can verify this with a quick “sh clock”:

R1#sh clock
16:08:04.536 UTC Wed Mar 15 2017
R1#

Now let’s make an access-list using this time range, and I’ll make it for telnet to demonstrate how to limit access to routers when “not needed” as though such a thing exists in the real world:

 

R1(config)#access-list 123 deny ?
  <0-255>  An IP protocol number
  ahp      Authentication Header Protocol
  eigrp    Cisco’s EIGRP routing protocol
  esp      Encapsulation Security Payload
  gre      Cisco’s GRE tunneling
  icmp     Internet Control Message Protocol
  igmp     Internet Gateway Message Protocol
  ip       Any Internet Protocol
  ipinip   IP in IP tunneling
  nos      KA9Q NOS compatible IP over IP tunneling
  ospf     OSPF routing protocol
  pcp      Payload Compression Protocol
  pim      Protocol Independent Multicast
  tcp      Transmission Control Protocol
  udp      User Datagram Protocol

R1(config)#access-list 123 deny tcp ?
  A.B.C.D  Source address
  any      Any source host
  host     A single source host

R1(config)#access-list 123 deny tcp any ?
  A.B.C.D  Destination address
  any      Any destination host
  eq       Match only packets on a given port number
  gt       Match only packets with a greater port number
  host     A single destination host
  lt       Match only packets with a lower port number
  neq      Match only packets not on a given port number
  range    Match only packets in the range of port numbers

R1(config)#access-list 123 deny tcp any any ?
  ack          Match on the ACK bit
  dscp         Match packets with given dscp value
  eq           Match only packets on a given port number
  established  Match established connections
  fin          Match on the FIN bit
  fragments    Check non-initial fragments
  gt           Match only packets with a greater port number
  log          Log matches against this entry
  log-input    Log matches against this entry, including input interface
  lt           Match only packets with a lower port number
  neq          Match only packets not on a given port number
  precedence   Match packets with given precedence value
  psh          Match on the PSH bit
  range        Match only packets in the range of port numbers
  rst          Match on the RST bit
  syn          Match on the SYN bit
  time-range   Specify a time-range
  tos          Match packets with given TOS value
  urg          Match on the URG bit
  <cr>

R1(config)#access-list 123 deny tcp any any eq ?
  <0-65535>    Port number
  bgp          Border Gateway Protocol (179)
  chargen      Character generator (19)
  cmd          Remote commands (rcmd, 514)
  daytime      Daytime (13)
  discard      Discard (9)
  domain       Domain Name Service (53)
  drip         Dynamic Routing Information Protocol (3949)
  echo         Echo (7)
  exec         Exec (rsh, 512)
  finger       Finger (79)
  ftp          File Transfer Protocol (21)
  ftp-data     FTP data connections (20)
  gopher       Gopher (70)
  hostname     NIC hostname server (101)
  ident        Ident Protocol (113)
  irc          Internet Relay Chat (194)
  klogin       Kerberos login (543)
  kshell       Kerberos shell (544)
  login        Login (rlogin, 513)
  lpd          Printer service (515)
  nntp         Network News Transport Protocol (119)
  pim-auto-rp  PIM Auto-RP (496)
  pop2         Post Office Protocol v2 (109)
  pop3         Post Office Protocol v3 (110)
  smtp         Simple Mail Transport Protocol (25)
  sunrpc       Sun Remote Procedure Call (111)
  syslog       Syslog (514)
  tacacs       TAC Access Control System (49)
  talk         Talk (517)
  telnet       Telnet (23)
  time         Time (37)
  uucp         Unix-to-Unix Copy Program (540)
  whois        Nicname (43)
  www          World Wide Web (HTTP, 80)

R1(config)#access-list 123 deny tcp any any eq 23 ?
  ack          Match on the ACK bit
  dscp         Match packets with given dscp value
  established  Match established connections
  fin          Match on the FIN bit
  log          Log matches against this entry
  log-input    Log matches against this entry, including input interface
  precedence   Match packets with given precedence value
  psh          Match on the PSH bit
  rst          Match on the RST bit
  syn          Match on the SYN bit
  time-range   Specify a time-range
  tos          Match packets with given TOS value
  urg          Match on the URG bit
  <cr>

R1(config)#access-list 123 deny tcp any any eq 23 time-range ?
  WORD  Time-range entry name

R1(config)#access-list 123 deny tcp any any eq 23 time-range CCNP
R1(config)#

LOTS of output, but I wanted to demonstrate a few things, and I’ve also highlighted in red all the commands that I used to create the ACL along the way.

First I wanted to demonstrate that since it is telnet, I used “tcp” instead of just “ip” traffic as we don’t need that general of a statement. Next I used any any because we will be applying this to our VTY lines for telnet access control, so the source and destination can be any as the connection might come from anywhere to here so no need to split hairs when its not needed.

Next and this is a big one, I’m not sure if this still works, but you used to be able to create an ACL and type eq ? at the end to get a list of port #’s to help you along the test if you forget a certain port #. If you get a simulator or something that allows this, it may be worth doing this once quick and jotting some down you don’t have committed to memory moving forward through the test it might just save your ass.

Finally after eq, I could have put telnet or 23, I personally always use port numbers to keep them fresh in  my head but you can put the service name if listed as well and that is a valid command. Finally the time-range is added onto the end of the ACL. Now lets check it out:

R1#sh access-list
Extended IP access list 123
    10 deny tcp any any eq telnet time-range CCNP (active)
R1#sh time-range
time-range entry: CCNP (active)
   periodic weekdays 8:00 to 17:00
   used in: IP ACL entry
R1#sh clock
16:21:50.366 UTC Wed Mar 15 2017
R1#

I was expecting it to say (Inactive) there, but I forgot the UTC thing, so lets go for the final step which is configuring it in telnet or more specifically on the VTY line configuration:

R1(config)#line vty 0 4
R1(config-line)#access-group ?
% Unrecognized command
R1(config-line)#access-class ?
  <1-199>      IP access list
  <1300-2699>  IP expanded access list
  WORD         Access-list name

R1(config-line)#access-class 123 ?
  in   Filter incoming connections
  out  Filter outgoing connections

R1(config-line)#access-class 123 in ?
  vrf-also  Same access list is applied for all VRFs
  <cr>

R1(config-line)#access-class 123 in
R1(config-line)#

(Quick note at the end of the access-class command, it can be applied to non-global VRF route tables as well, worth noting while on the subject)

I highlighted what I put in, and what was correct for a reason, because it’s so easy to mess up like I just did. Access-group is on interfaces, and access-class will always be for VTY line configuration of applying ACL’s. Notice we also had to define in our out, so because this router will be receiving the telnet connections, I specified “in” as my modifier option.

So one more time, access-class = applying ACL to vty lines, followed by ACL # and in/out.

Now that we have this all configured and everything seems to be working great, lets go to R5 and give our new found access-list a go:

R5#telnet 172.12.15.1
Trying 172.12.15.1 …
% Connection refused by remote host

R5#

Wow, duh, I put DENY on my ACL. Let me change that and try it again here:

R1(config)#no access-list 123 deny tcp any any eq 23 time-range CCNP
R1(config)#access-list 123 permit tcp any any eq 23 time-range CCNP
R1(config)#
ASR#5
[Resuming connection 5 to r5 … ]

R5#telnet 172.12.15.1
Trying 172.12.15.1 … Open

User Access Verification

Password:
R1>en
Password:
R1#

Works much better when you PERMIT telnet access during the hours you want it available, eh? Now lets throw a wrench into the mix, while telnet’d into R1 I am going to change the routers time to be outside the time-range and see if that immediately boots me out:

R1#clock set 22:31:45 15 mar 2017
R1#sh time-range
time-range entry: CCNP (inactive)
   periodic weekdays 8:00 to 17:00
   used in: IP ACL entry
R1#sh access-list
Extended IP access list 123
    10 permit tcp any any eq telnet time-range CCNP (inactive) (2 matches)
R1#
R1#
R1#exit

[Connection to 172.12.15.1 closed by foreign host]
R5#telnet 172.12.15.1
Trying 172.12.15.1 …
% Connection refused by remote host

R5#

There are a couple very important real world lessons here:

  • ACL’s will only block connection attempts after they are set, they will not break current connection attempts, so we would need to manually clear that vty line to kick the user out so to say – This is for any connection on any firewall basically at all in the real world so keep this very important concept in mind
  • This brings up the “no exec-t” command that is great for labs, but if the user never gets kicked out after so long idle, they have a loophole around that time-range
  • Notice the ACL says (inactive) – That is because the time-range is not engaged and using the ACL at the moment!

I have personally accidentally deleted ACL’s that showed inactive because I didn’t know that meant it was on a time-range schedule (or what it meant at all), so do not as I do, an (Inactive) ACL is not an unused ACL!

That completes this post and I think about wraps up ACL’s, we’ve been using them on other topics so hopefully they’re comfortable with CCNP candidates reading this by now.

Next up is going to be a bit more CCNA type of material, but for thorough sake you bet your beehive I will write up a quick refresher post on that as well, it’s nice to get a break in with basically refresher material right before I hit the BGP section (which I am oddly looking forward to).

If I don’t see ya, good afternoon, good evening, and good night!