Category Archives: CCNP – NTP

NTP: Demonstrated between 2 routers running IOS 15.x to demonstrate NTPv4 Authentication and other concepts! (Edited with more content 5/20/17)

NTPAs I only have two routers in my lab running IOS 15.x, and 12.x is not NTPv4 capable, I’ve decided to review it between just these two routers to demonstrate new features brought by 15.x code that I couldn’t in my previous lab involving the whole WAN Topology.

To start there are a quite a few things we require a network with synchronized time for:

  • Phone Systems absolutely require synchronized time across the network
  • Debugging output and logging files to compare to current “sh clock” time
  • Network Mgmt Protocols (Like SNMP)

Just to name a few, beyond just logs and such, it is absolutely necessary for phone systems to have as accurate a time as it can have without be hooked directly to an Atomic Clock.

Also, if you’ve ever pulled logs with the completely wrong time set, and had to do “sh clock” and try to piece the logs to the current time like a puzzle it quickly turns into a very frustrating waste of time when you’re busy.

That being said, all routers have a built in “System Clock” which is usually battery driven, so if you power cycle the router it will retain the correct time (not all, but most routers).

When a router is pulled out of the box, it has a default time that is probably very different from the current time, and needs to be configured in one of the following ways:

  • Manual Configuration – Configuring the time directly on the device
  • NTP (Network Time Protocol) – Getting Time from a Time Source
  • SNTP (Simple Network Time Protocol) – Receive only protocol for Time

Given that NTP can both receive and send network time to other devices, we will not discuss SNTP at all, but wanted to mention it because it is a thing (though I’ve never seen it in production networks).

The two major versions in use of NTP are Version 3 and Version 4, and the ROUTE exam will expect you to know some differences between them, in terms of what was introduced in NTPv4 (which is why I’m using 15.x routers only).

So lets get to the differences between the two versions right up front:

Version 3

  • Only worked with IPv4, not IPv6
  • Does not do Authentication
  • Both use UDP Port 123 for source and destination

Version 4

  • Introduces NTP Authentication
  • Works with IPv4 and IPv6
  • Again both uses UDP Port 123 as source and destination

NTP devices, aside from manually configuring a Master / Server source in your network, obtains their time from a reliable source. This can be an Atomic Clock (unlikely unless you have access to Military grade Atomic Clocks), NTP Servers available via ntp.org, or other network devices like a manually configured device.

NTP is 100% Client / Server model protocol, where the client polls the Server for time about every minute, and the Server (considered to be the Authortive Source for time) provides time to that Client.

The Authority that a Time Server has is based on something called Stratum, which is a measurement of hops away from an Atomic Clock, so a device directly connected to an Atomic Clock is a Stratum 1, another device 1 hop away from a Stratum 1 device is a Stratum 2.

So as can be seen, the Stratum increments based on where you get your time from, for example if my router gets it’s time from a Stratum 3 clock then it is Stratum 4.

When configuring a router manually to be an NTP server, it’s default stratum level if not defined, is Stratum 8 with no real explanation as to why – So you’ll want to set it to a lower value.

There are 4 different modes of NTP devices

  • NTP Server (Also referred to as a Master)
  • NTP Client
  • NTP Peers (Called Symmetric mode, two or more NTP servers that exchange information)
  • NTP Broadcast / Multicast mode (Can be configured on the Server or Client as a “push” or poll of time in either Broadcast or Multicast format of transmission)

So, all that being said, first lets look at how to manually configure our NTP Server:

R1#clock set ?
  hh:mm:ss  Current Time

R1#clock set 18:34:00 ?
  <1-31>  Day of the month
  MONTH   Month of the year

R1#clock set 18:34:00 May ?
  <1-31>  Day of the month

R1#clock set 18:34:00 May 19 ?
  <1993-2035>  Year

R1#clock set 18:34:00 May 19 2017 ?
  <cr>

R1#clock set 18:34:00 May 19 2017
R1#
*May 19 18:34:00.000: %SYS-6-CLOCKUPDATE: System clock has been updated from 00:29:07 UTC Sat May 20 2017 to 18:34:00 UTC Fri May 19 2017, configured from console by console.
R1#

As you can see, Cisco has predicated at the time of the making of this router, our species will either not make it past the year 2035, or figured the device would be antiquated by then. I agree with both theories.

So now that a network device has the correct time, we should make that the Server for the network, and assign it a proper Stratum:

R1#conf t
Enter configuration commands, one per line.  End with CNTL/Z.
R1(config)#ntp master ?
  <1-15>  Stratum number
  <cr>

R1(config)#ntp master 1
R1(config)#

As can be seen this is done in Global Configuration mode, which also has a couple of optional commands that can be entered here as well:

R1(config)#ntp peer 172.12.14.4 ?
  burst        Send a burst when peer is reachable
  iburst       Send a burst when peer is unreachable
  key          Configure peer authentication key
  maxpoll      Maximum poll interval
  minpoll      Minimum poll interval
  normal-sync  Disable rapid sync at startup
  prefer       Prefer this peer when possible
  source       Interface for source address
  version      Configure NTP version
  <cr>

R1(config)#ntp peer 172.12.14.4
R1(config)#int fa0/1
R1(config-if)#ntp ?
  broadcast  Configure NTP broadcast service
  disable    Disable NTP traffic (both IP and IPv6)
  multicast  Configure NTP multicast service

R1(config-if)#

So as shown the “ntp peer” command is entered now on the global config level, and I did go into the interface level to show the broadcast / multicast command, but did not issue it.

So now that R1 is saying its an NTP Peer with R4, lets issue that on R4 and see if it synchronizes:

R4#debug ntp events
NTP events debugging is on
R4#conf t
Enter configuration commands, one per line.  End with CNTL/Z.
R4(config)#ntp peer 172.12.14.1
R4(config)#
*May 19 23:40:26.315: %NTP : Drift Read : FFFFFFFF.FFFFD70D
*May 19 23:40:26.315: NTP: Initialized interface FastEthernet0/0
*May 19 23:40:26.315: NTP: Initialized interface FastEthernet0/1
*May 19 23:40:26.315: NTP: Initialized interface Loopback4
*May 19 23:40:26.315: NTP: Initialized interface VoIP-Null0
R4(config)#do sh clock
*23:40:35.299 UTC Fri May 19 2017
R4(config)#

This can be configured between NTP Servers, or between Server / Client, depending on whether you who you want to keep on the same page of what Time it is. Now lets try the interface level Broadcast command facing R4:

R4(config)#no ntp peer 172.12.14.1
R4(config)#
ASR#1
[Resuming connection 1 to r1 … ]

R1(config)#no ntp peer 172.12.14.4
R1(config)#int fa0/1
R1(config-if)#ntp broadcast
R1(config-if)#
ASR#4
[Resuming connection 4 to r4 … ]

.May 19 18:59:54.540: NTP: Uninitialized interface FastEthernet0/0
.May 19 18:59:54.540: NTP: Uninitialized interface FastEthernet0/1
.May 19 18:59:54.540: NTP: Uninitialized interface Loopback4
.May 19 18:59:54.540: NTP: Uninitialized interface VoIP-Null0
R4(config)#do sh clock
.19:00:35.060 UTC Fri May 19 2017
R4(config)#

Again without any Authentication or anything set, the Server broadcasts its time at a non-server (client) and it immediately syncs its time with the Server.

Remember, Peer and Broadcast are completely optional!

So I’ve removed all those settings, and we are back to the R4 having no Time Source information configured but R1 is our Stratum 1 NTP Server, lets look at the client configuration.

There are two ways to really go about requesting time from the Server:

  • “ntp server (master IP)”
  • “ntp broadcast client”

My IOS doesn’t seem to support the second command (dag nab it), so lets look at the first:

R4(config)#ntp server 172.12.14.1
R4(config)#
.May 19 19:08:12.908: %NTP : Drift Read : FFFFFFFF.FFFFD70D
.May 19 19:08:12.912: NTP: Initialized interface FastEthernet0/0
.May 19 19:08:12.912: NTP: Initialized interface FastEthernet0/1
.May 19 19:08:12.912: NTP: Initialized interface Loopback4
.May 19 19:08:12.912: NTP: Initialized interface VoIP-Null0
R4(config)#

R4 is of course still running “debug ntp events”, and if I remove the command:

R4(config)#no ntp server 172.12.14.1
R4(config)#
.May 19 19:08:26.472: NTP: Uninitialized interface FastEthernet0/0
.May 19 19:08:26.472: NTP: Uninitialized interface FastEthernet0/1
.May 19 19:08:26.472: NTP: Uninitialized interface Loopback4
.May 19 19:08:26.472: NTP: Uninitialized interface VoIP-Null0
R4(config)#

So from this output we gather Initialized = Good, Uninitialized = Bad with NTP.

The two verification commands for NTP as I’ve covered before will be:

  • “sh ntp status”
  • “sh ntp assoc”

So I’ll re-apply the server command to R4 pointing it to R1 again, and lets see what the output looks like on R4:

R4(config)#do sh ntp status
Clock is unsynchronized, stratum 16, no reference clock
nominal freq is 250.0000 Hz, actual freq is 250.0006 Hz, precision is 2**24
reference time is 00000000.00000000 (00:00:00.000 UTC Mon Jan 1 1900)
clock offset is 0.0000 msec, root delay is 0.00 msec
root dispersion is 0.10 msec, peer dispersion is 0.00 msec
loopfilter state is ‘FSET’ (Drift set from file), drift is -0.000002440 s/s
system poll interval is 64, never updated.
R4(config)#
R4(config)#
R4(config)#
R4(config)#do sh ntp assoc

  address         ref clock       st   when   poll reach  delay  offset   disp
 ~172.12.14.1     .INIT.          16      –     64     0  0.000   0.000 16000.
 * sys.peer, # selected, + candidate, – outlyer, x falseticker, ~ configured
R4(config)#exit

Wooooah, too soon?

R4#sh ntp status
Clock is synchronized, stratum 2, reference is 172.12.14.1
nominal freq is 250.0000 Hz, actual freq is 250.0006 Hz, precision is 2**24
reference time is DCC9C244.9328C9C9 (19:13:08.574 UTC Fri May 19 2017)
clock offset is 5.0405 msec, root delay is 1.83 msec
root dispersion is 5.83 msec, peer dispersion is 0.05 msec
loopfilter state is ‘CTRL’ (Normal Controlled Loop), drift is -0.000002435 s/s
system poll interval is 64, last update was 25 sec ago.
R4#

Yup, too soon, took about 30 seconds or so to synchronize. Lets look at “sh ntp assoc”:

R4#sh ntp assoc

  address         ref clock       st   when   poll reach  delay  offset   disp
*~172.12.14.1     .LOCL.           1     27     64     7  1.839   5.040  1.678
 * sys.peer, # selected, + candidate, – outlyer, x falseticker, ~ configured
R4#

That is what we want to see, not just the asterisk, but the wavy line (whatever that is) along with the asterisk which indicates it is synched with Server. Also note it is 1 hop away from the Stratum 1 server, so the client is Stratum 2.

So to go through the fields in the “sh ntp assoc” I’ll do it bullet point style here:

  • address = IP of the server this device is getting its time from
  • ref clock = Where that server is getting its time from (LOCL means manually configured)
  • st = Stratum of that server, in this case Stratum 1

Beyond that the fields are beyond the scope of CCNP and I don’t want to open that can of worms, but that is how to interpret those 3 fields.

Now I only have two routers so I cannot demonstrate this, but if there is a client one hop downstream from another client, they can share information with each other once the original client is synchronized with the Server.

You can simply type “ntp server 172.12.14.4” lets say if R5 was behind R4, and R4 would supply it time like a Server would.

Now for methods of Authentication or just NTP Server Protection, you can use Access-Lists to limit who can receive NTP from it, or configure Authentication

NTP Authentication uses MD5 Hash configured by Authentication Keys.

Multiple Keys can be configured for different peers, so you can configure a key per peer.

To configure this on the Server, this is how we do that in global configuration mode:

R1#conf t
Enter configuration commands, one per line.  End with CNTL/Z.
R1(config)#ntp authentication-key 1 md5 CCNP
R1(config)#ntp trusted-key 1
R1(config)#ntp authenticate
R1(config)#

The server will not start actually using authentication until you use the last command entered “ntp authenticate”.

Now on the client, the commands are exactly the same, with one additional command at the end:

R4#conf t
Enter configuration commands, one per line.  End with CNTL/Z.
R4(config)#ntp authentication-key 1 md5 CCNP
R4(config)#ntp trusted-key 1
R4(config)#ntp authenticate
R4(config)#ntp server 172.12.14.1 key 1
R4(config)#

And that is how it is done, that is all for now, later!

EDIT:

Had to cut that short due to company and actually enjoying my Friday, but back to it, and I wanted to first edit this with some output of what it looks like when Authentication is failing.

I wanted to demonstrate a couple of things after I stripped NTP completely off R4, and manually did a “clock set …” on it. I wanted to see if NTP will take precedence over a local manual configuration when you enable NTP, and also what the debug looks like when Authentication fails:

So first we strip the commands, reset the clock manually as it hangs onto its last known time even if you remove NTP (so it still had the same time as R1), so lets get to it quick:

Removal

R4(config)#do sh ntp assoc
R4(config)#do sh ntp stat
%NTP is not enabled.
R4(config)#do sh clock
.00:19:17.447 UTC Sun May 21 2017
R4(config)#exit

“clock set …” and “debug ntp events” started
R4#clock set 02:45:00 feb 27 2000
R4#
.Feb 27 02:45:00.000: %SYS-6-CLOCKUPDATE: System clock has been updated from 00:20:05 UTC Sun May 21 2017 to 02:45:00 UTC Sun Feb 27 2000, configured from console by console.
R4#debug ntp ?
  adjust    NTP clock adjustments
  all       NTP all debugging on
  core      NTP core messages
  events    NTP events
  packet    NTP packet debugging
  refclock  NTP refclock messages

R4#debug ntp events
NTP events debugging is on

Giving it a try with just “ntp server …”

R4#conf t
Enter configuration commands, one per line.  End with CNTL/Z.
R4(config)#ntp server 172.12.14.1
R4(config)#
.Feb 27 02:55:41.571: %NTP : Drift Read : FFFFFFFF.FFFFD70D
.Feb 27 02:55:41.571: NTP: Initialized interface FastEthernet0/0
.Feb 27 02:55:41.571: NTP: Initialized interface FastEthernet0/1
.Feb 27 02:55:41.571: NTP: Initialized interface Loopback4
.Feb 27 02:55:41.571: NTP: Initialized interface VoIP-Null0

………… still waiting…
R4(config)#do sh ntp assoc

  address         ref clock       st   when   poll reach  delay  offset   disp
 ~172.12.14.1     .INIT.          16      –     64     0  0.000   0.000 16000.
 * sys.peer, # selected, + candidate, – outlyer, x falseticker, ~ configured
R4(config)#do sh ntp stat
Clock is unsynchronized, stratum 16, no reference clock
nominal freq is 250.0000 Hz, actual freq is 250.0006 Hz, precision is 2**24
reference time is 00000000.00000000 (00:00:00.000 UTC Mon Jan 1 1900)
clock offset is 0.0000 msec, root delay is 0.00 msec
root dispersion is 0.17 msec, peer dispersion is 0.00 msec
loopfilter state is ‘CTRL’ (Normal Controlled Loop), drift is -0.000002440 s/s
system poll interval is 64, never updated.

Getting impatient, time to “debug ntp all”
R4(config)#do debug ntp all
NTP events debugging is on
NTP core messages debugging is on
NTP clock adjustments debugging is on
NTP reference clocks debugging is on
NTP packets debugging is on
R4(config)#
May 21 00:32:41.241: NTP message sent to 172.12.14.1, from interface ‘FastEthernet0/1’ (172.12.14.4).
May 21 00:32:41.241: NTP message received from 172.12.14.1 on interface ‘FastEthernet0/1’ (172.12.14.4).
May 21 00:32:41.241: NTP Core(DEBUG): ntp_receive: message received
May 21 00:32:41.241: NTP Core(DEBUG): ntp_receive: peer is 0x67444C00, next action is 1.
May 21 00:32:41.241: NTP Core(DEBUG): receive: packet given to process_packet

Something finally happened, lets check out what!
R4(config)#do sh clock
00:33:12.941 UTC Sun May 21 2017
R4(config)#do sh ntp assoc

  address         ref clock       st   when   poll reach  delay  offset   disp
*~172.12.14.1     .LOCL.           1     46     64     3  1.929   0.140 3937.9
 * sys.peer, # selected, + candidate, – outlyer, x falseticker, ~ configured
R4(config)#

What???

I am guessing this is another bug I ran into last time I was NTP labbing, but in hopes that it isn’t I reloaded R4 t give it a chance.

Essentially the bug is that it will allow the commands to be entered for NTPv4, but Authentication is not doing anything between them, like its running NTPv3 (and I don’t see an “ntp …” option to set NTPv4.

So after the reload I re-manually set the clock, confirmed it retained the ntp server command, and did a debug all:

R4#sh clock
00:44:53.522 UTC Sun May 21 2017
R4#clock set 02:00:00 jan 15 2000
R4#
.Jan 15 02:00:00.000: %SYS-6-CLOCKUPDATE: System clock has been updated from 00:45:23 UTC Sun May 21 2017 to 02:00:00 UTC Sat Jan 15 2000, configured from console by console.

R4#debug ntp all
NTP events debugging is on
NTP core messages debugging is on
NTP clock adjustments debugging is on
NTP reference clocks debugging is on
NTP packets debugging is on
R4#sh run | i ntp
ntp server 172.12.14.1
R4#
.Jan 15 02:00:39.051: NTP message sent to 172.12.14.1, from interface ‘FastEthernet0/1’ (172.12.14.4).
.Jan 15 02:00:39.051: NTP message received from 172.12.14.1 on interface ‘FastEthernet0/1’ (172.12.14.4).
.Jan 15 02:00:39.051: NTP Core(DEBUG): ntp_receive: message received
.Jan 15 02:00:39.051: NTP Core(DEBUG): ntp_receive: peer is 0x6743C8A0, next action is 1.
.Jan 15 02:00:39.051: NTP Core(DEBUG): receive: packet given to process_packet
.Jan 15 02:00:39.051: NTP Core(DEBUG): Peer becomes reachable, poll set to 6.

.Jan 15 02:00:39.051: NTP Core(INFO): peer 172.12.14.1 event ‘event_reach’ (0x84) status ‘unreach, conf, 3 events, event_reach’ (0x8034)
R4#
.Jan 15 02:00:39.051: NTP Core(INFO): synchronized to 172.12.14.1, stratum 1

Unbelievable, even with IOS code 15 I run into a bug.

Well that was my attempt to show more detail, just remember the commands AND DIFFERENCES between NTPv3 and NTPv4 for exam day 🙂

NTP Broadcast better demonstrated over an Ethernet segment, pretty brief and to the point to finish off the NTP section!

ACL_Refresher_R1toR5

I have done a “wr er” / “reload” on R1 and R5, configuring only loopbacks, and opened Fa0/1 interfaces to be Area 0 between the two. No NBMA, no oddities (hopefully), and see how it works with a lot of troubleshooting.

So to begin, I’ll set R1 first as the NTP master with a clock time:

R1#clock set 18:00:00 30 mar 2017
R1#
*Mar 30 18:00:00.000: %SYS-6-CLOCKUPDATE: System clock has been updated from 18:43:02 UTC Thu Mar 30 2017 to 18:00:00 UTC Thu Mar 30 2017, configured from console by console.
R1#conf t
Enter configuration commands, one per line.  End with CNTL/Z.
R1(config)#ntp master ?
  <1-15>  Stratum number
  <cr>

R1(config)#ntp master 3
R1(config)#int fa0/1
R1(config-if)#ntp broadcast ?
  client       Listen to NTP broadcasts
  destination  Configure broadcast destination address
  key          Configure broadcast authentication key
  version      Configure NTP version
  <cr>

R1(config-if)#ntp broadcast
R1(config-if)#

So that is all configured, unfortunately R1 is running on 12.x IOS code, so it is using NTPv3 (and is only capable of running version 3). So lets go to R5 and see if we can get this working:

R5#conf t
Enter configuration commands, one per line.  End with CNTL/Z.
R5(config)#int fa0/1
R5(config-if)#ntp ?
  broadcast  Configure NTP broadcast service
  disable    Disable NTP traffic (both IP and IPv6)
  multicast  Configure NTP multicast service

R5(config-if)#ntp broadcast ?
  client       Listen to NTP broadcasts
  destination  Configure broadcast destination address
  key          Configure broadcast authentication key
  version      Configure NTP version
  <cr>

R5(config-if)#ntp broadcast client ?
  <cr>

R5(config-if)#ntp broadcast client
R5(config-if)#^Z
R5#debug
*Mar 30 17:05:01.491: %SYS-5-CONFIG_I: Configured from console by console
R5#debug ntp pack
NTP packets debugging is on
R5#
*Mar 30 17:05:10.659: NTP message received from 172.12.15.1 on interface ‘FastEthernet0/1’ (255.255.255.255).
R5#sh clock
*17:05:27.923 UTC Thu Mar 30 2017
R5#

Just as easy as that, that is how it is supposed to work 🙂

One thing I noticed on R5 running 15.1 IOS code is the NTP messages are so much smaller and concise, a lot of the basic infrastructure works the same between IOS versions but I do like some of the subtle differences.

So now I’ll reverse roles with debug still running, and see what happens starting with R5:

R5#sh clock
.18:45:28.132 UTC Thu Mar 30 2017
R5#conf t
Enter configuration commands, one per line.  End with CNTL/Z.
R5(config)#ntp master ?
  <1-15>  Stratum number
  <cr>

R5(config)#ntp master 3 ?
  <cr>

R5(config)#ntp master 3
R5(config)#int fa0/1
R5(config-if)#ntp broadcast ?
  client       Listen to NTP broadcasts
  destination  Configure broadcast destination address
  key          Configure broadcast authentication key
  version      Configure NTP version
  <cr>

R5(config-if)#ntp broadcast version ?
  <2-4>  NTP version number

R5(config-if)#ntp broadcast version 4
R5(config-if)#^Z
R5#deb
.Mar 30 18:49:32.833: %SYS-5-CONFIG_I: Configured from console by console
R5#debug ntp pack
.Mar 30 18:49:38.737: NTP message sent to 255.255.255.255, from interface ‘FastEthernet0/1’ (172.12.15.5).
R5#debug ntp pack
NTP packets debugging is on
R5#

I changed it to Version 4 NTP to see if R1 who only understands up to version 3 can still pick up the time from R5, so debugs running lets see what happens with R1:

R1#conf t
Enter configuration commands, one per line.  End with CNTL/Z.
R1(config)#int fa0/1
R1(config-if)#ntp broadcast client
R1(config-if)#do sh clock
18:54:12.613 UTC Thu Mar 30 2017
R1(config-if)#
ASR#2
[Resuming connection 2 to r5 … ]

Mar 30 18:52:48.736: NTP message sent to 255.255.255.255, from interface ‘FastEthernet0/1’ (172.12.15.5).
R5#
Mar 30 18:53:54.736: NTP message sent to 255.255.255.255, from interface ‘FastEthernet0/1’ (172.12.15.5).
R5#
Mar 30 18:54:57.735: NTP message sent to 255.255.255.255, from interface ‘FastEthernet0/1’ (172.12.15.5).
R5#sh clock
18:55:04.327 UTC Thu Mar 30 2017
R5#clock set 21:00:00 30 mar 2017
R5#
.Mar 30 21:00:00.000: %SYS-6-CLOCKUPDATE: System clock has been updated from 18:55:35 UTC Thu Mar 30 2017 to 21:00:00 UTC Thu Mar 30 2017, configured from console by console.
R5#
.Mar 30 21:00:26.763: NTP message sent to 255.255.255.255, from interface ‘FastEthernet0/1’ (172.12.15.5).
R5#
ASR#1
[Resuming connection 1 to r1 … ]

R1(config-if)#do sh clock
18:56:17.338 UTC Thu Mar 30 2017
R1(config-if)#

So it is not updating, as I believe it just doesn’t understand NTP version 4, so I’ll change it to version 3 and see if we can get this working:

R5(config)#int fa0/1
R5(config-if)#no ntp broadcast version 4
R5(config-if)#ntp broadcast version 3
R5(config-if)#do sh clock
21:03:18.143 UTC Thu Mar 30 2017
R5(config-if)#
Mar 30 21:03:55.763: NTP message sent to 255.255.255.255, from interface ‘FastEthernet0/1’ (172.12.15.5).
R5(config-if)#
ASR#1
[Resuming connection 1 to r1 … ]

R1(config-if)#do sh clock
18:59:56.341 UTC Thu Mar 30 2017

So it’s not liking that, so I tried removing and re-adding the the “ntp broadcast client” on R1 to see if that’d kick it into gear but it did not so I did have to reboot R1, however:

R1>en
Password:
R1#sh clock
21:24:00.844 UTC Thu Mar 30 2017
R1#

Almost gets me teary how proud I am of my routers to work as expected 🙂

So that is it, done with NTP for now, we have officially beaten that dead horse into the ground!

One thing to note, as this NTP battle has underscored running IOS code 12.x sucks for the current CCNP ROUTE exam, all of the concepts we have troubleshot have been overcome by logic, and not incompatibility between the versions. I will definitely demonstrate on my 2 routers I have running IOS 15.x any critical to know material that are not on my NBMA routers (R1 / R2 / R3 are on 12.x IOS code), however it really does not alter the behaviors of most protocols beyond newer versions of them – So don’t take this as irrelevant due to the 12.x code over the NBMA I would just rather invest that money into the exams

NTP Finale – Configuring broadcast / multicast / version #, and disabling NTP all together – Time to see what breaks!

OSPF_Base_Topology_NTP

After quite a few days out of the study game due to illness and actually giving myself time to recover from it this time, I am back, and I am already sick of going over NTP – So this is the last but important concepts for NTP.

One oddity I have noticed working with NTP on my routers over the NBMA, is that R1 for some reason does not save the time after I “wr mem” so I have to use the “clock set …” command every time for it, however R2 and R3 never catch up after I reset the clock and get their time:

R3#sh clock
*00:33:12.545 UTC Sat Mar 2 2002
R3#sh ntp assoc

      address         ref clock     st  when  poll reach  delay  offset    disp
 ~172.12.123.1     0.0.0.0          16    45    64    0     0.0    0.00  16000.
 * master (synced), # master (unsynced), + selected, – candidate, ~ configured
R3#
ASR#2
[Resuming connection 2 to r2 … ]

R2#sh clock
*00:32:47.895 UTC Sat Mar 2 2002
R2#

That is, unless I reload them, I let them sit for 10 minutes after resetting R1 but still nothing. So, given the frame switch is a NON-BROADCAST MultiAccess network between all of them, I am going to change this up a bit to see if I can wake R2 and R3 up:

R1#conf t
Enter configuration commands, one per line.  End with CNTL/Z.
R1(config)#int s0/0
R1(config-if)#ntp ?
  broadcast  Configure NTP broadcast service
  disable    Disable NTP
  multicast  Configure NTP multicast service

R1(config-if)#ntp multicast ?
  A.B.C.D  Multicast group IP address
  client   Listen to NTP multicasts
  key      Configure multicast authentication key
  ttl      TTL of the multicast packet
  version  Configure NTP version
  <cr>

R1(config-if)#ntp multicast
R1(config-if)#
ASR#2
[Resuming connection 2 to r2 … ]

R2#conf t
Enter configuration commands, one per line.  End with CNTL/Z.
R2(config)#int s0/0
R2(config-if)#ntp multicast
R2(config-if)#
ASR#3
[Resuming connection 3 to r3 … ]

R3#conf t
Enter configuration commands, one per line.  End with CNTL/Z.
R3(config)#int s0/2
R3(config-if)#ntp multicast
R3(config-if)#
ASR#1
[Resuming connection 1 to r1 … ]

R1(config-if)#^Z

I am leaving R4 out of this lab, because it is late, and I want to play with a few of the extended command highlighted in red – specifically version # to see if NTP will jive between different version numbers.

So I threw on some debugging to see what we have going on with R2 and R3:
R1#debug ntp pack
NTP packets debugging is on
R1#
.Mar 29 20:43:46.821: NTP: rcv packet from 172.12.123.2 to 172.12.123.1 on Serial0/0:
.Mar 29 20:43:46.821:  leap 3, mode 3, version 3, stratum 0, ppoll 64
.Mar 29 20:43:46.821:  rtdel 0000 (0.000), rtdsp 10001 (1000.015), refid 00000000 (0.0.0.0)
.Mar 29 20:43:46.825:  ref 00000000.00000000 (00:00:00.000 UTC Mon Jan 1 1900)
.Mar 29 20:43:46.825:  org DC869AC2.D297792B (20:42:42.822 UTC Wed Mar 29 2017)
.Mar 29 20:43:46.825:  rec C02A9DA9.4FAC7075 (00:39:05.311 UTC Sat Mar 2 2002)
.Mar 29 20:43:46.825:  xmt C02A9DE9.42276D52 (00:40:09.258 UTC Sat Mar 2 2002)
.Mar 29 20:43:46.825:  inp DC869B02.D2BFA7C5 (20:43:46.823 UTC Wed Mar 29 2017)
R1#
.Mar 29 20:43:46.829: NTP: stateless xmit packet to 172.12.123.2:
.Mar 29 20:43:46.829:  leap 3, mode 4, version 3, stratum 0, ppoll 64
.Mar 29 20:43:46.829:  rtdel 0000 (0.000), rtdsp 10001 (1000.015), refid 00000000 (0.0.0.0)
.Mar 29 20:43:46.829:  ref 00000000.00000000 (00:00:00.000 UTC Mon Jan 1 1900)
.Mar 29 20:43:46.829:  org C02A9DE9.42276D52 (00:40:09.258 UTC Sat Mar 2 2002)
.Mar 29 20:43:46.829:  rec DC869B02.D2BFA7C5 (20:43:46.823 UTC Wed Mar 29 2017)
.Mar 29 20:43:46.829:  xmt DC869B02.D41D2E3F (20:43:46.828 UTC Wed Mar 29 2017)
R1#
.Mar 29 20:43:54.894: NTP: rcv packet from 172.12.123.3 to 172.12.123.1 on Serial0/0:
.Mar 29 20:43:54.894:  leap 3, mode 3, version 3, stratum 0, ppoll 64
.Mar 29 20:43:54.894:  rtdel 0000 (0.000), rtdsp 10001 (1000.015), refid 00000000 (0.0.0.0)
.Mar 29 20:43:54.894:  ref 00000000.00000000 (00:00:00.000 UTC Mon Jan 1 1900)
.Mar 29 20:43:54.898:  org DC869ACA.E5739DBB (20:42:50.896 UTC Wed Mar 29 2017)
.Mar 29 20:43:54.898:  rec C02A9DE2.574AB8D3 (00:40:02.340 UTC Sat Mar 2 2002)
.Mar 29 20:43:54.898:  xmt C02A9E22.46D2D084 (00:41:06.276 UTC Sat Mar 2 2002)
.Mar 29 20:43:54.898:  inp DC869B0A.E5776373 (20:43:54.896 UTC Wed Mar 29 2017)
R1#
.Mar 29 20:43:54.898: NTP: stateless xmit packet to 172.12.123.3:
.Mar 29 20:43:54.902:  leap 3, mode 4, version 3, stratum 0, ppoll 64
.Mar 29 20:43:54.902:  rtdel 0000 (0.000), rtdsp 10001 (1000.015), refid 00000000 (0.0.0.0)
.Mar 29 20:43:54.902:  ref 00000000.00000000 (00:00:00.000 UTC Mon Jan 1 1900)
.Mar 29 20:43:54.902:  org C02A9E22.46D2D084 (00:41:06.276 UTC Sat Mar 2 2002)
.Mar 29 20:43:54.902:  rec DC869B0A.E5776373 (20:43:54.896 UTC Wed Mar 29 2017)
.Mar 29 20:43:54.906:  xmt DC869B0A.E6CF23AF (20:43:54.901 UTC Wed Mar 29 2017)
R1#

So as can be seen, it shows their old time, and the new time, but we have seen this with the switch that stays insane on the Ethernet segment so let us check R2 and R3:

R2(config-if)#do sh clock
*00:47:10.551 UTC Sat Mar 2 2002
R2(config-if)#
ASR#3
[Resuming connection 3 to r3 … ]

R3(config-if)#do sh clock
*00:48:09.668 UTC Sat Mar 2 2002
R3(config-if)#do sh ntp assoc

      address         ref clock     st  when  poll reach  delay  offset    disp
 ~172.12.123.1     0.0.0.0          16    44    64    0     0.0    0.00  16000.
 * master (synced), # master (unsynced), + selected, – candidate, ~ configured
R3(config-if)#do sh ntp assoc det
172.12.123.1 configured, authenticated, insane, invalid, unsynced, stratum 16
ref ID 0.0.0.0, time 00000000.00000000 (00:00:00.000 UTC Mon Jan 1 1900)
our mode client, peer mode unspec, our poll intvl 64, peer poll intvl 64
root delay 0.00 msec, root disp 0.00, reach 0, sync dist 0.000
delay 0.00 msec, offset 0.0000 msec, dispersion 16000.00
precision 2**5, version 3
org time DC869CCA.E3EA868D (20:51:22.890 UTC Wed Mar 29 2017)
rcv time C02A9FE2.56ACF13C (00:48:34.338 UTC Sat Mar 2 2002)
xmt time C02A9FE2.45D78CAD (00:48:34.272 UTC Sat Mar 2 2002)
filtdelay =     0.00    0.00    0.00    0.00    0.00    0.00    0.00    0.00
filtoffset =    0.00    0.00    0.00    0.00    0.00    0.00    0.00    0.00
filterror =  16000.0 16000.0 16000.0 16000.0 16000.0 16000.0 16000.0 16000.0

R3(config-if)#

So it has the same issue as the switch, I’m having a hard time grasping what this issue is, however thinking upon it if you configure an interface for “ntp broadcast” or “ntp multicast” you don’t actually don’t have to have a server command pointing to R1 as it will listen on that interface for NTP broadcasts regardless.

So I am going to make a change to R1 and throw it into broadcast mode, as it should actually before forwarding broadcasts with given the ‘broadcast’ options on my frame relay statements, reload both of them to see what we get:

R3(config-if)#exit
R3(config)#no ntp server 172.12.123.1

R3(config)#int s0/2
R3(config-if)#ntp broadcast
R3(config-if)#^Z
R3#wr
*Mar  2 00:58:24.845: %SYS-5-CONFIG_I: Configured from console by console
R3#wr
Building configuration…
[OK]
R3#reload
Proceed with reload? [confirm]

ASR#2
[Resuming connection 2 to r2 … ]

R2(config-if)#ntp broadcast
R2(config-if)#exit
R2(config)#no ntp server 172.12.123.1
R2(config)#exit
R2#wr
Building configuration…

*Mar  2 00:58:18.581: %SYS-5-CONFIG_I: Configured from console by console[OK]
R2#reload
Proceed with reload? [confirm]

*Mar  2 00:58:35.857: %SYS-5-RELOAD: Reload requested by console. Reload Reason: Reload Command.
ASR#1
[Resuming connection 1 to r1 … ]

R1#conf t
Enter configuration commands, one per line.  End with CNTL/Z.
R1(config)#int s0/0
R1(config-if)#ntp broadcast
R1(config-if)#exit
R1(config)#exit
R1#debug
.Mar 29 21:02:31.933: %SYS-5-CONFIG_I: Configured from console by console
R1#debug ntp pack
NTP packets debugging is on
R1#

So now Serial0/0 on R1 is rocking broadcast mode, along with the serial interfaces of R2 and R3, and as they reload I have a debug running to see what output we get upon them coming up, lets see NTP come through for us just one more time so I can move on to a new topic:

(An off topic note, the slowness of a 2600 series router booting, mixed with a serial link speed to bring up an OSPF adjacency makes me want to jump off a roof. Fortunately my roof is only high enough to break my leg or something, so I won’t jump off it quite yet)

R1#debug
.Mar 29 21:02:31.933: %SYS-5-CONFIG_I: Configured from console by console
R1#debug ntp pack
NTP packets debugging is on
R1#
.Mar 29 21:03:38.574: %OSPF-5-ADJCHG: Process 1, Nbr 3.3.3.3 on Serial0/0 from FULL to DOWN, Neighbor Down: Dead timer expired
R1#
.Mar 29 21:04:14.165: %OSPF-5-ADJCHG: Process 1, Nbr 2.2.2.2 on Serial0/0 from FULL to DOWN, Neighbor Down: Dead timer expired
R1#sh access-list
Standard IP access list 10
    10 permit 172.12.123.0, wildcard bits 0.0.0.255 (149 matches)
    20 permit 172.12.34.0, wildcard bits 0.0.0.255 (26 matches)
    30 permit 172.12.23.0, wildcard bits 0.0.0.255 (73 matches)
R1#
.Mar 29 21:09:38.997: %OSPF-5-ADJCHG: Process 1, Nbr 3.3.3.3 on Serial0/0 from LOADING to FULL, Loading Done
R1#
.Mar 29 21:10:14.483: %OSPF-5-ADJCHG: Process 1, Nbr 2.2.2.2 on Serial0/0 from LOADING to FULL, Loading Done
R1#

So after 6 or so minutes of staring at the CLI, my adjacencies finally creaked their way out of their coffin to say Hello to R1, however no NTP debug output at all?

And as soon as I typed that, of course I DID get some output from the debug:

R1#
.Mar 29 21:12:07.224: NTP: rcv packet from 172.12.23.1 to 172.12.123.1 on Serial0/0:
.Mar 29 21:12:07.224:  leap 3, mode 3, version 3, stratum 0, ppoll 64
.Mar 29 21:12:07.224:  rtdel 0000 (0.000), rtdsp 10001 (1000.015), refid 00000000 (0.0.0.0)
.Mar 29 21:12:07.228:  ref 00000000.00000000 (00:00:00.000 UTC Mon Jan 1 1900)
.Mar 29 21:12:07.228:  org DC869F27.39F0071C (21:01:27.226 UTC Wed Mar 29 2017)
.Mar 29 21:12:07.228:  rec AF3BE488.1E7958AA (01:25:28.119 UTC Mon Mar 1 1993)
.Mar 29 21:12:07.228:  xmt AF3BE708.11A18045 (01:36:08.068 UTC Mon Mar 1 1993)
.Mar 29 21:12:07.228:  inp DC86A1A7.3A1C931F (21:12:07.226 UTC Wed Mar 29 2017)
R1#
.Mar 29 21:12:07.232: NTP: stateless xmit packet to 172.12.23.1:
.Mar 29 21:12:07.232:  leap 3, mode 4, version 3, stratum 0, ppoll 64
.Mar 29 21:12:07.232:  rtdel 0000 (0.000), rtdsp 10001 (1000.015), refid 00000000 (0.0.0.0)
.Mar 29 21:12:07.232:  ref 00000000.00000000 (00:00:00.000 UTC Mon Jan 1 1900)
.Mar 29 21:12:07.232:  org AF3BE708.11A18045 (01:36:08.068 UTC Mon Mar 1 1993)
.Mar 29 21:12:07.232:  rec DC86A1A7.3A1C931F (21:12:07.226 UTC Wed Mar 29 2017)
.Mar 29 21:12:07.232:  xmt DC86A1A7.3B65703B (21:12:07.232 UTC Wed Mar 29 2017)
R1#

So now the only debug information I am seeing is coming from the Ethernet segment which we did not alter, so I’m afraid to look at R2 and R3 but given they are configured only on the interface level to listen for NTP broadcasts maybe they just silently take the time and go with it like NTP ninjas?

Lets see:

R2>en
Password:
R2#sh clock
*01:05:28.941 UTC Sat Mar 2 2002
R2#

I did a “sh run” and forgot one of my cardinal rules, I’m already getting rusty – remove old configurations before entering new configurations for a command:

!
interface Serial0/0
 ip address 172.12.123.2 255.255.255.0
 encapsulation frame-relay
 ip ospf priority 0
 ntp broadcast
 ntp multicast
 frame-relay map ip 172.12.123.3 221
 frame-relay map ip 172.12.123.1 221 broadcast
 no frame-relay inverse-arp
 frame-relay lmi-type cisco

So I’ll save the output, but I removed NTP Multicast from all routers, and it didn’t make a difference.

And I just figured out what I goofed here:

R2(config-if)#ntp broadcast ?
  client       Listen to NTP broadcasts
  destination  Configure broadcast destination address
  key          Configure broadcast authentication key
  version      Configure NTP version
  <cr>

R2(config-if)#ntp broadcast client ?
  <cr>

R2(config-if)#ntp broadcast client

And just a few moments later:

R2(config-if)#do sh clock
.23:32:32.694 UTC Wed Mar 29 2017

So one big note worth mentioning on here that I completely derp’d, on the server you only need to set “broadcast”, however with clients listening on their interfaces you must add that “client” to the end as you are not pointing it to a master making it a client by default so you must configure it as a client. Clear as mud? Good, lets move on.

What a huge oversight, that is what facepalms are made of. However, lets see if the same works for multicasts as well then eh?

Again, after making the change on both clients to “ntp multicast client” on the serial interface I am surprisingly seeing no output from R1, and I’ve set it’s Serial0/0 interface as “ntp multicast” before configuring the others. Lets look at times around the network and see what we have:

R2(config-if)#do sh clock
.23:43:12.229 UTC Wed Mar 29 2017
R2(config-if)#do sh ntp assoc det

R2(config-if)#exit
R2(config)#exit
R2#sh n
.Mar 29 23:43:47.383: %SYS-5-CONFIG_I: Configured from console by console
R2#sh ntp ?
  associations  NTP associations
  status        NTP status

R2#sh ntp status
Clock is unsynchronized, stratum 16, no reference clock
nominal freq is 249.5901 Hz, actual freq is 249.5907 Hz, precision is 2**16
reference time is DC86C1C0.16D0FDAD (23:29:04.089 UTC Wed Mar 29 2017)
clock offset is 0.0000 msec, root delay is 0.00 msec
root dispersion is 7875.02 msec, peer dispersion is 7875.02 msec

So it has the time that it’s probably held on to from the broadcast configuration, but as can be seen there is no ntp assoc, no reference clock, notta.

So I’m going to set everything back into broadcast mode, and see if we can get everything at least in order again, and again I’ll spare the output of the commands to keep this post that should be brief, as brief as possible.

Also, I’m just leaving “debug ntp pack” on R1 the entire time, I want to see every 1 and 0 that passes through that dang NTP Master to see if I can spot a clue as to what is going on here:

R2(config)#
R2(config)#int s0/0
R2(config-if)#no ntp multicast client
R2(config-if)#ntp broadcast client
R2(config-if)#exit
R2(config)#exit
R2#wr
Building configuration…

.Mar 29 23:49:25.404: %SYS-5-CONFIG_I: Configured from console by console[OK]
R2#
ASR#3
[Resuming connection 3 to r3 … ]

R3(config-if)#no ntp multicast client
R3(config-if)#ntp broadcast client
R3(config-if)#^Z
R3#wr
Building configuration…

*Mar  2 01:42:09.543: %SYS-5-CONFIG_I: Configured from console by console[OK]
R3#
R3#reload
Proceed with reload? [confirm]

*Mar  2 01:42:33.590: %SYS-5-RELOAD: Reload requested by console. Reload Reason: Reload Command.
ASR#2
[Resuming connection 2 to r2 … ]

R2#reload
Proceed with reload? [confirm]

.Mar 29 23:50:47.210: %SYS-5-RELOAD: Reload requested by console. Reload Reason: Reload Command.
ASR#1
[Resuming connection 1 to r1 … ]

R1#conf t
Enter configuration commands, one per line.  End with CNTL/Z.
R1(config)#int s0/0
R1(config-if)#no ntp multicast
R1(config-if)#ntp broadcast
R1(config-if)#^Z
R1#
.Mar 29 21:52:28.573: %SYS-5-CONFIG_I: Configured from console by console
R1#debug ntp pack
NTP packets debugging is on
R1#

I did that in kind of a weird order, as I wanted R2 and R3 to reload at relatively the same time, then reset R1 for broadcast mode as they re-cook, and now that I have 6 minutes to wait I put on my debug on R1 and going to start my PS4 and get food ready, run a marathon, and go sky diving so when I get back it should finally be back up and cooking.

So I literally cleaned my kitchen, did dishes, and came back to this:

R1#debug ntp pack
NTP packets debugging is on
R1#
.Mar 29 21:53:54.190: %OSPF-5-ADJCHG: Process 1, Nbr 3.3.3.3 on Serial0/0 from FULL to DOWN, Neighbor Down: Dead timer expired
R1#
.Mar 29 21:54:03.565: %OSPF-5-ADJCHG: Process 1, Nbr 2.2.2.2 on Serial0/0 from FULL to DOWN, Neighbor Down: Dead timer expired
R1#
.Mar 29 21:59:32.168: NTP: rcv packet from 172.12.34.4 to 172.12.123.1 on Serial0/0:
.Mar 29 21:59:32.168:  leap 3, mode 3, version 4, stratum 0, ppoll 256
.Mar 29 21:59:32.172:  rtdel 0000 (0.000), rtdsp 2095 (127.274), refid 41555448 (65.85.84.72)
.Mar 29 21:59:32.172:  ref 00000000.00000000 (00:00:00.000 UTC Mon Jan 1 1900)
.Mar 29 21:59:32.172:  org 00000000.00000000 (00:00:00.000 UTC Mon Jan 1 1900)
.Mar 29 21:59:32.172:  rec 00000000.00000000 (00:00:00.000 UTC Mon Jan 1 1900)
.Mar 29 21:59:32.172:  xmt DC870C00.8A152270 (04:45:52.539 UTC Thu Mar 30 2017)
.Mar 29 21:59:32.172:  inp DC86ACC4.2BCC2DF4 (21:59:32.171 UTC Wed Mar 29 2017)
R1#
.Mar 29 21:59:32.176: NTP: stateless xmit packet to 172.12.34.4:
.Mar 29 21:59:32.176:  leap 3, mode 4, version 4, stratum 0, ppoll 256
.Mar 29 21:59:32.176:  rtdel 0000 (0.000), rtdsp 10001 (1000.015), refid 00000000 (0.0.0.0)
.Mar 29 21:59:32.176:  ref 00000000.00000000 (00:00:00.000 UTC Mon Jan 1 1900)
.Mar 29 21:59:32.176:  org DC870C00.8A152270 (04:45:52.539 UTC Thu Mar 30 2017)
.Mar 29 21:59:32.176:  rec DC86ACC4.2BCC2DF4 (21:59:32.171 UTC Wed Mar 29 2017)
.Mar 29 21:59:32.176:  xmt DC86ACC4.2D26835D (21:59:32.176 UTC Wed Mar 29 2017)
R1#
.Mar 29 21:59:54.624: %OSPF-5-ADJCHG: Process 1, Nbr 3.3.3.3 on Serial0/0 from LOADING to FULL, Loading Done
R1#
.Mar 29 22:00:03.884: %OSPF-5-ADJCHG: Process 1, Nbr 2.2.2.2 on Serial0/0 from LOADING to FULL, Loading Done
R1#

The seemingly never ending OSPF Adj reform, but also again no NTP messages between R1 and either spoke, so lets take a look at the spokes here:
R2>en
Password:
R2#sh clock
*23:57:14.213 UTC Wed Mar 29 2017
R2#
ASR#3
[Resuming connection 3 to r3 … ]

System Bootstrap, Version 12.2(8r) [cmong 8r], RELEASE SOFTWARE (fc1)
Copyright (c) 2003 by cisco Systems, Inc.
C2600 platform with 131072 Kbytes of main memory

R3>en
Password:
R3#sh clock
*01:50:15.588 UTC Sat Mar 2 2002
R3#

I just don’t get some of these behaviors, and yes that clock does show its actually now past midnight, and this is one of those things that will drive me up a freegin wall if I do not figure out this behavior (Although as seen in other lab posts, there are bugs (and screenshots of Cisco’s webpage describing the bug) for some of these IOS’s, so it very well may be a bug.

So lets get to troubleshooting here and figure out what in the world is going on. So first, I want to stare and compare any conflicts on the “sh run” of all 3 routers which I’ll display here:

R1:

R1#sh run int serial0/0
Building configuration…

Current configuration : 258 bytes
!
interface Serial0/0
 ip address 172.12.123.1 255.255.255.0
 encapsulation frame-relay
 ntp broadcast
 frame-relay map ip 172.12.123.3 123 broadcast
 frame-relay map ip 172.12.123.2 122 broadcast
 no frame-relay inverse-arp
 frame-relay lmi-type cisco
end

R1#

R2:
R2#sh run int s0/0
Building configuration…

Current configuration : 275 bytes
!
interface Serial0/0
 ip address 172.12.123.2 255.255.255.0
 encapsulation frame-relay
 ip ospf priority 0
 ntp broadcast client
 frame-relay map ip 172.12.123.1 221 broadcast
 frame-relay map ip 172.12.123.3 221
 no frame-relay inverse-arp
 frame-relay lmi-type cisco
end

R2#

R3:

R3#sh run int s0/2
Building configuration…

Current configuration : 247 bytes
!
interface Serial0/2
 ip address 172.12.123.3 255.255.255.0
 encapsulation frame-relay
 ip ospf priority 0
 ntp broadcast client
 frame-relay map ip 172.12.123.2 321
 frame-relay map ip 172.12.123.1 321 broadcast
 no frame-relay inverse-arp
end

R3#

These all look good, how about pings, I guess what has burnt me in the past is Layer 3 connectivity:

R1#ping 172.12.123.2

Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 172.12.123.2, timeout is 2 seconds:
!!!!!
Success rate is 100 percent (5/5), round-trip min/avg/max = 64/66/69 ms
R1#ping 172.12.123.3

Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 172.12.123.3, timeout is 2 seconds:
!!!!!
Success rate is 100 percent (5/5), round-trip min/avg/max = 65/67/68 ms
R1#

Nope. So to try to force some action, I have set “debug ntp pack” on R2 and R3, and will do so now and remove the ntp broadcast command and switch it to a version number not used by the other routers that are using NTPv3 from what I see in past output, and see what happens:

R1#conf t
Enter configuration commands, one per line.  End with CNTL/Z.
R1(config)#int s0/0
R1(config-if)#no ntp broadcast version 2
R1(config-if)#^Z
R1#deb
.Mar 29 22:20:34.140: %SYS-5-CONFIG_I: Configured from console by console
R1#debug ntp pack
NTP packets debugging is on
R1#conf t
Enter configuration commands, one per line.  End with CNTL/Z.
R1(config)#int s0/0
R1(config-if)#ntp broadcast version 2
R1(config-if)#
ASR#2
[Resuming connection 2 to r2 … ]

R2#sh clock
*00:12:33.711 UTC Thu Mar 30 2017
R2#
ASR#3
[Resuming connection 3 to r3 … ]

R3#sh clock
*02:05:31.201 UTC Sat Mar 2 2002

Across the NBMA this gets all sorts of buggy, I must have gotten cursed code for my routers when it comes to NTP, however lets point back to a master so I can sleep tonight knowing my clocks are in sync:

R2#conf t
Enter configuration commands, one per line.  End with CNTL/Z.
R2(config)#int s0/0
R2(config-if)#no ntp broadcast client
R2(config-if)#exit
R2(config)#ntp server 172.12.123.1
R2(config)#
ASR#3
[Resuming connection 3 to r3 … ]

R3#conf t
Enter configuration commands, one per line.  End with CNTL/Z.
R3(config)#int s0/2
R3(config-if)#no ntp broadcast client
R3(config-if)#exit
R3(config)#ntp server 172.12.123.1
R3(config)#
ASR#1
[Resuming connection 1 to r1 … ]

.Mar 29 22:25:05.950: NTP: rcv packet from 172.12.34.4 to 172.12.123.1 on Serial0/0:
.Mar 29 22:25:05.950:  leap 3, mode 3, version 4, stratum 0, ppoll 1024
.Mar 29 22:25:05.950:  rtdel 0000 (0.000), rtdsp 2679 (150.284), refid 41555448 (65.85.84.72)
.Mar 29 22:25:05.950:  ref 00000000.00000000 (00:00:00.000 UTC Mon Jan 1 1900)
.Mar 29 22:25:05.954:  org DC86AEC3.1A2B1433 (22:08:03.102 UTC Wed Mar 29 2017)
.Mar 29 22:25:05.954:  rec DC870DFF.9B9DF69E (04:54:23.607 UTC Thu Mar 30 2017)
.Mar 29 22:25:05.954:  xmt DC8711FE.89175915 (05:11:26.535 UTC Thu Mar 30 2017)
.Mar 29 22:25:05.958:  inp DC86B2C1.F3D1CFC4 (22:25:05.952 UTC Wed Mar 29 2017)
R1(config)#0F7FD7A (22:25:29.066 UTC Wed Mar 29 2017)
.Mar 29 22:25:30.057: NTP: rcv packet from 172.12.123.2 to 172.12.123.1 on Serial0/0:
.Mar 29 22:25:30.057:  leap 3, mode 3, version 3, stratum 0, ppoll 64
.Mar 29 22:25:30.057:  rtdel 0000 (0.000), rtdsp 10001 (1000.015), refid 00000000 (0.0.0.0)
.Mar 29 22:25:30.057:  ref 00000000.00000000 (00:00:00.000 UTC Mon Jan 1 1900)
.Mar 29 22:25:30.057:  org DC86B2D9.10F7FD7A (22:25:29.066 UTC Wed Mar 29 2017)
.Mar 29 22:25:30.062:  rec DC86CCEF.534DC1DA (00:16:47.325 UTC Thu Mar 30 2017)
.Mar 29 22:25:30.062:  xmt DC86CCF0.43AD06C4 (00:16:48.264 UTC Thu Mar 30 2017)
.Mar 29 22:25:30.062:  inp DC86B2DA.0F1B0C1B (22:25:30.059 UTC Wed Mar 29 2017)
.Mar 29 22:25:30.062: NTP: stateless xmit packet to 172.12.123.2:
.Mar 29 22:25:30.062:  leap 3, mode 4, version 3, stratum 0, ppoll 64
.Mar 29 22:25:30.062:  rtdel 0000 (0.000), rtdsp 10001 (1000.015), refid 00000000 (0.0.0.0)
.Mar 29 22:25:30.066:  ref 00000000.00000000 (00:00:00.000 UTC Mon Jan 1 1900)
.Mar 29 22:25:30.066:  org DC86CCF0.43AD06C4 (00:16:48.264 UTC Thu Mar 30 2017)
.Mar 29 22:25:30.066:  rec DC86B2DA.0F1B0C1B (22:25:30.059 UTC Wed Mar 29 2017)
.Mar 29 22:25:30.066:  xmt DC86B2DA.10675AC7 (22:25:30.064 UTC Wed Mar 29 2017)
.Mar 29 22:25:31.059: NTP: rcv packet from 172.12.123.2 to 172.12.123.1 on Serial0/0:
.Mar 29 22:25:31.059:  leap 3, mode 3, version 3, stratum 0, ppoll 64
.Mar 29 22:25:31.059:  rtdel 0000 (0.000), rtdsp 10001 (1000.015), refid 00000000 (0.0.0.0)
.Mar 29 22:25:31.059:  ref 00000000.00000000 (00:00:00.000 UTC Mon Jan 1 1900)
.Mar 29 22:25:31.059:  org DC86B2DA.10675AC7 (22:25:30.064 UTC Wed Mar 29 2017)
.Mar 29 22:25:31.059:  rec DC86CCF0.52B2A86D (00:16:48.323 UTC Thu Mar 30 2017)
.Mar 29 22:25:31.063:  xmt DC86CCF1.44195CA6 (00:16:49.266 UTC Thu Mar 30 2017)
.Mar 29 22:25:31.063:  inp DC86B2DB.0F712205 (22:25:31.060 UTC Wed Mar 29 2017)
.Mar 29 22:25:31.063: NTP: stateless xmit packet to 172.12.123.2:
.Mar 29 22:25:31.063:  leap 3, mode 4, version 3, stratum 0, ppoll 64
.Mar 29 22:25:31.063:  rtdel 0000 (0.000), rtdsp 10001 (1000.015), refid 00000000 (0.0.0.0)
.Mar 29 22:25:31.063:  ref 00000000.00000000 (00:00:00.000 UTC Mon Jan 1 1900)
.Mar 29 22:25:31.067:  org DC86CCF1.44195CA6 (00:16:49.266 UTC Thu Mar 30 2017)
.Mar 29 22:25:31.067:  rec DC86B2DB.0F712205 (22:25:31.060 UTC Wed Mar 29 2017)
.Mar 29 22:25:31.067:  xmt DC86B2DB.10BC944D (22:25:31.065 UTC Wed Mar 29 2017)
.Mar 29 22:25:32.061: NTP: rcv packet from 172.12.123.2 to 172.12.123.1 on Serial0/0:
.Mar 29 22:25:32.061:  leap 3, mode 3, version 3, stratum 0, ppoll 64
.Mar 29 22:25:32.061:  rtdel 0000 (0.000), rtdsp 10001 (1000.015), refid 00000000 (0.0.0.0)
.Mar 29 22:25:32.061:  ref 00000000.00000000 (00:00:00.000 UTC Mon Jan 1 1900)
.Mar 29 22:25:32.061:  org DC86B2DB.10BC944D (22:25:31.065 UTC Wed Mar 29 2017)
.Mar 29 22:25:32.061:  rec DC86CCF1.530371D5 (00:16:49.324 UTC Thu Mar 30 2017)
.Mar 29 22:25:32.065:  xmt DC86CCF2.4483F9C0 (00:16:50.267 UTC Thu Mar 30 2017)
.Mar 29 22:25:32.065:  inp DC86B2DC.0FE71261 (22:25:32.062 UTC Wed Mar 29 2017)
.Mar 29 22:25:32.065: NTP: stateless xmit packet to 172.12.123.2:
.Mar 29 22:25:32.065:  leap 3, mode 4, version 3, stratum 0, ppoll 64
.Mar 29 22:25:32.065:  rtdel 0000 (0.000), rtdsp 10001 (1000.015), refid 00000000 (0.0.0.0)
.Mar 29 22:25:32.065:  ref 00000000.00000000 (00:00:00.000 UTC Mon Jan 1 1900)
.Mar 29 22:25:32.069:  org DC86CCF2.4483F9C0 (00:16:50.267 UTC Thu Mar 30 2017)
.Mar 29 22:25:32.069:  rec DC86B2DC.0FE71261 (22:25:32.062 UTC Wed Mar 29 2017)
.Mar 29 22:25:32.069:  xmt DC86B2DC.1136D29C (22:25:32.067 UTC Wed Mar 29 2017)
.Mar 29 22:25:33.058: NTP: rcv packet from 172.12.123.2 to 172.12.123.1 on Serial0/0:
.Mar 29 22:25:33.058:  leap 3, mode 3, version 3, stratum 0, ppoll 64
.Mar 29 22:25:33.058:  rtdel 0000 (0.000), rtdsp 10001 (1000.015), refid 00000000 (0.0.0.0)
.Mar 29 22:25:33.058:  ref 00000000.00000000 (00:00:00.000 UTC Mon Jan 1 1900)
.Mar 29 22:25:33.058:  org DC86B2DC.1136D29C (22:25:32.067 UTC Wed Mar 29 2017)
.Mar 29 22:25:33.058:  rec DC86CCF2.5387E2A1 (00:16:50.326 UTC Thu Mar 30 2017)
.Mar 29 22:25:33.062:  xmt DC86CCF3.43E9BCB7 (00:16:51.265 UTC Thu Mar 30 2017)
.Mar 29 22:25:33.062:  inp DC86B2DD.0F48A96F (22:25:33.059 UTC Wed Mar 29 2017)
.Mar 29 22:25:33.062: NTP: stateless xmit packet to 172.12.123.2:
.Mar 29 22:25:33.062:  leap 3, mode 4, version 3, stratum 0, ppoll 64
.Mar 29 22:25:33.062:  rtdel 0000 (0.000), rtdsp 10001 (1000.015), refid 00000000 (0.0.0.0)
.Mar 29 22:25:33.062:  ref 00000000.00000000 (00:00:00.000 UTC Mon Jan 1 1900)
.Mar 29 22:25:33.066:  org DC86CCF3.43E9BCB7 (00:16:51.265 UTC Thu Mar 30 2017)
.Mar 29 22:25:33.066:  rec DC86B2DD.0F48A96F (22:25:33.059 UTC Wed Mar 29 2017)
.Mar 29 22:25:33.066:  xmt DC86B2DD.1094F81A (22:25:33.064 UTC Wed Mar 29 2017)
.Mar 29 22:25:37.818: NTP: rcv packet from 172.12.123.3 to 172.12.123.1 on Serial0/0:
.Mar 29 22:25:37.818:  leap 3, mode 3, version 3, stratum 0, ppoll 64
.Mar 29 22:25:37.818:  rtdel 0000 (0.000), rtdsp 10001 (1000.015), refid 00000000 (0.0.0.0)
.Mar 29 22:25:37.818:  ref 00000000.00000000 (00:00:00.000 UTC Mon Jan 1 1900)
.Mar 29 22:25:37.818:  org 00000000.00000000 (00:00:00.000 UTC Mon Jan 1 1900)
.Mar 29 22:25:37.818:  rec 00000000.00000000 (00:00:00.000 UTC Mon Jan 1 1900)
.Mar 29 22:25:37.818:  xmt C02AB2EA.4A4B18B3 (02:09:46.290 UTC Sat Mar 2 2002)
.Mar 29 22:25:37.822:  inp DC86B2E1.D1AC185A (22:25:37.819 UTC Wed Mar 29 2017)
.Mar 29 22:25:37.822: NTP: stateless xmit packet to 172.12.123.3:
.Mar 29 22:25:37.822:  leap 3, mode 4, version 3, stratum 0, ppoll 64
.Mar 29 22:25:37.822:  rtdel 0000 (0.000), rtdsp 10001 (1000.015), refid 00000000 (0.0.0.0)
.Mar 29 22:25:37.822:  ref 00000000.00000000 (00:00:00.000 UTC Mon Jan 1 1900)
.Mar 29 22:25:37.822:  org C02AB2EA.4A4B18B3 (02:09:46.290 UTC Sat Mar 2 2002)
.Mar 29 22:25:37.826:  rec DC86B2E1.D1AC185A (22:25:37.819 UTC Wed Mar 29 2017)
.Mar 29 22:25:37.826:  xmt DC86B2E1.D2F33CAE (22:25:37.824 UTC Wed Mar 29 2017)
.Mar 29 22:25:38.820: NTP: rcv packet from 172.12.123.3 to 172.12.123.1 on Serial0/0:
.Mar 29 22:25:38.820:  leap 3, mode 3, version 3, stratum 0, ppoll 64
.Mar 29 22:25:38.820:  rtdel 0000 (0.000), rtdsp 10001 (1000.015), refid 00000000 (0.0.0.0)
.Mar 29 22:25:38.820:  ref 00000000.00000000 (00:00:00.000 UTC Mon Jan 1 1900)
.Mar 29 22:25:38.820:  org DC86B2E1.D2F33CAE (22:25:37.824 UTC Wed Mar 29 2017)
.Mar 29 22:25:38.820:  rec C02AB2EA.593EA3B9 (02:09:46.348 UTC Sat Mar 2 2002)
.Mar 29 22:25:38.824:  xmt C02AB2EB.4AB74497 (02:09:47.291 UTC Sat Mar 2 2002)
.Mar 29 22:25:38.824:  inp DC86B2E2.D215FB3E (22:25:38.820 UTC Wed Mar 29 2017)
.Mar 29 22:25:38.824: NTP: stateless xmit packet to 172.12.123.3:
.Mar 29 22:25:38.824:  leap 3, mode 4, version 3, stratum 0, ppoll 64
.Mar 29 22:25:38.824:  rtdel 0000 (0.000), rtdsp 10001 (1000.015), refid 00000000 (0.0.0.0)
.Mar 29 22:25:38.824:  ref 00000000.00000000 (00:00:00.000 UTC Mon Jan 1 1900)
.Mar 29 22:25:38.824:  org C02AB2EB.4AB74497 (02:09:47.291 UTC Sat Mar 2 2002)
.Mar 29 22:25:38.828:  rec DC86B2E2.D215FB3E (22:25:38.820 UTC Wed Mar 29 2017)
.Mar 29 22:25:38.828:  xmt DC86B2E2.D36249E9 (22:25:38.825 UTC Wed Mar 29 2017)
.Mar 29 22:25:39.817: NTP: rcv packet from 172.12.123.3 to 172.12.123.1 on Serial0/0:
.Mar 29 22:25:39.817:  leap 3, mode 3, version 3, stratum 0, ppoll 64
.Mar 29 22:25:39.817:  rtdel 0000 (0.000), rtdsp 10001 (1000.015), refid 00000000 (0.0.0.0)
.Mar 29 22:25:39.817:  ref 00000000.00000000 (00:00:00.000 UTC Mon Jan 1 1900)
.Mar 29 22:25:39.817:  org DC86B2E2.D36249E9 (22:25:38.825 UTC Wed Mar 29 2017)
.Mar 29 22:25:39.817:  rec C02AB2EB.5BF27F97 (02:09:47.359 UTC Sat Mar 2 2002)
.Mar 29 22:25:39.822:  xmt C02AB2EC.4A1B24F3 (02:09:48.289 UTC Sat Mar 2 2002)
.Mar 29 22:25:39.822:  inp DC86B2E3.D189A67E (22:25:39.818 UTC Wed Mar 29 2017)

Woah! That is what I needed to see, some life from my spokes!! If this does not show the correct time on my NTP routers now, I might just throw my rack off the roof instead of myself. I did check R2 which has the correct time, but no NTP association.

The part that drives me crazy, and as seen below, I will do a “wr” with the new ntp master on their and the interface configs removed, and it will come back up as it should. I may revisit this on an Ethernet only segment to cut out the oddities NBMA might be throwing in there.

R3#wr
Building configuration…

*Mar  2 02:15:21.301: NTP: xmit packet to 172.12.123.1:
*Mar  2 02:15:21.301:  leap 3, mode 3, version 3, stratum 0, ppoll 64
*Mar  2 02:15:21.301:  rtdel 0000 (0.000), rtdsp 10001 (1000.015), refid 00000000 (0.0.0.0)
*Mar  2 02:15:21.301:  ref 00000000.00000000 (00:00:00.000 UTC Mon Jan 1 1900)
*Mar  2 02:15:21.305:  org DC86B3F0.D0EA111D (22:30:08.816 UTC Wed Mar 29 2017)
*Mar  2 02:15:21.305:  rec C02AB3F9.57D528A6 (02:14:17.343 UTC Sat Mar 2 2002)
*Mar  2 02:15:21.305:  xmt C02AB439.4DA1B594 (02:15:21.303 UTC Sat Mar 2 2002)
*Mar  2 02:15:21.506: NTP: rcv packet from 172.12.123.1 to 172.12.123.3 on Serial0/2:
*Mar  2 02:15:21.510:  leap 3, mode 4, version 3, stratum 0, ppoll 64
*Mar  2 02:15:21.510:  rtdel 0000 (0.000), rtdsp 10001 (1000.015), refid 00000000 (0.0.0.0)
*Mar  2 02:15:21.510:  ref 00000000.00000000 (00:00:00.000 UTC Mon Jan 1 1900)
*Mar  2 02:15:21.510:  org C02AB439.4DA1B594 (02:15:21.303 UTC Sat Mar 2 2002)
*Mar  2 02:15:21.510:  rec DC86B430.D46F260E (22:31:12.829 UTC Wed Mar 29 2017)
*Mar  2 02:15:21.510:  xmt DC86B430.D472979E (22:31:12.829 UTC Wed Mar 29 2017)
*Mar  2 02:15:21.514:  inp C02AB439.8260000E (02:15:21.509 UTC Sat Mar 2 2002)[OK]
R3#
R3#

This is what is driving me crazy, is it is receiving the correct time, but then reverts itself back to its old time.

So I will maybe revisit this quickly between R1 and R5 directly connected over a FastEthernet connection to see if it works any better on there, otherwise a few take aways from NTP:

  • Authentication doesn’t really force authentication, so use ACL’s to limit access
  • ACL command to apply to ntp is “ntp access-group #” from global config
  • On the interface level, at this point theoretically, you shouldn’t need to configure an NTP Server but just an interface level command “ntp (broadcast/multicast) client” to listen for NTP broadcasts from a master on that interface
  • NTP can also be disabled (which I did not get to) on the interface level in place of where you configure it to be broadcast or multicast
  • For all intensive purposes, over an NBMA, the server/client model is the way to go

I am going to go blink a few times before bed now, I am not sure exactly the subject matter of the next section, however I really would like to go over this subject again on just an Ethernet segment between R1 and R5 to show how it actually works.

I did want to keep this post up, just to show behaviors of ntp (configurations and command variables of course), but also its odd behaviors for me on my home lab, I assume in the Cisco exam environment they won’t have a shoddy 2621XM Frame Switch serving as their NBMA (I hope) 🙂 That is all for now!

NTP Authentication and ACL configuration, odd behaviors explained, and issues to troubleshoot as always!

OSPF_Base_Topology_NTP

So to begin this, I apparently completely spaced writing R1 so nothing got saved, and we are back in time again:

R1#sh clock
*22:38:45.666 UTC Fri Mar 1 2002
R1#

So I’ve decided to go right into Authentication first, as it’s fairly straight forward with an odd behavior to note, then continue my battle with R4 over the virtual-link to see if I can maybe use the “peer” command between R3 and R4 to resolve the issue.

So I’ve removed all NTP settings from all routers involved in the last lab, as I will need to reconfigure them for authentication. Now to configure authentication, it actually only takes 3 commands on the Master / Server, but 4 commands on the clients as seen here:

R1(config)#ntp authenticate
R1(config)#

^This command sets NTP authentication to run

R1(config)#ntp authentication-key ?
  <1-4294967295>  Key number

R1(config)#ntp authentication-key 1 ?
  md5  MD5 authentication

R1(config)#ntp authentication-key 1 md5 ?
  WORD  Authentication key

R1(config)#ntp authentication-key 1 md5 CCNP ?
  <0-4294967295>  Authentication key encryption type
  <cr>

R1(config)#ntp authentication-key 1 md5 CCNP
R1(config)#

^This command is literally one way to type it, straight forward, CCNP is my keys “password” to authenticate to NTP clients. I have no idea what the last value is, so I will just leave it as configured.
R1(config)#ntp trusted-key ?
  <1-4294967295>  Key number

R1(config)#ntp trusted-key 1 ?
  <cr>

R1(config)#ntp trusted-key 1
R1(config)#

^Again just a very straight forward command, just identifying which one of its keys is a trusted key.

And that is it for the server, it is now “offering” authentication for NTP to potential clients, which sounds odd for authentication as it definitely should.

So on R3 I repeat the same thing:

R3(config)#ntp authenticate
R3(config)#ntp authentication-key 1 md5 CCNP
R3(config)#ntp trusted-key 1
R3(config)#ntp server 172.12.123.1 ? <– The 4th command that is required for clients!
  key      Configure peer authentication key
  prefer   Prefer this peer when possible
  source   Interface for source address
  version  Configure NTP version
  <cr>

R3(config)#ntp server 172.12.123.1 key ?
  <0-4294967295>  Peer key number

R3(config)#ntp server 172.12.123.1 key 1 ?
  prefer   Prefer this peer when possible
  source   Interface for source address
  version  Configure NTP version
  <cr>

R3(config)#ntp server 172.12.123.1 key 1
R3(config)#

So really the NTP clients ONLY additional command to get its time from the server in any case is that mysterious 4th command while R1 has 3 (not including the clock set again).

After waiting a few minutes to see R3 populate, I realized two things: 1. I forgot that yesterday R1 had no neighbor statements for OSPF on my Hub for my spoke routers, and 2. I forgot “ntp master 1” on R1.

So really to set up its 4 commands on each if you include the ntp master on the time server, however it is 3 and 4 if you assume that is part of the normal configuration and adding “key 1” to the “ntp server x.x.x.x …” command.

Now that we’ve beat that horse to death, lets see whats happening on R3:

R3(config)#do sh ntp assoc

      address         ref clock     st  when  poll reach  delay  offset    disp
*~172.12.123.1     .LOCL.            1     1    64   17    65.4   -1.01  1877.2
 * master (synced), # master (unsynced), + selected, – candidate, ~ configured
R3(config)#do sh clock
18:32:54.346 UTC Tue Mar 21 2017
R3(config)#

Alright so that is now working as expected, and I just reconfigured R4 pointed at 172.12.123.1 because it makes no sense from yesterday that it cannot sync up with R1 (which I will visit shortly), but I also configured R2 with absolutely no authentication commands and shortly after I pointed it at R1 as the Time Server we get this:

R2(config)#do sh ntp assoc

      address         ref clock     st  when  poll reach  delay  offset    disp
*~172.12.123.1     .LOCL.            1     9    64  377    53.7   -0.43     0.3
 * master (synced), # master (unsynced), + selected, – candidate, ~ configured
R2(config)#do sh clock
18:39:25.526 UTC Tue Mar 21 2017
R2(config)#

So this is the weird thing with NTP Authentication, is when I say it is “offered” when set by the Master, it is usable without authenticating which sort of defeats the concept of authentication completely – REMEMBER THIS FOR EXAM DAY!

In not so red of text, a client can be configured with no authentication to the time server, and it will still get time from that server (defeating the purpose of authentication).

So I configured R4 with authentication commands and pointed it to R1, and to my surprise:

ASR#4
[Resuming connection 4 to r4 … ]

R4(config)#do sh ntp assoc

address         ref clock       st   when   poll reach  delay  offset   disp
*~172.12.123.1    .LOCL.           1     43     64    77 64.412 -38.023 188.77
* sys.peer, # selected, + candidate, – outlyer, x falseticker, ~ configured
R4(config)#
R4(config)#do sh ntp status
Clock is synchronized, stratum 2, reference is 172.12.123.1
nominal freq is 250.0000 Hz, actual freq is 250.0000 Hz, precision is 2**24
reference time is DC7BF1C5.0072A487 (18:39:01.001 UTC Tue Mar 21 2017)
clock offset is -38.0238 msec, root delay is 64.41 msec
root dispersion is 506.73 msec, peer dispersion is 5.72 msec
loopfilter state is ‘CTRL’ (Normal Controlled Loop), drift is -0.000000202 s/s
system poll interval is 64, last update was 471 sec ago.
R4(config)#

Hooray! No nitty gritty troubleshooting, the lab must know I feel like I am getting sick too!

In the above output of the two “show” verification commands we know of thus far, you see nothing about NTP authentication, but it is all in the “detail” so to say:

R4(config)#do sh ntp assoc detail
172.12.123.1 configured, authenticated, our_master, sane, valid, stratum 1
ref ID .LOCL., time DC7BF403.9FB9E5B6 (18:48:35.623 UTC Tue Mar 21 2017)
our mode client, peer mode server, our poll intvl 64, peer poll intvl 64
root delay 0.00 msec, root disp 0.03, reach 377, sync dist 67.60
delay 64.58 msec, offset -101.6407 msec, dispersion 2.66
precision 2**18, version 4
org time DC7BF405.D9A56AD6 (18:48:37.850 UTC Tue Mar 21 2017)
rec time DC7BF405.FBEF0202 (18:48:37.984 UTC Tue Mar 21 2017)
xmt time DC7BF405.EB34C2F5 (18:48:37.918 UTC Tue Mar 21 2017)
filtdelay =    64.58   64.68   65.65   64.73   64.64   64.76   64.60   64.63
filtoffset = -101.64  -95.55  -87.40  -80.41  -72.88  -65.64  -58.45  -51.39
filterror =     0.00    0.94    1.90    2.88    3.85    4.83    5.80    6.76
minpoll = 6, maxpoll = 10

R4(config)#

I tripped over this message a couple times, because the huge output made me think “sh ntp status” however it is “sh ntp assoc detail” and NOT detail”s”!

So back to authentication not really making hosts authenticate to use the server as a time source, to limit hosts receiving time from our NTP to who we want, Access-Lists’s come to  the rescue!

So the creation of the access-list:

R1(config)#access-list 10 permit 172.12.123.0 0.0.0.255
R1(config)#access-list 10 permit 172.12.34.0 0.0.0.255

Pretty simple, allows R2 / R3 / R4 to get time from R1 with of course the implicit deny at the end to deny all other networks, and now for the NTP portion of applying the ACL:

R1(config)#ntp ?
  access-group        Control NTP access
  authenticate        Authenticate time sources
  authentication-key  Authentication key for trusted time sources
  broadcastdelay      Estimated round-trip delay
  clock-period        Length of hardware clock tick
  logging             Enable NTP message logging
  master              Act as NTP master clock
  max-associations    Set maximum number of associations
  peer                Configure NTP peer
  server              Configure NTP server
  source              Configure interface for source address
  trusted-key         Key numbers for trusted time sources

R1(config)#ntp access-group ?
  peer        Provide full access
  query-only  Allow only control queries
  serve       Provide server and query access
  serve-only  Provide only server access

R1(config)#ntp access-group serve ?
  <1-99>       Standard IP access list
  <1300-1999>  Standard IP access list (expanded range)

R1(config)#ntp access-group serve 10

And that officially applies it, so I just reloaded R2 / R3 / R4 to see how they would come back up and get their time, meanwhile I went on the 172.12.23.0 Ethernet segment and set SW1 to also point to 172.12.123.1 as its NTP server and give it some time to sync until my routers reloaded.

Now all routers have reloaded, lets go around the room, and see who has the correct time:

R2#sh clock
18:34:48.760 UTC Wed Mar 22 2017
R2#
ASR#3
[Resuming connection 3 to r3 … ]

R3#sh clock
18:35:04.335 UTC Wed Mar 22 2017
R3#
ASR#4
[Resuming connection 4 to r4 … ]

R4#sh clock
18:35:14.577 UTC Wed Mar 22 2017
R4#
ASR#5
[Resuming connection 5 to sw1 … ]

SW1#sh clock
*01:30:37.361 UTC Mon Mar 1 1993
SW1#sh ntp assoc

      address         ref clock     st  when  poll reach  delay  offset    disp
 ~172.12.123.1     0.0.0.0          16     –    64    0     0.0    0.00  16000.
 * master (synced), # master (unsynced), + selected, – candidate, ~ configured

Our switch is existing in probably my favorite era of my lifetime, the 90’s, and will remain there unless we let it back into the present (but that might be cruel) 🙂

So I wanted to post up the output of SW1’s “sh ntp status” and “sh ntp assoc det” because there is what I found in my voice days hilarious, but serious NTP status:
SW1#sh ntp status
Clock is unsynchronized, stratum 16, no reference clock
nominal freq is 119.2092 Hz, actual freq is 119.2092 Hz, precision is 2**18
reference time is 00000000.00000000 (00:00:00.000 UTC Mon Jan 1 1900)
clock offset is 0.0000 msec, root delay is 0.00 msec
root dispersion is 0.00 msec, peer dispersion is 0.00 msec

Not a whole lot there, except showing the clock isn’t synchronized, but with “sh ntp assoc det” we see a very… odd and awesome way of putting it:
SW1#sh ntp assoc det
172.12.123.1 configured, insane, invalid, unsynced, stratum 16
ref ID 0.0.0.0, time 00000000.00000000 (00:00:00.000 UTC Mon Jan 1 1900)
our mode client, peer mode unspec, our poll intvl 64, peer poll intvl 64
root delay 0.00 msec, root disp 0.00, reach 0, sync dist 0.000
delay 0.00 msec, offset 0.0000 msec, dispersion 16000.00
precision 2**5, version 3
org time 00000000.00000000 (00:00:00.000 UTC Mon Jan 1 1900)
rcv time 00000000.00000000 (00:00:00.000 UTC Mon Jan 1 1900)
xmt time AF3BE5E5.8494D4E8 (01:31:17.517 UTC Mon Mar 1 1993)
filtdelay =     0.00    0.00    0.00    0.00    0.00    0.00    0.00    0.00
filtoffset =    0.00    0.00    0.00    0.00    0.00    0.00    0.00    0.00
filterror =  16000.0 16000.0 16000.0 16000.0 16000.0 16000.0 16000.0 16000.0

SW1#

This switch is INSANE. When I first saw that, it was so awesome, the terminology just tickled me right in my Cisco soft spot. So what is “insane” you might ask (in terms of Cisco switches)? It is when the a network device is configured with an NTP server to get time from, but cannot reach that time source.

So lets see if we can get this switch SANE, and back to the present date. My brain is already fried like chicken so I accidentally exited until I found myself back at “user priv” (square one) mode, so I reconfigured SW1 to point at 172.12.123.1 for NTP and lets see:

SW1#conf t
Enter configuration commands, one per line.  End with CNTL/Z.
SW1(config)#ntp server 172.12.123.1
SW1(config)#do sh clock
*01:46:27.731 UTC Mon Mar 1 1993

Hmm.. that’s been cooking for about 2-3 minutes prior to that output, so it’s time to investigate this, and as I’ve learned with R4 I need to make sure I can ping R1 first:

SW1#ping 172.12.123.1
% Unrecognized host or address, or protocol not running.

Crap. So I forgot I only gave this a name when I brought it online, so to make it able to ping over to R1 I input the following commands:

SW1#conf t
Enter configuration commands, one per line.  End with CNTL/Z.
SW1(config)#ip routing
SW1(config)#int vlan1
SW1(config-if)#ip address 172.12.23.1 255.255.255.0
SW1(config-if)#no shut
SW1(config-if)#exit
01:50:31: %LINK-3-UPDOWN: Interface Vlan1, changed state to up
01:50:32: %LINEPROTO-5-UPDOWN: Line protocol on Interface Vlan1, changed state to up
SW1(config)#exit
SW1#ping 172.12.123.1

Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 172.12.123.1, timeout is 2 seconds:
…..
Success rate is 0 percent (0/5)
SW1#

So we are routing now, but still no dice, I sense an OSPF network not included issue:

R1#sh ip route ospf
     2.0.0.0/32 is subnetted, 1 subnets
O IA    2.2.2.2 [110/65] via 172.12.123.2, 00:30:22, Serial0/0
     3.0.0.0/32 is subnetted, 1 subnets
O IA    3.3.3.3 [110/65] via 172.12.123.3, 00:30:22, Serial0/0
     4.0.0.0/32 is subnetted, 1 subnets
O IA    4.4.4.4 [110/66] via 172.12.123.3, 00:30:22, Serial0/0
     5.0.0.0/32 is subnetted, 1 subnets
O       5.5.5.5 [110/2] via 172.12.15.5, 01:48:21, FastEthernet0/1
     172.12.0.0/24 is subnetted, 4 subnets
O IA    172.12.34.0 [110/65] via 172.12.123.3, 00:30:22, Serial0/0
O IA    172.12.23.0 [110/65] via 172.12.123.3, 00:30:11, Serial0/0
                    [110/65] via 172.12.123.2, 00:30:11, Serial0/0
     44.0.0.0/32 is subnetted, 1 subnets
O IA    44.44.44.1 [110/66] via 172.12.123.3, 00:30:22, Serial0/0
R1#ping 172.12.23.1

Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 172.12.23.1, timeout is 2 seconds:
…..
Success rate is 0 percent (0/5)
R1#sh access-list
Standard IP access list 10
    10 permit 172.12.123.0, wildcard bits 0.0.0.255 (76 matches)
    20 permit 172.12.34.0, wildcard bits 0.0.0.255 (94 matches)
    30 permit 172.12.23.0, wildcard bits 0.0.0.255
R1#

So in the route table it knows of the 172.12.23.0 network via R2, so I hopped on there:
R2#ping 172.12.23.1

Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 172.12.23.1, timeout is 2 seconds:
!!!!!
Success rate is 100 percent (5/5), round-trip min/avg/max = 1/3/4 ms
R2#ping 172.12.123.1

Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 172.12.123.1, timeout is 2 seconds:
!!!!!
Success rate is 100 percent (5/5), round-trip min/avg/max = 64/66/68 ms
R2#
ASR#5
[Resuming connection 5 to sw1 … ]

SW1#ping 172.12.23.2

Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 172.12.23.2, timeout is 2 seconds:
!!!!!
Success rate is 100 percent (5/5), round-trip min/avg/max = 1/2/8 ms
SW1#

So R2 can ping R1 and SW1, and is the go between for them, and SW1 can ping R2 which is its middle man to R1. So I go back to SW1 to give a traceroute a try to see if it’s even getting a response from R2 when sending traffic:

SW1#traceroute 172.12.123.1

Type escape sequence to abort.
Tracing the route to 172.12.123.1

  1  *  *  *
  2  *  *  *
  3  *  *
SW1#sh ip route

Gateway of last resort is not set

     172.12.0.0/24 is subnetted, 1 subnets
C       172.12.23.0 is directly connected, Vlan1

There’s the issue, it has no route to 172.12.123.1, so at this point of going brain dead from work / study I will just create the static route to bring sanity back to this switch:

SW1(config)#ip route 172.12.123.0 255.255.255.0 172.12.23.2
SW1(config)#exit
SW1#ping 172.12.123.1

Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 172.12.123.1, timeout is 2 seconds:
!!!!!

I love when logic works, makes my head hurt less. So I issued a write and reload, and after it booted back up, this is how the clock is now looking:

SW1#sh clock
*00:01:08.501 UTC Mon Mar 1 1993
SW1#sh ntp assoc

      address         ref clock     st  when  poll reach  delay  offset    disp
 ~172.12.123.1     0.0.0.0          16     –    64    0     0.0    0.00  16000.
 * master (synced), # master (unsynced), + selected, – candidate, ~ configured
SW1#

So starting to lose my mind wondering why on Earth this thing will not sync, I got on R1 and ran “debug ntp packet” and I’ll spare you all the output EXCEPT WHEN SW1 FINALLY HIT THIS SUCKER AND GOT BROUGHT BACK FROM THE FUTURE… OR PAST… WHICHEVER:

.Mar 22 19:16:31.578: NTP: rcv packet from 172.12.23.1 to 172.12.123.1 on Serial0/0:
.Mar 22 19:16:31.578:  leap 3, mode 3, version 3, stratum 0, ppoll 64
.Mar 22 19:16:31.578:  rtdel 0000 (0.000), rtdsp 10001 (1000.015), refid 00000000 (0.0.0.0)
.Mar 22 19:16:31.578:  ref 00000000.00000000 (00:00:00.000 UTC Mon Jan 1 1900)
.Mar 22 19:16:31.578:  org 00000000.00000000 (00:00:00.000 UTC Mon Jan 1 1900)
.Mar 22 19:16:31.582:  rec 000
R1#00000.00000000 (00:00:00.000 UTC Mon Jan 1 1900)
.Mar 22 19:16:31.582:  xmt AF3BD148.1181126D (00:03:20.068 UTC Mon Mar 1 1993)
.Mar 22 19:16:31.582:  inp DC7D4C0F.945A1DF3 (19:16:31.579 UTC Wed Mar 22 2017)
.Mar 22 19:16:31.582: NTP: stateless xmit packet to 172.12.23.1:
.Mar 22 19:16:31.582:  leap 3, mode 4, version 3, stratum 0, ppoll 64
.Mar 22 19:16:31.582:  rtdel 0000 (0.000), rtdsp 6002 (375.031), refid 4C4F434C (76.79.67.76)
.Mar 22 19:16:31.586:  ref DC7D400B.EEB63084 (18:25:15.932 UTC Wed Mar 22 2017)
.Mar 22 19:16:31.586:  org AF3BD148.1181126D (00:03:20.068 UTC Mon Mar 1 1993)
.Mar 22 19:16:31.586:  rec DC7D4C0F.945A1DF3 (19:16:31.579 UTC Wed Mar 22 2017)
.Mar 22 19:16:31.586:  xmt DC7D4C0F.95A82567 (19:16:31.584 UTC Wed Mar 22 2017)
R1#u all
R1#

I left the big crap ton of output to see the exchange and references, I am not sure what most of it means, but it does show the switches old time and the current time being exchanged (as well as referring to 1900 for some odd reason).

So when we go back to SW1, we should finally have some sanity, before I lose mine:

SW1#sh clock
*00:10:24.859 UTC Mon Mar 1 1993
SW1#sh ntp assoc det
172.12.123.1 configured, insane, invalid, unsynced, stratum 16
ref ID 0.0.0.0, time 00000000.00000000 (00:00:00.000 UTC Mon Jan 1 1900)
our mode client, peer mode unspec, our poll intvl 64, peer poll intvl 64
root delay 0.00 msec, root disp 0.00, reach 0, sync dist 0.000
delay 0.00 msec, offset 0.0000 msec, dispersion 16000.00
precision 2**5, version 3
org time DC7D4D8F.933D330C (19:22:55.575 UTC Wed Mar 22 2017)
rcv time AF3BD2C8.1E2EB9CE (00:09:44.117 UTC Mon Mar 1 1993)
xmt time AF3BD2C8.10B78FA5 (00:09:44.065 UTC Mon Mar 1 1993)
filtdelay =     0.00    0.00    0.00    0.00    0.00    0.00    0.00    0.00
filtoffset =    0.00    0.00    0.00    0.00    0.00    0.00    0.00    0.00
filterror =  16000.0 16000.0 16000.0 16000.0 16000.0 16000.0 16000.0 16000.0

SW1#

What??? If you examine the details, other than insanity still being awesome terminology, you can see what I’ve highlighted in red that it does have the correct network time in the output which its getting from R1, but show clock does not show the correct time.

So at this point, I did try to put “router ospf 1” on the switch which it does show as a valid command, but it does not drop me into ospf configuration mode to see if that gives it the kick it needs. You can even see it getting hits on R1’s ACL for NTP requests:

R1#sh access-list
Standard IP access list 10
    10 permit 172.12.123.0, wildcard bits 0.0.0.255 (130 matches)
    20 permit 172.12.34.0, wildcard bits 0.0.0.255 (105 matches)
    30 permit 172.12.23.0, wildcard bits 0.0.0.255 (15 matches)
R1#

So at this point, the next segment of my course is running NTP in broadcast mode, and I want to see what that has to say there to see if we can maybe salvage this SW1 situation.

NTP – Network Time Protocol discussion, configuration, and some NTP / routing troubleshooting as usual!

OSPF_Base_Topology_NTP

For this lab I will be using this OSPF Topology, which has the virtual-link to R4 to bring it into the entirety of the network, and to demonstrate how to configure each of these routers so they keep in sync with the entire network via NTP even if their primary time source goes down.

So this is a subject near and dear to my heart, working previously a lot with Cisco Voice systems / servers, if the network NTP is not near perfectly synchronized your phones / voicemails / servers are going to go absolutely haywire.

In the CCNP ROUTE context, I believe it is mostly beneficial for troubleshooting so your logging is on the same time, so you don’t have to compare different time frames to guess where the event happened on different devices because they had different times – I know this because network time on Small / Medium sized businesses can be off an amazing amount of time.

To begin any NTP discussion you have to first begin with “Stratum” and what it is. Stratum is the metric (like hop count) as to how close to a Stratum 0 device you are to gauge how accurate your time is, this again is especially important for VOIP as I believe only Stratum 3 or lower is acceptable for time differential.

So with Stratum, the lower the better, the best obviously being Stratum 0 which are actually referred to as atomic clocks that are the size of datacenters on naval bases. You will not be able to connect directly to a stratum 0 device (or make a router stratum 0 as you will see), but there are “Time Servers” on the internet that are Stratum 1, that you can point your edge device to (or multiple of them in case one disappears).

Each hop you get away from that Time Server, the more your “Stratum #” will increase when you are running show commands for NTP on your router, and goes up to a maximum of 15 which means as unreliable as it gets before Stratum 16 which means unreachable or unreliable. Now for a couple important notes before we dive into some configuration:

  • When you “write erase” / “reload” a device to wipe it, you are wiping the time, so in the real world or in your lab don’t forget it needs to be reset or chaos will ensue!
  • NTP uses UDP port 123, so do block it on the devices on your network

Also worth a bullet point style explanation, are the 3 different types of NTP router:

  • NTP Server – Set time on this device, it will send out Time Sync messages to NTP clients on the network
  • NTP Client – Receives Time Sync messages from Server, DOES NOT send time sync messages back to server
  • NTP Peer – Can be both Client and Peer, Peers can share time with eachother

NTP can be run in broadcast mode, or multicast mode, depending on your network needs. Its odd that this part of the topic is kind of just left at you needing to figure out what works with your network best, so I imagine trying to lab it over an NBMA running OSPF should be fun!

Also a note on configuring an NTP Server or “Master” as your edge device, it is highly recommended that you not only use a public time server(s) as your time source as opposed to setting it, but also it is necessary to use authentication and / or ACL’s to stop other routers from using ours as an NTP time source even if it is just for time synching reasons we don’t want the extra workload on the edge device.

Ahhhh, the perfect segway into the security side of NTP 🙂 However for all this discussion, there has been no labbing, and not labbing bores me to tears so lets get into some configuration and see what we can break:

R1#sh ntp assoc
R1#sh clock
*23:21:30.839 UTC Fri Mar 1 2002
R1#

As show with “sh ntp associations” we have nothing configured with any other routers (including this one), and we have traveled back 15 years to 2002. So given that my lab is connected to the internet, I am going to set R1 as the NTP Server / Master, and all other routers as clients / peers.

So first, we need to get R1 rocking as the NTP Master of the network, so lets get that done:

(I actually forgot how to set the time and referred back to my time-based ACL’s post, and that’s why I say it over and over it’s so important to start your own blog or something equal in being able to refer back to examples of these things quickly!!)

R1#clock set 19:43:00 20 mar 2017
R1#
*Mar 20 19:43:00.000: %SYS-6-CLOCKUPDATE: System clock has been updated from 23:30:06 UTC Fri Mar 1 2002 to 19:43:00 UTC Mon Mar 20 2017, configured from console by console.
R1#conf t
Enter configuration commands, one per line.  End with CNTL/Z.
R1(config)#ntp ?
  access-group        Control NTP access
  authenticate        Authenticate time sources
  authentication-key  Authentication key for trusted time sources
  broadcastdelay      Estimated round-trip delay
  clock-period        Length of hardware clock tick
  logging             Enable NTP message logging
  master              Act as NTP master clock
  max-associations    Set maximum number of associations
  peer                Configure NTP peer
  server              Configure NTP server
  source              Configure interface for source address
  trusted-key         Key numbers for trusted time sources

R1(config)#ntp master ?
  <1-15>  Stratum number
  <cr>

R1(config)#ntp master 1 ?
  <cr>

R1(config)#ntp master
R1(config)#

A couple things here, I left the ? output after NTP so you can see the command modifiers of which I used master, but as I went on also that my options were 1-15 from most trusted to least. I was going to make it Stratum 1 but I left off the # as I curious what Stratum # it gets by default when configured as an NTP master:

R1(config)#do sh ntp assoc

      address         ref clock     st  when  poll reach  delay  offset    disp
*~127.127.7.1      127.127.7.1       7    51    64  377     0.0    0.00     0.0
 * master (synced), # master (unsynced), + selected, – candidate, ~ configured
R1(config)#

Ha, so this router isn’t giving me too much credibility, under “st” which refers to Stratum # is 7, so I am neither very trusted or very unreliable. I’ll switch it back so that it is a Stratum 1, because I like my NTP master of my network to have some credibility.

Also what I have highlighted in red is very important, especially the * when it comes to clients, because that means they are fully synced with that address. Being this is the master it uses its loopback 127.127.7.1, so if you see * in “sh ntp assoc” with a loopback address you know that router is set as the NTP Master.

Now lets configure R2 and R3 over our NBMA and our Area 0 in OSPF as clients of R1:

R2#conf t
Enter configuration commands, one per line.  End with CNTL/Z.
R2(config)#ntp ?
  access-group        Control NTP access
  authenticate        Authenticate time sources
  authentication-key  Authentication key for trusted time sources
  broadcastdelay      Estimated round-trip delay
  clock-period        Length of hardware clock tick
  logging             Enable NTP message logging
  master              Act as NTP master clock
  max-associations    Set maximum number of associations
  peer                Configure NTP peer
  server              Configure NTP server
  source              Configure interface for source address
  trusted-key         Key numbers for trusted time sources

R2(config)#ntp server ?
  Hostname or A.B.C.D  IP address of peer
  vrf                  VPN Routing/Forwarding Information

R2(config)#ntp server 172.12.123.1 ?
  key      Configure peer authentication key
  prefer   Prefer this peer when possible
  source   Interface for source address
  version  Configure NTP version
  <cr>

R2(config)#ntp server 172.12.123.1
R2(config)#
ASR#3
[Resuming connection 3 to r3 … ]

R3#conf t
Enter configuration commands, one per line.  End with CNTL/Z.
R3(config)#ntp server 172.12.123.1
R3(config)#do sh ntp assoc

      address         ref clock     st  when  poll reach  delay  offset    disp
 ~172.12.123.1     0.0.0.0          16     –    64    0     0.0    0.00  16000.
 * master (synced), # master (unsynced), + selected, – candidate, ~ configured
R3(config)#

I’ve highlighted the “prefer” option after pointing at 172.12.123.1, to show that it is there, and is used when assigning multiple NTP servers as backup but you prefer to use a specific server as the time source.

Also highlighted in red, before it could sync I did a quick “sh ntp assoc” to demonstrate what it looks like when a router is not synced, and why I stressed seeing the * next to the IP address means that it is fully synced.

Also, that Stratum 16 is the equivalent to RIP’s metric of 16, it’s not even barely reliable but is actually an invalid time source at Stratum 16 – This is important to note!

So let’s see if we have some synchronization going on with R2 and R3:

R2(config)#do sh ntp assoc

      address         ref clock     st  when  poll reach  delay  offset    disp
*~172.12.123.1     .LOCL.            1     1    64  377    53.5   -1.53     0.4
 * master (synced), # master (unsynced), + selected, – candidate, ~ configured
R2(config)#do sh clock
20:03:17.432 UTC Mon Mar 20 2017
R2(config)#
ASR#3
[Resuming connection 3 to r3 … ]

R3(config)#do sh ntp assoc

      address         ref clock     st  when  poll reach  delay  offset    disp
*~172.12.123.1     .LOCL.            1    18    64  377    53.5   -3.38     0.5
 * master (synced), # master (unsynced), + selected, – candidate, ~ configured
R3(config)#do sh clock
20:03:39.637 UTC Mon Mar 20 2017
R3(config)#

Both are looking good, except I do see it references this again as UTC time, and for lab purposes I do not intend to spend time digging into getting the time into my timezone 🙂

Another important command to know for checking NTP settings on the local router is “sh ntp status” as demonstrated here:

R3#sh ntp status
Clock is synchronized, stratum 2, reference is 172.12.123.1
nominal freq is 249.5901 Hz, actual freq is 249.5903 Hz, precision is 2**18
reference time is DC7AB544.FFF63BFD (20:08:36.999 UTC Mon Mar 20 2017)
clock offset is -3.8317 msec, root delay is 53.48 msec
root dispersion is 4.06 msec, peer dispersion is 0.18 msec
R3#

So we see it’s synchronized now back to the server from R2 and R3 which right now is just using the Client / Server model, however I had to have my first Derp of the night – Configuring R4 correctly for NTP however not testing connectivity to the server address:

R4(config)#do sh ntp assoc

  address         ref clock       st   when   poll reach  delay  offset   disp
 ~172.12.123.1    .INIT.          16      –     64     0  0.000   0.000 15937.
 * sys.peer, # selected, + candidate, – outlyer, x falseticker, ~ configured
R4(config)#do ping 172.12.123.1
Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 172.12.123.1, timeout is 2 seconds:
…..
Success rate is 0 percent (0/5)
R4(config)#

The sad part is I’ve been waiting for about 5 minutes or so for that to sync, because I know it can take 5+ minutes, but a ping even before ntp configuration would have been a good way to start the configuration! So lets take a look at R3:

R3#sh ip int bri
Interface                  IP-Address      OK? Method Status                Protocol
FastEthernet0/0            172.12.23.3     YES NVRAM  up                    up
FastEthernet0/1            172.12.34.3     YES NVRAM  up                    up
Serial0/2                  172.12.123.3    YES NVRAM  up                    up
Serial0/3                  unassigned      YES NVRAM  administratively down down
Loopback3                  3.3.3.3         YES NVRAM  up                    up
R3#ping 4.4.4.4

Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 4.4.4.4, timeout is 2 seconds:
!!!!!
Success rate is 100 percent (5/5), round-trip min/avg/max = 1/2/4 ms
R3#sh ip ospf nei

Neighbor ID     Pri   State           Dead Time   Address         Interface
44.44.44.1        0   FULL/  –           –        172.12.34.4     OSPF_VL0
2.2.2.2           1   FULL/BDR        00:00:38    172.12.23.2     FastEthernet0/0
44.44.44.1        1   FULL/DR         00:00:38    172.12.34.4     FastEthernet0/1
R3#sh ip proto
Routing Protocol is “ospf 1”
  Outgoing update filter list for all interfaces is not set
  Incoming update filter list for all interfaces is not set
  Router ID 3.3.3.3
  It is an area border router
  Number of areas in this router is 4. 4 normal 0 stub 0 nssa
  Maximum path: 4
  Routing for Networks:
    3.3.3.3 0.0.0.0 area 3
    172.12.23.0 0.0.0.255 area 23
    172.12.34.0 0.0.0.255 area 34
    172.12.123.0 0.0.0.255 area 0
 Reference bandwidth unit is 100 mbps
  Routing Information Sources:
    Gateway         Distance      Last Update
    44.44.44.1           110      01:58:30
  Distance: (default is 110)

R3#sh ip ospf virtual-link
Virtual Link OSPF_VL0 to router 44.44.44.1 is up
  Run as demand circuit
  DoNotAge LSA allowed.
  Transit area 34, via interface FastEthernet0/1, Cost of using 1
  Transmit Delay is 1 sec, State POINT_TO_POINT,
  Timer intervals configured, Hello 10, Dead 40, Wait 40, Retransmit 5
    Hello due in 00:00:03
    Adjacency State FULL (Hello suppressed)
    Index 1/2, retransmission queue length 0, number of retransmission 0
    First 0x0(0)/0x0(0) Next 0x0(0)/0x0(0)
    Last retransmission scan length is 0, maximum is 0
    Last retransmission scan time is 0 msec, maximum is 0 msec
R3#ping 172.12.123.1

Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 172.12.123.1, timeout is 2 seconds:
!!!!!
Success rate is 100 percent (5/5), round-trip min/avg/max = 64/66/68 ms
R3#

Uhhhh…. So R4 has some issue? It should have been ruled out by R3 pinging it’s loopback, but I’ll play ball, lets take a look:

R4#traceroute 172.12.123.1
Type escape sequence to abort.
Tracing the route to 172.12.123.1
VRF info: (vrf in name/id, vrf out name/id)
  1 172.12.34.3 4 msec 0 msec 4 msec
  2  *  *  *
  3  *  *  *
  4  *  *
R4#
R4#sh ip route ospf

Gateway of last resort is not set

      3.0.0.0/32 is subnetted, 1 subnets
O IA     3.3.3.3 [110/2] via 172.12.34.3, 02:04:27, FastEthernet0/1
      172.12.0.0/16 is variably subnetted, 4 subnets, 2 masks
O IA     172.12.23.0/24 [110/2] via 172.12.34.3, 02:03:49, FastEthernet0/1
O        172.12.123.0/24 [110/65] via 172.12.34.3, 02:04:27, FastEthernet0/1
R4#
R4#ping 172.12.123.3
Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 172.12.123.3, timeout is 2 seconds:
!!!!!
Success rate is 100 percent (5/5), round-trip min/avg/max = 1/2/4 ms
R4#ping 172.12.123.1
Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 172.12.123.1, timeout is 2 seconds:
…..
Success rate is 0 percent (0/5)
R4#

So it is being processed by R3, as it can ping the serial interface on the NBMA network, so the issue is either somewhere on the frame switch or on R1. It’s too late and I’m too fried to really want to dig into this weird behavior (I will just wipe / reconfigure them if push comes to shove), but for the heck of it lets look at R1:

!
interface Serial0/1
 ip address 172.12.13.1 255.255.255.252
 clock rate 2000000
!
router ospf 1
 log-adjacency-changes
 area 34 virtual-link 44.44.44.1 <— What???
 network 1.1.1.1 0.0.0.0 area 1
 network 172.12.15.0 0.0.0.255 area 15
 network 172.12.123.0 0.0.0.255 area 0
!
!

I am wondering if I was so tired configuring this, that I entered that command on R1, and that is what is jamming up the traffic, lets put a stop to this silliness:

R1#conf t
Enter configuration commands, one per line.  End with CNTL/Z.
R1(config)#router ospf 1
R1(config-router)#no area 34 virtual-link 44.44.44.1
R1(config-router)#

I can confirm it is now gone, lets see about some pings:

R1(config-router)#do ping 4.4.4.4

Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 4.4.4.4, timeout is 2 seconds:
…..
Success rate is 0 percent (0/5)
R1(config-router)#do ping 172.12.34.4

Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 172.12.34.4, timeout is 2 seconds:
…..
Success rate is 0 percent (0/5)
R1(config-router)#do ping 172.12.34.3

Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 172.12.34.3, timeout is 2 seconds:
…..
Success rate is 0 percent (0/5)
R1(config-router)#do ping 172.12.123.3

Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 172.12.123.3, timeout is 2 seconds:
!!!!!
Success rate is 100 percent (5/5), round-trip min/avg/max = 64/66/68 ms
R1(config-router)#

So R3 will process packets from R4 to the IP on its NBMA serial Interface, but R1 cannot ping to the Fa0/1 interface of Area 34. Let me stare at the show run for a moment, I am losing my sense of humor now 🙂

Wow, I must have been half asleep, I found some leftover access-lists from when I was doing that section on R3 that was messing with traffic, let me remove these AND LET THE TRAFFIC FLOWETH THROUGHOUT THE NETWORK!!! :

R3#conf t
Enter configuration commands, one per line.  End with CNTL/Z.
R3(config)#no access-list 15
R3(config)#no access-list 111
R3(config)#int fa0/1
R3(config-if)#no ip access-group 111 in
R3(config-if)#no ip access-group 15 out
R3(config-if)#
ASR#4
[Resuming connection 4 to r4 … ]

R4#ping 172.12.123.1
Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 172.12.123.1, timeout is 2 seconds:
…..
Success rate is 0 percent (0/5)

GAH!!!

So at this point, I’m giving R3 a write / reload, and if R4 cannot ping the server, I’ll have to wipe and confirm cabling / reconfigure the network as the previously labs are having impact somewhere in the network.

So as R3 loaded back up, and I watched the Adjacency form, I never saw the relationship back to Area 0 on the NBMA network form (I even waited the 10 years it takes over serial links):

R3>en
Password:
R3#
Mar 20 21:16:43.574: %OSPF-5-ADJCHG: Process 1, Nbr 44.44.44.1 on FastEthernet0/1 from LOADING to FULL, Loading Done
R3#
Mar 20 21:16:46.915: %OSPF-5-ADJCHG: Process 1, Nbr 2.2.2.2 on FastEthernet0/0 from LOADING to FULL, Loading Done
R3#
Mar 20 21:16:58.655: %OSPF-5-ADJCHG: Process 1, Nbr 44.44.44.1 on OSPF_VL0 from LOADING to FULL, Loading Done
R3#sh ip ospf nei

Neighbor ID     Pri   State           Dead Time   Address         Interface
44.44.44.1        0   FULL/  –           –        172.12.34.4     OSPF_VL0
2.2.2.2           1   FULL/DR         00:00:37    172.12.23.2     FastEthernet0/0
44.44.44.1        1   FULL/DR         00:00:37    172.12.34.4     FastEthernet0/1
R3#

So now it is an OSPF issue that R4 is not getting it’s time, which raises another good point, that time servers should not be reliant (if possible) on a dynamic protocol to reach it’s time source.

So for the hell of it since its this late and I’m already this friend, I might as well try to see this though to the gruesome end, so I took at what routes it DOES see and this is what I got:

R1#sh ip route

Gateway of last resort is not set

     1.0.0.0/32 is subnetted, 1 subnets
C       1.1.1.1 is directly connected, Loopback1
     5.0.0.0/32 is subnetted, 1 subnets
O       5.5.5.5 [110/2] via 172.12.15.5, 00:28:51, FastEthernet0/1
     172.12.0.0/24 is subnetted, 2 subnets
C       172.12.15.0 is directly connected, FastEthernet0/1
C       172.12.123.0 is directly connected, Serial0/0
R1#

Nothing over my NBMA. OMG I forgot the neighbor statements on R1 in my tired stupor, *slams head against desk* :

R1(config)#router ospf 1
R1(config-router)#neighbor 172.12.123.2
R1(config-router)#neighbor 172.12.123.3
R1(config-router)#
.Mar 20 21:31:26.994: %OSPF-5-ADJCHG: Process 1, Nbr 2.2.2.2 on Serial0/0 from LOADING to FULL, Loading Done
.Mar 20 21:31:27.102: %OSPF-5-ADJCHG: Process 1, Nbr 3.3.3.3 on Serial0/0 from LOADING to FULL, Loading Done
R1(config-router)#
ASR#4
[Resuming connection 4 to r4 … ]

*Mar 21 01:56:05.859: %OSPF-5-ADJCHG: Process 1, Nbr 3.3.3.3 on OSPF_VL0 from FULL to DOWN, Neighbor Down: Interface down or detached
R4#
*Mar 21 01:56:12.583: %LINEPROTO-5-UPDOWN: Line protocol on Interface FastEthernet0/1, changed state to down
*Mar 21 01:56:12.583: %OSPF-5-ADJCHG: Process 1, Nbr 3.3.3.3 on FastEthernet0/1 from FULL to DOWN, Neighbor Down: Interface down or detached
R4#ping 172.12.123.1
Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 172.12.123.1, timeout is 2 seconds:
!!!!!
Success rate is 100 percent (5/5), round-trip min/avg/max = 64/65/68 ms
R4#sh ntp assoc

  address         ref clock       st   when   poll reach  delay  offset   disp
 ~172.12.123.1    .INIT.          16      –   1024     0  0.000   0.000 15937.
 * sys.peer, # selected, + candidate, – outlyer, x falseticker, ~ configured
R4#sh ntp assoc

  address         ref clock       st   when   poll reach  delay  offset   disp
 ~172.12.123.1    .INIT.          16      –   1024     0  0.000   0.000 15937.
 * sys.peer, # selected, + candidate, – outlyer, x falseticker, ~ configured

So it is just not synching for whatever reason I don’t care to dig in now that its 10pm, so I will continue this battle in my next post, which will include NTP Authentication!