For this lab I will be using this OSPF Topology, which has the virtual-link to R4 to bring it into the entirety of the network, and to demonstrate how to configure each of these routers so they keep in sync with the entire network via NTP even if their primary time source goes down.
So this is a subject near and dear to my heart, working previously a lot with Cisco Voice systems / servers, if the network NTP is not near perfectly synchronized your phones / voicemails / servers are going to go absolutely haywire.
In the CCNP ROUTE context, I believe it is mostly beneficial for troubleshooting so your logging is on the same time, so you don’t have to compare different time frames to guess where the event happened on different devices because they had different times – I know this because network time on Small / Medium sized businesses can be off an amazing amount of time.
To begin any NTP discussion you have to first begin with “Stratum” and what it is. Stratum is the metric (like hop count) as to how close to a Stratum 0 device you are to gauge how accurate your time is, this again is especially important for VOIP as I believe only Stratum 3 or lower is acceptable for time differential.
So with Stratum, the lower the better, the best obviously being Stratum 0 which are actually referred to as atomic clocks that are the size of datacenters on naval bases. You will not be able to connect directly to a stratum 0 device (or make a router stratum 0 as you will see), but there are “Time Servers” on the internet that are Stratum 1, that you can point your edge device to (or multiple of them in case one disappears).
Each hop you get away from that Time Server, the more your “Stratum #” will increase when you are running show commands for NTP on your router, and goes up to a maximum of 15 which means as unreliable as it gets before Stratum 16 which means unreachable or unreliable. Now for a couple important notes before we dive into some configuration:
- When you “write erase” / “reload” a device to wipe it, you are wiping the time, so in the real world or in your lab don’t forget it needs to be reset or chaos will ensue!
- NTP uses UDP port 123, so do block it on the devices on your network
Also worth a bullet point style explanation, are the 3 different types of NTP router:
- NTP Server – Set time on this device, it will send out Time Sync messages to NTP clients on the network
- NTP Client – Receives Time Sync messages from Server, DOES NOT send time sync messages back to server
- NTP Peer – Can be both Client and Peer, Peers can share time with eachother
NTP can be run in broadcast mode, or multicast mode, depending on your network needs. Its odd that this part of the topic is kind of just left at you needing to figure out what works with your network best, so I imagine trying to lab it over an NBMA running OSPF should be fun!
Also a note on configuring an NTP Server or “Master” as your edge device, it is highly recommended that you not only use a public time server(s) as your time source as opposed to setting it, but also it is necessary to use authentication and / or ACL’s to stop other routers from using ours as an NTP time source even if it is just for time synching reasons we don’t want the extra workload on the edge device.
Ahhhh, the perfect segway into the security side of NTP 🙂 However for all this discussion, there has been no labbing, and not labbing bores me to tears so lets get into some configuration and see what we can break:
R1#sh ntp assoc
R1#sh clock
*23:21:30.839 UTC Fri Mar 1 2002
R1#
As show with “sh ntp associations” we have nothing configured with any other routers (including this one), and we have traveled back 15 years to 2002. So given that my lab is connected to the internet, I am going to set R1 as the NTP Server / Master, and all other routers as clients / peers.
So first, we need to get R1 rocking as the NTP Master of the network, so lets get that done:
(I actually forgot how to set the time and referred back to my time-based ACL’s post, and that’s why I say it over and over it’s so important to start your own blog or something equal in being able to refer back to examples of these things quickly!!)
R1#clock set 19:43:00 20 mar 2017
R1#
*Mar 20 19:43:00.000: %SYS-6-CLOCKUPDATE: System clock has been updated from 23:30:06 UTC Fri Mar 1 2002 to 19:43:00 UTC Mon Mar 20 2017, configured from console by console.
R1#conf t
Enter configuration commands, one per line. End with CNTL/Z.
R1(config)#ntp ?
access-group Control NTP access
authenticate Authenticate time sources
authentication-key Authentication key for trusted time sources
broadcastdelay Estimated round-trip delay
clock-period Length of hardware clock tick
logging Enable NTP message logging
master Act as NTP master clock
max-associations Set maximum number of associations
peer Configure NTP peer
server Configure NTP server
source Configure interface for source address
trusted-key Key numbers for trusted time sources
R1(config)#ntp master ?
<1-15> Stratum number
<cr>
R1(config)#ntp master 1 ?
<cr>
R1(config)#ntp master
R1(config)#
A couple things here, I left the ? output after NTP so you can see the command modifiers of which I used master, but as I went on also that my options were 1-15 from most trusted to least. I was going to make it Stratum 1 but I left off the # as I curious what Stratum # it gets by default when configured as an NTP master:
R1(config)#do sh ntp assoc
address ref clock st when poll reach delay offset disp
*~127.127.7.1 127.127.7.1 7 51 64 377 0.0 0.00 0.0
* master (synced), # master (unsynced), + selected, – candidate, ~ configured
R1(config)#
Ha, so this router isn’t giving me too much credibility, under “st” which refers to Stratum # is 7, so I am neither very trusted or very unreliable. I’ll switch it back so that it is a Stratum 1, because I like my NTP master of my network to have some credibility.
Also what I have highlighted in red is very important, especially the * when it comes to clients, because that means they are fully synced with that address. Being this is the master it uses its loopback 127.127.7.1, so if you see * in “sh ntp assoc” with a loopback address you know that router is set as the NTP Master.
Now lets configure R2 and R3 over our NBMA and our Area 0 in OSPF as clients of R1:
R2#conf t
Enter configuration commands, one per line. End with CNTL/Z.
R2(config)#ntp ?
access-group Control NTP access
authenticate Authenticate time sources
authentication-key Authentication key for trusted time sources
broadcastdelay Estimated round-trip delay
clock-period Length of hardware clock tick
logging Enable NTP message logging
master Act as NTP master clock
max-associations Set maximum number of associations
peer Configure NTP peer
server Configure NTP server
source Configure interface for source address
trusted-key Key numbers for trusted time sources
R2(config)#ntp server ?
Hostname or A.B.C.D IP address of peer
vrf VPN Routing/Forwarding Information
R2(config)#ntp server 172.12.123.1 ?
key Configure peer authentication key
prefer Prefer this peer when possible
source Interface for source address
version Configure NTP version
<cr>
R2(config)#ntp server 172.12.123.1
R2(config)#
ASR#3
[Resuming connection 3 to r3 … ]
R3#conf t
Enter configuration commands, one per line. End with CNTL/Z.
R3(config)#ntp server 172.12.123.1
R3(config)#do sh ntp assoc
address ref clock st when poll reach delay offset disp
~172.12.123.1 0.0.0.0 16 – 64 0 0.0 0.00 16000.
* master (synced), # master (unsynced), + selected, – candidate, ~ configured
R3(config)#
I’ve highlighted the “prefer” option after pointing at 172.12.123.1, to show that it is there, and is used when assigning multiple NTP servers as backup but you prefer to use a specific server as the time source.
Also highlighted in red, before it could sync I did a quick “sh ntp assoc” to demonstrate what it looks like when a router is not synced, and why I stressed seeing the * next to the IP address means that it is fully synced.
Also, that Stratum 16 is the equivalent to RIP’s metric of 16, it’s not even barely reliable but is actually an invalid time source at Stratum 16 – This is important to note!
So let’s see if we have some synchronization going on with R2 and R3:
R2(config)#do sh ntp assoc
address ref clock st when poll reach delay offset disp
*~172.12.123.1 .LOCL. 1 1 64 377 53.5 -1.53 0.4
* master (synced), # master (unsynced), + selected, – candidate, ~ configured
R2(config)#do sh clock
20:03:17.432 UTC Mon Mar 20 2017
R2(config)#
ASR#3
[Resuming connection 3 to r3 … ]
R3(config)#do sh ntp assoc
address ref clock st when poll reach delay offset disp
*~172.12.123.1 .LOCL. 1 18 64 377 53.5 -3.38 0.5
* master (synced), # master (unsynced), + selected, – candidate, ~ configured
R3(config)#do sh clock
20:03:39.637 UTC Mon Mar 20 2017
R3(config)#
Both are looking good, except I do see it references this again as UTC time, and for lab purposes I do not intend to spend time digging into getting the time into my timezone 🙂
Another important command to know for checking NTP settings on the local router is “sh ntp status” as demonstrated here:
R3#sh ntp status
Clock is synchronized, stratum 2, reference is 172.12.123.1
nominal freq is 249.5901 Hz, actual freq is 249.5903 Hz, precision is 2**18
reference time is DC7AB544.FFF63BFD (20:08:36.999 UTC Mon Mar 20 2017)
clock offset is -3.8317 msec, root delay is 53.48 msec
root dispersion is 4.06 msec, peer dispersion is 0.18 msec
R3#
So we see it’s synchronized now back to the server from R2 and R3 which right now is just using the Client / Server model, however I had to have my first Derp of the night – Configuring R4 correctly for NTP however not testing connectivity to the server address:
R4(config)#do sh ntp assoc
address ref clock st when poll reach delay offset disp
~172.12.123.1 .INIT. 16 – 64 0 0.000 0.000 15937.
* sys.peer, # selected, + candidate, – outlyer, x falseticker, ~ configured
R4(config)#do ping 172.12.123.1
Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 172.12.123.1, timeout is 2 seconds:
…..
Success rate is 0 percent (0/5)
R4(config)#
The sad part is I’ve been waiting for about 5 minutes or so for that to sync, because I know it can take 5+ minutes, but a ping even before ntp configuration would have been a good way to start the configuration! So lets take a look at R3:
R3#sh ip int bri
Interface IP-Address OK? Method Status Protocol
FastEthernet0/0 172.12.23.3 YES NVRAM up up
FastEthernet0/1 172.12.34.3 YES NVRAM up up
Serial0/2 172.12.123.3 YES NVRAM up up
Serial0/3 unassigned YES NVRAM administratively down down
Loopback3 3.3.3.3 YES NVRAM up up
R3#ping 4.4.4.4
Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 4.4.4.4, timeout is 2 seconds:
!!!!!
Success rate is 100 percent (5/5), round-trip min/avg/max = 1/2/4 ms
R3#sh ip ospf nei
Neighbor ID Pri State Dead Time Address Interface
44.44.44.1 0 FULL/ – – 172.12.34.4 OSPF_VL0
2.2.2.2 1 FULL/BDR 00:00:38 172.12.23.2 FastEthernet0/0
44.44.44.1 1 FULL/DR 00:00:38 172.12.34.4 FastEthernet0/1
R3#sh ip proto
Routing Protocol is “ospf 1”
Outgoing update filter list for all interfaces is not set
Incoming update filter list for all interfaces is not set
Router ID 3.3.3.3
It is an area border router
Number of areas in this router is 4. 4 normal 0 stub 0 nssa
Maximum path: 4
Routing for Networks:
3.3.3.3 0.0.0.0 area 3
172.12.23.0 0.0.0.255 area 23
172.12.34.0 0.0.0.255 area 34
172.12.123.0 0.0.0.255 area 0
Reference bandwidth unit is 100 mbps
Routing Information Sources:
Gateway Distance Last Update
44.44.44.1 110 01:58:30
Distance: (default is 110)
R3#sh ip ospf virtual-link
Virtual Link OSPF_VL0 to router 44.44.44.1 is up
Run as demand circuit
DoNotAge LSA allowed.
Transit area 34, via interface FastEthernet0/1, Cost of using 1
Transmit Delay is 1 sec, State POINT_TO_POINT,
Timer intervals configured, Hello 10, Dead 40, Wait 40, Retransmit 5
Hello due in 00:00:03
Adjacency State FULL (Hello suppressed)
Index 1/2, retransmission queue length 0, number of retransmission 0
First 0x0(0)/0x0(0) Next 0x0(0)/0x0(0)
Last retransmission scan length is 0, maximum is 0
Last retransmission scan time is 0 msec, maximum is 0 msec
R3#ping 172.12.123.1
Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 172.12.123.1, timeout is 2 seconds:
!!!!!
Success rate is 100 percent (5/5), round-trip min/avg/max = 64/66/68 ms
R3#
Uhhhh…. So R4 has some issue? It should have been ruled out by R3 pinging it’s loopback, but I’ll play ball, lets take a look:
R4#traceroute 172.12.123.1
Type escape sequence to abort.
Tracing the route to 172.12.123.1
VRF info: (vrf in name/id, vrf out name/id)
1 172.12.34.3 4 msec 0 msec 4 msec
2 * * *
3 * * *
4 * *
R4#
R4#sh ip route ospf
Gateway of last resort is not set
3.0.0.0/32 is subnetted, 1 subnets
O IA 3.3.3.3 [110/2] via 172.12.34.3, 02:04:27, FastEthernet0/1
172.12.0.0/16 is variably subnetted, 4 subnets, 2 masks
O IA 172.12.23.0/24 [110/2] via 172.12.34.3, 02:03:49, FastEthernet0/1
O 172.12.123.0/24 [110/65] via 172.12.34.3, 02:04:27, FastEthernet0/1
R4#
R4#ping 172.12.123.3
Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 172.12.123.3, timeout is 2 seconds:
!!!!!
Success rate is 100 percent (5/5), round-trip min/avg/max = 1/2/4 ms
R4#ping 172.12.123.1
Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 172.12.123.1, timeout is 2 seconds:
…..
Success rate is 0 percent (0/5)
R4#
So it is being processed by R3, as it can ping the serial interface on the NBMA network, so the issue is either somewhere on the frame switch or on R1. It’s too late and I’m too fried to really want to dig into this weird behavior (I will just wipe / reconfigure them if push comes to shove), but for the heck of it lets look at R1:
!
interface Serial0/1
ip address 172.12.13.1 255.255.255.252
clock rate 2000000
!
router ospf 1
log-adjacency-changes
area 34 virtual-link 44.44.44.1 <— What???
network 1.1.1.1 0.0.0.0 area 1
network 172.12.15.0 0.0.0.255 area 15
network 172.12.123.0 0.0.0.255 area 0
!
!
I am wondering if I was so tired configuring this, that I entered that command on R1, and that is what is jamming up the traffic, lets put a stop to this silliness:
R1#conf t
Enter configuration commands, one per line. End with CNTL/Z.
R1(config)#router ospf 1
R1(config-router)#no area 34 virtual-link 44.44.44.1
R1(config-router)#
I can confirm it is now gone, lets see about some pings:
R1(config-router)#do ping 4.4.4.4
Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 4.4.4.4, timeout is 2 seconds:
…..
Success rate is 0 percent (0/5)
R1(config-router)#do ping 172.12.34.4
Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 172.12.34.4, timeout is 2 seconds:
…..
Success rate is 0 percent (0/5)
R1(config-router)#do ping 172.12.34.3
Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 172.12.34.3, timeout is 2 seconds:
…..
Success rate is 0 percent (0/5)
R1(config-router)#do ping 172.12.123.3
Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 172.12.123.3, timeout is 2 seconds:
!!!!!
Success rate is 100 percent (5/5), round-trip min/avg/max = 64/66/68 ms
R1(config-router)#
So R3 will process packets from R4 to the IP on its NBMA serial Interface, but R1 cannot ping to the Fa0/1 interface of Area 34. Let me stare at the show run for a moment, I am losing my sense of humor now 🙂
Wow, I must have been half asleep, I found some leftover access-lists from when I was doing that section on R3 that was messing with traffic, let me remove these AND LET THE TRAFFIC FLOWETH THROUGHOUT THE NETWORK!!! :
R3#conf t
Enter configuration commands, one per line. End with CNTL/Z.
R3(config)#no access-list 15
R3(config)#no access-list 111
R3(config)#int fa0/1
R3(config-if)#no ip access-group 111 in
R3(config-if)#no ip access-group 15 out
R3(config-if)#
ASR#4
[Resuming connection 4 to r4 … ]
R4#ping 172.12.123.1
Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 172.12.123.1, timeout is 2 seconds:
…..
Success rate is 0 percent (0/5)
GAH!!!
So at this point, I’m giving R3 a write / reload, and if R4 cannot ping the server, I’ll have to wipe and confirm cabling / reconfigure the network as the previously labs are having impact somewhere in the network.
So as R3 loaded back up, and I watched the Adjacency form, I never saw the relationship back to Area 0 on the NBMA network form (I even waited the 10 years it takes over serial links):
R3>en
Password:
R3#
Mar 20 21:16:43.574: %OSPF-5-ADJCHG: Process 1, Nbr 44.44.44.1 on FastEthernet0/1 from LOADING to FULL, Loading Done
R3#
Mar 20 21:16:46.915: %OSPF-5-ADJCHG: Process 1, Nbr 2.2.2.2 on FastEthernet0/0 from LOADING to FULL, Loading Done
R3#
Mar 20 21:16:58.655: %OSPF-5-ADJCHG: Process 1, Nbr 44.44.44.1 on OSPF_VL0 from LOADING to FULL, Loading Done
R3#sh ip ospf nei
Neighbor ID Pri State Dead Time Address Interface
44.44.44.1 0 FULL/ – – 172.12.34.4 OSPF_VL0
2.2.2.2 1 FULL/DR 00:00:37 172.12.23.2 FastEthernet0/0
44.44.44.1 1 FULL/DR 00:00:37 172.12.34.4 FastEthernet0/1
R3#
So now it is an OSPF issue that R4 is not getting it’s time, which raises another good point, that time servers should not be reliant (if possible) on a dynamic protocol to reach it’s time source.
So for the hell of it since its this late and I’m already this friend, I might as well try to see this though to the gruesome end, so I took at what routes it DOES see and this is what I got:
R1#sh ip route
Gateway of last resort is not set
1.0.0.0/32 is subnetted, 1 subnets
C 1.1.1.1 is directly connected, Loopback1
5.0.0.0/32 is subnetted, 1 subnets
O 5.5.5.5 [110/2] via 172.12.15.5, 00:28:51, FastEthernet0/1
172.12.0.0/24 is subnetted, 2 subnets
C 172.12.15.0 is directly connected, FastEthernet0/1
C 172.12.123.0 is directly connected, Serial0/0
R1#
Nothing over my NBMA. OMG I forgot the neighbor statements on R1 in my tired stupor, *slams head against desk* :
R1(config)#router ospf 1
R1(config-router)#neighbor 172.12.123.2
R1(config-router)#neighbor 172.12.123.3
R1(config-router)#
.Mar 20 21:31:26.994: %OSPF-5-ADJCHG: Process 1, Nbr 2.2.2.2 on Serial0/0 from LOADING to FULL, Loading Done
.Mar 20 21:31:27.102: %OSPF-5-ADJCHG: Process 1, Nbr 3.3.3.3 on Serial0/0 from LOADING to FULL, Loading Done
R1(config-router)#
ASR#4
[Resuming connection 4 to r4 … ]
*Mar 21 01:56:05.859: %OSPF-5-ADJCHG: Process 1, Nbr 3.3.3.3 on OSPF_VL0 from FULL to DOWN, Neighbor Down: Interface down or detached
R4#
*Mar 21 01:56:12.583: %LINEPROTO-5-UPDOWN: Line protocol on Interface FastEthernet0/1, changed state to down
*Mar 21 01:56:12.583: %OSPF-5-ADJCHG: Process 1, Nbr 3.3.3.3 on FastEthernet0/1 from FULL to DOWN, Neighbor Down: Interface down or detached
R4#ping 172.12.123.1
Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 172.12.123.1, timeout is 2 seconds:
!!!!!
Success rate is 100 percent (5/5), round-trip min/avg/max = 64/65/68 ms
R4#sh ntp assoc
address ref clock st when poll reach delay offset disp
~172.12.123.1 .INIT. 16 – 1024 0 0.000 0.000 15937.
* sys.peer, # selected, + candidate, – outlyer, x falseticker, ~ configured
R4#sh ntp assoc
address ref clock st when poll reach delay offset disp
~172.12.123.1 .INIT. 16 – 1024 0 0.000 0.000 15937.
* sys.peer, # selected, + candidate, – outlyer, x falseticker, ~ configured
So it is just not synching for whatever reason I don’t care to dig in now that its 10pm, so I will continue this battle in my next post, which will include NTP Authentication!
The DEVNET GRIND! 