So to begin this, I apparently completely spaced writing R1 so nothing got saved, and we are back in time again:
R1#sh clock
*22:38:45.666 UTC Fri Mar 1 2002
R1#
So I’ve decided to go right into Authentication first, as it’s fairly straight forward with an odd behavior to note, then continue my battle with R4 over the virtual-link to see if I can maybe use the “peer” command between R3 and R4 to resolve the issue.
So I’ve removed all NTP settings from all routers involved in the last lab, as I will need to reconfigure them for authentication. Now to configure authentication, it actually only takes 3 commands on the Master / Server, but 4 commands on the clients as seen here:
R1(config)#ntp authenticate
R1(config)#
^This command sets NTP authentication to run
R1(config)#ntp authentication-key ?
<1-4294967295> Key number
R1(config)#ntp authentication-key 1 ?
md5 MD5 authentication
R1(config)#ntp authentication-key 1 md5 ?
WORD Authentication key
R1(config)#ntp authentication-key 1 md5 CCNP ?
<0-4294967295> Authentication key encryption type
<cr>
R1(config)#ntp authentication-key 1 md5 CCNP
R1(config)#
^This command is literally one way to type it, straight forward, CCNP is my keys “password” to authenticate to NTP clients. I have no idea what the last value is, so I will just leave it as configured.
R1(config)#ntp trusted-key ?
<1-4294967295> Key number
R1(config)#ntp trusted-key 1 ?
<cr>
R1(config)#ntp trusted-key 1
R1(config)#
^Again just a very straight forward command, just identifying which one of its keys is a trusted key.
And that is it for the server, it is now “offering” authentication for NTP to potential clients, which sounds odd for authentication as it definitely should.
So on R3 I repeat the same thing:
R3(config)#ntp authenticate
R3(config)#ntp authentication-key 1 md5 CCNP
R3(config)#ntp trusted-key 1
R3(config)#ntp server 172.12.123.1 ? <– The 4th command that is required for clients!
key Configure peer authentication key
prefer Prefer this peer when possible
source Interface for source address
version Configure NTP version
<cr>
R3(config)#ntp server 172.12.123.1 key ?
<0-4294967295> Peer key number
R3(config)#ntp server 172.12.123.1 key 1 ?
prefer Prefer this peer when possible
source Interface for source address
version Configure NTP version
<cr>
R3(config)#ntp server 172.12.123.1 key 1
R3(config)#
So really the NTP clients ONLY additional command to get its time from the server in any case is that mysterious 4th command while R1 has 3 (not including the clock set again).
After waiting a few minutes to see R3 populate, I realized two things: 1. I forgot that yesterday R1 had no neighbor statements for OSPF on my Hub for my spoke routers, and 2. I forgot “ntp master 1” on R1.
So really to set up its 4 commands on each if you include the ntp master on the time server, however it is 3 and 4 if you assume that is part of the normal configuration and adding “key 1” to the “ntp server x.x.x.x …” command.
Now that we’ve beat that horse to death, lets see whats happening on R3:
R3(config)#do sh ntp assoc
address ref clock st when poll reach delay offset disp
*~172.12.123.1 .LOCL. 1 1 64 17 65.4 -1.01 1877.2
* master (synced), # master (unsynced), + selected, – candidate, ~ configured
R3(config)#do sh clock
18:32:54.346 UTC Tue Mar 21 2017
R3(config)#
Alright so that is now working as expected, and I just reconfigured R4 pointed at 172.12.123.1 because it makes no sense from yesterday that it cannot sync up with R1 (which I will visit shortly), but I also configured R2 with absolutely no authentication commands and shortly after I pointed it at R1 as the Time Server we get this:
R2(config)#do sh ntp assoc
address ref clock st when poll reach delay offset disp
*~172.12.123.1 .LOCL. 1 9 64 377 53.7 -0.43 0.3
* master (synced), # master (unsynced), + selected, – candidate, ~ configured
R2(config)#do sh clock
18:39:25.526 UTC Tue Mar 21 2017
R2(config)#
So this is the weird thing with NTP Authentication, is when I say it is “offered” when set by the Master, it is usable without authenticating which sort of defeats the concept of authentication completely – REMEMBER THIS FOR EXAM DAY!
In not so red of text, a client can be configured with no authentication to the time server, and it will still get time from that server (defeating the purpose of authentication).
So I configured R4 with authentication commands and pointed it to R1, and to my surprise:
ASR#4
[Resuming connection 4 to r4 … ]
R4(config)#do sh ntp assoc
address ref clock st when poll reach delay offset disp
*~172.12.123.1 .LOCL. 1 43 64 77 64.412 -38.023 188.77
* sys.peer, # selected, + candidate, – outlyer, x falseticker, ~ configured
R4(config)#
R4(config)#do sh ntp status
Clock is synchronized, stratum 2, reference is 172.12.123.1
nominal freq is 250.0000 Hz, actual freq is 250.0000 Hz, precision is 2**24
reference time is DC7BF1C5.0072A487 (18:39:01.001 UTC Tue Mar 21 2017)
clock offset is -38.0238 msec, root delay is 64.41 msec
root dispersion is 506.73 msec, peer dispersion is 5.72 msec
loopfilter state is ‘CTRL’ (Normal Controlled Loop), drift is -0.000000202 s/s
system poll interval is 64, last update was 471 sec ago.
R4(config)#
Hooray! No nitty gritty troubleshooting, the lab must know I feel like I am getting sick too!
In the above output of the two “show” verification commands we know of thus far, you see nothing about NTP authentication, but it is all in the “detail” so to say:
R4(config)#do sh ntp assoc detail
172.12.123.1 configured, authenticated, our_master, sane, valid, stratum 1
ref ID .LOCL., time DC7BF403.9FB9E5B6 (18:48:35.623 UTC Tue Mar 21 2017)
our mode client, peer mode server, our poll intvl 64, peer poll intvl 64
root delay 0.00 msec, root disp 0.03, reach 377, sync dist 67.60
delay 64.58 msec, offset -101.6407 msec, dispersion 2.66
precision 2**18, version 4
org time DC7BF405.D9A56AD6 (18:48:37.850 UTC Tue Mar 21 2017)
rec time DC7BF405.FBEF0202 (18:48:37.984 UTC Tue Mar 21 2017)
xmt time DC7BF405.EB34C2F5 (18:48:37.918 UTC Tue Mar 21 2017)
filtdelay = 64.58 64.68 65.65 64.73 64.64 64.76 64.60 64.63
filtoffset = -101.64 -95.55 -87.40 -80.41 -72.88 -65.64 -58.45 -51.39
filterror = 0.00 0.94 1.90 2.88 3.85 4.83 5.80 6.76
minpoll = 6, maxpoll = 10
R4(config)#
I tripped over this message a couple times, because the huge output made me think “sh ntp status” however it is “sh ntp assoc detail” and NOT detail”s”!
So back to authentication not really making hosts authenticate to use the server as a time source, to limit hosts receiving time from our NTP to who we want, Access-Lists’s come to the rescue!
So the creation of the access-list:
R1(config)#access-list 10 permit 172.12.123.0 0.0.0.255
R1(config)#access-list 10 permit 172.12.34.0 0.0.0.255
Pretty simple, allows R2 / R3 / R4 to get time from R1 with of course the implicit deny at the end to deny all other networks, and now for the NTP portion of applying the ACL:
R1(config)#ntp ?
access-group Control NTP access
authenticate Authenticate time sources
authentication-key Authentication key for trusted time sources
broadcastdelay Estimated round-trip delay
clock-period Length of hardware clock tick
logging Enable NTP message logging
master Act as NTP master clock
max-associations Set maximum number of associations
peer Configure NTP peer
server Configure NTP server
source Configure interface for source address
trusted-key Key numbers for trusted time sources
R1(config)#ntp access-group ?
peer Provide full access
query-only Allow only control queries
serve Provide server and query access
serve-only Provide only server access
R1(config)#ntp access-group serve ?
<1-99> Standard IP access list
<1300-1999> Standard IP access list (expanded range)
R1(config)#ntp access-group serve 10
And that officially applies it, so I just reloaded R2 / R3 / R4 to see how they would come back up and get their time, meanwhile I went on the 172.12.23.0 Ethernet segment and set SW1 to also point to 172.12.123.1 as its NTP server and give it some time to sync until my routers reloaded.
Now all routers have reloaded, lets go around the room, and see who has the correct time:
R2#sh clock
18:34:48.760 UTC Wed Mar 22 2017
R2#
ASR#3
[Resuming connection 3 to r3 … ]
R3#sh clock
18:35:04.335 UTC Wed Mar 22 2017
R3#
ASR#4
[Resuming connection 4 to r4 … ]
R4#sh clock
18:35:14.577 UTC Wed Mar 22 2017
R4#
ASR#5
[Resuming connection 5 to sw1 … ]
SW1#sh clock
*01:30:37.361 UTC Mon Mar 1 1993
SW1#sh ntp assoc
address ref clock st when poll reach delay offset disp
~172.12.123.1 0.0.0.0 16 – 64 0 0.0 0.00 16000.
* master (synced), # master (unsynced), + selected, – candidate, ~ configured
Our switch is existing in probably my favorite era of my lifetime, the 90’s, and will remain there unless we let it back into the present (but that might be cruel) 🙂
So I wanted to post up the output of SW1’s “sh ntp status” and “sh ntp assoc det” because there is what I found in my voice days hilarious, but serious NTP status:
SW1#sh ntp status
Clock is unsynchronized, stratum 16, no reference clock
nominal freq is 119.2092 Hz, actual freq is 119.2092 Hz, precision is 2**18
reference time is 00000000.00000000 (00:00:00.000 UTC Mon Jan 1 1900)
clock offset is 0.0000 msec, root delay is 0.00 msec
root dispersion is 0.00 msec, peer dispersion is 0.00 msec
Not a whole lot there, except showing the clock isn’t synchronized, but with “sh ntp assoc det” we see a very… odd and awesome way of putting it:
SW1#sh ntp assoc det
172.12.123.1 configured, insane, invalid, unsynced, stratum 16
ref ID 0.0.0.0, time 00000000.00000000 (00:00:00.000 UTC Mon Jan 1 1900)
our mode client, peer mode unspec, our poll intvl 64, peer poll intvl 64
root delay 0.00 msec, root disp 0.00, reach 0, sync dist 0.000
delay 0.00 msec, offset 0.0000 msec, dispersion 16000.00
precision 2**5, version 3
org time 00000000.00000000 (00:00:00.000 UTC Mon Jan 1 1900)
rcv time 00000000.00000000 (00:00:00.000 UTC Mon Jan 1 1900)
xmt time AF3BE5E5.8494D4E8 (01:31:17.517 UTC Mon Mar 1 1993)
filtdelay = 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
filtoffset = 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
filterror = 16000.0 16000.0 16000.0 16000.0 16000.0 16000.0 16000.0 16000.0
SW1#
This switch is INSANE. When I first saw that, it was so awesome, the terminology just tickled me right in my Cisco soft spot. So what is “insane” you might ask (in terms of Cisco switches)? It is when the a network device is configured with an NTP server to get time from, but cannot reach that time source.
So lets see if we can get this switch SANE, and back to the present date. My brain is already fried like chicken so I accidentally exited until I found myself back at “user priv” (square one) mode, so I reconfigured SW1 to point at 172.12.123.1 for NTP and lets see:
SW1#conf t
Enter configuration commands, one per line. End with CNTL/Z.
SW1(config)#ntp server 172.12.123.1
SW1(config)#do sh clock
*01:46:27.731 UTC Mon Mar 1 1993
Hmm.. that’s been cooking for about 2-3 minutes prior to that output, so it’s time to investigate this, and as I’ve learned with R4 I need to make sure I can ping R1 first:
SW1#ping 172.12.123.1
% Unrecognized host or address, or protocol not running.
Crap. So I forgot I only gave this a name when I brought it online, so to make it able to ping over to R1 I input the following commands:
SW1#conf t
Enter configuration commands, one per line. End with CNTL/Z.
SW1(config)#ip routing
SW1(config)#int vlan1
SW1(config-if)#ip address 172.12.23.1 255.255.255.0
SW1(config-if)#no shut
SW1(config-if)#exit
01:50:31: %LINK-3-UPDOWN: Interface Vlan1, changed state to up
01:50:32: %LINEPROTO-5-UPDOWN: Line protocol on Interface Vlan1, changed state to up
SW1(config)#exit
SW1#ping 172.12.123.1
Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 172.12.123.1, timeout is 2 seconds:
…..
Success rate is 0 percent (0/5)
SW1#
So we are routing now, but still no dice, I sense an OSPF network not included issue:
R1#sh ip route ospf
2.0.0.0/32 is subnetted, 1 subnets
O IA 2.2.2.2 [110/65] via 172.12.123.2, 00:30:22, Serial0/0
3.0.0.0/32 is subnetted, 1 subnets
O IA 3.3.3.3 [110/65] via 172.12.123.3, 00:30:22, Serial0/0
4.0.0.0/32 is subnetted, 1 subnets
O IA 4.4.4.4 [110/66] via 172.12.123.3, 00:30:22, Serial0/0
5.0.0.0/32 is subnetted, 1 subnets
O 5.5.5.5 [110/2] via 172.12.15.5, 01:48:21, FastEthernet0/1
172.12.0.0/24 is subnetted, 4 subnets
O IA 172.12.34.0 [110/65] via 172.12.123.3, 00:30:22, Serial0/0
O IA 172.12.23.0 [110/65] via 172.12.123.3, 00:30:11, Serial0/0
[110/65] via 172.12.123.2, 00:30:11, Serial0/0
44.0.0.0/32 is subnetted, 1 subnets
O IA 44.44.44.1 [110/66] via 172.12.123.3, 00:30:22, Serial0/0
R1#ping 172.12.23.1
Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 172.12.23.1, timeout is 2 seconds:
…..
Success rate is 0 percent (0/5)
R1#sh access-list
Standard IP access list 10
10 permit 172.12.123.0, wildcard bits 0.0.0.255 (76 matches)
20 permit 172.12.34.0, wildcard bits 0.0.0.255 (94 matches)
30 permit 172.12.23.0, wildcard bits 0.0.0.255
R1#
So in the route table it knows of the 172.12.23.0 network via R2, so I hopped on there:
R2#ping 172.12.23.1
Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 172.12.23.1, timeout is 2 seconds:
!!!!!
Success rate is 100 percent (5/5), round-trip min/avg/max = 1/3/4 ms
R2#ping 172.12.123.1
Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 172.12.123.1, timeout is 2 seconds:
!!!!!
Success rate is 100 percent (5/5), round-trip min/avg/max = 64/66/68 ms
R2#
ASR#5
[Resuming connection 5 to sw1 … ]
SW1#ping 172.12.23.2
Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 172.12.23.2, timeout is 2 seconds:
!!!!!
Success rate is 100 percent (5/5), round-trip min/avg/max = 1/2/8 ms
SW1#
So R2 can ping R1 and SW1, and is the go between for them, and SW1 can ping R2 which is its middle man to R1. So I go back to SW1 to give a traceroute a try to see if it’s even getting a response from R2 when sending traffic:
SW1#traceroute 172.12.123.1
Type escape sequence to abort.
Tracing the route to 172.12.123.1
1 * * *
2 * * *
3 * *
SW1#sh ip route
Gateway of last resort is not set
172.12.0.0/24 is subnetted, 1 subnets
C 172.12.23.0 is directly connected, Vlan1
There’s the issue, it has no route to 172.12.123.1, so at this point of going brain dead from work / study I will just create the static route to bring sanity back to this switch:
SW1(config)#ip route 172.12.123.0 255.255.255.0 172.12.23.2
SW1(config)#exit
SW1#ping 172.12.123.1
Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 172.12.123.1, timeout is 2 seconds:
!!!!!
I love when logic works, makes my head hurt less. So I issued a write and reload, and after it booted back up, this is how the clock is now looking:
SW1#sh clock
*00:01:08.501 UTC Mon Mar 1 1993
SW1#sh ntp assoc
address ref clock st when poll reach delay offset disp
~172.12.123.1 0.0.0.0 16 – 64 0 0.0 0.00 16000.
* master (synced), # master (unsynced), + selected, – candidate, ~ configured
SW1#
So starting to lose my mind wondering why on Earth this thing will not sync, I got on R1 and ran “debug ntp packet” and I’ll spare you all the output EXCEPT WHEN SW1 FINALLY HIT THIS SUCKER AND GOT BROUGHT BACK FROM THE FUTURE… OR PAST… WHICHEVER:
.Mar 22 19:16:31.578: NTP: rcv packet from 172.12.23.1 to 172.12.123.1 on Serial0/0:
.Mar 22 19:16:31.578: leap 3, mode 3, version 3, stratum 0, ppoll 64
.Mar 22 19:16:31.578: rtdel 0000 (0.000), rtdsp 10001 (1000.015), refid 00000000 (0.0.0.0)
.Mar 22 19:16:31.578: ref 00000000.00000000 (00:00:00.000 UTC Mon Jan 1 1900)
.Mar 22 19:16:31.578: org 00000000.00000000 (00:00:00.000 UTC Mon Jan 1 1900)
.Mar 22 19:16:31.582: rec 000
R1#00000.00000000 (00:00:00.000 UTC Mon Jan 1 1900)
.Mar 22 19:16:31.582: xmt AF3BD148.1181126D (00:03:20.068 UTC Mon Mar 1 1993)
.Mar 22 19:16:31.582: inp DC7D4C0F.945A1DF3 (19:16:31.579 UTC Wed Mar 22 2017)
.Mar 22 19:16:31.582: NTP: stateless xmit packet to 172.12.23.1:
.Mar 22 19:16:31.582: leap 3, mode 4, version 3, stratum 0, ppoll 64
.Mar 22 19:16:31.582: rtdel 0000 (0.000), rtdsp 6002 (375.031), refid 4C4F434C (76.79.67.76)
.Mar 22 19:16:31.586: ref DC7D400B.EEB63084 (18:25:15.932 UTC Wed Mar 22 2017)
.Mar 22 19:16:31.586: org AF3BD148.1181126D (00:03:20.068 UTC Mon Mar 1 1993)
.Mar 22 19:16:31.586: rec DC7D4C0F.945A1DF3 (19:16:31.579 UTC Wed Mar 22 2017)
.Mar 22 19:16:31.586: xmt DC7D4C0F.95A82567 (19:16:31.584 UTC Wed Mar 22 2017)
R1#u all
R1#
I left the big crap ton of output to see the exchange and references, I am not sure what most of it means, but it does show the switches old time and the current time being exchanged (as well as referring to 1900 for some odd reason).
So when we go back to SW1, we should finally have some sanity, before I lose mine:
SW1#sh clock
*00:10:24.859 UTC Mon Mar 1 1993
SW1#sh ntp assoc det
172.12.123.1 configured, insane, invalid, unsynced, stratum 16
ref ID 0.0.0.0, time 00000000.00000000 (00:00:00.000 UTC Mon Jan 1 1900)
our mode client, peer mode unspec, our poll intvl 64, peer poll intvl 64
root delay 0.00 msec, root disp 0.00, reach 0, sync dist 0.000
delay 0.00 msec, offset 0.0000 msec, dispersion 16000.00
precision 2**5, version 3
org time DC7D4D8F.933D330C (19:22:55.575 UTC Wed Mar 22 2017)
rcv time AF3BD2C8.1E2EB9CE (00:09:44.117 UTC Mon Mar 1 1993)
xmt time AF3BD2C8.10B78FA5 (00:09:44.065 UTC Mon Mar 1 1993)
filtdelay = 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
filtoffset = 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
filterror = 16000.0 16000.0 16000.0 16000.0 16000.0 16000.0 16000.0 16000.0
SW1#
What??? If you examine the details, other than insanity still being awesome terminology, you can see what I’ve highlighted in red that it does have the correct network time in the output which its getting from R1, but show clock does not show the correct time.
So at this point, I did try to put “router ospf 1” on the switch which it does show as a valid command, but it does not drop me into ospf configuration mode to see if that gives it the kick it needs. You can even see it getting hits on R1’s ACL for NTP requests:
R1#sh access-list
Standard IP access list 10
10 permit 172.12.123.0, wildcard bits 0.0.0.255 (130 matches)
20 permit 172.12.34.0, wildcard bits 0.0.0.255 (105 matches)
30 permit 172.12.23.0, wildcard bits 0.0.0.255 (15 matches)
R1#
So at this point, the next segment of my course is running NTP in broadcast mode, and I want to see what that has to say there to see if we can maybe salvage this SW1 situation.
The DEVNET GRIND! 