If a trouble ticket pops up for these an issue with these protocols, to know if a protocols port # is not being forwarded correctly, or being blocked by an ACL somewhere, in addition to all the other misconfigurations that might go wrong.
For IP SLA, perhaps someone can correct me here in the comments, but I’ve seen it documented that the control port the devices “talk” on for mgmt traffic can be either UDP 1967 or UDP 5000 so I’d just associate either of those with IP SLA!
A quick review of “ip helper-address” and “ip forward-protocol”
“ip helper address” is mainly for DHCP in a network, as it will forward DHCP, DNS, and TFTP traffic as the main protocols we are interested in as Cisco Engineers, with the exception of TACACS also being allowed.
For everything else (basically all mgmt protocol traffic), the “ip forward-protocol …” command will be needed to forward the UDP Port #’s in the top graphic.
By default, the “ip helper-address x.x.x.x” command only forward 8 UDP port #’s:
- TIME (TimeServer) = 37 (Nothing to do with NTP Protocol)
- TACACS = 49
- DNS = 53
- BOOTP (DHCP SERVER) = 67
- BOOTP (DHCP CLIENT) = 68
- TFTP (Trivial File Transfer Protocol) = 69
- NETBIOS Name Service = 137
- NETBIOS Datagram Service = 138
So this is good information if the Client PC on TSHOOT is having an issue getting a DHCP Address, as the “ip helper-address x.x.x.x” command might be missing from an Ingress interface, or might be pointed at an IP other than that of the DHCP Server.
When pointing a protocol to a destination, that destination should always be the final destination IP, not the next hop IP!
Troubleshooting NTP Protocol
Some issues to look for right out of the gate:
- NTP Server is not reachable – This should be the first test, using “Divide and Conquer” troubleshooting by pinging the NTP Server from the device having issues, if it pings we know we are going upwards from Layer 3!
- ACL Blocking NTP Packets – This can be verified relatively quick with a “show access-list” issued on routers using “Follow the Path” troubleshooting, NTP uses UDP Port 23
- NTP Authentication – This is going to be mostly “Stare and Compare” troubleshooting between routers, NTP Authentication is a bit odd in how it works, for a quick look at configuration and behaviors I have a post on NTP Auth here.
- Wrong NTP Server being used – Many NTP Servers can be configured on a router, the router will choose the best NTP Server to use for time based on lowest Stratum #, the server being used can be verified with “sh ntp status”
- High CPU Utilization – This one is a stretch in my mind to see on exam day, however NTP Packets are processed by the CPU (as are all mgmt protocols), so it is possible if the CPU is pegged that NTP will go crazy, “sh proc cpu history” to see the CLI graph of CPU utilization (I would be amazed to see this in the TSHOOT exam)
- Time offset too high – This is the difference between the NTP “sh clock” time and the problem routers “sh clock” time if there is also a manual “clock set …” configuration on the router
- Stratum level is too high – Stratum 1 is considered Atomic Clock status, and the max level 15 is the highest acceptable NTP Stratum value, anything above that is considered “insane” in the “sh ntp status” output at the top (I love that term!)
- Server has ACL limiting who can use it – This is a security best practice so I’d be prepared to run into this issue on a Cisco exam, again a quick “sh access-list” and review of “sh run” configuration should lead a candidate down the right path to verify if this is the issue they are facing
Some helpful commands for NTP Troubleshooting:
- “sh ntp status” – All info regarding NTP Status on router
- “sh ntp assoc (detail)” – Info on devices associated to NTP
- “sh access-list” – Watch for UDP port 123 (or subnet of problem device)
- “sh run” – To view NTP Configuration
- “debug ntp all” – Will pretty clearly show all NTP issues if available
Again, I do have an overview of NTP to brush up a bit more in depth here.
Syslog Troubleshooting
I’ve covered configuration pretty in depth here for TSHOOT, but you’ll want to make sure no ACL’s are blocking UDP Port 514, and that the destination IP for a Syslog server is both reachable and correctly configured.
If remotely connected and not seeing console output messages, the issue is not having “term mon” or “terminal monitor” enabled, which allows you to see those messages when using Telnet or SSH to a Cisco device.
“sh logging” will give most info needed to troubleshoot, however “sh run” if available is a an excellent command to verify manually entered configs to spot an issue!
SNMP Troubleshooting
There are several pretty straight forward things that can go wrong with SNMP in terms of what you can identify on TSHOOT, as I doubt there will be access to the NMS (Network Management Server) that collects the SNMP information.
Here is a list of things to check for SNMP if a trouble ticket pops up for it:
- ACL’s – Check access-lists for blocking not only subnets in both directions, but also UDP Ports 161 and 162 to verify traffic is not being filtered!
- “enable snmp traps” not configured – You can have everything else configured perfectly but if SNMP Traps are not enabled on the client (router / switch) then SNMP will not be running despite any other configuration
- Make sure “community string” is present – If traps are enabled but there is no community string at all for an NMS to match on, SNMP will not work
- Incorrect NMS IP Address – Verify IP Address in running configuration
- Verify correct string matches – If provided the configuration from the NMS, use the “Stare and Compare” method to verify it matches the “community string” configured on the Cisco router or switch
- Index Shuffling – On device reboots or upgrades a thing called “index shuffling” may occur that makes your NMS think all monitored interfaces / services are down because their value changed, to fix this issue command “snmp-server ifindex persist” to prevent the NMS from going bonkers upon a device reboot
Some commands to utilize on exam day include:
- “sh run (| sec snmp)” – Manual SNMP configuration input
- “sh snmp group” – Shows Group Name / ACL applied / SNMP Ver and Security Level (Auth/Priv/AuthNoPriv/AuthPriv/NoAuthNoPrive)
- “sh access-list” – Review any ACL’s for subnets / ports being blocked
- “sh snmp user” – SNMP User config parameters
- “sh snmp host” – Shows NMS / Host IP, UDP Port used to communicate, Trap or Inform configured, and SNMP Version in use by NMS
- “sh snmp view” – Shows OID’s included in the MIB that is sent to the NMS
IP SLA Tracking troubleshooting
IP SLA and Tracking Objects are really two different things, as Tracking Objects are also used for things like FHRP’s, however it works with SLA as well to provide the same type of mechanism if IP SLA in terms of router WAN link redundancy.
It is made up of an IP SLA Prove and IP SLA Responder, the Responder being a device that is pinged (or probed with different kind of traffic) that reports back the measurement it received minus the latency experiencing while processing the packet.
You can use IP SLA to just ping a google server as well, and tie a Tracking Object to it, and finally tie that Tracking Object to say a default route out to the WAN that the IP stops pinging the link is configured down and the route is removed from the IP route table.
This is one good reason for floating static routes, as in this type of IP SLA Tracking configuration on a primary default route, the backup route / link can be used until the IP SLA gets a response again from whatever it is tracking to.
Some verification commands for exam day:
- “sh run (| sec sla)” – Manual IP SLA configuration
- “sh ip sla (monitor) application” – Shows application SLA parameters
- “sh ip sla (monitor) configuration” – Shows SLA Configuration parameters
- “sh ip sla (monitor) stat” – IP SLA Statistics
- “sh ip sla (monitor) responder” – Responder information
- “sh track” – Shows tracking objects configured
- “sh run | i ip route” – Track will only show in the running config of a manually configured IP route on a Cisco Router or Switch
There is a lot to IP SLA covered in other articles I have written for ROUTE studies, however it will be about the same drill as most management protocol troubleshooting, make sure correct parameters are configured and IP Addresses / Protocols are not being filtered anywhere on the network.
That will wrap up this article on Management Protocols!
I know there are a few I’ve missed in this article such as SPAN / RSPAN / Telnet / SSH, however those should be covered in SWITCH Security or related articles to the technology, so they are around this site waiting to be searched out 🙂
The DEVNET GRIND! 