I was going to demonstrate some concepts in packet tracer to make writing out explanations a bit quicker, but I need to speed up my grind a bit here, so I’m just going to foot to the floor these next few explanation before I take a break from blogging to really put my nose to the grind stone on studying to pass the TSHOOT in the next month or so!
With that I will get right to it!
Ping / Extended Ping command from PC and CLI
With any troubleshooting ticket this will immediately determine via “Divide and Conquer” troubleshooting method if Layers 3-1 are working, by sending a ping from the PC and getting a response from the destination IP, and if no response is gotten you may want to switch up to “Follow the Data Path” and ping one hop at a time (or Traceroute!).
Though because this is a Cisco test I will not review the possible PC ping responses and what they mean, however the following is a list of the CLI ping responses and what the response means to the network admin:
- !!!!! = Layer 3-1 Connectivity
- ….. = No route to destination IP Address
- !.!.! = Possible load balancing to destination with one path failing
- U.U.U = Destination Host IP Unreachable
- M.M.M = Packet MTU exceeds the MTU of the local egress interface
By the time you are at TSHOOT most of these should be familiar, however the last three are of particular interest in terms of the TSHOOT exam, as the indicative of a specific issue on the network.
!.!.!.!.!.!
This type of response I would look for Equal-Cost EIGRP Success Routes to the destination IP Address in the IP Route Table or “sh ip eigrp topology” to view all Successor and Feasible Successor routes, and confirming correct information on upstream devices in the data path for Security settings (key-chains) and correct IP information on interfaces.
U.U.U.U.U.U
This indicates that a device in the data path is trying to pass the ping along upstream, and a device in that path is reporting back that it has no route to the destination, in this instance I would use a ‘traceroute’ to determine which last hop is reporting back, and begin looking at connectivity from there to the next hop probably using the “Stare and Compare” method to find any configuration mismatches (and of course verify correct routing is in place).
M.M.M.M.M.M
Based off the Topology Cisco provides on its webpage for the TSHOOT exam I would expect to see this on the GRE Tunnel Topology, as issues will occur if the MTU’s are not properly configured for the GRE Tunnel / Egress interface on either side of the tunnel causing packet “Fragmentation” during transmission, so a network admin will want to confirm that the GRE Tunnel interface is lower than the Physical interface the tunnel connects on by 24 Bytes for GRE Tunnel Overhead / Encapsulation.
The Default MTU upon creating a Tunnel Interface will be 1476, as the default Physical Interface MTU is 1500, and if the Tunnel interface is manually set to be a higher value (meaning it is less than 24 bytes more than the physical interface), it will cause fragmentation of the data packets that will need to be reassembled by the receiving host.
So you will want to “sh int tu#” and “sh int gi1/0” to verify the physical interface is 24 bytes higher value than the tunnel mtu #, otherwise you want to reconfigure it for the interface using the “ip mtu #” command on the physical interface to increase the MTU size (or possibly decrease the Tunnel MTU Size depending on what the remote side shows).
Both sides of the tunnel should have matching MTU’s on both the physical and logical (Tunnel) interfaces!
To test this Extended Ping can be used with its “Sweep Range” option, if for some reason you are unable to verify the MTU with show commands, you can set the DF-Bit (Do not Fragment) to ‘yes’ and make a sweep range of 1400-1500 over the GRE Tunnel.
This will send 100 pings over the Tunnel interface while incrementing the MTU by 1 each ping, so it should come back as !!!!!!!! until it hits the configured MTU, at which point the ping response will turn into M.M.M.M.M.
In this way you can see how many successful ping responses were received, so if the range swept was 1400-1500 with 50 responses received that indicates the packets are beginning to be fragmented at 1450 MTU over the GRE tunnel.
Another good verification command for this is “sh ip traffic int gi1/0” to verify the “Fragmented Packets” of the output to compare the Received/Reassembled values to see if they match, and if they do not match there is an MTU configuration issue that needs to be corrected.
Traceroute
This command is “tracert x.x.x.x” on a PC, and “traceroute x.x.x.x” on an IOS device, and can tell you the last reported device that sent an ICMP echo response back in the data path to determine where to start troubleshooting.
Telnet
Telnet is used to confirmed Layer 4 protocol connectivity after Layer 3 tested successfully via ping, or really just in general as well, as ICMP traffic might be blocked by the ACL.
Also can be done from the PC using “telnet x.x.x.x:25” for example to test SMTP on a remote device is open / port forwarded / allowed through ACL’s and NAT’s, this command can also be issued on the CLI by using “telnet x.x.x.x 25” with a space instead of a : (socket) which should give the result as “Open…” if the protocol.
So again this is a technique to test for Layer 4 connectivity.
SNMP
This can be searched under my blog for better details in the ROUTE blogs, however the basics are as follows, bullet point style:
- Uses a “Pull” model, where device statistics (NOT network / traffic statistics) are sniffed by a Network Management Station or Server (NMS)
- This Pulls device statistics like interface behaviors, CPU utilization, and can only be used by setting up the NMS that sniffs the data and makes it available in a graphical format somewhere else on the network
Netflow
On the contrary of SNMP:
- This uses a “Push” model, where you configure Netflow on a Cisco device, and configure an IP Address and Port # (no standard port for netflow, refer to your collector documentation) to send it to an NMS
- This collects data on the Network Utilization ONLY, and nothing in terms of actual device statistics in terms of CPU / Hardware utilization
- Netflow can be configured and output reviewed right on the router, these commands are:
- “int fa#/#” to configure at interface level
- “ip flow ingress” on interface level
- “show ip cache flow” to see a huge graph of all ‘top talkers’ on the network off a certain interface, this is an excellent command for on the job!!!
EEM = Embeded Event Manager
I won’t go into the configuration of this as it can be quite cumbersome, I highly advise checking out the actual Cisco documentation once over to get a feel for it, however Embedded Event Manager is basically programmable automation on a Cisco device.
It is basically an “If” / “Then” sort of operation that can be configured, whether it is when someone displays the command “sh ip route” then display banner “Back up the running config if you alter my route table!” or something to that effect.
Being that its Router Progammability it is just way far out from what I’m going to dive into here, however it can be viewed at this link on Cisco’s website.
At its core it is an “Event Detection” system that configured to write syslog messages for commands issued / pop-up messages as posted above, but is not really aimed at full task automation that I am currently aware of.
Though the Automation Specialist Cert from Cisco will be one of my next pursuits after my CCNP, so I will be going very deeply into that topic sometime in 2020 if training is available at that time (fingers crossed!).
FTP to backup your devices / TSHOOT based on old configs
This is technically an IOS feature for troubleshooting as well as maintenance, and its configuration / verification is done from the CLI, so wanted to touch on the key points of FTP and how to restore configurations in a disaster recovery scenario:
- There are GUI Tools such as CCP (Cisco Config Pro) and ASDM for ASA’s to perform and automate FTP backups of a configs device to an FTP server
- “startup-config ftp://USER:PASS@192.168.1.100” – This is done from the Global Exec level (not global config!), requires the username / password of the FTP server the file is being sent to for authentication, followed by server IP addy!
- “ip ftp username (username)” / “ip ftp password (password)” – This sets credentials for the FTP server, CONFIGURED IN GLOBAL CONFIG(!), so that you can then just type “startup-config ftp://192.168.1.100” without the credentials (again this command is issued in Global Exec mode!)
- “ip http username (username)” / “ip http password (password)” – This sets the router password to allow the GUI apps like CCP and ASDM to connect via http, may also need to add “ip https …” to use HTTPS for a secured connection
Details on restoring a devices running config with “Archive” copies via FTP
One thing to be aware of is that copying a remote image / copy to a devices current running configuration will treat it as a “merge” rather than an erase and replace, and it will replace the “Running” config, so this is something to do in a maintenance window!
“config replace ftp://192.168.1.100/RouterA_backup8” – This command issued at Global Exec will call the FTP server to pull down the backup file “RouterA_backup8” to replace its running configuration.
FTP backups will be shown in archive history whether they are manually done or automatically set to execture at specific time intervals, if configured to automatically backup it will show in running config as:
!
archive
path ftp://192.168.1.100/RouterA
time-period 1440
write memory
!
This config sets automatic created of numbered backup files beginning with file name “RouterA” every 1440 minutes (24 hours) and issue command “write memory” after it finishes the operation.
“sh archive” – This will display all archived copies of the backup, it will only show up to 10 file names though there are likely many more than 10 backed up to the server, however the most recent file name (usually the last on the list / #10) will be the most recent that you will want to do a restore from.
This is something good to know for your work place to perform backups, but also on exam day if you are going down a path of believing a config replacement may be needed on a question, checking to confirm backups are configured / there are backups to use in the archive will be critical to verify to confirm or rule out the problem report solution!
That is it for this first part of TSHOOT CLI commands
From what I have read a lot of the next ones are your well known “ping” / “traceroute” type commands, however there are some good to know methods of attack when troubleshooting using these tools, and some responses that are not well known that may prove very good to know for one of the TSHOOT exam Topologies!
The DEVNET GRIND! 