I’ll be working between SW1 and SW3 for this lab, as it can work to cover both the lab topics of “EtherChannel Misconfig Guard” as well as Layer 3 EtherChannels, and reviewing some EtherChannel Load-Sharing and testing load-sharing to finish!
EtherChannel Misconfig Guard – Hiding in plain sight throughout all these labs!
Every time the Err-Disable would happen on one side of the EtherChannel while using the “channel-group # mode on” configuration (AKA manual EtherChannel), what was happening was Misconfig Guard placing ports into Err-Disable state, because the EtherChannel detected a possible loop.
Misconfig Guard is on by default, and is seen here in “show span summ” output, as it is an STP specific command for loop prevention:
SW1#sh span summ
Switch is in pvst mode
Root bridge for: VLAN0001
Extended system ID is enabled
Portfast Default is disabled
PortFast BPDU Guard Default is disabled
Portfast BPDU Filter Default is disabled
Loopguard Default is disabled
EtherChannel misconfig guard is enabled
UplinkFast is disabled
BackboneFast is disabled
EtherChannel Misconfig Guard was seen in this format several times while configuring static EtherChannel on the lab:
SW1(config-if-range)#channel-group 13 mode on
Creating a port-channel interface Port-channel 13
SW1(config-if-range)#
*Mar 1 00:07:42.027: %LINK-3-UPDOWN: Interface Port-channel13, changed state to up
*Mar 1 00:07:43.034: %LINEPROTO-5-UPDOWN: Line protocol on Interface Port-channel13, changed state to up
SW1(config-if-range)#
SW1(config-if-range)#
*Mar 1 00:08:23.710: %PM-4-ERR_DISABLE: channel-misconfig (STP) error detected on Fa1/0/1, putting Fa1/0/1 in err-disable state
*Mar 1 00:08:23.735: %PM-4-ERR_DISABLE: channel-misconfig (STP) error detected on Fa1/0/2, putting Fa1/0/2 in err-disable state
*Mar 1 00:08:23.752: %PM-4-ERR_DISABLE: channel-misconfig (STP) error detected on Fa1/0/3, putting Fa1/0/3 in err-disable state
*Mar 1 00:08:23.777: %PM-4-ERR_DISABLE: channel-misconfig (STP) error detected on Fa1/0/4, putting Fa1/0/4 in err-disable state
SW1(config-if-range)#
The closest I’ve seen to a semi-official time is 40 seconds while labbing, there is no official timer that I have found for this, and it works by detecting “Port IDs” from the remote interfaces on an EtherChannel.
How this works is Spanning-Tree sees the Port-Channel interface as one logical interface, and expects to receive a single Port ID / MAC Address / etc from the links bundled into that logical interface, so if it starts to see multiple Port IDs coming back on its bundled links it detects a loop is present and shuts everything down.
The reason a loop is detected is because of the switches operation in the flooding of frames, the Port-Channel is only going to flood a frame over one link in the bundle, and when the remote switch receives this frame it is going to flood it out all interfaces except the interface it rode in on!
To illustrate this, say SW3 receives a frame and floods it over to SW1:
When SW1 receives the frame on Fa1/0/1, it will flood it out Fa1/0/2 – 4 connected back to SW3, as per the normal frame forwarding nature of a layer 2 switch:
EtherChannel Misconfig Guard detects this issue, somewhere right around 40 seconds after the EtherChannel goes ‘Up’, and put all bundled links into an Err-Disable state:
Here is how the EtherChannel looks on the lab while in this Err-Disable state:
From this perspective, “channel-group 13 mode on” was configured on SW3, but not on SW1, so it shows the Amber ports indicating the Err-Disable state on SW3 – However SW1 ports are only logically down (No lights at all).
So only the side with the EtherChannel configured / having a problem will put its bundled links into Err-Disable, and this means the remote side of the links will go into a non-admin “Down” state, and do not need to get bounced to come back to an ‘Up’ state.
To remedy the situation on SW3, a quick shut / no shut can be done on the interfaces to bounce them out of Err-Disable, once the configuration is corrected on the remote side.
To enable or disable EtherChannel Misconfig Guard:
SW1(config)#span ?
backbonefast Enable BackboneFast Feature
etherchannel Spanning tree etherchannel specific configuration
extend Spanning Tree 802.1t extensions
logging Enable Spanning tree logging
loopguard Spanning tree loopguard options
mode Spanning tree operating mode
mst Multiple spanning tree configuration
pathcost Spanning tree pathcost options
portfast Spanning tree portfast options
transmit STP transmit parameters
uplinkfast Enable UplinkFast Feature
vlan VLAN Switch Spanning Tree
SW1(config)#span etherchannel ?
guard Configure guard features for etherchannel
SW1(config)#span etherchannel guard ?
misconfig Enable guard to protect against etherchannel misconfiguration
SW1(config)#span etherchannel guard misconfig ?
<cr>
Note that this is done from “Global Config” mode and not on an interface range or Port-Channel interface, and watch the configuration syntax because it is entered with those last two words backwards from its official Cisco name of Misconfig Guard.
This can be DISABLED in the same place:
SW1(config)#no span etherchannel guard misconfig
SW1(config)#do sh span summ
Switch is in pvst mode
Root bridge for: VLAN0001
Extended system ID is enabled
Portfast Default is disabled
PortFast BPDU Guard Default is disabled
Portfast BPDU Filter Default is disabled
Loopguard Default is disabled
EtherChannel misconfig guard is disabled
UplinkFast is disabled
BackboneFast is disabled
So to quickly review Misconfig Guard bullet point style:
- Misconfig Guard is Enabled by default on Cisco switches
- Misconfig Guard detects loops based on Port IDs received
- Puts links in bundle into Err-Disable, remote side goes logical down
- “span ether guard misconfig” at global level, “no” in front to disable
- “sh span summ” to verify if it is enabled or disabled
All there is to EtherChannel Misconfig Guard – Easy points on exam day!
Layer 3 EtherChannel fundamentals and configuration
Layer 3 EtherChannel is used as an alternative to a Layer 2 EtherChannel to limit the scope of of Broadcast traffic on the network, allowing for hosts a network to traverse the EtherChannel only when routed over it, while still providing link redundancy to the other side of the EtherChannel.
Separating Broadcast traffic can also be achieved by putting the EtherChannel ports in its own VLAN, however there are still Spanning-Tree impact on the logical link, such as Forward Delay timers and Path Cost that may cause the logical link to go into a non-FWD state.
Layer 3 EtherChannels do require a Layer 3 switch as will be seen below in the configuration, but it should be noted that only speed / duplex settings need to match across the interfaces, because the other Layer 2 information will not matter!
I labbed up a quick example of a layer 3 EtherChannel between SW1 and SW3:
SW1 Port-Channel Configuration
SW1(config)#int po13
*Mar 1 00:25:41.171: %LINK-3-UPDOWN: Interface Port-channel13, changed state to down
SW1(config-if)#no switchport
SW1(config-if)#ip add 10.0.0.1 255.255.255.252
The Port-Channel interface is created and configured before anything, and this logical interface must be configured with “no switchport” just like a physical interface, or there will be no option to apply “ip add x.x.x.x” to the logical interface.
SW1 Interface Range Configuration
SW1(config-if)#int ra fa1/0/1 – 4
SW1(config-if-range)#no switchport
SW1(config-if-range)#
(Fa1/0/1 – 4 all bounce Down and back Up)
SW1(config-if-range)#channel-group 13 mode on
SW1(config-if-range)#
*Mar 1 00:26:52.382: %LINK-3-UPDOWN: Interface Port-channel13, changed state to up
*Mar 1 00:26:53.389: %LINEPROTO-5-UPDOWN: Line protocol on Interface Port-channel13, changed state to up
SW1(config-if-range)#
I removed the flood of 4 physical interfaces bouncing from the “No switchport” command that turns them from a Layer 2 mode of operation to Layer 3, and only after the “channel-group” including the existing Port-Channel is applied does the Port-Channel interface come up.
A couple of important notes for configuration is that it goes backward from a Layer 2 EtherChannel, being you need to create the Port-Channel interface first, then apply it to the physical interfaces.
A very important note for exam day is that the channel-group # applied to the physical interfaces must match the Port-Channel #, and if the command “ip add x.x.x.x” is not being recognized on an interface, it needs “no switchport” command applied to it:
SW1(config-if)#ip add ?
% Unrecognized command
SW1(config-if)#no switchport
SW1(config-if)#
*Mar 1 00:25:41.171: %LINK-3-UPDOWN: Interface Port-channel13, changed state to down
SW1(config-if)#ip add 10.0.0.1 255.255.255.252
SW1(config-if)#
Be sure to watch for this on exam day!
SW3 Configuration
SW3(config)#int po31
SW3(config-if)#no switchport
SW3(config-if)#ip add 10.0.0.2 255.255.255.0
SW3(config-if)#int ra fa1/0/1 – 4
SW3(config-if-range)#no switchport
SW3(config-if-range)#
SW3(config-if-range)#channel-group 31 mode on
SW3(config-if-range)#
Verification from SW3 that we have a Layer 3 EtherChannel
SW3(config-if-range)#do ping 10.0.0.1
Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 10.0.0.1, timeout is 2 seconds:
!!!!!
Success rate is 100 percent (5/5), round-trip min/avg/max = 16/218/1024 ms
SW3(config-if-range)#do sh span
No spanning tree instance exists.
SW3(config-if-range)#
SW3(config-if-range)#do sh ether summ
Flags: D – down P – bundled in port-channel
I – stand-alone s – suspended
H – Hot-standby (LACP only)
R – Layer3 S – Layer2
U – in use f – failed to allocate aggregator
M – not in use, minimum links not met
u – unsuitable for bundling
w – waiting to be aggregated
d – default port
Number of channel-groups in use: 1
Number of aggregators: 1
Group Port-channel Protocol Ports
——+————-+———–+———————————————–
31 Po31(RU) – Fa1/0/1(P) Fa1/0/2(P) Fa1/0/3(P)
Fa1/0/4(P)
SW3(config-if-range)#
Pings are returned between the two sides, no spanning-tree instance is shown for the EtherChannel, and the “sh ether summ” output shows it is an Active Layer 3 EC.
So easy a caveman can do it. On to the final topic of this post, load-sharing!
EtherChannel Load-Balancing / Load-Sharing fundamentals and configuration
EtherChannel does “Load sharing” rather than true “Load balancing” with traffic, as each flow of traffic is divided up over the available links and not individual packets or frames for a flow, so it is referred to as load-sharing because it distributes the load as it can.
The traffic flow of particular Hosts will always take the same port / link in the EtherChannel, as a Cisco hash algorithm will run on the criteria defined in the load-balancing configuration, which I will work with from the following output:
SW1(config)#
SW1(config)#port-channel ?
load-balance Load Balancing method
SW1(config)#port-channel load-balance ?
dst-ip Dst IP Addr
dst-mac Dst Mac Addr
src-dst-ip Src XOR Dst IP Addr
src-dst-mac Src XOR Dst Mac Addr
src-ip Src IP Addr
src-mac Src Mac Addr
SW1(config)#
Highlighted in blue shows load balancing by IP Address, highlighted in green shows load balancing by MAC Address, and both must define either the source / destination / both source and destination parameter be used in calculating the link to be used.
My switches are not running CatOS, so I don’t have a Layer 4 option to load balance by Port #, however it does exist on Cisco 6k / 6500 series Catalyst switches.
Also to note for exam day, this is done at global configuration level, so this will effect the entire switch and how it distributes traffic across links!
There is a Cisco Proprietary hashing algorithm is really a topic of its own that I won’t delve into for the sake of getting this topic finish, however to learn how to calculate the nuts and bolts of how links are chosen for traffic flows, there is an excellent Cisco documented regarding the selection process located here.
One important note for exam day is that XOR refers to the computing of both source and destination values by the hashing algorithm, and there is no mix-and-match with XOR, both source and destination types must be the same (MAC, IP, Etc).
Essentially it depends on the # of links present, how much load they are able to handle, and this is run against the last 3 bits of the MAC / IP Address to create a value between 0-7 which indicates the Port in the EtherChannel that traffic flow will take.
From that Cisco article, a general idea of load balancing can be seen here, as to how much each link will carry at different amounts of links in the EtherChannel:
8 – 1:1:1:1:1:1:1:1
7 – 2:1:1:1:1:1:1
6 – 2:2:1:1:1:1
5 – 2:2:2:1:1
4 – 2:2:2:2
3 – 3:3:2
2 – 4:4
This is not a percentage of traffic going over the link, but the ratio of how much traffic will go over the links, highlighted in red shows the only true load balancing ratios are at 8 / 4 / 2 links bundled in the Port-Channel.
The hashing algorithm cannot be changed or reconfigured, only how load-balancing is performed on the switch per the global config can it the distribution be manipulated!
To view the current load-balancing configured, “show ether load-balance” is issued:
SW1#sh ether ?
<1-48> Channel group number
detail Detail information
load-balance Load-balance/frame-distribution scheme among ports in
port-channel
port Port information
port-channel Port-channel information
protocol protocol enabled
summary One-line summary per channel-group
| Output modifiers
<cr>
SW1#sh ether load ?
| Output modifiers
<cr>
A lot of output, but the command is “show ether-channel load-balance” in full.
SW1#sh ether load
EtherChannel Load-Balancing Configuration:
src-mac
EtherChannel Load-Balancing Addresses Used Per-Protocol:
Non-IP: Source MAC address
IPv4: Source MAC address
IPv6: Source MAC address
SW1#
By default, the load balancing on a Cisco switch is by MAC Address, which means the same hosts will ALWAYS use the same Port in the EtherChannel, as the MAC Address will normally never change on a Host.
This means that any traffic flow from Host A to Host B across an EtherChannel will always use the same link, and the only chance of variation would be to use the XOR or “both” command to run against the hash algorithm to possibly produce a different value.
So to achieve as close to the closest thing to perfect load distribution (equal load distribution of traffic flows), either 8 / 4 / 2 links is required, however for scenarios it will take some planning demonstrated at the bottom of the post!
The EtherChannel can be tested with the command “test ether …” from user exec mode, either by IP or MAC Address, which will be demonstrated using the following Topology:
In this example I plugged R1 and R2 into Fa1/0/12 on each switch to represent IP Hosts, and sent a ping across the wire to confirm connectivity:
R1#ping 10.0.0.2
Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 10.0.0.2, timeout is 2 seconds:
!!!!!
Success rate is 100 percent (5/5), round-trip min/avg/max = 1/2/4 ms
R1#
Connectivity!
To test the connectivity on a MAC or Layer 2 level, I first ran “sh mac add dyn” to get my dynamically learned MAC Address:
SW1#sh mac add dyn
Mac Address Table
——————————————-
Vlan Mac Address Type Ports
—- ———– ——– —–
1 001b.5336.f2cd DYNAMIC Po13
1 001e.f797.f14b DYNAMIC Fa1/0/12
1 5897.1eab.ce03 DYNAMIC Po13
1 5897.1eab.ce04 DYNAMIC Po13
1 5897.1eab.ce05 DYNAMIC Po13
1 5897.1eab.ce06 DYNAMIC Po13
Total Mac Addresses for this criterion: 6
SW1#
Pretty obvious from looking at this which MAC Addresses are part of the Port-Channel and which are unique, however using this information I ran through the following:
SW1#test ether ?
load-balance Load balancing information
SW1#test ether load ?
interface Port-channel interface
SW1#test ether load int po13 ?
ip IP address
ipv6 IPv6 address
mac Mac address
SW1#test ether load int po13 mac ?
H.H.H Source MAC address
SW1#test ether load int po13 mac 001e.f797.f14b ?
H.H.H Destination MAC address (egress value for routed unicast)
SW1#test ether load int po13 mac 001e.f797.f14b 001b.5336.f2cd ?
<cr>
The moment of truth:
SW1#test ether load int po13 mac 001e.f797.f14b 001b.5336.f2cd
Would select Fa1/0/2 of Po13
One good to know behavior for exam day, is this test can only be run on the criteria configured in terms of MAC / IP, when I try to test IP Load Balancing:
SW1#test ether load int po13 ip 10.0.0.1 10.0.0.2
Configured load-balance is “src-mac”, cannot select member of Po13 based on IP address
To test the IP load balancing to see if it selects the same Port for the communication, I configure the EtherChannel based on IP:
SW1(config)#port-channel ?
load-balance Load Balancing method
SW1(config)#port-channel load ?
dst-ip Dst IP Addr
dst-mac Dst Mac Addr
src-dst-ip Src XOR Dst IP Addr
src-dst-mac Src XOR Dst Mac Addr
src-ip Src IP Addr
src-mac Src Mac Addr
SW1(config)#port-channel load src-ip ?
<cr>
SW1(config)#port-channel load src-dst-ip ?
<cr>
SW1(config)#port-channel load src-ip
Highlighted in blue is to demonstrate all of these return only a <cr> after entering the load balancing parameter, whether it is source / destination or using both!
Now to perform a Layer 3 test to see if the traffic flow will hit the same port:
SW1#test ether load int po13 ip 10.0.0.1 10.0.0.2
Would select Fa1/0/3 of Po13
SW1#
Indeed it does hit a different port this time!
One good to note caveat with this, is that a source AND destination is required for the EtherChannel test command, whether its configured for only source or destination, both values are still required for the hash algorithm to determine which port is used.
The following is a scenario where you would want to change the default load balancing:
Â
SW1 will need to be changed from the default “src-mac” as all traffic flows will be coming from the same source MAC of the email server, and thus taking the same single link every time.
In this scenario, the default “src-mac” could be left on SW2, but SW1 would need to be changed to “dst-mac” or src-dst-mac” to perform best practice load balancing!
Ideally you will use “src-dst-ip” on switches to get the closest to equal cost load balancing, however if presented with limited options, make sure to consider a situation like the one above where you can accomplish semi-equal cost load balancing with “dst-mac”!
That is it for this one, up next will be Multi-Layer switching!
The DEVNET GRIND! 