STP EtherChannel – Fundamentals illustrated, “On” mode basic config, troubleshooting, and verification on lab!

During the lab I will only use the FastEthernet interfaces between SW1 and SW3 to demonstrate EtherChannel “On” configurations and behaviors, as I have found Multi-Mode Fiber interfaces do not interact the same way, as shown at the bottom of post!

 

STP EtherChannel Fundamentals

 

STP EtherChannel is the answer to STP’s Loop Prevention mechanism preventing the use of various links between switches, due to how it prevents loops, by putting all Non-Root / Non-Designated ports into blocking mode.

Consider the following topology:

With 4 links / 8 interfaces that can possible send data between SW1 and SW2, STP will put all ports on the Root Bridge into Desg / FWD, but on the remote switch it will only allow one link back to the Root Bridge making 3 links unavailable for throughput.

EtherChannel addresses this by allowing the Aggregation of links (2-8 links) to be “bundled” into one logical interface called a “Port-Channel” interface, and the logical Port-Channel interface will be treated by STP just as it would treat a physical interface.

To address 3 links being blocked, the following adjustment could be made with an EtherChannel configuration:

This allows all 4 FastEthernet links to be aggregated or “bundled” to allow 4 x 100mbps Full-Duplex throughput, rather than a single 100mbps Full-Duplex link, if any links in the bundle fail it will be taken out of the channel and the logical interface will remain Up.

EtherChannel supports FastEthernet (100mbps) / 1Gig / 10Gig interfaces, however interface types / links speeds and duplex must match across all links in the channel, or you will see this error:

SW1(config-if)#int gi1/0/1
SW1(config-if)#channel-group 13 mode on
SW1(config-if)#
*Mar 1 01:02:25.068: %EC-5-CANNOT_BUNDLE2: Gi1/0/1 is not compatible with Fa1/0/1 and will be suspended (speed of Gi1/0/1 is 1000M, Fa1/0/1 is 100M)
*Mar 1 01:02:26.050: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet1/0/1, changed state to down

EtherChannel interfaces must all be either Layer 2 or Layer 3 interfaces, you cannot have some Layer 3 interfaces on a Layer 2 EtherChannel, and vice versa.

(Layer 3 EtherChannel will not be covered in this post beyond that mention)

EtherChannel does not have any STP Forward Delay timers upon link failure within the bundle, the failed link will be “Suspended” from the EtherChannel, and the traffic flow that was going over the link will be re-initiated over another link in the bundle.

There are some exceptions to this with configuring EtherChannel Negotiation modes, however those will be covered in depth in the next post, I will focus on the EtherChannel mode as “On” mode in this post.

 

Speaking of which, here is a very basic EtherChannel “On” mode configuration

 

Step by step on SW1

SW1(config)#int ra fa1/0/1 – 4

Define range of links to be added

SW1(config-if-range)#channel-group ?
<1-48> Channel group number

Locally significant only, can be any # within this range

SW1(config-if-range)#channel-group 13 ?
mode Etherchannel Mode of the interface

Channel mode

SW1(config-if-range)#channel-group 13 mode ?
active Enable LACP unconditionally
auto Enable PAgP only if a PAgP device is detected
desirable Enable PAgP unconditionally
on Enable Etherchannel only
passive Enable LACP only if a LACP device is detected

“On” mode is a static mode that forces all bundled interfaces to be trying to form an EtherChannel with the remote side of the Port-Channel ONLY if it is running in “On” mode as well, this will not negotiation at all to form an EtherChannel!

The other 4 sub-commands are 2 different negotiation modes for 2 different EtherChannel negotiation protocols, which will be reviewed in a separate post.

Configuration on SW1

SW1(config-if-range)#channel-group 13 mode on
Creating a port-channel interface Port-channel 13

SW1(config-if-range)#

The 4 FastEthernet links are now bundled into Port-Channel 13 or Po13, my usual naming scheme to indicate where the links connect (SW1 – SW3).

Configuration on SW3

SW3(config-if-range)#channel-group 31 mode on
Creating a port-channel interface Port-channel 31

Just to note, on SW3 the locally significant # assigned for the Po# interface is 31, and is now configured so back to SW1 to verify!

Verifying on SW1

SW1(config-if-range)#do sh span

(…)

Interface Role Sts Cost Prio.Nbr Type
——————- —- — ——— ——– ——————————–
Gi1/0/1 Desg FWD 4 128.1 P2p
Gi1/0/2 Desg FWD 4 128.2 P2p
Po13 Desg FWD 8 128.608 P2p

There is our EtherChannel in all of its glory!

 

How to break an EtherChannel in static “On” mode – It isn’t difficult!

 

(Note: This err-disable lab only pertains to “On” mode, negotiation modes do not tank as aggressively as “On” mode)

To play with breaking some stuff, I decided to try removing an interface from an existing EtherChannel “On” mode bundle, and once it started going wrong, it kept going wrong:

Removing Interface Fa1/0/2 from channel-group

SW1(config)#int fa1/0/2
SW1(config-if)#no channel-group 13 mode on
SW1(config-if)#do sh span

(…)

Interface Role Sts Cost Prio.Nbr Type
——————- —- — ——— ——– ——————————–
Gi1/0/1 Desg FWD 4 128.1 P2p
Gi1/0/2 Desg FWD 4 128.2 P2p
Fa1/0/2 Desg FWD 19 128.4 P2p
Po13 Desg FWD 9 128.608 P2p

So Fa1/0/2 is in FWD status, Po13 is still in FWD status, life is good!

Then about 30 seconds later all interfaces bit the dust

*Mar 1 00:59:23.237: %LINEPROTO-5-UPDOWN: Line protocol on Interface FastEthernet1/0/1, changed state to down
*Mar 1 00:59:23.262: %LINEPROTO-5-UPDOWN: Line protocol on Interface FastEthernet1/0/2, changed state to down
*Mar 1 00:59:23.346: %LINEPROTO-5-UPDOWN: Line protocol on Interface FastEthernet1/0/3, changed state to down
*Mar 1 00:59:23.363: %LINEPROTO-5-UPDOWN: Line protocol on Interface FastEthernet1/0/4, changed state to down
*Mar 1 00:59:23.379: %LINEPROTO-5-UPDOWN: Line protocol on Interface Port-channel13, changed state to down

Every interface configured for the Port-Channel, along with the logical Port-Channel interface itself, is all goes into a non-admin “Down” state.

I tried adding the interface back to the Port-Channel, but it did not dynamically bring the Port-Channel back up:

SW1(config-if)#
SW1(config-if)#channel-group 13 mode on
SW1(config-if)#
(2 minutes later…)
SW1(config-if)#

I attempted the reliable “shut” / “no shut” to bounce the interfaces:

FastEthernet interfaces admin-down

SW1(config)#int ra fa1/0/1 – 4
SW1(config-if-range)#shut
*Mar 1 01:22:10.664: %LINK-5-CHANGED: Interface FastEthernet1/0/1, changed state to administratively down
*Mar 1 01:22:10.664: %LINK-5-CHANGED: Interface FastEthernet1/0/2, changed state to administratively down
*Mar 1 01:22:10.672: %LINK-5-CHANGED: Interface FastEthernet1/0/3, changed state to administratively down
*Mar 1 01:22:10.681: %LINK-5-CHANGED: Interface FastEthernet1/0/4, changed state to administratively down

Port-Channel interface admin-down

SW1(config-if-range)#int po13
SW1(config-if)#shut
SW1(config-if)#
*Mar 1 01:22:15.101: %LINK-5-CHANGED: Interface Port-channel13, changed state to administratively down

FastEthernet interfaces admin Up

SW1(config-if)#int ra fa1/0/1 – 4
SW1(config-if-range)#no shut
% Range command terminated because it failed on FastEthernet1/0/1
SW1(config-if-range)#int po13
*Mar 1 01:22:27.718: %EC-5-STAYDOWN: Fa1/0/1 will remain down as its port-channel Po13 is admin-down

The error highlighted in blue indicates that as long the logical Port-Channel interface is admin down, any interfaces bundled in that Port-Channel will also remain down, so I tried a “no shut” on the logical Port-Channel interface but continued to see “down” console messages for all bundled interfaces and the Port-Channel interface:

SW1(config-if-range)#int po13
SW1(config-if)# no shut
SW1(config-if)#
*Mar 1 01:28:56.614: %LINK-3-UPDOWN: Interface Port-channel13, changed state to down
*Mar 1 01:28:56.614: %LINK-3-UPDOWN: Interface FastEthernet1/0/1, changed state to down
*Mar 1 01:28:56.622: %LINK-3-UPDOWN: Interface FastEthernet1/0/2, changed state to down
*Mar 1 01:28:56.630: %LINK-3-UPDOWN: Interface FastEthernet1/0/3, changed state to down
*Mar 1 01:28:56.630: %LINK-3-UPDOWN: Interface FastEthernet1/0/4, changed state to down

It is at this point I looked over at my lab wondering what on Earth is going on, and I behold this sight, which led me right to the issue:

I did not stop to consider what may be causing this on the remote side of the link!

When moving back to SW3 I was met with a flood of err-disable

SW3(config-if-range)#
*Mar 1 00:59:13.900: %PM-4-ERR_DISABLE: channel-misconfig (STP) error detected on Fa1/0/1, putting Fa1/0/1 in err-disable state
*Mar 1 00:59:13.925: %PM-4-ERR_DISABLE: channel-misconfig (STP) error detected on Fa1/0/2, putting Fa1/0/2 in err-disable state
*Mar 1 00:59:13.942: %PM-4-ERR_DISABLE: channel-misconfig (STP) error detected on Fa1/0/3, putting Fa1/0/3 in err-disable state
*Mar 1 00:59:13.958: %PM-4-ERR_DISABLE: channel-misconfig (STP) error detected on Fa1/0/3, putting Fa1/0/4 in err-disable state
SW3(config-if-range)#

The EtherChannel will go into Err-Disable when it detects a configuration change / mismatch on the EtherChannel, to remedy this a “shut” / “no shut” on the Port-Channel interface itself is needed to correct:

Correcting via Interface Po31

SW3(config-if)#int po31
SW3(config-if)#shut
SW3(config-if)#no shut
SW3(config-if)#
*Mar 1 01:10:46.682: %LINEPROTO-5-UPDOWN: Line protocol on Interface Port-channel31, changed state to down
*Mar 1 01:10:47.630: %LINK-5-CHANGED: Interface FastEthernet1/0/1, changed state to administratively down
*Mar 1 01:10:47.647: %LINK-5-CHANGED: Interface FastEthernet1/0/2, changed state to administratively down
*Mar 1 01:10:47.663: %LINK-5-CHANGED: Interface FastEthernet1/0/3, changed state to administratively down
*Mar 1 01:10:47.672: %LINK-5-CHANGED: Interface FastEthernet1/0/4, changed state to administratively down
SW3(config-if)#
*Mar 1 01:10:47.697: %LINK-5-CHANGED: Interface Port-channel31, changed state to administratively down
SW3(config-if)#
*Mar 1 01:10:49.785: %LINK-3-UPDOWN: Interface FastEthernet1/0/1, changed state to up
*Mar 1 01:10:49.785: %LINK-3-UPDOWN: Interface FastEthernet1/0/2, changed state to up
*Mar 1 01:10:49.810: %LINK-3-UPDOWN: Interface FastEthernet1/0/3, changed state to up
*Mar 1 01:10:49.810: %LINK-3-UPDOWN: Interface FastEthernet1/0/4, changed state to up
*Mar 1 01:10:49.810: %LINK-3-UPDOWN: Interface Port-channel31, changed state to up

Being that int Fa1/0/2 is added back to the channel-group, and all interfaces are logically down and waiting to form an EtherChannel, it came right back up as verified on SW1:

SW1 “sh span” verification for Po13

SW1(config-if)#do sh span

(…)

Interface Role Sts Cost Prio.Nbr Type
——————- —- — ——— ——– ——————————–
Gi1/0/1 Desg FWD 4 128.1 P2p
Gi1/0/2 Desg FWD 4 128.2 P2p
Po13 Desg FWD 8 128.608 P2p

With negotiation protocols for EtherChannel, removing an interface from a Channel-Group simply raises the STP path cost and removes the link to allow it to being operating independent from the EtherChannel again – However “On” mode is an all or none mode.

Another way to verify EtherChannel is with “sh etherchannel summ”:

SW1#sh etherchannel summ
Flags: D – down P – bundled in port-channel
I – stand-alone s – suspended
H – Hot-standby (LACP only)
R – Layer3 S – Layer2
U – in use f – failed to allocate aggregator

M – not in use, minimum links not met
u – unsuitable for bundling
w – waiting to be aggregated
d – default port

 

Number of channel-groups in use: 2
Number of aggregators: 2

Group Port-channel    Protocol      Ports
——+————-+———–+———————————————–
12             Po12(SU)           PAgP         Fa1/0/13(P) Fa1/0/14(P) Fa1/0/15(P)
13              Po13(SD)              –              Fa1/0/1(D) Fa1/0/2(D) Fa1/0/3(D)
Fa1/0/4(D)

It is very important to note there are both Upper and Lower case letters, meaning S could mean either Layer 2 or Suspended depending on whether it is UPPER or lower case, so watch that carefully for exam day.

Also note the physical interfaces simply show bundled or down, while the Port-Channel interface shows what Layer the EtherChannel is, and what its current state is!

(Note if an interface just goes down, it will not cause an err-disable state, this only pertains to removing an interface from an existing Port-Channel in “On” mode)

I setup a quick negotiated channel (PAgP) for comparison of output, shown here:

• On mode should only – in the Protocol field, PAgP or LACP will have their acronym
• Only the Port-Channels show a “Layer 2” designated, interfaces only show states
• Group 13 is actually in Err-Disable, so there is no special designation for that, these interfaces will simply show as “D” or “Down”

So the command “sh ether summ” will give a great snapshot of EtherChannels and information pertaining to them , but for full details, you can use “sh ether det” and it will give you the full output of all details of all EtherChannels on the switch.

The output is huge so I won’t post it here, but it shows very detailed information like how long an interface has been in a certain status, how long its been since an interface has been added to the bundle, and other very detailed output regarding the Port-Channel interface as well – Be sure to review “sh ether det” output!

 

Adding a link to an “On” mode bundle, and Trunk issues to note for exam day

 

I wanted to try adding a link to the “On” EtherChannel to see if ti would go into Err-Disable, which it did not with adding a link as it did removing one, however I did completely forget to make the new interfaces a Trunk:

Configuring Fa1/0/5 on both SW1 and SW3 for EtherChannel

SW1(config-if)#int fa1/0/5
SW1(config-if)#channel-group 13 mode on
SW1(config-if)#
ASR#3
[Resuming connection 3 to sw3 … ]

SW3(config-if)#
SW3(config-if)#int fa1/0/5
SW3(config-if)#channel-group 31 mode on

EtherChannel errors yell at me about a couple different issues

SW3(config-if)#
*Mar 1 01:45:31.788: %EC-5-CANNOT_BUNDLE2: Fa1/0/5 is not compatible with Fa1/0/1 and will be suspended (trunk encap of Fa1/0/5 is auto, Fa1/0/1 is dot1q)
*Mar 1 01:45:32.794: %LINEPROTO-5-UPDOWN: Line protocol on Interface FastEthernet1/0/5, changed state to down
*Mar 1 01:46:01.517: %EC-5-CANNOT_BUNDLE2: Fa1/0/5 is not compatible with Fa1/0/1 and will be suspended (trunk mode of Fa1/0/5 is access, Fa1/0/1 is trunk)

Trunk encap configured, the interface is now “compatible”

SW1(config-if)#switchport trunk encap dot
SW1(config-if)#
*Mar 1 01:46:41.875: %EC-5-COMPATIBLE: Fa1/0/5 is compatible with port-channel members

Trunk “mode” configured, the interface is no “Up”

SW1(config-if)#switchport mode trunk
SW1(config-if)#
*Mar 1 01:46:42.898: %LINEPROTO-5-UPDOWN: Line protocol on Interface FastEthernet1/0/5, changed state to up

So that actually would have gone smoothly, had I already had Trunk configurations in place, but I think stepping through that is a good example that may come in handy for TSHOOT or SWITCH.

A quick review of some “On” mode concepts in regards to EtherChannel:

• Removing an interface puts state into Err-Disable for Port-Channel and interfaces
• An interface can be added, even only on one side, and it will join the EtherChannel without causing an issue (or an err-disable issue at least)
• Non-Trunk interfaces (or any mismatching values) will be suspended for incompatibility until the incompatible value is adjusted
• Not every interface must have a matching configuration on the remote end of the link for the Port-Channel to come up and an EtherChannel to form

The last one wasn’t covered yet, so just for a quick visual of what I mean there:

Remove EtherChannel from SW3

SW3(config-if-range)#no channel-group 31 mode on
SW3(config-if-range)#exit
SW3(config)#no int po31

Adding EtherChannel back to SW3

SW3(config)#int ra fa1/0/1 – 2
SW3(config-if-range)#channel-group 31 mode on
Creating a port-channel interface Port-channel 31

SW3(config-if-range)#int fa1/0/4
SW3(config-if)#channel-group 31 mode on
SW3(config-if)#

Now SW1 is configured with Fa1/0/1 – Fa1/0/4 configured for EtherChannel, while SW3 has Fa1/0/1, 2, 4 configured for EtherChannel and this is the output we get:

SW3

SW3(config-if)#do sh span

(…)

Interface Role Sts Cost Prio.Nbr Type
——————- —- — ——— ——– ——————————–
Fa1/0/3 Desg FWD 19 128.5 P2p
Fa1/0/12 Desg FWD 19 128.14 P2p
Po31 Root FWD 9 128.752 P2p

SW1

SW1(config-if)#do sh span

(…)

Interface Role Sts Cost Prio.Nbr Type
——————- —- — ——— ——– ——————————–
Po12 Desg FWD 47 128.600 P2p
Po13 Desg FWD 8 128.608 P2p

This demonstrates the forceful nature of “On” mode, as the remote end of Fa1/0/3 does not have a matching EtherChannel configuration, but it does remain “Up” and Forwarding and does not put the EtherChannel into err-disable state.

How to address making changes to EtherChannel values (speed, duplex, etc)

 

With any changes, it should be made on the Port-Channel interface, which will then “dynamically” change the corresponding values of all its bundled interfaces, however I did find a sort of “quirk” when playing with changing the values.

First I changed the entire physical interface range, which only bounced the port down and back up:

SW1(config-if)#int ra fa1/0/1 – 4
SW1(config-if-range)#speed 10
SW1(config-if-range)#
*Mar 1 01:14:01.222: %LINEPROTO-5-UPDOWN: Line protocol on Interface FastEthernet1/0/1, changed state to down
*Mar 1 01:14:01.247: %LINEPROTO-5-UPDOWN: Line protocol on Interface FastEthernet1/0/2, changed state to down
*Mar 1 01:14:01.314: %LINEPROTO-5-UPDOWN: Line protocol on Interface FastEthernet1/0/3, changed state to down
*Mar 1 01:14:01.339: %LINEPROTO-5-UPDOWN: Line protocol on Interface FastEthernet1/0/4, changed state to down
*Mar 1 01:14:01.356: %LINEPROTO-5-UPDOWN: Line protocol on Interface Port-channel13, changed state to down
SW1(config-if-range)#
*Mar 1 01:14:02.371: %LINK-3-UPDOWN: Interface Port-channel13, changed state to down
SW1(config-if-range)#
*Mar 1 01:14:06.012: %LINEPROTO-5-UPDOWN: Line protocol on Interface FastEthernet1/0/1, changed state to up
*Mar 1 01:14:06.113: %LINEPROTO-5-UPDOWN: Line protocol on Interface FastEthernet1/0/2, changed state to up
*Mar 1 01:14:06.155: %LINEPROTO-5-UPDOWN: Line protocol on Interface FastEthernet1/0/3, changed state to up
*Mar 1 01:14:06.171: %LINEPROTO-5-UPDOWN: Line protocol on Interface FastEthernet1/0/4, changed state to up
SW1(config-if-range)#
*Mar 1 01:14:06.993: %LINK-3-UPDOWN: Interface Port-channel13, changed state to up
*Mar 1 01:14:08.000: %LINEPROTO-5-UPDOWN: Line protocol on Interface Port-channel13, changed state to up

So I did NOT need to go into the Port-Channel interface configuration, as long as I was configuring the entire range at once it behaved as though I changed in on the Port-Channel, so if the ports are consecutive you CAN change the speed with Interface Range configuration mode – HOWEVER.

However, if you try to change it on one or more interfaces individually or in small groups, those links will be suspended:

SW1(config-if-range)#int fa1/0/2
SW1(config-if)#speed 100
SW1(config-if)#
*Mar 1 01:15:02.752: %LINEPROTO-5-UPDOWN: Line protocol on Interface FastEthernet1/0/2, changed state to down
SW1(config-if)#
*Mar 1 01:15:06.578: %EC-5-CANNOT_BUNDLE2: Fa1/0/2 is not compatible with Fa1/0/1 and will be suspended (speed of Fa1/0/2 is 100M, Fa1/0/1 is 10M)
SW1(config-if)#

So in a case like with SW3 where its Fa1/0/1, Fa1/0/2, and Fa1/0/4 in the EtherChannel – The Port-Channel interface would HAVE to be used to adjust all channel-group members values at once to avoid suspending any links!

 

One odd quirk between Ethernet interfaces vs Fiber when it comes to Err-Disable

 

When configuring the FastEthernet faces initially on SW1, I didn’t get to SW3 immediately, and it actually threw the Port-Channel into err-diable on SW1:

SW1(config-if-range)#channel-group 13 mode on  <— FastEthernet EtherChannel
Creating a port-channel interface Port-channel 13

SW1(config-if-range)#
*Mar 1 00:07:42.027: %LINK-3-UPDOWN: Interface Port-channel13, changed state to up
*Mar 1 00:07:43.034: %LINEPROTO-5-UPDOWN: Line protocol on Interface Port-channel13, changed state to up
SW1(config-if-range)#
SW1(config-if-range)#int ra gi1/0/1 – 2
SW1(config-if-range)#channel-group 12
*Mar 1 00:08:23.710: %PM-4-ERR_DISABLE: channel-misconfig (STP) error detected on Fa1/0/1, putting Fa1/0/1 in err-disable state
*Mar 1 00:08:23.735: %PM-4-ERR_DISABLE: channel-misconfig (STP) error detected on Fa1/0/2, putting Fa1/0/2 in err-disable state
*Mar 1 00:08:23.752: %PM-4-ERR_DISABLE: channel-misconfig (STP) error detected on Fa1/0/3, putting Fa1/0/3 in err-disable state
*Mar 1 00:08:23.777: %PM-4-ERR_DISABLE: channel-misconfig (STP) error detected on Fa1/0/4, putting Fa1/0/4 in err-disable state
SW1(config-if-range)#

As mentioned above, if absolutely no Channel-Group mode On is detected on the remote end of this logical Port-Channel bundle it will go into Err-Disable state, and before I could even finish my Gig Interface configuration they all went into Err-Disable.

However this is not the case when configuring Multi-Mode Fiber interfaces:

SW1(config)#int ra gi1/0/1 – 2
SW1(config-if-range)#channel-group 12 mode on
Creating a port-channel interface Port-channel 12

SW1(config-if-range)#
(Couple of minutes go by)
SW1(config-if-range)#

The Port-Channel interface does not come up, however with “sh span” :

SW1(config-if-range)#do sh span

(…)

Interface Role Sts Cost Prio.Nbr Type
——————- —- — ——— ——– ——————————–
Po12 Desg FWD 3 128.600 P2p <—- Uhhh, what? (1 sided Gig interface config)
Po13 Desg FWD 7 128.608 P2p

On SW2 this gets even more confusing:

SW2(config-if-range)#do sh span

(…)

Interface Role Sts Cost Prio.Nbr Type
——————- —- — ——— ——– ——————————–
Gi1/0/1 Root FWD 4 128.1 P2p
Gi1/0/2 Desg FWD 4 128.2 P2p

All lights are green on the interfaces, several minutes have elapsed without any errors, apparently with Multi-Mode Fiber it will allow for a one-sided configuration of EtherChannel “On” mode.

I am fairly certain this is a special condition for multi-mode fiber connections, but given its not exactly an exam topic, I wanted to share more for the sake of knowledge.

That concludes my review of basic EtherChannel “On” mode config and its quirks, next will be a much more detailed review with an emphasis on negotiation modes!