Putting Multicast labbing on pause to hit Nexus 9K series / ACI / SDN studies and lab it up!
After a slight hiccup in continuing on with Multicast studies, I’ve decided to table that to learn the Cisco Nexus 9000 Data Center switching hardware platform and ACI Software that runs on it, as going back to lab Nexus and ACI configuration I remember surprisingly not much – So time to start from square one!
Being this concept is a bit more advanced there is not exactly a “Beginner” level to this, I highly encourage anyone wanting to learn ACI fully to start with Cisco Learning Labs linked below:
https://developer.cisco.com/learning/labs/tags/Networking
I will be writing more expansive articles on ACI, this is really just the 10 minute intro to how ACI works at a very high level view, but I definitely encourage utilizing DevNet resources to help learn these concepts if this article interests you – Cisco has excellent free resources for training!
This article will also need some cleanup as my understanding of ACI evolves, please leave a comment if you see incorrect info on this page so it can be corrected – Thank you!
The fundamentals of Cisco Nexus 9000 Switches, ACI, and SDN!
CIsco Nexus 9000 series switches is the Nexus switch model / family that supports ACI or Application Centric Infrastructure, which is Cisco SDN (Software Defined Network) Data Center solution, that allows for Network Automation through a policy based model that is configured on an ACI Controller (APIC) to then push those policies into the ACI Fabric devices for configuration per the policy defined.
So what exactly is ACI?
ACI is software that provides “Programmatic Configuration” of the Cisco Nexus 9K Spine and Leaf switches by changing objects on the APIC (Application Policy Infrastructure Controller), which then applies the policies to the ACI “Spine” and “Leaf” switches that it controls, providing the Leaf switches instruction on how to route traffic while Spine switches focus on passing traffic between Leafs.
The APIC can be configured via REST API, Cobra SDK, SSH, Ansible, and a few other Orchestration tools similar to Ansible – I will be using a DevNet Sandbox for ACI Labbing as CML 2.0 is not capable of booting into ACI from its Nexus 9K images.
The underlying protocols and “Policy Model” that makes up ACI
ACI uses an Automated VXLAN “Overlay” Tunnel System that supports both Layer 2 and Layer 3 VXLAN Gateways, meaning that it can operate both as a Layer 2 Switch and Layer 3 Router using VXLAN Tunnels, while using IS-IS and MP-BGP Routing Protocols to form the “Underlay” of the network between Spine and Leaf switches, while Leaf switches build the VXLAN Tunnels to other Leaf Switches which forms the “Overlay” of the ACI Fabric.
One thing I noticed was different with the Nexus 9K / ACI model upon getting into the Switch CLI:
switch# sh run
!Command: show running-config
!Running configuration last done at: Thu Apr 22 03:59:36 2021
!Time: Thu Apr 22 04:01:54 2021
version 9.2(3) Bios:version
vdc switch id 1
limit-resource vlan minimum 16 maximum 4094
limit-resource vrf minimum 2 maximum 4096
limit-resource port-channel minimum 0 maximum 511
limit-resource u4route-mem minimum 128 maximum 128
limit-resource u6route-mem minimum 96 maximum 96
limit-resource m4route-mem minimum 58 maximum 58
limit-resource m6route-mem minimum 8 maximum 8
no password strength-check
username admin password 5 $5$2GdRNGTv$Jb4MQlfMd/8WjfiWB6et7XK2dBdz8xxRTc9Fiu43on
3 role network-admin
username cisco password 5 $5$m6hp4wl4$uywwW/VWIYxeqDoDNfS2xdSxzORPp9VG772zCGiJug
1 role network-admin
username cisco passphrase lifetime 99999 warntime 14 gracetime 3
ip domain-lookup
snmp-server user admin network-admin auth md5 0x10ed70d94fb029b5c7cd2b288653772f
priv 0x10ed70d94fb029b5c7cd2b288653772f localizedkey
snmp-server user cisco network-admin auth md5 0x10ed70d94fb029b5c7cd2b288653772f
priv 0x10ed70d94fb029b5c7cd2b288653772f localizedkey
rmon event 1 description FATAL(1) owner PMON@FATAL
rmon event 2 description CRITICAL(2) owner PMON@CRITICAL
rmon event 3 description ERROR(3) owner PMON@ERROR
rmon event 4 description WARNING(4) owner PMON@WARNING
rmon event 5 description INFORMATION(5) owner PMON@INFO
vlan 1
vrf context management
interface Ethernet1/1
interface Ethernet1/2
interface Ethernet1/3
interface Ethernet1/4
interface Ethernet1/5
interface Ethernet1/6
interface Ethernet1/7
interface Ethernet1/8
interface Ethernet1/9
interface Ethernet1/10
Etc – The entire running configuration is blank!
This is because ACI uses a “whitelist” model where policies must be configured to explicitly allow traffic and connections by default, so if traffic is not explicitly allowed, its implicitly denied again by default.
In this way ACI really provides the services of a Switch, Router, and a Firewall!
The Hardware Components and Network Design of ACI
There are 3 main hardware components to the ACI Network:
- The APIC (Application Policy Infrastructure Controller) – Brains of the ACI System, this communicates routing policies to Leaf switches, and dynamically discovers and builds the Underlay and Overlay that creates the ACI Fabric across the devices
- Spine Switches – These only handle Data Transport between the Leaf Switches and Underlay connection
- Leaf Switches – These handle the Overlay VXLAN encapsulation between other Leafs, perform a majority of the data processing as set by Policy on the APIC
The APIC is the Controller which runs on either a dedicated server or a Virtual Machine, it is programmed via GUI / API / CLI, and the APIC then discovers / builds the ACI Fabric dynamically and implements routing on Leaf switches as defined the Policies configured on the APIC.
APICs connect to Leaf Switches and only Leaf switches connect to APICs, this is how ACI dynamically discovers the fabric devices, as described at the bottom of this post!
The Spine and Leaf switches are accessible via CLI but is read-only, as these are configured programatically via the APIC, so configuration of the Spine / Leaf switches is done on the APIC.
Leaf Switches are the VXLAN Tunnel Endpoints (VTEPs) which are “plug and play” in that ACI dynamically sets up the IS-IS / MP-BGP Underlay and VXLAN Overlay, these configurations are handled in the background by the ACI System, meaning Leaf switches should be able to plug into a “Spine” and begin forwarding once the APIC delivers its forwarding instructions / policies.
APICs run in clusters of 3, 5, or 7 servers per the guidelines in the following Cisco article:
The ACI Database is “Sharded” across all APICs that are in use (that control the same set of data), in that it distributes the Configuration / Policies across all APICs similar to a RAID schema, this provides both a distributed workload and redundancy as in the event of APIC failure the traffic will still be forwarded without interruption – However if all but a single APIC fail the Fabric will convert to “read-only”.
Traffic will continue to be forwarded because APICs are in the “Control” Plane, but not the “Data” Plane, so even if the traffic goes into a “read-only” mode in which data is forwarded with the existing configuration.
Nexus 9K Switch models 9500 and 9300 are able to run ACI, but the 9200 series is NX-OS only.
The big take away from this other than 9200 series cannot run ACI, is that the other 9500 and 9300 can run both NX-OS or ACI, however they can only run NX-OS or ACI at once.
Direction of communication and protocols used between devices (And Collapsed Core concept!)
As shown in the Topology there are no connections between Spine to Spine or Leaf to Leaf switches, meaning there is no “East/West” bound traffic, all traffic must flow from Leaf to Spine to Leaf.
The Spine switches form what is called a “Collapsed Core” which performs extremely high speed forwarding between Leaf switches, which means it is a 2-Tier model rather than the usual 3-Tier model taught in a traditional Network Topology of switches.
The Spines also host the “Endpoint Database” which uses Council of Oracle Protocol (COOP), which will be reviewed in much more depth later, but wanted to mention that in this Intro article.
The Leaf switches connect Northbound to Spines, and Southbound to nodes or other devices, they do not have east/west communication but rather communicate through the Spines to talk Leaf-to-Leaf:
Leafs form the IS-IS / MP-BGP Underlay to the Spines, however the Spines do not participate in VXLAN at all (they only forward data between Leafs), so Leafs perform the encapsulation and decapsulation of VXLAN between Access Ports (Southbound) or Fabric Ports (Northbound).
This means the Spines are not aware of and do not participate in VXLAN at all, they simply forward data on the “Underlay” connections to Leaf Switches, and the Leafs handle the VXLAN “Overlay” tunnels.
A review of how the ACI Fabric is designed in terms of connections and scalability
ACI uses what is called a “CLOS” Topology which entails the following attributes:
- All Leaf switches uplink to all Spine switches
- APICs only connect to Leaf switches
- APICs dual connect to Leafs to dynamically discover the Topology
- No cross-connects between Leafs or Spines (East/West traffic)
- Data flows Leaf to Spine to Leaf (No cross connects for Spine to Spine)
- Bandwidth is Scalable by adding more Spine switches to the “Collapsed Core”
A point of clarity on the APICs dual connecting to Leaf Switches
APICs are plugged into the Leaf switches, and perform discovery by building and maintaining dual connections to Leafs, which is how ACI dynamically discovers spines and other Leafs to build the Underlay and Overlay of the ACI Fabric.
This will conclude a very brief overview of ACI fundamentals and design concepts!
My CML 2.0 does not support ACI on the Nexus 9K switches, so I will be reserving a lab after work to dig into how to configure / troubleshoot / verify ACI in a Cisco DevNet Sandbox environment!
Tons of excellent Sandboxes on-demand or free to reserve, go check them out:
https://developer.cisco.com/site/sandbox/
Being this is a very new technology I will be jotting notes as I understand them at the time and clean them up as my understanding evolves, please leave a comment if any information is incorrect or incomplete, to help me keep this information as accurate and concise as possible – I greatly appreciate it!
Next Up – Labbing Cisco ACI in DevNet Sandbox!
The DEVNET GRIND! 