Being that it has been quite awhile since my ROUTE, this will contain a lot of core / foundation concepts, and tie them into troubleshooting as I go.
That said, here we go!
OSPFv2 (IPv4) review and troubleshooting
OSPF is configured with a Process # that is locally significant only “router ospf #” and networks / interfaces are OSPF enabled if their network is defined within the OSPF config with a “network 10.10.1.0 0.0.0.255 area 0” statement, which this can also be defined directly on an interface to enable ospf with “ip ospf (process #) area (#).
OSPF discovers neighbors / forms adjacencies / propagates routes by exchanging LSA’s which of course are different for each Area type, which a great link can be found here for LSA types (and lots of them can be found using the search function:
This is something I need to re-read myself big time!
The first command you will want to be familiar with is “sh ip ospf nei” as this will show the following values of the neighbor:
- Neighbor ID – The RID of the neighbor
- Priority – The Priority of the neighbor
- State – Will be in format of State/Router Type (Full/BDR), will discuss much more in depth regarding all states and neighbor types
- Dead Time – The countdown until the next Hello needs to be received or they neighbor will be considered dead / Adjacency will drop (default dead time 40 sec)
- Address – The actual IP Address of the neighboring link interface
- Interface – The LOCAL interface off which the neighbor is found
(Passive Interfaces will not show in ospf neighbor table!)
This is a great link for OSPF neighbor Adjacency formation!
“sh ip ospf” is another very powerful command displaying the following:
- Each OSPF Process #
- The RID for each Router Process #
- Whether Authenticaton is enabled
- The Areas within that router
- The # of times the SPF algorithm has ran
- The last time the SPF algorithm ran
- Lots of LSA information
- What types of Area the Areas are within the process (Backbone / NSSA / etc)
All that being said, lets get into some reasons why an Adjacency now form, or why some routes may be missing that are expected to be in the IP Route table:
- Interface is down – Of course needs to be Up/Up
- Interface not running OSPF – Must be OSPF enabled to Send / Receive OSPF Hellos
- Mismatched Timers – Timers must match between neighbors to from Adjacency
- Mismatched Area # – Area # must match between neighbors to form Adjacency
- Mismatched Area type – Neighbors must agree on type of area they are in such as NSSA, Stub, or normal Area
- Mismatched Subnets – Neighbor interfaces must reside in same subnet to form an Adjacency between them
- Passive interface – Will not Send or Receive Hellos if interface is Passive
- Authentication Mismatch – Authentication info is not exact on both nbr interfaces
- ACL – ACL possibly blocking Multicast address 224.0.0.5 for OSPF
- MTU Mismatch – MTU of neighboring interfaces must match for Adjacency to form
- Duplicate Router IDs – RIDS must be unique
- Mismatched network type – NOT SAME AS AREA TYPE, this refers to Point to Point, Multipoint, NBMA and other different network segment types
Adjacencies are not automatically made upon the receipt of Hellos from two neighbors, there is a series of states the neighbors transition through to get to Full / Formed:
- Down – No Hellos received from neighbor
- Attempt – Router sends a Unicast Hello
- Init – When router receives a Hello from neighbor
- 2way – Both routers have received Hellos
- ExStart – When routers begin to exchange Route Information based on which router has the highest RID, Highest RID becomes the “Master” and the other end the “Slave” between the two, this process continues for all neighbor Adjacencies to find the DR / BDR
- Exchange – When the two routers are sending each other DBD or DataBase Descriptors that contain info about the routers Link-State Database, as all OSPF neighbors in the same Area must have matching LSDB’s
- Loading – Based on LSDB’s Routers will request any missing LSA’s from the LSDB during the “Loading” state of forming the Adjacency to fill in any gaps
- Full – This state indicates the LSDB / LSA exchange is completed, and adjacency has formed between OSPF neighbors
When a neighbor is not being listed in the “sh ip ospf nei” table, you will want to use “sh cdp nei (det)” as this will first show your neighbors in a brief mode and use “detail” to find IP Addresses / Subnet information to confirm they are on the same subnet.
To verify the interface is running OSPF use “sh ip ospf int brief” to see currently enabled OSPF interfaces, and if not showing in the output as expected it can be verified a few way with “sh ip proto” to see if the network encompassing the interface IP is entered, “sh run | sec router ospf” to see if the network statement is incorrect (network / mask / area), and “sh run int (int)” to see if OSPF is enabled correctly on the interface.
(Again, remember passive interfaces do not show up in “sh ip ospf int” output)
*If an interface is configured for OSPF with “ip ospf # area #” it will show up in the “sh ip proto” as a host address in that area, for example “network 10.10.10.1 0.0.0.0 area 0” UNLESS a network statement exists with that network segment / area # already configured.
Timers are a bit different with OSPF, as they change depending on the Topology, and the Dead Timer automatically adjusts to be 4 x the Hello timer when changed, however this does not dynamically change the OTHER SIDE OF THE LINK – Thus Timer Mismatches can occur!
The default timers are as follows per network type:
- Hello 10sec / Dead 40sec – Broadcast and Point-to-Point
- Hello 30sec / Dead 120sec – NBMA and Point-to-Multipoint
“sh ip ospf int (int)” can be used to see all OSPF parameters for an interface (including its timers), also “debug ip ospf hello” will reveal if there is a mismatched parameter and will even provide the hello / dead timers received (R) to its own configured (C) in the output.
“ip ospf hello-interval” is the interface level command to change timers, cannot be changed in router configuration for entire process / area, must be configured per interface and Dead timer will auto-adjust with it to be 4x Hello (in seconds).
Mismatched Area #’s can be found via “sh ip ospf int (int)” as well to see the Area it belongs to, along with “sh ip ospf int brief” to get a condensed view of the OSPF interfaces and information such as Subnet / Area / State / Cost / Local Interfaces.
“debug ip ospf adj” will also reveal Area mismatches in the “mismatched area” portion of the output represented oddly enough in dotted decimal format (so 0.0.0.1 = Area 1).
Mismatched Area types will cause Adjacaency issues if the connected interfaces on neighbor routers think they are in different OSPF Area types, either Normal / Stub / NSSA Area types to be specific. To view Area Type information, use “sh ip proto” to see the # of different areas in the OSPF Process, and “sh ip ospf” to see which Areas / Interfaces are configured as what type of OSPF Area.
Different subnets is pretty easy to verify, a quick “sh ip int brief” or “sh run int (int)”
Passive Interfaces are shown in “sh ip proto” output, by doing a “debug ip ospf hello” to see if a certain interface is not sending Hellos, and in the running config OSPF section.
OSPF has 3 authentication levels to know in debug output:
- Type 0 – Null, no authentions
- Type 1 – Authentication sent in plain text
- Type 2 – Authentication sent in MD5
Both types are configured on the interface directly, with the following commands:
Clear Text (Type 1)
“ip ospf authentication-key secret”
“ip ospf authentication”
MD5 Encrypted (Type 2)
“ip ospf message-diget-key # md5 secret”
“ip ospf authentication message-digest”
To verify if authentication is set “show ip ospf” can be used to see if Authentication is in use in an Area at all, and “sh ip ospf int (int)” is used to drill down a bit further to see what type of Authentication is configured, finally “debug ip ospf adj” can be used to see what “Type #” is being locally used and what is being received.
ACL’s can also rain on your OSPF parade if you see the Multicast IPv4 address of 224.0.0.5 in an ACL or Prefix-List applied to a filter-list, which you can verify if an ACL is applied via “access-group …” command with “sh ip int (int)” and then review the ACL, or with “sh ip proto” to see if a filter list is being used and verify a Prefix-List with “sh ip prefix-list” to verify the Multicast Address is not being filtered.
One very important point is recognizing an MTU mismatch simply by seeing neighbors stuck in one of two states: EXSTART or EXCHANGE
If you see neighbors stuck in this, do not confuse it with neighbors being “Stuck in 2way” that are on a point-to-point Ethernet segment, if you see neighbors stuck in a state starting with “EX…” it indicates an MTU mismatch!
“sh run int (int)” to show the MTU on both sides, default should be 1500, unless there is a GRE Tunnel. To fix this either issue “ip mtu #” on the interface, or you can also use the command “ip ospf mtu-ignore” command on the interface as well!
Duplicate Router-ID’s will cause an adjacency not to form, and will kick out a syslog indicating if it detects a duplicate RID from a neighbor, the RID can be verified in “show ip proto” output and can be changed in the “router ospf #” process with “router-id x.x.x.x” and yes it must be in IPv4 format even for IPv6 RID’s!
If changing a RID while troubleshooting, must do a “clear ip ospf process” which will drop all adjacencies to that router!
A listing of the different Network Types, and how to configure them in OSPF
Different types of networks work differently with OSPF, including having different default timers and need neighbors statically configured, so I’ll list them off here:
- Broadcast Network – Default on LAN interfaces, Neighbors Discovered Automatically, DR / BDR Elected Automatically, Timers Hello 10 / Dead 40 seconds
- Non-Broadcast Multi-Access (NBMA) – Default on Frame Relay and Point-to-Multipoint interfaces, Neighbors Statically Configured, DR must be manually configured on Hub router, Timers Hello 30 / Dead 120 seconds
- Point-to-Point – Default on Point-to-Point serial and Point-to-Point Frame-Relay subinterfaces, Neighbors Discovered Automatically, No DR or BDR, Timers Hello 10 / Dead 40 seconds
- Point-to-Multipoint – (Not a default) Optimal for Hub-and-Spoke (Frame-Relay) networks, Neighbors Discovered Automatically, No Dr or BDR, Timers Hello 30 / Dead 120 seconds
- Point-to-Multipoint NonBroadcast – (Not a default) Optimal for Hub-and-Spoke (Frame-Relay) that don’t support Broadcast or Multicast, Neighbors Statically configured, No DR or BDR, Timers Hello 30 / Dead 120 seconds
A quick recap of the DR (Designated Router) and BDR (Backup DR) Election
By default OSPF Priority is 1 for all routers, and the DR / BDR is elected by highest RID configured in the Area, or if a RID is not manually configured the highest IP Address is considered the RID.
You can influence which routers become the DR / BDR, or even particpate at all by configuring a static priority # on the interface in the OSPF Area, with the command “(ip/ipv6) ospf priority #” where 0 will make the interface not participate in the election at all while higher values will cause the router to win the DR / BDR election.
As stated in the network segments some network types don’t need a DR / BDR, for example a PtP or PtMP networks, as there is no need for a DR and BDR because there are only two OSPF Routers as per the type of network!
In these instances you will see states of “2way/DROTHER” which is completely fine, they are neighbor adjacent but not fully adjacent (because they do not need to be), and they exchange the appropriate LSA’s to keep their Link State Databases synchronized.
Non-Broadcast Multi-Access however, you must statically configure neighors in the router process for ospf with the “neighbor x.x.x.x x.x.x.x” command on the Hub, as these type of networks are not meant to dynamically find neighbors so they must be statically configured on these segment types – So if your missing a neighbor on an NBMA segment you may need the “neighbor …” statement to correct the issue.
On an NBMA Frame-Relay network (which states manual configuration of DR on Hub), you can really just issue the “ip ospf pri 0” on the spoke interfaces so they don’t participate in the election, and the Hub will become the DR with no BDR on the spokes.
Back to troubleshooting, why OSPFv2 routes may not be appearing as expected
Now that neighbor adjacencies have been covered in detail, what if routes are missing despite an adjacency being present?
Lets take a look at some reasons why OSPF Routes may go missing:
- Interface is not running OSPF Process – Either it is not enabled in the router process configuration via network statement, not configured for the OSPF process on the interface, OR POSSIBLY CONFIGURED IN THE WRONG OSPF PROCESS #!
- Better source of information – Two words: Administrate Distance!
- Route Filtering – ACL’s and Prefix-Lists set in a Filter List in “sh ip proto”
- Stub Area Configuration – Depending on the type of Stub (Total Stub or NSSA) it will share different types of routes, more on this to come
- Interface is shut down – So obvious it may be easy to overlook!
- Wrong DR was elected – This is an issue with Hub-and-Spoke / Frame-Relay / NBMA networks where neighbors do not automatically discover each other!
- Duplicate RID – “debug ip ospf adj” or “sh ip proto” to verify duplicate RIDs
There isn’t much that needs to be touched on that isn’t fairly obvious to me (for the sake of keeping this article as short as possible), one of them being the Administrative Distance and the other being Stub networking with OSPF.
Administrative Distance does not change with OSPF when Redistributed or OSPFv3, it is always 110 unless changed with the “distance …” command which there is two flavors:
- “distance #” – This changes the AD for all OSPF routes in the IP Route Table
- “distance ospf external # intra-area # inter-area #” – This changes the AD for the 3 different types for OSPF, External being Redistributed Routes, Intra-Area routes being within the same Area, and Inter-Area being OSPF routes brought in from other Areas
Both of these are configured in the OSPF router process configuration, and verified with “sh ip proto” or “sh ip ospf” on OSPF enabled routers.
Time for complete DEEP DIVE LSA / Link State Database review!!!
I am going to borrow a huge explanation from an article on LSA I wrote long ago that can be reviewed in totality here, however this is so important to know I will repeat it here!
So first, I will start with a description of each LSA type of the 7 of them:
- LSA Type 1 “Router” – “Router Link States” will be its header in the LSA DB, and the name is self explanatory, these LSA’s are generated by each router with updates on its local Link States, all router types generate and flood this LSA Type.
- LSA Type 2 “Network” – “Net Link States” are only generated and sent by DR’s and BDR’s to routers in the Same Area, that are also on the same multi-access network type, LSA type stays within its own Area, only seen in NON-Point-to-Point network types
- LSA Type 3 “Summary” – “Summary Net Link States” has nothing to do with summarization, but floods its summary of networks from one Area into others except for the Area it is part of – Not flooded into Total-Stub’d Areas (Stub or NSSA)
- LSA Type 4 “Summary ASB” – “Summary ASB Link States” LSA type is only created by ABR’s back to the ASBR, so when redistribution is configured on the ASBR Router it flips a bit in its “Router LSA” (Type 1!), and the ABR(s) then create LSA type 4’s to pass along throughout the network giving OSPF neighbors the path back to the ASBR – Not flooded into Stub Areas.
- LSA Type 5 “Autonomous System External Link State” – or “AS External Link States” in the OSPF LSA DB, these are your “O E1” and “O E2” Redistributed routes, generated from the ASBR itself OUTSIDE an NSSA Area – Not flooded into Stub Areas.
- LSA Type 6 – Not needed for the CCNP ROUTE, but it is for Multicast Extensions of OSPF (MOSPF), but again is not referenced in the the ROUTE exam, just wanted to mention for the sake of thoroughness
- LSA Type 7 “NSSA LSA’s” – This type of LSA is generated by the ASBR INSIDE an NSSA Area does Redistribution, as Type 5 Redistribution LSA’s cannot enter an NSSA Area
Phew. So to cover what type of routers create which type of LSA’s ONE MORE TIME:
- Type 1 – All Routers
- Type 2 – All DR’s
- Type 3, 4 – All ABR’s
- Type 5 – ASBR’s OUTSIDE the NSSA Areas (NSSA’s don’t allow LSA type 5)
- Type 6 – Reserved for MOSPF
- Type 7 – ASBR’s INSIDE the NSSA Areas (Type 7 LSA’s [N1, N2 in route table])
Given this new found understanding of LSA’s, lets review a Link State Database!
Here is the Topology and below it will be its OSPF Link State Database:
R2#sh ip ospf database
OSPF Router with ID (2.2.2.2) (Process ID 1)
Router Link States (Area 0)
Link ID ADV Router Age Seq# Checksum Link count
1.1.1.1 1.1.1.1 350 0x80000002 0x00DCA2 1
2.2.2.2 2.2.2.2 350 0x80000007 0x00E65D 2
3.3.3.3 3.3.3.3 351 0x80000007 0x001D10 2
Net Link States (Area 0)
Link ID ADV Router Age Seq# Checksum
172.12.123.1 1.1.1.1 346 0x80000004 0x0023BE
Summary Net Link States (Area 0)
Link ID ADV Router Age Seq# Checksum
1.1.1.1 1.1.1.1 351 0x80000004 0x0041EF
2.2.2.2 2.2.2.2 1406 0x80000003 0x00F633
3.3.3.3 3.3.3.3 1373 0x80000003 0x00AA77
172.12.15.0 1.1.1.1 351 0x80000005 0x0072F9
172.12.34.0 3.3.3.3 1373 0x80000005 0x0064EC
Router Link States (Area 2)
Link ID ADV Router Age Seq# Checksum Link count
2.2.2.2 2.2.2.2 1406 0x80000003 0x00A571 1
Summary Net Link States (Area 2)
Link ID ADV Router Age Seq# Checksum
1.1.1.1 2.2.2.2 340 0x80000001 0x00AB44
3.3.3.3 2.2.2.2 340 0x80000001 0x004F98
22.2.2.2 2.2.2.2 1164 0x80000002 0x00F323
33.3.3.3 2.2.2.2 340 0x80000001 0x00C702
172.12.15.0 2.2.2.2 340 0x80000001 0x00DE4D
172.12.34.0 2.2.2.2 340 0x80000001 0x000D0C
172.12.123.0 2.2.2.2 340 0x80000001 0x002C94
R2#
LSDB’s do not display the LSA Type / Number, so it is important to know the names of each LSA Type, or at least what they appear as in the LSDB so you can read it properly.
The LSDB will display LSA Types 1 – 7 started with Area 0 and go downward by Area # with the LSA’s it knows for each area.
A quick break down of the most common LSA #’s to know about and how they work:
- LSA Type 1 – This is created when a network is added to an OSPF Area, the router creates an LSA Type 1 and floods it to all Routers within the Area (Intra-Area), shows in IP Route Tables as O next to the route
- LSA Type 3 – These are created by ABR Routers, it takes the OSPF Routes or LSA Type 1’s from one Area and creates an LSA Type 3 to flood into another area (Inter-Area) which shows in the IP Route Table as OI next to the route, the cost advertised is the cost back to the ABR to reach the Destination in the Inter-Area
- LSA Type 5 – Non-OSPF / External routes learned by OSPF via Redistribution
- LSA Type 7 – NSSA Specific LSA’s that are basically LSA Type 5’s converted into LSA Type 7’s for this specific Area
I won’t dive into the details of LSA types and how to find them out, for that you will need to review my other articles that do delve into them once again here and here.
However the take away from LSA’s and the LSDB is being able to first verify the route is contained within it, and then to find the “ADV Router” or the “Advertising Router” that the route came from!
You can also use “sh ip route x.x.x.x x.x.x.x” to get a detailed output of the routes details including where it originated from, if not in the LSDB, that is a powerful troubleshooting tool for exam day (it definitely saved my butt on the ROUTE exam).
A review of Stub, Total Stub, and NSSA Area types with LSA info is fresh!
I want to hit this quick while the LSA types and LSDB is right there, as these OSPF Area types are configured to only accept / advertise certain LSA types, and this needs to be understood as well when passing an exam or working with OSPF on the job.
Again, coming back to this Topology:
First to define the different types of OSPF Routers as shown in the Topology:
- Backbone Area = Area 0
- Backbone Router = OSPF Router has at least one interface in Area 0
- Internal Router = All OSPF Interfaces are in a single Area
- ABR / Area Boarder Router = At least one interface in Area 0 and one interface in a non-backbone Area
- ASBR / Autonomous System Boundary Router = At least one interface in an OSPF Area and at least one interface in another Routing Protocol performing Redistribution into OSPF
- Transit Area = Area which a Virtual Link connects an Internal Router (R4 and R5 in the Topology) to an ABR that has an interface in Backbone Area 0
- Virtual Link = Allows Internal Router to connect to an ABR Router, thus allowing it connectivity to the rest of the OSPF Network
Now a review of Stub, Total Stub, and Not So Stubby Areas
Borrowing this from a Cisco document located here for a total (and much better) explanation of all things OSPF Stub Areas:
- Stub = “area # stub” = Does not allow External (Redistributed) LSA’s / Routes
- Total Stub = “area # stub no-summary” = Only type of Summary LSA / Route allowed is a “default information originate” summary route
- NSSA = “area # nssa” = No External Routes allowed in this Area, are replaced by Type 7 LSA’s, so External Routes show as N1 and N2 routes instead of E1 and E2 in any Non-NSSA Area’s route table
- NSSA Totally Stubby = “area # nssa no-summary” = Only default summary route “default information” and Type 7 LSA’s advertised to this router type
Whew.
Now when it comes DR / BDR Election, the one time it is critical to make sure the DR is elected correctly, is when the Topology is not full mesh / neighbors cannot all reach each other like an NBMA / DMBPN / Frame Relay network. The Hub needs to be the DR, and the spokes need to NOT have a chance to become the DR, so the interfaces should be “ip ospf pri 0” and a secondary Hub should be connected if redundancy is needed.
The whole point of Hub and Spoke networks is that communication MUST go through the Hub, so there is no reason for a Spoke to even be considered to become a DR.
Redistributed and Summary Routes in OSPF have an AD of 110, EIGRP Summary Routes change AD to 5, but OSPF always has the AD of 110 unless changes with “distance …” command in router ospf configuration.
Summarization – Area Range vs Summary Address in OSPF
The difference between configuring “area range x.x.x.x x.x.x.x” and “summary-address x.x.x.x x.x.x.x” is really where they are used.
Summary Address is for advertising External Routes (Type 5 and Type 7 LSA’s), so it should be configured on ASBR’s, whereas Area Range is for summarizing routes Internal to OSPF to create LSA Type 3’s so they are configured on OSPF ABRs.
OSPF Virtual Links
A quick summary is creating a link between two Autonomous Systems where its generally an Internal Router with no interface in the Backbone Area 0, so it connects to an ABR that has an interface in the Backbone Area.
“area 1 virtual-link 1.1.1.1” in the OSPF router process on one side, and “area 1 virtual-link 3.3.3.3” on the other side, assuming Area 1 is the “transit area” between the two interfaces and both routers have a Loopback Interface with 1.1.1.1 on one side and 3.3.3.3 on the other side to create the Virtual-Link.
OSPF Default Route
This is configured with “default-information originate” to advertise to a default route back to the router configured on, this is not a default static route that is Redistributed into OSPF.
OSPFv3 as seen with EIGRP for IPv6, works exactly the same as IPv4
The only differences is your verification commands, otherwise it is all exactly the same as OSPFv2, some of these verification commands being:
- “sh ipv6 proto”
- “sh ipv6 ospf”
- “sh ipv6 ospf int”
- “sh ipv6 ospf int (int)”
- “sh ipv6 ospf database”
- “sh ipv6 ospf nei”
A few other unique things to IPv6:
- Multicast Address is FF02::5
- LSA Type 8 = Link-Local information, not flooded by other routers
- LSA Type 9 = Intra-Area Prefix Advertisements, similar to Type 3 LSA
Address-Families with OSPFv3
This is the same concept as EIGRP Address Families where it is a centralized place to configure both IPv4 and IPv6 OSPF Processes / Areas, so I won’t go deep into details as this article is hitting a pretty good length, but an interesting thing to me is that where EIGRP replace “ip” and “ipv6” in show commands, OSPF just uses “ospfv3” in its verification commands instead of just “ospf” even though it is for both IPv4 and IPv6:
- “sh run | sec router ospfv3”
- “sh ospfv3”
- “sh ospfv3 nei”
- “sh ospfv3 database”
- “sh ip proto” and “sh ipv6 proto” are still separate with OSPF
- “sh ospfv3 int (brief)”
- “sh ospv3 int (int)”
And that will do it for OSPF TSHOOT notes before exam day
I feel like I just crammed my entire OSPF studies into a single day, my brain is officially dead, so I am going to end this here!
This is kind of a mix of everything not in great detail, I really highly suggest the TSHOOT OCG Book for Trouble Ticket sections that tie all this information together, that I just do not have the will to type up after going through allllllll that material 🙂
Until next time!
The DEVNET GRIND! 