Category Archives: CCNP – Authentication

Part 2: Deep Dive into EIGRP / OSPF / RIP Authentication, Route Summarization, OSPF troubleshooting to end lab!

labbers_delight_rev1

Tonight I had a whole agenda of tasks to configure as much as I could on this network, however I was only able to demonstrate authentication configuration for all 3 protocols thoroughly, create a Summary Route for R5, and troubleshoot an OSPF issue that was a bit tricky to catch but finally got it (which I added as its own section to demonstrate how I troubleshoot the issue).

And with that lets get rocking on some configurations, I will separate each topic before configuration with a header, and note that the topology now reflect loopbacks and may be revised as the lab moves forward to reflect changes made / configs added.

Prepare yourself for a lot of router output, with explanations tucked between the output, so without further ado:

 

Summary Route Configuration on R5

 

R5#conf t
Enter configuration commands, one per line.  End with CNTL/Z.
R5(config)#
R5(config)#interface Loopback101
R5(config-if)# ip address 100.1.0.1 255.255.0.0
R5(config-if)#!
R5(config-if)#interface Loopback102
R5(config-if)# ip address 100.2.0.1 255.255.0.0
R5(config-if)#!
R5(config-if)#interface Loopback103
R5(config-if)# ip address 100.3.0.1 255.255.0.0
R5(config-if)#!
R5(config-if)#interface Loopback104
R5(config-if)# ip address 100.4.0.1 255.255.0.0
R5(config-if)#!
R5(config-if)#interface Loopback105
R5(config-if)# ip address 100.5.0.1 255.255.0.0
R5(config-if)#!
R5(config-if)#interface Loopback106
R5(config-if)# ip address 100.6.0.1 255.255.0.0
R5(config-if)#!
R5(config-if)#interface Loopback107
R5(config-if)# ip address 100.7.0.1 255.255.0.0
R5(config-if)#!
R5(config-if)#
*Feb  2 05:20:35.351: %LINEPROTO-5-UPDOWN: Line protocol on Interface Loopback1, changed state to up
*Feb  2 05:20:35.427: %LINEPROTO-5-UPDOWN: Line protocol on Interface Loopback101, changed state to up
*Feb  2 05:20:35.563: %LINEPROTO-5-UPDOWN: Line protocol on Interface Loopback102, changed state to up
*Feb  2 05:20:35.635: %LINEPROTO-5-UPDOWN: Line protocol on Interface Loopback103, changed state to up
*Feb  2 05:20:35.711: %LINEPROTO-5-UPDOWN: Line protocol on Interface Loopback104, changed state to up
*Feb  2 05:20:35.771: %LINEPROTO-5-UPDOWN: Line protocol on Interface Loopback105, changed state to up
R5(config-if)#
*Feb  2 05:20:35.843: %LINEPROTO-5-UPDOWN: Line protocol on Interface Loopback106, changed state to up
*Feb  2 05:20:35.911: %LINEPROTO-5-UPDOWN: Line protocol on Interface Loopback107, changed state to up

As you can tell by the !’s I’ve pulled these summary addresses from prior configs and put them in a notepad file, along with the “network …” commands to make quick summary routes, work smart not hard 🙂

R5(config)#router eigrp 100
R5(config-router)#network 100.1.0.0 0.0.255.255
R5(config-router)#network 100.2.0.0 0.0.255.255
R5(config-router)#network 100.3.0.0 0.0.255.255
R5(config-router)#network 100.4.0.0 0.0.255.255
R5(config-router)#network 100.5.0.0 0.0.255.255
R5(config-router)#network 100.6.0.0 0.0.255.255
R5(config-router)#network 100.7.0.0 0.0.255.255
R5(config-router)#int fa0/1
R5(config-if)#ip summary-address eigrp 100 ?
  A.B.C.D  IP address

R5(config-if)#ip summary-address eigrp 100 100.0.0.0 ?
  A.B.C.D  IP network mask

R5(config-if)#ip summary-address eigrp 100 100.0.0.0 255.248.0.0
R5(config-if)#
*Feb  2 05:47:05.207: %DUAL-5-NBRCHANGE: EIGRP-IPv4 100: Neighbor 172.12.15.1 (FastEthernet0/1) is resync: summary configured
R5(config-if)#

Neighbor change message, do I detect a disturbance in the force?


R5(config-if)#do sh ip eigrp nei
EIGRP-IPv4 Neighbors for AS(100)
H   Address                 Interface       Hold Uptime   SRTT   RTO  Q  Seq
(sec)         (ms)       Cnt Num
0   172.12.15.1             Fa0/1             10 01:01:03    1   200  0  14
R5(config-if)#

Nope, we are good to go, and to confirm lets check out R1’s EIGRP routes:

R1#sh ip route eigrp
     100.0.0.0/13 is subnetted, 1 subnets
D       100.0.0.0 [90/156160] via 172.12.15.5, 00:03:06, FastEthernet0/1
     5.0.0.0/32 is subnetted, 1 subnets
D       5.5.5.5 [90/156160] via 172.12.15.5, 01:02:36, FastEthernet0/1
R1#

Rock and Roll, Summary Route accomplished, onto setting authentication in each domain!

I will have to refer to my notes for this unfortunately at least for EIGRP, I will need to start configuring authentication even if not needed in more labs to get this down.

 

EIGRP Authentication for AS 100 and AS 200

 

EIGRP is set with a Key Chain config, then set on the interface with two commands as shown below.

R5(config)#key chain CCNP
R5(config-keychain)#key ?
  <0-2147483647>  Key identifier

R5(config-keychain)#key 1
R5(config-keychain-key)#?
Key-chain key configuration commands:
  accept-lifetime  Set accept lifetime of key
  default          Set a command to its defaults
  exit             Exit from key-chain key configuration mode
  key-string       Set key string
  no               Negate a command or set its defaults
  send-lifetime    Set send lifetime of key

R5(config-keychain-key)#key-string CISCO
R5(config-keychain-key)#int fa0/1
R5(config-if)#ip eigrp ?
% Unrecognized command
R5(config-if)#ip auth
R5(config-if)#ip authentication ?
  key-chain  key-chain
  mode       mode

R5(config-if)#ip authentication mode ?
  eigrp  Enhanced Interior Gateway Routing Protocol (EIGRP)

R5(config-if)#ip authentication mode eigrp ?
  <1-65535>  AS number

R5(config-if)#ip authentication mode eigrp 100 ?
  md5  Keyed message digest

R5(config-if)#ip authentication mode eigrp 100 md5 ?
  <cr>

R5(config-if)#ip authentication mode eigrp 100 md5
R5(config-if)#
*Feb  2 06:07:11.879: %DUAL-5-NBRCHANGE: EIGRP-IPv4 100: Neighbor 172.12.15.1 (FastEthernet0/1) is down: authentication mode changed
R5(config-if)#ip authentic
R5(config-if)#ip authentication key-chain ?
  eigrp  Enhanced Interior Gateway Routing Protocol (EIGRP)

R5(config-if)#ip authentication key-chain eigrp 100 CCNP
R5(config-if)#

Couple of things to note quick:

  • In global config the command syntax is “key chain” while on interface it’s “key-chain” in the commands
  • On the interface configs, the command defining the key-chain can be set without impacting the adjacency, however once the command configuring authentication itself on the interface is entered it will drop the adj if not config’d on the other side as can be seen in this snippet of output:

R5(config-if)#no ip authentication mode eigrp 100 md5
*Feb  2 06:11:56.323: %DUAL-5-NBRCHANGE: EIGRP-IPv4 100: Neighbor 172.12.15.1 (FastEthernet0/1) is up: new adjacency
R5(config-if)#no ip authentication key-chain eigrp 100 CCNP
R5(config-if)#ip authentication key-chain eigrp 100 CCNP
R5(config-if)#ip authentication mode eigrp 100 md5
R5(config-if)#
*Feb  2 06:12:36.791: %DUAL-5-NBRCHANGE: EIGRP-IPv4 100: Neighbor 172.12.15.1 (FastEthernet0/1) is down: authentication mode changed
R5(config-if)#

  • Also to note is that you use the key chain name, not the key name on interface config

And over on R1:

R1(config)#router eigrp 100
R1(config-router)#exit
R1(config)#key chain CCNP
R1(config-keychain)#key 1
R1(config-keychain-key)#key-string ?
0     Specifies an UNENCRYPTED password will follow
7     Specifies a HIDDEN password will follow
LINE  The UNENCRYPTED (cleartext) user password

R1(config-keychain-key)#key-string CISCO
R1(config-keychain-key)#int fa0/1
R1(config-if)#ip authentication mode eigrp 100 md5
R1(config-if)#ip authentication key-chain eigrp 100 CCNP
R1(config-if)#
*Mar  1 18:33:43.996: %DUAL-5-NBRCHANGE: IP-EIGRP(0) 100: Neighbor 172.12.15.5 (FastEthernet0/1) is up: new adjacency
R1(config-if)#

That was a struggle once again to remember those commands, I deleted a lot of ? output from R1, but got it Authenticated, now to slap that on R3 to R4 hopefully without the need of ?’s at all here:

R3(config)#key chain CCNP
R3(config-keychain)#key 1
R3(config-keychain-key)#key-string CISCO
R3(config-keychain-key)#exit
R3(config-keychain)#int fa0/1
R3(config-if)#ip authen
R3(config-if)#ip authentication mode eigrp 200 md5
*Mar  2 01:38:07.304: %DUAL-5-NBRCHANGE: IP-EIGRP(0) 200: Neighbor 172.12.34.4 (FastEthernet0/1) is down: authentication mode changed

R3(config-if)#ip authentication key-chain CCNP
                                          ^
% Invalid input detected at ‘^’ marker.

R3(config-if)#ip authentication key-chain ?
  eigrp  Enhanced Interior Gateway Routing Protocol (EIGRP)

R3(config-if)#ip authentication key-chain eigrp 200 CCNP
R3(config-if)#

Almost had it! Now over to R4 to complete without any errors or ?’s:

R4(config)#key chain CCNP
R4(config-keychain)#key 1
R4(config-keychain-key)#key-string CISCO
R4(config-keychain-key)#int fa0/1
R4(config-if)#ip authentication mode eigrp 200 md5
R4(config-if)#ip authentication key-string eigrp 200 CCNP
                                    ^
% Invalid input detected at ‘^’ marker.

R4(config-if)#ip authentication key-chain eigrp 200 CCNP
R4(config-if)#
*Feb  2 05:32:37.855: %DUAL-5-NBRCHANGE: EIGRP-IPv4 200: Neighbor 172.12.34.3 (FastEthernet0/1) is up: new adjacency
R4(config-if)#

As can be seen, really unneeded error to receive, I think my brain is already hitting exhaustion as I started later in the night today (almost midnight), so time to boogie.

EIGRP Autonomous System Authentication Completed! Onto OSPF!

 

OSPF Authentication for Area 0

 

So here is a big mess of output to explain:

R1(config-router)#area 0 ?
  authentication  Enable authentication
  default-cost    Set the summary default-cost of a NSSA/stub area
  filter-list     Filter networks between OSPF areas
  nssa            Specify a NSSA area
  range           Summarize routes matching address/mask (border routers only)
  sham-link       Define a sham link and its parameters
  stub            Specify a stub area
  virtual-link    Define a virtual link and its parameters

R1(config-router)#area 0 authentication ?
  message-digest  Use message-digest authentication
  <cr>

R1(config-router)#area 0 authentication messa
R1(config-router)#area 0 authentication message-digest ?

  <cr>

R1(config-router)#area 0 authentication message-digest
R1(config-router)#int s0/0
R1(config-if)#ip ospf authent
R1(config-if)#ip ospf authentication ?
  message-digest  Use message-digest authentication
  null            Use no authentication
  <cr>

R1(config-if)#ip ospf authentication message-dig
R1(config-if)#ip ospf authentication message-digest ?

  <cr>

R1(config-if)#ip ospf ?
  <1-65535>            Process ID
  authentication       Enable authentication
  authentication-key   Authentication password (key)
  bfd                  Enable BFD on this interface
  cost                 Interface cost
  database-filter      Filter OSPF LSA during synchronization and flooding
  dead-interval        Interval after which a neighbor is declared dead
  demand-circuit       OSPF demand circuit
  flood-reduction      OSPF Flood Reduction
  hello-interval       Time between HELLO packets
  lls                  Link-local Signaling (LLS) support
  message-digest-key   Message digest authentication password (key)
  mtu-ignore           Ignores the MTU in DBD packets
  network              Network type
  priority             Router priority
  resync-timeout       Interval after which adjacency is reset if oob-resync is
                       not started
  retransmit-interval  Time between retransmitting lost link state
                       advertisements
  transmit-delay       Link state transmit delay

R1(config-if)#ip ospf
*Mar  1 18:57:13.326: %OSPF-5-ADJCHG: Process 1, Nbr 22.22.22.2 on Serial0/0 from FULL to DOWN, Neighbor Down: Dead timer expired
R1(config-if)#ip ospf
*Mar  1 18:57:18.695: %OSPF-5-ADJCHG: Process 1, Nbr 33.33.33.3 on Serial0/0 from FULL to DOWN, Neighbor Down: Dead timer expired
R1(config-if)#ip ospf messag
R1(config-if)#ip ospf message-digest-key ?
  <1-255>  Key ID

R1(config-if)#ip ospf message-digest-key 1 ?
  md5  Use MD5 algorithm

R1(config-if)#ip ospf message-digest-key 1 md5 ?
  <0-7>  Encryption type (0 for not yet encrypted, 7 for proprietary)
  LINE   The OSPF password (key) (maximum 16 characters)

R1(config-if)#ip ospf message-digest-key 1 md5 CCNP ?
LINE    <cr>

R1(config-if)#ip ospf message-digest-key 1 md5 CCNP
R1(config-if)#

Do the commands highlighted in red on the interface and in router configuration to “set” authentication look familiar? That’s because they are, it can either be “set” in router config or on the interface, however the actual Key name must be on the interface as can eventually be seen there.

I’m going to set R2 and R3 with the “set” portion on the serial interface instead of in router config and see if the adjacency comes back up:

R2#conf t
Enter configuration commands, one per line.  End with CNTL/Z.
R2(config)#int s0/0
R2(config-if)#ip ospf authentican ?
% Unrecognized command
R2(config-if)#ip ospf au
R2(config-if)#ip ospf authentication ?
  message-digest  Use message-digest authentication
  null            Use no authentication
  <cr>

R2(config-if)#ip ospf authentication mess
R2(config-if)#ip ospf authentication message-digest ?
  <cr>

R2(config-if)#ip ospf authentication message-digest
R2(config-if)#ip ospf mess
R2(config-if)#ip ospf message-digest-key ?
  <1-255>  Key ID

R2(config-if)#ip ospf message-digest-key 1 ?
  md5  Use MD5 algorithm

R2(config-if)#ip ospf message-digest-key 1 md5 ?
  <0-7>  Encryption type (0 for not yet encrypted, 7 for proprietary)
  LINE   The OSPF password (key) (maximum 16 characters)

R2(config-if)#ip ospf message-digest-key 1 md5 CCNP
R2(config-if)#
*Mar  1 17:22:31.104: %OSPF-5-ADJCHG: Process 1, Nbr 11.11.11.1 on Serial0/0 from LOADING to FULL, Loading Done

R2(config-if)#

  • So no matter where Authentication is set, as long as it’s on the proper interface it will work with another neighbor that has it configured in the router config

Will slap an all interface configuration on R3, and we are done with that unless there are any issues:

R3(config-if)#int s0/2
R3(config-if)#ip ospf authentication message-digest
R3(config-if)#ip ospf mess
R3(config-if)#ip ospf message-digest-key 1 md5 CCNP
R3(config-if)#
*Mar  2 01:59:41.266: %OSPF-5-ADJCHG: Process 1, Nbr 11.11.11.1 on Serial0/2 from LOADING to FULL, Loading Done
R3(config-if)#

Done deal. Now onto RIP Authentication, though it’s been so long I don’t even remember if it has an authentication to set.

 

RIP Authentication… maybe?

 

A quick google search and good old Cisco documentation shows RIP authentication to be the same as EIGRP, with a key chain, and directly on the interface.

So for this I already have a key chain CCNP for EIGRP on R3, so I am going to try to use that same key chain and just apply it on the interface, though I will of course have to configure the key chain on R2 so I will actually start there:

R2(config)#router rip
R2(config-router)#exit
R2(config)#key chain CCNP
R2(config-keychain)#key 1
R2(config-keychain-key)#key-string CISCO
R2(config-keychain-key)#int fa0/1
R2(config-if)#ip rip ?
  advertise       Specify update interval
  authentication  Authentication control
  receive         advertisement reception
  send            advertisement transmission
  v2-broadcast    send ip broadcast v2 update

R2(config-if)#ip rip authe
R2(config-if)#ip rip authentication ?
  key-chain  Authentication key-chain
  mode       Authentication mode

R2(config-if)#ip rip authentication mode ?
  md5   Keyed message digest
  text  Clear text authentication

R2(config-if)#ip rip authentication mode md5 ?
  <cr>

R2(config-if)#ip rip authentication mode md5 <- Look familiar?

R2(config-if)#ip rip authentication key-chain ?
  LINE  name of key-chain

R2(config-if)#ip rip authentication key-chain CCNP ?
LINE    <cr>

R2(config-if)#ip rip authentication key-chain CCNP <- Slight difference from EIGRP

R2(config-if)#

Just a very slight difference in the syntax in that the command starts off with the protocol name instead of almost ending with it, so lets see if we can re-use R3’s key chain – But one thing to note:

R3(config)#do sh ip route rip
     22.0.0.0/24 is subnetted, 1 subnets
R       22.22.22.0 [120/1] via 172.12.23.2, 00:00:11, FastEthernet0/0
R3(config)#do sh ip route rip
     22.0.0.0/24 is subnetted, 1 subnets
R       22.22.22.0 [120/1] via 172.12.23.2, 00:00:15, FastEthernet0/0
R3(config)#do sh ip route rip
     22.0.0.0/24 is subnetted, 1 subnets
R       22.22.22.0 [120/1] via 172.12.23.2, 00:00:24, FastEthernet0/0
R3(config)#do sh ip route rip
     22.0.0.0/24 is subnetted, 1 subnets
R       22.22.22.0 [120/1] via 172.12.23.2, 00:00:26, FastEthernet0/0
R3(config)#do sh ip route rip
     22.0.0.0/24 is subnetted, 1 subnets
R       22.22.22.0 [120/1] via 172.12.23.2, 00:00:01, FastEthernet0/0
R3(config)#

As can be seen there is no adjacency or neighbor relationship to lose, it only cares about Hello’s and hop counts, so I am wondering what exactly the authentication is even for. However I am too tired to care to be honest, I just wanted to point out the remote router with no interface config yet for Authentication still is getting RIP route updates.

Now to config int Fa0/1 on R3 for RIP Authentication with a shared EIGRP authentication key:

R3(config-if)#no ip rip authentication key-chain CCNP
R3(config-if)#do ping 2.2.2.2 (<- Ooops, in OSPF domain)

Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 2.2.2.2, timeout is 2 seconds:
….. (<- … What??)
Success rate is 0 percent (0/5)
R3(config-if)#do sh ip route rip

     22.0.0.0/24 is subnetted, 1 subnets
R       22.22.22.0 [120/1] via 172.12.23.2, 00:00:01, FastEthernet0/0

R3(config-if)#do ping 22.22.22.2

Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 22.22.22.2, timeout is 2 seconds:
!!!!!
Success rate is 100 percent (5/5), round-trip min/avg/max = 1/2/4 ms
R3(config-if)#

So as can be seen I found out 2 things:

  • RIPv2 Authentication appears to be worthless, as I can still ping R3’s loopback IP 22.22.22.2 from R3 and continue to get Hellos and Routing updates
  • That for some reason when I accidentally pinged 2.2.2.2 thinking it was in the RIP domain, I got no response, so what is going on there

So I am calling this the end of Authentication for all domains are now authenticating (I put the config back on R3’s RIP interface just because I guess, but now I have an OSPF route propagation issue to look at, and I just love these end of lab issues when I am fried 🙂

 

Troubleshooting what happened to R2 broadcasting 2.2.2.2 via OSPF

 

As mentioned I did a quick verification and R2 has no route to it, so I went over to R2 to confirm it has the network in it’s OSPF config:

R2(config-if)#
R2(config-if)#do sho ip proto
Routing Protocol is “ospf 1”
  Outgoing update filter list for all interfaces is not set
  Incoming update filter list for all interfaces is not set
  Router ID 22.22.22.2
  Number of areas in this router is 1. 1 normal 0 stub 0 nssa
  Maximum path: 4
  Routing for Networks:
    2.2.2.2 0.0.0.0 area 0
    172.12.123.0 0.0.0.255 area 0 (<- Confirmation of correct NBMA network)
 Reference bandwidth unit is 100 mbps
  Routing Information Sources:
    Gateway         Distance      Last Update
    11.11.11.1           110      00:49:52
    33.33.33.3           110      00:49:52
  Distance: (default is 110)
R2(config-if)#do sh ip ospf nei

Neighbor ID     Pri   State           Dead Time   Address         Interface
11.11.11.1        1   FULL/DR         00:01:53    172.12.123.1    Serial0/0

R2(config-if)#

Being that both of those two highlighted parts are in the config / output, I am betting the house the lo2 is network 2.2.2.0 /24 :

R2(config-if)#do sh int lo2
Loopback2 is up, line protocol is up
  Hardware is Loopback
  Internet address is 2.2.2.2/32

Well what the fudge, this is the end of lab / night stuff that just kills my brain, cause I cannot stop until I find out what the deal is. So I look at R2’s route table, and look at it in all it’s majesty:

R2(config-if)#do sh ip route

Gateway of last resort is not set

     2.0.0.0/32 is subnetted, 1 subnets
C       2.2.2.2 is directly connected, Loopback2
     33.0.0.0/24 is subnetted, 1 subnets
R       33.33.33.0 [120/1] via 172.12.23.3, 00:00:03, FastEthernet0/0
     172.12.0.0/24 is subnetted, 3 subnets
R       172.12.34.0 [120/1] via 172.12.23.3, 00:00:03, FastEthernet0/0
C       172.12.23.0 is directly connected, FastEthernet0/0
C       172.12.123.0 is directly connected, Serial0/0
     22.0.0.0/24 is subnetted, 1 subnets
C       22.22.22.0 is directly connected, Loopback22
R2(config-if)#

Not a single OSPF route, even though it is neighbors with the hub, and the hub is getting the loopback route from R3 via OSPF:

R1(config-if)#do show ip route ospf
     3.0.0.0/32 is subnetted, 1 subnets
O       3.3.3.3 [110/65] via 172.12.123.3, 00:58:54, Serial0/0
R1(config-if)#

And in R1’s running config the neighbor statement is verified there:

router ospf 1
 log-adjacency-changes
 area 0 authentication message-digest
 network 1.1.1.1 0.0.0.0 area 0
 network 172.12.123.0 0.0.0.255 area 0
 neighbor 172.12.123.2
 neighbor 172.12.123.3

So all that’s left in my mind is “debug ip ospf pack” and hope to catch it in there, lets see if any output smokes out this weasel so I can call it a night with a clear conscience:

R1#debug ip ospf pack
OSPF packet debugging is on
R1#
*Mar  1 20:17:13.495: OSPF: rcv. v:2 t:1 l:48 rid:22.22.22.2
      aid:0.0.0.0 chk:0 aut:2 keyid:1 seq:0x3C7FC8A0 from Serial0/0
*Mar  1 20:17:13.515: OSPF: rcv. v:2 t:1 l:48 rid:22.22.22.2
      aid:0.0.0.0 chk:0 aut:2 keyid:1 seq:0x3C7FC8A0 from Serial0/0
R1#
*Mar  1 20:17:18.839: OSPF: rcv. v:2 t:1 l:48 rid:33.33.33.3
      aid:0.0.0.0 chk:0 aut:2 keyid:1 seq:0x3C8040C7 from Serial0/0
R1#
*Mar  1 20:17:33.856: OSPF: rcv. v:2 t:4 l:76 rid:33.33.33.3
      aid:0.0.0.0 chk:0 aut:2 keyid:1 seq:0x3C8040D6 from Serial0/0
*Mar  1 20:17:33.888: OSPF: rcv. v:2 t:4 l:60 rid:33.33.33.3
      aid:0.0.0.0 chk:0 aut:2 keyid:1 seq:0x3C8040D6 from Serial0/0
R1#
*Mar  1 20:17:41.288: OSPF: rcv. v:2 t:5 l:64 rid:22.22.22.2
      aid:0.0.0.0 chk:0 aut:2 keyid:1 seq:0x3C7FC8BB from Serial0/0
R1#
*Mar  1 20:17:43.496: OSPF: rcv. v:2 t:1 l:48 rid:22.22.22.2
      aid:0.0.0.0 chk:0 aut:2 keyid:1 seq:0x3C7FC8BE from Serial0/0
*Mar  1 20:17:43.516: OSPF: rcv. v:2 t:1 l:48 rid:22.22.22.2
      aid:0.0.0.0 chk:0 aut:2 keyid:1 seq:0x3C7FC8BE from Serial0/0
R1#
*Mar  1 20:17:48.841: OSPF: rcv. v:2 t:1 l:48 rid:33.33.33.3
      aid:0.0.0.0 chk:0 aut:2 keyid:1 seq:0x3C8040E4 from Serial0/0

Absolutely nothing, so I am going to hammer out a clear ip ospf proc on R1 and R2 to see what happens, and sure enough it did nothing, but in looks at the peers after doing it I found a hintiditty hint hint:

R1#show ip ospf nei

Neighbor ID     Pri   State           Dead Time   Address         Interface
22.22.22.2        0   FULL/DROTHER    00:01:54    172.12.123.2    Serial0/0
33.33.33.3        1   FULL/DR         00:01:29    172.12.123.3    Serial0/0 (<- No Bueno)

So I actually made sure R2 had “ip ospf pri 0” on it’s interface before the clear ip ospf proc, but this proves that there was an issue there, as R3 should never become DR if it has that configured. So I am not sure if I was just dead tired (like now) and forgot to add it, it didn’t save, or what happened so I made sure both spokes are now rocking the config on their serial interface and “clear ip ospf proc”s all around.

Now lets see if that finally resolved this issue, please oh please let this be done:

R1#show ip route ospf
     2.0.0.0/32 is subnetted, 1 subnets
O       2.2.2.2 [110/65] via 172.12.123.2, 00:01:35, Serial0/0
     3.0.0.0/32 is subnetted, 1 subnets
O       3.3.3.3 [110/65] via 172.12.123.3, 00:01:35, Serial0/0
R1#

Yes!!!!!! I won even in my deliriously tired state!!! This is how troubleshooting is done my friends, keep looking at it from the different angles, and if you know what to look for you will eventually find it 🙂

Okay, it is 1:30am, I am going to call it for tonight and wr mem across the board on the routers.