VPN: DEEP Dive into IPSec, configurations / functions, the VPN fails, but is troubleshot with debugs / verification commands to fix the issue explained!

This is more a Part 2 of 3 in the series of 1 being building a GRE tunnel which we now have, 2 building an IPSec Tunnel which we will have shortly, and 3 placing the GRE traffic into the IPSec VPN for transmission – As IPSec only sends Unicast but GRE takes any type of traffic and encapsulates it into a Unicast packet.

As this is an incredibly long lab given the robust amount of configuration, and the troubleshooting at the end, I want to post up here at the top all of the configurations used in the lab so you don’t need to read the whole post to get the base configs / troubleshooting / verification however I highly recommend you read the entire page:

 

• access-list 123 permit ip 172.12.34.0 0.0.0.255 172.12.23.0 0.0.0.255
• crypto isakmp policy 10
• – encr 3des
• – hash md5
• – authentication pre-share
• crypto isakmp key 0 CCNP address 172.12.123.3 (use # 6 for encrypted key)
• crypto ipsec security-association lifetime seconds 86400
• crypto ipsec transform-set VPNLAB ah-md5-hmac
• – mode tunnel
• crypto map CCNP 100 ipsec-isakmp
• – set peer 172.12.123.2
• – set transform-set VPNLAB
• – match address 123
• int s0/0
• – crypto map VPNLAB
• TROUBLESHOOTING AND VERIFICATION COMMANDS BELOW
• debug crypto ipsec
• show crypto isakmp sa
• show crypto ipsec sa

I color coded things that are configured together as they drop you into different configuration modes, and put the few troubleshooting and verification commands at the bottom, however I again really encourage to read through the lab as it contains a lot of important information for exam day.

Or in the event you are like me and have already gone through the lab, and just want a quick reference to all commands used handy at the top of the page 🙂

 

 

First, we enable and create an ISAKMP / IKE policy for the VPN configurations:

 

Enabling ISAKMP for the IKE policy:

R2#conf t
Enter configuration commands, one per line.  End with CNTL/Z.
R2(config)#crypto isakmp enable
R2(config)#

Check! Verifying there IS a default policy on the router:

R2#sh cry isa policy

Global IKE policy
Default protection suite

        encryption algorithm:   DES – Data Encryption Standard (56 bit keys).
        hash algorithm:         Secure Hash Standard
        authentication method:  Rivest-Shamir-Adleman Signature
        Diffie-Hellman group:   #1 (768 bit)
        lifetime:               86400 seconds, no volume limit
R2#

Check! Note the default policy is called the “Default Protection Suite” and that the command “sh crypto isakmp policy” can be used almost as a cheat sheet of the what needs to be defined when making a new policy! Now to configure our own policy:

R2#conf t
Enter configuration commands, one per line.  End with CNTL/Z.
R2(config)#crypto isakmp policy ?
  <1-10000>  Priority of protection suite

R2(config)#crypto isakmp policy 10
R2(config-isakmp)#

Quick note – The lower the policy #, the higher priority it has in being used for IKE, moving on:

R2(config)#crypto isakmp policy 10
R2(config-isakmp)#authentication pre-share
R2(config-isakmp)#encryption ?
  3des  Three key triple DES
  aes   AES – Advanced Encryption Standard.
  des   DES – Data Encryption Standard (56 bit keys).

R2(config-isakmp)#encryption 3des ?
  <cr>

R2(config-isakmp)#encryption 3des
R2(config-isakmp)#hash ?
  md5  Message Digest 5
  sha  Secure Hash Standard

R2(config-isakmp)#hash md5
R2(config-isakmp)#lifetime ?
  <60-86400>  lifetime in seconds

R2(config-isakmp)#lifetime 86400

I gave some ?’s just to see modifiers, note you get a <cr> after you choose an encryption type, so you can’t throw them all into one policy like you can in transform sets (covered shortly), and that lifetime is in seconds but it is always good to check with IOS help.

We now have 2 IKE policies:

R2#sh cry isa pol

Global IKE policy
Protection suite of priority 10

        encryption algorithm:   Three key triple DES

        hash algorithm:         Message Digest 5

        authentication method:  Pre-Shared Key

        Diffie-Hellman group:   #1 (768 bit)

        lifetime:               86400 seconds, no volume limit

Default protection suite
        encryption algorithm:   DES – Data Encryption Standard (56 bit keys).
        hash algorithm:         Secure Hash Standard
        authentication method:  Rivest-Shamir-Adleman Signature
        Diffie-Hellman group:   #1 (768 bit)
        lifetime:               86400 seconds, no volume limit
R2#

These are the parameters that are sent to the remote peer as part of Phase 1 to match IKE criteria, and needless to say if Phase 1 goes bad, we will not make it to Phase 2.

One very important thing to note, when these policies are exchanged, the only thing that DOES NOT NEED TO MATCH IS LIFETIME!

If the recipients Lifetime is equal to or lower than the initiators, the lower value is used on the initiator.

 

Now to set the field of the IKE’s ISAKMP Policy:

 

First up is setting the pre-shared key, notice as you go through it my IOS is buggy and doesn’t recognize me setting it to encrypt the key, so I have to go clear text with it:

R2#conf t
Enter configuration commands, one per line.  End with CNTL/Z.
R2(config)#crypto isakmp key 6 ?

% Unrecognized command

R2(config)#crypto isakmp key ?

  0  Specifies an UNENCRYPTED password will follow

  6  Specifies an ENCRYPTED password will follow

R2(config)#crypto isakmp key 0 ?
  WORD  The UNENCRYPTED (cleartext) user password

R2(config)#crypto isakmp key 0 CCNP ?
  address   define shared key with IP address
  hostname  define shared key with hostname

R2(config)#crypto isakmp key 0 CCNP address ?
  A.B.C.D  Peer IP address
  ipv6     define shared key with IPv6 address

R2(config)#crypto isakmp key 0 CCNP address 172.12.123.3 ?
  A.B.C.D   Peer IP subnet mask
  no-xauth  Bypasses XAuth for this peer
  <cr>

R2(config)#crypto isakmp key 0 CCNP address 172.12.123.3
R2(config)#

When configuring the Transform-Set, one important note is that the Policies must agree exactly on the encryption and algorithm being used (Top two lines in ISAKMP Policy) to create an IPSec SA (Security Association).

As I mentioned briefly earlier, multiple Transform-Set’s can be configured and sent, and the remote peer will compare each set against its own until it finds a match, once a match is found the IPSec SA is built.

Now to look at Transform-Set configuration:

R2(config)#crypto ipsec transform-set VPNLAB ?
  ah-md5-hmac   AH-HMAC-MD5 transform
  ah-sha-hmac   AH-HMAC-SHA transform
  comp-lzs      IP Compression using the LZS compression algorithm
  esp-3des      ESP transform using 3DES(EDE) cipher (168 bits)
  esp-aes       ESP transform using AES cipher
  esp-des       ESP transform using DES cipher (56 bits)
  esp-md5-hmac  ESP transform using HMAC-MD5 auth
  esp-null      ESP transform w/o cipher
  esp-seal      ESP transform using SEAL cipher (160 bits)
  esp-sha-hmac  ESP transform using HMAC-SHA auth

Look familiar? That is because it was covered in depth in our DEEP Dive of VPN Packet Types / Headers, which we reviewed the difference between them, however in the list itself it does show the word ‘cipher’ in the ESP commands indicating it will encrypt them.

For lab purposes, I am going to use an AH scheme, cause I am not overly worried about the data payload:

R2(config)#crypto ipsec transform-set VPNLAB ah-md5-hmac
R2(cfg-crypto-trans)#?
Crypto transform configuration commands:
  default  Set a command to its defaults
  exit     Exit from crypto transform configuration mode
  mode     encapsulation mode (transport/tunnel)
  no       Negate a command or set its defaults

R2(cfg-crypto-trans)#mode ?
  transport  transport (payload encapsulation) mode
  tunnel     tunnel (datagram encapsulation) mode

R2(cfg-crypto-trans)#mode tunnel ?
  <cr>

R2(cfg-crypto-trans)#mode tunnel

This is also in the VPN Packet / Header DEEP Dive, as it describes whether you are using Transport or Tunnel mode, which for this I will use Tunnel mode.

So with the “crypto ipsec transform set (name) …” command we are configuring exactly what we looked at in the VPN Packet inspection, it is all contained in the Transform-Set!

Also a side note, I could have added every transform-set type there is on the original command to configure it, this is very often seen in the real world and may end up on the exam as well, and its a perfectly legal command to use all transform-set modifiers.

Moving along, lets look at the SA lifetime configuration command:

R2(config)#crypto ipsec security-association ?
  idle-time  Automatically delete IPSec SAs after a given idle period.
  lifetime   security association lifetime

  replay     Set replay checking.

R2(config)#crypto ipsec security-association lifetime ?
  kilobytes  Volume-based key duration
  seconds    Time-based key duration

R2(config)#crypto ipsec security-association lifetime seconds ?
  <120-86400>  Security association duration in seconds

R2(config)#crypto ipsec security-association lifetime seconds 86400 ?
  <cr>

R2(config)#crypto ipsec security-association lifetime seconds 86400
R2(config)#

This is an optional, global configuration to say how long IPSec SA’s will stay up, either with the “… idle-time” set to delete the SA, or if lifetime is used to tear down the tunnel by either a matter of time, or traffic limit.

This command is not seen a lot in the real world as no company I’ve seen wants their VPN to drop in regular intervals (or ever) so this configuration is mainly exam oriented.

 

Next up is Crypto ACL’s (Interesting Traffic)

 

This is a type of ACL is created for outbound traffic flows, not to permit or deny traffic, but to define what traffic should be encrypted / go through the tunnel, and all other traffic will be permitted as normal with encryption.

On the inbound side, a permit statement on the ACL means that if traffic shows up with a source that should be encrypted but is not, it will drop the traffic. Again this will not deny all other types of traffic, but if the inbound Crypto ACL has that subnet on it, it better be encrypted, or it is getting dropped!

***One important concept with a Crypto ACL is that it won’t be applied on the “in” or “out” on an interface, the same ACL (one line usually) is read forwards (normally) for outbound traffic and backwards for inbound traffic.***

If that does not make sense right away, read it until it does. On the local router, outbound the Crypto ACL is read normally, and inbound traffic reads the same ACL backwards:

R2(config)#
R2(config)#access-list 123 permit ip 172.12.23.0 0.0.0.255 172.12.34.0 0.0.0.255
R2(config)#

BAM! Yes that was anti-climatic, but that is it, this defines traffic coming from the x.x.23.x network and destined to the x.x.34.x network as “Interesting Traffic” that should cause the VPN to build if not already up and encrypt the traffic.

Needless to say, we will need an inverse ACL on the remote peer 🙂

 

Now for the final configuration, the almighty Crypto Map!

 

This is where we call out the Crypto ACL, set the Peer IP Address, and Transform-Set to be used:

R2(config)#crypto map CCNP ?
  <1-65535>       Sequence to insert into crypto map entry
  client          Specify client configuration settings
  isakmp          Specify isakmp configuration settings
  isakmp-profile  Specify isakmp profile to use
  local-address   Interface to use for local address for this crypto map
  redundancy      High availability options for this map

R2(config)#crypto map CCNP 100 ?
  gdoi          GDOI
  ipsec-isakmp  IPSEC w/ISAKMP
  ipsec-manual  IPSEC w/manual keying
  <cr>

R2(config)#crypto map CCNP 100 ipsec-isakmp ?
  dynamic  Enable dynamic crypto map support
  profile  Enable crypto map as a crypto-profile
  <cr>

R2(config)#crypto map CCNP 100 ipsec-isakmp
% NOTE: This new crypto map will remain disabled until a peer
        and a valid access list have been configured.
R2(config-crypto-map)#

Notice it gave us a little warning on the end, that is just saying your not done yet. Also I highlighted a few things as there is a jumble of output there.

I gave it a sequence number after its unique map name, chose the ipsec-isakmp option, and that was it before I hit enter to drop me into crypto-map configuration as seen – This is where you will set your ACL / Peer IP / Transform-Set (we are getting close to done if you can believe that):

R2(config-crypto-map)#match address 123    <- Crypto ACL
R2(config-crypto-map)#set ?
identity              Identity restriction.
ip                    Interface Internet Protocol config commands
isakmp-profile        Specify isakmp Profile
nat                   Set NAT translation
peer                  Allowed Encryption/Decryption peer.

pfs                   Specify pfs settings
security-association  Security association parameters
transform-set         Specify list of transform sets in priority order

R2(config-crypto-map)#set peer ?
A.B.C.D  IP address of peer

WORD     Host name of the peer

R2(config-crypto-map)#set peer 172.12.123.3 ?
default  Reset to default peer in list in the event of a failure
<cr>

R2(config-crypto-map)#set peer 172.12.123.3

R2(config-crypto-map)#set ?
identity              Identity restriction.
ip                    Interface Internet Protocol config commands
isakmp-profile        Specify isakmp Profile
nat                   Set NAT translation
peer                  Allowed Encryption/Decryption peer.
pfs                   Specify pfs settings
security-association  Security association parameters
transform-set         Specify list of transform sets in priority order

R2(config-crypto-map)#set transform-set ?
WORD  Proposal tag

R2(config-crypto-map)#set transform-set VPNLAB ?
WORD  Proposal tag
<cr>

R2(config-crypto-map)#set transform-set VPNLAB

R2(config-crypto-map)#

Now the veeeeery last thing is to apply it to the outside interface of the router:

R2(config)#int s0/0
R2(config-if)#crypto map CCNP

R2(config-if)#
*Mar 30 23:57:59.226: %CRYPTO-6-ISAKMP_ON_OFF: ISAKMP is ON
R2(config-if)#

Like I said no “in” or “out” with this ACL, apply the Crypto Map by its name, and you should see that ISAKMP is ON message if we a rockin – Now I will configure this on the other side without displaying that huge mess of output again!

I have shut down int fa0/0 on R3 so it only knows the route to 172.12.23.0 via OSPF across the NBMA, I have also remove the static route from R2 from the previous lab that allowed traffic GRE traffic to take the tunnel.

I have set “debug ipsec crypto” on R2 and R3 just int case something starts happening immediately when configured to the outside Serial face, lets have a look:

R2#debug crypto ipsec
Crypto IPSEC debugging is on
R2#
ASR#3
[Resuming connection 3 to r3 … ]

R3#
R3#debug crypto ipsec
Crypto IPSEC debugging is on
R3#conf t
Enter configuration commands, one per line.  End with CNTL/Z.
R3(config)#int s0/2
R3(config-if)#crypto map VPNLAB
R3(config-if)#
*Mar 31 20:08:19.690: %CRYPTO-6-ISAKMP_ON_OFF: ISAKMP is ON
R3(config-if)#

Honestly I was hoping to see some spur of life there, but like I said previously the ACL will detect VPN traffic, and that will then build the VPN tunnel (so the first few pings might … on you):

Debug output from R3:

*Mar 31 20:15:22.692: IPSEC(sa_request): ,
  (key eng. msg.) OUTBOUND local= 172.12.123.3, remote= 172.12.123.2,
    local_proxy= 172.12.34.0/255.255.255.0/0/0 (type=4),
    remote_proxy= 172.12.23.0/255.255.255.0/0/0 (type=4),
    protocol= AH, transform= NONE  (Tunnel),
    lifedur= 86400s and 4608000kb,
    spi= 0x0(0), conn_id= 0, keysize= 0, flags= 0x0
*Mar 31 20:15:24.126: IPSEC(validate_proposal_request): proposal part #1
*Mar 31 20:15:24.130: Crypto mapdb : proxy_match
        src addr     : 172.12.34.0
        dst addr     : 172.12.23.0
        protocol     : 0
        src port     : 0
        dst port     : 0
R3#
*Mar 31 20:15:24.134: IPSEC(key_engine): got a queue event with 1 KMI message(s)
*Mar 31 20:15:24.134: Crypto mapdb : proxy_match
        src addr     : 172.12.34.0
        dst addr     : 172.12.23.0
        protocol     : 0
        src port     : 0
        dst port     : 0
*Mar 31 20:15:24.138: IPSEC(crypto_ipsec_sa_find_ident_head): reconnecting with the same proxies and peer 172.12.123.2
*Mar 31 20:15:24.138: IPSEC(policy_db_add_ident): src 172.12.34.0, dest 172.12.23.0, dest_port 0
*Mar 31 20:15:24.142: IPSEC(create_sa): sa created,
  (sa) sa_dest= 172.12.123.2, sa_proto= 51,
    sa_spi= 0x616C073C(1634469692),
    sa_trans= ah-md5-hmac , sa_conn_id= 2
*Mar 31 20:15:24.142: IPSEC(update_current_outbound_sa): updated peer 172.12.123.2 current outbound sa to SPI 616C073C
R3#

 

The debug output on R2’s side is much more brief:
*Mar 30 23:21:33.479: IPSEC(key_engine): got a queue event with 1 KMI message(s)
*Mar 30 23:21:33.703: IPSEC(validate_proposal_request): proposal part #1
*Mar 30 23:21:33.703: IPSEC(validate_proposal_request): proposal part #1,
  (key eng. msg.) INBOUND local= 172.12.123.2, remote= 172.12.123.3,
    local_proxy= 172.12.23.0/255.255.255.0/0/0 (type=4),
    remote_proxy= 172.12.34.0/255.255.255.0/0/0 (type=4),
    protocol= AH, transform= ah-md5-hmac  (Tunnel),
    lifedur= 0s and 0kb,
    spi= 0x0(0), conn_id= 0, keysize= 0, flags= 0x0
*Mar 30 23:21:33.707: Crypto mapdb : proxy_match
        src addr     : 172.12.23.0
        dst addr     : 172.12.34.0
        protocol     : 0
        src port     : 0
        dst port     : 0
*Mar 30 23:21:33.731: IPSEC(key_engine): got a queue event with 1 KMI message

We seem to have a mismatch somewhere along the configuration it would appear by our debug output, and the continuous pings of course kept failing, but here is one thing to definitely note:

R2#sh crypto isa sa

IPv4 Crypto ISAKMP SA
dst             src             state          conn-id slot status
172.12.123.2    172.12.123.3    QM_IDLE           2001    0 ACTIVE

IPv6 Crypto ISAKMP SA

R2#
ASR#3
[Resuming connection 3 to r3 … ]

R3#sh cry isa sa

IPv4 Crypto ISAKMP SA
dst             src             state          conn-id slot status
172.12.123.2    172.12.123.3    QM_IDLE           1001    0 ACTIVE

IPv6 Crypto ISAKMP SA

R3#
ASR#4
[Resuming connection 4 to r4 … ]
………..
…
Success rate is 0 percent (0/73)
R4#

As you can see R4 continue to timeout, however the “sh crypto isakmp sa” verifies that Phase 1 negotiated and created the IPSec SA, however we are not agreeing on Phase 2 parameters, particularly the transform-set that would actually be used to encrypt the data.

THIS IS A VERY IMPORTANT DISTINCTION, AS WE IN THE VPN’S BUSINESS MUST KNOW WHAT THE FOLLOWING OUTPUT MEANS, AND HOW TO INTERPRET IT!!! :

R3#sh cry ipsec sa

interface: Serial0/2
    Crypto map tag: VPNLAB, local addr 172.12.123.3

   protected vrf: (none)
   local  ident (addr/mask/prot/port): (172.12.34.0/255.255.255.0/0/0)
   remote ident (addr/mask/prot/port): (172.12.23.0/255.255.255.0/0/0)
   current_peer 172.12.123.2 port 500
     PERMIT, flags={origin_is_acl,}
    #pkts encaps: 72, #pkts encrypt: 72, #pkts digest: 72
    #pkts decaps: 0, #pkts decrypt: 0, #pkts verify: 0
    #pkts compressed: 0, #pkts decompressed: 0
    #pkts not compressed: 0, #pkts compr. failed: 0
    #pkts not decompressed: 0, #pkts decompress failed: 0
    #send errors 1, #recv errors 0

     local crypto endpt.: 172.12.123.3, remote crypto endpt.: 172.12.123.2
     path mtu 1500, ip mtu 1500, ip mtu idb Serial0/2
     current outbound spi: 0x616C073C(1634469692)

     inbound esp sas:

     inbound ah sas:

R3#
ASR#2
[Resuming connection 2 to r2 … ]

R2#sh cry ipsec sa

interface: Serial0/0
    Crypto map tag: CCNP, local addr 172.12.123.2

   protected vrf: (none)
   local  ident (addr/mask/prot/port): (172.12.23.0/255.255.255.0/0/0)
   remote ident (addr/mask/prot/port): (172.12.34.0/255.255.255.0/0/0)
   current_peer 172.12.123.3 port 500
     PERMIT, flags={origin_is_acl,}
    #pkts encaps: 0, #pkts encrypt: 0, #pkts digest: 0
    #pkts decaps: 72, #pkts decrypt: 72, #pkts verify: 72
    #pkts compressed: 0, #pkts decompressed: 0
    #pkts not compressed: 0, #pkts compr. failed: 0
    #pkts not decompressed: 0, #pkts decompress failed: 0
    #send errors 0, #recv errors 0

     local crypto endpt.: 172.12.123.2, remote crypto endpt.: 172.12.123.3
     path mtu 1500, ip mtu 1500, ip mtu idb Serial0/0
     current outbound spi: 0xF0BC088B(4038854795)

     inbound esp sas:

     inbound ah sas:
 –More–

When you see one side that is having encaps but not decaps, your first step is to verify that destination is reachable to the router performing the tunnel / encryption:

R3#ping 172.12.34.100

Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 172.12.34.100, timeout is 2 seconds:
..

Yep. This is not necessarily ever a VPN config problem, so don’t jump straight to debugs necessarily, in this example I had turned my lab off to do dishes and forgot to “wr mem” on SW1 so its default route pointed at R2 got wiped out.

After re-applying that default route, and pinging once again from R4:

R4#ping 172.12.23.100
Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 172.12.23.100, timeout is 2 seconds:
!!!!!

🙂

R3#sh cry ipsec sa

interface: Serial0/2
    Crypto map tag: VPNLAB, local addr 172.12.123.3

   protected vrf: (none)
   local  ident (addr/mask/prot/port): (172.12.34.0/255.255.255.0/0/0)
   remote ident (addr/mask/prot/port): (172.12.23.0/255.255.255.0/0/0)
   current_peer 172.12.123.2 port 500
     PERMIT, flags={origin_is_acl,}
    #pkts encaps: 138, #pkts encrypt: 138, #pkts digest: 138
    #pkts decaps: 61, #pkts decrypt: 61, #pkts verify: 61

R2#sh cry ipsec sa

interface: Serial0/0
    Crypto map tag: CCNP, local addr 172.12.123.2

   protected vrf: (none)
   local  ident (addr/mask/prot/port): (172.12.23.0/255.255.255.0/0/0)
   remote ident (addr/mask/prot/port): (172.12.34.0/255.255.255.0/0/0)
   current_peer 172.12.123.3 port 500
     PERMIT, flags={origin_is_acl,}
    #pkts encaps: 342, #pkts encrypt: 342, #pkts digest: 342
    #pkts decaps: 418, #pkts decrypt: 418, #pkts verify: 418

As you can see I did a happy dance before I got over to R2 to confirm it is encap’ing and decap’ing as well, meaning the traffic from R3 to R2’s Ethernet segment are now being encrypted before transmission, and my job to see this lab through to the end is done!

Now that it’s 2am I am officially fried for the night, I will be sure to wr mem on all devices this time, and our next step is to get GRE configured to take the IPSec tunnel!

To summarize, here are all the commands by themselves, no more explanations to configure and troubleshoot an IPSec VPN:

• access-list 123 permit ip 172.12.34.0 0.0.0.255 172.12.23.0 0.0.0.255
• crypto isakmp policy 10
• – encr 3des
• – hash md5
• – authentication pre-share
• crypto isakmp key 0 CCNP address 172.12.123.3 (use # 6 for encrypted key)
• crypto ipsec security-association lifetime seconds 86400
• crypto ipsec transform-set VPNLAB ah-md5-hmac
• – mode tunnel
• crypto map CCNP 100 ipsec-isakmp
• – set peer 172.12.123.2
• – set transform-set VPNLAB
• – match address 123
• int s0/0
• – crypto map VPNLAB
• TROUBLESHOOTING AND VERIFICATION COMMANDS BELOW
• debug crypto ipsec
• show crypto isakmp sa
• show crypto ipsec sa

I color coded things that are configured together as they drop you into different configuration modes, and put the few troubleshooting and verification command at the bottom, I will actually put this list at the top as well to just stare at if need be.