TSHOOT – GRE DEEP DIVE!!! MTU / Fragmentation / TCP-MSS / PMTUD / Recursive Routing / Interface States / must know information for exam day!

The graphic above shows a VPN Tunnels two modes (Transport and Tunnel), along with how their payloads differ when configured with AH (Authentication only) and ESP (Encryption) on the tunnel, and what protocols correlate with all of it.

(After watching the content of the CCIE R/S VPN Technology course, this article more than sums up what is needed there, so I will also be using it for CCIE)

The basics of GRE and IPSec Tunnel Modes and Packet Formats

Basic GRE tunnel configuration has absolutely no security involved in terms of authentication or encryption, it is purely a transport vessel, as shown here:

It simply encapsulates data packets for transport with a GRE Header added to the packet so the remote endpoint can identify GRE Traffic, at its core it is used for transporting certain types of traffic (such as IPv6 over an IPv4 network), however it can be Authenticated (AH) or Encrypted (ESP) as well to secure the payload.

To review this graphic once more in detail to point out the differences observed:

The big difference between Tunnel Mode vs Transport Mode, is that Tunnel Mode uses a new IP Header in front of the Authenticated / Encrypted packet, whereas Transport mode uses the original IP Header when “Transporting” the data packet over the VPN Tunnel.

So on exam day if you see anything about “adding headers” / “additional headers” / etc, you know that we are talking about Tunnel mode, as Transport Mode DOES NOT add any new IP Headers to packets on their way across the VPN Tunnel!

AH and ESP are flavors of IPSec, not GRE, which first requires the configuration of IPSec:

R2(config)#int tu0
R2(config-if)#tunnel
*Mar 31 00:11:11.825: %LINEPROTO-5-UPDOWN: Line protocol on Interface Tunnel0, changed state to down
R2(config-if)#ip add 10.1.1.2 255.255.255.0
R2(config-if)#tunnel source s0/0
R2(config-if)#tunnel destination 172.12.123.3
R2(config-if)#
*Mar 31 00:12:16.912: %LINEPROTO-5-UPDOWN: Line protocol on Interface Tunnel0, changed state to up
R2(config-if)#
R2(config-if)#tunnel mode ipsec ipv4

GRE itself can be configured in Tunnel mode as well, with “tunnel mode gre ip” in the tunnel interface configuration, but it offers absolutely zero security to the payload!

Once IPSec is defined as the “Tunnel Mode” on the tunnel interface, these two different configurations are what defines whether AH or ESP is being used:

AH

R2(config)#crypto ipsec transform-set VPNLAB ah-md5-hmac

This defines the Authentication that must be agreed upon on each side, or with IPSec VPN, this is a piece of “Phase 1” of an IPSec VPN Tunnel.

ESP

R2(config)#crypto isakmp policy 10
R2(config-isakmp)# encr 3des
R2(config-isakmp)# hash md5
R2(config-isakmp)# authentication pre-share

This defines the Encryption to be used by the VPN Tunnel, or with IPSec, this is defined as “Phase 2” of a VPN Tunnel.

There is a lot more to the configuration of IPSec, which can be found here, however to keep this concise and focus on GRE topics I won’t go further than this into configuration.

Protocols used in GRE and IPSec VPNs

These are not TCP or UDP, but rather IP Protocol #’s, which are used for VPN technology:

• 47 – GRE
• 50 – ESP
• 51 – AH
• 500 – ISAKMP
• 4500 – NAT-T (Nat Traversal)

For exam day, remember that ESP and AH are just flavors of IPSec and not the actual Tunnel itself, so if asked what port # (singular) needs to be open for a VPN Tunnel to form it will be Port 47 first then 50 and 51 if additional answers are available.

Also a quick note on NAT-T – This is used for IPSec VPN traffic because the Tunnel Endpoints will detect if packets are malformed via NAT, so this mechanism is used to avoid the traffic being touched by NAT all together.

What is Fragmentation, and why does it matter?

The entire issue with Fragmentation and VPN technologies is NOT that it will not work at all, but fragmentation of packets from the sender means virtual re-assembly before decryption by the receiver, which if heavily utilized can lead to dropping traffic and severe performance issues on drop sensitive traffic – So we want to avoid it!

If the DF-Bit (Don’t Fragment Bit) is set on drop sensitive traffic, it will actually be dropped before even being sent, though this can be changed via Policy Based Routing (PBR) by changing this bit on the Ingress interface before it hits the Egress Interface.

That may be beyond what is tested on the CCNP R/S, but good to know 🙂

Default MTU Values, Double Fragmentation, PMTUD, and TCP-MSS

By default the MTU (Maximum Transmission Unit) on a Physical Interface is 1500 bytes, and a logical Tunnel Interface will have a default MTU value of 1476, as basic GRE adds 24 bytes in overhead to a packet (20 byte IP Header and 4 byte GRE Header).

This ensures that traffic sent to the physical interface via the Tunnel interface does not exceed the MTU of 1500, thus avoiding Fragmentation for normal packet throughput.

Double-Fragmentation happens when using IPSec over GRE, and the packet is fragmented twice (once for GRE and once for IPSec) due to the overhead, there are specific values (1440 Transport and 1420 Tunnel Mode MTU size) that can be set, however an established baseline of 1400 MTU is the generally used MTU.

TCP-MSS (TCP Maximum Segment Size) is configured on the Tunnel interface to “set” the Maximum Data Packet size it will accept, and is sent to the remote host within a TCP Header option field in a packet to its neighbor, basically telling it “if you send me packets larger than X I am going to drop them” to the remote VPN Peer.

TCP-MSS values do not have to match on either side, and from what I’ve found, a value of 40 bytes under the Tunnel interface MTU is the standard TCP-MSS configuration.

This is configured with the command: “tcp-adjust mss #” on the logical Tunnel interface.

However this is only done between the VPN endpoints, thankfully we have an answer for that, called PMTUD!

PMTUD (Path MTU Discovery) works by determining the lowest value MSS / MTU in the data path, and dynamically adjusting its own Send MSS for that link to avoid Fragmentation, and it monitors (in both directions) every packet to either:

1. Packet makes it fine to the end point
2. ICMP Response returned to sender advising DF Bit set / Packet Discarded

PMTUD receives these responses from devices in the path to constantly adjust its Send MSS, again this happens in both directions, it gets a bit more technical but that is a pretty good technical overview of it WHEN ITS WORKING.

However, what happens when PMTUD is NOT working? What causes that?

Excellent question!

There are a few causes that can stop PMTUD from working, and it all revolves around the ICMP Message sent advising of the packet drop due to exceeding the MSS / MTU:

1. Router that drops packet cannot generate an ICMP response to sender
2. Router somewhere in the path blocks the ICMP response sent
3. Router receiving the ICMP response ignores the packet

There are a lot of things like ACL’s blocking pings, dropping certain network traffic, etc that can be potential causes. However, there is one very defined culprit that can be configured on a router in the path that will cause PMTUD to fail:

“no ip host unreachable” – The Host Unreachable responses are what fuels PMTUD!

If it is not working, this is likely causing your issue, though anything blocking this ping response along the path (perhaps a routing loop expiring the ICMP TTL?) can cause issue.

Things that DO, and DO NOT have to match on opposite sides of a tunnel

Things that DO have to match or line up correctly:

• Authentication / Encryption if configured (IPSec)
• Subnets configured in Crypto Maps (IPSec again)
• Tunnel mode IPv4 or IPv6 on both sides
• Tunnel source IP version and defined Tunnel mode must match
• Tunnel IP’s (Overlay) and Tunnel Source / Destinations must be on same subnet
• The Underlay (Physical IP connection) must be reachable for Overlay to work

A couple of kind of tricky things is one side of a tunnel having “tunnel mode gre ipv6” configured on the tunnel interface, when both sides are using an interface as the “source” for the Underlay, which has an IPv4 Address.

In this situation the fix is not to configure “tunnel mode gre ipv6” on the other side because they are using IPv4 for the Underlay, so the fix is to remove that ipv6 config.

The Underlay information must match IP versions, and be reachable, an IPv6 source / destination cannot be used on one side and an IPv4 source / destination on the other end.

Outside of IPSec which I won’t get into here, those are the biggies of what must jive.

Things that DO NOT have to match or line up properly:

Timers / Keep Alives – GRE is stateless meaning the Tunnel interface will stay Up/Up even if the remote Tunnel interface is Down, so unless an IGP hold timer brings down the Tunnel interface, it will stay up and sending information into a Data Blackhole.

This is where “keep alive 10 3” comes in to configure on the Tunnel interface, as this sets a hold timer of 10 and will send keep alives every 3 seconds, as you might have guessed after 3 missed keep alives the Tunnel interface will be brought down.

The very interesting part about keep alives (or I thought this was pretty cool)!

The sender double encapsulates the keep alive message sent, which tricks the receiver into auto-sending a response back to the sender, so even if the peer doesn’t have keep alives set the local router will still have its keep alives answered.

Sneaky routers! I thought that was very interesting.

Also GRE Timers / Keep Alives are NOT compatible with mGRE / DMVPN, so if asked about this on exam day, the answer is no!

MTU / TCP-MSS – This does not need to match configuration wise either, though its important that both routers are aware of each others limitations, so they are not dropping tons of traffic or taking on a ton of fragmentation that drops the network!

I think that is about it for “DO NOT” have to matches, if I think of more I will add them.

Recursive Routing, what causes it to happen, and possible fixes for it!

Recursive Routing is an issue when traffic destined for the Tunnel Interfaces destination (the Underlay) is sent to the Tunnel Interface itself (The Overlay), this causes the router lookup process to freak out and kicks out the following console message:

%TUN-RECURDOWN Interface Tunnel 0
temporarily disabled due to recursive routing

This will occur when the tunnel destination IP is pointed to the actual Tunnel interface itself rather than out the physical interface (Underlay) to its destination.

Or a more tricky scenario, an IGP advetises the the Tunnels Overlay Address via the routing protocol, so you now have a Dynamically Learned Tunnel Destination Address (Underlay) with a next-hop of Tunnel0 (to itself / the Overlay), which makes the IP Route table melt down and bring the Tunnel down due to Recursive Route lookup.

A fix for this is a static IP to the Tunnel Destination Address, and if there is already a static route but you are having Recursive Routing issues, make a longer prefix static route to the Tunnel Destination address.

There may be other scenarios on exam day, but the bottom line to Recursive Routing, is that traffic is trying to Route over Tunnel0 (The Overlay), that really needs to Route over the Underlay instead – So consider this when choosing an answer if asked about this!

GRE Tunnel Interface States and causes (Pulled from Cisco’s website)

There are four possible states in which a GRE tunnel interface can be:

Up/up – This implies that the tunnel is fully functional and passes traffic. It is both administratively up and it’s protocol is up as well.

Administratively down/down – This implies that the interface has been administratively shut down.

Up/down – This implies that, even though the tunnel is administratively up, something causes the line protocol on the interface to be down.

Reset/down – This is usually a transient state when the tunnel is reset by software. This usually happens when the tunnel is misconfigured with a Next Hop Server (NHS) that is it’s own IP address.

A couple of my own, non-Cisco website notes on these interface states:

Up/Down can be caused by – Keep Alive not getting a response, mismatched “tunnel mode gre (ip/ipv6)” on either side, IGP running over Tunnel hold timer expires / brings the tunnel interface logically down, or Recursive Routing as just discussed!

Reset/Down – This is generally just descirbed as “caused by software resets” in any exam prep or training materials I’ve seen, don’t let fancy wording confuse you on exam day!

One last piece of information, an example of GRE Tunnel “sh cry ipsec sa” output

local  ident (addr/mask/prot/port): (2.2.2.2/255.255.255.255/47/0)
remote ident (addr/mask/prot/port): (3.3.3.3/255.255.255.255/47/0)  —- omitted —-
inbound ah sas:
spi: 0x6580F6CA(1702950602)
transform: ah-md5-hmac ,
in use settings ={Transport, }
conn id: 41, flow_id: 41, sibling_flags 80000010, crypto map:

I actually stole this demo from a Cisco Learning Network post found here.

I wanted to point out when looking at GRE “sh cry ipsec sa” local ident / remote ident information highlighted in red we can see 47 as the Protocol # used indicating GRE.

The local ident = The Tunnel Source (Underlay) information
The remote ident = The Tunnel Destination (Underlay) information

This may come in handy when reviewing output, so I’d maybe click on that link and check out the full configuration in that post to understand that output, if you are not familiar with what configs would give that “sh cry ipsec sa” output!

That does it for this DEEP DIVE into all things GRE, you are now an Expert!

This is not necessarily a HUGE part of any CCNP exam I’ve taken to date, however if questions are seen on exam day for ROUTE or TSHOOT, its a waste of point not to give this a healthy once over and understand the concepts and how everything works.

Will keep updating as I think of good tid bits to help push towards exam day 🙂