Understanding IPSEC VPN tunnels

There is a big difference between just following the configuration steps to make IPSec work and being able to actually understand what is happening in the background when you type those configuration commands.

In this article I will try to uncover that and also will try to make it very simple. Once you finish this article, you will have a good understanding of the basic components of a network that has an active IPSec tunnel. You will also have a clear overview on the related protocols and the way they interact with each other.

People are concerned about securing the communication in their networks. Starting from the application layer and down to the network layer.

Now Why would you need to encrypt something? just to add an overhead. Well of course not ! it is all about confidentiality which gives you the privilege of securing your data in a way that even if your packets got sniffed, the attacker will look at the encrypted data and will not be able to make any sense of it.

There are two common solutions that implements Confidentiality.

Way 1:

The first way is one that you use every day with HTTPS and FTPS. It is called the Transport level security which works at the transport layer by encrypting application data.

Way 2:

The second one is called IPSEC which provides gateway to gateway encryption where the end host/application are not aware of the encryption process and the encryption happens to the network layer.

So you get the advantage of tunneling and it allows you to connect more than two sites together over the internet. it can also be used for software clients too.

Behind the scenes, IPSec is not just one protocol rather than is a collection of protocols/processes that interact together to make it work. Before tackling those protocols, let’s have discuss some of the related cryptography terminologies:

Authentication is the process of verifying the identity of the peer you are speaking to. How you know it is not any random peer that is spoofing the identity of the valid peer? For authentication you can use:

• Pre-shared keys where the same key is provided to each end of the tunnel.
• Another option is using X509v3 certificates. Each end has a certificate and the remote peer should be able to verify that certificate which means he should trust the issuer of the remote peer certificate.

In this article i will discuss the preshared key authentication method as we will not discuss the PKI here [at least for now]. ??????

Confidentiality is about encrypting the plain text data so that no one can know what you are sending in case you are using an insecure channel. In other words it is that the no one can have an idea about the clear-text by looking at the cipher-text.

There are two types of encryption algorithms:

• Symmetric where the two sides share the same key and that key will be used for encryption and decryption. An example of those are DES, 3DES, and AES.
• The other type is Asymmetric where there are two keys, a public one that you share with anyone and a private one that should be kept secret. Whatever you encrypt with one key it can be decrypted with the other key[RSA, DSA].

Generally speaking symmetric ones are much faster than asymmetric algorithms and it is used with IPSEC for data encryption.

Integrity is being able to verify that the data you are getting has not been tampered in transient. This is being done using one way functions [aka hash functions] which produces a fixed length output from an input with a variable size. Hash functions can be used to verify the integrity of the files you downloads where you have a calculated hash provided before you download the file. After you download it you can compare the new calculated value with the original one. That will not be enough for verifying the integrity of the data packets moving across the network. You need to add another piece to this which is a key that will be fed to the input of the hash function with the message [known as Hashed message authentication code]. No one can tamper the message and generate a new hash and then attach it to it because he has no knowledge of the key used with the HMAC generation process. So the receiver will still be able to verify that the message has been changed or no.

After explaining those terms related to cryptography, it should be very clear why do we need a protocol to negotiate a specific set of parameters [encryption algorithms, authentication method and session keys]. For example we need ;

--A method to share a key that will be used for symmetric encryption of the traffic.

--A way to agree on which algorithms to be used for encryption and authentication.

--A way to authenticate the peer you are talking to.

--A protocol to encapsulate the encrypted payloads and to guarantee the integrity of those packets.

The last point talks about the actual protocol that will be used to encapsulate the encrypted packets. We will focus on encapsulation security payload [ESP] which is the most common one.

The first protocol that we are going to discuss here is IKE [internet key exchange protocol]. This is a keying management protocol. it will achieve the following :

a) Share the session key between the two peers.

b) Authenticate the peer.

c) Establish a secure channel to negotiate further security associations.

IKE has two versions. We will start discussing version 1. Version 1 has two phases. Phase 1 authenticates the peer, negotiate the Proposals to be used and finally generate the session key. Before explaining each packet in this phase i would like to explain the Diffie-Hellman key agreement protocol that is a very essential one when it comes to cryptography.

Diffie Hellman is used to generate a session key between two peers over an insecure channel. In this example we have Alice and Bob how wants to generate a session key over an insecure channel. let us take a look at a DH example here so share a key between Alice and Bob.

1. Alice and Bob agree to use a modulus p = 23 and base g = 5. These two values are shared between Alice and Bob.
2. Alice chooses a secret integer a = 6, then sends Bob A = ga mod p A = 56 mod 23 = 8
3. Bob chooses a secret integer b = 15, then sends Alice B = gb mod p, B = 515 mod 23 = 19
4. Alice computes s = Ba mod p, s = 196 mod 23 = 2 which is the session key in this case.
5. Bob computes s = Ab mod p,s = 815 mod 23 = 2
6. Alice and Bob now share a secret (the number 2).

You can ignore the math if you want. Just remember that it allows you to generate a shared key over an insecure channel. Now we are ready to look at IKEv1 main mode :

Here is the description of each message :

MM1 : Main mode message number 1 which contains the proposals aka security associations payloads that we will use to secure the main mode exchange. We negotiate the encryption, hash, DH group and authentication mode. We sends all the proposals we have.

MM2 : Main mode message number 2 where the peer checks our suggested proposals and reply back with the one he have chosen. At this point we know what encryption, hash, authentication and DH group will be used. The DH group specify the p and g that we will use.

MM3 : is contains two things, the DH material generated by the initiator and a nonce which is a random value that will be used later in keys generation and also it is used to prove the liveness of the peer.

MM4 : Main mode message number 4 where the peer reply with its Nonce and the DH keying material.

Note : After this message both devices can now generate the DH key. With the Pre-shared key that is already configured and the DH generated key they calculate another key that will be used as a material for generating the rest of the keys. Hence the next two messages are encrypted and authenticated.

MM5 : Main mode message number 5 which contains the identity [ip address or hostname] and a hash that is computed on specific inputs, calculated using the key too [this is where the authentication happens].

MM6 : Main mode message number 6. Same content as MM5 but in the other direction. After this message Phase 1 is completed. the peer is authenticated and now we can proceed to negotiate another phase that is concerned about the data encryption itself.

Quick Mode

which is known as phase 2. Here are the messages :

QM1 : quick mode message number 1 which contains the proposals for phase 2 encryption/integrity algorithms, the networks that should be protected by this security association, a nonce that will be used in the hash calculation and a HASH that is used to authenticate the message.

Note :

SA : here it means security association and it is the payload that carries the proposals [AES,3DES, SHA1, MD5....etc]

ID : here refers to the IP subnets for which you wanna apply encryption.

QM1 : quick mode message number 2. It contains the proposal selected, a HASH, a responder nonce and the subnets to be encrypted.

QM3 : quick mode message number 3. it contains a hash for authentication and it ensures the sender of QM2 that QM2 has been received as the HASH is calculated based on values in QM2.

Note

The quick mode three messages are encrypted.

So far we have negotiated what encryption services should be applied to the traffic, what traffic should be protected, and the keys to be used. We have also authenticated the remote peer. After that we are ready to start encrypting the traffic but we need a protocol to do the encapsulation for us. There are two protocols ESP and AH. ESP provides Integrity and confidentiality while AH provides integrity only. As it is the most common one we will talk about ESP only. First the ESP header which is very important as it will appear in every packet where the services negotiated were applied :

Security parameter index : This is a value that is negotiated in phase 2. it is very important as the gateway uses it with the peer ip address to identify the security association it should use to serve this packet. Every Encapsulated data packet has this value. Each security association has one inbound and one outbound SPI.

Sequence number: This is included in each packet and will be incremented by one for each encrypted packet we send. This helps protecting against replay attacks. it help the router to avoid getting replayed ESP packets thus avoid wasting more resources on this packets.

Note :

There is a field in the header that tells the device decrypting the packet what is the next header to be expected. so that it knows how to parse it.

The last thing in the header is the HMAC calculated on the encrypted packet which helps detecting if the packet was changed in transient.

Next we are going to discuss ESP modes, the first one is tunnel mode in which a new ip header is added to the encrypted payload. This encapsulation does the tunneling part of the protocol as any device will be forwarding the packet based on the new added ip header and the original ip header is hidden with encryption. So the private addresses are not revealed to the outsiders. The second mode is the transport mode where the original IP header is not encrypted it is only the IP payload that is encrypted. This is good when you do host to host encryption and notice that the routing will not be a problem here as the original IP headers are not hidden. In the next figure we can see the two modes.

Figure 2.1 ESP tunnel mode

Figure 2.2 ESP transport mode

Finally we need to mention Network address translation effects on IPSEC VPN. ESP is running over IP directly so it is not encapsulated with TCP or UDP. If you do static or dynamic address translation at the ip header only ESP will work as the authentication is not computed on the new added header while AH will fails because of that.

But we still have one problem. What if the device between the two peers is doing a port translation. If the device doesn't find a port in the packet then it will drop it. The solution of this is NAT traversal which is here to solve this problem. during IKE phase 1 the two peers send nat discovery payloads which contains a hash of the source port/IP and destination port/IP. The remote peer calculate the hash manually and then compares it with what has been received, if there is a mismatch then it knows that there is a NAT device. If NAT is detected then it switches to port 4500. MM5 , MM6 and the Quick mode exchange plus the data packets encapsulated with ESP will be encapsulated inside UDP 4500 header. So the problem with the NAT will be resolved.