IPSec VPN for Linux

IPsec Protocols

Authentication Header (AH) and/or Encapsulating Security Payload (ESP) are the two protocols that we use to actually protect user data. Both of them can be used in transport or tunnel mode.

Authentication Header Protocol

AH offers authentication and integrity but it doesn’t offer any encryption. It protects the IP packet by calculating a hash value over almost all fields in the IP header. The fields it excludes are the ones that can be changed in transit (TTL and header checksum).

Encapsulating Security Payload Protocol

ESP is the more popular choice of the two since it allows you to encrypt IP traffic. We can use it in transport or tunnel mode.

ESP in Transport Mode

Configurations

There are many ways to configure IPsec VPN between two hosts. Here we are using  racoon to configure IPsec VPN between two hosts in transport mode. 

Step 01: Install racoon and ipsec-tools in each server

Step 02: Configure IPsec 

• Machine A IP: 192.168.1.170 add below content to /etc/ipsec-tools.conf

 

#!/usr/sbin/setkey -f
flush;
spdflush;

 

spdadd 192.168.1.170 192.168.1.171 any -P out ipsec
    esp/transport//require;
spdadd 192.168.1.171 192.168.1.170 any -P in ipsec
    esp/transport//require;

•  Machine B IP: 192.168.1.171 add below content to /etc/ipsec-tools.conf

 

#!/usr/sbin/setkey -f

flush;

spdflush;

 

spdadd 192.168.1.170 192.168.1.171 any -P in ipsec

    esp/transport//require;

spdadd 192.168.1.171 192.168.1.170 any -P out ipsec

    esp/transport//require; 

• To apply the policies restart the “setkey” service in both servers 

systemctl restart setkey 

• To verify the policies applied correctly you can use below command.

setkey -PD 

• Machine A IP: 192.168.1.170 put below content to “raccoon.conf” file. /etc/racoon/racoon.conf

log notify;

path pre_shared_key "/etc/racoon/psk.txt";

path certificate "/etc/racoon/certs";

 

remote anonymous {

exchange_mode main,aggressive;

lifetime time 2 min;

proposal {

encryption_algorithm aes 256;

hash_algorithm sha256;

authentication_method pre_shared_key;

dh_group modp1024;

}

generate_policy off;

} 

sainfo anonymous {

lifetime time 2 min;

pfs_group 2;

encryption_algorithm aes 256;

authentication_algorithm hmac_sha256;

compression_algorithm deflate;

}

• Machine B IP: 192.168.1.171 put below content to “raccoon.conf” file. /etc/racoon/racoon.conf

log notify;

path pre_shared_key "/etc/racoon/psk.txt";

path certificate "/etc/racoon/certs";

 

remote anonymous {

exchange_mode main,aggressive;

lifetime time 2 min;

proposal {

encryption_algorithm aes 256;

hash_algorithm sha256;

authentication_method pre_shared_key;

dh_group modp1024;

}

generate_policy off;

}

sainfo anonymous {

lifetime time 2 min;

pfs_group 2;

encryption_algorithm aes 256;

authentication_algorithm hmac_sha256;

compression_algorithm deflate;

}

• Add below line to /etc/racoon/psk.txt Host A: 192.168.1.170

192.168.1.171 a9993e364706816aba3e

• Add below line to /etc/racoon/psk.txt Host B: 192.168.1.171

192.168.1.170 a9993e364706816aba3e

If the tunnel is configured correctly, when you ping from each host you should be able to ping and can see the traffic go through the tunnel from wiresharq dump.

AH + ESP in Transport Mode

With transport mode we will use the original IP header, followed by an AH and ESP header. The transport layer, payload and ESP trailer will be encrypted.

Because we also use AH, the entire IP packet is authenticated.

In Transport Mode we can user AH and ESP together. With transport mode it will use the original IP header, followed by an AH and ESP header. The transport layer, payload and ESP trailer will be encrypted.
AH does not encrypt the content of the message but it makes sure that the sender is known and that the IP-header is untouched.

Configurations

• To enable “Authentication Headers” we have to add few policies to ipsec-tools.conf file

Host-A 192.168.1.170

 

#!/usr/sbin/setkey -f

flush;

spdflush;

# Put the ESP SAs in the DB

add 192.168.1.170 192.168.1.171 esp 31031 -E aes-cbc

0xdcc6981b0787002abf5dfc28d8b2221063b949316b35c51a7bdb65753e369c66;

add 192.168.1.171 192.168.1.170 esp 43241 -E aes-cbc

0x3a9fc0f3d7081ae4f4184250b658f69c0d5b220367a160b8592871d2eeef1523;

# AH SAs In the DB

add 192.168.1.170 192.168.1.171 ah 33362 -A hmac-sha256

0xf93c4f2b0034dd236e77d9aca68014d09dd7c2ef0571d7831082617b1b560324;

add 192.168.1.171 192.168.1.170 ah 30759 -A hmac-sha256

0xd3e451082bb8837e88f238c153f16664bc0b33542515d6d8d4b79050e06662c8;

spdadd 192.168.1.170 192.168.1.171 any -P out ipsec

esp/transport//require

ah/transport//require;

spdadd 192.168.1.171 192.168.1.170 any -P in ipsec

esp/transport//require

ah/transport//require;

Host-B 192.168.1.171

#!/usr/sbin/setkey -f

flush;

spdflush;

# Put the ESP SAs in the DB

add 192.168.1.170 192.168.1.171 esp 31031 -E aes-cbc

0xdcc6981b0787002abf5dfc28d8b2221063b949316b35c51a7bdb65753e369c66;

add 192.168.1.171 192.168.1.170 esp 43241 -E aes-cbc

0x3a9fc0f3d7081ae4f4184250b658f69c0d5b220367a160b8592871d2eeef1523;

# AH SAs In the DB

add 192.168.1.170 192.168.1.171 ah 33362 -A hmac-sha256

0xf93c4f2b0034dd236e77d9aca68014d09dd7c2ef0571d7831082617b1b560324;

add 192.168.1.171 192.168.1.170 ah 30759 -A hmac-sha256

0xd3e451082bb8837e88f238c153f16664bc0b33542515d6d8d4b79050e06662c8;

spdadd 192.168.1.170 192.168.1.171 any -P in ipsec

esp/transport//require

ah/transport//require;

spdadd 192.168.1.171 192.168.1.170 any -P out ipsec

esp/transport//require

ah/transport//require; 

• To apply the policies restart the “setkey” service in both servers 

systemctl restart setkey 

ESP in Transport Mode vs ESP + AH in Transport Mode  

This is packet capture for ESP in Transport mode

This is packet capture for AH + ESP in Transport Mode