Here's a how to about different possibilities to make a IPsec VPN under GNU/Linux environment. (the goal is to make a communication with a Windows) computer.
Transport Mode : VPN between two hosts (generally inside a same LAN).
Tunnel Mode : Secured Tunnel between two (or more) routers in order to make secure communication different LAN through unsecured links (generally via Internet).
ESP : Encapsulating Security Payload, allow confidentiality (3DES, AES...), integrity (MD5, SHA1.., but not on the entire frame) and authentication. It's the IP protocol n°50.
AH : Authentication Header, allow integrity on the entire packet. It's the IP protocol n°51.
I've made my tests under VMware environment. To make it works we need to correctly set vSwitches, (see few lines below).
OS : Debian GNU/Linux 6.0.5 (squeeze)
Role : Gate + Certificate authority
IP : 192.168.0.84
Bob or Regis :
OS : Debian GNU/Linux 6.0.5 (squeeze) or Windows XP professionnel SP3 or Windows 7 SP1.
Role : IPsec Host
IP : 192.168.0.85
OS : Debian GNU/Linux 6.0.5 (squeeze).
Role : Network packet analyzer between IPsec hosts.
Debian Bridge Configuration
root@host:~# apt-get install bridge-utils
# The loopback network interface
auto lo br0
iface lo inet loopback
# The primary network interface
#NetworkManager#iface eth0 inet dhcp
iface eth1 inet manual
iface eth2 inet manual
iface br0 inet static
bridge_ports eth1 eth2
Under a VMware architecture we need to set the following sets to the vSwitch to make it works :
Promiscuous Mode : avoid virtual machines to listen trafic on the network interface connected to the vSwitch. So if the Reject option is set we won't be able to use Wireshark.
MAC Address Changes : Reject packets where the MAC address doesn't match with the one which is recorded inside the .vmx file of the VM. Avoid MAC spoofing attack.
Forged Transmits : Avoid a VM to send forged ARP packets in order to redirect traffic. Avoid ARP spoofing and Cache Poisonning attacks.