Guide to Establishing an IPsec Transport Mode VPN with X509 authentication Between Two Windows Computers

Last updated: Jun 5, 2024

I've already discussed how to establish an IPsec VPN between two hosts in a Microsoft Windows environment using a PSK. For more details, see here. Now, let's enhance security by replacing the PSK with x509 certificates. In this guide, we'll explore how to set up a secure IPsec VPN between Microsoft Windows hosts within the same LAN to significantly improve network security. With this configuration, all communications between machines will be encrypted without exception.

Network Architecture

For the purposes of this article, we will maintain a simple architecture. We will establish a secure connection using an IPsec VPN with x509 authentication between a computer running Windows 11 and a server running Windows Server 2022. Both computers will be on the same local network.

Network diagram showing ipsec tunnel X509 between two Windows Hosts

Windows Server

PKI

As discussed previously, we will use X.509 certificates to establish an IPsec connection between our server and client. To do this, we need to create a PKI: a Root CA which will sign the certificates, and two certificates, one for each of our two hosts.

All the commands below will be executed from a PowerShell console with administrator rights.

Certificate for the Server

Create the Root CA, which will enable us to sign server and client certificates:

PS C:\> $rootCert = New-SelfSignedCertificate -CertStoreLocation Cert:\LocalMachine\My -DnsName RootCA -KeyUsage CertSign,CRLSign,DigitalSignature -NotAfter (Get-Date).AddYears(10)

Create server certificate, and sign it with RootCA (Note: we can change "ServerName" to the actual server name.):

PS C:\> $serverCert = New-SelfSignedCertificate -CertStoreLocation Cert:\LocalMachine\My -DnsName ServerName -Signer $rootCert -NotAfter (Get-Date).AddYears(5)

Retrieve server certificate information to the $serverCert variable:

PS C:\> $serverCert = Get-ChildItem -Path Cert:\LocalMachine\My | Where-Object { $_.Subject -eq "CN=ServerName" }

Client Certificate

Create a client certificate and sign it with RootCA (Note: "W11" can be replaced with the actual client name.):

PS C:\> $cert = New-SelfSignedCertificate -DnsName "W11" -CertStoreLocation "cert:\LocalMachine\My" -Signer $rootCert -KeyExportPolicy Exportable -NotAfter (Get-Date).AddYears(5)

Set a password to protect the pfx file, which will be used to store the client's certificate with the private key:

PS C:\> $password = ConvertTo-SecureString -String "PassPass" -Force -AsPlainText

Export the client certificate to a pfx container, exemple here to C:\W11.pfx:

PS C:\> Export-PfxCertificate -Cert $cert -FilePath "C:\W11.pfx" -Password $password

Check

Open the Local Computer Certificates Manager console:

Windows run window with certlm.msc written inside

From the Personal > Certificates folder, check that the previously created certificates are present:

Windows Local Computer Certificates Manager with the Personal Certificates folder opened

IPsec Configuration

Graphical User Interface (GUI)

Open the Windows Defender Firewall with Advanced Security console:

Windows run window with wf.msc written inside

Do a right click on Connection Security Rules and select New Rule…:

Windows Defender Firewall with Advanced Security console with the connection security rules drop-down menu

Select Custom and click Next:

New Connection Security Rule Wizard window with custom option selected

In the Which computers are in Enpoint 1 section, select These IP addresses and click on Add…:

New Connection Security Rule Wizard window, Endpoints step

Here we add the Host or Network with which we need have to establish an IPsec connection to communicate:

IP Address window, with network 192.168.1.0/24 filled in

Do the same with Enpoint 2 entering the server's IP address then click Next:

For maximum security and impose the use of IPsec select Require authentication for inbound and outbound connections:

New Connection Security Rule Wizard window, Requirements step

As we'll be using a certificate as our Authentication Method in this example, click on Customize…:

New Connection Security Rule Wizard window, Authentication Method step

Follow this steps to set the Certificate:

New Connection Security Rule Wizard window, the three steps to set the certificate

We can specify one or more specific protocols to be encapsulated inside IPsec, here we encapsulate everything:

New Connection Security Rule Wizard window, Protocol and Ports step

We can restrict the rule to specific profiles:

Finally specify the Name of the rule:

New Connection Security Rule Wizard window, Name step

PowerShell

We can create the same IPsec Rule as the one configured just above, in a three PowerShell commands line:

PS C:\> $cert1Proposal = New-NetIPsecAuthProposal -Machine -Cert -Authority "CN=RootCA" -AuthorityType Root PS C:\> $certAuthSet = New-NetIPsecPhase1AuthSet -DisplayName "Computer Certificate Auth Set" -Proposal $cert1Proposal

PS C:\> New-NetIPsecRule -DisplayName "IPsec" -Name "IPsec" -Mode Transport -InboundSecurity Require -OutboundSecurity Require -LocalAddress 192.168.1.200 -RemoteAddress 192.168.1.30 -Enable True -Phase1AuthSet $certAuthSet.Name

FireWall Configuration

We will create a Firewall rule to allow encrypted flows.

Graphical User Interface (GUI)

Do a right click on Inbound Security Rules and select New Rule…:

Windows Defender Firewall with Advanced Security console with the Inbound rules drop-down menu

Select Custom and click Next:

New Inbound Rule Wizard window, Rule Type step

Choose All programs and click on Next:

New Inbound Rule Wizard window, Program step

We can specify one or more specific protocols to be matched by the rule, here we will match Any protocols:

New Inbound Rule Wizard window, Protocol and Ports step

Restrict rule application to defined ip addresses concerned:

New Inbound Rule Wizard window, Scope step

In the Action step, configure the Require the connections to be encrypted condition:

New Inbound Rule Wizard window, Action step, selecting Require the connections to be encrypted

We can limit the application of the rule to specific users, which will not be used for our purposes:

New Inbound Rule Wizard window, Users step

We can do the same to limit the rule to specific computers, which won't be used in this example either:

New Inbound Rule Wizard window, Computers step

We can restrict the rule to specific profiles:

Finally specify the Name of the rule:

New Inbound Rule Wizard window, Name step

PowerShell

We can create the same Firewall Rule sas the one configured just above, in a single PowerShell command line:

PS C:\> New-NetFirewallRule -DisplayName "IPsec ALLOW" -Direction Inbound -Enabled True -Action Allow -LocalAddress 192.168.1.200 -RemoteAddress 192.168.1.0/24 -Protocol Any -Encryption Dynamic -Authentication Required

We've now finished configuring the server. Now it's time to move on to client configuration.

Windows 11

Importing the Certificate

First, copy the W11.pfx file previously generated to the Windows 11 system and double-click on it:

Select Local Machine then click Next:

Certificate Import Wizard, Welcome to the Certificate Import Wizard Step

Click Next:

Certificate Import Wizard, File to Import Step

Type the password previously set when exporting certificates to pfx and click Next:

Certificate Import Wizard, Private Key Protection Step

Choose “Automatically select the certificate store” and click Next:

Certificate Import Wizard, Certificate Store Step

IPsec Rule

I'm going to describe them all, but the steps are pretty much the same as those we followed for the server.

Open the Windows Defender Firewall with Advanced Security console:

Do a right click on Connection Security Rules and select New Rule…:

Select Custom and click Next:

In the Which computers are in Enpoint 1 section, select These IP addresses and click on Add…:

Here we add our client's IP address:

IP Address window, with network 192.168.1.30 filled in

Do the same with Enpoint 2 entering the server's IP address then click Next:

For maximum security and impose the use of IPsec select Require authentication for inbound and outbound connections:

As we'll be using a preshared key as our Authentication Method in this example, click on Customize…:

Follow this steps to set the Certificate:

New Connection Security Rule Wizard window, the three steps to set the Preshared key

We can specify one or more specific protocols to be encapsulated inside IPsec, here we encapsulate everything:

We can restrict the rule to specific profiles:

Finally specify the Name of the rule:

Do a ping to the server and check security association is present:

Windows Defender Firewall with Advanced Security console inside the security associations menu

Improve Security

From here, everything should work and connections are already encrypted. Here, we'll look at how to further enhance security by choosing more robust IPsec protocols thant those offered by default. All command listed below must be entered on both client and server.

IKEv2

Enable IKEv2 mode:

PS C:\> Set-NetIPsecRule -DisplayName "IPsec" -KeyModule IKEv2 -ForwardPathLifetime 120

Enforce Security by Changing Protocols with Graphical User Interface (GUI)

From the Windows Defender Firewall with Advanced Security console, right-click on Windows Defender Firewall with Advanced Security, then select Properties:

Windows Defender Firewall with Advanced Security console

In the IPsec Settings tab, click Customize…:

Windows Defender Firewall with Advanced Security, properties window

Select Advanced and click Customize…:

Windows Defender Firewall with Advanced Security, Customize IPsec Defaults window

Main Mode

First, Remove existing entries, then enable Diffie-Hellman and finally click Add…:

Windows Defender Firewall with Advanced Security, Customize Advanced Key Exchange Settings

Change default algorithms:

Windows Defender Firewall with Advanced Security, Add Security Method window

Quick Mode

Enable Require encryption, then Remove the predefined algorithms and finally click Add…:

Windows Defender Firewall with Advanced Security, Customize Data Protection Settings window

Select ESP, then modify algorithms:

Windows Defender Firewall with Advanced Security, Add Integrity and Encryption Algorithms window

Enforce Quick Mode with PowerShell

Change protocols:

PS C:\> $proposal1 = (New-NetIPsecQuickModeCryptoProposal -Encapsulation ESP -Encryption AES256 -ESPHash SHA256) PS C:\> $mMCryptoSet=(New-NetIPsecQuickModeCryptoSet -DisplayName "Quick Mode Rule" -Proposal $proposal1) PS C:\> Set-NetIPsecRule -DisplayName IPsec -QuickModeCryptoSet $mMCryptoSet.Name

Some Commands

Gets IPsec properties from IPsec rule:

PS C:\> Get-NetIPsecRule -DisplayName IPsec

Remove IPsec rule named IPsec:

PS C:\> Remove-NetIPsecRule -DisplayName IPsec

Gets the IPsec main mode properties for the IPsecMain rule:

PS C:\> Get-NetIPsecMainModeRule -DisplayName IPsecMain

Remove IPsec main profile:

PS C:\> Remove-NetIPsecMainModeRule -Name "{XXX-XXX-XXX}"

Gets active Main Mode SA (security association):

PS C:\> Get-NetIPsecMainModeSA

Gets active Quick Mode SA (security association):

PS C:\> Get-NetIPsecQuickModeSA

Computing

Music

Misc

Guide to Establishing an IPsec Transport Mode VPN with X509 authentication Between Two Windows Computers

Network Architecture

Windows Server

PKI

Certificate for the Server

Client Certificate

Check

IPsec Configuration

Graphical User Interface (GUI)

PowerShell

FireWall Configuration

Graphical User Interface (GUI)

PowerShell

Windows 11

Importing the Certificate

IPsec Rule

Improve Security

IKEv2

Enforce Security by Changing Protocols with Graphical User Interface (GUI)

Main Mode

Quick Mode

Enforce Quick Mode with PowerShell

Some Commands