Guide to Establishing an IPsec Transport Mode VPN with X509 authentication Between Two Windows Computers

Network Architecture

For the purposes of this article, we will maintain a simple architecture. We will establish a secure connection using an IPsec VPN with x509 authentication between a computer running Windows 11 and a server running Windows Server 2022. Both computers will be on the same local network.

Windows Server

PKI

As discussed previously, we will use X.509 certificates to establish an IPsec connection between our server and client. To do this, we need to create a PKI: a Root CA which will sign the certificates, and two certificates, one for each of our two hosts.

All the commands below will be executed from a PowerShell console with administrator rights.

Certificate for the Server

• Create the Root CA, which will enable us to sign server and client certificates:

PS C:\> $rootCert = New-SelfSignedCertificate -CertStoreLocation Cert:\LocalMachine\My -DnsName RootCA -KeyUsage CertSign,CRLSign,DigitalSignature -NotAfter (Get-Date).AddYears(10)

• Create server certificate, and sign it with RootCA (Note: we can change "ServerName" to the actual server name.):

PS C:\> $serverCert = New-SelfSignedCertificate -CertStoreLocation Cert:\LocalMachine\My -DnsName ServerName -Signer $rootCert -NotAfter (Get-Date).AddYears(5)

• Retrieve server certificate information to the $serverCert variable:

PS C:\> $serverCert = Get-ChildItem -Path Cert:\LocalMachine\My | Where-Object { $_.Subject -eq "CN=ServerName" }

Client Certificate

• Create a client certificate and sign it with RootCA (Note: "W11" can be replaced with the actual client name.):

PS C:\> $cert = New-SelfSignedCertificate -DnsName "W11" -CertStoreLocation "cert:\LocalMachine\My" -Signer $rootCert -KeyExportPolicy Exportable -NotAfter (Get-Date).AddYears(5)

• Set a password to protect the pfx file, which will be used to store the client's certificate with the private key:

PS C:\> $password = ConvertTo-SecureString -String "PassPass" -Force -AsPlainText

• Export the client certificate to a pfx container, exemple here to C:\W11.pfx:

PS C:\> Export-PfxCertificate -Cert $cert -FilePath "C:\W11.pfx" -Password $password

Check

• Open the Local Computer Certificates Manager console:

• From the Personal > Certificates folder, check that the previously created certificates are present:

IPsec Configuration

Graphical User Interface (GUI)

• Open the Windows Defender Firewall with Advanced Security console:

• Do a right click on Connection Security Rules and select New Rule…:

• Select Custom and click Next:

• In the Which computers are in Enpoint 1 section, select These IP addresses and click on Add…:

• Here we add the Host or Network with which we need have to establish an IPsec connection to communicate:

• Do the same with Enpoint 2 entering the server's IP address then click Next:

• For maximum security and impose the use of IPsec select Require authentication for inbound and outbound connections:

• As we'll be using a certificate as our Authentication Method in this example, click on Customize…:

• Follow this steps to set the Certificate:

• We can specify one or more specific protocols to be encapsulated inside IPsec, here we encapsulate everything:

• We can restrict the rule to specific profiles:

• Finally specify the Name of the rule:

PowerShell

• We can create the same IPsec Rule as the one configured just above, in a three PowerShell commands line:

PS C:\> $cert1Proposal = New-NetIPsecAuthProposal -Machine -Cert -Authority "CN=RootCA" -AuthorityType Root

PS C:\> $certAuthSet = New-NetIPsecPhase1AuthSet -DisplayName "Computer Certificate Auth Set" -Proposal $cert1Proposal

PS C:\> New-NetIPsecRule -DisplayName "IPsec" -Name "IPsec" -Mode Transport -InboundSecurity Require -OutboundSecurity Require -LocalAddress 192.168.1.200 -RemoteAddress 192.168.1.30 -Enable True -Phase1AuthSet $certAuthSet.Name

FireWall Configuration

We will create a Firewall rule to allow encrypted flows.

Graphical User Interface (GUI)

• Do a right click on Inbound Security Rules and select New Rule…:

• Select Custom and click Next:

• Choose All programs and click on Next:

• We can specify one or more specific protocols to be matched by the rule, here we will match Any protocols:

• Restrict rule application to defined ip addresses concerned:

• In the Action step, configure the Require the connections to be encrypted condition:

• We can limit the application of the rule to specific users, which will not be used for our purposes:

• We can do the same to limit the rule to specific computers, which won't be used in this example either:

• We can restrict the rule to specific profiles:

• Finally specify the Name of the rule:

PowerShell

• We can create the same Firewall Rule sas the one configured just above, in a single PowerShell command line:

We've now finished configuring the server. Now it's time to move on to client configuration.

Importing the Certificate

• First, copy the W11.pfx file previously generated to the Windows 11 system and double-click on it:

• Select Local Machine then click Next:

• Click Next:

• Type the password previously set when exporting certificates to pfx and click Next:

• Choose “Automatically select the certificate store” and click Next:

IPsec Rule

I'm going to describe them all, but the steps are pretty much the same as those we followed for the server.

• Open the Windows Defender Firewall with Advanced Security console:

• Do a right click on Connection Security Rules and select New Rule…:

• Select Custom and click Next:

• In the Which computers are in Enpoint 1 section, select These IP addresses and click on Add…:

• Here we add our client's IP address:

• Do the same with Enpoint 2 entering the server's IP address then click Next:

• For maximum security and impose the use of IPsec select Require authentication for inbound and outbound connections:

• As we'll be using a preshared key as our Authentication Method in this example, click on Customize…:

• Follow this steps to set the Certificate:

• We can specify one or more specific protocols to be encapsulated inside IPsec, here we encapsulate everything:

• We can restrict the rule to specific profiles:

• Finally specify the Name of the rule:

• Do a ping to the server and check security association is present: