rss logo

How to Configure an IPsec Transport Mode Tunnel Between Two Windows Machines

This article explains how to set up a secure IPsec VPN between Microsoft Windows hosts on the same LAN, in order to significantly enhance network security. With this configuration, all communications between machines will be fully encrypted, without exception.

Network Architecture

For the purposes of this article, we will keep the network architecture as simple as possible. A secure IPsec VPN connection using a pre-shared key (PSK) will be established between a computer running Windows 11 and a server running Windows Server 2022, both located on the same local network.

Network diagram showing an IPsec Transport Mode tunnel between a Windows 11 workstation and a Windows Server 2022 host on the same LAN
Windows Server Logo

Windows Server

IPsec Configuration

Graphical User Interface (GUI)

  • Open the Windows Defender Firewall with Advanced Security console:
Windows Run dialog with wf.msc entered to open the Windows Defender Firewall with Advanced Security
  • Do a right click on Connection Security Rules and select New Rule…:
Creating a new connection security rule in Windows Defender Firewall for IPsec configuration
  • Select Custom and click Next:
Selecting the Custom rule type in the Windows Firewall security rule wizard for IPsec setup
  • In the Which computers are in Endpoint 1 section, select These IP addresses and click on Add…:
Specifying IP addresses for Endpoint 1 in the IPsec connection rule setup in Windows Firewall
  • Here we add the Host or Network with which we need to establish an IPsec connection to communicate:
Entering a specific IP address to define an IPsec rule endpoint in Windows Firewall
  • Do the same with Endpoint 2 entering the server's IP address then click Next:
Defining IP address and subnet ranges as endpoints for a Windows IPsec security rule
  • For maximum security and impose the use of IPsec select Require authentication for inbound and outbound connections:
Windows IPsec rule configuration requiring authentication for both inbound and outbound connections
  • As we'll be using a preshared key as our Authentication Method in this example, click on Customize…:
IPsec rule configuration window showing advanced authentication method selection in Windows
  • Follow these steps to set the Preshared key:
Windows IPsec advanced authentication setup with preshared key entered as first method
  • We can specify one or more specific protocols to be encapsulated inside IPsec, here we encapsulate everything:
Windows IPsec connection rule setup selecting protocol type and port ranges
  • We can restrict the rule to specific profiles:
IPsec rule profile selection screen with domain, private, and public network types enabled
  • Finally specify the Name of the rule:
Final step of the IPsec rule creation in Windows Firewall with rule name set to 'IPsec'

PowerShell

  • We can create the same IPsec Rule as the one configured just above, in a three PowerShell commands line:
PS C:\> $PSKAuthProp = New-NetIPsecAuthProposal -Machine -PreSharedKey "IPsuperSECRET"
PS C:\> $PSKAuthSet = New-NetIPsecPhase1AuthSet -DisplayName "PSK auth" -Proposal $PSKAuthProp
PS C:\> New-NetIPsecRule -DisplayName "IPsec" -Name "IPsec" -Mode Transport -InboundSecurity Require -OutboundSecurity Require -LocalAddress 192.168.1.200 -RemoteAddress 192.168.1.30 -Enable True -Phase1AuthSet $PSKAuthSet.Name

FireWall Configuration

We will create a Firewall Rule to allow encrypted flows.

Graphical User Interface (GUI)

  • Do a right click on Inbound Security Rules and select New Rule…:
Windows Defender Firewall interface showing how to create a new inbound rule
  • Select Custom and click Next:
Windows Inbound Rule Wizard showing the selection of a custom rule type
  • Choose All programs and click on Next:
Firewall rule configuration in Windows set to apply to all programs on the system
  • We can specify one or more specific protocols to be matched by the rule, here we will match Any protocols:
Windows Firewall rule configured to allow any protocol and any port
  • Restrict rule application to defined ip addresses concerned:
Scope settings in Windows Firewall rule specifying local and remote IP addresses for IPsec connection
  • In the Action step, configure the Require the connections to be encrypted condition:
Windows Firewall rule configuration set to require encrypted IPsec connections for secure communications
  • We can limit the application of the rule to specific users, which will not be used for our purposes:
Windows Firewall rule wizard showing user-based connection authorization options for IPsec rules
  • We can do the same to limit the rule to specific computers, which won't be used in this example either:
Windows Firewall wizard displaying computer-based connection restrictions for IPsec rule setup
  • We can restrict the rule to specific profiles:
Firewall rule wizard in Windows showing domain, private, and public network profiles selected for IPsec policy application
  • Finally specify the Name of the rule:
Final step of the Windows Firewall wizard showing an IPsec inbound rule named 'IPsec ALLOW' ready to be created

PowerShell

  • We can create the same Firewall Rule as the one configured just above, in a single PowerShell command line:
PS C:\> New-NetFirewallRule -DisplayName "IPsec ALLOW" -Direction Inbound -Enabled True -Action Allow -LocalAddress 192.168.1.200 -RemoteAddress 192.168.1.0/24 -Protocol Any -Encryption Dynamic -Authentication Required

We've now finished configuring the server. Now it's time to move on to client configuration.

Windows 11

Windows 11 Logo

I'm going to describe them all, but the steps are pretty much the same as those we followed for the server.

  • Open the Windows Defender Firewall with Advanced Security console:
Windows Run dialog with 'wf.msc' entered to open the Windows Defender Firewall interface
  • Right-click on Connection Security Rules and select New Rule…:
Windows Defender Firewall showing context menu to create a new connection security rule under Connection Security Rules
  • Select Custom and click Next:
Windows Firewall wizard screen showing selection of custom rule type for IPsec configuration
  • In the Which computers are in Endpoint 1 section, select These IP addresses and click on Add…:
Windows IPsec configuration showing how to specify IP addresses for secure endpoint connections
  • Here we add our client's IP address:
IP Address window, with network 192.168.1.30 filled in
  • Do the same with Endpoint 2 entering the server's IP address then click Next:
IPsec configuration wizard showing selected IP addresses for Endpoint 1 and Endpoint 2 on Windows
  • For maximum security and impose the use of IPsec select Require authentication for inbound and outbound connections:
Windows IPsec wizard showing option to require authentication for both inbound and outbound connections
  • As we'll be using a preshared key as our Authentication Method in this example, click on Customize…:
Windows IPsec wizard with Advanced authentication method selected and Customize option highlighted
  • Follow this steps to set the Preshared key:
Advanced IPsec authentication settings in Windows with Pre-shared Key (PSK) configuration and confirmation
  • We can specify one or more specific protocols to be encapsulated inside IPsec, here we encapsulate everything:
Windows IPsec connection rule setup selecting protocol type and port ranges
  • We can restrict the rule to specific profiles:
Windows Firewall IPsec rule wizard showing selected network profiles: Domain, Private, and Public
  • Finally specify the Name of the rule:
Final step of IPsec rule creation in Windows Firewall showing rule name 'IPsec to 192.168.1.200'
  • Do a ping to the server and check security association is present:
Windows Firewall showing IPsec Main Mode association with preshared key and AES-CBC 128 encryption between 192.168.1.30 and 192.168.1.200

Improve Security

From here, everything should work and connections are already encrypted. Here, we'll look at how to further enhance security by choosing more robust IPsec protocols than those offered by default. All command listed below must be entered on both client and server.

IKEv2

  • Enable IKEv2 mode:
PS C:\> Set-NetIPsecRule -DisplayName "IPsec" -KeyModule IKEv2 -ForwardPathLifetime 120

Enforce Security by Changing Protocols with Graphical User Interface (GUI)

  • From the Windows Defender Firewall with Advanced Security console, right-click on Windows Defender Firewall with Advanced Security, then select Properties:
Windows Defender Firewall interface showing context menu with Properties option selected
  • In the IPsec Settings tab, click Customize…:
Windows Defender Firewall properties window with IPsec Settings tab open and Customize button highlighted
  • Select Advanced and click Customize…:
Customize IPsec Defaults window showing advanced options enabled for key exchange and data protection with Customize buttons highlighted

Main Mode

  • First, Remove existing entries, then enable Diffie-Hellman and finally click Add…:
Advanced IPsec key exchange settings showing SHA-1 integrity, AES-CBC 128 and 3DES encryption with Diffie-Hellman Group 2, and options to add or remove security methods
  • Change default algorithms:
Security method configuration in Windows IPsec with SHA-384 integrity, AES-CBC 256 encryption, and Elliptic Curve Diffie-Hellman P-384 key exchange

Quick Mode

  • Enable Require encryption, then Remove the predefined algorithms and finally click Add…:
Customize Data Protection Settings window in Windows IPsec with selected encryption requirement and ESP algorithm using SHA-1 and AES-CBC
  • Select ESP, then modify algorithms:
Configuration of ESP encryption and integrity algorithms in Windows IPsec with AES-GCM 256 selected for both parameters

Enforce Quick Mode with PowerShell

  • Change protocols:
PS C:\> $proposal1 = (New-NetIPsecQuickModeCryptoProposal -Encapsulation ESP -Encryption AES256 -ESPHash SHA256)
PS C:\> $mMCryptoSet=(New-NetIPsecQuickModeCryptoSet -DisplayName "Quick Mode Rule" -Proposal $proposal1)
PS C:\> Set-NetIPsecRule -DisplayName IPsec -QuickModeCryptoSet $mMCryptoSet.Name

Some Commands

  • Gets IPsec properties from IPsec rule:
PS C:\> Get-NetIPsecRule -DisplayName IPsec
  • Remove IPsec rule named IPsec:
PS C:\> Remove-NetIPsecRule -DisplayName IPsec
  • Gets the IPsec main mode properties for the IPsecMain rule:
PS C:\> Get-NetIPsecMainModeRule -DisplayName IPsecMain
  • Remove IPsec main profile:
PS C:\> Remove-NetIPsecMainModeRule -Name "{XXX-XXX-XXX}"
  • Get active Main Mode SA (security association):
PS C:\> Get-NetIPsecMainModeSA
  • Gets active Quick Mode SA (security association):
PS C:\> Get-NetIPsecQuickModeSA
Creative Commons License
This work is licensed under a Creative Commons Attribution-NonCommercial-ShareAlike 4.0 International License.

Contact :

contact mail address