Overview of IPSec

Security is a key part of any network. Companies need to protect their network from eavesdropping and tampering. Windows provides ACLs (Access Control Lists) which can be used to prevent access to data over the network. However, ACLs don’t protect data travelling over the network cable.

Use the buttons below to navigate through the lesson

Network data can easily be recorded, viewed or tampered with. It can include username and passwords and other sensitive information. Even if a file is encrypted on the machine, it has to be unencrypted to be sent across the network. Most companies can’t afford this sort of security hole. Logon data can be recorded and replayed by a hacker to break into the network. Data can also be modified and placed back on the network.  To overcome this problem, Windows Server 2008 provides Internet Protocol Security Extensions (IPsec). IPsec is a set of extensions to TCP/IP used to protect network data.

IPSec has two separate features which work together: authentication and encryption. They can be used together or separately to protect a network.

Authentication protects the data on the network from tampering. This tampering could be a hacker sitting between a client and server and altering the contents of data packets. IPsec uses an authentication header (AH) to digitally sign the entire contents of each packet. This provides the following protection:

Protection against replay attacks: A hacker can capture network packets and save them until a later time. These packets can then be sent again to impersonate a machine. This is known as a replay attack. AH prevents this by including the sender’s signature on all packets.

Protection against tampering. IPsec’s signatures provide data integrity checking. Any data that has been modified is rejected by the receiving machine. IPsec provides mutual authentication which means both communicating sides can verify the other’s identity.

The Authentication Header is used to protect each network packet sent across the network. The AH contains information that can be used to verify the origin and time the packet was sent.

The IP Authentication Header (AH) protocol does not encrypt the data it guards. AH provides basic security with a relatively low overhead. AH ensures that any data exchanged between two systems hasn’t been tampered with and that the data was sent from the originator specified in the IP header. The Authentication Header guarantees the data. However it cannot prevent the data being captured and read.

The Authentication Header guarantees the data. However it cannot prevent the data being captured and read.


IPsec provides encryption using the Encapsulating Security Payload (ESP). ESP is used to encrypt the entire payload of an IPsec packet, rendering it unreadable to anyone other than the intended recipient. ESP only provides confidentiality. Packets might still be recorded and modified. ESP can be used in combination with AH to provide maximum security for the network. ESP only provides confidentiality.

All the data following the ESP header and up to the ESP trailer is encrypted to protect it from unauthorized viewing.

To prevent tampering, an ESP client has to sign the ESP header, payload data and ESP trailer into one unit. ESP is used to encrypt the payload data and the ESP trailer is used to provided confidentiality.

Components of IPSec

The Internet Security Agreement/Key Management Protocol (ISAKMP) provides a way for two computers to agree on security settings and exchange a security key. A Security Association (SA) provides all the information needed for two computers to communicate securely. The SA contains a policy agreement which controls which algorithms and key lengths the two machines will use, e.g. 128-bit RC5 and SHA-1.

Overview of IPSec

IPSec sits on the network layer of the OSI model and encrypts the datagrams. At the data link layer all the addressing information is added. The encrypted datagram is only decrypted at the destination. All the addressing and routing information is outside the encryption therefore intervening network hardware does not have to be IPSec aware.

The Windows 2008 IPSec supports policy-based security. A group policy can be applied to machines to enable IPSec and secure communications. IPSec’s operation is called end-to-end mode (also known as transport mode.) This is because data is protected before it leaves the originating machine and it remains secured until it reaches its destination.  IPSec can also be used to secure traffic passing through other networks (e.g. The Internet). This is known as tunnelling mode. Here a tunnel is created between two computers through the other networks. All data inside this tunnel is encrypted using IPsec. Tunnels are used with virtual private networks (VPNs).

IPSec supports three separate authentication methods:

  1.  Kerberos, which is the default authentication for Windows 2003.
  2. Certificate Services. Machine certificates can be used for authentication.
  3. Pre-shared keys. The pre-shared key is just a word or code that both computers know. The two machines use this password to establish a trust.


Encrypting and decrypting are both processor intensive. This could degrade the application performance of machines. Also, IPSec increases network traffic.

An administrator must consider whether IPSec is required for all machines, the Remote Access server or just machines that transfer sensitive data between each other.