Setting up a Virtual Private Network (VPN) can be tedious. There are multiple phases involved and numerous parameters to be considered. When things don’t go as planned, having the right tools and commands to diagnose and resolve issues is essential. This guide will walk you through useful commands to help troubleshoot new VPN tunnels effectively.
Verifying VPN Phases
When establishing a VPN tunnel, the communication process is split into two main phases: Phase 1 and Phase 2. These phases are integral in setting up a secure channel for data transmission.
Phase 1: IKE Phase
In Phase 1, also known as the IKE Phase, the two endpoints of the VPN tunnel establish a secure channel to communicate securely. This is done using the Internet Key Exchange (IKE) protocol. The main objectives of
Phase 1 is to:
- Authenticate the identities of the two VPN gateways or endpoints.
- Negotiate the security parameters that will be used to protect the communications.
- Generate the encryption and authentication keys.
This phase is crucial since it forms the foundation for the secure communication channel on which the data will be transmitted.
Note about IKE:
IKEv1 and IKEv2 are two Internet Key Exchange protocol versions used to establish IPsec VPNs.
- IKEv1: It is the first version of the Internet Key Exchange protocol. It is a part of the IPsec protocol suite and is used to set up security associations and keying materials for IPsec. IKEv1 consists of two sub-protocols, ISAKMP (Internet Security Association and Key Management Protocol) and OAKLEY.
- IKEv2: This is an improved version of IKE and is also used for setting up VPNs and securing IP communications. It has several advantages over IKEv1, such as a simplified SA negotiation, improved performance, and support for NAT traversal and EAP authentication. IKEv2 is also less bandwidth-intensive and generally considered more secure.
Phase 2: IPSec Phase
In Phase 2, the endpoints use the secure channel established in Phase 1 to negotiate the security associations (SAs) for the data transmission. This is done using the IPSec protocol. The main objectives of Phase 2 are to:
- Negotiate the encryption and authentication methods to be used for the data.
- Generate the keys for encrypting the data.
- Periodically refresh the encryption keys and algorithms during the communication.
In essence, while Phase 1 sets up the secure channel, Phase 2 deals with the secure transmission of the actual data between the two endpoints.
Verification Phase 1: IKE
show crypto isakmp sa detail | be {Peer IP}
Verification Phase 2: IPsec
show crypto ipsec sa peer {Peer IP}
Verify Phase 1 & 2:
show vpn-sessiondb detail l2l filter ipaddress {Peer IP}
Debug IKE/IPsec for v1 and v2:
v1:
debug crypto condition peer 107.180.50.236 debug crypto ikev1 127 debug crypto ipsec 127
v2:
debug crypto condition peer 107.180.50.236 debug crypto ikev2 protocol 127 debug crypto ikev2 platform 127
NOTE:
I’m specifically looking for a peer in the first command. This way you only see debugs for that peer.
Verify Tunnel is up:
v1:
show crypto ikev1 sa
v2:
show crypto ikev2 sa
Verify traffic is flowing with the peer IP Address from the above command:
show crypto ipsec sa peer {PEER_IP_ADDRESS}
Look at “pkts encaps“, pkts encrypt“, “pkts decaps“, and “pkts decrypt“.
Run a Capture or a Trace:
Packet Capture and Packet Tracer are network diagnostic tools for troubleshooting and analyzing network traffic. Each tool serves a different purpose and is used in different scenarios.
Packet Capture:
- Purpose: Packet capture is the process of intercepting and logging traffic that passes over a digital network. Its primary use is for network troubleshooting, analysis, software and communications protocol development, and monitoring network performance.
- Functionality: Packet capture tools intercept the packets on the network and store them in a file, allowing you to analyze the actual data being transmitted over the network.
- Output: You get a capture file which can be analyzed using various tools (such as Wireshark). This file will contain the actual packets that were transmitted or received by the network interface.
- Usage Scenarios: Packet captures are invaluable for deep-diving into network issues, understanding the behavior of network protocols, monitoring for security breaches, compliance, etc.
- Example Tools: Wireshark, tcpdump, and various vendor-specific tools.
Packet Tracer (In Cisco context):
- Purpose: Cisco’s Packet Tracer is a network simulation tool that allows users to create network topologies and emulate how data would flow in the network. It is primarily an educational tool used for learning networking and preparing for Cisco certifications.
- Functionality: Packet Tracer allows users to design networks with routers, switches, and other devices and visually see how packets would traverse the network. It can also refer to a diagnostic tool available on Cisco ASA devices that allows administrators to simulate how the firewall will handle a packet.
- Output: In the case of Cisco’s educational Packet Tracer, you get a visual representation of how your packet traverses through the network, what decisions are being made at each point, etc. In the case of the diagnostic tool on Cisco ASA devices, you get a textual output showing how a packet would be handled.
- Usage Scenarios: Packet Tracer is primarily used for educational purposes and for individuals studying for Cisco certifications. The diagnostic tool version of Packet Tracer on Cisco ASA devices is used to predict how the device will handle packets in real-time, which helps troubleshoot and verify configurations.
- Example Tools: Cisco Packet Tracer (educational tool), `packet-tracer` command on Cisco ASA devices (diagnostic tool).
In summary, Packet Capture is about capturing real traffic for detailed analysis. At the same time, Packet Tracer can refer to a network simulation tool for educational purposes or a diagnostic tool to trace the path of a packet through the security appliance.
Packet Capture:
Use the Inside interface for a capture:
capture CORDERO interface INSIDE match ip any host 8.8.8.8 capture CORDERO interface INSIDE match ip host 8.8.8.8 any show capture CORDERO
Use the Outside interface:
capture CORDERO interface OUTSIDE match ip any host 8.8.8.8 capture CORDERO interface OUTSIDE match ip host 8.8.8.8 any show capture CORDERO
Flags are some combination of:
S (SYN)
F (FIN)
P (PUSH)
R (RST)
W (ECN CWR)
E (ECN-Echo)
single `.’ (no flags)
There are times where you will need to run a capture on the “Accelerated Security Path“. ASP within Cisco ASA has two main components:
Session Management Path:
The Session Management Path (SMP) is responsible for processing the first packet of a new connection. It performs various functions such as Access Control List (ACL) checks, route lookups, Network Address Translation (NAT) building, and establishing sessions. It is also responsible for creating entries in the state table for the connection.
- Performing the ACL checks: This is likely part of the session management path, as ACLs (Access Control Lists) can filter traffic and define what should be allowed or denied.
- Performing route lookups: Yes, figuring out where to forward the packet is part of session management.
- Building NAT / XLAT Translations: Creating NAT (Network Address Translation) entries is part of session management.
- Establishing sessions for the Fast Path: This is crucial for the session management path, enabling the fast path to function efficiently.
The Fast Path:
The Fast Path is responsible for processing subsequent packets in a connection. Once the Session Management Path has processed the first packet of a connection and made the necessary entries in the state table, the Fast Path can process the remaining packets more efficiently. This is because it doesn’t need to perform the same level of inspection and processing as the SMP. It performs IP checksums, session lookups, and TCP sequence number checks and uses NAT translations.
- Performing IP checksums: This is important for ensuring the integrity of packet headers.
- Performing session lookup: The fast path must look up session information to forward traffic efficiently.
- Performing TCP sequence number checks: This is important for ensuring that the packets are in the correct order and no packets are missing.
- Using NAT / XLAT translations based on existing Session Management: NAT translations that have been established in the session management path are used in the fast path.
- Performing Layer 3 and 4 header checks: This ensures the packet headers are well-formed and correct before forwarding them.
Running an ASP Capture:
clear cap /all :clear all captures capture asp type asp-drop all circular-buffer :run a capture on ALL asp options sh cap asp | in {ipaddress}.{port}
Packet Tracer
Syntax:
packet-tracer input ifc_name tcp [SRC_HOST] [SRC_PORT] [DST_HOST] [DST_PORT]
For example, below we are looking at RDP traffic.
TPA-FW-A# packet-tracer input INSIDE tcp 172.16.1.5 1024 4.2.2.2 5589 !!! output truncated Phase: 2 Type: ACCESS-LIST Subtype: log Result: DROP <---- ASA Dropped the traffic Config: access-group INSIDE_IN in interface INSIDE access-list INSIDE_IN extended deny ip any4 any4 log <---- This rule denied the traffic Additional Information: Result: input-interface: INSIDE input-status: up input-line-status: up output-interface: OUTSIDE output-status: up output-line-status: up Action: drop Drop-reason: (acl-drop) Flow is denied by configured rule <----Reason for Drop
Verify what Policy is being used:
“show service-policy” is a great command to see which policy is applied to any given flow.
CORDERO-ASA1# show service-policy flow tcp host 192.168.5.100 host 10.100.20.50 eq 80
Global policy:
Service-policy: global_policy
Class-map: web-class
Match: access-list WEB_TRAFFIC_ACL
Access rule: permit tcp any any eq www
Action:
Input flow: inspect http
Class-map: class-default
Match: any
Action:
Look at the ACTIVE ASA Connections
“show connection” is a great troubleshooting command which displays the ACTIVE ASA connection table. All traffic that passes through the ASA will create a connection.
Quick Reference:
UIO = Outbound Connection
UIOB = Inbound Connection
Flags:
A – awaiting inside ACK to SYN,
a – awaiting outside ACK to SYN,
B – initial SYN from outside,
b – TCP state-bypass or nailed,
C – CTIQBE media,
D – DNS, d – dump,
E – outside back connection,
F – outside FIN,
f – inside FIN,
G – group,
g – MGCP,
H – H.323,
h – H.225.0,
I – inbound data,
i – incomplete,
J – GTP,
j – GTP data,
K – GTP t3-response
k – Skinny media,
M – SMTP data,
m – SIP media,
n – GUP
O – outbound data,
P – inside back connection,
p – Phone-proxy TFTP connection,
q – SQL*Net data,
R – outside acknowledged FIN,
R – UDP SUNRPC,
r – inside acknowledged FIN,
S – awaiting inside SYN,
s – awaiting outside SYN,
T – SIP,
t – SIP transient,
U – up,
V – VPN orphan,
W – WAAS,
X – inspected by service module
Examples:
INBOUND CONNECTION:
TCP Outside:172.30.200.24/50323 inside:172.16.200.1/6061,
flags UIOB, idle 27s, uptime 1D5h, timeout 1h0m, bytes 20155
U = the connection UP
I = there’s INBOUND data
O = there’s OUTBOUND data
B = initiated from the outside
OUTBOUND CONNECTION:
TCP outside:10.255.6.22/1433 inside:172.16.200.10/51033,
flags UIO, idle 24s, uptime 5m25s, timeout 1h0m, bytes 1982
U = the connection UP
I = there’s INBOUND data
O = there’s OUTBOUND data
INCOMPLETE:
TCP Outside:172.30.200.24/57630 inside:10.65.10.100/0,
flags Ti, idle 5m49s, uptime 5m49s, timeout -, bytes 0
T = this is SIP traffic
i = incomplete
Look at the order of your NATs
Understanding how NAT policies are processed when configuring Network Address Translation (NAT) on Cisco ASA devices is crucial. This post will dive deeply into how NAT policies are evaluated and provide recommendations for optimizing your configurations.
NAT Processing Order
NATs on Cisco ASA are processed based on the “First Match” principle. This means that when a packet arrives at the firewall, it will traverse the NAT rules from the top to the bottom. The first rule that matches the packet’s attributes is applied, and the rest are ignored. Knowing this, the order in which you configure your NAT rules is paramount.
Order of Operations:
1. Manual NAT Policies (Before Auto NAT)
2. Auto NAT Policies
3. Manual NAT Policies (After Auto NAT)
Manual NAT Policies:
Manual NAT policies allow you to have granular control over the translation process. They are beneficial when you need to implement complex NAT scenarios. Manual NAT policies that are defined before Auto NAT policies will be evaluated first.
Auto NAT Policies:
Auto NAT, or Object NAT, simplifies the NAT configuration by binding NAT rules to network objects. They are easier to configure and manage but might not be as flexible as Manual NAT policies.
Manual NAT Policies (After Auto NAT):
Finally, if you have Manual NAT policies that you want to evaluate after Auto NAT policies, you can configure them as such. This can be useful in cases where general rules are defined in Auto NAT and specific exceptions are handled in Manual NAT.
Auto NAT Policies Sub-Order:
Even within Auto NAT policies, there is an order in which they are evaluated:
1. Static NAT – Longest Prefix Match to Shortest Prefix Match
2. Dynamic NAT – Longest Prefix Match to Shortest Prefix Match
Static NAT policies are evaluated first, with the longest prefix match taking precedence. After all Static NAT rules are evaluated, the ASA moves on to Dynamic NAT rules, again considering the longest prefix matches first.
Recommendations:
1. Use Manual NAT for Complexity: When complex NAT requirements need granular control, use Manual NAT.
2. Leverage Auto NAT for Simplicity: For general NAT requirements without complexity, leverage Auto NAT for easier configuration and management.
3. Be Mindful of Prefix Lengths: Within Auto NAT, remember that rules with longer prefix lengths will take precedence. So, if you want a specific subnet to be translated using a different rule than the rest of the traffic, ensure it has a longer prefix length.
4. Review and Reorder: Regularly review your NAT policies and ensure they are ordered logically. This is key to preventing unintended consequences and ensuring the NAT process is as efficient as possible.
5. Document your NAT Rules: For easier maintenance and troubleshooting, document why each NAT rule is in place and what it’s designed to accomplish.
Conclusion
The order in which NAT rules are processed plays a significant role in how packets are handled. By understanding the NAT processing order and carefully planning your NAT policies, you can optimize your network’s performance and security. Whether your environment calls for Manual NAT, Auto NAT, or a combination of both, being mindful of the order in which rules are evaluated is critical to effective NAT configuration on Cisco ASA devices.
Verification:
show run nat show nat
Look at order of ikev1 crypto’s since the ASA will go in order:
In IKE (both v1 and v2), policies define how the VPN devices negotiate security parameters for the VPN tunnel. Each policy contains a set of security parameters such as encryption algorithm, hash algorithm, authentication method, and Diffie-Hellman group.
The devices at both ends of a VPN tunnel must have at least one matching IKE policy in order to establish a secure connection. If there are multiple policies defined, the device lists them in order of priority. When the negotiation starts, the device offers its highest priority policy. If the remote endpoint supports it, they use that policy. If not, it offers its next policy in the list.
v1:
sh run crypto ikev1 crypto ikev1 policy 10 authentication pre-share encryption aes-256 hash sha group 2 lifetime 86400 crypto ikev1 policy 20 authentication pre-share encryption aes-256 hash sha group 2 lifetime 28800
v2:
sh run crypto ikev2 crypto ikev2 policy 1 encryption aes-256 integrity sha group 5 2 prf sha lifetime seconds 86400 crypto ikev2 policy 10 encryption aes-192 integrity sha group 5 2 prf sha lifetime seconds 86400 crypto ikev2 policy 20 encryption aes integrity sha group 5 2 prf sha lifetime seconds 86400 crypto ikev2 policy 30 encryption 3des integrity sha group 5 2 prf sha lifetime seconds 86400
Verify the Lifetimes
Lifetime in the context of VPN refers to the duration for which the security associations (SAs) remain in effect. Once the lifetime expires, the SAs are regenerated. This is essential for security because it prevents attackers from having enough time to break the encryption.
There are two aspects of lifetime:
- Time-Based: This is the duration for which the SA remains in effect before it’s automatically renegotiated.
- Traffic-Based: This refers to the amount of data that can be transferred using a single SA before it must be renegotiated.
Having shorter lifetimes makes the VPN connection more secure but might impact performance due to more frequent renegotiations. Conversely, longer lifetimes improve performance but reduce security.
In practice, lifetimes are often set as a trade-off between security and performance based on the specific requirements of the VPN implementation.
The lifetimes do not need to be the same on both sides of a VPN tunnel, but it is generally recommended for optimal stability and performance. When the lifetimes are different on each side, the tunnel will renegotiate the security associations (SAs) based on the shorter of the two lifetimes.
Here’s what happens in more detail:
- If Phase 1 Lifetime is Different: When the side with the shorter Phase 1 lifetime reaches its expiration, it will initiate a rekey for the IKE SA. The other side will still consider the old SA to be valid until its own lifetime expires, but will accept the new SA from the first side.
- If Phase 2 Lifetime is Different: The side with the shorter Phase 2 lifetime will initiate the rekey for the IPSec SA. Again, the other side will accept the new SA even though it hasn’t reached its own lifetime expiration.
When the lifetimes are significantly different, this can lead to more frequent renegotiations, which might affect the performance of the VPN tunnel. It’s generally a good practice to keep the lifetimes the same or at least similar to avoid unnecessary renegotiations and to ensure the stability of the VPN connection.
Verification:
show isakmp sa detail
show crypto ikev1 sa detail
1 IKE Peer: 139.177.229.3
Type : L2L Role : initiator
Rekey : no State : MM_ACTIVE
Encrypt : aes-256 Hash : SHA
Auth : preshared Lifetime: 28800
Lifetime Remaining: 1873
2 IKE Peer: 72.32.69.25
Type : L2L Role : responder
Rekey : no State : MM_ACTIVE
Encrypt : aes-256 Hash : SHA
Auth : preshared Lifetime: 28800
Lifetime Remaining: 11931
Default values to keep in mind. When using the CLI, remember to add “all” to the commands:
vpn-idle-timeout 30: This means that the VPN session will be terminated if there is no activity for 30 minutes.
sh run all group-policy
group-policy DfltGrpPolicy attributes
vpn-idle-timeout 30
vpn-idle-timeout alert-interval 1
vpn-session-timeout none
vpn-session-timeout alert-interval 1
crypto ipsec security-association lifetime seconds 28800: Specifies the lifetime of the security association in seconds. 28,800 seconds equals 8 hours, which is a common default.
crypto ipsec security-association lifetime kilobytes 4608000: Specifies the lifetime of the security association in kilobytes of data transferred. 4,608,000 kilobytes is also a common default.
sh run all | inc ipsec security-association crypto ipsec security-association lifetime seconds 28800 crypto ipsec security-association lifetime kilobytes 4608000
isakmp keepalive threshold 10 retry 2: This is related to Dead Peer Detection (DPD). The device will send a keepalive message every 10 seconds and retry 2 times if no response is received.
sh run all tunnel-group
isakmp keepalive threshold 10 retry 2